Appears In
The Department of Veterans Affairs (VA) enhanced several IT systems to stay ahead of the unprecedented surge in the demand for video appointments as record numbers of Veterans and their families opted for virtual medical appointments during the COVID-19 pandemic. VA’s Office of Information and Technology (OIT) quintupled the capacity of VA’s primary telehealth system, VA Video Connect, to allow more clinicians and patients to conduct remote appointments simultaneously. Veterans Health Administration clinicians now conduct close to 30,000 telehealth video appointments a day, compared to fewer than 2,500 video appointments each day in January and February.
To ensure that VA Video Connect would continue to operate smoothly, Site Reliability Engineers from the Operations Triage Group within OIT created a robust set of new monitoring dashboards that aggregate data from multiple sources to shed light on telehealth system behavior. These new dashboards allow the IT monitoring team to easily gather disparate data and work with Site Reliability Engineers and other IT engineers to investigate and resolve issues within just hours, rather than days. Alongside a real-time view of call and video quality, the dashboards let the monitoring team see call and conference counts, errors and warnings, and calls by caller location. As of early June, OIT’s monitoring experts have been using this new tool to examine data from 10 different sources and analyze over 72 million events every day.
The unprecedented rapid quintupling of VA Video Connect’s capacity initially introduced a few errors and intermittent system slowdowns; however, the newly stood-up monitoring capabilities immediately alerted IT experts and video engineers to these issues. Working quickly to keep functionality available, OIT was able to use the new monitoring dashboards to isolate and resolve problems and return to a nearly error-free state. The systems have also helped troubleshoot and resolve several other issues, including an issue that was impacting some Veterans’ ability to connect to their appointments.
The Site Reliability Engineers have been hard at work identifying any and all issues related to capacity, which then inform the engineering team about how to adjust the VA Video Connect system’s environment. By leveraging these insights, OIT has reduced connection errors from as many as 1,000 users per day to fewer than 10. Teams also established event thresholds that would trigger alerts as VVC use approached its maximum capacity.
Fortunately, VA has a great deal of experience to draw from to develop and maintain such a robust telehealth dashboard, and a strong relationship with industry partners who specialize in aggregating and integrating data. When the pandemic began to affect Veterans earlier this year, these existing teams pivoted seamlessly to assemble the telehealth monitoring dashboard to respond to emerging needs.
With a unified goal and exceptional work by all teams, VA was able to fast track an authority to operate (ATO) to push the new monitoring system into the cloud, securing it in three weeks instead of the standard six months. This collaboration has had positive real-world impacts on Veterans and the clinicians who provide care, including the ability to predict user needs from system trends, mitigate outages or problems for Veterans, and give VHA’s Office of Connected Care more visibility into the systems it depends on.
Now that OIT has implemented this sophisticated telehealth monitoring solution during the COVID-19 pandemic, VA will expand to many systems across the enterprise to improve not only monitoring practices, but the interactions between them as well.
