Medical record retrieval in minutes, not months.
“Why does it take days, weeks, or months to retrieve my own medical records or to transfer them between providers?”
“Why does the burden of moving records from one provider to another fall on me, the patient?”
Datavant isn’t trying to answer those thorny questions.
Our goal is to eliminate the need to ask them in the first place.
Broadly speaking, the Datavant Switchboard enables data requesters to connect to two types of data: de-identified data and identified data. Connecting identified data through the Switchboard solves the underlying problems for patients retrieving their medical records – problems most people have encountered interacting with the many nodes of healthcare in the US. But the problem of record retrieval doesn’t just exist for patients. It’s a problem for any entity attempting to locate medical records for a specific individual (i.e. other healthcare providers, insurance companies, law firms, etc.).
By way of our merger with Ciox, Datavant has moved into the center of the ecosystem of medical record retrieval. In 2022, we retrieved more than 60 million charts for health systems and insurers across the country. Most of this work was fulfilled through a time-consuming series of manual steps.
In order to retrieve electronic medical records (EMRs), Datavant connects to individual record provider APIs, but health data is fragmented and one API connection into an EMR provider will not provide access to all available patient data. A big step in this process was bringing our FHIR Worker online, but even with FHIR Worker, integrations remain difficult and bespoke, each requiring its own implementation.
Building a system that can facilitate rapid medical record retrieval at scale demands that our Switchboard design be absolutely bottleneck-free. With that in mind, we came to the conclusion that our legacy API retrieval system was too slow, especially when dealing with millions of patient search requests. We decided to redesign our API retrieval system so that it could turn around requests in less than a day with minimal manual intervention, regardless of the size of the request.
To scale up to thousands of providers and 10 times our current chart volume with ease, we outlined three specific goals for the redesign:
In order to retrieve individuals’ records, we must connect directly to EMR vendors, such as Epic, that manage the electronic records for healthcare providers. The entity making the record request (a person, an insurance company, etc.) needs to provide the demographic information for the healthcare provider from which they are requesting records. That is, in a traditional manual record request, the requester needs to know where the record is located so that a human being can either call that location, or walk into that location to retrieve the record. Without that information, the request cannot be fulfilled because the proverbial haystack of records is, so to speak, invisible.
For seamless, on-demand record requests, we should be able to locate any record for any patient anywhere in the US, even if we don’t know where it is located. This is person-based flow, and it’s something that our Convenet product has already achieved in England because of England’s centralized health system and Convenet’s integration with the NHS-Spine.
Our first steps toward realizing a person-based flow system was to set up geolocation matching to find potential healthcare providers and use our embedded API connections for patient search and chart retrievals. Using Lightweight instructions to check the patient’s existence and encounters within the given site, and repeat for all sites within Datavant’s network under the given patient’s geological proximity (based on zip code), we can locate records for individuals without prior knowledge of their location.
In our legacy system, record requests were accepted with our Request & Intake Services, dropped into a Scheduler Queue, then saved to a database by Scheduler On_Message. Scheduler On_Batch, as its name suggests, sent batch-by-batch requests to the corresponding Retrieval Queue. The corresponding FHIR Worker* listened to these requests to retrieve records.
*Sometimes our FHIR Workers are called EHR Workers because not all Electronic Health Record systems use the HL7 FHIR Standard…apologies if this creates any confusion. Trust us, we feel it too.
In this organization, every client (every hospital, EMR provider, etc.) has its own Retrieval Queue and dedicated FHIR Worker. As a result of this duplication within this part of this process, more clients equaled bigger bottlenecks.
After testing, we discovered that our FHIR Workers consumed patient search requests faster than the Scheduler could provide them. FHIR Workers completed the requests in queue and then idled, potentially for hours, waiting for more requests to arrive.
During the first runs of our person-based flow design, when ~500k requests entered the system, the Scheduler On_Batch process became a bottleneck. After 30—45 mins, Scheduler On_Batch had not cycled through all the queues and more retrieval queues slowed this process even further. This meant that we were not fully utilizing the provider API bandwidth and retrieval time was not as fast as the provider API would allow.
The biggest change in system design was to make FHIR Workers/EHR Workers actively pull requests to process rather than have them wait for requests to be pushed to them.
Requests are submitted to the Submission Queue by the Dispatcher, and the API Chase Submitter, which listens to the queue, and submits POST requests to the Controller, an API service that manages the lifecycle of requests sent to the provider API. The Controller populates a Redis Zset with requests and their associated processing priority. FHIR/EHR Workers pull requests to process from this Redis ZSet and, upon completion of the request processing, send update and retrieval results back to the Controller. The Controller then updates the request status and sends a message to the Dispatcher Notification Queue.
Redis Zset is a sorted set that holds a type of key and value.
The keys (called members) are unique, and the values (called scores) are limited to floating-point numbers. ZSETs have the unique property in Redis of being able to be accessed by members (like a HASH), but items can also be accessed by the sorted order and values of the scores. To store requests in Redis Zset, the key/member is the request body JSON string and the value/score is the associated priority of the request. Requests are then sorted by order of priority.
The redesigned system processed significant at-scale loads in minutes that would have taken hours in the legacy system. (Under extreme circumstances, the legacy system could not run on its own and required human intervention to continue running.) Because the system is horizontally scalable, we can multiply this throughput with more pods and provider APIs. In addition to scalability, this redesign also improves efficiency and maintainability as it makes use of only one deployment of the FHIR/EHR Worker to handle API chart retrievals for all of the Datavant’s Identified Switchboard.
This essential component of the Switchboard creates a single request starting point and routes requests where they need to go – whether that be the health system, EHR vendor, or data aggregator – maximizing yield and minimizing turnaround time.
In other words, we’re beginning to chip away at the question,
“Why is it so hard to retrieve…?”
The API System redesign was driven by Sherry Liu, with support from Datavant’s Connection Pod and the IDSB team, including:
Victor Cai
Abhishek Goyal
Micheal Stensby
Aidan Higginbotham
Russell Kirmayer
Nola Gordon
Alec Ruiz-Ramon
Ryan Ma
Sherry Liu
Authored by Sherry Liu and Nicholas DeMaison, with input from Victor Cai.
Sherry Liu has a background in system management and data strategy, and is a software engineer at Datavant. Connect with Sherry on LinkedIn.
Nicholas DeMaison writes for Datavant where he leads talent branding initiatives. Connect with Nick on LinkedIn.
Considering joining the Datavant team? We’re hiring remotely across teams and we would love to speak with any potential new Datavanters who are nice, smart, and get things done, and want to build the future tools for securely connecting health data and improving patient outcomes.
AnalyticsIQ, a marketing data and analytics company, recently adopted Datavant’s state de-identification process to enhance the privacy of its SDOH datasets. By undergoing this privacy analysis prior to linking its data with other datasets, AnalyticsIQ has taken an extra step that could contribute to a more efficient Expert Determination (which is required when its data is linked with others in Datavant’s ecosystem).
AnalyticsIQ’s decision to adopt state de-identification standards underscores the importance of privacy in the data ecosystem. By addressing privacy challenges head-on, AnalyticsIQ and similar partners are poised to lead clinical research forward, providing datasets that are not only compliant with privacy requirements, but also ready for seamless integration into larger datasets.
"Stakeholders across the industry are seeking swift, secure access to high-quality, privacy-compliant SDOH data to drive efficiencies and improve patient outcomes,” says Christine Lee, head of health strategy and partnerships at AnalyticsIQ.
“By collaborating with Datavant to proactively perform state de-identification and Expert Determination on our consumer dataset, we help minimize potentially time-consuming steps upfront and enable partners to leverage actionable insights when they need them most. This approach underscores our commitment to supporting healthcare innovation while upholding the highest standards of privacy and compliance."
As the regulatory landscape continues to evolve, Datavant’s state de-identification product offers an innovative tool for privacy officers and data custodians alike. By addressing both state-specific and HIPAA requirements, companies can stay ahead of regulatory demands and build trust across data partners and end-users. For life sciences organizations, this can lead to faster, more reliable access to the datasets they need to drive research and innovation while supporting high privacy standards.
As life sciences companies increasingly rely on SDOH data to drive insights, the need for privacy-preserving solutions grows. Data ecosystems like Datavant’s, which link real-world datasets while safeguarding privacy, are critical to driving innovation in healthcare. By integrating state de-identified SDOH data, life sciences can gain a more comprehensive view of patient populations, uncover social factors that impact health outcomes, and ultimately guide clinical research that improves health.
Both payers and providers are increasingly utilizing SDOH data to enhance care delivery and improve health equity. By incorporating SDOH data into their strategies, both groups aim to deliver more personalized care, address disparities, and better understand the social factors affecting patient outcomes.
Payers increasingly leverage SDOH data to meet health equity requirements and enhance care delivery:
Payers’ consideration of SDOH underscores their commitment to improving health equity, delivering targeted care, and addressing disparities for vulnerable populations.
Capital District Physicians’ Health Plan (CDPHP) incorporated SDOH, partnering with Papa, to combat loneliness and isolation in older adults, families, and other vulnerable populations. CDPHP aimed to address:
By integrating SDOH data, CDPHP enhanced their services to deliver comprehensive care for its Medicare Advantage members.
Value-based care organizations face challenges in fully understanding their patient panels. SDOH data significantly assists providers to address these challenges and improve patient care. Here are some examples of how:
By leveraging SDOH data, providers gain a more comprehensive understanding of their patient population, leading to more targeted and personalized care interventions.
While accessing SDOH data offers significant advantages, challenges can arise from:
To overcome these challenges, providers must have robust data integration strategies, standardization efforts, and access to health data ecosystems to ensure comprehensive and timely access to SDOH data.
With Datavant, healthcare organizations are securely accessing SDOH data, and further enhancing the efficiency of their datasets through state de-identification capabilities - empowering stakeholders across the industry to make data-driven decisions that drive care forward.
Explore how Datavant can be your health data logistics partner.
Contact us
.svg)
