Building an internal philosophy around the high-security cloud infrastructure necessary for our Authorization to Operate
In December 2022, Datavant achieved a major information security milestone: Federal Risk and Authorization Management Program (FedRAMP) Authorization to Operate at the Moderate Impact level. FedRAMP is a government-wide program that promotes the adoption of secure cloud services across the Federal Government by providing a standardized approach to security and risk assessment for cloud technologies and federal agencies, and establishes a framework for public-private partnership to promote innovation via more secure information technologies. Partnering with government and public organizations is a core component of the Datavant platform and health data network, and FedRAMP Authorization to Operate is a key enabler for us to connect public health data. We spoke with Lucienne Roe, Dongting Yu, and Jasmin Phua about the process, and what it means for Datavant.
The FedRAMP-approved mission statement about the program is given in the intro above, but in your own words, how would you describe what the program is?
FedRAMP provides a standardized approach to security authorizations for cloud service offerings. You could say it’s the mother of all compliance frameworks. We went from having programs like SOC (System and Organization Controls) 2 — Type 2, where we had 50 security controls in place, to FedRAMP Moderate, where we now have 326 controls in place. All of the controls are very prescriptive and together touch almost every team in the company from Security to Product Engineering to People.
Why is FedRAMP important to Datavant?
FedRAMP Authorization to Operate status is the vehicle through which the Federal Government accelerates the adoption of cloud computing. It creates transparent standards and processes for security authorizations on a government-wide scale. This is not a healthcare technology-specific certification, but a certification for any SaaS company that wants to work with the Federal Government. It’s kind of a golden ring because it’s very difficult to get and not a lot of companies of our size, or our stage, have managed to get it.
Obtaining FedRAMP Authorization to Operate is also critical to our mission to connect the world’s health data to improve patient outcomes. Having the ability to work across federal agencies very directly helps break down data silos. The National COVID Cohort Collaborative (N3C), for example, is one of the largest collections of secure and de-identified clinical data in the US for COVID-19 research. Datavant provides the privacy-preserving infrastructure tools for the N3C, and provides subject matter expertise on privacy-preservation enterprise architectures, data governance, and policy considerations. By achieving FedRAMP certification, Datavant and NIH are able to provide its intramural, extramural, federal and state agency stakeholders with strong reassurance that the privacy-preserving infrastructure and tools required to accomplish NIH’s mission are operating on a private, secure, reliable infrastructure.
FedRAMP provides what looks to be a very streamlined flow chart (above) describing the approval process, but what does the process look like from the inside?
First you undertake a gap assessment to find the areas where you fall short of their protocols, and you work with a sponsor at the Agency Level (in our case, it was NIH-NCATS) who agrees to review your package. You write a 450-page security plan and then undergo a really rigorous audit, supported by an accredited auditor (a 3PAO — Third Party Assessment Organization). This takes at minimum 3 months, and in the audit process they say things like, “For control X you say Y in one document, but you provide a slightly different version of Y in another document. Go fix it.”
What are the implications of this for the Security team?
Even though the program is meant for SaaS and the cloud age, a lot of the controls are what the Federal Government uses to manage their own on-premise environment, and they originate from very early days of cloud computing. Just translating the controls into a cloud-first environment takes time, and the translations are not always obvious.
For example, Federal Information Processing Standards (FIPS 140–2, a set of U.S. Government security requirements for data encryption) requires that all encryption use Validated Modules governed by NIST, the government’s standards body. As we are using AWS as our infrastructure, we also need to make API requests to AWS using their FIPS-specific endpoints, and configure our AWS resources to use FIPS encryption as directed by each service.
We discovered there are non-obvious consequences to this requirement. S3’s FIPS endpoints only support Virtual Hosted-style addressing, in the format of:
https://<bucket>.s3-fips.us-east-1.amazonaws.com
…but not Path-style in the form of:
https://s3.us-east-1.amazonaws.com/<bucket>
At the same time, AWS’s wildcard TLS certificate is only valid for:
*.s3-fips.us-east-1.amazonaws.com
…but any dots in an S3 bucket name would be interpreted as a subdomain, and result in a client-side TLS certificate validation error. (To overcome this limitation, AWS would need to include wildcards for all possible depths of subdomains!) The practical outcome of this situation was that we had to rename all of our S3 buckets that previously included dots in their names to use dashes. As some buckets are customer-facing or are parts of automated data pipelines, we also had to figure out how to migrate them without causing downtime.
Whether or not we are working with the Federal Government, FedRAMP security practices are good for the company.
Obtaining Authorization to Operate was a major administrative and engineering lift, but why was it ultimately good for Datavant for reasons beyond working with the Federal Government?
By nature of our mission, we are inherently a privacy/security-centric company. The FedRAMP approval process forced us to look at all of our current security controls and become even more regimented than we already were. One of the most significant things the approval process introduced was a vulnerability management program. With FedRAMP, you draw a little diagram of your company, and then you build a moat around it.
No federal data or metadata that’s inside the moat is allowed outside the moat. This means that you can’t use just any SaaS tool: you have to use vendors who also have FedRAMP approval (for example, Datadog for Government, Snowflake for Government, or AWS and Google, who both have Strong FedRAMP Authorization to Operate). The FedRAMP editions of Snowflake and Datadog are separate from their commercial editions. All of the existing data connectors into Snowflake and Datadog accounts must be migrated to point to a different instance, but after doing this, you inherit those companies’ authorization to operate.
How did the team manage to meet FedRAMP requirements in a way that did not handicap itself
(…or cause a revolt…)?
This was a big question for us, given the speed with which the Datavant engineering team usually works and the quantity of updates required. It was important to instill a philosophy around the process: “We’re not making these changes because the framework tells us we have to, and we also have no intention of ripping this work out 6 months from now, so let’s do it properly and make these obligations fit our needs while meaningfully maturing our security posture.”
Our adoption of Datadog, a cloud monitoring service, is a good example to illustrate the architectural thinking that was necessary along the way. Before our FedRAMP Authorization, we maintained security logs in a variety of systems. Now, all of our logs are maintained through Datadog for Government. It’s not particularly fun or glamorous to implement, but it lets us dashboard everything with extreme visibility in one place, and we can now perform cross-correlation between different streams of logs. Multi-system parameters unify alert triggers, whereas before each system had its own individual alert triggers. As a result we have much better visibility into our logging and alerting. This solution was not provided to us by the Federal Government, rather, it’s what we determined to best sit at the intersection of their requirements and our needs.
The Federal Government trusts us to operate securely, so we believe you can trust us to do the same.
This is also true for vulnerability management: we now have scanners like Qualys and Sysdig, and now constantly scan for vulnerabilities in…everything, and we’re tied to a schedule for fixing vulnerabilities. In FedRAMP there’s no such thing as a won’t-fix vulnerability. We must either fix, wait for upstream to fix, or show something to be a false-positive, and we have to show the proof. We have a monthly deliverable to the government where we upload our raw scans, a plan of action for fixes, and milestones that fully disclose any vulnerabilities we have not fixed on the prescribed timeline. Whether or not we are working with the Federal Government, this is good for the company.
Finally, we did not differentiate between commercial deployments and federal deployments, so all of our customers are also inside the FedRAMP “moat.” This allows us to say to our customers, “The Federal Government trusts us to operate securely, so we believe you can trust us to do the same.”
What would you like another company to know when considering the FedRAMP approval process?
There is a monthly cost to this beyond the regular deliverables to the government. FedRAMP editions of SaaS tools cost about thirty percent more when you transition to them. We have a number of commitments to show that we are putting the right things in practice, like making sure there are no changes to security groups or to ports and protocols. The calendar of continuous monitoring is ongoing: there are things you have to do constantly, quarterly, and every three years. The 3PAO assesses roughly one-third of the 326 controls every year, but we undertake our own mini audits more frequently, because every time we request a significant change (which must be approved by the government), we need to map it to all of the controls the particular change impacts.
FedRAMP approval requires significant support from multiple functions and at all levels of the company. There is an initial as well as ongoing financial investment, and it takes patience and persistence to complete the process. We could not have done this if Founder and Board President Travis May and Head of Engineering Aneesh Kulkarni had not been fully supportive of the effort.
In addition to Datavant’s commitment, the agency sponsor also commits to stay engaged. As a partner in the process, agency sponsors must be aware of our information security status and roadmap with regard to the FedRAMP impact at all times. Datavant’s achievement would not have been possible without the steadfast commitment of NCATS as our agency sponsor. Many thanks go to NCATS CIO, Sam Michael, and his team Nam Ngo, Hsuan Ou, Josephine Kennedy, Mark Backus (Zero Trust), and Ramon Alvarez.
About the Authors
Lucienne Roe has a background in IT Security and vulnerability management, and is currently Head of Compliance & Quality at Datavant. Connect with Lucienne via Linkedin.
Dongting Yu has a background in computer science and holds a Ph.D. in security from the University of Cambridge. He is currently Head of Security Engineering at Datavant. Connect with Dongting via Linkedin.
Jasmin Phua has a background in enterprise architectures and user-centered design, and is the Head of Government Solutions at Datavant. Connect with Jasmin via Linkedin.
Interview compiled/edited by Nicholas DeMaison and Rebecca Slisz.
Considering joining the Datavant team? We’re hiring remotely across teams and we would love to speak with any potential new Datavanters who are nice, smart, and get things done, and want to build the future tools for securely connecting health data and improving patient outcomes.
.svg)
