Errors writing data due to AWS incident

Our data write pipeline cleared out its queues as of 6:40pm last night and has been operating normally since.

With AWS largely recovered, our services are processing data writes normally. Data queued during the incident should be processed over the next two hours. Our intelligent refresh process and patient history jobs are processing normally.

We will be continuing to monitor and will post a final update to this incident tomorrow morning.

As AWS continues to recover from this incident, their team has begun increasing the ability to launch new compute instances, which is enabling recovery across their systems. You can continue to follow their updates here.

As their services recover, we are seeing our write processing slowly return to normal capacity, with significant increases over the past hour. Our FHIR write APIs are functioning normally, and our EHR integrations and third-party data services have backlogs which the systems are working through.

We began to see signs of improvement at 1:50pm eastern today and are monitoring the situation as our systems return to normal operation.

Beginning at approximately 11am eastern time this morning, it appears that AWS began to experience additional significant issues as part of the fallout of the early-morning AWS incident.

This is causing significantly elevated errors from Zus’s FHIR write APIs, which impacts writing data directly to our APIs and via our web applications as well as Zus ingesting data from customer EHRs and third-party data sources.

We are continuing to monitor the situation and will evalute necessary remediations as AWS systems recover. AWS’s status page is available at https://health.aws.amazon.com/health/status