Incident Summary

Grand Challenge was completely unreachable for approximately 51 minutes. The outage began at the same time as a routine deployment, which initially led us to suspect our own changes. Investigation ruled out our infrastructure, and we mitigated the issue by switching the main API from a CloudFront VPC origin to a global CloudFront origin. It later turned out to be an unrelated, unannounced AWS outage affecting CloudFront VPC Origins connectivity.

Timeline

Incident Summary

A problem with the grand-challenge backend prevents reader study cases to load in common scenarios for some users.

Friday evening: After an initial analysis of the issue it seems that it is contained to only a few users of the platform. However, for those users it seems to be a persistent problem which needs further investigation. Right now, as evident from the application log files, only developers seem to have observed the problem. Therefore, we lowered the severity from major to minor because it does not seem that users are directly affected at this moment.

Monday: We have fully tracked down the issue to a code change that added an alternative way of counting cases in a reader study. The issue affected users for grand-challenge who:

1. are editors of a reader study,
2. are readers of that same reader study,
3. had saved annotations,
4. and were using regular, non-admin accounts.

We reverted the change which resolved the issue. Reader studies are fully operational again.

Timeline

Incident Summary

We experienced a viewer session outage affecting all viewing of items on the platform. This included reader studies, archive items, algorithm-job results, and others.

Timeline

Incident Summary

We experienced a viewer session outage affecting all viewing of items on the platform. This included reader studies, archive items, algorithm-job results, and others.

Timeline

Incident Summary

We are experiencing issues with our queues. As a result, algorithms currently cannot be queued for execution and emails can't be sent. We are working on fixing the issue.

Timeline