The S3 mirror responses were not served due to unexpected errors caused by analytics reporting to Cloudflare. We are implementing a fix to prevent this from happening during future Cloudflare R2/Analytics outages.

The Cloudflare Incident has been resolved. https://www.cloudflarestatus.com/incidents/pgz7g5xlpxzr

We can no longer see any errors within our monitoring. We are still doing an internal investigation on why our S3 mirror did not taking over when R2 was unavailable.

We are seeing a reduction in errors being reported within our monitoring system. The CDN is recovering.

Cloudflare posted a status claiming to resolve the availability issues: "A fix has been implemented and we are monitoring the results" (https://www.cloudflarestatus.com/incidents/pgz7g5xlpxzr).

We have a theory on why the S3 mirror requests are not served, calls to the Cloudflare analytics reporting (which is also affected by the outage) might let requests fail before serving the result.

We are currently blocked from deploying and testing that change to our systems by a GitHub incident (https://www.githubstatus.com/incidents/yhn3m0yqdxmc).

Schema publishes and reads the supergraph/schema/services from CDN that are affected by the Cloudflare outages.

We are investigating why our S3 mirror is failing partially failing to serve the artifacts if it fails to serve them from R2.

Cloudflare is having an ongoing incident in the past hour (https://www.cloudflarestatus.com/incidents/pgz7g5xlpxzr). Our solution is based on R2 as primary data store, so schema publish and schema checks are affected at the moment.

CDN is expected to perform without issues, as we have a mirror for reading data from the storage.

We are monitoring and tracking the incident in CloudFlare, and we will post updates here soon.