Service is fully restored and new WebRTC sessions are succeeding. The cause was a misconfiguration in our traffic-routing layer that was inadvertently rolled out to our production and staging environments at the same time, disrupting request routing. Once we traced it to that change, we reverted it and confirmed recovery across the affected regions. To prevent a recurrence, we're strengthening our infrastructure rollout process so changes must be validated in staging before they can reach production.
Resolved
Service is fully restored and new WebRTC sessions are succeeding. The cause was a misconfiguration in our traffic-routing layer that was inadvertently rolled out to our production and staging environments at the same time, disrupting request routing. Once we traced it to that change, we reverted it and confirmed recovery across the affected regions. To prevent a recurrence, we're strengthening our infrastructure rollout process so changes must be validated in staging before they can reach production.
Monitoring
Fix has rolled out, EU and US sessions appear to be starting without issues again. We'll continue to monitor before sharing a post-mortem.
Identified
Sessions in EU and US currently failing to start. A fix is being deployed.
Investigating
We're investigating the size of the impact and the RC.
