You've Got (Too Much) Mail: Behind the Scenes of the 3/25/26 Voice Outage
On March 25th, voice and video on Discord suffered major degradation beginning at 12:13 PDT, lasting a little over three hours. Learn how the issue originated, how it affected systems across Discord, how we recovered, and how we’re preventing the same problem from reoccurring.
Opening excerpt (first ~120 words) tap to expand
This article is a collaborative piece by Bo Ingram, Senior Staff Engineer on Realtime Infrastructure, and Stephen Birarda, Senior Staff Engineer on Audio/Video infrastructure. The sad trombone haunts our dreams. Womp, womp, womp, woOoOoOoOmp. Womp, womp, womp, woOoOoOoOmp. Womp, womp, womp, woOoOoOoOmp. A cascading outage manifests in many ways: processes crashing, users reconnecting. As an on-call engineer, you often see it firsthand when the alert notifications reach your phone. On March 25th, voice and video on Discord suffered major degradation beginning at 12:13 PDT until 15:30 PDT.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Discord.
Discussion
0 commentsMore from Discord
