AI Changed How Spotify Builds. What We Learned (and Fixed) About Quality at Higher Velocity

Spotify migrated Java across its backend fleet in just three days using AI agents, yet zero of its recent major incidents were directly caused by AI-authored code. 👇

The real issue wasn't buggy AI code—it was raw velocity. Shipping changes much faster exposed hidden weaknesses in Spotify's testing, capacity planning, and regional failovers across an ecosystem serving 12 million backend requests per second.

To keep pace with this surge in code and content, Spotify overhauled its workload prioritization, broadened its release guardrails, and added stricter safety controls to its automated update pipeline.

When Spotify analyzed its incident retrospectives, it found that AI didn't write worse code, but the volume of change overwhelmed existing verification controls across four key areas:

- Content processing: Handling over 500,000 daily uploads stalled when batch jobs competed with new episodes and a scheduling bug cut throughput by 10%. Spotify fixed the scheduler, added end-to-end failure alerts, deprioritized batch jobs, and throttled spam from bad actors to ensure high-priority uploads process first. - Automated fleet updates: Spotify's agentic Fleet Management framework automated massive refactors, but an automated dependency upgrade passed checks and still failed in production. Spotify responded by expanding rollback capacity, strengthening verification checks, and scheduling automated merges during owning teams' working hours. - Regional failovers: Global AI demand stripped away spare CPU and GPU capacity, making regional failovers impact end users when backup regions ran out of compute. Spotify doubled its reserved edge capacity and built controls to shift service-mesh traffic gradually between regions. - Mobile app guardrails: Faster release cycles allowed smaller regressions to pass standard release checkpoints and accumulate over time on specific devices. Spotify broadened its guardrail metrics and added long-term trend tracking to catch subtle degradation earlier.

By upgrading its safety checks and workload controls to match AI delivery speeds, Spotify maintained platform stability across nearly 3,000 production services without sacrificing engineering velocity.