Spotify Engineering·unknown·11 min read·Neville Li, Claire McGinty, Sahith Nallapareddy, & Joel Östlund49
How Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020 | Spotify Engineering
In this post we’ll discuss how Spotify optimized and sped up elements from our largest Dataflow job, , for using a technique called Sort Merge Bucket (SMB) join. We’ll present the design and implementation of SMB and how we incorporated it into our data pipelines.
Discussion around the web
- Hacker News196 · 125 💬
Score breakdown
- Technical depth52
- Practical value45
- Originality60
- Writing quality100
- Source reputation84
- Recency0
- External engagement34
- On-site engagement0
More like this
Shopify Engineering72
Lessons Learned From Running Apache Airflow at Scale - Shopify
unknown·15 min read
Shopify Engineering67
Shopify’s journey to faster breadth-first GraphQL execution (2026) - Shopify
unknown·16 min read
Yelp Engineering46
mrjob: Distributed Computing for Everybody
unknown·4 min read
Uber Engineering57
Scaling Exact COUNT(DISTINCT) for High-Cardinality Non-Rollup Metrics in Distributed Data Pipelines
unknown·14 min read
Engradar shows a summary and links to the original article. The full article is hosted on engineering.atspotify.com.