Data Engineering

Lessons Learned From Running Apache Airflow at Scale - Shopify

Shopify shares some of the lessons we learned and solutions we built in order to run Airflow at scale.

Billions of Messages a Day - Yelp's Real-time Data Pipeline

Billions of Messages a Day - Yelp

Data Wrangling at Slack

For a company like Slack that strives to be as data-driven as possible, understanding how our users use our product is essential. The Data Engineering team at Slack works to provide an ecosystem to help people in the company quickly and easily answer questions about usage, so they can make better and data informed decisions:…

How Airbnb built “Wall” to prevent data bugs

Gaining trust in data with extensive data quality, accuracy, and anomaly checks.

Streaming Messages from Kafka into Redshift in near Real-Time

Streaming Messages from Kafka into Redshift in near Real-Time Shahid C., Software Engineer Oct 17, 2016 This post is part of a series covering Yelp

Apache Beam for Search: Getting Started by Hacking Time - Shopify

To create relevant search, processing clickstream data is key: you frequently want to promote search results that are being clicked on and purchased, and demote those things users don’t love. Typically search systems think of processing clickstream data as a batch job run over historical data, perhaps using a system like Spark. But on Shopify’s Discovery team, we ask the question: What if we could auto-tune relevance in real-time as users interact with search results—not having to wait days for a large batch job to run? At Shopify—this is what we’re doing! We’re using streaming data processing systems that can process both real-time and historic data to enable real-time use cases ranging from simple auto boosting or down boosting of documents, to computing aggregate click popularity statistics, building offline search evaluation sets, and on to more complex reinforcement learning tasks. But this article is introducing you to the streaming system themselves. In particular, to Apache Beam. And the most important thing to think about is time with those streaming systems. So let’s get started! What Exactly is Apache Beam? Apache Beam is a unified batch and stream processing system. This lets us potentially unify historic and real-time views of user search behaviors in one system. Instead of a batch system, like Spark, to churn over months of old data, and a separate streaming system, like Apache Storm, to process the live user traffic, Beam hopes to keep these workflows together. For search, this is rather exciting. It means we can build search systems that both rely on historic search logs while perhaps being able to live-tune the system for our users’ needs in various ways. Let’s walk through an early challenge everyone faces with Beam: that of time! Beam is a kind of time machine that has to reorder events in their right spot after getting annoyingly delayed by lots of intermediate processing and storage step. This is one of the core complications of a streaming system

Leveraging Spark 3 and NVIDIA’s GPUs to Reduce Cloud Cost by up to 70% for Big Data Pipelines

Lyft’s Feature Store: Architecture, Optimization, and Evolution

Kafka App? There’s a Skill for That

Etsy is home to over 100 million listings from 5.6 million active sellers. Because the items for sale are unique and creative, there is no standard product catalog that tells us what they are. When someone searches for “light linen dress for summer,” our models must infer what the shopper means and what the listings contain. They do that using data from user visits and Etsy listings. We use Kafka for streaming data. These data streams provide fresh features and embeddings that power our machine learning models. If a shopper favorites a hand-painted ceramic mug, that action can shape their recommendations within minutes. But building these streaming applications hasn’t always been a smooth process. A new app requires templated code, configs, tests, validation, deployment steps, and careful review. That developer experience has been a strain on engineers across the company. The Olden Days Before coding agents were broadly available at Etsy, engineers turned machine learning designs into Kafka applications by hand. A streaming feature might require filtering events from terabyte-scale topics, writing transformation logic, and passing outputs through filtered topics to make them available in the feature store. The challenge wasn't writing Java or YAML. It was choosing the right schema, identifying the correct base class, validating input, anticipating edge cases, and running local tests. While static templates could generate files, they couldn't streamline the iterative, time-consuming process. Old process, human developer does everything Kafka Skills to the Rescue Our team provides the infrastructure and platform support for Etsy engineers to create and manage new features and embeddings with Kafka, among other streaming data products. We decided to make this easier with Agent Skills, reusable playbooks that teach language models how to execute specific tasks. For each workflow, the agent uses the Skills and tools we built to generate code, validate configs, run apps locally, and create PRs. Collaborative process between Human developer and Kafka agent The Kafka Skills we created are conversational workflows that allow developers and LLMs to collaborate. They can be invoked with natural language or a slash command such as /new-streaming-feature. Conversation with Claude using Kafka Skill to create a new ML feature For each streaming feature pipeline, the agent creates a Java class that implements a shared interface. The YAML configuration files are fill-in-the-blanks. The agent uses scaffolding scripts, templated code generation, and other tools that we created to output deterministic code. We saw an opportunity to have the Kafka Skills orchestrate all the steps to run an application locally. The human developer no longer needs to create test topics, build JARs, fill in manual commands and read from test topics. The agent will do all that, report back on findings, and generate all the relevant output links. Pull request and verification links posted for the developer’s review Observability We wanted to know which Kafka Skills are being used, how often they make it to pull requests, which environments and application types are involved, and where developers may still need support. To do this, we added lightweight telemetry around the workflow: when a Skill is invoked, a hook fires and a Python helper emits a started event. When the agent opens a pull request, a wrapper emits a completion event. Both events flow through an existing Etsy service that publishes a protobuf message to Kafka and stores it in a BigQuery table. Recording developer use of Kafka Skills in BigQuery Build It and They Won't Come We tested and demoed Kafka Skills and were very pleased with our “Agentic Improvements”. When we looked at adoption rates though, we were surprised to find that developers weren’t using them. They weren’t using the Skills to create features, generate embeddings, or even turn Kafka topic records into HTTP requests. Feeling a bit defeated, we asked devs: why? They responded that they simply didn’t know about them. The devs were using coding agents, but the agents never offered the Kafka Skills. Before the Skills existed, the standard workflow for feature creation was to find the closest existing feature, copy the code for it, and modify the fields. Instead of starting the process by opening the LLM and saying "I'd like to create a new streaming feature," an engineer might engage with the agent halfway through the old process and ask to "copy this class and change it to count cart moves," or "edit this file" with a path lifted from a previous pull request. Another challenge we found was that requests often arrive in whatever vocabulary the developer's team uses. When creating a new feature with Kafka, the dev might refer to a specific feature-store entry, an internal event name, a parent Java class name needed, or an output data type. To address these issues, we enumerated entry points for different vocabulary, lifecycle stages, and copy-paste-shaped requests. Plus, we added repository-level routing so that touching related files is itself a trigger for the Kafka Skill. The developer describes the task however they naturally would, and the agent offers the paved path. Guardrails In developing the Kafka Skills, we hoped to prevent agents from inventing Kafka applications from scratch. The agent handles the ambiguous parts: understanding the developer's request, identifying source schemas and fields, and deciding which validation steps to run. The agent then passes structured inputs into versioned scaffolding scripts and validation tools. Anthropic describes this pattern in its tool-use documentation: the agent returns a structured tool call that application-owned code executes, and strict tool use can constrain those inputs to a JSON schema. The agent handles the orchestration, while our scripts and hooks generate the templated code. Security research reinforced that decision. In a 2023 Stanford study, developers using an assistant wrote less secure code and were more likely to think their code was secure. BaxBench found a similar gap between code that runs and code that’s safe: across 392 backend tasks and 11 models, the best model was functionally correct 62% of the time, but no model produced code that was both correct and secure on more than 37% of the tasks. Researchers could exploit about half of the functionally correct programs. Now that we had agents running commands and committing code on our behalf, we also found that we needed to tighten Kafka ACLs, repository permissions, and deployment controls. We designed the system around least privilege: agents only get the access they need for the task they are performing. This was to make sure that neither human nor agent could delete an important topic, write to an unexpected one, change an unrelated pipeline, or bypass the normal production review path. What’s Next? Kafka supports a wide range of critical data workflows at Etsy, including machine learning feature generation, embeddings, filtering, fan-out, and storing data. We’ve built Skills for seven Kafka workflows so far. Adoption is the near-term focus: we're continuing to add entry points so the Skills are offered no matter how a developer phrases the task. Among our earliest insights from the observability table will be whether the discoverability fixes described above worked, and we’ll continue to iterate as we learn. We built the Kafka Skills to fix a developer-experience problem: too many steps stood between an idea for an ML model and a working pipeline with all the data. Our machine learning models shape the Etsy user experience for buyers and sellers. Each streaming feature that ships sooner helps our models understand a little earlier what shoppers mean and what listings contain. That means more fresh signals behind every search for a light linen dress, and behind the next hand-painted ceramic mug someone favorites. Acknowledgements A huge thanks to Kamya Shethia, Shannon Hoang, Nicollette Lui, Andrey Polyakov, and Virginia Poundstone for all their hard work designing and building Kafka Skills and agent tools, setting up observability, as well as locking down access control to make the world safer for robots and humans alike.

A Tale of Two Flink Autoscalers

From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines

Excerpt By 2024, Slack’s data platform had accumulated 700+ SSH-based operators orchestrating critical data pipelines. We’re talking daily search indexing that processed terabytes of data, analytics jobs powering business intelligence, the whole shebang. Every single one of these jobs required direct SSH access to production AWS Elastic MapReduce (EMR) clusters. We had a massive security…

Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework

Reducing experiment duration with predicted control variates

In 2021, we published a blog post titled “Increasing experimentation accuracy and speed by using control variates,” describing how we reduce the variance of metrics using CUPED in our experimentation platform. This is a follow-up on how CUPED has evolved at Etsy since then. Spoiler – It’s changed a lot, decreasing our average experiment duration by 3 days! Etsy’s mission is to Keep Commerce Human. To achieve this, we need to understand the impact each change to our platform has on our buyers' and sellers' experience. Whether that involves changing the color of the “Buy Now” button on the Etsy app or updating elements of how our algorithms rank search results, we leverage large-scale online experimentation to iterate on and improve the things we build. However, running an experiment can be a long process. From design and setup to running the experiment and analyzing results, the entire experimentation process can take weeks to months. Experiments must run long enough to collect sufficient data for the results to be statistically significant – ensuring we can confidently attribute observed changes to the treatment, rather than random chance. On the other hand, being able to learn from an experiment quickly is a crucial step in the product development lifecycle, enabling faster improvements to Etsy. Fortunately, there are tools to reduce experiment runtime. CUPED is one of them! Variance reduction techniques like CUPED can help reduce the time to run an experiment, shortening the overall experimentation lifecycle and time to learning, as visualized below. A recap of CUPED CUPED is a variance reduction technique that estimates experiment outcomes with greater speed and accuracy compared to a direct comparison between control and treatment groups. In 2021, Etsy implemented CUPED (Controlled-Experiment Using Pre-Experiment Data) for key metrics like Conversion Rate (the percentage of visitors that make a purchase). CUPED leverages historical visitor data collected before the experiment begins — for example, the number of purchases in the week prior to the experiment – to explain some natural variation in the outcome metric. The pre-experiment factors are used as covariates in a linear regression model to remove some of the “noise” that is not attributable to the treatment. By accounting for this variation, CUPED reduces the variance of the treatment effect estimator, increasing statistical power and improving sensitivity without introducing bias. The CUPED correction can be conceptualized as: The CUPED-adjusted metric will have a smaller variance than the original metric, as visualized below, providing more precise estimates of a mean or treatment effect. Sample size, power, and variance are all related. Holding everything else unchanged, the smaller the variance of a metric, the smaller the sample size required to reach a desired power. Since we can reduce the variance of our metric by applying CUPED, we can achieve the same amount of power with a smaller sample size. In practice, a smaller sample size corresponds to a shorter experiment duration. Etsy’s initial implementation of CUPED yielded an average variance reduction of 7% across all experiments, with some experiments achieving up to 30% variance reduction. Experiments that used CUPED-adjusted metrics in decision-making yielded a decision about 1 day earlier, on average. However, we’re always iterating to improve our buyers’ and sellers’ experience on Etsy, and we knew we could do even better. Enter: CUPAC. Leveling up further with CUPAC During our research and implementation of CUPED in 2020, scientists at DoorDash published a blog post describing a novel statistical method, building on CUPED, called Control Using Predictions as Covariate, or “CUPAC.” When performing CUPAC, the pre-experiment data is first input into a non-linear machine learning model that captures more complex relationships than a linear model. The non-linear model is trained to predict the outcome metric of interest – for example, if an experiment is measuring the observed Conversion Rate, the model would predict Conversion Rate. The prediction more effectively captures the impact of pre-experiment behaviors on our experimental outcomes than the raw pre-experiment data because it captures complex relationships in the data that linear regression alone cannot. The prediction is then used as an “ML-based covariate” in a linear regression to perform the CUPED correction: The CUPAC-adjusted outcome has an even smaller variance than the CUPED-adjusted outcome, as visualized below. Empirically, our CUPAC-adjusted metrics showed even lower variance than CUPED. Our initial prototype demonstrated that CUPAC produced an adjusted metric with an additional 10% smaller variance when compared to our original CUPED estimator. Despite the added complexity, these results justified incorporating CUPAC into our experimentation pipeline. We hypothesized it would cut average experiment duration by an additional day, enabling teams to run more experiments and ship changes to Etsy faster. Training and implementation The first step was to train the CUPAC models to predict the ML-based covariate. We identified over 100 pre-experiment features, increasing from 3 features in CUPED, to capture more behavior prior to the experiment. Using these features, we iteratively trained and tuned the models in Vertex AI. Hyperparameters were optimized on a validation dataset to maximize the median correlation between the model’s predictions and the observed in-experiment metrics across experiments. Initially we trained XGBoost, a popular gradient boosted tree model, but then found LightGBM, a similar non-linear, tree-based model, was better suited to predict the covariate. When testing the models at scale with billions of predictions, LightGBM demonstrated both rapid training and prediction times, along with strong validation results. Once the models were trained, our next challenge was to implement them at scale. Our experimentation pipeline runs batch jobs for hundreds of experiments each day. From our original implementation, we had an Airflow DAG (directed acyclic graph) to orchestrate the CUPED variance reduction pipeline, as visualized below: We evolved this pipeline to support CUPAC by adding a batch prediction step to produce the ML-based covariate. In the above CUPAC pipeline, we perform the following steps: Calculate pre-experiment features and in-experiment data using BigQuery SQL jobs. Predict ML-based covariates with our trained LightGBM models via parallel Dataflow jobs using the pre-experiment features. Perform variance reduction with a Spark job that fits a linear regression model between the ML-based covariates and in-experiment data, creating the CUPAC-adjusted metrics. Apply statistical t-tests using the CUPAC-adjusted metric to calculate the treatment effect, p-value, and power of the experiment. Impact: Shortening average experiment duration by 3 days We measured success through variance reduction. Variance reduction is the percent change between the: Variance of the metric without CUPAC, and Variance of the CUPAC-adjusted metric. The original CUPED implementation showed 7% variance reduction, reducing overall experiment duration by almost 1 day, on average. After implementing CUPAC, we observed an average of 27% variance reduction, nearly 4x as much variance reduction, when compared to CUPED, exceeding our early research estimates. The additional variance reduction shortens our average experiment duration by almost 3 days. This means a 10-day experiment could conclude in only 7 days due to the ability to reach power on a smaller sample size with CUPAC. These marginal time savings allow many teams to run 10 or more additional experiments each year. That translates to more opportunities to test and faster insights into how we can deliver the best experience for our community of millions of sellers and buyers. Notably, there was a substantial spread in variance reduction among different metrics and experiments, ranging from 2% to 77%. In the chart below, each blue bar displays the percent variance reduction for a sampled metric on an experiment. The large range is expected because variance reduction can be influenced by several factors, such as metric definition, data accessibility, experimental design, and market characteristics. These factors impact how predictive the pre-experiment data is of the outcome metric, resulting in the degree of variance reduction. For example, two common experimentation metrics are Mean Visits and Purchase Rate. In the e-commerce setting, an individual's visit behavior will almost always be more stable over time than their purchasing behavior. This implies that pre-experiment data is more correlated with in-experiment data for a visit-related metric than for a purchase-related metric. Therefore, CUPAC is more effective at reducing variance in a metric like Mean Visits than in a metric like Purchase Rate. What’s next? Aligned with Etsy’s culture of experimentation, we’ll continue to evolve our pipeline to be nimble and flexible based on the needs of the teams that use them. One challenge we face is that teams use metrics curated to specific parts of the Etsy experience – like search, recommendations, seller features, etc. – to make decisions on their experiment results. However, our CUPAC models take significant time to train and maintain for each metric, consequently limiting the number of CUPAC-adjusted metrics we can develop. While we continue to grow CUPAC use, we also encourage teams to continue to use CUPED, which is more scalable and has lower maintenance costs. To account for this, we plan to increase the flexibility of CUPED to more metrics by automatically collecting pre-experiment data based on the metric definition to reduce noise. In tandem with our work on CUPAC, this CUPED expansion will enable teams across Etsy to benefit from variance reduction across all their team-specific metrics, not just a select few. Despite the success of CUPED and CUPAC thus far, there remains a need to explore additional variance reduction techniques for the current metrics that leverage CUPAC. In 2024, we released research findings exploring a novel approach: Variance reduction combining pre-experiment and in-experiment data. As we look to generalize our variance reduction architecture, we expect that incorporating such techniques will continue to strengthen our experimentation platform and enable product teams to iterate more quickly. Lastly, it is important to recognize that applying variance reduction in practice can be a never-ending race to squeeze the most noise out of these estimators. In our experience, the craft lies in finding the sweet spot between variance reduction, implementation cost, and the impact on experimentation velocity. That intersection is context-dependent and what makes experimentation code as craft. We hope our experience inspires you to try out variance reduction techniques and determine which one is best suited to your needs! Acknowledgements Thank you to Alexander Tank and Stephane Shao for their work on initial research and implementation of CUPAC. Thanks to Pablo Crespo for his research into extending our CUPAC models with more predictive features. And, thanks to Julie Beckley, Kevin Gaan, and Mary Hu for supporting and prioritizing this project. References A. Deng, Y. Xu, R. Kohavi, T. Walker (2013). Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. J. Li (2020). Improving Experimental Power through Control Using Predictions as Covariate (CUPAC).

Data Projects: Managing Data Assets at Netflix Scale