Topic
50 articles
We're building AI Gateway into a unified inference layer for AI, letting developers call models from 14+ providers. New features include Workers AI binding integration and an expanded catalog with multimodal models.
We are excited to launch Workers AI - an AI inference as a service platform, empowering developers to run AI models with just a few lines of code, all powered by our global network of GPUs
A feature platform that offers observability and management tools, allows ML practitioners to use a variety of data sources, while handling…
Machine learning based marketing automation to improve cost and volume efficiency in an ever-changing marketplace.
By Hamel Husain & Nick Handel
How we built and iterated on a machine learning Search Ranking platform for a new two-sided marketplace and how we helped it grow.
Authors: Dóra Jámbor and Chen Karako There is a good chance you have come across a “recommended for you” statement somewhere in our data-driven world. This may be while shopping on Amazon, hunting for new tracks on Spotify, looking to decide what restaurant to go to on Yelp, or browsing through your Facebook feed — ranking and recommender systems are an extremely important feature of our day-to-day interactions. This is no different at Shopify, a cloud-based, multi-channel commerce platform that powers over 600,000 businesses of all sizes in approximately 175 countries. Our customers are merchants that use our platform to design, set up, and manage their stores across multiple sales channels, including web, mobile, social media, marketplaces, brick-and-mortar locations, and pop-up shops. Shopify builds many different features in order to empower merchants throughout their entrepreneurial lifecycle. But with the diversity of merchant needs and the variety of features that Shopify provides, it can quickly become difficult for people to filter out what’s relevant to them. We use recommender systems to suggest personalized insights, actions, tools and resources to our merchants that can help their businesses succeed. Every choice a merchant makes has consequences for their business and having the right recommendation at the right time can make a big difference. In this post, we’ll describe how we design and implement our recommender system platform. Methodology Collaborative Filtering (CF) is a common technique to generate user recommendations for a set of items. For Shopify, users are merchants, and items are business insights, apps, themes, blog posts, and other resources and content that merchants can interact with. CF allows us to leverage past user-item interactions to predict the relevance of each item to a given user. This is based on the assumption that users with similar past behavior will show similar preferences for items in the future. The first step of designing our recommender system is choosing the right representation for user preferences. One way to represent preferences is with user-item interactions,
Merlin, Shopify’s machine learning platform that can handle different (often conflicting) requirements, inputs, data types, dependencies, and integrations.
Tangle saves months of compute time, makes every experiment automatically reproducible, and allows teammates to share computation without coordination.
Learn how Shopify Data built new online inference capabilities into its Machine Learning Platform to deploy and serve models for real-time prediction at scale.
Introduction Each year, Etsy hosts an event known as “CodeMosaic” - an internal hackathon in which Etsy admin propose and build bold advances quickly in our technology across a number of different themes. People across Etsy source ideas, organize into teams, and then have 2-3 days to build innovative proofs-of-concept that might deliver big wins for Etsy’s buyers and sellers, or improve internal engineering systems and workflows. Besides being a ton of fun, CodeMosaic is a time for engineers to pilot novel ideas. Our team’s project this year was extremely ambitious - we wanted to build a system for stateful machine learning (ML) model training and online machine learning. While our ML pipelines are no stranger to streaming data, we currently don’t have any models that learn in an online context - that is, that can have their weights updated in near-real time. Stateful training updates an already-trained ML model artifact incrementally, sparing the cost of retraining models from scratch. Online learning updates model weights in production rather than via batch processes. Combined, the two approaches can be extremely powerful. A study conducted by Grubhub in 2021 reported that a shift to stateful online learning saw up to a 45x reduction in costs with a 20% increase in metrics, and I’m all about saving money to make money. Day 1 - Planning Of course, building such a complex system would be no easy task. The ML pipelines we use to generate training data from user actions require a number of offline, scheduled batch jobs. As a result it takes quite a while, 40 hours at a minimum, for user actions to be reflected in a model’s weights. To make this project a success over the course of three days, we needed to scope our work tightly across three streams: Real-time training data - the task here was to circumvent the batch jobs responsible for our current training data and get attributions (user actions) right from the source. A service to consume the data stream and learn incrementally - today, we heavily leverage TensorFlow for model training. We needed to be able to load a model's weights into memory, read data from a stream, update that model, and incrementally push it out to be served online. Evaluation - we'd have to make a case for our approach by validating its performance benefits over our current batch processes. No matter how much we limited the scope it wasn't going to be easy, but we broke into three subteams reflecting each track of work and began moving towards implementation. Day 2 - Implementation The real-time training data team began by looking far upstream of the batch jobs that compute training data - at Etsy’s Beacon Main Kafka stream, which contains bot-filtered events. By using Kafka SQL and some real-time calls to our streaming feature platform, Rivulet, we figured we could put together a realistic approach to solving this part of the problem. Of course, as with all hackathon ideas it was easier said than done. Much of our feature data uses the binary avro data format for serialization, and finding the proper schema for deserializing and joining this data was troublesome. The team spent most of the second day munging the data in an attempt to join all the proper sources across platforms. And though we weren't able to write the output to a new topic, the team actually did manage to join multiple data sources in a way that generated real-time training data! Meanwhile the team focusing on building the consumer service to actually learn from the model faced a different kind of challenge: decision making. What type of model were we going to use? Knowing we weren’t going to be able to use the actual training data stream yet - how would we mock it? Where and how often should we push new model artifacts out? After significant discussion, we decided to try using an Ad Ranking model as we had an Ads ML engineer in our group and the Ads models take a long time to train - meaning we could squeeze a lot of benefit out of them by implementing continuous training. The engineers in the group began to structure code that pulled an older Ads model into memory and made incremental updates to the weights to satisfy the second requirement. That meant that all we had left to handle was the most challenging task - evaluation. None of this architecture would mean anything if a model that was trained online performed worse than the model retrained daily in batch. Evaluating a model with more training training periods is also more difficult, as each period we’d need to run the model on some held-out data in order to get an accurate reading without data leakage. Instead of performing an extremely laborious and time-intensive evaluation for continuous training like the one outlined above, we chose to have a bit more fun with it. After all, it was a hackathon! What if we made it a competition? Pick a single high-performing Etsy ad and see which surfaced it first, our continuously trained model or the boring old batch-trained one? We figured if we could get a continuously trained model to recommend a high-performing ad sooner, we’d have done the job! So we set about searching for a high-performing Etsy ad and training data that would allow us to validate our work. Of course, by the time we were even deciding on an appropriate advertised listing, it was the end of day two, and it was pretty clear the idea wasn’t going to play out before it was time for presentations. But still a fun thought, right? Presentation takeaways and impact Day 3 gives you a small window for tidying up work and slides, followed by team presentations. At this point, we loosely had these three things: Training data from much earlier in our batch processing pipelines A Kafka consumer that could almost update a TensorFlow model incrementally A few click attributions and data for a specific listing In the hackathon spirit, we phoned it in and pivoted towards focusing on the theoretical of what we’d been able to achieve! The 1st important potential area of impact was cost savings. We estimated that removing the daily “cold-start” training and replacing it with continuous training would save about $212K annually in Google Cloud costs for the 4 models in ads alone. This is a huge potential win - especially when coupled with the likely metrics gains coming from more reactive models. After all, if we were able to get events to models 40 hours earlier, who knows how much better our ranking could get! Future directions and conclusion Like many hackathon projects, there's no shortage of hurdles getting this work into a production state. Aside from the infrastructure required to actually architect a continuous-training pipeline, we’d need a significant number of high-quality checks and balances to ensure that updating models in real-time didn’t lead to sudden degradations in performance. The amount of development, number of parties involved, and the breadth of expertise to get this into production would surely be extensive. However, as ML continues to mature, we should be able to enable more complex architectures with less overhead.
Through our experience building our first few models, Shopify Data carved out a pragmatic step-by-step guide that has enabled us to successfully scale machine learning across our organization.
We share five steps on approaching machine learning from a business perspective that helped to build and scale our suite of machine learning products.
Meta’s Generative Ads Recommendation Model (GEM), the foundation model behind ads recommendations across Instagram and Facebook, now trains at LLM scale on several thousand of the latest-generation GPUs. This post goes into the details on how we achieved: doubling end-to-end (E2E) training efficiency to 20–25% Model FLOPs Utilization (MFU) while scaling training FLOPs 4x in [...] Read More... The post GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model appeared first on Engineering at Meta.
Ever searched for something specific, only to be met with results that are close, but not quite? On Etsy’s Search Relevance team, that frustration is exactly what we are tackling. Our goal is simple yet ambitious: to help buyers find exactly what they’re looking for, and to help sellers reach the people seeking their special products. Search plays a central role in that mission. Historically, Etsy’s search models have relied heavily on engagement signals – such as clicks, add-to-carts, and purchases – as proxies for relevance. These signals are objective, but they can also be biased: popular listings get more clicks, even when they’re not the best match for a specific query. To address this, we introduce semantic relevance as a complementary perspective to engagement, capturing how well a listing aligns with a buyer’s intent as expressed in their query. We developed a Semantic Relevance Evaluation and Enhancement Framework, powered by large language models (LLMs). It provides a comprehensive approach to measure and improve relevance through three key components: High quality data: we first establish human-curated “golden” labels of relevance categories (we’ll come back to this) for precise evaluation of the relevance prediction models, complemented by data from a human-aligned LLM that scales training across millions of query-listing pairs Semantic relevance models: we use a family of ML models with different trade-offs in accuracy, latency, and cost; tuned for both offline evaluation and real-time search Model-driven applications: we integrate relevance signals directly into Etsy’s search systems enabling both large-scale offline evaluation and real-time enhancement in production Together, this framework brings a more intent-aware search experience that better serves both buyers and sellers across our marketplace. Figure 1. Overview of the Semantic Relevance Evaluation and Enhancement Framework Capturing Shades of Relevance Let’s return to the idea of relevance categories. Based on user research, we define three categories for semantic relevance of query-listing pairs: Relevant: listing matches all parts of the query, accounting for meaning and proper nouns Partially relevant: listing matches part of the query or is thematically related but not a full match Irrelevant: listing has no meaningful connection to the query; its presence in top results would make the search feel broken Figure 2. Examples for the three relevance categories. Text highlighted in green shows how the product aligns with the search query, whereas red highlights indicate mismatches.* In an ideal world, we’d rely on human judgments for all query-listing pairs. But large-scale human annotation is time-consuming and expensive, rendering it infeasible. Instead, language models unlock the ability to generate these judgments at scale, transforming our ability to make every search on Etsy produce more relevant results. Data: Anchored by Humans, Scaled by LLMs With recent advances in LLMs, a promising approach to evaluate search relevance is to use LLM-as-a-judge: directly using LLMs to judge the relevance of our search system without looping in humans. However, this approach faces two main challenges: Domain shift: off-the-shelf LLMs may not capture the unique preferences and vocabulary of Etsy users Performance-cost tradeoff: larger LLMs offer stronger reasoning but are expensive for large-scale inference, while smaller LLMs are faster and cheaper, but less accurate To address these challenges, we start with human-curated golden labels to evaluate and align a powerful LLM with these human-labels, then use a full dataset scaled up by the LLM for training our relevance judge. In other words, humans define what good looks like, and LLMs help us scale it. LLMs do not replace human judgment, instead they align with and amplify it. We maintain a detailed, evolving relevance labeling guideline, continuously refined through user research and annotation feedback. What relevance means in our marketplace shifts over time and social context. For example, people searching for “face masks” pre-2020 were primarily looking for masks for costumes or fashion, which is a completely different intent from protective masks post-2020. These guidelines ensure our definitions of relevance accurately reflect Etsy users’ intent and capture cultural trends over time. Query-listing pairs are sampled from search logs using a mix of approaches, including both random, stratified sampling for broad coverage, and targeted sampling for challenging cases. Each query-listing pair is labeled by two Etsy admins, with an ongoing review process to both break ties and adjust labeling guidelines accordingly. For quality control, we continuously track metrics such as row-level disagreement rates, which measures how often multiple annotators disagree with each other for the same query-listing pair. To scale beyond manual annotation, we introduced a few-shot, chain-of-thought (CoT) prompting strategy using the o3 model, implemented in LangGraph. The prompt instruction is inspired by the annotation guidelines described above, and includes comprehensive query and listing features, like title, images, text description, attributes, variations, and extracted entities (read more about listing extracted entities in another one of our posts). We also applied self-consistency sampling to improve reliability. This model, known as the LLM annotator (as seen in Figure 1), is first validated against the human-labeled golden data to ensure its judgement aligns with humans. Once validated, we use it to generate large-scale training data to develop the production models. The LLM annotator thus serves as the foundation for our teacher-student modeling pipeline, bridging the gap between expensive manual labeling and scalable automated annotation. Models: Balancing Accuracy, Latency and Cost Our modeling pipeline uses a three-tier cascaded distillation design, where each model balances accuracy and efficiency for a specific purpose. The stack includes: The LLM annotator: our most accurate and cost-intensive model, aligned closely with human-labeled golden data The teacher model: a fine-tuned smaller LLM (Qwen 3 VL 4B) that delivers high-throughput annotation at scale The student model: a lightweight, BERT-based two-tower model optimized for real-time inference The LLM annotator aligns best with the golden labels, but is too costly for recurrent, large-scale inference. To reduce cost while maintaining quality, we performed supervised fine-tuning (SFT) with a smaller LLM, Qwen 3 VL 4B, using the training data generated by the LLM annotator. This teacher model preserves human alignment while enabling us to label millions of query-listing pairs daily, which is ideal for recurring evaluation and monitoring. The teacher, however, is too slow to surface relevant search results quickly, which is critical for helping our sellers reach potential buyers. As such, we further distilled the teacher into a student model with a two-tower architecture. The distillation process aligns the student’s output with that of the teacher, so that the student judges relevance labels nearly as accurately as the teacher, while being lightweight and fast. The resulting model ensures we deliver search results almost as fast as before, with only <10ms additional latency. All three models – the LLM annotator, teacher, and student – are evaluated against the same golden dataset to ensure traceable performance and consistent alignment with human judgment. Figure 3 shows their accuracy measured using multi-class Macro F1, and individual class F1 scores. Figure 3. Performance of semantic relevance models against human golden labels Applications: From Evaluation to Action With these models in place, we can both measure and enhance search relevance across Etsy. Search relevance evaluation We use the teacher model to measure how well our search system surfaces relevant listings. Each day, we sample search requests and perform offline inference using the teacher model, then aggregate the predicted relevance labels into summary metrics. These metrics are reviewed regularly by our team, and if we observe unexpected trends like a sudden decline of relevance, we work to quickly diagnose and address the problem. Similarly, we monitor relevance metrics in A/B tests. The computed relevance metrics are discussed when we decide whether to roll out a new change to our search system, to ensure those changes affect semantic relevance of search results in a neutral to positive way. We sample sufficient amounts of requests separately from control and treatment variants, to ensure statistical power. Using vLLM for high-throughput inference, we process millions of query-listing pairs daily at a very low cost, maintaining both statistical power and operational efficiency. Improving search in production The lightweight student model is embedded directly into Etsy’s real-time search stack. It improves relevance through several integration points: Filtering: removes retrieved listings predicted as irrelevant before downstream ranking Feature enrichment: contributes model-predicted relevance scores as features for the downstream ranking model Loss weighting: adjusts training weights of the ranking model based on predicted relevance Relevance boosting: promotes listings deemed highly relevant using heuristic rules among the final returned search results How Semantic Relevance is Changing Etsy Search The Semantic Relevance Evaluation and Enhancement Framework is fully deployed in Etsy’s search stack, and continues to evolve. We’ve observed a measurable uplift in semantic relevance: the percentage of fully relevant listings (as defined by the relevance categories described earlier) has increased from 58% to 62% between August and October 2025. Figure 4. Improvement of semantic relevance metrics over time This improvement reflects Etsy’s growing ability to align search results with buyer intent. For instance, in searches like “fall decor,” the enhanced search engine now focuses on surfacing seasonal decor items, while deprioritizing loosely related listings like clothing, which appeared before the enhancement on relevance. Figure 5. Before and after comparison when searching for “fall decor” * Beyond these immediate gains, semantic relevance has shifted how we evaluate and improve search at Etsy, by adopting a user-centered approach. By grounding our evaluation in semantic intent in addition to behavioral signals, we move closer to our goal of connecting buyers with the relevant products, not just the most popular ones. While search results are influenced by multiple factors, and outcomes may vary, on the seller side, improving semantic relevance can also help surface items from small or new sellers who may not yet have the visibility of more established shops. What’s Next In ongoing and future efforts, we hope to explore the following directions: Better understanding of relevance-engagement dynamics. In online experiments, we often observe engagement metrics decline even as semantic relevance improves (a pattern also noted by other e-commerce platforms). We suspect this results from applying uniform relevance treatments despite contextual variation. Next, we plan to explore adaptive strategies that tailor adjustments by query type. Refining partial relevance. Inspired by Amazon’s ESCI framework, we’re exploring finer-grained labels, for example, introducing new subcategories of complements and substitutes. This could potentially improve evaluation precision and power new user search experiences. Reducing annotation effort through LLM facilitation. When LLM judgments are self-consistent, they align better with human labels. This may indicate easier query-listing pairs. We are exploring using LLMs for these easy cases, focusing human effort on more complex cases. Simplifying the multi-stage model stack. Our current three-tier distillation pipeline provides flexibility but adds operational complexity. We plan to simplify this setup by exploring better performance-efficiency tradeoffs and potentially merging model tiers. Improving relevance in retrieval. So far, post-retrieval filtering is the first stage where our semantic relevance model applies. We see strong potential to enhance both inference and measurement further upstream in the retrieval layer. Conclusion Key takeaways: LLMs can meaningfully evaluate search relevance when grounded in human judgment. Aligning LLM assessments with human-labeled data ensures we measure, and continually improve, the search experience that is so essential to connecting buyers and sellers on Etsy. Semantic relevance redefines how Etsy optimizes search. By complementing engagement metrics with semantic relevance, we address real customer pain points and deliver more satisfying search experiences. Teacher-student distillation offers a flexible and efficient way to apply relevance modeling across diverse performance, latency and cost requirements. Ultimately, improving semantic relevance strengthens the human connections that define Etsy. By understanding what shoppers truly mean, we can help them find the right items. And by emphasizing relevant listings over popular ones, we can help create fairer opportunities on the search relevance factor of search visibility for our sellers – 89% of whom are businesses of one. Acknowledgments This work is brought to you in a collaborative effort by the Search Relevance Team, enabled by ML Enablement, and the Merchandising teams. Thanks to the following contributors Data: Susan Liu, Jugal Gala, David Blincoe, Yuqing Zhang, Taylor Hunt, Liz Mikolaj Models: David Blincoe, Oriane Cavrois, Orson Adams, Yuqing Zhang Application: Grant Sherrick, Kaushik Bekal, Haoming Chen, Patrick Callier, Davis Kim, Marcus Daly Product leadership: Julia Zhou, Willy Huang, Argie Angeleas Engineering leadership: Yinlin Fu, Congzhe Su, Xiaoting Zhao ML Enablement partners: Ari Carter, Stan Schwertly, Shreya Agarwal, K Ogilvie, Marvin Wang, etc. Other cross-team partners: Will Beckman, Karl Yokono, Audrey Chen, Heather Campbell, David Le, Khadeeja Din, etc. Early contributors: Ethan Benjamin, Cung Tran, Maggie Matsui, Jack Gammack, Yogeeta Chatoredussy, Austin Clapp, Benjamin Russell, Khaled Jabr Special thanks to Oriane Cavrois & David Blincoe for helping this piece come to life. * Images are provided for illustrative purposes. Item availability on Etsy may vary.
When buyers search on Etsy, they need to quickly and easily find the perfect item. At the same time, sellers need to be confident their unique products are being seen by the right customers. Our Ads Search ranking model, which is built on a multitask learning foundation, is the critical link in this connection. Recently, we identified an opportunity to drive more meaningful buyer engagement by enhancing our model’s ability to predict purchase intent. We achieved this via a dual-pronged improvement: introducing Multigate Mixture of Experts (MMoE) to our model architecture and leveraging add-to-cart as an auxiliary signal. By providing our downstream systems with more accurate predictions, we improved matching in our marketplace, surfacing more relevant listings for buyers while helping sellers reach customers who are genuinely interested in their products. Background When a buyer searches for an item on Etsy, we want them to find exactly what they’re looking for from our inventory containing tens of millions of listings. In order to help them do this, we surface high-quality listings that are relevant to a user’s search query by ranking a small subset of items from a much larger group. This includes advertisements purchased by sellers that enables them to promote their listings across Etsy placements, including search. While these results are sponsored, the items go through their own ranking process to surface the listings most likely to meet a buyer’s needs. The final result on the search page utilizes our auto-bidding system, which helps decide which listings get shown and the cost-per-click. After a user views an ad (known as an “impression”), clicking on the ad is often the first engagement in their purchase journey. However, each subsequent step – from click to cart addition to purchase – represents a progressively smaller subset of users. The increasing data sparsity that exists further along in the purchase journey can make it difficult for our model to pick up on a strong signal to learn from. When ranking ads, our machine learning models optimize for click-through rate (CTR) and post-click conversion rate (PCCVR). Clicks and purchases are the primary behaviors we use to predict and drive user engagement, but other actions in the buyer’s purchase journey, such as adding an item to a cart, are important and often predictive of a purchase. Figure 1. The Ads Search user journey. Some post-impression actions, such as favoriting an item, are not directly related to a buyer's purchase journey but can provide valuable signals to enhance our model's predictive capability. A click can be a strong indicator of a future purchase, but it can also be noisy – meaning it doesn’t always reliably predict purchase intent. For example, a user may click on an ad purely out of curiosity with no intention to buy. These are just a few reasons why user behavior is complex, and we are constantly trying to improve our prediction models to better capture these patterns and recommend the most relevant ads. Multitask Model Architecture The Ads Search ranking model is a multitask learning framework containing four major components: feature representation, explicit feature interaction, implicit feature interaction, and task prediction. Figure 2 is a depiction of our model architecture prior to the enhancements this post will describe. We start with raw numerical, categorical, and high-cardinality ID features for query, user, and listing entities, which are converted through the feature representation layer - including text embeddings and sequence encodings - to generate dense feature representations. These are concatenated and fed to a Deep and Cross Network (DCN) that learns explicit feature interactions. The explicitly crossed features then pass through a shallow feed-forward network for the model to learn additional implicit feature interactions. Finally, the latent feature representations are fed into task-specific towers to output CTR and PCCVR predictions. Figure 2. The initial multitask architecture used for the Ads Search ranking model which has since been upgraded with Multigate Mixture of Experts (MMoE). Since the CTR and PCCVR predictions are used in downstream ads ranking and auto-bidding systems, we need the predictions to be well-calibrated. After the underlying model is trained, we individually calibrate the CTR and PCCVR towers to probability distributions using Platt scaling layers. As user behaviors vary significantly across ad placements, the model learns distinct parameters for different placements. Optimizing for Purchase Intent The multitasked architecture we use has several advantages, as it helps the model learn shared patterns across tasks, reduces overfitting by allowing the model to learn more generalizable features, and decreases training and serving infrastructure costs by consolidating the two separate models into a single model. We originally deployed this multitask ranking model online in July 2023 and had not made major changes to its architecture since then. In the second half of 2025, the team identified an opportunity to better optimize for meaningful buyer engagement beyond click and purchase signals alone. Our goal was to not only surface listings that resonate with buyers and drive conversions but also encourage them to return to Etsy – creating a positive feedback loop that benefits both buyers and sellers. We hypothesized that optimizing our models for engagement actions that signaled both purchase intent and buyer satisfaction would surface more relevant ads to buyers. Engagement actions include behavior that goes beyond a simple click, such as an add-to-cart or a favorited listing. To more effectively prioritize listings that led to this meaningful engagement for our buyers, we experimented with: A model architecture that would better predict purchase intent and Adding additional signals to boost high-quality listings that buyers are more likely to purchase These two enhancements, in the form of Multigate Mixture of Experts (MMoE) and add-to-cart as an auxiliary task, worked well together in our model to drive a sizable product improvement in Q4 2025. Enabling Task-Specific Learning with Multigate Mixture of Experts (MMoE) While the introduction of our initial multitask model was a large success overall, it also had a limitation: since the model learns and shares the same feature representations across tasks, it is not always able to learn task-specific nuances, and this is more pronounced the less related the tasks are. When one task sees improved performance, other tasks can see performance degradation. This behavior is known as the “seesaw phenomenon,” and we encountered this when we first brought the multitask model online. One solution to this limitation is to add a Multigate Mixture of Experts (MMoE) layer. In this architecture, the model still employs a shared bottom architecture where the feature representation and interaction layers remain unchanged. However, the MMoE layer introduces two key additional components in place of the shared feed-forward network: experts and gates. Experts are parallel subnetworks that, unlike the shared representations before them in the network, are able to specialize and learn different patterns of the data. Experts are not specifically assigned tasks but rather this learning happens organically during training - some experts learn more about click-specific behavior, others learn more about purchase-specific behavior, and others learn patterns that are important for both of these actions. Each task has one softmax gating network which controls how that task combines expert outputs. This allows the CTR and PCCVR tasks to use and activate different subsets of experts differently, and this weighted expert information is then sent to our task-specific towers. Figure 3. A comparison of our shared bottom multitask architecture for the Ads Search ranking model (left) and our MMoE architecture (right). Tuning the Experts The main hyperparameters to tune in the MMoE layer are the number of experts, the size of the experts, and the expert type. The number of experts used depends on several factors. One factor is a tradeoff between having too few experts – which can underfit and fail to capture distinct patterns needed for each task – and too many – which can overfit to training data and fail to generalize well. Another factor to consider is that adding experts increases model capacity which in turn increases latency and infrastructure costs. Our initial configuration included only multilayer perceptrons (MLP)-based experts (i.e., feed-forward neural networks), but we experimented offline with heterogenous experts and saw an offline lift in purchase and click metrics through introducing a mixture of DCN- and MLP-based experts. There are other pitfalls when employing an MMoE architecture which oftentimes require additional hyperparameter tuning to resolve. Specifically, common issues in an MMoE structure are expert utilization (the ability of each task to use multiple experts) and expert specialization (the ability of each expert to learn differently). If experts are not well-utilized, the model has wasted capacity and fails to leverage the full representational power of the architecture. If experts are not specialized, the model effectively reduces to a shared bottom architecture with redundant experts. On the other hand, if experts specialize too strictly and are not shared between tasks, the model loses some of the benefits of multitask transfer learning. We ran into these issues when training our new model. To build a successful model with MMoE, we needed each task to utilize more experts and utilize some of the same experts as the other tasks so that they could benefit from both specialized and shared learning. We experimented offline with two regularization techniques to try and solve this issue: expert dropout and temperature scaling. In expert dropout, we randomly disable some experts during training to force the model to learn more diverse representations. Expert dropout differs from typical “dropout” in neural networks (where we randomly remove a percent of connections in a given layer during training), as we fully remove the utilization of a number of experts during the forward pass. Using expert dropout, utilization did improve a bit: each gate was selecting a primary and a secondary expert for each task. Still, we did not see any sharing of experts between the tasks. We then tried temperature scaling, which modifies the raw logits of the expert gates by dividing them by a temperature (T) to control the smoothness of the resulting probability distribution. By applying this before the softmax function (which converts logits to probabilities) in the gates with a T > 1, we softened the distribution, making it more likely to select multiple experts. Expert dropout is random and only applied at training while temperature scaling is deterministic and applied at both training and inference. Temperature scaling achieved better utilization and specialization than expert dropout, leading us to deploy this approach. Auxiliary Tasks Our multitask model already leveraged user click and purchase engagements to train CTR and PCCVR towers. However, we also have access to other rich user interactions, namely add-to-cart and favorites, that reflect the meaningful buyer engagement described above. Purchases are quite sparse compared to clicks, and one of the major benefits of our original multitasking model was for this sparse purchase action to benefit with additional signal from the more common click action. Our goal with adding auxiliary tasks was to help the model learn more generalizable representations of user engagement, again leaning on actions that are more plentiful than purchase. We hypothesized that add-to-cart and favorite actions were indications of high purchase intent that would help the model better learn the purchase task without hurting the click task. Since we do not use add-to-cart and favorite predictions for downstream use cases like ranking or bidding, we did not need to calibrate these predictions or serve them online. This makes them relatively straightforward to add to our existing uncalibrated model architecture. In the shared bottom version of our model, we simply add one tower for each additional task. In the MMoE version of our model, we add one gate and one tower for each additional task. We experimented with both versions offline and found that MMoE in combination with auxiliary tasks performed better than the shared bottom model with auxiliary tasks. It makes sense that MMoE would outperform the shared bottom when we added more tasks due to the “seesaw phenomenon” described earlier. Through experimentation, we learned that while add-to-cart as an auxiliary task boosted purchase metrics by bridging the gap between purchases and clicks in terms of relatedness, favorites actually had a negative impact (known as negative transfer) on the model. With further analysis, we found that favorites can actually be quite noisy and not indicative of high purchase intent. As a result, the version of the model was ultimately ramped up in production only included add-to-cart as an auxiliary task. Figure 4. A simplified version of the MMoE piece of our model architecture with add-to-cart (ATC) as an auxiliary task. Results and Impact Offline, our new model showed promising improvements in Purchase and Click Area Under the Precision-Recall Curves (PR AUCs) and Purchase Area Under the Receiver Operating Characteristic Curve (ROC AUC) metrics. Together, these metrics measure how well our model predicts buyer behavior – PR AUC evaluates its ability to rank relevant listings at the top of the search results, and Purchase ROC AUC evaluates its ability to distinguish between listings buyers will and will not purchase. We saw average increases of 3.5% and 1% to Purchase and Click PR AUCs, respectively, and a 0.5% increase to Purchase ROC AUC, meaningful lifts for an industry-level ranking system. When we deployed the model online, we saw three meaningful improvements across the marketplace. First, the model drove purchases, improving buyer experience by more accurately predicting which listings from our inventory would resonate with them. Second, the ads marketplace became more efficient due to an improvement in purchase calibration metrics. More accurate PCCVR predictions served as better inputs to our auto-bidding system, which helped sellers reach buyers who are genuinely interested in their listings. Finally, the MMoE architecture is more flexible than the shared bottom architecture, so we were able to keep the overall model size flat by pruning other parts of the model when adding in MMoE. At serving time, inference became less costly, likely due to differences in the distribution of compute across model components. What’s Next The MMoE architecture provides the flexibility to add a variety of tasks to our ranking model by reducing the risk for negative transfer by encouraging some experts to learn task-specific patterns and others to learn shared representations. After seeing success with the add-to-cart task in our new modeling framework, we plan on experimenting with several additional auxiliary tasks, such as dwell time, to further improve our model’s ability to connect buyers with listings they’ll love. Ads give sellers an additional opportunity to stand out to buyers seeking their unique creations on Etsy. With each improvement to our ranking model, we continue to strengthen the marketplace connection between buyers and sellers – facilitating matches that help our sellers’ businesses grow and buyers discover products that feel made for them.
Learn how we solved a major search engineering dilemma—running machine learning models at native C++ speed.
At Meta, we’ve built a system that enhances ads recommendations by leveraging idle inference capacity in our fleet. Two elasticity levers in the system allow ads recommendation work to expand across opportunistic capacity, without compromising baseline performance. This post describes how we designed and now operate the system at Meta’s scale and how it drives [...] Read More... The post Turning Opportunistic AI Inference Capacity Into Better Ads Recommendations appeared first on Engineering at Meta.
Etsy features a diverse marketplace of unique handmade and vintage items. It’s a visually diverse marketplace as well, and computer vision has become increasingly important to Etsy as a way of enhancing our users’ shopping experience. We’ve developed applications like visual search and visually similar recommendations that can offer buyers an additional path to find what they’re looking for, powered by machine learning models that encode images as vector representations. Figure 1. Visual representations power applications such as visual search and visually similar recommendations Learning expressive representations through deep neural networks, and being able to leverage them in downstream tasks at scale, is a costly technical challenge. The infrastructure required to train and serve large models is expensive, as is the iterative process that refines them and optimizes their performance. The solution is often to train deep learning architectures offline and use the pre-computed pretrained visual representations in downstream tasks served online. (We wrote about this in a previous blog post on personalization from real-time sequences and diversity of representations.) In any application where a query image representation is inferred online, it's important that you have low latency, memory-aware models. Efficiency becomes paramount to the success of these models in the product. We can think about efficiency in deep learning along multiple axes: efficiency in model architecture, model training, evaluation and serving. Model Architecture The EfficientNet family of models features a convolutional neural network architecture. It uniformly optimizes for network width, depth, and resolution using a fixed set of coefficients. By allowing practitioners to start from a limited resource budget and scale up for better accuracy as more resources are available, EfficientNet provides a great starting point for visual representations. We began our trials with EfficientNetB0, the smallest size model in the EfficientNet family. We saw good performance and low latency with this model, but the industry and research community have touted Vision Transformers (ViT) as having better representations. We decided to give that a try. Transformers lack the spatial inductive biases of CNN, but they outperform CNN when trained on large enough datasets and may be more robust to domain shifts. ViT decomposes the image into a sequence of patches (16X16 for example) and applies a transformer architecture to incorporate more global information. However, due to the massive number of parameters and compute-heavy attention mechanism, ViT-based architectures can be many times slower to train and inference than lightweight Convolutional Networks. Despite the challenges, more efficient ViT architectures have recently begun to emerge, featuring clever pooling, layer dropping, efficient normalization, and efficient attention or hybrid CNN-transformer designs. We employ the EfficientFormer-l3 to take advantage of these ViT improvements. The EfficientFormer architecture achieves efficiency through downsampling multiple blocks and employing attention only in the last stage. This derived image representation mechanism differs from the standard vision transformer, where embeddings are extracted from the first token of the output. Instead, we extract the attention from the last block for the eight heads and perform average pooling over the sequence. In Figure 2 we illustrate these different attention weights with heat maps overlaid on an image, showing how each of the eight heads learns to focus on a different salient part. Figure 2. Probing the EfficientFormer-l3 pre-trained visual representations through attention heat maps. Model Training Fine-Tuning With our pre-trained backbones in place, we can gain further efficiencies via fine tuning. For the EfficientNetB0 CNN, that means replacing the final convolutional layer and attaching a d-dimensional embedding layer followed by m classification heads, where m is the number of tasks. The embedding head consists of a new convolutional layer with the desired final representation dimension, followed by a batch normalization layer, a swish activation and a global average pooling layer to aggregate the convolutional output into a single vector per example. To train EfficientNetB0, new attached layers are trained from scratch for one epoch with the backbone layers frozen, to avoid excessive computation and overfitting. We then unfreeze 75 layers from the top of the backbone and finetune for nine additional epochs, for efficient learning. At inference time we remove the classification head and extract the output of the pooling layer as the final representation. To fine-tune the EfficientFormer ViT we stick with the pretraining resolution of 224X224, since using sequences longer than the recommended 384X384 in ViT leads to larger training budgets. To extract the embedding we average pool the last hidden state. Then classification heads are added as with the CNN, with batch normalization being swapped for layer normalization. Multitask Learning In a previous blog post we described how we built a multitask learning framework to generate visual representations for Etsy's search-by-image experience. The training architecture is shown in Figure 3. Figure 3. A multitask training architecture for visual representations. The dataset sampler combines examples from an arbitrary number of datasets corresponding to respective classification heads. The embedding is extracted before the classification heads. Multitask learning is an efficiency inducer. Representations encode commonalities, and they perform well in diverse downstream tasks when those are learned using common attributes as multiple supervision signals. A representation learned in single-task classification to the item’s taxonomy, for example, will be unable to capture visual attributes: colors, shapes, materials. We employ four classification tasks: a top-level taxonomy task with 15 top-level categories of the Etsy taxonomy tree as labels; a fine-grained taxonomy task, with 1000 fine-grained leaf node item categories as labels; a primary color task; and a fine-grained taxonomy task (review photos), where each example is a buyer-uploaded review photo of a purchased item with 100 labels sampled from fine-grained leaf node item categories. We are able to train both EfficientNetB0 and EfficientFormer-l3 on standard 16GB GPUs (we used two P100 GPUs). For comparison, a full sized ViT requires a larger 40GB RAM GPU such as an A100, which can increase training costs significantly. We provide detailed hyperparameter information for fine-tuning either backbone in our article. Evaluating Visual Representations We define and implement an evaluation scheme for visual representations to track and guide model training, on three nearest neighbor retrieval tasks. After each training epoch, a callback is invoked to compute and log the recall for each retrieval task. Each retrieval dataset is split into two smaller datasets: “queries” and “candidates.” The candidates dataset is used to construct a brute-force nearest neighbor index, and the queries dataset is used to look up the index. The index is constructed on the fly after each epoch to accommodate for embeddings changing between training epochs. Each lookup yields K nearest neighbors. We compute Recall@5 and @10 using both historical implicit user interactions (such as “visually-similar ad clicks”) and ground truth datasets of product photos taken from the same listing (“intra-item”). The recall callbacks can also be used for early stopping of training to enhance efficiency. The intra-item retrieval evaluation dataset consists of groups of seller-uploaded images of the same item. The query and candidate examples are randomly selected seller-uploaded images of an item. A candidate image is considered a positive example if it is associated with the same item as the query. In the “intra-item with reviews” dataset, the query image is a randomly selected buyer-uploaded review image of an item, with seller-uploaded images providing candidate examples. The dataset of visually similar ad clicks associates seller-uploaded primary images with primary images of items that have been clicked in the visually similar surface on mobile. Here, a candidate image is considered a positive example for some query image if a user viewing the query image has clicked it. Each evaluation dataset contains 15,000 records for building the index and 5,000 query images for the retrieval phase. We also leverage generative AI for an experimental new evaluation scheme. From ample, multilingual historical text query logs, we build a new retrieval dataset that bridges the semantic gap between text-based queries and clicked image candidates. Text-to-image generative stable diffusion makes the information retrieval process language-agnostic, since an image is worth a thousand (multilingual) words. A stable diffusion model generates high-quality images which become image queries. The candidates are images from clicked items corresponding to the source text query in the logs. One caveat is that the dataset is biased toward the search-by-text production system that produced the logs; only a search-by-image-from-text system would produce truly relevant evaluation logs. The source-candidate image pairs form the new retrieval evaluation dataset which is then used within a retrieval callback. Of course, users entering the same text query may have very different ideas in mind of, say, the garment they’re looking for. So for each query we generate several images: formally, a random sample of length 𝑛 from the posterior distribution over all possible images that can be generated from the seed text query. We pre-condition our generation on a uniform “fashion style.” In a real-world scenario, both the text-to-image query generation and the image query inference for retrieval happen in real time, which means efficient backbones are necessary. We randomly select one of the 𝑛 generated images to replace the text query with an image query in the evaluation dataset. This is a hybrid evaluation method: the error inherent in the text-to-image diffusion model generation is encapsulated in the visually similar recommendation error rate. Future work may include prompt engineering to improve the text query prompt itself, which as input by the user can be short and lacking in detail. Large memory requirements and high inference latency are challenges in using text-to-image generative models at scale. We employ an open source fast stable diffusion model through token merging and float 16 inference. Compared to the standard stable diffusion implementation available at the time we built the system, this method speeds up inference by 50% with a 5x reduction in memory consumption, though results depend on the underlying patched model. We can generate 500 images per hour with one T4 GPU (no parallelism) using the patched stable diffusion pipeline. With parallelism we can achieve further speedup. Figure 4 shows that for the English text query “black bohemian maxi dress with orange floral pattern” the efficient stable diffusion pipeline generates five image query candidates. The generated images include pleasant variations with some detail loss. Interestingly, mostly the facial details of the fashion model are affected, while the garment pattern remains clear. In some cases degradation might prohibit display, but efficient generative technology is being perfected at a fast pace, and prompt engineering helps the generative process as well. Figure 4. Text-to-image generation using a generative diffusion model, from equivalent queries in English and French Efficient Inference and Downstream Tasks Especially when it comes to latency-sensitive applications like visually similar recommendations and search, efficient inference is paramount: otherwise, we risk loss of impressions and a poor user experience. We can think of inference along two axes: online inference of the image query and efficient retrieval of top-k most similar items via approximate nearest neighbors. The dimension of the learned visual representation impacts the efficient retrieval design as well, and the smaller 256d derived from the EfficientNetB0 presents an advantage. EfficientNet B0 is hard to beat in terms of accuracy-to-latency trade-offs for online inference, with ~5M parameters and around 1.7ms latency on iPhone 12. The EfficientFormer-l3 has ~30M parameters and gets around 2.7ms latency on iPhone 12 with higher accuracy (while for example MobileViT-XS scores around 7ms with a third of accuracy; very large ViT are not considered since latencies are prohibitive). In offline evaluation, the EfficientFormer-l3-derived embedding achieves around +5% lift in the Intra-L Recall@5 evaluation, a +17% in Intra-R Recall@5, and a +1.8% in Visually Similar Ad clicks Recall@5. We performed A/B testing on the EfficientNetB0 multitask variant across visual applications at Etsy with good results. Additionally, the EfficientFormer-l3 visual representations led to a +0.65% lift in CTR, and a similar lift in purchase rate in a first visually-similar-ads experiment when compared to the production variant of EfficientNetB0. When included in sponsored search downstream rankers, the visual representations led to a +1.26% lift in post-click purchase rate. Including the efficient visual representation in Ads Information Retrieval (AIR), an embedding-based retrieval method used to retrieve similar item ad recommendations caused an increase in click-recall@100 of 8%. And when we used these representations to compute image similarity and included them directly in the last-pass ranking function, we saw a +6.25% lift in clicks. The first use of EfficientNetB0 visual embeddings was in visually similar ad recommendations on mobile. This led to a +1.92% increase in ad return-on-spend on iOS and a +1.18% increase in post-click purchase rate on Android. The same efficient embedding model backed the first search-by-image shopping experience at Etsy. Users search using photos taken with their mobile phone’s camera and the query image embedding is inferred efficiently online, which we discussed in a previous blog post. Learning visual representations is of paramount importance in visually rich e-commerce and online fashion recommendations. Learning them efficiently is a challenging goal made possible by advances in the field of efficient deep learning in computer vision. If you'd like a more in-depth discussion of this work, please see our full accepted paper to the #fashionXrecsys workshop at the Recsys 2023 conference.
At Etsy, we’re focused on elevating the best of our marketplace to help creative entrepreneurs grow their businesses. We continue to invest in making Etsy a safe and trusted place to shop, so sellers’ extraordinary items can shine. Today, there are more than 100 million unique items available for sale on our marketplace, and our vibrant global community is made up of over 90 million active buyers and 7 million active sellers, the majority of whom are women and sole owners of their creative businesses. To support this growing community, our Trust & Safety team of Product, Engineering, Data, and Operations experts are dedicated to keeping Etsy's marketplace safe by enforcing our policies and removing potentially violating or infringing items at scale For that, we make use of community reporting and automated controls for removing this potentially violating content. In order to continue to scale and enhance our detections through innovative products and technologies, we also leverage state-of-the-art Machine Learning solutions which we have already used to identify and remove over 100,000 violations during the past year on our marketplace. In this article, we are going to describe one of our systems to detect policy violations that utilizes supervised learning, a family of algorithms that uses data to train their models to recognize patterns and predict outcomes. Datasets In Machine Learning, data is one of the variables we have the most control over. Extracting data and building trustworthy datasets is a crucial step in any learning problem. In Trust & Safety, we are determined to keep our marketplace and users safe by identifying violations to our policies. For that, we log and annotate potential violations that enable us to collect datasets reliably. In our approach, these are translated into positives, these were indeed violations, and negatives, these were found not to be offending for a given policy. The latter are also known as hard negatives as they are close to our positives and can help us to better learn how to partition these two sets. In addition, we also add easy or soft negatives by adding random items to our datasets. This allows us to give further general examples to our models for listings that do not violate any policy, which is the majority in our marketplace and improve generalizability. The number of easy negatives to add is a hyper-parameter to tune, more will mean higher training time and fewer positive representations. For each training example, we extract multimodal signals, both textual and imagery from our listings. Then, we split our datasets by time using progressive evaluation, to mimic our production usecase and learn to adapt to recent behavior. These are split into training, used to train our models and learn patterns, validation to fine tune our training hyper-parameters such as learning rate and to evaluate over-fitting, and test to report our metrics in an unbiased manner. Model Architecture After usual transformations and extraction of a set of offline features from our datasets, we are all set to start training our Machine Learning model. The goal is to predict whether a given listing violates any of our predefined set of policies, or in contrast, it doesn’t violate any of them. For that, we added a neutral class that depicts the no violation class, where the majority of our listings fall into. This is a typical design pattern for these types of problems. Our model architecture includes a text encoder and an image encoder to learn representations (aka embeddings) for each modality. Our text encoder currently employs a BERT-based architecture to extract context-full representations of our text inputs. In addition, to alleviate compute time, we leverage ALBERT, a lighter BERT with 90% fewer parameters as the transformer blocks share them. Our initial lightweight representation used an in-house model trained for Search usecases. This allowed us to quickly start iterating and learning from this problem. Our image encoder currently employs EfficientNet, a very efficient and accurate Convolutional Neural Network (CNN). Our initial lightweight representation used an in-house model for category classification using CNNs. We are experimenting with transformer-based architectures, similar to our text encoders, with vision transformers but its performance has not been significantly improved. Inspired by EmbraceNet, our architecture then further learns more constrained representations for both text and image embeddings separately, before they are concatenated to form a unique multimodal representation. This is then sent to a final softmax activation that maps logits to probabilities for our internal use. In addition, in order to address the imbalanced nature of this problem, we leverage focal loss that penalizes more hard misclassified examples. Figure 1 shows our model architecture with late concatenation of our text and image encoders and final output probabilities on an example. Model Architecture. Image is obtained from @charlesdeluvio on Unsplash Model Evaluation First, we experimented and iterated by training our model offline. To evaluate its performance, we established certain benchmarks, based on the business goal of minimizing the impact of any well-intentioned sellers while successfully detecting any offending listings in the platform. This results in a typical evaluation trade-off between precision and recall, precision being the fraction of correct predictions over all predictions made, and recall being the fraction of correct predictions over the actual true values. However, we faced the challenge that recall is not possible to compute, as it’s not feasible to manually review the millions and millions of new listings per day so we had to settle for a proxy for recall from what has been annotated. Once we had a viable candidate to test in production, we deployed our model as an endpoint and built a service to perform pre-processing and post-processing steps before and after the call to our endpoint that can be called via an API. Then, we ran an A/B test to measure its performance in production using a canary release approach, slowly rolling out our new detection system to a small percentage of traffic that we keep increasing while we validate an increase in our metrics and no unexpected computation overload. Afterwards, we iterated and every time we had a promising offline candidate, named challenger, that improved our offline performance metrics, we A/B tested it with respect to our current model, named champion. We designed guidelines for model promotion to increase our metrics and our policy coverage. Now, we monitor and observe our model predictions and trigger re-training when our performance degrades. Results Our supervised learning system has been continually learning as we train frequently, run experiments with new datasets and model architectures, A/B test them and deploy them in production. We have added violations as additional classes to our model. As a result, we have identified and removed more than 100,000 violations using these methodologies, in addition to other tools and services that continue to detect and remove violations. This is one of our approaches to identify potentially offending content among others such as explicitly using the policy information and leverage the latest in Large Language Models (LLMs) and Generative AI. Stay tuned! "To infinity and beyond!" –Buzz Lightyear, Toy Story
When Spotify launched in 2008 in Sweden, and in 2011 in the United States, people were amazed that they could access almost the world’s entire music catalog instantaneously. The experience felt like magic and as a result, music aficionados dug in and organized that content into millions of unique playlists. Early on, our users relied on playlists and rudimentary recommendation features like a related artists feature to surface new music. Over time Spotify got more advanced in our recommendations and user-favorite features like Discover Weekly started to significantly improve new music discovery and the overall Spotify experience (more on that ). More users and more features led to more systems that relied on Machine Learning to scale inferences across a growing user base.
In the dynamic landscape of commerce, Shopify merchants rely on our platform's ability to seamlessly and reliably deliver highly relevant products to potential customers. Therefore, a rich and intuitive search experience is an essential part of our offering. Over the past year, Shopify has been integrating AI-powered search capabilities into our merchants’ storefronts. Shopify Storefront Search has transformed the way consumers can shop online. With Semantic Search, we went beyond keyword matching. We improved our understanding of the intent behind a consumer’s search, so that we could match a search with the most relevant products. The net result is helping our merchants boost their sales while offering positive interactive experiences for their consumers. It’s a win-win! Building ML assets with real-time Embeddings Around the same time, Shopify also started investing in creating foundational machine learning (ML) assets. These assets are built as a shared repository of ML primitives which are used as reusable building blocks for more sophisticated AI systems. Shopify Storefront Search is the perfect use case for these ML assets. Complex systems like this need primitives that can transform both text and images into data formats it can process. How do we do that? Enter embeddings, which translate textual and visual content into numerical vectors in a high-dimensional space. This transformation allows us to measure the similarity between different pieces of content, whether text or images, enabling more accurate and context-aware search results. Simplified example of using embeddings to compare search text to result corpus Now that we have a clear idea of the ML primitives that Shopify Storefront Search relies on, let’s take a look at how we send these embedding updates to their systems. Designing ML inference streaming pipelines Currently, Shopify processes roughly 2,500 embeddings per second (or roughly 216 million per day) across our image and text pipelines in near real time. These embeddings include net-new or updated content from across our merchants. Since most of our data platform infrastructure is built on BigQuery,