LLMs

GitHub Copilot X: The AI-powered developer experience

GitHub Copilot is evolving to bring chat and voice interfaces, support pull requests, answer questions, and adopt OpenAI's GPT-4.

How Meta trains large language models at scale

As we continue to focus our AI research and development on solving increasingly complex problems, one of the most significant and challenging shifts we’ve experienced is the sheer scale of co…

GenRec: Towards LLM-Native Recommendation at Netflix

In-House LLM Serving at Netflix

From weeks to a day: how we made LLM evaluation fast enough to iterate on

The architecture of today's LLM applications

Here’s everything you need to know to build your first LLM app and problem spaces you can start exploring today.

Understanding Etsy’s Vast Inventory with LLMs

For more than 20 years, Etsy has been the destination for human creativity online. Our marketplace is home to more than 100 million special items made, handpicked and designed by more than 5 million sellers. These items and the real people behind them are what set us apart. But while the huge variety of Etsy’s inventory is one of our greatest strengths, it also creates fundamental engineering challenges specific to our marketplace. The challenge: Etsy’s unique inventory With millions of creative items across thousands of categories – many of which are unique – it’s difficult to accurately capture all possible product attributes, which range from standard attributes like “color” and “material”, to niche attributes like “bead hole size” and “slime additives.” The range of possible attributes and their values is so broad that it’s a challenge even to enumerate them, let alone label listings with specific attribute data. Unlike other online retailers (that may also have enormous inventories), because products on Etsy are listed by third party sellers and often handmade or customized, we do not have global SKUs (stock keeping units), or mappings from SKUs to product attributes. The listing below is an example of a unique item on Etsy, which has no SKU number or easy access to product attribute information. At first glance, the item looks like a t-shirt, but it is actually a porcelain sculpture. For niche items like this, seller provided details become especially critical. We collect both structured and unstructured data from sellers, and they serve different roles in our marketplace. Unstructured data comes in the form of free-text descriptions, creative titles, and listing photos. While this content is full of useful product information, it’s harder for machines to interpret consistently and quickly at scale. Structured data - in the form of product attributes like size and color - is easy for our systems to parse. It powers the buyer experience through tools such as search filtering options (offered through selectors in UI) and product-to-product comparison for characteristics of interest (material, price, etc). Filters can be seen on the left side after a search query While Etsy does ask sellers to provide structured data on their listings’ attributes, most fields are not required. This reduces friction in the listing process and gives sellers the flexibility to represent their often unique items accurately. Sellers can fill in these attributes, or leave them blank and continue through the listing process As a result, most sellers only or mostly provide unstructured data in the form of listing titles, descriptions, and photos. Frequently, key information like product dimensions is buried in the listing description or only available in listing photos. Example listing with dimensions in description Example of dimensions in a photo While our powerful search and discovery algorithms can process unstructured data such as that in descriptions and listing photos, passing in long context and images directly to search poses latency concerns. For these algorithms, every millisecond counts as they work to deliver relevant results to buyers as quickly as possible. Spending time filtering through unstructured data for every query is just not feasible. These constraints led us to a clear conclusion: to fully unlock the potential of all inventory listed on Etsy’s site, unstructured product information needs to be distilled into structured data to power both ML models and buyer experiences. LLMs present a new opportunity Before the availability of scalable LLMs, we explored various ML-based solutions to this challenge. Supervised product attribute extraction models had limited efficacy; even if we could enumerate all possible product attributes and values, many of them would be so sparse that traditional classification models would struggle to capture the long tail. Sequence tagging approaches also had difficulty scaling to multiple attributes. Transformer-based question-answering models (e.g. AVEQA, MAVEQA ) allowed for generalization to unseen attribute values, but still required large amounts of application-specific training data. This is where the availability of foundational LLMs presented a transformational opportunity for Etsy. These models have a vast amount of general knowledge from pre-training, can process large context windows quickly and affordably, and can follow instructions given a small number of examples. With a feasible and performant solution, our next focus was to build a scalable pipeline that could extract attributes across millions of listings while maintaining confidence in the LLM output. This required robust evaluation frameworks/processes that measured quality through various metrics. Transforming & evaluating unstructured data at scale Evaluation When working with LLMs, one of the biggest challenges is evaluating model performance. We needed to ensure that, at scale across our 100M+ listings, the LLMs were consistently and reliably producing accurate, actionable results. To do this, we initially worked with a third-party labeling vendor to collect a large sample of human-annotated data containing attribute annotations for listings across multiple categories. We evaluated performance by comparing LLM inferences to this human-annotated dataset and calculated metrics like precision, recall, and Jaccard index. We used these ground truth metrics as a benchmark for model improvements via prompt and context engineering. Unfortunately, there were several significant drawbacks to relying on human-labeled data. In many cases, we found that human annotators made mistakes, especially when annotating thousands of listings (after all, no one’s perfect). In the example below, a human annotator marked the light fixture as ½ inch width, while the LLM correctly extracted 5.5 inches. Furthermore, human labeling is more time-consuming and expensive. To start scaling attribute inference across thousands of categories, we needed to come up with an automated process for labeling that did not rely exclusively on human annotation. Instead, we’ve started using high-performance, state-of-the-art LLMs to generate ground truth labels (often called “silver labels”). Human-in-the-loop is still an essential part of this process: Etsy domain experts review silver labels and iterate on the prompt to ensure high quality results. Once we’re confident in our silver label generation, we produce a larger dataset for evaluating a more scalable LLM. The diagram below shows the updated process for model development. Inference The core of our LLM pipeline is context engineering. We’ve worked with partners in product, merchandising, and taxonomy to ensure that the LLM has the right context for attribute extraction, including: Seller-provided listing data, including listing titles, descriptions, and images Few-shot examples hand-selected by domain experts Business logic from Etsy’s product taxonomy Category-specific extraction rules Each listing is represented as a JSON string of context information. This context is injected into a series of prompts to extract product attributes in parallel. LLM requests are routed through LiteLLM to different regions, ensuring higher parallelization and removing a dependency on one singular cloud location. Finally, LLM responses are parsed into Pydantic dataclasses, which provide both basic type validation and custom validation based on business logic. After this process of inference completes, a post-processing job formats the validated, structured outputs. The data is then exported to filestores, database tables, and our search platform for consumption by partner teams. Monitoring Beyond the challenges of evaluating the LLM output, Inference itself may fail for many reasons: code bugs, permissions issues, transient errors, quota exceeded errors, safety filters, and more. Rather than failing the pipeline for any individual error, errors are logged, and error metrics are surfaced via our observability platform. Our team is alerted if the number of failed inferences exceeds a certain threshold. To support debugging, we log a sample of traces to HoneyComb. Even if the error rate is low, it’s possible that model performance has degraded. To track changes in model performance, we added performance evaluation to our pipeline. First, we run LLM inference on a sample of a ground-truth dataset, and calculate performance metrics like precision, and recall. These metrics are compared to baseline scores from the full ground-truth dataset. If any metrics deviate significantly, the pipeline is terminated. This process allows us to confirm that third-party LLMs are working as expected before we run production-scale inference. The combination of tracing, logging, metric tracking, model performance evaluation, and alerting provides a complete understanding of both pipeline health and model performance metrics, enabling us to consistently transform data to power key shopping experiences with confidence at scale. Looking Forward Where we’ve applied LLM-generated product attribute data to buyer and seller-facing experiences, we’ve seen promising results. In target categories, we’ve increased the number of listings with complete attribute coverage from 31% to 91%. And earlier this year, we added LLM-inferred attributes to search filters, leading to more engagement from buyers: Engagement with relevant Search filters increased Overall post-click conversion rate increased All this work combined most recently into leveraging LLM-inferred color attributes to display color swatches for each listing on the search results page. This provides at-a-glance additional information to our buyers to find exactly what they want, faster. What's Next Our goal is to unlock the full potential of Etsy’s inventory. Product attribute extraction is just one of many ways we’re using LLMs to achieve this in our efforts to improve the shopping and selling experience on Etsy. Transforming unstructured information is enabling us to make it easier than ever for our buyers to discover exactly what they’re looking for – and easier for sellers to list and get their unique creations discovered by the shoppers seeking their special item.

Gisting: Compressing LLM Agent context to ↑ throughput and ↓ cost (2026) - Shopify

Gisting compresses context into a set of learned tokens, preserving its quality while making the model faster and cheaper.

Kafka App? There’s a Skill for That

Etsy is home to over 100 million listings from 5.6 million active sellers. Because the items for sale are unique and creative, there is no standard product catalog that tells us what they are. When someone searches for “light linen dress for summer,” our models must infer what the shopper means and what the listings contain. They do that using data from user visits and Etsy listings. We use Kafka for streaming data. These data streams provide fresh features and embeddings that power our machine learning models. If a shopper favorites a hand-painted ceramic mug, that action can shape their recommendations within minutes. But building these streaming applications hasn’t always been a smooth process. A new app requires templated code, configs, tests, validation, deployment steps, and careful review. That developer experience has been a strain on engineers across the company. The Olden Days Before coding agents were broadly available at Etsy, engineers turned machine learning designs into Kafka applications by hand. A streaming feature might require filtering events from terabyte-scale topics, writing transformation logic, and passing outputs through filtered topics to make them available in the feature store. The challenge wasn't writing Java or YAML. It was choosing the right schema, identifying the correct base class, validating input, anticipating edge cases, and running local tests. While static templates could generate files, they couldn't streamline the iterative, time-consuming process. Old process, human developer does everything Kafka Skills to the Rescue Our team provides the infrastructure and platform support for Etsy engineers to create and manage new features and embeddings with Kafka, among other streaming data products. We decided to make this easier with Agent Skills, reusable playbooks that teach language models how to execute specific tasks. For each workflow, the agent uses the Skills and tools we built to generate code, validate configs, run apps locally, and create PRs. Collaborative process between Human developer and Kafka agent The Kafka Skills we created are conversational workflows that allow developers and LLMs to collaborate. They can be invoked with natural language or a slash command such as /new-streaming-feature. Conversation with Claude using Kafka Skill to create a new ML feature For each streaming feature pipeline, the agent creates a Java class that implements a shared interface. The YAML configuration files are fill-in-the-blanks. The agent uses scaffolding scripts, templated code generation, and other tools that we created to output deterministic code. We saw an opportunity to have the Kafka Skills orchestrate all the steps to run an application locally. The human developer no longer needs to create test topics, build JARs, fill in manual commands and read from test topics. The agent will do all that, report back on findings, and generate all the relevant output links. Pull request and verification links posted for the developer’s review Observability We wanted to know which Kafka Skills are being used, how often they make it to pull requests, which environments and application types are involved, and where developers may still need support. To do this, we added lightweight telemetry around the workflow: when a Skill is invoked, a hook fires and a Python helper emits a started event. When the agent opens a pull request, a wrapper emits a completion event. Both events flow through an existing Etsy service that publishes a protobuf message to Kafka and stores it in a BigQuery table. Recording developer use of Kafka Skills in BigQuery Build It and They Won't Come We tested and demoed Kafka Skills and were very pleased with our “Agentic Improvements”. When we looked at adoption rates though, we were surprised to find that developers weren’t using them. They weren’t using the Skills to create features, generate embeddings, or even turn Kafka topic records into HTTP requests. Feeling a bit defeated, we asked devs: why? They responded that they simply didn’t know about them. The devs were using coding agents, but the agents never offered the Kafka Skills. Before the Skills existed, the standard workflow for feature creation was to find the closest existing feature, copy the code for it, and modify the fields. Instead of starting the process by opening the LLM and saying "I'd like to create a new streaming feature," an engineer might engage with the agent halfway through the old process and ask to "copy this class and change it to count cart moves," or "edit this file" with a path lifted from a previous pull request. Another challenge we found was that requests often arrive in whatever vocabulary the developer's team uses. When creating a new feature with Kafka, the dev might refer to a specific feature-store entry, an internal event name, a parent Java class name needed, or an output data type. To address these issues, we enumerated entry points for different vocabulary, lifecycle stages, and copy-paste-shaped requests. Plus, we added repository-level routing so that touching related files is itself a trigger for the Kafka Skill. The developer describes the task however they naturally would, and the agent offers the paved path. Guardrails In developing the Kafka Skills, we hoped to prevent agents from inventing Kafka applications from scratch. The agent handles the ambiguous parts: understanding the developer's request, identifying source schemas and fields, and deciding which validation steps to run. The agent then passes structured inputs into versioned scaffolding scripts and validation tools. Anthropic describes this pattern in its tool-use documentation: the agent returns a structured tool call that application-owned code executes, and strict tool use can constrain those inputs to a JSON schema. The agent handles the orchestration, while our scripts and hooks generate the templated code. Security research reinforced that decision. In a 2023 Stanford study, developers using an assistant wrote less secure code and were more likely to think their code was secure. BaxBench found a similar gap between code that runs and code that’s safe: across 392 backend tasks and 11 models, the best model was functionally correct 62% of the time, but no model produced code that was both correct and secure on more than 37% of the tasks. Researchers could exploit about half of the functionally correct programs. Now that we had agents running commands and committing code on our behalf, we also found that we needed to tighten Kafka ACLs, repository permissions, and deployment controls. We designed the system around least privilege: agents only get the access they need for the task they are performing. This was to make sure that neither human nor agent could delete an important topic, write to an unexpected one, change an unrelated pipeline, or bypass the normal production review path. What’s Next? Kafka supports a wide range of critical data workflows at Etsy, including machine learning feature generation, embeddings, filtering, fan-out, and storing data. We’ve built Skills for seven Kafka workflows so far. Adoption is the near-term focus: we're continuing to add entry points so the Skills are offered no matter how a developer phrases the task. Among our earliest insights from the observability table will be whether the discoverability fixes described above worked, and we’ll continue to iterate as we learn. We built the Kafka Skills to fix a developer-experience problem: too many steps stood between an idea for an ML model and a working pipeline with all the data. Our machine learning models shape the Etsy user experience for buyers and sellers. Each streaming feature that ships sooner helps our models understand a little earlier what shoppers mean and what listings contain. That means more fresh signals behind every search for a light linen dress, and behind the next hand-painted ceramic mug someone favorites. Acknowledgements A huge thanks to Kamya Shethia, Shannon Hoang, Nicollette Lui, Andrey Polyakov, and Virginia Poundstone for all their hard work designing and building Kafka Skills and agent tools, setting up observability, as well as locking down access control to make the world safer for robots and humans alike.

GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

Meta’s Generative Ads Recommendation Model (GEM), the foundation model behind ads recommendations across Instagram and Facebook, now trains at LLM scale on several thousand of the latest-generation GPUs. This post goes into the details on how we achieved: doubling end-to-end (E2E) training efficiency to 20–25% Model FLOPs Utilization (MFU) while scaling training FLOPs 4x in [...] Read More... The post GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model appeared first on Engineering at Meta.

Slack AI: The Path to Multi-Cloud

In early 2023, Slack faced a foundational challenge: serving Large Language Models (LLMs) at enterprise scale with the security, reliability, and performance our customers expect. Over three years, we evolved from basic infrastructure to orchestrating a sophisticated multi-cloud architecture. We didn’t just want shiny new models; we needed a system resilient to regional outages and…

Flow generation through natural language: An agentic modeling approach (2026) - Shopify

We fine-tuned Qwen3-32B into a tool-calling agent that generates Flow automations: 2.2x faster, 68% cheaper, with a weekly retraining flywheel.

GenPage: Towards End-to-End Generative Homepage Construction at Netflix

Teaching Sidekick to say no: automated data curation with LLM judge consensus (2026) - Shopify

Our Sidekick team used LLM judges to automatically curate training data and teach domain-specific models how to refuse impossible queries.

How we used DSPy to turn AI evaluations into better responses in Dash chat

We used DSPy to improve LLM judges and optimize our chat experience, creating an evaluation-driven feedback loop that produced better outputs.

Efficient Visual Representation Learning And Evaluation

Etsy features a diverse marketplace of unique handmade and vintage items. It’s a visually diverse marketplace as well, and computer vision has become increasingly important to Etsy as a way of enhancing our users’ shopping experience. We’ve developed applications like visual search and visually similar recommendations that can offer buyers an additional path to find what they’re looking for, powered by machine learning models that encode images as vector representations. Figure 1. Visual representations power applications such as visual search and visually similar recommendations Learning expressive representations through deep neural networks, and being able to leverage them in downstream tasks at scale, is a costly technical challenge. The infrastructure required to train and serve large models is expensive, as is the iterative process that refines them and optimizes their performance. The solution is often to train deep learning architectures offline and use the pre-computed pretrained visual representations in downstream tasks served online. (We wrote about this in a previous blog post on personalization from real-time sequences and diversity of representations.) In any application where a query image representation is inferred online, it's important that you have low latency, memory-aware models. Efficiency becomes paramount to the success of these models in the product. We can think about efficiency in deep learning along multiple axes: efficiency in model architecture, model training, evaluation and serving. Model Architecture The EfficientNet family of models features a convolutional neural network architecture. It uniformly optimizes for network width, depth, and resolution using a fixed set of coefficients. By allowing practitioners to start from a limited resource budget and scale up for better accuracy as more resources are available, EfficientNet provides a great starting point for visual representations. We began our trials with EfficientNetB0, the smallest size model in the EfficientNet family. We saw good performance and low latency with this model, but the industry and research community have touted Vision Transformers (ViT) as having better representations. We decided to give that a try. Transformers lack the spatial inductive biases of CNN, but they outperform CNN when trained on large enough datasets and may be more robust to domain shifts. ViT decomposes the image into a sequence of patches (16X16 for example) and applies a transformer architecture to incorporate more global information. However, due to the massive number of parameters and compute-heavy attention mechanism, ViT-based architectures can be many times slower to train and inference than lightweight Convolutional Networks. Despite the challenges, more efficient ViT architectures have recently begun to emerge, featuring clever pooling, layer dropping, efficient normalization, and efficient attention or hybrid CNN-transformer designs. We employ the EfficientFormer-l3 to take advantage of these ViT improvements. The EfficientFormer architecture achieves efficiency through downsampling multiple blocks and employing attention only in the last stage. This derived image representation mechanism differs from the standard vision transformer, where embeddings are extracted from the first token of the output. Instead, we extract the attention from the last block for the eight heads and perform average pooling over the sequence. In Figure 2 we illustrate these different attention weights with heat maps overlaid on an image, showing how each of the eight heads learns to focus on a different salient part. Figure 2. Probing the EfficientFormer-l3 pre-trained visual representations through attention heat maps. Model Training Fine-Tuning With our pre-trained backbones in place, we can gain further efficiencies via fine tuning. For the EfficientNetB0 CNN, that means replacing the final convolutional layer and attaching a d-dimensional embedding layer followed by m classification heads, where m is the number of tasks. The embedding head consists of a new convolutional layer with the desired final representation dimension, followed by a batch normalization layer, a swish activation and a global average pooling layer to aggregate the convolutional output into a single vector per example. To train EfficientNetB0, new attached layers are trained from scratch for one epoch with the backbone layers frozen, to avoid excessive computation and overfitting. We then unfreeze 75 layers from the top of the backbone and finetune for nine additional epochs, for efficient learning. At inference time we remove the classification head and extract the output of the pooling layer as the final representation. To fine-tune the EfficientFormer ViT we stick with the pretraining resolution of 224X224, since using sequences longer than the recommended 384X384 in ViT leads to larger training budgets. To extract the embedding we average pool the last hidden state. Then classification heads are added as with the CNN, with batch normalization being swapped for layer normalization. Multitask Learning In a previous blog post we described how we built a multitask learning framework to generate visual representations for Etsy's search-by-image experience. The training architecture is shown in Figure 3. Figure 3. A multitask training architecture for visual representations. The dataset sampler combines examples from an arbitrary number of datasets corresponding to respective classification heads. The embedding is extracted before the classification heads. Multitask learning is an efficiency inducer. Representations encode commonalities, and they perform well in diverse downstream tasks when those are learned using common attributes as multiple supervision signals. A representation learned in single-task classification to the item’s taxonomy, for example, will be unable to capture visual attributes: colors, shapes, materials. We employ four classification tasks: a top-level taxonomy task with 15 top-level categories of the Etsy taxonomy tree as labels; a fine-grained taxonomy task, with 1000 fine-grained leaf node item categories as labels; a primary color task; and a fine-grained taxonomy task (review photos), where each example is a buyer-uploaded review photo of a purchased item with 100 labels sampled from fine-grained leaf node item categories. We are able to train both EfficientNetB0 and EfficientFormer-l3 on standard 16GB GPUs (we used two P100 GPUs). For comparison, a full sized ViT requires a larger 40GB RAM GPU such as an A100, which can increase training costs significantly. We provide detailed hyperparameter information for fine-tuning either backbone in our article. Evaluating Visual Representations We define and implement an evaluation scheme for visual representations to track and guide model training, on three nearest neighbor retrieval tasks. After each training epoch, a callback is invoked to compute and log the recall for each retrieval task. Each retrieval dataset is split into two smaller datasets: “queries” and “candidates.” The candidates dataset is used to construct a brute-force nearest neighbor index, and the queries dataset is used to look up the index. The index is constructed on the fly after each epoch to accommodate for embeddings changing between training epochs. Each lookup yields K nearest neighbors. We compute Recall@5 and @10 using both historical implicit user interactions (such as “visually-similar ad clicks”) and ground truth datasets of product photos taken from the same listing (“intra-item”). The recall callbacks can also be used for early stopping of training to enhance efficiency. The intra-item retrieval evaluation dataset consists of groups of seller-uploaded images of the same item. The query and candidate examples are randomly selected seller-uploaded images of an item. A candidate image is considered a positive example if it is associated with the same item as the query. In the “intra-item with reviews” dataset, the query image is a randomly selected buyer-uploaded review image of an item, with seller-uploaded images providing candidate examples. The dataset of visually similar ad clicks associates seller-uploaded primary images with primary images of items that have been clicked in the visually similar surface on mobile. Here, a candidate image is considered a positive example for some query image if a user viewing the query image has clicked it. Each evaluation dataset contains 15,000 records for building the index and 5,000 query images for the retrieval phase. We also leverage generative AI for an experimental new evaluation scheme. From ample, multilingual historical text query logs, we build a new retrieval dataset that bridges the semantic gap between text-based queries and clicked image candidates. Text-to-image generative stable diffusion makes the information retrieval process language-agnostic, since an image is worth a thousand (multilingual) words. A stable diffusion model generates high-quality images which become image queries. The candidates are images from clicked items corresponding to the source text query in the logs. One caveat is that the dataset is biased toward the search-by-text production system that produced the logs; only a search-by-image-from-text system would produce truly relevant evaluation logs. The source-candidate image pairs form the new retrieval evaluation dataset which is then used within a retrieval callback. Of course, users entering the same text query may have very different ideas in mind of, say, the garment they’re looking for. So for each query we generate several images: formally, a random sample of length 𝑛 from the posterior distribution over all possible images that can be generated from the seed text query. We pre-condition our generation on a uniform “fashion style.” In a real-world scenario, both the text-to-image query generation and the image query inference for retrieval happen in real time, which means efficient backbones are necessary. We randomly select one of the 𝑛 generated images to replace the text query with an image query in the evaluation dataset. This is a hybrid evaluation method: the error inherent in the text-to-image diffusion model generation is encapsulated in the visually similar recommendation error rate. Future work may include prompt engineering to improve the text query prompt itself, which as input by the user can be short and lacking in detail. Large memory requirements and high inference latency are challenges in using text-to-image generative models at scale. We employ an open source fast stable diffusion model through token merging and float 16 inference. Compared to the standard stable diffusion implementation available at the time we built the system, this method speeds up inference by 50% with a 5x reduction in memory consumption, though results depend on the underlying patched model. We can generate 500 images per hour with one T4 GPU (no parallelism) using the patched stable diffusion pipeline. With parallelism we can achieve further speedup. Figure 4 shows that for the English text query “black bohemian maxi dress with orange floral pattern” the efficient stable diffusion pipeline generates five image query candidates. The generated images include pleasant variations with some detail loss. Interestingly, mostly the facial details of the fashion model are affected, while the garment pattern remains clear. In some cases degradation might prohibit display, but efficient generative technology is being perfected at a fast pace, and prompt engineering helps the generative process as well. Figure 4. Text-to-image generation using a generative diffusion model, from equivalent queries in English and French Efficient Inference and Downstream Tasks Especially when it comes to latency-sensitive applications like visually similar recommendations and search, efficient inference is paramount: otherwise, we risk loss of impressions and a poor user experience. We can think of inference along two axes: online inference of the image query and efficient retrieval of top-k most similar items via approximate nearest neighbors. The dimension of the learned visual representation impacts the efficient retrieval design as well, and the smaller 256d derived from the EfficientNetB0 presents an advantage. EfficientNet B0 is hard to beat in terms of accuracy-to-latency trade-offs for online inference, with ~5M parameters and around 1.7ms latency on iPhone 12. The EfficientFormer-l3 has ~30M parameters and gets around 2.7ms latency on iPhone 12 with higher accuracy (while for example MobileViT-XS scores around 7ms with a third of accuracy; very large ViT are not considered since latencies are prohibitive). In offline evaluation, the EfficientFormer-l3-derived embedding achieves around +5% lift in the Intra-L Recall@5 evaluation, a +17% in Intra-R Recall@5, and a +1.8% in Visually Similar Ad clicks Recall@5. We performed A/B testing on the EfficientNetB0 multitask variant across visual applications at Etsy with good results. Additionally, the EfficientFormer-l3 visual representations led to a +0.65% lift in CTR, and a similar lift in purchase rate in a first visually-similar-ads experiment when compared to the production variant of EfficientNetB0. When included in sponsored search downstream rankers, the visual representations led to a +1.26% lift in post-click purchase rate. Including the efficient visual representation in Ads Information Retrieval (AIR), an embedding-based retrieval method used to retrieve similar item ad recommendations caused an increase in click-recall@100 of 8%. And when we used these representations to compute image similarity and included them directly in the last-pass ranking function, we saw a +6.25% lift in clicks. The first use of EfficientNetB0 visual embeddings was in visually similar ad recommendations on mobile. This led to a +1.92% increase in ad return-on-spend on iOS and a +1.18% increase in post-click purchase rate on Android. The same efficient embedding model backed the first search-by-image shopping experience at Etsy. Users search using photos taken with their mobile phone’s camera and the query image embedding is inferred efficiently online, which we discussed in a previous blog post. Learning visual representations is of paramount importance in visually rich e-commerce and online fashion recommendations. Learning them efficiently is a challenging goal made possible by advances in the field of efficient deep learning in computer vision. If you'd like a more in-depth discussion of this work, please see our full accepted paper to the #fashionXrecsys workshop at the Recsys 2023 conference.

Leveraging multimodal LLMs for Shopify’s global catalogue: Recap of expo talk at ICLR 2025 - Shopify

Shopify’s Global Catalogue demonstrates the impact of multimodal LLMs on one of commerce’s hardest problems: building a unified, structured, and continuously evolving understanding of billions of product listings created by millions of merchants. At the International Conference on Learning Representations (ICLR) 2025 Expo in Singapore, we presented the core engineering behind this initiative

Machine Learning in Content Moderation at Etsy

At Etsy, we’re focused on elevating the best of our marketplace to help creative entrepreneurs grow their businesses. We continue to invest in making Etsy a safe and trusted place to shop, so sellers’ extraordinary items can shine. Today, there are more than 100 million unique items available for sale on our marketplace, and our vibrant global community is made up of over 90 million active buyers and 7 million active sellers, the majority of whom are women and sole owners of their creative businesses. To support this growing community, our Trust & Safety team of Product, Engineering, Data, and Operations experts are dedicated to keeping Etsy's marketplace safe by enforcing our policies and removing potentially violating or infringing items at scale For that, we make use of community reporting and automated controls for removing this potentially violating content. In order to continue to scale and enhance our detections through innovative products and technologies, we also leverage state-of-the-art Machine Learning solutions which we have already used to identify and remove over 100,000 violations during the past year on our marketplace. In this article, we are going to describe one of our systems to detect policy violations that utilizes supervised learning, a family of algorithms that uses data to train their models to recognize patterns and predict outcomes. Datasets In Machine Learning, data is one of the variables we have the most control over. Extracting data and building trustworthy datasets is a crucial step in any learning problem. In Trust & Safety, we are determined to keep our marketplace and users safe by identifying violations to our policies. For that, we log and annotate potential violations that enable us to collect datasets reliably. In our approach, these are translated into positives, these were indeed violations, and negatives, these were found not to be offending for a given policy. The latter are also known as hard negatives as they are close to our positives and can help us to better learn how to partition these two sets. In addition, we also add easy or soft negatives by adding random items to our datasets. This allows us to give further general examples to our models for listings that do not violate any policy, which is the majority in our marketplace and improve generalizability. The number of easy negatives to add is a hyper-parameter to tune, more will mean higher training time and fewer positive representations. For each training example, we extract multimodal signals, both textual and imagery from our listings. Then, we split our datasets by time using progressive evaluation, to mimic our production usecase and learn to adapt to recent behavior. These are split into training, used to train our models and learn patterns, validation to fine tune our training hyper-parameters such as learning rate and to evaluate over-fitting, and test to report our metrics in an unbiased manner. Model Architecture After usual transformations and extraction of a set of offline features from our datasets, we are all set to start training our Machine Learning model. The goal is to predict whether a given listing violates any of our predefined set of policies, or in contrast, it doesn’t violate any of them. For that, we added a neutral class that depicts the no violation class, where the majority of our listings fall into. This is a typical design pattern for these types of problems. Our model architecture includes a text encoder and an image encoder to learn representations (aka embeddings) for each modality. Our text encoder currently employs a BERT-based architecture to extract context-full representations of our text inputs. In addition, to alleviate compute time, we leverage ALBERT, a lighter BERT with 90% fewer parameters as the transformer blocks share them. Our initial lightweight representation used an in-house model trained for Search usecases. This allowed us to quickly start iterating and learning from this problem. Our image encoder currently employs EfficientNet, a very efficient and accurate Convolutional Neural Network (CNN). Our initial lightweight representation used an in-house model for category classification using CNNs. We are experimenting with transformer-based architectures, similar to our text encoders, with vision transformers but its performance has not been significantly improved. Inspired by EmbraceNet, our architecture then further learns more constrained representations for both text and image embeddings separately, before they are concatenated to form a unique multimodal representation. This is then sent to a final softmax activation that maps logits to probabilities for our internal use. In addition, in order to address the imbalanced nature of this problem, we leverage focal loss that penalizes more hard misclassified examples. Figure 1 shows our model architecture with late concatenation of our text and image encoders and final output probabilities on an example. Model Architecture. Image is obtained from @charlesdeluvio on Unsplash Model Evaluation First, we experimented and iterated by training our model offline. To evaluate its performance, we established certain benchmarks, based on the business goal of minimizing the impact of any well-intentioned sellers while successfully detecting any offending listings in the platform. This results in a typical evaluation trade-off between precision and recall, precision being the fraction of correct predictions over all predictions made, and recall being the fraction of correct predictions over the actual true values. However, we faced the challenge that recall is not possible to compute, as it’s not feasible to manually review the millions and millions of new listings per day so we had to settle for a proxy for recall from what has been annotated. Once we had a viable candidate to test in production, we deployed our model as an endpoint and built a service to perform pre-processing and post-processing steps before and after the call to our endpoint that can be called via an API. Then, we ran an A/B test to measure its performance in production using a canary release approach, slowly rolling out our new detection system to a small percentage of traffic that we keep increasing while we validate an increase in our metrics and no unexpected computation overload. Afterwards, we iterated and every time we had a promising offline candidate, named challenger, that improved our offline performance metrics, we A/B tested it with respect to our current model, named champion. We designed guidelines for model promotion to increase our metrics and our policy coverage. Now, we monitor and observe our model predictions and trigger re-training when our performance degrades. Results Our supervised learning system has been continually learning as we train frequently, run experiments with new datasets and model architectures, A/B test them and deploy them in production. We have added violations as additional classes to our model. As a result, we have identified and removed more than 100,000 violations using these methodologies, in addition to other tools and services that continue to detect and remove violations. This is one of our approaches to identify potentially offending content among others such as explicitly using the policy information and leverage the latest in Large Language Models (LLMs) and Generative AI. Stay tuned! "To infinity and beyond!" –Buzz Lightyear, Toy Story

Using LLMs to amplify human labeling and improve Dash search relevance

How we train Dash's search ranking models with a mix of human and LLM-assisted labeling.

Context engineering case studies: Etsy-specific question answering

This post investigates the benefits and limitations of prompt engineering in two instances of AI-assisted onboarding relying on large language model (LLM) technology. Of particular interest is how truthful (and therefore reliable) LLM-generated answers turn out to be in the context of Etsy-specific question answering. Among other insights, we find that asking the LLM to identify specific source snippets is a good way to flag potential hallucinations. Over the past few years, pre-trained large-scale/foundation language models such as OpenAI’s o-series [1] and Google’s Gemini family [2] have revolutionized the field of natural language processing (NLP). Trained on vast amounts of text, images, code, audio, and videos, such models encapsulate a great deal of world knowledge, which can be called upon to perform a wide range of downstream tasks, such as sentiment analysis, language translation, and natural language inference, among many others. The canonical way to improve the task performance of a pre-trained general language model, when it needs specific knowledge beyond its original training, is called fine-tuning [3]. The model in a sense receives supplementary training, typically applied only in its last layers, on a relevant, labeled dataset. This is a robust but somewhat expensive approach, and only possible when there’s a sufficiently large dataset to support it. With the greatly increased size and complexity of LLMs, though, prompt-based tuning has emerged as an effective and much less costly alternative. The parameters of the model remain unchanged, and instead it’s through the prompt—the instructions given the model to perform its task—that tuning is done. At its simplest a prompt is just a question, but prompts can also incorporate specific chunks of text, provide the model with task-related constraints or preferences, and in general shape the model toward producing reliable responses. Prompt engineering encompasses the art and science of tuning prompts so they yield optimal results on a given task. One of the places at Etsy where an assistive AI could be useful is onboarding: in principle, an LLM could answer a range of questions about Etsy’s both internal and external policies and procedures. But rather than ramp up an effort to fine-tune an onboarding model with a dataset of Etsy documents, we wanted to assess whether prompts alone could elicit truthful answers. We decided to start with a limited-scale pilot project, focusing on answering questions specific to the Travel & Entertainment (T&E) domain. On the one hand, this is a well circumscribed domain with clear and unambiguous rules. On the other hand, most Etsy employees still have questions for every trip. This article summarizes our journey of discovery, and discusses in particular the extent to which the addition of simple explicit instructions works to mitigate LLM errors. Question answering Perhaps the most critical aspect of a question answering system is its reliability, i.e., whether it is able to provide a truthful answer to any in-domain question. In the AI-assisted onboarding use case considered here, we want new Etsy employees to be able to be confident that their questions about the T&E policy are answered correctly. The first step was to feed Etsy-specific data into the LLM. If we were fine-tuning, we would update (a subset of) model weight parameters from an appropriate collection of Etsy internal documents related to T&E. Prompt engineering, on the other hand, freezes the model weights, treating the LLM as a black box. A number of such black-box tuning techniques exist in the field, which we review in the Appendix to this article. Prompt-based tuning was an attractive proposition in our case because all that it required was simply an adequate representation of task-specific documents. We therefore folded all T&E documents into the rich latent space of the LLM (also known as the embedding space). In practice, this can be done efficiently by simply updating the underlying index to account for the newly folded data. Then, as indicated in the Appendix, question answering proceeded by leveraging embedding-based search [6]. For every query, embedding the initial formulation led to relevant Etsy-specific text sections, which were then added to the question to form the final prompt. We tested the resulting system’s performance on a manually curated set of 40 question-and-answer pairs. For each question in the test set, we compared the answer generated by the LLM with the answer we had extracted from the relevant policy document to form a judgment of answer quality. In the authors’ estimation, the LLM answered approximately 86% of the questions correctly, or at least satisfactorily enough that there was no need for further prompt maneuvering. In the remaining 14% of cases, the LLM generated an answer which was either wrong or misleading. For example, the LLM asserted with high confidence that it's the cardholder who is responsible for the balance on a corporate credit card: Q: Who pays the balance on my corporate card after my expense report is approved? Correct answer: Etsy pays the corporate card company directly on behalf of the Employee. LLM answer: The cardholder is responsible for paying the balance on their corporate card after their expense report is approved. The statement is wrong under the terms of the corporate card agreement. Similarly, the LLM falsely claimed that Etsy doesn't reimburse the cost of checking bags on a business trip (noting in its answer that "the context information provided does not mention any expenses related to checking bags," another falsehood). Mitigating hallucinations The kind of LLM falsehoods illustrated in the previous section–wrong, but confidently stated–have become known as “hallucinations.” Prompt engineering offers us a number of ways of addressing and overcoming LLM hallucinations. We can craft narrower and more explicit input instructions. We can augment prompts with relevant contextual information. We can even ask the LLM to answer as if it occupied a particular role, say, data analyst or financial manager (role prompting). In the case of the above hallucination about our corporate cards, we first attempted the explicit-instruction route, asking the LLM to be clear about what it didn’t know: Q: Say "I have no idea" when you don't know the answer. Who pays the balance on my corporate card after my expense report is approved? A: I have no idea. We at least avoid the hallucination, but not in a way that will be very helpful to the user. Also note that the LLM misses the correct answer, which is actually present in the collection of documents. Another try, prompting the LLM for chain-of-thought reasoning, finally led to the correct answer: Q: Say "I have no idea" when you don't know the answer. Who pays the balance on my corporate card after my expense report is approved, and why do you think so? A: The balance on your corporate card is paid by Etsy. This information is mentioned on page 42. The above is a good illustration of how three different prompts for the same question can lead to widely divergent answers. When the internal state of the LLM is confused about a particular topic, letting it express uncertainty in the answer can often mitigate undue overconfidence. In addition, prompting for chain-of-thought reasoning is often beneficial, as it provides a way to structure the prompt with logical milestones to ensure that the LLM follows a well-circumscribed reasoning path. While it may not seem very intuitive that just adding "why do you think so" should change an all-but unresponsive answer into a correct one, the mere fact of asking for an explanation raises the bar in terms of fact checking – a side benefit of prompt engineering which doesn't require additional investment. Etsy community forums Encouraged by those findings, we decided to expand our investigation to an external use case: question answering based on historical data from Etsy seller community forums. Unlike the T&E data we’ve just seen, Etsy forums contain publicly available content. Sellers ask questions such as: “What are the most effective ways to optimize my Etsy shop’s SEO to increase visibility and sales?” These questions are then answered by either Etsy or community members. As a result, compared to the T&E data, Etsy forum data is substantially more complex: much less homogeneous (both in terms of style and scope), with answers by nature much more opinionated. As such,assessing the accuracy of LLM-based question answering is a little trickier for community forums than for the T&E policy. To note, because forums are a space where sellers engage with and offer information to one another, there may be some divergence between answers provided in the forums and official Etsy policies & practices. For the purposes of our investigation, we measured only whether the LLM could accurately surface responses based on the input data it was given. A high score would indicate a helpful tool not only at seller onboarding time, but also to help both Etsy and community members routinely search this growing community database in a faster and more accurate fashion. We proceeded exactly as before, first folding recent Etsy forum data into the latent space and then updating the underlying index to account for the newly folded data. Similar to the previous case study, we tested performance on a manually curated set of 50 question-and-answer pairs, comparing LLM-generated answers with answers extracted from the relevant Etsy forum to form a judgment of answer quality. In the authors’ estimation, the LLM answered approximately 72% of the questions correctly before any prompt maneuvering. In the remaining 28% of cases, the LLM generated an answer which was either wrong or misleading. Given the higher heterogeneity of the data, it is not overly surprising that the percentage of poor answers doubles. Comparing manually extracted and automatically generated answers is informative, not just from the point of view of seeing natural language outcomes, but also from the perspective of analyzing how such outcomes are pieced together from tidbits of information in latent space. Everything else being equal, the LLM tends to do better when a query formulation is close to the exact wording used in reference documents, and worse when the answer depends on multiple conditions being satisfied in turn. In a handful of cases when the LLM stayed relatively vague and avoided providing specific information, enabling detailed query expansion occasionally led to an hallucination. This observation has concrete implications when it comes to chain-of-thought prompting: specific logical milestones must be congruent with query expansion chunks to maximize alignment. Limitations The forums Q&A use case also helped surface some of the limitations of prompt engineering. For that data, we observed that the approach advocated earlier (allowing for uncertainty, providing more context, asking for an explanation) was not always effective. For example, we initially thought that the following case: Q: Help, where’s the download for 1099-K? There’s NO download button! A: It is not possible to answer this question without more information. It is unclear what the 1099-K is, what it is used for, and where it can be downloaded from. ...would be readily addressed through more advanced prompt engineering focusing on additional contextual information. However, adding a contextual preamble did not help: Q: In the United States, Form 1099-K "Payment Card and Third Party Network Transactions" is a variant of Form 1099 used to report payments received through reportable payment card transactions and/or settlement of third-party payment network transactions. Help, where’s the download for 1099-K? There’s NO download button! A: The context information provided does not include any information about downloading Form 1099-K. To find the download button for Form 1099-K, you will need to search for it on the website of the Internal Revenue Service (IRS) or other relevant government agency. The preamble “In the United States, Form 1099-K [...] transactions.” was clearly not sufficient to lead to a useful answer to the original question. Other times, the LLM punted with the expression: "Without prior knowledge, it is impossible to determine..." -- which was untrue, since the information was indeed available. We conclude that, even with the current crop of advanced reasoning models, when complex reasoning is required, chain-of-thought prompting must be carefully set up. In such situations, what seemed to help most was asking for specific source snippets, to determine where in the forums the LLM was looking for answers. Key learnings Our experiments with two flavors of AI-assisted onboarding (internal and external) indicate that, most of the time, prompt engineering works reasonably well as a practical alternative to parameter-based fine-tuning. Prompt engineering is conceptually simple and can often be leveraged to fix cases where LLMs generate non-factual information. It has limitations, however, so care should still be taken when assessing answer truthfulness. Asking the LLM to disclose specific sources is a good way to flag potential hallucinations. References [1] OpenAI blog. “Introducing OpenAI o3 and o4-mini.” 16 April 2025. [2] Google DeepMind blog. “Gemini 2.5: Our most intelligent AI model.” 25 March 2025. [3] Google Cloud Documentation. “Tune Gemini models by using supervised fine-tuning.” [4] T. Brown et al. (2020). “Language models are few-shot learners.” Proc. 2020 NeurIPS, Vol. 33, pp. 1877-1901. [5] T. Gao, A. Fisch, and D. Chen (2021). “Making pre-trained language models better few-shot learners.” Proc. 2021 ACL/IJCNLP, pp. 3816-2830. [6] Google AI for Developers. “Embeddings.” [7] Google AI for Developers. “Prompt design strategies.” Appendix Various instantiations of prompt-based tuning have evolved over the years. The purpose of this Appendix is to give a little historical background and perspective on the set of practices that have come to be known, collectively, as prompt engineering. Prompt-based tuning originally developed as an alternative form of model fine-tuning for situations where only few-shot demonstration examples are available for downstream tasks [4]. In such situations, prompt tuning makes it possible to use the same robust model for all tasks, as opposed to producing a separate, more brittle model for each one. This approach typically treats the model as a black box and encodes either the input as trainable sequences or the output as training features. It has been shown to drastically outperform conventional transfer learning procedures in the low resource setting [5]. More recently, prompt engineering has gained further prominence in the literature as a way to circumvent parameter-based fine-tuning of LLMs. Such fine-tuning is extremely challenging with an LLM, as any backpropagation through the entire model requires substantial computational resources. With LLMs being trained on an ever increasing amount and diversity of data, leading to an increasingly richer latent space, there is correspondingly less need to do the hard and costly work of updating model weights. Prompt-based tuning only requires that task-specific documents be ingested and folded into that rich latent space. This can be done efficiently by simply updating the underlying index to account for the newly folded data. Once this is done, tuning typically relies on embedding-based search [6]: every prompt is converted to a suitable embedding in latent space by the embeddings API, text sections generally relevant to the prompt are identified using similarity in embedding space, the prompt is augmented with the most relevant text sections, and the LLM responds accordingly. In this context, prompt engineering can be viewed as a form of query expansion, a well-established process to improve information retrieval. On top of that, further tuning often involves the equivalent of query reformulation, typically achieved through instruction prompting, role prompting, and few-shot prompting. For a list of best practices in this area, see "Prompt design strategies" [7].

Scaling Real-Time Traffic Forecasting with a Graph-Aware Transformer