Topic
29 articles
As we continue to focus our AI research and development on solving increasingly complex problems, one of the most significant and challenging shifts we’ve experienced is the sheer scale of co…
Most of what an AI coding agent does for me isn't thinking. It's I/O. The post Portal by Spotify cut my Claude Code token usage by 90% appeared first on Spotify Engineering.
In this post, learn how Shopify's Sidekick solves markdown rendering jank and response delay in LLM chatbots with buffering parser and async content resolution.
Gisting compresses context into a set of learned tokens, preserving its quality while making the model faster and cheaper.
Here’s everything you need to know to build your first LLM app and problem spaces you can start exploring today.
Etsy is home to over 100 million listings from 5.6 million active sellers. Because the items for sale are unique and creative, there is no standard product catalog that tells us what they are. When someone searches for “light linen dress for summer,” our models must infer what the shopper means and what the listings contain. They do that using data from user visits and Etsy listings. We use Kafka for streaming data. These data streams provide fresh features and embeddings that power our machine learning models. If a shopper favorites a hand-painted ceramic mug, that action can shape their recommendations within minutes. But building these streaming applications hasn’t always been a smooth process. A new app requires templated code, configs, tests, validation, deployment steps, and careful review. That developer experience has been a strain on engineers across the company. The Olden Days Before coding agents were broadly available at Etsy, engineers turned machine learning designs into Kafka applications by hand. A streaming feature might require filtering events from terabyte-scale topics, writing transformation logic, and passing outputs through filtered topics to make them available in the feature store. The challenge wasn't writing Java or YAML. It was choosing the right schema, identifying the correct base class, validating input, anticipating edge cases, and running local tests. While static templates could generate files, they couldn't streamline the iterative, time-consuming process. Old process, human developer does everything Kafka Skills to the Rescue Our team provides the infrastructure and platform support for Etsy engineers to create and manage new features and embeddings with Kafka, among other streaming data products. We decided to make this easier with Agent Skills, reusable playbooks that teach language models how to execute specific tasks. For each workflow, the agent uses the Skills and tools we built to generate code, validate configs, run apps locally, and create PRs. Collaborative process between Human developer and Kafka agent The Kafka Skills we created are conversational workflows that allow developers and LLMs to collaborate. They can be invoked with natural language or a slash command such as /new-streaming-feature. Conversation with Claude using Kafka Skill to create a new ML feature For each streaming feature pipeline, the agent creates a Java class that implements a shared interface. The YAML configuration files are fill-in-the-blanks. The agent uses scaffolding scripts, templated code generation, and other tools that we created to output deterministic code. We saw an opportunity to have the Kafka Skills orchestrate all the steps to run an application locally. The human developer no longer needs to create test topics, build JARs, fill in manual commands and read from test topics. The agent will do all that, report back on findings, and generate all the relevant output links. Pull request and verification links posted for the developer’s review Observability We wanted to know which Kafka Skills are being used, how often they make it to pull requests, which environments and application types are involved, and where developers may still need support. To do this, we added lightweight telemetry around the workflow: when a Skill is invoked, a hook fires and a Python helper emits a started event. When the agent opens a pull request, a wrapper emits a completion event. Both events flow through an existing Etsy service that publishes a protobuf message to Kafka and stores it in a BigQuery table. Recording developer use of Kafka Skills in BigQuery Build It and They Won't Come We tested and demoed Kafka Skills and were very pleased with our “Agentic Improvements”. When we looked at adoption rates though, we were surprised to find that developers weren’t using them. They weren’t using the Skills to create features, generate embeddings, or even turn Kafka topic records into HTTP requests. Feeling a bit defeated, we asked devs: why? They responded that they simply didn’t know about them. The devs were using coding agents, but the agents never offered the Kafka Skills. Before the Skills existed, the standard workflow for feature creation was to find the closest existing feature, copy the code for it, and modify the fields. Instead of starting the process by opening the LLM and saying "I'd like to create a new streaming feature," an engineer might engage with the agent halfway through the old process and ask to "copy this class and change it to count cart moves," or "edit this file" with a path lifted from a previous pull request. Another challenge we found was that requests often arrive in whatever vocabulary the developer's team uses. When creating a new feature with Kafka, the dev might refer to a specific feature-store entry, an internal event name, a parent Java class name needed, or an output data type. To address these issues, we enumerated entry points for different vocabulary, lifecycle stages, and copy-paste-shaped requests. Plus, we added repository-level routing so that touching related files is itself a trigger for the Kafka Skill. The developer describes the task however they naturally would, and the agent offers the paved path. Guardrails In developing the Kafka Skills, we hoped to prevent agents from inventing Kafka applications from scratch. The agent handles the ambiguous parts: understanding the developer's request, identifying source schemas and fields, and deciding which validation steps to run. The agent then passes structured inputs into versioned scaffolding scripts and validation tools. Anthropic describes this pattern in its tool-use documentation: the agent returns a structured tool call that application-owned code executes, and strict tool use can constrain those inputs to a JSON schema. The agent handles the orchestration, while our scripts and hooks generate the templated code. Security research reinforced that decision. In a 2023 Stanford study, developers using an assistant wrote less secure code and were more likely to think their code was secure. BaxBench found a similar gap between code that runs and code that’s safe: across 392 backend tasks and 11 models, the best model was functionally correct 62% of the time, but no model produced code that was both correct and secure on more than 37% of the tasks. Researchers could exploit about half of the functionally correct programs. Now that we had agents running commands and committing code on our behalf, we also found that we needed to tighten Kafka ACLs, repository permissions, and deployment controls. We designed the system around least privilege: agents only get the access they need for the task they are performing. This was to make sure that neither human nor agent could delete an important topic, write to an unexpected one, change an unrelated pipeline, or bypass the normal production review path. What’s Next? Kafka supports a wide range of critical data workflows at Etsy, including machine learning feature generation, embeddings, filtering, fan-out, and storing data. We’ve built Skills for seven Kafka workflows so far. Adoption is the near-term focus: we're continuing to add entry points so the Skills are offered no matter how a developer phrases the task. Among our earliest insights from the observability table will be whether the discoverability fixes described above worked, and we’ll continue to iterate as we learn. We built the Kafka Skills to fix a developer-experience problem: too many steps stood between an idea for an ML model and a working pipeline with all the data. Our machine learning models shape the Etsy user experience for buyers and sellers. Each streaming feature that ships sooner helps our models understand a little earlier what shoppers mean and what listings contain. That means more fresh signals behind every search for a light linen dress, and behind the next hand-painted ceramic mug someone favorites. Acknowledgements A huge thanks to Kamya Shethia, Shannon Hoang, Nicollette Lui, Andrey Polyakov, and Virginia Poundstone for all their hard work designing and building Kafka Skills and agent tools, setting up observability, as well as locking down access control to make the world safer for robots and humans alike.
Meta’s Generative Ads Recommendation Model (GEM), the foundation model behind ads recommendations across Instagram and Facebook, now trains at LLM scale on several thousand of the latest-generation GPUs. This post goes into the details on how we achieved: doubling end-to-end (E2E) training efficiency to 20–25% Model FLOPs Utilization (MFU) while scaling training FLOPs 4x in [...] Read More... The post GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model appeared first on Engineering at Meta.
In early 2023, Slack faced a foundational challenge: serving Large Language Models (LLMs) at enterprise scale with the security, reliability, and performance our customers expect. Over three years, we evolved from basic infrastructure to orchestrating a sophisticated multi-cloud architecture. We didn’t just want shiny new models; we needed a system resilient to regional outages and…
We fine-tuned Qwen3-32B into a tool-calling agent that generates Flow automations: 2.2x faster, 68% cheaper, with a weekly retraining flywheel.
Our Sidekick team used LLM judges to automatically curate training data and teach domain-specific models how to refuse impossible queries.
Etsy features a diverse marketplace of unique handmade and vintage items. It’s a visually diverse marketplace as well, and computer vision has become increasingly important to Etsy as a way of enhancing our users’ shopping experience. We’ve developed applications like visual search and visually similar recommendations that can offer buyers an additional path to find what they’re looking for, powered by machine learning models that encode images as vector representations. Figure 1. Visual representations power applications such as visual search and visually similar recommendations Learning expressive representations through deep neural networks, and being able to leverage them in downstream tasks at scale, is a costly technical challenge. The infrastructure required to train and serve large models is expensive, as is the iterative process that refines them and optimizes their performance. The solution is often to train deep learning architectures offline and use the pre-computed pretrained visual representations in downstream tasks served online. (We wrote about this in a previous blog post on personalization from real-time sequences and diversity of representations.) In any application where a query image representation is inferred online, it's important that you have low latency, memory-aware models. Efficiency becomes paramount to the success of these models in the product. We can think about efficiency in deep learning along multiple axes: efficiency in model architecture, model training, evaluation and serving. Model Architecture The EfficientNet family of models features a convolutional neural network architecture. It uniformly optimizes for network width, depth, and resolution using a fixed set of coefficients. By allowing practitioners to start from a limited resource budget and scale up for better accuracy as more resources are available, EfficientNet provides a great starting point for visual representations. We began our trials with EfficientNetB0, the smallest size model in the EfficientNet family. We saw good performance and low latency with this model, but the industry and research community have touted Vision Transformers (ViT) as having better representations. We decided to give that a try. Transformers lack the spatial inductive biases of CNN, but they outperform CNN when trained on large enough datasets and may be more robust to domain shifts. ViT decomposes the image into a sequence of patches (16X16 for example) and applies a transformer architecture to incorporate more global information. However, due to the massive number of parameters and compute-heavy attention mechanism, ViT-based architectures can be many times slower to train and inference than lightweight Convolutional Networks. Despite the challenges, more efficient ViT architectures have recently begun to emerge, featuring clever pooling, layer dropping, efficient normalization, and efficient attention or hybrid CNN-transformer designs. We employ the EfficientFormer-l3 to take advantage of these ViT improvements. The EfficientFormer architecture achieves efficiency through downsampling multiple blocks and employing attention only in the last stage. This derived image representation mechanism differs from the standard vision transformer, where embeddings are extracted from the first token of the output. Instead, we extract the attention from the last block for the eight heads and perform average pooling over the sequence. In Figure 2 we illustrate these different attention weights with heat maps overlaid on an image, showing how each of the eight heads learns to focus on a different salient part. Figure 2. Probing the EfficientFormer-l3 pre-trained visual representations through attention heat maps. Model Training Fine-Tuning With our pre-trained backbones in place, we can gain further efficiencies via fine tuning. For the EfficientNetB0 CNN, that means replacing the final convolutional layer and attaching a d-dimensional embedding layer followed by m classification heads, where m is the number of tasks. The embedding head consists of a new convolutional layer with the desired final representation dimension, followed by a batch normalization layer, a swish activation and a global average pooling layer to aggregate the convolutional output into a single vector per example. To train EfficientNetB0, new attached layers are trained from scratch for one epoch with the backbone layers frozen, to avoid excessive computation and overfitting. We then unfreeze 75 layers from the top of the backbone and finetune for nine additional epochs, for efficient learning. At inference time we remove the classification head and extract the output of the pooling layer as the final representation. To fine-tune the EfficientFormer ViT we stick with the pretraining resolution of 224X224, since using sequences longer than the recommended 384X384 in ViT leads to larger training budgets. To extract the embedding we average pool the last hidden state. Then classification heads are added as with the CNN, with batch normalization being swapped for layer normalization. Multitask Learning In a previous blog post we described how we built a multitask learning framework to generate visual representations for Etsy's search-by-image experience. The training architecture is shown in Figure 3. Figure 3. A multitask training architecture for visual representations. The dataset sampler combines examples from an arbitrary number of datasets corresponding to respective classification heads. The embedding is extracted before the classification heads. Multitask learning is an efficiency inducer. Representations encode commonalities, and they perform well in diverse downstream tasks when those are learned using common attributes as multiple supervision signals. A representation learned in single-task classification to the item’s taxonomy, for example, will be unable to capture visual attributes: colors, shapes, materials. We employ four classification tasks: a top-level taxonomy task with 15 top-level categories of the Etsy taxonomy tree as labels; a fine-grained taxonomy task, with 1000 fine-grained leaf node item categories as labels; a primary color task; and a fine-grained taxonomy task (review photos), where each example is a buyer-uploaded review photo of a purchased item with 100 labels sampled from fine-grained leaf node item categories. We are able to train both EfficientNetB0 and EfficientFormer-l3 on standard 16GB GPUs (we used two P100 GPUs). For comparison, a full sized ViT requires a larger 40GB RAM GPU such as an A100, which can increase training costs significantly. We provide detailed hyperparameter information for fine-tuning either backbone in our article. Evaluating Visual Representations We define and implement an evaluation scheme for visual representations to track and guide model training, on three nearest neighbor retrieval tasks. After each training epoch, a callback is invoked to compute and log the recall for each retrieval task. Each retrieval dataset is split into two smaller datasets: “queries” and “candidates.” The candidates dataset is used to construct a brute-force nearest neighbor index, and the queries dataset is used to look up the index. The index is constructed on the fly after each epoch to accommodate for embeddings changing between training epochs. Each lookup yields K nearest neighbors. We compute Recall@5 and @10 using both historical implicit user interactions (such as “visually-similar ad clicks”) and ground truth datasets of product photos taken from the same listing (“intra-item”). The recall callbacks can also be used for early stopping of training to enhance efficiency. The intra-item retrieval evaluation dataset consists of groups of seller-uploaded images of the same item. The query and candidate examples are randomly selected seller-uploaded images of an item. A candidate image is considered a positive example if it is associated with the same item as the query. In the “intra-item with reviews” dataset, the query image is a randomly selected buyer-uploaded review image of an item, with seller-uploaded images providing candidate examples. The dataset of visually similar ad clicks associates seller-uploaded primary images with primary images of items that have been clicked in the visually similar surface on mobile. Here, a candidate image is considered a positive example for some query image if a user viewing the query image has clicked it. Each evaluation dataset contains 15,000 records for building the index and 5,000 query images for the retrieval phase. We also leverage generative AI for an experimental new evaluation scheme. From ample, multilingual historical text query logs, we build a new retrieval dataset that bridges the semantic gap between text-based queries and clicked image candidates. Text-to-image generative stable diffusion makes the information retrieval process language-agnostic, since an image is worth a thousand (multilingual) words. A stable diffusion model generates high-quality images which become image queries. The candidates are images from clicked items corresponding to the source text query in the logs. One caveat is that the dataset is biased toward the search-by-text production system that produced the logs; only a search-by-image-from-text system would produce truly relevant evaluation logs. The source-candidate image pairs form the new retrieval evaluation dataset which is then used within a retrieval callback. Of course, users entering the same text query may have very different ideas in mind of, say, the garment they’re looking for. So for each query we generate several images: formally, a random sample of length 𝑛 from the posterior distribution over all possible images that can be generated from the seed text query. We pre-condition our generation on a uniform “fashion style.” In a real-world scenario, both the text-to-image query generation and the image query inference for retrieval happen in real time, which means efficient backbones are necessary. We randomly select one of the 𝑛 generated images to replace the text query with an image query in the evaluation dataset. This is a hybrid evaluation method: the error inherent in the text-to-image diffusion model generation is encapsulated in the visually similar recommendation error rate. Future work may include prompt engineering to improve the text query prompt itself, which as input by the user can be short and lacking in detail. Large memory requirements and high inference latency are challenges in using text-to-image generative models at scale. We employ an open source fast stable diffusion model through token merging and float 16 inference. Compared to the standard stable diffusion implementation available at the time we built the system, this method speeds up inference by 50% with a 5x reduction in memory consumption, though results depend on the underlying patched model. We can generate 500 images per hour with one T4 GPU (no parallelism) using the patched stable diffusion pipeline. With parallelism we can achieve further speedup. Figure 4 shows that for the English text query “black bohemian maxi dress with orange floral pattern” the efficient stable diffusion pipeline generates five image query candidates. The generated images include pleasant variations with some detail loss. Interestingly, mostly the facial details of the fashion model are affected, while the garment pattern remains clear. In some cases degradation might prohibit display, but efficient generative technology is being perfected at a fast pace, and prompt engineering helps the generative process as well. Figure 4. Text-to-image generation using a generative diffusion model, from equivalent queries in English and French Efficient Inference and Downstream Tasks Especially when it comes to latency-sensitive applications like visually similar recommendations and search, efficient inference is paramount: otherwise, we risk loss of impressions and a poor user experience. We can think of inference along two axes: online inference of the image query and efficient retrieval of top-k most similar items via approximate nearest neighbors. The dimension of the learned visual representation impacts the efficient retrieval design as well, and the smaller 256d derived from the EfficientNetB0 presents an advantage. EfficientNet B0 is hard to beat in terms of accuracy-to-latency trade-offs for online inference, with ~5M parameters and around 1.7ms latency on iPhone 12. The EfficientFormer-l3 has ~30M parameters and gets around 2.7ms latency on iPhone 12 with higher accuracy (while for example MobileViT-XS scores around 7ms with a third of accuracy; very large ViT are not considered since latencies are prohibitive). In offline evaluation, the EfficientFormer-l3-derived embedding achieves around +5% lift in the Intra-L Recall@5 evaluation, a +17% in Intra-R Recall@5, and a +1.8% in Visually Similar Ad clicks Recall@5. We performed A/B testing on the EfficientNetB0 multitask variant across visual applications at Etsy with good results. Additionally, the EfficientFormer-l3 visual representations led to a +0.65% lift in CTR, and a similar lift in purchase rate in a first visually-similar-ads experiment when compared to the production variant of EfficientNetB0. When included in sponsored search downstream rankers, the visual representations led to a +1.26% lift in post-click purchase rate. Including the efficient visual representation in Ads Information Retrieval (AIR), an embedding-based retrieval method used to retrieve similar item ad recommendations caused an increase in click-recall@100 of 8%. And when we used these representations to compute image similarity and included them directly in the last-pass ranking function, we saw a +6.25% lift in clicks. The first use of EfficientNetB0 visual embeddings was in visually similar ad recommendations on mobile. This led to a +1.92% increase in ad return-on-spend on iOS and a +1.18% increase in post-click purchase rate on Android. The same efficient embedding model backed the first search-by-image shopping experience at Etsy. Users search using photos taken with their mobile phone’s camera and the query image embedding is inferred efficiently online, which we discussed in a previous blog post. Learning visual representations is of paramount importance in visually rich e-commerce and online fashion recommendations. Learning them efficiently is a challenging goal made possible by advances in the field of efficient deep learning in computer vision. If you'd like a more in-depth discussion of this work, please see our full accepted paper to the #fashionXrecsys workshop at the Recsys 2023 conference.
Shopify’s Global Catalogue demonstrates the impact of multimodal LLMs on one of commerce’s hardest problems: building a unified, structured, and continuously evolving understanding of billions of product listings created by millions of merchants. At the International Conference on Learning Representations (ICLR) 2025 Expo in Singapore, we presented the core engineering behind this initiative
At Etsy, we’re focused on elevating the best of our marketplace to help creative entrepreneurs grow their businesses. We continue to invest in making Etsy a safe and trusted place to shop, so sellers’ extraordinary items can shine. Today, there are more than 100 million unique items available for sale on our marketplace, and our vibrant global community is made up of over 90 million active buyers and 7 million active sellers, the majority of whom are women and sole owners of their creative businesses. To support this growing community, our Trust & Safety team of Product, Engineering, Data, and Operations experts are dedicated to keeping Etsy's marketplace safe by enforcing our policies and removing potentially violating or infringing items at scale For that, we make use of community reporting and automated controls for removing this potentially violating content. In order to continue to scale and enhance our detections through innovative products and technologies, we also leverage state-of-the-art Machine Learning solutions which we have already used to identify and remove over 100,000 violations during the past year on our marketplace. In this article, we are going to describe one of our systems to detect policy violations that utilizes supervised learning, a family of algorithms that uses data to train their models to recognize patterns and predict outcomes. Datasets In Machine Learning, data is one of the variables we have the most control over. Extracting data and building trustworthy datasets is a crucial step in any learning problem. In Trust & Safety, we are determined to keep our marketplace and users safe by identifying violations to our policies. For that, we log and annotate potential violations that enable us to collect datasets reliably. In our approach, these are translated into positives, these were indeed violations, and negatives, these were found not to be offending for a given policy. The latter are also known as hard negatives as they are close to our positives and can help us to better learn how to partition these two sets. In addition, we also add easy or soft negatives by adding random items to our datasets. This allows us to give further general examples to our models for listings that do not violate any policy, which is the majority in our marketplace and improve generalizability. The number of easy negatives to add is a hyper-parameter to tune, more will mean higher training time and fewer positive representations. For each training example, we extract multimodal signals, both textual and imagery from our listings. Then, we split our datasets by time using progressive evaluation, to mimic our production usecase and learn to adapt to recent behavior. These are split into training, used to train our models and learn patterns, validation to fine tune our training hyper-parameters such as learning rate and to evaluate over-fitting, and test to report our metrics in an unbiased manner. Model Architecture After usual transformations and extraction of a set of offline features from our datasets, we are all set to start training our Machine Learning model. The goal is to predict whether a given listing violates any of our predefined set of policies, or in contrast, it doesn’t violate any of them. For that, we added a neutral class that depicts the no violation class, where the majority of our listings fall into. This is a typical design pattern for these types of problems. Our model architecture includes a text encoder and an image encoder to learn representations (aka embeddings) for each modality. Our text encoder currently employs a BERT-based architecture to extract context-full representations of our text inputs. In addition, to alleviate compute time, we leverage ALBERT, a lighter BERT with 90% fewer parameters as the transformer blocks share them. Our initial lightweight representation used an in-house model trained for Search usecases. This allowed us to quickly start iterating and learning from this problem. Our image encoder currently employs EfficientNet, a very efficient and accurate Convolutional Neural Network (CNN). Our initial lightweight representation used an in-house model for category classification using CNNs. We are experimenting with transformer-based architectures, similar to our text encoders, with vision transformers but its performance has not been significantly improved. Inspired by EmbraceNet, our architecture then further learns more constrained representations for both text and image embeddings separately, before they are concatenated to form a unique multimodal representation. This is then sent to a final softmax activation that maps logits to probabilities for our internal use. In addition, in order to address the imbalanced nature of this problem, we leverage focal loss that penalizes more hard misclassified examples. Figure 1 shows our model architecture with late concatenation of our text and image encoders and final output probabilities on an example. Model Architecture. Image is obtained from @charlesdeluvio on Unsplash Model Evaluation First, we experimented and iterated by training our model offline. To evaluate its performance, we established certain benchmarks, based on the business goal of minimizing the impact of any well-intentioned sellers while successfully detecting any offending listings in the platform. This results in a typical evaluation trade-off between precision and recall, precision being the fraction of correct predictions over all predictions made, and recall being the fraction of correct predictions over the actual true values. However, we faced the challenge that recall is not possible to compute, as it’s not feasible to manually review the millions and millions of new listings per day so we had to settle for a proxy for recall from what has been annotated. Once we had a viable candidate to test in production, we deployed our model as an endpoint and built a service to perform pre-processing and post-processing steps before and after the call to our endpoint that can be called via an API. Then, we ran an A/B test to measure its performance in production using a canary release approach, slowly rolling out our new detection system to a small percentage of traffic that we keep increasing while we validate an increase in our metrics and no unexpected computation overload. Afterwards, we iterated and every time we had a promising offline candidate, named challenger, that improved our offline performance metrics, we A/B tested it with respect to our current model, named champion. We designed guidelines for model promotion to increase our metrics and our policy coverage. Now, we monitor and observe our model predictions and trigger re-training when our performance degrades. Results Our supervised learning system has been continually learning as we train frequently, run experiments with new datasets and model architectures, A/B test them and deploy them in production. We have added violations as additional classes to our model. As a result, we have identified and removed more than 100,000 violations using these methodologies, in addition to other tools and services that continue to detect and remove violations. This is one of our approaches to identify potentially offending content among others such as explicitly using the policy information and leverage the latest in Large Language Models (LLMs) and Generative AI. Stay tuned! "To infinity and beyond!" –Buzz Lightyear, Toy Story
By: Jeet Mehta and Kathy Ge With over 1M business owners now on Shopify, there are billions of products being created and sold across the platform. Just like those business owners, the products that they sell are extremely diverse! Even when selling similar products, they tend to describe products very differently. One may describe their sock product as a “woolen long sock,” whereas another may have a similar sock product described as a “blue striped long sock.” How can we identify similar products, and why is that even useful? Applications of Product Categorization Business owners come to our platform for its multiple sales/marketing channels, app and partner ecosystem, brick and mortar support, and so much more. By understanding the types of products they sell, we provide personalized insights to help them capitalize on valuable business opportunities. For example, when business owners try to sell on other channels like Facebook Marketplace, we can leverage our product categorization engine to pre-fill category related information and save them time. In this blog post, we’re going to step through how we implemented a model to categorize all our products at Shopify, and in doing so, enabled cross-platform teams to deliver personalized insights to business owners. The system is used by 20+ teams across Shopify to power features like marketing recommendations for business owners (imagine: “t-shirts are trending, you should run an ad for your apparel products”), identification of brick-and-mortar stores for Shopify POS, market segmentation, and much more! We’ll also walk through problems, challenges, and technical tradeoffs made along the way. Why is Categorizing Products a Hard Problem? To start off, how do we even come up with a set of categories that represents all the products in the commerce space? Business owners are constantly coming up with new, creative ideas for products to sell! Luckily, Google has defined their own hierarchical Google Product Taxonomy (GPT) which we leveraged in our problem. The particular task of classifying over a large-scale hierarchical taxonomy presented
As computing use grows, so does its environmental footprint. While hardware and software have become significantly more efficient, AI demand is growing even faster, outpacing those efficiency gains and increasing the overall environmental impact of computing. What’s more, measurement standards are still playing catch up, making it difficult for companies to measure and reduce their environmental impact. We’ve grappled with this challenge for years (starting with cloud computing and now AI), and we know we’re not alone. Environmental impact has been growing exponentially as computational power has grown in complexity. At Etsy, we believe taking responsibility for the environmental impact of powering our marketplace is part of keeping commerce human. Our approach to sustainable engineering has evolved as the world and our industry has changed, but our commitment has remained the same. Rapid AI development and adoption is the latest (and perhaps most consequential) industry shift, so we are rising to this new challenge. In this article we'll discuss how we've managed the environmental impact of our computing in the past, how we've begun to approach it in this new world of AI, and how we think other companies can join us. For more on our full Impact & Sustainability work see our 2025 Integrated Report. Disclaimer and legend No generative AI was involved in the making of this article: the images were hand-drawn by Emily Sommer, and the text was arduously written by Emily Sommer and Sam Brundrett. We both leverage AI frequently in our work, but this article warranted a different type of rigor and craftsmanship, so we decided to take a more manual approach. The scope of this article is limited to Etsy employees’ compute and AI usage and does not include application of AI outside of internal tools, such as Etsy sellers' use of AI outside of Etsy-provided tools. Despite looking so, the computers are not sad – just working hard. Where we've been: transition to the cloud We set a 2025 goal to improve our compute energy efficiency (kWh per site visit) by 25%. In 2015, Etsy formed an internal working group to address the environmental footprint of running Etsy's website and related systems. At the time, our servers were running in several co-located data centers, so we had a strong sense of how much energy our servers were using because we were billed for it directly. We were seeing meaningful increases in our energy use as we expanded our big data processing platform, so we set a goal to help improve our efficiency. We committed to improving our kWh per Etsy site visit (session) by 25% by 2025 – and we did! In 2018, we began moving our compute workloads out of our co-located data centers and onto Google Cloud Platform (GCP). From a sustainability perspective, moving to GCP was a big win for several reasons: Google's data centers operate much more efficiently (as measured by Power Usage Effectiveness (PUE)) than colocated datacenters. The scale at which they operate enables them to invest in optimizing custom machines, racks, power distribution, sensors, etc. Google matched 100% of its energy consumption with renewable energy. In aggregate, overall server idle time is lower for cloud platform customers than colocated data center customers, because cloud providers often run customer workloads on shared platforms and are able to schedule virtual resources to optimize for efficiency. However, at the time, Google didn’t provide customers with data about how much energy their servers were using, meaning we were rapidly losing the ability to measure ourselves against our goal. We got creative Cloud Jewels energy coefficients can be applied to billed usage data to estimate energy consumption. Without compute energy use data, we worked with what we had: billed usage data by machine type. We used public data about the energy efficiency of typical large-scale servers as well as industry research about the energy efficiency of hard drives. From these, we developed a set of coefficients for estimating the energy consumption of compute and storage use. We called it Cloud Jewels, a play on cloud computing, the joule, and crown jewels. We published this work and collaborated with Thoughtworks, a technology consultancy, which used Cloud Jewels in the early development of their Cloud Carbon Footprint. We also heard from several companies that Cloud Jewels had inspired them to take action! Cloud Jewels opened the door to meaningful measurement and efficiency wins: We worked with our auditors to review the approach, and we used it to report on our energy use and progress towards our 2025 goal in filings. By estimating the energy used by our cloud services, we were able to apply renewable energy certificates (RECs) from our long-term solar agreement to that energy use, reducing our market-based cloud emissions. We used Cloud Jewels to estimate the impact of a few significant efficiency projects like snoozing developer virtual machines (VMs) at night (Cloud Snooze). But Cloud Jewels had practical limitations. We explored tying data into our experimentation platform, but it proved challenging: it was hard to communicate what our partial data represented and what it omitted. Google took the lead in transparency Google began providing emissions and energy data to customers – incredible! But our adoption of AI has grown beyond just GCP as an AI provider. Cloud Jewels worked well as we migrated onto GCP: for the first few years, a lot of our workloads ran directly on cloud VMs and storage. However, as we adopted more managed services (like BigQuery and Dataflow), Cloud Jewels became less and less applicable – we had no way to apply concepts like "BigQuery slots" to server wattage coefficients. The proportion of our total energy that we were able to measure was declining. In part thanks to our influence, in 2021, Google launched the first iteration of its Carbon Footprint tool, providing customers with energy and carbon emissions data associated with their GCP use. This was incredible. They were first in the industry to release granular customer environmental data! This tool provides customers with location-based (LB) and market-based (MB) Greenhouse Gas (GHG) emissions, broken down by scope, aggregated by month, service, project, and region. The tool has a few shortcomings that limit its actionability: data is not broken down by day or SKU, and lag time can be up to 45 days. Still, its release has been a game-changer for Etsy’s emissions measurement, and this year (2026), Google added energy and emissions data from AI model inference -- another industry-leading feature we’re excited to leverage! But challenges persist. As our adoption of AI has expanded beyond Google products, we’re working with fewer insights: many suppliers don’t provide energy or emissions data, and there’s no standard approach to accurately estimate this impact. We find ourselves in a familiar bind! We're focused on what we CAN do We're tackling environmental responsibility up the supply chain, internally within our services, and outward amongst our peers. This is an unprecedented age, and AI is an extraordinary tool. We're leaning into adoption and exploration: how can these tools support our buyers and sellers and unlock internal productivity? As we grow our use of AI, we're developing a sustainability strategy in real time, without all the answers or decisions in place. To do so, we're observing our adoption and considering: How are we currently leveraging AI? (In flux!) What data is available for measuring AI's environmental impact? (Not much.) What are others in the industry doing about their footprint? (Also stumped.) What options exist for influencing environmental impact closer to the source? (Some!) Instead of being paralyzed by all that we don't know, we're using it as a call to action: how can we mobilize ourselves and the industry to learn more about measuring and managing the environmental impact of AI? To tackle this, we developed a new sustainable engineering strategy centered around three levers: Catalyze the industry, Influence the supply chain, and Optimize our internal resource use. You can read more about our work to activate on the first two levers and how it fits into Etsy’s larger sustainability efforts in this post on the Etsy News Blog. Below, we dive into the third lever: the ways we’re driving more sustainable AI use inside Etsy. Optimizing our internal resource use We're approaching resource optimization from three directions: engaging our engineers, educating all employees, and holding ourselves accountable for our use. Engage our engineers Educate all employees Hold ourselves accountable Sustainable Engineering Council Effective AI use Efficiency target where we have reliable data Amplifying existing tooling Environmental (and monetary) resource efficiency Impact accounting per model and token Engaging our engineers Etsy’s values are deeply embedded in our culture: many Etsy engineers care about the environmental impact of AI and computing and proactively look to participate in efficiency initiatives. The challenge is in harnessing that interest into actionable initiatives. Our efforts currently underway include: Sustainable Engineering Council We've formed a new internal group with rotating members from across Engineering who have a mix of expertise, including both product and platform teams. We're working with this group to identify and act on opportunities for enhanced tooling and data to better inform decision making around cloud and AI use. Amplifying existing tooling We’re improving how teams across Etsy discover, evaluate, and adopt tools that can increase efficiency and reduce environmental impact. For example: We're exploring the effectiveness (e.g., performance improvement and energy reduction) of tools that automatically direct a user's prompts to appropriately-sized models based on keyword matching. We're enhancing the granularity of Google's Carbon Footprint data with regression modeling and surfacing the estimated data alongside teams' cost data. We're designing accessible tooling to guide all employees in choosing the most appropriate AI and workflow tools for their tasks. Educate all employees AI has lowered the barrier to entry for automating work and coding. From a sustainability perspective, this expansion means guidance about resource use must be accessible and applicable to a broad audience of varying technical expertise. To that end, we've developed internal guidance to address effective AI use and resource efficiency – both of which relate to sustainability. Effective AI use We have many AI tools available to us, many engineering tools that can amplify AI's utility, and a wide array of options for models, reasoning effort, and speed. It's hard to know what to use for what, and decisions about models and tools can have a large impact on resource use. Our guidance seeks to assist employees who lack engineering experience make decisions that minimize engineering concerns – such as safety, maintenance, and reliability – while encouraging engineers to right-size models and tools for the task at hand. It contains several sections: Guidance on selecting an appropriate model and reasoning effort for the task at hand, including a matrix of model tier and reasoning effort combinations with example tasks with the aim of directing users to the lightest-weight model and effort applicable to their task. A decision tree that maps example categories of tasks to appropriate AI tools, optimizing for the lightest weight tool and recommending products with the most impact transparency and commitment to renewable energy. Tooling that recommends the most effective engineering options for a given objective. This is important because engineers, non-engineers, and especially agents are often unfamiliar with the basic workflow automation tooling that we already have available, thus agents often generate needlessly complex solutions. Resource efficiency Our recommendations for using AI efficiently aim to be specific enough to be useful and generic enough to be broadly applicable. We organized the guidance around two audiences: individuals using AI in the workplace including coding tools (many) and employees building products powered by AI (some engineers). We've published an external version of our guidance in the sustainable-ai-guide GitHub repo in the hopes that it’s useful to other teams navigating this journey. Hold ourselves accountable for AI use Incomplete data is better than no data! We want to hold ourselves accountable for our environmental impact and be able to measure and validate our successful optimizations – even if we know we're missing some information. So we're setting a carbon efficiency target using the data that we have, noting that it will (hopefully) need to change as we expand our coverage. And to that end, we're partnering with industry experts to develop a more comprehensive AI impact measurement methodology. Efficiency target where we have reliable data GCP provides us with reliable emissions data covering the vast majority of our cloud and storage use and some of our AI applications via Gemini. For these compute resources, we’ve set an efficiency target: keep our GCP emissions intensity flat through 2027, with intensity defined as metric tonnes of LB (location-based) emissions per $M in sales. We use LB emissions as a way to track real-world pollution that more clearly reflects our decision making, such as machine and region selection. This goal is intentionally set for a short timeline: technology and our usage patterns are changing quickly, making it difficult to reliably predict beyond 2027. As the landscape and research evolves, we'll continue to adapt our approach. Impact accounting per model and token Our goal is to account for all of our energy and emissions impact across compute and AI vendors at a level of granularity that our teams can action on. Until all vendors provide this, we need a way to more accurately estimate our AI emissions. To address this, we’ve partnered with Sustainable AI Group to expand, refine, and pilot an open-source methodology for estimating energy and emissions per LLM token, using Etsy as the initial test case. The SAIG emissions coefficients are mapped to our internal AI usage data to surface emissions alongside token usage and costs – at the model level. This means employees can see the estimated environmental impact of using different models across providers and make informed decisions about when the complexity of a task they're tackling is worth a specific increase in carbon emissions! We're thrilled to be collaborating on this groundbreaking methodology and to have the opportunity to shape its future through our partnership with SAIG. We hope to inspire other companies to adopt this approach and thereby urge AI providers into disclosing more granular impact data. In parting and partnership We aspire to remain optimistic and action-oriented in a time of uncertainty. In line with Etsy's mission to keep commerce human, we're leaning into creativity and problem solving to conscientiously embrace AI and take responsibility for our adoption of it. We seek to be transparent in our work and would love to partner with other companies and individuals. Join us! We're actively seeking collaborators, both formal and informal! We invite you to: Partner with us to test and adopt a standard AI emissions approach Share insight on sustainable software engineering practices and AI deployment Reach out to us at sustainability@etsy.com. Forward-Looking Statements: This post contains forward-looking statements about our sustainability strategy, goals, targets and planned initiatives. These statements are based on current expectations and assumptions and are subject to risks and uncertainties.
How we train Dash's search ranking models with a mix of human and LLM-assisted labeling.
Author: Jeff Wu Key Contributors : Lily Sierra, Nour Alkhatib, Steven Gross, Jacquelene Obeid, Kyle Swint, Monta Shen, Gary Song, Riddhima Sejpal, Jatin Jain, Shishir Kumar Prasad, Ayesha Saleem Introduction Millions of people rely on...
This post investigates the benefits and limitations of prompt engineering in two instances of AI-assisted onboarding relying on large language model (LLM) technology. Of particular interest is how truthful (and therefore reliable) LLM-generated answers turn out to be in the context of Etsy-specific question answering. Among other insights, we find that asking the LLM to identify specific source snippets is a good way to flag potential hallucinations. Over the past few years, pre-trained large-scale/foundation language models such as OpenAI’s o-series [1] and Google’s Gemini family [2] have revolutionized the field of natural language processing (NLP). Trained on vast amounts of text, images, code, audio, and videos, such models encapsulate a great deal of world knowledge, which can be called upon to perform a wide range of downstream tasks, such as sentiment analysis, language translation, and natural language inference, among many others. The canonical way to improve the task performance of a pre-trained general language model, when it needs specific knowledge beyond its original training, is called fine-tuning [3]. The model in a sense receives supplementary training, typically applied only in its last layers, on a relevant, labeled dataset. This is a robust but somewhat expensive approach, and only possible when there’s a sufficiently large dataset to support it. With the greatly increased size and complexity of LLMs, though, prompt-based tuning has emerged as an effective and much less costly alternative. The parameters of the model remain unchanged, and instead it’s through the prompt—the instructions given the model to perform its task—that tuning is done. At its simplest a prompt is just a question, but prompts can also incorporate specific chunks of text, provide the model with task-related constraints or preferences, and in general shape the model toward producing reliable responses. Prompt engineering encompasses the art and science of tuning prompts so they yield optimal results on a given task. One of the places at Etsy where an assistive AI could be useful is onboarding: in principle, an LLM could answer a range of questions about Etsy’s both internal and external policies and procedures. But rather than ramp up an effort to fine-tune an onboarding model with a dataset of Etsy documents, we wanted to assess whether prompts alone could elicit truthful answers. We decided to start with a limited-scale pilot project, focusing on answering questions specific to the Travel & Entertainment (T&E) domain. On the one hand, this is a well circumscribed domain with clear and unambiguous rules. On the other hand, most Etsy employees still have questions for every trip. This article summarizes our journey of discovery, and discusses in particular the extent to which the addition of simple explicit instructions works to mitigate LLM errors. Question answering Perhaps the most critical aspect of a question answering system is its reliability, i.e., whether it is able to provide a truthful answer to any in-domain question. In the AI-assisted onboarding use case considered here, we want new Etsy employees to be able to be confident that their questions about the T&E policy are answered correctly. The first step was to feed Etsy-specific data into the LLM. If we were fine-tuning, we would update (a subset of) model weight parameters from an appropriate collection of Etsy internal documents related to T&E. Prompt engineering, on the other hand, freezes the model weights, treating the LLM as a black box. A number of such black-box tuning techniques exist in the field, which we review in the Appendix to this article. Prompt-based tuning was an attractive proposition in our case because all that it required was simply an adequate representation of task-specific documents. We therefore folded all T&E documents into the rich latent space of the LLM (also known as the embedding space). In practice, this can be done efficiently by simply updating the underlying index to account for the newly folded data. Then, as indicated in the Appendix, question answering proceeded by leveraging embedding-based search [6]. For every query, embedding the initial formulation led to relevant Etsy-specific text sections, which were then added to the question to form the final prompt. We tested the resulting system’s performance on a manually curated set of 40 question-and-answer pairs. For each question in the test set, we compared the answer generated by the LLM with the answer we had extracted from the relevant policy document to form a judgment of answer quality. In the authors’ estimation, the LLM answered approximately 86% of the questions correctly, or at least satisfactorily enough that there was no need for further prompt maneuvering. In the remaining 14% of cases, the LLM generated an answer which was either wrong or misleading. For example, the LLM asserted with high confidence that it's the cardholder who is responsible for the balance on a corporate credit card: Q: Who pays the balance on my corporate card after my expense report is approved? Correct answer: Etsy pays the corporate card company directly on behalf of the Employee. LLM answer: The cardholder is responsible for paying the balance on their corporate card after their expense report is approved. The statement is wrong under the terms of the corporate card agreement. Similarly, the LLM falsely claimed that Etsy doesn't reimburse the cost of checking bags on a business trip (noting in its answer that "the context information provided does not mention any expenses related to checking bags," another falsehood). Mitigating hallucinations The kind of LLM falsehoods illustrated in the previous section–wrong, but confidently stated–have become known as “hallucinations.” Prompt engineering offers us a number of ways of addressing and overcoming LLM hallucinations. We can craft narrower and more explicit input instructions. We can augment prompts with relevant contextual information. We can even ask the LLM to answer as if it occupied a particular role, say, data analyst or financial manager (role prompting). In the case of the above hallucination about our corporate cards, we first attempted the explicit-instruction route, asking the LLM to be clear about what it didn’t know: Q: Say "I have no idea" when you don't know the answer. Who pays the balance on my corporate card after my expense report is approved? A: I have no idea. We at least avoid the hallucination, but not in a way that will be very helpful to the user. Also note that the LLM misses the correct answer, which is actually present in the collection of documents. Another try, prompting the LLM for chain-of-thought reasoning, finally led to the correct answer: Q: Say "I have no idea" when you don't know the answer. Who pays the balance on my corporate card after my expense report is approved, and why do you think so? A: The balance on your corporate card is paid by Etsy. This information is mentioned on page 42. The above is a good illustration of how three different prompts for the same question can lead to widely divergent answers. When the internal state of the LLM is confused about a particular topic, letting it express uncertainty in the answer can often mitigate undue overconfidence. In addition, prompting for chain-of-thought reasoning is often beneficial, as it provides a way to structure the prompt with logical milestones to ensure that the LLM follows a well-circumscribed reasoning path. While it may not seem very intuitive that just adding "why do you think so" should change an all-but unresponsive answer into a correct one, the mere fact of asking for an explanation raises the bar in terms of fact checking – a side benefit of prompt engineering which doesn't require additional investment. Etsy community forums Encouraged by those findings, we decided to expand our investigation to an external use case: question answering based on historical data from Etsy seller community forums. Unlike the T&E data we’ve just seen, Etsy forums contain publicly available content. Sellers ask questions such as: “What are the most effective ways to optimize my Etsy shop’s SEO to increase visibility and sales?” These questions are then answered by either Etsy or community members. As a result, compared to the T&E data, Etsy forum data is substantially more complex: much less homogeneous (both in terms of style and scope), with answers by nature much more opinionated. As such,assessing the accuracy of LLM-based question answering is a little trickier for community forums than for the T&E policy. To note, because forums are a space where sellers engage with and offer information to one another, there may be some divergence between answers provided in the forums and official Etsy policies & practices. For the purposes of our investigation, we measured only whether the LLM could accurately surface responses based on the input data it was given. A high score would indicate a helpful tool not only at seller onboarding time, but also to help both Etsy and community members routinely search this growing community database in a faster and more accurate fashion. We proceeded exactly as before, first folding recent Etsy forum data into the latent space and then updating the underlying index to account for the newly folded data. Similar to the previous case study, we tested performance on a manually curated set of 50 question-and-answer pairs, comparing LLM-generated answers with answers extracted from the relevant Etsy forum to form a judgment of answer quality. In the authors’ estimation, the LLM answered approximately 72% of the questions correctly before any prompt maneuvering. In the remaining 28% of cases, the LLM generated an answer which was either wrong or misleading. Given the higher heterogeneity of the data, it is not overly surprising that the percentage of poor answers doubles. Comparing manually extracted and automatically generated answers is informative, not just from the point of view of seeing natural language outcomes, but also from the perspective of analyzing how such outcomes are pieced together from tidbits of information in latent space. Everything else being equal, the LLM tends to do better when a query formulation is close to the exact wording used in reference documents, and worse when the answer depends on multiple conditions being satisfied in turn. In a handful of cases when the LLM stayed relatively vague and avoided providing specific information, enabling detailed query expansion occasionally led to an hallucination. This observation has concrete implications when it comes to chain-of-thought prompting: specific logical milestones must be congruent with query expansion chunks to maximize alignment. Limitations The forums Q&A use case also helped surface some of the limitations of prompt engineering. For that data, we observed that the approach advocated earlier (allowing for uncertainty, providing more context, asking for an explanation) was not always effective. For example, we initially thought that the following case: Q: Help, where’s the download for 1099-K? There’s NO download button! A: It is not possible to answer this question without more information. It is unclear what the 1099-K is, what it is used for, and where it can be downloaded from. ...would be readily addressed through more advanced prompt engineering focusing on additional contextual information. However, adding a contextual preamble did not help: Q: In the United States, Form 1099-K "Payment Card and Third Party Network Transactions" is a variant of Form 1099 used to report payments received through reportable payment card transactions and/or settlement of third-party payment network transactions. Help, where’s the download for 1099-K? There’s NO download button! A: The context information provided does not include any information about downloading Form 1099-K. To find the download button for Form 1099-K, you will need to search for it on the website of the Internal Revenue Service (IRS) or other relevant government agency. The preamble “In the United States, Form 1099-K [...] transactions.” was clearly not sufficient to lead to a useful answer to the original question. Other times, the LLM punted with the expression: "Without prior knowledge, it is impossible to determine..." -- which was untrue, since the information was indeed available. We conclude that, even with the current crop of advanced reasoning models, when complex reasoning is required, chain-of-thought prompting must be carefully set up. In such situations, what seemed to help most was asking for specific source snippets, to determine where in the forums the LLM was looking for answers. Key learnings Our experiments with two flavors of AI-assisted onboarding (internal and external) indicate that, most of the time, prompt engineering works reasonably well as a practical alternative to parameter-based fine-tuning. Prompt engineering is conceptually simple and can often be leveraged to fix cases where LLMs generate non-factual information. It has limitations, however, so care should still be taken when assessing answer truthfulness. Asking the LLM to disclose specific sources is a good way to flag potential hallucinations. References [1] OpenAI blog. “Introducing OpenAI o3 and o4-mini.” 16 April 2025. [2] Google DeepMind blog. “Gemini 2.5: Our most intelligent AI model.” 25 March 2025. [3] Google Cloud Documentation. “Tune Gemini models by using supervised fine-tuning.” [4] T. Brown et al. (2020). “Language models are few-shot learners.” Proc. 2020 NeurIPS, Vol. 33, pp. 1877-1901. [5] T. Gao, A. Fisch, and D. Chen (2021). “Making pre-trained language models better few-shot learners.” Proc. 2021 ACL/IJCNLP, pp. 3816-2830. [6] Google AI for Developers. “Embeddings.” [7] Google AI for Developers. “Prompt design strategies.” Appendix Various instantiations of prompt-based tuning have evolved over the years. The purpose of this Appendix is to give a little historical background and perspective on the set of practices that have come to be known, collectively, as prompt engineering. Prompt-based tuning originally developed as an alternative form of model fine-tuning for situations where only few-shot demonstration examples are available for downstream tasks [4]. In such situations, prompt tuning makes it possible to use the same robust model for all tasks, as opposed to producing a separate, more brittle model for each one. This approach typically treats the model as a black box and encodes either the input as trainable sequences or the output as training features. It has been shown to drastically outperform conventional transfer learning procedures in the low resource setting [5]. More recently, prompt engineering has gained further prominence in the literature as a way to circumvent parameter-based fine-tuning of LLMs. Such fine-tuning is extremely challenging with an LLM, as any backpropagation through the entire model requires substantial computational resources. With LLMs being trained on an ever increasing amount and diversity of data, leading to an increasingly richer latent space, there is correspondingly less need to do the hard and costly work of updating model weights. Prompt-based tuning only requires that task-specific documents be ingested and folded into that rich latent space. This can be done efficiently by simply updating the underlying index to account for the newly folded data. Once this is done, tuning typically relies on embedding-based search [6]: every prompt is converted to a suitable embedding in latent space by the embeddings API, text sections generally relevant to the prompt are identified using similarity in embedding space, the prompt is augmented with the most relevant text sections, and the LLM responds accordingly. In this context, prompt engineering can be viewed as a form of query expansion, a well-established process to improve information retrieval. On top of that, further tuning often involves the equivalent of query reformulation, typically achieved through instruction prompting, role prompting, and few-shot prompting. For a list of best practices in this area, see "Prompt design strategies" [7].