Topic
14 articles
Not long ago I needed to benchmark the performance of Golang on a many-core machine. I took several of the benchmarks that are bundled with the Go source code, copied them, and modified them to run on all available threads.
Is it ok to have if clauses that will basically never be run? Surely, there must be some performance cost to that...
Cloudflare investigated CPU performance benchmark results for Workers, uncovering and fixing issues in infrastructure, V8 garbage collection, and OpenNext optimizations. These improvements have made Cloudflare Workers faster for all customers.
High-performance applications form the backbone of the modern web. At LinkedIn, a number of internal high-throughput services cater to thousands of user requests per second. For optimal user experience, it is very important to serve these requests with low latency. For example, a product our members
We are open-sourcing DCPerf, a collection of benchmarks that represents the diverse categories of workloads that run in data center cloud deployments. We hope that DCperf can be used more broadly b…
For more than 20 years, Etsy has been the destination for human creativity online. Our marketplace is home to more than 100 million special items made, handpicked and designed by more than 5 million sellers. These items and the real people behind them are what set us apart. But while the huge variety of Etsy’s inventory is one of our greatest strengths, it also creates fundamental engineering challenges specific to our marketplace. The challenge: Etsy’s unique inventory With millions of creative items across thousands of categories – many of which are unique – it’s difficult to accurately capture all possible product attributes, which range from standard attributes like “color” and “material”, to niche attributes like “bead hole size” and “slime additives.” The range of possible attributes and their values is so broad that it’s a challenge even to enumerate them, let alone label listings with specific attribute data. Unlike other online retailers (that may also have enormous inventories), because products on Etsy are listed by third party sellers and often handmade or customized, we do not have global SKUs (stock keeping units), or mappings from SKUs to product attributes. The listing below is an example of a unique item on Etsy, which has no SKU number or easy access to product attribute information. At first glance, the item looks like a t-shirt, but it is actually a porcelain sculpture. For niche items like this, seller provided details become especially critical. We collect both structured and unstructured data from sellers, and they serve different roles in our marketplace. Unstructured data comes in the form of free-text descriptions, creative titles, and listing photos. While this content is full of useful product information, it’s harder for machines to interpret consistently and quickly at scale. Structured data - in the form of product attributes like size and color - is easy for our systems to parse. It powers the buyer experience through tools such as search filtering options (offered through selectors in UI) and product-to-product comparison for characteristics of interest (material, price, etc). Filters can be seen on the left side after a search query While Etsy does ask sellers to provide structured data on their listings’ attributes, most fields are not required. This reduces friction in the listing process and gives sellers the flexibility to represent their often unique items accurately. Sellers can fill in these attributes, or leave them blank and continue through the listing process As a result, most sellers only or mostly provide unstructured data in the form of listing titles, descriptions, and photos. Frequently, key information like product dimensions is buried in the listing description or only available in listing photos. Example listing with dimensions in description Example of dimensions in a photo While our powerful search and discovery algorithms can process unstructured data such as that in descriptions and listing photos, passing in long context and images directly to search poses latency concerns. For these algorithms, every millisecond counts as they work to deliver relevant results to buyers as quickly as possible. Spending time filtering through unstructured data for every query is just not feasible. These constraints led us to a clear conclusion: to fully unlock the potential of all inventory listed on Etsy’s site, unstructured product information needs to be distilled into structured data to power both ML models and buyer experiences. LLMs present a new opportunity Before the availability of scalable LLMs, we explored various ML-based solutions to this challenge. Supervised product attribute extraction models had limited efficacy; even if we could enumerate all possible product attributes and values, many of them would be so sparse that traditional classification models would struggle to capture the long tail. Sequence tagging approaches also had difficulty scaling to multiple attributes. Transformer-based question-answering models (e.g. AVEQA, MAVEQA ) allowed for generalization to unseen attribute values, but still required large amounts of application-specific training data. This is where the availability of foundational LLMs presented a transformational opportunity for Etsy. These models have a vast amount of general knowledge from pre-training, can process large context windows quickly and affordably, and can follow instructions given a small number of examples. With a feasible and performant solution, our next focus was to build a scalable pipeline that could extract attributes across millions of listings while maintaining confidence in the LLM output. This required robust evaluation frameworks/processes that measured quality through various metrics. Transforming & evaluating unstructured data at scale Evaluation When working with LLMs, one of the biggest challenges is evaluating model performance. We needed to ensure that, at scale across our 100M+ listings, the LLMs were consistently and reliably producing accurate, actionable results. To do this, we initially worked with a third-party labeling vendor to collect a large sample of human-annotated data containing attribute annotations for listings across multiple categories. We evaluated performance by comparing LLM inferences to this human-annotated dataset and calculated metrics like precision, recall, and Jaccard index. We used these ground truth metrics as a benchmark for model improvements via prompt and context engineering. Unfortunately, there were several significant drawbacks to relying on human-labeled data. In many cases, we found that human annotators made mistakes, especially when annotating thousands of listings (after all, no one’s perfect). In the example below, a human annotator marked the light fixture as ½ inch width, while the LLM correctly extracted 5.5 inches. Furthermore, human labeling is more time-consuming and expensive. To start scaling attribute inference across thousands of categories, we needed to come up with an automated process for labeling that did not rely exclusively on human annotation. Instead, we’ve started using high-performance, state-of-the-art LLMs to generate ground truth labels (often called “silver labels”). Human-in-the-loop is still an essential part of this process: Etsy domain experts review silver labels and iterate on the prompt to ensure high quality results. Once we’re confident in our silver label generation, we produce a larger dataset for evaluating a more scalable LLM. The diagram below shows the updated process for model development. Inference The core of our LLM pipeline is context engineering. We’ve worked with partners in product, merchandising, and taxonomy to ensure that the LLM has the right context for attribute extraction, including: Seller-provided listing data, including listing titles, descriptions, and images Few-shot examples hand-selected by domain experts Business logic from Etsy’s product taxonomy Category-specific extraction rules Each listing is represented as a JSON string of context information. This context is injected into a series of prompts to extract product attributes in parallel. LLM requests are routed through LiteLLM to different regions, ensuring higher parallelization and removing a dependency on one singular cloud location. Finally, LLM responses are parsed into Pydantic dataclasses, which provide both basic type validation and custom validation based on business logic. After this process of inference completes, a post-processing job formats the validated, structured outputs. The data is then exported to filestores, database tables, and our search platform for consumption by partner teams. Monitoring Beyond the challenges of evaluating the LLM output, Inference itself may fail for many reasons: code bugs, permissions issues, transient errors, quota exceeded errors, safety filters, and more. Rather than failing the pipeline for any individual error, errors are logged, and error metrics are surfaced via our observability platform. Our team is alerted if the number of failed inferences exceeds a certain threshold. To support debugging, we log a sample of traces to HoneyComb. Even if the error rate is low, it’s possible that model performance has degraded. To track changes in model performance, we added performance evaluation to our pipeline. First, we run LLM inference on a sample of a ground-truth dataset, and calculate performance metrics like precision, and recall. These metrics are compared to baseline scores from the full ground-truth dataset. If any metrics deviate significantly, the pipeline is terminated. This process allows us to confirm that third-party LLMs are working as expected before we run production-scale inference. The combination of tracing, logging, metric tracking, model performance evaluation, and alerting provides a complete understanding of both pipeline health and model performance metrics, enabling us to consistently transform data to power key shopping experiences with confidence at scale. Looking Forward Where we’ve applied LLM-generated product attribute data to buyer and seller-facing experiences, we’ve seen promising results. In target categories, we’ve increased the number of listings with complete attribute coverage from 31% to 91%. And earlier this year, we added LLM-inferred attributes to search filters, leading to more engagement from buyers: Engagement with relevant Search filters increased Overall post-click conversion rate increased All this work combined most recently into leveraging LLM-inferred color attributes to display color swatches for each listing on the search results page. This provides at-a-glance additional information to our buyers to find exactly what they want, faster. What's Next Our goal is to unlock the full potential of Etsy’s inventory. Product attribute extraction is just one of many ways we’re using LLMs to achieve this in our efforts to improve the shopping and selling experience on Etsy. Transforming unstructured information is enabling us to make it easier than ever for our buyers to discover exactly what they’re looking for – and easier for sellers to list and get their unique creations discovered by the shoppers seeking their special item.
How the GitHub Issues team used client-side caching, smart prefetching, and service workers to make navigation feel instant. The post From latency to instant: Modernizing GitHub Issues navigation performance appeared first on The GitHub Blog.
Monorepo performance can suffer due to the sheer number of files in your working directory. Git’s new builtin file system monitor makes it easy to speed up monorepo performance.
Gisting compresses context into a set of learned tokens, preserving its quality while making the model faster and cheaper.
Rarely are there opportunities for big, bold, game-changing improvements in web performance. The Speculation Rules API (SRA) is a recent browser development that offers just such an opportunity. This post details a joint effort between the search and the web performance teams at Etsy to implement SRA on Etsy search pages and drastically improve the performance of product listing pages with some metrics seeing 20-24% improvements and some dropping all the way to 0ms values. Prefetching Options There are two main ways to predictively fetch resources for the next page: <link rel="prefetch"> allows us to instruct the browser to download a resource we believe we'll need soon. The resource can be anything, like a static asset or an HTML page. Speculation Rules API (SRA) is a recently updated browser API which allows for a JSON definition to dictate what page A wants to do with page B. The "do"-ing can be either prefetching (just download the HTML) or prerendering (load the page, including its static assets, and render it completely). The prerendering happens in a new browser process and page B is ready to be swapped with the current page A instantaneously when the user navigates to B. While implementing full prerendering is likely to yield more impressive performance improvements, it is a bigger and riskier investment, mostly related to the side effects of executing the JavaScript on the target page B. Starting with prefetching is a good first step into exploring SRA. The benefits of using SRA over link prefetch will become evident further in the article but the topline highlights include: a simpler API to define what is prefetched (via a CSS selector rather than do-it-yourself bespoke JavaScript), convenient utilities to define when, as well as where (memory and HTTP cache) prefetching happens, and a built-in upgrade path to full on prerendering. Implementing Speculation Rules The Search team at Etsy recently ran an A/B experiment to use the Speculation Rules API to prefetch the listing page when hovering over organic listing cards on the desktop search page on Chromium browsers. To do this, we added a new <script type="speculationrules"> tag to the search page with JSON that defines how we want the prefetching to work, like so: <script type="speculationrules"> { "prefetch": [{ "where": { "and": [ { "href_matches": "/{*/}?listing/*" }, { "selector_matches": "[data-sr-prefetch='1']" } ] }, "eagerness": "moderate" }] } </script> This instructs the browser to download the HTML for a listing page when: the user hovers over a link to a listing page for 200 milliseconds (defined by the “moderate” eagerness property), and the link has a data-sr-prefetch attribute The data attribute allows us to more precisely opt in pages that are eligible for prefetching. Lessons learned In terms of changes to the page’s code, the implementation of SRA was straightforward. As such, we spent most of our time testing that everything was working as expected and that our systems and analytics were not inadvertently affected. And we found some surprises along the way, related to all the little details modern web pages use (such as cookies, redirects, new tabs). Allow us to share a few lessons in prefetching... Two ways to prefetch As mentioned earlier, prefetching can happen one of two ways: <link rel="prefetch"> or speculative prefetch (the one using SRA). So what sets them apart? They do work mostly the same, except that the speculative prefetch caches the page in both memory and the HTTP cache. The <link> only uses the HTTP cache and merely downloads the specified resource. This makes the SRA way of prefetching more advantageous than the <link> prefetch because of the memory cache. Two speculative prefetches only We also discovered the number of prefetched pages that are kept in memory is restricted to two. When you prefetch a third page, the first one is evicted from the memory cache. The HTTP cache still works as usual. So again, the SRA prefetching is preferable to the <link> prefetch due to the difference in caching we just described. It’s helpful while debugging to be aware of the eviction of the prefetched page from memory. But rest assured, the downloaded page is still cached locally. Eagerness While <link rel="prefetch"> advises the browser to load a resource as soon as it sees the <link> in the DOM, the speculative prefetch is more nuanced, offering eager, immediate, conservative and moderate loading. We selected moderate eagerness, which prefetches after the user has hovered over a link for 200ms. Exploring our options we found that the immediate eagerness would trigger a significantly larger number of prefetches (since it executes immediately and prefetches all eligible pages), and we wanted to avoid creating new server requests for listing cards with a low likelihood of being clicked. However, the immediate eagerness setting could be worth considering if the cost of additional requests is very low. The conservative eagerness executes on pointer or touch down, providing a very small head start over normal browser behavior and therefore greatly reducing the potential benefits of prefetching. Conservative eagerness may only be suitable for a use case in which it is necessary to avoid unused prefetches altogether. Note that eager and immediate were synonyms in the initial SRA implementation, but that is changing. Keep an eye on the official docs for updates. Speculations and new browser tabs Initially, SRA launched without the ability to prerender pages that open in new tabs, as Etsy listings do. This option was added later, but only for values _blank of the target attribute of the link elements, not named target attributes such as the ones that Etsy uses, for example <a href="/listing/123" target="etsy.123456">. Fortunately, the target restriction doesn't apply to prefetching, so for SRA prefetching (unlike SRA prerendering) there's no problem, regardless of whether or how you specify a target at all. For developers who may be considering moving from prefetching to prerendering, this distinction is something to bear in mind. 5-minute rule Because of the complex nature of listing pages, Etsy's HTML pages are non-cacheable. However, the speculative prefetch keeps the prefetched pages cached in memory for five minutes. This was a helpful learning, as there would be no point of using speculative prefetches at all if they expire immediately. After five minutes, the normal caching rules apply, set via HTTP headers such as Max-age or Expires. Given that only two pages are currently kept in memory cache and all others expire because they are non-cacheable, the benefits are greatly reduced when, for example, a person hovers over 3 links and eventually clicks the first one which leads to a page that's already expired from the prefetch memory (and HTTP!) cache. To aid with the two-page restriction, one strategy we devised is to make our pages cacheable for five minutes when we detect a prefetch request. Such requests are identifiable because the browser sends Sec-Purpose: prefetch HTTP header when prefetching. This helps preserve downloaded pages that would’ve otherwise expired from both memory and HTTP cache. Video links and shadow DOM Often, listings on Etsy include product videos, which start to play on the search results page when a user hovers their mouse over them. In these cases, prefetching doesn't work: the mouse hover is effectively "swallowed," disappearing into the shadow DOM of the browser's video player. One workaround is to overlay a div on top of the video for 200ms to let the hover register in the DOM. Then, after the 200ms has elapsed, remove the extra div to let the browser video controls (e.g., on right click) work as usual. You can find a demonstration of this technique here. Cookies If a page sets cookies, prefetching it will set those cookies as well (as demonstrated here). This is something to be aware of, as the prefetch may end up being unused. This may confuse your application (and/or analytics) to thinking a page has been visited where in reality it was not. Again, you can use Sec-Purpose: prefetch HTTP header to detect prefetch requests and avoid setting the cookie as part of the prefetching process. Redirects If the link to the page being prefetched goes through a redirect, the actual page after the redirect is still being prefetched. Let’s say you have a sequence that looks like this: Link on Page A -> redirect -> Page B Here the browser follows the redirect during prefetching and still caches Page B. When the user then clicks the link on page A leading to Page B, the browser follows the usual process of going through the redirect. Normal HTTP cache rules still apply, meaning that if the redirect is cached, it won’t need to be requested again. So, even though redirects are a bad performance practice, if you need to do them, they do not affect prefetching as long as you set appropriate caching headers. Mutating hrefs Sometimes the href attributes of <link> elements get modified by JavaScript on mouse hover. This does not play well with prefetching. Imagine you have: <a href="link.html">Follow me</a> … which changes on hover to: <a href="link.html?source=footer">Follow me</a> When the user hovers over the link, the browser starts working on prefetching link.html but realizes that the link to that page is no longer in the DOM and abandons the process. So the page is not prefetched even if ?source=footer doesn’t change the target page in any way other than reporting analytics. The browser has no way of knowing this and considers the two as separate pages. Additionally, the failed attempt at prefetching link.html counts in the “two speculations only” rule and evicts the older speculative load from the memory cache. For best results, avoid modifying links on hover. Analytics and Event Logging This is the elephant in the room. Many sites on the web today were built in a world where prefetching did not exist. So there is one big assumption: that a page load is always initiated by the user and the load can be counted as such – either server-side during page construction or client-side by JavaScript after the page is loaded (or, as it often happens, a combination of the two). With prefetching, this assumption is no longer true. A page constructed on the server-side and downloaded by the browser does not necessarily mean the page has been seen (and therefore its JavaScript has been executed). This can result in a number of miscalculations when it comes to analytics. Luckily, browser APIs such as the Sec-Purpose HTTP header and JavaScript APIs (document.prerendering and prerenderingchange event) allow us to tell prerender requests from user-generated ones, as well as when a prerendered page is "activated" (when the user actually sees a prefetched page). For prefetches, Performance Resource Timing’s deliveryType method of navigational-prefetch can be used for the purposes of analytics. We (and our analytics partners) found this to be the hardest part: ironing out the required analytics updates so that numbers remain true after implementing speculation rules. In our particular use case, we intentionally pursued a strategy of prefetching the destination page instead of prerendering it, meaning that no assets would be loaded and JavaScript would not execute on our prefetches. This gave us a relatively simple way to handle the accuracy of our analytics. A foundational piece of our analytics is event logging. For example, in the controller of the listing page we log a view_listing event that contains key information such as the listing ID, user ID, etc. This informs not only our site analytics, but also our search training pipeline, recently viewed listing data for users, and more. We ended up creating a system to cache the payload of all events within a request to avoid firing those events during prefetches. We were then able to move that event logging to the destination page’s JavaScript bundle, deferring them until after page “activation” and mitigating the impact of prefetching on our analytics. Results We were thrilled with the performance results of the prefetching experiment. We saw a 20-24% improvement in many performance metrics we care about: TTFB, DOMContentLoaded, FCP, LCP. The 75th percentile time to first byte (TTFB) on the listing page improved by 23.6% We saw similar improvements throughout the request: First Contentful Paint -20.7%, Largest Contentful Paint -21.1%, DOMContentLoaded -20.4%, and Page Load -10.6% In the cumulative distribution function below, we see the control of our experiment (no speculation rules prefetches) in blue, and the treatment (speculation rules prefetches) in orange, with the treatment dramatically faster than the control at every percentile. Remarkably, about 40% of eligible browsers saw their TTFB drop nearly to zero: We saw small but detectable improvements in some business metrics, which is promising given that listing page views come from many sources, only some of which are search results. As we implement more prefetching in more places, we hypothesize that the numbers will further improve. When people approach SRA implementation they may be worrying about unused prefetches and resource waste. In our experiment we saw a ratio of about 14:1 for the number of prefetches requested to subsequently activated pages (i.e., about 1 in 14 prefetch requests was navigated to by the user). We’re encouraged by these results, and are looking forward to new opportunities to improve performance across additional surfaces. Opportunities to iterate and expand One clear opportunity is to try implementing prefetching on other pages beyond Search. Shoppers end up on product listing pages from various other referral surfaces: shop pages, our SEO-optimized landing pages, home page, etc. Prefetching could improve performance on these surfaces, leading to a better experience for Etsy buyers. Another opportunity is to consider upgrading our prefetching to prerendering in the future. This would be a significant change to client-side JavaScript code operating during prefetches. However (and it's hard to contain the excitement about this!) Chrome is working on prerender-until-script update, which means prerendering stops at the first <script>. Even if you have <script> high up in the <head> of your page and prerendering halts early, the browser will still download page resources (scripts, styles, images, fonts) and have them ready. For our use case, enabling prerender-until-script would mean that frontend performance metrics downstream of TTFB, such as First/Largest Contentful Paint, would likely see even larger improvements, and users would be able to interact with the listing page even earlier. This would further reduce friction for users when browsing on Etsy, letting them spend less time watching web pages load and more time engaging directly with our sellers’ amazing inventory of items. Acknowledgements Implementing SRA was truly a cross-team effort, not only by the search front-end and web performance teams but also people from infrastructure, analytics, ranking, and recommendations. Special thanks to Paul Calvano from our Web Performance team, Diana Sanchez Urban from Search Experience Web, and Eileen Toomer from the Visits team! This project also benefited from input from members of Recommendations and Listing Page teams, as well as members of our internal Architecture Advisory Group.
Ever searched for something specific, only to be met with results that are close, but not quite? On Etsy’s Search Relevance team, that frustration is exactly what we are tackling. Our goal is simple yet ambitious: to help buyers find exactly what they’re looking for, and to help sellers reach the people seeking their special products. Search plays a central role in that mission. Historically, Etsy’s search models have relied heavily on engagement signals – such as clicks, add-to-carts, and purchases – as proxies for relevance. These signals are objective, but they can also be biased: popular listings get more clicks, even when they’re not the best match for a specific query. To address this, we introduce semantic relevance as a complementary perspective to engagement, capturing how well a listing aligns with a buyer’s intent as expressed in their query. We developed a Semantic Relevance Evaluation and Enhancement Framework, powered by large language models (LLMs). It provides a comprehensive approach to measure and improve relevance through three key components: High quality data: we first establish human-curated “golden” labels of relevance categories (we’ll come back to this) for precise evaluation of the relevance prediction models, complemented by data from a human-aligned LLM that scales training across millions of query-listing pairs Semantic relevance models: we use a family of ML models with different trade-offs in accuracy, latency, and cost; tuned for both offline evaluation and real-time search Model-driven applications: we integrate relevance signals directly into Etsy’s search systems enabling both large-scale offline evaluation and real-time enhancement in production Together, this framework brings a more intent-aware search experience that better serves both buyers and sellers across our marketplace. Figure 1. Overview of the Semantic Relevance Evaluation and Enhancement Framework Capturing Shades of Relevance Let’s return to the idea of relevance categories. Based on user research, we define three categories for semantic relevance of query-listing pairs: Relevant: listing matches all parts of the query, accounting for meaning and proper nouns Partially relevant: listing matches part of the query or is thematically related but not a full match Irrelevant: listing has no meaningful connection to the query; its presence in top results would make the search feel broken Figure 2. Examples for the three relevance categories. Text highlighted in green shows how the product aligns with the search query, whereas red highlights indicate mismatches.* In an ideal world, we’d rely on human judgments for all query-listing pairs. But large-scale human annotation is time-consuming and expensive, rendering it infeasible. Instead, language models unlock the ability to generate these judgments at scale, transforming our ability to make every search on Etsy produce more relevant results. Data: Anchored by Humans, Scaled by LLMs With recent advances in LLMs, a promising approach to evaluate search relevance is to use LLM-as-a-judge: directly using LLMs to judge the relevance of our search system without looping in humans. However, this approach faces two main challenges: Domain shift: off-the-shelf LLMs may not capture the unique preferences and vocabulary of Etsy users Performance-cost tradeoff: larger LLMs offer stronger reasoning but are expensive for large-scale inference, while smaller LLMs are faster and cheaper, but less accurate To address these challenges, we start with human-curated golden labels to evaluate and align a powerful LLM with these human-labels, then use a full dataset scaled up by the LLM for training our relevance judge. In other words, humans define what good looks like, and LLMs help us scale it. LLMs do not replace human judgment, instead they align with and amplify it. We maintain a detailed, evolving relevance labeling guideline, continuously refined through user research and annotation feedback. What relevance means in our marketplace shifts over time and social context. For example, people searching for “face masks” pre-2020 were primarily looking for masks for costumes or fashion, which is a completely different intent from protective masks post-2020. These guidelines ensure our definitions of relevance accurately reflect Etsy users’ intent and capture cultural trends over time. Query-listing pairs are sampled from search logs using a mix of approaches, including both random, stratified sampling for broad coverage, and targeted sampling for challenging cases. Each query-listing pair is labeled by two Etsy admins, with an ongoing review process to both break ties and adjust labeling guidelines accordingly. For quality control, we continuously track metrics such as row-level disagreement rates, which measures how often multiple annotators disagree with each other for the same query-listing pair. To scale beyond manual annotation, we introduced a few-shot, chain-of-thought (CoT) prompting strategy using the o3 model, implemented in LangGraph. The prompt instruction is inspired by the annotation guidelines described above, and includes comprehensive query and listing features, like title, images, text description, attributes, variations, and extracted entities (read more about listing extracted entities in another one of our posts). We also applied self-consistency sampling to improve reliability. This model, known as the LLM annotator (as seen in Figure 1), is first validated against the human-labeled golden data to ensure its judgement aligns with humans. Once validated, we use it to generate large-scale training data to develop the production models. The LLM annotator thus serves as the foundation for our teacher-student modeling pipeline, bridging the gap between expensive manual labeling and scalable automated annotation. Models: Balancing Accuracy, Latency and Cost Our modeling pipeline uses a three-tier cascaded distillation design, where each model balances accuracy and efficiency for a specific purpose. The stack includes: The LLM annotator: our most accurate and cost-intensive model, aligned closely with human-labeled golden data The teacher model: a fine-tuned smaller LLM (Qwen 3 VL 4B) that delivers high-throughput annotation at scale The student model: a lightweight, BERT-based two-tower model optimized for real-time inference The LLM annotator aligns best with the golden labels, but is too costly for recurrent, large-scale inference. To reduce cost while maintaining quality, we performed supervised fine-tuning (SFT) with a smaller LLM, Qwen 3 VL 4B, using the training data generated by the LLM annotator. This teacher model preserves human alignment while enabling us to label millions of query-listing pairs daily, which is ideal for recurring evaluation and monitoring. The teacher, however, is too slow to surface relevant search results quickly, which is critical for helping our sellers reach potential buyers. As such, we further distilled the teacher into a student model with a two-tower architecture. The distillation process aligns the student’s output with that of the teacher, so that the student judges relevance labels nearly as accurately as the teacher, while being lightweight and fast. The resulting model ensures we deliver search results almost as fast as before, with only <10ms additional latency. All three models – the LLM annotator, teacher, and student – are evaluated against the same golden dataset to ensure traceable performance and consistent alignment with human judgment. Figure 3 shows their accuracy measured using multi-class Macro F1, and individual class F1 scores. Figure 3. Performance of semantic relevance models against human golden labels Applications: From Evaluation to Action With these models in place, we can both measure and enhance search relevance across Etsy. Search relevance evaluation We use the teacher model to measure how well our search system surfaces relevant listings. Each day, we sample search requests and perform offline inference using the teacher model, then aggregate the predicted relevance labels into summary metrics. These metrics are reviewed regularly by our team, and if we observe unexpected trends like a sudden decline of relevance, we work to quickly diagnose and address the problem. Similarly, we monitor relevance metrics in A/B tests. The computed relevance metrics are discussed when we decide whether to roll out a new change to our search system, to ensure those changes affect semantic relevance of search results in a neutral to positive way. We sample sufficient amounts of requests separately from control and treatment variants, to ensure statistical power. Using vLLM for high-throughput inference, we process millions of query-listing pairs daily at a very low cost, maintaining both statistical power and operational efficiency. Improving search in production The lightweight student model is embedded directly into Etsy’s real-time search stack. It improves relevance through several integration points: Filtering: removes retrieved listings predicted as irrelevant before downstream ranking Feature enrichment: contributes model-predicted relevance scores as features for the downstream ranking model Loss weighting: adjusts training weights of the ranking model based on predicted relevance Relevance boosting: promotes listings deemed highly relevant using heuristic rules among the final returned search results How Semantic Relevance is Changing Etsy Search The Semantic Relevance Evaluation and Enhancement Framework is fully deployed in Etsy’s search stack, and continues to evolve. We’ve observed a measurable uplift in semantic relevance: the percentage of fully relevant listings (as defined by the relevance categories described earlier) has increased from 58% to 62% between August and October 2025. Figure 4. Improvement of semantic relevance metrics over time This improvement reflects Etsy’s growing ability to align search results with buyer intent. For instance, in searches like “fall decor,” the enhanced search engine now focuses on surfacing seasonal decor items, while deprioritizing loosely related listings like clothing, which appeared before the enhancement on relevance. Figure 5. Before and after comparison when searching for “fall decor” * Beyond these immediate gains, semantic relevance has shifted how we evaluate and improve search at Etsy, by adopting a user-centered approach. By grounding our evaluation in semantic intent in addition to behavioral signals, we move closer to our goal of connecting buyers with the relevant products, not just the most popular ones. While search results are influenced by multiple factors, and outcomes may vary, on the seller side, improving semantic relevance can also help surface items from small or new sellers who may not yet have the visibility of more established shops. What’s Next In ongoing and future efforts, we hope to explore the following directions: Better understanding of relevance-engagement dynamics. In online experiments, we often observe engagement metrics decline even as semantic relevance improves (a pattern also noted by other e-commerce platforms). We suspect this results from applying uniform relevance treatments despite contextual variation. Next, we plan to explore adaptive strategies that tailor adjustments by query type. Refining partial relevance. Inspired by Amazon’s ESCI framework, we’re exploring finer-grained labels, for example, introducing new subcategories of complements and substitutes. This could potentially improve evaluation precision and power new user search experiences. Reducing annotation effort through LLM facilitation. When LLM judgments are self-consistent, they align better with human labels. This may indicate easier query-listing pairs. We are exploring using LLMs for these easy cases, focusing human effort on more complex cases. Simplifying the multi-stage model stack. Our current three-tier distillation pipeline provides flexibility but adds operational complexity. We plan to simplify this setup by exploring better performance-efficiency tradeoffs and potentially merging model tiers. Improving relevance in retrieval. So far, post-retrieval filtering is the first stage where our semantic relevance model applies. We see strong potential to enhance both inference and measurement further upstream in the retrieval layer. Conclusion Key takeaways: LLMs can meaningfully evaluate search relevance when grounded in human judgment. Aligning LLM assessments with human-labeled data ensures we measure, and continually improve, the search experience that is so essential to connecting buyers and sellers on Etsy. Semantic relevance redefines how Etsy optimizes search. By complementing engagement metrics with semantic relevance, we address real customer pain points and deliver more satisfying search experiences. Teacher-student distillation offers a flexible and efficient way to apply relevance modeling across diverse performance, latency and cost requirements. Ultimately, improving semantic relevance strengthens the human connections that define Etsy. By understanding what shoppers truly mean, we can help them find the right items. And by emphasizing relevant listings over popular ones, we can help create fairer opportunities on the search relevance factor of search visibility for our sellers – 89% of whom are businesses of one. Acknowledgments This work is brought to you in a collaborative effort by the Search Relevance Team, enabled by ML Enablement, and the Merchandising teams. Thanks to the following contributors Data: Susan Liu, Jugal Gala, David Blincoe, Yuqing Zhang, Taylor Hunt, Liz Mikolaj Models: David Blincoe, Oriane Cavrois, Orson Adams, Yuqing Zhang Application: Grant Sherrick, Kaushik Bekal, Haoming Chen, Patrick Callier, Davis Kim, Marcus Daly Product leadership: Julia Zhou, Willy Huang, Argie Angeleas Engineering leadership: Yinlin Fu, Congzhe Su, Xiaoting Zhao ML Enablement partners: Ari Carter, Stan Schwertly, Shreya Agarwal, K Ogilvie, Marvin Wang, etc. Other cross-team partners: Will Beckman, Karl Yokono, Audrey Chen, Heather Campbell, David Le, Khadeeja Din, etc. Early contributors: Ethan Benjamin, Cung Tran, Maggie Matsui, Jack Gammack, Yogeeta Chatoredussy, Austin Clapp, Benjamin Russell, Khaled Jabr Special thanks to Oriane Cavrois & David Blincoe for helping this piece come to life. * Images are provided for illustrative purposes. Item availability on Etsy may vary.
GitHub has done a lot of work in the last six months to improve performance for large monorepos.
TL; DR At Meta’s scale, a few milliseconds of latency degradation can have a significant negative impact on ads performance. When a Linux kernel upgrade risked regressing latency across Meta’s ad serving fleet, we turned to sched_ext — the upstream, BPF-based extensible scheduling framework — to build a scheduling policy customized to the Ads delivery [...] Read More... The post Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler appeared first on Engineering at Meta.
In 2020, Etsy concluded its migration from an on-premise data center to the Google Cloud Platform (GCP). During this transition, a dedicated team of program managers ensured the migration's success. Post-migration, this team evolved into the Etsy FinOps team, dedicated to maximizing the organization's cloud value by fostering collaborations within and outside the organization, particularly with our Cloud Providers. Positioned within the Engineering organization under the Chief Architect, the FinOps team operates independently of any one Engineering org or function and optimizes globally rather than locally. This positioning, combined with Etsy's robust engineering culture focused on efficiency and craftsmanship, has fostered what we believe is a mature and successful FinOps practice at Etsy. Forecast Methodology A critical aspect of our FinOps approach is a strong forecasting methodology. A reliable forecast establishes an expected spending baseline against which we track actual spending, enabling us to identify deviations. We classify costs into distinct buckets: Core Infrastructure: Includes the costs of infrastructure and services essential for operating the Etsy.com website. Machine Learning & Product Enablement: Encompasses costs related to services supporting machine learning initiatives like search, recommendations, and advertisements. Data Enablement: Encompasses costs related to shared platforms for data collection, data processing and workflow orchestration. Dev: Encompasses non-production resources. The FinOps forecasting model relies on a trailing Cost Per Visit (CPV) metric. While CPV provides valuable insights into changes, it's not without limitations: A meaningful portion of web traffic to Etsy involves non-human activity, like web crawlers that’s not accounted for in CPV. Some services have weaker correlations to user visits. Dev, data, and ML training costs lack direct correlations to visits and are susceptible to short-term spikes during POCs, experiments or big data workflows. A/B tests for new features can lead to short-term CPV increases, potentially resulting in long-term CPV changes upon successful feature launches. Periodically, we run regression tests to validate if CPV should drive our forecasts. In addition to visits we have looked into headcount, GMV(Gross Merchandise Value) and revenue as independent variables. Thus far, visits have consistently exhibited the highest correlation to costs. Monitoring and Readouts We monitor costs using internal tools built on BigQuery and Looker. Customized dashboards for all of our Engineering teams display cost trends, CPV, and breakdowns by labels and workflows. Additionally, we've set up alerts to identify sudden spikes or gradual week-over-week/month-over-month growth. Collaboration with the Finance department occurs weekly to compare actual costs against forecasts, identifying discrepancies for timely corrections. Furthermore, the FinOps team conducts recurring meetings with major cost owners and monthly readouts for Engineering and Product leadership to review forecasted figures and manage cost variances. While we track costs at the organization/cost center level, we don't charge costs back to the teams. This both lowers our overhead and more importantly, provides flexibility to make tradeoffs that enable Engineering velocity. Cost Increase Detection & Mitigation Maintaining a healthy CPV involves swiftly identifying and mitigating cost increases, to achieve this we: Analysis: Gather information on the increase's source, whether from specific cloud products, workflows, or usage pattern changes (ie variance in resource utilization). Collaboration: Engage relevant teams, sharing insights and seeking additional context. Validation: Validate cost increases from product launches or internal changes, securing buy-in from leadership if needed. Mitigation: Unexpected increases undergo joint troubleshooting, where we outline and assign action items to owners, until issues are resolved. Communication: Inform our finance partners about recent cost trends and their incorporation into the expected spend forecast post-confirmation or resolution with teams and engineering leadership. Cost Optimization Initiatives Another side of maintaining a healthy CPV involves cost optimization, offsetting increases from product launches. Ideas for cost-saving come as a result of collaboration between FinOps and engineering teams, with the Architecture team validating and implementing efficiency improvements. Notably we focus on the engineering or business impact of the cost optimization rather than solely on savings, recognizing that inefficiencies often signal larger problems. Based on effort vs. value evaluations, some ideas are added to backlogs, while major initiatives warrant dedicated squads.Below is a breakout of some of the major wins we have had in the last year or so. GCS Storage Optimization - In 2023 we stood up a squad focused on optimizing Etsy’s use of GCS, as it has been one of the largest growth areas for us over the past few years. The squad delivered a number of improvements including improved monitoring of usage, automation features for Data engineers, implementation of TTLs that match data access patterns/business needs and the adoption of Intelligent tiering. Due to these efforts, Etsy’s GCS usage is now less than it was 2 years ago. Compute Optimization - Migrated over 90% of Etsy infrastructure that is serving traffic to the latest and greatest CPU platform. This improved our serving latency while reducing cost. Increased Automation for model deployment - In an effort to improve the developer experience, our machine learning enablement team developed a tool to automate the compute configurations for new models being deployed, which also ended up saving us money. Network Compression - Enabling network compression between our high throughput services both improved the latency profile and drastically reduced the networking cost. What's Next While our core infrastructure spend is well understood, our focus is on improving visibility into our Machine Learning platform's spend. As these systems are shared across teams, dissecting costs tied to individual product launches is challenging. Enhanced visibility will help us refine our ROI analysis of product experiments and pinpoint future areas of opportunity for optimization.