Topic
17 articles
We’re sharing lessons learned from shifting our Android development from Java to Kotlin. Kotlin is a popular language for Android development and offers some key advantages over Java. As of today,…
Meta has been on a years-long undertaking to translate our entire Android codebase from Java to Kotlin. Today, despite having one of the largest Android codebases in the world, we’re well pas…
We developed a new static analysis tool called Nullsafe that is used at Meta to detect NullPointerException (NPE) errors in Java code. Interoperability with legacy code and gradual deployment model…
I’m excited to announce the next step in LinkedIn’s service infrastructure: the Play Framework. Play is a modern web framework that combines the performance and reliability of Java and Scala, the power of reactive programming, and the productivity of full hot reload support. We’ve been running Play
By Harshad Sane
High-performance applications form the backbone of the modern web. At LinkedIn, a number of internal high-throughput services cater to thousands of user requests per second. For optimal user experience, it is very important to serve these requests with low latency. For example, a product our members
Getting real with virtual threads
I had this conversation over and over before I really understood it. It goes: “X language can be as fast as a compiled language because it has a JIT compiler!” “Wow! It’s as fast as C?” “Well, no, we’re working on it. But in theory, it could be even FASTER than C!” “Really? How would that work?” “A JIT can see what your code does at runtime and optimize for only that. So it can do things C can’t!” “Huh. Uh, okay.” It gets hand-wavy at the end, doesn’t it? I find that frustrating. These days I work on YJIT, a JIT for Ruby. So I can make this extremely NOT hand-wavy. Let’s talk specifics. I like specifics. Wait, What’s JIT Again? An interpreter reads a human-written description of your app and executes it. You’d usually use interpreters for Ruby, Python, Node.js, SQL, and nearly all high-level dynamic languages. When you run an interpreted app, you download the human-written source code, and you have an interpreter on your computer that runs it. The interpreter effectively sees the app code for the first time when it runs it. So an interpreter doesn’t usually spend much time fixing or improving your code. They just run it how it’s written. An interpreter that significantly transforms your code or generates machine code tends to be called a compiler. A compiler typically turns that human-written code into native machine code, like those big native apps you download. The most straightforward compilers are ahead-of-time compilers. They turn human-written source code into a native binary executable, which you can download and run. A good compiler can greatly speed up your code by putting a lot of effort into improving it ahead of time. This is beneficial for users because the app developer runs the compiler for them. The app developer pays the compile cost, and users get a fast app. Sometimes people call anything a compiler if it translates from one kind of code to another—not just source
At the beginning of this year, we ran several experiments aimed at reducing the latency impact of the Ruby garbage collector (GC) in Shopify's monolith. In this article, Jean talks about the changes we made to improve GC performance, and more importantly, how we got to these changes.
We’ve spent a lot of time on performance tuning our Apache Flink application. We’ll walk you through key lessons for optimizing large stateful Apache Flink applications
Every once in a while I run into an obscure computer technology that is a hidden gem, which over the years has become mostly forgotten. This is exactly how I feel about the tcpdump tool and its kernel counterpart the packet filter interface.
We all know the stereotypes of working at enterprise tech companies: endless bureaucracy and behemoth codebases running on a years-old language version. Most organizations know they should modernize quickly, but haven’t figured out how to build an upgrade path compatible with a large codebase. The recurring cost of upgrading keeps most large JVM codebases trailing behind the latest Java versions.
Etsy is home to over 100 million listings from 5.6 million active sellers. Because the items for sale are unique and creative, there is no standard product catalog that tells us what they are. When someone searches for “light linen dress for summer,” our models must infer what the shopper means and what the listings contain. They do that using data from user visits and Etsy listings. We use Kafka for streaming data. These data streams provide fresh features and embeddings that power our machine learning models. If a shopper favorites a hand-painted ceramic mug, that action can shape their recommendations within minutes. But building these streaming applications hasn’t always been a smooth process. A new app requires templated code, configs, tests, validation, deployment steps, and careful review. That developer experience has been a strain on engineers across the company. The Olden Days Before coding agents were broadly available at Etsy, engineers turned machine learning designs into Kafka applications by hand. A streaming feature might require filtering events from terabyte-scale topics, writing transformation logic, and passing outputs through filtered topics to make them available in the feature store. The challenge wasn't writing Java or YAML. It was choosing the right schema, identifying the correct base class, validating input, anticipating edge cases, and running local tests. While static templates could generate files, they couldn't streamline the iterative, time-consuming process. Old process, human developer does everything Kafka Skills to the Rescue Our team provides the infrastructure and platform support for Etsy engineers to create and manage new features and embeddings with Kafka, among other streaming data products. We decided to make this easier with Agent Skills, reusable playbooks that teach language models how to execute specific tasks. For each workflow, the agent uses the Skills and tools we built to generate code, validate configs, run apps locally, and create PRs. Collaborative process between Human developer and Kafka agent The Kafka Skills we created are conversational workflows that allow developers and LLMs to collaborate. They can be invoked with natural language or a slash command such as /new-streaming-feature. Conversation with Claude using Kafka Skill to create a new ML feature For each streaming feature pipeline, the agent creates a Java class that implements a shared interface. The YAML configuration files are fill-in-the-blanks. The agent uses scaffolding scripts, templated code generation, and other tools that we created to output deterministic code. We saw an opportunity to have the Kafka Skills orchestrate all the steps to run an application locally. The human developer no longer needs to create test topics, build JARs, fill in manual commands and read from test topics. The agent will do all that, report back on findings, and generate all the relevant output links. Pull request and verification links posted for the developer’s review Observability We wanted to know which Kafka Skills are being used, how often they make it to pull requests, which environments and application types are involved, and where developers may still need support. To do this, we added lightweight telemetry around the workflow: when a Skill is invoked, a hook fires and a Python helper emits a started event. When the agent opens a pull request, a wrapper emits a completion event. Both events flow through an existing Etsy service that publishes a protobuf message to Kafka and stores it in a BigQuery table. Recording developer use of Kafka Skills in BigQuery Build It and They Won't Come We tested and demoed Kafka Skills and were very pleased with our “Agentic Improvements”. When we looked at adoption rates though, we were surprised to find that developers weren’t using them. They weren’t using the Skills to create features, generate embeddings, or even turn Kafka topic records into HTTP requests. Feeling a bit defeated, we asked devs: why? They responded that they simply didn’t know about them. The devs were using coding agents, but the agents never offered the Kafka Skills. Before the Skills existed, the standard workflow for feature creation was to find the closest existing feature, copy the code for it, and modify the fields. Instead of starting the process by opening the LLM and saying "I'd like to create a new streaming feature," an engineer might engage with the agent halfway through the old process and ask to "copy this class and change it to count cart moves," or "edit this file" with a path lifted from a previous pull request. Another challenge we found was that requests often arrive in whatever vocabulary the developer's team uses. When creating a new feature with Kafka, the dev might refer to a specific feature-store entry, an internal event name, a parent Java class name needed, or an output data type. To address these issues, we enumerated entry points for different vocabulary, lifecycle stages, and copy-paste-shaped requests. Plus, we added repository-level routing so that touching related files is itself a trigger for the Kafka Skill. The developer describes the task however they naturally would, and the agent offers the paved path. Guardrails In developing the Kafka Skills, we hoped to prevent agents from inventing Kafka applications from scratch. The agent handles the ambiguous parts: understanding the developer's request, identifying source schemas and fields, and deciding which validation steps to run. The agent then passes structured inputs into versioned scaffolding scripts and validation tools. Anthropic describes this pattern in its tool-use documentation: the agent returns a structured tool call that application-owned code executes, and strict tool use can constrain those inputs to a JSON schema. The agent handles the orchestration, while our scripts and hooks generate the templated code. Security research reinforced that decision. In a 2023 Stanford study, developers using an assistant wrote less secure code and were more likely to think their code was secure. BaxBench found a similar gap between code that runs and code that’s safe: across 392 backend tasks and 11 models, the best model was functionally correct 62% of the time, but no model produced code that was both correct and secure on more than 37% of the tasks. Researchers could exploit about half of the functionally correct programs. Now that we had agents running commands and committing code on our behalf, we also found that we needed to tighten Kafka ACLs, repository permissions, and deployment controls. We designed the system around least privilege: agents only get the access they need for the task they are performing. This was to make sure that neither human nor agent could delete an important topic, write to an unexpected one, change an unrelated pipeline, or bypass the normal production review path. What’s Next? Kafka supports a wide range of critical data workflows at Etsy, including machine learning feature generation, embeddings, filtering, fan-out, and storing data. We’ve built Skills for seven Kafka workflows so far. Adoption is the near-term focus: we're continuing to add entry points so the Skills are offered no matter how a developer phrases the task. Among our earliest insights from the observability table will be whether the discoverability fixes described above worked, and we’ll continue to iterate as we learn. We built the Kafka Skills to fix a developer-experience problem: too many steps stood between an idea for an ML model and a working pipeline with all the data. Our machine learning models shape the Etsy user experience for buyers and sellers. Each streaming feature that ships sooner helps our models understand a little earlier what shoppers mean and what listings contain. That means more fresh signals behind every search for a light linen dress, and behind the next hand-painted ceramic mug someone favorites. Acknowledgements A huge thanks to Kamya Shethia, Shannon Hoang, Nicollette Lui, Andrey Polyakov, and Virginia Poundstone for all their hard work designing and building Kafka Skills and agent tools, setting up observability, as well as locking down access control to make the world safer for robots and humans alike.
Enterprise Java developers have a new superpower—drive GitHub Copilot from idiomatic Java code with annotations, virtual threads, and more. The post Using the GitHub Copilot SDK for Java appeared first on The GitHub Blog.
In the early fall our infrastructure team was considering Kafka, a highly available message bus. We were looking to solve several infrastructure problems that had come up around that time. We were looking for a reliable way to collect event data and send it to our data warehouse. We were considering a more service-oriented architecture, and needed a standardized way of message passing between the components. We were starting to evaluate containerization of Shopify, and were searching for a way to get logs out of containers. We were intrigued by Kafka due to its highly available design. However, Kafka runs on the JVM, and its primary user, LinkedIn, runs a full JVM stack. Shopify is mainly Ruby on Rails and Go, so we had to figure out how to integrate Kafka into our infrastructure.
Shopify's involvement with TruffleRuby began half a year ago, with the goal of furthering the success of the project and Ruby community. TruffleRuby is an alternative implementation of the Ruby language (where the reference implementation is CRuby, or MRI) developed by Oracle Labs. TruffleRuby has high potential in speed, as it is nine times faster than CRuby on optcarrot, a NES emulator benchmark developed by the Ruby Core Team. I’ll walk you through a simple feature I investigated and implemented. It showcases many important aspects of TruffleRuby and serves as a great introduction to the project! Introduction to Ruby Lazy Initialization Ruby developers tend to use the double pipe equals operator ||= for lazy initialization, likely somewhat like this: Syntactically, the meaning of the double pipe equals operator is that the value is assigned if the value of the variable is currently not set. The common use case of the operator isn’t so much “assign if” and more “assign once”. This idiomatic usage is a subset of the operator’s syntactical meaning so prioritizing that logic in the compiler can improve performance. For TruffleRuby, this would lead to less machine code being emitted as the logic flow is shortened. Analyzing Idiomatic Usage To confirm that this usage is common enough to be worth optimizing for, I ran static profiling on how many times this operator is used as lazy initialization. For a statement to count as a lazy initialization for these profiling purposes, we had it match one of the following requirements: The value being assigned is a constant (uses only literals of int, string, symbol, hash, array or is a constant variable). An example would be a ||= [2 * PI]. The statement with the ||= operator is in a function, an instance or class variable is being assigned, and the name of the instance variable contains the name of the function or vice versa. The function must accept no params. An example would be def get_a; @a ||= func_call. These criteria
Most Java developers have probably run into problems where their code throws a NoSuchMethodError or a NoClassDefFoundError at runtime, despite compiling perfectly well. These issues can be very frustrating and hard to solve. This post tries to explain how they happen and explores some things that can be done to fix them.