Beyond the model: Engineering AI infra with scientific judgement

Airbnb turned customer support investigations that used to take months into workflows that finish in a matter of days — and most of the people running them now have never written a line of code. 👇

Before launching an AI customer service assistant, Airbnb needed to catch rare and risky edge cases hidden inside unstructured support conversations. Evaluating these scenarios required months of hand-crafted work. Data scientists manually queried tables, reviewed sample rows with domain experts, and copy-pasted analytical methods across separate notebooks. When Airbnb needed to repeat these audits for new languages, regions, and products every week, the manual process completely broke down.

To scale their methodology without losing rigor, Airbnb built an agent harness called Insight Miner. It wraps proven data science processes into infrastructure, turning bespoke data exploration into a repeatable system guided by AI agents.

Insight Miner handles the heavy mechanical work under the hood. It runs the underlying extract → embed → cluster pipeline, manages scaled labeling, and tracks data movements so users don't need to touch infrastructure.

On top of that core execution layer, the harness encodes expert data science techniques — like prompt tuning, hard-example mining, and contrastive labeling. A user starts an investigation by talking to an AI agent in a chat session. The agent acts as an executor and methodology expert, applying those built-in techniques to any unstructured text dataset.

To keep the system reliable over time, separate background agents maintain the environment. They monitor real-world usage, update prompt instructions when new base models release, and automatically reproduce and propose fixes for bugs in the harness itself.

Investigations that once took months now take days. Dozens of Airbnb teams now use Insight Miner across hundreds of investigation types, analyzing several orders of magnitude more data. The tool expanded far beyond data science, allowing non-technical operations and product teams to run rigorous, large-scale data investigations on their own.