ML based ranking using Nrtsearch

Shipping candidate features over the network to a dedicated ML scoring service was bottlenecking Yelp's search performance, so they brought the entire model server inside the search engine. 👇

Yelp's search architecture originally retrieved candidate documents from Nrtsearch, fetched features from remote stores, and sent large payloads to a standalone scoring service. But as candidate pools grew and feature sizes expanded, network transfer overhead and serialization bottlenecks made real-time re-ranking unsustainable.

To fix this, Yelp built the Nrtsearch Inference Plugin to run ML models directly inside the search engine layer. Co-locating feature extraction and model inference removed external network calls entirely.

Standard Lucene relies on term-based BM25 scoring, which cannot learn non-linear feature interactions or handle rich user signals. Moving to models like XGBoost and neural networks usually requires external service calls, but Yelp bypassed this by embedding a TensorFlow-based model server directly inside the search nodes' Java Virtual Machine (JVM).

When an Nrtsearch replica node boots up, it reads its cluster configuration and loads MLeap-packaged model bundles directly from MLflow. While primary nodes handle document indexing, replica nodes process search requests and host this embedded inference engine.

When a query arrives, the search process flows in sequence: * Filter candidates: The replica node runs initial query filters to isolate eligible document IDs. * Extract features: The plugin pulls features directly from the indexed documents or incoming query parameters, skipping external feature stores. * Run inference: An internal ML Scorer passes these features to the in-JVM model server to score each candidate document. * Rescore results: For multi-model requests, the plugin aggregates individual model outputs into a final score before building the response object.

Engineering teams can use an out-of-the-box generic scorer configured with Lucene query expressions, or write custom Java modules when fine-grained scoring logic is required.

By performing inference directly on search replicas, Yelp eliminated network serialization overhead across Search, Ads, and Home Feed. Queries now evaluate complex ML models directly against index data in milliseconds, while simplifying how teams deploy and test new models.