MAPS: Netflix’s Multimodal Asset Personalization at Scale

Combining visual embeddings with model consolidation gave Netflix a 5.69% offline boost on data-starved artwork, even though trying either idea on its own yielded zero lift in online A/B tests. 👇

Netflix recommends custom artwork and video previews for every title, but brand-new assets have no interaction history. Recommendation engines previously treated every asset as an opaque ID, completely blind to what was actually inside an image or video clip. For new launches, Netflix had to fall back on generic popularity rules or force random exploration until members clicked enough times to build a data history.

To solve this cold-start problem, Netflix built MAPS (Multimodal Asset Personalization at Scale). Instead of relying strictly on asset IDs, the system feeds multimodal embeddings directly into ranking models so personalization kicks in on day one.

For still artwork, Netflix encodes each image into a 768-dimensional CLIP vector and concatenates it with the asset's learned ID embedding. Because CLIP captures visual concepts like faces, colors, and framing, a member's preference for a specific actor carries over to a brand-new title instantly.

This visual representation solved a secondary problem: model fragmentation. Netflix previously ran five separate models for five image canvases (like wide billboards and vertical boxes) because ID-based models could not tell that a cropped box shot was the same scene as a wide billboard. Because CLIP vectors stay nearly identical under cropping and resizing, Netflix merged all five models into a single unified system.

To keep high-volume canvases from dominating training, Netflix weighted examples by their long-term reward scores rather than raw impression counts.

Testing revealed a crucial dependency. Adding CLIP to the five separate models failed online because data-starved canvases lacked enough traffic to learn how to use the vectors. Merging the models without CLIP also failed because new assets still lacked history. Only doing both together worked: mature canvases taught the shared model how to interpret CLIP vectors, and those learnings immediately transferred to brand-new assets on sparse canvases.

The same CLIP vectors unlocked query-aware search artwork without extra model training. Netflix simply calculates the cosine similarity between the CLIP text embedding of a search query and the CLIP image embedding of candidate artwork, adding that alignment score directly to the personalization ranker.

For video previews, static images were not enough. Netflix built an in-house model called MediaFM that encodes each video shot across three modalities: * Visual: SeqCLIP embeddings averaged across video frames. * Audio: Pretrained speech and audio representations. * Text: Captions processed through a large text model.

Evaluating new embeddings through full online A/B tests is expensive. To filter candidates quickly, Netflix built a cheap proxy task using a linear probe. The probe attempts to predict an asset's debiased popularity winner using only its content embedding. If an embedding cannot beat random guessing on this test, it gets dropped before touching production pipelines.

The unified CLIP artwork model delivered statistically significant lifts in core discovery metrics and streaming hours, successfully absorbing Netflix's major TV home-screen redesign. MediaFM outperformed visual-only models both offline and online, making it the default video preview encoder across all Netflix platforms.