Rerouting the Stream: How Lyft Moved to the Apache Flink Operator

Lyft was overprovisioning its streaming fleet by a few million dollars per year, largely to avoid breaking jobs during custom deployments. Yet when they tried switching to the open-source Apache Flink operator, standard deploys caused up to 20 minutes of downtime. 👇

By 2024, Lyft's custom Flink operator had become a wall of technical debt. Built in 2020 before an official Kubernetes control plane existed, the custom operator relied on brittle savepoint logic that failed without retries. It forced developers to guess memory limits and left the team unable to use modern features like autoscaling without rebuilding what open source had already solved.

Lyft decided to retire its internal operator and migrate hundreds of production streaming jobs to the open-source Apache Flink Kubernetes Operator. Instead of forcing teams to rewrite their application configs, Lyft built an automated translation layer inside their deploy API to convert legacy configurations on the fly.

The translation layer intercepts legacy custom resources and converts them into native Flink deployment specs:

* Spec translation: The API packs flat fields into nested open-source specs, mapping `jarName` → `jarURI` and converting inline container settings into proper `PodTemplateSpec` objects. * Resilient recovery: Deployments default to `last-state` upgrade mode, restoring jobs directly from high-availability metadata or checkpoints rather than failing on savepoint timeouts. * Backward compatibility: The API injects legacy environment variables like `APP_NAME` and `HOST_NAME` so underlying application code continues to run without modification.

Getting onto the new operator was only half the battle. Default open-source deployments took down old TaskManagers before starting new ones, causing 3 to 20 minutes of downtime per deploy. To fix this, Lyft adopted the `FlinkBlueGreenDeployments` CRD to spin up new clusters alongside old ones before cutting traffic over.

Enabling memory autotuning revealed another surprise. Python-based Apache Beam jobs started crashing with out-of-memory errors. Lyft had historically hidden Python memory inside JVM overhead allocations; the new autotuner saw that unused JVM overhead, reclaimed it, and killed the TaskManager pods. Lyft resolved this by pulling the Python harness out of the Flink container and running it as a dedicated sidecar.

Finally, to enable seamless autoscaling, Lyft upgraded to Flink 1.19 for in-place rescaling without restarts. They introduced AWS Karpenter to handle the shifting CPU and memory ratios created by dynamic tuning.

The migration eliminated a few million dollars in annual overprovisioning costs while drastically reducing upgrade maintenance. Instead of maintaining an isolated state machine, Lyft now contributes bug fixes directly back to the upstream Apache Flink project.