Introduction

Picture this — you’re home after a long day at work and can’t wait to catch up on Bridgerton. You log on to Netflix, search for “Bridgerton” and land on this screen:

If this sounds familiar, you already know what a synopsis is — it’s a short glimpse into what a viewer can expect to see in a show. We see two types of synopses here: a series-level synopsis describing the overarching plot, and multiple episode-level synopses describing each episode.

A synopsis plays a very important role for us — it helps viewers decide whether a title is worth watching or not. Every synopsis needs to strike a balance: sharing just enough to give viewers a sense of what to expect, while leaving enough intrigue to draw them in. But of course, we can’t just write anything.

“In the fog-drenched streets of Victorian London, Detective Daphne Bridgerton races to unmask a serial killer known only as the elusive Lady Whistledown.”

Intriguing? Definitely. But it is most certainly not the premise of Bridgerton. Which leads us to the first (and most important) rule of writing a good synopsis: it must be factual. At Netflix’s scale, with thousands of titles, multiple episodes per season and multiple languages, fact-checking all of these by hand simply doesn’t scale. This is where our factuality service comes in: an automated way to judge whether a synopsis is factual or not.

Our current factuality checker feeds a reasoning model everything we know about a title (like metadata and text assets) and asks it to judge synopsis factuality according to a rigorous rubric. There are two main limitations of this approach that we want to optimize:

1. High cost of long-context reasoning calls: Each decision requires expensive, long-context model calls to reasoning models, which doesn’t scale well when we want to check multiple titles, episodes, languages, and style variants.
2. Misaligned rationales: The model’s explanations often diverge from our annotators’ reasoning, which makes its factuality verdicts less reliable. We want these rationales to be better aligned so that (1) the model can consistently flag nuanced errors, and (2) its explanations can be used downstream to automatically correct issues in the synopsis.

Our new approach tackles both problems at once. On the cost side, it delivers a 52.4% reduction per synopsis. On the quality side, it significantly improves both verdict accuracy and reasoning alignment: Recall–Fail (“Of all truly non-factual synopses, how many does the system correctly reject?”) rises from 0.77 to 0.86, and the new factuality service’s explanations now show ~82% overlap with human rationales (up from ~69%). In other words, it’s not just cheaper and more accurate — it’s also much better aligned with how our expert annotators actually judge synopses.

So, what exactly does factuality mean?

Let’s go back to the Bridgerton example. Say you see this synopsis about Bridgerton Season 1:

Synopsis A: “Bridgerton follows Daphne Bridgerton, the eldest daughter of the powerful Bridgerton family as she makes her debut onto Regency London’s competitive marriage market.”

It’s easy to judge that this synopsis is factually correct — no wrong facts, and it gives the reader a clear, accurate glimpse of what to expect from the show.

Let’s take another example:

Synopsis B: “Set in modern-day London, this gritty crime drama follows Detective Daphne Bridgerton as she hunts a serial killer while hiding her own criminal past.”

Another easy one — the plot is so blatantly off the mark that it is definitely not factual.

But what about this?

Synopsis C: “Bridgerton follows the mysterious, sharp-tongued Lady Whistledown as she exposes the secrets of London’s high society through her scandalous gossip sheet.”

Technically, every statement here is true — Lady Whistledown does indeed publish a gossip sheet, and if there’s one thing she is known for, it’s her scathing exposés. So then, is this a factual synopsis?

Not quite! While technically true, it paints a misleading picture of the show. A new viewer might reasonably expect it to revolve around Lady Whistledown, rather than Daphne navigating the marriage market. This, too, is a type of non-factuality we want to catch.

Lastly, here’s the trickiest of all:

Synopsis D: “When Daphne Bridgerton makes her society debut, she catches the eye of the aloof Duke of Hastings, and the two begin a passionate love affair that sets the whole ton abuzz.”

Again, every statement is true, but the phrasing is misleading. A new viewer might assume they fall in love immediately, missing the fact that their relationship begins as a fake, mutually beneficial arrangement. This is another type of nuanced error our system should flag.

So now that we have an idea of the kinds of errors we care about, the real question is: how do we build a system that is able to catch both glaring as well as subtle factuality violations?

Current Service and Its Limitations

Our current factuality pipeline is a straightforward, holistic LLM call. We feed a reasoning model everything we know about a title (text assets, metadata, and a detailed factuality rubric) and ask it to judge the factuality of a synopsis. We repeat the call several times to reduce output randomness, and if any run flags the synopsis as non-factual, it fails. This approach has two main limitations:

1. High cost: Each judgment requires multiple (five, to be exact) calls to a large reasoning model over extremely long contexts (100k+ tokens), which does not scale well.
2. Misaligned reasoning: The model can fail synopses for nitpicky reasons (e.g., “This is a show about wonderful magic” gets failed because the characters never literally call the magic “wonderful”) while overlooking nuanced errors by convincing itself they “don’t really make the synopsis false.” This misalignment makes verdicts less reliable and rationales less useful for downstream correction.

Optimizing the Holistic Pipeline

To understand exactly where we should optimize the current holistic pipeline, we study the current pipeline both qualitatively and quantitatively on a custom factuality benchmark that we built. This benchmark comprises synopses hand-labeled by human annotators as factual or non-factual, along with their rationales. This analysis reveals several key insights:

Insight 1: Do we really need five holistic calls?

Let’s think back to why we added multiple holistic calls in the first place: to reduce sampling noise. In practice, though, we see an interesting dichotomy:

• Blatantly true / false synopses (Bridgerton A, B): All runs agree on the verdict with near-identical reasoning.
• Nuanced errors (Bridgerton C, D): The model is much more likely to disagree with itself — some runs catch the error, some don’t. This is exactly where additional sampling helps.

This suggests a simple optimization: A and B don’t really need all 5 calls. If we can somehow identify these “easy” synopses upfront, we can cut down on the number of calls for them, while still giving harder cases like C and D the extra calls they need.

The easiest way to do this is to ask the model to output its confidence alongside its verdict and rationale. We can then separate “easy” (high confidence) synopses from “hard” (low confidence) ones and only send the hard cases through additional checks. Think of it like airport security: most bags go straight through, but those that trigger the alarm (low confidence) get extra screening.

Implementing this easy fix makes our pipeline go from the current holistic model to a new “confidence gated” holistic model:

Insight 2: Five identical judges calls give us no new information

With confidence gating, our pipeline only escalates harder synopses to the full five holistic calls. But extra calls only help with sampling noise, not systematic reasoning errors: if every run follows the same flawed rationale, they’ll all reach the same wrong verdict, rendering the extra calls useless. How can we ensure that each call counts?

Idea: Diversify the judges and make them specialized

Instead of running five holistic judges with the same generic factuality prompt, we introduce two specialized judges that individually target the two aspects of factuality we care about:

• Literal Judge: Is the synopsis literally true?
• Implicature Judge: Does this synopsis give a false picture to a new reader who has not watched this show?

Explicitly searching for specific types of errors helps us better flag both literal and subtler implicature errors. In practice, we observe that this works better than repeatedly querying a single holistic judge, hoping that sampling stochasticity will surface nuanced errors.

Insight 3: The rationale reveals mistakes the verdict hid

At this point, we have a pretty good pipeline, but we’re still underusing one of the richest signals available to us: the model’s rationale behind its verdict. For nuanced errors, we often see the model identify a point of contention, then brush it aside based on a faulty assumption or because it deems the issue too minor to affect the verdict. Can we use these shaky spots in the rationale to catch (and correct) errors the verdict misses?

Idea: Secondary atomic-claim checking via a router:

Since our literal and implicature judges are already strong reasoning models, we ask them to additionally critically analyse their own reasoning and surface shaky spots — any doubts, concessions, or assumptions that might need rechecking. These include cases where the model:

• Makes an assumption.
• Raises a doubt that the available evidence can’t resolve (e.g., an external critic quote).
• Notices a contradiction but dismisses it as insufficient to change the verdict.

We pass all concessions from both judges to a router model, which:

1. Reframes them as self-contained, verifiable atomic claims, each with a materiality field specifying the evidence needed to support or refute it.
2. Routes each claim to the appropriate secondary Stage-2 judge: a web judge for external facts (e.g., citations, media quotes, or prior work), or a source-material recheck focused on the reframed atomic claim. By pinpointing the exact point of contention, the recheck is more targeted, making it easier to mark the claim as true or false.

Putting It All Together: Proposed Cascade Factuality Service

Our findings lead to three key improvements: specialized judges, confidence-gated escalation, and targeted rechecking of shaky reasoning using a secondary stage when escalated.

We implement these improvements as a tiered system shown in the figure below: every synopsis first passes through both the literal and the implicature judges. High-confidence verdicts are retained, while low-confidence cases escalate to the router and Stage-2 judges for further checking. Stage 2 can overturn the initial verdict in two cases:

• FAIL → PASS: All concessions, including the original failure reasons, are supported by the secondary check.
• PASS → FAIL: At least one concession is contradicted.

Results

Based on our discussions from above, we evaluate three different pipelines:

Evaluation: Datasets and Metrics

We evaluate these systems on two datasets: show-level and episode-level synopses. Each datapoint includes a synopsis, a human gold label (factual / not factual), and the annotator’s rationale. For episode-level synopses, we additionally require that the synopsis be specific to that particular episode, not just a generic synopsis of the overall show.

We focus on two main axes: Quality and Cost. The quality metrics are listed below:

Recall-FAIL is our primary metric: higher recall means fewer Gold FAILs are missed. We also track Precision-PASS, which measures how reliably PASS verdicts identify Gold PASSes, and alignment with human rationales.

On the cost side, we want to reduce both the average number of LLM calls per synopsis and the end-to-end cost of producing a final verdict. Latency is not our primary optimization target, but we do want to ensure the median (P50) latency does not blow up to impractical judgement times.

Evaluation Results

The best numbers per metric per dataset are shown in bold.

Note: We compared the cascade against two holistic ablations — increased reasoning effort and a smaller, cheaper model — to test if simple ablations could achieve better performance. For episode-level synopses, we also added a full-show summary to the prompt to provide context from previous episodes; we observe summaries substantially improve performance when compared to non-summary context variants, so we report only summary-based variants.

Our takeaways from this experiment are:

• Cascade improves quality. Our primary metric, Recall-FAIL improves on both show and episode level synopses, with a limited impact on Recall-PASS. In other words, we now catch more Gold FAIL synopses without sacrificing the reliability of our PASS verdicts.
• Cascade is cheaper. Cascade cuts total cost by 52.4% on the show-level dataset. Cascade (with summary) is also 15% cheaper than the current (no summary) holistic pipeline on episodic dataset.
• Cascade’s P50 latency increases with respect to the holistic baseline. Even so, this remains within what we consider a reasonable range for factuality judgment.
• Simple tweaks aren’t enough. Increasing reasoning effort makes the factuality checker too lenient, while using a smaller model makes it too strict — neither addresses the underlying problems of the holistic baseline.
• Summary context matters for episodic tasks. Without a full-show summary, the model tends to over-fail because it lacks the context to determine whether a detail from the episode summary fits the broader storyline.

Alignment with Human Rationale

So far, we’ve established that the cascade produces better verdicts: it catches more non-factual synopses and produces more reliable PASS decisions. But are these verdicts correct for the right reasons?

To measure this, we use an LLM-as-a-Judge to compare each pipeline’s reasoning against the human rationale on Gold-FAIL (GF) synopses. The judge assigns a Reasoning Similarity Score (RSS) of 0 (dissimilar), 0.5 (partially similar), or 1 (very similar) based on rationale overlap.

We observe that cascade is significantly better aligned with human rationale in all cases. This is a big gain — not only is the cascade giving us more accurate factuality verdicts, but it also identifies the same points of contention in the synopsis as a specialized human annotator.

Conclusion + Future Work

Overall, our proposed cascade pipeline improves both cost and quality of the existing factuality service:

• On the cost side, confidence gating lets us avoid unnecessary calls, reducing the average cost of factuality evaluation.
• On the quality side, two diverse judges allow us to collect complementary informative verdicts, while the Stage-2 judges recheck and correct the assumptions behind uncertain decisions. This results in higher verdict accuracy and substantially better alignment between model and expert annotator reasoning.

The cascade gives us a path toward a factuality system that is not just more accurate, but also more interpretable and self-correcting. Some future directions to consider would be:

• Adding multimodal inputs to help verify claims that text assets alone cannot confirm.
• Use the rationale from factuality checking to correct the faulty synopsis.

Fact-Checking at Scale: Building a More Accurate and Efficient Synopsis Factuality System was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Netflix, data and machine learning practitioners run millions of workloads each day across orchestration systems and compute engines such as Titus, Spark, Trino, and Druid. On peak days, roughly 100,000 workloads fail. When workloads fail, the failure itself is only the beginning. Someone still needs to find the useful signal in several logs, decide whether the cause sits in their code or the platform, and determine whether to retry, reconfigure, or escalate. Even a small share of difficult failures can consume engineering time, waste compute, and interrupt critical data pipelines.

Our first answer to that problem was Pensive, an auto-diagnosis and remediation system in Netflix’s data platform. Pensive collected logs from failed workloads, used regular expressions and hand-authored Java rules to extract and classify errors, then passed retry or remediation guidance to products such as Maestro. It made ~ 50% of failures self-service and, in some cases, automatic.

Rules work well for known patterns, but their coverage grows only when someone recognizes a new failure mode, understands the logs, writes a rule, and keeps it current as engines and log formats change. Over one three-month period, Pensive processed roughly seven million job-failure classifications, yet about half fell into the unknown bucket. We needed a way to expand coverage without giving up the deterministic parts of Pensive that already worked.

To solve this ambiguous and complex problem at Netflix scale, we invested in Pensive AI to add AI-assisted error extractions and focused diagnosis and remediation guidance to our data and ML practitioners. The goal was not simply to place a model in front of a large collection of logs. We needed bounded extraction, predictable evaluation inputs, responsive product behavior, and quality checks that could follow the system from local development through production. Those requirements shaped every layer of the design, which we will discuss in this blog. Pensive AI wraps AI around deterministic processes, which gives broader coverage, low latency, controlled AI costs, and easier maintenance across the software development lifecycle. Building on this foundation, we plan to explore more agentic debugging experiences, as Part 6 describes.

Early results are encouraging. AI error extraction now covers 100% of failures, including the failures that legacy Pensive classified as unknown. Failed step attempts where no one used Pensive AI averaged 321% more manual debug-log exploration actions than attempts where someone did (an observational comparison, not a causal claim).

The two paths complement each other. Error extraction improves an existing capability and is always on. It lets the user of a failed Maestro workload quickly check the parts of the logs that may point to an immediate root cause. It has low latency and low cost, and it can help in a large share of cases. On-demand debugging enables a new capability. It looks at symptoms across the logs and adds the LLM’s own knowledge and reasoning for a deeper analysis, at slightly higher latency and cost. Extraction gives a fast first look at every failure, and on-demand debugging triggered by the user goes deeper when that first look is not enough.

Part 1: Finding the useful errors

Legacy Pensive rules are precise, but each one captures a particular log shape. A simplified rule looks like this:

- name: SparkContainerOOM
logSource: GENIE_STDERR
restartable: false
regex: "Container.*is running beyond physical memory limits.*Killing container"
summary: Spark AM container runs out of memory

With Pensive AI, we wanted to find useful errors that had no corresponding rule and adapt more easily when familiar errors changed shape after a platform upgrade. Data Portal (our UI for data pipelines) surfaces these extracted errors when available and links back to the source logs, giving practitioners a focused starting point for triage. That raises a question: how can we ask a model to identify an error without asking it to repeat a long stack trace token by token?

Producing output tokens is much slower than reading input tokens. A direct transcription approach would make long errors expensive and slow. Instead, Pensive numbers the lines of each log and gives the model a sequence of {lineNumber, text} records. The model identifies the first and last lines of the error span, and Pensive reconstructs the selected text from its own log copy, leading to lower cost and better latency. This way, the model always returns two integers (start and end line number), whether the selected logs span a few lines or hundreds of lines. The displayed content comes directly from the original log, reducing hallucinations. The model can still choose the wrong range, which is why extraction quality remains an integral measurement for our AI evals, as we will discuss later.

For orchestrated workflows, “Pensive AI Error Extraction” runs automatically whenever a step fails. A step attempt is one execution of a step within a Maestro workflow. Maestro already calls Pensive for a regex-derived error and restartability classification, so AI extraction runs in the same request and returns its results alongside the established fields. The overall Pensive response must return within Maestro’s hard 90-second post-processing timeout for a failed step attempt.

That deadline shaped the extraction path. Pensive extracts errors from the logs in parallel to reduce latency, and the per-log calls share one deadline set well inside Maestro’s timeout limit. A per-log extraction call that misses the deadline is excluded from the result, allowing the overall request to return on time.

The data contract between Pensive AI and the various consumers needed the same care. The legacy data model contained one flattened error string; AI extraction instead returns a list of errors, one for each log source, with deep links back to the originating logs. Changing the old field in place would have broken existing clients. We introduced the new data model beside the legacy non-AI error extraction data model and continued producing both. Data Portal, our user interface for monitoring and observing data pipelines, preferred AI errors when available and fell back to the legacy value when they were not during the rollout. That allowed a gradual rollout and ensured that historical failures still display an error, since we chose not to backfill the old data due to log retention issues.

The migration is still in progress today. The Data Portal UI surface used for failed Maestro steps reads the new AI extracted error field today, while the query editor (for editing and running SQL), Workbench notebooks (managed compute platform and UI to edit and run jupyter based notebooks) and some slack clients calling Pensive still use the legacy error extraction path. Each client needs a compatible schema change before it can consume the new data. Keeping both fields populated lets teams adopt the new behavior at their own pace without forcing a coordinated cutover as we continue evolving Pensive AI.

Part 2: Full failure diagnosis and the product experience

Beyond error extraction, we built Pensive AI Debugging, a tool that analyzes failed step attempts and provides root-cause analysis and suggested remediations. It is available through programmatic APIs, MCP, and the product UI (Data Portal), all backed by a common entry point across the Netflix Data Platform. This shared model supports consistent behavior and custom product experiences.

This diagnosis still has to arrive quickly and remain consistent across investigations. A debugging response can take tens of seconds or a minute to generate, compared with a few seconds for error extraction. We run full AI-based debugging on demand instead of automatically, which keeps costs under control because most failed steps are never opened. We treat the failed step attempt, not the person who triggered it, as the unit of work, so everyone investigating the same failed attempt sees the same diagnosis.

That choice gives us a stable unit of work for caching. We cache each debugging response by step attempt. A second request becomes a key-value read instead of another AI inference call. The cache also prevents duplicate work, reducing latency and lowering costs further. This also means that a debugging generated in the Data Portal is cached and immediately retrieved for the MCP path instead of incurring another AI inference call. A conditional write creates an in-progress record, one caller starts generation, and other callers wait for the shared result to avoid conflicts when concurrent requests come in for the same step attempt failure. If generation remains in progress past an abandonment timeout, Pensive treats the record as abandoned and reclaims it. Completed responses expire after 90 days to help control costs and optimize storage because debugging requests for much older workloads are uncommon.

Pensive AI supports multiple ways to initiate debugging responses. A caller can (1) read a cached response, (2) generate one if no cache entry exists, or (3) regenerate and overwrite an existing response. Those semantics support the product UI, programmatic access, and the evaluation harness while keeping the response associated with a single immutable workload failure.

For users, the most important behavior is that a long-running answer feels alive. Pensive streams the response as it is produced. The service emits server-sent events, and the GraphQL layer republishes them as a WebSocket subscription for the Data Portal client.

Each event carries the complete response generated so far. That duplicates a small amount of text, but it keeps the client simple: it does not need a replay buffer or token reassembly logic, and the next event corrects the display after a dropped or late event. Generation is independent of the connection that requested it. A disconnect does not cancel the work, and the final answer still reaches the cache. A hard reload can show a loading state and poll for the completed result. Within the single-page application backing the Data Portal UI, active streams are maintained independently of view and component lifecycle, so navigation does not interrupt a response in progress.

Part 3: Measuring and improving the behavior we want

LLM-assisted systems need more than a demo or a few working examples. Before shipping in production, we need confidence in both error extraction and debugging guidance, and we need to know that we have not regressed on failures that legacy Pensive already handled. We built separate evaluation frameworks focused on the two distinct jobs.

Building labeled data is often the expensive part of evaluation. For error extraction, legacy Pensive’s hand-written rules provide reference labels for failures they recognize. They also serve as excellent ground truth in most cases. We run those rules over production logs and compare their extracted spans with Pensive AI’s output. This gives us a reliable regression set without relabeling every example by hand, though it is not an absolute ground truth (since our human-built rules could also have been stale). For now, we keep these rules frozen as a label source. This lets us continue to build comparison evals and run them on top of the existing system.

Stable inputs are essential for repeatable evaluation experiments where we A/B test new models/configurations or context injections to improve response quality. The logs Pensive AI uses can expire after as few as seven days or as many as 30 days. Production Pensive AI requests archive fetched logs to S3 object storage asynchronously during workload execution time so the upload is not on the user-facing critical path. Uploads do not overwrite an existing object, preserving the exact snapshot used to create an evaluation case. Evals use those archived logs, so score changes remain attributable to the system under test. The archive also prevents parallel experiments from overloading the source systems and lets us run them without affecting production. Cached S3 log inputs further reduce costs when running evaluations.

Concretely, we measure four complementary signals for error extraction:

• Inclusion: An LLM judge compares the legacy Pensive error for a failure with the errors Pensive AI extracted and determines whether the known error is still represented, even when wording differs.
• Precision: A second judge sees only the extracted AI errors and evaluates whether each is meaningful and actionable on its own.
• Repeated noise: A script looks across the whole dataset for lines that recur too often. A generic status line may look harmless in one answer but become distracting if it appears in a large share of failures. Lines appearing in more than about 30% of workflows are candidates for explicit exclusion. In practice, the exclusion list has only three lines, each of which otherwise appeared in nearly every extracted error.
• Latency: We record the latency of every extraction call. Quality scores alone cannot reveal when an apparent improvement requires too much additional work, which often correlates with increased AI costs.

This combination gives us an actionable feedback loop. Prompt changes can reduce general noise, while the dataset-level view exposes the stubborn boilerplate that prompt changes do not eliminate. Existing rules act as a regression floor, and archived logs give us a repeatable way to compare changes. At Pensive AI’s scale, model and system choices must satisfy quality, latency, maintainability and cost constraints together.

To seed the debugging ground-truth dataset for evaluation, we mine internal engineering-support Slack threads and use an LLM to identify threads that link to an orchestrated workflow, describe a resolved problem, and match the class of issue Pensive AI handles. A second model synthesizes a reference debugging response from the relevant discussion. Paired with logs from that workflow failure, it becomes a concrete evaluation row in our evals dataset. This approach works only retrospectively, because Pensive AI in production aims to provide debugging guidance before such a Slack discussion happens.

We also made the debugging generation process visible. Tracing follows the request through Pensive, with child spans for each log fetch and extraction tool call. Our tracing backend based on Braintrust brings the prompt, completion, tool calls, and latency into a single view. During early product development, that visibility helped us inspect low-performing debugging responses in context instead of guessing which input or operation caused them. Tracing helped us improve the responses and fill context gaps even before we had comprehensive evals. In production, the API and UI return the trace identifier with the response, so feedback can lead directly to the trace for in-depth analysis.

Part 4: Keeping quality in the release path

Evaluation is not only for experiments with LLM orchestration, models, prompts, or context changes. Changes to log selection, trimming logic, or a source system can change what the model sees and therefore change extraction quality. We use the same evaluation harness and evaluators throughout local development and Pensive AI deployment so those effects are visible before users encounter them.

For each pull request, the code branch is deployed to an isolated preview environment and a CI job runs an experiment with our evals against that isolated environment. Precision and inclusion thresholds gate the merge. The job also runs after a build is promoted, catching interactions that were not present when individual pull requests were evaluated in isolation. If the promoted build misses its thresholds, it alerts the on-call engineer to investigate or roll back.

The evaluator runs on a schedule against production to surface gradual drift or external changes. Developers can also trigger evaluation experiments remotely while iterating, using the same deployed-environment path as CI. Consistent infrastructure makes the results comparable across local iteration, preview builds, promoted builds, and production.

Part 5: Looking for feedback signal from users

The system is only valuable if it reduces the amount of manual debugging and log chasing done by our users. We wanted a measurement that was available from product telemetry and close to the behavior Pensive AI was designed to change. Manual debug-log exploration became one of the main signals. The Pensive interface already offers links to logs for a failed step attempt. To estimate whether the feature reduced raw-log hunting, we compared failed step attempts where someone interacted with Pensive AI with those where no one did. The second group averaged 321% more manual debug-log exploration actions. This observational comparison does not establish causality or a specific reduction in mean time to resolution, but it suggests that focused diagnosis often allows practitioners to inspect fewer raw logs. Programmatic MCP access also lets teams build operational flows with their terminals or custom agentic experiences around the same diagnosis service.

Another more explicit signal we collect is thumbs up and thumbs down on the debugging response. These feedback signals land directly alongside the internal traces so we can quickly associate feedback to the complete log of inputs/outputs and the path the AI took to arrive at the answer. More importantly, we created an internal business-hours alerting system where each thumbs down response sends us a Slack alert. This was valuable to resolve a lot of rough edges and context gaps initially in the product and is helping us improve the product continuously even today.

Part 6: Future Work building on these foundations

Pensive AI is the foundation for more work that goes beyond diagnosis and recommended remediations. We are exploring “Pensive AI Deepthink,” an opt-in managed agentic experience with higher cost and latency. It builds on the precise, low-latency analysis produced by Pensive AI Debugging and gathers additional context from internal documentation, manuals, past Slack discussions, source code, read-only warehouse queries, and Spark execution details. A routing step directs the investigation to the appropriate skill, such as Spark failures, data-audit failures, or late workflows. This keeps the shared starting point stable and deterministic while allowing the investigation to become more specialized, further improving recall. We are also in talks with the Spark team to leverage Pensive AI responses for failure debugging analysis and A/B comparison for upcoming platform migrations.

We also plan to invest in “Pensive AI Error Classification”. Hard-to-maintain legacy rules also classify failures by restartability and by whether they are platform or user caused. Maestro acts on those labels and changes its retry behavior accordingly like failing fast or tuning the default number of retries (platform errors are tried more than user errors by default). Error classifications must meet a higher bar than error extractions and debugging alone. Moving the majority of the error classifications to AI would let us retire most of the remaining rule-maintenance burden and avoid retrying unknown failures that were unlikely to succeed. For restartability, the costliest error is classifying a recoverable failure as non-restartable, because Maestro will turn a step that could have been recovered into a hard failure, leading to end user alerts and manual remediations. The evaluation framework will need to be tuned around that outcome before AI-enabled classifications can take on those decisions. Equally important, we must avoid retrying non-transient failures that will not succeed, because retries waste compute and delay data pipelines.

Our work on Pensive AI in Data Platform addresses concrete user pain points in debugging, improves maintainability for platform owners, and improves precision and recall for error extraction. Pensive AI extends coverage to failures that legacy Pensive previously classified as unknown. Reliability comes from the system around the AI model calls: bounded error extractions, compatible client data contracts, responsive delivery, reproducible evaluations, caching and tracing, and release controls that keep quality and cost visible as the platform changes.

Part 7: Acknowledgements

This work would not have been possible without our stunning colleagues across Data Platform and our Data Engineering and Analytics partners who provided early feedback. While we cannot name everyone individually, we are deeply grateful for their contributions, which shaped both response quality and the product experience. Special thanks to Adith Swaminathan for advising on evaluations and inference improvements and acting as a thought partner. Thank you to the Orchestration team members, especially Deepak Ramalingam and Malcolm Ramirez for their additional engineering contributions.

For dogfooding and integrating Pensive AI as part of user support, we also thank the engineering support organization (ESO Team). We also thank Abhishek Kapatkar (Data Developer Experiences Team) and Emmie Dong (Data Movement Team) for their feedback to improve the Pensive AI integrations with the rest of the ecosystem. Special thanks to the AI Platform team (Kyle Ahn, Rob Mann) and Braintrust Team for helping with distributed tracing, evals infrastructure and model gateway runtime support.

We also thank Kamran Masood, Ely Spears, Ronald Leung, Anjali Norwood, Prashanth Ramdas, Charles Smith, Harikrishna Menon and other leaders for their constructive feedback and support on the Pensive AI project.

Making Failure Diagnosis Reliable with Pensive AI was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

At Netflix, a crucial part of our mission is ensuring our content looks stunning on any device. Vertical formats are increasingly important for Netflix, as we have 1.2B followers across our social media accounts, and have recently rolled out Clips, our vertical video feed in the Netflix mobile app. Traditionally, creating vertical assets meant editors and producers manually reframed horizontal videos in editing tools, a process that doesn’t scale as our catalog and use cases grow. To support both our in‑app feed and external marketing campaigns at scale, we built an automated multi‑ratio video cropping system, tightly integrated into our internal production tools, that can generate high‑quality vertical and other aspect‑ratio variants with minimal manual effort.

Our Video Resizing Approach

Converting cinematic 16:9 footage into mobile‑friendly aspect ratios takes more than a simple center crop. Filmmaking often involves dynamic character blocking and complex camera movement. We need an intelligent “virtual camera” that can follow the key action, respond to how each scene is composed, and produce crops that feel as intentional as the original shot.

We address this challenge by framing video resizing as a tracking and temporal optimization problem. Our pipeline processes each asset in three main stages, designed to produce 9:16, 4:5, and 1:1 crops in a single pass, without incurring additional computational overhead.

Shot Detection and Splitting

First, we segment the video into atomic shots using a neural‑network–based shot boundary detector. In many cases, shots contain multiple characters conversing on opposite sides of the frame. To better handle these scenarios, we use an Active Speaker Detection based approach to logically split a single continuous shot into multiple sub‑shots. This allows our virtual camera to cut and re-focus on different speakers as the conversation flows. Our Active Speaker Detection leverages an internal face detector and LipFlap to reliably identify active speakers in both live‑action and animated titles.

Focus Region Discovery

Once shots and sub‑shots are defined, we identify the most important region in each frame. We combine face detections with visual saliency maps to filter and prioritize the faces that should be in focus. In scenes with multiple characters, we further rank faces using signals such as active speaker status and relative size. Our model aggregates these cues with shot‑level content understanding to establish a continuous “focus region” for the virtual camera.

Temporal Reframing

Once we’ve established the focus region, the next challenge is deciding how the crop should move over time. Tracking the focus region on a strict frame-by-frame basis would produce a shaky, distracting viewing experience, so we apply a set of heuristics that consider how the focus region moves over the course of a shot. When its displacement is small, we lock a static crop window that consistently covers the relevant area. When there is significant motion, we compute a dynamic tracking window and smooth its path so that any panning feels intentional and cinematic.

Tooling and Workflow Integration

We integrated the resizing algorithm directly into our internal Clips tool, so editors and producers can access it as part of their existing workflow. The interface enables them to automatically generate multiple aspect‑ratio versions of a clip with smart reframing. A unified in‑app module provides intuitive shot‑reframing controls, allowing users to review the algorithm’s crops, and quickly make targeted adjustments where needed.

For more complex shots or custom campaigns, producers can export the resizing parameters as XML and load them directly into external timeline editors such as Adobe Premiere. This allows agencies and external partners to start from the algorithm’s framing decisions instead of recreating crops manually or relying on tools like Premiere’s Auto Reframe. In our internal test, our resizing algorithm outperforms Auto Reframe on a curated evaluation set, so the XML export effectively brings those higher‑quality crops into external workflows. By bringing automated resizing and manual refinement into a single place, this pipeline significantly reduces production time and manual effort while preserving creative control.

Conclusion

Our work on multi-ratio video resizing is one step in a broader effort to make content creation more scalable, without compromising creative intent. By pairing algorithm-driven reframing with editor-friendly tools, we’ve reduced the time teams spend on repetitive resizing tasks and unlocked faster experimentation across mobile feeds and marketing campaigns. As we continue to broaden these capabilities to new formats and workflows, our goal remains the same: to give creatives more time to tell great stories, and to deliver a consistently high-quality viewing experience for our members and fans.

Multi‑Ratio Video Resizing for Mobile Feed and Marketing was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Netflix, shared Java libraries support applications developed across a large, independently evolving repository ecosystem. A library change begins in one repository, but its impact can reach many others. Producers can see the API diff in front of them. What they often cannot see is who still depends on the symbols that changed.

The need for that visibility grows as coding agents take on more of the software development lifecycle. An agent can accelerate a local edit, but without cross-repository context it cannot judge downstream risk or help plan consumer migrations. The evidence a producer uses to assess a change can also guide the agent working on it.

Consider a common case: removing a deprecated API. Deprecated entry points expand the surface producers must test, document, and support, and they can constrain how a library evolves.

Suppose a replacement has been available for several releases and the old method cannot express options required by new behavior. The library’s tests pass, and a compatibility checker flags the removal. The producer still cannot answer the questions that determine what to do next:

• Does any consumer still call this method?
• Are consumers concentrated on one version or spread across several generations?
• Does another library implement or wrap the API, multiplying its impact?
• Is direct coordination sufficient, or does this require a deprecation and migration campaign?

At small scale, producers can ask nearby teams or search a handful of repositories. Across many repositories, it is harder to find every caller of a method and the library version each one uses. Those answers are scattered across consumer codebases and builds.

Without that evidence, a producer may release a breaking change and discover the impact through consumer failures, or preserve an old API indefinitely because the risk is unknown. Libraries need to evolve, and their producers need evidence that helps them do so intentionally.

We built Quasar to index symbol definitions and references across Netflix repositories. For library producers, that shared index helps answer two questions:

1. What changed in the library’s public API?
2. Which repositories use the exact symbols that changed?

The first identifies a potentially incompatible change. The second estimates its observed blast radius. Together, they turn a generic compatibility warning into evidence a producer can act on before merging or releasing.

From API Changes to Consumer Impact

Existing techniques each see a different part of the problem. Binary and source compatibility tools identify contract changes, but cannot tell who uses the changed API. Dependency graphs show declared reach, but cannot prove that code calls a particular method. Text search may miss overloads, inheritance, or the artifact version behind a call. Consumer builds provide strong validation, but running them across the fleet is expensive.

Quasar builds on SCIP, an open interchange format for semantic code intelligence. Language-specific indexers encode definitions, references, and relationships using stable symbol identifiers. Quasar retains API definitions for library versions and commits alongside references from consumer repositories. That shared identity lets it connect an API change in one repository to exact usages elsewhere, even when those repositories were indexed independently.

On each enrolled CI build, language-specific indexers submit SCIP indexes to Quasar’s ingestion service. It writes API definitions and consumer references as Parquet in Amazon S3, retaining the symbol identity, dependency version, and source location needed to investigate a change.

During pull-request review, Quasar compares indexed base and head definitions and counts direct consumer references with Amazon Athena. Quasar exposes those findings through its REST and GraphQL APIs and MCP server; follow-up usage queries provide dependency-version breakdowns and source locations. A separate compatibility-impact assessment can also follow implementations and their callers when a producer supplies a changed JVM symbol.

Consider this simplified API change:

public interface Client {
- Result getLegacyValue(Request request);
+ Result getValue(Request request, Options options);
}

For a detailed compatibility-impact assessment of this removal, a producer-side tool supplies the old method descriptor. Quasar constructs its canonical symbol identity. This abridged excerpt from the direct-reference portion of Quasar’s Amazon Athena query shows the essential match:

SELECT c.change_id, 'DIRECT_REFERENCE' AS evidence_kind,
       r.scm_type, r.org, r.repo, r.dep_version, r.file_path
FROM requested_changes c
JOIN quasar.symbol_refs r
  ON r.symbol_canonical = c.target_canonical

requested_changes contains the API changes submitted by the producer. The SQL excerpt finds repositories that call the removed method. Quasar also checks implementations and calls through them, so it can count each affected repository once. It returns sample dependency versions and file locations to help investigate the result.

For the Client change, this tells the producer which repositories still use getLegacyValue, which versions they use, and where some of those calls occur. That helps the producer decide whether to coordinate a migration before removing the method.

Bringing the Analysis Before Release

Version-to-version comparison is valuable when planning an upgrade or writing a migration guide. Producers benefit even more when feedback arrives during code review.

For pull-request analysis, Quasar stores API definition snapshots keyed by commit. Given a base and head commit, it classifies symbols as added, removed, or signature-modified. Removals and modifications then become candidates for consumer-impact analysis.

An incompatible change is not automatically a high-risk change. Quasar reports the observed impact so the producer can decide what to do; it does not enforce a release policy. “No references observed” means only that Quasar found none in the indexed population. Reflection, configuration, dynamic loading, stale branches, or incomplete indexing can hide consumers. The result needs to show whether the requested commits were indexed, how fresh the consumer indexes are, and whether evidence was truncated.

Pull-request review requires indexed snapshots for the exact base and head commits. If either is unavailable, Quasar reports that it cannot compare them; it does not substitute a default-branch snapshot. This keeps a missing comparison from looking like a clean API diff.

Return to our Client change. This illustrative report brings together signals available through Quasar's API comparison, consumer-usage queries, and library-level criticality. It is not a literal response from one API:

{
  "change": "REMOVED",
  "symbol": "Client#getLegacyValue(Request)",
  "affectedRepositories": 137,
  "versionDistribution": [
    { "version": "4.2.1", "repositories": 86 },
    { "version": "4.1.7", "repositories": 38 },
    { "version": "3.9.4", "repositories": 13 }
  ],
  "sampleConsumerLocations": [
    {
      "repository": "checkout-service",
      "filePath": "src/main/java/com/example/checkout/CheckoutClient.java",
      "lines": [84, 129]
    },
    {
      "repository": "client-wrapper",
      "filePath": "src/main/java/com/example/client/ClientAdapter.java",
      "lines": [57]
    }
  ],
  "libraryCriticality": {
    "tier": 1,
    "basis": "highest-tier downstream application"
  },
  "comparison": {
    "baseShaFound": true,
    "headShaFound": true
  }
}

The repository count is symbol-specific: in this example, 137 indexed repositories refer to the removed method. The version distribution reveals a dominant cohort on 4.2.1 and a smaller long tail still on 3.9.4. A producer can inspect affected repositories and retrieve indexed file paths, with line numbers where the index provides them. The JSON shows only two sample locations to keep the example readable. The filePath and lines shape mirrors Quasar's query contract. The tier is broader context about the library, inherited from its highest-criticality downstream application; it does not imply that all 137 callers are Tier 1.

Together, those signals change the producer’s plan. Instead of merging the removal and waiting for failures, the producer can retain and deprecate the old method, publish a migration path, coordinate a broader migration, and track the long-tail versions before removal. File paths and available line numbers turn that scope into a concrete work list for engineers or coding agents. The same API diff paired with weak or stale evidence would lead to a different action: refresh the indexes before deciding. The value is not merely detecting the removal; it is choosing a proportionate next step.

A Polyglot Foundation with SCIP

The shared SCIP format also lets Quasar index several language ecosystems. Each language indexer resolves its own code, while Quasar can use the resulting symbol identities to find references across repositories.

Conceptually, client-lib indexes Client#getLegacyValue(Request) as a definition, while checkout-service indexes a reference to the same symbol. Quasar can then connect the library API to its consumer across repositories.

Relationships are especially useful when generated code separates the API a producer owns from the symbols consumers invoke. For example, Quasar can trace a Protobuf definition through its generated Java members to the repositories that use them, even when consumer code never names the .proto element.

SCIP records symbols and their references; it does not determine whether an API change violates an ecosystem’s compatibility rules. Today, Quasar compares indexed JVM and Protobuf API definitions and traces Protobuf changes through generated Java to find affected repositories. A separate JVM compatibility-impact path uses breaks identified by a producer-side tool to query affected repositories. Go, Python, JavaScript/TypeScript, and Rust support usage queries, but Quasar does not yet assess their breaking API changes.

Moving Risk Assessment Earlier

Today, Quasar surfaces API-change findings on pull requests, including observed consumer counts and confidence. Producers can use its other queries to inspect dependency versions and source locations, with library criticality as broader context. Together, those signals support a decision before merge.

We want to move that context earlier. An agent writing library code should recognize when it is changing a public symbol and consult Quasar before the change becomes a pull request. Quasar’s APIs and Model Context Protocol (MCP) integration provide an access path, but access alone does not create awareness. The coding workflow must also recognize public API edits so the agent knows when to query impact and when to involve the producer.

Earlier guidance will not replace pull-request analysis. The PR assessment will still verify the concrete diff and catch changes that were not recognized during coding. Coverage, freshness, ambiguity, unsupported platforms, and truncation must travel through both checks in machine-readable form. An agent should not turn “no observed usages in the available index” into “safe to remove.”

Today, agents can query Quasar through MCP when prompted. We are working toward making that context part of the coding workflow, then using identified call sites to prepare migration changes. Quasar supplies semantic evidence, an agent reduces migration toil, and the library producer remains responsible for API and release decisions.

What We Learned

Building this workflow led us to four practical lessons.

• Separate compatibility from reach. A removed method can break the API contract even when Quasar finds no callers. A list of affected repositories and their library versions shows the migration work ahead. Producers need both signals to choose a release path.
• Make the evidence inspectable. A repository count is more useful with sample call sites and a version breakdown. The result also needs to show which commits were indexed, how fresh the consumer indexes are, and whether evidence was truncated. Otherwise, “no usage found” can be mistaken for proof that nobody is affected.
• Keep usage and compatibility separate across languages. SCIP lets Quasar find symbol references in several ecosystems. Deciding whether an API change is breaking requires compatibility rules or a trusted producer-side signal for that ecosystem.
• Bring evidence to the producer while choices are open. Pull-request findings can prompt a producer to retain an API, publish a migration path, or coordinate consumers before merge. Moving the same evidence into the coding workflow could shape the edit earlier. After release, it still helps locate consumers that need to migrate.

Today, Quasar indexes JVM, Go, Python, JavaScript/TypeScript, and Rust code for cross-repository usage analysis. It compares JVM APIs and traces Protobuf changes through generated Java. Producers can review change findings through Quasar’s REST and GraphQL APIs or its MCP server, then investigate affected repositories and call sites. We are extending this foundation toward more ecosystem-specific compatibility checks, deeper code-review integration, and agent-assisted consumer migrations that keep release approval with the producer.

Quasar is internal to Netflix, but the pattern is portable: index code semantically, retain API definitions, compare them at the point of change, query affected consumers, and report the quality of the evidence. Library producers need more than a compatibility verdict. They need to know who may be affected and have enough time to act before release.

A huge thank you to the Java Platform team at Netflix. Their expertise, collaboration, and support made this work possible

Related Talks

• From Lag to Lightning: Transforming Dependency Update Timelines explores how Netflix shortened dependency-update timelines.
• Netflix’s Journey to Confident Automated Code Changes discusses building confidence in automated changes at scale.

Beyond API Compatibility: Building a Tool to Measure the Blast Radius of Library Changes was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

You click play on a Friday night, and in the second that follows, dozens of services answer in milliseconds. Most of them run on Titus, our internal container platform. Titus is, in essence, a Kubernetes-based container hosting platform: we keep both the control plane and the data plane (the container hosts that actually run containers) patched and up to date, so application teams get a consistent and reliable container experience. They hand us a container, and we run it, with no host, cluster, or kernel to own. Titus launches millions of containers every week across tens of thousands of EC2 instances in multiple AWS regions. It is multi-tenant by default: many containers from many teams share each host. That density is what makes container compute efficient, and it is why Titus has been the default home for so much of Netflix’s compute for years.

For most workloads, sharing a host works well. For a few, it does not, and those few include some of the most demanding real-time systems we run. This post is about them, and about single-tenant Titus, a mode that gives a task its own virtual machine while keeping everything else about Titus the same: the API, the deployment path, and the managed experience where the Compute/Platform team, not the application team, owns the host.

We will walk through why shared-kernel isolation runs out of road for certain workloads, how we measure isolation rather than asserting it, the options we weighed (microVMs, gVisor, EC2, and more shared-kernel tuning) and why a dedicated VM inside Titus won, how the mode is built on the Titus federation layer and autoscaling, and what we have seen along three axes: isolation, fleet efficiency, and the operational cost of keeping the fleet patched.

The cost of good neighbors

Multi-tenancy is a tradeoff. Packing many containers onto a large bare-metal instance keeps utilization high, but it also means those containers share a single Linux kernel and the underlying hardware. Titus isolates them with the tools the kernel provides: cgroups cap CPU, memory, and disk; namespaces give each container its own view of the system; seccomp and AppArmor restrict the system calls a container can make; the Completely Fair Scheduler divides CPU time; the iocost controller meters disk IO; and traffic shaping meters each container’s slice of the network. Each container also gets its own identity, IAM role, and security groups, so tenants are isolated for security and resource management.

These mechanisms provide isolation for what the kernel can partition. They cannot fully partition what is shared in hardware. A group of cores shares a last-level cache and a path to memory, so a container that suddenly pulls a lot of data from RAM evicts a neighbor’s cache lines and slows it down, even when both stay inside their CPU limits. There is no cgroup for memory bandwidth or cache residency, as there is for CPU shares. The interference is real, it is hardware, and it does not show up as either container exceeding its quota.

A noisy neighbor and a co-located app each stay within their cgroup CPU and memory limits, but both draw on a shared last-level cache and memory bandwidth that lack cgroup fences. The noisy neighbor takes most of it, so the co-located app stalls and its tail latency rises even though no one exceeded a quota.

We have written about this before. Predictive CPU isolation of containers at Netflix moved some placement decisions out of the kernel and into a service that forecasts each container’s CPU demand and re-pins containers to cores to reduce cache interference, formulated as a combinatorial optimization solved every minute. Noisy Neighbor Detection with eBPF instruments the kernel scheduler to surface per-container run-queue latency, so we can see a victim being starved in close to real time. And Mount Mayhem traced container-startup slowness to a global kernel lock amplified by modern CPU cache topology. Each of these raised the floor for everyone on the platform, and we are continuing that work.

A decade of shared-kernel isolation, at Netflix and across the industry, has not made co-tenancy invisible, because a noisy neighbor is not one problem but a family of them. Almost every physical resource a host shares is its own axis of interference, and a cgroup can fence very few of them:

• CPU performance indeterminacy. Co-located threads contend for the run queue, so scheduling latency rises even while a container remains within its CPU quota. Separately, a neighbor with a large working set evicts a tenant’s last-level cache lines and saturates memory bandwidth, neither of which a CPU share can partition.
• Network bandwidth and packets per second. Per-container shaping competes for link bandwidth, and receives processing runs in softirq context, charging one container’s packet work to whatever CPU a neighbor is running on. A flood of small packets exhausts a core’s softirq budget long before any bandwidth cap is reached.
• Storage bandwidth and IOPS. A neighbor saturating a device’s queue serializes everyone’s I/O, and on cloud block storage, the per-volume and per-instance throughput ceilings are a shared budget in their own right.
• Long tail. Page-cache reclaim, NUMA and cross-socket interconnect traffic, TLB-shootdown interrupts, PCIe and GPU contention, package-level power and thermal throttling, and contention on kernel global locks (the class of problem behind Mount Mayhem) each degrade neighbors in their own way.

What unites them is that they are physical, shared, and below the level a quota can reach. A platform can cap how much CPU time or memory a container is granted; it cannot hand each tenant its own slice of the memory bus, its own softirq budget, or its own portion of the last-level cache.

Of these, the network dimension has bitten hardest in our experience. Per-container bandwidth shaping uses a hierarchical token bucket (HTB) queueing discipline that serializes traffic with a global lock to account for each container’s bytes and packets. Under a flood of small packets from a single container, contention for that lock can burn kernel CPU and starve co-located containers of the time they need to process their own traffic, resulting in TCP retransmits and tail-latency spikes on services that did nothing wrong. The mechanism meant to prevent noisy neighbors becomes one.

The workloads most exposed to this are stateful, high-throughput streaming jobs: real-time data pipelines and stream processors that continuously move large volumes of data and maintain in-memory state. When one of these is perturbed, recovery is not cheap. A stream processor that falls behind or loses its place may have to restore from a checkpoint and pause processing for many minutes to catch up. For these jobs, intermittent interference from a neighbor is not a small slowdown that averages out; it is a stop-the-world event that pages an on-call engineer. The same teams report, accurately, that they never saw this on a plain EC2 instance.

Why we kept coming back to the VM

Full VM isolation, where a workload gets its own host and kernel, is the standard our customers are already happy with. The question was not whether VM isolation works; it was how to offer it without giving up what makes Titus worth using. We weighed four paths.

1. Keep improving shared-kernel isolation. This helps every workload on the platform, needs no opt-in, and we are still investing in it (more on the roadmap below). Its limitation is structural. Each round of tuning narrows the gap without closing it, because the underlying hardware remains shared. Cache, memory bandwidth, and the kernel’s own shaping and scheduling locks are not things a container boundary can fully contain. Shared-kernel isolation is the right default and the wrong promise for a workload that cannot tolerate a bad minute.
2. microVMs (Firecracker, Kata Containers). Lightweight VMs boot in about 150 milliseconds by emulating far less hardware than traditional VMs, which is why they power serverless platforms. They were the most appealing alternative on paper, and we carefully reviewed them. Three things made the cost hard to justify. First, the hardware virtualization they rely on is available only on bare-metal instances or through nested virtualization, and nested virtualization was not ready on the instance types we run at Netflix at the time, so adopting microVMs broadly would have constrained us to a narrow slice of the fleet. Second, there is no managed microVM service we could build on, which means we would have to run and maintain a virtualization layer ourselves: patching the hypervisor, live-migrating guests, and qualifying emulated devices. That is exactly the kind of operational surface we prefer not to own when we can avoid it. Third, even setting those aside, microVMs add virtualization overhead on CPU- and IO-bound work and do not support GPUs. Excellent isolation, but at a price in flexibility and operational load we were not willing to pay.
3. gVisor. A user-space kernel that intercepts system calls provides sandbox-grade isolation without a hypervisor, thereby avoiding the bare-metal constraint. The catch is performance and dependency. Intercepting syscalls in user space adds overhead on IO-heavy paths, which are precisely the workloads we most need to protect, and the open-source project lags behind its internal counterpart in fixes and is effectively maintained by a single vendor. The isolation we would gain came with a throughput tax on the exact jobs we were trying to help.
4. Move the workload to EC2. It is possible to run containers directly on EC2 instances. The problem is that the application team takes back the operational load that Titus exists to carry: choosing instance types, patching hosts, and sizing and scaling the fleet. For a team that came to Titus precisely to avoid those things, this is a regression in the experience, and the move between Titus and EC2 has historically been a one-way door that teams are reluctant to cross.

Spectrum of isolation options from lighter and more shared to stronger isolation: shared-kernel containers, then gVisor, then microVMs (Firecracker or Kata), then a full VM as used by single-tenant Titus, which is highlighted. Each option is annotated with its launch latency, overhead, and the entity that operates it; single-tenant Titus is a dedicated EC2 instance with no additional virtualization overhead.

There is also a question of what kind of isolation we actually need. microVMs and gVisor are built for security isolation between mutually distrusting tenants, the case where one tenant might actively try to attack or escape into another. Among Netflix teams, that threat model is more than the situation warrants. What these workloads need is resource and performance isolation: a guarantee that a neighbor cannot steal their CPU, cache, memory bandwidth, or network, not protection from a hostile co-tenant. A dedicated EC2 VM delivers exactly that, with hardware we do not have to emulate and a kernel we do not have to sandbox. We pay for the isolation we need rather than for a stronger security boundary.

microVMs and gVisor asked for a large, bespoke investment with real risk in order to approximate something we could get directly: a VM. Using EC2 to run containers provides the VM, but takes away the managed experience. So we took the direct route and brought single-tenant VM isolation into Titus itself, under the existing API and the existing shared-responsibility model.

How single-tenant Titus works

The idea is simple to state: a single-tenant task runs on its own EC2 instance, sized to fit it, with nothing else scheduled alongside. That instance is still a Titus container host that the Compute/Platform team owns and patches. The team running the workload sees the same Titus API and the same deployment path. Nothing about how they package or ship the job changes.

Side-by-side anatomy of a multi-tenant Titus, where multiple teams’ containers share one Linux kernel and the hardware beneath it and contend with one another, versus a single-tenant node where one task plus its platform sidecars owns the whole machine with no co-tenants.

Most of the work happens in the federation layer, the part of Titus that decides where a job runs (which cell, which capacity) before it is scheduled onto a node.

When a job is eligible for single tenancy, three things happen.

1. Federation decides eligibility and the envelope. Eligibility is governed by an allowlist maintained by the Compute/Platform team; a team opts in by targeting a single-tenant workload group rather than flipping a flag in the public API. For an eligible task, federation tags it as single-tenant and records the set of instance families it may run on, ensuring consistent performance across the workload. Consolidating this decision in federation follows the same principle we use for routing in general: placement intelligence belongs in one place, above the individual cells, where it can account for capacity and availability across the region.
2. Autoscaling provisions a dedicated node. Given a pending single-tenant task, our autoscaling implementation launches the smallest instance that satisfies the task’s requested CPU, memory, and GPU from the allowed families for the task just in time. Taking the tightest fit keeps the unused remainder of each dedicated node small. The next section covers this in detail.
3. The scheduler enforces one task per node. Single-tenant pods are configured with pod anti-affinity to prevent colocation with other pods, along with a priority class that exempts the task from preemption. Single-tenant nodes are also tainted to prevent unrelated workloads from being placed there, with only single-tenant workloads tolerating this taint. We also disable defragmentation/repacking for these nodes, so a single-tenant task is never relocated to repack the fleet. Together, these mean a single-tenant task has its host to itself for its entire life, undisturbed by a neighbor or by the platform’s own bin-packing.

Two design choices are worth dwelling on. First, single tenancy is allowlisted on purpose. If it were a free checkbox in the API, most teams would tick it, because who wouldn’t want a quieter host? And the efficiency that multi-tenancy exists to provide would erode across the fleet. Keeping the Compute/Platform team in control lets us direct dedicated isolation to workloads that need it and steer the rest toward shared capacity, while keeping the public API surface small as we learn. Second, we do not silently resize anyone’s job to fill its node. The job keeps the resources it requested; we recommend sizing it close to a node shape so the dedicated instance is well utilized, while keeping the workload portable and running identically whether in single-tenant mode or not, regardless of the host’s dimensions. Moving in and out of the mode is a configuration change, not a rewrite.

A few perpetrators, a lot of collateral damage

Noisy-neighbor pain is not spread evenly. In practice, a small number of workloads cause most of the disruption: a job that floods the host with small packets, churns the page cache, or saturates memory bandwidth degrades the performance of every other container unlucky enough to share its host. Some of these are also among the workloads most affected by contention, because a job can be both a source of interference and sensitive to it. The shape of the problem is a few perpetrators and many victims.

That shape is what makes single tenancy a precise lever rather than a blunt one. We do not need to give every workload its own machine. We need to pull the handful of perpetrators and the most disruption-sensitive victims onto dedicated nodes. Doing so protects their former neighbors directly and removes the reason the platform was spreading sensitive workloads apart with anti-affinity placement in the first place. The hosts left behind are quieter, and because they no longer have to be kept partly empty to honor spread rules, they pack more densely.

Before-and-after view of the Titus fleet. Before: a few red perpetrators are scattered across shared hosts, the amber workloads beside them are disrupted, and anti-affinity spreading leaves dashed stranded capacity. After: each perpetrator runs on its own dedicated single-tenant node, the remaining shared hosts are fully loaded with healthy workloads, and one host’s worth of capacity is reclaimed.

On the left is the fleet before: perpetrators sit on shared hosts, the workloads next to them suffer, and anti-affinity rules leave gaps that cannot be filled. On the right is the same fleet after: each perpetrator has its own right-sized node where it can be as noisy as it likes, with no one to disturb; the remaining hosts run full of undisturbed workloads; and the capacity that used to be stranded comes back. A few dedicated nodes buy isolation for the perpetrators and a calmer, denser fleet for everyone else.

Provisioning many small nodes

Single tenancy trades fewer large, shared instances for more small, dedicated ones, and that shift has consequences across the stack.

The first is the control-plane load. More nodes mean more for the Titus control plane to track and more VM launches against the EC2 control plane, the latter mattering most during a regional failover, when we scale up the fastest and can least afford a throttle. Titus is already a federated system: each region is divided into independent cells, which limit how large any single control plane can get, and let us add capacity by adding cells rather than growing a single stack without limit. We migrate workloads in stages rather than all at once, and we exercise the launch-rate path under controlled load on a schedule rather than discovering its limits during an incident.

The second consequence is capacity management, where the design changed the most. With one task per node, the fleet now spans many instance shapes whose demand shifts continuously as jobs come and go. Pre-staging static pools of every shape would be slow to manage and wasteful, so we instead provision single-tenant nodes on demand. It handles provisioning end-to-end, and several of its features line up with exactly what single tenancy needs:

• Just-in-time, right-sized provisioning. Single-tenant jobs are configured with a list of supported instance families based on their performance requirements. At launch, they are mapped to the best-fit node size by comparing requested resources against the selected instance families. New nodes of that instance type are brought up by the autoscaler to meet the pending demand. Single-tenant pods are configured with anti-affinity to all other pods, so the scheduler naturally lands one pod per autoscaled node. Newly vacated nodes may be reused by any pending deploys before they’re automatically torn down. The node pools themselves are shared across single-tenant workloads and describe a catalog of available instance families and sizes.
• Single tenant node pools. Single tenant may run in separate clusters or be shared with multitenant clusters. The pools are configured with a list of supported instance-type permutations and are pinned to a Titus container host AMI. This is where the managed experience lives: the host image and its configuration are ours, declared once and applied to every node created.
• Consolidation management. In multi-tenant environments, we default to a repack/defrag behavior that periodically relocates workloads. This doesn’t apply in single tenant, where nodes are explicitly each running at most one workload. We therefore reclaim only nodes vacated by workloads that have scaled in or terminated. Node termination is performed in two stages, allowing nodes to be reused for pending workloads before they are fully terminated.
• Drift-based patching. When we publish a new container host AMI, we support a “mutable” deploy to existing nodes without affecting any hosted containers, and an “immutable” deploy that updates the target AMI for newly created nodes. This means that many node upgrades don’t require disrupting workloads, but we reserve the right to roll the fleet with our users if needed.

The launch-latency tax, and why warm pools are harder here

Provisioning on demand does not make a node appear instantly, and launch latency is the tradeoff single tenancy asks us to manage most carefully. Bringing a single-tenant task to the point where it can serve traffic has two parts: standing the node up (booting a fresh EC2 instance, then pulling and extracting the container image, which is the largest part of host setup for most jobs) and the application’s own startup (a JVM loading and JIT-compiling its classes, a cache filling, and so on). A fresh instance takes tens of seconds before the image pull even begins.

In multi-tenant Titus, most of this is hidden. A new task usually lands on a Titus container host that is already running and already has the image layers cached, so starting it takes only a few seconds rather than minutes; the host boot was paid for once, long ago, and amortized across every task that has run there since. Single tenancy gives up that hiding place. Every single-tenant task waits for its own machine to boot and pull its image.

The usual remedy is a warm pool: keep some instances booted and idle so a pending task lands on one immediately. This works well for the multi-tenant fleet precisely because that fleet is close to homogeneous. A handful of large shapes cover almost everything, since any task can be packed onto a big shared host. A small amount of warm capacity in a few shapes gives a high hit rate, and its cost is spread across the many tasks that will use it.

Single tenancy inverts those economics, and this is the part worth being explicit about.

Press enter or click to view image in full size

Because every task gets a right-sized node, the fleet spans many instance shapes (several families across many sizes), and a warm instance only helps a pending task whose shape matches it. A warm r7i.2xlarge does nothing for a task that needs an m7a.8xlarge. To achieve a good hit rate, we would need to hold idle capacity in every family-and-size bucket, and idle capacity in a bucket with no task to fill is pure cost. The very property that makes single tenancy efficient at steady state, tight best-fit sizing, works against warm-pool reuse: more distinct shapes mean more buckets and a lower hit rate per dollar of standing capacity. And the pressure peaks exactly when it is least welcome. During a regional failover or a live event, when many tasks need fresh nodes at once, a thin warm pool spread across many shapes drains immediately, and the rest of the demand pays the full cold-boot cost.

So we treat warm capacity as a small, targeted lever rather than a default. We keep a little pre-warmed capacity for the most latency-sensitive single-tenant starts, and we deliberately narrow the menu of shapes to a curated set of sizes rather than the full continuum, so warm capacity concentrates into fewer buckets and trades a little best-fit efficiency for a better hit rate. Everything else provisions just-in-time and accepts the boot.

The more durable wins are in cutting the launch cost itself rather than pre-paying for it, and they help both single- and multi-tenant nodes, which is the better place to spend the effort:

• Faster image distribution. The image pull dominates the host setup, so newer OCI image formats and lazy-pulling snapshotters such as stargz and SOCI, which fetch only the bytes a container actually reads at startup rather than the whole image up front, reduce the largest component of the gap.
• Checkpoint and restore. For processes that are slow to initialize, snapshotting a warmed-up application with a tool like CRIU and restoring it, rather than cold-starting it, removes most of the second half of launch latency.
• Instance type selection. In multitenant configurations, we have historically used metal instances, which offer slightly better performance but come with a slower startup time. In a single-tenant configuration, node startup latency is more important, so we have moved away from these instances.
• Scheduling optimization. Through regular stress testing of fleet scaling, we found a scheduling bottleneck caused by Kubernetes preemption logic, leading to nodes being scaled up quickly but then left idle while waiting for workloads to be scheduled on them. This was ultimately unnecessary, as our single-tenant pools do not run preemptible workloads. Disabling preemption greatly improved scheduling performance when under strain.

None of this makes the tradeoff disappear, and it is worth stating plainly: a single-tenant node is slower to stand up than a slot on an already-warm shared host, and hiding that with warm pools costs more per node here than it does on the multi-tenant fleet, because the warm pool cannot be as homogeneous. But for the long-running services that single tenancy targets, paying that cost once at launch in exchange for a quiet life afterward is a trade these teams are glad to make.

Measuring the difference

Isolation claims are easy to assert and hard to trust, so we measure them with a repeatable benchmark and a single score.

The method is a differential one. We run a reference container, a healthy and representative application, and record its throughput and tail latency while it runs alone. Then we run it again next to a havoc container engineered to be a bad neighbor: a UDP flood through XDP to stress the network-shaping path, pointer-chasing and STREAM to pollute the last-level cache and saturate memory bandwidth, fsync storms against the disk, stress-ng to abuse the scheduler, and GPU-burn to monopolize an accelerator. We only trust a run once the host has reached a steady state, which we confirm with the kernel’s Pressure Stall Information: we wait until cpu_some or mem_some shows that tasks are actually stalling on a contended resource, not merely that the resource is busy. Each scenario is run many times with fixed seeds, and we report the median with a bootstrapped confidence interval, because noisy-neighbor distributions are heavy-tailed and a simple average can mask the spikes that matter.

From the reference container’s numbers, we derive a simple measure of how well isolation held. Throughput efficiency is the fraction of operations per second the app retains under attack; latency efficiency is the ratio of its undisturbed P99 to its P99 under pressure. A neighbor that doubles the reference app’s P99 drops the latency efficiency. As an illustration, a reference app whose P99 latency goes down next to a havoc container achieves a certain latency efficiency, and one whose throughput falls from a to b operations per second achieves a certain throughput efficiency. The composite score is a weighted blend of the two, with a latency-sensitive preset that gives greater weight to tail latency for interactive and streaming-path services. A perfectly isolated system scores 100%, and a single hard failure invalidates the run regardless of the average.

Noisy-neighbor isolation

cgroups container isolation isn’t perfect, and applications may experience worse performance due to sharing a machine with a determined neighbor. This could extend to disk and/or network IO where the same volumes or interfaces may be shared across workloads. For a distributed workload, this can lead to measurable inconsistencies in performance depending on where the replicas are placed. For more sensitive workloads, this variance can mean the difference between stability and getting paged.

The single-tenant option makes it possible to better isolate workloads that cause problems for their neighbors and/or are more sensitive to such problems. This can be used as a stopgap until the specific regression problem is resolved and consistency is restored, or kept permanent for load patterns that aren’t worth supporting in a shared environment.

Efficiency

The second effect is fleet efficiency, and it is the one people expect to go the wrong way. Dedicating a node to a single task consumes that node’s slack, so the naive expectation is that utilization falls. In aggregate, it can move the other way, for a reason related to how we dealt with poor isolation before single tenancy existed.

To protect sensitive workloads on shared hosts, the platform used anti-affinity placement, scattering them across the fleet to reduce colocation of the same workload on the same nodes. Spreading fights bin-packing: it strains capacity, because a host kept partly empty to honor an anti-affinity rule cannot be filled by a workload that would otherwise have fit there. When isolation-sensitive workloads move to dedicated nodes, the need for spreading and fragmentation goes away, and the remaining multi-tenant fleet can pack more densely.

Another source of fragmentation is services that are sensitive to disruption. These are typically stateful services where recovery is expensive. As these services become more common in a given population, they can lead to nodes being held up by pods with strict disruption requirements. This can lead to an inability to defrag and recompact the pool during off-peak times. The symptoms and resulting drop in fleet efficiency caused by disruption-sensitive workloads mirror what’s observed for workloads with anti-affinity rules.

Depending on the magnitude of the anti-affinity and/or disruption budget problem, moving such workloads to single-tenant pools can result in equivalent or even improved placement when all costs have been accounted for. It also becomes easier to quantify the cost to end-users. Rather than trying to attribute fleet-wide fragmentation to individual workloads, we can directly quantify waste as single-tenant node overhead.

The theory and the practice diverge here, and it was the practice that decided it. In principle, the efficiency ceiling is higher for multi-tenant bin-packing: pack enough tasks together, and a host fills almost completely. We do not reach that ceiling. AWS offers a discrete set of instance sizes, so a large shared host rarely matches the exact sum of the requests we place on it, and the gap between a host’s capacity and what we can actually pack onto it is a bin-packing hole we cannot fill. At the granularity of large instances, those holes are sizable. Right-sizing an instance to a single workload, picking the closest fit from many shapes instead of draping many workloads over one big host, captured more of the available efficiency in our experience than the higher theoretical ceiling ever delivered. The clean theory favored multi-tenancy; our measured fleet favored right-sizing.

Container-host maintenance

The third effect is operational, and it shows up in how we keep the fleet patched. A multi-tenant host is hard to take down for maintenance precisely because it is full: draining it means relocating many tenants at once, several of which carry disruption budgets that limit how fast they can move, so a routine patch becomes a careful, coordinated evacuation. A single-tenant host runs exactly one task, with consolidation and defragmentation turned off, so draining it means relocating only the one workload, and we can automatically handle that when a new AMI ships. The remaining multi-tenant hosts are easier to maintain, too, because the workloads that were most sensitive to disruption are no longer stranded on them.

What it changes, and what it doesn’t

Single-tenant mode is a configuration, not a migration. A team moves a workload onto a dedicated node, and back, with a labeling change, without rewriting anything and without leaving Titus. That turns the isolation decision into a reversible setting, which is the opposite of the one-way door between Titus and EC2 that teams used to face. A team can try single tenancy for a quarter, measure it, and step back if the trade-off isn’t worth it for them.

What does not change is the contract. The Compute/Platform team still selects the instance type, patches the host, and sizes and scales the fleet underneath. The application team still uses the Titus API, the same delivery pipeline, and the same observability. Single tenancy is a new point on the isolation axis offered within the existing managed experience, not a separate product that the team has to learn and operate.

When a dedicated node is the wrong call

A dedicated node is a real cost, and single tenancy is not free isolation. Two things are worth weighing before reaching for it.

First, a single-tenant task does not get the whole machine; it gets the machine minus overhead. Every Titus container host, dedicated or shared, runs the platform sidecars, system services, the kubelet, and the host environment itself, and that overhead is a roughly fixed slice of CPU and a few gigabytes of memory. On a shared host, that slice is paid once and amortized across the many tasks running there. On a dedicated node, a single task handles it. Because the cost is fixed rather than proportional, it is punishing as a fraction of a small instance and negligible on a large one: a small dedicated node can lose tens of percent of its resources to overhead, while on the largest shapes it rounds to a couple of percent.

Second, because a task is placed on the smallest instance that fits it, there is usually a gap between what the task asked for and the next available shape, and on a dedicated node, that gap is pure waste, with no neighbors to soak it up. The two effects compound: fixed overhead plus the best-fit remainder set a ceiling on how much of a dedicated node a workload can actually use, and that ceiling is lowest exactly where people are tempted to reach for isolation casually, on small instances. However, this sizing waste can be reduced through coordination with the service owner as needed.

The guidance follows from the arithmetic. Single tenancy pays off for large, long-running, performance-sensitive, disruption-sensitive workloads, sized close to a node shape and run as fewer, larger replicas rather than many small ones. It is the wrong call for small, bursty, or highly elastic workloads, where the overhead fraction is high, the dedicated node sits mostly idle, and multi-tenant packing is both cheaper and faster to scale. This is also why the mode is allowlisted rather than a self-serve checkbox: the platform can keep workloads on the side of that line where they belong, and leave the ones that would gain little from a node of their own on the shared fleet, where they are cheaper to run.

Who’s using it

The first adopters are the workloads that motivated the work: real-time data infrastructure. Managed Apache Flink stream processing, streaming data connectors, and telemetry and logging pipelines were both the most frequent sources of network contention and the most damaged by it, because their recovery is expensive and their disruption budgets are tight. These are stateful, high-throughput jobs where a few seconds of starvation can cascade into minutes of catch-up, and where the platform’s own repacking used to be a source of disruption in its own right.

Moving them onto dedicated nodes removed the class of problem rather than mitigating it. The teams running them report stable operation, having previously seen intermittent, hard-to-attribute interference, and their jobs are no longer relocated by fleet repacking, removing a second, self-inflicted source of disruption. Latency-sensitive serving paths and a handful of GraphQL and edge services have followed for the same reason: a predictable tail matters more to them than a few points of bin-packing efficiency.

This is an ongoing rollout, not a finished one. We are widening the set of eligible workloads as we learn, using the largest production jobs as the stress test for the platform’s scaling assumptions, and watching the control-plane and EC2 launch-rate metrics as the count of small nodes grows. We have also been running periodic stress tests to exercise scaling ahead of real user workload, and making improvements to autoscaling and scheduling logic along the way.

What’s next

We are not done on either side of the tradeoff. On single tenancy, we are reducing launch latency with smarter pre-warming of common shapes, broadening the allowlist as scaling data holds up, and extending the model beyond long-running services as cost attribution and capacity stories mature. On the multi-tenant side, which remains the right default for most workloads, we are addressing the dimensions that hurt the most. The largest is the network: replacing HTB’s global-lock shaping with an eBPF-based Expected Departure Time approach that rate-limits by timestamping packets rather than contending on a lock, and pairing it with a lower-overhead container network device. We are also evaluating hardware cache and memory-bandwidth partitioning (Intel RDT and AMD PQoS) to give the kernel a real lever on the last-level cache, hardware GPU partitioning (NVIDIA MIG) for accelerator isolation, and continued work on the cgroup v2 IO controller for disk. This isolation benchmark lets us rank these by return on effort rather than by intuition, and show customers, in the numbers, that the shared fleet is getting quieter over time.

Single-tenant Titus is also a step toward something larger. Netflix’s Platform teams offer two ways to get compute: virtual machines on EC2 & containers on Titus. We want to replace that with a single Compute API that takes intent rather than mechanism into account: how much isolation a workload needs, how quickly it must launch, how long it can run without disruption, and what business domain it belongs to. The platform then makes placement and runtime choices, and can make globally optimal ones because it sees the whole picture.

Conclusion

Multi-tenancy is the right default for a container platform, and we will keep making it better for everyone. But the right default is not right for everything. Single-tenant Titus gives the workloads that need it the isolation of a dedicated VM, provisioned and patched by our existing node management, under the same API and the same managed experience, without taking away the freedom to change their mind. We can now point to the difference rather than describe it: a tail that stops moving under a noisy neighbor, a fleet that packs more densely once spreading is retired, and hosts that are faster and safer to patch. For the streaming and data systems at the heart of Netflix’s real-time platform, a room of one’s own turned out to be the feature that mattered.

Single-tenant Titus: VM-level isolation on a multi-tenant container platform was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

When Jake Paul and Mike Tyson stepped into the ring in November 2024, we delivered the fight to roughly 65 million concurrent streams. On an ordinary day our infrastructure takes in more than 2 billion requests and handles upwards of 38 million real-time logging events every second. At that scale, the whole picture doesn’t fit into a single engineering team’s head or onto a single dashboard. With more Live content, Podcasts and real-time streams, our need to understand and update our systems as quickly as possible becomes imperative to provide a good user experience. Understanding the system at this scale and within the time constraints we have, often becomes a daunting exercise in finding a needle in a haystack, in a dark jungle, blindfolded.

This post is about that search for the needle.

The Four hours before a fix

Consider a regression of the kind we see in a given week. The home screen where you see the list of lists of movies, starts taking about 1.5 seconds longer to render. An alert fires. And then we go looking for what caused this issue and how do we fix it?

As it so happens, a regression on the application could be related to anywhere in the system — a client app regression from a new version deployment or A/B test rollout, a load shedding in the infrastructure, a GraphQL query responding with bad data because of a service rollout 6 hops downstream, are all suspects at this point. So, answering “what caused the issue?” needs insights from across the system, and this involves combing through layers of data in a meticulous manner.

The MELT Layer

“What caused the issue?”

Metrics, events, logs and traces, or MELT, in the industry’s shorthand, are the right raw materials for such an investigation, and each of them answers roughly a quarter of the question. We collect observability data across these four pillars, unified into a single observability data layer. This lets us observe our systems in real time, across what our users are experiencing, how our applications are doing and what our backend infrastructure and services are processing and responding to.

The investigation that follows may look like this in each layer:

The timeline of an investigation

So how long does it take to investigate an issue, figure out the root cause and fix it, so our users stop running into this issue again?

In our chosen example, four hours. Very little of it went into computing anything difficult. Most of it went into reconstructing relationships that our observability systems already knew but we had never written down anywhere a human or machine could simply read and discover.

The disagreement underneath

The instinct here could have been, around 2023, to reach for more data or better dashboards, and while both help at the margins, neither addresses what makes these investigations slow.

It’s 2026 now, so we might say, “hey, agents”, but without the right context delivered to the agents, AI systems hallucinate, and especially when teams are on-call and need a fix “now”, we cannot let inaccurate agentic analysis lead us down the wrong path. We need the right needle, not any needle.

Each of our systems is internally coherent and externally isolated. Atlas, our telemetry platform, knows applications and their time series. Radar, our alerting system, knows thresholds and firing history. incident.io knows severities and responders. PagerDuty knows rotations. Spinnaker knows what shipped and when. And our Slack history knows, in a form no query engine can reach, which engineer solved something very similar last quarter and what they found.

The difficulty is that they do not agree on what they are talking about. A thing that is an application in one system is a service in a second and a cluster in a third, with names differing by a suffix, a casing convention, or a rename only a couple of people remember (SMEs, ring a bell?). Joining across them requires a human carrying the mapping in their head, and that person becomes the bottleneck in such cross-domain investigations. The hard part of end-to-end observability is probably better understood as a data modelling problem.

What we lacked was an explicit, machine-readable agreement about the entities in our operational world.

The Ontology

This agreement has a name: an ontology. Ontology is a formal specification of types, the properties they carry, and the relationships they may have. Its unit is the triple, one fact expressed as subject, predicate, object. Consider the following example triples.

api-gateway   rdf:type       ops:Application .
api-gateway   ops:ownedBy    "Team Bedrock" .
api-gateway   ops:deployedIn "us-east-1" .

INC-5377      rdf:type       ops:Incident .
INC-5377      ops:affects    api-gateway .
INC-5377      ops:severity   "SEV 2" .

In the example here, we have already encoded six facts, and can answer in one query, a question that used to span three tools.

“Which incidents affected applications owned by Team Bedrock in us-east-1?” is a single query in place of looking up incidents, Spinnaker and Radar.

We call our ontology layer, The Observability Ontology. It acts as the schema, rule set and shared agreement that standardizes how different systems understand entities like applications, services, incidents, deployments, etc.

We built it on the W3C stack for this: RDF for the data model, OWL for the ontology, SHACL for validation, and SPARQL for querying. These standards are well specified and widely implemented, which keeps the model portable across graph stores.

The data from the ontology itself, is stored as a knowledge graph, persisted to QuipuDB, our homegrown RDF graph database, a distributed quad store based on redb, a Rust-based embedded key value db. Data here is generated by parsing raw telemetry, logs, and incident discussions into triples (subject-predicate-object facts).

Every harvest also produces a full Turtle serialization for archival, making it human-readable and diff-friendly, which has proven more useful for debugging than we expected.

Our ontology is organized into twelve namespaces, each owning one operational domain:

There are two decisions about this ontology that mattered while we built it, so they’re worth sharing.

The first is that we refused a generic Service type. “Service” means four different things across our tooling, so applications resolve to exactly one type, ops:Application, and every one of them is validated against our software catalog of roughly 49,000 entries before the entity is created. Unvalidated strings like usernames, team names, dependency names scraped from network topology, never become applications. We also cap dependency edges in both directions, because one popular service can carry thousands of reverse dependencies and would otherwise dominate the graph.

The second decision concerns what we chose to model. Incident chat carries the reasoning behind an incident. The hypotheses tried and discarded, the mitigation that worked, who knew why, is all context in the incident’s Slack channel and that knowledge evaporates when the channel goes quiet. So a 200-message, 50-person, 15-service incident channel becomes a structure, converting the operational chaos of an incident into ordered triples:

Creating determinism from non-determinism

Building this graph means using tools that are inherently stochastic. LLMs summarize discussions. Fuzzy matching resolves service names. A probabilistic engine proposes causal chains. None of these produce the same output twice with certainty, but what comes out at the end of our system is a deterministic graph that answers a SPARQL query the same way every time.

The ontology is the mechanism that creates this determinism through its pipelined approach. We think of the pipeline as a ladder that drives entropy down, where entropy means the proportion of information still trapped in unstructured form. For example, at the start of this pipeline, an LLM could say anything, but it cannot emit a triple that violates the type system. We constrain the output space tightly enough and a stochastic process yields a deterministic contract:

# The model may produce any prose. Its output MUST fit these predicates:
category  IN ["incident", "question", "discussion", "decision"]
sentiment IN ["positive", "neutral", "negative", "urgent"]

# Result — a deterministic triple:
thread:123  slack:category          "incident" .
thread:123  slack:isIncidentRelated "true"^^xsd:boolean .

There are five layers that get us down this ladder. Any single step of this ladder is non-deterministic. But in aggregate, the chain becomes deterministic. That inversion is the whole trick. We describe them below:

Enrichers: Agents as entropy reducers

We use what we call, enrichers, to reduce entropy across the knowledge graph. Dozens of enrichers run over the graph. Each one of them takes noisy input and emits typed triples, and each reads what earlier enrichers wrote. The contract for an Enricher is deliberately narrow:

class Enricher:
    async def _enrich_async(self, ...):
        raw    = await self.fetch(...)      # 1. read noisy input
        result = self.analyze(raw)          # 2. process (may be stochastic)
        graph.add((uri, RDF.type, OPS.X))   # 3. emit TYPED triples
        graph.add((uri, OPS.pred, Literal(result.value)))
        return EnrichmentResult(success=True, triples_added=count)

Three invariants that are standard across the enrichers:

• every entity carries an rdf:type,
• every URI follows a single naming scheme, and
• the catalog gates every ops:Application.

Two operational choices are worth naming here, as well. The enrichment pipeline runs sequentially — load the graph, add triples, and save.

This sequencing removes an entire category of race conditions at a cost in wall-clock time we are happy to pay. And calls out to external systems like metrics, logs, events and the rest could be best-effort, so a slow downstream API doesn’t block the graph harvest, albeit some degradation. To that end, we are also working to translate these separate systems into the ontology, such as the Metrics Ontology and the Alerts Ontology, so everything could be queried from the Ontology itself.

The Correlation Engine: finding the correlation chains

The correlation engine assimilates the vast number of signals from diverse input sources and condenses them into causal or correlatory chains of events. Today, the engine reads thirteen signal types, including log errors, metric anomalies, slow and failed traces, alerts, incidents, deployments, config changes, experiments, autoscaling, commits, end-to-end bottlenecks, and error patterns. It then searches for paths among them. Some ordered pairs are known to be causal and carry a high prior weight. For example, deployment before log error, config change before metric anomaly, failed trace before incident, and so on. Others, which show high correlation are considered correlation chains. Such chains are scored on temporal proximity and severity, inside a ten-minute window, with anything under 0.6 confidence getting discarded.

The role of the engine here is to propose with evidence, not to make conclusions. Hence, the engine stores its chains along with the graph records that were used to create them.

A 0.87 chain in the above example, is a strong hypothesis for a human or an agent to check, and the graph records the confidence alongside the claim so that consumers can understand why a chain is what it is and weigh their options to make decisions.

The knowledge flywheel

Each harvest cycle turns the same four steps, and each rotation leaves the graph richer for the next. This is the Knowledge Flywheel, leading to continuous enrichment and improvement of the context about our system, stored in the knowledge graph.

Observe collects the signals. Enrich adds typed triples. Infer finds patterns invisible in the raw data. Adapt feeds knowledge back, so that established patterns become detection rules. The compounding result of the four phases leads to a self-improving system. A causal chain discovered in one incident becomes a known pattern for the next, and the ontology gradually stops being a schema and starts being a memory.

Consider the example above. When we first see an incident in api-gateway caused due to a timeout and resolve it, we convert this knowledge into entries to the ontology. This becomes encoded knowledge. As we go through multiple cycles of this encoding, the graph starts learning patterns. After N cycles, patterns become wisdom — “api-gateway deploys cause 40% of SEV-2s” and “30% of deployment issues in api-gateway come from Connection Timeouts”. Surface this as an insight and we proactively start reducing incidents, not just react quickly to new ones.

Putting the graph to work

On top of the graph sits AutoRCA, our internal agentic AI solution, built with a set of specialist agents coordinated around one question. Walking the home page regression from the top of this post, the agentic system’s reasoning chain might look something like this:

Each specialization to the inputs is built by people who understand their respective domain deeply. That division of labour only works if the specialists can pass findings to one another without having to agree on the terms of communication or even have knowledge of each other. The ontology acts as the interface. Each specialist learns one vocabulary — the graph’s. A team writing specialized use cases for how they use traces record their findings as triples about an ops:Application, the logs specialization reads triples about that same ops:Application, and neither needs to know the other exists. A new specialist learns the same vocabulary once and can immediately read everything the other six have written.

Set against the investigation we opened with, the change is narrower than it might sound. The alert still fires at 3:30pm, an engineer still has to weigh the evidence, and the fix still has to be written by someone who understands the code. The hour or two that used to go into assembling context — finding the owning team, reconstructing what shipped, correlating information from different parts of the system, recalling that something similar happened last quarter — all that work is the part the ontology and the graph absorbs. We are early enough that the ranked hypotheses are sometimes wrong, which is why each one carries the evidence that produced it. But when the engineer now starts the investigation, they are already holding the picture that used to take half an afternoon to assemble.

If you want to build one of these

A fair amount of what we have built, we learned by getting it wrong first. Based on this experience, these are the principles we would carry into a second attempt.

• Model identity before anything else. Choose one authoritative catalog per entity kind and validate against it before creating an entity.
• Treat the ontology as code. Version it, review it, test it, and let changes arrive as pull requests. Treating the ontology as code lends itself to agentic development and lets teams scale and contribute to it.
• Separate concepts from instances. Concepts change on a human timescale and instances change continuously, conflating them makes both harder to evolve.
• Build on open standards. RDF, OWL, SHACL and SPARQL are specified, implemented many times over, and portable. A bespoke graph model ties a future consumer to one implementation.
• Constrain stochastic tools with a type system. This is the principle we would defend hardest. Models are useful precisely where determinism is impossible, and a sufficiently narrow output contract converts their output into something a query engine can trust.
• Make enrichment best-effort and ordered. A slow dependency should utmost degrade the graph, never block it. And ordering enrichers so each builds on the last buys more accuracy and that’s worth the tradeoff with parallelism that buys speed.
• Model human reasoning alongside machine telemetry. The most valuable edges in our graph came from discussion. Telemetry records what the system did, discussion records what people understood and interpreted. The reasoning is more valuable to store and serve as wisdom when we need it.
• Keep humans as the approvers. Let agents propose fixes, enrichers, and ontology changes, and route every proposal through a review process with humans in the loop.

What comes next

The graph makes three things possible. First, automatic root cause analysis, where a regression arrives with a ranked set of graph-backed hypotheses attached. Next, automated remediation, where a sufficiently confident diagnosis carries a proposed action like a targeted rollback, a traffic shift, a code change. And eventually infrastructure that heals itself, acting on a predicted failure before members feel it.

Alongside this, we plan to integrate the Observability Ontology with Upper, the domain-modeling language at the heart of Netflix’s Unified Data Architecture (UDA). Upper lets teams describe business and system domains as interconnected, machine-readable models and preserve their meaning and constraints as those models are projected into concrete schemas and systems. This integration would connect operational concepts such as applications, alerts, deployments, and incidents to the broader set of domains modeled across Netflix, giving engineers and agents a shared semantic layer for querying, validation, and reasoning. We will begin with a proof of concept that models incidents in Upper and use what we learn to determine where the two systems should converge.

Acknowledgements

None of this would exist without the teams who share their data, expertise, and patience with an early system. We want to specially acknowledge the efforts of our own team members who made this possible from very early on in this lifecycle, Kaela Marsheck and Aditya Gaur. We are also especially grateful to our collaborators in Netflix’s ontology working group, Peter Royal, Alex Hutter, and to Alexandre Bertails and members of the Unified Data Architecture (UDA)/Knowledge Foundations team for sharing Upper and their domain-modeling expertise, and for helping us explore how the Observability Ontology can become part of a broader semantic foundation at Netflix.

Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The motivation for building the Netflix Foundation Model for personalization was laid out in this previous blog post. The Netflix homepage is powered by several specialized models and it requires significant time and resources to maintain and continue innovating on these individual models. Therefore, we decided to centralize member preference learning and have one powerful model learned from comprehensive user interaction histories and content data at a large scale and have this model distribute its learnings to other models.

The previous two blog posts(1, 2) focus on the development of the Foundation Model and here we talk about how we integrate the Foundation Model into various applications. We found that there is extensive literature focused on the training and inference of large-scale transformer-based models. However, there remains a significant gap in practical guidance and research on how to effectively and efficiently integrate large models into existing production systems. At Netflix, we experimented with 3 distinct integration approaches — embeddings, subgraph and fine-tuning. The motivation for trying different approaches stems from varying application needs. Specifically, different teams have different latency requirements, tech stack constraints and different appetites for leveraging the full power of a large foundation model.

In this blog, we go over the 3 approaches we experimented with and which are now used in production for different use cases. Each integration approach has its own set of motivations, tradeoffs, complexities and learnings. The following sections go into the details of each of these integration approaches.

Embeddings from Foundation model

Transformer style foundation models naturally lend themselves to generating comprehensive user profile and item representations. Our approach for generating embeddings includes getting the hidden state of the last user event as the profile embedding and weights from the item tower as item embeddings.

Netflix has an elaborate set up for refreshing the embeddings every day. First, the Foundation Model is pre-trained from scratch every month. Then, the monthly pre-trained model is fine-tuned everyday based on the latest data. The daily fine-tuning also expands the entity id space to include the newly launching titles. After the daily fine-tuned model is ready, we run batch inference to refresh the profile and item embeddings and publish them to the Embedding Store.

An important part of this setup is the embedding stabilization technique (paper). When the pretraining job retrains the model from scratch, the embedding spaces between different runs are completely different due to the random initialization in the model. In addition, the embeddings start to drift when the daily job fine-tunes on the latest data, despite the daily fine-tuning job warm-starting from the previous day’s model. Therefore, it is important to map the embeddings generated everyday into the same embedding space, so that downstream models can consume pre-computed embeddings as features.

This embedding approach relies heavily on the Embedding Store, which is a feature store specializing in embeddings built by the platform team at Netflix. The embedding store makes producing and consuming embeddings easy by taking care of versioning and timestamping the embeddings. The Embedding Store also provides various interfaces to access the embeddings offline and online.

This approach is often the initial and most straightforward way for applications to interact with the Foundation Model. Embeddings can serve as powerful features for other models or for candidate generation, helping to retrieve appealing titles for a user or facilitate title-to-title recommendations. Integrating embeddings as features into existing pipelines is generally well-supported and offers a low barrier to entry. Compared to other integration approaches, using embeddings from the Embedding Store is expected to have a relatively smaller impact on training and inference costs.

However, this approach also has some cons. The time gap between embedding computation and downstream model inference can lead to staleness, impacting the freshness of recommendations. This may prevent applications from fully unlocking the benefit of the Foundation Model, especially for use cases requiring real-time adaptability.

Overall, the default starting point for leveraging the Foundation Model is through its profile and item embeddings via the Embedding Store, provided that the embeddings’ freshness and stability meet application-specific needs. While embeddings may not leverage the full power of the Foundation Model, they offer a good initial step.

Our biggest learning from working on this approach is that embeddings are a low cost and high leverage way of using the Foundation Model. A resilient embedding generation framework and embedding store are essential for updating the embeddings daily and serving them to application models. We found this infrastructure to be so valuable that we stepped up our investment here to build a near-realtime embedding generation framework. This new framework also enables us to update the embeddings based on user actions during the session, making the embeddings and our downstream models more adaptive. Although the near-real time framework cannot be used for very large models, it represents an important direction for addressing embedding staleness and adaptiveness of recommendations.

Foundation Model as a Subgraph

This approach involves using the Foundation Model as an input to the downstream model i.e. the Foundation Model decoder stack is a subgraph of the application model’s full model graph and hence the name “subgraph integration”. The Foundation Model subgraph processes raw user interaction sequences and outputs representations that are then fed into the downstream model graph.

Applications can fine-tune the Foundation Model subgraph as part of their own training process, potentially leading to better performance than just using static embeddings. In this approach, there is no time gap or staleness between the Foundation Model inference and the application model’s inference, ensuring that the most up-to-date learnings are used. This approach also allows applications to leverage specific layers from the Foundation Model that may not be exposed through the Embedding Store, uncovering more application-specific value.

However, the subgraph approach comes with some complexities and trade-offs. To use the Foundation Model as a subgraph, application models need to generate all the features necessary for the subgraph as part of their feature generation process. This can add time, compute and complexity to their feature gen jobs. Merging the Foundation Model as a subgraph also increases the application model size and inference time. To alleviate some of these complexities, the Foundation Model team provides some reusable code and jobs that make the feature generation more compute efficient. To optimize the inference performance, we split the subgraph and ensure it only runs once per profile per request and is shared by all items in the request.

Overall, the subgraph approach allows for a deeper integration and enables the applications to harness the full power of the Foundation Model. But it can also make the application’s training code more complex, requiring a trade-off between metric improvement, compute cost, and time. Therefore, we would reserve this approach for high impact use cases where the increase in metric wins compensate for the cost and complexity.

Fine-tuning Foundation Model and use directly

This approach is similar to fine-tuning LLMs with domain specific data. In the previous blog post, we described how the Foundation Model is trained on a next-token prediction objective, with the tokens being different user interactions. Different interactions are of different importance to various surfaces on the Netflix website. For example, for the ‘Trending now’ row, recent interactions on trending titles might be more important than old interactions. Hence, the Foundation Model can be fine-tuned on more product specific data and then the fine-tuned model can be used to directly power the product. During fine-tuning, application teams can choose to do full parameter fine-tuning or freeze some layers. They can also choose to add a different output head with a different objective. We built a fine-tuning framework to make it easy for application teams to develop their own custom fine-tuned versions of the Foundation Model.

This approach offers the ability to adapt the Foundation Model to application’s specific data and objectives, making it optimized for their use case. A side benefit of this integration approach is that it provides a de facto baseline for new models/applications. Instead of coming up with a new model stack and spending months on feature engineering, new applications can directly utilize fine-tuned foundation models. However, this approach can lead to more models and pipelines to maintain across the organization.The latency and Service Level Agreements (SLAs) of the fine-tuned model must be carefully optimized for the specific application use case.

Conclusion

We are continuing to innovate on these integration approaches over time. For example, the Machine Learning Platform team is working on near-real time embedding inference capabilities to address the staleness issues with the embedding approach. We are also working on a smaller distilled version of the Foundation Model to reduce the inference latency of the subgraph approach. We also provide a fine-tuning framework to make it easy to fine-tune and leverage the Foundation Model. We also continue to refine and standardize the APIs used in these approaches to make it easy for application teams to use them. Overall, these three integration patterns provide a comprehensive framework for how Netflix applications can interact with and extract value from the Foundation Model.

Acknowledgements

We thank our stunning colleagues in the Foundation Model team for developing and continuing to innovate on the Foundation Model. We thank our colleagues in the AIMS applications teams for driving the integrations and their valuable feedback and discussions. We also thank our Machine Learning Platform team for building and supporting the infra components to get these approaches up and running in production.

Contributors to this work (name in alphabetical order): Davis Shepherd Inbar Naor Jin Wang Jonathan Indig Kevin Zielnicki Luke Wang Mario Garcia-Armas Maryam Esmaeli Michael Tu Nicholas Hortiguera Qiuling Xu Robbie Greuner Roger Menezes Sejoon Oh Trung Nguyen Vito Ostuni Yesu Feng.

Integrating Netflix’s Foundation Model into Personalization applications was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Intro

Our Netflix journey from data to dashboards, particularly within the space of the experimentation platform, always presented a unique set of challenges. This blog post aims to illuminate these hurdles and explain how an innovative open-source metric platform became instrumental in overcoming some of them. We’ll showcase the critical role that DataJunction played in streamlining our data workflows and enhancing our ability to derive meaningful insights from our vast datasets.

Metrics are the foundation of data-driven decision-making, but at many organizations, they are scattered, inconsistently defined, and difficult to discover or trust. At Netflix, this challenge first became apparent in our experimentation platform, where reliable metrics are essential for interpreting test results. But the complexity of the original tooling made it very difficult to understand and introspect these metrics both in terms of their definitions as well as their values.

Initial Problem

Scattered metric definitions, redundant datasets, and pipelines are often a natural effect of a fast-moving company with a large and powerful analytical workforce. This is a common problem in big data-driven firms, and it is not easy to solve across multiple departments, which are often governed by somewhat different rules and regulations. An easy-to-use, well-governed, and centralized semantic layer can help with these issues. Our initial problem was most evident in our Experimentation Platform team, where the previous system required:

• extensive business knowledge
• familiarity with experimentation conventions
• and complex engineering practices

This led to metric authoring processes that would take weeks, which was a prohibitive requirement for Data Scientists and Analytics Engineers who wanted to onboard new metrics to the Experimentation Platform.

Enter DataJunction

To address this problem and to set ourselves on the path to a successful semantic layer solution in the future, we looked at several candidates. Among other solutions we evaluated Mando (an internal framework), Minerva (an open source semantic layer from AirBnB), and DataJunction (another open source solution, “DJ” for short). All of them had some strength, but DJ offered one thing that the others did not at the time: a solid SQL parsing and SQL generation engine. At the time of the POC, DJ was also the only open-source solution that stored all its definitions in a connected graph, similar to how relational databases store metadata on their views and queries. Easy “SQL wrangling” with rich metadata was the main feature that convinced us to try it for our immediate use case.

DataJunction originated from a similar internal Facebook project created (around 2015) by the data science and infrastructure (DSI) team, mainly as an answer to the need for metrics portability. During several iterations of the Deltoid project — which was the main experimentation platform at Facebook at the time — users were required to rebuild and repopulate their metrics when they wanted to move to the next version of Deltoid. But after implementing DJ and abstracting all metric definitions the team offered seamless migrations by separating the concerns between the metrics and the stats computation platform. That project was subsequently renamed to MDF (Metrics and Dimensions Factory) and promoted as a generic semantic layer used widely by various departments. In 2021, one of the Facebook engineers (Beto Dealmeida), started the DJ Github project and built out some basic functionality. Then together with some folks at Netflix expanded the scope further, and soon after that the project was introduced at Netflix. Following that introduction, three Netflix engineers (authors of this blog) together with a couple of external contributors, most notably Nick Ouellet, have been building out the remaining features of the platform.

Architecture Decisions

A critical design constraint for building DJ at Netflix was resource scarcity. We refused to compromise on product quality, choosing instead to limit the feature set. Although DJ was clearly capable of addressing many simultaneous problems across diverse use cases, our lack of adequate resources and official support meant we developed it as a side project for over two years. Our hope was that demonstrating its value would eventually secure management attention.

API First, UI and Other Interfaces Later

A metrics platform doesn’t need a flashy front-end to deliver value — its core is the semantic layer, which stores a consistent set of dimensions and metrics and serves this information through a rich API service. You can see below how we structured the main components of DataJunction in the Netflix implementation. Our DJ services (in the middle) interact with the underlying data (on the left) and supply our access layer with the business definitions in the form of SQL w/ metadata or directly with data depending on the client request.

Consistent Graph First, Other Features Later

A complete semantic layer system can seem like an octopus, with its tentacles reaching out into many functional areas and can expand from a basic semantic metadata stored for some datasets to full dashboard functionality. Let’s call out some of them:

• semantic metadata authoring and management and monitoring
• metric and dimension query / definition generation
• metric and dimension data generation (requires a query engine)
• data visualization and data tracking
• performance optimization (aka caching or materialization)
• integration layer with other tools and
• any / all the above functionality served as: UI, API, client libraries, MCP servers and more.

Of course it is extremely hard to provide the best in class functionality across all these layers, so with our small “unofficial” team we decided to focus on the most important piece: solid semantic layer engine (and API) with the ability to parse SQL as the input to the graph and generate SQL as the serving layer. That is the core of the system and everything else can be added later with less effort and w/o the need to rebuild the core.

Cube as The Perfect Consumption Protocol

Analytical multi-dimensional cubes have been a fundamental concept for decades, with their representation integrated across various data processing and analysis platforms. While definitions and features may vary between tools, they all serve the common purpose of encapsulating, or abstracting, a subset of semantic objects.

In DataJunction, we adopted cubes as first-class features for two main reasons: their familiarity to users and their suitability as a serving interface. They efficiently deliver both the semantic definitions (of metrics and dimensions) and fast data access. See an example of data “flow” graph (dotted lines) and dimensional relationships graph (solid lines) below.

We simplified the creation of basic cubes, allowing them to easily aggregate arbitrary collections of shared metrics and dimensions. Furthermore, accessing, managing, and visualizing these cubes is straightforward for any client integrating with DJ. Custom integrations, such as for Netflix’s internal fast Druid data analysis tool (Data Explorer → see below), became trivial to build, and we also made it easy to materialize these cubes directly in our Druid clusters.

One of DJ’s most powerful client integrations is the ability to link any DJ-originating metric definition back to the DJ ecosystem. This creates a valuable feedback loop: troubleshooting and introspecting business dashboards becomes simple if they display DJ-defined metrics that include links back to the definition source. This link then makes inspecting and running ad-hoc analysis, by modifying existing metrics, a trivial process. This integration is not only possible with our inhouse tools, but also with other well known open-source products, like Superset.

Experimentation at Netflix

Metrics are the backbone of experimentation. They translate data into signals that tell us whether a change is meaningful, safe, and worth scaling. Without trustworthy metrics and dimensions, experiments risk becoming inconclusive or misleading, so a robust metrics layer becomes crucial for data-driven decisions.

At Netflix, the original metrics layer integrated into the experimentation platform was overly complicated. Creating new metrics not only required tribal knowledge of experimentation data artifacts, but also familiarity with complex engineering workflows. This unnecessary friction meant that onboarding a single new metric could take weeks, slowing adoption of new metrics and hindering experimentation.

This is where DataJunction comes in: it provides a central hub for defining metrics and dimensions that can easily be shared and explored via the semantic graph. It also provides flexible, user-friendly interfaces (including a UI, API, and various clients), making it easy for users to author and manage their metrics and other semantic models.

Conclusions

With experimentation domain well supported by DataJunction, we would like to take on next steps to unifying of the general analytics at Netflix and to make it easier for Netflix users to both:

• find and use metric definitions across verticals (defined in DJ)
• easily access, modify and experiment with new metrics across Netflix data stack

That will require building out strong integrations between DJ and other tools, but that has been underway and it is one of the most enjoyable parts of this project, where most of the feature exists and we are simply enabling existing systems to exchange information in a structured and reliable way. With a system like DataJunction our internal LLMs can deliver accountable information about the business metrics as well as explanation of how they are defined and used across Netflix.

Related resources:

• 2023 : DataJunction at Netflix (slide deck)
• 2025 : DataJunction talk at Data Engineering Open Forum 2025 (video)
• 2025 : DataJunction: Unifying Experimentation and Analytics (blog)

DataJunction as Netflix’s answer to the missing piece of the modern data stack was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Personalization is everywhere on the internet. It shapes the products we discover, the stories we read, the music we hear, and the movies and TV shows we watch. At Netflix, personalization has been central to the member experience for decades — from the Netflix Prize to our modern recommender system that helps members navigate a large and constantly changing catalog.

But personalization is not only an algorithmic problem, it is also a measurement problem. How do we know how much value personalization creates? More precisely, how do we separate the appeal of a title from the value created by recommending that title to the right member at the right time?

In a new article in the International Journal of Industrial Organization — which we are also presenting at this year’s ACM Conference on Economics & Computation (EC ‘26) — we develop a model of member behavior, title choice, and personalized recommendations. The model lets us estimate how recommendations affect both whether members choose to watch and, if so, which titles they choose. In this post, we summarize our approach and main findings.

The Measurement Challenge

A natural first attempt at valuing recommendations is to ask: what share of viewing comes from titles that were recommended? This is an intuitive metric, but it fails to answer the causal question at hand.

The problem is that recommendations are not randomly assigned. A title appears on a member’s homepage precisely because the recommender system predicts that the member might want to watch it. If a member watches that title, was it because the recommendation caused the member to watch it? Or would she have watched it anyway because of her underlying preference for that kind of title? This is the core measurement challenge: recommendations are targeted to demand, so viewing a recommended title combines the effect of personalization with the member’s underlying preferences.

One way to solve this would be to run a large experiment that replaces Netflix’s personalized experience with an unpersonalized one. That would create clean variation — but it would also be a poor member experience. Personalization is a central part of how members locate valuable titles in a large catalog, so turning it off broadly is not an acceptable measurement strategy.

Instead, we take a hybrid approach. We build a behavioral model that separates members’ underlying preferences from the incremental effect of recommendations. The model uses the natural variation in the recommendation system, for example due to algorithmic exploration, to estimate how recommendations affect choice. Then, to validate the model predictions about title substitutability, we use small-scale A/B tests that “nudge” the salience of a handful of titles exogenously, generating ground-truth substitution patterns for a subset of titles. This lets us combine the scale and richness of modeling with the credibility of experimental evidence.

A Model of Member Choice

To measure the value of personalization, we need a model of what members would watch under different recommendation systems. Our model treats Netflix viewing as a day-by-day choice process. On each day, a member chooses among the titles available in the catalog, or she chooses not to watch anything. This gives us a concrete target to estimate: the probability that a member watches each title on a given day, as a function of their preferences, their viewing history, and the recommendations they see.

A simple version of this model would be too restrictive. It might assume, for example, that removing one title from the catalog shifts viewing toward all other titles in roughly proportional ways. But that isn’t how member choice works. If a member is considering a romantic comedy, the closest substitutes are likely other romantic comedies, not every title on Netflix equally. For similar reasons, two members may react very differently to the same recommendation.

To capture this complexity, we use a low-rank choice model. The model learns a representation of each title and a corresponding representation of each member’s current preferences. These representations are not limited to manually specified features like genre, cast, or language. Instead, they are learned from viewing behavior itself, allowing the model to capture nuanced similarities between titles that may be difficult to describe with coarse features, such as genre or release date, alone.

Member preferences are dynamic. What someone wants to watch today depends on what they have watched recently. A member who just finished a crime thriller may be in a different preference state than the same member after watching a stand-up special or a kids’ movie. Rather than estimate a fixed preference vector for each member, the model summarizes each member’s past viewing history into a current preference state. This lets us capture heterogeneous and time-varying tastes while also scaling to millions of members.

The key component for our purposes is that the model explicitly parameterizes the effect of recommendations. When a title is recommended, the model gives it an additional recommendation effect, or “bonus,” that increases its probability of being chosen. This bonus is a reduced-form way to capture two related mechanisms. First, recommendations can create awareness by helping members discover titles they may not have known about. Second, recommendations can provide information by signaling that a title is likely to be a good match.

Once estimated, the model becomes an engine for predicting counterfactuals. We can ask what would happen if a title were not recommended, if a different set of titles were recommended, or if the current recommender system were replaced with simpler alternatives. Because the model separately represents member preferences, title appeal, and recommendation effects, it lets us estimate not only how recommendations affect total viewing, but also how they change which titles are watched.

The model is only useful, however, if it captures real substitution patterns. The next step is therefore validation: comparing the model’s predictions to randomized experiments that directly perturb, on a small scale, the visibility that each title receives.

Validating the Model with Randomized Experiments

The model relies on an important assumption. After conditioning on the member, title, and state information available to the recommender system, we treat the remaining variation in whether a title is recommended as if it were randomly assigned. In our setting, this assumption is reasonable: we observe all of the information available to the recommender system, so we should be able to model its behavior accurately. But it is still an assumption. A useful model should not only fit historical viewing patterns; it should also predict what happens when recommendations change for external reasons.

Thus, to validate the model, we used randomized experiments that introduce variation in title visibility exogenously. These experiments mirror the kinds of variation that occur routinely when recommendation algorithms are updated: some titles receive more impressions, while others receive fewer. The key difference is that, in the validation experiment, this variation was prespecified so that we could estimate clean experimental benchmarks.

Specifically, we randomly assigned members to treatment arms that slightly nudged recommendations towards and away from different categories of titles. This let us measure diversion ratios: when a set of titles becomes less visible, how much viewing shifts to other titles, and how much shifts out of Netflix viewing altogether?

We then asked whether the model could reproduce those experimentally measured substitution patterns. We estimated the model on the control group, simulated the same recommendation changes, and compared the model-implied diversion ratios to the experimental ones. The agreement was strong: the model’s predicted diversion ratios had a correlation of 0.86 with the experimental estimates and an R-squared of 0.73.

This validation gives us confidence that the model captures counterfactual substitution patterns accurately. Subsequently, we ask what would happen under more dramatic changes, such as different recommender systems.

Counterfactuals: Aggregate Consumption and Diversity

With a validated model in hand, we can move from measurement to counterfactuals. Instead of asking only what happened under the current Netflix experience, we can ask what would have happened if members had seen recommendations generated by a different system.

We compare the current recommender system to three baselines:

1. The first is random recommendations: every member receives titles selected uniformly at random from the catalog. This is not intended as a realistic product experience, but it helps isolate the value of having recommendations that are relevant.
2. The second is popularity-based recommendations: every member receives broadly popular titles. This is a natural non-personalized benchmark.
3. The third is a basic matrix factorization model, a classic personalized recommendation approach and a useful baseline for older generations of recommender systems.

In each counterfactual, we replace the recommended titles while keeping the rest of the choice environment fixed. This lets us estimate how much viewing would change if the recommendation policy changed, holding constant the catalog, members, and model of member choice.

The results show that the current recommender system generates substantially more engagement than each benchmark. Relative to the current system, random recommendations reduce engagement by 16%, popularity-based recommendations reduce engagement by 12%, and matrix factorization reduces engagement by 4%. These are large differences at Netflix scale. Even the comparison to matrix factorization — a personalized baseline — suggests that continued improvements in personalization translate into meaningful gains in how effectively members find titles they want to watch.

Engagement is only part of the story. A recommender system could increase viewing while concentrating attention on a narrow set of already-popular titles. That would be a worse experience for members whose tastes are not well represented by those titles, and it would make poorer use of a broad catalog.

We therefore also examine consumption diversity: how broadly viewing is distributed across titles. The current system performs well on this dimension too. Popularity-based and matrix-factorization recommendations both substantially increase concentration, meaning that fewer titles receive meaningful viewership. Random recommendations produce slightly more diversity, as expected, but at a large cost to engagement.

Taken together, these counterfactuals show that the value of personalization is not just that members watch more. It is also that members are matched to a broader set of titles. The current recommender system increases engagement while preserving a wide distribution of viewing, reflecting the diversity of member tastes across the Netflix catalog.

Decomposing the Impact on Titles

The aggregate counterfactuals show that personalization increases engagement while maintaining broad consumption across the catalog. We can also use the model to ask a more granular question: why does a recommended title get watched?

There are three conceptually distinct channels:

1. The first is exposure: simply showing a title to a member can increase the probability that the member watches it. This captures the basic value of surfacing a title that may otherwise have been hard to find.
2. The second is selection: recommended titles are shown to members who already have a relatively high baseline probability of watching them. This channel doesn’t lead to incremental engagement, but it can smooth the discovery experience and help users feel that the recommender system understands their preferences.
3. The third is targeting: the recommender system may identify members for whom the recommendation itself is especially effective, meaning members who are much more likely to watch the title when it is recommended than when it is not.

The distinction matters because not all recommendation-driven viewing should be interpreted the same way. If recommendations worked only through exposure, then any reasonably visible surface could generate much of the same value. If they worked mostly through selection, then recommendations would mainly be finding members who were already likely to watch a title. Targeting is different: it captures whether the system is placing titles in front of members for whom the recommendation changes the decision.

Our model lets us estimate these channels title by title. For each title, we can compare the predicted probability that a targeted member watches it when recommended to the probability that an average member watches it when not recommended. We then decompose that difference into selection, exposure, and targeting.

We first show that exposure is meaningful: showing a title to an average member roughly triples its consumption probability relative to the baseline. Exposure is not the main driver of differences in consumption, as mechanical exposure explains only about 7% of the gap under the current recommender system, while selection and targeting account for the vast majority of the difference and the targeting effect is nearly seven times larger than the exposure effect.

This pattern is especially informative when compared with simpler recommendation systems. Random recommendations are dominated by exposure because they do not personalize: differences in consumption mostly reflect differences in what titles are shown. Matrix factorization changes this by making recommendations personal, reducing the role of exposure and leaving exposure, selection, and targeting to contribute nearly evenly to consumption differences. The current recommender system goes further: its gains come less from mechanical exposure, but more from targeting the right titles to the right viewers.

Targeting is also not uniform across the catalog. It is strongest for mid-popularity titles: titles with enough potential audience to be broadly relevant, but not so much mass appeal that everyone already knows about them. Very popular titles need less help from targeting, while very niche titles may have too small an audience to target effectively. The largest gains come from the middle of the catalog, where personalization helps connect specific titles with specific member tastes.

Conclusion

Personalized recommendations are valuable because they help members navigate a large catalog and find titles they want to watch. They are also valuable to Netflix because they make the service more useful, more engaging, and better able to serve a wide range of tastes.

But measuring that value is not straightforward. Recommendations are targeted by design, so we cannot simply compare recommended and non-recommended viewing. At the same time, turning off personalization at scale would create a worse member experience.

Our approach combines a large-scale model of member choice with true experimental validation. The model lets us separate underlying title preferences from the incremental effect of recommendations, and the experiments verify that the model captures real substitution patterns.

Using this framework, we find that personalization affects both how much members watch and which titles they watch. The value of recommendations comes not just from exposure, but from matching the right titles to the right members.

Measuring the Impact of Personalized Recommendations was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

In an earlier post, we introduced the TimeSeries Abstraction, our high-throughput, low-latency platform designed to store and query temporal event data at scale. It has since become the operational backbone for Netflix’s largest datasets, which now span multiple petabytes.

However, managing such large datasets in hot storage, tuned for low-millisecond latency access, is not cheap. At Netflix, cost-efficiency is a continuous discipline rather than a temporary initiative. This post shares two solutions that significantly reduce storage costs by continuously moving older, less frequently accessed data into cheaper object storage:

• Solution 1: Uses Apache Kafka and Flink to stream data directly to S3
• Solution 2: Leverages Cassandra backups in S3 and pluggable Memtables to read data directly from S3

We’ll start with the Kafka/Flink-based design, then describe how we evolved to a native Cassandra-based cold tier.

Challenges

As Netflix’s TimeSeries datasets have expanded, we’ve encountered several compounding challenges:

• Growing retention demands: Users often push for increasing data retention to serve a broader range of use cases, which in turn increases the storage costs.
• Escalating costs: Maintaining massive, dense clusters for infrequently accessed historical data is prohibitively expensive.
• Sparse historical access: Most queries target recent events, but older data must remain readily available for occasional audits and investigations.
• Operational strain: High storage density on hot nodes complicates scaling, deployment, and cluster stability.
• Latency requirements: Even historical data sometimes requires sub-second query performance for internal tooling and analysis.

This creates a clear opportunity: keep the most recent data on fast storage and migrate older, rarely read data to cheaper storage while preserving a consistent API. Done well, this can meet user and operational needs while greatly reducing our infrastructure spend.

TimeSeries Abstraction

Before we dive into the cold storage solutions, here is a quick recap of how TimeSeries partitions its dataset into multiple tables, each covering a window of time we call Time Slices.

This enables several key optimizations, most notably allowing a table to become immutable once it crosses the write boundary, which lets us safely apply further optimizations. For more details, refer to our previous post.

This slice information is critical and is maintained in a dedicated metadata table:

Creating new Time Slices and maintaining their metadata is a primary responsibility of the TimeSeries control plane. The TimeSeries data plane uses this information to correctly route the read and write traffic to the provisioned tables.

With that background, let’s look at the different solutions we built for data tiering.

Solution 1: Streaming with Apache Kafka and Flink

Our first solution followed a familiar streaming pattern using Apache Kafka and Flink, where we streamed data to S3 in realtime while simultaneously persisting it to Cassandra.

Getting Data Into S3 Efficiently

To optimize data streaming to S3, the system hash-partitions Time Series IDs into buckets. This partitions the stream and lets dedicated Flink tasks handle each bucket, ultimately batching and reducing the write throughput to S3. Flink workers batch events into Parquet files grouped by event time within the hour. These files are great for high-throughput streaming writes, but not for low-latency reads.

Enter the Compaction job.

Compaction Job to Optimize Reads

The compaction job runs daily and merges small Parquet files into a single, optimized daily Parquet file per namespace.

The job further optimizes the Parquet files in the following ways:

• Small Row Groups: Creates small (4–8 MB) row groups to minimize lookup cost. We can then use the Parquet Java library to perform range reads of row groups from S3. This is intentionally well below Parquet’s recommended row group floor — smaller groups were better because our reads are dominated by point/range lookups rather than full scans.
• Event Sorting: Sorts events within a row group by time, eliminating brute-force scans when querying for a time range.
• Bloom Filters: Caches Bloom filters and file metadata in EVCache, allowing the TimeSeries query planner to efficiently skip unnecessary reads from S3.

Once the compaction job completes, all compacted files per partition are recorded in the TimeSeries control plane, and the hot storage slices can now be truncated!

On some of our largest datasets, this job typically takes about 12 hours, prioritizing lower compute costs over execution speed.

Streaming Cold Tier Read

Once compacted files are optimized in cold storage, they are ready to be read. The TimeSeries data plane fans out a given query across both the hot and cold tiers and merges their result. The users still use the same query interface:

The Cold Tier Data Plane uses compaction metadata for aggressive planning and pruning, which dramatically reduces the S3 query fan-out. As a result, retrieving a full day of data requires at most three sequential S3 reads, even in the uncommon case where an ID spans two row groups. This yields a p50 latency of ~150 ms and a p99 latency of ~500 ms per read.

At the time of writing, the streaming cold tier implementation was processing up to 1,250 reads per second from cold storage globally:

Why not stop here?

The streaming cold tier has delivered real savings, shrinking both Cassandra’s EBS footprint and the Cassandra backup footprint in S3. But the streaming infrastructure itself is expensive and adds moving pieces that widen the team’s on-call scope:

We wanted to improve this further and asked ourselves:

• Can we avoid streaming data into the object store and the associated data transfer costs?
• Can we reduce the compute and operational overhead of new components?

That led to a more radical question: Can we read directly from existing Cassandra backups in S3?

Solution 2: Cassandra Cold Storage for Immutable Data

At Netflix, we back up Cassandra data into S3 at regular intervals for disaster recovery. The idea for tiering is simple:

Read directly from these S3 backups instead of paying a separate data transfer cost to S3.

Once TimeSeries marks a hot Time Slice as ready for cold storage, we can customize Cassandra to read directly from its backups, allowing us to delete the on-disk data copy. This would also incur minimal changes to the TimeSeries read path!

To make this viable in production, we had to answer a series of “How…?” questions.

How Can We Read the Backups Efficiently?

We introduced a new compaction strategy in Cassandra, which we call ReadOnlyCompactionStrategy (ROCS). Here is how it works:

Once a time slice crosses the user-configured cold-tier boundary, TimeSeries stops accepting writes for that time slice making it immutable. This immutability unlocks significant optimization opportunities:

• SSTable overlap: Standard strategies (LCS, TWCS) often produce SSTables that overlap in the token space, and so a single partition key may exist across multiple files.
• Read fan‑out: Because a read must touch every SSTable covering the key’s token range, this overlap turns a single point-read into multiple lookups, increasing latency across all tiers.

ROCS re-compacts all SSTables once, aligning them precisely with token ranges:

ROCS applies more broadly to all TimeSeries use cases, driving tail latency down on all storage tiers by reducing the SSTable fan-out during read operations:

Once the immutable Time Slices have been optimized, it is time to trigger backups.

How Would We Leverage Backups?

Cassandra backs up every node in the cluster, organizing each backup on S3 as a declarative manifest, along with the SSTables backing the keyspace’s tables:

 {bucket}/
  └── {prefix}/                             ← cluster identity (env_backup/hash_app)
      └── {token}/                          ← node identity (ring token)
          ├── META_V2/                      ← manifests, one per snapshot
          │   ├── {snapshotInstantMs}.json
          │   ├── {snapshotInstantMs}.json
          │   └── ...
          └── SST_V2/                       ← actual SSTable files
              └── {backupTimestamp}/        ← snapshot timestamp
                  └── {keyspace}/
                      └── {table-with-uuid}/ 
                          └── {compression}/
                              └── {encryption}/
                                  ├── {component-file}
                                  └── ...

The backup manifest orchestrates the backup, listing the paths to all its SSTables:

manifest.json
  ├── info
  │   ├── appName              "cass-cluster"
  │   ├── region               "us-east-1"
  │   ├── snapshotInstantMs    1748908800000
  │   ├── backupPathPrefix     "prod_backup/1234_cass/token"
  └── data[]
      └── [0]
          ├── keyspaceName     "ks"
          ├── columnfamily     "tbl-a1b2..."
          └── sstables[]
              └── [0]
                  └── components[]
                      ├── na-1-big-Data.db
                      ├── na-1-big-Index.db
                      ├── na-1-big-Filter.db
                      ├── na-1-big-Statistics.db
                      ├── na-1-big-Summary.db
                      ├── na-1-big-CompressionInfo.db

Alongside the large Data.db file, backups include a compact set of index, metadata, and integrity files. This structure allows us to quickly locate, validate, and restore specific partitions or data ranges without scanning the entire dataset. Furthermore, since the manifest catalogs every S3 path, we can download most of these components directly to the Cassandra node (excluding the Data.db file) where they occupy negligible space. These local components enable efficient query planning before any data is read from S3.

How Would Cassandra Read From Backups?

Cassandra reads typically pull data from two sources: the Memtable, which holds recent unflushed writes, and flushed SSTables on durable storage. As of Cassandra 4.1, we can leverage pluggable Memtables to redirect reads from “cold” tables directly to the backup storage in S3.

-- create table for a given Time Slice
CREATE TABLE data_20260616_00_04 (
    ...
    ...,
    PRIMARY KEY ((time_series_id, time_bucket, event_bucket), event_time, event_id, key)
) WITH CLUSTERING ORDER BY (event_time DESC, event_id DESC, key DESC)
    ...
    AND compaction = ...
    AND compression = ...
    AND memtable = 'backupmemtable:timestamp=1781953200000' ← Custom memtable
    ...

The timestamp marks the point at which the backup is guaranteed to cover the full table, and the table is guaranteed to be immutable.

Opening a backup Memtable kicks off a state-machine initialization that pulls the index data from S3 onto the node once:

This process ensures all SSTable components are downloaded reliably through several resilience mechanisms:

• Retries and backoff: All S3 operations and file-system cache writes use retries with exponential backoff.
• Resource isolation: Initialization runs on a dedicated thread pool, isolated from normal Cassandra operations.
• Partial download protection: If a downloaded component’s size doesn’t match the manifest, it is deleted and re-downloaded.
• DDL change protection: A .timestamp file marks completed initialization and ties it to the backup timestamp. If the table’s backup timestamp changes, this file is removed and initialization restarts for the new timestamp.
• Read protection: Memtable initialization is asynchronous; reads to an initializing memtable block for at most half the read RPC timeout.

Once initialization has pulled every SSTable component (except Data.db) from S3 onto the local disk, the Memtable is ready to serve reads. Here is what the read process looks like:

To minimize latency, the system parallelizes S3 reads across requests, SSTables, and individual chunks. A shared cache deduplicates fetches and prefetches data, so chunks are ready before they are needed.

We further optimize cluster and table settings to reduce the number of S3 calls:

• Larger compression chunks: Reduce S3 calls per partition by reading fewer, larger chunks (accepting some over-fetch on small reads).
• Read repair disabled: Avoid expensive, unnecessary repair traffic against immutable, pre‑repaired backup data.
• Speculative retries disabled: Prevent S3’s higher latency from triggering redundant, overlapping reads.

How Would We Know When It Is Safe To Read From Backups?

However, before the Memtable can point to a backup, we must confirm two things across the whole ring, and we use Cassandra Virtual Tables to do that.

First, that ROCS re-compaction succeeded:

cqlsh> SELECT * FROM netflix_views.cluster_view                                                                                                                                                          
       WHERE keyspace_name = 'system_views' AND table_name = 'sstable_tasks';

keyspace_name | table_name    | host     | value                                                                                                                                                        
--------------+---------------+----------+------------------------------------------------
system_views  | sstable_tasks | 10.0.0.1 | [{"table_name":"<.>","kind":"compaction", ...}]
system_views  | sstable_tasks | 10.0.0.2 | [] <---- no active compactions
system_views  | sstable_tasks | 10.0.0.3 | []

And second, that backups were fully captured (denoted by uploaded=True):

cqlsh> SELECT * FROM netflix_views.complete_backups                                                                                                                                                      
       WHERE keyspace_name = 'my_ks' AND table_name = 'my_table';

keyspace_name | table_name | timestamp     | num_tokens | num_uploaded | uploaded                                                                                                                       
--------------+------------+---------------+------------+--------------+----------
my_ks         | my_table   | 1765211400000 |          3 |            3 |     True
my_ks         | my_table   | 1765213200000 |          3 |            2 |    False

How Would We Automate This Entire Flow?

The TimeSeries control plane automates the entire transition as slices cross the immutable boundary:

Each slice can be in any lifecycle stage at any time without impacting system availability, and the system remains resilient to delayed backups or lagging compactions.

How Would Cassandra Handle The Additional Load?

Serving data from S3 is much more CPU-intensive than reading from local storage. A sudden spike in read requests can overwhelm the Cassandra cluster and affect the hot path, turning a cost-saving feature into a reliability risk.

To avoid this, cold reads can optionally run on a separate Cassandra cluster that points its backup Memtables at the hot cluster’s backups:

The cold Cassandra cluster must match the original cluster in topology but need not match it in hardware! We can run the cold cluster on much smaller, cheaper, storage-light instances.

Real World Performance

Cassandra Cold Tier was rolled out to the most storage‑intensive TimeSeries Abstraction clusters. It currently stores over 15 PB of compressed data in Cassandra cold storage and has allowed us to cut attached storage significantly on all affected clusters:

At the same time, it serves thousands of reads per second from cold tier storage in any given region:

Cassandra Cold Tier improves p90 latency by ~30% over the previous cold storage implementation, thanks to more efficient S3 access patterns and smaller read sizes, while keeping p99 latency under ~500 ms. Cassandra already handles bursts of cold reads gracefully via its chunk cache, so this design preserves that same resilience for the S3-backed cold tier:

Acknowledgements: Special thanks to our stunning colleagues who contributed to the TimeSeries Abstraction’s success and collaborated with us on the cold storage solutions: Kaidan Fullerton, Matt Lehman, Sumanth Pasupuleti, Carlos Matias Herrero, Nick Pavlakis, Bryan Keller, Francesco Di Chiara

TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.