Runway couldn't fix a bug in its AI video model, so it turned the bug into a feature

Runway spent weeks trying to engineer its way out of a stubborn bug: AI-generated avatars would drift off-center during real-time video generation. The fix wasn't a back-end patch — it was a new front-end feature that just worked around the problem. That's the kind of lesson Ryan Phillips, head of enterprise product at Runway ML, walked through at VB Transform 2026, arguing that even companies not building foundation models themselves can learn from how Runway builds, evaluates, and ships them.

"I think even if you are not all building models yourselves, it's helpful to learn how we do it because I think almost all of the lessons are applicable to what you all are doing day-to-day," Phillips said.

Runway is an applied AI research company building general world models to power generative tools. During his presentation, Phillips showcased Runway Characters, a real-time video model that enables zero-latency, back-and-forth interactions with AI-generated avatars. Five years ago, creating a video with illegible text and low framerates took artists hundreds of hours of stitching individual frames together, he said. Today, Runway’s models generate interactive video on the fly.

“Studying how we build these real-time models can inspire how you build and deploy real-time experiences, whether agentic or not, in your companies today,” he said.

Demystifying evals

Building a robust AI product starts with a high-quality evaluation set. However, creating this set cannot be treated solely as an engineering task. It requires deep cross-functional alignment across product, design, research, and sales to define what "quality" actually looks like.

Phillips emphasized running internal workshops where team members review generated examples together. The goal is to align the entire organization on specific failure modes so everyone shares a unified definition of a successful generation.

“We spent a lot of time working with our team, running through examples... of what success and failure looks like, down to the very, very detailed and picky things,” Phillips said.

The resulting evaluation set must cover broad customer use cases alongside extreme edge cases. For instance, Phillips highlighted that to ensure the model behaves predictably when pushed beyond standard human facial structures, they used “Tooth,” a non-human character with no nose and very unusual teeth.

When grading these generations, the Runway team looks for subtle artifacts. In one example, a video where a character’s face remained intact but background elements, such as a net, began morphing was strictly graded as a failure.

Despite the cutting-edge nature of the product, the tool Runway uses to track these evaluations is simple: an Excel spreadsheet. The team logs tests daily, categorizing outputs as "minor" or "major" failures against a predetermined pass rate. 

“We set a bar before we get started on what percentage we need to pass, and when we hit that, we ship the model,” Phillips said. “So it's not magical.”

For enterprise developers facing non-deterministic quality drift in their own real-time pipelines, manual evaluation at scale is a bottleneck. To solve this, Phillips noted that developers can rely on language models to automate the visual grading process. 

“LLMs are getting quite good at being a judge for a lot of this content, especially the types of morphing or changing that you would see in an evaluation set,” he said. Teams can also feed an LLM behind-the-scenes context (e.g., a hand-drawn sketch or an ad's structural layout) to guide the generation and validation processes, ensuring quality without adding cognitive load to the end user.

Model training and turning bugs into features

Delivering real-time generative video requires a highly optimized technical stack. The process begins with pre-training a massive foundation model, which is resource-intensive and slow to generate outputs. To achieve real-time latency, Runway relies on distillation, where a smaller, faster "student" model is trained to mimic the large "teacher" model. According to Phillips, distillation helps Runway cut down “80 to 90% of the generation time.”

The team then applies adversarial post-training (APT) to the distilled model. This technique forces the model to continuously improve by testing it against a system designed to find its flaws, helping regain the visual sharpness lost during the distillation process.

However, altering the model architecture introduces new problems. The distillation and APT phases introduced a stubborn bug: characters would sway or drift from the center of the frame during real-time generation.

The team spent weeks attempting to fix the core model to eliminate the drift, he said. Ultimately, they discovered that if the user's initial input image was perfectly centered, the generated video remained stable. Instead of spending more time on a backend engineering patch, Runway pivoted to a user experience solution.

“What we did was, when we noticed this in our evaluations, we then said, 'What if we just offered that as a feature?' If a user gives us a character that is turned to the left, we know the video is going to morph. Let's just fix it for them,” Phillips said. 

They introduced a frontend feature called "Optimize for Image Quality," which automatically re-centers the user's image before generation begins. By wrapping a backend model limitation in a frontend tool, users perceived a helpful feature rather than an engineering flaw.

“Turn model limitations into product features so that you can actually expand how the model works,” Phillips advised. “It might feel like a limitation internally, but your customers will not see it that way if you're kind of building this in as a product feature.”

The devil is in the infrastructure details

Delivering video globally at 24 frames per second requires optimizing every layer of the infrastructure stack. This ranges from caching and parallel decoding to making deep kernel changes in partnership with hardware providers like Nvidia.

Shortly after launching Runway Characters, he said the team noticed that 8% of API calls were dropping to 16 frames per second, causing the video to stutter for customers. 

Finding the root cause required deep observability. The team used an AI agent powered by Claude alongside monitoring tools like Datadog and Sentry to trace the anomaly. The debugging session isolated the problem to a single data center in the us-east-1 region.

“The solution actually wasn't [to] go fix anything or change a config,” Phillips explained. “They actually went and physically replaced those GPUs in the data center to fix it, and that ultimately solved the problems.”

For enterprise teams deploying real-time applications, the takeaway is clear: hardware and infrastructure anomalies will directly impact model performance, requiring rigorous, full-stack debugging capabilities. 

“Don't forget about all the small details, because there's so many of them when you're deploying these models,” Phillips said.

Surviving "failure hell" and the future of world-building

Developing AI systems is rarely a linear process. Teams often find themselves stuck for weeks on a single problem with no end in sight, a phase Phillips referred to as "failure hell.”

“We think you have to go through that pain and really struggle with the problem for a little bit before you can get the breakthrough,” he said. Consistent iteration eventually flattens the difficulty curve, triggering sudden, exponential improvements.

As the underlying models overcome these technical hurdles, the role of enterprise creatives is also fundamentally changing. Traditionally, marketing and design teams have focused on creating single assets, like a specific advertisement or illustration. In an era of real-time generation and agentic workflows, that paradigm is shifting toward defining parameters, aesthetics, and intellectual property.

“You might not be designing a single ad, but you might design a world that then the agent or a real-time video model can generate ads from,” Phillips said.