GENERATIVE AI

Runway Labs Introduces Diffusion-UI, a Research App That Generates Interactive Interfaces in Real Time and Can Use Structured Data

A floating dark app panel labeled Diffusion-UI shows a map route, weather icon, slider and buttons, fed by a small code card, over a blue-purple glow.

Runway Labs introduced Diffusion-UI, a research application in which a real-time model generates every pixel of an interactive interface, and shared an early look at generated interfaces that can use structured data, maintain states and generate controls. Precise map routes, live weather or any API can be passed in as JSON objects, which gives the model specific information to render a dynamic interface.

  • Four layers drive each generation. Runway Labs says Diffusion-UI generates using data sources, reference layouts, styles and text prompts.
  • Scenes are laid out by moving a box. Generated interfaces render each frame as you interact, so you can design a scene by moving and resizing a bounding box.
  • Text prompts edit the scene live. You can type a prompt to change the scene, add objects or edit anything in real time.
  • The model highlights its own controls. On hover, white outlines highlight nearby buttons, sliders and other controls to show which elements are interactive.
  • It is an early research application. Runway Labs calls Diffusion-UI a research application and an early look, and the thread shares no pricing, access or release details.

Diffusion-UI's demo clips cover live data, generated controls and 3D game scenes

In the thread, Runway Labs shows each of these abilities in short demo clips, including a generated word-tile board.

Runway Labs canvas labeled 'Word Tiles · live' showing a generated word-tile board spelling PLAY, YES and WOW, with a QUIRKTO tile rack
A generated, interactive "Word Tiles" board from the Diffusion-UI demo video, spelling PLAY, YES and WOW. Image: X (formerly Twitter)

Runway Labs also says Diffusion-UI is versatile enough to render complex 3D interfaces, pointing to immersive game scenes as an example.

The thread presents all of this as demo clips. It gives no resolution, frame rate or latency figures, and it doesn't name the model behind Diffusion-UI.

Runway Labs turns Runway's video and world models into prototypes

Diffusion-UI comes out of Runway Labs, an incubator inside Runway that turns video and world models into prototypes people can use.

Runway launched the Labs incubator to explore new applications for generative video and General World Models. It named Co-founder and Chief Innovation Officer Alejandro Matamala Ortiz to lead it with a team dedicated to incubating experimental products.

The thread closes with Runway Labs saying it is exploring tools that make composing and refining real-time interfaces "feel direct and familiar," with more prototypes and tools to share.

Runway's Solaris study favored generated interfaces over coded ones, with text and accessibility still unsolved

Runway has worked on model-generated interfaces before. Its Solaris research post introduced Interface World Models, real-time models that generate the interface itself frame by frame with no code underneath.

Solaris builds on Runway's Gen-4.5 video architecture and generates 720p frames in under 500 milliseconds, and in a 250-person Runway study participants preferred it over Claude Opus 5-coded interfaces 61% to 24% for following the requested interaction, CMSWire reported. For natural behavior within the scene, the split was 71% to 21%.

Runway also acknowledged open problems for Solaris in stable, legible text and long-session coherence, plus accessibility, since a generated interface has no DOM for screen readers to hook into.

The Diffusion-UI thread doesn't say how the new application handles those limits.

KEEP READING