NVIDIA's Spatial Intelligence Lab published ARDY, a framework that generates 3D human motion in real time from text prompts a user can change while the character is still moving. The lab released the code and model weights publicly rather than keeping the system behind a demo.
Streaming, not batch. ARDY synthesizes motion frame by frame while accepting new instructions mid-sequence, so a character can shift from a stealthy walk to a victory dance without stopping to re-render.
Two control paths. Direction comes from online text prompts and from kinematic constraints such as root trajectories, full-body keyframes, and end-effector positions.
Open release. The code is public on GitHub, the model weights are on Hugging Face, and the work publishes in ACM Transactions on Graphics.
Real-time generation steered by live text and sparse constraints
ARDY, short for Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation, targets applications that cannot wait for an offline render: animation, simulation, and humanoid robotics. "Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics," the researchers write.
The framework accepts two kinds of direction at once. Text prompts drive behaviors the team demonstrates, including a limp, a pick-and-put action, a stealthy walk, a victory dance, a lean-and-peek, and a zombie sit. Kinematic constraints handle spatial precision: root trajectories and waypoints, full-body keyframes, end-effector positions and rotations, or arbitrary combinations of those. The system also supports long-horizon goal reaching, where a constraint is specified beyond the current generation window, and real-time locomotion control through mouse waypoints and keyboard commands.
A hybrid representation and a two-stage denoiser
The method splits the motion into two representations. It combines explicit global root motion features with latent body embeddings, which the team frames as a way to balance trajectory control against generative efficiency. On top of that sits a two-stage autoregressive transformer denoiser that predicts the root motion first, then conditions the body-motion prediction on that root output.
Two additional pieces support the streaming behavior. A variable-length history context captures longer-term semantics to improve generation quality, and masked kinematic constraints allow spatiotemporally sparse conditioning that extends past the current window. The generative core follows the diffusion approach that has moved into character work elsewhere; we covered Motorica's raise to bring generative AI to character animation.
From on-screen characters to a physical humanoid
ARDY does not stop at rendered figures. The team integrated its output with a Unitree G1 humanoid robot through the SONIC physical tracking policy, using the generated motion to drive interactive robot control. That connection to hardware fits NVIDIA's broader robotics push; we covered the company's GTC keynote and its open humanoid robot dataset for training foundation models.
For production teams, the interactive framing separates ARDY from capture-first pipelines. Where markerless systems reconstruct motion from a performance, ARDY generates it from instructions; we covered Move AI's Gen 2 markerless capture as one point on that spectrum.
Published at SIGGRAPH with code and weights available
The paper appears in ACM Transactions on Graphics, Volume 45, Issue 4, Article 86, presented at SIGGRAPH, with DOI 10.1145/3811284. The authors are Kaifeng Zhao (NVIDIA, ETH Zürich), Mathis Petrovich (NVIDIA), Haotian Zhang (NVIDIA), Tingwu Wang (NVIDIA), Siyu Tang (ETH Zürich), and Davis Rempe (NVIDIA). The model weights are posted on Hugging Face alongside the code.
Because the code and weights are public, studios and robotics teams can test real-time, promptable motion generation against their own constraints instead of waiting for a product. The open release lets teams check how the models hold up on custom rigs and live control loops directly, rather than through the paper's demo clips.
