Generate long horizon predicted driving video from an input clip using a hierarchical world model
Steer generated future driving footage with a specific speed, yaw rate, or 2D trajectory
Reproduce the experiments and rollouts described in the Orbis 2 research paper
Study a two-tier autoregressive video prediction architecture for driving scenes
| lmb-freiburg/orbis2 | 0c33/agentic-ai | adennng/stock_strategy_lab | |
|---|---|---|---|
| Stars | 14 | 14 | 14 |
| Language | Python | Python | Python |
| Setup difficulty | hard | hard | hard |
| Complexity | 5/5 | 4/5 | 4/5 |
| Audience | researcher | developer | researcher |
Figures from each repo's GitHub metadata at analysis time.
Requires a GPU, a Conda environment, and downloading separate pretrained checkpoints from Hugging Face.
Orbis2 is the official code release for a research paper from the University of Freiburg that presents a driving world model, an AI system trained to generate realistic future video of what a car's camera would see next, based on past video frames it has already seen. It is designed for long stretches of predicted driving footage, not just a few frames ahead. The model works in two layers. A first component looks at video sampled at a low frame rate to build a rough, long range sense of what is likely to happen further into the future. A second component then uses that rough context together with recent high frame rate frames to generate the next frames one at a time, filling in fine detail as it goes. On top of this, the model can be steered: you can tell it to follow a specific driving speed and turning rate, or hand it a two dimensional path and have it generate video that follows that trajectory. Using it requires setting up a Python environment through Conda, then downloading pretrained model weights and configuration files from a separate Hugging Face repository. Once those pieces are in place, a single script rolls the model forward from an existing input video, optionally guided by a trajectory file that steers where the generated future frames go. A faster, distilled version of one of the two model components is available for quicker inference at some cost to quality, and several command line options control things like how many future steps to generate, which device handles decoding, and whether to compile the model for speed. This is a research artifact meant to accompany a published paper rather than a polished consumer tool: it assumes familiarity with running Python machine learning code, GPUs, and command line tools. It requires a GPU to run at a reasonable speed. All of the code in the repository is released under the MIT license, so it can be freely reused, modified, and built upon, including commercially.
The official research code for Orbis 2, an AI model that generates long, steerable future driving video from past camera frames.
Mainly Python. The stack also includes Python, PyTorch, Conda.
Use, modify, and redistribute the code freely, including commercially, as long as you keep the copyright notice.
Setup difficulty is rated hard, with roughly 1day+ to a first successful run.
Mainly researcher.
This repo across BitVibe Labs
double-check against the repo, no cap.