A Sense of Space and Time: Optimizing Interactive Video Generation
This blog series explores how biological neural networks can be integrated into modern AI systems. The posts progress from practical applications to increasingly general principles, moving from perception to representation, dynamics, and ultimately algorithm discovery. In this post, we extend the ideas behind the Post 3 video adapter into a new adapter shaped by our work with living neurons, running inside an interactive, playable video model that generates in real time. We show how biological structure lets the adapter preserve quality and long-horizon stability throughout aggressive serving optimization.
Note: This is a living document, and we are building in the open. We expect frequent updates and actively welcome feedback from the community.
Key Takeaway
For the first time, we show that a small, biologically derived adapter keeps a fast, interactive video generation model sharp and stable while it is played in real time, staying coherent for longer than the base model. Building on our prior work keeping generated video stable over long runs, this brings our adapter into a live, playable system. This matters for games, simulation, robotics training, and virtual worlds, where responsiveness and stability must coexist. Structure drawn from biology lets us push hard for speed without compromising on quality, pointing to high-quality, real-time video at much lower cost.
Key Result
With the Hypercolumn Neural Optimizer running, the TBC-optimized model generates video several times faster while keeping the video quality sharp and stable for much longer periods. Without the optimizer to offset inference speed improvements, the base model runs slowly, and the video drifts and deteriorates as the clip goes on. You can compare the optimized and base models side by side in the live demo below.
Play the live demo: tbc.co/demo
Call To Action
This proof of concept shows that a small, biologically derived adapter delivers real gains in a compact interactive video model, in both quality and speed, and holds those gains under real-time serving optimization. Next, we are applying the same approach to a production text-to-video model, one of the leading open models for creative video. Creators, marketers, filmmakers, game studios, and developers will soon be able to access TBC-optimized video models through APIs and cloud marketplaces. Sign up to be among the first to test our text-to-video product.
Introduction
Interactive video generation models face a persistent tension between quality, speed, and stability. High-fidelity generation is possible when a model is allowed to compute extensively, but interactive applications such as games, simulations, and robotics training must produce each frame in real time and in direct response to a user's actions. Autoregressive video models add a further difficulty: as a rollout extends, small errors accumulate, internal representations lose structure, and outputs drift, forcing a tradeoff between how long a scene remains coherent and how quickly it can be produced.
In Post 3 we showed that a small, biologically based adapter improves long-horizon stability in an open-source video model, measured offline on saved checkpoints. In this post we take the next step. We extend the result to a live, interactive model, and we do it with a new adapter that adds both spatial and temporal structure drawn from biology. Its two ingredients mirror the brain’s visual cortex: local spatial mixing, like the local receptive fields of early visual areas, and a per-location memory over time, like the columns that integrate a patch of space across successive moments. We make the system interactive and real time, measure the deployed path end to end, and describe both the adapter that makes it work and the biology it comes from.
AI Approach
The base model is the open-source Oasis Minecraft video generation model. A Vision-Transformer autoencoder compresses each frame into a compact latent. A causal Diffusion Transformer then takes that latent and the previous latents within a history window and predicts the next frame from the player's action. Frames are generated one at a time. The base model is an excellent real-time per-frame generator, but for long, interactive rollouts it has two structural limitations. Its spatial attention within a frame is all-to-all, with no built-in preference for local structure, so the model has to learn spatial relationships across a large number of pixels. Its temporal attention treats every past frame alike, which is computationally expensive and error prone, so video quality drifts as a rollout extends.
Both limitations have a natural reference in the visual cortex. Early visual areas do not have the all-to-all connections of a silicon spatiotemporal attention mechanism. The visual cortex processes a scene through local receptive fields that expand in space and time as it performs successive transforms. These transforms extract lines from pixels, objects from lines, and moving scenes from objects. Global structure is built up from local operations rather than by densely connecting all spatial and temporal information at once, the way silicon architectures typically do. Natural scenes are dominated by local correlations, which the visual cortex has learned to exploit. The cortex is organized into columns, each integrating a local patch of space across time. Two design lessons follow: mix information locally in space and let depth carry the long range, and give each spatial location its own memory over time.
Our Hypercolumn Neural Optimizer instantiates these lessons directly. The hypercolumn adapter runs as a small branch in parallel with the base model and factors computation at every spatial location into two parts: a local convolution that mixes space, and a per-location causal attention with a fixed-size memory that carries time. Spatial mixing happens only through the convolution and temporal coupling only through the attention, mirroring the way a cortical column separates a local patch of space from its integration over time. The temporal context narrows with depth, from a long window reaching into the distant past in the early layers to short windows covering only adjacent frames in the later ones, so the adapter holds long-range context where it helps and precise frame-to-frame transitions where they matter (Figure 1). The adapter joins the base as a residual branch: its output is added back into the base at each block, and it starts with almost no effect on the base model's output, so training moves the system gently away from the base rather than disrupting it. We keep the base weights frozen and train the branch on a large video corpus, then fine-tune the base and adapter jointly. Training the base alone, even for far longer, did not improve quality; only joint training did, which is what shows the gains come from the adapter rather than added base capacity. In total the adapter adds roughly 14 million parameters, about 3% of the base model, and supplies exactly the local-space and structured-time behavior the base lacks.
Figure 1. Architecture of the hypercolumn adapter. At each spatial location a local convolution mixes space while a per-location causal attention with a fixed-size cache carries time. Context windows narrow with depth, from long-range context in the early layers to temporally adjacent frames in the later ones.
Bio Approach
The visual cortex shaped the overall design of the adapter. Our work with living neurons shaped its two key ingredients: a space constant and a time constant. Read through its interpretable parts, the hypercolumn adapter behaves like a discretized neural field: its spatial pathway is a convolution that acts as a lateral-interaction kernel with a fixed extent, and its temporal pathway carries memory over time. Spatially, every layer's free k × k convolution is replaced by a distance-parameterized difference-of-Gaussians kernel with a space constant λ, the canonical spatial descriptor in neural field theory. Temporally, we add a leaky recurrent excitatory/inhibitory layer after the convolution stack that decays with a time constant τ. A space constant and a time constant are exactly the quantities that describe how activity spreads and decays in living neural tissue (Figure 2).
This marks a progression across the TBC blog series. The adapter in Post 2 drew on neural connectivity patterns to capture spatial structure and sharpen image reconstruction, and Post 3 extended it with neural dynamics to hold video together over long rollouts. The hypercolumn adapter carries both, a space constant and a time constant, the spatial and temporal parameters that also characterize living tissue.
We characterize both directly on our platform. Using living neurons cultured on high-density multi-electrode arrays, we stimulate a single site and record how evoked activity spreads across the array and decays over time. Fitting the spatial spread gives a space constant (Figure 3), and fitting the post-stimulus time course gives a time constant (Figure 4). These are measured properties of living networks, not analogies applied after the fact, and, as in earlier posts, they carry substantial trial-to-trial variability. The hypercolumn adapter is built around these same two quantities, guided by the structure the measurements reveal.
Figure 2. A space constant and a time constant. The spatial spread and temporal decay of evoked activity in living tissue are captured by a space constant λ and a time constant τ. In the neural-field view of the adapter, a space constant sets the width of a difference-of-Gaussians convolution kernel and a time constant sets the decay of a leaky recurrent layer; long-range content mixing stays as standard attention.
Figure 3. Measuring spatial decay. (A) The MEA-derived difference-of-Gaussians (DoG) kernel across connectivity orders and the per-plate distribution of the fitted space constant across cultures. (B) The difference-of-Gaussians form a neural-field model uses for spatial mixing, shown against the MEA-measured profile.
Figure 4. Measuring temporal decay. (A) Normalized MEA response as a function of time after stimulation across stimulation amplitudes, with the fitted time constant, and the distribution of that constant across sites. (B) The leaky-decay form a neural-field model uses for temporal memory, shown against the MEA-measured profile.
Results
We evaluated our hypercolumn adapter on long-horizon rollouts using the open-source Oasis Minecraft video generation model. Across 200 randomly-seeded action sequences of 600 frames each, we compared the base model and the optimized model against a real-gameplay reference, measuring five complementary metrics at the ten-second point of each rollout. These cover how close the generated video stays to the distribution of real video, how much visual detail it retains over time, and the standard perceptual dimensions of subject consistency, aesthetic quality, and imaging quality.
The optimized model improves on the unoptimized base on every metric (Figure 5). It produces sharper, more stable rollouts, retains substantially more visual detail over time, and sits markedly closer to real gameplay, while the base collapses fastest on every axis. As autoregressive video models roll out, they add more and more noise to each successive frame and quality declines. This happens quickly with the base model, within about five seconds, but is largely curbed in the adapted model. Training the base alone, even for much longer, did not produce these gains; only adding and jointly training the adapter did, so the improvement comes from the adapter, which adds under 3% more parameters, not from added base capacity. The speed gains come from the efficiency of that adapter together with serving optimizations tailored to its architecture.
Figure 5. Quality across the full 600-frame rollout. Top: per-frame mean with ±SEM shading for five evaluation metrics (dispersion, Fréchet Video Distance (FVD), subject consistency, aesthetic quality, imaging quality). Real video stays flat, the base model degrades fastest and keeps falling across the rollout, and both optimized configurations track far closer to real video throughout. Bottom: example frames at 0.5, 2.5, 5, 10, 20, and 30 seconds, showing ground truth (top), the Oasis base model (middle), and the hypercolumn model (bottom). The base darkens and drifts within a few seconds, while the hypercolumn model stays sharp and close to ground truth across the whole clip.
Real-Time Serving
Turning this into something a person can play is an engineering problem as much as a modeling one. The adapter runs as a parallel branch alongside the base model, inside the existing encoder and decoder, and we rebuilt the runtime around the combined model into an interactive, multi-session serving system (Figure 6). Rather than rendering a fixed clip in a single pass, the system generates one frame at a time in response to a live action, the same way a game engine would.
Figure 6. The Hypercolumn in the serving pipeline. The adapter runs as a parallel branch alongside the base model, inside the existing encoder and decoder, and a residual combination merges the two before decoding each frame.
On a single stream, this takes the model from roughly two frames per second in its original serving configuration to nearly ten in the deployed configuration, a 4.4-fold improvement despite under 3% more parameters (Figure 7). Because per-frame cost is dominated by the number of sampling steps, we chose the deployed configuration to sit at the point where quality is fully preserved and speed is highest.
Figure 7. Real-time serving throughput. Single-stream frame rate on the same model weights across four sampler and step-count configurations. The deployed configuration reaches 4.4 times the throughput of the original serving path. DDIM and DPM++ are diffusion samplers, the algorithms that turn noise into each frame over a set number of steps; DPM++ is a higher-order sampler that reaches the same quality in fewer steps.
Optimizing for Speed Without Losing Quality
The speed comes from serving the same model more efficiently, not from a different or smaller one. We combined a streaming cache that avoids recomputing shared context on every step, a higher-order sampler that reaches the same quality in fewer steps, and multi-session serving across GPUs.
What matters here is the relationship between optimization and quality. Optimizing an autoregressive video model for speed normally costs quality. Fewer sampling steps and faster schedules tend to produce visibly worse frames. With the adapter in place, that tradeoff largely disappears (Figure 8). At six sampling steps the optimized system matches the quality of a ten-step configuration while running roughly 1.5 times faster, and the fully optimized system, at nearly ten frames per second, still improves on the unoptimized base across all five metrics. In effect, the biologically based structure absorbs the quality cost that aggressive optimization would otherwise impose.
Figure 8. Quality holds as the system is optimized for speed. Moving from left to right the serving path becomes faster. Distributional distance to real video stays low and retained visual information stays high across the optimized configurations, rather than degrading as speed increases.
Limitations and Future Directions
This work is an early step toward serving biologically based video generation models in real time, and several questions remain open. Our current evaluation focuses on a single Minecraft environment. Confirming that these gains generalize across environments, action distributions, and inference regimes, particularly at the low step counts favored in production, will be important. On the biology, we are continuing to characterize how the measured space and time constants vary across cultures and stimulation conditions. We are also quantifying the effect on energy demand, including whether the adapter enables comparable rollout quality at reduced model size or compute, which would translate the speed gains reported here into direct reductions in the cost of interactive generation.
Conclusions
We show that a compact, biologically based adapter, trained jointly with the base model and served in real time, runs an open-source video generation model several times faster than its original serving path while improving quality across long rollouts, adding only about 3% to the base model's parameters. Two constants govern the adapter, one for how far it mixes across space and one for how long it holds memory, the same quantities that describe how activity spreads and fades in living tissue. Rather than trading quality for speed, the optimized model holds quality as speed increases. Next, we apply the same approach to text-to-video and larger models. This moves biological computation from offline demonstration toward deployed, interactive systems, and points to a path for high-quality, long-horizon generation that is both real time and cheaper to run.
