From Vision and Instructions to Robot Actions: Two Emerging Paths

一分钟结论

许多讨论把视觉—语言—动作模型（Vision-Language-Action, VLA）和世界—动作模型（World-Action Model, WAM）当作两个边界清楚的产品类别来比较。我认为这种框架太僵硬。两类系统都可以接收图像、语言指令、观测历史和机器人状态，最终也都必须输出可执行的机器人动作。真正更有用的问题，是这两个端点之间究竟发生了什么。

在收到观测与指令之后，模型在生成动作之前，主要被训练去理解或预测什么？

为了便于说明，我把现有方法概括为两条宽泛路线。第一条把机器人控制建立在视觉—语言模型学习到的语义表征之上。第二条引入视频预测或世界建模，让策略学习物理世界如何随时间变化。两条路线的最终输出可能相同，但传给动作生成器的表征，以及产生这种表征的学习目标，可能非常不同。

关于这套分类。 这两条路线是分析框架，不是互斥的模型家族。今天的系统越来越多地混合语义、时间和动作目标。只有当这套分类能让训练目标与推理路径更容易被检查时，它才有价值。

这张图按输入、中间目标和推理时计算来组织。选择图片可打开完整分辨率版本。

共同起点

两条路线通常都从某种组合开始：视觉观测、自然语言指令，以及机器人自身的状态。

视觉输入可以是一帧相机图像，也可以是一小段历史帧。指令描述任务，例如“把杯子放进水槽”。机器人状态则可能包含关节位置、夹爪状态、末端执行器位姿或其他本体感觉信息。

这些输入是数据，不是彼此独立的模型模块。这看似显而易见，却很重要：架构图常把“观测”和“语言”画在 VLM 或 WM 方框旁边，让它们看起来像独立的架构组件。实际上，它们只是提供给模型的信息。

输入进入系统之后，才出现第一个真正有意义的分类点：模型首先被训练去解决什么问题？

路线 A：先理解任务，再生成动作

第一条路线从预训练视觉—语言模型出发，把它扩展为视觉—语言—动作模型。

VLM 的主要目标，是把视觉与语言信息组织成具有语义用途的表征。给定一张图像和一条指令，它可以识别物体、理解关系、解释请求，并判断场景中的哪些部分与任务有关。

但在机器人系统里，VLM 并不一定先生成一句人类可读的文字，再把它交给独立控制器。大多数 VLA 系统里的动作模块，会直接读取内部隐藏状态或 token embedding。

这些隐藏表征编码了模型认为场景里有什么、指令是什么意思，以及应该追求哪个语义目标。它们是机器可读的表征，而不是普通语言。动作解码器或动作专家随后把这段上下文转换成动作块：机械臂、夹爪、移动底盘或其他执行器在短时间内要执行的一串连续指令。

路径： 观测与指令 → 语义隐藏表征 → 动作生成

π0 是代表性例子。它在预训练 VLM 之上接入一个基于 flow matching 的动作专家，让互联网规模的语义知识与连续动作生成在同一个紧耦合策略中相互作用。不能把它理解成两个互相独立的程序——一个先写出任务说明，另一个再控制机器人。

这条路线的优势很清楚。预训练 VLM 把语言理解、物体识别、场景知识、指令遵循和网络规模的语义先验带进机器人学习。机器人不必只靠昂贵的示范数据，从头学会每一个概念。

一个可能的限制是语义瓶颈。为物体识别和指令理解优化的表征，可能压缩掉操控所需的一部分精细时间与物理细节。它可以理解“需要拿起杯子”，却未必以同等保真度表征接触将怎样改变杯子位姿，或未来数秒内场景会如何演化。

这并不意味着 VLM 表征完全不包含空间或物理信息；更准确的说法是，它们的主导预训练目标通常是语义理解，而不是显式预测物理演化。

路线 B：先学习世界如何变化，再生成动作

第二条路线从视频模型或世界模型出发。它的核心目标不只是识别当前看见了什么，还要建模接下来可能发生什么：物体怎样运动，机器人怎样改变环境，接触怎样影响场景，以及视觉状态如何随时间演化。

路径： 观测与指令 → 未来世界表征 → 动作生成

“未来世界表征”并不总是指一段人可以直接观看的完整 RGB 视频。根据架构不同，它可以是显式解码的视频、一串视频 latent、内部预测视觉特征，或只在训练阶段出现而部署时被移除的目标。

这些方法的共同点不是某一种输出格式，而是把未来视觉动力学作为学习信号。相比高度压缩的语义表征，视频表征可以保留更稠密的空间与时间结构：不仅说明一个物体是什么，还描述它可能移动到哪里、机器人可以怎样与它交互，以及可能产生什么物理后果。

这正是世界—动作模型背后的动机。动作生成器不只依赖对当前场景的语义描述，还可以利用对潜在物理演化的表征。

代价同样重要。视频建模通常比紧凑的动作序列消耗更多 token、内存与计算量。显式生成未来视频还可能引入明显延迟，尤其当它需要迭代式 diffusion 或去噪时。

第二个分岔：推理时还保留哪些计算？

模型已经从未来视觉动力学中学到东西之后，在控制机器人时是否仍需生成未来，或显式计算对未来的估计？

这个问题把一些常被过度归为一类的设计区分开来。有些模型在推理时联合生成未来视觉状态与动作；另一些模型保留预测性视觉特征，但不解码完整视频；还有一类只在训练时使用视频预测，部署时移除迭代式未来生成。

DreamZero、VPP、Fast-WAM 与 GigaWorld-Policy 很好地展示了这种区别。

DreamZero：联合生成未来世界与动作

DreamZero 是这组比较中最显式的世界—动作建模例子。它建立在预训练视频 diffusion 骨干之上，联合建模未来视频与机器人动作。给定观测、历史、指令和相关机器人状态，模型在生成对应动作的同时，也预测视觉世界可能如何演化。

这不应被理解成严格串行的流水线：一个独立世界模型先做完视频，再把视频交给另一个独立控制器。视频流与动作流在生成过程中彼此耦合。未来视觉预测为物理动力学提供稠密监督，动作信息则把视觉演化与可执行行为连接起来。

用最直白的话说，DreamZero 一边想象未来画面，一边生成可能实现这一未来的动作。

这套设计很直观：如果模型能表征任务成功在时间上的样子，它或许能学到比只模仿动作标签的策略更丰富的感知—动力学—控制关系。挑战在于计算成本。联合生成视频和动作，需要充分的系统优化才能快到适合闭环控制。

VPP：预测视觉特征，但不画出完整视频

Video Prediction Policy 位于显式视频生成和 latent→动作推理之间。

VPP 把视频 diffusion 模型用作预测视觉表征的编码器。这些内部特征既包含当前场景信息，也包含对未来演化的结构化估计。Video Former 压缩高维特征，随后由 diffusion policy 生成机器人动作。

关键技术点是：主控制路径只需让视频模型前向一次，得到单步预测表征；它不需要执行为了渲染精致、可观看的未来视频而使用的完整多步去噪流程。30 步视频可用于可视化，却不是部署策略所必需的表征。

因此，把 VPP 描述成“先生成完整视频，再从视频推断动作”并不准确。更好的说法是：VPP 构造一个面向未来的视觉表征，并让动作策略以此为条件。

用最直白的话说，VPP 预测未来，但不把未来完整画出来。

Fast-WAM：从未来预测中学习，但不在部署时展开未来

Fast-WAM 追问：世界—动作模型的主要价值，来自推理时显式想象，还是来自训练阶段视频建模所塑造的表征？

它的受控比较表明，大部分价值可能来自训练目标。Fast-WAM 保留视频协同训练：未来视频 latent 与动作块同时提供学习信号，把视频骨干塑造成具有物理信息的世界编码器。

但在部署时，Fast-WAM 会移除未来视频分支，也移除对未来帧进行迭代去噪的过程。它并没有取消所有面向未来的计算。干净观测 token 仍会通过一次预训练视频 DiT——此时它作为单次前向的世界编码器——产生 latent 世界表征，再以此为条件驱动动作专家。

路径： 当前观测 → 单次前向的 latent 世界表征 → 机器人动作

最简洁的描述是：训练时联合建模视频，测试时从 latent 直接推动作。 模型从未来预测中学习，却不会在控制机器人时展开未来视频。

这项设计试图保留世界建模带来的表征收益，同时移除其最大部署成本之一：测试时的迭代式未来生成。

GigaWorld-Policy：让动作在因果上独立于未来视频

GigaWorld-Policy 通过另一种架构选择实现只生成动作的推理。

训练时，它从当前观测预测未来动作；与此同时，它学习以观测和预测动作共同为条件生成未来视频。未来视觉动力学由此提供稠密物理监督，并鼓励模型产生与合理场景演化一致的动作。

因果注意力掩码阻止未来视频 token 影响动作 token。依赖只沿一个方向流动：预测动作可以作为未来视频生成的条件，但动作预测本身不依赖未来视频 token。因此，推理时可以关闭视频分支，而不改变动作所使用的因果路径。

路径： 当前观测 → 仅解码动作

GigaWorld-Policy 在训练时仍从世界建模中获益，但它把动作生成视为中心任务，把未来视频视为辅助物理监督。它的策略可以概括为：训练时利用视觉动力学，部署时只解码动作。

Fast-WAM 和 GigaWorld-Policy 都避免在推理时生成未来视频，但两者逻辑并不相同。Fast-WAM 强调视频协同训练的经验价值，并在推理时保留单次前向的世界编码器；GigaWorld-Policy 则把动作分支与视频分支之间的因果独立直接写进架构。

这组比较真正说明了什么

VLA 与 WAM 的区别，不能被简化成输入或最终输出不同。两者都可能接收图像、语言、观测历史和机器人状态，也都最终产生机器人动作。

更深层的区别在于：模型被训练去预测什么，哪一种表征到达动作生成器，以及部署时哪些计算仍然存在。

VLM→VLA 路线主要围绕语义理解组织：眼前有什么、指令是什么意思、机器人应该做什么。视频／世界模型路线则围绕时间与物理演化组织：接下来可能发生什么、场景会怎样变化，以及一个动作是否与这种变化相一致。

基于 VLM 的系统继承了强语义先验、语言 grounding 与互联网规模知识；它们的表征也可以更紧凑，更容易进行高频部署。基于世界模型的系统引入更稠密的时间与空间监督，可能改善对运动、交互、接触和物理后果的表征。代价则是更高的计算与内存成本，尤其当系统在推理时显式生成未来视觉状态时。

因此，前沿问题不是在 VLA 和 WAM 之间二选一。更有用的问题是：需要多少世界建模，它应在学习过程的哪里进入，以及部署时是否必须继续运行。

DreamZero 在推理中保留未来视频与动作的联合生成；VPP 提取单步预测视觉特征，但不完整解码未来视频；Fast-WAM 在训练时使用视频建模，推理时依赖单次前向的世界编码器；GigaWorld-Policy 则让动作生成在因果上独立于可选的未来视频分支。

连续谱： 显式未来生成 → 预测视觉表征 → 训练时世界建模 → 仅动作推理

相关术语仍未稳定。“World-Action Model” 是正在形成的标签，并非普遍标准化的架构类别；VLA、视频策略、世界模型与 WAM 之间的边界也越来越模糊。

因此，我不会只凭一篇论文使用的标签来评价新的机器人基础模型。我会追问三个具体问题：

模型被训练去预测什么？
什么表征到达动作生成器？
机器人运行时，哪些计算仍然活跃？

这三个问题解释了图中分类背后的逻辑，而且即使名称继续变化，它们依然有用。

一手资料

技术描述依据截至 2026 年 7 月 29 日可获得的以下论文版本核对。

One-minute takeaway

Most discussions compare Vision-Language-Action models and World-Action Models as though they were two cleanly separated product categories. I find that framing too rigid. Both kinds of systems can receive images, language instructions, observation histories, and robot states. Both ultimately need to produce executable robot actions. The more useful question is what happens between those two endpoints.

After receiving an observation and an instruction, what is the model primarily trained to understand or predict before it generates an action?

For exposition, I use two broad routes. The first builds robot control on the semantic representations learned by a Vision-Language Model. The second introduces video prediction or world modeling so that the policy can learn how the physical world changes over time. The final output may be the same, but the representation passed to the action generator—and the learning objective that produces it—can be very different.

A note on the taxonomy. These are analytical routes, not mutually exclusive model families. Contemporary systems increasingly hybridize semantic, temporal, and action objectives. The classification is useful only if it makes the training target and inference path easier to inspect.

The diagram is organized around inputs, intermediate objectives, and inference-time computation. Select it to open the full-resolution version.

The common starting point

Both routes usually begin with some combination of visual observations, a natural-language instruction, and the robot’s own state.

The visual input may be a single camera frame or a short history of frames. The instruction describes the task, such as “place the cup in the sink.” The robot state may include joint positions, gripper state, end-effector pose, or other proprioceptive information.

These inputs are data, not separate model modules. This sounds obvious, but it matters because diagrams often place “observation” and “language” next to VLM or WM boxes in ways that make them look like architectural components. They are simply the information supplied to the model.

The first meaningful classification point comes after these inputs enter the system: what problem is the model trained to solve first?

Route A: understanding the task before generating the action

The first route begins with a pretrained Vision-Language Model and extends it into a Vision-Language-Action model.

A VLM is primarily designed to organize visual and linguistic information into a semantically useful representation. Given an image and an instruction, it can identify objects, interpret relationships, understand the request, and determine which parts of the scene are relevant to the task.

In a robotic system, however, the VLM does not necessarily generate a human-readable sentence and hand it to a separate controller. In most VLA systems, the action module reads internal hidden states or token embeddings directly.

These hidden representations encode what the model believes is present, what the instruction means, and which semantic goal should be pursued. They are machine-readable representations, not ordinary language. The action decoder or action expert then converts this context into an action chunk: a short sequence of continuous commands for the arm, gripper, mobile base, or other actuators.

Route: Observation and instruction → semantic hidden representation → action generation

π0 is a representative example. It places a flow-matching action expert on top of a pretrained VLM so that internet-scale semantic knowledge and continuous action generation interact inside one tightly coupled policy. It should not be read as two independent programs—one that writes a task description and another that controls the robot.

The advantage is clear. A pretrained VLM brings language understanding, object recognition, scene knowledge, instruction following, and web-scale semantic priors into robot learning. The robot does not need to learn every concept exclusively from expensive demonstrations.

A possible limitation is a semantic bottleneck. A representation optimized for recognizing objects and understanding instructions may compress some of the fine-grained temporal and physical details required for manipulation. It may understand that a cup must be picked up without representing, with equal fidelity, how contact will change the cup’s pose or how the scene will evolve over the next several seconds.

This does not mean that VLM representations contain no spatial or physical information. It means that their dominant pretraining objective is usually semantic understanding rather than explicit prediction of physical evolution.

Route B: learning how the world changes before generating the action

The second route begins with a video model or world model. Its central objective is not simply to recognize what is currently visible. It attempts to model what may happen next: how objects move, how the robot changes the environment, how contact affects the scene, and how visual states evolve over time.

Route: Observation and instruction → future-world representation → action generation

“Future-world representation” does not always mean a complete RGB video that a person can watch. Depending on the architecture, it may be an explicitly decoded video, a sequence of video latents, internal predictive visual features, or a training-only target that is absent during deployment.

What these approaches share is not a single output format. They use future visual dynamics as a learning signal. Compared with highly compressed semantic representations, video-based representations can retain denser spatial and temporal structure: not only what an object is, but where it may move, how the robot could interact with it, and what physical consequences may follow.

This is the motivation behind World-Action Models. The action generator can benefit from a representation of possible physical evolution, rather than only a semantic account of the present scene.

The cost is equally important. Video modeling generally involves more tokens, memory, and computation than a compact action sequence. Explicit future-video generation can introduce substantial latency, especially when it requires iterative diffusion or denoising.

The second split: what remains active during inference?

Once a model has learned from future visual dynamics, does it still need to generate—or explicitly compute—the future while controlling the robot?

This question separates designs that are often grouped together too loosely. Some models jointly generate future visual states and actions at inference time. Others retain predictive visual features without decoding a complete video. A third group uses video prediction during training but removes iterative future generation during deployment.

DreamZero, VPP, Fast-WAM, and GigaWorld-Policy illustrate the distinction particularly well.

DreamZero: jointly generating the future world and the action

DreamZero is the most explicit World-Action Modeling example in this comparison. Built on a pretrained video diffusion backbone, it jointly models future video and robot actions. Given an observation, history, instruction, and relevant robot state, the model predicts how the visual world may evolve while generating the associated actions.

This should not be interpreted as a strict serial pipeline in which a standalone world model finishes a video and then hands it to an independent controller. The video and action streams are coupled inside the generative process. Future visual prediction provides dense supervision for physical dynamics, while action information connects visual evolution to executable behavior.

In plain language, DreamZero imagines future frames while producing actions that could bring about that future.

The formulation is intuitive: if the model can represent what successful task completion looks like over time, it may learn a richer connection between perception, dynamics, and control than a policy trained only to imitate action labels. The challenge is computational. Joint video-action generation requires systems optimization to run fast enough for closed-loop control.

VPP: predictive visual features without drawing the full video

Video Prediction Policy occupies a middle position between explicit video generation and latent-to-action inference.

VPP uses a video diffusion model as an encoder of predictive visual representations. These internal features contain current scene information and a structured estimate of future evolution. A Video Former compresses the high-dimensional features, after which a diffusion policy generates robot actions.

The important technical point is that the main control path uses one forward pass through the video model to obtain a single-step predictive representation. It does not require the complete multi-step denoising process used to render a polished, human-viewable future video. Thirty-step videos are useful for visualization, but they are not the representation required by the deployed policy.

Describing VPP as “generate a complete video first, then infer the action from it” would therefore be inaccurate. A better description is that VPP constructs a future-oriented visual representation and conditions the action policy on that representation.

In plain language, VPP predicts the future, but it does not fully draw it.

Fast-WAM: learning from future prediction without rolling it out

Fast-WAM asks whether the main value of a World-Action Model comes from explicit imagination at inference time or from the representations learned through video modeling during training.

Its controlled comparisons suggest that much of the value comes from the training objective. Fast-WAM retains video co-training: future video latents and action chunks both provide learning signals, shaping the video backbone into a physically informed world encoder.

At deployment, however, Fast-WAM removes the future-video branch and the iterative process of denoising future frames. It does not eliminate all future-aware computation. The clean observation tokens still pass once through the pretrained video DiT, now used as a single-pass world encoder, and the resulting latent world representation conditions the action expert.

Route: Current observation → single-pass latent world representation → robot action

The concise description is: training-time video co-modeling, test-time latent-to-action inference. The model learns from future prediction without rolling out future video while controlling the robot.

This design aims to preserve the representational benefit of world modeling while removing one of its largest deployment costs: iterative test-time future generation.

GigaWorld-Policy: making actions causally independent of future video

GigaWorld-Policy reaches action-only inference through a different architectural choice.

During training, it predicts future actions from the current observation. In parallel, it learns to generate future video conditioned on both the observation and the predicted actions. Future visual dynamics therefore supply dense physical supervision and encourage actions that are consistent with plausible scene evolution.

A causal attention mask prevents future-video tokens from influencing action tokens. The dependency runs in one direction: predicted actions can condition future-video generation, but action prediction itself does not depend on future-video tokens. The video branch can therefore be disabled at inference without changing the causal path used for actions.

Route: Current observation → action-only decoding

GigaWorld-Policy still benefits from world modeling during training, but it treats action generation as the central task and future video as an auxiliary source of physical supervision. Its strategy can be summarized as: train with visual dynamics; deploy with action-only decoding.

Although Fast-WAM and GigaWorld-Policy both avoid future-video generation at inference time, their logic is not identical. Fast-WAM emphasizes the empirical value of video co-training and retains a single-pass world encoder. GigaWorld-Policy builds causal independence between the action and video branches into the architecture itself.

What the comparison is really showing

The distinction between VLA and WAM should not be reduced to different inputs or final outputs. Both may consume images, language, observation histories, and robot states. Both eventually produce robot actions.

The deeper difference lies in what the model is trained to predict, what representation reaches the action generator, and which computations remain active at deployment.

The VLM-to-VLA route is organized primarily around semantic understanding: what is present, what the instruction means, and what the robot should do. The video/world-model route is organized around temporal and physical evolution: what may happen next, how the scene could change, and whether an action is consistent with that change.

VLM-based systems inherit strong semantic priors, language grounding, and internet-scale knowledge. Their representations can also be more compact, making high-frequency deployment easier. World-model-based systems introduce denser temporal and spatial supervision, potentially improving representations of motion, interaction, contact, and physical consequences. The trade-off is higher computation and memory cost, especially when future visual states are explicitly generated at inference time.

The frontier is therefore not a binary choice between VLA and WAM. The more useful question is how much world modeling is necessary, where it should enter the learning process, and whether it must remain active during deployment.

DreamZero keeps joint future-video and action generation inside inference. VPP extracts one-step predictive visual features without fully decoding a future video. Fast-WAM uses video modeling during training, then relies on a single-pass world encoder at inference. GigaWorld-Policy makes action generation causally independent of its optional future-video branch.

Spectrum: Explicit future generation → predictive visual representation → training-time world modeling → action-only inference

The terminology remains unsettled. “World-Action Model” is an emerging label rather than a universally standardized architecture category, and the boundaries among VLA, video policy, world model, and WAM are increasingly blurred.

For that reason, I would not evaluate a new robot foundation model only by the label used in its paper. I would ask three concrete questions:

What is the model trained to predict?
What representation reaches the action generator?
Which computations remain active when the robot is running?

Those three questions explain the classification logic behind the diagram—and remain useful even as the names change.

Primary sources

Technical descriptions were reviewed against the following paper versions available on July 29, 2026.

从视觉与指令到机器人动作：两条正在形成的路径From Vision and Instructions to Robot Actions: Two Emerging Paths