dreamforge
world-model imagination → corrective training data for robot policies
mining episodes
150
base policy rollouts, fixed seeds
failure states
360
decision points mined
base policy
61.0%
10-episode protocol (paired)
imagined vs none
+4.2pp
exact McNemar p = 0.032
imagined vs random
-1.1pp
equal budget · p = 0.608
01Pipeline
six stages · every claim is measurable in the real simulator
stage 1
mine
roll out the base VLA, record every decision point with its exact simulator state
→
stage 2
imagine
CEM over a latent world model, 0.5σ trust region, seeded by the policy's own chunk
→
stage 3
verify
paired replay of the saved state: none / imagined / random, then hand back to the policy
→
stage 4
build
correction datasets (all / gained / lifted) mixed with expert replay
→
stage 5
train
SmolVLA fine-tune, flow matching, 16 GB single GPU
→
stage 6
eval
paired closed-loop evaluation with Wilson intervals and exact McNemar
02World-model pre-flight
Imagination is only meaningful if the model responds to actions. This diagnostic is shipped as a CLI command and is run before any experiment.
| world model | score std (random plans) | action-swap Δprogress | latent swap | mean |z0| | verdict |
|---|---|---|---|---|---|
base | 0.0003 | 0.0002 | 0.41 | 34 | ignores actions |
aso | 0.0163 | 0.0101 | 17.88 | 11 | used |
full | 0.0000 | 0.0003 | 14.56 | 2961 | ignores actions |
An unregularized checkpoint scores ≈0 on both response metrics — it ignores the action entirely. CEM cannot optimize such a model, so results would be meaningless.
03What imagined corrections are worth (real simulator)
Each mined state is restored exactly and replayed with three arms, then the unchanged policy resumes. The none arm reproduces the original episode, which anchors the comparison.
Imagined progress along the CEM plan
world-model read-out · mean ± sd over mined failure states (not simulator success)
What this is: the world model rolls a candidate plan forward 50 steps in latent space (no simulator); a small head then predicts episode-time progress p∈[0,1] from each imagined latent. The head is trained on demonstration frames labelled by their normalised time inside the episode, so p is a time proxy for task progress. CEM maximises the mean of the last 3 imagined steps. Read-outs like this correlate only weakly with real outcomes (+0.09), which is the finding this repository is about.
Arms
| arm | n | success | rate | 95% CI |
|---|---|---|---|---|
| none | 360 | 34 | 9.4% | [6.8, 12.9] |
| imagined | 360 | 49 | 13.6% | [10.5, 17.5] |
| random | 360 | 53 | 14.7% | [11.4, 18.8] |
Paired tests (exact McNemar)
| comparison | delta | improve/worsen | p |
|---|---|---|---|
| imagined vs none | +4.2pp | 29 / 14 | 0.032 |
| imagined vs random | -1.1pp | 15 / 19 | 0.608 |
By decision step
| bucket | n | imagined | none | delta |
|---|---|---|---|---|
| start | 60 | 32% | 17% | +15pp |
| early | 120 | 23% | 18% | +6pp |
| late | 180 | 1% | 2% | -1pp |
04Does training on imagined corrections help?
Fine-tuned policies are evaluated on the same suite and seeds as the base policy (paired by task and episode).
Closed-loop success
| policy | n | success | 95% CI |
|---|---|---|---|
smolvla_libero | 100 | 61.0% | [51.2, 70.0] |
ft_all | 100 | 57.0% | [47.2, 66.3] |
ft_early | 100 | 45.0% | [35.6, 54.8] |
ft_replay | 100 | 58.0% | [48.2, 67.2] |
Paired tests (exact McNemar)
| comparison | delta | W/L | p |
|---|---|---|---|
| smolvla_libero vs ft_all | +4.0pp | 11 / 7 | 0.481 |
| smolvla_libero vs ft_early | +16.0pp | 22 / 6 | 0.004 |
| smolvla_libero vs ft_replay | +3.0pp | 11 / 8 | 0.648 |
| ft_all vs ft_early | +12.0pp | 16 / 4 | 0.012 |
| ft_all vs ft_replay | -1.0pp | 12 / 13 | 1.000 |
| ft_early vs ft_replay | -13.0pp | 11 / 24 | 0.041 |
⚑Verdict
Verdict. imagined corrections change outcomes (+4.2pp vs doing nothing), but they are indistinguishable from an equal-budget random perturbation (-1.1pp, p = 0.61) — the model's direction is not where the value is. Training on the corrections (ft_all 57.0%) matches a replay-only control (ft_replay 58.0%), and concentrating corrections where the decision-point metric looked best is actively harmful (ft_early 45.0% vs smolvla_libero 61.0%). Offline value at a state does not tell you what to teach there.
dreamforge
世界模型想象 → 机器人策略的纠错训练数据
采矿回合
150
基线策略 rollout，固定随机种子
失败状态
360
挖掘到的决策点
基线策略
61.0%
标准 10 回合协议（配对）
想象 vs 不纠正
+4.2pp
exact McNemar p = 0.032
想象 vs 等预算随机
-1.1pp
同尺度扰动 · p = 0.608
01管线
六个阶段 · 每一条结论都能在真实仿真器里测量
阶段 1
mine
跑基线 VLA，记录每个决策点及其精确仿真状态
→
阶段 2
imagine
在潜空间世界模型上做 CEM，按维度 0.5σ 信任域，以策略自身动作为种子
→
阶段 3
verify
同状态三臂配对回放：none / imagined / random，随后交回原策略
→
阶段 4
build
构造纠错数据集（all / gained / lifted），与专家回放混合
→
阶段 5
train
微调 SmolVLA（flow matching），单卡 16GB
→
阶段 6
eval
配对闭环评测：Wilson 区间 + exact McNemar
02世界模型预检
只有模型对动作有响应，想象才有意义。该诊断已工具化，实验前必跑。
| 世界模型 | 随机计划分数标准差 | 动作交换 Δ进度 | 潜状态位移 | 平均 |z0| | 判定 |
|---|---|---|---|---|---|
base | 0.0003 | 0.0002 | 0.41 | 34 | 忽略动作 |
aso | 0.0163 | 0.0101 | 17.88 | 11 | 采用 |
full | 0.0000 | 0.0003 | 14.56 | 2961 | 忽略动作 |
未正则化的检查点在两个响应指标上均≈0——它完全忽略动作。CEM 无法优化这样的模型，任何结论都会失去意义。
03想象纠错的真实价值（真实仿真器）
精确恢复每个挖掘状态并执行三臂回放，随后交回未修改的策略。none 臂复现原始回合，作为配对比较的锚点。
想象进度随 CEM 计划的变化
世界模型读出值 · 在挖掘到的失败状态上（均值 ± 标准差）· 并非仿真器判定的成功
这是什么：世界模型在潜空间把候选计划前滚 50 步（不进仿真器），再由一个小预测头从每个“想象出的”潜状态读出 回合时间进度 p∈[0,1]。预测头的训练标签是演示帧在回合内的归一化时间占比，所以 p 是任务进度的时间代理量。CEM 最大化想象最后 3 步的均值。这类读出值与真实结局的相关性很弱（+0.09）——这正是本仓库要说明的问题。
各组
| 组 | n | 成功 | 成功率 | 95% CI |
|---|---|---|---|---|
| none | 360 | 34 | 9.4% | [6.8, 12.9] |
| imagined | 360 | 49 | 13.6% | [10.5, 17.5] |
| random | 360 | 53 | 14.7% | [11.4, 18.8] |
配对检验（exact McNemar）
| 对比 | 差值 | 改善/变差 | p |
|---|---|---|---|
| imagined vs none | +4.2pp | 29 / 14 | 0.032 |
| imagined vs random | -1.1pp | 15 / 19 | 0.608 |
按决策步分层
| 区间 | n | imagined | none | 差值 |
|---|---|---|---|---|
| 开局 | 60 | 32% | 17% | +15pp |
| 早期 | 120 | 23% | 18% | +6pp |
| 晚期 | 180 | 1% | 2% | -1pp |
04用想象纠错训练是否有用？
微调后的策略在与基线相同的任务与随机种子上评测（按任务+回合配对）。
闭环成功率
| 策略 | n | 成功 | 95% CI |
|---|---|---|---|
smolvla_libero | 100 | 61.0% | [51.2, 70.0] |
ft_all | 100 | 57.0% | [47.2, 66.3] |
ft_early | 100 | 45.0% | [35.6, 54.8] |
ft_replay | 100 | 58.0% | [48.2, 67.2] |
配对检验（exact McNemar）
| 对比 | 差值 | 胜/负 | p |
|---|---|---|---|
| smolvla_libero vs ft_all | +4.0pp | 11 / 7 | 0.481 |
| smolvla_libero vs ft_early | +16.0pp | 22 / 6 | 0.004 |
| smolvla_libero vs ft_replay | +3.0pp | 11 / 8 | 0.648 |
| ft_all vs ft_early | +12.0pp | 16 / 4 | 0.012 |
| ft_all vs ft_replay | -1.0pp | 12 / 13 | 1.000 |
| ft_early vs ft_replay | -13.0pp | 11 / 24 | 0.041 |
⚑结论
结论。 想象纠错确实改变结果（相对不纠正 +4.2pp），但与等预算随机扰动无法区分（-1.1pp，p = 0.61）——模型的方向不是价值所在。用纠错训练（ft_all 57.0%）与仅回放对照（ft_replay 58.0%）持平，而把纠错集中到决策指标最看好的位置反而有害（ft_early 45.0% vs smolvla_libero 61.0%）。状态上的离线价值并不告诉你在该状态该教什么。