View + tactile world model: ground truth vs. rollout

A three-stream Diffusion-Forcing world model (view_tactile_wan22_action36) jointly predicting the RGB camera view and both GelSight-style tactile sensors from 2 latent frames of history plus a 36-D action (18-D projected view-space position + 18-D relative SE(3) sensor step). Every clip is ground truth on the left, model prediction on the right; tactile streams decode through the Wan2.2 VAE with the encode-time LoRA, the view stream through the base Wan2.2 VAE.

Drift: how error evolves over the 16 s rollout

Tactile (mean of both sensors)

held-out (test) train

View (RGB camera)

held-out (test) train

Per-latent-frame MSE across the 16 s rollouts (n = 6 per split, shaded ±1 sd). One latent frame = 0.67 s. The context frames are given, so their error is ~0.

Metrics

Tactile pixel PSNR

held-out (test) train

View pixel PSNR

held-out (test) train

Error bars ±1 sd across samples. Tactile = mean of left and right sensors.

Regime	Split	n	View PSNR	TL PSNR	TR PSNR	View lat. MSE	Tac lat. MSE

Long-rollout numbers are not on the same scale as short-window ones. The long-rollout ground truth is stitched from different VAE encode windows than the rollout's seed, which adds a reconstruction offset to every long-rollout error. It does not affect the train-vs-test or 8 s-vs-16 s comparisons, which are within-regime.

Short window — 2.7 s, 2 latent frames given, 2 predicted

held-out episodes 005/006 · 6 samples per split

Long rollout — autoregressive, predictions fed back

test episodes 005 & 006 · train episodes 001 & 004 · 3 start offsets each

Browse every rollout

Pick a cell, sample, and stream — each sample has view / left / right × gt / rollout / side-by-side.

How it was run

RUN=.../runs/view_tactile_wan22_action36
CKPT=$RUN/checkpoints/last.ckpt   # epoch 99, val_loss 0.0502

# short window, 6 samples per split (latents only; the built-in decoder
# targets 16ch VideoVAE+, wrong for these 48ch Wan2.2 latents)
python -m vm_diffusion.scripts.inference_rollout --ckpt $CKPT \
    --split {val,train} --num_samples 6 --no_decode

# long rollout — long_rollout.py patched to emit the 36-D view_tactile action
# (view half read from view_actions.pt, absolute per-transition-start position).
# --fps 6 is MANDATORY: without it fps derives from native 30fps timestamps.
python -m vm_diffusion.scripts.long_rollout --config $RUN/config.yaml --ckpt $CKPT \
    --episode <ep>.pt --start <offset> --seconds {8,16} --fps 6

# decode (effecterase env): tactile through Wan2.2 VAE + encode-time LoRA,
# view through the base VAE (visual latents were encoded with adapter=base)
python decode_rollout_wan22.py --rollout-dir <dir> --fps 6

Caveats

Long-rollout cells are n = 6 (two episodes × three start offsets); short-window cells are n = 6. All episodes are the same motherboard task domain, so the train/test contrast isolates memorization rather than task difficulty. "n.s." means the difference is smaller than twice its standard error — with these sample sizes that rules out large effects, not small ones. View PSNR is not comparable to tactile PSNR: the RGB scene is far more complex than a tactile gel image, and its latents were encoded/decoded with the base VAE rather than a finetuned adapter.

Model: 3 streams fused as a token sequence, 4 latent-frame window, 2 frames of history, pyramid sampling, 100 denoise steps, 36-D per-frame causal action conditioning. One latent frame = 4 frames @ 6 fps = 0.67 s.