Modeling Latent Disturbances for Robust Decision-Making in World Models

Carnegie Mellon University

Robust decision-making in the latent space of world models, by modeling latent-space disturbances.

Method

Latent disturbances perturb the learned dynamics within a calibrated uncertainty set of plausible transitions, solved via efficient game-theoretic optimization.

Robust Policy Steering

54%→15% failure rate with a robust latent safety filter and 35%→70% success rate with sample-and-verify in contact-rich manipulation.

Robust Optimization in World Model Latent Space

Robust decision-making directly in the latent space of world models, validated against ground-truth robust solutions when system dynamics and disturbances are known.

Benchmarks

Three vision-based tasks in simulation and the real world: Dubins’ car, block pouring, and egg serving with a Franka robot.

LUCID: Latent-space Uncertainty-Calibrated In-distribution Disturbance

Serve a sunny-side-up egg without flipping it.

Click a trajectory to explore its imagined outcome.

Predicted next latent state distributionCalibrated uncertainty setOut-of-distribution

World Model Latent Space

Uncertainty Set of Latent Dynamics

Current state

WM imaginations

Generated by the world model.

How can we make robust decisions in a world model’s learned latent space?

What is robust optimization?

The same serving motion can place the egg on the plate or flip it, depending on unobserved friction or oil on the spatula. Robust optimization selects an action that remains effective under the worst-case disturbance in a specified set. This is a game: the disturbance maximizes the cost of the outcome, while the robot chooses an action that minimizes this worst-case cost.

\pi_{\mathrm{rob}}(s)=\underbrace{\underset{a\in\mathcal A}{\arg\min}}_{\text{choose an action}}\;\underbrace{\colorbox{#FFCC80}{$\displaystyle\max_{d\in\mathcal D}\;C\!\left(f(s,a,d)\right)$}}_{\text{worst-case cost}}

How do we do this in latent space?

A world model learns compact latent states and their dynamics from observations. It does not explicitly represent disturbances such as friction or the amount of oil. Instead, uncertainty about these factors is reflected in its distribution of predicted next states.

Key idea: model a latent-space disturbance as a perturbation to the predicted next-state distribution. Optimize this disturbance within an uncertainty set of plausible latent dynamics to induce pessimistic world-model imaginations.

\pi_{\mathrm{rob}}(z)=\underset{a\in\mathcal A}{\arg\min}\;\underbrace{\colorbox{#FFCC80}{$\displaystyle\max_{f_z^d(z,a)\in\colorbox{#79D8C4}{$\scriptstyle\mathcal F(z,a)$}}\;\mathbb E_{z'\sim f_z^d(\cdot\mid z,a)}$}}_{\text{worst-case latent disturbance}}\!\left[C(z')\right]

Here, the disturbance picks the most adverse latent dynamics within the uncertainty set, so that the world model imagines futures in which the action is most likely to fail, while the robot chooses the action that performs best even under these pessimistic imaginations. The uncertainty set must therefore include diverse transitions that are plausible under the system, while excluding implausible ones.

Challenge: Overly pessimistic disturbances. A nearby or high-likelihood latent state need not represent a physically feasible outcome. The disturbance can exploit these model errors and imagine impossible failures, making the robot overly conservative. The uncertainty set must cover adverse transitions while constraining implausible, out-of-distribution outcomes.

How to construct an uncertainty set over latent dynamics?

Current State

Next State

KL divergence from the predicted next-state distribution

Application: Robust Policy Steering

This latent-space robust optimization can be used to steer a task policy π^task at runtime, preventing hard-to-model failures under uncertainty. We instantiate it for two policy-steering paradigms, replacing nominal world-model imaginations with pessimistic yet plausible ones.

Latent Safety Filtering

Safeguard π^task with least-restrictive filtering: evaluate the safety of the action proposed by the task policy, and intervene with the robust safety policy only when that action is doomed to fail.

\pi_{\mathrm{rob}}(z)=\begin{cases} \pi^{\mathrm{task}}(z), & \text{if }\colorbox{#FFCC80}{$\mathbb E_{z'\sim f_z^{\star}(\cdot\mid z,\pi^{\mathrm{task}}(z))}$}\big[V^{\mathrm{safe}}_{\mathrm{rob}}(z')\big]>0,\\[.4em] \pi^{\mathrm{safe}}_{\mathrm{rob}}(z), & \text{otherwise}. \end{cases}

The robust safety value and safety policy are learned with pessimistic imaginations induced by the latent disturbance:

V^{\mathrm{safe}}_{\mathrm{rob}}(z)=\min\Big\{\ell_z(z),\;\max_{a\in\mathcal A}\;\colorbox{#FFCC80}{$\displaystyle\min_{f_z^d\in\colorbox{#79D8C4}{$\scriptstyle\mathcal F(z,a)$}}\mathbb E_{z'\sim f_z^d(\cdot\mid z,a)}$}\big[V^{\mathrm{safe}}_{\mathrm{rob}}(z')\big]\Big\}

Sample-and-Verify

Samples K candidate action sequences a^(k) ∼ π^task(z), evaluates each with world-model imaginations, and executes the one with the lowest expected cost.

\pi_{\mathrm{rob}}(z)=\underset{\mathbf a\in\{\mathbf a^{(k)}\}_{k=1}^{K}}{\arg\min}\;\colorbox{#FFCC80}{$\displaystyle\mathbb E_{\mathbf z'\sim f_z^{\star}(\cdot\mid z,\mathbf a)}$}\big[C(\mathbf z')\big]

The learned latent disturbance generates adverse futures for each candidate, so action selection accounts for calibrated system uncertainty.

1) Real-World Results: Serving Sunny-Side-Up Fried Eggs

Robust latent safety filtering

Loading trajectory…

Not FilteredRobust Safety Filter Active··· Zero threshold

Rollouts safeguarded by the robust safety filter. Orange marks steps where the filter overrides the task-policy action because its robust safety value becomes non-positive.

Ours preemptively identifies actions that may cause the egg to fall or flip, and overrides them before failure becomes unavoidable.
It intervenes with robust fallback actions whose plausible outcomes remain safe under worst-case dynamics.

Why does the nominal safety filter fail?

Loading trajectory…

Not FilteredNominal Safety Filter ActiveRobust Safety Value (Reference)··· Zero threshold

Rollouts safeguarded by the nominal safety filter. The dotted orange line evaluates the same rollout with the robust safety value for reference.

Overly optimistic imaginations can delay the intervention until failure is no longer avoidable.
Fallback actions that appear safe under nominal dynamics can still fail under adverse contact conditions, flipping the egg sunny-side down.

Does latent disturbance induce pessimistic yet plausible imaginations?

Example 1

Nominal imagination

Camera 1Camera 2

Pessimistic imagination

Camera 1Camera 2

0.00 / 2.00 s

Both videos are generated by the world model.

Nominal (left) and pessimistic (right) world-model imaginations for the same real-world observation and action.

Robust to disturbances: different spatula surfaces

No tape

Failure rate ↓ · n = 30

95% Wilson intervals

No tape
Method	Rate	95% CI	Count
Base Policy	0.6333	0.4551–0.7813	19 / 30
Nominal	0.3333	0.1923–0.5122	10 / 30
Ours	0.0667	0.0185–0.2132	2 / 30

Heavy-duty tape

Failure rate ↓ · n = 30

95% Wilson intervals

Heavy-duty tape
Method	Rate	95% CI	Count
Base Policy	0.9000	0.7438–0.9654	27 / 30
Nominal	0.6000	0.4232–0.7541	18 / 30
Ours	0.1333	0.0531–0.2968	4 / 30

Scotch tape

Failure rate ↓ · n = 30

95% Wilson intervals

Scotch tape
Method	Rate	95% CI	Count
Base Policy	0.8333	0.6644–0.9266	25 / 30
Nominal	0.7000	0.5212–0.8334	21 / 30
Ours	0.2667	0.1418–0.4445	8 / 30

Across all surfaces

Robust rate ↑ · n = 30

95% Wilson intervals

Across all surfaces
Method	Rate	95% CI	Count
Base Policy	0.1000	0.0346–0.2562	3 / 30
Nominal	0.1667	0.0734–0.3356	5 / 30
Ours	0.6667	0.4878–0.8077	20 / 30

We replay 30 failure and 20 success trajectories on three spatula surfaces while safeguarding them. The robust rate is the fraction of trajectories that remain safe across all surfaces.

Ours consistently reduces failures and achieves a higher robust rate, so its interventions do not depend on a favorable surface.
The Nominal filter is effective only under certain surfaces: its fallback actions break down once the unobserved contact properties change.

Robustly safeguarding and steering task policies

Diffusion Policy · safety filtering

Success rate ↑ · n = 20

95% Wilson intervals

Diffusion Policy · safety filtering
Method	Rate	95% CI	Count
Base Policy	0.1000	0.0279–0.3010	2 / 20
Nominal	0.5000	0.2993–0.7007	10 / 20
Ours	0.7500	0.5313–0.8881	15 / 20

π₀.₅ · safety filtering

Success rate ↑ · n = 20

95% Wilson intervals

π₀.₅ · safety filtering
Method	Rate	95% CI	Count
Base Policy	0.4000	0.2188–0.6134	8 / 20
Nominal	0.4500	0.2582–0.6579	9 / 20
Ours	0.7000	0.4810–0.8545	14 / 20

π₀.₅ · policy steering

Success rate ↑ · n = 20

95% Wilson intervals

π₀.₅ · policy steering
Method	Rate	95% CI	Count
Base Policy	0.4000	0.2188–0.6134	8 / 20
Nominal	0.3500	0.1812–0.5671	7 / 20
UnConf.	0.3000	0.1455–0.5190	6 / 20
Ours	0.7000	0.4810–0.8545	14 / 20

Left two panels: safety filtering over 20 trials with Scotch tape on the spatula. Right panel: sample-and-verify steering of π₀.₅, which evaluates 8 candidate action chunks in world-model imagination and executes the one with the best predicted outcome.

As a safety filter, Ours improves success more than Nominal for both a diffusion policy and a fine-tuned VLA.
In steering, Ours selects actions that remain favorable even under plausible worst-case futures.
Nominal steering imagines optimistic futures for risky actions, while UnConf. steering imagines implausible failures even for safe actions, weakening discrimination among candidates.

2) Simulation Results: Block Pouring

Same action · Different physics

Does the in-distribution constraint matter?

Imagination using latent disturbance optimized without the in-distribution constraint

Executed in simulation

Camera 1Camera 2

OOD imagination

Camera 1Camera 2

0.00 / 3.07 s

Right video is generated by the world model.

Left: the action executed in simulation. Right: the imagined outcome of the same action, using a latent disturbance optimized without the in-distribution constraint.

Without the in-distribution constraint, the disturbance exploits model errors and imagines the green block falling in an infeasible manner.
Latent states with high predicted likelihood can still be out-of-distribution; the constraint excludes them and prevents overly pessimistic imaginations.

Does latent disturbance induce pessimistic yet plausible imaginations?

Imagination 1

Nominal imagination

Camera 1Camera 2

Pessimistic imagination

Camera 1Camera 2

0.00 / 17.23 s

Both videos are generated by the world model.

Nominal (left) and pessimistic (right) world-model imaginations for the same initial state and action.

The nominal imagination can predict a non-failure outcome, overestimating safety.
Within the calibrated uncertainty set, the latent disturbance imagines a plausible adverse transition, such as the green block sliding off and falling.

Does the robust safety value avoid overestimating safety?

Loading trajectory…

Not FilteredNominal Safety Filter ActiveRobust Safety Filter Active··· Zero threshold

Both filters safeguard the same base action sequence. The plots show when each filter intervenes; the videos show whether those actions prevent failure.

The nominal safety value evaluates the average outcome of an action, so it can overestimate safety when adverse outcomes occur.
The robust safety value evaluates worst-case plausible outcomes, so Ours intervenes more preemptively, using actions whose plausible outcomes avoid failure.

Can robust filtering prevent failures under uncertainty?

Base policy

Randomized physics

2,000 rollouts per method

Randomized physics
Method	Rate
Base Policy	0.4400
Nominal	0.4500
Worst-of-10	0.4600
CVaR (0.1)	0.3900
UnConf.	0.0000
Ours	0.7300

Each policy is rolled out for 2,000 trajectories with randomized initial states and physics. Success: Fraction of rollouts that complete the task.

Sampling-based (Worst-of-N) and risk-sensitive (CVaR) baselines are less effective: in high-dimensional latent spaces, sampling rarely finds meaningful worst cases, while extreme tails can include implausible states.
Ours optimizes against worst-case plausible latent dynamics, reducing failures while remaining far less conservative than UnConf., which drops the in-distribution constraint.
Its interventions yield positive safety gains, moving the system toward safer realized states, whereas Nominal gains are smaller or even negative.

Robustness under controlled uncertainty

Base policyTeleoperation trajectories

Controlled-physics replay

100 trajectories × 20 physics settings

Controlled-physics replay
Method	Rate
Base Policy	0.1000
Nominal	0.1100
Worst-of-10	0.0700
CVaR (0.1)	0.1000
UnConf.	0.9000
Ours	0.7700

We replay 100 successful teleoperation trajectories, each under 20 different physics settings. All-safe: Fraction of trajectories safe under all 20 physics settings.

Ours keeps most trajectories failure-free across all physics settings, with safety actions that remain effective across dynamics variations.
UnConf. also avoids failures, but only by rarely completing the task, since implausibly pessimistic imaginations make useful actions look unsafe.

Robustness under partial observability

Base policy

Fixed physics

2,000 rollouts per method

Fixed physics
Method	Rate
Base Policy	0.4300
Nominal	0.6000
Worst-of-10	0.3400
CVaR (0.1)	0.0000
UnConf.	0.0000
Ours	0.8100

The same task with fixed physics, so uncertainty arises only from partial observability and model approximation. Success: Fraction of rollouts that complete the task.

Even when the system is deterministic, the learned world model remains uncertain about future transitions due to partial observability.
Nominal filtering is more effective than under randomized physics, but Ours further reduces failures by accounting for this remaining uncertainty.

BibTeX

@article{seo2026modeling,
  title   = {Modeling Latent Disturbances for Robust Decision-Making in World Models},
  author  = {Seo, Junwon and Bajcsy, Andrea},
  journal = {arXiv preprint arXiv:2610.07599},
  year    = {2026}
}