Phd position f/m structured ego-world models for scaling dexterous robot learning

PhD Position F/M Structured Ego-World Models for Scaling Dexterous Robot Learning

Type de contrat : CDD

Niveau de diplôme exigé : Bac + 5 ou équivalent

Fonction : Doctorant

Niveau d'expérience souhaité : Jeune diplômé

Mission confiée

Context & Motivation: Recent breakthroughs in generalist robot learning are increasingly driven
by behavior cloning on large-scale heterogeneous teleoperation data across diverse robotic
platforms. However, a fundamental architectural bottleneck remains: these systems overwhelmingly
reduce perception and control to a monolithic, direct pixel-to-action mapping. This paradigm tightly
entangles scene semantics, camera viewpoint, environmental dynamics, and robot-specific morphology
within a single latent space. Consequently, while these models excel at interpolating within their
training distributions, they lack grounded structural priors. This fundamental entanglement leaves
even state-of-the-art policies severely brittle to cross-embodiment shifts, viewpoint perturbations, out-
of-distribution visual environments, and the compounding errors inherent to long-horizon inference.
Moreover, large-scale human-video / egocentric datasets encode rich priors on object affordances,
temporal structure, and scene dynamics, yet remain underutilized for robotics because we lack
robust mechanisms to map human trajectories onto robot actions. This PhD proposes an alternative
solution that relies on persistent 3D/4D scene representations, factorized world models that
decouple scene dynamics from the embodiment dynamics, and transferable interaction represen-
tations grounded in geometry rather than raw pixels.

Work program: This requires two structural changes to how we design dexterous robot models:
(i) 3D/4D Geometric Grounding: World models / visuomotor policies cannot rely solely on 2D visual
observations. They must be grounded in structural 3D geometry to form an embodiment-agnostic latent
belief state. This ensures the model understands physical spatial dynamics independently of the specific
camera viewpoint or robot morphology. (ii) Ego-World Factorization: The internal representation must
decouple the environment from the agent by factoring the latent space into persistent world dynamics
(how objects behave) and ego kinematics (how the specific body moves).
In the first year of the PhD we will design good mechanisms to learn 3D/4D latent representations that
reliably capture geometric consistency, temporal scene memory, affordance modeling, object-centric un-
derstanding, and viewpoint/embodiment-invariance. In the second year we will extend this to latent
world models factorized into separate world-dynamics and ego-kinematic streams. This will enable
interaction dynamics pretrained on large-scale human video to transfer to robots via lightweight embod-
iment adaptation? And finally, in the third year we will extend this to Cross-Embodiment Adaptation
and design appropriate benchmarks. In particular, we will examine how factorized latent world models
enable robust transfer from human to robot demonstrations, and what formal benchmarks are needed
to evaluate embodiment transfer robustness, geometric consistency, and long-horizon performance.

Principales activités

Main activities:

Analyse and implement related work.
Design novel innovative solutions.
Write progress reports and papers.
Present work at conferences.

Compétences

Technical skills and level required : programming skills are required.

Languages : English and possibly French.

Relational skills : Good communication skills.

Avantages