Student Projects

Human videos encode rich procedural knowledge about how tasks are performed. However, most robot learning approaches extract only low-level, isolated skills from short demonstrations [1, 2], missing the long-horizon structure. As a result, robots struggle to generalize to extended, multi-step tasks. This project proposes to learn long-horizon task representations directly from human videos. Given a video demonstration, a Vision–Language Model (VLM) will be used to segment the video into semantically meaningful action fragments and infer symbolic preconditions and postconditions that describe the state required for the action to begin and the state achieved upon completion. Then, the fragments are used as demonstrations to distill low-level policies [1, 2] and finally composed into a behavior tree [3, 4] that captures the hierarchical and conditional structure of the task. By representing demonstrations as behavior trees rather than monolithic policies, the proposed approach enables modularity, interpretability, and reuse of learned skills. The resulting trees will be validated on a Stretch3 mobile manipulator. [1] Chen, Hanzhi, et al. "VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025. [2] Zhang, Anran, et al. "Actron3D: Learning Actionable Neural Functions from Videos for Transferable Robotic Manipulation." arXiv preprint arXiv:2510.12971 (2025). [3] Iovino, Matteo, et al. "A survey of behavior trees in robotics and ai." Robotics and Autonomous Systems 154 (2022): 104096. [4] Styrud, Jonathan, et al. "Automatic behavior tree expansion with llms for robotic manipulation." 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025.

- Extend works [1, 2] to consider relative motion (e.g., relative placement of objects) - Develop a pipeline based on VLMs to parse the video and generate labelled fragments - Combine policies distilled with [1, 2] to backchained construction of Behavior Trees [4] - Validate the framework to learn long-horizon tasks from human videos on the Stretch3 robot

miovino@ethz.ch anzhang@ethz.ch

To apply, please send your CV, transcripts, and a few sentences about why you would be a fit for this kind of project to bmicha@ethz.ch, joshua.naef@mrl.ethz.ch and alexander.hansson@mrl.ethz.ch. We look forward to hearing from you!