LeRobot & MuJoCo Build a Robot Arm Policy (SO-101)

Robot imitation learning in simulation: MuJoCo digital twin, LeRobot datasets, ACT training, honest evaluation
What you'll learn

---

Build a complete robot learning loop in MuJoCo: define a task, record demonstrations, train an ACT policy, and evaluate it against a fixed success gate.
Write success checkers and contact gates that reject a faked grasp, so your success rate measures the robot's behaviour rather than a proxy that looks good.
Run controlled experiments that change one variable at a time: camera choice, action horizon, data volume versus coverage, and randomisation with purpose.
Map a policy's real operating envelope on held-out layouts, objects, and lighting, then decide with measured cost whether a VLA is worth it over a baseline.
Requirements

---

Comfortable with Python. No robot hardware required: everything runs in MuJoCo simulation on a normal laptop, and the training baseline fits a 6 GB GPU.
No prior robotics, MuJoCo, or LeRobot experience needed. Every model, dataset, and evaluation script is built from scratch in the course.
Description

---

This course contains the use of artificial intelligence.
Most robot learning courses end with a video of an arm completing a task once. This course focuses on something more useful: measuring whether the robot actually works reliably, understanding why it fails, and defining the conditions under which the result can be trusted.
You will build one complete robot learning pipeline in simulation using an SO-101 arm, MuJoCo, and LeRobot.
In this course, you will
Define a clear pick-and-place task before training any policy
Write measurable success criteria for the task
Build an automated pass/fail checker
Validate grasps using contact information
Detect false or visually misleading successes
Test simulation determinism
Study timestep sensitivity
Choose observation cameras based on what they can actually resolve
Test whether the policy depends on vision or proprioception
Collect and document a robot learning dataset
Train an ACT policy using LeRobot
Build a repeatable rollout and evaluation harness
Measure success rate across repeated trials
Calculate confidence intervals
Analyze failure cases instead of showing only successful rollouts
Study the effect of camera input
Experiment with action horizon
Compare dataset size against dataset coverage
Apply randomization with a specific purpose
Evaluate on held-out layouts
Test unfamiliar objects
Measure performance under lighting changes
Compare ACT with a vision-language-action approach
Consider both model performance and practical compute cost
Package the final experiment so another person can reproduce the result
Training loss is not treated as the final metric. You will see why a model with lower training loss can still perform worse during actual robot rollouts, and why repeated evaluation matters more than a single successful demonstration.
The final sections of the course are structured as controlled experiments. You will change one variable at a time and study how those changes affect robot performance and reliability.
By the end of the course, you will have a complete robot learning project with a defined task, dataset, trained policy, evaluation system, failure analysis, experimental results, and reproducibility package.
This is a simulation-only course. No physical robot hardware is used, and no sim-to-real transfer claim is made or evaluated.
Requirements

---

Comfortable with Python
No previous robotics experience required
No previous MuJoCo experience required
No previous LeRobot experience required
No previous robot learning experience required
A normal laptop for simulation
Up to a 6 GB GPU for the training baseline
Who this course is for

---

Python developers and ML engineers moving into robot learning who want a reproducible imitation-learning project instead of a demo that only looks good.
Robotics students and researchers who need an honest evaluation workflow: success contracts, failure galleries, and confidence intervals, not cherry-picked runs.
Anyone curious about VLAs and ACT who wants to see measured trade-offs on one task before committing to a large model.
Homepage

---