On-Robot Reinforcement Learning with Goal-Contrastive Rewards

Ondrej Biza1,2, Thomas Weng1,*, Lingfeng Sun1,*, Karl Schmeckpeper1,*, Tarik Kelestemur1,*,
Yecheng Jason Ma3,†, Robert Platt1,2,† Jan-Willem van de Meent4,† Lawson L. S. Wong2,†
1 Robotics and AI Institute, 2 Northeastern University,
3 University of Pennsylvania, 4 University of Amsterdam
ICRA 2025
*Equal contribution: implementation and experiments. †Equal contribution: technical advising.
Paper

We propose Goal-Contrastive Rewards (GCR), a method for learning dense reward functions from passive video demonstrations. GCR combines two loss functions: an implicit value loss function that models how the reward increases when traversing a successful trajectory, and a goal-contrastive loss that discriminates between successful and failed trajectories.

Abstract

Reinforcement Learning (RL) has the potential to enable robots to learn from their own actions in the real world. Unfortunately, RL can be prohibitively expensive, in terms of on-robot runtime, due to inefficient exploration when learning from a sparse reward signal. Designing dense reward functions is labour-intensive and requires domain expertise. In our work, we propose GCR (Goal-Contrastive Rewards), a dense reward function learning method that can be trained on passive video demonstrations. By using videos without actions, our method is easier to scale, as we can use arbitrary videos. GCR combines two loss functions, an implicit value loss function that models how the reward increases when traversing a successful trajectory, and a goal-contrastive loss that discriminates between successful and failed trajectories. We perform experiments in simulated manipulation environments across RoboMimic and MimicGen tasks, as well as in the real world using a Franka arm and a Spot quadruped. We find that GCR leads to a more-sample efficient RL, enabling model-free RL to solve about twice as many tasks as our baseline reward learning methods. We also demonstrate positive cross-embodiment transfer from videos of people and of other robots performing a task.

BibTeX

@inproceedings{biza25onrobot,
  author       = {Ondrej Biza and
                  Thomas Weng and
                  Lingfeng Sun and
                  Karl Schmeckpeper and
                  Tarik Kelestemur and
                  Yecheng Jason Ma and
                  Robert Platt and
                  Jan{-}Willem van de Meent and
                  Lawson L. S. Wong},
  title        = {On-Robot Reinforcement Learning with Goal-Contrastive Rewards},
  booktitle    = {ICRA},
  year         = {2025}
}