Learned Robot Representations (RoboReps)

Overview

General-purpose robotic systems require powerful representations and abstractions. In deployment, such robots are expected to encounter diverse and complex scenarios. While recent large-scale learned models exhibit remarkable generalization, similarly getting representations that can flexibly generalize to all the unanticipated situations a robot might face remains challenging, especially given the cost of robot data. Thus, it is important to investigate how to best learn generalizable representations, evaluate their effectiveness, and leverage them for downstream robotics tasks.

Ideally, these representations should capture: (1) spatial-dynamic information needed for fine-grained control, (2) semantic information required for common-sense reasoning and scene understanding, and (3) knowledge of conventions needed for smooth human-robot interactions. Additionally, these representations must be robust to the diversity of tasks, scenes, and operators the robot will encounter. In this workshop, we aim to explore the following: What makes a good robot representation, how can we learn them, and how can we most effectively make use of them?

Our speakers and panelists are pioneering robotics and machine learning researchers defining the state of the art on a range of topics, including: end-to-end control, task and motion planning (TAMP), human-robot interaction (HRI), scene understanding / SLAM, and more. We invite the community for submissions in these areas as well as from a wider set of perspectives – for example, submissions addressing how the following fields might guide robotics research: (1) deep representation learning in vision and language; (2) learning representations for field robotics and AI, where data is extremely scarce or noisy; or (3) bias and robustness in neural representations.

Areas of Interest

We aim to investigate the following topics and research questions:

• What sorts of pre-training tasks, objectives, data, or models yield good representations for robotics?
  • How do we evaluate or choose good pre-trained models to fine-tune into robot policies or leverage in a perception or control stack?
  • Is “embodied” or “grounded” data necessary for good robot representations, or is general pre-training data sufficient?
  • When is fine-tuning large models from other domains enough for robotic foundation models and when do we need robot-specific large-scale training?
  • How do we elicit representations at a level of granularity and semantic abstraction that is appropriate for robot tasks?
• What functionalities do pre-trained representations grant robots? E.g.:
  • Common-sense semantic understanding (“what” to look for)
  • Reasoning about semantics, dynamics, spatial features, or motion (“how” to act)
  • Conventions for interacting with humans (dialogue, goal specification, information gathering, interpretability)
• How are these pre-trained representations best integrated into robot systems? E.g.:
  • End-to-end learned policies learned with BC or RL (e.g., RT-2, OpenVLA)
  • Zero-shot controllers (e.g., Code as Policies, MOKA)
  • Hand-crafted scene or task representations (e.g., semantic scene graphs, LeRFs, neural TAMP, image generation as subgoals)
• How can learned representations be de-biased such that the robots using them can:
  • Robustly operate in underrepresented environments (e.g., households from cultures around the world, not just “conventional” households)
  • Understand language and behaviors from diverse human operators

Submission Guidelines

Submission Portal: OpenReview

We are accepting workshop submissions of the following types

• Papers - up to 6 pages plus unlimited references / appendices
• Extended Abstracts - up to 2 pages plus unlimited references / appendices

We request that submissions should be in the RSS, CoRL, ICRA, or IROS format. They should not be anonymized.

Important Dates

Schedule

Invited Speakers

Mahi Shaffiulah

New York University
USA

Andreea Bobu

Massachusetts Institute of Technology
USA

Chelsea Finn

Stanford University & Physical Intelligence
USA

Sourav Garg

University of Adelaide
Australia

Wolfram Burgard

University of Technology Nuremberg
Germany

Georgia Chalvatzaki

TU Darmstadt
Germany

Liam Paull

University of Montreal
Canada

Organizing Committees

William Chen

U.C. Berkeley

Dominic Maggio

Massachusetts Institute of Technology

Mara Levy

University of Maryland

Dhruv Shah

Google DeepMind

Jared Strader

Massachusetts Institute of Technology

Kuan Fang

Cornell University

Accepted Papers (Read them here)

• Paper1
  Author1
• Paper2
  Author2

Program Committee