Report · Page 38

38
Center for Research on Foundation Models (CRFM)
Given the challenging closed-loop nature of learning control, it is possible that collecting datasets
of size comparable to those used in vision and language is insufficient for robotics. One exciting
option is to additionally leverage external, non-robotic sources of data such as videos of humans or
existing vision and natural language datasets. Such data is diverse and exists in large quantities
on the web [Deng et al. 2009; Lee et al. 2012; Heilbron et al. 2015; Goyal et al. 2017a; Damen et al.
2018; Gao et al. 2020a; Grauman et al. 2021], affording the possibility of broad generalization if
properly leveraged. Elegantly addressing the gap between the robot’s domain and those found in
videos or language on the web remains an open challenge; however, recent progress in domain
adaptation [Smith et al. 2019; Schmeckpeper et al. 2020] and using pretrained video and language
models in robotics [Lynch and Sermanet 2020; Shao et al. 2020; Chen et al. 2021c] present promising
directions towards closing this gap.
Finally, simulation presents a boundless source of rich interactive data that robots can learn from,
with a range of sensor modalities like rendered visuals, point-clouds, and simulated touch/audio.
However, a major challenge lies in bridging the gap between simulation and the real world, both in
the underlying physics and in the semantic distribution of environments and tasks. Recent work
has shown that by using extensive domain randomization, tasks ranging from flight [Sadeghi and
Levine 2017] to contact-rich manipulation [Mahler et al. 2017; OpenAI et al. 2019] and locomotion
[Peng et al. 2020; Hwangbo et al. 2019] skills learned in simulation can be transferred to real robots
with some success, and that the semantic and visual distribution of the real world can be simulated
by scanning the real world into a simulation [Chang et al. 2017; Kolve et al. 2017b; Savva et al. 2019a;
Szot et al. 2021; Shen et al. 2021a]. While these are promising steps towards closing the sim-to-real
gap, effective and general sim-to-real learning of manipulation and locomotion skills remains an
open challenge. Simulation data, real robot data, videos of humans, and natural language data could
all be essential to learning robotic foundation models.
Safety & robustness. Further complicating the development of new foundation models for robotics
is ensuring their safety and robustness when training or deploying them in the real world. We
can expect the safety risks from these models for robotics to be different from their language
counterparts given that embodied agents are empowered to manipulate and interact with their
surroundings directly in the physical world. One core safety challenge for learning-based systems is
the chicken-and-egg problem of needing to specify system constraints for safety prior to collecting
data, after which unforeseen unsafe behaviors requiring additional constraints may emerge. For
instance, an agent adapting to a new kitchen outside of the training distribution requires sufficient
safety guarantees to ensure safe data collection, which may either adversely affect task performance
or cause the agent to fail in novel ways. One way to resolve this is restricting the complexity of
the environment or increasing the complexity of the robot such that irrecoverable states or unsafe
actions are avoided by construction. The robot can also be tasked with autonomously resetting the
envir