What Do VLAs Actually Learn through In-Context Failure Conditioning?

Abstract

Vision-language-action (VLA) policies often repeat the same mistakes after failure, as they lack memory of prior attempts. Whether a VLA can learn from its own failure through in-context conditioning, and what information in failure contexts is useful, remains unclear. We study this with RoboRetry, a controlled probe that inserts failure information directly into the policy context window via block-diagonal attention. Across 12 RLBench tasks, we evaluate failure-conditioned retry under a paired protocol and compare it with success-demonstration context under the same retry budget. Our results show that failure-conditioned context improves cumulative retry success over matched blind retry.

Our interventions point to two sources of retry lift: a slot-presence effect that provides a structural retry cue, and a content-dependent effect driven primarily by high-quality corrective language. We further identify remaining limits around semantic target/progress re-grounding and contact-precision recovery, where free-form corrective text is not sufficiently actionable for the required recovery. To support further study, we release FailureSlot, a dataset of 1,334 naturally occurring VLA failure trajectories with human-labeled corrective annotations and an 8-category taxonomy. Together, the probe and dataset provide a starting point for studying failure-driven in-context robot learning, where adaptation happens through experience in context rather than through parameter updates.