In-Context Learning for Robots: What Should Actually Go in the Context?

GEN-1.5 takes a few seconds of sensorimotor data as a physical prompt and one-shots a new task. π0.7 stuffs language, goal images, episode metadata, and history into a single prompt. RoboTTT pushes visuomotor context out to 8K steps. Over the past year, “in-context learning” has quietly become one of the highest-frequency phrases in robot learning.

But lumping all of this under “robot ICL” hides something important: the things being loaded into the context are not the same kind of thing.

Some contexts carry a task demonstration. Some carry human motion intent. Some carry a behavioral style. Some carry nothing but the robot’s own aimless flailing from five seconds ago. They share a word. They are not solving the same problem.

1. Why a policy needs context at all

Almost every VLA policy today can be written as the same function, $\pi(a_t \mid o_t, l)$: given the current frame plus proprioception $o_t$, and a language instruction $l$, output the next action $a_t$.

The form is clean and it trains well. But it smuggles in a strong Markov assumption: everything you need to decide the next action is already present in the current frame. History doesn’t have to appear explicitly — its role has been compressed into the weights.

Statistical language models made exactly the same bet. The probability of a sentence should factor by the chain rule, each word conditioned on every word before it. That’s unestimable, so n-gram models stepped back and assumed each word depends only on the previous one or two.

That approximation died of two things.

Long-range dependency. “The keys to the cabinet ___ on the table.” The blank is are, agreeing with keys five words back. The nearest noun is cabinet, which pulls hard toward is. A model that sees only the last two words is guaranteed to get it wrong.

Data sparsity. With a 10K vocabulary, bigrams give you 10⁸ combinations and trigrams 10¹². No corpus is large enough; the overwhelming majority of counts are zero. Every token you add to the context makes sparsity worse by an order of magnitude — unless you stop counting and start generalizing.

Robotics is standing in roughly the same place.

Inside the training distribution, the Markov assumption works fine. Fixed camera, fixed arm, fixed table height, bounded task set: the model only has to learn one mapping from observation to action. But deployment produces changes it cannot absorb:

• the current frame doesn’t reveal how far into the task you already are;
• the camera moved, so the relationship between pixel motion and robot coordinates changed;
• the arm or the end-effector was swapped, so the same action command now means different kinematics;
• the same task admits many styles — fast, slow, careful, brutal;
• new objects, new scenes, combinations that never appeared in training.

These share one property: the current observation is not sufficient to determine the action, and the missing piece happens to be available somewhere else.

So the policy gets one more input:

$\mathcal{C}$ is the context. It might be a demonstration trajectory, a human video, a goal image, a few seconds of the robot’s own poking around, or the execution history of this episode so far.

Which leaves exactly one question: what should go in $\mathcal{C}$?

2. Which uncertainty is the context resolving?

The term in-context learning started with language models. Give the model this:

Translate English to French:

sea otter    =>  loutre de mer
peppermint   =>  menthe poivrée
plush girafe =>  girafe peluche

cheese =>

and it completes fromage.

The model was never trained as an English–French translator. Not one byte of the weights moved. It inferred from a handful of examples what was going on here, and then did the fourth one.

It’s in-context because the acquisition happens inside the context window. It’s learning because the model ends up with a mapping it didn’t have before, and that mapping holds on inputs it never saw.

The robot analogue is nearly one-to-one: the language instruction is the spec, the demonstration trajectory is the worked example — a run of (observation, action) pairs from which the model figures out what this task is, then does it in a new scene. Weights still frozen.

Robot context windows keep getting fuller, but the contents are not resolving the same uncertainty. There are roughly three kinds.

(1) Which mapping to use. Language instructions, goal images, URDF and other embodiment metadata. These say what the objective is — but the mapping that achieves it was already in the weights. The instruction just selects one.

(2) What the current state is. The last several frames of execution. The cup went into the left drawer; three of the five steps are done. The mapping is unchanged; what changed is the state you feed into it.

(3) What the mapping itself is. Demonstration trajectories, human video. The model didn’t know how to do this. After reading the context, it does.

Only the third one is learning. It resolves uncertainty about the mapping itself. The first two are just as useful and just as context-dependent, but after the model finishes reading, the function it uses to turn observations into actions has not changed.

Written out, the difference lands in one place. Split the policy into the part frozen at training time and the part determined by context:

The question is whether the context gives you the name of $z$, or sample points on $f_z$. “Put the red block in the tray” names $z$ directly. A demonstration hands you a set of $(o, a)$ pairs from which $z$ has to be inverted — and the recovered $f_z$ has to keep holding on observations that were never in the context.

3. Learning from robot demos: teleop trajectories as context

The most direct source of context is the robot’s own demonstration trajectory, usually collected by human teleoperation. Observations and actions are recorded in the robot’s own frame. No representational gap to cross.

For a policy to recover a task mapping from such a trajectory, several things have to hold at once: the model must be capable of induction from pairs; the demo has to enter the context in a form the model can read; and training needs enough demo–query pairs to learn from.

3.1 The induction ability comes from the training objective

The ability to invert a task mapping out of a demonstration does not appear by itself. It’s shaped by the objective.

One-Shot Imitation Learning constructs a large number of distinct tasks at training time, pairs each with one demonstration and one execution, and optimizes performance after reading the demo. The demonstration here is a task condition, not a history buffer. The model is repeatedly forced to infer a task rule from one trajectory and apply it to new objects or scenes.

3.2 What form the demo takes

A teleop trajectory is a mixed sequence of images, proprioception, and continuous actions. The form it takes on entry determines how hard the induction is.

ICRT goes with the most literal option: tokenize images, states, and actions, concatenate into one sequence, train a causal Transformer with next-token prediction. At test time you prepend one or two teleop trajectories, append the current observation, and the model just keeps writing actions. Parameters unchanged. The paper validates adaptation to unseen tasks on a Franka. The takeaway: recovering a task mapping from a demonstration does not require robot-specific machinery — sequence modeling alone can carry it.

[Fig 3-2] ICRT interleaves image / state / action into a single token stream.

Instant Policy swaps in a graph. Demonstration, current observation, and the actions to be generated all become nodes; actions are produced by diffusion over the graph. The structure explicitly encodes correspondence between demo and current scene, whereas the sequence form makes the model discover that correspondence at the token level itself.

[Fig 3-3] Instant Policy: demo nodes, current-observation nodes, and action nodes to be generated.

Keypoint Action Tokens abstracts further out: visual observations become keypoint tokens, action trajectories become action tokens, and a pretrained text Transformer does the few-shot imitation directly. The keypoint layer strips the dimensionality out of pixels and continuous control, and what’s left looks a lot like the pattern matching a text Transformer already does.

Tokenization on the action side matters too. Action Tokenizer Matters compares several action tokenizers and finds that a scheme that reconstructs trajectories well does not necessarily preserve temporal smoothness — after quantization, adjacent actions can land far apart in latent space. Their LipVQ-VAE imposes a Lipschitz constraint on the latent action space. When action tokens are geometrically discontinuous, the local correspondences the model extracted from the demo can’t turn into smooth control output.

[Fig 3-4] Trajectory smoothness under different action tokenizers — source: Action Tokenizer Matters paper

Behavior Prompting Policy (BPP) also compresses the demo into an embedding, but drops the single-causal-Transformer setup: a prompt encoder cross-attends over the demo embedding with the current observation as query, and hands the result to an action decoder. It also ships with a handheld collection UMI, which moves the source of demonstrations from the lab teleop station to the deployment site.

[Fig 3-5] Architecture of BPP.

StellaVLA goes further still: raw trajectories are converted into task plans, subgoal descriptions, and 3D motion information, entering the context in structured form. What the demonstration carries shifts from what the expert did to what steps this task consists of.

[Fig 3-5] Overview of StellaVLA, which conditions a VLA policy on in-context structured demonstrations.

3.3 Where the training pairs come from

All of the above requires large numbers of demo–query pairs, and real robot data does not contain them natively — one collection session gives you one trajectory, not a pair. Besides collecting paired data on purpose, there are two routes.

Recombine what you already have. If a task has multiple demonstrations, take one as prompt and the rest as queries; the pairs appear for free (ICRT, BPP).

Generate them. One-Shot Imitation Learning procedurally generates task instances in simulation, manufacturing the task distribution itself. Instant Policy generates pseudo-demonstrations in sim to push pair counts to trainable scale. SynthICL goes all the way and trains ICIL policies on purely synthetic RGB.

3.4 Bolting the ability onto an existing model

ICL doesn’t have to wait for scale to make it emerge. RICL starts from a trained π0-FAST, runs one round of small-scale in-context post-training, then retrieves snippets from the ten-to-twenty new-task demos a user provides and drops them into the context. No parameter updates at test time. Cheaper than training a dedicated ICL policy from scratch.

[Fig 3-5] Architecture of RICL.

4. Human video as an in-context demo

Teleop demonstrations are expensive on both ends: you need a robot, and you need somebody who can drive it. Human video is far cheaper. One person, one camera — and there is already an enormous backlog of it sitting on the internet.

The cost is equally clear. Video has no action labels. You can see how the hand moved; you cannot see joint angles, and you cannot see how hard the gripper squeezed. Worse, a human body is not a robot body. The mapping the model needs to induce has an embodiment gap running through the middle of it.

Three ways across.

4.1 Vid2Robot: align the two representations directly

Vid2Robot takes a human manipulation video plus the robot’s current state and emits actions through cross-attention. Training uses video–trajectory pairs plus a contrastive loss that pulls human-video and robot-video representations together.

[Fig 4-1] Vid2Robot’s cross-attention architecture and contrastive alignment loss.

The paper reports cross-object motion transfer: a manipulation demonstrated on one object in video gets applied by the robot to a different object in its own environment. That transfer happening at all says the aligned representation carries more than pixel-level similarity.

4.2 MimicDroid: skip the pairing entirely

MimicDroid uses only continuous, unlabeled human play video. It mines the video for trajectory pairs with similar manipulation behavior and trains the model to watch one and predict the other. Human video serves as both context and supervision; no robot data is needed to form the pairs at all.

To narrow the appearance and kinematics gap, it uses wrist-pose retargeting and random patch masking, the latter reducing dependence on appearance detail.

[Fig 4-2] How MimicDroid builds trajectory pairs out of play video.

The question it attacks isn’t how do we back-fill action labels onto video. It’s which part of human behavior is structurally reusable across embodiments.

4.3 Point Policy: pick a representation both sides can read

Point Policy routes around raw pixels: human hand pose and object state become semantic keypoints, lifting the human manipulation into a morphology-agnostic representation. Humans and robots look very different in raw images and quite similar in keypoint space, which makes the mapping easier to induce. The paper validates robustness to novel objects and background distractors across 8 real tasks.

[Fig 4-3] Point Policy translates sparse human hand and object annotations into key points, predicts future 3D point tracks with a transformer policy.

Same road: Gen2Act first imagines a human manipulation video with a video generation model, then hands it to a video-conditioned policy — the context becomes an imagined motion. And HumanEgo converts egocentric video into entity-level hand–object interaction representations.

5. Task-agnostic random motion is also a good demonstration

The camera moved twenty centimeters to the left. The task didn’t change. The demonstration didn’t change. The image still looks reasonable to the model, and the output is still confident — and the action lands in the wrong place.

What the robot is missing here isn’t task information. It’s this: under this camera and this body, if I issue an action command, how does the image change?

That’s a system identification problem, not a task inference problem. And everything above — teleop demos, human video — silently assumes the system configuration is fixed. They assume the robot already knows how its actions map to visual change, and the demo only has to say where to go.

5.1 ICWM: let the robot move at random for a few seconds first

In-Context World Modeling for Robotic Control (ICWM) sets it up like this: before the task starts, let the robot move randomly for a few seconds.

Not demonstrating the task. Not approaching the target object. Just moving. Record three things per action — image before, the action, image after — and pack these $(o_{\text{start}}, a, o_{\text{end}})$ triplets into the context.

[Fig 5-1] ICWM: a self-exploration segment before task execution, used as in-context interaction evidence.

These are task-agnostic prompts. They don’t tell the model where to put the object; they tell it what a given action does to the image in this particular system. After reading them, the way the model turns observations into actions has changed — and what changed is the mapping itself, not a selection among existing mappings, and not an estimate of current state. So this is still in-context learning. The object of induction has just switched: not the task, the system.

Being task-agnostic drives the collection cost down further: no human present, no labels, no task binding. One exploration segment serves every task at that workstation, and when the camera gets bumped, the gripper gets swapped, or the bench gets moved, you re-run a few seconds and you’re recalibrated.

6. When context reaches pretraining scale

6.1 Qwen-RobotManip: inducing behavioral style from execution history

Generalization from cross-embodiment pretraining does not solve fast behavioral adaptation when you deploy onto a new robot or a new environment. Qwen-RobotManip adds an in-context policy adaptation mechanism for exactly this: recent $(o_h, s_h, a_h)$ triplets from within the same episode — what was seen, what state it was in, which $K$ steps it executed — are strung into a context that conditions the current action prediction. No parameter updates at deployment, no task-specific finetuning.

[Fig 6-1] Qwen-RobotManip: canonical representation, action expert, and history context.

Qwen-RobotManip also reports a context shortcut, which is the most instructive result in the section. If training always supplies the $H$ chunks immediately preceding the current step, the model takes the cheap path: the last chunk is temporally closest, so copying its action block is enough to drive training loss down. The context mechanism degenerates into a recency-based copy heuristic — and it collapses the moment recent history is ambiguous, atypical, or unrepresentative of how the robot executes overall.

The fix is stochastic context sampling: during training, the context window no longer abuts the current step but is sampled from random positions in the episode. Sampled chunks may be far away in time, so recency is no longer exploitable and the model has to extract behavioral style that runs through the whole episode. At deployment it gets a rolling window of the most recent $H$ chunks. The paper reports that removing this term gives you very low training loss and very poor task success.

Unlike a task demonstration, this context doesn’t describe what to do. It describes how this machine is currently doing things.

6.2 GEN-1.5: a one-shot ability nobody trained for

GEN-1.5’s physical prompt is a sensorimotor sequence — sensor data plus action trajectory — which can come from handheld-gripper human collection or from the robot’s own rollout. The prompt goes into a 30-second context window; the rest is left for the rolling current observation. Once the prompt is in context, the model just executes. Zero training steps.

Across 10 tasks, one-shot in-context averages 59% success (std 10%). With five minutes of per-task data (~50 demos) and 10 gradient steps, it goes to 83% (std 9%). On some tasks, in-context beats 1–5 gradient steps on the same data. The write-up is also candid about limits: the test tasks are short-horizon atomic manipulations, success rates aren’t high, and in-context-acquired skills are more brittle than finetuned ones — though they do handle some perturbations and recover from mistakes.

What separates this from everything above is where the ability came from. The earlier ICL methods write “performance after reading a demo” directly into the training objective. GEN-1.5 did none of that — no architectural modification for in-context learning, no auxiliary objective encouraging fast adaptation. Pretraining used randomly sampled continuous segments from a data engine, with no special handling for packing examples into context, and a physical prompt actually introduces a temporal discontinuity that never occurred during training. The capability showed up over eight months of pretraining.

The offered explanation is explicitly a hypothesis: the distribution of physical observations and actions may have the same burstiness and Zipfian structure as language, structure that has previously been linked to ICL in language models. Alternatively, physical labor contains repeated cycles, and the model learned to detect and continue those patterns.

A few derivative results around the physical prompt:

• Composition. Put two independently recorded prompts for different tasks into the context, with no transition between them, and the model chains them into one continuous behavior — inventing intermediate actions that appear in neither demo: repositioning, regrasping, error recovery.
• Sim-to-real prompting. A prompt built entirely from simulation rollouts (scripted policy, RL agent, or a human teleoperating in sim) drives a real robot to complete the task — despite the pretraining data containing no simulation at all.
• Cross-embodiment. A human demonstrates with their own hands inside the robot’s camera view; the robot then reproduces it with its own.
• Few-step adaptation. 1–10 gradient steps and 1–5 minutes of data adapt to a new task; after 10 steps, weights change by less than 0.15% on held-out tasks. At one step and one minute of data, held-out success is 66.5%.

A physical prompt is sensorimotor pairs, and what the model induces from it is a task mapping — same category as the human-video section. What’s distinctive is that the prompt can come from a human hand, the robot itself, or a simulator: the source of the context does not have to share an embodiment with the executor.

7. Two other things you can do with a context window

The other two kinds of uncertainty also depend on context, and are also useful. There just isn’t any learning in them.

7.1 Selection: which mapping to use

π0.7 is the most complete example. Language instruction, next semantic subtask, episode metadata like speed and quality, multi-view subgoal images — all in one prompt, forming a genuinely powerful control interface over the policy. The same task can be done at different speeds and with different grasps; the metadata says which one this time. The subgoal image says what the scene should look like a few seconds from now.

[Fig 7-1] π0.7’s context includes multiple distinct modalities, including language commands, episode metadata that describes the data quality and strategy, and multimodal inputs such as subgoal images.

These inputs describe objectives and style. The mappings that produce those behaviors already exist in the weights; the metadata picks one out. The input–output function is unchanged.

7.2 State: the part the current frame can’t see

The second kind is memory. MemoryVLA separates short-term working memory from long-term episodic memory. MemER has a high-level policy retrieve relevant keyframes and emit a language instruction for a low-level executor. ContextVLA compresses multi-frame history into a single context token. MEM pairs a video encoder for dense short-horizon visual memory with a language memory the policy rewrites itself, reaching fifteen-minute tasks. HiMe splits memory by timescale across three models — a fast Executor, a Sentry watching for subtask completion, and a slow Planner that edits episodic memory with explicit Add / Update / Delete.

What these solve is partial observability inside a single task: where did I just put the cup, which stage of this multi-step task am I on. Same task, same system — the current frame just doesn’t carry enough.

[Fig 7-2] The MEM equips VLAs with long-horizon memory.

Execution history written out is $(o_1,a_1),(o_2,a_2),\dots$ — formally identical to a demonstration trajectory. The difference is what changes after you read it. Read a demonstration and the way the model turns observations into actions changes. Read eight frames of history and that way is unchanged; what changed is the state you plug into it.

The real difficulty on this line isn’t reading the context either — it’s selection and compression. Longer history isn’t automatically better, and spurious correlations inside long sequences teach policies the wrong dependencies. Big Picture Policies uses a VLM to pick the minimal set of keyframes; Gated Memory Policy explicitly learns when to read and what to read. Both are attacking this.

7.3 And one orthogonal road

The same deployment-time evidence can either sit in the context window or be written into the weights by gradient. The latter is test-time training: RoboTTT compresses 8K steps of visuomotor history into fast weights; VANE isolates candidate updates from the live policy and only commits when subsequent visual outcomes support the update. This path changes $\theta$, not $z$. The goals overlap heavily with ICL; the costs differ. It needs gradients, it’s hard to roll back, and it imposes extra online safety requirements.

8. What’s still open

1. How does in-context learning emerge?

ICL in language models was not a trained capability. It showed up after pretraining whose only objective was next-token prediction. Robotics is running the opposite way: most of the work above trains the ability explicitly — writing “performance after reading a demo” into the objective, post-training ICL into an existing VLA, or manufacturing large numbers of demo–query pairs.

GEN-1.5 is currently the only reported case of the ability appearing on its own, and the mechanism is unstudied. What data distributions produce it? Is there a predictable relationship to data volume and model scale? And do explicitly trained ICL and emergent ICL differ systematically in how they generalize?

2. What form does a demo eventually take?

For the same demonstration, existing work has proposed wildly different encodings: interleaved token sequences, graph nodes, keypoints, compressed chunk embeddings, task plans and subgoal descriptions, semantic keypoints, task-agnostic interaction triplets, raw sensorimotor sequences. Each works in some setting. Almost none of them have been compared head to head.

Behind that is an unresolved tradeoff. Highly abstracted representations (keypoints, structured plans) make induction easier, but what they throw away may be exactly the contact, force, and material properties that resist symbolization. Low-abstraction representations (raw sensorimotor sequences) keep everything, at the cost of making the model establish demo-to-scene correspondence by itself. It’s not obvious where this converges.

3. How do we scale context?

A VLA has to emit actions at a fairly high rate, and every token you add to the context adds inference cost to every single decision. This is not the long-context problem language models have: there, the cost is latency. Here, the cost is that the control loop may stop closing. Which information must be kept frame by frame, which can be squeezed into one token, which can be dropped — long-context scaling in robotics is its own problem.

Conclusion

Robot ICL is starting to move past the phase of transplanting the GPT prompt analogy wholesale.

What it actually changes is what the policy looks at. A robot no longer sees only the current image and one sentence. It can read a demonstration, read a video, read a goal state, read the results of its own poking around — and from that decide which parts of the scene in front of it are task information, which are system information, and which are just noise.

So the question worth caring about is not how many tokens is the context. It’s:

Does the context contain the specific information the policy is missing but the task requires?

Sometimes that’s a task demonstration. Sometimes it’s a human video. And when the camera or the body has changed, it might just be the few seconds the robot spent moving at random.

9. Cheat sheet

Shares the context window, but is not ICL: π0.7 (resolves which mapping), MemoryVLA / MemER / ContextVLA (resolve what state), RoboTTT / VANE (the function does change — but what changes is $\theta$).

References

§3 — Teleop demonstrations and task-level ICL

• One-Shot Imitation Learning
• One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
• In-Context Imitation Learning via Next-Token Prediction
• RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Models
• Instant Policy: In-Context Imitation Learning via Graph Diffusion
• SynthICL: Scalable In-context Imitation Learning with Synthetic Data
• Behavior Prompting Policy: Demonstrations as Prompts for Manipulation
• StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models
• Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics
• Action Tokenizer Matters in In-Context Imitation Learning
• A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Robotics
• TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies

§4 — Human video and cross-embodiment transfer

• Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers
• MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos
• Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation
• Gen2Act: Human Video Generation in Novel Scenarios Enables Generalizable Robot Manipulation
• HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos

§5 — System identification and interaction context

• In-Context World Modeling for Robotic Control
• RoboMorph: In-Context Meta-Learning for Robot Dynamics Modeling
• LILAD: Learning In-context Lyapunov-stable Adaptive Dynamics Models
• Reflective VLA: In-Context Action Consequences Make VLAs Generalize
• VICX: Generalizable Robot Manipulation via Video Generation and In-Context Operator Network

§6 — Large-scale pretraining

• Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models
• GEN-1.5: Embodied Foundation Models are One-Shot Learners

§7 — Specification, memory, and test-time adaptation

• π0.7: A Steerable Generalist Robotic Foundation Model with Emergent Capabilities
• MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation
• MemER: Scaling Up Memory for Robot Control via Experience Retrieval
• ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Context
• MEM: Multi-Scale Embodied Memory for Vision Language Action Models
• HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control
• BPP: Long-Context Robot Imitation Learning by Focusing on Key History Frames
• Gated Memory Policy: In-Context Memorization and Adaptation
• RoboTTT: Context Scaling for Robot Policies
• VANE: Reliable Test-Time Training for Vision-Language-Action Models via Future Visual Representation Prediction

Cite this

@article{Wang2026RobotICL,
  title   = "In-Context Learning for Robots: What Should Actually Go in the Context?",
  author  = "Wang, Siyin",
  year    = "2026",
  url     = "https://sinwang20.github.io/blog/robot-icl-en/"
}