Xuji Zhang (张徐吉)

M.S. student (3rd year), Department of Automation, University of Science and Technology of China

Email: zhangxuji@mail.ustc.edu.cn

Research interests: embodied AI and robot learning — memory for generative policies, learning from human videos, and world models.

This page collects my recent projects, with real-robot videos.

RecallFlow: Episodic Memory as the Source Distribution of Generative Robot Policies

Xuji Zhang, Zichen Zhang, Wei Zhai, Jian Di, Kun Li, Yang Cao, Wenjun Lv

Preprint, 2026

Shell game: memory as condition starts generation from noise; memory as source starts generation from recall anchors.
Start from what you remember. In memory-dependent manipulation, the evidence that decides which action to take (cue) disappears before the robot acts (now). (a) Feeding the memory to the policy as a condition still starts generation from uninformed noise. (b) RecallFlow starts generation from what it remembers: the episodic memory is the source, and the transport network sees only the present.

Abstract

Diffusion and flow-matching policies have become a standard choice for visuomotor imitation learning, owing to their ability to model multimodal action distributions. In memory-dependent manipulation, the robot must act on information that is no longer visible. Existing methods feed the observation history to the policy as an additional condition, while action generation still starts from Gaussian noise. However, the policy must then pick out the relevant target from a long and redundant history, leading to spurious correlations and unreliable decisions as the memory load grows. Additionally, the initial noise carries no information about what the robot has seen, leaving generation without guidance toward the remembered target. To address these issues, we propose RecallFlow, a flow-policy framework that builds the source distribution from episodic memory. Recall–Relocalization relates the memory to the current view through a few recall slots, so that the source reflects where the remembered targets are now. Recall Anchors then turns each slot into a Gaussian component of the source, enabling generation to start close to the intended action instead of from noise. The transport network then completes the action from the current observation alone, keeping inference nearly as cheap as a memoryless policy. We further build a suite of memory-dependent tasks grouped by memory load and compare all policies under a matched protocol. Extensive experiments show that RecallFlow achieves a state-of-the-art average success rate of 80.4% on the public MIKASA-Robo memory benchmark, and outperforms the strongest baseline by 20.3 points on our suite and by 18.8 points on four real-robot tasks.

Method Overview

Overview of RecallFlow: encoding and memory, Recall–Relocalization, Recall Anchors, and the transport network.
Overview of RecallFlow. The present observation is encoded into tokens Zt. Recall–Relocalization lets Zt read the episodic memory bank and decodes K recall slots, each attending to the current view through a heatmap Hk. Recall Anchors turns each slot into a Gaussian anchor in action space; with a standard Gaussian, the anchors form the source p0. A transport network conditioned only on Zt carries x0 ∼ p0 to the action chunk, so memory reaches the action solely through the source.

Real-Robot Experiments

RecallFlow on the four real-robot tasks (third-person view, 2× speed). Videos play automatically without sound.

Remember Colorpick the block of the shown color
No Repeattouch each cup once
Marked Plateplace the cube into the marked plate
Shell Gamelift the cup that hides the block
Real-robot experiments: platform, four memory tasks, success rates, and rollouts on No Repeat.
Real-robot experiments. (a) We deploy the policies on an xArm 7 with a wrist camera and a third-person camera. (b) We design four tasks that differ in what must be remembered. (c) RecallFlow achieves the highest success rate on all four tasks. (d) The three behaviors observed on No Repeat: repeating one cup, touching one cup twice, and touching each cup once.

Simulation Benchmarks

Evaluation tasks in simulation: RecallSuite, MIKASA-Robo, and RoboMimic.
Evaluation tasks in simulation. (a) RecallSuite consists of five memory-dependent tasks: the cue is seen only earlier in the episode, and the candidates look identical when the robot must decide. (b), (c) Tasks from MIKASA-Robo and RoboMimic.
MethodSGTIMRC3RC5RC9Avg.
Octo-small463945171131.6
OpenVLA‡47145916628.4
π0334235221529.4
SpatialVLA232727171121.0
CronusVLA3253113918.0
MemoryVLA882444302041.2
GMP988380611767.8
SimpleMemVLA∗998371585974.0
Chameleon96.758.994.472.253.375.1
RecallFlow (ours)100.087.3100.078.336.380.4
Comparison with published methods on MIKASA-Robo. Success rate (%) on ShellGameTouch (SGT), InterceptMedium (IM), and RememberColor with 3, 5, and 9 colors (RC3–RC9). RecallFlow is averaged over three seeds and achieves the highest average (80.4 ± 1.1). Results of prior methods are taken from the literature. ∗Available only on arXiv. ‡OpenVLA-OFT, as reported in MemoryVLA.

Analysis

Generation steps of long-history DP and RecallFlow on a No Repeat decision, with samples at target and success rate versus generation steps.
Memory decides where generation starts. (a) At a decision of a No Repeat episode, all thirty samples of RecallFlow head to the untouched container from the source on, whereas long-history DP starts from noise and commits only in its last generation steps. (b) Over 15 decisions, the source samples of RecallFlow are already at the target. (c) RecallFlow therefore keeps its success rate with a single generation step.

Which Representation Fits the Task? Adaptive Selection for Robot Learning from Human Videos

Xuji Zhang, Zichen Zhang, Wei Zhai, Jian Di, Kun Li, Yang Cao, Wenjun Lv

Preprint, 2026

Project video (about 3 minutes), including real-robot rollouts on all five tasks.

Abstract

Human videos offer a scalable source of manipulation experience to complement limited robot demonstrations. Existing approaches typically commit to one intermediate representation, such as object or scene trajectories, hand keypoints, contact affordances, or latent actions, across tasks with different control requirements. We study how the level of abstraction in these representations can be matched to the information needed for robot control in each task. We introduce two diagnostics: embodiment residue, measuring the distinguishability of human and robot representations, and task information, measuring how much a representation helps predict robot actions. We compare six representation families under a common protocol to examine how these properties vary across tasks. We select the representation with the least embodiment residue among those that measurably improve action prediction, then use its predictor, trained on both data sources, to condition a policy trained only on robot demonstrations. We find that the trade-off between embodiment residue and task information varies across tasks, leading to different choices of abstraction level. Across five real-world manipulation tasks, our method achieves 64% average success, improving over the strongest external baseline by 11 percentage points and over the best fixed representation by 4 percentage points.

Method Overview

From diagnostics to a robot policy: cross-embodiment data, candidate representations, diagnostics and task-level selection, prediction and control.
From diagnostics to a robot policy. We extract six candidate families from human (H) and robot (R) demonstrations and select f⋆ using an information gate followed by embodiment-residue ranking. The predictor gf⋆ learns from both domains and outputs the future representation; the policy πf⋆ uses it with current observations to generate the action chunk At, learning only from robot actions. The Pour Granules example selects 3D hand keypoints, and this family remains fixed during execution. Curves inside the diagnostic cards are schematic. Checkmarks and crosses denote whether the lower 95% information-advantage bound exceeds zero.

A Telepresence Mobile Robot Built from Scratch

Tsinghua University, Institute for AI Industry Research (AIR) · Research intern, 2023–2024 · Related paper: IMWUT 2025

A differential-drive mobile robot built full-stack from scratch, from mechanical design and embedded control to ROS-based navigation.

(a) System architecture, built bottom-up

Application
Remote telepresenceNavigation research for visually impaired people
Control
STM32 velocity closed loopMotors and driversHost–controller communication (serial, ROS 1/2)
Body
BuildModeled in SolidWorksFabricated by a local workshop in Beijing
StructureWelded aluminum-alloy frameDifferential drive wheelsThree decks
SensorsRGB-D cameras (front and side)3D LiDARIMUSide force sensors

(b) Black version: 3D LiDAR

Black version of the robot seen from above, with the 3D LiDAR, aluminum frame, electronics and drive wheels labeled.
  1. 13D LiDAR
  2. 2Aluminum-alloy frame
  3. 3Electronics
  4. 4Drive wheels with independent suspension
Real-robot architecture. (a) Four layers, from the robot body up to applications. (b) Black version: 3D LiDAR, with the components that are visible in the photo labeled.
Corridor test (4× speed).

The robot navigates stably through narrow corridors with many turns and avoids pedestrians. It was used in research on remote telepresence and on navigation for visually impaired people.

Silver version of the robot, full view with mast.
Silver version: full robot (with mast)
Black version: independent suspension and drive wheels.
Black version: independent suspension and drive wheels
Electronics: main controller and motor drivers.
Electronics: main controller and motor drivers

Click a photo to view it at full size.

Co-authored Papers

  1. ART: Bridging the Embodiment Gap in 3D Human-to-Robot Affordance Transfer.
    Zichen Zhang, Wei Zhai, Xuji Zhang, Yang Cao, Yu Kang.
    IEEE Transactions on Artificial Intelligence, 2026 (accepted).
  2. Guiding Multiple Remote Users in Physical Tasks with Language-driven Robotic Telepresence.
    Ruyi Li, Jingfei Guo, Xinyi Zhang, Xuji Zhang, Zeqing Li, Jiannan Li, Jiangtao Gong.
    Proc. ACM IMWUT 9(3), 2025. DOI: 10.1145/3749472