From Human Skill to Robotic Mastery

Introduction

The Beginning of Human Data

Historically, advances in artificial intelligence (AI) have been driven by the scaling of training data and computational resources. Embodied intelligence stands apart from established fields such as autonomous driving and large language models in a critical aspect: it lacks a pre-existing large-scale data repository, and cannot naturally accumulate task-relevant data through commercial deployment. Consequently, scaling up high-quality training data has become the paramount challenge facing the field. Humans continuously perform dexterous bimanual manipulations in daily life and industrial operations, rendering human behavioral data a natural and invaluable data source for training embodied agents. Yet learning from human demonstration data has long been plagued by two fundamental limitations. First, the inherent embodiment gap induces inconsistencies between human and robotic systems in kinematic structures and dynamic profiles. Second, the majority of existing human behavioral data is sourced from unconstrained egocentric internet videos or limited open-source datasets, such that neither the quantity nor quality of data meets the requirements of large-scale pretraining. The effective utilization of human data and its potential following large-scale deployment thus remain critical open questions. Psi-R2 and Psi-W0 represent the first suite of embodied intelligence models pretrained on human demonstration data at the 100,000-hour scale. In the subsequent sections, we elaborate on their architectural designs, the core design principles that motivate them, and the unique functional roles each model fulfills within the overall framework.

Chapter 1

1. Psi-R2: What Matters in Learning from Human Data

Psi-R2 accepts visual images and natural language instructions as inputs, and generates predictions of future video frames alongside executable robot action sequences. Its backbone network is built upon a pre-trained video generative model, thereby aligning with the conceptual definition of the emerging World Action Model (WAM). Concurrently, by processing visual and linguistic inputs to output robotic actions, it can also be categorized as a vision-language-action (VLA) model. These two characterizations are not mutually exclusive; the distinction lies solely in its adoption of a video generative architecture as the core backbone.

Psi-R2 is initialized and trained on the open-source Wan2.2-IT2V-5B-480P backbone, with the training objective defined as the joint prediction of future video frames and robot control actions. During the pre-training phase, we integrate both real-robot demonstration data and human behavioral data. The robot data is exclusively sourced from PsiBot’s Psi-MobiDex dataset, totaling 5,417 hours. The human data combines motion recordings collected via PsiBot’s in-house exoskeleton glove and uninstrumented bare-hand data, amounting to 95,472 hours that cover 294 distinct scenes, 4,821 task types, and 1,382 object categories. Following pre-training, the model can be fine-tuned using an extremely small volume of real-robot data—fewer than 100 trajectories—to execute long-horizon, fine-grained manipulation tasks including smartphone assembly, industrial packaging, and paper box folding.

For the integration of human data and real-robot data, we adopt a deliberately minimalist processing paradigm: human joint configurations are aligned to the robot kinematic chain via direct kinematic mapping, raw image inputs remain unaltered, and the aligned representations are directly fed into the model as state observations and action labels. In essence, our research focuses on leveraging human data to boost pre-training for a target specific robot embodiment, rather than developing a universal model capable of generalizing across heterogeneous embodiments. The former is a critical enabler for commercial robotic deployment, while the latter remains an open research problem.

Over the course of this research, we explored numerous specialized processing pipelines for human data, including image inpainting, keypoint-guided auxiliary loss functions, and shared latent space alignment between human and robot data. Ultimately, we adhere to the core insight of the Bitter Lesson: raw data in, raw data out. Once the training corpus reaches sufficient scale, any hand-engineered module becomes a performance bottleneck. Methods that artificially homogenize human and robot data improve learning efficiency on simple tasks such as pick-and-place, yet severely degrade performance on long-horizon, high-precision manipulation tasks. Most such approaches attempt to blur the distributional discrepancy between human and robot data to facilitate shared representation learning, but ignore the inherent divergence in their dynamic profiles. For fine manipulation tasks, indistinguishability between the two data domains introduces amplified error and task failure. Instead of deploying complex hand-crafted processing, we only perform input-output dimensional alignment and enable the model to learn end-to-end from large-scale raw data. The results demonstrate a clear trend: under limited data scale, the advantage is negligible, yet with sufficient data volume, the minimalist design consistently yields superior performance. Direct utilization of raw inputs and outputs yields the strongest generalization capability, most robust long-horizon behavior, and highest performance ceiling for manipulation tasks.

A key insight derived from this work is that model performance depends less on sophisticated architectural design than on principled understanding and curation of training data, especially human behavioral data. This includes data mixture ratios, diffusion schedulers, curriculum learning strategies, data retrieval mechanisms, and quantitative evaluation protocols. The core challenges involve designing staged training data combinations, retrieving semantically paired data samples, and validating whether actionable knowledge from human data is effectively transferred into the robot policy. Many of these capabilities rely on Psi-W0 as an Action-Conditioned World Model, particularly for rigorous policy evaluation. Performance improvement is only attainable through measurable evaluation, which serves as the definitive verification that the model has authentically absorbed knowledge from human demonstrations. This motivates the joint training of Psi-W0 alongside Psi-R2, with detailed elaboration provided in Chapter 2.

The end-to-end data pipeline is equally critical to overall system performance. We developed dedicated pipelines for data cleaning, automatic annotation, quality assurance, and expert review. All annotation and quality control procedures are implemented via auto-labeling, with human intervention only for final validation. We further integrate Psi-W0 into the data quality assurance stage, leveraging the world model’s visual predictive capability to assign data quality scores. From the pipeline’s inception, our objective was for human data to provide not only high-level semantic cues from visual inputs but also fine-grained manipulation details, driving us to maximize the precision of 3D human hand trajectories. For purely egocentric video data, we trained an end-to-end first-person hand detection model that fuses spatiotemporal information to accurately determine whether the left and right hands are in the frame, and directly predicts their MANO parameters and poses in the camera coordinate system. To acquire camera intrinsic and extrinsic parameters as well as image depth—thereby unifying the MANO trajectories into the world coordinate system—we innovatively adopted a technical stack combining DPVO and Any4D. For frames where hands are temporarily out of the frame, the system completes frame filling through smooth interpolation, abandoning the error-prone Infiller module in HaWoR. Hand motion recovery from purely egocentric video typically incurs millimeter-scale errors, while PsiBot’s exoskeleton glove reduces this error to sub-millimeter precision. For instance, we calibrate kinematic mappings via teleoperation with the glove prior to data collection, and further correct pose drift or inaccuracies within the post-processing pipeline to ensure final trajectories can be faithfully replayed on physical robots. A thorough analysis of the data pipeline is presented in Chapter 3. Finally, slow inference speed is a well-documented limitation of WAM-based systems; we therefore invested extensive engineering efforts into DiT caching, Torch Compile, quantization, and other system-level optimizations, reducing single-pass inference latency from 2.2 seconds to under 100 ms and enabling smooth, dexterous robotic manipulation.

Chapter 2

2. Psi-W0: Doing RL in the World Model

Psi-W0 accepts visual observations, natural language instructions, and robot action trajectories as inputs, and predicts corresponding future video frames. Analogous to Psi-R2, it is constructed upon a pre-trained video-generation backbone network. The core distinction lies in that action signals are incorporated as conditional inputs to regulate the generated video content, which renders it formally denoted as an Action-Conditioned World Model (AC-WM).

One may question the necessity of an AC-WM given that Psi-R2 already enables future video prediction. The critical justification resides in the modeling of counterfactual outcomes. All training trajectories employed for Psi-R2 are purposeful and successful executions, rendering the model inherently incapable of representing failure scenarios. Nevertheless, failure cases and other counterfactual predictions are indispensable for robust policy learning, particularly within reinforcement learning paradigms. This renders Psi-W0 invaluable not merely as a predictive module, but as a foundational instrument for policy evaluation, performance improvement, and the construction of a closed training flywheel, with reinforcement learning (RL) serving as its core mechanism.

Psi-W0 is initialized and trained using the identical Wan2.2-IT2V-5B-480P backbone as Psi-R2, and adheres to the same fundamental data formatting schema. Beyond the training data utilized for Psi-R2, Psi-W0 additionally incorporates approximately 30% supplementary failure trajectories—either explicitly collected or generated during routine data acquisition and inference procedures—to enable accurate modeling of counterfactual scenarios.

Psi-W0 fulfills two primary functions. The first is policy evaluation during the training of Psi-R2: we execute rollouts of Psi-R2 within scenes derived from human demonstration data, and employ Psi-W0 to verify whether the policy has genuinely acquired the actionable knowledge embedded within such demonstrations. The second, and more foundational, function is the translation of human data into robot-compatible trajectories. The core principle underlying this capability was initially proposed in our 2024 publication Object-Centric Dexterous Manipulation from Human Motion Data: the key to deploying human data on robotic systems is to transfer human dynamics to robot dynamics via reinforcement learning. As a representative example, consider the task of grasping an apple: direct kinematic retargeting of human apple-grasping motion to the robot typically yields slightly biased robot joint configurations due to the embodiment gap, despite the apparent subtlety of such errors.

The most straightforward resolution is to introduce an additional reinforcement-learning fine-tuning stage, such that the retargeted robot trajectory can accomplish physically plausible object grasping. Prior methodologies rely on object reconstruction within simulation environments to conduct RL, yet such pipelines suffer from excessive computational overhead, while the sim-to-real gap further impedes reliable modeling of deformable objects. Our framework replaces the simulator with an AC-WM to overcome these limitations. Given pre-trained Psi-R2 and Psi-W0 models, we feed the trajectory inferred by Psi-R2 from a human demonstration into Psi-W0 for rollout execution, switch the visual and dynamic representations of Psi-W0 to the robot domain, and conduct reinforcement-learning fine-tuning to enable the policy to complete the target task under robot-specific dynamics. In this process, the knowledge embedded in the human demonstration is absorbed into the policy, model performance is enhanced, and novel valid trajectories are generated. We subsequently filter high-quality samples from the generated data and reintroduce them into the training loops of Psi-R2 and Psi-W0 to construct a self-sustaining data flywheel. Furthermore, we can introduce stochastic variations within Psi-W0 to synthesize novel scenes and generate additional training trajectories.

Consequently, Psi-W0 is not designed as a secondary task-execution model. In contrast to Psi-R2, its primary objective is not direct task completion, but rather policy evaluation, performance optimization, and simultaneous translation of human data into robot-executable trajectories. This dictates that Psi-W0 requires not only successful task trajectories, but also failed and even non-meaningful samples, to provide the comprehensive distribution of outcomes necessary for effective policy learning within the AC-WM framework.

Chapter 3

3. What Really Determines Data Value

Although human behavioral data have been utilized for training embodied intelligence models since the early stages of the field, this research direction has witnessed a remarkable resurgence in recent years. Concurrent with this resurgence, a diverse array of data collection paradigms has emerged. From the hardware perspective, some systems employ head-mounted cameras for unconstrained recording; others leverage high-precision optical motion capture to track full-body keypoints; and certain setups integrate tactile gloves to acquire haptic feedback signals. The data collection environments also span a wide spectrum, including unconstrained in-the-wild settings, mobile manipulation scenarios, confined laboratory rooms, and dedicated fixed data acquisition factories. A critical question thus arises: which categories of human data are genuinely beneficial for large-scale model pre-training? While rigorous theoretical validation remains challenging, extensive empirical investigations over months of research lead us to a definitive conclusion: within our training pipeline, the signal-to-noise ratio (SNR) stands as the paramount metric governing whether human data exert a positive or detrimental impact on model learning, with low-SNR data actively degrading training stability and performance. Concretely, we identify a clear hierarchy in dataset composition: task diversity dominates object diversity, which in turn is vastly more influential than scene diversity. For sensory modalities, the priority follows precise 3D human pose >> tactile feedback >> 2D visual features.

For structured dataset construction, scene diversity exhibits the least significance for model performance. Benefiting from the strong representational power of pre-trained video-generation models, both the policy model and the world model achieve surprisingly robust cross-background generalization, even when the background diversity of the training corpus is severely limited. Our experimental results validate that models initialized with pre-trained Wan weights readily generalize to unseen scenes in both video generation and policy execution tasks, whereas models without such initialization suffer from frequent generalization failures. This indicates that cross-scene generalization capability is predominantly inherited from the pre-trained video foundation model. In contrast, robotic manipulation is inherently object-centric, as every manipulation task can be formulated as interacting with a target object and executing a specific interaction pattern. Accordingly, object diversity constitutes a critical factor for model generalization. Nevertheless, the core determinant of pre-training dataset quality is task diversity, as it directly governs the model’s coverage of downstream real-world tasks. Our experiments further reveal that cross-task generalization represents the most challenging capability to acquire, with a clear empirical trend: the greater the variety of tasks observed during pre-training, the stronger the model’s cross-task generalization performance. For this reason, we prioritize frequent variations in task types and object categories during human dataset construction, while regarding background variation as the least critical factor.

Regarding sensory modalities, we adopt a multi-modal collection strategy to acquire as many complementary signals as possible, enabling the model to autonomously discover task-relevant cues. In addition to language instructions, visual observations, and robot joint states, our input pipeline incorporates full-hand fabric-based tactile sensing. Among all modalities, full-hand 3D pose tracking serves as the most critical signal, acting as the essential bridge for upgrading 2D video-generation models to 3D physical manipulation models. We thus devoted substantial engineering efforts to optimizing this module, including high-precision sensing hardware, post-hoc pose error compensation, and systematic kinematic calibration. Psi-W0 provides a reliable paradigm for validating the significance of high-precision 3D pose data. We conduct comparative experiments by feeding 3D trajectories of the same task—collected via devices with varying precision—into the human-to-robot data conversion pipeline described in Chapter 2, and use Psi-W0 to synthesize corresponding robot manipulation trajectories. The results are unambiguous: trajectories captured via PsiBot’s exoskeleton glove yield optimal transfer performance, as they are inherently well-aligned with robot dynamic profiles.

We defer an in-depth discussion of tactile sensing to Chapter 5. As for 2D visual features including keypoints and segmentation masks, they exhibit characteristics analogous to the hand-engineered modules discussed previously: their contribution to model performance becomes negligible under large-scale data regimes, leading us to omit these features entirely in the final pipeline. In practice, we categorize human behavioral data into two distinct classes. The first class is precision-centric data: trajectories defined in the robot kinematic space that, after standardized processing, are functionally equivalent to real-robot demonstration data and can be faithfully replayed on physical robotic platforms. Such data hold the potential to replace real-robot data during model post-training, with high-precision data collected via PsiBot’s exoskeleton glove under controlled conditions serving as a representative instance. The second class is generalization-centric data: these data exhibit lower precision but enable extreme scalability; while they are unsuitable for direct post-training, they can be rapidly accumulated to endow pre-trained models with strong cross-domain generalization capabilities. We posit that both categories of human data are indispensable for building high-performance embodied intelligence systems.

Chapter 4

4. Human Data or Not Human Data, That Is the Question

The community frequently raises two fundamental questions regarding the utilization of human data. First, given sufficient real-robot teleoperation data in the future, can we entirely rely on robot data for model training and dispense with human data? Second, can we pursue the alternative paradigm of training deployable models exclusively from human data, without incorporating any real-robot data? In our perspective, both questions converge to an identical conclusion: for practical commercial deployment, the capability to train models solely from human data may ultimately become indispensable.

Commercial robotic deployment differs fundamentally from laboratory demo development. A core metric that is frequently overlooked in laboratory research is takt time, the standardized cycle time of industrial operations. The standard operating procedure (SOP) of frontline workers is the product of iterative optimization and typically approximates operational optimality. In industrial factories, even a single redundant step or minor slowdown in a workflow can result in financial losses amounting to tens of millions. In laboratory demonstrations, non-functional behaviors are often circumvented by switching tasks or adopting longer yet simpler workflows; such compromises are generally infeasible in real-world deployment. For this reason, the most valuable training data are often those generated by workers executing practical operations in authentic industrial environments.

Human data resolve this dilemma through two key advantages. First, the data collection process can be nearly perfectly aligned with the actual operational workflows of workers: for instance, production line workers can wear data-collection gloves during routine operations, and cashiers can use the same equipment during checkout. This setup eliminates the demand for specialized data collectors, minimizes data acquisition costs, and yields frontline operational data that are highly consistent with the requirements of real-world deployment. Second, and more critically, human data can capture task execution tempos that are unattainable via teleoperation. Teleoperation relies on operators to monitor and control robotic manipulation, which inherently limits operational speed; if the physical velocity limit of a robotic arm is denoted as 1200 units, teleoperation typically only achieves 800 units or lower. In contrast, human workers can generate data at the full physical velocity (1200 units) during natural operation. From a commercial deployment perspective, this renders human data of higher quality than teleoperated robot data, as they faithfully preserve the authentic SOP and operational tempo.

This underscores the critical importance of integrating high-tempo, high-quality human data into model training, and validates the significance of the exclusive human-data training paradigm. This scenario is highly analogous to the sim-to-real transfer problem: verifying the utility of simulated data requires zero-shot sim-to-real evaluation, and validating the effectiveness of human data likewise demands zero-shot human-to-robot transfer, with the simulator replaced by the world model in this framework. This necessitates the collaborative operation of Psi-R2 and Psi-W0, which leverage the human-to-robot data conversion pipeline introduced in the preceding chapter to translate human data into robot-executable trajectories. In our internal experiments, we have attempted to directly substitute real-robot data with converted human data during the post-training stage. Encouraging results have been achieved on simple manipulation tasks such as pick-and-place and bag grasping; however, the success rate remains relatively low for high-precision tasks including smartphone assembly and box insertion. While further research and iteration are required for this direction, we maintain that it represents an indispensable path for the commercialization of embodied intelligence.

Chapter 5

5. Tactile: A Medium for Cross-Embodiment Transfer

From an intuitive perspective, tactile sensing is undeniably indispensable for dexterous human manipulation. It is thus natural to ask why the most advanced general-purpose robot “brain” models integrate little to no tactile feedback. The primary constraint lies in the extreme scarcity of high-quality tactile data from physical robots. In practical deployments, tactile sensors mounted on robotic platforms frequently suffer from instability, and their output formats vary drastically across hardware systems, rendering large-scale scaling of robotic tactile data highly challenging. Human-sourced tactile data present a stark contrast: the cost of data collection is less than one-tenth that of robotic tactile acquisition, and the corresponding sensing devices are relatively portable. This unique advantage grants us a viable opportunity to scale up tactile data collection on the human side and leverage such data to enhance the training of embodied intelligence models.

We posit that tactile sensing holds particular significance for human behavioral data, because the fundamental essence of robotic manipulation resides in physical contact. Despite the discrepancies in visual appearance, kinematic architectures, and dynamic profiles between human and robotic systems, the binary and continuous signals that encode the occurrence and state of physical contact are fundamentally shared across embodiments. For this reason, we believe tactile information can serve as a critical enabler for knowledge transfer from human hands to robotic end-effectors. This motivated the integration of tactile sensing capabilities into the design of our custom data-collection glove: we aim to employ tactile signals to compensate for the dynamic gap between human and robotic hands within the world model.

Given that tactile data formats from physical robots are generally incompatible with those from our data-collection glove, and considering that most deployed robotic platforms lack tactile sensing hardware altogether, we adopt a mask-training strategy. When real-robot data are fed as input, we directly mask the tactile modalities; rather than treating tactile signals as observational inputs, we reformulate them as predictive action labels for model training. Experimental results clearly demonstrate that the integration of tactile data yields substantial performance improvements to the world model, while also endowing the model with the ability to anticipate robot-object interactions, indicating that the model has effectively learned contact and collision cues.

Closing

Collaborate with Us

If you are interested in collaborating, please reach out.

We are also hiring. If you would like to join us, we would love to hear from you.

If you are interested in our work, potential collaborations, or any other questions, please write to market@psirobot.ai.