HumanCOIN: Egocentric Manipulation Dataset for Human Intent

HumanCOIN: Egocentric Manipulation Dataset for Human Intent

Shihan Wu, You Liu, Chunyu He, Minghui Qin, Longhan Jiao +46 more

13 min read•Nov 22, 2026

Egocentric Vision Corpora

Assembly101 instrumented tabletop assembly with 4,321 multi-view videos; ARCTIC captured bimanual manipulation of articulated objects with 2.1 million frames and 3D hand–object annotations; HOI4D recorded 2.4 million RGB-D frames of category-level hand–object interaction; and HoloAssist annotated 166 hours of human–assistant collaboration, including mistakes and their corrections.

Episode Duration and Task Stages

Episode duration determines how much context and how many steps a single sample carries, and it largely constrains the kind of training and evaluation the sample can support. HumanCOIN’s duration is not set uniformly; it varies systematically with tier, task type, and difficulty.

Duration distribution. The distribution is unimodal with a long tail: the density peaks at 52 seconds and the mean is 88 seconds, with the shortest episodes near 10 seconds and an open tail beyond five minutes. The tiers differ systematically. L1 has the widest range and the heaviest tail, with a mean of 101 seconds; L2 and L3 are more concentrated, at 65 and 60 seconds. That gap belongs to the recording protocol rather than to the tasks: L1 is segmented after the fact from long continuous recordings, whereas L2 and L3 are scripted episodes with bounded sessions; electrode stability further shortens the deepest sessions. The gap is therefore not evidence about task difficulty, nor about depth as such.

Task stages. Subtasks per episode count the stages a task is composed of rather than the temporal granularity of annotation. The count is highest at L1, at 11.0 per episode, where natural activity is split into many short steps; it falls to 8.3 at L2 and 7.9 at L3, where a fixed protocol groups the work into fewer, longer stages.

Overall, HumanCOIN shows clear tier differences along both duration and task structure: the shallow tiers hold long, loosely segmented, and diverse everyday activity, while the deep tiers concentrate on scripted, standardized tasks in which the same work is organized into fewer, longer stages. Duration characterizes the recording window, stage count characterizes task structure, and together they form the corpus’s spectrum from natural behaviour to complex controlled tasks.

Task Distribution

The tiers treat the vocabulary differently. L3 keeps 40 task verbs (40%), and those verbs cover 77.3% of L1 task occurrences. A protocol fixes the task set, so the verb list contracts to the tasks it scripts, while the steps that are recorded stay labelled in full: the deep tier is selective in vocabulary but not in the record.

Taken together, all three vocabularies are long-tailed, and what the tiers trade is breadth against depth rather than vocabulary size: the shallow tiers spread the full vocabulary over everyday activity, while the deep tier contracts to the classes its protocol needs and concentrates the record there.

Hardware Configuration

The intent proxies summarized above do not appear in a recording by themselves. Each becomes a record only through an instrument that converts a physical quantity into a measurable signal, so the kit is a set of measurement proxies: one class of instrument per component of the measurement framework, each chosen for what it can physically reach. Every instrument keeps its own clock, its own native sampling rate, and its own coordinate frame. This section describes what is worn at each tier, the two settings in which the kit is deployed, the two-phase protocol that organizes a controlled session, and how the resulting streams are stored.

Settings and Protocol

A tier’s collection is governed by two requirements: how much of the everyday scene it must preserve, and how much signal fidelity its instrumentation demands. The two pull in opposite directions, and the corpus resolves them tier by tier.

Two settings. L1 and L2 privilege mobility over signal depth. Every instrument they add is wearable and connects to a headband-mounted computer, so the operator moves through everyday environments—standing, turning, walking, and gesturing—without a fixed bench or cable constraints. This is the open setting: L1 and L2 are collected in it throughout, and L3 also uses it, with a lighter configuration, trading EEG density for scene coverage. Open here names the acquisition setting, not the task: an open-setting session can still use a scripted task sheet and a staged scene.

L3 is where the trade-off reverses. Dense electrophysiology requires a stable posture: electrode contact and impedance degrade with movement, and the denser the recording, the less it tolerates motion. The eye tracker, the gloves, and the wrist trackers load the same platform, so acquisition calls for a bench that does not move. Sessions run in short blocks, with a re-baseline between blocks to limit drift. L3 is collected under both settings: in the controlled setting the amplifier and the host remain on a dedicated desk and the operator holds still, and the session runs imagery then execution; in the open setting the kit runs on the headband computer in everyday scenes, and the session runs execution only.

Controlled-session imagery and execution phases

The two cued phases of a controlled session (for example, picking up a cup) are imagery and execution. During imagery, shown on a blue screen, the operator sits still with hands on the desk and rehearses the task; only EEG and gaze are recorded. During execution, shown on a green screen, the same operator manipulates the objects, with the head held as still as the grasp allows. The eyes move in both phases; the hands move only in the second.

Each controlled episode therefore comprises three segments, not two cues alone: preparation while the operator settles, imagery of the task sequence, and execution of that sequence. Because the first two are acquired while the body is still, the EEG records the task as it is planned rather than as it is performed: the step sequence and the action each step will require, with no movement to mask them. The high-density record exists for that pairing: the same task as a plan, acquired while the body is still, and as a performance seconds later. Open settings (L1, L2, and L3 with the lighter configuration) record the execution segment alone.

Collection procedure. L1 is recorded without guidance: operators are asked only to carry out the activity as they normally would. L2 and L3 use task sheets: the operator arranges the objects as drawn and follows the step order marked on the sheet, so that the same task is staged identically across sessions. At L3, tasks are drawn in random order rather than blocked, which sustains engagement and limits drift of the EEG towards drowsiness. Operators also rotate between scenes at a fixed interval, so that no operator works a single scene for an entire shift and no scene is shaped by an individual’s habits.

All operators are trained to a single standard before collection: how to wear each device, how to run a session, and how to record it. L3 adds two requirements: operators are trained to keep the head still during acquisition and are responsible for seating and maintaining the electrodes so that contact quality stays within range; participants assigned to L3 are also screened so that the EEG is clear and stable. Each hour of collection is followed by fifteen minutes of rest, to sustain attention and energy.

Data Storage

Streams are stored as recorded and linked by reference, never merged into a single file. Each device keeps its own clock, its own native sampling rate, and its own coordinate frame; forcing them into one stream would discard the timing the corpus exists to keep. Synchronization is therefore a property the data declares rather than one it enforces: no resampling and no alignment happen at write time, and the corpus is written as many asynchronous files, each at its own rate.

Quality is enforced on the recording, not on the downstream analysis. Every session is checked stream by stream once it is written—dropped or out-of-order samples, black or frozen video, dead channels, impossible value jumps, a device missing from the set—and a session that fails is recaptured rather than repaired, so a released episode never mixes a good half with a patched one. The verdicts travel with the episode in its metadata; the appendix lists the checks, their thresholds, and what each level costs.

Data Processing

Processing runs per stream, because the noise and the failure modes of each modality are set by its own measurement physics; the streams meet only at the end. Each stream is reduced to a timestamp, a unit, a declared frame, and an alignment report, and the five groups follow the dependency order of the data—synchronization and calibration, geometry and motion, gaze, electrophysiology—with annotation last, since it consumes the others. Chains are fixed, their parameters are logged per episode, and a failing step is flagged rather than filled, so any run can be replayed.

Geometry and Motion

Depth is built from RGB. The left-front and right-front lenses are hardware-timestamp synchronized, so every stereo pair comes from the same instant; the pair, together with the camera parameters, is passed to FoundationStereo, which returns a depth map stored with the intrinsics that produced it. Point clouds are formed per frame by projecting depth through those intrinsics and are subsampled to a fixed budget.

Gaze

Gaze is obtained by acquisition and by estimation. Both routes write the same fields—a timestamp and a pixel coordinate—in the pixels of the image each route reads: the scene camera of the eye tracker in the controlled L3 subset, the rectified head view at L1, at L2, and in the open L3 subset.

In the controlled L3 subset the eye tracker is calibrated per participant and per session and logs at 200 Hz; its coordinates are pixels of the scene-camera image, whose video runs at 30 fps. The calibration that produced each record—the intrinsics and distortion of that camera, with the transform that carries a ray into the head frame—ships with the episode, so a pixel can be back-projected and placed in the head frame. Because L3 records a reference measurement, it also serves as the test bed for the estimator used at L1 and L2.

HumanCOIN scenes used for egocentric recordings

Electrophysiology

Both streams are cleaned with a fixed chain. EMG is filtered in zero phase and reduced to an envelope. The trace is mirror-padded at both ends, band-passed between 20 and 450 Hz with second- and fourth-order Butterworth sections applied forward and backward, and comb-notched at 50 Hz with a 2 Hz bandwidth, with periodic extension and period alignment at the edges so that the notch does not ring there. The cleaned trace is rectified and reduced to one envelope per channel, and the raw trace is kept beside it as *_emg_raw. The appendix lists the orders, cut-offs, and padding steps.

Motion cannot be removed from a recording in which the operator moves, and even the deepest tier is stationary only by protocol. The cleaned EEG is therefore signal plus residual motion noise: we release the metrics that bound it—impedance pass rate, share of interpolated channels, and share of samples marked artefactual—and leave the choice of a stricter rejection threshold to the user. EMG is recorded at 2000 Hz in millivolts, eight channels per arm; EEG at 1000 Hz over 128 saline channels or at 250 Hz over 32 or 8 gel channels, in microvolts, with every channel named by brain region and its impedance logged in kilohms as a quality input.

Annotation and Segmentation

L1 needs a different route. Because passive collection does not intervene, a raw L1 recording is one continuous stream that lasts hours, and episodes are cut after the fact: frames and the available annotation are passed to a VLM, which proposes episode boundaries and step segments, and a human reviewer confirms or adjusts every boundary before release. This is why L1 episodes vary widely in length while controlled L3 sessions are scripted, and why an L1 boundary carries the uncertainty of a segmentation rather than the certainty of a cue.

Discussion

An egocentric corpus pairs an observation with an action and so answers what happened; why the action took that form belongs to the intent that preceded it, which is often settled before execution. HumanCOIN collects that intent, recording proxies for it at three signal depths. Whether it holds up depends on work we have not done; we state its limitations first, its openings second.

Limitations

The premise is untested. That intent sits between observation and action and carries information the observation does not is untested here: the corpus only makes it testable, and policy training, lead-time baselines, and cross-site quality gates lie outside its scope. Nor can the opposite be excluded: intent may be redundant with observation, since gaze is predictable from the scene and preparatory EEG may follow the kinematics. What fails then is not the concept but the proxies; the negative result turns “is intent redundant?” from a position into a measurement.

Coverage and representativeness. L3 accounts for 4% of the hours (500 hours); its controlled subset is about 350 hours (2.8%) and is the part concentrated on the desk, and electrode stability bounds how long the densest episodes can run. The scene and task mix is not a random sample of manipulation. L1 is assembled from fourteen heterogeneous units whose camera counts, resolutions, calibration, and declared frames differ, so how much of an L1 statistic reflects behaviour rather than the device mix cannot yet be separated.

Annotation and comparability. Step boundaries and object attributes are human judgements—L1 boundaries from automatic segmentation with human review, attributes measured only at L2 and L3. Hand motion is estimated at L1, at L2, and in the open L3 subset and measured in the controlled L3 subset, so the estimated part carries a reconstruction error. Comparability holds only in part: timestamps, units, frames, and calibration are released as each source declares them and the declarations are uneven, while quality metrics such as the impedance pass rate exist only at L3.

Long tails. The vocabularies are long-tailed: rare classes have few samples within a tier, cross-tier comparisons are unevenly powered, and the L3 share limits the deep tier’s rare classes, so “the deep tier is more concentrated” holds only qualitatively.

Research Directions

None of these questions can be answered on the collected data alone.

Evaluating before the fact. With only observations and actions a corpus scores after the fact; with intent one can ask before: how the action would change under a different intent, whether the decision stays readable when no action occurs, and whether a model predicts the action or the intent behind it. Such answers do not depend on

Frequently Asked Questions

What is HumanCOIN? HumanCOIN is an egocentric manipulation corpus that records proxies for human intent at three signal depths.

How do the HumanCOIN tiers differ in episode duration? L1 episodes average 101 seconds, compared with 65 seconds at L2 and 60 seconds at L3. The differences reflect recording protocols, not task difficulty or depth alone.

What is recorded during a controlled L3 session? A controlled episode includes preparation, imagery, and execution. Imagery records EEG and gaze while the operator remains still; execution records the operator manipulating objects.

How are L1 episodes segmented? L1 recordings are continuous streams lasting hours, so a VLM proposes episode boundaries and step segments. A human reviewer confirms or adjusts every boundary before release.