Related Work
We study how the objects retained, their estimated positions, movement histories, and contextual descriptions help answer different spatial questions. Errors in geometry can misplace an object or make it appear to have moved. We therefore evaluate how these errors affect answers, alongside their effect on localization.
Method
Ledger (Long-horizon Egocentric Descriptions, Geometry, and Event Records) constructs a persistent object memory offline from an egocentric recording and supplied or estimated camera poses. It is constructed once per recording and independently of any question. The memory is then reused for spatial question answering and named-object localization. On UCS-Bench, reading it costs about 2.0k input tokens per question, compared with 9.6k for 48 sampled frames.
Offline Memory Construction
Discovery and geometry. Qwen3.5-9B names objects in sampled frames. Synonym merging produces a vocabulary for YOLO-World, so objects enter memory whether or not the wearer touches them, within the detection and 3D-lifting budgets. From the image, detection box, and label, WildDet3D predicts a 3D box in camera coordinates, and the camera pose places its centre in scene coordinates.
Query-Time Reasoning
Answering. Retrieved labels, ranking signals, and segment histories (times, positions, and descriptions) are rendered as text. A single call to Qwen3.5-9B or GPT-5.4 answers using this evidence, the question, and its options.
HD-EPIC queries use the completed memory. For timestamped UCS-Bench queries, snapshots remove later observations and descriptions, truncate histories and the wearer path, and recompute positions. This filters what the answerer sees without making construction causal.
Experiments
We first show three memories answering on one stream. Five research questions then connect benchmark results to interventions on the memory, and an appendix analyses failures by stage.
Conclusion and Limitations
Ledger makes past object observations reusable for spatial answering and localization. Admitting untouched objects answers fixture questions, and persistence and contextual descriptions contribute more than coordinate refinement; missing event evidence and fragmented identities limit interaction and counting. Comparisons between memories depend on the reader, and scene changes break both retrieval and construction, which per-scene construction partly repairs. Evaluations should measure answerable evidence alongside localization, and should state retrieval budgets and scene boundaries.
Camera Poses and Object Lifting
We use calibrated device trajectories on HD-EPIC and registered EgoLoc poses on VQ3D. To estimate poses, we use FastVGGT, a training-free acceleration of VGGT. It processes 96-frame chunks with 16 overlapping frames. We align adjacent chunks using a Sim(3) transformation estimated from their shared frames.
These estimated poses have arbitrary scale. To recover metric scale, the pipeline compares pairs of observations of the same static object with their predicted metric positions in camera coordinates. This step fails for most UCS-Bench recordings and all TeleEgo recordings. Distances in those memories are therefore expressed in scene units.
Given an image, a 2D box, and an object label, WildDet3D predicts a metric 3D box in camera coordinates. We use its centre as the object’s position. This avoids selecting pixels from a dense depth map, where object and background surfaces can be mixed. Although the predicted box is metric, its position in scene coordinates is only metric if the camera poses also have a known scale.
In the earlier HD-EPIC analysis, the median localization error was 0.39 m, with a median along-ray component of 0.37 m. This motivates using multiple views with known camera poses to refine object positions.
On the UCS-Bench pilot, two Depth Anything 3 reconstructions were rejected. Their chunk scale ratios were 1.16 and 1.58, and their alignment residuals at chunk boundaries were 0.67 and 7.10 m, respectively. We fall back to FastVGGT, which also handles fisheye recordings but does not provide metric scale on its own.
Association and Consolidation
Trajectory memory clusters observations in 3D, associating them chronologically by label and proximity to an instance’s most recent position. Detections in the same frame remain separate unless they are overlapping duplicates.
A second pass, called consolidation, reconnects tracks that may belong to the same object. It groups tracks with synonymous labels, uses the SAM 3 video tracker to propagate their masks across frames, and merges tracks whose propagated masks agree. Co-occurrence and 3D-distance checks block incompatible merges.
We use consolidation on HD-EPIC and UCS-Bench. On VQ3D, we compare raw, trajectory, and consolidated memories, using a trajectory association radius of 1.0 m and a consolidation mask-IoU threshold of 0.3. On UCS-Bench, merges are blocked beyond a 3D distance of 1.0, measured in scene units when metric scale is unavailable. Even with consolidation, an object may receive a new identity after a long absence or a large move.
Persistence and Contextual Descriptions
Distance-threshold segmentation starts a new segment as soon as an observation lies more than the specified distance threshold from the current resting position. Our persistence rule waits until the specified number of successive observations outside that radius before confirming a new segment. If observations return within the radius before confirmation, they remain in the current segment.
On HD-EPIC, we use the specified distance threshold in metres and the specified persistence count. We use the same values on UCS-Bench, but measure the radius in scene units when metric scale is unavailable. An appendix compares the two rules.
For up to six segments per object, Qwen3.5-9B describes a representative frame with the object outlined. These descriptions record details such as the object’s contents, its supporting surface, and interactions that positions alone do not capture.
In the reported HD-EPIC runs, we first generated descriptions for distance-threshold segments. Each persistence segment then inherited the description of the segment with the greatest temporal overlap. The persistence ablation therefore also changes which descriptions appear in the records; it does not keep all textual evidence fixed. Additional activity descriptions for sampled frames in the UCS-Bench extensions are detailed in an appendix.
Multi-View Refinement
Observing a resting object from several viewpoints helps constrain its 3D position. Each detected box centre defines a viewing ray from the camera. Given camera centres for the observations and unit ray directions for the corresponding viewing rays, we estimate the point that minimizes the sum of squared distances to these rays.
Movement and Stationary-State Analysis
The threshold rule over-counts movement in 64% of these questions at 0.3 m. A larger distance threshold suppresses localization noise but can miss small movements. In the annotations, 19% of moves are shorter than 0.3 m and 50% are shorter than 0.8 m. Meanwhile, the 95th percentile of localization error is 0.84 m. Persistence improves movement counts on questions whose target is already in memory, separating segmentation errors from failures to retain the target. Bias and mean counts are rounded separately.
Stationary-localization questions ask when an object stopped moving and remained still. In the earlier analysis, choosing the answer timestamp closest to a rest-segment start is correct for only 22% of questions, compared with 20% chance. This suggests that sampled observation times do not reliably identify when continuous rest began. Memory does not always hurt this question type, however: the current Qwen and GPT runs show opposite effects.
Development Data and Cost
We developed the segmentation and retrieval rules, triangulation thresholds, and fixture synonym groups on 5–10 pilot videos from a calibration dataset kept separate from evaluation.
In an earlier measurement on a 24 GB GPU, WildDet3D took 0.85, 0.99, and 0.94 seconds per detection at input resolutions of 1802, 960, and 480 pixels, respectively. Each box is processed at a fixed internal resolution, so reducing the input resolution does not consistently reduce processing time. Generating descriptions can dominate total runtime in videos with many objects. These component timings do not measure end-to-end throughput or memory compression.
VQ3D Protocol and Interpretation
We evaluate all 164 queries from the 44 Ego4D clips in the VQ3D validation split. Each target has two 3D annotations, the two benchmark-provided annotations, in the scan coordinate frame. We retrieve an object instance by its name and return its stored world position. We do not compare this with an otherwise identical system that retrieves objects from query crops.
We measure error relative to the midpoint of the two annotations. Conditional success uses the benchmark criterion the benchmark’s specified success threshold.
This threshold is permissive. Combined with excluding unanswered queries, it produces near-ceiling conditional success even when localization errors are several metres. Before making a leaderboard comparison, we still need to compute success over all queries and the official benchmark aggregation metric using the benchmark’s evaluation code. Published results and the remaining protocol differences are listed in an appendix.
We score a query only when retrieval returns a candidate. Raw SAM 3 memory returns candidates for 143 of 164 queries, and trajectory memory for 153. Each variant is therefore scored on its own returned queries, rather than a shared set. A 99% conditional success rate or a median error below one metre does not imply successful localization on 99% of all queries. These results do not establish full-query success or an end-to-end improvement on the official VQ3D task.
VQ3D Implementation Settings
We sample every twentieth frame from a 5 fps grid, giving one frame every four seconds. YOLO-World uses v8x-worldv2 at 640 pixels, with a confidence threshold of 0.25 and at most 12 detections per frame. We lift at most 25 detections per tag and 400 per scene into 3D. Trajectory association uses a 1.0 m radius, and consolidation uses a mask-IoU threshold of 0.3.
The localization tables use GPT-5.4 to select from at most four candidates. We evaluate Qwen only for tagging and judging coverage, so these results do not demonstrate a localization pipeline built entirely from open models. We also evaluate FastVGGT as an alternative source of camera poses.
UCS-Bench Protocol
We sample one frame every four seconds and use the same pipeline as on HD-EPIC: Qwen3.5-9B for tagging, YOLO-World for detection, WildDet3D for 3D lifting, SAM 3 for consolidation, followed by persistence segmentation and segment descriptions.
We use Depth Anything 3 poses when they pass three consistency checks: a chunk-alignment residual of at most 0.5 m, chunk scales within the specified scale range, and camera steps of at most 6 m. Otherwise, we use FastVGGT. Circular-fisheye recordings go directly to FastVGGT. We attempt metric scale recovery using pairs of observations of the same object. This fails for most recordings and all TeleEgo recordings, so their distances remain in scene units.
For each question, we filter the memory at its timestamp. We remove later observations, segments, and descriptions, truncate the current segment and wearer path, and recompute positions. We also withhold object summaries written using all segments. The adapter ranks entries by a score based on question-word matches in names and descriptions, plus the time gap to the nearest observation in seconds. The answerer receives the top four entries and the wearer’s path.
Counting
No use of the memory helps counting, and showing its counts to the answerer costs 6.4 points. We evaluate counts on 257 questions with numeric answers whose target noun appears in memory. Counting tracks gives the correct answer on 11% of questions and is within one on 22%. The median estimate is seven objects, compared with a true count of three. Using the largest number of tracks visible in a single frame improves these rates to 16% and 45%, respectively.
The memory both splits one object into several tracks and misses other objects. On recordings without metric scale, the consolidation step’s fixed 3D-distance threshold blocks 98% of proposed merges. Removing this check reduces the number of stored objects from 6,490 to 4,404 across 12 recordings, without lowering measured overall accuracy. Counting remains unresolved: the two configurations score 33.7% and 31.6% on 95 counting questions, both below the 35.8% blind score. Frames help with action and people counts, but object counts remain near the blind baseline. These results point to instance association and consistent geometric scale during construction as priorities for improving object counts.
Development Pilot and Causal Construction
We developed the adapter on a ten-video pilot with 138 questions. These results are therefore not a held-out evaluation. The first version scored 36.2%, compared with 25.4% for the blind control, but included future evidence. On 57 questions, it retrieved a candidate first observed after the question time, and 43.6% of the displayed path waypoints were in the future. We exclude this gain from our claims.
The timestamp filtering described above removes explicit future evidence from the prompt. On the pilot, the filtered memory scores 38.4% with GPT-5.4, compared with 31.9% blind: a gain of 6.5 points with an interval of the reported interval. Direction labels, relation episodes, and place context provide no measurable additional gain.
However, construction still uses the full recording. The vocabulary draws on frames throughout the video, and pose estimation, consolidation, and segmentation all run offline. Filtering their outputs does not show that earlier memory states are independent of later frames. To establish this, we would need to build every stage using only footage available by each query time and verify that appending footage does not change those earlier states. We therefore describe these runs as timestamp-filtered, rather than as a validated streaming evaluation.
Repeating GPT-5.4 on identical inputs changes the selected option on about a quarter of pilot questions. Our main claims therefore rely on paired comparisons across thousands of questions, rather than these pilot estimates.
UCS-Bench Implementation Settings
Tagging uses at most 80 frames per recording, with a budget of 130 tokens per frame. Detection runs at 1280 pixels with a confidence t
Frequently Asked Questions
What is Ledger? Ledger is an offline system that builds persistent object memory from egocentric video for spatial question answering and named-object localization.
How does Ledger estimate object positions? WildDet3D predicts 3D boxes in camera coordinates, and camera poses place their centres in scene coordinates. Multiple views can help refine positions.
Why does the system use persistence segmentation? Persistence waits for successive observations outside a distance threshold before confirming movement, reducing movement over-counting caused by localization noise.
What are the main limitations reported? Fragmented identities, missing event evidence, and counting errors remain challenges. Offline construction also means timestamp filtering does not establish a validated streaming evaluation.