ProactEgoMem

Benchmarking Proactive Memory in Streaming Egocentric Video

Ruotong Liao1,*, Meikel Kokowski4,*, Xiaoxiao Sun2, Yongqin Xian3, Laura Bravo Sánchez2, Shiye Su2, Federico Tombari3, Serena Yeung-Levy2, Volker Tresp1

1 LMU Munich 2 Stanford University 3 Google 4 Technical University of Munich * equal contribution

LMU Munich Stanford University Google Technical University of Munich
▶ See the tasks Paper Code Data Leaderboard
1,000QA items+500 in the lab
300clips · 16.0 h+100 in the lab
3×2tasks × levels
13models
The idea · proactive memory

The Proactive Paradigm

Today's assistants are reactive: they answer when spoken to and stay silent otherwise. But what you will want to know later is often gone from view by the time you ask. A proactive assistant watches your first-person stream in the background, decides on its own when something matters, and keeps it in memory before it leaves the frame.

one stream,
two assistants
0:12You turn on the faucet.
3:20
9:05You ask: “How long was the faucet on?”
Reactive
silent
silent
“I can't see the kitchen from here.” the faucet left the frame long ago
Proactive
▶ start state tracking · faucet, on nobody asked
■ stop state tracking · faucet “Faucet tracked until off.” stops when the event ends
“About three minutes, from 0:12 to 3:20.” answered from memory
An illustration of the idea, not a benchmark clip. The proactive row is the Intervention Level: the agent starts a memory tool when the event begins, keeps it running in the background, stops it when the event ends and reports. At the Notification Level it sends one timely message instead, such as “The faucet is still running.” ProactEgoMem scores when it acts, against human-annotated events, and what it kept, through multiple-choice questions asked afterwards.
Streaming egocentric It watches

Your view, one frame at a time. No rewinding and no looking ahead.

Proactive It decides

Nobody prompts it. It picks the moment to speak or act, and stays quiet otherwise.

Co-existent It remembers

It starts, keeps and stops memory tools in the background while you carry on.

Notification → Intervention

Proactivity is a ladder

Two rungs on the same stream, each asking more of the agent than the last.

01

Notification

Say the right thing at the right moment, unprompted.

Watching — nothing worth saying yet.
02

Intervention

Act on it — run the right tool for exactly as long as the event lasts.

Three tasks · two levels · two domains

The annotated data, played back

Pick a task on the left and press play. Each bar is a human-annotated segment on the real timeline; the shaded sweep is the window an agent would have seen so far. The level switch replays the same clip either way.

Results · Daily

Even the strongest models miss the moment

Across proprietary, open-source, video-language, and streaming baselines, memory-backed accuracy averages 47.6 / 23.6 / 24.2% on IC / ST / LT — yet handing the same models oracle memory lifts them to 76.5 / 49.3 / 57.6%, averaged over all thirteen. Acquiring the right memory, not answering from it, is the bottleneck.

Model performance radar, ProactEgoMem Daily track Radar chart of thirteen models on the ProactEgoMem Daily benchmark, macro-averaged over Information Capture, State Tracking and Location Tracking. Axes are start@0.2s, absolute start error in seconds (inverted, so a smaller error sits further out), recall and precision, plus mIoU and QA accuracy with memory at the intervention level only — mIoU is undefined without a predicted interval, and QA has no proactivity level. Each axis has its own range, printed under its name, so the rim is the best of the thirteen on that axis and not 100%, and the rings are quarters of that range. Timing axes are omitted where a model matched fewer than ten events and are drawn at the centre. Every value is also in the table beside it.
Figure 1 — Thirteen models on ProactEgoMem-Daily, macro-averaged over IC / ST / LT. Each axis has its own range, shown under its name: the rim is the best of the thirteen, not 100%. |Δstart| is inverted, so further out is closer to the right moment. Hover an axis for all thirteen values.

Table 1 — Main results, Daily track. Bars scale within each column; the best value in a column is bold. Greyed timing cells rest on fewer than 10 matched events, so start@0.2, |Δs| and σ are not comparable there and are left out of the column best.

The dataset

Grounded in real first-person activity

Two parallel tracks, one pipeline. ProactEgoMem-Daily is 300 Ego4D clips of everyday first-person activity across 39 scenes, from cooking and laundry to carpentry and car repair. ProactEgoMem-Sci is 100 laboratory clips. Annotators correct, delete or add events on the real timeline, and every question is reviewed by at least three annotators. All annotations and QA are new, so none of it can have leaked into pre-training data.

300/ 100clips
16.0/ 6.5 hof video
1,242/ 1,377annotated events
292/ 89distinct objects
1,000/ 500QA items
31.8/ 13.3 smean event length

Daily / Sci

What each task contributes

Bar length is the count; segments split it by task. Science has fewer clips, but its lab protocols are dense: 831 annotated state events in just 35 ST clips.

ClipsAnnotated eventsQA items
Daily
Sci
IC · Information Capture ST · State Tracking LT · Location Tracking
ProactEgoMem-Sci

From daily life to the laboratory

ProactEgoMem-Sci evaluates the proactivity of models on egocentric recordings from biochemistry laboratories.

Dailyassistant

Ego4D · everyday first-person activity

  • 300 clips spanning 16.0 h · 100 per task
  • 1,242 annotated events · 292 distinct objects
  • 1,000 multiple-choice QA items
Scienceco-scientist

FineBio + BioVL-QR + Ego4D lab clips · wet labs

  • 100 clips spanning 6.5 h · 30 IC / 35 ST / 35 LT
  • 1,377 annotated events · 89 distinct objects
  • 500 multiple-choice QA items

A lab clip, played back

Cite

BibTeX

bibtex
@article{liao2026proactegomem,
  title   = {ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video},
  author  = {Liao, Ruotong and Kokowski, Meikel and Sun, Xiaoxiao and Xian, Yongqin
             and Bravo S\'anchez, Laura and Su, Shiye and Tombari, Federico
             and Yeung-Levy, Serena and Tresp, Volker},
  journal = {arXiv preprint arXiv:2610.30089},
  year    = {2026},
  url     = {https://proactegomem.github.io/}
}