Praxis

Prototype v0 · real model output on public data

Instructions that can finally see how you're doing.

AI guidance for the physical world. Praxis watches a first-person video stream, works out the task, tracks which step you're on and coaches you through it, catching mistakes with a specific fix.

Watch the clipsBrowse all 48 evaluation clips
Engine output on held-out clip 1_43_0. Praxis said “Use the knife to cut the muffin so you get clean, even halves.” The dataset's annotation for that moment: “Using hand instead of knife”.

Measured on 48 held-out clips

235 minutes of first-person video from the public CaptainCook4D dataset: 8 recipes, 302 annotated steps and 170 annotated mistakes. All labels are the dataset's. Each figure names the results file it comes from.

54%
step accuracy, task known
A frontier model called every 5 s without state: 48%.
results/runs/fused_oracle.json
$0.018
per minute of video
The stateless baseline costs $0.085. Most checks are fast typed questions.
results/runs/fused_oracle.json
73%
recipes recognized with no setup
Median time to recognize: 40 s.
results/l3b/recognition.json
88%
of model calls answered by the fast tier
Jev answers in 151 ms (median). Only 14% of frames leave the local layers at all.
results/metrics.json (call mix, from the run logs)
0.8 s
worst lag running live
A recording fed through the live loop at real-time speed. Local layers: 50 ms per frame; recipe recognized at 22 s.
results/live/live_replay_async.json
77–98%
asked for a better view when the view was bad
Blur, occlusion or the step cropped out, on 60 checks; it was never confidently wrong. The stateless baseline named the wrong step at 50%+ confidence on 12–73%.
results/l5/visual_uncertainty.json

Where it's still weak

  • Mistakes. Strictly judged, 4 of Praxis's 17 mistake alerts named the actual annotated error; the stateless baseline managed 2 of 3. Both catch only a small share of the 170 annotated errors. results/l6/mistake_semantic_fused_oracle.json
  • Recognition. 27% of clips were announced as the wrong recipe or not recognized. One is in the clips below.
  • Hardware. Not tested on smart glasses yet. It runs live on a camera stream; glasses are one more input adapter.

Real engine output

Every clip is drawn from a saved run log, and the rule that picked it is written under it. Development clips were excluded from selection. The two failure cases are picked by rule too.

Catches a mistake, with the fix

“Use the knife to cut the muffin so you get clean, even halves.”

Annotated error in the dataset: Using hand instead of knife

Clip 1_43_0 · chosen by rule: first mistake alert the strict judge rated 'yes' (matches an annotated error) · engineer view

Catches the wrong ingredient

“That looks like a zucchini, not a cucumber. Put it back and grab the cucumber instead.”

Annotated error in the dataset: Use zucchini instead of cucumber

Clip 17_40_0 · chosen by rule: second mistake alert the strict judge rated 'yes'

Follows you step by step

Clip 10_18_0 · chosen by rule: 45 s window with the most annotated step changes the engine followed within 10 s (task recognized by Praxis), ties broken by step accuracy in the window · engineer view

Recognizes the task by itself

Clip 22_24_0 · chosen by rule: highest step accuracy after recognition among clips recognized within 40 s · engineer view

Asks for a better view instead of guessing

“Look back at the sink and show the cucumber in your hands”

Clip 17_43_0 · chosen by rule: first view request

Failure case: a false alarm

“Give the ramekin a quick coat of cooking spray first so the egg won't stick.”

Clip 1_25_0 · chosen by rule: first mistake alert with no annotated error within the matching window

Failure case: the wrong task

Clip 8_50_0 · chosen by rule: first clip whose first announced task was wrong

Running live, in real time

A recorded clip fed through the live loop at wall-clock speed, with model calls running in the background so the loop never waits on the network. Source: results/live/live_replay_async.json

Look closer

Clip viewer →

All 48 evaluation clips. Each page syncs the video with the engine's belief over the task's steps, the event log (verifier calls and coaching actions) and the clip's metrics.

Studio session replays →

3 sessions run through Praxis Studio's live loop on test clips: the coaching HUD, the engineer panel, flags and the full run log.

How it works

Layers of our own run on the laptop on every frame. A frontier model is a swappable verifier, only asked when the local layers aren't sure.

  1. L1
    Perception. SigLIP 2 and V-JEPA 2 features and hand landmarks, computed locally.
  2. L2
    Events. A learned detector for step boundaries and grasps.
  3. L3
    Task graphs. Instructions compiled into typed steps, visual end states, checks and common mistakes. A recognizer picks the task from video.
  4. L4
    State tracker. A Bayesian belief over where you are in the task, updated from local evidence and verifier answers.
  5. L5
    Verifier. Fast typed checks (Jev, calibrated), with ambiguous ones escalated to Claude Opus in the background.
  6. L6
    Policy. When to speak, how much detail, when to flag a mistake, when to ask for a better view.
  7. L7
    Flywheel. Run logs become training records; a local student distilled from the verifier takes routine checks.