StreamDecisionBench (SDB)
A benchmark for fast, state-conditioned decisions in ongoing applications.
Presentation guidance, live conversation assistance, recording and privacy control, IDE guidance, and assembly assistance all call for decisions that can be acted on immediately. In these applications, a correct answer can become useless or disruptive if it arrives too late.
SDB starts from these application needs. Rather than repurposing tasks designed for general language models or classifiers, it focuses on judgments that a sufficiently capable model should be able to make quickly.
What makes a task suitable?
- Latency matters to the application. The decision is used during an ongoing human or environmental process. It needs to arrive in time to be useful.
- The decision requires judgment. The model must apply explicit rules or criteria to the current state. Semantic similarity between the input and an answer label is insufficient.
- The task should admit an immediate judgment. The target is a reactive decision, rather than a problem requiring substantial deliberation (Astra achieves 99.86% mean untimed accuracy at low reasoning effort).
- The reference decision is unambiguous and executable. Scenario policies are public. Reference decisions are generated by code rather than subjective annotation.
The tasks should be easy for a capable model, but still require judgment based on the current state.
From task design to evaluation
These applications continue while the model responds. New evidence can change the appropriate action. The application may keep applying the model's latest decision until a replacement arrives.
SDB therefore evaluates the decision in force throughout an evolving stream. Untimed accuracy measures whether an answer was correct for the state the model saw. In-force accuracy measures whether the decision being applied is correct for the current state. This includes the time spent waiting for a response and the time an outdated decision remains in use.
The challenge is to make the right judgment quickly and keep it synchronized with an evolving state.
Leaderboard
Table view
In-force accuracy across update intervals
Curves replay recorded answers and latencies at update intervals from 0.5 to 8 s; Wity auto/off each have one pass; other settings average three. The selected range also sets the in-force leaderboard above.