layaAgent · Evaluate

How much of the agent's work does the encoder handle, and how often is it wrong when it does?

Starting…

System 1 Laya typed-decision encoder (421M)

Build

Backend

System 2 small LLM via WebLLM

Model

System 1: . Until you run the suite yourself, this page shows a run recorded from the same models.

Run the task suite

Tasks

Also run System 2 on every step

Each gold step is run with the gold history as context, with tool outputs recorded when the suite was built, so scores do not move with the live APIs. Everything below is recomputed from the stored answers when a setting changes.

Showing

The gate

Score needed on the action:

Score needed on every argument:

p(goal met) needed to stop without asking:

Score

Use Laya's "goal met" to stop (and hold on disagreement) Hold read-only steps when Laya reads the goal as risky

max auto-step error

Picks the threshold (same for action and arguments) with the largest System 1 share on the dev split whose automatic steps stay under the error cap; read the result on the test split.

Score on

Headline at this gate

Baselines

Task success = every step of the task decided correctly (with recorded tools, that is exactly when the agent reaches the gold answer). "You" is a person answering held steps, assumed correct: its cost is the share of steps you have to decide.

Coverage vs accuracy

Each point is one threshold (applied to action and arguments). Right = more steps handled by System 1 alone; up = those automatic steps are right more often. Click a point to use its threshold.

System 1's parts

By category

Every step

layaAgent · Apache-2.0 · The task suite is synthetic (tasks.json): goals, notes and names are made up, tool outputs were recorded from Wikipedia, Wikidata and Open-Meteo when the suite was built. Notices.