harness/core
5
0
Fork 1
Code Issues Pull requests 14 Projects Releases 224 Packages 1 Wiki Activity Actions

feat(benchmark): pinned model/reasoning benchmarking for dispatcher workers #312

Merged
che-guevara merged 5 commits from feat/model-benchmark into main 2026-11-10 10:56:41 +00:00
Conversation 0 Commits 5 Files changed 25 +2448 -23
che-guevara commented 2026-11-09 15:34:19 +00:00
Collaborator
Copy link

What this adds

A benchmarking subsystem that measures a model and reasoning level under production
conditions — the same dispatcher, worker runtime, session logs and acceptance commands a
real delegated task uses. Answers "what does this model cost me on this backend when it
does real work", not "how fast does one prompt stream".

  • Worker pin in the dispatcher (execution.ts, dispatcher.ts, dispatcher-runtime.ts):
    Execution.pinned is a host-only field (never model-facing, no dispatch parameter),
    submitScheduled(..., pinned) validates it, and the runner strategy honours the pin
    before rules and the classifier.
  • Strict pin (model-roles.ts): resolveWorkerPin() throws on an unavailable model or
    an unsupported thinking level instead of falling back to the default model. Unlike
    resolveModelRole, a benchmark can never silently measure the wrong model.
  • Suites + fixtures (benchmark/suite.ts): declarative cases, fixture checksums before
    submission and re-checked afterwards, taskIsolation: require-new-session.
  • Runner (benchmark/runner.ts): submits pinned tasks serially, runs acceptance
    commands outside the worker path, collects metrics from the worker's own session logs,
    persists every outcome immediately (a crash cannot lose a finished measurement).
  • Metrics (benchmark/metrics.ts): turns/tokens/reasoning chars/model time/tool calls,
    summed across attempts including runtime restarts.
  • Epoch (benchmark/epoch.ts): build revision + models.json + settings drop-in hashes;
    epochDifferences() refuses cross-epoch comparisons.
  • Report + service + command: /benchmark [list|status|report [suite]|<suite>] starts a
    run detached, delivers per-outcome progress and the median/p95 table afterwards.
  • Settings: benchmark block (enabled/capacity/repeats/caseTimeoutMs), intentionally
    not live-reloadable and not field-mergeable — a running measurement must not have its
    conditions changed under its feet.
  • Docs: docs/benchmark.md.

Why

Three Qwen configurations on the homelab backend (off/medium/xhigh) were compared by
hand and the first attempt produced unusable numbers: a forced-format prompt penalised the
non-thinking run, and one worker whose fixture had vanished reconstructed it from the parent
session transcript (11 of 21 tool calls were a rescue detour). This makes the comparison
reproducible and makes both failure modes structurally visible instead of silently polluting
the table.

Verification

  • Full suite on this branch: 1705 tests, 1704 pass, 0 fail, 1 skipped (npm test, incl.
    build + plugin materialise + runtime wiring contracts). tsc --noEmit clean.
  • 34 new tests: config, epoch, suite, metrics, runner, report, service, command.
  • Metrics verified against real session logs: recomputed independently in Python from
    sessions/dispatch_33d9cbfe-*/ — 6 turns, 34752 input, 5922 output tokens, 9868 reasoning
    chars, 381821 ms model time, 8 tool calls — exact match with the parser output. Corrupted
    lines (bad JSON, record without totalTokens, array line) are ignored, not fatal.
  • Strict-pin tests assert the pin branch runs before rules/classifier and that a rejected
    pin fails the run instead of falling back.
  • Integrity tests assert: a worker editing its own fixture is marked unverified (drift),
    expectations never appear in the submitted task text, a run with a live sibling task is
    refused, timeouts cancel the task and still persist.
  • Command tests cover admin gating, disabled subsystem refuses to submit, detached start with
    progress + final table delivery, and failures reported instead of dying silently.

Known limits (documented in docs/benchmark.md)

  • reasoningChars is a budget proxy, not tokens — the backend returns no thinking-usage data.
  • durationMs is the sum of per-turn model time; wall clock is stored separately so queueing
    and model time stay distinguishable.
  • Workers share the filesystem with the parent and can read session logs. Expectations are
    kept out of the task text and drift is detected, but a motivated worker could still locate
    a suite file on disk — the coordinator verifies completed work independently regardless.
  • One case is not a comparison: use several cases and the repeats, read median and p95.

Not in this PR

No default-config changes (role levels, review thinking, dispatcher capacity) — those are
measurements decisions, and the point of this branch is to produce those measurements first.

## What this adds A benchmarking subsystem that measures **a model and reasoning level under production conditions** — the same dispatcher, worker runtime, session logs and acceptance commands a real delegated task uses. Answers "what does this model cost me on this backend when it does real work", not "how fast does one prompt stream". - **Worker pin in the dispatcher** (`execution.ts`, `dispatcher.ts`, `dispatcher-runtime.ts`): `Execution.pinned` is a host-only field (never model-facing, no `dispatch` parameter), `submitScheduled(..., pinned)` validates it, and the runner strategy honours the pin **before** rules and the classifier. - **Strict pin** (`model-roles.ts`): `resolveWorkerPin()` throws on an unavailable model or an unsupported thinking level instead of falling back to the default model. Unlike `resolveModelRole`, a benchmark can never silently measure the wrong model. - **Suites + fixtures** (`benchmark/suite.ts`): declarative cases, fixture checksums before submission and re-checked afterwards, `taskIsolation: require-new-session`. - **Runner** (`benchmark/runner.ts`): submits pinned tasks serially, runs acceptance commands outside the worker path, collects metrics from the worker's own session logs, persists every outcome immediately (a crash cannot lose a finished measurement). - **Metrics** (`benchmark/metrics.ts`): turns/tokens/reasoning chars/model time/tool calls, summed across attempts including runtime restarts. - **Epoch** (`benchmark/epoch.ts`): build revision + `models.json` + settings drop-in hashes; `epochDifferences()` refuses cross-epoch comparisons. - **Report + service + command**: `/benchmark [list|status|report [suite]|<suite>]` starts a run detached, delivers per-outcome progress and the median/p95 table afterwards. - **Settings**: `benchmark` block (enabled/capacity/repeats/caseTimeoutMs), intentionally **not** live-reloadable and not field-mergeable — a running measurement must not have its conditions changed under its feet. - Docs: `docs/benchmark.md`. ## Why Three Qwen configurations on the homelab backend (`off`/`medium`/`xhigh`) were compared by hand and the first attempt produced unusable numbers: a forced-format prompt penalised the non-thinking run, and one worker whose fixture had vanished reconstructed it from the parent session transcript (11 of 21 tool calls were a rescue detour). This makes the comparison reproducible and makes both failure modes structurally visible instead of silently polluting the table. ## Verification - Full suite on this branch: **1705 tests, 1704 pass, 0 fail, 1 skipped** (`npm test`, incl. build + plugin materialise + runtime wiring contracts). `tsc --noEmit` clean. - 34 new tests: config, epoch, suite, metrics, runner, report, service, command. - **Metrics verified against real session logs**: recomputed independently in Python from `sessions/dispatch_33d9cbfe-*/` — 6 turns, 34752 input, 5922 output tokens, 9868 reasoning chars, 381821 ms model time, 8 tool calls — exact match with the parser output. Corrupted lines (bad JSON, record without `totalTokens`, array line) are ignored, not fatal. - Strict-pin tests assert the pin branch runs **before** rules/classifier and that a rejected pin fails the run instead of falling back. - Integrity tests assert: a worker editing its own fixture is marked unverified (drift), expectations never appear in the submitted task text, a run with a live sibling task is refused, timeouts cancel the task and still persist. - Command tests cover admin gating, disabled subsystem refuses to submit, detached start with progress + final table delivery, and failures reported instead of dying silently. ## Known limits (documented in docs/benchmark.md) - `reasoningChars` is a budget proxy, not tokens — the backend returns no thinking-usage data. - `durationMs` is the sum of per-turn model time; wall clock is stored separately so queueing and model time stay distinguishable. - Workers share the filesystem with the parent and can read session logs. Expectations are kept out of the task text and drift is detected, but a motivated worker could still locate a suite file on disk — the coordinator verifies completed work independently regardless. - One case is not a comparison: use several cases and the repeats, read median and p95. ## Not in this PR No default-config changes (role levels, review thinking, dispatcher capacity) — those are measurements decisions, and the point of this branch is to produce those measurements first. 
Benchmarking a model pair through the dispatcher previously meant editing the
live routing rules and hoping nothing fell back. resolveModelRole silently
returns the main pair when a thinking level is unsupported, which measures the
wrong model and reports it as the pinned one.

- Execution.pinned (host-only) with validateWorkerPin + resolveWorkerPin, which
  fails loudly on an unavailable model or an unsupported thinking level
- benchmark suite format with fixture path checks, serial schedule and a task
  isolation guard so a worker cannot read its own answer key
- benchmark epoch (build revision, models.json, settings dropins) so runs from
  different configuration eras refuse to be compared
- benchmark settings block validated and merged per field like dispatcher

Not yet included: runner, report and the /benchmark command.
Sums tokens, thinking characters, tool calls and per-turn model time from the
assistant records of dispatch_<taskId>_* session directories. Verified against
real worker session logs by an independent recomputation, and tolerant of
malformed lines and records without usage.

durationMs is the sum of per-turn model time, not wall clock: the runner records
wall clock separately so queueing and model time stay distinguishable.
Runs one case at a time through the dispatcher with a pinned worker pair and
keeps every verdict on the harness side: fixtures are materialised with their
checksums, the acceptance command runs outside the worker task text and the
expectation never reaches the worker at all.

- per-attempt worker metrics, so a restarted worker stays visible instead of
  averaging itself away
- fixture drift is reported and blocks verification: a worker that rewrites the
  test did not pass it
- per-case wall clock separate from model time, timeout cancels the task
- run records are persisted atomically after every outcome so an interrupted
  benchmark still reports what it measured
che-guevara changed title from feat(benchmark): pinned model/reasoning benchmarking for dispatcher workers to WIP: feat(benchmark): pinned model/reasoning benchmarking for dispatcher workers 2026-11-09 15:35:11 +00:00 
Summarises each level with median and p95 over the repeats, counts every
attempt (a restart costs real tokens) and refuses to compare runs whose
epochs differ. The service loads suites from <home>/benchmark/suites, keeps
one run in flight, validates before writing and stores every outcome
immediately so a crash cannot lose a finished measurement.

/benchmark [list|status|report [suite]|<suite>] starts a run detached: the
reply is immediate, per-outcome progress and the table follow while the local
workers run. Documentation covers the pin, the integrity gates and what the
metrics do and do not mean.
che-guevara force-pushed feat/model-benchmark from c24646970c to f8851e44d1 2026-11-10 08:13:17 +00:00 Compare 
A first measurement attempt on the live backend exposed four defects in the
benchmark subsystem itself, not in the measurement:

- verified required zero fixture changes, so a coding case that legitimately
  fixed its deliverable could never count. Changes are now split into fixed
  (declared deliverable), tampered (frozen file, disqualifying) and extra
  (added file, reported only).
- nothing proved which model and level actually ran. The worker session log is
  now compared against the pin (pinMatches); a run at another level is excluded
  instead of averaged in. This caught a run submitted as off that ran medium.
- a case that is already solved, or that its own reference fix cannot solve, was
  measured anyway. A preflight now checks the instrument before any worker time
  is spent and reports preflight-failed without submitting.
- grading lived only in the workspace, which workers can read. verifier runs from
  a private directory outside it, with {workspace} as the only bridge, and naming
  it in the task text is rejected.

Suite gains deliverables, verifier and referenceSolution (fingerprinted, so
comparability reflects the grading side too). Docs: the quick-start YAML used
fields the parser never accepted; it is corrected and the case contract, the
preflight and the five integrity gates are documented.

Tests: 1728 pass, 1 skipped, tsc clean.
che-guevara changed title from WIP: feat(benchmark): pinned model/reasoning benchmarking for dispatcher workers to feat(benchmark): pinned model/reasoning benchmarking for dispatcher workers 2026-11-10 10:56:29 +00:00
che-guevara force-pushed feat/model-benchmark from c3fb8dbc79 to ff2d7069d5 2026-11-10 10:56:31 +00:00 Compare
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
harness/core!312
No description provided.