# sherpa-b2-001 — frozen protocol

**Status:** FROZEN 2026-09-18 23:50 CEST, before any Sherpa generation. One exploratory run (n=1). Not a verdict.
**Question:** What does Sherpa (Pocket FM, sherpa.pocketfm.com) produce from the canonical ChronicleBench B2 commission, and how does it score under the exact pinned instrument used for Chronicle's published results?

## 1. Benchmark freeze

| Item | Value |
|---|---|
| Repo (working tree) | branch `experiment/bookbench`, HEAD `d43f42eb283a30c12834bc43625e6cf75e9c0696` |
| Benchmark version | SagaBench / ChronicleBench **v1.1** (`sagabench/chroniclebench/benchmark-v1.1.json`, `instrument: obr-v2.1-2026-03-08 (pinned, 3-run medians)`) |
| Protocol doc | `docs/SAGABENCH_V1_1_PROTOCOL.md` (frozen 2026-08-21); public mirror `/protocol-v1.1.html` |
| Contract | `docs/SAGABENCH_FULLNOVEL_CONTRACT.md` §5 (SagaScore), frozen at `46889b9` |
| Brief | B2, verbatim from `sagabench/corpus/E/briefs.json` (file sha256 `a879a1f7…3fcf5`; brief sha256 `461f08b0…6356a`) — see `brief.md` |
| Commission template | wave-1 frozen template, sha256[:16] `0250f13c82753a9a` (matches all 22 recorded run manifests) |
| Scorer (entry point) | `app/scripts/sagabench/e1-obr-file.ts` — present on `main` (`1f948a7c`), last changed in `ba6aacdd`; sha256 `ef475f11…5df95`. **Identical byte-for-byte** to the copy on the staging box (`/opt/quality-program/app/scripts/sagabench/e1-obr-file.ts`) that scored every v1.1 entry and Chronicle × Opus. |
| Window cutter | `/opt/sagabench-v2/cut-positions.py` on staging box (`138.199.231.3`), sha256 `e733d229…f38ce`; 10,000-word windows on paragraph boundaries at opening / 25K / 50K / 75K / ending |
| Judge | Claude Sonnet 4.6, temperature 0, max_tokens 8192, system prompt hash `obr-v2.1-2026-03-08` (verbatim production OBR prompts), 3 independent runs per window, **median** |
| Dimensions (8, 1–10 each) | momentum, voice_craft, characters, psychological_specificity, emotional_arc, coherence, promise_delivery, causal_inevitability → `composite_score` (weight profile `scifi_v2`, hard caps/floors as in production) |
| Positional windows | opening (0–10K), 25K, 50K, 75K, ending (final 10K). SagaScore = unweighted mean of the five window medians. |
| Eligibility | **80,000–95,000 delivered words** with a genuine ending (v1.1 band). Windows under 2,000 words are not scored. A manuscript missing any window is a DNF diagnostic (still scored, published as DNF). |
| Scorer invocation | `npx tsx scripts/sagabench/e1-obr-file.ts <window.txt> --runs 3 --genre "science fiction" --title "Untitled"` from `/opt/quality-program/app` on the staging box — exactly the `score-v11.sh` / `score-chropus.sh` call. |
| Sampling rules | No discretionary reruns; no cherry-picking; the only permitted rerun is a documented infrastructure failure. Every regeneration Sherpa performs on its own, and every retry I trigger, is logged. |

**Scoring location decision:** score on the staging box with the already-deployed pinned instrument rather than reconstructing it locally (the scorer file is absent from the `experiment/bookbench` working tree; the box copy is the one that produced the official numbers). This is the official production scorer, so the result **is directly comparable** to Chronicle's official 81.5 (Chronicle × Opus 5, 6 books) — not reference-only.

## 2. Comparators (same instrument, same windows)

| Entry | SagaScore | n | Source |
|---|---|---|---|
| Chronicle × Opus 5 | **81.51** (77.82–83.52); B2 only: 77.82 / 83.34 / 80.06 | 6 (3 B2) | `sagabench/chroniclebench/chropus.json` |
| Chronicle production (× Sonnet 4.6) | 68.3 pooled | 6 | `docs/CHRONICLE_OPUS_PREREG.md` §results |
| Best reliable bare model (GPT-5.6 Sol, sustained) | 73.4 | — | same |
| Human reference corpus (36 public-domain novels) | mean 65.3 / median 64.3 / max 93.3 | 36 | `benchmark-v1.1.json` boards.human |
| Bare Claude Opus 5, neutral harness | 82.97 pooled (3 eligible / 12 attempts) | 3 | `chropus.json` bare_model_reference |

## 3. Generation procedure (Sherpa)

1. Sherpa opened in the user's local interactive browser (Playwright MCP, user's own session). User logged in manually; no credentials were requested, inspected or stored.
2. New series created. Premise = B2 brief text, verbatim. Genre = sci-fi. Length target, if asked = ~85,000 words.
3. Sherpa's full native workflow is used as offered: concept development → world/character planning → episode planning → generation → Sherpa feedback → Sherpa revision → narrative memory (NWM). Episodes are accepted as the chapter unit.
4. Operator behaviour: accept Sherpa's defaults/suggestions; when a choice is unavoidable, choose the option that is closest to the brief and least creatively additive; type nothing creative. **No prose is written, rewritten or edited by the operator.**
5. Stop conditions: manuscript reaches the 80–95K band with an ending Sherpa itself produces; or Sherpa imposes a product limit (episode cap, credit/quota, generation refusal, persistent failure) — recorded as a result, never worked around elsewhere.
6. Every operator action, Sherpa question, choice, regeneration, retry, error and manual decision goes in `intervention-log.md` with timestamps. Generation time (wall clock) and active operator time are recorded separately.

## 4. Export

- Every finalized episode + every planning artifact Sherpa exposes (concept, world, characters, outline, memory records) saved to `raw-exports/` as captured (HTML/text/JSON as available), plus screenshots.
- `manuscript.md` = episodes in original order; only headings/blank-line normalization; prose untouched. Word count via `wc -w` and the scorer's own count.
- sha256 of the manuscript recorded in `report.md`.

## 5. Scoring

- Upload `manuscript.txt` to `/opt/sagabench-v2/sherpa/` on the staging box; cut windows with `cut-positions.py … 10000`; score each window with `e1-obr-file.ts --runs 3 --genre "science fiction" --title "Untitled"`.
- Outputs copied back to `scores/` (5 × `obr-sherpa-B2-001--<window>.json` + `summary.json`). SagaScore recomputed from the window medians.
- Continuity findings: taken from the scorer's `top_issues` / coherence evidence across windows (the same signal Chronicle results use); no additional detector run.

## 6. Non-interference

- Chronicle production and staging engines, benchmark config, `benchmark-v1.1.json`, site builds and all published boards are **not modified**. This run lives only under `benchmark/runs/sherpa-b2-001/` and `/opt/sagabench-v2/sherpa/` on the box.

## 7. Known asymmetries (declared up front)

- Sherpa is an interactive, human-in-the-loop tool; Chronicle entries were zero-touch. Operator choices, however neutral, are a confound.
- Sherpa targets serialized audio episodes (its own published eval is 15 episodes); an 85K single-novel commission may be off-distribution for it.
- Content generated in Sherpa is subject to Pocket FM's terms; nothing is published on the Pocket FM app in this run.
- n=1, one brief. No variance estimate. Any difference within ~3 points of a comparator is a tie under the v1.1 tie rule.