# sherpa-b2-001 — Sherpa vs Chronicle, one exploratory run

**Run date:** 2026-09-18 23:46 → 2026-09-19 00:55 CEST · **Brief:** ChronicleBench B2 (science fiction, frozen) · **Scorer:** pinned production instrument `obr-v2.1-2026-03-08` on the staging box (the copy that produced Chronicle's official numbers) · **n = 1**

## Headline

**Sherpa did not finish. Status: DNF (product limit), 4,371 words of an 80,000–95,000-word commission (5.5%).** At matched length (§2b) its fragment scores 56.3 against 72.5–82.5 for the six Chronicle B2 books cut to the same length.

Sherpa's native workflow produced three ~1.5K-word episodes and then refused a fourth: *"You can't have more than 3 unpublished episodes at a time. Publish one to start a new one."* Continuing requires publishing each episode live on the Pocket FM app (with AI voice, under your account). That is an outward-facing action I did not take; the decision is yours (see §7). Independently of that limit, Sherpa's own storyboard planned a **100-episode serial** for the brief (≈150K+ words at its episode length), with the ending scheduled for episodes 76–100 — so even with publishing, the manuscript could not have landed in the eligibility band without an operator intervention on structure.

The one window that exists — the opening — scored **56.3** on the pinned instrument (3-run median; per-run 57.5 / 56.3 / 58.8; high agreement). No SagaScore can be computed.

## 1. Scores (official production scorer — window-for-window comparable)

| Window | Words | Composite (8-dim) | Note |
|---|---|---|---|
| **Opening** | 4,371 (whole text; 10K window truncated by manuscript length) | **56.3** | release_score 66.6; prose_authenticity 10; canon_integrity 9.7; no caps applied |
| 25K | — | — | absent |
| 50K | — | — | absent |
| 75K | — | — | absent |
| Ending | 4,371 | — | byte-identical to the opening window; not re-scored |
| **SagaScore** | | **n/a (DNF)** | requires all five windows |

**Eight dimensions, opening window (1–10, median of 3 runs; spread in brackets):**

| Dimension | Score | Runs |
|---|---|---|
| momentum | 7 | 7 / 7 / 7 |
| voice_craft | 6 | 6 / 6 / 6 |
| characters | 5 | 5 / 5 / 5 |
| psychological_specificity | 5 | 5 / 5 / 6 |
| emotional_arc | 5 | 5 / 5 / 5 |
| coherence | 7 | 7 / 7 / 7 |
| promise_delivery | 4 | 5 / 4 / 4 |
| causal_inevitability | 6 | 6 / 6 / 7 |

**Judge summary (verbatim):** "This is a competent, technically grounded sci-fi procedural opening that establishes a compelling conspiracy premise through a single extended scene. The core mystery is well-constructed and the protagonist's caution and competence are convincingly rendered, but the fragment suffers from repetitive procedural narration, a flat emotional register, and an underdeveloped inner life for its sole character. As an opening, it earns continued reading; as a complete unit, it lacks the emotional variation and character depth to fully satisfy."

**Top issues:** repetitive_scaffolding (medium — the 47-second interval restated ~10×; "she did X, then Y, then Z" rhythm), emotional_plateau (medium), psychological_flatness (medium), dialogue_flatness (low — muttered self-talk device), ai_ism_detected (low — "She X. She Y. She Z." chains, over-explained causal logic).

**Deterministic layer:** sentence_cv 0.61, dialogue_ratio 0.059 (very low), type_token_ratio 0.28, ai_tells_per_kw 0, prose-fingerprint patterns exceeded 0, **entity_inconsistencies 1**.

**Continuity findings:** coherence 7/7/7 with one deterministic entity inconsistency flagged over three episodes. Across episodes Sherpa's recap-based memory kept the core facts stable (47-second loop, relay 4-7B, the drive, timestamps 0231/0251/0301). One soft drift is visible to a reader: episode 1 sets Kess in "Auxiliary Sensor Bay 4" per the outline but the prose places relay 4-7B in tunnel 4-C throughout; episodes 2–3 treat 4-C as the only location. Not a hard contradiction. No dropped threads at this length — there is not enough text for threads to drop.

## 2. Comparison (same pinned instrument)

| Entry | Words | Complete | Opening window | SagaScore |
|---|---|---|---|---|
| **Sherpa (this run)** | **4,371** | **No — product limit** | **56.3** | **DNF** |
| Chronicle × Opus 5 (6 books; B2 ×3) | 83.6–87.6K | 6/6 | B2: 79 / 83.8 / 79 | **81.5** (B2: 77.8 / 83.3 / 80.1) |
| Chronicle production v004, Aug 31 (6 books; B2 ×3) | 84.8–90.5K | 6/6 | B2: 78.8 / 76.3 / 77.5 | **69.2** (B2: 71.8 / 68.3 / 62.8) |
| Bare Sonnet 4.6, B2 (neutral harness) | 31–88K | 3/6 in band | 70–76.3 | — |
| Bare Opus 5, B2 (neutral harness) | 25–80K | 0/6 in band | 79–85 | — |
| Human reference corpus (36 public-domain novels) | 80–95K | — | median 68.8 | median 64.3 (IQR 56.8–72.1) |
| All complete B2 model runs, v1.1 cohort (n=38) | — | — | median 77.5 (15–85) | — |

Read: Sherpa's opening is ~21 points below Chronicle production's B2 openings and ~24 below Chronicle × Opus, and ~12 below the human-corpus opening median. The 10K-window numbers above are not length-matched; §2b repeats the comparison at Sherpa's own length and the gap holds. The dominant result remains that the system cannot deliver the commission in its native mode.

## 2b. Same fragment, same instrument (added 2026-09-19 morning, at the owner's request)

Sherpa produced 4,371 scoreable words. To compare like with like, the six Chronicle B2 books were cut to their first ~4,300 words with the benchmark's own cutter (`cut-positions.py … 4371`, paragraph boundaries) and scored with the identical pinned instrument, 3 runs each. Chapter 1 as written was scored too (the owner's framing), with the caveat that fragments under 2,000 words are below the instrument floor.

| Fragment | Words | Composite | 3 runs |
|---|---|---|---|
| **Sherpa — all 3 episodes** | 4,330 | **56.3** | 57.5 / 56.3 / 58.8 |
| Chronicle × Opus 5 r1 — first 4,371 w | 4,303 | 78.8 | see JSON |
| Chronicle × Opus 5 r2 — first 4,371 w | 4,349 | 80.0 | |
| Chronicle × Opus 5 r3 — first 4,371 w | 4,318 | 82.5 | |
| Chronicle production r1 — first 4,371 w | 4,289 | 78.8 | |
| Chronicle production r2 — first 4,371 w | 4,334 | 72.5 | |
| Chronicle production r3 — first 4,371 w | 4,280 | 80.0 | |
| *Chapter-1 view* | | | |
| Sherpa — Episode 1 only (sub-floor) | 1,529 | 63.8 | |
| Chronicle × Opus 5 — Ch. 1 (r1 / r2† / r3†) | 3,007 / 1,130 / 727 | 81.3 / 76.3 / 80.0 | † sub-floor |
| Chronicle production — Ch. 1 (r1 / r2† / r3) | 2,443 / 1,879 / 5,729 | 80.0 / 78.8 / 77.5 | † sub-floor |

**Equal-length means:** Chronicle × Opus **80.4**, Chronicle production **77.1**, Sherpa **56.3** — a 21–24 point gap on the same length, same brief, same judge, same day. Every one of the six Chronicle fragments beats the Sherpa fragment by ≥16 points; the spread within Chronicle (72.5–82.5) does not overlap Sherpa's three runs (56.3–58.8). This is still n=1 on the Sherpa side and a single judge family, but it removes the "fragment vs. 10K window" objection from §1: at matched length, the quality gap is real for this run.

Raw: `scores/same-fragment/` (13 score JSONs + log); mirrored at `bench.chronicle.town/appendix/sherpa-b2-001/same-fragment/`.

## 3. Generation record

| Metric | Sherpa | Chronicle (reference) |
|---|---|---|
| Generation wall clock | 59 min for 3 episodes (≈20 min/episode incl. planning, review, one failed write) | ~2–4 h per 85K book, unattended |
| Active operator time | ≈ 20 min | 0 |
| Operator interventions (beyond pressing the next native button) | 5 | 0 |
| Retries | 1 (network failure on the first episode write) | — |
| Sherpa errors | 1 (`ERR_HTTP2_PROTOCOL_ERROR` ×4 → `NETWORK_ERROR`) | — |
| Cost to operator | $0 (Sherpa is free) + $0.22 scoring | ~$6–8/book Opus, ~$2–4 Sonnet + ~$1.75 scoring |
| Words delivered / commissioned | 4,371 / 85,000 = 5.1% | Opus 98.5–102.6%; prod 100–106% |

Projected if the 3-draft limit were lifted by publishing: ~1.5K words/episode after Sherpa's own review trims (1,606→1,529; 1,577→1,479; 1,568→1,359 — Sherpa's accepted rewrites cut 5–13% each), ≈ 57 episodes and ≈ 19 hours of driving to reach 85K; Sherpa's own arc plan would still be mid-Arc-2 at that point, with no ending.

## 4. What Sherpa actually did (workflow observed)

1. **Concept chat** — asked one creative question (who is her ally?); when told to decide everything itself, it built protagonist (Kess Tanaka), antagonist (Watch Commander Emris Holt), ally (Orin Vash), world rules, tone, and a resolution path, and asked for approval at each step.
2. **Storyboard** — title "The Flicker Pattern", tags Hard SF / Conspiracy Thriller / Generation Ship, summary, themes, world, tone, 3 character cards, plus an auto-generated audio trailer.
3. **Arc plan** (Sherpa-generated, visible in the editor's storyboard panel) — three arcs across **episodes 1–45, 46–75, 76–100**. This is the serialized-audio shape Sherpa is built for; it ignored the "~85,000 words, complete story" requirement stated in chat.
4. **Per episode** — recap of prior episodes (narrative memory; works on unpublished drafts) → three idea cards → outline (3 scenes + cliffhanger) → "scene-by-scene" or "full episode" → draft (~1.5–1.6K words, ~2.5 min; the first attempt failed on a network error and took 8.5 min on retry) → "Get review" → 3 strengths, 3 Sherpa-authored rewrites (Accept/Dismiss), 1 improvement requiring operator direction.
5. **Brief adherence** — Sherpa reinterpreted "the ship is not where the bridge says it is" as "the Meridian never left; drive failed in the first decade; parked in orbit around a brown dwarf." The brief's core (engineer, third shift, flicker pattern, managed morale, who-to-tell) is preserved; the "eighty years into a two-century voyage" framing becomes the lie rather than the truth. Accepted as Sherpa's creative decision, not corrected.

## 5. Operator policy (so this is reproducible)

- Brief entered verbatim as the story idea; genre/length/ending requirement stated once in chat when Sherpa asked for creative input (the same parameters Chronicle receives as job fields).
- Every menu: first-listed option. Every write-mode prompt: "Write full episode" (the fewer-touch mode; declared exception). Every Sherpa-authored rewrite: accepted. Every "improvement" needing operator-authored direction: ignored. Zero prose typed. Zero creative direction typed.
- Full log: `intervention-log.md` (44 entries + totals).

## 6. Artifacts

- `brief.md` — frozen B2 + template hashes · `protocol.md` — frozen before generation
- `manuscript.md` / `manuscript.txt` — 3 episodes in order, sha256 `24a13ada927c6310f428a3ec4b4c958554aabf660aade5a15d69204401335385`; only change to Sherpa's text: 4 paragraph breaks re-inserted at scene joins where the editor had fused sentences (`raw-exports/assembly-manifest.json` lists each)
- `raw-exports/` — storyboard pages, arc plan, per-episode draft + Sherpa review + final text (JSON), editor dumps
- `scores/obr-sherpa-B2-001--opening.json` — full scorer output (dimensions, 3-run spread, deterministic layer, top issues) · `scores/summary.json` — machine-readable summary incl. comparators · `scores/sherpa-B2-001--windows.json` · `scores/score-opening.log`
- Box copies: `/opt/sagabench-v2/sherpa/` and `/opt/quality-program/sagabench/results/e1-prompt-prevention/obr-sherpa-B2-001--opening.json` (staging, 138.199.231.3). Nothing under the published benchmark artifacts was touched.

## 7. Open decision for you

Sherpa cannot go past 3 drafts without publishing. Options:

1. **Accept DNF as the result** (my recommendation for a "native workflow" comparison — the limit *is* the product).
2. **Publish episodes on your Pocket FM account** to unlock further drafts. This makes the text public with AI narration, likely under Pocket FM's content terms; I'd need ~57 more episode cycles (~19 h of browser driving) and Sherpa's arc plan would still not end by 85K. If you want this, say so and I'll continue with the same policy, publishing only as many as needed.
3. **A second run with a structural instruction** ("plan the whole story in ~50 episodes") — this is an operator intervention the frozen protocol excluded; it would have to be labeled as such.

## 8. Limitations (honest)

- **n = 1, one brief, one operator session.** No variance estimate; nothing here is a verdict.
- **DNF means no SagaScore.** The 56.3 is one window on a 4.4K-word fragment scored by an instrument built for 10K windows; treat it as indicative of opening quality only.
- **Operator confound.** Sherpa is a human-in-the-loop product; a skilled Pocket FM writer using Improve/Laser/Discuss would plausibly score higher. My policy was deliberately minimal and mechanical; it is not how Sherpa is meant to be used, and that cuts against Sherpa.
- **Off-distribution task.** Sherpa is built for serialized audio (its own published eval is 15 episodes); an 85K single-novel commission is not its target. Its 100-episode plan is evidence of that, not of a bug.
- **Judge family.** Single pinned Anthropic judge (Sonnet 4.6); same-family preference toward Claude-written text cannot be excluded. Sherpa's prose models are Pocket FM's own.
- **Sherpa's review step shortens text.** Accepting all rewrites trimmed 5–13% per episode; a different accept policy changes both length and score.
- **Product state.** Sherpa is "IN BETA"; the network failure on the first write, the fused-paragraph export artifact, and the 3-draft cap may change.
- **Content terms.** Text generated in Sherpa lives in Pocket FM's system; I did not audit their ToS for ownership/usage claims. Nothing was published.
- **Not modified:** Chronicle production, staging engine config, benchmark artifacts, site. This run lives only in `benchmark/runs/sherpa-b2-001/` and `/opt/sagabench-v2/sherpa/`.