What is under test

Same brief, six configurations

Every configuration receives the exact same prompt text, with no system prompt beyond what the provider and the CLI force. The brief asks for one self-contained HTML file, from a playable browser game to a world map drawn from memory or a timed fireworks show. Each recoverable reply is extracted to one artifact and rendered in a sandboxed frame you open yourself. The prompt sent is shown verbatim on each round page.

A configuration, not a model

Six model configurations are reached through two different command-line clients: Anthropic models through the Claude Code CLI, OpenAI models through the Codex CLI. The client is part of what is being tested, so it is named rather than hidden. Both are run in one-shot print mode with customization stripped, but they are not the same instrument:

ANTHROPIC PATH

claude --print --model <model> --effort max --safe-mode --disable-slash-commands --no-session-persistence --tools "" --output-format text The command requests safe mode, disables slash commands and session persistence, and passes an empty tools list.

OPENAI PATH

codex exec --model <model> -c model_reasoning_effort="max" --sandbox read-only --ephemeral --skip-git-repo-check --ignore-user-config --ignore-rules --color never The session is ephemeral, user config and rules are ignored, and the sandbox is read-only. Read-only tools remain available.

EFFORT

Both paths request maximum reasoning effort and record the flag in argv (--effort max, model_reasoning_effort="max"). What is recorded is the request. No provider response confirms the level actually applied, so no such claim is made.

LIVE AUTH

Each configuration is reached through its own provider login (transport kind cli-live-auth). Credentials are never copied into a throwaway config dir: a refresh from a copy would invalidate the real token. Isolation therefore comes from flags plus an empty temp working directory, not from a fake config dir.

EMPTY CWD

Every call runs from a fresh, empty temp directory, so nothing local reaches the context.

RECORDED SHAPE

Each protocol v1 run writes its transport kind, full argv, isolation string, CLI version, timeout, and output cap to meta.json. Pilot records predate those fields, so this page is checkable against the data rather than trusted.

Where the two paths differ

The two clients have carried different constraints, and the retained archive includes failures attributed to the harness rather than the model. Those differences are stated here in full:

OUTPUT CAP

Pilot runs on the Claude path hit a 64,000 output-token ceiling. It caused at least one recorded failure, classified output-limit: a harness cap and not a model failure. The Codex path had no equivalent harness cap. Protocol v1 raises the Claude cap to 128,000 and records the cap on every run.

TOOL ACCESS

Anthropic runs pass --tools "" and get no tools. Codex runs keep read-only tools inside a read-only sandbox. That is not symmetric. It is disclosed rather than smoothed over, and it is one reason a result belongs to a configuration and not to a model.

TIMEOUT

Pilot runs carried no harness timeout, so a stalled call surfaced as the CLI's own error rather than a recorded timeout. Protocol v1 sets an explicit harness timeout and records it per run.

PROTOCOL

Pilot evidence predates the levelled harness. Any released pilot round is shown as-is and excluded from cross-vendor conclusions; protocol v1 is the canonical series.

One shot. No retries. Failures stay visible.

Honesty is the product. Nothing in the pipeline retries a call or repairs an artifact, and a broken result is not removed from a released round. Failures are classified by source where the surviving evidence supports it. Pilot runs predate raw-output persistence, and one pilot failure remains unclassified, so a harness cap or a timeout is never read as a model failing the brief when the evidence establishes its source.

• ONE SHOT One request per model. The first reply is the result.
• NO RETRIES A failed or empty reply is never re-rolled.
• ROUND-LEVEL CURATION Victor chooses which complete rounds to release. Inside a released round, all six first replies are shown; no individual result is selected or dropped.
• FAILURES STAY VISIBLE Broken or empty output stays in any released round, flagged by source.
• NO REPAIR The pipeline never edits, completes, or fixes an artifact.

The rubric

Each entry is scored on four criteria, 0 to 5, each with a one-line justification. Four fives is a perfect 20. Victor built the test and grades it himself, by eye, so the scores are one declared person's judgment rather than a panel. Scores sit next to the artifact on the round page; an unscored round shows an ungraded state rather than a placeholder number.

1. 01 Loads clean boots with no console errors or blank frame 0–5
2. 02 Complete every required feature present, nothing stubbed 0–5
3. 03 Behaves plays or simulates correctly, not just renders 0–5
4. 04 Feel polish, responsiveness, and craft of the result 0–5

What this does not measure

A benchmark is only worth reading if its limits are as loud as its results. These are the ones that matter:

SINGLE SAMPLE

One run per configuration per brief. No repeats, no variance estimate, no confidence intervals. A rerun could reorder the results.

NO ITERATION

One shot means no follow-up turn, no debugging, no agentic loop. That loop is how these tools are normally used, and none of it is measured here.

ONE JUDGE

Victor built the test and grades it. Single evaluator, declared, on a subjective rubric. No blind panel, no inter-rater check.

NARROW DOMAIN

One self-contained HTML file per brief. Nothing here speaks to reasoning, long context, retrieval, writing, or real codebase work.

CONFIGURATION, NOT MODEL

Every result is a model as reached through one CLI with one set of flags. The same model through a different client can produce a different artifact.

PILOT CAPS

Any released pilot evidence ran under the older asymmetric caps described above. Read it as a specimen, not as a cross-vendor verdict.

EFFORT IS A REQUEST

Run metadata proves which effort flag was sent. It does not prove which level the provider applied or that max was best for the brief.

NO GLOBAL WINNER

A round result is a result for that brief, that day, that configuration. It is not a ranking of model intelligence.

Run, review, curate, rebuild

A round is a Node harness fanning one prompt out to every model. Each run writes a meta.json under version control. An artifact.html is written only when the reply contains a recoverable artifact; a hand-graded scores.yaml joins the run records when the round is scored. Publishing rebuilds the static site from those records, so each artifact stays tied to its run.