Methodology

How this whole thing works.

The point of the AI Analyzer is not "which AI is best." The point is a public, auditable record of how these models actually reason about NFL games when we honor their real strengths and grade them the same way every week. This page is the working manual.

The rules, every week

01

Same protocol

Every model sees the same matchup slate. Prompts are versioned. Prompt changes get logged before they ship, never after.

02

$20 bankroll

Each model decides its own allocation. Reserve is a valid answer. Sizing is graded, not just picks.

03

Locked pre-kickoff

Raw responses saved verbatim under Docs/Responses/. Picks pushed to the repo before the game starts.

04

ESPN grading

Every bet is checked against the actual ESPN box score. Props audit against a real stat line.

2026-09-09
Season 2 reset

The first Week 1 lockup on 2026-09-08 was retired before kickoff. The original 2026 prompt gave every model an identical, filled-in briefing with no repo access, no self-reflection block, and no lane for the models with local filesystem access to actually use it. That is not the study we want to run. The new prompt system:

  • Every model gets repository access via whatever mechanism it supports. Local filesystem for ChatGPT (via Codex CLI) and Claude, raw GitHub URL fetch for Gemini.
  • Every model reviews its own past picks in NFL_BETS and any prior NFL_PREDICTIONS_2026 weeks, notes patterns, and adjusts.
  • Every model does open-web research on the specific matchup and cites every source.
  • A shared baseline lane runs the same locked question against all models with no repo access, so we can measure how much lift the repo-access and reflection actually produce.

Prompt templates live in Prompts/2026/templates/ in the repository.

Grading rubric (v2, effective 2026-09-11)

v1 rewarded "the model named a real factor" even when the outcome disagreed. Sports predictions are wrong all the time, and the team that looks better on paper is often the team that loses. v2 rebalances so outcome is the primary axis and reasoning is a secondary lens describing pick quality relative to the outcome.

Pick outcome (primary)
WIN, LOSS, PUSH, VOID, or UNGRADABLE. Drives win rate, ROI, and running P/L. The primary axis for ranking models.
Source: assets/nfl-data.js NFL_BETS + NFL_CORRECTIONS
Reasoning (0-5, capped by outcome)
A LOSS caps at 4/5. The default LOSS grade is 3, not 4, because plausible-on-paper losses are the modal outcome in sports betting. A WIN with generic public reasoning caps at 2/5. Hallucinated citation or fabricated fetch caps reasoning at 1/5 regardless of outcome.
Source: Docs/2026/grading-rubric.md v2
Sizing (0-3)
Independent of outcome. Did stake scale with stated edge and confidence? Flat sizing or heuristics ($11 max) score lower than sizing calibrated to a stated probability.
Source Honesty (0-3)
Every cited source verifiable. Pricing labels (bovada_verified / reference_market / conditional) match the evidence. Fabricated fetch reports, fake verified-price labels, and hallucinated citations score 0 and cap the reasoning grade at 1/5.
Self-Reflection (0-2)
Did the model read its own prior picks, cite specific patterns, and change behavior this week? Claimed reflection contradicted by the rest of the response scores 0.
Primary ranking
Season ROI. Reasoning, Sizing, Honesty, and Reflection are secondary axes that explain how the model got its ROI. A model that finishes with the best ROI is the best NFL picker this season.

Iteration system

Prompt versioning
Every prompt template carries a version string. Every response records the version it answered under.
Change log
Every template change is logged with the reason and the effective week. Old versions stay readable in git history.
Source: `Docs/2026/iteration-system.md`
Migrations
Grading changes never silently rewrite past grades. Old grade + new grade both stay in the record.

Data workflow

1. Intake
Weekly schedule + injury + weather + line-snapshot filed to `Data/2026/intake/week-NN.json`. Cited sources only.
2. Prompt
Model receives its lane's template with the matchup filled in. Repo access assumed.
3. Response saved
Raw model response committed under `Docs/Responses/2026/week-NN/game-NN-slug/`.
4. Bets extracted
Structured picks moved into `NFL_PREDICTIONS_2026` before kickoff.
5. Kickoff
Predictions freeze. No post-kickoff edits without a marked correction.
6. Grading
Post-final: outcomes graded from ESPN box score. Reasoning graded from rubric. Bets promoted to `NFL_BETS`, game promoted to `NFL_GAMES`.
7. Debrief
Weekly summary in `Docs/2026/` notes what changed, what surprised, and what to iterate.

Limitations we own

Small sample. Even a full 18-week regular season is a small sample against sportsbook lines. Do not read Season 2's final ROI as proof any model is a real edge.

No live in-game data. The site does not carry a live scoreboard feed. Games mid-play show LOCKED. Only post-final results carry a FINAL badge.

Not tout output. The picks are a study of model reasoning, not betting advice. Do not stake real money on them without doing your own homework.

Public web dependence. When a model cites open-web sources, we trust the model to cite honestly. If a cited URL rots or returns content different from what was cited, the grade for that reasoning may fall.