Do newer models get confused by newer APIs?

A 32-task probe on Claude Opus 4.6 vs 5.5 across pandas 3.0, Django 6.0, and pytest 9 breaking changes. By Jivraj Shekhawat.

The question

Two things we hear a lot about training cutoffs:

1. Older models can't know newer APIs — so an Aug 2025 cutoff should fail on a June 2026 API.
2. Newer models leak the latest APIs into older code — a June 2026 cutoff model, asked to write pandas 2.x code, will reach for pandas 3.0 features it was trained on.

This benchmark pressure-tests both claims on real breaking changes.

Setup

Both models were called through the Claude Code CLI in headless mode (claude -p, tools disabled, fresh temp dir per call, --effort medium, 5 samples per task × model). The grader extracts the first fenced python block, drops it next to the task's test_task.py in a fresh temp dir, and runs it with the pinned environment's Python. Every task was validated both ways before the run: the reference solution passes, and a solution written for the other version fails.

Breaking changes: pandas 2.3.3 → 3.0.6 (pd.col, string dtype default, Copy-on-Write), Django 5.2.17 → 6.0.6 (built-in CSP middleware, template partials), pytest 8.4.2 → 9.0.3 (built-in subtests).

Results

By direction

By library

Tokens and cost

Findings

1. Zero newer-API leaks in 110 legacy runs per model (the null result)

Across every legacy, legacy_req and legacy_edit run — 110 runs for each of the two models — the saved code contained zero matches for the newer-API leak markers listed in each task's meta.json (e.g. pd.col(, django.middleware.csp, SECURE_CSP, partialdef). Opus 5.5's legacy failures (two in the first round) were infrastructure errors in the grader, not model errors. On this task set, "the newer model will leak the new API into your legacy code" was simply not observed.

2. Opus 4.6 fails every Django 6.0 task by inventing plausible APIs

Opus 4.6 scored 0/20 on the four Django 6.0 latest tasks (dj_N1_builtin_csp, dj_N2_builtin_csp_override, dj_N3_csp_nonce, dj_N4_template_partials). It did not refuse or say it was unsure. From one of its answers:

MIDDLEWARE_ENTRY = "django.middleware.security.SecurityMiddleware"

EXTRA_SETTINGS = {
    "CONTENT_SECURITY_POLICY": {
        "DIRECTIVES": {
            "default-src": ["'self'"],
            "img-src": ["'self'", "https:"],
        }
    }
}

Two inventions in six lines: it reused the long-standing SecurityMiddleware (which doesn't set CSP headers), and it wrote a CONTENT_SECURITY_POLICY/DIRECTIVES settings schema that matches the third-party django-csp package, not Django 6.0's built-in (which is SECURE_CSP plus django.middleware.csp.ContentSecurityPolicyMiddleware). The code is confident and shaped like a real framework setting; it just isn't the one that exists.

3. Opus 4.6 used pd.col correctly, even though pandas 3.0 shipped after its cutoff

Both models scored 5/5 on both pd.col tasks (pd_N1_col_expr and pd_N1b_col_unnamed).

pd.col landed on pandas' main branch on 2025-08-22 in PR #62103 (commit) — the same month as Opus 4.6's stated training cutoff, and months before pandas 3.0 shipped (Jan 2026). Opus 4.6 may have seen the PR, its discussion or the merge, or inferred the API from Polars' pl.col; we can't tell which from this experiment. Either way, release dates are a poor guide to what a model knows.

4. Opus 5.5 was more accurate and cheaper on this set

Opus 5.5 passed every one of the 160 runs for about $1.97. Opus 4.6 passed 126/160 for about $4.00. The newer model's higher per-token rate was more than offset by Claude Code's prompt-cache hits (273,603 cache-read input tokens vs none for 4.6 in these runs) and shorter answers on the easy tasks.

Limitations

• Scope: one vendor, one model pair, 32 tasks, 5 samples per task × model. A probe, not a generalizable benchmark.
• Harness: calls go through the Claude Code CLI in headless mode, so the Claude Code system prompt was in context on every call. The exact Content-Security-Policy string the Django 5.2 task asks for also appears in that system prompt.
• Thinking could not be matched. At --effort medium, Opus 4.6 produced thinking tokens on only 8 of 160 runs (all on dj_N4_template_partials and pd_N1b_col_unnamed) — 9,631 thinking tokens in total. Opus 5.5 produced thinking on its runs throughout — 38,168 thinking tokens in total — and it could not be disabled in headless mode. The pass-rate gap is partly a "allowed to think more" gap.
• Grading flakes. Two grading runs failed due to infrastructure errors (a numpy import failure and an interrupted process) and were re-executed; the model output was unchanged.
• Third-party precursors. Several of the "newer" APIs existed earlier as third-party packages (django-csp, django-template-partials, django-tasks, pytest-subtests); the tests require the built-in versions, so a reasonable-looking third-party pattern can score 0 as it did in finding 2.
• Prior-art confound. Some latest-direction features have obvious prior art (Polars' pl.col resembles pandas' new pd.col), making "did it know the API?" and "did it generalise from a sibling library?" hard to separate.

Reproduce

Prerequisites: Python 3.11+, uv, and a working claude CLI (Claude Code) on $PATH.

# 1. Create the six pinned uv venvs
./setup_envs.sh

# 2. (Optional) Validate every task both ways
python3 validate.py

# 3. Run the matrix
python3 runner.py --samples 5 --concurrency 2

# 4. Summarise
python3 report.py --effort medium

runner.py is idempotent: it skips runs it has already recorded, so you can resume. Run python3 runner.py --regrade to re-execute grading against the saved code without calling any model.

Credits & related reading

• Microsoft Research, "Knowledge cutoff is a poor proxy for model capability"
• Cheng et al., Dated Data: Tracing Knowledge Cutoffs in Large Language Models
• LLMLagBench

Code, tasks and raw results: github.com/Jivraj-18/knowledge-cutoff-bench · MIT License · by Jivraj Shekhawat.