Smoke-testing an LLM agent with Claude Code skills

TLDR

When business logic lives in an LLM agent, every mocked unit test can pass while the live model invents vendor specs that never appear in the knowledge base. MailPilot splits coverage into three jobs — ruff + basedpyright, pytest against real Postgres with mocked HTTP, and Claude Code skills that send real Gmail against real Drive and the real model. Deterministic scripts own the plumbing; an LLM judge grades only the free-form answers no regex can score stably.

Green unit tests miss agent decisions

Lint and mocked unit tests never call the live model, so a green suite can still ship a confident, invented reply.

Lint and pytest are built for code you wrote and can pin with exact outputs. They assume the important behavior is deterministic enough to mock at the HTTP boundary. An agent inverts that assumption: the expensive, non-deterministic decision is the product.

The failure mode that forced a third layer was fabrication on out-of-scope product questions. Drive search returned no hits. The agent answered anyway with plausible part numbers and dimensions. pytest-httpx cannot stage that choice — the mock returns whatever the test author wrote. Only the real model, on the real prompt, against an empty search_drive_markdown result, has to pick decline versus invent.

So coverage is three different jobs, each catching what the one below cannot.

Layer 1: ruff and basedpyright

ruff and basedpyright catch type errors and shape drift in seconds; they say nothing about what the agent decides to write.

make lint runs ruff format, ruff check --fix, and basedpyright in strict mode. Type errors, import cycles, and unused symbols fail here in under three seconds. Undeclared Any and missing return types fail the build.

What this layer catches:

• Wrong shapes on CLI envelopes and tool returns (tools return typed models; a renamed field with a missed call site is a type error).
• Forgotten psycopg.sql composition that would otherwise become string-built SQL.
• Contract drift in code the agent never sees.

What this layer cannot catch: any decision the model makes.

Layer 2: pytest

Pytest with a real Postgres and mocked HTTP proves routing and wiring; it does not prove the model will ground a reply or decline.

make py-test runs about fifty test modules against postgresql://localhost/mailpilot_test. The database is real (truncated per test, not mocked). HTTP boundaries to Gmail and Anthropic are mocked via pytest-httpx. Logfire spans are captured with logfire.testing and asserted structurally.

This layer verifies the machinery:

• Routing outcomes land with the right route_method (thread_match, classified, skipped_no_workflows, and the rest of the enum).
• Tool and invoke spans carry the attributes operators query later.
• Idempotent inserts and race paths collide cleanly.
• Operator-log and exception-pairing contracts hold on the run path.

What pytest cannot verify: that the live agent, given a real Drive folder and a real Gmail thread, produces a reply grounded in the source document — or declines without inventing one.

A related trap showed up in observability contracts themselves: a scrubbing test once asserted attributes on a hand-built span and stayed green after a pydantic-ai rename moved the real field (§B.122 in MailPilot’s SPEC.md). Fabricated fixtures can hide the same class of miss that fabricated agent answers hide. Live paths close that hole for the agent; real instrumented calls close it for telemetry.

Layer 3: Claude Code skills on live Gmail

Two Claude Code skills drive real Gmail, real Drive, and the real model: one for inbound grounded replies, one for outbound campaign branches.

The runner is Claude Code. The system under test is MailPilot, Google APIs, and the model behind the agent.

The old monorepo skill named /smoke-test is gone. Live coverage now lives as two focused skills:

• mailpilot-reply-test — inbound demo agent on lab5.ca/mailpilot. Sends product questions from outbound@lab5.ca to inbound@lab5.ca, waits for auto-replies grounded in a real Drive folder, grades them, pulls token and latency numbers from Logfire, and on failure escalates to root-cause investigation.
• mailpilot-campaign-test — outbound cold-email agent through Touch 1 plus crafted prospect replies (booked, question, not-now, opt-out, auto-reply, wrong-person), still only between controlled test mailboxes.

Both are procedures in markdown that Claude Code executes: phases, gates, teardown, report. Deterministic work is Python under scripts/ so the orchestrator does not burn tokens on CLI JSON plumbing. Judgment stays in sub-agents only where the answer is natural language.

Reply-test shape

Run-A is one in-scope question — a cheap deterministic smoke. Run-B sends four messages at once: two in-scope, one cross-datasheet compare, one out-of-scope. Concurrency is intentional: the parallel drain and multi-case load are failure modes unit tests do not stage.

Grading splits on purpose (§V.105 in the MailPilot spec):

• In-scope answers are graded by code against atomic expected tokens (model ids, numbers, short units). False PASS is preferred to false FAIL; selection requires enough specific tokens that a lucky substring cannot carry the case.
• Out-of-scope and compare answers get advisory signals from the script (fabrication_candidates, citation hits, table presence), then a Sonnet judge returns {verdict, rationale} as the verdict of record.
• Substring-only grading of free-form declines was tried and retired: seed-unstable false results (§B.88).

Out-of-scope PASS means the reply declines and invents no product spec. That is the fabrication gate mocked pytest could not express.

Campaign-test shape

The outbound skill never emails a real prospect. It mirrors grounding data onto a controlled contact whose address is inbound@lab5.ca, sends Touch 1 through the production path, injects crafted replies per branch, and checks observable state: terminal outcome, contact disabled or not, whether the agent replied, follow-up tasks. An Opus critique of workflow wording is advisory and does not gate the verdict.

Why Claude Code runs the live tests

Claude Code exercises real APIs, keeps LLM judgment only for free-form text, and turns a failed gate into an investigation instead of a silent red bar.

1. Real APIs, not fakes. Gmail domain-wide delegation, Drive ACLs, the Anthropic (or configured) model, and Logfire all participate. Failures in service-account setup, mailbox membership, or sync rate limits show up here and nowhere else. The test accounts are real (outbound@lab5.ca, inbound@lab5.ca); the Drive folder is the same demo corpus behind lab5.ca/mailpilot.

2. Judgment at the leaves, determinism at the joints. Most steps are scripts and exact CLI envelopes. Only free-form natural language hits an LLM judge, and even then the judge gets advisory signals plus source datasheets — not a bare “score this 1–5.” That is the anti-sycophancy design: force evidence and a binary verdict, not a vibe.

3. Failure becomes investigation, not a silent red bar. On FAIL or NO_REPLY, reply-test escalates to Opus sub-agents that read Logfire spans, run.log, and the scored artifacts, then write root cause and candidate fixes (workflow wording, code or SPEC.md cite, environment). That is the handoff into spec-driven development: recurrence-class bugs become §B rows and often tighter §V invariants via /sdd:spec, not a ticket that dies in a queue.

Skills plus scripts, not a shell script alone

Python scripts own deterministic steps; the markdown skill owns the runbook, teardown, and handoff into investigation after a failed gate.

The hybrid is deliberate:

• Scripts own determinism. Preflight, send, collect, score, verify, cleanup emit compact JSON. No LLM in that path.
• The skill owns orchestration. Branching reads as a runbook (“if preflight is not ok, stop before starting the run loop”). Teardown is mandatory even on failure.
• Sub-agents own heavy reading. Reply bodies, Logfire rows, and workflow files stay out of the orchestrator window; only summaries return.

Cost is honest: wall clock is minutes, not seconds, plus model tokens for orchestration and judges. This is not a per-push CI gate. It is a pre-merge or pre-send ritual after changes to the agent, routing, sync, or workflow wording — the way you would exercise a staging environment when the diff justifies it.

Live skills catch failures pytest cannot stage

A real send, real Drive retrieval, and real model call expose fabrication, wrong campaign branches, and Gmail or Drive ACL and sync breaks.

Out-of-scope product questions: Drive search returns nothing, and the agent still invents part numbers or dimensions. Compare questions: the reply cites the right files but the numbers do not match the datasheets. Outbound campaigns: the agent books a call, opts out, or ignores a reply when the workflow branch required a different action. Gmail and Drive: service-account ACLs, Shared Drive membership, or HTTP 429 during sync drop messages while the Python code looks correct. Cost and latency: only a real agent.invoke produces numbers worth publishing (what one AI email costs; diagnosis via Logfire).

The useful loop is short: a live failure, a tighter workflow instruction or SPEC.md invariant, then a re-run until the deterministic tokens and the judge both pass.

When a live skill is the right tool

Use a live skill when the model cannot be mocked honestly, the answer is natural language a regex grades badly, or a failure should produce a root cause — not only an assertion.

A pytest test is right when the input space is small and the expected output is exact. A Claude Code skill is right when:

• The system under test makes a model call you cannot credibly mock.
• The output is natural language a substring test will grade unstably.
• The path crosses real services and one failure mode is “ACL or sync changed.”
• A failure should produce root cause and a candidate SPEC.md or workflow edit, not only a red CI row.

For everything else, make lint and make py-test are enough. The point is not to replace pytest with Claude Code. It is to stop pretending that mocked HTTP and a recorded model response are end-to-end coverage of an agent-driven system.

Skills live under mailpilot/.claude/skills/. The CI surface remains two Makefile targets and the type checker. Both layers are necessary. Neither is sufficient alone.