Deep research, with receipts.

End-to-end walkthrough real recorded run

This is the output of the actual pipeline running on the query “How do frontier coding agents verify the code they write?” Below is exactly what the CLI printed, the sandbox job it executed, and the audit trail it wrote. The recorded run used the deterministic mock shim (TRACE_MOCK=1) — the identical code path that runs live. Update, Oct 25: the live path is now proven: a real Tavily search + extract call was verified through the client the same night its auth header was fixed to Bearer (direct Tavily Researcher account, not the still-pending Nebius Builders credit), and real Nebius Token Factory inference (Nemotron-3-Nano-30B) has been verified through the client. The transcript below remains the recorded mock run.

1 · The command

$ TRACE_MOCK=1 python -m trace.cli research "How do frontier coding agents verify the code they write?" \
    --auto-approve --note "human reviewed"

2 · The brief it produced

# Research brief: How do frontier coding agents verify the code they write?

## 1. Tests first, trust later

Frontier coding agents verify code primarily by executing it: reproducing failing
tests, applying a patch, and re-running the suite until green [1][3].
Self-critique alone is unreliable.

## 2. Published harnesses set the bar

SWE-bench Verified reports top agent pass rates of 74.7%, while SWE-bench
Multimodal drops to 41.9% — a 32.8 percentage-point gap recomputed in our
sandbox [1][2][sandbox:t1].

## 3. Failure modes

LLM self-verification fails when the same model grades its own work without
external execution; calibration is restored by independent test runs in
isolated sandboxes [3].

## Verification

- Agents verify code by executing tests in sandboxes — confidence 0.92 ✅
- 74.7% vs 41.9% pass rates, 32.8pp gap — confidence 0.97 ✅
- Self-critique alone is unreliable — confidence 0.88 ✅

## Sandbox computations

### `t1`
SWE-bench Verified: 74.7%
SWE-bench Multimodal: 41.9%
delta_pp: 32.8

## Sources

[1] SWE-bench Verified — https://openai.com/index/introducing-swe-bench-verified/
[2] SWE-bench Multimodal — https://www.swebench.com/multimodal.html
[3] Why LLM self-verification fails — https://arxiv.org/abs/2409.12345

_Approved by human: yes (human reviewed)_

---
tiers used: planner=nvidia/nemotron-3-super-120b-a12b
            drafter=nvidia/nemotron-3-nano-30b-a3b
            verifier=nvidia/nemotron-3-super-120b-a12b
sandbox backend: local | mock: True

3 · The sandbox job, recomputing the numbers

$ python -m trace.cli sandbox-demo
backend=local exit=0 28ms
SWE-bench Verified: 74.7%
SWE-bench Multimodal: 41.9%
delta_pp: 32.8

In production this code runs in an isolated Token Factory Sandbox microVM after a static safety gate (allowlisted imports, no dangerous builtins, bounded timeout). The audit log records which backend executed every job.

4 · The audit trail (8 events, JSONL)

start    query: How do frontier coding agents verify the code they write?
plan     llm:planner (tier=planner, nvidia/nemotron-3-super-120b-a12b)
search   tavily:search — 3 questions → 3 unique sources
analyze  sandbox:execute job t1: Compare reported pass rates → exit=0 70ms
draft    llm:drafter (tier=drafter, nvidia/nemotron-3-nano-30b-a3b)
verify   llm:verifier (tier=verifier) — 3 sections → claim-level checks
approve  human:approve — "human reviewed"
assemble brief assembled: 3 sections

The eval suite (11 pytest tests) asserts exactly these properties: every section cites sources, every verification carries a confidence score with supporting evidence, sandbox numbers appear in the brief, and the audit trail covers all 7 stages.

High-level architecture

                      ┌────────────────────────────┐
                      │  TRACE CLI / Orchestrator  │
                      │  (trace/pipeline.py)       │
                      └──────────────┬─────────────┘
                                     │
   PLAN ──▶ SEARCH ──▶ ANALYZE ──▶ DRAFT ──▶ VERIFY ──▶ HUMAN APPROVAL ──▶ brief
     │         │           │           │           │
     ▼         ▼           ▼           ▼           ▼
  Nemotron  Tavily    Token Factory  Nemotron  Nemotron
  3 Super   search +  Sandbox:       3 Nano    3 Super
  120B      extract   sandbox:       30B       120B
  reasoning execute   code            reasoning reasoning
  on        (adv      isolated        low       on
            depth)    microVM

   audit trail: every tool call, tier, latency → runs/*.jsonl
   policy: scoped tool permissions per stage (trace/policy.py)

• Tavily powers retrieval: search (advanced depth) per research question plus extract for full source text the verifier checks claims against — also the basis of the Best Use of Tavily prize entry.
• Token Factory Sandboxes execute the generated data-analysis jobs: static safety gate first, then isolated cloud microVM via the account's sandbox endpoint (or TRACE_SANDBOX_BACKEND=local for dev/tests).
• Observability is structural: audit trail records run IDs, input hashes, model IDs, backends, and latencies for every stage.