378 Iterations Overnight. One Winner.

We needed the optimal system prompt for Qwen3-coder on our NVIDIA GB10. Not "good enough" — the optimal one. The one that produces 100% CACP compliance with the shortest possible prompt.

We used Andrej Karpathy's autoresearch pattern: modify one thing, test for 5 minutes, keep improvements, discard regressions. Let the loop run overnight.

The Pattern

while True:
    1. Mutate one parameter (system prompt, temperature, etc.)
    2. Run benchmark (PawBench, 5 minutes)
    3. Compare metric (CACP compliance, quality, tok/s)
    4. If better → keep. If worse → git reset.
    5. Log to results.tsv
    6. Repeat

The key insight from Karpathy: fixed time budget + single metric + autonomous iteration. No human in the loop. No "let me think about this." Just brute-force exploration of the parameter space while you sleep.

What We Tuned

The system prompt for Hermes/Qwen3-coder. This is the instruction that tells the model how to format its response after using tools. Without a good system prompt, the model outputs 2,000 tokens of prose instead of CACP structured fields.

The Iterations

The Winning Prompt

You are a coding agent. Use tools to build what is asked.

MANDATORY OUTPUT FORMAT — violation causes automatic rejection:
When you are done using tools, your response MUST be:
STATUS:ok
FILES_CREATED:file1,file2
FILES_MODIFIED:
TESTS:pass:0
BUILD:pass
LEARNED:one sentence

Rules:
- First line MUST start with STATUS:
- Do NOT write any text before STATUS:
- Do NOT summarize your work
- Do NOT explain what you did
- ONLY the CACP fields, nothing else

What We Learned About Autoresearch

Running Your Own Autoresearch

The autoresearch loop is built into servingcard's CLI. Point it at your vLLM endpoint and let it run overnight:

$ servingcard benchmark \
    --model qwen3-coder \
    --hardware nvidia-gb10 \
    --endpoint http://localhost:8000

# Runs PawBench, produces a servingcard YAML
# with your hardware-specific results

The results become a serving card that others can apply. Your overnight experiment saves everyone else the same work.

The Model Comparison Run

We also ran autoresearch across multiple models to find which ones work best for coding agent dispatch on GB10:

Qwen3-coder wins on quality by a wide margin. Smaller models are 2x faster but unusable for production coding dispatch — 29% quality means the agent produces broken code most of the time.