Mean code coverage
Maintained across submissions in both pilot rounds.
Team 30978 / 2025-2026
NIST GenAI Code Challenge (Pilot)
An automated test-generation pipeline for producing reliable Python pytest suites from natural-language specifications.
Lead author and system architectTemitayo OgunsusiRound 1 (Pilot 01) and Round 2 (Pilot 02)
NIST evaluation task two pilot rounds
Given a Python function specification:
H function headerD behavior description
Generateoptimal pytest suitecorrect · comprehensive · executable
Best correct-test rate
Reached in Round 2 after targeted prompt refinement.
Best CI1 / CIT detection
The strongest result for detecting both fault categories.
01 / The challenge
Reliable tests need more than code that compiles.
NIST evaluated AI-generated unit tests for elementary Python functions. Each suite had to run directly in pytest, cover meaningful execution paths, and catch incorrect implementations without introducing faulty assertions of its own.
02 / The system
A fully automated generation and quality pipeline.
The system turns a NIST specification into a validated submission entry through five controlled stages.
Prompt construction
Combine the Python specification with a fixed or custom quality prompt.
Test generation
Use GPT-5.1-Codex to produce a candidate pytest suite.
Output extraction
Isolate runnable tests from the model response using standard delimiters.
QA pipeline
Validate syntax, repair imports, and remove accidental implementations.
Submission packaging
Assemble schema-valid JSON entries for NIST evaluation.
03 / Quality assurance
Three checks prevent silent submission failures.
Generated code is treated as a candidate, not a finished artifact. Every suite passes through lightweight post-processing before packaging.
Syntax validation
Parse every generated suite with Python AST. Malformed code triggers a retry with the failure context.
Import correction
Verify that pytest and the function under test are imported so each suite runs in isolation.
Implementation removal
Strip accidental copies of the target function before they can distort coverage measurements.
04 / What improved
Round 1 exposed the gap. Round 2 targeted it.
Coverage was already near-perfect. The harder problem was generating tests that were logically correct and precise enough to detect faults. Dry-run analysis turned recurring failures into focused prompt and QA refinements.
Best correct-test rate
Round 1
48.98%
Round 2
+31.46 percentage points through targeted refinement80.44%
Exception precision
Require the exact exception type described by the specification.
Boundary reasoning
Explicitly cover edge cases and validate expected values before output.
Runnable output
Repair missing imports and reject malformed generated test suites.