Design
CEO
Frames the requirement
Uncertainty-aware multi-agent software development
UA-ChatDev gives software agents a reliability reflex.
Our framework extends agent collaboration with uncertainty quantification. Every development phase can measure response reliability and trigger targeted knowledge retrieval before an uncertain answer travels downstream.
Role-based collaboration Phase-specific thresholds Active retrieval
Live collaboration trace uncertainty gate active
01 / Our starting point
We extend ChatDev with a missing reliability layer.
ChatDev is the prior foundation: a virtual software company where specialist agents move a requirement through design, coding, review, and testing. UA-ChatDev is our extension. It adds a principled uncertainty check to every agent interaction so the pipeline can recognize when its own knowledge falls short.
Plan
CTO
Shapes the architecture
Code
Programmer
Builds the implementation
Review
Reviewer
Finds concrete failures
Test
Tester
Validates the result
02 / The research gap
A fluent answer is not always a reliable answer.
Existing role-based pipelines improve coordination, but still act on every response with equal confidence. A plausible-looking mistake can move from design to code to testing before the system realizes something is wrong.
agent_response.pyunchecked
Without a reliability check, the pipeline cannot distinguish knowledge from a confident fabrication.
03 / Our uncertainty module
UA-ChatDev turns uncertainty into action.
UA-ChatDev measures token-level confidence after each interaction, compares the resulting uncertainty score against a calibrated threshold for that development phase, and retrieves external context only when the model needs it.
Token confidence
importflasksocketmethod
Log-probabilities reveal where generation becomes unreliable.Phase check
u > tauphase
Each phase uses its own reliability bar.Targeted retrieval
Seek external context
Relevant knowledge is supplied before work continues.04 / The UA-ChatDev workflow
Reliable collaboration, phase by phase.
Our framework preserves the useful role-based workflow while adding the ability to recognize uncertainty, retrieve external knowledge, and keep weak assumptions from propagating into later development phases.
Generate
A specialist agent produces the next response in the development chain.
Measure
Token log-probabilities become an uncertainty score for the current phase.
Retrieve when needed
If uncertainty crosses the calibrated threshold, external context is requested.
Continue stronger
The retrieved knowledge flows forward into the next software-development turn.
Interactive research prototype
Watch the factory think in real time.
Follow live agent exchanges, inspect token uncertainty, and see retrieval activate when a phase crosses its confidence threshold.
Launch the live factory