The Problems of AI Coding
The move from compilers to AI coding brings with it two main problems: non-determinism, and the loss of our mental models of software. The first problem means that prompt outputs aren’t stable, and in long-term agentic work the compounding instability leads to chaotic systems with highly divergent outcomes. As agent errors are often not mechanistically detectable, the system needs to act as a feedback loop with the agent being the detector and corrector, aided by mechanistic detection and correction, like compilers and tests, where possible....