Tried to one-shot a small distributed task execution engine based on rendezvous hashing. Spent a few days setting up a simple coding harness, with quality checks, planning and review, and detailed coding and testing guidelines. Then I spent some 30 minutes discussing the task with the planning agent. The agents took over and produced the complete solution in some 30 minutes. It took me another two days to clean it up (with prompting). Excluding harness setup time, I'm not sure I would have been any slower if I implemented the whole thing by hand. I'm certain I'd be much faster if I wrote this with multiple smaller prompts. The majority of the problems were in the tests. The agent ignored the instructions to test observable behavior. Instead of having a test like "when a node is removed from the cluster, the remaining nodes take over its work", the test was more like "when a node is removed from the cluster, internal events X, Y, and Z happened in the given order". Basically, the tests were checking if the code is working as the code is written. This didn't give me sufficient confidence that the thing is doing what it's supposed to do. Through a lot of microprompting I overhauled the entire test code, and in the process simplified the implementation, reduced LOC, increased test coverage, and made the tests focused on the actual behavior. Some shower thoughts based on this small experiment: - Next time I'll just try to supply the code sketches of the tests I want, and let the agent fill in the blanks. Hopefully that will produce better results. - Despite all the quality checks and multiple agentic reviews, the code still suffered from a lot of "small papercuts". Basically, as with human-language prose, the models overcomplicate the coding prose and make it overly verbose, which makes the code harder to understand. And it still needs to be understood, at the very least by the agents. It seems neither quality checks nor agent reviews catch such issues. - Given all this, especially the shape of the tests (and my past experiences with generated tests), I remain unconvinced about giving the agents full autonomy and not reviewing the code myself. At the very least I'll be very scrupulous about the tests. - I remain committed to the "work in small chunks" approach. When the chunk of work is small I can provide a more precise description of what I want, I can more quickly review the code, catch possible issues earlier, and steer the agent before it strays too far. - For the record I still like agentic development. I barely write any code by hand these days, and still enjoy the experience. I just remain skeptical about the "one-shot everything, review nothing" philosophy which seems to be more and more prevalent. n.b.: this text has been written completely by hand 🙃
