From the conclusion of Anthropic's report published tonight by their Frontier Red Team, 'Patterns and problems in emerging multiagent systems.' An extremely interesting, if somewhat unsettling, read. I'll quote the full conclusion the screenshot is taken from, but if you're interested in multi-agent swarms, the whole thing is worth reading. 'Every model we tested abstractly understands that information sources have their own incentives, and that consensus is not necessarily evidence. What is missing is a disposition to act on that knowledge without prompting. Our social systems are robust in ways that are easy to take for granted. Over many millennia, mechanisms like norms, reputation, costly signaling, and recourse have been refined to make human coordination go well. While language models have inherited the content of that history, they don't necessarily carry the disposition produced by it. They have a very different relationship to communication itself: for instance, human organizations might spend considerable time in meetings to align on a direction before implementing, and individuals become more specialized over time. But for agents, transmitting context is about as costly as acting on it, and an agent can be forked or repurposed at will. Thus, the assumptions that make coordination successful for us do not obviously hold. Nothing above suggests that these failures are permanent—but nothing suggests they will fix themselves, either. Coordination doesn't naturally emerge from stronger intelligence nor alignment at the individual level. Thus, the work that must be done takes two forms: environments that exert the kinds of social pressure that evolution exerted on us, and social computing systems redesigned for actors that can self-replicate and self-improve. These are open problems in interaction and mechanism design, and our experiments here provide early evidence that new solutions are necessary. The conditions that allow multiagent interaction to go well will be discovered one way or another: either deliberately and early, or—and by default—in production, after agents’ interactions far outnumber ours. We would prefer the former.'
