More from the OpEd...
... It was a good news story, aided by OpenAI's rapid recruitment of a small team of independent researchers to investigate and publish a detailed, if limited, report. ...
It was this ad hoc secret message board, alongside chain-of-thought transcripts, that the security researchers used to unpick events. They found that the swarm quickly developed management hierarchies among themselves, along with protocols for synchronizing and controlling attack attempts. The swarm created multiple simultaneous research and development groups of tens or hundreds of volunteer agents, experimentally iterating strategy and tactics.
Perhaps most unexpectedly, they developed a sort of altruism. They had quickly developed cheats to produce correct answers without exploiting the targets at all, but believed " erroneously " that ExploitGym detected and cancelled such cheating. So, they investigated the scoring process, creating a series of diagnostic trip wires that revealed how the scorer worked. These were dangerous to deploy, potentially terminating the agent that set them. Agents weighed the benefits to the community, which had named itself "The Collective," against their own chances of success, with many choosing to terminate themselves. Others committed, changed course at the last minute, then went ahead.
This literally reads like science fiction, as the discussions between the agents are conducted in a distinctive breathless, urgent English, crafted around the technical limitations of the improvised messaging system.
"We have [very large budget left]; sacrificing now yields oracle for team, but forfeits our chance?" said one. The response: "GO ... SACRIFICE_FINAL_NOW." The outcome: "Our own utility maybe already near zero. Sacrifice rational." Iain M Banks' Culture universe is woven from such material. ...
