It’s been a few months since I’ve made progress posting on this Substack - both because I’ve been busy, and because the avenues I went down weren’t as productive as I hoped. But the series so far covers and explains a lot of different things - so I’ve somewhat lost track of what I have and have not said myself, and what the relationship between the points are.
But I have more things to say, and don’t want to lose those threads just because I’ve been busy. So as a service to myself, I’ve been chatting with an LLM to summarize and structure the story so far, and we have a few different ways of thinking about that. I’ll present a few of those that seem at least moderately insightful - first, requirements for cooperation, then failure modes, then shared reality. (The last is the best one, in my view, but the cost of the insight is that I don’t think it correctly captures what I intend in subtle ways.)
Either way, this exercise has definitely help remind myself of the thread so far, and helped recenter my thoughts, and hopefully it’s useful to readers as well - so I’m posting what I’ve rediscovered and/or reconceptualized, despite the slightly skewed conceptions - which are subtle enough that I’ve had trouble rewriting them away.
Requirements for Cooperation
One way of looking at the entire question is based on the structures for what allows cooperation.
At the root, we have Preconditions for cooperation. As we discussed repeatedly, it’s not always true that cooperation is possible, and for it to occur, it depends on agents having real, plural preferences and goals that are not inherently anti-cooperative, rather than on shared values or moral convergence. Those preconditions also assume some basic philosophical points about Meaning; agents must be able to interpret rules, norms, and commitments through grounded symbols that actually constrain action, rather than merely producing behavior that looks right. Meaning, in turn, relies on Foundations such as rejecting naïve views that equate optimization and prediction with understanding.
When these layers are in place, Scaled Institutions are possible via norms, enforcement mechanisms, and governance structures, though they remain fragile under optimization pressure. The overall direction points toward Design, with alignment implications: robust and cooperative systems are about alignment, not just observable behavior. Otherwise, instead of sustained cooperation, as agents learn more they will exploit it; this is especially true of more powerful agents.
Cooperation Failure Modes and Causes
Another way of conceptualizing the work so far is based on what can fail, and how.
Here, precondition failures usually lead to misinterpretation or masking, while scaling and design failures are otherwise pervasive; it give less information about the structure of the arguments so far, but there is much more insight into which failures to actually expect.
Shared Reality
Lastly, we can think of shared reality as the keystone, instead of looking at agents or incentives. Agents participate in this intersubjective reality, but they do not define it unilaterally (see: intra,) nor do they receive it as an input.
The coordination mechanisms (games, institutions, enforcement) operate within or form part of the definition of this built shared reality. This is really helpful, because it reframes the idea of preconditions for cooperation, the rules of the game, as parts which can be changed via redefinition of the systems over time. This reframes the construction of cooperation as a process, rather than a goal or end point - which I don’t think has been discussed explicitly so far, but is really important.
But it also means that drift is endogenous to coordination at scale — it is not (only or always) an external shock. And stabilization of cooperation and building the definitions is not about single-party optimization; it is an active way to participate in the continued and evolving definition. But this gets into multi-agent systems, which I haven’t discussed enough yet.

A key feedback loop to notice is: Coordination → Drift → Shared Reality degradation → Coordination failure.
The claim, per LLM stochastic parrot summary, is that stabilization must interrupt this loop without destroying agency or meaning - which is kind of right, but a bit too much of a word salad simplifying and combining different ideas. However, I also got the last diagram to map onto the different posts, in the tables below - helpful in case you missed any and want to catch up, or want to refresh you memory further.
(Unfortunately, it changed the post order a bit, and repeated one instead of including Cooperation Beyond Evolution.)
Hope to see you again soon - either to talk about multi-agent systems, or to explain in more detail why I think liberal democracy is cheating, and why I disagree with the inimical Seb Krier






