Defining and Starting to Scale Cooperation
Abandoning Prisoner's Dilemmas in favor of Stag Hunts, then abandoning Stag Hunts once they get too large.
The question of what to do in a stag hunt was brought to my attention by Abram Demski, who studies how future AI systems might reason and how they could cooperate. Abram got started studying cognitive architecture at the University of South California’s Institute for Creative Technologies, where I first met him, but then moved north to Silicon Valley to think about artificial intelligence. We’ll get back to that in a different post, but one topic that interested him was how groups act rationally, and in a long running online conversation, the topic of stag hunts in game theory was a recurring theme.
I’ll explain the question, but it’s also a central theme I’m going to return to repeatedly on what is needed to cooperate at scale. And to get there, I need to define cooperation.
What is cooperation?
In previous posts, I talked about how it arises naturally - the game theory for prisoner’s dilemmas describes viruses as much as it describes humans. Changing directions for a bit, I want to venture a definition for what cooperation means.
I’d start with two requirements for cooperation: potential for either mutual benefit or defection, and active decisions towards reciprocation. For active decisions that are intentionally reciprocal, though, we need something like consciousness on the part of both actors. Unsurprisingly, SMBC Comics nails this; “…which requires the species to possess three [things]: Awareness of oneself as a moral actor, awareness of other beings as similarly endowed, and awareness that there exists understanding between beings about the appropriateness of actions.” But for this to be cooperation, as opposed to a more general notion, I’ve added to Zach’s claim that both must have not just an idea of appropriateness, but the possibility of benefitting the other via cooperation, or defecting for more short term gain.
What does this exclude? When I tell my younger children that I want them to cooperate and help clean up the room, I mean that they should do what I want instead of making me do everything. They might have an incentive to “cooperate” because I reward them, but the other side is absent - I have no willingness to defect, and if they do so, I will need to clean up on my own. It’s not cooperation, it’s just a polite demand on my part for them to do something. On the other hand, both sides can, in theory, defect in the mutualist interaction I have with my gut bacteria - they can reproduce out of control or mutate in ways that don’t benefit me, and I can eat poorly in ways that hurt them and myself, or I can kill them off with antibiotics. The difference here is that they can’t decide to defect, since they don’t make conscious decisions - so the lack of defection isn’t cooperation.
Fake Prisoner’s Dilemmas
The above definition is in some sense all about Prisoner’s Dilemmas. But most such scenarios are fake, in some sense. Decision-theorist-turned-Cassandra-of-AI Eliezer Yudkowsky points out that most Prisoner’s Dilemmas aren’t, really, the claimed dilemma; there are too many reasons that people have incentives other than just whether to cooperate and defect. And the reason this happens matches much of the discussion about why people actually cooperate.
There is a TV show, Golden Balls, which makes contestants play a monetary version of this game in practice. To start, there are several rounds of play, in which the players can lie or not, building to the final game, where the two final contestants choose either to split the prize, or to steal. If they cooperate, as the name implies, they split the money. If one steals, that player takes the entire total, while if both steal, no-one wins. As a prisoner’s dilemma, each player is better off being the only one who steals, but if everyone is selfish, playing leaves everyone going home empty handed. Overall, about half split the money. Similarly, in monetary prisoner’s dilemmas conducted online, around half of the public cooperates.
In some sense, the narrowly “rational” decision, in each case, is not to cooperate - and despite this, people often cooperate. Why?
For the game show, we can imagine that people might not want to be known as someone who steals - and since the whole thing is televised, they might not want to steal. And incentives matter! But for the online experiments, no-one will ever know what the player chose, so it’s hard to imagine that is the motive.
Another theory is that people cooperate because they simply aren’t selfish - they like the idea that others will get money, and so they are happier to risk losing some amount of money. And this is definitely part of the answer; we have evolved and been culturally conditioned to be less than completely selfish. Thankfully.
But a third theory, advanced by Douglas Hofstadter in his column in Scientific American, is that people are super-rational. That is, they realize that everyone is better off in general when people cooperate, and knowing that the other players are smart and aware of this, they will cooperate - not because of empathy or future behavior, but as a form of enlightened self interest. To test this idea, he wrote to twenty friends with a monetary prisoner’s dilemma, played by the friends, with all payments coming from Scientific American. Each player has a choice of sending everyone three dollars by cooperating, getting five dollars by defecting when others cooperate, or having everyone receive only one dollar if both defect. If everyone cooperated, they would all get $57, while if all defected, they would all get $19. After explaining, he says, “why am I doing the multiplication or any of this figuring for you? You’re very bright. So are all of you! All about equally bright, I’d say, in fact. So all you need to do is tell me your choice.”
To explain the logic a bit more, Hofstandter reasoned that there should be a correct answer. If there was, it seemed to him that these smart people should agree about what that answer is. And, he assumed, these smart people should understand that whichever answer they got would, likely, be the right one. Clearly, they’d be better off if that answer got them $57 instead of $19, so they should all cooperate. Hofstadter’s logic is a bit tricky, since it seems to think that your decision causes other people to decide differently. The idea of this kind of mutual causation gets confusing, but it also starts getting really important towards the end of the book, so we’ll leave it as somewhat confusing, and return to it then. But for now, while Hofstadter’s friends were very smart people, only 6 of the 14 cooperated, notably, a bit below the level of cooperation in the general public.
Unfortunately, it seems that smart people are at most no more likely to cooperate, and arguably are worse off. Even more concerning, the literature implies that learning about game theory and self-interested behavior make people less likely to cooperate. And if that’s true, perhaps Hofstadter’s experiment showed that I have just done you all a disservice by explaining this - but I think there’s a better answer, and it’s about knowing too little game theory, just enough to lose the evolved basis, and not enough to go beyond it, entering the uncanny valley of rationality.
To escape the valley, we’ll need to learn to hunt together.
Stag Hunts
The idea comes from Rosseau, the originator of the “Social Contract” theory of society and morality, in his Discourse on the Origin of Inequality. And that’s a critical discussion, but to skip the social science and ethics for the remainder of the post, he imagines a group of “savage men,” peaceful hunter-gatherers just trying to keep themselves fed. This group would go out hunting, and see two types of animals - deer, and rabbits. Deer are harder to hunt, but have far more food, and can keep everyone in the tribe fed for days, while rabbits will only provide enough food to survive. Obviously, if the group could agree to hunt deer, they should do so - but “if a deer was to be taken, every one saw that, in order to succeed, he must abide faithfully by his post: but if a hare happened to come within the reach of any one of them, it is not to be doubted that he pursued it without scruple, and, having seized his prey, cared very little, if by so doing he caused his companions to miss theirs.” Rabbit hunting was safe, but cooperation would be better, if only they could cooperate.
Game theorists distilled the idea of a stag-hunt to a 2-player game, where the players can either cooperate to hunt stag, and benefit, or they can each decide to hunt rabbits, and still survive. The problem is when they are supposed to hunt stag, but “a hare happened to come within the reach” of one, and he defects. If the two people cannot trust that the other will cooperate, they cannot risk hunting stag.
This sounds a lot like a prisoner’s dilemma, but it is critically different, because in this case, even the person who grabs the hare is worse off - there’s no reward for defecting, just a fear that cooperation will fail. Unlike the Prisoner’s dilemma, two people can agree to the better outcome, and the level of trust needed is pretty minimal - no-one individually benefits by switching. But Duncan Sabien, a former instructor for the Center for the Study of Rationality, pointed out that “It is a defining characteristic of stag hunts that when everybody actually buys in, the payoff is pretty huge. It is also a defining characteristic of stag hunts that when critical mass fails to cohere, those who chose stag get burned, and feel cheated, and lose big.” And this explains why Rosseau’s stag hunt is part of his discussion of inequality.
But Demski pointed out that Rosseau’s example isn’t really fair, especially in a world where there is enough food, and where we need lots of people to cooperate. So, he explained, most prisoner’s dilemmas are actually stag hunts. That is, most of the time the situations where defecting seems to be better for you are actually times where smart people would find ways to cooperate.
Scale and Society
The reason people manage to successfully hunt stags, in a metaphorical sense, include the fact that most interactions are repeated, the way people can later be punished for defection because they develop a reputation, and because the modern world has invented enforceable contracts. So if Demski is right, and most prisoner’s dilemmas are stag hunts, the question stops being about why people cooperate so often, and becomes why people don’t cooperate even more!
And we do see cooperation across scales. Small communities hold barn-raisings or similar, where everyone pitches in majorly to help each other, with social rewards deferred, often indefinitely. There are often communal meals, and richer individuals build social capital by providing things for others and donating to community organizations. And the young and the elderly are often cared for at least quasi-communally. Somewhat larger communities have community organizations with dues and funds to build shared infrastructure and provide for the needy in other ways, ensure the elderly are cared for by relatives or extended families, and often have groups who do cooperative babysitting and informal education. At an even larger scale, countries have governments and taxes which help people in sometimes similar ways; tax subsidies for home purchases instead of barn raising, universal education instead of communal childcare, social security and disability insurance instead of a social safety net.
In the latter cases, cooperation depends much more on size and structures. The larger the scale, the more the burden shifts towards formalized cooperation and enforcement. But why?
The reason stag hunts work is because they are only required to coordinate a dozen people or so. In larger groups, if we can sign a contract saying we’ll split the money no matter what, the incentive to defect disappears - it’s a win-win situation. But if you need hundreds of people to all cooperate to catch the stag, not only is the risk that at least one will fail far higher, but everyone knows it, and is less willing to try. As long as there are small groups where reputation matters, or there is a society to strongly enforce rules and contracts, cooperation should be possible, if not easy.
And to give even more foreshadowing of what I want to talk about in other posts, coordinating deals in order to share benefits either takes time to negotiate, or requires shared understanding. This works well in small groups, when people share goals, and when people trust social norms or contracts can be enforced. But it needs more and more overhead and structure, and additional coercion - like jail time for unpaid taxes - for it to work. And this isn’t because of the form of government or reduced trust, it really is fundamental to scale.
In future posts, I want to go back to cooperation at each scale, and talk more about what has been done to build it - and what more will be needed at even larger scales in the future. I also need to talk about what people really want, and whether and how cooperation can or cannot help, depending on goals and values. There’s a lot to talk about, so stay tuned!