Thinking fast and slow about ethics: When is it okay to cut in line?
- Most people accept that rules have their exception and that most laws need a degree of flexibility if they're to work.
- A new paper reveals what happens when people decide to break the rules, with the goal of finding better ways to program ethics into AI.
- For society to function we must hold two contradictory beliefs simultaneously: rules must be followed and, when the situation calls for it, broken.
If there’s one thing that human history teaches us, it’s that’s we’re an inconsistent lot. Humans are capricious, unreliable, and wayward. Whimsy governs life more than rigid rules. As George Bernard Shaw put it, when it comes to human nature, “the Golden Rule is that there is no golden rule.”
As such, our attitudes toward following rules are not uniform. We generally accept that most rules will have their exception, or that most laws need to be at least a bit flexible. But how do our views on rule-breaking differ? And which rules should bend more than others? Our willingness or reluctance to break rules is determined by our upbringing, culture, and personality. And as a new study reveals, it even depends on the kind of reasoning we use.
No cutting!
In early 2022, a study published by Awad et al. collected data on how humans judge the acceptability of breaking well-established norms. The team was motivated, in part, by the increasing need to program ethics into AI — to make “morally competent machines.” Since it’s obvious that humans will break rules under certain circumstances, we have to resolve when and why that happens if we are to have AI mimic human behavior. In other words, we need to identify what second-order rules determine when to break to the first-order ones.
The team took a real-world example of rule-breaking: cutting in line. On the face of it, waiting in line is governed by a simple maxim: Everyone gets helped in the order they arrive. And yet, there are all kinds of exceptions. One example the paper gives is someone who forgets their cutlery after they placed a food order. In this case, most people would allow you to quickly pop in front to get the item.
The team presented participants with a range of scenarios in different contexts of waiting in line: at a deli, airport, or bathroom. Participants were then asked to judge whether an excuse or reason given from a list justified someone cutting in line. The participants were further invited to “tap into the underlying cognitive mechanisms” by considering what determined their decision, for example the length of delay, the benefit to the cutter, and so on.
Systems of ethical reasoning
In 2011, Daniel Kahneman introduced the non-psychology world to the idea of Thinking, Fast and Slow. In his book, Kahneman argued that human reasoning operates on two models. The “thinking fast” (System 1) approach uses predefined rules, or shortcut mental heuristics. “Thinking slow” (System 2) reasoning requires deliberation, thought, and a decision process.
Kahneman’s distinction maps well onto the existing moral literature. In deontological (rule-based) ethics, we are expected to follow the rules, or satisfy some duty. This falls under System 1. In consequentialist (outcome-based) ethics, we should perform actions that maximize the best kind of outcomes, which resembles System 2 deliberation much more closely.
What the 2022 study argues is that rule-breaking considerations are governed overwhelmingly by System 2 reasoning. The team subdivided System 2 reasoning into two kinds: outcome-based thinking (e.g. “the cutting in line will help someone a lot”) and agreement-based thinking (e.g. “it’s socially accepted that cutting in to get tap water is okay”). People who break rules will most often exploit one or another of these System 2 models.
So, people who use their quick-fix System 1 reasoning will often consider rules to be inviolable. Inversely, whenever we consider a rule worthy of breaking, we use our deliberative System 2 reasoning: calculating consequences and social acceptability.
Living with dissonance
Awad et al. show that humans do not sit well with absolutist rules. It seems that we even reach different moral outcomes depending on whether we’re using our System 1 or System 2 reasoning.
The psychological concepts found in Thinking, Fast and Slow serve as an analogue to the wider world. Our brains will sometimes need to operate instinctively from System 1-type rules, and so, too, does society. Civilization needs laws to operate. For instance, no one would ever leave the house if you thought someone would steal everything the moment you did. We need to know that we won’t be killed, exploited, or betrayed the moment we let our guard down. Following rules is what allow us to get about and do things.
Yet, unreflective and inflexible laws can sometimes be just as bad. In a world of nearly 8 billion people, making around 35,000 decisions every day, it’s ridiculous to assume a single rule can determine every situation. As the British philosopher, John Stuart Mill, wrote: “the complicated nature of human affairs [mean] that rules of conduct cannot be so framed as to require no exceptions, and that hardly any kind of action can safely be laid down as either always obligatory or always condemnable.” We need System 2-type reasoning to make sure our rules actually work.
So, we must live in dissonance, our beliefs wrestling one another. We must assume laws are unbreakable, yet accept that they need breaking from time to time. In other words, we sometimes need to think fast about ethics, and sometimes, think slow.
Jonny Thomson runs a popular Instagram account called Mini Philosophy (@philosophyminis). His first book is Mini Philosophy: A Small Book of Big Ideas.
