A trolley, five people and a switch
In 1967 the philosopher Philippa Foot left moral philosophy with a long-lived question: a runaway trolley is heading toward five people; if you pull a switch, it will go down another track and hit one person. What do you do? In 1985 Judith Jarvis Thomson developed the question under the name 'the trolley problem' and discussed dozens of new versions. Most people find pulling the switch acceptable; but when the same outcome requires pushing a person with their own hands, most people refuse.
For a long time this thought experiment was discussed mainly in philosophy classes; self-driving cars brought it back. If an engineer programming a car had to write in advance what the car should do in such a situation, which principle should they pick?
Moral Machine: 40 million decisions
Edmond Awad, Iyad Rahwan and colleagues at the MIT Media Lab built an online platform called the Moral Machine. Visitors were shown two possible outcomes for a self-driving car with failed brakes and asked which they preferred. The scenarios varied along nine dimensions: human or pet, more or fewer characters, young or old, crossing legally or on a red light, higher or lower status, fit or not, female or male, passengers or pedestrians, and whether the car stays on course or swerves.
In ten languages the platform collected 39.61 million decisions from 233 countries and territories; the results were published in Nature in 2018. The three strongest preferences were for sparing humans over animals, more lives over fewer, and the young over the old. The authors reported that the four most spared characters were the baby, the little girl, the little boy and the pregnant woman. Among the 492,921 users who filled in the optional demographic survey, individual characteristics such as age, gender, education, income, and political and religious views did not change preferences in any sizable way.
Three moral clusters
The main differences were between countries. Countries with at least 100 participants fell into three clusters based on their preference profiles: a Western cluster including North America and many European countries; an Eastern cluster including far eastern countries such as Japan and Taiwan as well as many Islamic countries; and a Southern cluster made up of Central and South American countries together with France and some countries with French influence. The preference for sparing the young was much weaker in the Eastern cluster and much stronger in the Southern cluster; the Southern cluster also showed a weaker preference for sparing humans over pets than the other two.
Two preferences appeared to be shared across clusters: sparing pedestrians over passengers (weak) and sparing the lawful over the unlawful (moderate). In this experiment your profile, this lab's crowd and the overall ranking from the paper are shown side by side.
Utilitarian for everyone, but not for me
In six online studies published in Science in 2016, Bonnefon, Shariff and Rahwan found a curious contradiction. Participants morally approved of cars that sacrifice their passengers to save more people and wanted others to buy such cars; but they themselves preferred to ride in a car that protects its passengers at all costs. They also opposed regulations that would make such a rule mandatory.
The authors stressed that this is a social dilemma: everyone may want utilitarian cars to become common, but nobody wants to be the first to buy one. Making utilitarian rules mandatory could put people off adopting a safer technology and so indirectly increase the number of accidents.
Why is making rules so hard?
In 2017 Germany became the first country to draw up official principles for the ethics of self-driving cars. The ethics commission of the Federal Ministry of Transport banned classifying people by personal features such as age, gender or physical and mental condition in unavoidable accidents, while ranking the protection of human life above property damage and animal life. As Awad and colleagues noted, the principle of putting humans before animals agrees with the crowd; but the ban on age distinctions clashes with the crowd's strong preference for saving children.
That is why we ask you a question about rules at the end of this experiment. Saving the young in the dilemmas and still rejecting age distinctions as a lawmaker is not a contradiction: a decision in a single case and a rule applied to everyone answer different questions. The result screen shows the two side by side.
Forced choice, or choice by force?
One of the most debated objections to the Moral Machine concerned the format of its questions. In their 2020 comment in Nature, Yochanan Bigman and Kurt Gray showed that when an option to 'treat both sides equally' was added to similar dilemmas, many participants chose it. In their view, offering only two options can make people appear to want distinctions they do not actually want. Awad and colleagues published a reply in the same issue; the debate is about what forced choices, and an equal-treatment option, actually measure.
Another criticism is that such thought experiments describe situations with certain outcomes that almost never happen in real traffic; a real car distributes risk under uncertainty. Think of this experiment not as a test of your morals but as a mirror that makes your preferences visible.