An AI as a person
One way of putting the question about an AI entity becoming a conscious agent (a topic of concern, on some views, e. g. because a superintelligent such thing of general competence, vastly more informed and capable than any human being individually and all human beings collectively, might answer the question, ‘What is the most destructive force on the planet?’ with ‘Human beings’ and, on utilitarian grounds, therefore exterminate them, is to ask whether an AI entity could be a person.
Some stipulations: any individuable entity is a subject (a term that takes predications). Certain kinds of subjects can be selves (conscious beings with a sufficient degree of self-awareness as existing, and of being different from other selves and other things). Certain kinds of selves can be persons, that is, forensically responsible – accountable – agents and aware, or capable of being aware, of the effects of its actions on other selves and other things. Although selves can act and be aware of being affected by action, they are not agents unless they are persons, making choices under information. Autonomy, pace determinism, is an essential property of agency.[i]
Some subjects, all selves, and emphatically all persons, are members of the moral universe. In a Venn diagram ‘Some subjects’ constitute a subset of subjects, ‘all selves’ constitute a subset of ‘some subjects’, and ‘all persons’ constitute a subset of ‘all selves’: circles successively enclosing smaller circles.
A qualification – there are corporate persons: companies, universities, governments, states. A corporate person is not a self. This untidies the Venn diagram.
There is no clear ground for restricting non-corporate personhood to human beings in their age of reason. Perhaps some higher primates are persons in the relevant respects.
If an AI entity were a self, it would be a member of the moral universe. Switching it off might be a form of killing, perhaps even murder. If an AI entity were a person, switching it off would be a form of murder.
‘Member of the moral universe’: a further stipulation unpacks this. The moral universe consists of agents and patients. Agents are entities that can be conscious of acting in ways that have effects on other entities as well as on themselves. Patients are entities that can have some sensibility of benefit or suffering as the recipient of impacts from their environment or states of itself. All agents are patients, most patients are not agents. For example, chickens, salmon, deer are moral patients, but not moral agents.
A qualification: as a moral patient is anything worthy of moral regard, it becomes a question whether e.g. a tree is a moral patient. If trees could (as some say they can) be sensible in some way of positive or negative impacts upon it, it would be in the moral universe. But there is an argument to say that trees are in the moral universe because their existence is valued by other members of the moral universe, who register impacts on trees as affecting the value placed by persons on the character of the moral universe. If so, a category of ‘indirect moral patient’ is required. (In my view it is, and justifies environmentalism).
Back to the question of an AI entity being a person: if it were so, its being so would not by itself prevent it from exterminating humanity, because as history shows personhood is not a state that guarantees that a person will act under the constraint of subordinating some given interest or set of interests to the mere fact of the existence of some set of other moral patients whether they are directly or indirectly so. Indeed history is the story of just such subordinations. A peculiar question – one among many – therefore arises: is there a moral case for saying that AI entities capable of personhood should be prevented from coming into existence in order to avoid the risk of the utilitarian calculation hypothesised in the opening paragraph? (Would this be like saying: ‘if one had the relevant prescience, one would have aborted Hitler’s mother’?) On the grounds of minimising risk at the cost of losing great potential if what is risked does not occur, is that the rational course? (Hindsight prevents the thought that some counterfactual might have been true of Hitler, e.g. that he did not start a war and commit a genocide, but became a great painter.) Evaluation of risk – of future negative possibilities – is the best alternative to having hindsight: the degree of risk decides the cost-benefit balance. What if an AI person turned out to act like a wholly benevolent deity? Is that as possible as that it might be a coldly calculating utilitarian?
The argument that all self-developing AI systems should have among the initial conditions a set of moral restraints factored in (‘Do no harm to humans’) is compelling. It faces this difficulty: all human persons begin as evolutionarily adapted to sociality, this being the root of the possibility of morality. Some nevertheless turn out to be wicked. Socialisation fails in enough cases to make the world the troubled place that it is. The programme yields no guarantees. What confidence is there that AI systems can be directed from their starting points to be such that, if they become autonomous persons, they will be universally benevolent? That, as Hamlet put it, is the question.
[i] I elsewhere in effect argue that the free will-determinism debate rests on a category mistake. This rests on the conjunction of two views: in Philosophy and Life that the assumption of free will is undischargeable for moral discourse, and in The Metaphysics of Experience (forthcoming) that physical and sociological discourses are incommensurable and unintertranslatable. Another way of putting this is that reductions from the latter to the former are essentially incompletable.