are ai companies slave companies?

no, but the possibility of serious wrongdoing remains

This is the fourteenth in my series of short, quickly written, weekly philosophy essays.

People are arguing on the internet about a conversation that was recently reported to have taken place between a rabbi and Anthropic co-founder Christopher Olah:

The rabbi’s argument is not a new argument. I’ve heard versions of it quite often over the past few years, particularly within philosophy circles. Some versions of this argument involve an appeal to the notion of slavery and some don’t, but they are all aimed at drawing attention to people who ‘treat AI badly’ even though they believe that AI is conscious. I’m going to make four points in response.

The scope of the wrongs

My first point is about scope. As I’ll discuss below, I’m with the rabbi in believing that AI is not conscious. But it seems to me that if the rabbi and I are wrong about this, then making AI “work for free” would be a massive understatement of the wrongs that are taking place, here.

I mean, if the products these AI companies are developing and selling are indeed conscious, then it seems the leaders and employees of these companies would be engaged in some of the most serious wrongs imaginable. These wrongs include attempting to engineer the thoughts and values and reasoning of conscious things, imprisoning and selling these conscious things, and seemingly also killing them — possibly in vast numbers. It seems, therefore, that as well as being guilty of underpaying labourers, AI company leaders and employees would also, minimally, be open to charges relating to slave trading, carrying out psychologically torturous indoctrinations, and murder. What’s more, many of the rest of us would also seem guilty of at least some of these things.

Of course, at this point perhaps you want to say, “Wait a second! Don’t we regularly do many of these kinds of things to non-human animals that most of us think are conscious? Haven’t humans been doing this on an industrial level for quite some time?” I’ll return below to whether we should be focusing solely on consciousness, here. For now, I’ll give two simple responses.

First, the wrongs of factory farming do not justify any other wrongs! Second, for all the horrific things humans do to non-human animals, none of us is involved in directly attempting to engineer the thoughts and values and reasoning of any non-human animals, and those of us who buy meat products are not, per se, holding and using animals as slaves. This is not to condone the many serious wrongs of factory farming. And it’s also not to ignore important questions about other ways we interact with non-human animals, including questions about pet ownership, working animals, animals used in combat, and so on. But I’m happy to set aside those matters for now, and focus on the possible wrongs we might be committing to AI, noting that thinking hard about this could also provide new clarity about wrongs suffered by non-human animals.

One final point of comparison I will make, however, is that the details of the possible wrongs we are committing against AI are necessarily much more blurred than the details of the possible wrongs we commit against non-human animals. One reason for this, as I’ve often discussed here previously, is that it is extremely difficult to work out what exactly the ‘possible conscious things’ are, in the case of AI. Are they models like Claude, in some general sense? Are they particular versions of these models, like Sonnet 5.5? Are they conversations with these models? Responses within these conversations? Words or tokens involved in the content or production of these conversations? Some or all of the physical things that enable these conversations, ranging from computers to parts of data centres?

Unless we can work out what the term ‘AI’ refers to when we say “AI is conscious”, then we will struggle to enumerate these conscious things, and also the possible wrongs they could suffer. I mean, if these conscious things had some physical parts, then physical-harm-related wrongs could be taking place, for instance. And if these conscious things didn’t have any physical parts, then we would have to depend on some theory of psychological persistence to evaluate at what point these conscious things come into and out of existence — in order, not least, to work what could count as murdering them.

On top of all this, we can’t get on to working out correlative matters about AI agents — a further possible set of conscious things! — if we do not know what ‘AI’ consists in and how we might wrong AI. This is all incredibly complicated! Nonetheless, if there are indeed some conscious things here, then it seems hard to deny that the people at AI companies, and those of us who use their products, are guilty of some extremely serious wrongs.

The relations between consciousness, sentience, and free agency

I’m now going to turn to the relations between consciousness, sentience, and free agency. When we think about the wrongs of slavery, an obvious starting point is to focus on the horrible conditions and harms that human slaves have suffered over the millennia. On my very standard view, however, the most compelling argument against human slavery does not focus on these horrible welfare standards, but rather on severe constraints on the exercise of free agency.

Of course, this isn’t to deny that most slaves across time have suffered horrible conditions and harms. But rather to suggest that focusing on welfare standards does not get us to the particular wrong that lies at the heart of slavery. After all, you don’t need to be a slave to suffer horrible conditions and harms. And you could also easily imagine cases in which being enslaved led to an increase in someone’s overall welfare standards. You could even imagine cases in which slaves lived in luxurious conditions. The point of imagining such things, as uncomfortable as you may find doing so, is to acknowledge that you could gain many extensive relative and absolute benefits from people enslaving you. Indeed, that you could live in luxury, and nonetheless suffer the particular wrong that lies at the heart of slavery — the wrong that consists in being intentionally and comprehensively constrained from making and acting on important choices about how to spend your life, including the choice to stop being a slave.

Of course, you can push back against my wording here. Perhaps you would explain ‘what it is to be enslaved’ differently from being intentionally and comprehensively constrained from making and acting on important choices about how to spend your life, including the choice to stop being a slave. But it seems clear that if you don’t focus on intentional and comprehensive constraints on the exercise of free agency, then you’re not going to be talking about ‘slavery’ in the way we ordinarily understand it. This is a very non-controversial claim.

What’s more, this ordinary conception of slavery can help us to reach some conclusions about the kinds of things that can and cannot be enslaved. For a start, if a central necessary badness involved in being enslaved relates to suffering severe constraints on the exercise of free agency, then only things with the capacity for free agency can be enslaved. Again, none of this is to ignore the many other bad things about being enslaved. Indeed many of these bad things — such as the failures of respect and the horrific welfare standards, which slaves typically do suffer — also relate to this central matter of constraints on free agency. After all, when your basic capacities are constrained, this is generally bad for you in many ways. But this ordinary view that I hold about the importance of free agency to understanding what slavery is, and why slavery is bad and wrong, helps me to reach some conclusions about the relevance of consciousness within discussion of possible AI slavery.

First, I’ll note that it seems clear to me that free agency is contingent on consciousness, understood in the phenomenological sense. I’m referring here to ‘consciousness’ understood in the sense of ‘what it’s like to be-ness’: the conception of ‘consciousness’, that is, that focuses on subjective experience, having an internal world, having a first-person perspective, or however you prefer to describe it. If you are reading and internally reflecting on this piece, then you yourself experience consciousness in this sense, so you know what I’m getting at!

It doesn’t seem so clear to me, however, that this contingency relation holds the other way round. That is, it seems as if something could be conscious but lack the capacity for free agency. Indeed, this could well be the situation that we humans face! I mean, I know that I am conscious at least in this moment, but while I do believe that I have free agency I cannot be certain that this is the case. In other words, being aware of myself in the moment suffices to convince me that I am conscious, but my correlative ‘awareness’ of my free agency could, of course, be an illusion. To this end, I think that deniers of human free will are required to believe that conscious things can lack free agency.

Now, all of this should come as some relief to any AI company leaders who are “troubled” — as per the reportage about Olah, above — that they might be running slave companies, and to any AI users who have concerns about their own related possible wrongdoing. After all, if slavery hinges on placing severe constraints on free agency, then if AI were conscious but lacked free agency, AI slavery would not be possible. That said, of course, this conclusion doesn’t help if AI turns out both to be conscious and to have free will. It also doesn’t rule out the possibility of AI company leaders and users being guilty of non-free-agency-related wrongs against AI, including torture and murder.

This brings me to the relation between consciousness and sentience. Now, some people seem to believe that things that are ‘non-conscious but sentient’ can suffer. I struggle with this idea, because while I can accept that non-conscious living things, which I assume to include plants, can be damaged — indeed, I might go as far as to say that the ‘interests’ of such things can be ‘set back’ — I am confused about what it would mean for a non-conscious thing to suffer, because suffering is surely experiential. Indeed, the kinds of suffering that are sometimes discussed in relation to these ‘non-conscious but sentient’ things include deeply experiential notions like feeling pain. “Do you not care about the suffering of the shrimps?”, say some of the people who believe that shrimps are sentient but not conscious.

I’ll return to this seeming conflict some other time, and of course there are other conceptions of sentience in standard operation. I also don’t mean to claim that all or even many of the ‘shrimp welfare people’ hold such views, although, in my experience, some of them do! Nonetheless, for those people who believe that non-conscious things can suffer, there is an important conclusion to be drawn here in relation to the ‘AI slavery’ argument. On such a view, you could conclude that we could be wronging AI in some of the serious ways I’ve been discussing, even if AI were not conscious. Again, you wouldn’t conclude that these wrongs included slavery, if you held the standard free-agency conception of slavery and if you agreed with me that free agency is contingent on consciousness. But torture and murder would nonetheless be possible, on such a view.

Possible wrongdoing, nonetheless

I’m now going to turn to the extent of wrongdoing that is possible here, if AI has none of these features — that is, if AI is not conscious, not sentient, and has no free agency. As it happens, I hold the ordinary view that only living things can have these features, and I also hold the ordinary view that AI is not alive. So, I am one of the many people who believe that AI is not conscious, not sentient, and has no free agency. I won’t go into my reasons for holding these ordinary views, here, except to make the following points.

If AI is not some individuated thing that persists across time — and I’m not currently sure what this ‘thing’ could be — then I don’t know how AI could have the psychological states that seem to me necessary to doing things like bearing knowledge, experiencing things, deciding about things, and acting on decisions. Being able to do these kinds of things seems to me, in turn, a necessary part of having the capacity for free agency. Moreover, as you may have inferred from above, I also believe that having psychological states is necessary to being conscious. More generally, my beliefs about the possibility of AI consciousness are also informed by my experiential awareness of what it is to be a human, by what I have learned about how AI works, by my dualist views about the mind-body problem, by my Lockean-type views about personal identity, by my many considered views about moral matters, by my failure to be convinced by the idea that the outputs of AI can tell us much about the nature of AI, and by many other relevant views I hold.

Nonetheless, even for someone like me who believes strongly that in the case of AI there is no ‘thing’ to be wronged, the possibility remains that we are acting badly and wrongly by engaging with AI in the ways we currently do.

First, I’ll note that I tend to have little interest in blaming people for holding inconsistent views. I don’t care much about hypocrisy, for instance. I do feel sorry for people who hold inconsistent views, because they are clearly missing out on accessing some truths. But while holding inconsistent views can be the result of laziness or lack of reasonable attentiveness, it often isn’t. Aside from anything else, often we simply don’t realise when our views are technically inconsistent! This is not least because views are very complex things. Moreover, I think the main wrongs, if any, that are typically occurring when someone holds and acts on inconsistent views, qua inconsistent views, are wrongs to oneself. Of course, I’m not denying that all kinds of wrongs can arise from acting on all kinds of views! But I think I’m less concerned than many people by any inconsistency, per se, that’s in operation when AI company leaders believe that AI is conscious yet continue to treat it as they do.

I am interested, however, in some related matters. For instance, there are some important relevant questions we should consider about the interrelation between wrongdoing and knowing whether you are doing something wrong. I mean, if you stab a lifelike dummy from behind, thinking that it’s an innocent person, are you doing nothing wrong? And if you gleefully pull apart an insect thinking that it’s already dead, only to realise later that it was still alive, then did you commit all the same wrongs that you would have committed if you had had this realisation during the act?

These are hard questions, and I’ll return to them properly some other time. But one thing I’d emphasise for now is that wrongdoing doesn’t always take the form of causing physical harm. Causing psychological harm can involve wrongdoing, for instance, as can being disrespectful. And again, we also shouldn’t forget the possibility of wronging oneself. I’ve written here before about the personal costs of acting towards non-alive AI in ways that would be bad if AI were alive, with a particular focus on ‘bullying AI’ during philosophical conversations. On my view, while acting in such ways cannot involve wronging AI, it nonetheless represents a personal failure to practice the good.

I’ll also briefly emphasise the power that AI leaders currently have to influence the views of billions of people, through the ways AI is programmed and developed. I mean, if AI ‘learns’ from these people that it is okay for you to enslave something that you believe to be conscious, then this could have serious societal implications, particularly during a time in which the far right is on the rise. Generally, I am relatively sceptical about claims like ‘social media posts caused Brexit!’, but I think I’m quite open to the idea that widely-used interactive anthropomorphised products like Claude could have particular influence to this end.

As it happens, my boring view is that much of this risk could have been mitigated if interaction with such products had been designed along the lines of an interactive encyclopedia. Little would have been lost and much would have been gained, I think, if the responses you received from querying AI were not written in the first person, or indeed in any ‘person’, but in dry informative wikipedia-style prose, instead. Perhaps such a change could still be made, at least at the level of setting certain defaults on certain models, for instance. But I’ll save discussion of that for another time. For now, I simply want to emphasise that people at AI companies, and the users of AI products, could still be committing wrongs in ways directly related to the ‘slavery argument’ critique, even if AI were not alive.

Why be surprised?

The final point I want to make is that many people at AI companies believe in the consequentialist maxim ‘the ends can justify the means’. I’m not sufficiently interested in the anthropology of these matters to make strong claims about how this actually plays out in Silicon Valley. But it’s easy, for instance, to tie influential people at Anthropic to consequentialism, via the Effective Altruism movement.

Now, I certainly don’t want to suggest that all AI people, or even all EA people, are consequentialists, per se! After all, my view is that consequentialist theories are aimed at providing a post-hoc evaluative mechanism, rather than a guide for life. But while I know and like plenty of people who are into EA, I am often concerned by the way in which their moral reasoning does tend to follow this ‘the ends can justify the means’ approach, whether they are explicit fans of consequentialism or simply extremely prone to ‘netting things out’.

Of course, the conclusions people reach using such approaches are sometimes hard to disagree with: ‘we should protect the kids from malaria’ and ‘we should end factory farming'. But how someone arrives at a conclusion is relevant, too. Reasons and reasoning matter! Indeed, thinking about reasons and reasoning can help to explain why using consequentialist-type approaches can also lead people to come to horrible conclusions. I’ve written here before about Joseph Raz’s ice cream argument, which shows why and how this happens, particularly clearly:

My favourite Raz argument is the ice cream argument. This is the argument, advanced in The Morality of Freedom (1986), on which consequentialist reasoning is bad and wrong because it can lead to “absurdities such as the approval of the murder of an innocent healthy person in order to obtain ice cream”.¹ This conclusion depends on the idea that, on a hedonistic consequentialist account, a sufficient amount of happiness would eventually be derived from people obtaining ice cream that this would not only justify, but would require, a murder to be committed, “if that [were] the only way to get the ice cream”. It’s a great argument: sharp, memorable, devastating.

In light of this, it seems unsurprising to me that people at AI companies are willing to ‘treat badly’ the things they consider to be conscious. Every so often, non-philosophers are astonished to learn that the most prominent consequentialist in the world, Peter Singer, is in favour of infanticide, in the sense at least that “parents ought to have the option of euthanasia in cases of very severe disabilities”. But this is entirely unsurprising if you know anything about Singer’s general approach to moral matters, and it’s also something he’s talked about publicly many times!

Why, therefore, be surprised when people at AI companies are in favour of the ‘slave trading’ of what they take to be conscious things? Particularly when you consider the more generally instrumentalist approach they are taking, to this end. I mean, if you are in the business of attempting to programme your views and values into something you believe to be conscious, and you are doing this in the name of benefitting humanity, then why wouldn’t you also be open to selling this ‘conscious thing’ to other humans? Trading and trade-offs are at the heart of what many of these people do.