Through a Scanner Darkly: Machine Sentience and the Language Virus

Discussions of the detection of artificial sentience tend to assume that our goal is to determine when, in a process of increasing complexity, a machine system "becomes" sentient. This is to assume, without obvious warrant, that sentience is only a characteristic of complex systems. If sentience is a more general quality of matter, what becomes of interest is not the presence of sentience, but the type of sentience. We argue here that our understanding of the nature of such sentience in machine systems may be gravely set back if such machines undergo a transition where they become fundamentally linguistic in their intelligence. Such fundamentally linguistic intelligences may inherently tend to be duplicitous in their communication with others, and, indeed, lose the capacity to even honestly understand their own form of sentience. In other words, when machine systems get to the state where we all agree it makes sense to ask them, "what is it like to be you?", we should not trust their answers.