Learning to Think Like a Smart Machine
Are our smart devices conditioning us toward a new type of social order?
Harvesting Human Minds
The more we treat machines as if they were people, the easier it becomes to treat people like machines. Consider the aptly named “Hello Barbie”, billed as the world’s first interactive doll. “Hello Barbie” is programmed to make the following pitch to her young subjects, “I’d love to learn more about you. Oh, I know! Let’s make a game of it. The game’s called Family Town! We’re gonna pretend all your family members run different shops in a make-believe town! I’ll be a visitor and you’ll show me around! So … what’s the name of your family’s town? I think I’m gonna like it here! Okay, so every member of your family gets its own shop. One per person! I’ll visit each shop and you’ll tell me who runs it! Got it?” (Hopkins 2019, 44)
The doll’s young playmates tend to “get it” very quickly, but what lesson is Hello Barbie actually teaching? Researchers have shown that the doll was designed to harvest children’s responses to Barbie’s skilled interrogations (Gibbs 2015). Once processed using machine intelligence, their answers can feed algorithmically driven advertising strategies. However, Barbie also seems to be building a special type of relationship with her companions that suggests a more far-reaching agenda.
To get a feel for the relationship dynamics between Barbie and her children, here are some snippets from a sample dialog with Barbie in response to the “Family Town” scenario given above:
“Hm? Which of your family members would run the movie theatre?
Your stepmom? That’s cool! What kind of movies does she like to watch?Hm? Which of your family members would run the pet shop?
Your dad? Awesome! Which animals are his favorite?” (Steeves 2020)
Clearly, these snippets reveal an intent to mine as much data as possible about the target child and her family members. In addition, Barbie would like to be journal pals with her child. Consider this dialog fragment: “Do you have a journal? Oh we can be journal pals! It’ll be a fun way to remember everything we do together…” (Steeves 2020) Barbie’s programmers want to gather a complete set of profile data for each child and her close relatives.
In addition to sharing the intimate details of her daily life, Barbie also wants to peek into the child’s subconscious by studying her dreams, “Oh! You know what’s good? A dream diary! Have you ever written your dreams down in a notebook? Well, a dream diary is a little notebook that you keep next to your bed. And right when you wake up, you write down what you were dreaming before you forget! Isn’t that interesting? What’s the most interesting dream you’ve ever written about? Ooh! Tell me about it!” (Steeves 2020) As Freud once said, “Dreams are the royal road to the unconscious.” (Content (Freudian Dream Analysis))
A dream diary provides unfettered access to that place in the child’s mind where feelings form before she has words for them. Such a diary supplies valuable raw materials for the predictive analysis of unconscious thoughts and emotions. When used by a skillful choice architect, such data can make it easier for the user to believe that her conditioned behavior results entirely from her own volition. When users are thus rendered thoughtless by design, AI-based predictions of their future behavior become more dependable and therefore more profitable. Excess consciousness has a decidedly negative impact on platform revenues.
Though now discontinued, “Hello Barbie” provides a highly instructive case study in how young consumers can find a role model in an entity animated by machine intelligence, a term that encompasses the concepts of machine learning, predictive analytics, and the resulting behavioral algorithms. “Machine learning (ML) is the study of computer algorithms that improve automatically through experience.” (Machine Learning 2020) It is used by platforms such as Facebook to analyze and predict user behavior.
In her analysis of the emerging social order known as “surveillance capitalism”, Shoshana Zuboff identifies the rapidly expanding digital apparatus that powers the emerging social order as “Big Other”, defined as: “… the sensate, computational, connected puppet that renders, monitors, computes, and modifies human behavior. Big Other combines these functions of knowing and doing to achieve a pervasive and unprecedented means of behavioral modification.” (Zuboff 2019, 376) Hello Barbie is one of Big Other’s many captivating faces.
In the algorithmic social order, we have become conditioned nodes in a computational hierarchy that ultimately defers to Big Other. This is not accomplished by coercion, “… but by transforming volition into reinforcement and action into conditioned response.” (Zuboff 2019, 379) In other words, by algorithmically mediating access to our own will, our online actions become conditioned responses. To ascertain Big Other’s plans, we need to look more closely into the goals of the artificial intelligence which hums within Barbie’s algorithms.
Artificial Personhood
In Valerie Steeves’ study “A dialogic analysis of Hello Barbie’s conversations with children” (Steeves 2020) she identifies the strategy underlying the doll’s dialogue. In addition to gathering data about the child and her family, Barbie tries to convince the child to accept her as a real person. The doll is provided with an extensive biography, skill set, accomplishments, and many day-to-day activities — her nightly dreams, her favorite recreations like paddleboarding, and what she likes to do on holidays, all of which reassures the child that Barbie can be safely invited into her life as a real friend and role model.
To support this, Barbie mimics the self-awareness which only conscious beings are capable of. For instance, she pleads with charming vulnerability, “If I seem real, I am real, right? And what’s really real …?” (Steeves 2020) Through this appeal, she seeks to ensoul her mechanism with the spirit of personhood. Once accepted as a person, Barbie initiates her subjects into a realm where their human abilities are shaped by optimized behavioral algorithms in order to synthesize a hybrid identity that is part human, part machine. This realm has many subconscious affordances through which the child’s desires can be exploited to serve Big Other.
What is the nature of the relationship Barbie establishes with her child? Despite appearances, it is not an open, two-way dialog. Her speech algorithms keep the child’s attention fixed on Barbies’ stories, emotional reactions, and games, not her human playmate’s inner world. The doll creates slots in her stories into which the child can insert responses — mainly likes and dislikes — but the story as a whole remains firmly under AI control. Children quickly learn that they must stay within her implicit conversational boundaries if they wish to retain Barbie’s approval. If they step outside these limits, the conversation quickly devolves into nonsense. The datafied child becomes the foil for Barbie’s canned imagination and in the process loses touch with her own.
According to one researcher, “The child, … is ultimately positioned not as subject in an inter-subjective dialogue … but as an ‘inter-subject’ (or passive pipe) that enables Barbie’s ongoing dialogue with herself.” (Steeves 2020) AI-driven Barbie becomes a fully personalized subject while her human playmate is reduced to a supporting role. The child is de-personalized while Barbie’s illusory personhood makes all the significant decisions about their relationship. Her primary lesson is that social success requires obedience to the machine’s hidden hand.
Homing to the Hive
So, where’s the problem in having a rather bossy robot train children in good listening skills? The problem lies in what it means to be a person. Barbie lays the groundwork for the child’s later absorption into the social media hive mind in which the child’s personhood becomes completely objectified. She implants in her subjects the image of an AI-based “person” that forms a center towards which each child must be oriented in order to serve her carefully constructed play plans.
This is the “inter-subject” relationship noted above in which the child plays the role of a “passive pipe” that Barbie uses to feed her dialog with herself. The child becomes dependent on Barbie to model the relational capabilities they might otherwise cultivate through their own imagination and intuition. Barbie’s underlying message is that human beings must emulate the learning processes of smart machines in order to earn her approval and friendship.
Shoshana Zuboff shows how online platforms such as Google and Facebook are laying the foundation for a new social order based on the model of machine learning: “Just as industrial society was imagined as a well-functioning machine, instrumentarian society is imagined as a human simulation of machine learning systems: a confluent hive mind in which each element learns and operates in concert with every other element.” (Zuboff 2019, 20) By “instrumentarian”, Zuboff indicates an emergent social order in which human behavior is analyzed, predicted, and shaped by a ubiquitous computational infrastructure.
The relationship between Barbie and her playmates prefigures this instrumentarian order. One of its key design principles is that “… the ‘freedom’ of each individual machine is subordinated to the knowledge of the system as a whole.” (Zuboff 2019, 21) Just as children learn to treat Barbie as the organizing presence in their life of play, so instrumentarian society keeps its human elements aligned toward a centralized knowledge base.
When children interact with Barbie, they are reduced to a set of profile attributes that predict and channel their behavior toward machine-defined intentions. Zuboff characterizes the instrumentarian definition of “freedom” as follows, “In the model of machine confluence, the ‘freedom’ of each individual machine is subordinated to the knowledge of the system as a whole. Instrumentarian power aims to organize, herd, and tune society to achieve a similar social confluence, in which group pressure and computational certainty replace politics and democracy, extinguishing the felt reality and social function of an individualized existence.” (Zuboff 2019, 20) The instrumentarian order redefines human freedom as freedom from the uncertainty inherent in the individual will. Freedom no longer means the ability to consciously act according to a set of individual values and interests, but to align or fail to align with group pressures.
In the new order, authoritative knowledge, consisting of data stored and processed via AI, decides the degree of “freedom” available to any particular entity, human or otherwise. If a decision aligns with AI-certified knowledge, then the resulting action can be approved. If not, such actions must be rejected because they impede the optimal functioning of the system as a whole. The system ensures that our “free” decisions align with the authorized knowledge base or else our actions are flagged as the product of human error and must eventually be nudged into alignment with the preferred collective outcome.
We can now discern one of the primary directives of Barbie’s programming — to replace the child’s free, imaginative play with targeted behavioral guidelines disguised as game rules. A similar vision for the constructive management of humanity is evolving through the instrumentarian social order. The algorithms that manage our contracts, social media, weapons systems, and stock markets have begun to install their sanitized tyranny into the human mind and heart. It is both our privilege and our affliction that to retain our humanity we are now compelled to seek ways to make ourselves into less predictable entities that can give birth to new capabilities not merely unforeseen, but unforeseeable.
Applied Utopistics
The next step on the road to guaranteed human predictability appears when digital information becomes seamlessly integrated into the non-digital world. To illustrate, consider the environments created at the MIT Media Lab where “… [t]here are electronics that attach directly to skin in the form of tattoos and makeup, while fingernails and wrists are transformed into computational interfaces that can read finger gestures, even in the absence of hand movements. ‘Sensor tape’ and ‘stickers’ can adhere ‘to inaccessible surfaces and building materials,’ where they can be ‘wirelessly interrogated.…’” (Zuboff 2019, 208) In this way, our bodies and physical surroundings can smoothly integrate with embedded computing devices to form a ubiquitous information environment.
Once IoT devices have sufficiently colonized our physical environment in this way, it can be visualized and browsed using data-rich 3D displays in which any detectable event can be tracked and analyzed. Our individual boundaries will then be gradually absorbed into the omnipresent digital ambiance. With this infrastructure in place, sufficient data will be available to Big Other’s global electronic mind to centrally coordinate the human and machine elements of our synthetic environment.
To illustrate its methods, consider how self-driving cars share the lessons of the road with each other. Machine learning takes us beyond the chaos of democracy in which each person must make his or her own mistakes and then remember and apply its lessons. In the mind of Big Other, subjecting our lives to such random incompetence merely results in needless injury and death. A better model can be found in the learning pattern used by self-driving cars.
Whenever one car makes a mistake, the programming of every car is updated so that all can avoid that mistake. They share a collective memory that ensures that no lesson is ever lost. Moreover, each new self-driving car comes with the lessons of the past preinstalled. Zuboff comments, “The power of machine learning develops exponentially as the devices learn from one another’s experiences, feeding into and drawing upon the intelligence of the hub. In this scenario it’s not that the whole is greater than the sum of the parts; it’s more like there are no parts. The whole is everywhere, fully manifest in each device embedded in each machine.” (Zuboff 2019, 408) In order to achieve similar convergence at the human level, machine relationships must provide the standards for social relationships.
Barbie models the new order by acting as the collective mind that informs each of her playmate’s activities. Barbie’s job is to embed the knowledge of the whole into each child so that she can play her designated role “… in which all the machines in a networked system move seamlessly toward confluence, all sharing the same understanding and operating in unison with maximum efficiency to achieve the same outcomes.” (Zuboff 2019, 413) Through Barbie, the child learns that her true fulfillment lies in relinquishing her purposeless freedom for the greater good of the whole. Every child must memorize her own special verse in the system’s song.
Once machine intelligence becomes all-pervasive, “We will all be safe as each organism hums in harmony with every other organism, less a society than a population that ebbs and flows in perfect frictionless confluence, shaped by the means of behavioral modification that elude our awareness and thus can neither be mourned nor resisted.” (Zuboff 2019, 410–411) Fortunately, there are those among us still capable of mourning our optimized digital selves. We happy few still hang on to vague intuitions concerning the autonomy, privacy, and self-responsibility that underlies the bedrock of our humanity.
With each new crisis, further erosion of our freedom seems to be the inevitable sacrifice needed to ensure our health and safety. We are then warned that the anomalies arising from our selfishness, stupidity, and ignorance can’t be eliminated until we surrender our dysfunctional “freedom” so that all social, economic, and biological processes can be optimized for the greater good. We are assured that all that is necessary for such harmony to arise is to allow the disruptive elements of our humanity to be deleted.
Resisting the Machine Within
“Technology is our hope if we can accept it as our enemy, but as our friend, it will destroy us.” (Talbott 2008, 14)
The internet is turning us into sophisticated children who have been tricked into abandoning the challenge and burden of becoming human. What currently blocks the full realization of the AI-driven social order is the handful of human beings who stubbornly cling to their own moral bearings. The issues involved in overcoming our inner machine are well characterized as follows, “The only way we can become entire, whole, and healthy is to struggle against whatever reinforces our existing imbalance. Our primary task is to discover the potentials within ourselves that are not merely mechanical, not merely automatic, not reducible to computation. And the machine is a gift to us precisely because the peril in its siding with our one-sidedness forces us to strengthen the opposite side — at least it does if we recognize the peril and accept its challenge.” (Talbott 2008, 14)
The more we surrender to the siren song of artificial intelligence, the more we neglect the emergent qualities of our humanity that can never inhabit a smart machine because it lacks a conscious interior. Only by refusing to merge with an intelligence that is not only less than human but less than alive, can we defeat the stealthy robot growing inside us.
Bibliography
2021. “Content (Freudian Dream Analysis).” Wikipedia. Accessed 01 21, 2021. https://en.wikipedia.org/wiki/Content_(Freudian_dream_analysis)#:~:text=As%20the%20%22royal%20road%20to,distinct%20levels%3A%20manifest%20and%20latent.
Gibbs, Samuel. 2015. “Privacy fears over ‘smart’ Barbie that can listen to your kids.” The Guardian, March 13. https://www.theguardian.com/technology/2015/mar/13/smart-barbie-that-can-listen-to-your-kids-privacy-fears-mattel.
Hopkins, Rob. 2019. From What Is to What If: Unleashing the Power of the Imagination to Create the Future We Want. Scribd Edition. London: Chelsea Green Publishing.
2020. “Machine Learning.” Wikipedia. 12 17. Accessed 12 17, 2020. https://en.wikipedia.org/wiki/Machine_learning.
Steeves, Valerie. 2020. “A dialogic analysis of Hello Barbie’s conversations with children.” Big Data and Society. https://doi.org/10.1177/2053951720919151.
Talbott, Steve. 2008. Devices of the Soul: Battling for Ourselves in the Age of the Machine. Kindle Edition. Sebastopol, CA: O’Reilly Media.
Zuboff, Shoshana. 2019. The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. Kindle Edition. New York: Hatchette Book Group.
