The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power

Chapter 13 dives deeper into the mechanisms of instrumentarianism. Instrumentarianism relies on the apparatus—the “Big Other”—to monitor and datify individuals’ behavior. This is the first step in modifying behavior with the eventual goal of producing guaranteed behavioral outcomes. Zuboff explains that the Big Other sees people as “organisms that behave,” fulfilling Skinner’s goals of radical behaviorism and his interpretations of a person merely as an “Other-One” (354).

Because instrumentarianism does not express itself through violence as totalitarianism did, people are less prone to view it as a serious threat. But, Zuboff warns, the apparatus operates in other dangerous ways. If surveillance capitalism’s instrumentarian goals come into fruition, Big Other will present the illusion of choice in all aspects of life, and the wrong decision will be made at an individual’s peril.

Instrumentarian power can already be seen at work today. Chapter 13 turns its attention towards China, whose government is working towards a “social credit” system for its internet. This social credit system tracks the personal data of citizens, scores their behavior (by what they buy, what sites they visit, etc.), and stores these behavioral scores on a database that can be accessed by the government and private enterprises alike. This is all for the ostensible purpose of “improv[ing] citizens’ behavior” (364). Zuboff argues that although this is popularly critiqued as totalitarianism, it is more aptly understood as instrumentarian power, because its operations follow surveillance capitalism’s imperatives and economies of action.

However, Zuboff also points out that that while the Chinese model is dismal picture of a new, emergent global reality, cultural differences—such as less emphasis placed on ideas of privacy in the East versus the West—and political differences—democratic systems versus China’s one-party state—suggest that their model is not a direct image of the entire world’s future. Nevertheless, Zuboff asks readers to recall the “fork in the road” that was before the 20th century citizens of the 1930s who were facing down the barrel of totalitarianism, with no clue as to what was ahead. The 21st century fork is between the synthetic declaration, which proposes a third modernity where technology is harnessed and used for democratic public benefit, and that of an instrumentarian third modernity, which is already foreshadowed by the Chinese social credit system.

Linking to the image she drew in the previous chapter’s conclusion with a “fork in the road” leading today’s society down instrumentarianism or democracy, Zuboff opens Chapter 14 by warning that the processes of normalization and habituation are already well underway, drawing instrumentarianism closer and closer.

Surveillance capitalism seeks to create a utopia of certainty. This notion has been described by Facebook CEO Mark Zuckerberg, who pushed for Facebook to lay the groundwork for a new global community that will create a contemporary streamlined utopia. Zuckerberg insisted on three company goals for the future: connecting everyone, understanding the world, and building a knowledge economy. Such comments are important because they reflect how surveillance capitalists can be classified as utopians.

Zuboff argues that surveillance capitalists often employ images of social progress that fit nearly all of the categories of utopian philosophy, as defined by utopian scholars Frank and Fritzie Manuel. However, the most important aspect is how surveillance capitalism defies utopian thought. Utopian theory and putting such theory into practice involves an “unavoidable gap” which allows for critical thought and reflection as to the proper way forward. As Zuboff explains, “we can question whether a law or governmental practice is consistent with a nation’s constitution [...] because we can inspect, interpret, and debate [such] documents. If the gap is too great, citizens act to close the gap by challenging the law or practice” (381). Surveillance capitalists reverse this relationship, as their actions occur before any theory can be formulated, thus avoiding the critical thought and analysis of the people.

Microsoft CEO Satya Nadella’s partnership with a Swedish manufacturer illustrates how theory and practice are applied simultaneously by surveillance capitalists. Microsoft forged a partnership with a manufacturer to install AI technology at a Swedish factory where they cut metal. Microsoft linked factory machines together in a telemetric network that streamed data to a Microsoft cloud where anomalies that put machines at risk could be located. The AI then preemptively shut down a potentially dangerous or nonfunctioning piece of equipment in a matter of seconds. Because it displays Microsoft’s power of anticipating an event and preemptively responding to that event, Nadella then “translat[ed] these facts into their practical application, observing that once you have lots of devices around, an ‘ad hoc center’ is created ‘on a factory floor, at home, or anywhere else. [...] You can turn any place into a safe, AI-driven place’” (383).

Similar ideas were used to create a patent application by Microsoft that proposed a device to monitor users’ behavior for preemptive detection of deviations from normal behaviors, reflecting how the theory of instrumentarianism is already being put into practice. Zuboff concludes by noting that surveillance capitalists wish to draw humanity closer to machinery, where human freedom is set aside for the desires of a collective predictability, and where society marches along the same path of guaranteed action.

Chapter 15 reviews the work of Alex Pentland, the director of the Human Dynamics Lab at MIT’s Media Lab. Pentland is of special importance because his studies reflect the burgeoning institutionalization of instrumentarian theory and practice.

Pentland favored Skinner’s ideas of objective, distanced observations of people—something he referred to as the “God’s eye view” (392). Working with groups of doctoral students, Pentland identified social relations as a promising, untapped resource of data. In 2005, his team saw the cell phone as an important device that would extend the reach of the digital network in unprecedented ways. The scientists sought to explore the potentials of tapping into this ubiquitous network and, in a study with 100 MIT students and faculty equipped with Nokia phones, found that one could predict behavior with great accuracy using digital behavioral records alone.

Continuing his work in 2009, Pentland experimented with a wearable computing platform that observed social relationships and analyzed them through machine intelligence. The computer revealed behaviors that would otherwise go unknown, causing Pentland to “argue strongly for the use of automatic sensing data collection tools to understand social systems” (397) and propose that employers have their employees wear the devices to record performance metrics and extract productivity behavior.

Zuboff conducts a close read of Pentland’s 2011 essay, “Society’s Nervous System: Building Effective Government, Energy, and Public Health Systems,” in an effort to better understand his ideology. In this essay, the scientist argued that the apparatus of ubiquitous technology would not be truly successful until it understood human behavior on a global level. The essay reflected Pentland’s true goal of monitoring and controlling behavior on a worldwide scale.

Zuboff terms Pentland’s ideology as “social physics.” Pentland’s social physics reflects a theoretical vision for a new society determined by establishing laws of social behavior, similar in objectivity to the laws of physics. These social physics, Zuboff argues, rely on five foundational principles that would determine a new, harmonious utopian society.

These five principles include behavior for the greater good, where everybody’s behavior must be monitored and controlled for the supposed purpose of bettering humanity at large; plans replacing politics, wherein behavioral prediction and modification are perfected and total control is achieved, thus removing the need for debate and democratic conversations over who will be in power; social pressure for harmony, Pentland’s version of reinforcement which includes encouraging certain behaviors within relationships as opposed to targeting individuals’ behaviors; applied utopistics, which dictates that instrumentarian control over the means of behavioral modification will help society achieve utopia; and finally, the death of individuality. Under social physics, individual thoughts, beliefs, and feelings only serve as friction to collective harmony; the individual’s free will is viewed as an irrational concept, and a total social cohesion is its rational alternative.

Zuboff concludes Chapter 15 by reflecting on the fact that instrumentarianism is the social culmination of surveillance capitalism’s division of learning. Therefore, Pentland’s work, which encourages the institutionalization of instrumentarianism, is a deathly threat to third modernity and the individual’s free will.

Zuboff describes an experiment involving 1,000 students around the globe who were asked to go 24 hours without using digital media. Many of the experiment’s subjects reported that they felt isolated, lonely, and lost over the course of their 24 hours.

Zuboff reflects on this experiment and notes that the emotional strain these students felt foreshadows what the world’s future might look like in an instrumentarian society, where surveillance capitalists work to create dependency and constant use of their digital media.

This chapter focuses on youth under surveillance capitalism. It examines the psychological impacts younger generations have suffered growing up in the age of social media. Zuboff notes that Generation Z—those born in 1996 and afterwards—are a particularly important demographic to analyze because they grew up entirely under surveillance capitalism, whereas those born before 1996 were adults by the time surveillance capitalism sprouted and flourished.

Recent studies on social media have made the alarming discovery that engaging with these sites encourages an addiction, uncovering how social media drives younger generations to engage in automated behavior. This happens because social media is specifically designed to manipulate the psychology of the adolescent mind, which typically grapples with questions of self-identity more than the adult mind.

Important in the discussion of how social media sites target the adolescent mind is the concept of “emerging adulthood,” when those between ages 18 and 29 struggle to reconcile their understanding of self with respect to others. At this juncture of life, emerging adults tend to rely on social comparison, looking to others to resolve their internal conflict. Facebook is constructed to exploit the psychological vulnerabilities that can exist in social comparison through tools such as the “Like” button, targeted ads, and curation tools that create specific feeds unique to different users.

Social media heightens the sense of comparison to others to a nearly unbearable degree, creating dynamics such as “FOMO” (fear of missing out), whereby individuals become compulsive users of social media and obsessively check their feeds. As Zuboff explains, “[C]ompulsive behavior is intended to produce relief in the form of social reassurance, but it predictably breeds more anxiety and more searching” (434). Because posts on websites such as Facebook present curated, idealized images of peoples’ lives, adolescent users can become insecure and anxious about their own life, breeding feelings of inferiority—thus only fueling the cycle of social comparison and compulsive use.

Researchers who have studied the mental and emotional tolls of FOMO and compulsive use found that these dynamics lead to depression and self-objectification, a phenomenon which occurs when a person sees oneself as an object to “market” to others in social media posts. Zuboff notes that these alarming findings about Facebook’s effects on the current youth are not a prophecy of something to come; rather, they are the first steps of an instrumentarian future, in which everybody will suffer the emotional turmoil that young social media users are burdened with today.

In the final chapter of Part 3, Zuboff posits that the idea of “sanctuary” will be the first casualty of surveillance capitalism, because privacy, solitude, and individual space—both physical and psychological—are enemies of behavioral data surplus collection.

Zuboff then tackles the question of how sanctuary can be protected. In the United States, government-level protections of one’s sanctuary are thin, largely because the American legal system depends on the Fourth Amendment when dealing with cases concerning digital attacks on existing protections. Zuboff observes that the issue with the Fourth Amendment with regard to surveillance capitalism is that it is specifically about protecting citizens from the state; however, surveillance capitalism is a corporate enemy, not a governmental one.

She then argues that a better model of future protections lies in the European Union. In May 2018, the General Data Protection Regulation was introduced to the EU. Its regulations require companies to notify users when personal data is compromised, levels heavy fines for violations, and even allows for users to pursue class-action lawsuits against companies.

Even still, Zuboff insists that more systemic change is needed if sanctuary is truly to be protected, because individuals alone will not be able to fight the assertive tide of surveillance capitalism. To make her point, Zuboff turns to the example of Paul-Olivier Dehaye, a Belgian mathematician and data protection activist who, in 2016, requested the sum of his personal data from Facebook. Dehaye wanted to determine the body of information that Facebook had compiled on him through his years of using the website. After a long chain of emails, Facebook told him that his data was not readily available because Facebook stored it in the Hive, “where it is retained for ‘data analytics’ and maintained as separate from ‘the data bases that power the Facebook site.’ […] ‘This data,’ the company writes, ‘is also not used to directly serve the live Facebook website which users experience’” (452). Zuboff notes that this statement from Facebook is especially important because it is a direct admission of the existence of the “shadow text” that maintains surveillance capitalism’s asymmetries of knowledge and power, rendering individuals virtually powerless. Thus, only large-scale, systemic change can offer true protection of sanctuary.

Zuboff dedicates the remainder of Chapter 17 to describing different modes of resistance—all in the form of the act of hiding—that an emerging generation of artists, activists, and students have turned to. The younger generations have worked to perfect the art of hiding by using devices such as visors and masks that prevent facial recognition, false fingerprint prosthetics, and signal-blocking phone cases. While these are significant actions that represent future generations’ drive to fight for their liberties, Zuboff argues that such measures are also indicative that society is on an extremely concerning path toward instrumentarianism.