Big Data Isn’t Just Watching You—It’s Making You Poorer

Cathy O’Neil’s new book, Weapons of Math Destruction, shows mathematical models aren’t free of ideology.

Pankaj Mehta

Big Data isn’t just watching you, O’Neil writes—it’s making you poorer. (Shutterstock)

By some estimates, humanity now produces 2.5 quintillion bytes of data every day — more than a hundred times the amount of data in the entire Library of Congress. This data ranges from Facebook posts to military-grade satellite photos. Increasingly, this data is analyzed by complex mathematical models that determine more and more aspects of our lives, from the advertisements we see to whether we have access to private insurance. Yet despite their growing importance, these models often remain hidden. 

O’Neil makes a convincing case that many mathematical models today are engineered to benefit the powerful at the expense of the powerless.

Advocates of such mathematical modeling, in both the public and private sectors, portray it as a neutral and efficient alternative to fallible and biased human decision-making. Mathematician, data scientist and popular blogger Cathy O’Neil, author of Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, doesn’t agree. She argues that many mathematical models are ideological tools that exacerbate oppression and inequality. Her examples range from the crime models used by police departments to determine which neighborhoods to patrol, to the recidivism models used by judges to hand out prison sentences.

O’Neil is passionate about exposing the harmful effects of Big Data – driven mathematical models (what she calls WMDs), and she’s uniquely qualified for the task. She earned a Ph.D. in math from Harvard and landed a tenure-track at Barnard. But she became bored with the pace and insularity of academia, and left to work as a quantitative analyst at the hedge fund D.E. Shaw. There, she had a front-row seat for the 2008 financial crisis. 

This experience fundamentally changed O’Neil’s relationship to mathematics. She realized that far from being a neutral object of study, mathematics was not only deeply entangled in the world’s problems but also fueling many of them.” People in power were deliberately [wielding] formulas to impress rather than clarify.” This disillusionment led O’Neil to get involved with Occupy Wall Street and start educating the public about the dangers of WMDs through her blog, MathBabe.

She is careful to point out that there is nothing inherently destructive about mathematical modeling. Sophisticated data modeling enables much of modern technology, from wireless communication to drug discovery. How can one distinguish a destructive math model from an ordinary, or even helpful, one? O’Neil identifies three key features of WMDs: lack of transparency, lack of fairness and, most importantly, operation on a massive scale.

O’Neil grounds her argument in case studies of WMDs in a variety of settings: finance, higher education, the criminal justice system, online advertising, employment decisions and scheduling, and credit and insurance provision. The value-added” model for teacher evaluation, which looks at improvements in individual students’ test scores — a favorite of the so-called educational reform” movement — is touted as an objective measure of a teacher’s worth. Yet this is far from the truth. O’Neil cites an analysis by blogger and educator Gary Rubinstein of New York’s 2010 value-added scores for public school teachers. O’Neil explains that Of teachers who taught the same subject in consecutive years, one in four registered a 40-point difference. [This] suggests that the evaluation data is practically random.” O’Neil argues that this is because the value-added model, which relies on predictions of student performance, suffers from a built-in logical flaw: No statistical model can accurately make predictions about a class of 25 or 30 students — the sample size is too small. Yet the high-stakes testing regime continues to wreak havoc on the trajectories of students and teachers alike.

Other WMDs, known as e-scores, use data such as ZIP codes, web-surfing patterns and recent purchases to evaluate a person’s credit-worthiness. Unlike the more familiar FICO credit scores that are freely available and regulated by the government, these secretive e-scores are unaccountable, unregulated and often unfair.” Whereas FICO scores are based on your own financial history, e-scores compare you to other people with similar profiles. This may seem benign, but it can result in feedback loops that reinforce existing social inequities. If you live in a poor ZIP code, then your e-score will drop, meaning less credit and higher interest rates — essentially, an algorithmic redlining of the poor and working class.

Through these examples, O’Neil makes a convincing case that many mathematical models today are engineered to benefit the powerful at the expense of the powerless. The problem with most WMDs, O’Neil argues, is that it’s almost always their objective” to maximize the bottom line” rather than fairness” or other ethical priorities.” By enacting these choices, WMDs serve an important ideological function in modern capitalism. Far from eliminating human bias, O’Neil suggests, WMDs camouflage it with technology.” 

To paraphrase the great evolutionary biologist Richard Lewontin, WMDs are math as ideology.” This argument is implicit throughout Weapons of Math Destruction, but the book would have benefited from an explicit discussion of the relationship between WMDs and capitalism. This is a minor shortcoming, however. Cathy O’Neil has written an entertaining and timely book that gives readers the tools to cut through the ideological fog obscuring the dangers of the Big Data revolution.

Help In These Times Celebrate & Have Your Gift Matched!

In These Times is proud to share that we were recently awarded the 16th Annual Izzy Award from the Park Center for Independent Media at Ithaca College. The Izzy Award goes to an independent outlet, journalist or producer for contributions to culture, politics or journalism created outside traditional corporate structures.

Fellow 2024 Izzy awardees include Trina Reynolds-Tyler and Sarah Conway for their joint investigative series “Missing In Chicago," and journalists Mohammed El-Kurd and Lynzy Billing. The Izzy judges also gave special recognition to Democracy Now! for coverage that documented the destruction wreaked in Gaza and raised Palestinian voices to public awareness.

In These Times is proud to stand alongside our fellow awardees in accepting the 2024 Izzy Award. To help us continue producing award-winning journalism a generous donor has pledged to match any donation, dollar-for-dollar, up to $20,000.

Will you help In These Times celebrate and have your gift matched today? Make a tax-deductible contribution to support independent media.

Pankaj Mehta is a physicist and activist living in the Boston area.
Democratic Rep. Summer Lee, who at the time was a candidate for the state House, at a demonstration in Pittsburgh for Antwon Rose, who was killed by police, in 2018. Lee recently defeated her 2024 primary challenger.
Get 10 issues for $19.95

Subscribe to the print magazine.