Big Data Isn’t Just Watching You—It’s Making You Poorer

Cathy O’Neil’s new book, Weapons of Math Destruction, shows mathematical models aren’t free of ideology.

Pankaj MehtaSeptember 6, 2016

Big Data isn’t just watching you, O’Neil writes—it’s making you poorer. (Shutterstock)

By some esti­mates, human­i­ty now pro­duces 2.5 quin­til­lion bytes of data every day — more than a hun­dred times the amount of data in the entire Library of Con­gress. This data ranges from Face­book posts to mil­i­tary-grade satel­lite pho­tos. Increas­ing­ly, this data is ana­lyzed by com­plex math­e­mat­i­cal mod­els that deter­mine more and more aspects of our lives, from the adver­tise­ments we see to whether we have access to pri­vate insur­ance. Yet despite their grow­ing impor­tance, these mod­els often remain hidden. 

O’Neil makes a convincing case that many mathematical models today are engineered to benefit the powerful at the expense of the powerless.

Advo­cates of such math­e­mat­i­cal mod­el­ing, in both the pub­lic and pri­vate sec­tors, por­tray it as a neu­tral and effi­cient alter­na­tive to fal­li­ble and biased human deci­sion-mak­ing. Math­e­mati­cian, data sci­en­tist and pop­u­lar blog­ger Cathy O’Neil, author of Weapons of Math Destruc­tion: How Big Data Increas­es Inequal­i­ty and Threat­ens Democ­ra­cy, doesn’t agree. She argues that many math­e­mat­i­cal mod­els are ide­o­log­i­cal tools that exac­er­bate oppres­sion and inequal­i­ty. Her exam­ples range from the crime mod­els used by police depart­ments to deter­mine which neigh­bor­hoods to patrol, to the recidi­vism mod­els used by judges to hand out prison sentences.

O’Neil is pas­sion­ate about expos­ing the harm­ful effects of Big Data – dri­ven math­e­mat­i­cal mod­els (what she calls WMDs), and she’s unique­ly qual­i­fied for the task. She earned a Ph.D. in math from Har­vard and land­ed a tenure-track at Barnard. But she became bored with the pace and insu­lar­i­ty of acad­e­mia, and left to work as a quan­ti­ta­tive ana­lyst at the hedge fund D.E. Shaw. There, she had a front-row seat for the 2008 finan­cial crisis. 

This expe­ri­ence fun­da­men­tal­ly changed O’Neil’s rela­tion­ship to math­e­mat­ics. She real­ized that far from being a neu­tral object of study, math­e­mat­ics was not only deeply entan­gled in the world’s prob­lems but also fuel­ing many of them.” Peo­ple in pow­er were delib­er­ate­ly [wield­ing] for­mu­las to impress rather than clar­i­fy.” This dis­il­lu­sion­ment led O’Neil to get involved with Occu­py Wall Street and start edu­cat­ing the pub­lic about the dan­gers of WMDs through her blog, MathBabe.

She is care­ful to point out that there is noth­ing inher­ent­ly destruc­tive about math­e­mat­i­cal mod­el­ing. Sophis­ti­cat­ed data mod­el­ing enables much of mod­ern tech­nol­o­gy, from wire­less com­mu­ni­ca­tion to drug dis­cov­ery. How can one dis­tin­guish a destruc­tive math mod­el from an ordi­nary, or even help­ful, one? O’Neil iden­ti­fies three key fea­tures of WMDs: lack of trans­paren­cy, lack of fair­ness and, most impor­tant­ly, oper­a­tion on a mas­sive scale.

O’Neil grounds her argu­ment in case stud­ies of WMDs in a vari­ety of set­tings: finance, high­er edu­ca­tion, the crim­i­nal jus­tice sys­tem, online adver­tis­ing, employ­ment deci­sions and sched­ul­ing, and cred­it and insur­ance pro­vi­sion. The val­ue-added” mod­el for teacher eval­u­a­tion, which looks at improve­ments in indi­vid­ual stu­dents’ test scores — a favorite of the so-called edu­ca­tion­al reform” move­ment — is tout­ed as an objec­tive mea­sure of a teacher’s worth. Yet this is far from the truth. O’Neil cites an analy­sis by blog­ger and edu­ca­tor Gary Rubin­stein of New York’s 2010 val­ue-added scores for pub­lic school teach­ers. O’Neil explains that Of teach­ers who taught the same sub­ject in con­sec­u­tive years, one in four reg­is­tered a 40-point dif­fer­ence. [This] sug­gests that the eval­u­a­tion data is prac­ti­cal­ly ran­dom.” O’Neil argues that this is because the val­ue-added mod­el, which relies on pre­dic­tions of stu­dent per­for­mance, suf­fers from a built-in log­i­cal flaw: No sta­tis­ti­cal mod­el can accu­rate­ly make pre­dic­tions about a class of 25 or 30 stu­dents — the sam­ple size is too small. Yet the high-stakes test­ing régime con­tin­ues to wreak hav­oc on the tra­jec­to­ries of stu­dents and teach­ers alike.

Oth­er WMDs, known as e‑scores, use data such as ZIP codes, web-surf­ing pat­terns and recent pur­chas­es to eval­u­ate a person’s cred­it-wor­thi­ness. Unlike the more famil­iar FICO cred­it scores that are freely avail­able and reg­u­lat­ed by the gov­ern­ment, these secre­tive e‑scores are unac­count­able, unreg­u­lat­ed and often unfair.” Where­as FICO scores are based on your own finan­cial his­to­ry, e‑scores com­pare you to oth­er peo­ple with sim­i­lar pro­files. This may seem benign, but it can result in feed­back loops that rein­force exist­ing social inequities. If you live in a poor ZIP code, then your e‑score will drop, mean­ing less cred­it and high­er inter­est rates — essen­tial­ly, an algo­rith­mic redlin­ing of the poor and work­ing class.

Through these exam­ples, O’Neil makes a con­vinc­ing case that many math­e­mat­i­cal mod­els today are engi­neered to ben­e­fit the pow­er­ful at the expense of the pow­er­less. The prob­lem with most WMDs, O’Neil argues, is that it’s almost always their objec­tive” to max­i­mize the bot­tom line” rather than fair­ness” or oth­er eth­i­cal pri­or­i­ties.” By enact­ing these choic­es, WMDs serve an impor­tant ide­o­log­i­cal func­tion in mod­ern cap­i­tal­ism. Far from elim­i­nat­ing human bias, O’Neil sug­gests, WMDs cam­ou­flage it with technology.” 

To para­phrase the great evo­lu­tion­ary biol­o­gist Richard Lewon­tin, WMDs are math as ide­ol­o­gy.” This argu­ment is implic­it through­out Weapons of Math Destruc­tion, but the book would have ben­e­fit­ed from an explic­it dis­cus­sion of the rela­tion­ship between WMDs and cap­i­tal­ism. This is a minor short­com­ing, how­ev­er. Cathy O’Neil has writ­ten an enter­tain­ing and time­ly book that gives read­ers the tools to cut through the ide­o­log­i­cal fog obscur­ing the dan­gers of the Big Data revolution.

Pankaj Mehta is a physi­cist and activist liv­ing in the Boston area.
Limited Time: