Weapons of Math Destruction by Cathy O'Niel

Data is big business today. Algorithms are used to sift through information and make evaluations and predictions. But can it be misused? Can automatic systems of information yield wrong results?

O'Niel says yes. She provides plenty of anecdotal evidence showing how attempting to reduce human behavior to algorithms can be destructive. The math can be misused. The results can be misinterpreted. The models devised can actually be detrimental.

She gives us a little background on models, helping us understand how they are created and adjusted. Then she takes us through examples of misuse of models and the information they produce. Examples include teacher assessment, prisoner evaluation, risk for financial institutions, college ranking, online marketing (predatory advertising), filtering job or loan candidates, determining working hours, and more.

She shows how corporations and institutions can lose sight of the actual individual. Algorithms can be used to impress rather than clarify (as happened with the 2008 financial collapse). Wrong data can be on a credit report. The possible errors go on and on.

If you have wondered why some of your friends on Facebook see your latest post and others do not, it is because of the algorithm. If you wondered how your car insurance price was determined, it may well have nothing to do with your actual driving record.

O'Niel has given readers an understandable critique of current data collection and analysis techniques. But the system could not be dismantled easily. Since is is usually the poor and disenfranchised who are hurt most frequently, the push for change is weak. O'Neil does give some suggestions for regulation of mathematical models. She also suggests that models having significant impact on the public should be open and available to the public. There may be hope for the future as some are even now seeing the problems. Whether users of models will admit to the responsibility that use entails is yet to be seen.

My rating: 4/5 stars.

Cathy O'Niel is a data scientist and bogs at https://mathbabe.org/. She has a PhD in mathematics and taught at Barnard College before moving to the private sector. She worked at a hedge fund and then for various start-ups. She started the Lede Program in Data Journalism at Columbia and is the author of Doing Data Science. She appears weekly on the Slate Money podcast.

Crown Publishing, 272 pages.

I received a complimentary copy of this book from the publisher. My comments are an independent and honest review.