Mlodinow: The Drunkard's Walk

From Scienticity

---

Revision as of 03:58, 22 March 2009

Scienticity:
Readability:
Hermeneutics:
Charisma:
Recommendation:
Ratings are described on the Book-note ratings page.

Leonard Mlodinow, The Drunkard's Walk : How Randomness Rules our Lives. New York : Pantheon Books, 2008. xi + 252 pages; illustrated, with notes and index.

The central concepts of probability and statistics are relatively simple to say, and yet comprehending their meaning, and their implications, in practical situations is a challenge for a great many people. Accepting the challenge of helping normal people understand those concepts, Mlodinow does a good job of it by blending small and palatable sections of exposition with anecdotes and illustrative examples that actually do illustrate the ideas being discussed. From the beginning he understands both his challenge and the importance of the task.

To swim against the current of human intuition is a difficult task. As we'll see, the human mind is built to identify for each event a definite cause and can therefore have a hard time accepting the influence of unrelated or random factors. And so the first step is to realize that success or failure sometimes arises neither from great skill nor from great incompetence but from, as the economist Armen Alchaian wrote, "fortuitous circumstances." Random processes are fundamental in nature and are ubiquitous in our everyday lives, yet most people do not understand them or think much about them. [p. xi]

Mlodinow's goal is to arm the reader with enough awareness of how randomness and statistics and probability work so that she can spot weak and shady reasoning, charlatans who would take advantage of statistical innocence, and the mendacious who try to exploit any bit of ignorance they can to further their goals.

Here's a fine instance of the author's using an example that will be familiar to many to show one of the many ways that bad reasoning about probability is commonly used to exploit the naive. In this case it's the "Prosecutor's Fallacy", in which a lawyer tries to make a case using faulty reasoning about how probable events are.

The renowned attorney and Harvard Law School professor Alan Dershowitz also successfully employed the prosecutor's fallacy—to help defend O.J. Simpson in his trial for the murder of Simpson's ex-wife, Nicole Brown Simpson, and a male companion. The trial of Simpson, a former football star, was one of the biggest media events of 1994—95. The police had plenty of evidence against him. They found a bloody glove at his estate that seemed to match one found at the murder scene. Bloodstains matching Nicole's blood were found on the gloves, in his white Ford Bronco, on a pair of socks in his bedroom, and in his driveway and house. Moreover, DNA samples taken from blood at the crime scene matched O.J.'s. The defense could do little more than accuse the Los Angeles Police Department of racism—O.J. is African American—and criticize the integrity of the police and the authenticity of their evidence.

The prosecution made a decision to focus the opening of its case on O.J.'s propensity toward violence against Nicole. Prosecution spent the first ten days of the trial entering evidence of his history of abusing her and claimed that this alone was a good reason to suspect him of her murder. As they put it, "a slap is a prelude to homicide." The defense attorneys used this strategy as a launch pad for their accusations of duplicity, arguing that the prosecution had spent two weeks trying to mislead the jury and that the evidence that O.J. had battered Nicole on previous occasions meant nothing. Here is Dershowitz's reasoning: 4 million women are battered annually by husbands and boyfriends in the United States, yet in 1992, according to the FBI Uniform Crime Reports, a total of 1,432, or 1 in 2,500, were killed by their husbands or boyfriends. Therefore, the defense retorted, few men who slap or beat their domestic partners go on to murder them. True? Yes. Convincing? Yes. Relevant? No. The relevant number is not the probability that a man who batters his wife will go on to kill her (1 in 2,500) but rather the probability that a battered wife who was murdered was murdered by her abuser. According to the Uniform Crime Reports for the United States and its Possessions in 1993, the probability Dershowitz (or the prosecution) should have reported was this one: of all the battered women murdered in the United States in 1993, some 90 percent were killed by their abuser. That statistic was not mentioned at the trial. [p. 120]

All that's called for from the reader is interested attention and a clear head and, perhaps, not reading too quickly. The discussions and examples are clear, precise, and easy enough to follow, but they are not best-seller breezy. Important ideas often take time to digest; there's no harm in slow reading.

American politics and social responsibility can be difficult to follow honestly and knowledgeably at the best of times, but we need the tools to evaluate arguments in the public sphere that rely heavily on concepts from statistics and probability, often wielded by people who shouldn't be let near a statistic without adult supervision.

Similar issues [of perceiving patterns in random data] arise frequently in reports of cancer clusters. If you divide any city or country into parcels and randomly distribute incidents of cancer, some parcels will receive less than average and some more. In fact, according to Raymond Richard Neutra, chief of the Division of Environmental and Occupational Disease Control in California's Department of Health, given a typical cancer registry—a database on local rates for dozens of different cancers—for California's 5,000 census tracts, you could expect to find 2,750 with statistically significant but random elevations of some form of cancer. And if you look at a large enough number of such parcels, you'll find some regions in which cancer occurred at many times the normal rate.

The picture looks even worse if you draw the parcel boundaries after the cancers are distributed. What you get then is called the sharpshooter effect, after the apocryphal fellow who excels in his aim because he shoots at blank paper and draws the target afterward. Unfortunately that is how it usually happens in practice: first some citizens notice neighbors with cancer; then they define the boundaries of the area of issue. Thanks to the availability of data on the Internet, America these days is being scoured for such clusters. Not surprisingly, they are being found. Yet the development of cancer requires successive mutations. That means very long exposure and/or highly concentrated carcinogens. For such clusters of cancer to develop from environmental causes and show themselves in concert and before the victims have moved away from the affected area is quite a long shot. According to Neutra, to produce the kind of cancer clusters epidemiologists are typically called on to investigate, a population would have to be exposed to concentrations of carcinogens that are usually credible only in patients undergoing chemotherapy or in some work settings—far greater concentrations than people receive in contaminated neighborhoods and schools. Nevertheless, people resist accepting the explanation that the clusters are random fluctuations, and so each year state departments of health receive thousands of residential cancer-cluster reports, which result in the publication of hundreds of exhaustive analyses, none of which has convincingly identified an underlying environmental cause. Says Alan Bender, an epidemiologist with the Minnesota Department of Health, those studies "are an absolute, total, and complete waste of taxpayer dollars." [pp. 184—185]

What's a concerned citizen to do? Read this book and learn something interesting and useful would be one possibility.

As I write this crises, in financial markets are making headlines and a good dose of understanding and perspective would help the average citizen make sense of it all. One place where randomness is far more significant than many financial professionals would care to admit is the stock market, and the author draws a number of useful examples from that domain.

People systematically fail to see the role of chance in the success of ventures and in the success of people like the equity-fund manager Bill Miller. And we unreasonably believe that the mistakes of the past must be consequences of ignorance or incompetence and could have been remedied by further study and improved insight. That's why, for example, in spring 2007, when the stock of Merrill Lynch was trading around $95 a share, its CEO E. Stanley O'Neal could be celebrated as the risk-taking genius responsible, and in the fall of 2007, after the credit market collapsed, derided as the risk-taking cowboy responsible—and promptly fired. We afford automatic respect to superstar business moguls, politicians, and actors and to anyone flying around in a private jet, as if their accomplishments must reflect unique qualities not shared by those forced to eat commercial-airline food. And we place too much confidence in the overly precise predictions of people—political pundits, financial experts, business consultants—who claim a track record demonstrating expertise. [pp. 199—200[

Having thoroughly discussed the mathematical concepts of his subject, the author also takes a very useful look at human expectations and how we like to see patterns where there are none. Self-awareness has a significant role to play in understanding—or, often, misunderstanding—randomness in our world.

If it is easy to fall victim to expectations, it is also easy to exploit them. That is why struggling people in Hollywood work hard to look as though they are not struggling, why doctors wear white coats and place all manner of certificates and degrees on their office walls, why used-car salesmen would rather repair blemishes on the outside of a car than sink money into engine work, and why teachers will, on average, give a higher grade to a homework assignment turned in by an "excellent" student than to identical work turned in by a "weak" one. Marketers also know this and design ad campaigns to create and then exploit our expectations. one arena in which that was done very effectively is the vodka market. vodka is a neutral spirit, distilled, according to the U.S. government definition, "as to be without distinctive character, aroma, taste or color." Most American vodkas originate, therefore, not with passionate, flannel-shirted men like those who create wines, but with corporate giants like the agrochemical supplier Archer Daniels Midland. And the job of the vodka distiller is not to nurture an aging process that imparts finely nuanced flavor but to take the 190-proof industrial swill such suppliers provide, add water, and subtract as much of the taste as possible. Through massive image-building campaigns, however, vodka produces have managed to create very strong expectations of difference. As a result, people believe that this liquor, which by its very definition is without a distinctive character, actually varies greatly from brand to brand. Moreover, they are willing to pay large amounts of money based on those difference. Lest I be dismissed as a tasteless boor, I wish to point out that there is a way to test my ravings. You could live up a series of vodkas and a series of vodka sophisticates and perform a blind tasting. As it happens, The New York Times did just that. And without their labels, fancy vodkas like Grey Goose, and Ketel One didn't fare so well. Compared with conventional wisdom, in fact, the results appeared random. Moreover, of the twenty-one vodkas tasted, it was the cheap bar brand, Smirnoff, that came out at the top of the list. Our assessment of the world would be quite different if all our judgments could be insulated from expectation and based only on relevant data. [pp. 214—215]

My only complaints are few and really not significant. There were a number of terms familiar to practitioners of statistics that the author avoided. I'm sure he did so to tone down the jargon; it sounded peculiar at times to my ear but most readers won't be bothered by it. I also thought an equation or two might serve some purpose, but publishers disagree and the author avoided equations; again, most readers probably won't be upset by the omissions.

I thought the book a very successful, complete, and coherent introduction to the ideas of randomness (i.e., statistics and probability) and how it pops up all over the place. Reading it could provide painless yet vital self-defense skills for modern citizens.

-- Notes by JNS