Steve Kass

In July, I griped here about misleading news reports of a higher crash risk among texting truckers than among non-texting ones. I pointed out that the research quoted had not shown an increased crash risk, and in fact observed fewer crashes (zero, in fact) among the texting truckers. The data might suggest a decreased crash risk among texting truckers, I noted. The reason for the confusion was that crashes and near-crashes (which included sudden and possibly crash-preventing maneuvers) hadn’t been separated in the calculations. Nevertheless, the increased crash-or-near-crash risk was widely reported as an increased crash risk. I wondered whether more near crashes might in fact be a positive thing; those who occasionally swerve suddenly might be paying more attention to the road than those who rarely do.

Today, researchers and others are expressing surprise that just-in real crash data doesn’t support what they don’t realize the earlier research didn’t show in the first place. For example:

• Results of Study on Cellphone Use Surprise Researchers
• Cell phone bans do little to reduce crashes, study finds

If researchers (or journalists) are surprised by today’s news, they probably didn’t examine the research very closely. They may have believed the misleading headlines instead.

Disclaimers: I haven’t read all the research, and some studies might have in fact shown an increased crash risk, unlike the one I mentioned. That might be reason for real surprise. In addition, today’s data doesn’t specifically show that less phone use means no fewer accidents, because laws don’t always change behavior. But some of the reports I read did suggest that there was less phone use, yet no lower crash rate, in places that instituted bans.

Ironically, the crash data out today could make things worse. If drivers think phone use while driving is not as unsafe as previously thought, they might be less careful when using a phone while driving, and phone-related crashes might increase. Common sense suggests that multitasking requires greater concentration. If you do use a phone while driving, drive with even more care than usual. In other words, this might be one situation where being wary might have benefits, even when its not warranted by the facts – especially because being somewhat over-cautious while driving has no serious down side, it seems to me. (It’s not like it infringes upon hundreds of millions of people’s civil rights, like acting on other unwarranted fears can and does…)

A headline in today’s Washington Post is “Fewer Americans think Obama has advanced race relations, poll shows.” My statistics students know what I have to say about that: “fewer than what?” Yes, headlines are generally false, and the Post reports the statistics somewhat more carefully in the article. “On the eve of President Obama’s inauguration a year ago, nearly six in 10 Americans said his presidency would advance cross-racial ties. Now, about four in 10 say it has done so.”

According to the actual poll results, a year ago, 60 percent of Americans answered “help” to the question “Do you think Obama’s presidency will do more to (help) or more to (hurt) race relations in this country, or not make much of a difference?” Last week, 40 percent of Americans answered “help” TO A DIFFERENT QUESTION: “Do you think Obama’s presidency has done more to (help) or more to (hurt) race relations in this country, or has it not made much of a difference?”

Big deal. Not a story. Fewer Americans today also think they have died than thought last year that they would die.

Original title: Over 90% of Research Studies Make Me Want to Scream (P < 3E-12).

Shania Twain is in the news today. No, her new album still isn’t out, but her face is in the spotlight. It turns out someone “applied” the latest “research” to “determine” that she has the perfect face, “scientifically” speaking. The distance between her eyes and mouth are precisely 36% of the length of her face, and her interocular distance is exactly 46% of its width. These proportions, according to an article in press at Vision Research, are universally optimal (among low-resolution, mostly Photoshopped images of a few white women).

Garbage. Poppycock. Nonsense. Balderdash. Crap, crap, crap of a research paper, right from sentence 1: “Humans prefer attractive faces over unattractive ones.”

But you came here for the pictures. (more…)

If you read my last post, you know I’ve been looking at some very fishy survey data from Strategic Visions, LLC. The data seems to stink no matter how anyone looks at it, and mathematicians, statisticians, and programmers have been looking hard and every which way. Instead of throwing yet another heavy mathematical brick at the poor numbers, Let’s see how it stands up to a feather.

You don’t have to read many poll or survey results to be familiar with the phrase “totals may not equal 100% because of rounding.”

So guess what? The numbers in Strategic Visions’ results all add to exactly 100. Oops. That’s a huge red flag. Huge, like big enough to wrap a planet in.

Ok, I didn’t check all their polls. Just the most recent 73, which is how many I checked before stopping. For the record, I didn’t stop because poll #74 added to 99 or 101. That would have been statistical misconduct on my part. I didn’t look at poll #74 or any others, because I wanted to write this up. (If anyone checks further, let me know and I’ll post an update.)

If Strategic Visions were not lying (or being extremely sloppy in a systematic way, which is their only hope of explaining this—see below), the chance the 73 most recent polls would add to 100 is less than 1 in 2,000. (That assumes intentional rounding to minimize 99s and 101s. For any pre-determined rounding rule, however, we’re talking 1 in 10,000,000 or worse. Maybe a real pollster can fill me in on what’s industry practice among those who aren’t lying.)

One-in-two-thousand-ish stuff happens all the time, but believe me, I didn’t find this on a data dredging excursion. I noticed that the first few poll results I saw added to exactly 100, I formulated a plausible hypothesis based on all the evidence at hand, I carried out an experiment, and I calculated the p-value.* Small enough to be incriminating in my home court, and I tend to be a benefit-of-the-doubt guy.

Just two more things. First, the possibility of systematic sloppiness:

The consistent adding-to-100 could be the result of a systematic error, as opposed to cheating. Of the possible excuses, this is the one I suggest SV choose if they decide not to come clean. Logically it can’t be distinguished from lying, and they can attribute it to a whipping boy like the web site designer. (This excuse doesn’t defend against the mountain of mathematical bricks I mentioned earlier, however.) They can say that they made a regrettable decision that to avoid the appearance of error, they calculated one percentage in each survey from the other percentages, not from the survey results. I won’t believe it for a minute (though if they show me their programs or confirm that some commercial product makes this error, I’ll reconsider), but it might get them out of hot water.

And second and last, an example and a bit of the math behind my calculations:

Most survey results are short lists of whole number percentages that express fractions to the nearest whole percentage point. Suppose 600 likely voters were polled in a tight race between Tintin and Dora the Explorer. Tintin had the support of 287 people, Dora was close behind with 286, and the rest of the 600 people surveyed (that would be 27 of them) said they weren’t sure. To the nearest whole percentages, that’s 48% for Tintin, 48% for Dora, and 5% undecided. The sum of the rounded percentages is 101%, and that’s due to honest mathematics, not fraud.

Let me skip some really fun mathematics and tell you that for survey questions that have three answers, the percentages add to 100 most, but by no means all of the time. Exactly how often they don’t add to 100 depends on several factors, two of which matter much at all in this case: the number of people surveyed, and how numbers ending in .5 are rounded to whole numbers. Strategic Visions, LLC’s usual survey sample size is 800, and even if ending-in-.5 numbers are rounded differently in each poll to avoid a sum of 99 or 101 when possible, three-choice result percentages should still add to 99 or 101 at least one time out of ten.

* Not to be taken as evidence of a non-Bayesian persuasion on my part. The frequentist approach seemed to me pretty straightforward and justifiable here, that’s all.

Today’s clicking (especially from fivethirtyeight.com) led me to two strikingly similar declamatory reports about high school student’s knowledge of civics, complete with chart-laden survey results.

“Arizona schools are failing at [a] core academic mission,” concludes this Goldwater Institute policy brief.

“Oklahoma schools are failing at a core academic mission,” announces this Oklahoma Council of Public Affairs article.

When asked to name the first president of the United States, only 26.5% of the Arizona high school students surveyed answered correctly. Only 49.6% could correctly name the two major political parties in the United States. An even smaller percentage of Oklahoma high school students gave correct answers to these and other questions from the U.S. citizenship test study guide. None of the thousands of students surveyed in either state answered all ten questions correctly.

The shocking thing is that these are garbage studies. Made-up numbers, probably. The acme of vulpigeration. Evil. Makes me sick. (Glad I coined the word, though.)

No way these are real studies. Danny Tarlow over at This Number Crunching Life has taken a mathematical hammer to the Oklahoma “study” quite effectively. (The blatant similarity of the Arizona “study” blows away any shred of possibility that the Oklahoma study is legit. I’d love to see Danny’s face when he sees the Arizona report.)

What’s frightening is that this kind of snake oil has far too good a chance of surviving as fact (which it isn’t) and influencing public policy.

The guilty parties? The Goldwater Institute, which as you might guess is a conservative “think” tank. The OCPA, which describes itself as “the flagship of the conservative movement in Oklahoma.” Matthew Ladner, the author of both reports, who is vice president of research for the Goldwater Institute. And last but not least, Strategic Vision, LLC, which Ladner says “conducted” the studies. In my opinion, the word is concocted. Read about them yourself.

[Updated with correct business name: Strategic Vision, LLC.]

Don’t get me wrong. The GM Volt is probably a good thing. But headlines like G.M.: Chevy Volt Gets 230 M.P.G. are misleading, vulpigerating. Using the same logic, the vehicle that takes me to work and back most days (photo at left, used with permission from http://www.flickr.com/photos/rudiriet/ / CC BY-SA 2.0) gets infinitely-many M.P.G.

What’s doesn’t fit in the headline is the fact that—unlike a Toyota Prius, say—the Volt needs to be topped up with electricity as well as gas. You have to plug it in and charge it. With a cord. And pay for the electricity. Like your computer and cell phone, except the battery is way bigger. Way bigger. According to this Wall Street Journal article, GM says the car will use 25 kWh of electricity for every 100 miles driven. That’s on top of the just-over-a-penny per mile you’ll pay (at today’s prices) for gas. Where I live, electricity costs about $0.17 per kWh and fuel costs about $2.55 per gallon, and here’s a more useful analysis, still based on a slew of questionable assumptions. I didn’t break it down, but just so you know, the electricity a Volt needs will cost you about 4 times as much as the gas.

Also not clear is where the number 230 comes from, except that it appeared on a green banner (assuming the smiley-faced electric plug is supposed to be a zero) behind Fritz Henderson in a photo in the New York Times. Google returned no hits for any of these phrases at gm.com: “230 MPG”, “230 M.P.G.”, “230 miles per gallon.”

Notes:

• A couple of years ago, my 1989 Honda Civic, which I’d had for 8 years, gave up the ghost. It got about 44 mpg on the highway and 32 mpg in the city.  The car I had for 5 years before it was a 1981 VW Rabbit (Diesel), which got about 45 mpg on the highway and no less in the city. My 2007 Honda Fit only gets about 32 mpg on the highway, largely because of an extra 1000 pounds (over the Civic and Rabbit) needed to make it safer should an SUV or Hummer crash into me. It also pollutes less than either the Civic or Rabbit, which I like.
• I haven’t called State Farm to ask them whether I would have to pay more for my personal liability insurance if I stretched an extension cord out the door, down the sidewalk in front of 12 neighbors’ doorways, and out into the parking lot.
• GM is reported to have said you can buy a kWh of electricity off-peak for a nickel (that’s $0.05 for readers outside the U.S.). I don’t know where they sell that, but I’m sure I can’t afford the extension cord. There’s no off-peak potential savings here.

What the Daily News says.

A few minutes ago, I tried posting a comment in response to the article “OMG! Driving while texting might soon be illegal” on the Christian Science Monitor web site. Just like this I did:

I wasn’t expecting the pithy retort:

Headlines today, all citing the same study by the Virginia Tech Transportation Institute:

• Texting While Driving Increases Crash Risk by 20X
• Texting increases crash risk 23 times: study
• Texting Drivers 23 Times More Likely to Wreck

From the press release for the study (emphasis mine):

In VTTI’s studies that included light vehicle drivers and truck drivers, manual manipulation of phones such as dialing and texting of the cell phone lead to a substantial increase in the risk of being involved in a safety?critical event (e.g., crash or near crash). However, talking or listening increased risk much less for light vehicles and not at all for trucks. Text messaging on a cell phone was associated with the highest risk of all cell phone related tasks.

From a New York Times article containing further detail about the VTTI study (emphasis again mine):

In the two studies, there were 21 crashes and 197 near crashes — defined as an imminent collision narrowly avoided — from a variety of causes, including texting. There were also about 3,000 other near crashes that were somewhat easier for the truckers to avoid, the researchers said. There also were 1,200 unintended lane deviations.To determine how the events compared to safer conditions, the researchers compared what occurred in the dangerous situations to 20,000 segments of videotape chosen at random.

In the case of texting, the software detected 31 near crashes in which the cameras confirmed the trucker was texting, though no actual collisions occurred. In the random moments of videotape, there were six instances of truckers’ texting that did not result in the software detecting a dangerous situation.

I couldn’t find the study data on the VTTI web site, but I found some PowerPoint presentations and other documents about the study. None of them quoted a “crash risk.” The increased risks were for “crash/near crash” incidents. The numbers quoted by the Times make it pretty clear that “crash/near crash” included crashes, near crashes, “other near crashes” and unintended lane deviations, because that’s the only way the odds ratio comes out to 23:

Do I think texting while driving is a bad idea? Yes. Do I think you’re 23 times more likely to have an accident while texting than while not texting? I don’t know. The VTTI study suggests not, and it might suggest the opposite: while texting, you’re less likely to have an accident (but more likely to brake, swerve, or change lanes unintentionally). I can’t say whether the study suggests either of these things with statistical significance, because I don’t have the data, but it does seem to be the case that the study subjects, when texting, were more likely to swerve or slam on their brakes, but not crash.

I can think of some possible reasons for this, too: if you’re texting, you’re probably not sleeping (sleeping being a significant risk factor for actual crashes, but probably not for swerving and braking), and if you’re texting, you might know you’re creating a dangerous situation and be more prepared to swerve or brake suddenly.

A headline on blog.aflcio.org says “CEOs Get One-Third of All Pay.” Not surprisingly, it’s flat-out false. The group of Americans “getting” all this money isn’t CEOs. It’s everyone earning more than the Social Security wage base, currently $106,800, according to the AFL-CIO blog’s second-hand source, a Wall Street Journal article. (Google’s cache has the full article, which the WSJ site won’t show you unless you register.)

Credit to the WSJ for drawing attention to data that could support an argument for payroll taxes on higher wages. (I don’t know what the WSJ’s editorial stance on taxes is; I just know they tend to be on the other side of most issues from me, but they usually do their homework more thoroughly than most media outlets.)

The data suggest that the payroll tax ceiling hasn’t kept up with the growth in executive pay. As executive pay has increased, the percentage of wages subject to payroll taxes has shrunk, to 83% from 90% in 1982. Compensation that isn’t subject to the portion of payroll tax that funds old-age benefits now represents foregone revenue of $115 billion a year.

I don’t know why the WSJ decided that if you’re earning a lot of money, you must be an executive. Maybe it has to do with who they perceive as their readership. (Personally, I’m not this “highly paid” nor am I an executive, but that’s not relevant to the argument.)

I no idea what fraction of All Pay this country’s CEOs earn, but now I know it’s less than a third.

[Thanks, Lucinda, for sharing this article.]