Steve Kass

Need to run a one-sample t-test or z-test? Here’s a little calculator written in Excel to help you out.

Need to calculate z-scores, percentiles, or scores based on a normal distribution? Here’s little calculator for that, too.

Kevin Underhill* emailed me today about the Schwarzenegger veto letter. Specifically, Kevin wondered whether “it might be possible to calculate the odds against this arrangement of letters being entirely random, as the Governor’s office has claimed.”

Kevin wasn’t the only one to wonder about the odds and contact a math professor.

Several hours after Kevin wrote me, SFweekly.com published “Odds Schwarzenegger’s ‘I F–k You’ Message Was Coincidental? About One in Two Billion, Says Math Prof.” SFweekly.com quoted Stephen Devlin, a mathematician at University of San Francisco, and Gregory McColm, a mathematician at the University of South Florida, and printed various tiny chances, including a) 1 in 10 million, b) 1 in 100 million, and c) 1 in 2 billion. These are (using rough estimates of initial-letter frequencies in English words) the chances a) that seven randomly selected lines of English text begin with f, u, c, k, y, o, and u, in that order, b) that eight randomly selected lines of English text begin with i, f, u, c, k, y, o, and u, in that order, and c) that the first letters of eight randomly selected lines of English text and two blank lines separating those lines into three groups appear in the sequence i, blank, f, u, c, k, blank, y, o, u. The chances of c) are 1 in 2.1 billion, by my calculation, but no matter—only the order of magnitude is of interest here.

Whether many visitors to SFweekly.com (well, male ones of any persuasion) read the piece is questionable. It appeared next to two photo links with more click appeal than “Blah Blah F–k Blah Blah Math Prof.”: Exotic Erotic Ball (image of exotically costumed ladies kissing) and San Franciso’s Hottest Chefs (image of boyish chef wearing five o’clock shadow and a uniform, smiling seductively).

Similar calculations appear with less distraction elsewhere.

For the record, even before making any calculations, I was convinced beyond a reasonable doubt that the governor meant full well to say “fuck you.” In a later blog post, however, I’ll try to explain that the unlikelihood of “fuck you” or “i fuck you” appearing in a letter isn’t in and of itself a smoking gun.

*I’ve e-known Kevin for a couple of months now, ever since I introduced myself with an unceremonious, if politely worded email to the effect of “your arithmetic is wrong.” More details here. It turns out to be a better way to meet people that you might think.

Kevin finds fortune at Shook, Hardy, and Bacon (shb.com, Alexa US traffic rank 292,302, with 132 sites linking in), and he finds fame at Lowering the Bar (www.loweringthebar.net, Alexa US traffic rank 59,588, with 145 sites linking in).

A colleague recently drew my attention to this CNN.com article, titled “Confident students do worse in math; bad news for U.S.”

It’s interesting, I guess, but the analysis is flawed.

The article reports the results of a Brookings Institute study based on the 2003 Trends in International Science and Mathematics Study (TIMSS).

It’s the culture, stupid.

Where to start? First off, the countries with the best average math scores were East Asian countries. That confirms other studies and general perceptions, and it didn’t surprise anyone. Obviously, then, anything that correlates with East Asianness (straightness of hair, size of epicanthal folds, or facility in Chinese or Japanese) will also correlate with math scores when East Asian country-wide averages are compared with other country averages. One trait that correlates with East Asianness is self-effacement. Self-effacement tends to be highly valued in countries like Taiwan and Japan, but praising one’s self is not, and there was a survey question that measured self-effacement: “How strongly do you agree with the statement ‘I usually do well in mathematics.’”

It shouldn’t have surprised anyone that students in East Asian countries were less likely to answer “strongly agree” when asked how much they agree with the statement “I usually do well in mathematics,” so it shouldn’t have surprised anyone that country-wide averages on this survey question correlated (inversely) with mathematical ability. The correlation is simply explained by culture and the known difference between countries in mathematical ability.

Hurting the chances for a fair airing in the press (if there is ever any chance), the Brookings analysts repeatedly confuse expressed self-confidence with true self-confidence, too: “In the TIMSS data, when one looks at the math scores of students within each country, those who express confidence in their own math abilities do indeed score higher than those lacking in confidence,” and “The world’s most confident eighth graders are found…”, and “students in [East Asian] countries do not believe that they do very well in math.” Combine this with the fact that while the Brookings folks do notice the culture question, they dance around it enough to let reporters to come away with conclusions that are a good six fallacious leaps away from the data and statistics, like “Happy, confident students do worse in math” headlining an article by Association Press education writer Ben Feller. Nothing in the data suggests that confidence in mathematical ability is inversely related to actual mathematical ability at the level of individuals, but the headline gives that impression, strongly.

Another problem with the study is that it commits the ecological fallacy. It speculates about how confidence and ability are related in individual students from country-wide aggregate data, and the press drag these wrong conclusions in the wrong direction. The aggregate data is screwy to begin with, given the very strange list of countries surveyed. Many have small populations: Jordan, Israel, Bahrain, Cyprus, Latvia, the Palestinian Authority, Moldova, Lithuania, Scotland, Norway, Flemish Belgium, Botswana, Macedonia, and Serbia, and a few have large populations: Russian Federation, Japan, Indonesia, Malaysia, the United States, and the Philippines, but each country’s average mathematics score counted the same in the statistical analysis to which the press is paying the most attention. The country list was hardly comprehensive, either. Chile was the only American country on the list besides the U.S.

A regression weighted by population would have been a little better, but drawing any conclusions about individual students from country-wide averages is invalid. Here are a couple of good articles about this fallacy: (link) (link).

The TIMSS data contains plenty of useful information to support a conclusion opposite to the one reported. The data showed that all other things being equal (that is, among students within any one individual country) higher student confidence (which, ceteris paribus, should now correlate with expressed confidence) in math ability tended to be associated with higher math ability. The Brookings folks observed this, and they called it paradoxical: “So an interesting paradox emerges from the international data on student confidence and achievement. The relationships are the opposite depending on whether within-nation or between-nation data are examined.”

When Brookings writes “The international evidence makes at least a prima facie case that self-confidence, liking the subject, and relevance are not essential for mastering mathematics at high levels,” it’s easy to think they are suggesting that the ecological fallacy analysis is telling us something, and they are diminishing the different conclusions supported by the ceteris paribus analysis. And the reporters take the hook. I wonder how good the press corps and the Brookings researchers think they are at statistics.