It was Disraeli who said "There are lies, damn lies, and statistics", but he didn't know the half of it. Way out in the thickest underbrush of statistical hocus-pocus, there are academic test scores. There has been a flurry of news stories recently about how well our students are performing on various kinds of tests. Few non-specialist readers, I imagine, have the patience to hack their way through the thickets of acronyms and figures, but here is a brief primer on the most important points.

SAT/ACT. About half of those who apply to selective colleges in the United States take the SAT (Scholastic Assessment Test), favored by Eastern colleges. In the Midwest and South, the ACT (American College Testing) is more often used.

Re-calibration. Both the SAT and the ACT have been re-calibrated, the ACT in 1989, the SAT in 1995. When comparing statistics across the re-calibration points, responsible journalists (like Richard Rothstein in the New York Times, 8/30/00) are careful to use adjusted figures that can be legitimately compared. However, nobody believes that the re-calibration was without consequences. Statistically speaking, it at least had the effect of blurring the high-end results — that is, of making it more difficult to separate out the superb performers from those who are merely excellent.

Pitfalls in interpreting the statistics. There are many. In comparing results across years, for example, you must make allowance for the fact that the population of test-takers for SAT/ACT has been different at different times. More students of all levels of ability take SAT/ACT now than ten years ago; and the proportion of black and Hispanic students taking them is higher. There is also the public school / private school gap: what proportion of test-takers are from each? It changes with time. Another problem is "cramming"; that is, the use of SAT/ACT-preparation courses, whose popularity and effectiveness both seem to be increasing rapidly. There is endless rancorous debate among educationalists and psychometricians about the weight that should be accorded to these factors.

How are we doing? The overall impression from recent results, with due attention given to the factors above, is that verbal scores have been constant for some years, while math scores have improved slightly.

The gender gap. Much jiggery-pokery here. In a report released August 24th, the NCES claimed that "the gap between the average mathematics scores of male and female 17-year-olds on national tests has disappeared". Now, any statistician will tell you that in matters like this, the average is only half the story. Test scores are spread among the tested population in a certain pattern called a distribution, of which the average is only one measure. When the distribution is our old friend the Bell Curve, as in cases like this it usually is, there is a second measure just as important: the standard deviation. This measures the "flatness" (or conversely the "spikiness") of the curve. On most characteristics, women have a "spikier" bell curve than men, their scores more bunched around the mean, with less out at the extremes — the "tails" of the bell curve. Another way of saying this is that men's results are more spread out — men have more variation. So it is with math scores, if you look closely at the NCES figures. Out at the high end of the NCES test scores, for example ("scale scores" at or above 350) we find 10 per cent of males, but only 7 per cent of females in the 1999 figures. (Earlier figures: 1996 — 9 and 5 per cent, 1994 — 9 and 6 per cent.) In math, there are less high-performing women than men. SAT/ACT indicate the same result.

The race gap. Persistent and so far intractable. Not, as Rothstein implies in his otherwise-honest piece, a case of "disadvantaged minority students" versus "middle-class whites". The differences appear when figures are controlled for family income and family educational level. Whites out-score blacks. Hispanics are intermediate between whites and blacks. East Asians out-score whites. Ashkenazi Jews out-score everybody. Get used to it.