Tuesday, 7 December 2010

Statistics - not always black and white

I was a little startled by the front page of the Guardian this morning. It featured an article claiming that David Lammy, MP for Tottenham, had uncovered shocking evidence of racism in the admissions procedures of Oxford and Cambridge - Britain's two most prestigious universities.

Some of the figures are certainly cause to raise an eyebrow - just one black Briton of Caribbean descent accepted by Oxford last year? One college hasn't admitted a black student in five years? Surely this is evidence of institutionalised racism at its worst! Or is it? That one black Briton of Caribbean descent was of just 35 applicants, and a spokeswoman for Oxford points out that "black students apply disproportionately for the most oversubscribed subjects". This is before you start thinking about how many people don't disclose their ethnicity (on all the forms you're sent when you apply), and so on.

Clearly this is somewhere where some better statistical thinking would help, but it does not seem to be forthcoming. There are plenty of points that can be dissected and discussed, but I'm just going to pick on one quote (posted on this blog) from the honourable member, which I think highlights the quality of the analysis:

"Why is it that 25 of 84 Black applicants received offers from Keble College but just 5 of 64 Black applicants received offers from Jesus College over the same 11 year period?"

On face value, this seems quite a big difference. 25 out of 84 is 30%, whilst 5 out of 64 is just 8%. Surely that's not down to chance? That is presumably what we're supposed to think, but let's dig a little deeper. The Guardian have published the admissions data for each college, and it's from these that these two figures come. A quick look reveals that Lammy has picked out the two colleges with the highest and lowest rate of admission for black applicants, and this is when alarm bells should start ringing.

The sharpshooter fallacy is a classic. An old Texan (not a great shot) fires at the broad side of a barn, and then draws a big target whose centre is where his biggest cluster of shots happened to land. He points to this as proof of his superb marksmanship. This is an essential aspect of statistics - if you don't decide what you're looking for before you collect your data, it's easy to find results that seem implausible. If Lammy had had some reason to chose Keble and Jesus before he'd looked at the data, then the difference he highlights might mean something, but as it is could it just be down to chance?

Fortunately, it's a pretty easy thing to check. Let's assume that a black applicant has the same chance of being admitted to any Oxford college. On average, 22% of black applicants over the last 11 years were admitted, so I'll use this as my baseline. I'm going to simulate new versions of the real dataset, where I take the (real) total number of black applicants to each college, and then see how many get accepted by giving everyone a 22% chance. If we do this, then on average the college with the highest success rate admits 35% of its black applicants, whilst the average lowest success rate is 11% - a spread of 24%. David Lammy highlights a difference of 22% between best and worst as indicative of... something, when in fact it's pretty much what you'd expect.

There are doubtless plenty of valid issues - some of which Lammy does try to raise - that these data could highlight, had they been analysed properly and not obfuscated by a cloud of sensationalism. Lammy says that "the variations between colleges in their admissions statistics is a pertinent point and Oxbridge should be doing more to find out why such variations exist". Perhaps if he'd employed a statistician he would be able to answer this one for himself.

Saturday, 30 October 2010

The Sound of Silence

It's all been a bit quiet here lately. Too quiet, if you ask me. Alas, my output has been somewhat reduced by an exciting new website from the Royal Statistical Society, namely the online home of their quarterly magazine, Significance. (Do you see what they did there?)

So yes, what would have been content for Statscream has turned into content for Significance, and so whilst I have various things in the works that I'm pretty sure will only be suitable for this corner of the Internet, there will be numerous posts where I merely link to my pieces elsewhere.

My opening gambits, therefore, are:

Anyone Fore Golf - a look at whether there is evidence of home advantage in the Ryder Cup; and
Putting the Man in Man Booker - investigating the gender gap in literary prizes.

I have even become dangerously web 2.0 and set up a Twitter account - statacake - thus increasing my statistical Internetular outlets to 3. The hypothesis "do more means of communication result in less content" will be tested in due course...

Saturday, 18 September 2010

Rank Outsiders

The latest round of international University rankings recently came by, so I thought I'd whip up a quick graph to look at how countries performed compared with their GDP. As usual, I didn't fancy doing anything complicated, so I just totalled up the scores for every university from each country, and plotted them against their GDP according to the IMF in 2009.
So there you go. Fairly scattered around a (least squares) line of best fit, with an R-squared of 0.7, but the UK is by far the most outlying country, ostensibly punching well above its weight. Other things to compare university rankings with are available.

Sunday, 12 September 2010

Notorious Voting Scenes

One of the unceasing complaints about the Eurovision Song Contest is that ever since the introduction of televoting the contest has been dominated by 'political' (particularly diaspora) voting. In an attempt to combat this, the organisers recently introduced a part jury, part televote system, where the points awarded by a jury of 'experts' would be given equal weight to those from the televote. In theory the juries will be much more objective, and less easily swayed by 'extramusical' matters.

After the event the scores awarded by the juries are revealed, so it's possible to compare the juries' scores with the televote scores, theoretically giving us an indication of how much the voting public are affected by, say, a country's foreign policy, or who's controlling their gas supply. To make it easy to visualise, I've made a map showing all the countries who competed in the 2010 contest, and colour-coded it according to how much the televote rankings differed from the jury vote. Darker red means a country did much better in the televote than the jury vote, darker blue means the opposite. (These data are taken from the semi-finals as well as the final, taking the final results if a country appeared in both.)

The common theories are that Eastern European countries dominate the contest because "they all vote for each other" and that western nations are unfairly disadvantaged. So what does our map look like? (Click to make it bigger.)


My initial reaction to this is that there's no particularly obvious pattern - both east and west seem to 'benefit' from the televote with Turkey, Serbia and Moldova in the east as red as France and Spain in the west. Russia - often perceived as the biggest beneficiary of political voting - is a lighter red, finishing 4 places higher in the telephone than the jury vote, whilst the Netherlands are almost in the same boat finishing 3 places higher. Eastern Europe is a bit of a mix, whereas it's the Scandinavian countries - Iceland, Norway, Sweden, Finland and Denmark - who seem to merit their 'usual suspects' status. Of these, only Norway are a shade of blue, and as the hosts this might be expected anyway (a common theory is that because a country hosts as a consequence of winning the previous year's competition, voters are less willing to vote for them to win again so soon).

So maybe the public aren't as politically minded after all, or maybe the juries are less objective than the organisers would hope. Alternatively, with so many countries densely packed into Eastern Europe it's inevitable that some will benefit and some will lose out from votes being shared around. These are only the results from one year, so it's not a great way to try and separate patterns from anomalies. For instance, this year Spain got to perform twice in the final after a stage invasion, which seems a fairly reasonable explanation for their 'overperformance' in the televote. A few more years of split voting should hopefully clarify the picture.

A final observation, which is rather cute (and perhaps a little reassuring) is that both Germany and the UK have a neutral colour in this map - they finished in the same position in the televote as the jury vote. For those who don't follow these things, Germany won the contest whilst the UK finished dead last. You can say what you like about dodgy voting, but they get the winner and loser spot on.

Monday, 2 August 2010

A thorough investigation into popular opinion of statistics

(In the interests of full disclosure, I removed a bunch of results for things such as "9 out of 10 statistics are made up" which generated artificial hits for lower numbers (10 in this example).)

I think the graph speaks for itself: most people think most statistics are made up. A couple of bonus (not entirely made up) stats are that the modal hit was "87% of statistics are made up", and the mean hit was "74.8% of statistics are made up". So next time someone tells you an alarming statistic don't worry, three-quarters of the time it will have just been made up. Possibly.

Saturday, 10 July 2010

Mega Football Versus Small Octopus

On my previous post about Paul the octopus, a commenter asked a couple of questions which I thought merited a separate post to address. The first:

"There is of course a detail that you've (likely intentionally) overlooked, which is that the chance of winning a football match is not usually exactly 50-50. Given that Germany are one of the world's top teams they could be expected to win more matches than they lose.

Having done a bit of research, I've discovered that since Paul started making his predictions (at least for the public) at the start of Euro 2008, Germany have won 22 games, lost only seven and drawn four. Their win record thus stands at 66.7% over the last two years, which is probably a fairer representation of their chances of victory in a randomly determined match.

Can your analysis take account of this?"


This is an interesting question, and it boils down (as happens surprisingly often with probability) to a matter of perspective.

Suppose you have a friend called Peter who knows a bit about football. He's successfully predicted the results of the same six games that Paul has. Since Peter knows about football, he knows that the chance of Germany beating Australia (for example) was probably not exactly 50%. Does this matter?

Well, not really. The analysis we carried out last time was testing a specific hypothesis - that Paul was picking teams at random. This was our 'null' hypothesis, our default state of belief, if you will. Our 'alternative' hypothesis was that he has done better than you'd expect him to by chance alone. In testing this I claimed that Paul's chance of predicting the winner - if he's just picking at random - is 50/50. Crucially, this doesn't depend on the real chances of either outcome. This might not seem intuitive at first, but imagine Paul was picking the team after the game had happened - at this point the winner is known, so if he's picking at random he has a 50% chance of picking the right team. Since Paul's picking doesn't (we presume) interfere with the outcome of the game, if we're assuming he's picking 'blind' then it doesn't matter whether he chooses before or after the result is determined.

So what about Peter? We could test the same hypothesis and we would come to the same conclusion. The only difference is that we're not (as) impressed because we would expect him to be doing better than chance anyway - he has extra information to help make his decisions. Paul, meanwhile, is just an octopus, and so no-one would expect him to know anything (except possibly how to count to eight).

On a separate note, and with regards to the probabilities we've calculated telling us that Paul has some apparently incredible ability, it's worth stressing that that isn't what we've shown either. All we've done is shown that if Paul was picking at random (as - call me a sceptic - he probably was) he's just got quite lucky. This in itself isn't really that remarkable though - Paul was only brought to our attention after a string of successful predictions. There may well have been hundreds of other octopuses/coins/babies making similar predictions and getting them wrong, and we've just got to see the one who got them right. If you see a golfer hit a hole in one it seems remarkably improbable, but if you think about all the millions of shots that didn't go in, that single event occurring doesn't seem so incredible.

But anyway, on to the second question:

"Secondly, I would note that there are three possible results in most football matches (win, loss, draw) rather than two, although there seems to be no way for Paul to predict anything other than a win or loss. So far none of the matches he has made predictions for have resulted in a draw, but the possibility exists nonetheless. How does that affect the overall dataset?"

This is a good point (and one which I ignored previously for the sake of keeping things simple), and an interesting one to discuss.

As we've discussed, if our (null) hypothesis remains that Paul is picking at random, his probability of picking either team is just 0.5. However, since it's not certain that one of those teams will go on to win the game, his chance of picking the winner is actually going to be less than that. For instance, if 2 in 3 games end in one of the two teams winning, Paul then has a 1 in 3 chance of picking the team that wins, a 1 in 3 chance of picking the team that loses, and a 1 in 3 chance of there being no winning team to be picked at all.

What this amounts to is that Paul's chances of correct predictions are in fact even lower than those we'd already calculated, but unless you are willing to believe an octopus has been keeping an eye on the football pages of Bild, the chances are he's just very lucky.