Killing Pregnant Women is un-Westerosian

A recent statement from a spokesperson for House Lannister:

House Lannister has heard of the recent events at The Twins, and are shocked by the brutality rumoured to have occurred there. Suggestions that the Lannister family had any connection with these horrendous events are a slander on our good name, and anyone repeating them is obviously subscribing to the craziest and most preposterous of conspiracy theories. What is said to have happened at The Twins is abhorrent and inconsistent with the moral principles by which all Westerosians live, and there is no way that the Lannister family would have any connection with such cruel and un-Westerosian behaviour.

At this point, an enterprising journalist should ask, “if this behavior is ‘un-Westerosian’, how come we have a song called The Rains of Castermere?” Or, “If this brutality is so inconsistent with Westerosian ethics, what do you say to claims that your own son was forced to kill the mad king to prevent him burning down all of King’s Landing?” But they never do ask, do they? They take these glib denials as fact, when in fact the very essence of the claim that the behavior being decried is “un-Westerosian” is completely false.

The above statement follows the lines of standard dismissals of bad behavior that are issued by politicians all the time, and a common element of those dismissals is a call to a common ideal of the ethics of the country in question that is, in general, a historical fallacy. Opposition figures also use the same call to fantasies of goodness when they decry acts that they were either complicit in during the previous government, or will continue when in government. For example, after today’s revelations that the NSA is spying on our communications[1],  one of the authors of the Patriot Act, Jim Sensenbrenner, said:

I do not believe the broadly drafted Fisa order is consistent with the requirements of the Patriot Act. Seizing phone records of millions of innocent people is excessive and un-American.

Really? UnAmerican? What about Cointelpro? What about the House un-American Affairs Committee? What about the Patriot Act that you wrote? It seems to me that the USA has a long history of this kind of thing, and it’s not “un-American” at all to get up to this sort of mischief. The scope might be greater but so what? It’s a matter of degrees, and it doesn’t seem unlikely to me that if he could have, McCarthy would have spied on emails.

The same thing applies with the recent expression of regret (and pathetic compensation) to Mau Mau rebels by the British government, which refuses to directly apologize because it might set a legal precedent, but which also somehow manages to claim that

It is an enduring feature of our democracy that we are willing to learn from our history.

Really? The Mau Mau tortures occurred in the 1950s, but the UK was still operating secret torture camps in Northern Ireland in the 1980s. Is that how the British government “learn from our history”? Is burying the documents in a secret location (“if we are going to sin, we should sin quietly”) and fighting compensation in the courts how the government “learn from our history”? I don’t think so. Meanwhile the Guardian refer in an editorial to Britain’s “battered moral authority.” Really? You think your country has moral authority? Does that perhaps derive from your willingness to send people you don’t like to Libya to be tortured by Ghadaffi? Your cooperation with secret rendition and torture programs run by the USA? Or your long history of torturing and murdering political opponents in the colonies? Where, exactly, does Britain’s “moral authority” derive from – having sinned quietly rather than publicly?

For the same reason I am expected to believe that when the two major parties in Australia try to find ways to “stop the boats” they are doing so out of genuine concern for human welfare – not because they are the same two parties that oversaw the White Australia Policy and a long term policy of racially-based child abduction. I am supposed to believe them when they assure me that racism has no place in modern Australian politics, and really they’re just trying to solve a problem.

It seems to me that when politicians make these claims, drawing on long traditions of moral authority, they’re either:

1. setting themselves up to deny something they’re actively involved in, or
2. trying to gain false moral ground so they can use a bipartisan policy for partisan political gain

So the best thing to do, as soon as a politician calls on a long-standing “principle” of their own society as defense or source of moral outrage, is to assume they’re lying, they will do that very thing if they get a chance, and they’re already up to their neck in it. And for final proof, I present this perfect quote from Jack Straw when he denied – in parliament, I think – the initial rumours of British involvement in rendition and torture:

Unless we all start to believe in conspiracy theories and that the officials are lying, that I am lying, that behind this there is some kind of secret state which is in league with some dark forces in the United States… there simply is no truth in the claims that the United Kingdom has been involved in rendition.

All those claims about complicity with rendition were subsequently shown to be true. Perhaps Jack Straw should apply for a job at King’s Landing …

—

fn1: note to my secret NSA readers: don’t bother using PRISM to catch this subversive document, I upload them to a public site.

 

Inequality in British State Schools: A Completely Avoidable Situation?

Figure 1: Absenteeism by level of deprivation, England, 2004

The Guardian today reports that Britain’s top 50 state-funded comprehensive schools and academies have become more unequal over recent years, and are not reflective of the social composition of their surrounding areas, or of the remainder of the schools in England. Those of us from more equal societies might think this is not a big deal but the research is quite stark in showing very large differences between the schools and their surrounding communities. Of course, inequality in educational outcomes in the UK is stark and scary compared to other OECD nations, and to help digest this I’ve provided two figures. Figure 1 above shows rates of authorized (i.e. with parental request) and total absenteeism (i.e. including truancy) for small areas in the UK, by the level of poverty of the area; the further left you go, the poorer the community becomes. Figure 2 below shows GCSE achievement on the same scale. In this case, “deprivation” is measured by the Index of Multiple Deprivation, which I think is the scale for measuring poverty that is favoured by the UK Office of National Statistics.

Figure 2: GCSE Scores by level of deprivation, England, 2004

School outcomes in the UK are obviously heavily determined by wealth. The Guardian report suggests that amongst state-funded schools this effect is most obvious in the elite schools, the comprehensives and academies. This, it suggests, is due to increasing income inequality in the UK, and because of the power of house prices. Basically, middle class families in the UK are able to buy houses in the catchment areas of the best schools, ensuring their children can access those schools. This in turn has the effect of pushing up property prices in those areas, forcing out poorer people and preserving the schools for the wealthier incomers. It appears that some of these schools have a policy of guaranteeing access not just on the basis of catchment area but on distance from the school, which guarantees that people with better purchasing power can push out poorer people.

The statistics about differences between school socioeconomics and that of the surrounding communities are pretty stark. They report that

uptake of free school meals – which is most often linked to parents receiving low-income benefits – was lower than half the national average: 7.6% in the 500 leading schools compared with 16.5% in almost 3,000 state secondary schools in England.

Just putting aside the fact that this suggests 16.5% of British families are too poor to provide their children with lunch, we can see that the communities served by these schools are, on average, wealthier than the rest of the country. They are also wealthier than the communities they are embedded in. Measured in terms of whether the schools enrol equal or higher numbers of students on free school meals as are present in the local community, the report found

only 25 also exceeded their local average, and they were well outnumbered by the 106 schools that had fewer than 3% of their pupils eligible.

Most of these elite state-funded schools were somehow managing to recruit on income, even though they are ostensibly open for all. This isn’t inevitable, and some schools have shown that it is possible both to recruit above-average numbers of poorer children and to have good academic results. For example, Chesterton community sports college in Staffordshire:

Chesterton college in Newcastle-under-Lyme has 22% of its pupils on free school meals, compared with its local authority average of just 9.8%. In 2012, 72% of its pupils achieved five good grades at GCSE, well above the national and Staffordshire local authority average of 59%.

This shows that in a good school, poverty is neither a barrier to access nor to success. So what’s going on? This Guardian article is citing a report by the Sutton Trust, which recommends some interesting solutions to the problem, including the use of lotteries or banding (basically, stratified random sampling) to ensure equal access (or, at least, better access). These are interesting ideas for short term solutions, but they don’t address the basic problem: massive inequality in British society somehow ensures that even with free-to-access services (like health and education), those with the assets manage to seize the advantage. The report makes this clear through one simple stark claim: some proportion of this elitism in state-funded schools is only possible because some parents are willing and able to move houses to be in the catchment area (and to push others out of that catchment area). People are required and willing to move homes just to get these superior education services. Should a good high school education be worth that much? Why are people moving homes to secure education outcomes? And should they have to?

I think this problem is driven by two factors: 1) investment in the majority of British state-funded education is so poor that people are willing to move homes to ensure their kids don’t have to go to some schools; and 2) the middle class in Britain now see their situation as so precarious that they are willing to make major asset purchase decisions (home purchase) simply to guarantee their children continued membership of the class they grew up in.  It seems to me that neither of these things should be necessary, and that there are alternative ways to manage society that would prevent these two situations – in my opinion, in a way that benefits everyone.

Increase investment in the worst schools

Looking at the two charts above, and considering the success reported by some of these elite academies, it’s pretty obvious that there must be some terrible schools in the UK, and some schools in serious need of extra investment. This won’t work by itself, since a lot of these areas need major cultural and economic change of their own, but better schools, and better teachers in those schools, supported in their work and properly able to deal with challenging students, will make a difference to the outcomes at those schools. It won’t completely change the phenomenon of rich and middle class parents fleeing to the state-funded comprehensives, but it will reduce the incentive as parents realize that attending a completely ordinary local school won’t kill their child’s future. I’m willing to bet as well that part of the reason poor schools in poor areas do so badly is a lack of educational diversity – no high achieving children, no historic record of achievement to inspire subsequent generations of students, and no reward for teachers to encourage them. If all these teachers have to look forward to is another year full of future criminals and children whose parents make no effort, then they will soon give up. And parents with any desire for their children to achieve will see that and move on. I’m also suspicious that the worst schools in Britain aren’t just educationally tatty: their facilities are, I’m willing to bet, also terrible, and the entire community lack pride in them. That can be fixed.

Increase attention on negative outcomes

Figure 1 shows rates of absenteeism in the poorest schools, but unauthorized absenteeism is something that police and social services can intervene in. Why don’t they? Because they’re dealing with other pressing problems. I think a lot of people in politics in the UK don’t realize just how pressing those problems are, or how much they degrade poor communities and depress the people living in them. Better attention on those problems, and greater efforts to ensure that the community in which children live is supportive of the learning needs of children, will in time lead to reductions in inequalities in behavior related to childhood delinquency – less absenteeism, less casual violence, less malicious fires, less vandalism. But there is no easy way to achieve this except through more funding: more funding for social services, police, teachers, council beautification programs, and activities for children. I don’t think any political party in the UK sees these things as essential state services anymore, and instead of funding these services they’re squeezing them, at the same time as they squeeze the general education budget and the welfare budget. While that happens, sensible people will take their children out of poor areas, making those areas more intense areas of community dislocation, reducing the likelihood that the existing social services will be successful in fighting the problems, and creating a vicious circle of social exclusion. I don’t see this vicious circle being stopped without concerted community effort.

Reduce the social mobility hard scrabble

Why is an education in Britain so crucial that parents will buy a new house in a new area just to ensure it? I think it’s because the middle class in the UK and US has become precarious, and a lot of people in that class are aware that their children risk falling out of it. Securing a position in that class is becoming a desperate struggle, with increasing numbers of losers who are falling out the bottom end of the class and into the increasing pool of poor and socially excluded. This is Ed Milliband’s “squeezed middle,” the middle class who in America and the UK have increasingly turned to debt and the housing “ladder” ponzi scheme to stay ahead of the Joneses. This race has to end, and there is a very simple way to end it: shift from a society focused on social mobility to one focused on social sustainability. I’ve written about this on my blog before: social mobility is a false promise of wealth and advancement, and a better alternative is to find ways to ensure that all jobs are socially sustainable. That is, find ways to ensure that even people at the “bottom” of the ladder can raise a family and live a halfway decent life, rather than having to scrabble up. In such a society education is still important, but because there is less urgency to achieve a ticket to success – because all careers are sufficient to support a happy life – education is not commoditized. Such societies exist, in Northern Europe and Japan, and to a certain extent Australia and Canada; and in these societies, people do not have to fight their neighbours to push them out of a precious school place. And if they do, the people pushed out will still grow up to a functioning life. The UK needs to move away from its competitive, inequalitarian social model towards these models.

Engage corporate power

A society built on social sustainability can only be built in two ways[3]: through a powerful system of taxes and transfers, or through a system in which corporations agree to some kind of social contract. Of course, in reality most such societies see a little of both, but I think a lot of thinkers in the anglosphere see social sustainability as only possible through the former, and I think they see it this way because they think corporations will not give up their wealth for a greater good, but need to be coerced into it. This is, I think, fundamentally defeatist. An alternative to a punitive system of taxes and transfers is a Japanese style system of shared corporate responsibility, in which companies pay their lowest staff a living wage, and don’t pay their highest staff stellar wages. Just because corporations won’t do these things of their own volition doesn’t mean they have to be forced to at gunpoint, but I think the natural assumption in the UK is that no one will give up anything without being forced to. That needs to change. In this respect I think Britain could learn a huge amount from Japan, which has a very strong social contract based around individual and corporate responsibility – something which I think a lot of British people don’t believe is possible.

I think Britain’s inequality is heading into a very bad place, and it’s not going to be an option to ignore it for much longer. It’s cruel, counter-productive and embarrassing. The huge inequalities developing in education can’t be solved just by throwing money at the poorest schools, though this is an essential minimum: changes need to be made in the way that the government tackles social disunity in poor areas, and also in the way that British society views “upward mobility,” competition and social sustainability. But with proper attention on improving schools in the short term, and a shift in social and economic priorities in the long term, Britain can reverse its inexorable slide into a failed state. Can they do it? I’m not hopeful, but I think it can be done.

—

fn3: that I can see. I think a third option is colonialism and theft of other nations resources, but let’s put that side for now.

 

Hypocrisy, Idiocy and Online Porn

Friday’s Guardian editorial featured a spit-flecked rant against internet pornography, starting and finishing with a demand to ban all of it. At the same time, the Daily Mail was putting up a strident demand for more efforts to police child porn. These articles are both profoundly wrong on facts and science, and breath-takingly hypocritical, not to mention steeped in conservative morality.

These articles are inaccurate in both their description of the content of internet pornography (and pornography generally), and the science of its effects. In its first incarnation, the Guardian editorial claimed that all internet pornography was abusive and violent, a claim it updated within hours on the same day to reduce the focus of the article to “violent pornography.” Internet pornography covers a very, very wide gamut and is not necessarily violent or abusive at all, and characterizing it as such is ignorant at best, misleading at worst. The Daily Mail claimed that

Experts say Google can combat abuse by paedophiles by simply popping up messages when users type in search terms such as ‘teen sex’ or ‘barely legal’, warning them that they may be about to access illegal material.

There is almost zero chance of finding kiddy porn by those search terms, just a huge number of sites with young adults pretending to be teenagers, and any “expert” who thinks targeting these search terms is going to stop child porn is a fool. I note that the Daily Mail doesn’t bother identifying these so-called “experts.”

These opinion pieces are even more misleading and disingenuous when they talk about the science of porn’s supposed effects. The Guardian provides a range of links to its claim that the science is under dispute and there may be good effects to porn, but is strangely lacking links to any evidence when it makes the ludicrous claim that

there is strong evidence that at the very least it is addictive, can normalise violence, and at the same time diminishes sympathy for its victims

The Daily Mail at least tries to give some science, when it cites a British scientist and says

He pointed to a British study from 2007, which found a ‘substantial minority’ of those who watch child porn go on to attack children.

David Middleton, of De Montfort University, analysed 213 online offenders and 191 paedophiles who had physically abused children.

‘The majority of people [who watch child porn] don’t appear to escalate their behaviour. But a substantial minority do,’ Professor Middleton said.

‘Various studies have looked at this and put it somewhere between one in six and one in ten.

The problem with drawing the conclusion in the first line from the findings of this study is that it runs afoul of Bayes’s theorem. Unless the prevalence of child abuse is very high, the probability of abusing children after watching child porn is much, much lower than the one in six found in the study of child abusers, and the link can’t be properly described until we know how many people view child porn – a figure that is never going to be known, despite the best efforts of the Daily Mail to suggest that watching “barely legal” videos is somehow the same thing.

The big problem with claims that internet porn or child porn desensitize criminals to their victims or encourage crime is that we don’t know the causal order. We know that Mark Bridger had a violent rape scene from a slasher film on loop the night before he went and abducted April Jones. But was this the cause, or the symptom, of his violent disorder? This is a thorny scientific question, and one that will be very hard to answer because of the difficulty of collecting data (and the woeful state of scientific research).

The thing about these articles that really flabbergasts me though, and makes me angry, is their rank hypocrisy. They complain that internet porn is a big business and people have vested interests against stopping it, and that this needs to be fought, but on the morning that I read the Guardian article the newspaper was saturated with adverts for a Thai dating website that featured an upskirt shot of a barely legal Thai girl in pigtails and semi school uniform. What’s that trying to tell readers about the content of the women on that site? And isn’t the Guardian a strong anti-trafficking campaigner? What’s the subtext of Thai dating websites in the UK, if not trafficking? So the Guardian can demand action against internet porn on moral grounds, at the same time as it is broadcasting adverts with barely legal girls showing their crotch? Meanwhile, read any article on the Daily Mail website and you will see the disgustingly named “Femail” sidebar, that contains links to hundreds of articles salivating over young women’s bodies. Yesterday when I read the article on child porn there was a link to an article about a barely legal starlet which started with the words “no daddy’s girl anymore!” And isn’t this the newspaper that, more than any other, reduced the Duchess of Cambridge’s sister to an arse? A crime that all the British media participated in.

The Daily Mail has a heavy investment in salivating over barely dressed women, putting up pictures of wardrobe malfunctions (i.e. upskirts and nipple shots), and reducing “it” girls to a collection of body parts. This is the moral equivalent of porn, just dressed up enough to escape the moral outrage associated with page 3 girls. Make no mistake: the Daily Mail is up to its eyeballs in fetishization of “barely legal” girls; and if the Guardian want to protect “vulnerable women” by banning things, they can start with the advertising on their own website.

It’s also not clear what the Daily Mail hopes to gain by hounding google about child porn, which is already highly illegal and hard to get. The implication of the article is that there is lots of child porn out there, just a google search away. I think the only such “child porn” that anyone will find is actually legal pictures of legal age women pretending to be 16. Is that what the Daily Mail wants to ban? And why can’t they say so?

The reality is that we have no evidence that porn is addictive, desensitizing or dangerous, and porn has been around a lot longer than the internet. There are strong reasons to be uncomfortable with the messages that modern children are getting from online porn, and to think that child porn is linked to child abuse, but the causal nature of these links is far from established. Also, let’s look at some things that are absent from the discussion of children’s safety and causative agents in these articles: there is no mention anywhere of parental supervision, of educating children about sexuality (rather than just sex education), or of ways of “protecting” children other than by banning porn. There is also no mention, anywhere, of the fact that Mark Bridger – who killed April Jones horribly and probably sexually abused her – had spent years working in abbatoirs, killing and dismembering animals. The evidence that cruelty to animals is linked with cruelty to adult women is just as strong as the evidence on child porn (i.e. weak and subject to huge assumptions) and banning abbatoirs is very easy to do. Why are we hearing no calls for this? Or at least, for careful monitoring of and intervention in abbatoir workers? Could it be, to quote the Guardian article, because the meat industry “is a global business” and one that the Guardian supports? Would that be hypocrisy?

Like attacks on sex workers, calls to ban porn are one of the easiest and most successful moral scares for small-minded people to drum up. But they aren’t going to protect women and children, and certainly are going to be of very limited effect compared to the huge benefits to be derived from careful police work tracing and capturing child pornographers. Furthermore, there is almost no link between the mainstream porn industry and child porn, and targeting the former is simply going to divert resources from the latter. The British tabloid media are eager to show that they are strongly against child porn – so long as you don’t look too closely at the barely legal smut they’re peddling in their sidebars. Are these articles a distraction from the real issues in media representation of women? I’m sure they wouldn’t like you to ask …

 

Athenian Punk Rockers Meet 300!

Outside the city …

I am finally away from my Greek Island and the “five star” resort with no internet access, so I am able to resume blogging. Yesterday evening I arrived in Athens for a three day stay, and as is my wont in a new city, the first thing I did was go out for a wander. My hotel has a rooftop bar with a view direct to the acropolis, which is pretty amazing, and is on the temple slopes so it’s a short walk to the old town. Walking through the old town one can catch regular glimpses of the acropolis from the streets, and also experience the pleasures of a summer night in the city. The streets were heaving with people, all out to enjoy the evening air. All the restaurants in Greece seem to be open to the sky, and alfresco dining is the norm, so everywhere you look people are enjoying eating under the stars. I passed a Suleimanese punch-and-judy show, where the puppets are dressed in Persian-style pantaloons and curled hats (but still beating each other) and the horde of gathered children scream at the villain in Greek. I passed a concert being held in an old temple ruin, all lit up with red spotlights. Every square was full of people sitting chatting and drinking; the main square was absolutely heaving with young people in groups just enjoying the night air. The weather was dry and warm, the temperature perfect, the sky a million miles away and clear and the whole balmy evening cupped within the bowl of the distant mountains, with the Acropolis the gleaming jewel set in the middle of that frame, seen occasionally between buildings and lit up against the night sky.

I found a stylish open restaurant in the old town, that served excellent food and had a massively camp Swiss host. They serve a chicken cooked whole inside a loaf of bread and cut up on your plate for you, and an exquisite lemon-flavoured pumpkin soup garnished with little cthulhu-esque octopuslets. I didn’t have my camera with me so didn’t order the cockerel; I may return to experience this strangeness this evening. I have to say, the way Greek people use lemons in their cooking – and the predominance of citrus throughout their cuisine – is excellent and commendable.

After dinner I wandered a little more, enjoying the chaos and light-heartedness of the city. I found myself in the area just west of the Syntagma square, which is supposedy full of bars and night clubs, and in front of a rock bar called Six Dogs. They were hosting an American band called The Shrine, some sort of classic heavy rock outfit that I’ve never heard of, so in I went, for my first experience of Greek punk/metal fans.

What is on your playlist, Archilokos?

The band was average, I have to say, and somewhat hamstrung by the fact that their singer has exactly the same accent as the weird zoo-owner from the Mighty Boosh. They were a pacey, hard rocking classic metal outfit with a bit of skate-punk overtone, so pretty likeable overall. The crowd, however, were fascinating. First of all they were really lively and cheerful, bouncing around with way more energy than the band deserved, and managing to do spontaneous crowd-surfing efforts even though there were only about 50 of them. This meant that whenever one of their number wanted to go up, he had to get the others to lift him, and then a group of 10 or 15 fans would go charging around the room in a little chaotic loop, carrying the surfer aloft, and then drop him. It’s not quite lollapalooza, is it? But they were really into it. But the best thing about them was the way they looked so … classical.

I think every second rocker in the crowd was basically a classical Greek stereotype, come to life then covered in tattoos and stuffed into a pair of skate-punk shorts and a band t-shirt. They all had the broad shoulders and narrow waste of the classic Greek pottery or statues, and that particular style of Greek beard that you see in the classic pictures: the one that is cropped close to the skin along the jaw and near the ears, but extends to a block or point out from the chin, and merges in a perfect gradient with short-cropped hair. It works perfectly with the classical Greek profile of aquiline nose and strong jaw. The rockers also had the same classical hair style, that is neatly cropped at the back but then a little unruly or longer and forward-pointing near the front.

It was like moshing with the guys from 300, if they had bothered to put on t-shirts. It was one of those classic moments, like when a French waiter pulls a 110% expressive face, or a German man says very precisely about one of his most memorable experiences, “it was in general perfect” with German precision, or a Japanese person bows on the phone – one of those moments where the person you are talking to is subconsciously channelling a million years of cultural history and to the rest of the world they’re a stereotype of fantastic proportions, but to them it is so completely normal that they would never realize they were doing it, even if you could play them a video of the moment. So it was that these Greek rockers were moshing not to the tune of an ordinary Venice Beach band, but to a couple of thousand years of classical Greek history. The Pelopennese war through hardcore, or something. I think I will dub this style of Greek counter-culture “300-core.” I hope to see more of it as I wander this city of romance and history!

 

In Praise of Excel

Recently a major economics paper was found to contain basic excel errors, among other problems, and an amusing storm of controversy is growing around the paper. The controversy arises because the paper was apparently quite influential in promoting the most recent round of austerity politics in the western world, and the authors themselves used it actively in this way. The authors even managed to find a magic number – 90% – at which government debt (as a proportion of GDP) throttles growth, a threshold that many small government activists and “sustainable deficit” types have been seeking for years. It’s like manna from heaven!

There’s been a lot of hilarity arising from this, about how woeful the economics field is and about how vulnerable policy-makers on crucial issues like government spending can be to even quite terrible research that supports their agenda. But there has also been some criticism on statistics and academic blogs about the use of excel for advanced analysis, and what a bad idea it is. Andrew Gelman has a thread debating how terrible excel is as a computational tool, and Crooked Timber has a post (with excellent graphic!) expressing incredulity that anyone would do advanced stats in excel. While I agree in general, I feel an urgent need to jump to the defense of MS Excel.

MS Excel is great. It’s much, much more convenient than a calculator, and it can be used to do quite complex calculations (as in sums and multiplications) in multiple directions that would take ages on a calculator. On most computers now the calculator is buried or, if you’re a windows user, crap, and if you need anything more than addition it’s much more convenient to drag out excel. Sure it takes a moment to load compared to your calculator function, but it is so much easier to compare numbers, to calculate exponents and logs, and to present simple results in excel than in a calculator. As a simple case in point: if you get regression coefficients from Stata you can copy and paste them into excel and exponentiate to get relative risks, etc.; then you copy the formulas below, run a new regression model (with, e.g. random effects that weren’t in the previous one) and paste the results to enable you to compare between models quickly and easily. Similarly, if you’re checking a paper to see if they calculated odds ratios or relative risks, you can chuck those numbers into excel and do the comparisons with the contingency table right there in front of you. It offers a simple, graphically convenient way to visualize numbers. This is especially useful when the task you’re approaching is conceptually very simple (like a contingency table) but takes a bit of time to do on a hand calculator, and takes a bit of time to convert to the file formats required in Stata or R. In the time it takes me to think about how to structure the problem, input four lines of data to R, and then write the code to calculate the odds ratios, I can do the whole thing in excel, have the contingency table in front of me to check I’ve made no transcription errors from the paper, and fiddle quickly with changing numbers.

If you’re doing cost-effectiveness analysis in TreeAge (shudder) or R, excel is a really useful tool both for outputting results to something that is vaguely attractive to use, and for doing ballpark calculations to check that your models are behaving reasonably. This is especially useful if you’re doing stochastic Markov models, that can take hours or days to run in TreeAge, because you can’t trust software like that to give you the correct answer if you try to treat your stochastic model as a simple decision tree (because of the way that TreeAge faffs around with probability distributions, which is non-intuitive). Make a few simple assumptions, and you can do approximate calculations yourself in excel, and fiddle with key numbers – cohort size or a few different parameters – and see what effect they have.

Recently I was helping someone with survival analysis and she was concerned that her definition of time to drop out was affecting her results. She conducted a sensitivity analysis in Stata to see what effect it was having, and although with correct programming she could have produced all the material she needed in Stata, the time it takes to do this and debug your code can be time-consuming if you aren’t a natural. It’s much easier with modern machines to just run the regression 10 times with different values of drop-out time and plot the output hazard ratios in excel.

So, I think excel is a very useful tool for advanced modeling, precisely because of its ease of use and its natural, intuitive feel – the properties that recent excel bashers claim make it such a terrible device. While I definitely think it should not be used for advanced models themselves, I find it a hugely valuable addition to the model-building process. Reproducible code and standardized tools are essential for publishable work, but unless you are one of those people who never does any fiddling in the background to work out what’s going on in your model, excel will turn out to be your go-to tool for a wide range of support tasks.

In any case, the bigger problem with Rogoff and Reinhart’s work was not the excel error. Even if they had got the excel code right, their results would still have been wrong because their modeling method was absolutely appalling, and should never have seen the light of day, even at a conference. The main flaws in their work were twofold:

• They binned years together, essentially giving different years different weights in the final model
• They stripped the years out of their time series context, so crucial information contained in the time ordering of deficits and growth was lost

I think the second flaw is the most specifically terrible. By using this method they essentially guaranteed that they would be unable to show that Keynesian policies work, and they stripped the cause-effect relationship from all data collected in the Keynesian era (which lasted from the start of their data series to about 1970). In the Keynesian era, we would expect to see a sequence in which deficit increases follow negative growth, so unless the negative growth periods are very short and random, Reinhart and Rogoff’s method guarantees that this looks like an association between negative growth and higher deficits. If Keynesian policies actually work, then we would subsequently see an increase in growth and a reduction in deficits – something that by design in Reinhart and Rogoff’s model would be used to drive the conclusion that higher debt causes lower growth.

In short, no matter what package they used, and no matter how sophisticated and reproducible their methods, Reinhart and Rogoff’s study was designed[1] to show the effect it did. The correct way to analyze this data was through the presentation of time series data, probably analyzing using generalized least squares with a random effect for country, or something similar. Using annual data I think it would probably be impossible to show the relationship between debt and growth clearly, because recessions can happen within a year. But you could probably achieve better, more robust results in excel using proper time series data than you could get in R from Reinhart and Rogoff’s original method.

The problem here was the operator, not the machine – something which should always be remembered in statistics!

—-

fn1: I use the term “was designed” here without any intention to imply malfeasance on the part of the authors. It’s a passive “was designed”.

 

Abraham Lincoln’s Secret Writings

We here highly resolve that these dead shall not have died in vain

Historians often struggle to come to grips with the more pathological eras of human history. The holocaust, world war 1, Stalin’s terror and the history of slavery are terrifying periods that are difficult for ordinary people to grasp. Why did people do these things? How did whole populations get caught up in frantic social movements with such a destructive and chaotic bent? Usually, balanced and reasonable historians attempt to explain these movements through a mixture of social, economic and institutional phenomena, with varying amounts of great man theory and good or bad luck thrown into the mix. But ultimately, somewhere along the line, historical theory often loses grip on that most simple of facts: that all historical movements depend in the end on individuals to enact them, and these individuals sometimes have to do terrible things. Though it is a terrible work of scholarship, Goldhagen’s Hitler’s Willing Executioners at least attempts to grapple with this vexing problem when, describing the events at Babi Yar, Goldhagen personalizes the actions of the policemen, and asks the reader to imagine what these men must have been thinking when they marched children to their deaths. But in general, somewhere in historical scholarship there must be some gap of understanding, between the grand social and economic movements that produced a pathology, and the willingness of ordinary people to execute that pathology.

A vivid example of this situation is the problem of slavery in the USA, and the war that brought its end. Scholars may characterize the conflicting forces driving this movement – economic, cultural, historical, geographic – and they can describe with some certainty the opinions and goals of the people involved in the fateful decision to go to war, but they still in the end cannot explain the simple personal fact of slavery in a way we can comprehend. How is it that an ordinary human being – similar, one assumes, in fundamental disposition to my own gentle reader(s) – can go to bed happy, dream of happy things, then wake up, step outside, and reduce another perfectly ordinary human being to a state of complete and humiliating servitude? What was that man thinking when he purchased another human being as chattel, and beat them if they refused to cooperate in their own subjugation? And then, when the rest of the world had made it clear that the trade would have to end, what were these men thinking when they decided that it were better to go to war rather than give up this obvious immorality? And why did ordinary men agree to join such a fight, and kill other ordinary men, over something so simply and obviously wrong as the right to get up in the morning and use another human being as if they were an animal?

Many explanations have been given for slavery and its acceptance, but to me they are fundamentally implausible because they fail to connect the broad social and cultural forces claimed to underlie the slave state with those intimately personal moments of cruelty and brutality that link an ordinary human being and his slave. Any such explanation must necessarily be fanciful and involve a reasonable degree of magical thinking: how can the economic forces of the tobacco plantation change the mind of the plantation owner so that he cannot see the humanity of his victim? There is no explanation for this in modern historical thought, and as such all extant theories of the reasons for slavery seem to me fundamentally implausible.

Last night, however, I watched a documentary that finally presented a plausible explanation for this whole era of American history: Vampires. Abraham Lincoln Vampire Hunter offers the first, and I think we can all agree, the most reasonable explanation for slavery yet contrived: the slave holders were not human. They were vampires, and they built a massive slave empire so that they could feed. While other recent films have presented a shabby depiction of Abraham Lincoln as an ordinary man, compromised by the (unexplained) mores of his time and forced to barter over the freedom of millions, in this documentary we see a more nuanced view of our hero, as a great man of unbending will on a heroic mission to use his super-powers against the forces of darkness. I ask you, which is more believable? Could the man who wrote those speeches have been a shabby compromiser, or a secret demon-slayer?

This documentary helps us to understand many things about the civil war. It shows us that white Southerners were not racists of some fantastic creed, but themselves victims of inhuman monsters. It also helps us to understand how such an abominable slave empire could have been established: no humans, with their simple morality, were involved, but instead the whole edifice was constructed by undead. We also see some insight into why Americans take this civil war so seriously. Where previously I had thought it strange that Americans treated this period of their history with so much importance, given it is so terribly embarrassing to have had to go to war in order to simply abolish an institution that the rest of the world already knew was so thoroughly wrong. But in fact, this war really was a titanic battle between the forces of good and evil, and though not all Americans will tell you about the Vampire aspect, they obviously have some sense of it, and thus are justly proud of the efforts of Americans to win and then recover from that cataclysmic struggle. Finally, this movie also helps to answer a question that has long vexed me about British politics. We all know that Tony Blair is a vampire, but many doubt their own sense of his deep and unrelenting evil because they ask themselves, “how can a vampire be abroad in daylight?” This movie answers that question, simply: sunscreen. Vampires use a special sunscreen, which is why Tony Blair always looks so heavily made up. Other objections to his undead nature (“a vampire could not possibly succeed in the labor party because it has neither the fortitude nor the appropriate level of deviousness” or “why did he become PM of England instead of America”) are either easily answered from the record (“Tony Blair is a devious bastard”) or are also answered by this documentary (“he was kicked out of the USA by Abraham Lincoln and he is mighty pissed about it”). So this documentary doesn’t teach us about only America, but also shows us some important insights into why things went so wrong in the UK. I strongly recommend this documentary as a way of better understanding the true nature of the undead who live amongst us.

As an aside, the documentary is also very entertaining, tells the story clearly and simply without confusion, and has such good footage from the battles of the time that you could almost believe it was made in a film studio. It is an entertaining and enlightening look at a very important part of American history, and has given me a newfound respect for Abraham Lincoln. Others may complain that it excuses white racism of the era, or reduces history to the story of one great man, but I think those criticisms fail to adequately take into account one important fact: Vampires. So if you want to learn more about the evil that threatened America, or you want to reassess the evidence for Lincoln’s greatness, I strongly recommend this documentary.

 

Health-related Financial Catastrophe in Bangladesh

This week a student and I published an article in PLOS ONE examining the relationship between healthcare-related expenditure and financial catastrophe in Bangladesh. Because PLOS ONE is an open access journal it is possible to read the entire article free online, here. Our study was a statistical analysis of data from a probability-sampled survey of households in Rajshahi, an urban area in Northwest Bangladesh. We collected data on their self-reported illness, household consumption and healthcare-related payments, and used it to estimate the prevalence and risks of financial catastrophe.

Bangladesh doesn’t really have any effective risk-pooling mechanisms, and a large portion of all health financing is derived from direct payments by individuals, usually referred to as out-of-pocket (OOP) payments. The lack of risk-pooling mechanisms mean that households with limited savings are at risk of financial catastrophe from unexpected healthcare costs, and may have to use a wide range of quite unpleasant coping mechanisms to deal with the costs. Our research project aimed to identify the drivers of costs, factors associated with financial catastrophe, and the coping mechanisms used to deal with high costs.

These kinds of research projects have a lot of challenges, and are necessarily flawed as a result. In low-income nations like Bangladesh it is difficult to assess wealth directly, since households often obtain income in kind or through bartering or intensive production (family gardens, etc); and often official income is not declared in order to avoid taxes or other costs. This is usually dealt with through assessing household consumption, rather than income, adjusting for fixed and productive assets. It’s also difficult to assess illness, which is usually done through self-report, and obviously also medical expenses can be hard to keep track of. There is an extensive body of literature on how to deal with those problems though, and we used mostly quite standard methods to handle them. Despite the obvious limitations in such a survey, I think this one presents fairly robust results.

We found a high prevalence of financial catastrophe, with an average of 11% of household consumption spent on healthcare and 9% of households facing financial catastrophe under our definition. Financial catastrophe was much more likely in the poorest households, even though these households spent considerably less on healthcare, and financial catastrophe was also associated with inpatient service use. Chronic illness was associated with higher OOP payments. Bangladesh is currently passing through the “epidemiological transition,” in which chronic non-communicable disease (NCD) prevalence is rising, but infectious diseases remain a significant problem, so the finding that chronic illness is associated with increased OOP payments is concerning: with a baseline proportion of their income already consumed by such illnesses, households will be less able to adapt to unexpected sudden illness or injury, both of which are relatively common in low-income countries compared to high-income countries.

Our findings suggest that Bangladesh needs to move rapidly to implement and scale-up risk-pooling mechanisms; deal with problems in public facilities that mean they don’t seem to be protective against financial catastrophe even though they are ostensibly free or heavily-subsidized; and prioritize NCDs in its health policy agenda. We’re currently conducting more research on disease-specific costs, coping mechanisms, and other aspects of the health-financing challenges facing Bangladesh. Other countries in Asia are moving towards universal health coverage (UHC) and Bangladesh lags some of them; but with care, a little reform, and some coordinated action to target NCDs, there’s no reason that despite its poverty Bangladesh can’t follow in the footsteps of other countries like Vietnam in reducing risk of financial catastrophe and improving healthcare access for the poorest members of its population.

As an aside, 9% is a very high prevalence of financial catastrophe, but I’d be interested to see how it compared with the USA (which also doesn’t have widespread and effective risk-pooling mechanisms). I don’t think the research is done the same way for US systems as in low-income countries, but there appears to be some evidence that financial catastrophe can be high, at least amongst the poor. For example, this New England Journal of Medicine article suggests that Medicare provides limited protection against financial catastrophe, and shows figures indicating that 4% of recipients pay >$5,000 on medical expenses in any one year, which would probably qualify them for financial catastrophe (since most Medicare users have low incomes). I would be interested to see the rates of financial catastrophe amongst the uninsured in the USA, and to compare them before and after Obamacare is introduced, but I don’t think research on the topic is done in the same way in the high-income countries, so I doubt it will be possible. Although health insurance (private or public) is supposed to protect against unexpected medical expenses, it can still be ineffective, and furthermore access to health insurance enables people to purchase healthcare they might otherwise have neglected, which could put them at risk of financial catastrophe where the insurance system fails to provide adequate coverage. Obamacare is going to extend no-frills coverage to the currently uninsured, but this doesn’t mean they’ll get benefits sufficient to prevent financial catastrophe, so it will be interesting to see whether it meets both of the goals of a health-financing system (improving access and reducing financial risk), just one of them, or neither. And if it fails on either or both of those goals, does this mean that Bangladesh will achieve effective UHC before the USA? That would be interesting … but first Bangladesh needs to start the move toward UHC, and hopefully this research will provide useful information and a little impetus in support of that process …

 

The NHS Begins its Great Reform Adventure

Next week sees the release of the Ken Loach movie The Spirit of ’45, which describes the UK’s attempts to implement socialism through the ballot box between 1945 and 1951. Ultimately a failed project, this revolution has left one enduring and much-loved symbol, the UK National Health Service (NHS). In the same month as the release of Loach’s movie, the NHS is going to undergo what are generally touted as the largest reforms in a generation, the Health and Social Care Bill, which see significant market reforms introduced to the NHS, and a major reorganization of its hierarchy. This is all happening against a backdrop of unprecedented government austerity, recent reports finding significant failings in the way the NHS cares for patients, and a Conservative-run government with a very strong ideological bent towards radical experimentation with the UK’s institutions. It is also happening against the backdrop of a worldwide movement towards universal health coverage (UHC) which is even taking hold in the USA – given this global movement, and the UK’s central role as a model for that movement, it’s unlikely that even the most ideological of governments is going to attack the basic principle of the NHS: to provide healthcare on the basis of need, not ability to pay.

It’s easy to make predictions about how these reforms will turn out, without even analyzing the policy, because a cynical outside observer of the UK can always fall back on three simple principles: nothing in the UK works very well, the British government is terribly incompetent regardless of its ideological stripe, and you can’t improve healthcare by reducing the amount you spend on it[1]. However, I’ve written before about what I think will happen as a result of the new Bill, and the specific good and bad points I see in it; I won’t repeat these in detail here. For those interested in the Bill itself, the Guardian gives an outline of its main points, and since I wrote my post the Bill has been beefed up a little. The revised Bill includes a sneaky little clause that supposedly forces clinical commissioning groups (CCGs) to make all health services they purchase subject to tender, rather than allowing existing NHS services to be preferred providers. CCGs are organizations supposedly formed by doctors which are charged with disbursing government money to providers of healthcare – they are the main purchasers of health services after the reforms have been passed. By forcing them to open all non-emergency services to tender, the Bill will (it is claimed) force existing NHS hospitals and GPs to compete with private services for government money, ideally driving down costs. It’s not clear to me how the contracts between these services and the CCGs will be negotiated, and this aspect of the reforms gets my spider-senses tingling, because it just stinks of “a potentially good idea done badly.” Some of the background on the way the Health and Social Care Bill forces CCGs to use competitive tendering is presented in this opinion piece (but beware, huge amounts of this piece are just factually wrong or very misleading, so take everything it says with a grain of salt). Below are a few reasons why I think this particular competition reform is going to fail.

• It puts family doctors in charge of paying family doctors: a Clinical Commissioning Group is meant to be a group of general practitioners (GPs, or family physicians in the American parlance) who will be given money by the government with the task of purchasing all health services for patients in their area. This is supposed to put the health service back into the hands of GPs. The problems here are two-fold and, I should think, blindingly obvious. Much of the money that these CCGs need to distribute will be spent on purchasing GP services, so it puts GPs in charge of purchasing services from GPs. This is a notoriously tightly-knit community with strong common interests, and I find it hard to believe the money will be dispensed wisely. The second problem with this plan is that GPs may be good doctors, but that doesn’t make them good at resource-allocation decisions, and often doctors are the worst people to decide how to spend money wisely. A good health financing system should find ways to efficiently and equitably enable doctors to make good clinical decisions. It’s not obvious to me how putting doctors in charge of the financing decisions is congruent with this. A lot of commentators in the Guardian are decrying the role of major accounting services, who are being contracted by some of these CCGs to handle the decision-making process[2], but to me this seems like a good thing: the further you can move the pot of money from a GP the better, in my opinion.
• It seems to rely on bulk contracts: The NHS to date seems to have structured a lot of its purchasing decisions on contracts that offer bulk funding – that is, a hospital or GP contracts to provide a service for a specified fee, but the service is very generalized and not broken down into its particulars. In the case of GPs, this usually means they are paid a fixed amount per year to provide services to patients on their list, but no detail is specified as to how they should provide services or what they should provide – not even a minimum basket of services. This is why many GPs in the UK operate single-handed surgeries with inconvenient opening times, have very little time for patients, and don’t provide services (like chest x-rays or chlamydia testing) that are taken for granted as routine in other countries. These contracts don’t encourage efficiency, and when these types of contracts are negotiated with large providers (like private healthcare organizations or big hospitals) it’s likely they will be highly beneficial to the provider unless the CCG negotiating the contract has a very adept team of lawyers. Unless the new Bill includes very strong support for this contract writing framework (and see my further point below) then I expect we will see profligate misuse of funds as the providers take these naive and poorly-supported CCGs to town.
• The financing system is not obvious: It’s not clear to me how the CCGs are expected to decide what is the most economically effective (or, for that matter, clinically effective) service, what benchmarks will be established for comparing services that they are accepting tenders from, and how they are expected to make contracting decisions. This isn’t all the fault of the people writing the Bill: within the field of health services management, there is still much dispute about how best to assess the quality of care provided by a large service. For example, in-hospital mortality might be considered an important measure of quality, but how does one account for the mix of patients and the severity of their illness in comparing two hospitals? Should one trust the figures the hospital presents, and if not who is the central provider of assessment services to which a CCG should turn when attempting to compare hospitals on this measure? And if two hospitals have slightly differing mortality rates, how much extra should a provider be expected to value the difference at? Can a provider make a judgment to buy services from a cheaper hospital with higher age/sex-adjusted mortality rates, or is that decision unethical on its face? Have CCGs got any expertise on these issues, or received any guidance? It could be that the Health and Social Care Bill provides extensive information on this, and supposedly the reforms will include the establishment of a new organization to help CCGs with this task. But the reality is that no one really knows the answers to many of these questions, and it’s not clear that the structure for health financing proposed by the Health and Social Care Bill is going to be invulnerable to problems because of them.
• It lacks centralized guidance and pricing structures: At the moment there is a single contract that all GPs sign with the NHS when they provide services. Will this contract be used by the CCGs? This contract basically gives a fixed pricing system for obtaining GP services – if it is used, how can the CCGs claim to be operating a competitive tendering system? If it is not used, and GPs are to negotiate their services on the basis of the price negotiations, how are the CCGs going to decide the correct price? We’re not talking about a fully free market here, since most CCGs operate to purchase services within a given area and, in general, patients won’t be going outside that area for services, so there won’t be multiple CCGs competing in the area, and patients won’t be voting with their feet if prices are too high (in fact the patients won’t even see the prices). If the system is going to operate without market signals, then it’s going to need some very carefully arranged pricing mechanisms to ensure it doesn’t waste money. These are not likely to be most optimally set at the level of individual CCGs, but would be better off set by the government. It’s not clear to me that this is going to happen, or that the CCGs are going to get much guidance at all on how to fix prices. By way of comparison, Japan operates a system in which private hospitals and clinics charge patients for services on a fee-for-service basis, then charge the government’s insurance system for 70% of the total cost. However, the government maintains a strict schedule of service fees, so it’s extremely difficult for doctors to over- or under-charge. Essentially in this market the government provides a very strong centralized pricing guideline to keep the market at a stable price. The Australian government uses a similar mechanism through setting a minimum fee for GP services provided and allowing patients to vote with their hip pocket when choosing GPs. It’s not clear that the NHS will be using any such system, but in the absence of a strictly market-based mechanism for setting prices[4], how are the CCGs going to be able to choose what to charge, or even to know that they are getting value for money?

Given these concerns, I can see all sorts of disasters befalling the revised network of CCGs, including (for example) the possibility that they set up contracts that take up all their budget, only to find their area massively underserved with health services but no funds left to purchase more. What are they going to do then? How is the government determining how much money each CCG needs, and how can they be sure they have gotten it right? Does the government have any fallback position or plan B if over the first few years of the system the CCGs massively cock-up their purchasing decisions?

On the face of it the reforms appear to consist of a poorly-structured semi-marketization, to be managed by inexperienced and new organizations that have been given arbitrary budgets in an environment with very limited centralized guidance, to purchase services in a marketplace where even the experts are uncertain about how to define value for money or quality of service provision, but with an extremely limited set of real market mechanisms as an alternative way of providing pricing signals. It’s like a healthcare Frankenmarket. How can this story possibly end well? I predict we’ll know before the next election, and I suspect that the results of this grand experiment are going to form the obituary for this government.

—

fn1: well, theoretically you can – lots of governments have built health policy on “efficiency savings.” Practically, however, health systems improve by spending more to gain more, rather than spending less to gain the same.

fn2: I provide no citation for this because I can’t be bothered looking, but really this shouldn’t require a citation, it’s the factual equivalent of saying “the sun rises in the morning.”[3]

fn3: I know I know.

fn4: I don’t maintain here that these systems are necessarily the best, simply that they are a strict guideline and they do roughly seem to work.

 

What to do About Maths?

On Monday I was required to monitor at the Tokyo University undergraduate entrance exams. I shepherded 60 terrified 17 year olds through a 2.5 hour Japanese language test and then a 100 minute maths test. These tests were part of a two day examination process for those want to enter the humanities faculty of Tokyo University. About the Japanese test I can say nothing, but the maths test interested me, and can be found online (in Japanese) at the Mainichi Shinbun newspaper. In order, based on my feeble attempts at translating the exam, the four questions were:

• A straightforward but nasty calculation of the properties of a line intersecting with a cubic function, including elucidation of all minima and maxima of the products of the lengths of two line segments
• A geometry question with two proofs
• A constrained linear programming problem
• A simple Markov model with a slight twist

The students had 100 minutes, and to their credit quite a few of the students managed all four, though a lot also stumbled and didn’t get past two. I would say that for a well-trained student with good maths skills, these four questions can all be done inside their allotted 25 minutes, but it’s a pretty risky process – even a small error at the start, or misconception of how to do the problem, and you have basically lost the whole question because you only have time to attack the problem once. And these problems are probably about the same level of difficulty as the questions on a standard year 12 maths exam in Australia – where usually we would have three hours.

But these questions were for the Humanities Faculty of this university. If you want to study Japanese literature at Tokyo University, you first have to get through that 100 minutes of high level mathematics. It says something, I think, about the attitude of Japanese people towards mathematics, and towards education in general, that they would even set a mathematics test for access to a Humanities Faculty; and it says even more about the national aptitude for maths that the students could tackle this exam.

At about the same time as these exams were being held, the Guardian and the Sydney Morning Herald released articles slamming the mathematical and science abilities of the average student in the UK and Australia, respectively. The Guardian reported on a new study that found English star students were two years behind their Asian counterparts in mathematics, with 16 year old English students at the same level as 14 year old Chinese. The study also found that

The research also found England’s most able youngsters make less progress generally than those of similar abilities across the 12 other countries studied. The other countries studied were Singapore, Hong Kong, Taiwan, Japan, Australia, Slovenia, Norway, Scotland, the US, Italy, Lithuania and Russia.

Meanwhile, the Sydney Morning Herald reported on a new study showing that the proportion of students doing mathematics is falling fast, with apparently only 19% of students studying maths, science or technology in their final year of school, and a rapid fall in mathematics enrollments amongst girls especially. The corresponding figure in Japan in 2002 was 64%.

So is this a problem, why is it common to the English speaking world and viewed so differently in Asia, and what can be done about it? Obviously as a statistician I think this is woeful[1], and it certainly is my personal opinion that understanding mathematics is a good thing, but is it bad for a society as a whole to neglect mathematics education? I don’t know if that’s objectively verifiable. So let’s skip that question, assume for now that improving the number of people taking mathematics is good, and just jump onto the question of why it is unpopular in Australia, and why the British are so bad at it.

First, I would like to dispute the possible explanation provided in the Guardian article by “the researchers”:

In east Asian cultures education has historically been highly valued. This can be seen not only in teachers’ high salaries, but also in the heavy investment of families in private tutoring services

While it may be true that “social and cultural factors” affect maths achievement, the idea that Asians are better at maths because they value education more highly is a very weak one. If this were the case, would it not also be the case that Japanese would universally be better at foreign languages than the British or Australians? Japanese get a long exposure to English teaching but are generally woeful at it, despite all the money they sink into private tutoring services. No, there’s something else going on here, something about the Asian approach to maths and the way it is taught that is important.

It is certainly the case that private tutoring services need to be considered in the mix. When comparing a 16 year old English student to a 14 year old Japanese student, for example, you are comparing someone who does a 9 – 5 study day with very long winter and summer holidays against someone who does an 8 – 8 study day with two-week holidays, and who gets 2-on-1 or small group tutoring in key subjects for up to 3 hours a day, and on weekends. This process starts at age 10 and really ramps up at about age 15-16, just when the linked article finds the biggest gap between English and Asian students. It’s also the kind of process that benefits the “brightest” students most, and would explain the gap very nicely.

It may be that if the UK wants to compete with the sleeping giants of Asia on basic educational outcomes, it’s just going to have to face up to a simple fact: British students need to study harder. A lot harder.

There are some more nebulous cultural factors that come into play, however, and I am going to go out on a limb here and name a few factors in Japanese society (the part of Asia I am familiar with) that I believe make Japanese so much better at maths than their western counterparts.

• It isn’t about native talent: A pet hate of mine about western approaches to mathematics is the idea that some people are talented at it, and most people aren’t. I don’t think this is true at all, and I think it’s not something that Japanese believe very strongly. The reality is that getting good at maths is a long, hard slog that involves a huge amount of repetition of basic skills (things like completing the square, substitution, differentiation, interpreting graphs, sign diagrams, etc.) – just like learning a language. Sure, solving maths problems requires creativity and intuition, but these are only of any value if you know the tools you can apply them to, and are familiar enough with those tools to recognize when and how to use them. Mathematics – and especially high school mathematics – is a process of drilling, drilling, drilling, and I think that Japanese recognize this. In Japan the default assumption is that if you pay attention at school and do your homework, you will be good at maths. Sure, they recognize that advanced maths requires extra commitment and talent, but there is a fundamental assumption here that the broad body of maths (up to and including differentiation, integration, limits, and basic probability theory) are things that anyone can learn.
• The teacher is important: the flip side of the idea that education is important is an increased stress on the value of the teacher, and their role as a guide. The role of the guide is also viewed very differently if they are teaching something that they believe anyone can do, compared to if they are teaching a subject that everyone believes is impossible for most mortals to comprehend. Find me a westerner under the age of 30 who is “terrible at maths” and I will show you someone who was humiliated by an arrogant maths teacher at a crucial time in high school, usually around when they were 14. I was in the bottom class in mathematics when I was 14, expecting to drop out as soon as possible, until a good teacher put some time into teaching me, and I found that I really loved it. In Japan, teachers can be bullies and they can be cold and hard, but I would also argue that they have a much greater burden of personal care and responsibility placed on them compared to western teachers, and the failure of their students is treated more like a professional failure (rather than due to the student’s personal talents) than it is in the west. I think this is especially important with mathematics, because when you don’t get it it really hurts – like a kind of itching in the back of your brain – and the failures pile up rapidly. Just a single year between 12 and 14 in which you give up on maths is enough to make all the subsequent years ever more challenging, meaning the damage and the attendant confidence failures compound.
• Being nerdy is cool: In Japan, it’s okay to be a nerd, and being good at mathematics is admired and respected. It’s virtually unheard of to find someone here who looks down on a man who can do maths, or thinks that it is beyond the female brain, or thinks that being interested in mathematics is weird. Furthermore, the nerd world in Japan is much more gender neutral than in the west, so there’s nothing unusual about girls doing maths. Good mathematics skill – up to and including being able to rearrange equations or solve systems of equations, for example – is not seen as a weird foible, but as an admirable sign that you are a rounded human being.
• There is a social expectation of mathematical skill: In addition to nerdiness being much more acceptable, the range of mathematical abilities that qualify you as a nerd in Japan is much more esoteric and advanced than in the west. There is a general expectation that ordinary people can solve maths problems, that they understand the basic language of mathematics so that even if they can’t solve a problem they know roughly what it is and where it sits in the pantheon. Parents assume that their kids will learn mathematics, and don’t dismiss it as the too hard subject that only the special or the weird get ahead in. Whereas in Australia having a kid who is good at maths is unusual, in Japan it is unusual (and embarrassing!) to have a kid who is not good at maths.

I think these properties add up to a society in which mathematical achievement is encouraged and widespread. I think that Australia and the UK need to change some cultural factors so that the intellectual and educational landscape is closer to that in Asia if they want to keep up on mathematics and technology achievement – especially since China’s education system is maturing, and other Asian nations like Vietnam, Singapore and India are getting wealthier, with all the educational gains that implies. So what should Australia do?

• Ditch the nerd-baiting: there’s something really wrong with the way the English-speaking world treats people who do nerdy things. I’m sure it’s mellowed a lot since I was a kid but it’s still there, the kind of ugly-four-eyes assumption about anyone who is interested in anything that isn’t sport or fashion. Until this weird attitude dissipates – and until the nerd world becomes more gender-balanced, to boot – it’s going to be hard to encourage the kind of cultural changes needed to make maths achievement standard across the board
• Less intuition and initiative, more drills: I think it’s very sweet that maths teachers want to encourage their charges to think about the broader world of maths, about creative problem-solving, about applying maths to the real world, etc. But I think those are natural talents all humans possess, that cannot be unlocked without a robust background in the basic skills that make mathematics work. So leave the creativity for people who need it, and stuff kids’ heads full of “useless” rote learning of techniques and drills. It’s boring, but it’s essential to the bigger stuff. If you aren’t able to immediately see when and how to complete a square, then any problem which requires this basic technique is going to be beyond you, no matter how intuitive you are. Maths, possibly more than any other discipline, is built from the ground up, tiny block by tiny block, and all those blocks are essential. So ram them down every kid’s throat, and make every kid think that knowing the quadratic formula is not a test of some kind of obscure talent, but a basic expectation of every 12 year old
• Force mathematics at higher school levels: When I finished school our balance of subjects had to include at least one science/technology subject, but it didn’t have to include maths. This is wrong, and part of the reason that so many students in Japan do mathematics is that you can’t get into a good university if you take this approach: every one of the better universities includes mathematics in its entrance exam. My personal belief is that completion of higher school certificates should require one foreign language, mathematics, and English. That leaves two other subjects to choose from, and guarantees that you have to do some kind of mathematics to the end of school. Not only will this very quickly lead to a society where entire generations of people are generally familiar with mathematics, it will also put a real focus on the quality of teaching at the earlier years, since any student who is doing badly in years 8 – 10 is going to fail their higher school certificate. [Probably this suggestion for a national curriculum is completely unreasonable, but at the very least students could be forced to do mathematics up until year 11, for example].
• Make school more robust: The Japanese school system is about to shift to a “tougher” system that will include Saturday morning classes, because the previous system was considered “relaxed” compared to earlier years. This is, frankly, ridiculous, but so is the attitude towards education of most of the English-speaking world. Summer holidays are way too long and relaxed, there is a real lack of extension classes and tutoring, and expectations are altogether too low. Education isn’t valued enough, and until this changes anyone who wants their child to do better is going to be swimming against a strong current. Educational achievement is partly supported through the shared goals of a whole society, not just through the targets of individual families, and the expectations we hold for education are primarily set through the school system. So toughen it up – not in the sense of making teachers scarier or bringing back outdated “three Rs” educational styles, but by increasing the amount of time students spend at school, setting tougher standards for graduation and university entrance, making schools compete with each other (as Japanese schools partly do) and forcing parents to take greater responsibility for and involvement in their children’s education. This change isn’t specific to mathematics, but it would certainly help.

I don’t think there’s anything special about Asian students, or about Asian culture, that we can’t adopt. Asians’ mathematics achievements aren’t some kind of native or racial talent. It’s just a collection of attitudes towards education, mathematics and nerdiness that we can adopt if we want. Obviously there will be (potentially challenging) institutional changes required as well, and many people may judge it not worth the effort, but I personally think a world where everyone is good at mathematics is a better world, and we should be aiming for it. With these cultural changes maybe one day everyone will know the obvious thrill of being able to complete a challenging mathematics exam … and enjoying it!

—

fn1: Though obviously, the less people doing maths, the longer I will remain competitive in the marketplace …

 

Importing Japanese Data into Stata in Windows and Apple

It’s all Greek to you, isn’t it?

I received a very interesting hospital dataset recently, in excel format and containing some basic variable names and values in Japanese. These included the sex of the patient, the specialty under which they were admitted to hospital, and all variable names. Initially this would be reasonably easy to convert to English in excel before import, but it would require making a pivot table and fiddling a bit (my excel-fu) is a bit rusty, but also I have address data and though at this stage it’s not important it may be in the future. So, at some point, I’m going to have to import this data in its Japanese form, so I figured I should work out how to do it.

The problem is that a straight import of the data leads to garbled characters, completely illegible, and very little information appears to be available online about how to import Japanese-labeled data into Stata. A 2010 entry on the statalist suggests it is impossible:

Unfortunately Stata does not support Unicode and does not support other multi-byte character sets, such as those necessary for Far Eastern Languages. If you are working with a data set in which all of the strings are in a language that can be represented by single byte characters (all European languages) just choose the appropriate output encoding. However, if your dataset contains strings in Far Eastern langages or multiple languages that use different character sets, you will simply not be able to properly represent all of the strings and will need to live with underscores in your data.

This is more than a little unfortunate but it’s also not entirely correct: I know that my students with Japanese operating systems can import Stata data quite easily. So I figured there must be something basic going wrong with my computer that was stopping it from doing a simple import. In the spirit of sharing solutions to problems that I find with computers and stats software, here are some solutions to the problem of importing far Eastern languages for two different operating systems (Windows and Mac OS X), with a few warnings and potential bugs or problems I haven’t yet found a solution for.

Case 1: Japanese language, Windows OS

In this case there should be no challenge importing the data. I tried it on my student’s computer: you just import the data any old how, whether it’s in .csv or excel format. Then in your preferences, set the font for the data viewer and the results window to be any of the Japanese-language OS defaults: MS Mincho or Osaka, for example.

This doesn’t work if you’re in an English language Windows, as far as I know, and it doesn’t work in Mac OS X (this I definitely know). In the latter case you are simply not able to choose the Japanese native fonts – Stata doesn’t use them. No matter what font you choose, the data will show up as gobbledigook. There is a solution for Mac OS X, however (see below).

Case 2: English language, Windows OS

This case is fiddly, but it has been solved and the solution can be found online through the helpful auspices of the igo, programming and economics blogger Shinobi. His or her solution only popped up when I did a search in Japanese, so I’m guessing that it isn’t readily available to the English language Stata community. I’m also guessing that Shinobi solved the problem on an English-language OS, since it’s not relevant on a Japanese-language OS. Shinobi’s blog post has an English translation at the bottom (very helpful) and extends the solution to Chinese characters. The details are on Shinobi’s blog but basically what you do is check your .csv file to see how it is encoded, then use a very nifty piece of software called iconv to translate the .csv file from its current encoding to one that can be read by Stata: in the example Shinobi gives (for Chinese) it is GB1030 encoding, but I think for Japanese Stata can read Shift-JIS (I found this explained somewhere online a few days ago but have lost the link).

Encoding is one of those weird things that most people who use computers (me included!) have never had to pay attention to, but it’s important in this case. Basically there are different ways to assign underlying values to far Eastern languages (this is the encoding) and although excel and most text editors recognize many, Stata only recognizes one. So if you have a .csv file that is a basic export from, say, excel, it’s likely in an encoding that Stata doesn’t recognize on an English-language OS. So just change the encoding of the file, and then Stata should recognize it.

Working out what encoding your .csv file is currently in can be fiddly, but basically if you open it in a text editor you should be able to access the preferences of the editor and find out what the encoding is; then you can use iconv to convert to a new one (see the commands for iconv in Shinobi’s blog).

Unfortunately this doesn’t work on Mac OS X: I know this, because I tried extensively. Mac OS X has iconv built in, so you can just open a terminal and run it. BUT, no matter how you change the encoding, Stata won’t read the resulting text file. You can easily interpret Shinobi’s solution for use on Mac but it won’t work. This may be because the native encoding of .csv files on Mac is unclear to the iconv software (there is a default “Mac” encoding that is hyper dodgy). However, given the simplicity of the solution I found for Mac (below), it seems more likely that the problem is something deep inside the way Stata and the OS interact.

Case 3: English-language, Mac OS X

This is, of course, something of a false case: there is no such thing as a single-language Mac OS X. Realizing this, and seeing that the task was trivial on a Japanese-language Windows but really fiddly on an English-language windows, it occurred to me to just change the language of my OS (one of the reasons I use Apple is that I can do this). So, I used the language preferences to change the OS language to Japanese, and then imported the .csv file. Result? Stata could instantly read the Japanese. Then I just switched my OS back to English when I was done with Stata. This is a tiny bit fiddly in the sense that whenever you want to work on this file you have to switch OS languages, but doing so on Apple is really trivial – maybe 3 or 4 clicks.

When you do this though, if you aren’t actually able to read Japanese, you’ll be stuffed trying to get back. So, before you do this, make sure you change your system settings so that the language options are visible on the task bar (you will see a little flag corresponding to your default locale appear next to the date and time). Then, make sure you know the sequence of clicks to get back to the regional language settings (it’s the bottom option of the language options menu in your taskbar, then the left-most tab inside that setting). That way you can change back easily. Note also that you don’t, strictly speaking, have to change the actual characters on the screen into Japanese! This is because when you select to change your default OS language, a little window pops up saying that the change will apply to the OS next time you log in but will apply to individual programs next time you open them. So you can probably change the OS, open Stata, fiddle about, close Stata, then change the OS back to English, and so long as you don’t log out/restart, you should never see a single Japanese-language menu! Weird, and kind of trivial solution!

A final weird excel problem

Having used this trick in Mac OS X, I thought to try importing the data from its original excel format, rather than from the intermediate .csv file. To my surprise, this didn’t work! In programming terms, running insheet to import .csv files translates the Japanese perfectly, but running import to import the excel file fails to translate properly! So, either there is something inaccessible about excel’s encoding, or the import program is broken in Stata. I don’t know which, but this does mean that if you receive a Japanese-language excel file and you’re using Mac OS X, you will need to export to .csv before you import to Stata. This is no big deal: before Stata 12, there was no direct excel import method for Stata.

A few final gripes

As a final aside, I take this as a sign that Stata need to really improve their support for Asian languages, and they also need to improve the way they handle excel. Given excel’s importance in the modern workplace, I think it would be a very good idea if Microsoft did more to make it fully open to other developers. It’s the default data transfer mechanism for people who are unfamiliar with databases and statistical software and it is absolutely essential that statisticians be able to work with it, whatever their opinions of its particular foibles or of the ethics of Microsoft. It also has better advanced programming and data manipulation properties than, say, OpenOffice, and this makes it all the more important that it match closely to standards that can be used across platforms. Excel has become a ubiquitous workplace tool, the numerical equivalent of a staple, and just as any company’s staplers can work with any other company’s staples if the standards match, so excel needs to be recognized as a public good, and made more open to developers at other companies. If that were the case I don’t think Stata would be struggling with Asian-language excel files but dealing fine with Asian-language .csv files.

And finally, I think this may also mean that both Apple and Microsoft need to drop their proprietary encoding systems and use an agreed, open standard. And also that Windows need to grow up and offer support for multiple languages on all their versions of Windows, not just the most expensive one.

Lastly, I hope this post helps someone out there with a Japanese-language import (or offers a way to import any other language that has a more extensive encoding than English).