Hey, just what you see, pal!

Hey, just what you see, pal!

With the sad events recently in America, and president Obama’s newfound (apparent) determination to do something about assault weapons, a lot of attention has focused on Australia’s National Firearms Agreement (NFA), which outlawed certain categories of guns and included a program to buy back guns already in circulation (commonly called the “gun buyback program”). Several studies have been conducted in the aftermath of the agreement, and these seem to have found that it was associated with a reduction in suicides and possibly homicides, with some evidence that the widely expected substitution in suicide methods didn’t offset the gains. The suicide gains appear to be quite large. I think the definitive paper is probably that by Simon Chapman and colleagues, which can be read free online (I think) through pubmed and gives some impressive charts showing declines in gun-related mortality. It is obvious from those charts though that homicide rates were declining anyway, so the results on homicide are weak. Leigh and Neill published a state-by-state analysis of similar trends, with details on the magnitude of the buyback program in each state, and also found tentative evidence in support of the law’s effects on homicides and suicides. However, there is a side debate about whether the law had an effect on mass killings, these events having been the spur for the law in the first place and also the primary focus of any gun laws that Obama will introduce.

The figures for mass killings are fairly impressive: Australia had 12 between 1980 and 1996, but has had zero since. New Zealand had 3 between 1980 and 1996, and had 1 since (in 1997). The USA has a continuing rate of mass killings, and these comparisons are suggestive of an effect of the law. Here I will argue that we need to wait another 10 years before we can make any statement about the effectiveness of the laws in this regard, and probably another 20 years before we can rule out the possibility that the sudden cessation of mass killings was due to other reasons. I will be basing a lot of this analysis on a dubious paper by McPhedran and Baker in the Justice Policy Journal, which contains the data on mass killings for the two countries.

Methods

Data on all mass shootings in New Zealand (NZ) and Australia were collected from the McPhedran and Baker paper for the period 1980-2012. In this paper a mass shooting was defined as an event where at least 4 people were killed by gunshots, not including the perpetrator. McPhedran and Baker give the number of mass shootings and the total death toll for each country. The study period was divided into pre-NFA (1980-1996) and post-NFA (1997-2012) periods. For the pre-NFA period the number of mass shootings was turned into a rate per unit of NZ population per year, on the assumption that although populations changed over time during the study period, the ratio of Australia’s to NZ’s population remained roughly static at approximately five times. Practically, this means that the number of mass shootings in NZ was divided by the length of the pre-NFA period (16 years) to get an annual rate of shootings, while the the number of mass shootings in Australia was divided by five times the length of the period to get its rate. This enables a sloppy, back-of-the-envelope calculation without fiddling with tricky population changes[1].

Once the rates had been calculated for the pre-NFA period, the probability that the observed number of shootings would be seen in the post-NFA period was calculated, on the assumption that mass shootings follow a poisson distribution and that the rate of shootings was not affected by the NFA. If the probability of observing the given number of shootings in Australia in the post-NFA period was less than 5%, we conclude that the NFA was effective; if the probability of observing the given number of shootings in NZ in the post-NFA period was less than 5%, we conclude that some other cultural change was occurring to drive down rates of mass shootings.

Finally, if the probability of observing the given number of shootings was greater than 5%, the additional number of years required to get the probability below 5% was calculated for both countries. Dubious conclusions were drawn on the basis of this[2].

Results

There were 12 shootings in Australia in the pre-NFA period, with 96 deaths, and 0 in the post-NFA period. There were 3 shootings in the pre-NFA period in NZ, with 24 deaths, and 1 in the post-NFA period with 5 deaths. Using the bodgy standardization method, we would expect to observe 15 mass shootings causing 120 deaths in Australia in the pre-NFA period if it had the same mass shooting rate as NZ, which suggests that shootings were less prevalent in Australia before 1996.

The rate of mass shootings per unit of NZ population per year in the pre-NFA period were 0.19 in NZ and 0.15 in Australia. Under the assumption that mass shootings are Poisson distributed, if these rates were to continue in the 15 year post-NFA period, there would be a 22% probability of observing only 1 or 0 mass shootings in NZ, and an 11% probability of observing 0 mass shootings in Australia. That is, we cannot say that the large reduction in the number of shootings post-NFA was statistically significant in Australia.

In fact, in order to conclude that the NFA had a significant effect on mass shootings, we would need to observe no mass shootings in Australia between 1997 and 2017. Conversely, if there are no mass shootings in NZ between now and 2023, we can conclude that a cultural change occurred in NZ that led to a reduction in mass shootings. So really, we have to wait until 2023 before we can draw any statistical conclusions about the effect of the NFA on the rate of mass shootings, unless we are willing to argue that New Zealanders are so radically different to Australians that they are not an appropriate control group for such a comparison.

Conclusion

It is not possible to conclude at this stage that the National Firearms Agreement reduced the rate of mass shootings, and we will need a much larger period of study before we can. Note that were we to approach this problem in 2017, when we had sufficient post-NFA data, we would also be best advised to extend the pre-NFA study period to 20 years, and this would likely reduce the rate of shootings in Australia (I think the 90s were a high water mark for this), which would again mean that we would likely again find no effect. It’s a bit of a hare and tortoise thing, this.

This study has limitations. Its primary limitation is that it is extremely bodgy. Secondary limitations include its having bought into the dubious framing of the McPhedran and Baker paper, its lack of attention to mortality, and the fact that it used a Poisson rather than a negative binomial distributional assumption – I think the results would be even less conclusive if a negative binomial distribution were used but I don’t now that much about those distributions, so can’t comment.

As a final note, it’s my impression that the NFA is well liked in Australia, easy to implement, cheap, and apparently effective in reducing suicides and homicides, possibly meaning it was more effective than originally intended. This law was implemented very quickly by a new and quite inexperienced prime minister, in a pressure cooker public environment and often against the interests of a constituency that was naturally inclined to vote for him. I think this is a testament to the policy-making skills of Australia’s bureaucracy at the time, and also to Howard’s political skill in that period. This exactly the kind of situation in which bad legislation will be drawn up, but he seems to have done a good job. I think it would be really fascinating if he and a few of his advisors and bureaucrats of that era were to get together and write a perspective piece for a major medical journal, describing the pressures they faced and how they managed to produce the legislation given the public and policy challenges they faced. I think it would get into the BMJ – they should do it! Whether the legislation was implemented as a cynical popularity boost by Howard, or for genuine public policy purposes, the results have been surprising and I think it would be interesting to see how such a beneficial law can be developed and implemented in such an environment.

It’s also worth remembering that the previous government under Bob Hawke faced a similar challenge with HIV/AIDS in 1984, and responded similarly effectively. This suggests that in health and law enforcement, Australia’s policy development framework is robust, effective and well liked by the population independent of the political leanings of the party in power. Let’s hope it stays that way long beyond the post-NFA period!

fn1: opinions are easy to have, and one shouldn’t have to work too hard to arrive at them.

fn2: For those tempted to try this at home, this is definitely not how the experimental method works. It’s a thought[3] experiment!

fn3: Obviously I’m using this term quite loosely.

Advertisements