Today the Guardian reported on a new study that claims a large sale of legal ivory in 2008 actually led to an increase in illegal elephant poaching. Basically in 2008 China and Japan were allowed to pay for a large stockpile of legally-obtained ivory, in the hopes that this would crash the market and drive ivory traders out of business. Instead, the study claims, the sale led to a big increase in poaching – approximately a 66% increase in elephants killed, according to the study. This is interesting because it appears to put a big dent in a common libertarian idea for preserving endangered species – that allowing a regulated trade in them would lead to their preservation. It is also one of those cute findings that puts a hole in the standard just-so story of “Economics 101” that everything is driven by supply and demand. We all know that in reality there are many factors which moderate the effect of supply and demand on crucial markets, and on the surface this study appears to suggest a quite contradictory supply and demand relationship in illegal poaching markets, in which increasing supply boosts poaching. But is it true?
The Guardian report links to the original study, which is held at the National Bureau of Economic Research behind a paywall, but which I managed to get a copy of through my work. I thought I would check the statistical methods and see if the study really did support this conclusion. My judgment is that this study is quite poor, and that the data doesn’t support that conclusion at all, due primarily to three causes:
- A poor choice of measure for illegal poaching that doesn’t clearly measure illegal poaching
- The wrong choice of statistical method to analyze this measure
- The wrong experimental design
I will go through each of these reasons in turn. Where equations are needed, I have used screenshots from the original paper because I’m terrible at writing equations in html. Let’s get started.
The PIKE is a terrible measure of illegal poaching
The study is based around analysis of a data set of “legal” and “illegal” carcasses observed at search sites in 40 countries. Basically a “legal” carcass is an elephant that died on its own, while an illegal one is one that was shot and looted. Apparently poachers don’t bother to clean up the corpse, they just cut off the ivory and run, so it’s easy to see when an elephant has been poached. However, because no one knows the full details of elephant populations, the authors study an outcome variable called the PIKE, which is defined as the ratio of illegal carcasses to total carcasses. In their words (screenshot):
They say that this enables them to remove the unknown population from the outcome by “normalizing” it out in top and bottom of the ratio. They justify this with a little proof that I am not convinced by, since the proof assumes that probability of discovering carcasses is independent of the number of carcasses, and that legal mortality and illegal mortality are not related in any way. But even if it factors out population, this PIKE measure doesn’t tell you anything about illegal poaching. Consider the following hypothetical scenario, for example:
Imagine a population of elephants in which all the older elephants have been killed by poachers, so only the pre-adult elephants remain. Every time an elephant becomes mature enough to have decent tusks a poacher kills it and the corpse is found. Further, suppose that the population is not subject to predation or other causes of legal mortality – it is young, and the environment is in good shape so there are large stocks of easier prey animals for lions to target. This population is at high risk of collapse due to adults being killed as they mature; indeed, let’s suppose no babies are born because adults are poached as soon as they reach sexual maturity. Thus every time an elephant is killed, the population drops by one towards its inevitable crash.
In this case, at every time point the PIKE would be 1, because there are no legal carcasses. The PIKE will remain 1 until there are no elephants left to die, at which point it will jump to infinity. It doesn’t tell us anything about the impending population collapse.
Consider now a situation where there are a great many more legal deaths than illegal deaths. Denoting illegal carcasses by y and legal carcasses by x, we have y/(y+x) where y<<x. In this case we can approximate the PIKE by y/x, and if e.g. the number of illegal carcasses suddenly doubles we will see an approximate doubling in the PIKE. But suppose y is approximately the same as x. Then we have that the PIKE is approximately 1/2. Now suppose that the number of illegal carcasses doubles; then the PIKE increases to 2/3, i.e. it nowhere near doubles. If the number of illegal carcasses again doubles, it increases to 4/5. But if all deaths drop to 0 it then increases to infinity … So the magnitude of the increase in PIKE is not a direct reflection of the size of the change in poaching, and in at least one case even the direction is not meaningful. That is not a well-designed measure of poaching. It is also scale free, which in this case is a bad thing because it means we cannot tell whether a value of 1 indicates a single illegal carcass or 10 illegal carcasses. Similarly we don’t know if a value of 1/2 corresponds to 1 or a million illegal carcasses; only that however many there are, they are half of the total.
The authors say that this variable is constrained between 0 and 1, but this is not strictly true; it actually has an additional non-zero probability mass at infinity. This strange distribution of the variable has implications for model choice, which leads us to the second problem with their data.
All the models in this study were poorly chosen
The authors choose to model the PIKE using an ordinary least squares (OLS) model with fixed effects for country and a (separate) fixed effect for each year. An OLS model is only valid if the residuals of the model are normally distributed, which is a very strong assumption to make about a variable that has lots of values of 0 or 1. The authors claim their residuals are normally distributed, but only by pooling them across years – when you look at residuals within individual years you can see that many years have much more normally distributed residuals. They also don’t show us the crucial plot of residuals against predicted values, which is where you get a real idea of whether the residuals are well-behaved.
An additional consequence of using an OLS model is that it is possible to predict values of the PIKE that are unphysical – values bigger than 1 or less than 0 – and indeed the authors report this in 5.6% of their data points. This is indicative of another problem – the PIKE shows a non-linear response to increased illegal kills (see my example from 1/2 to 2/3 to 4/5 above), so that for a fixed number of legal kills each additional illegal kill has a diminishing effect on the value of PIKE, but a linear OLS model assumes that the PIKE changes by a uniform amount across its range. Given that the goal here is to identify increases in the PIKE over time, this runs the risk of the model over- or under-estimating the true effect of the 2008 ivory sale, because it is not properly modeling the response of the PIKE score.
The authors try to test this by fitting a new model that regresses ln(illegal carcasses+1) against a function that includes ln(legal carcasses+1) like so:
This introduces a new set of problems. The “+1” has been added to both variables here because there are many zero-valued observations, and ln(0) doesn’t exist. But if there are lots of zero-valued observations, adding one to them is introducing a big bias – it’s effectively saying there was an illegal carcass where previously there wasn’t one. This distorts low numbers and changes the patterns in the data. The authors claim, furthermore, that “The coefficient on legal carcasses φ will be equal to unity if the ratio of illegal carcasses to legal carcasses is fixed”, but this is both nonsensical and obscures the fact that this model is no longer testing PIKE. It’s nonsensical because that is not how we interpret φ. If φ=1, then we can rewrite their equation (8) so that the left hand side becomes the natural logarithm of (illegal carcasses+1)/(legal carcasses+1). Then we are fitting a linear model of a new variable that is not the PIKE. We are not, however, assuming the ratio of illegal carcasses to legal carcasses is fixed. If φ is not 1, we are modeling the natural logarithm of (illegal carcasses+1)/(legal carcasses+1)^φ. The ratio here is still fixed, but the denominator has been raised to the power φ. What does “fixed” even mean in such a context, and why would we want to model this particular strange construction?
The authors do, finally, propose one sensible model, which is similar to equation (8) (they say) but uses a Poisson distribution for the illegal carcasses, and still fits the same right hand side. This is better but it still distorts the relationship between illegal and legal carcasses by adding a 1 to all the legal (but not the illegal) carcasses. It also doesn’t properly account for elephant populations, which is really what the legal carcasses serve as a proxy for. There is a much better way to use the legal carcass data and this is not it.
Finally there are two other big problems with the model: It uses fixed rather than random effects for country and site, which reduces its power, and also it doesn’t include any covariates. The authors instead chose to model these covariates separately and look for similar spikes in specific possible predictors of ivory usage, such as Chinese affluence. The problem with this is that you might not see a strong spike in any single covariate, but multiple covariates could move together at the same time to cause a jump in poaching. It’s better to include them in the model and report adjusted poaching numbers.
The wrong experimental design
An expert cited in the original article noted this interesting fact:
The Cites spokesman also noted that there had never been a one-off sale of rhino horn: “However, the spike in the number of rhinos poached for horn largely mirrors what has been seen with ivory. The illegal killing of rhino for its horn in South Africa alone increased from 13 in 2007 to close to 1,200 last year.”
This suggests that there has been an upsurge in illegal poaching across Africa that is independent of the ivory sale, and could reflect changing economic conditions in Africa (though it could also reflect different markets for ivory and rhino horn). It’s possible to test this using a difference-in-difference approach, in which rhino poaching data is also modeled, but is treated as not having been exposed to an intervention. The correct model specification then enables the analyst to use the rhino data to estimate a general cross-species increase in poaching; the elephant data identifies an additional, elephant-specific increase that could be said to be due to the ivory sale. The authors chose not to do this, which means that they haven’t rigorously ruled out a common change in poaching practice across Africa. If the CITES spokesman’s point is correct, then I think it likely that we would conclude the opposite to what this study found: that compared to rhinos, elephant poaching did not increase nearly as much, and in fact the ivory sale protected them from the kind of increased poaching observed with rhinos.
Indeed, it’s possible that there were poachers flooding into the market at around that time for other reasons (probably connected to development and increasing demand in Asia), but after the ivory sale most of them switched to killing rhinos. That would suggest the sale was successful, provided you aren’t judging that success from the standpoint of a rhino.
A better model: Bayesian population estimation followed by Poisson regression
It’s possible to build a better model using this data, by putting the legal carcass data to proper use and then using a correctly-specified Poisson regression model on the illegal carcass data. To see how different the results might then look, consider Figure 1, taken from the Appendix of the paper, which shows the actual numbers of illegal carcasses in each year.
Figure 1: Distribution of illegal elephant kills, 2002 – 2013 (year is above its corresponding histogram)
Does it look to you like the number of elephants killed has increased? It certainly doesn’t to me. Note that between 20 and 50% of observed data are 0 kills in all years except 2002 (which the authors say was the start year of the data, and exclude from their analysis). Can you strongly conclude any change from these figures? I haven’t shown the legal kill data but it is broadly similar in scale. Certainly, if there is any upward step in illegal kills in 2008, it could potentially be explained simply by changes in populations of elephants – if even a small change in elephant density leads to an extra 1 or 2 extra kills per site per year, it would lead to distributions like those in Figure 1. To me it seems likely that the single biggest determinant of elephant kills will be the number of elephants and the number of poachers. If we assume the number of poachers (or the pace of their activity) changed after 2008, then surely we need to consider what happened to the population of elephants overall in 2008. If it declined, then poachers might catch the same number as 2007; if it increased, they would catch more.
The best way to analyze this data is to directly adjust for the population of elephants. We can use the legal kill data to do this, assuming that it is mostly reflective of elephant population dynamics. It’s not easy, but if from published sources one can obtain some estimate of the mortality rate of wild elephants (or their life expectancy), a Bayesian model could be built to estimate total population of elephants from carcasses. This would give a credible interval for the population that could then be used as what is called an offset in a Poisson regression model that simply modeled counts of illegal kills directly against time. The advantage of this is that it uses all 0 count events, because a Poisson model allows for zeros, but it adjusts for the estimated population. I think the whole thing could be done in a single modeling process, but if not one could obtain first a distribution of the elephant population, then use this to simulate many different possible regression model coefficients for the effect of the ivory sale. In this model, the effect of the ivory sale would simply represent a direct estimate of the relative increase in mortality of elephants due to poaching.
Then, to complete the process, one would add in the rhino data and use a difference-in-difference approach to estimate the additional effect of the ivory sale on elephant mortality compared to rhinos. In this case one would find that the sale was protective for elephants, but potentially catastrophic for rhinos.
Based on looking at this data and my critical review of the model, I cannot conclude that the ivory sale led to an increase in poaching. I think CITES should continue to consider ivory sales as a tool to reduce elephant poaching, though with caution and further ongoing evaluation. In addition, based on the cited unnamed CITES spokesman, evidence from rhino culling at the time suggests the sale may even have been protective of elephants during a period of increased poaching; if so, a further big sale might actually crush the business, although there would be little benefit to this if it simply drove poachers to kill more rhinos.
With regard to the poor model design here, it shows a lot of what I have come to expect from economics research: poor definition of an outcome variable that seems intuitive but is mathematically useless (in health economics, the incremental cost effectiveness ratio shows a similar set of problems); over-reliance on OLS models when they are clearly inappropriate; poor model specification and covariate adjustment; and unwillingness to use Poisson or survival models when they are clearly most suited to the data.
I think there is lots of evidence that legal markets don’t necessary protect animals from over-exploitation (exhibit A, the fishing industry), but it is also obviously possible that economic levers of supply and demand could be used to kill an illegal industry. I suspect that more effective, sustainable solutions to the poaching problem will involve proper enforcement of sales bans in China and Japan, development in the regions where poaching happens, and better monitoring and implementation of anti-poaching measures. If market-crushing strategies like the 2008 ivory sale are going to be deployed, development is needed to offer affected communities an opportunity to move into other industries. But I certainly don’t think on the evidence presented here that such market-crushing strategies would have the exact opposite of the intended effect, and I hope this poor quality, non-peer-reviewed article in the NBER doesn’t discourage CITES from deploying a potentially effective strategy to stop an industry that is destroying a majestic and beautiful wild animal.