Max Dalton
Max Dalton did this work while working on a research internship with Owen Cotton-Barratt at the Global Priorities Project. It is part of our project about prioritisation under uncertainty. It interfaces with Giving What We Can in providing estimates of the cost-effectiveness of interventions in developing-world health.
Summary
This post examines the returns to funding medical research into a variety of developing world diseases. It estimates returns of 13.9 DALYs/$1000 for the sector as a whole, but finds median estimates for certain diseases that are almost ten times better than this.These calculations also exclude two known considerations which would increase the cost-effectiveness. These figures are, however, susceptible to significant uncertainty both in-model and out-of-model, meaning that they cannot be taken as a settled answer. However, the fact that these interventions show potential to be four or more times better than charities currently recommended by Giving What We Can suggests that further, more detailed research is necessary to improve our prioritisation in global health.
Introduction
Donors often give money to prevent or treat diseases in the developing world. But many of the available preventions or treatments that they choose from aren’t particularly effective, or are quite costly.
One approach to this problem is to compare treatments, to find those which are most effective for their cost, which Giving What We Can does. Another approach is to focus on developing new vaccines, controls, and treatments which may be more effective or cheaper than current treatments, thus providing more cost-effective solutions in the long term.
However, it is difficult to estimate the efficacy of this second approach, because it is uncertain how hard it is to produce new treatments, or what results increasing funding has.
But by funding only the existing treatments, we effectively ignore difficult-to-evaluate research funding, which would be a mistake if research funding were on average much more effective than funding treatments. By failing to extend our analysis to more difficult and uncertain problems we may fail to choose the best opportunities.
In this post we explain our attempt to model the cost-effectiveness of funding medical research for developing world diseases (neglected medical research). It stands alongside a spreadsheet which details our sources and calculations [1]. The investigation is on quite a general level. We find that funding research into vaccines, controls, and treatments for some diseases is plausibly four times more cost effective than the charities currently recommended by Giving What We Can, but that there is a lot of statistical and model uncertainty around this estimate, such that it should not be taken at face value. Our main conclusion is that these estimates, which are encouraging yet uncertain, show a need for more detailed and specific investigation in this area.
We begin by explaining our reasons for investigating this area rather than others. We then discuss the derivation and results of our key model, before discussing some other considerations which may improve those results further. We critically analyse the assumptions and approaches used, before concluding.
Why study medical research into specific diseases?
Owen Cotton-Barratt has built a model for estimating the cost-effectiveness of research. Although he has done some quick applications to things such as medical research, these were neither detailed nor particularly promising. We decided to apply the model in more depth, and wanted a specific problem to apply it to.
What we were looking for was a problem which is plausibly very valuable to solve, that we could get data on the problem (including indications of what the benefits would be to solving the problem, and how hard it would be to solve), and that there wouldn’t be too much research on this area already. We additionally wanted to be sure that Cotton-Barratt’s model was applicable to the problem.
From some existing work ( Owen Cotton-Barratt’s preliminary work on medical research generally, and a quick survey of returns to vaccine funding), we were aware that there was data available for some of the parameters we were interested in, that there was not much existing research on the cost-effectiveness of medical research, and that this area was susceptible to analysis using Cotton-Barratt’s model.
An additional reason for pursuing this line of research was that the analysis could be carried out by examining the impact of funding in terms of DALYs. This DALY output made it compatible with a lot of other prioritisation research, both in health prioritisation, and in the charity prioritisation carried out by GiveWell and Giving What We Can.
How did we select diseases to be studied?
We had reasons for thinking that diseases which were prevalent in developing countries would have higher burdens of disease, and lower funding for vaccine research. We were aware that much medical research is funded by private companies, who have little incentive to research diseases prevalent in less prosperous areas, because the market for any potential drug is likely to be small. So research does not always focus on those diseases which cause the greatest loss of health.
Thus, funding medical research into such high-burden but neglected diseases seemed like it might be valuable. Research into these relatively neglected, developing-world diseases filled all of our criteria for investigation.
In selecting a particular set of neglected diseases to study, we copied the set used in the survey cited above, which in turn was derived from one of the studies cited.
How can we assess the returns to neglected medical research?
Summary of approach
Research is an example of a one-off problem: we’ve never made a vaccine for malaria before, and once we’ve made a good one, we won’t need to again, because having multiple (equally effective) vaccines against the same disease isn’t very helpful.
Research is also an uncertain problem: we don’t know how difficult it is to make a vaccine for malaria until we’ve made one. This is because before making a vaccine, it is generally unclear what challenges we may face, whether they are surmountable, and how they are surmountable. Without this information, it is very difficult to predict how many resources it will take to meet all of the challenges of producing the vaccine. This is in part a result of the fact that it’s a one-off problem – we haven’t solved it before, so we don’t have any past evidence of how hard it is to solve.
Cotton-Barratt’s model for dealing with such problems of unknown difficulty thus fits the medical research situation well, and so this was a chance to apply the model in detail for the first time, as well as to investigate a promising priority area. This model is the theoretical basis for what follows. Details of the analysis will become clear as we go through this post, but our approach can be summarised now. We focused on the funding of neglected medical research. This is research into vaccines, and other control mechanisms for a wide variety of diseases prevalent in developing countries. Funding is modelled as shifting forward medical research somewhat, so that discoveries are made earlier than they otherwise would have been, and the solution (say, a vaccination for malaria) is reached sooner. Shifting forward the date of vaccination discovery also shifts forward the date of vaccination implementation, and so of eradication or near-eradication of the disease. Because the disease is eradicated sooner, fewer people suffer from it, which is a benefit, and a potentially significant one, given the spread and severity of many of the diseases we study.
In the rest of this section I will briefly discuss the sources for the empirical data we used to estimate such things as the impacts of disease, the productivity of research, and current funding. I will then explain some further empirical assumptions made as part of the model, before presenting our key findings, and noting some diseases for whom research looks more promising.
Sources of data
One of the reasons that this topic was chosen was the availability of relatively complete, recent, and good-quality data. We found data on three key empirical values in the model: burden of disease, current funding, and funding growth.
We found two complete surveys of global disease, both measuring their total impact in terms of DALYs. The Global Burden of Disease (GBD) study, from 2010, provided mean estimates and 95% confidence intervals of the DALY cost of a variety of diseases. However, on looking at the 2012 World Health Organisation (WHO) study of the same phenomena, we noticed that several of the WHO estimates were outside the 95% confidence intervals of the GBD study. Consequently, while we focused on the GBD study since it provided estimates of uncertainty, we treated their confidence intervals as as 90% confidence intervals instead of 95% intervals, to reflect our lack of confidence in these figures. It may be that there are simple explanations for the different figures which we are unaware of; if so this would increase our confidence in our final estimates.
Another key source was the G-FINDER survey of research funding for neglected diseases. This annual survey takes into account funding for all sorts of medical research (not only vaccines), from private, governmental, and charitable sources. We also based the list of charities we investigated on this study.
We used the G-FINDER reports from previous years to build up a picture of the growth in research funding for each disease for the past 5 years. We then constructed growth rates for neglected diseases as a whole over that time. We also calculated the growth in total US medical research funding over a 10 year period. When estimating the growth in funding for, say, Malaria, we weighted the growth in funding for Malaria with the neglected disease average, and the US average, to give a more moderate indication of how fast funding may grow in the future. We would have liked to have included an estimate of longer-term growth in this weighted average, but we didn’t find data on this.
Assumptions
There are many assumptions underlying this model. Here we discuss some of the most important empirical assumptions that we have made. More details are provided in this spreadsheet, whilst Cotton-Barratt’s post examines the theoretical assumptions made here.
A variety of assumptions feed into our estimate of the parameter A, which represents resources going into medical research. The G-FINDER study uses double-checks to ensure that all funding has actually been used, meaning that we can treat their figure as a reliable lower bound for funding. There may be organisations that they fail to survey, and some organisations may fail to respond, so the figures they report are likely to be downward biased, and this effect could be particularly significant proportionally for diseases which receive low levels of funding. Consequently, we specified a median higher than the G-FINDER figure: we added $1m to each figure, before multiplying reported figures by a factor of 1.06. This is an assumption, based only on the proportion of funding in the survey reported by occasional participants. Uncertainty about this assumption has not been accounted for in the model.
Another assumption about which we could find little data was the elasticity of research with funding. That is, how much the volume of research changes as the volume of funding changes. Facing a lack of empirical evidence, we believed this factor to be between 0 and 1, because we expected research output to double if all inputs (including funding as well human capital, willingness to research, etc) are doubled (i.e. research output to have constant returns to scale). We expected other inputs, especially human capital, to be very important to research productivity, and also expected marginal returns to research to be declining, so we assumed a value of 0.35, and imputed a reasonable amount of uncertainty to this estimate.
Similar reasons prompted us to count certain estimates of the resources needed to produce vaccines as underestimates. Many of these studies were based on the costs of producing vaccines which have actually been produced. However, we expect vaccines which were easier to produce to be more likely to be funded, and thus expect there to be a sample selection bias occurring here which causes costs to be underestimated. Furthermore, these figures may exclude vaccines that we’ve tried but failed to produce, causing an even more severe selection effect. We also expected such estimates to exclude the value of primary research which may be vital to producing vaccines, and so we expected people to be overoptimistic about how easy it was to produce vaccines and other control resources. Thus, we modelled all estimates for the cost of producing vaccines as under-representing costs by one third.
Also note that we neglected the costs of distributing any new treatments or vaccines generated by research. Since, by assumption, we are simply shifting these costs forward, rather than bringing about new costs, this should not be very significant. If we shift these costs forward we assume that the funders of the project forgo the interest on those costs for one year. This figure should be small, because interest will be low. We then discount this value, because we assume that the funders of control costs will not produce as much good as the most cost-effective charities. This results in a very low figure. We could not estimate this figure very well, so thought it best to flag it up in discussion than include it.
Findings
For neglected disease research as a whole, our calculations give a median impact of 13.9 DALYs/$1000, which is around one quarter less than the estimates used by Giving What We Can, of around 20 DALYs/$1000 for their top charities.
However, it is interesting to note that there is a surprising variety between diseases. For instance, returns to research into diarrhoeal disease, at 121 DALYs/$1000, are over 1000 times higher than research into leprosy, at 0.058 DALYs/$1000. These different returns are largely driven by widely divergent burdens of disease (89 million DALYs annually for diarrhoeal disease, versus 5.5 thousand DALYs for leprosy). Funding research into diseases such as malaria, hookworm, ascariasis, trichuriasis, lymphatic filariasis, meningitis, typhoid, and salmonella, as well as diarrhoeal diseases, all come out as more effective than 20 DALYs/$1000.
It must be acknowledged that these figures are highly uncertain, in two ways. First, there is model uncertainty: there are effects left out of the model in its current form. For instance, the model we use is very general, and misses out a lot of disease-specific considerations, which may strengthen or weaken these claims. The model discussed thus far also omits two factors which may substantially improve the cost-effectiveness of these research programmes. These factors are discussed below.
Second, there is a substantial amount of statistical uncertainty tracked in our model. We have estimated and tracked the variance of the different variables that feed into our estimate, and we can use this to construct rough error bounds around the estimate. It is worth noting that the construction of these error bounds relies on a number of assumptions about (in)dependence between variables, which are made in order to allow the calculation to be possible, rather than because they are likely to be exactly true. Given the warning that these bounds are themselves very rough, the 95% confidence interval for neglected diseases as a whole is 1.43-130 DALYs/$1000. This is a wide interval, indicating that there is also a significant amount of uncertainty in the model, much of it deriving from our estimates of total current funding to date, and of the difficulty of continuing with research. It is also worth noting that this interval overlaps the 20 DALYs/$1000 figure.
For these reasons, we take these figures to provide a compelling case for more detailed research, but not for any immediate change in priorities. In the rest of this post we list other considerations that might change this estimate, or change our uncertainty about the estimate. These other considerations, like the assumptions made thus far, are invitations to a more detailed analysis.
Findings by disease
It is worth discussing in more detail some of the diseases for which research looks most promising on this figure. Below we explain our estimates for the most promising diseases, and discuss their associated health cost and current treatments available for them. As will become further apparent in the remainder of these posts, these estimates are unlikely to be accurate, and miss out several important factors, but this section may provide some indication of which diseases are most promising for further investigation. If the sources for the figures below are not given here, they are given at the accompanying spreadsheet.
Diarrhoeal diseases
Diarrhoeal diseases are a group of diseases which are a leading cause of malnutrition and death for children under five. They can be prevented with sanitary measures, and with a vaccine that treats Rotavirus, and can be treated with rehydration tablets.These diseases include cholera and dysentry (Source: WHO). It is responsible for around 90 million DALYs lost each year (the highest of any disease we study), and around $127m are spent on research in this area annually. According to our analysis, research in this area may have the highest median impact, at 121 DALYs/$1000, although a rough 95% confidence interval suggests that this could vary between 12 and 1,216 DALYs/$1000.
Multiple Salmonella Infections
Salmonella is a mainly foodborne disease, resulting in fever and vomiting from which most people recover, but which cause problems in the young, ill and elderly (Source WHO). It is responsible for a relatively small burden of disease, at only 4.8 million DALYs lost, with commensurately lower spending at around $11m. We estimated a median impact of 74 DALYs/$1000 in this area, with a confidence interval of 1.3 to 4,179 DALYs/$1000.
Soil transmitted Helminth infections
Ascariasis, Hookworm, and Trichuriasis are three major types of Helminth-infection, that is, infection with worm-like organisms, which are particularly prevalent in children, and delay physical and mental development. Prevention is currently most effectively carried out by sanitary measures, and there are relatively cheap treatments available (Source, WHO research). Together they account for around 5 million DALYs lost annually, and research spending stands at around $16m. We estimated research into Ascariasis to produce a median impact of 63 DALYs/$1000 (lower bound, 6, upper bound 703), Hookworm research to produce 52 DALYs/$1000 (5, 552), and Trichuriasis research to produce 49 DALYs/$1000 (4, 638).
Partially modelled considerations
Our full model accounts for two further types of benefit. They are arrived at using the same model as that used above. The first type is plausibly large, but very uncertain, whilst the second type, is more accurately estimated and appears to be relatively small. It is because they are uncertain and indirect that we omitted them from our headline figures stated above. Regardless, they could be important drivers of the value of these interventions. Note that because they are benefits we have omitted, including them can only increase the cost-effectiveness.
Both of these types of effect arise from accounting for displaced resources. When a disease is eradicated, two sorts of funding are ended because they are no longer needed: funding on research and development to find tools to control the disease(for instance, funds to find new types of vaccines, vector controls, and deterrents), and funding on disease control, prevention and treatment (for instance, funds to buy pills, bednets, or to sanitise water supplies)
However, it seems likely that when the disease is eradicated, these funding streams don’t simply disappear: instead they are redirected to other interventions. Thus, when the eradication of a disease is brought forward, so is the redirection of these sorts of funding. This means that bringing forward the eradication of a disease brings forward the funding of other interventions, which causes more good to be done.
We now examine in detail the impacts due to these two considerations.
Research funding
Our assumptions about the level and growth of research funding carry over from the previous discussion. However, we also make some further assumptions.
First, we assume that dollars diverted away from funding a specific disease return to research, but that this research has a slightly lower value than the research that we are currently funding (we therefore multiply it by an uncertain factor between 0 and 1, median 0.6). We make this assumption for two reasons: first, the resources will be reallocated by organisations which may not focus primarily on effectiveness, and therefore might not fund the most effective sorts of research. Second, we might expect there to be declining returns to research over time, so that investment in research at the time the money is diverted may be less useful than investment in research now.
We also make assumptions about the discount rate on capital. We use as a median figure the UK Treasury rate, but the result is not very sensitive to this assumption.
Our results suggest that this effect could significantly increase the impact of funding research. Our median estimates range from $0.46-$0.48 of funding is diverted per dollar invested. This alone could mean that the effects mentioned above are twice as high. However, there is a very significant amount of uncertainty around this estimate. Because the model used here does not fit the distribution very well, it is difficult to give confidence intervals (and little weight should be put on the ‘mean’ number for this column in the spreadsheet), but the effect may be as low as $0.03, or as high as $0.80. These are evidently significant differences, and this effect will need to be better quantified in order for the figures given above to be useful. Much of this uncertainty arises from the complexity of the calculation, which involves a number of uncertain variables. If this effect could be calculated more directly, or the variables that its based on calculated more accurately, there is a hope of reducing this uncertainty.
Control funding
When we analyse control funding, instead of research funding, we have different sources for how much money is spent on control, and we make different assumptions about where the diverted money goes after diversion.
We found current control spending estimated in an IHME study, and used that as a baseline, dividing its very broad categories evenly between our more specific ones. We think that this study may not capture all spending accurately, and we think that our assumptions of even divisions between, say childhood diseases, is likely to be wrong. Thus, we impute a lot of uncertainty to our estimate of current control spending.
We also make different assumptions about how diverted control funding is spent. We assume that the diverted funding goes directly to the community that the control treatment would have gone to. Based on ballpark figures for the value of a DALY in these communities, we assume that each $1000 is as good as one DALY. This is a very rough, and very debatable assumption, but we don’t know which direction it’s wrong in. Our main conclusions are not sensitive to this number.
We found this effect to be relatively small, in the order of 5 to 17 DALYs/$1000 at the median, depending on which disease is examined. 95% confidence intervals exist around ten times higher and lower than this, on average, although there is significantly more uncertainty for specific diseases. Once again, these figures are just indicative. They show that this effect is relatively well quantified, and though likely less significant than the other two factors, it may be worth calculating in future analysis.
Criticism and limitations
There are many potential criticisms that could be made of our assumptions. Where we think that our assumptions are particularly questionable, we have tried to reflect this by associating it with a high level of variance. However, there are some assumptions for which that was not possible, and some for which we thought special note was required.
One of the most questionable assumptions for which we can’t record our uncertainty is that increasing funding now does not affect funding decisions in the future. However, it is easy to imagine that an increase in funding to a specific disease, say malaria, could cause other funders to consider this disease, when they had not done so before, thus increasing funding. We could equally imagine that increasing funding for malaria would make other donors believe that this area is receiving sufficient funds, so that marginal returns are low, and so cease to fund malaria. Thus, the assumption could fail either because our increase of funding increases funding, or because it decreases funding. Thus, we think that this assumption is a reasonable median, but it remains uncertain. We therefore make this assumption primarily because it makes the model easier to apply.
Another crucial assumption, for which we fail to track uncertainty is that shifting research funding forward shifts the eradication of diseases forward by the same amount. This may itself be decomposed into two assumptions: First, we assume that these diseases will eventually be eradicated. This may be optimistic, or even close to impossible for some of the diseases we study, particularly those where there is some transfer between animal and human populations. However, it seems like a good approximation: most of these diseases have been eradicated, or controlled at near-eradication levels, in parts of the world in which they were once endemic. It seems reasonable that with advancing technology and growing economic resources this (near-)eradication will become possible worldwide.
Second, we assume that it is research, rather than currently available control methods, which will solve this problem. This is a much more questionable assumption, particularly because there are existing, often quite cheap, treatment or prevention methods available for some of these diseases, and these have, as already mentioned, eradicated diseases in some parts of the world. For instance, the Schistosomiasis Control Initiative can treat seven neglected tropical diseases for less than a dollar [2]. Having such a cheap control method means that research into vaccines may not be necessary, since transmission may be reduced, and cases treated at low cost with currently available methods. Indeed, schistosomiasis may be effectively eradicated ten years before a vaccine could be prepared. If this is the case, then shifting forward the creation of a vaccine by two years would have almost no impact. However, under our model, shifting forward the creation of the vaccine is modelled as eradicating twice the annual global burden of disease. So it is clear that this assumption may be unrealistic. Whether it is realistic or not depends on features of the disease and current treatments available. For this, a more specific, more medically informed investigation is necessary.
Many of these problems draw from the same source. To make this analysis possible, it was made without reference to particular medical or technical aspects of disease, drawing on only a few data sets (and preferring complete data sets), and constructed using a general model (with only a few modifications made). The situation fitted quite well with the model, and this is proof that the model is tractable with relatively limited data, and without an extensive existing literature. Moreover, the model was relatively easy to apply, particularly for estimating the median values of the benefits (estimating the range of the variables is much more difficult to do accurately, and involves technicalities). Around two weeks of Max’s time and half a week of Owen’s time were spent on this work, which produced a relatively complete first-pass analysis. However, the brevity and generality of the analysis, whilst increasing tractability and simplicity, have necessarily created many of the problems discussed above. This was to be expected, and while these problems limit the analysis, they do not make us question its worth. A relatively simple method has given us an indication of the magnitudes of impacts we might expect in this area, and has provided a foundation for further analysis.
Finally, it is worth being open to the possibility that we have not caught all of our mistakes!
Conclusion
The questions raised above indicate how uncertain these estimates are. But the order of magnitude of these estimates indicates that this topic merits further investigation.
Further investigation into the specific problems raised seems to have the potential to reduce the uncertainty present in this analysis. In particular, an investigation of particular diseases, conscious of available treatment methods and specific technical problems with creating new treatments, would reduce a significant amount of the uncertainty around these estimates.
We have discussed many diseases here, and the figures calculated suggest that some have higher impacts than others. Since future work will likely need to focus on some subset of these diseases, it is worth considering which diseases are most promising for further research. Our spreadsheet gives some indication of which diseases are likely to be most promising. However, other considerations should include the cost-effectiveness of current control measures, and the likelihood of them eliminating the disease before research becomes fruitful. It is also worth considering how difficult research is for particular diseases, and conversations with experts may prove fruitful here. Another useful avenue would be to explore what opportunities exist for funding research into the high-priority areas.
This study has been fruitful in a number of ways. It has proved that our framework for analysing uncertainty is relatively easily applicable. It has shown that this neglected medical research is plausibly very high impact. And it has suggested areas for further investigation and the problems they must address. The limitations of the study are discussed extensively, but they do not look to be insurmountable with future research.
[1] The spreadsheet is hopefully comprehensible, but we’re afraid it hasn’t been optimised for readability.
[2] I do not know whether any of the empirical claims that follow do hold for schistosomiasis, and I don’t intend to suggest that they do: I use it as an example of a possible effect.