Many of us scientists are convinced that scientific results are omnipotent and should have the highest priority when decision-making is taking place, especially for drafting new or revising existing policies. But in fact, this might be thinking of things too black and white.
Not all evidence is made equal
You might hear from many scientists that “Evidence is evidence” and it all should be considered. And while this is technically true, in practice things are a little bit different. While it is true that all evidence should absolutely be taken into account during a decision-making process, different pieces of evidence certainly weight differently in the specific context at hand.
The "Weight of Evidence" Approach
There is a whole field of science trying to estimate using different methods the weight if each evidence in answering a scientific question. In many cases, people use interchangeably the term weight of an evidence with the strength of the evidence compared to the strength of other pieces of evidence.
This is often misleading as the strength of an evidence actually relates to the statistical significance attached to this evidence – the more stringent the statistical analysis of the experimental data, and the higher the statistical certainty – the stronger the evidence.
In contrast, the weight of the evidence is a more complex estimate for the value of the evidence taken into account together with all other available evidence for the same question and sometimes having to take into account philosophical factors too. and this is where it gets really complicated. But often the weight of the evidence can be related to its strength.
This approach is often applied in risk assessment and risk management.
Douglas Weed has written a beautiful article on weight if evidence, entitled “Weight of Evidence: A Review of Concept and Methods“.
Expand here for an example
Let's say that we are trying to establish the effect of a supplement on it population it is intended for. For simplicity, let's also say that there are three trials which studied this effect. Trial 1 has recruited 20 subjects, age 20 to 60, and their data say with very strong statistical significance that the supplement has the intended positive effect. Trial 2 has recruited 40 college students and their data shows that the supplement has the desired effect, but their statistics are a bit less strong about it. Trial 3 has recruited 100 adolescent boys and their data show that the supplement has no effect whatsoever and their statistics are very certain of that.What do you do? Do you approve that supplement for use with the advertisement that it does have the claimed effect when one out of three studies conducted, and one with quite large study group, shows that it doesn't? You might be tempted to say "no", but this is where the weight of evidence concept might play you a trick.
Think about it - first and foremost you need to think of the application. For whom is this supplement intended? If it's target consumer are adolescent males then you are in the clear and you can probably safely refute the claim of the product. But what if the supplement was originally created for elderly people? Or for people with obesity? Does any of the studies you could find has focused on such group? Is their cohort number big enough for their result to be statistically significant? Is the supplement going to be affordable for its intended consumers? Are there any studies in any groups that show the supplement having unintended effects? All these things you ought to consider before you approve this product on your market and this is aside of lobbyists trying to present you their versions of reality and of activists demanding theirs. And often , the people deciding don't have the scientific knowledge to conduct a detailed weight of evidence analysis (they often have scientific advisors for that at least), and in most cases the conundrums are much more complex than my example.
The cost of the best solution
From purely scientific perspective, it is very hard to imagine that the cost of a solution would play a vital role in its implementation to a problem at hand, especially if the given solution is really really going to solve the problem. If there is an ultimate cure for cancer, it is unfathomable why a decision-maker might decide against fully funding it from the health security funds in favour of something else. But what if that cure is so expensive that funds need to be cut for other state-funded therapies for other diseases? Who’s to say which group of patients deserves more to be cured?
With finite resources, one might decide to opt for funding types of medication which are in demand for more people of their electorate. Another might choose to finance preferentially medicaments for children’s diseases. And unfortunately, some will decide to fund the medicaments made by an overly-friendly/strong lobbying company. And while the last thing I want to do is defend decision-makers regarding questionable practices, it is worth taking the situation in its whole complexity, and that complexity always involves the cost of the solution and its implementation.
Evidence-based policy is not only based on evidence
Which is why, when we as scientist demand evidence-based policy, we ought to be aware of the weight our evidence has in the real-world context. It is absolutely mandatory that policy is based on scientifically sound conclusions. But it is important for us, also as members of society, to remember that our scientific evidence is not the only evidence which contributes to the complexity of the problem, and thus, it cannot possibly be the only one contributing to the solution as well.
The policy-making teams
It is also important for policy-makers to remember that their understanding of scientific facts and evidence is not always optimal. And this is not an attack. We have long departed from the age when one person could know and understand all the science there is to be known and understood (check this Wikipedia entry on the science during Renaissance).
If there is criticism to be made, it ought to be constructive – when selecting advisors, policy-makers should include scientists and ones:
• with ability to weight the existing evidence;
• the understanding of the importance of the weight;
• and the skill to convey this information.
It is a tall order, but nowadays more and more scientists have left the ivory tower in the interest of society and have learned to do all these things (here’s a great piece from Scientific American). So, it’s now “only” left for the policy-makers to actually listen to their scientific advisors and take into account what they have to say from science-evidence perspective and incorporate that into the new drafts.
The policy from and for the people
The people are the ones for whom policies are drafted (ideally, most often, technically…). Thus, it only makes sense that people who are not directly involved in the policy-making machinery, nor are members of a lobbying group, should be also involved in the process of drafting policies.
Increasing number of citizen projects and movements have demanded, and in various magnitudes succeeded, to involve interested members of the publics in debates related to decision-making. One rather positive example, is the more frequent involvement of patient organizations together with the policy-makers, pharma companies and medical specialists in discussions on drugs regulations.
For the people to take an active and successful role in the legislative process though, action needs to be taken. In the EU, there are more and more occasions in which The Commission for example opens calls for public consultations on specific topics. At the time of writing of this post, there are more than 400 open calls (European Commission public consultation site). Absolutely everyone who has an opinion or information they think is valid for the topic of each open call can submit their views directly. Additionally, you can (rather you should!) write on important issues to your representatives. For the European context – it would be you MEP in the European Parliament (here’s how you can find them). You can be certain they’d never listen your voice only if you don’t speak up. Some scientific data on the acceptance of public opinions in the policy-making process also shows hopeful trends:
The bottom line is – policies absolutely should be based on scientific evidence and we as scientists and citizens ought to do our part of the job to get to that point. Staying informed about the problems at hand, seeking information on the possible solutions and their respective positive and negative sides, and trying to stay on top of the current situation to understand how to best influence decisions you care about is a demanding, but obligatory for us as involved members of society, scientists or not.
Comments