“Research says…”, but does it? How to tell how good a study is. Research quality is an important factor in deciding to use a study but how do you know how good a particular bit of research really is?
- “Research says…”
- Science proves nothing
- There are lots of factors that influence things
- Science is useless
- How it works
- Confirmation bias
- How to tell how good a piece of research is
- The 4 levels of evidence
- The 5 GRADE Domains
- Tool for Assessing Risk of Bias
- What we are looking for
- Our research briefings, research quality and the review panel
- Citing evidence
Lots of people like quoting research. Usually the quote is vague, something like “There is a study that proves x”. There are a number of problems with using research in this way. The first and most obvious issue here is ‘what research’? A vague reference to some study could refer to a 4th grade children’s school project, someone’s ideas written on the back of an envelope or a major international study conducted by subject matter experts and professional researchers. There is usually no way of telling what the research quality is until we know which study it actually is, how the research was conducted and by whom.
For example there is a big difference in research quality between a study that interviews 5 people in the same office and a study that observes the actual behaviour of 2,000 employees over time and not what they tell you their behaviour is.
Science proves nothing
The other issue here is the idea that research is there to ‘prove’ things. It isn’t. It is trying to get to as close to the truth about a particular topic as possible, but that is very different to ‘proving something to be true’. To say something has been ‘proved’ means that there is no doubt left, that this is the truth.
Science doesn’t work like that[i]. Life is complex. There are rarely simple causes of things and finding real causal relationships is notoriously hard. For example, ‘smoking causes lung cancer’. Firstly, not everyone who smokes will get lung cancer. There are other factors which, when combined with smoking, make the chances of developing cancer more or less likely, like excessive drinking, exercise, living in a polluted environment, having a diet of fast food and fizzy drinks etc. Some people have a greater genetic susceptibility to the carcinogens contained in smoke than others. Even then, all the smoke does is to help to create the conditions within the lungs that start changes in the person’s physiology that can then result in cancer. Additionally, there are many types of cancer.
The best we can say is that smoking significantly increases the chance of developing cancer compared to people who don’t smoke. But it’s complex and we don’t have all the answers.
There are rarely simple causes to things – life is complex
There are lots of factors that influence things
On an organisational note, saying something like ‘it has been proved that transformational leadership is better than transactional leadership’ has the same problem. There are so many variables involved that it is impossible to ‘prove’. For example, a poor transformational leader could be worse than a good transactional leader, or transformational leadership may not work in certain situations or with certain people (which appears to be the case). Notice I say, ‘appears to be the case’, not ‘has been proved’. There is always an element of doubt and, as research progresses and gets better, we start to see flaws in our research.
The problem with life, organisations, people etc. is that the number of variables or factors involved in the relationship between most things is so complex it is really hard to unravel.
Science is useless
Does that mean science is useless if it doesn’t prove things? No, not at all. Think about all the life-saving drugs that have been developed, for example. Does it mean the drugs work all the time and in every case? No, because there are so many variables, which is why most medicines come with a leaflet to tell you to stop taking them if you get certain reactions.
How it works
So, what researchers do when they are testing something is, firstly, they try to work out what factors may be involved (usually from previous research) and then they don’t test it. They form a hypothesis, something like we think that x is related to y in z direction or way. For example, higher levels of work engagement result in higher levels of productivity. That’s the hypothesis. However, the researcher will turn that statement around and test the null hypothesis. So, for example something like engagement (however that is measured) doesn’t result in higher productivity (however that is measured). If we then find that the null hypothesis can’t be accepted, we accept the hypothesis.
The reason for this is that it prevents things like confirmation bias, where people start looking for the answer they want / expect. This way, researchers are trying to break their hypothesis. If they can’t break it then the hypothesis is accepted. And this is important. It is only accepted – for now. It is always possible (and this happens all the time) that someone else comes along and either finds a flaw in your findings or research method or discovers a relationship or intervening factor you hadn’t. Science and research is dynamic and constantly changing.
The problem is that just looking at findings doesn’t tell you how good or what quality a study is. For example, there is a big difference between someone publishing a single case study based on one particular situation, say a factory or one office or even one person, and a study taking in 1,000s of people across the world and using robust and valid research and analysis methods.
99% of everything you are trying to do...
...has already been done by someone else, somewhere - and meticulously researched.
Get the latest research briefings, infographics and more from The Oxford Review - Free.
How to tell how good a piece of research is
In evidence-based practice circles, the accuracy and reliability of the research (research quality) is an important factor when deciding to include a study in the evidence-base for a decision, especially clinical or engineering decisions where people’s lives are on the line.
One way that evidence-based practitioners judge the quality of research is to use the GRADE system or framework[i]. GRADE stands for Grading of Recommendations, Assessment, Development and Evaluations, and is used extensively in medical scenarios.
GRADE is now being used in engineering, organisational evidence-based practice to judge the research and is the basis of many systematic reviews.
The 4 levels of evidence
When choosing studies to include in a decision GRADE has four levels of evidence, or four levels of certainty, in the quality of the evidence/study:
- Very low – The real effect is probably very different from the reported findings.
- Low – The true effect is quite likely to be different from the findings of this study.
- Moderate – The true effect is probably close to the findings of this study.
- High – A high level of confidence that, the findings represent the true effect / represents reality.
The 5 GRADE Domains
There are five overall factors which are used to help a practitioner work out what the level of evidence a particular study is using :
- Risk of bias
- Publication bias
All of these potential biases downgrade the research quality of the study.
Tool for Assessing Risk of Bias
The Cochrane Collaboration’s Tool for Assessing Risk of Bias usually uses a 3-point grading system for judging bias and is an important factor in working the quality of research :
- Low risk of bias
- High risk of bias
- Unclear risk of bias
Risk of bias
In 2005 The Cochrane Institute in Oxford produced a tool that has become the standard bias risk assessment tool – The Cochrane Collaboration’s Tool for Assessing Risk of Bias – and looks specifically at a range of different biases that can affect a study’s findings. The tool looks for whether the study being looked at:
- Uses quality scales. Quality scales are often inherently biased and based on opinions.
- Looks at the internal validity of the study. What this means is that the researchers extensively look for and report any potential biases the method or study might have. In other words, are the methods used in the study likely to lead to bias or less so?
- Has actively looked for sources of bias or influence in their results. Double blind trials, where neither the participants (subjects) nor the researchers know which subject is getting which treatment or is in which group are considered to be at the least risk of bias.
- That the assessor / evaluator understands the methods used and can make a good judgement about the methods used.
- Is there a risk of bias in the way the data is being presented or represented?
- Does the use or group to which you are putting the study introduce a risk of bias not associated to the study?
How precise are the data and methods used for the study?
Does the study show inconsistences openly or do they try to mask them? Are there inherent inconsistencies that haven’t been reported?
How close is the situation / population of the study being looked at to the one it is being applied to? Is this study talking directly into and about the organisation or population of the study, or is this being used in a more indirect way?
- Does the publisher have a stake in the findings? For example, is this a study by a consultancy attempting to show how good its own methods are?
All of these potential biases downgrade the quality of the research study.
What we are looking for
When assessing a study, we are looking for good research that is applicable to the situation to which the findings are being put. It needs to be valid (the methods used actually give good data and is measuring what we think it is measuring) and reliable (you keep getting the same results). Being able to spot bias in all its forms in research studies is essential if you are using research to inform organisational decisions.
Our research briefings, research quality and the review panel
At the end of all of all of our research briefings we have an assessment/review of the study being reported. We use the following simplified panel to grade the quality of the study under consideration:
Example Review panel
- Research Quality – 3/5 A good literature review and overview of the subject. Based on a meta-analysis, rather than primary research.
- Confidence – 4/5 Consistent with the current research thinking and developments.
- Usefulness – 4/5 Particularly useful to HR practitioners.
- Comments – The current focus on differentiated HR architecture (structures and systems) to support talent management is showing a lot of promise in improving organisational performance generally.
And we always fully cite which studies we have used, so that you can check it and do further reading
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ (Clinical research ed). 2008;336(7650):924-6.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. Journal of clinical epidemiology. 2011;64(4):383-94.
Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. Journal of clinical epidemiology. 2011;64(4):395-400.
Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. Journal of clinical epidemiology. 2011;64(4):401-6.
Other references used
Guyatt, G. H., Oxman, A. D., Kunz, R., Vist, G. E., Falck-Ytter, Y., & Schünemann, H. J. (2008). What is “quality of evidence” and why is it important to clinicians?. Bmj, 336(7651), 995-998.
BMJ Best practice series: What is GRADE? Accessed at: https://bestpractice.bmj.com/info/toolkit/learn-ebm/what-is-grade/ on 3rd November 2019
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., … & Jaeschke, R. (2011). GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables. Journal of clinical epidemiology, 64(4), 383-394.
Be impressively well informed
Get the very latest research intelligence briefings, video research briefings, infographics and more sent direct to you as they are published
Be the most impressively well-informed and up-to-date person around...