Melnyk & Fineout-Overholt (2015, p. 10), Step 3 = "Critically appraise the evidence".
When appraising or evaluating the evidence (information sources) it is important to pause and be critical because it is not only about a source being reliable or authoritative. The primary goal of appraising or evaluating is to determine the appropriateness and value of a source in context with your purpose.
This can be challenging! And just because you have found your evidence in a library database, this does not mean there isn't a need for you to scrutinize it carefully. So we have provided a number of ways to make this task easier. The CRAAP Test is a quick and easy way to weed out the unsuitable evidence; the other critical appraisal tools provided help you match the evidence more closely to your clinical question.
Please remember that this stage of the EBP framework also includes using the Checklist as indicated by your Faculty!
DID YOU KNOW that when you first start the research process, there is no need to read everything you find? When applying a quick CRAAP test, you usually look at only the author, title, abstract, and reference list or bibliography. This means you can discard many of the articles without wasting too much time by reading what you don't need from beginning to end!
If you are using the PRISMA flow diagram, use the CRAAP test to SCREEN your RECORDS.
When it comes to finding the best evidence for your clinical question, the CRAAP test is only the first step. Sometimes articles published in peer-reviewed journals have issues with methodology, reporting, or conclusions drawn. Sometimes, the articles even contain research that is actually deliberate academic fraud.
A quick Google search of Alirio Melendez, for e.g., will show that this ex-NUS professor is reported to have made up his research in 21 published articles. Yes, 21 flawed articles! He denies the allegations, but Googling him provides some surprising insight.
The CRAAP test may not pick up on this type of fraud as you can apply the CRAAP test without even reading the article. A more in-depth critical appraisal, therefore, is a crucial part of the EBP process.
Which Checklist to use?
There are many tools available to help you critically appraise the literature you find, but there doesn't seem to be a “gold standard” - they are all just tools to help you decide on the validity of research that has been carried out by others.
Our suggestion would be to start with a general tool and then move on to the tools used for specific types of research. The general tool will also help you identify the type of research being conducted.
An example of a useful general tool...
Ingham-Broomfield, R. (2014). A nurse's guide to the critical reading of research. Australian Journal of Advanced Nursing, 32(1), 37-44. https://www.ajan.com.au/archive/Vol32/Issue1/5Ingham-Broomfield.pdf
Read these articles if you need help with your choice of the more targeted critical appraisal tools...
Buccheri, R. K., & Sharifi, C. (2017). Critical appraisal tools and reporting guidelines for evidence‐based practice. Worldviews on Evidence‐Based Nursing, 14(6), 463-472. http://dx.doi.org/10.1111/wvn.12258
Majid, U., & Vanstone, M. (2018). Appraising qualitative research for evidence syntheses: A compendium of quality appraisal tools. Qualitative Health Research, 28(13), 2115–2131. https://doi.org/10.1177 1049732318785358
Please remember, when in doubt, ask your supervisor or teacher.
A List of Appraisal Tools
PRISMA stands for Preferred Reporting Items for Systematic Reviews and Meta-Analyses, and was developed to help researchers conduct and then describe their search strategies in academic writing.
A question commonly asked about PRISMA is when to start applying the exclusion criteria like year of publication. As one of the reasons for providing a PRISMA flow chart is to give others a way to repeat your search strategy, you can choose to apply your exclusion dates at almost any stage - as long as you state this clearly. Just remember, the sooner you exclude by date, the fewer articles you will need to screen!
Figure 1. Date exclusion at identification stage of PRISMA.
Figure 2. Adapted PRISMA 2009 Flow Diagram with dates excluded during screening stage. Reprinted from "A model explaining refugee experiences of the Australian healthcare system: A systematic review of refugee perceptions," by M. Au, A.D. Anandakumar, R. Preston, R.A. Ray, & M. Davis, 2019, BMC International Health and Human Rights, 19(1), p.4. Copyright 2019 by the Authors.
Figure 3. PRISMA 2009 Flow Diagram with dates excluded at eligibility stage. Reprinted from "Poor quality in systematic reviews on PTSD and EMDR - An examination of search methodology and reporting," by E. Opheim, P.N. Andersen, M. Jakobsen, B. Aasen, & K. Kvaal, 2019, Frontiers in Psychology, 10(1588), p. 3. Copyright 2019 by the Authors.
Follow the links to see published examples of the PRISMA flow diagram and the search strategy descriptions.
Au, M., Anandakumar, A.D., Preston, R., Ray, R.A., & Davis, M. (2019). A model explaining refugee experiences of the Australian healthcare system: A systematic review of refugee perceptions. BMC International Health and Human Rights, 19(1), 1-23. https://doi.org/10.1186/s12914-019-0206-6
de Filippis, R., Carbone, E.A., Gaetano, R., Bruni, A., Pugliese, V., Segura-Garcia, C., & De Fazio, P. (2019). Machine learning techniques in a structural and functional MRI diagnostic approach in schizophrenia: A systematic review. Neuropsychiatric Disease and Treatment, 15, 1605–1627. https://doi.org/10.2147/NDT.S202418
Forte, G., Favieri, F., & Casagrande, M. (2019). Heart rate variability and cognitive function: A systematic review. Frontiers in Neuroscience, 13(710), 1-11. https://doi.org/10.3389/fnins.2019.00710
Khademi, F., Vaez, H., Momtazi-Borojeni, A.A., Majnooni, A., Banach, M., & Sahebkar, A. (2019). Bacterial infections are associated with cardiovascular disease in Iran: A meta-analysis. Archives of Medical Science, 15(4), 902–911. https://doi.org/10.5114/aoms.2019.85509
Mulugeta, H., Wagnew, F., Dessie, G., Biresaw, H., & Habtewold, T. D. (2019). Patient satisfaction with nursing care in Ethiopia: a systematic review and meta-analysis. BMC nursing, 18(27), 1-12. https://doi.org/10.1186/s12912-019-0348-9
Opheim, E., Andersen, P.N., Jakobsen, M., Aasen, B., & Kvaal, K. (2019). Poor quality in systematic reviews on PTSD and EMDR - An examination of search methodology and reporting. Frontiers in Psychology, 10(1558). 1-11. https://doi.org/10.3389/fpsyg.2019.01558
If you have used any of these checklists or tools to critically appraise studies used in your paper, it is important to show your usage of the tools and to provide proper attribution of the ideas used.
Most tools are available as online document on their respective sites and can be referenced as Websites; for published tools found in books or articles, use the relevant referencing based on the format.