Subject Guides: Evidence Based Practice: 3. APPRAISE

APPRAISE the evidence

Melnyk & Fineout-Overholt (2015, p. 10), Step 3 = "Critically appraise the evidence".

When appraising or evaluating the evidence (information sources) it is important to pause and be critical because it is not only about a source being reliable or authoritative. The primary goal of appraising or evaluating is to determine the appropriateness and value of a source in context with your purpose.

This can be challenging! And just because you have found your evidence in a library database, this does not mean there isn't a need for you to scrutinize it carefully. So we have provided a number of ways to make this task easier. The CRAAP Test is a quick and easy way to weed out the unsuitable evidence; the other critical appraisal tools provided help you match the evidence more closely to your clinical question.

Please remember that this stage of the EBP framework also includes using the Checklist as indicated by your Faculty!

The quick CRAAP test

DID YOU KNOW that when you first start the research process, there is no need to read everything you find? When applying a quick CRAAP test, you usually look at only the author, title, abstract, and reference list or bibliography. This means you can discard many of the articles without wasting too much time by reading what you don't need from beginning to end!

If you are using the PRISMA flow diagram, use the CRAAP test to SCREEN your RECORDS.

Currency

Relevance

Accuracy

Authority

Purpose

Critical Appraisal Tools

When it comes to finding the best evidence for your clinical question, the CRAAP test is only the first step. Sometimes articles published in peer-reviewed journals have issues with methodology, reporting, or conclusions drawn. Sometimes, the articles even contain research that is actually deliberate academic fraud.

A quick Google search of Alirio Melendez, for e.g., will show that this ex-NUS professor is reported to have made up his research in 21 published articles. Yes, 21 flawed articles! He denies the allegations, but Googling him provides some surprising insight.

The CRAAP test may not pick up on this type of fraud as you can apply the CRAAP test without even reading the article. A more in-depth critical appraisal, therefore, is a crucial part of the EBP process.

fraud alert

Which Checklist to use?

There are many tools available to help you critically appraise the literature you find, but there doesn't seem to be a “gold standard” - they are all just tools to help you decide on the validity of research that has been carried out by others.

Our suggestion would be to start with a general checklist or tool and then move on to the tools used for specific types of research. The general tool will also help you identify the type of research being conducted.

If you are having problems identifying the type of research or study used, the following articles may help...

Ingham-Broomfield, R. (2016). A nurse's guide to mixed methods research.
Australian Journal of Advanced Nursing, 33(4), 46-52. https://www.ajan.com.au/archive/Vol33/Issue4/5Broomfield.pdf

Ingham-Broomfield, R. (2015). A nurse's guide to qualitative research.
Australian Journal of Advanced Nursing, 32(3), 34-40. https://www.ajan.com.au/archive/Vol32/Issue3/4Broomfield.pdf

Ingham-Broomfield, R. (2014). A nurse's guide to quantitative research.
Australian Journal of Advanced Nursing, 32(2), 32-38. https://www.ajan.com.au/archive/Vol32/Issue2/4Broomfield.pdf

Read these articles if you need help with your choice of the more targeted critical appraisal tools...

Buccheri, R. K., & Sharifi, C. (2017). Critical appraisal tools and reporting guidelines for evidence‐based practice. Worldviews on Evidence‐Based Nursing, 14(6), 463-472. http://dx.doi.org/10.1111/wvn.12258

(OneSearch link)

Majid, U., & Vanstone, M. (2018). Appraising qualitative research for evidence syntheses: A compendium of quality appraisal tools. Qualitative Health Research, 28(13), 2115–2131. https://doi.org/10.1177 1049732318785358

(OneSearch link)

Please remember, when in doubt, ask your supervisor or teacher.

A List of Appraisal Tools

Critical Appraisal Skills Programme (CASP)
CASP offers critical appraisal skills training, workshops and tools. These help you read and check health research for trustworthiness, results & relevance.
Joanna Briggs Institute - Critical Appraisal Tools
JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance, and results of published papers.
Mixed Methods Appraisal Tool
A presentation from the McGill University Department of Family Medicine on the MMAT.
Scottish Intercollegiate Guidelines Network (SIGN) Checklists
SIGN develops evidence based clinical practice guidelines for the National Health Service (NHS) in Scotland. The site contains a list of methodology checklists for the different study designs.
University of Glasgow - General Practice and Primary Care
These checklists are designed by UOG with adaptation from a number of sources including the JAMA series on "How to read the Medical Literature", the UK Workshops on Teaching Evidence-based Health Care and the Critical Appraisal Skills Programme (CASP), Oxford.

General Tools and Checklists

If you need help evaluating an article, and do not want to use an in-depth critical appraisal tool, read this article to see an example of a useful general tool...

Ingham-Broomfield, R. (2014). A nurse's guide to the critical reading of research. Australian Journal of Advanced Nursing, 32(1), 37-44. https://www.ajan.com.au/archive/Vol32/Issue1/5Ingham-Broomfield.pdf

You could also try the general checklist below.

Is the work peer reviewed?
Are there any conflicts of interest, e.g. between the authors and the institution providing the funding?
Are the research questions and objectives clearly defined?
Is the work a systematic review or meta analysis? (useful for citation mining but not normally used in your assignments).
Is the study design appropriate for the research question?
Is the sample size justified and representative of the wider population?
Do the researchers describe how they collected their data?
Does the work clearly describe the measurements used?
Did the researchers use appropriate statistical measures?
Are the research questions and objectives answered?
Did the researchers account for confounding factors that may have influenced both the dependent and independent variables?
Have the researchers drawn conclusions that can be applied beyond the groups represented in the research?
Have the authors declared any conflicts of interest?

Questions 1 to 13 Mostly YES	Questions 1 to 13 Mostly NO	Questions 1 to 13 half & half
The research can probably be used as evidence.	It may be a good idea to find another article.	You need to decide based on your own experience.

CARE Case Report Guidelines
Case reports have historically been important in (1) recognizing new or rare diseases, (2) evaluating the therapeutic effects, adverse events, and costs of interventions; and (3) improving problem-based medical education. Like CAT, it help to identify novelty areas and are cornerstone of medical progress.
CONSORT
CONsolidated Standards of Reporting Trials: An evidence-based tool to help researchers, editors and readers assess the quality of the reports of trials.
PEDro scale
A specialist checklist relating to publications from the PEDro database (physiotherapy). Considers the validity and interpret-ability of trials.
STROBE Statement
Checklists for observational studies in epidemiology

Cochrane Handbook for Systematic Reviews of Interventions
Official guide that describes in detail the process of preparing and maintaining Cochrane systematic reviews on the effects of healthcare interventions.
PRISMA-P (PRISMA for systematic review protocols)
PRISMA-P was published in 2015 aiming to facilitate the development and reporting of systematic review protocols.
ROBIS systematic review bias tool
A new tool for assessing the risk of bias in systematic reviews (rather than in primary studies). It aims at four broad categories of reviews: interventions, diagnosis, prognosis, and etiology.

PRISMA stands for Preferred Reporting Items for Systematic Reviews and Meta-Analyses, and was developed to help researchers report their search strategies in academic writing.

*** With the increasing usage of artificial intelligence (AI) in academia, we need to think about how to acknowledge the use of AI in the PRISMA flow diagram. This example was found in a preprint...

Gabashvili, I. S. (2023). Artificial intelligence in biomedicine: Systematic review. MedRxiv. https://doi.org/10.1101/2023.07.23.23292672

Apart from the Extensions for Scoping Reviews, for example, there are basically 4 PRISMA 2020 flowcharts or flow diagrams...

new systematic reviews for searching databases and registers
new systematic reviews for searching databases, registers and other sources
updated systematic reviews for searching databases and registers
updated systematic reviews for searching databases, registers and other sources

A combination of all 4 flow diagrams can be found here https://www.bmj.com/content/bmj/372/bmj.n71/F1.large.jpg

A Report is a journal article, preprint, conference abstract, study register entry, clinical study report, dissertation, unpublished manuscript, government report, or any other document providing relevant information
A Register is a resource like https://clinicaltrials.gov/ and https://covid-19.cochrane.org/ where researchers list or register their studies so others can easily find out what they are doing.

See the articles below for published examples of the PRISMA 2020 flow diagram

Mulyadi, M., Tonapa, S.I., Luneto, S., Lin, W.-T., & Lee, B.-O. (2021). Prevalence of mental health problems and sleep disturbances in nursing students during the COVID-19 pandemic: A systematic review and meta-analysis. Nurse Education in Practice, 57, 103228. https://doi.org/10.1016/j.nepr.2021.103228

Kukreti, S., Rifai, A., Padmalatha, S., Lin, C.-Y.., Yu, T., Ko, W.-C., Chen, P.-L., Strong, C., & Ko, N.-Y. (2022). Willingness to obtain COVID-19 vaccination in general population: A systematic review and meta-analysis. Journal of lGobal Health, 12, 05006. https://jogh.org/2022/jogh-11-05006

PRISMA 2020 (updated 2024)
A guide on how to use the new PRISMA 2020 flow diagram
PRISMA 2020 Explained
Very neat explanations of the PRISMA 2020 flow diagram provided the Librarians from the Jerry Falwell Library at the Liberty University in Lynchburg, VA.
PRISMA 2009 vs 2020
A summary of the major differences between the PRISMA 2009 and PRISMA 2020 flow diagram.

How to properly use the PRISMA statement
It is important to remember that the PRISMA statement does not help to design or conduct literature searches. It helps to report research.
PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews
Another guide to the PRISMA 2020 flowchart, with definitions of Records, Reports, and Studies.
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Advances in systematic review methodology made necessary the development of an updated flow diagram. This paper looks at the PRISMA 2020 Statement in more detail.
PRISMA 2020: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
The PRISMA Statement consists of a 27-item checklist and a three-phase flow diagram.
Systematic Reviews: Using PRISMA for Reporting
A very detailed guide from the UNC University Libraries, Chapel Hill, NC.