LibGuides: How To Conduct A Systematic Review: Appraisal

Appraise abstracts

Searches in databases result in citations and abstracts. Most systematic review preparation information suggest that authors begin by appraising abstracts only, not the full text of the item, as it can be time-consuming and costly to obtain the full text of every item retrieved in a search.

All reviewers should have access to the inclusion and exclusion criteria created earlier in the process.
Remove duplicate abstracts from your search results. You may find the same citation in two or more databases.
Each abstract should be reviewed by at least 2 reviewers, independently.
If an abstract meets the inclusion criteria, mark it as INCLUDED. If it does not meet the criteria, mark it as EXCLUDED. If you are not sure, mark it as MAYBE.
Not all abstracts contain all the needed information to make a decision. If you cannot determine if the abstract meets the criteria, mark it as MAYBE.
Any citations that do not include an abstract should be marked as MAYBE.

After each abstract has been reviewed by 2 people, the whole group of authors should examine the reviewers' marks.

Abstracts where both reviewers agreed to include the abstract should be included in the next step.
Abstracts that both reviewers marked as Maybe should be included in the next step.
Abstracts where both reviewers agreed to exclude the abstract should be excluded from the next step.
If there is disagreement, the abstract should be read by a third reviewer. All three reviewers should then discuss the abstract as it relates to the inclusion/exclusion criteria to make the final decision.

Appraise full text

After appraising abstracts, your team will have a set of abstracts they wish to include in the review and a set of citations without abstracts which may be included in the review. You must obtain the full items for all these citations.

Two authors should independently read each item to determine if they meet the inclusion/exclusion criteria. The reviewer opinions should be examined by the entire team, and disagreements in inclusion/exclusion should be discussed by the entire team. You can use similar processes as you did for examining abstracts to record the reviewers' opinions of the full-text items. All co-authors need to have access to the full text of each item.

Assess methodologies

In addition to using the inclusion/exclusion criteria previously established, you can examine the quality of studies and data analyses.

The following list provides worksheets for critical appraisal of different study types:

AMSTAR Checklist
AMSTAR is an instrument used in assessing the methodological quality of systematic reviews
Best Bets Critical Appraisal Worksheets
Assessing the Risk of Bias in Systematic Reviews of Health Care Interventions
Critical Appraisal Skills Programme
Evidence-Based Medicine Toolbox

After you have completed this step, you will have identified all the acceptable articles and other items that meet your criteria. These items should now be moved onto the next step, data extraction and synthesis.

Appraisal Software Available For Purchase

Covidence
Helpful for screening through data abstraction. Streamlines evidence synthesis.
Distiller-SR
Free student program available & discounted faculty program.
Helps with duplicate removal, screening references with AI capabilities, helps with data abstraction and gives you full control of your data.
RevMan (Reviews Manager)
Cochrane software for preparing protocols and maintaining reviews. Free for academic use, license must be purchased for commercial use.
Rayyan
A web and mobile app, that helps expedite the initial screening of abstracts and titles.
Open Science Framework
A scholarly commons to connect the entire research cycle
Zenodo
Built and developed by researchers, to ensure that everyone can join in Open Science.
WebPlotDigitizer
A computer vision assisted software that helps extract numerical data from images of a variety of data visualizations.
JBI SUMARI
Facilitates the entire systematic review process from protocol to report, and includes team and contributor management for effective and efficient collaboration.
figshare
A repository where users can make all of their research outputs available in a citable, shareable and discoverable manner

Citation Tools

In the process of conducting your review, you will build a large collection of original research articles that you will need to keep organized and eventually cite in your completed project.

Citation management software enables you to create your own personal database of references, allowing you to keep track of your research and then easily incorporate these citations into your writing. These programs allow you to save, sort, and take notes on your sources. Generated citations are only as good as the input information, so always check your citations against a style manual.

Touro subscribes to RefWorks. Zotero, Mendeley and Endnote Basic are free options with limited storage.

Creating a list of all abstracts in a spreadsheet is another option. As abstracts are read, reviewers add notes into cells on the spreadsheet indicating whether the items represented by each abstract should be included or excluded.

Which citation manager should I use?

RefWorks
An institutional account is available for free to anyone with a Touro email. It includes unlimited document storage and sharing.
Mendeley
Free citation manager and academic social network.
Zotero
Free, easy-to-use tool to help you collect, organize, cite, and share your research sources.

What is PRISMA?

PRISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. PRISMA primarily focuses on the reporting of reviews evaluating the effects of interventions, but can also be used as a basis for reporting systematic reviews with other objectives (e.g. evaluating aetiology, prevalence, diagnosis or prognosis).

Who should use PRISMA?

Authors - PRISMA aims to help authors improve the reporting of systematic reviews and meta-analyses.
Journal Peer reviewers and editors - PRISMA may also be useful for critical appraisal of published systematic reviews, although it is not a quality assessment instrument to gauge the quality of a systematic review.

Reporting Standards

There are a number of reporting standards for systematic reviews. These can serve as guidelines for protocol and manuscript preparation and journals may require that these standards are followed for systematic reviews.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses.

PRISMA-P Standards

An updated version of the original PRISMA standards for protocol development.

PRISMA-IPD Standards

Extension of the original PRISMA standards for systematic reviews and meta-analyses of individual participant data.

EQUATOR Network

The EQUATOR (Enhancing the Quality and Transparency Of health Research) Network is an international initiative that seeks to improve the reliability and value of published health research literature by promoting transparent and accurate reporting and wider use of robust reporting guidelines. They provide a list of various standards for reporting in systematic reviews.

PRISMA Checklist

PRISMA Flow Diagram

The PRISMA flow diagram depicts the flow of information through the different phases of a systematic review. It maps out the number of records identified, included and excluded, and the reasons for exclusions. Most systematic reviews include a PRISMA flow diagram to track the search, screening and selection process. See below for resources to help you generate your own PRISMA flow diagram.

PRISMA Flow Diagram in PDF or Word