Skip to main content

How To Conduct A Systematic Review: Appraisal

Appraise abstracts

Searches in databases result in citations and abstracts.  Most systematic review preparation information (Prisma Flow Diagram, Cochrane Handbook for Systematic Reviews of Interventions) suggest that authors begin by appraising abstracts only, not the full text of the article or other item, as it can be time-consuming and costly to obtain the full text of every item retrieved in a search.

  • All reviewers should have access to the inclusion and exclusion criteria created earlier in the process.
  • Remove duplicate abstracts from your search results. You may find the same citation in two or more databases.
  • Each abstract should be reviewed by at least 2 reviewers, independently.
  • If an abstract meets the inclusion criteria and should be included, mark it as INCLUDED. If it does not meet the criteria, mark it as EXCLUDED. If you are not sure, mark it as MAYBE.
  • Not all abstracts contain all the needed information to make a decision. If you cannot determine if the abstract meets the criteria, mark it as MAYBE.
  • Any citations that do not include an abstract should be marked as MAYBE.

After each abstract has been reviewed by 2 people, the whole group of authors should examine the reviewers' marks. 

  • Abstracts where both reviewers agreed to include the abstract should be included in the next step.
  • Abstracts that both reviewers marked as Maybe should be included in the next step.
  • Abstracts where both reviewers agreed to exclude the abstract should be excluded from the next step.
  • If there is disagreement, the abstract should be read by a third reviewer. All three reviewers should then discuss the abstract as it relates to the inclusion/exclusion criteria to make the final decisions step.

Appraise full text

After appraising abstracts, your author team will have a set of abstracts they wish to include in the review and a set of citations without abstracts which may be included in the review.  You must obtain the full items for all these citations.

Two authors should independently read each item to determine if they meet the inclusion/exclusion criteria. The reviewer opinions should be examined by the entire team, and disagreements in inclusion/exclusion should be discussed by the entire group of authors. You can use similar processes as you did for examining abstracts to record the reviewers' opinions of the full-text items.  All co-authors need to have access to the full text of each item. 

Assess methodologies

In addition to using the inclusion/exclusion criteria previously established, you can examine the quality of studies and data analyses.

The following websites provide worksheets for critical appraisal of different study types:

After you have completed this step, you will have identified all the acceptable articles and other items that meet your criteria. These items should now be moved onto the next step, data extraction and synthesis.

Citation Tools

In the process of conducting your review, you will build a large collection of original research articles that you will need to keep organized and eventually cite in your completed project.

Citation management software enables you to create your own personal database of references, allowing you to keep track of your research and then easily incorporate these citations into your writing.  These programs allow you to save, sort, and take notes on your sources. Generated citations are only as good as the input information, so always check your citations against a style manual. 

The Sirota Library subscribes to RefWorks.  Zotero, Mendeley and Endnote Basic are other free options.

Creating a list of all abstracts in a spreadsheet is another option.  As abstracts are read, reviewers add notes into cells on the spreadsheet indicating whether the items represented by each abstract should be included or excluded. 

Appraisal Software Available For Purchase

Touro College of Osteopathic Medicine and College of Pharmacy
Harlem Campus 230 West 125th Street New York, NY 10027 (212) 851-1199