Skip to Main Content

Systematic Reviews

Grey literature


What is grey literature?

For a fully comprehensive systematic review search you need to include grey literature.  

"There are many definitions of grey literature, but it is usually understood to mean literature that is not formally published in sources such as books or journal articles."
Cochrane Handbook 6.2.1.8

 

Grey literature can cover clinical trials registers (for unpublished trial data), conference material, theses, patents, datasets, statistics, government and local government papers, charity websites.


Clinical Trials registers 

US Clinicaltrials.gov

ISRCTN registry 
Primary clinical trail registry recognised by WHO and ICMJE.

National Institute for Health Research (NIHR) list of studies recruiting in the UK
 

Conference proceedings 

Conference proceedings can be found in a number of databases that can be accessed from the library's databases list. 

  • Web of Science Core Collection - Search Conference Proceedings Index 
  • Embase
  • BIOSIS
  • CINAHL
  • PsycINFO 

 

Theses 

PhD theses and postgraduate dissertations can be useful grey literature.  

 

Other sources

Through Overton you can find grey literature from leading global universities, IGOs, NGOs, research funders, publishers and think tanks, government reports and legislation. You can also trace the academic literature used in creating the policy documents.

 

Policy Commons could be another source of useful grey literature. 

 

Our Grey Literature quick reference guide has details of many sources of grey literature. It is available with our database guides to health and biomedical databases.

Hand-searching

Hand-searching is also required in a systematic review.  It involves checking within the Table of Contents of selected key journals or searching them in full text. This will identify any relevant articles that have been missed by the database searches. You can analyse your search results to identify key journals for the review.  You will need to explain why you've hand-searched the particular journals and also indicate the year range of your search.

You should also check the references of articles that you are including in your analysis.  You could do this via Web of Science (available via the Library's databases list) using the cited references function.

 

Next step: Managing your search results

Assessing the quality of your grey literature

Grey literature can be really useful because it is often more current - it bypasses the peer review/ publishing process which means that it can be useful when looking at new or emerging areas of research. However, as it's not peer reviewed it can be of variable quality. You will need to assess the quality of this information yourself and decide if it appropriate to include in your review. Below are some useful tools for evaluating the information your are looking at. 

The Four Moves

infographic for the Four Moves. The word SIFT with symbols underneath that represent the step explained in the text below. Mike Caulfield. CC BY 4.0, via Hapgood.US

STOP – What is the source of the information? What is the reputation of the information and the source?

INVESTIGATE – What are you reading? Who produced the information? Knowing the expertise and agenda of the source is crucial to understanding the significance and trustworthiness of the information.

FIND BETTER COVERAGE – look for trusted sources to verify the information – “go out and find the best source you can on this topic, or, just as importantly, to scan multiple sources and see what the expert consensus seems to be”

TRACE – try and find the original context for the information you are looking at. Viewing it in it’s original context will help you get a sense of how accurately the information it being presented.

Full details of The Four Moves can be found here.  

CRAAP Test

Developed by staff from Meriam Library at California State University, the CRAAP Test looks at five criteria and suggests questions to ask yourself about of the criteria to help you establish the value of the work. Follow the link to see the full details for each of the criteria. 

Currency

  • When was the information published? Has the information been revised? 

Relevance

  • Is the information relevant to you? Who is the intended audience? Is the information at an appropriate level? Have you looked at a variety of sources before determining this is one you will use? 
  • Would you be comfortable citing this source in your research paper?

Authority

  • Who is the author/publisher/source/sponsor?
  • What are the author's credentials or organizational affiliations? Is the author qualified to write on the topic? Is there contact information, such as a publisher or email address?
  • Does the URL reveal anything about the author or source? examples: .com .edu .gov .org .net

Accuracy

  • Where does the information come from? Has the information been reviewed or refereed? 
  • Is the information supported by evidence? Can you verify any of the information in another source or from personal knowledge?
  • Does the language or tone seem unbiased and free of emotion? Are there spelling, grammar or typographical errors?

Purpose

  • Do the authors/sponsors make their intentions clear? What is the purpose of the information - inform, teach, sell, entertain or persuade? Is the information fact, opinion or propaganda?
  • Does the point of view appear objective and impartial? Are there political, ideological, cultural, religious, institutional or personal biases?

Meriam Library. CC BY 4.0, via California State University