Tab 3

Were the factors of interest (concepts, variables, etc.) clearly stated?

DESCRIPTION: Factors of interest may be defined as concepts, variables, target population, health care problem, etc. Factors may or may not be defined in terms of PICO(S) elements (population, intervention, comparison, outcome, setting).

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
There is no explicit statement of factors of interest, and variables cannot be inferred.	Factors of interest are not explicitly stated but may be inferred.	There is an explicit statement of the factors of interest, but factors are not clearly defined within specific contexts (population, outcomes, etc.).	There is an explicit statement of the factors of interest, and factors are clearly defined within specific contexts (population, outcomes, etc.).

Was the problem stated unambiguously and was it easy to identify?

DESCRIPTION: A problem statement outlines the areas under examination by the review. Where the statement of a preliminary search describes the action which was taken to identify the need for the integrative review, the problem statement is an explicit statement of the problem the review seeks to address. An easily identifiable problem statement may begin with a statement such as “This review addresses the problem of…”, or “Previous literature reviews highlighted problems surrounding…”, etc.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
There is no clearly stated problem.	A problem statement is alluded to but is not clearly stated.	The problem statement is clear but is not easily identifiable.	The problem statement is clear and is easily identifiable.

Were the inclusion/exclusion criteria for the eligible sources clearly stated?

DESCRIPTION: Specific eligibility criteria for inclusion and exclusion are clearly stated. Inclusion and exclusion criteria should be aligned with the purpose and aim of the review. Ideally, rationale will be provided for all limits, particularly those such as excluding non-English sources or limiting to a recent span of years.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
Inclusion/exclusion criteria for the eligible sources are not clearly stated.	Inclusion/exclusion criteria are provided but may not be clearly aligned with the review purpose. Or, rationale for eligibility criteria may not be clearly stated.	Inclusion/exclusion criteria which are clearly aligned with the review purpose are explicitly stated. Rationale for eligibility criteria is not clearly stated.	Inclusion/exclusion criteria which are clearly aligned with the review purpose are explicitly stated. Rationale for eligibility criteria is stated.

If applicable to the purpose and/or type of literature included, was the sampling frame clearly stated?

DESCRIPTION: A sampling frame for an integrative review may include (but is not limited to): the type of empirical studies, specific research design(s), inclusion of methodological or theoretical literature/framework, etc. Providing a rationale for choice of sampling frame strengthens the choice of using a sampling frame. Attention should also be paid to the alignment of the choice of sampling frame with the integrative review methodology. For example, the choice to use a sampling frame which limits the retrieved sources to only quantitative results does not align with the choice to use an integrative review methodology, which implies inclusion of varying result types (empirical [quantitative, qualitative, mixed methods], methodological, or theoretical).

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES	N/A
There is no clear statement of sampling frame nor is rationale provided. Or, the sampling frame does not align with the integrative review methodology, i.e.,limiting to only randomized controlled trials, where a systematic review would be more appropriate.	There is no clear statement of whether a sampling frame was used, nor is rationale provided; however, a sampling frame may be alluded to in the screening (inclusion/exclusion) or data analysis descriptions.	A sampling frame is clearly stated but no rationale is provided.	A sampling frame with rationale is clearly stated.	The use of a sampling frame is not applicable to the review purpose and/or type of literature included.

Were detailed descriptions of the database search processes provided?

DESCRIPTION: A detailed description of a database search will include (at a minimum): database name(s), keywords, whether subject headings were used, any applied limits (year, language), etc. A more comprehensive description will include (at a minimum): database name AND vendor (i.e. OVID MEDLINE; EBSCOhost MEDLINE with Full Text, etc.), keywords AND variations, which controlled vocabularies were used, list of all limiters WITH rationale, etc.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
There is no description of the database search process. There may be a statement such as “Two databases were searched…”, but the database names, keywords, etc. are not provided.	There is a minimal description of the database search process provided, which may include: · database name(s) OR · keywords OR · whether or not subject headings were used OR · any applied limits, etc.	The following are explicitly stated: · database name(s) AND · keywords AND · whether or not subject headings were used AND · any applied limits, etc.	The following are explicitly stated: · database name(s) AND vendor AND · keywords ANDvariations AND · which controlled vocabularies were used AND · list of all limiters WITH rationale, etc.

Are reproducible line-by-line search strategies (or a sequence of terms for simpler interfaces) provided?

DESCRIPTION: A reproducible line-by-line search strategy includes sufficient detail that a reader may recreate the search process and results in a manner that the search results are replicated. The line-by-line search strategies may be supplied as online-only supplements

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
There is no search strategy provided. There may be a statement such as “A comprehensive search was conducted …”, but keywords or terms, etc. are not provided sufficiently that a search could be replicated.	A reproducible search strategy is not explicitly described; however, there may be enough details provided that the search could possibly be replicated.	Reproducible line-by-line search strategies are provided for ONEdatabase with sufficient detail that a reader may recreate the search process.	Reproducible line-by-line search strategies are provided for ALLdatabases with sufficient detail that a reader may recreate the search processes.

Were the search results screened for relevance using a pre-specified set of eligibility criteria?

DESCRIPTION: The pre-specified eligibility criteria for inclusion/exclusion are used to screen retrieved sources. A strong review will include a statement of both title/abstract and full text relevance screening.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
Retrieved sources were not screened for inclusion/exclusion using eligibility requirements.	Insufficient detail is provided to determine whether retrieved sources are screened. Or, there is a statement of screening but no statement of eligibility criteria.	Retrieved sources are screened for inclusion/exclusion, but there is not an explicit statement that screening occurred in two phases (title/abstract and full text). Or, the set of eligibility requirements was not pre-specified.	Retrieved sources are screened for inclusion/exclusion in two phases – title/abstract and full text – using a pre-specified set of eligibility requirements.

Is the data extraction process explicit, unbiased, and reproducible?

DESCRIPTION: Like the information retrieval/search process, explicit detail should be presented about the data extraction process so that it could be reproduced, with an emphasis on minimizing error. An unbiased data extraction process would involve more than one reviewer extracting and/or verifying data and a process for resolving disagreements.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
The data extraction process is not described explicitly so that it is reproducible.	The data extraction process was completed by more than one person but is not sufficiently described so that it is reproducible.	The data extraction process was completed by only one person and is described explicitly so that it is reproducible.	The data extraction process was completed by more than one person and is described explicitly so that it is reproducible. The process for resolution of disagreements is described.

Were data analyzed using a review matrix?

DESCRIPTION: A review matrix arranges the extracted data from all sources into columns and rows for a visual representation of data and as a means of arranging data for further analysis. If using subgroups, several matrices may be presented as opposed to one single matrix.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
Data were not analyzed using a review matrix.	Data matrix is not readily available or is provided only via online supplement not accessible to reader.	Data were analyzed using a review matrix, but not all extracted data (or subgroups, if applicable) are present.	Data were analyzed using a review matrix, and all extracted data (including subgroups, if applicable) are present.

Is a data display assembling the data from retrieved sources provided?

DESCRIPTION: Examples of data displays may include (but are not limited to): matrices, graphs, charts, networks, etc.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
There is no visualization of data extracted from retrieved sources.	Data display is not readily available or is provided only via online supplement not accessible to reader.	Only partial data extracted from retrieved sources are displayed visually.	All extracted data from all subgroups (if applicable) are provided via visual data display.

Was a systematic analytic method explicitly identified?

DESCRIPTION: There is an explicit statement of the systematic analytic method (such as constant comparison, content analysis, or thematic analysis, etc.) used for data analysis.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
A systematic analytic method is not named and is not vaguely described.	Analysis is only vaguely described or may be inferred through context but is not explicitly identified by name.	A systematic analytic method process is thoroughly described but is not explicitly named.	A systematic analytic method is explicitly identified by name (constant comparison, content analysis, thematic analysis, etc.), and the process is thoroughly described.

Were commonalities and differences identified?

DESCRIPTION: Identification of commonalties and differences most commonly refers to an examination and comparison between retrieved sources but may also refer to identifying commonalties and differences between the retrieved sources and other extant materials, such as background literature or statistics.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
Commonalities and differences are not identified and cannot be inferred from the text.	Commonalities and differences are identified may be inferred from the text but are not explicitly described.	Commonalities and differences are identified and present in the text but are not explicitly described.	Commonalities and differences are identified and explicitly described.

Was conflicting evidence addressed?

DESCRIPTION: Addressing conflicting evidence differs from looking at identified differences in that an examination of conflicts seeks to determine a reason or explanation for the differences.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
Conflicting evidence is clearly present but is not addressed.	It cannot be determined if conflicting evidence is present. Or, conflicting evidence is present and discussed but not adequately.	Conflicting evidence is examined with possible explanations for differences able to be inferred but not explicitly stated.	Conflicting evidence is examined with possible explanations for differences explicitly stated.

Was the quality of retrieved sources addressed in a meaningful way?

DESCRIPTION: Quality of retrieved sources must be addressed in a meaningful way, with attention paid to aligning assessment of quality with the methodology or research design of identified sources. Ideally, there is an explicit statement of the methods and tools used to assess quality and how quality assessments were addressed (i.e. were any sources removed or interpreted differently due to quality, etc.). Thorough assessment of quality will also include an examination of risk of bias and study reliability, if applicable.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
Quality of retrieved sources was not addressed.	Quality of retrieved sources was addressed but not in a meaningful way. Alignment of tools with research designs of sources may be inadequate or may not be addressed. There may not be a statement of how sources were handled based on quality appraisal.	Quality of retrieved sources was addressed in a meaningful way. If numerous research designs were included, appropriate tools are used to assess quality. Risk of bias and study reliability were not assessed. There may not be a statement of how sources were handled based on quality appraisal.	Quality of retrieved sources was addressed in a meaningful way. If numerous research designs were included, appropriate tools are used to assess quality. Risk of bias and study reliability are also assessed. How sources were handled based on quality appraisal is stated.

Was quality verified by two independent reviewers?

DESCRIPTION: The use of two independent raters is preferable as this ensures accuracy and reduces bias.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
There is no evidence that quality was verified as part of the review.	It cannot be determined if quality was verified. Or, there is evidence that quality was verified but the number of reviewers is not stated.	There is evidence that quality was verified by one reviewer.	There is evidence that quality was verified by two reviewers. Inter-rater agreement calculations may have also been provided.

Were important elements or conclusions (for all subgroups, if applicable) synthesized into a summation of the topic or phenomenon?

DESCRIPTION: The review is concluded by explicitly synthesizing elements or conclusions into a summation of the topic or phenomenon. Merely listing identified elements is not an explicit means of synthesis. If applicable, synthesized elements or conclusions are present for all subgroups.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
Important elements or conclusions are not synthesized into a summation of the topic or phenomenon.	Important elements or conclusions are explicitly listed or discussed separately but are not clearly synthesized into a summation of the topic or phenomenon.	Important elements or conclusions are presented in a manner that is more consistent with listing than synthesis or summation of the topic or phenomenon. Or, if applicable, synthesized summations are not present for all subgroups.	Important elements or conclusions are explicitly synthesized into a summation of the topic or phenomenon. If applicable, synthesized summations are present for all subgroups.

Were methodological or other limitations of the review explicitly stated?

DESCRIPTION: Methodological limitations of the review may include (but are not limited to): only one individual for screening, quality appraisal, thematic analysis, etc.; limiting to English-language or recent years with no rationale; limited access to databases or full text resources for retrieval of sources, etc. Methodological limitations of the review do notinclude limitations of the retrieved sources, which is more appropriately discussed during quality appraisal.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
There are no methodological limitations listed.	Limitations are only mentioned regarding retrieved sources, not methodological limitations of the review itself.	Methodological limitations of the review may be inferred but are not explicitly stated. Or, methodological limitations of the review are present but are not discussed.	All methodological or other limitations of the review are explicitly stated.

Were implications discussed for: research, practice, education, or policy?

DESCRIPTION: Implications for research, practice, education, or policy are explicitly stated. Reviews may be strengthened by directly aligning implications with both the purpose and synthesized findings of the review.

NO	PARTIAL NO or UNKNOWN	PARTIAL YES	YES
Implications for research, practice, education, or policy are not explicitly stated nor may they be inferred from the text.	Implications for research, practice, education, or policy may be inferred but do not clearly align to the purpose of the review or its findings.	Implications and alignment with review purpose may be inferred but are not explicitly stated. Or, implications are stated but do not clearly align to the purpose of the review or its findings.	Implications for research, practice, education, or policy are explicitly stated and discussed, with clear alignment with purpose and findings of the review.