Embarking on a systematic literature review is a critical step in evidence synthesis, providing a foundation for robust decision-making in the healthcare landscape. As with most things, there are ways to cut corners on systematic reviews. Sometimes there are true efficiencies to be made. Yet, all compromises have consequences.
In this article, we delve into 11 common problems that may arise when cost-cutting compromises the quality of your systematic literature review.
Jump to:
- Choosing the wrong type of systematic review methodology
- Beginning with a poorly constructed search strategy
- Missing supplementary searches
- Missing subgroups in the PICO
- Not identifying a need for considering real-world evidence
- Using real-world evidence data from the wrong market
- Using the wrong bias tools
- Missed data and opportunities for new ideas
- Not planning for updates to meet HTA timelines
- Beyond the systematic literature review – missing the bigger picture
- Unexpected surprises
1. Choosing the wrong type of systematic review methodology
Selecting the wrong type of methodology for your review can have profound implications. A robust, gold standard Cochrane methodology involves two reviewers at all stages of the review, to avoid bias in the identification and presentation of the evidence, and ensure that decision-making has a solid foundation. It also includes an evaluation of the bias present within each study. This is then incorporated into the review.
Alternative methods to a gold standard Cochrane review exist, such as targeted and rapid reviews. These aim to speed up the pace of the review or focus on a specific subset of included studies. Both will likely incur bias.
The most common biases are selection bias and reporting bias. Selection bias can result in studies being missed, whilst reporting bias delivers an incorrect focus on specific outcomes. Therefore, selecting the wrong methodology may result in a failure to address the research question adequately, leading to costly consequences. Moreover, it may introduce bias into the review, potentially skewing the findings and compromising the validity of the synthesised evidence.
Selection of the correct methodology depends on whether the review is required for external decision-making or to inform internal strategies and prioritisation. It also depends on your level of concern regarding the introduction of bias. If you are compiling evidence for a decision-making process, such as a health technology assessment (HTA) submission, you will need a full, gold standard systematic literature review. If this is not the case, and time is of the essence, a targeted review may be more appropriate.
Our experts will help you explore all the options and identify which review will achieve your objectives. Learn more about the different types of review here.
2. Beginning with a poorly constructed search strategy
A poorly constructed search strategy can set the tone for the entire review process. If the search is restricted to a single database, selection bias may occur. A search that is either too broad or too narrow may fail to address the research question effectively. Additionally, language bias may emerge, limiting the inclusivity of relevant literature.
We begin every systematic literature review project by developing a comprehensive search strategy. We seek expert advise from our in-house Information Specialist, who advises on resources to search, language and date restrictions, and search term development for the specific disease area. This strategy is reviewed by a senior member of our team, before it is presented to our client. Only then do we make a start on your systematic literature review.
3. Missing supplementary searches
Relying solely on database searches may lead to incomplete or biased results. Failure to include supplementary searches can introduce publication bias. As articles with positive or significant results are more likely to be published in journals and subsequently picked up in database searches, other results can potentially be missed. This oversight undermines the comprehensiveness of the literature review.
We ensure that our clients are always asked about the inclusion of supplementary or grey literature searches before we start. We often recommend that they include supplementary searches in sources such as conferences, clinical trial registries, HTA body websites, etc. Some of these searches may not be indexed in databases as a formal publication, so are often not retrieved by database searches. If our client is unsure, we will suggest some supplementary searches that could enhance their review.
4. Missing subgroups in the PICO
Overlooking subgroups in the PICO (Population, Intervention, Comparison, Outcome) framework can result in incomplete analyses. Failure to consider all relevant subgroups may lead to a skewed understanding of the evidence. This can impact the applicability of findings to diverse patient populations.
This is a key part of the search strategy development. We work collaboratively with our clients and with clinical Key Opinion Leaders (KOLs) in our network to identify relevant subgroups to include in the PICO. This is a particularly important consideration, as the data will be used for a meta-analysis later on, which requires a good level of heterogeneity between populations.
5. Not identifying a need for considering real-world evidence
In the ever-evolving landscape of healthcare, real-world evidence (RWE) is crucial, providing additional evidence or validation of trial date. Failing to recognise the need for RWE in the review process may limit the applicability of findings to real-world scenarios, hindering the translation of evidence into practical clinical insights.
Increasingly, real-world evidence is being recognised by HTA bodies and payers. Particularly, if evidence from clinical trials is limited, for example with rare diseases or mutation specific oncology indications. (RWE was identified as one of ISPOR’s top 10 HEOR trends for 2024-25). Again, this is a key part of our early scoping and search strategy development process at Mtech Access.
6. Using real-world evidence data from the wrong market
If the review includes RWE, sourcing data from the wrong market can be detrimental. The applicability of findings to a specific healthcare system or region may be compromised, rendering the evidence less relevant and potentially leading to misguided decision-making. If RWE is submitted to a national HTA body, they will want to see that the RWE is collected from the healthcare system of that country and not from a different country with a different health care system.
When preparing evidence for global value dossiers for use in several markets, we recommend ensuring your evidence is tailored for each market and features RWE from each local healthcare system and population. Our interactive value platform enables market-specific systematic reviews to be linked to a central global value dossier. Local teams will have access to the evidence of relevance to their market, linked to the global value proposition.
7. Using the wrong bias tools
Assessing the risk of bias is a critical aspect of a systematic review; however, cutting costs may lead to a superficial or inaccurate evaluation of bias. This can jeopardise the overall quality of the review, potentially influencing the perceived strength of the evidence.
Our experts draw on years of experience conducting reviews and assessing different risk of bias tools for a wide range of disease areas, including ultra-rare diseases, to ensure that your risk of bias assessment is detailed and accurate.
8. Missed data and opportunities for new ideas
Rushing through the review process to save costs may result in overlooking valuable data and opportunities for innovative ideas. A thorough review should not only synthesise existing evidence, but also identify gaps and potential areas for future research. Finding no data is a finding in itself, and it is important that gaps are not missed.
Our experts strategically analyse their findings and offer recommendations to support your ongoing evidence generation, reimbursement, and market access strategy.
9. Not planning for updates to meet HTA timelines
Neglecting to plan for updates to the systematic review can have implications for HTA timelines. There can be significant delays between submissions for a wide range of reasons. There may be 6 months between your submission to the National Institute for Health and Care Excellence (NICE) in England and the Scottish Medicines Consortium (SMC) in Scotland. Outdated or incomplete evidence may impact the assessment process, potentially delaying the adoption of new technologies or interventions.
We work with our clients to schedule in updates for each HTA submission and key reimbursement milestone. When updating reviews, we apply lessons from previous submission feedback, whilst adding in evidence from any new or updated publications.
10. Beyond the systematic literature review – missing the bigger picture
When systematic literature reviews are conducted by novices with little support, or by automated systematic review platforms, they are sometimes done so in a vacuum. They do not benefit from the support and experience of senior colleagues or those with other related expertise (e.g. Health Economists). This can result in a final report that misses the bigger reimbursement picture. For example, the reviewer may fail to appreciate the broader health economic implications and how the evidence will be presented alongside economic models.
Our Systematic Review team work closely with our Market Access and HTA experts, Health Economists, and Medical Writers to understand how the review fits into your broader strategy or submission. This enables our team to ensure that the review is fit for purpose and to offer suggestions and identify points of strategic value.
11. Unexpected surprises
Cutting corners in a systematic literature review can result in a host of unexpected surprises, ranging from overlooked methodological flaws to unanticipated gaps in the evidence. These surprises can manifest as inconsistencies, contradictions, or unforeseen biases, undermining the reliability of the review. By opting for a budget-driven approach, rather than a best-in-class service, the risk of encountering unexpected surprises increases substantially.
These surprises may include sudden shifts in the evidence landscape, emerging data that challenges established assumptions, or the revelation of critical studies missed during the initial review. Such surprises not only jeopardise the integrity of the synthesised evidence, but also introduce uncertainties into decision-making processes.
Our experts employ rigorous methodologies and robust quality control measures to minimise the likelihood of unexpected surprises.
We have learnt to be proactive in dealing with common challenges and to expect the unexpected. For example, we map key study characteristics to ensure appropriate prioritisation. We also look ahead and scope upcoming trials to ensure that no study due to be published imminently is missed. This proactive approach mitigates the risk of being blindsided by unforeseen challenges, ultimately leading to more informed and resilient decision-making.
If you find that your review is not fit for purpose or that you require updates for a new submission or market, it is possible to update and improve an existing review; however, the level of checks and additional work required are often as substantial as conducting a de novo review. Therefore, it is always cost effective to have the initial review completed to a high standard.
By investing in a thorough review process, organisations can navigate the complexities of the literature landscape with confidence, ensuring that decision-makers are equipped with a reliable and predictable evidence base.
The Mtech Access approach – a best-in-class systematic literature review
In the realm of systematic literature reviews, cutting costs can be a perilous but increasingly necessary decision. The consequences of such compromises can be far-reaching, affecting the quality, validity, and applicability of the synthesised evidence. Choosing a best-in-class service, such as Mtech Access, ensures a comprehensive and rigorous approach, mitigating the risks associated with cost-cutting measures in systematic literature reviews.
Our Evidence team provide a bespoke response to each and every project. We adjust our approach to meet our client’s needs. We can meet all the requirements of health technology agencies or, where appropriate, will recommend a targeted review or rapid review, or suggest other approaches to meet your budget or tight deadlines. The priorities and needs of our client stakeholders are front and centre in our approach.
Whichever review you choose, we always offer a premium service conducted by experts who actively consider your strategic objectives and wider evidence needs.
To explore your evidence synthesis requirements with our experts, email info@mtechaccess.co.uk.
