Pragmatic Free Trial Meta
Pragmatic Free Trail Meta is an open data platform that allows research into pragmatic trials. It collects and distributes clean trial data, ratings and evaluations using PRECIS-2. This allows for diverse meta-epidemiological analyses that examine the effect of treatment across trials of different levels of pragmatism.
Background
Pragmatic studies are increasingly recognized as providing real-world evidence for clinical decision-making. The term "pragmatic" however, is not used in a consistent manner and its definition and measurement require further clarification. Pragmatic trials must be designed to guide clinical practice and policy decisions, rather than to prove a physiological or clinical hypothesis. A pragmatic trial should try to be as close as possible to actual clinical practices which include the recruitment of participants, setting, design, implementation and delivery of interventions, determination and analysis results, as well as primary analysis. This is a key distinction from explanation trials (as described by Schwartz and Lellouch1) which are designed to provide more thorough confirmation of a hypothesis.
Studies that are truly pragmatic should be careful not to blind patients or the clinicians in order to lead to distortions in estimates of the effects of treatment. The pragmatic trials also include patients from various healthcare settings to ensure that the results can be applied to the real world.
Furthermore, trials that are pragmatic must be focused on outcomes that matter to patients, such as quality of life and functional recovery. This is especially important for trials that involve surgical procedures that are invasive or may have serious adverse impacts. The CRASH trial29 compared a 2-page report with an electronic monitoring system for hospitalized patients suffering from chronic cardiac failure. The catheter trial28, however, used symptomatic catheter associated urinary tract infection as its primary outcome.
In addition to these features, pragmatic trials should minimize the requirements for data collection and trial procedures to cut costs and time commitments. Additionally pragmatic trials should try to make their findings as applicable to clinical practice as is possible by ensuring that their primary analysis is the intention-to-treat approach (as described in CONSORT extensions for pragmatic trials).
Many RCTs that do not meet the requirements for pragmatism however, they have characteristics that are in opposition to pragmatism, have been published in journals of various kinds and incorrectly labeled pragmatic. This could lead to false claims of pragmatism and the usage of the term should be made more uniform. The development of a PRECIS-2 tool that can provide an objective, standardized assessment of pragmatic features is a first step.
Methods
In a practical study it is the intention to inform policy or clinical decisions by demonstrating how an intervention could be integrated into routine care in real-world situations. Explanatory trials test hypotheses concerning the causal-effect relationship in idealized environments. In this way, pragmatic trials could have lower internal validity than explanatory studies and are more susceptible to biases in their design, analysis, and conduct. Despite these limitations, pragmatic trials may contribute valuable information to decisions in the context of healthcare.
The PRECIS-2 tool evaluates an RCT on 9 domains, ranging from 1 to 5 (very pragmatic). In this study the areas of recruitment, organisation, flexibility in delivery, flexibility in adherence, and follow-up scored high. However, the main outcome and method of missing data were scored below the practical limit. This suggests that a trial can be designed with effective practical features, yet not harming the quality of the trial.
It is difficult to determine the amount of pragmatism that is present in a trial because pragmatism does not have a single characteristic. Some aspects of a research study can be more pragmatic than others. A trial's pragmatism could be affected by modifications to the protocol or logistics during the trial. In addition 36% of 89 pragmatic trials discovered by Koppenaal and colleagues were placebo-controlled, or conducted prior to licensing, and the majority were single-center. This means that they are not quite as typical and can only be described as pragmatic in the event that their sponsors are supportive of the lack of blinding in such trials.
A common feature of pragmatic research is that researchers attempt to make their findings more meaningful by analyzing subgroups of the trial sample. This can lead to imbalanced analyses and less statistical power. This increases the chance of omitting or ignoring differences in the primary outcomes. This was a problem in the meta-analysis of pragmatic trials due to the fact that secondary outcomes were not corrected for covariates that differed at the baseline.
Furthermore, pragmatic studies may pose challenges to collection and interpretation safety data. It is because adverse events are typically self-reported, and are prone to delays, errors or coding variations. It is crucial to increase the accuracy and quality of outcomes in these trials.
Results
While the definition of pragmatism does not require that all clinical trials are 100% pragmatic There are advantages of including pragmatic elements in trials. These include:
Increased sensitivity to real-world issues, reducing study size and cost as well as allowing trial results to be more quickly implemented into clinical practice (by including patients from routine care). However, pragmatic trials may also have disadvantages. For example, the right type of heterogeneity can help a trial to generalise its findings to a variety of settings and patients. However the wrong kind of heterogeneity could reduce assay sensitivity, and thus lessen the ability of a trial to detect even minor effects of treatment.
Many studies have attempted classify pragmatic trials using a variety of definitions and scoring methods. Schwartz and Lellouch1 developed a framework to discern between explanation-based studies that prove a physiological hypothesis or clinical hypothesis and pragmatic studies that inform the selection of appropriate therapies in real world clinical practice. The framework was comprised of nine domains, each scoring on a scale ranging from 1 to 5 with 1 indicating more lucid and 5 indicating more practical. The domains were recruitment, setting, intervention delivery, flexible adherence, follow-up and primary analysis.
The original PRECIS tool3 had similar domains and scales from 1 to 5. Koppenaal et al10 created an adaptation to this assessment dubbed the Pragmascope that was easier to use in systematic reviews. They found that pragmatic systematic reviews had a higher average scores in the majority of domains but lower scores in the primary analysis domain.
The difference in the analysis domain that is primary could be due to the fact that most pragmatic trials analyse their data in an intention to treat method, whereas some explanatory trials do not. The overall score was lower for pragmatic systematic reviews when the domains on organisation, flexible delivery, and follow-up were merged.
It is crucial to keep in mind that a study that is pragmatic does not necessarily mean a low-quality study. In fact, there is an increasing number of clinical trials which use the term 'pragmatic' either in their abstract or title (as defined by MEDLINE however it is neither sensitive nor precise). The use of these terms in abstracts and titles may suggest a greater awareness of the importance of pragmatism but it is unclear whether this is manifested in the content of the articles.
additional resources
As the value of real-world evidence becomes increasingly widespread and pragmatic trials have gained momentum in research. They are randomized clinical trials that evaluate real-world alternatives to care instead of experimental treatments in development, they include patients that more closely mirror the ones who are treated in routine care, they employ comparisons that are commonplace in practice (e.g. existing drugs) and depend on participants' self-reports of outcomes. This method is able to overcome the limitations of observational research for example, the biases that come with the reliance on volunteers and the lack of the coding differences in national registry.
Pragmatic trials also have advantages, such as the ability to leverage existing data sources, and a greater likelihood of detecting meaningful differences than traditional trials. However, they may still have limitations which undermine their reliability and generalizability. For example the participation rates in certain trials might be lower than expected due to the healthy-volunteer influence and incentives to pay or compete for participants from other research studies (e.g. industry trials). Many pragmatic trials are also limited by the need to enroll participants on time. Certain pragmatic trials lack controls to ensure that any observed differences aren't caused by biases in the trial.
The authors of the Pragmatic Free Trial Meta identified RCTs published from 2022 to 2022 that self-described themselves as pragmatic. The PRECIS-2 tool was used to assess pragmatism. It covers areas such as eligibility criteria, recruitment flexibility as well as adherence to interventions and follow-up. They discovered that 14 trials scored highly pragmatic or pragmatic (i.e. scoring 5 or more) in at least one of these domains.
Studies with high pragmatism scores tend to have broader criteria for eligibility than conventional RCTs. They also have patients from a variety of hospitals. These characteristics, according to the authors, can make pragmatic trials more useful and applicable in the daily practice. However, they don't guarantee that a trial is free of bias. The pragmatism principle is not a definite characteristic; a pragmatic test that does not have all the characteristics of an explanation study could still yield valid and useful outcomes.