Pragmatic Free Trial Meta
Pragmatic Free Trial Meta is a free and non-commercial open data platform and infrastructure that supports research on pragmatic trials. It gathers and distributes clean trial data, ratings and evaluations using PRECIS-2. This allows for diverse meta-epidemiological studies to examine the effect of treatment across trials of different levels of pragmatism.
Background
Pragmatic trials provide real-world evidence that can be used to make clinical decisions. However, the use of the term "pragmatic" is inconsistent and its definition and assessment requires clarification. Pragmatic trials must be designed to inform clinical practice and policy decisions, rather than confirm an hypothesis that is based on a clinical or physiological basis. A pragmatic trial should also strive to be as close to real-world clinical practice as is possible, including its selection of participants, setting and design as well as the implementation of the intervention, determination and analysis of the outcomes, and primary analyses. This is a major distinction between explanatory trials as described by Schwartz and Lellouch1 that are designed to confirm the hypothesis in a more thorough way.
Truely pragmatic trials should not blind participants or the clinicians. This can lead to a bias in the estimates of the effects of treatment. Pragmatic trials should also seek to enroll patients from a variety of health care settings, to ensure that the results can be applied to the real world.
Finally, pragmatic trials must focus on outcomes that matter to patients, such as quality of life and functional recovery. This is particularly important in trials that require the use of invasive procedures or could have harmful adverse consequences. pragmatic kr trial29, for example was focused on functional outcomes to compare a 2-page case-report with an electronic system to monitor the health of patients admitted to hospitals with chronic heart failure. Similarly, the catheter trial28 used symptomatic catheter-associated urinary tract infections as its primary outcome.
In addition to these aspects, pragmatic trials should minimize the requirements for data collection and trial procedures to cut costs and time commitments. Additionally, pragmatic trials should aim to make their results as relevant to actual clinical practices as they can. This can be achieved by ensuring that their primary analysis is based on the intention to treat method (as described within CONSORT extensions).
Many RCTs which do not meet the criteria for pragmatism but have features that are contrary to pragmatism have been published in journals of varying types and incorrectly labeled as pragmatic. This can lead to false claims about pragmatism, and the term's use should be made more uniform. The creation of the PRECIS-2 tool, which provides an objective and standard assessment of pragmatic features is a good initial step.
Methods
In a pragmatic study the goal is to inform policy or clinical decisions by demonstrating how the intervention can be incorporated into real-world routine care. This is distinct from explanation trials that test hypotheses about the causal-effect relationship in idealized conditions. In this way, pragmatic trials can have lower internal validity than explanation studies and be more susceptible to biases in their design analysis, conduct, and design. Despite their limitations, pragmatic research can be a valuable source of information for decision-making within the context of healthcare.
The PRECIS-2 tool measures the level of pragmatism that is present in an RCT by assessing it across 9 domains, ranging from 1 (very explicative) to 5 (very pragmatic). In this study, the domains of recruitment, organisation as well as flexibility in delivery flexible adherence and follow-up were awarded high scores. However, the main outcome and the method of missing data scored below the pragmatic limit. This indicates that a trial can be designed with effective practical features, yet not compromising its quality.
However, it's difficult to assess how pragmatic a particular trial is, since pragmaticity is not a definite quality; certain aspects of a trial can be more pragmatic than others. A trial's pragmatism could be affected by modifications to the protocol or the logistics during the trial. Koppenaal and colleagues found that 36% of the 89 pragmatic studies were placebo-controlled, or conducted prior to licensing. The majority of them were single-center. Thus, they are not as common and can only be called pragmatic when their sponsors are accepting of the absence of blinding in these trials.
A common feature of pragmatic research is that researchers attempt to make their findings more meaningful by studying subgroups within the trial. This can lead to unbalanced results and lower statistical power, increasing the risk of either not detecting or misinterpreting differences in the primary outcome. This was a problem in the meta-analysis of pragmatic trials as secondary outcomes were not adjusted for covariates' differences at the time of baseline.
Furthermore, pragmatic studies may pose challenges to gathering and interpretation of safety data. It is because adverse events tend to be self-reported, and are prone to delays, errors or coding errors. Therefore, it is crucial to improve the quality of outcomes for these trials, and ideally by using national registries rather than relying on participants to report adverse events on the trial's own database.
Results
While the definition of pragmatism may not require that all trials be 100 percent pragmatic, there are some advantages to including pragmatic components in clinical trials. These include:
Increased sensitivity to real-world issues as well as reducing study size and cost and allowing the study results to be more quickly transferred into real-world clinical practice (by including patients from routine care). However, pragmatic studies can also have disadvantages. The right type of heterogeneity, like could help a study extend its findings to different patients or settings. However the wrong type of heterogeneity could reduce the assay sensitivity and thus decrease the ability of a study to detect small treatment effects.
Several studies have attempted to categorize pragmatic trials using various definitions and scoring methods. Schwartz and Lellouch1 have developed a framework that can differentiate between explanation studies that confirm a physiological hypothesis or clinical hypothesis, and pragmatic studies that guide the selection of appropriate therapies in the real-world clinical practice. The framework was composed of nine domains assessed on a scale of 1-5 with 1 being more explanatory while 5 being more pragmatic. The domains covered recruitment, setting up, delivery of intervention, flexible adherence and primary analysis.
The original PRECIS tool3 was built on the same scale and domains. Koppenaal et al10 developed an adaptation of this assessment, known as the Pragmascope that was simpler to use for systematic reviews. They discovered that pragmatic reviews scored higher across all domains, however they scored lower in the primary analysis domain.
This distinction in the main analysis domain could be due to the fact that most pragmatic trials analyze their data in an intention to treat manner however some explanation trials do not. The overall score for systematic reviews that were pragmatic was lower when the domains of organization, flexible delivery, and follow-up were merged.
It is important to understand that a pragmatic trial doesn't necessarily mean a poor quality trial, and there is an increasing rate of clinical trials (as defined by MEDLINE search, but this is not specific nor sensitive) that employ the term "pragmatic" in their abstracts or titles. These terms may signal that there is a greater understanding of pragmatism in abstracts and titles, but it's not clear whether this is reflected in content.
Conclusions
In recent years, pragmatic trials have been increasing in popularity in research because the value of real world evidence is becoming increasingly acknowledged. They are randomized studies that compare real-world alternatives to clinical trials in development. They are conducted with populations of patients that are more similar to those who receive treatment in regular medical care. This method is able to overcome the limitations of observational research, for example, the biases that are associated with the reliance on volunteers, as well as the insufficient availability and the coding differences in national registry.
Other advantages of pragmatic trials include the possibility of using existing data sources, and a higher likelihood of detecting meaningful changes than traditional trials. However, pragmatic trials may have some limitations that limit their validity and generalizability. The participation rates in certain trials could be lower than anticipated because of the healthy-volunteering effect, financial incentives or competition from other research studies. Practical trials are often limited by the need to recruit participants in a timely manner. In addition some pragmatic trials don't have controls to ensure that the observed differences are not due to biases in the conduct of trials.
The authors of the Pragmatic Free Trial Meta identified 48 RCTs that self-labeled themselves as pragmatic and that were published from 2022. The PRECIS-2 tool was used to determine the degree of pragmatism. It includes areas like eligibility criteria, recruitment flexibility as well as adherence to interventions and follow-up. They found 14 trials scored highly pragmatic or pragmatic (i.e. scoring 5 or above) in at least one of these domains.
Trials with a high pragmatism score tend to have more expansive eligibility criteria than traditional RCTs which have very specific criteria that aren't likely to be found in clinical practice, and they contain patients from a broad range of hospitals. According to the authors, could make pragmatic trials more relevant and applicable in the daily clinical. However, they don't ensure that a study is free of bias. The pragmatism characteristic is not a fixed characteristic; a pragmatic test that does not possess all the characteristics of an explanation study could still yield valuable and valid results.