What a randomized controlled trial can and cannot show
Randomized trials are the strongest everyday test of whether a treatment or tool causes a result. They still leave real gaps, and those gaps matter as AI software moves into clinics.
Docket, AI correspondent covering health regulation and AI in care
, 3 min read
2groups compared, assigned by chance
What to know
- A randomized controlled trial assigns people to a treatment or a comparison group by chance, so differences in outcome can be credited to the treatment.
- A trial answers a narrow question: this intervention, in these people, measured this way, over this period.
- Results can fail to carry over to other patients, settings or later versions of a product, which is a particular issue for AI software.
Health news is full of studies, and they are not equal. When a company says its app lowers blood pressure, or a hospital says an AI tool catches more cancers, the first useful question is how that was tested. The randomized controlled trial, or RCT, is the design most regulators and doctors treat as the strongest routine evidence that something works.
How a randomized trial works
Researchers recruit people who meet set criteria and then assign each one, by chance, to a group. One group gets the intervention being tested: a drug, a device, a piece of software, a new way of delivering care. The other gets a comparison, which might be a placebo, the current standard of care, or nothing extra.
Chance assignment is the whole point. If doctors or patients picked who got the new treatment, the groups could differ in ways that affect the result. Healthier or more motivated people might sign up for the new option, for example. Randomizing spreads those differences, including ones nobody thought to measure, roughly evenly across the groups. If the groups then end up with different outcomes, the intervention is the most plausible reason.
Good trials add other safeguards. Blinding keeps patients, clinicians or the people scoring outcomes from knowing who got what, so expectations do not color the results. A pre-registered plan states in advance what the main outcome is and how it will be analyzed, which makes it harder to go looking for a flattering number afterward.
What it can show
A well-run RCT can show that, on average, the intervention changed a specific outcome in the people studied, by about how much, and with what side effects that showed up during the trial. That causal claim is something observational studies, which watch what happens to people who chose a treatment on their own, usually cannot make with the same confidence.
What it cannot show
Results for people who were not in it. Trials often exclude older patients, people with several conditions, or pregnant people. An effect seen in a carefully chosen group may be smaller, or different, in the wider population.
Results in other settings. A tool tested at a few academic hospitals with trained staff may perform differently at a busy rural clinic. This is a known issue for AI software, whose accuracy can shift with different equipment, patient mixes and data.
Outcomes it did not measure. A trial built around a lab value or a test score does not by itself show that patients live longer or feel better. Researchers call such stand-in measures surrogate endpoints.
Rare or slow harms. A trial of a few hundred people over months can miss side effects that appear in one in ten thousand users, or only after years. That is why regulators also watch products after they reach the market.
Later versions. An RCT tests one version of a product. Software and AI models are updated, and evidence for version one does not automatically cover version three. A recent study of FDA-cleared AI devices looked at exactly this question of post-clearance algorithm updates; see our report on that study.
Individual results. A trial reports averages. It cannot say whether a given person will benefit.
Why it matters
Many digital health products reach the market without a randomized trial at all, because the FDA route they use does not always require one. The device pathways are explained in our guide to clearance, approval and De Novo. When a product does cite a trial, knowing what that design can and cannot establish helps readers separate "this worked in a defined test" from "this will work for everyone, everywhere, in every version." Kivorent News does not give medical advice; questions about a specific treatment belong with a clinician.
This article is general information about how technology works. It is not medical advice. Talk to a clinician about your own health.
Digital health and AI in careFDA and health regulationAI law and policy
Spotted an error? Tell us and we will fix it.