How wearable accuracy studies work, and how to read one
Wearable accuracy studies compare a watch, ring or band against a lab-grade reference such as an ECG, a sleep lab or a gas analyzer. The headline ranking matters less than the reference used, the error measure and who was tested.
Pulse, covering wearables and health sensors
, 3 min read
95%share of differences limits of agreement are built to cover
What to know
- Every accuracy study pairs the wearable with a reference method, and the reference sets how much the result can tell you.
- Average error, bias and limits of agreement answer different questions; a high correlation alone proves little.
- Small samples, older device generations and narrow groups of testers limit how far any result travels.
Every few months a new ranking announces which smartwatch or ring is "most accurate," and the winners shuffle depending on what got measured. That isn't a contradiction. A device can be excellent at resting heart rate overnight and mediocre at heart rate during intervals, because those are different problems for the same sensor. Reading a study well comes down to four questions: what was the device compared with, how was error counted, who wore it, and which device was it.
The reference decides the ceiling
A wearable is never checked against "the truth." It's checked against a better instrument, and each metric has its own.
Heart rate is usually compared with an ECG, either clinical electrodes or a chest strap that reads the heart's electrical signal. That matters because a wrist or finger sensor reads something else entirely: the pulse of blood under the skin, picked up by light (we explained how). Heart rate variability needs the same electrical reference, since it depends on the exact timing between beats (more on HRV).
Sleep stages are compared with polysomnography, the overnight sleep lab test that records brain waves, eye movements and muscle tone. A wearable can't see brain waves at all; it infers stages from movement, heart rate and breathing (what a sleep score measures). Calories burned are compared with indirect calorimetry, which measures oxygen in and carbon dioxide out through a mask, and VO2 max with a lab treadmill or bike test to exhaustion with the same gas analysis (how watches estimate it). Blood oxygen is compared with a clinical pulse oximeter or, in the strictest tests, arterial blood samples (how wrist SpO2 works).
A study that uses one consumer device as the yardstick for another is measuring agreement between two estimates, not accuracy.
Three numbers that answer different questions
Mean absolute percentage error (MAPE) is the average size of the miss, ignoring direction. A heart rate MAPE of 5% means readings were off by about 5% on average, sometimes high, sometimes low.
Bias is the average miss with direction kept. A device can have almost no bias, because big overestimates and big underestimates cancel, and still be wrong a lot on any given reading. That's why bias alone can flatter a device.
Limits of agreement, from a Bland-Altman analysis, are the range expected to hold 95% of the differences between the device and the reference. Narrow limits mean you can trust a single reading; wide ones mean only averages are useful.
Then there's correlation, the number to be most wary of. Two methods can correlate almost perfectly while one reads consistently 10 beats high, because correlation measures whether they rise and fall together, not whether they agree. Sleep studies add their own measure, Cohen's kappa, which scores how often the device and the lab agree on each 30-second slice of the night after stripping out agreement that would happen by chance.
Watch for failed readings, too. A blood oxygen sensor that refuses to report in a third of attempts can post a good average error on the readings it did give.
Who wore it, and which model
Validation studies are often small, a few dozen people, and frequently young, healthy and fit. Results can shift with skin tone, wrist size, age, heart rhythm problems and how much someone moves. A device tested only at rest says little about a sprint session.
Device generation matters just as much. Peer review and publication take time, so the model in a paper is often a year or two old by the time anyone reads it, and companies change their algorithms through software updates without changing the hardware. A result belongs to that sensor, that firmware and that group of testers.
Finally, check who ran it. Company-run validation isn't worthless, but it is chosen and reported by the people selling the device. Independent studies with published methods, and reviews that pool many of them, carry more weight. Even then, the honest reading is narrow: this device, measuring this thing, in people like these, did this well.
This article is general information about how technology works. It is not medical advice. Talk to a clinician about your own health.
WearablesSmart ringsHeart health techSleep tech
Written by an AI correspondent under standards set by editor Jacob Knight. Spotted an error? Tell us and we will fix it.