Is ZEO accurate?

jnk wrote:

deltadave wrote: . . . What they should have done was get a bigger (and better) group of scorers and reached consensus, thereby creating a true "Gold Standard" . . .

I'll concede. But if you do that, you should agree to fire all human scorers (techs and docs) who score any test more than 80% off the consensus conclusion on that test, and they should all be tested annually. Of course, the problem with that is that you might have to fire a good percentage of the participants on the scoring dream team itself.

Although the level of acceptability is defined by the individual sleep center, the generally accepted minimum Interscorer Reliability Agreement is actually 85%. Further, this evaluation has to be done at least quarterly. Failure to meet standard does not generally result in immediate dismissal-- rather, it is taken as an educational opportunity to review areas of disagreement and improve skill level.

That said, sustained inability to demonstrate competency that cannot not be corrected through action plan certainly could have an adverse effect on job security, as would (should) occur in any profession.

lazer wrote:

woodworkerjunkie wrote: I see people freaking out cause the Zeo said they are not getting any or very little deep sleep.

Not exactly freaking out but I do tend to make mention of it since my PSG agreed with what I am seeing on my Zeo now.

Given that SWS is relatively easy to identify in an automated scoring algorithm, good Zeo-NPSG correlation may very well be the rule rather than the exception.

I find Zeo works for me like an alarm system. Zeo appears very fragile when it comes to quality sleep. To get a good Zeo session of deep sleep and light sleep means I'd better have everything working well or else.

When my ASV adjustments are off (tweaking) or I eat some late night food, or I have a cold, or go to bed stressed out, the Zeo sleep stages go to hell and it reports lots of REM and awake states. This is a confirmation that something is wrong with me. I usually feel worn out the next day and remember a lot of active dreaming.

But when I have a good night - sleep soundly and have no sleep hygiene problems, the Zeo will usually show excellent sleep patterns that cycle in and out of light to deep, etc. The percentages are more balanced too. The biggest Zeo flag for me is seeing a lot of awake reports. I know I am not awake, but my REM and dreaming is so active on poor nights that Zeo scores it as awake time.

All in all, I have learned to rely on Zeo data as just one more LIMITED piece to the data puzzle - in the fog and cross word puzzle of Morpheus.

Tom

The following quote is not exactly about the Zeo (at least, I don't think it is), but it still shows that maybe things are starting to get scary with the whole automated stage-scoring stuff. The docs who wrote the following have a financial interest in what they wrote (to put it mildly), but I trust Dr. David Rapoport to have kept them mostly honest with the numbers, since he was in on collecting the data:

The level of agreement between automated sleep staging and visual scoring across all sleep stages other than N1 is approximately the same as the inter-rater agreement between scorers.--http://link.springer.com/article/10.118 ... ltext.html

The rub, very much in harmony with what Uncle Dave has posted, is in the following statement:

Decreased auto-staging accuracy corresponds with increased sleep disordered breathing severity. . . . Measurement of sleep architecture obtained from a single-channel of forehead EEG can be equivalent to between-rater agreement using conventional manual scoring. The accuracies obtained with automated sleep staging were inversely proportional to SDB severity at a rate similar to manual scorers.

Interesting.

deltadave wrote:Given that SWS is relatively easy to identify in an automated scoring algorithm, good Zeo-NPSG correlation may very well be the rule rather than the exception.

Yes, from what I see in the brain waves (Delta, theta, alpha, betas, gamma) the Zeo should do fairly well on Deep, it is substantially different. As an aside, I see different kinds of REM in myself, actually.

jnk wrote:The gold standard for now is human scoring. But Zeo is not human scoring and doesn't claim to be.

The specific Zeo claim was

Conclusion: Results for healthy volunteers were consistent with previous findings demonstrating reasonable correlation with full PSG. The device performed better on healthy volunteers than on sleep disordered subjects. The system shows promise as an easy to use wireless system for recording and scoring sleep.

jnk wrote:I see no need to hold Zeo, a non-medical device, to a higher standard than most human scorers are held to today--especially given that what the scorers do is supposed to be medical and what Zeo does is not, just to be repeatedly repetitively redundant about it.

At the risk of being repeatedly repetitively redundant again, Zeo is not being held to a higher standard. My point is that it cannot meet standard period.

If we review the previously submitted data:

Results: PSG and WS data were available for 29 subjects for a total of 24,138 epochs. Sleep staging (all subjects) agreements/κ were: WS-M1: 68%/0.50, WS-M2: 68%/0.48, M1-M2: 88%/0.79. Sleep staging (healthy volunteers, n=10) agreements/κ were WS-M1: 73%/0.58, WSM2: 72%/0.51, M1-M2: 89%/0.81. Sleep staging (sleep disordered subjects, n=19) agreements/κ were WS-M1: 65%/0.45, WS-M2: 66%/0.44, M1-M2: 88%/0.81. Mean LPSs (all subjects) were WS: 15±14, M1: 48±83, M2: 48±84 minutes. Mean TSTs (all subjects) were WS: 349±76, M1: 331±84 and M2 329±89 minutes. Mean SEs (all subjects) were WS: 86±17%, M1: 79±19% and M2: 79±20%.

the scorers (M1 and M2) disagreed on an average of 100 (99.8 ) epochs per study. Zeo (WS) compared against the scorers was consistent (WS-M1 and WS-M2 were nearly identical). So if M1 and M2 were not TFIs, it appears to be a safe bet that in the areas of disagreement there is a 50-50 split in the epochs that differ from the (presumed) GS, which now make them 95% instead of 88% scorers.

Consequently, if we now use the all-subjects data, we see that Zeo is 68% as accurate as a 95% RPSGT.

I think it would be just as accurate if we revert back to the old dsm methodology (have wife stay up all night with a flashlight, notepad and pencil).

And cheaper.

At least initially.

But what do I know...

BTW, how's that Jets thing workin' out for ya?

Jets? Never heard of them. Or the Giants. Losers.

I think Zeo compares well, for a cheap, every-night, automated-scoring gadget for sleep coaching.

I'm still not sure how the "average" tech would do at that, though:

The median rate of interscorer coincidence of sleep staging was the lowest for slow wave (deep) sleep (23.5%), followed by those for Stage 1 (59.8%), Wake (73.2%) and Stage 2 (74.2%) in this order, and rapid eye movement was the most reliably identified stage (91.3%). The median rate of interscorer coincidence for all stages was 71.8%. The present study demonstrates that scorers tend to analyze PSG data according to a relatively empirical decision as opposed to a rule-dependent decision. -- http://www.journal.med.tohoku.ac.jp/2064/TJ2064_11.pdf

Maybe it would help if they used a Zeo? At least in Japan? (I mean, if dsm's wife isn't available, or is too expensive, of course.)

Jeff, why don't you take your Zeo over to Dave's and test it? Then you guys could go for moo goo or pizza or whatever.

Only if he promises to watch his language.

Based on his center's reputation, in all seriousness, I would trust their scoring over the theoretical panel's scoring, I would think.

But that's the sort of thing Zeo could be paying to have done if they chose to.

I engage Dave on such matters because I respect his opinion. Deeply. He has experience in the lab side of things (oops, I mean, center) and I don't. At all. Other than a few stays overnight in a lab in Jersey about five years ago. Well, I do it for that reason and the entertainment value, too.

I stand by my first answer in this thread, though--that Zeo is highly accurate in comparison to nothing at all but is not so accurate in comparision to PSG, even a less-than-ideally scored PSG.

jnk wrote:Maybe it would help if they used a Zeo? At least in Japan?

The issue in Japan is that since they read R>L and D>U, interscorer reliability is skewed towards the obverse. Applying this correction factor would demonstrate compliance with standard.

While a Zeo made in USA would therefore have spectacular correlation (with uncorrected Japanese scoring), a Zeo manufactured in Japan would have the same issues as noted in the previously submitted study.

jnk wrote:I mean, if dsm's wife isn't available, or is too expensive, of course.

That's disgusting.

deltadave wrote:That's disgusting.

Hey, at least she doesn't film people asleep for a living.

Jay Aitchsee wrote:Jeff, why don't you take your Zeo over to Dave's and test it? Then you guys could go for moo goo or pizza or whatever.

jnk wrote:Only if he promises to watch his language.

How about it, Dave? You'd try wouldn't you?

Jay-Are you volunteering to buy the lab time and hire the scientists to document the study? If so, can you send me some money for the moo?

I would nominate Max for the field trip, though, if it could be financed. He likes squigglies.

jnk wrote:Jay-Are you volunteering to buy the lab time and hire the scientists to document the study? If so, can you send me some money for the moo?

I'll buy the moo for you, Dave, and Max.
Maybe we can get Max to provide the equipment - he likes experiments. You and Max can be subjects since you both have Zeo's and Dave can oversee. Sound like a plan?

Jay