Is ZEO accurate?

jnk wrote:Woodworker,

I think you are asking all the right questions and pointing out all the major concerns, and I think your posts are moving the discussion along nicely.

And I'm not just saying that because I had many of the exact same concerns myself, as expressed in this old thread: viewtopic.php?f=1&t=42560&st=0&sk=t&sd=a#p375048

As Derek from Zeo said back then in that thread:

Derek@Zeo on Fri Jul 24, 2009 3:39 pm wrote:At occasional points in time, Zeo can have some difficulty in determining if you are in Wake, Light sleep or REM sleep – but the aggregate data (your total Z, time in REM, time in Deep over the course of the entire night) compares very well with the results of a PSG (the gold standard for assessing sleep).

I have no argument with his wording, myself, but I can understand why some might.

Like me, such as like for instance:

viewtopic/p749226/viewtopic.php?f=1&t=7 ... eo#p665296

deltadave wrote:

Derek@Zeo on Fri Jul 24, 2009 3:39 pm wrote: . . . aggregate. . . total Z, time in REM, time in Deep over the course of the entire night . . . compares very well with the results of a PSG . . .

. . . such as like for instance:

viewtopic/p749226/viewtopic.php?f=1&t=7 ... eo#p665296

Which of the three (as general estimates for sleep coaching) does it seem to have most trouble with in your experience--total sleep, time in REM, or time in deep? Or all of the above?

jnk wrote:

deltadave wrote:

Derek@Zeo on Fri Jul 24, 2009 3:39 pm wrote: . . . aggregate. . . total Z, time in REM, time in Deep over the course of the entire night . . . compares very well with the results of a PSG . . .

. . . such as like for instance:

viewtopic/p749226/viewtopic.php?f=1&t=7 ... eo#p665296

Which of the three does it seem to have most trouble with in your experience--total sleep, time in REM, or time in deep? Or all of the above?

How many chickens ya got so far?

I'm one big chicken. And I'm from Kentucky, so I should get a discount.

I could loan you my Zeo?

jnk,

You might say I was playing devils advocate on this matter. I wasn't trying to talk bad about the Zeo, but when someone wants advice on a somewhat new product, I try to point out the concerns I would have. Before I spend any money on a product, I would want to know the answers to my concerns. I posted what I saw as concerns that was in the video, hoping that people will take a little time to investigate the product and do their own research. While getting other peoples opinions on a product is a good idea, it shouldn't replace doing their own research to satisfy any questions or concerns. I have no interest in this item, so I haven't spent any time to do research on the product. But, I do like to put some doubt in peoples minds, to get them to do the research. I don't like seeing anyone getting ripped off by scams. And no, I'm not saying that the Zeo is a scam! But, there are a lot of products out there, that wouldn't even make a good paper weight, just waiting to take peoples money!

I guess my biggest concern is all the discussions in the forums about the Zeo, that people are trying to use the Zeo as medical information. To me, the Zeo should be used to supplement the information of the Xpap and Oximeter. They should be taking the results as a reference, not as absolute! I see people freaking out cause the Zeo said they are not getting any or very little deep sleep.

jnk wrote:And I'm from Kentucky, so I should get a discount.

I think you should keep that a secret, lest they throw us out (of F&J's). You'd do better to let them know you're from Brooklyn.

Like they couldn't figure that out in a nanosecond.

Also, keep up with the cues. The next part of the bit should have been a play off "beads and trinkets".

jnk wrote:I could loan you my Zeo?

For why? I already know what my ZQ would be:



Further:

• That value is totally out of context in that I am a short-sleeping ASPS;
• I imagine I could put in a fudge factor (http://www.bulletproofexec.com/zeo-hack/), make it 96.3 and pretty much wipe everybody out;
• Although it might be accurate (general stage-wise) (The ZEO Irony - ZQ Is Accurate If You're A Good Sleeper) what's the point? And
• If I'm a good sleeper now, what happens when I stick crap on my head?

Also, that Zeo correlation study is stoopid. ZEO staging is automatic and constant. RPSGT scoring could (and did) float. What they should have done was get a bigger (and better) group of scorers and reached consensus, thereby creating a true "Gold Standard" (note Derek's misleading use of "GS" above).

Then let's see how Zeo compares to GS.

woodworkerjunkie wrote:I see people freaking out cause the Zeo said they are not getting any or very little deep sleep.

Good point.

Zeo is accurate only if you're a good sleeper and about 25.

deltadave wrote:. . . Zeo is accurate only if you're a good sleeper and about 25.

. . . and if you wear beads and trinkets.

There, is that better?

deltadave wrote: . . . What they should have done was get a bigger (and better) group of scorers and reached consensus, thereby creating a true "Gold Standard."

Why? Are sleep studies often done that way, with a bunch of scorers reaching a consensus? If not, why compare Zeo to something that never (I would assume, or rarely) happens?

The automated scoring people run into the same kind of issues, as I understand it:

Even with the AASM manual, sleep scoring is still plagued by experts who don't agree, says [Max Hirshkowitz, PhD, DABSM, associate professor in the Department of Medicine and Psychiatry, at Baylor College]. While one expert may look at an event and see a hypopnea, another will disagree and identify the event as a respiratory effort related arousal; or where one expert identifies the event as stage 3, another will argue it doesn't have enough delta.

"So here's the dilemma: the expert opinion is the standard; but the standard is inconsistent. You're trying to make a computer agree with two people who disagree. You can't agree with both of them. In engineering, that's called the 'rubber ruler problem,' because the standard is not standard—it's rubber. It stretches this way and that and bends," says Hirshkowitz.--http://www.sleepreviewmag.com/issues/ar ... -03_06.asp

To me, as long the boys are up front that their product is not as accurate as a sleep study, I'll give them a break on the hype of their own product.

For me, the trending info from Zeo, though much less accurate than an attended PSG, could still be useful for keeping an eye on one's own sleep hygiene--in much the same way that home-PAP-machine trending info can be useful for assessing changes in night breathing, even though that collection of data, too, is not nearly as accurate as attended NPSG.

It measures what it measures and trends what it trends and is not marketed as something to be used as a mini sleep study either for diagnosing OSA or for titrating PAP.

And as for it only being accurate for a normal sleeper, I think the argument could be made that is true of ALL scoring and definitions in sleep medicine, since they are all based on standards that are based on normal sleep in normal sleepers. Maybe the argument would be wrong, but I think it could be made.

Then again, hey, what do I know? I'm just a Kentuckyorker who can't afford good moo.

jnk wrote:

deltadave wrote:. . . Zeo is accurate only if you're a good sleeper and about 25.

. . . and you if wear beads and trinkets.

There, is that better?

deltadave wrote: . . . What they should have done was get a bigger (and better) group of scorers and reached consensus, thereby creating a true "Gold Standard."

Why? Are sleep studies often done that way, with a bunch of scorers reaching a consensus? If not, why compare Zeo to something that never (I would assume, or rarely) happens?

The automated scoring people run into the same kind of issues, as I understand it:

Even with the AASM manual, sleep scoring is still plagued by experts who don't agree, says [Max Hirshkowitz, PhD, DABSM, associate professor in the Department of Medicine and Psychiatry, at Baylor College]. While one expert may look at an event and see a hypopnea, another will disagree and identify the event as a respiratory effort related arousal; or where one expert identifies the event as stage 3, another will argue it doesn't have enough delta.

"So here's the dilemma: the expert opinion is the standard; but the standard is inconsistent. You're trying to make a computer agree with two people who disagree. You can't agree with both of them. In engineering, that's called the 'rubber ruler problem,' because the standard is not standard—it's rubber. It stretches this way and that and bends," says Hirshkowitz.--http://www.sleepreviewmag.com/issues/ar ... -03_06.asp

To me, as long the boys are up front that their product is not as accurate as a sleep study, I'll give them a break on the hype of their own product.

For me, the trending info from Zeo, though much less accurate than an attended PSG, could still be useful for keeping an eye on one's own sleep hygiene--in much the same way that home-PAP-machine trending info can be useful for assessing changes in night breathing, even though that collection of data, too, is not nearly as accurate as attended NPSG.

It measures what it measures and trends what it trends and is not marketed as something to be used as a mini sleep study either for diagnosing OSA or for titrating PAP.

And as for it only being accurate for a normal sleeper, I think the argument could be made that is true of ALL scoring and definitions in sleep medicine, since they are all based on standards that are based on normal sleep in normal sleepers. Maybe the argument would be wrong, but I think it could be made.

Then again, hey, what do I know? I'm just a Kentuckyorker who can't afford good moo.

Is this your final answer?

jnk wrote:The automated scoring people run into the same kind of issues, as I understand it:

That's fine, but recognizing this, AASM Standard says you can't leave automated scoring without manual visual review (F5: If used, computer-assisted scoring of polysomnography must be reviewed and edited for accuracy.)

jnk wrote:

deltadave wrote: . . . What they should have done was get a bigger (and better) group of scorers and reached consensus, thereby creating a true "Gold Standard."

Why? Are sleep studies often done that way, with a bunch of scorers reaching a consensus? If not, why compare Ceo to something that never (I would assume, or rarely) happens?

All sleep studies are reviewed epoch-by-epoch by the board-certified sleep specialist. More important, all scorers must participate in an Interscorer Reliability Program where they are evaluated against a Gold Standard.

ASME has a Program for this:

http://www.aasmnet.org/isr/

So in answer to your question, no, not every study receives this attention, but the competency of the program does.

Consequently, demonstrating the validity of a sleep scoring device by comparing it against a GS (and not frontline scorers) is precisely in keeping with AASM Standards of Care.

So in returning to a previous point and the question posed by the original poster, if we look at the previously submitted material by Zeo:

Results: PSG and WS data were available for 29 subjects for a total of 24,138 epochs. Sleep staging (all subjects) agreements/κ were: WS-M1: 68%/0.50, WS-M2: 68%/0.48, M1-M2: 88%/0.79. Sleep staging (healthy volunteers, n=10) agreements/κ were WS-M1: 73%/0.58, WSM2: 72%/0.51, M1-M2: 89%/0.81. Sleep staging (sleep disordered subjects, n=19) agreements/κ were WS-M1: 65%/0.45, WS-M2: 66%/0.44, M1-M2: 88%/0.81. Mean LPSs (all subjects) were WS: 15±14, M1: 48±83, M2: 48±84 minutes. Mean TSTs (all subjects) were WS: 349±76, M1: 331±84 and M2 329±89 minutes. Mean SEs (all subjects) were WS: 86±17%, M1: 79±19% and M2: 79±20%.

the point should not be, "Well, Zeo is sometimes close to the scoring by a couple of guys we pulled off the street", but rather, how accurate is it?

That can simply be accomplished by comparing it against a GS by the method I described, and certainly would have made a great deal of sense.

Unless the results were 66% for Zeo vs. the GS (which would be 100%), and that certainly wouldn't look particularly appealing.

Of course what do I know? I'm just a *** from ****.

deltadave wrote: . . . Is this your final answer?

Of course not. I participate here so people, like you, can change my mind on things. You often do.

deltadave wrote: . . . AASM Standard says you can't leave automated scoring without manual visual review . . .

Of course not. Jobs is jobs. And that is actually my point. It is brave for them to compare it to anything human-reviewed, since Zeo isn't. They might do better to simply compare their product to the most respected medical technology as demonstrated by commercial automated PSG scoring of sleep stages. That would be a fairer comparison, since there is no reason to expect Zeo to exceed what the medically used automated scoring technology does. After all, Zeo is just for coaching, not providing something automated for medical people to review for medical reasons.

deltadave wrote: . . . What they should have done was get a bigger (and better) group of scorers and reached consensus, thereby creating a true "Gold Standard" . . .

I'll concede. But if you do that, you should agree to fire all human scorers (techs and docs) who score any test more than 80% off the consensus conclusion on that test, and they should all be tested annually. Of course, the problem with that is that you might have to fire a good percentage of the participants on the scoring dream team itself.

The gold standard for now is human scoring. But Zeo is not human scoring and doesn't claim to be. I don't consider it dishonest for Zeo to compare their product to average everyday human scoring, not a dream team, although I do concede the sorry state of affairs with human scoring today in backwoods Bob's Sleep Lab. I hope insurance never figures out that widespread inconsistency and doesn't start requiring scorers to prove consistency to some dream-team standard created by the insurance companies. That could get really messy in a hurry.

I see no need to hold Zeo, a non-medical device, to a higher standard than most human scorers are held to today--especially given that what the scorers do is supposed to be medical and what Zeo does is not, just to be repeatedly repetitively redundant about it.

But that's not my final answer. So you may not want to reply until I take the next half hour to edit and re-edit my messy thoughts above, which I for some reason always seem to post before I'm completely through with them. It's one of my many bad habits on the board. I'm working on it, though. So don't judge. I don't think that's mentioned in the user agreement.

woodworkerjunkie wrote: I see people freaking out cause the Zeo said they are not getting any or very little deep sleep.

Not exactly freaking out but I do tend to make mention of it since my PSG agreed with what I am seeing on my Zeo now.