S9 flow skew bug... Anyone else seeing this?

[Nord]

Dave, this is a good idea. Just to clarify, by a "session" I meant one 24-hour period from 12:00 noon one day until 12:00 noon the next day. I believe this is how ResScan defines a session when it reports how many detailed sessions are being downloaded. This session may include both daytime naps (if any) and nighttime sleep. I was curious if everyone is only downloading one detailed data session each time. You're probably right that it's not the source of the problem, but I think every variable is worth looking at.

You are right DreamOn...
A Session is one 24 hour period from 12 noon to 12 noon as defined by ResMed.

I have been using the term "sleep" to define anytime during that 24 hour period that I turn the machine on and off again... but I think you're terms of "Sleep" and "Nap" further define when the S9 is on and off for a short or longer period and that is probably better.

I also like your idea of becoming more organized about how we collect data to solve this skew problem. Starting with asking set questions of people using S9's to get a better picture about the differences and consistency of our "hygiene".

I have taken a pretty thorough look at ResScan 3.10 as the problem, and I have tested DL's in different ways and have not been able to skew the data. I DO NOT lock the SD card for ResScan and have checked data multiple ways for errors... nada.
Corruption of the card can be attained when proper ejection or removal process is not followed.
Data "missing" can happen when first opened in graph windows at the widest time frame. It seems to be more of: "data slowing". I have got the data to return after changing time frames to shorter periods (ie. 24 hours reduced to 1 hour or shorter then back to 24 hours) in the detailed window.

I am looking at the SD cards a little. I just started using a 2 Gig Scandisk card that was newly formatted by PC and used last night for the first time.
ResMed SD 1 Gig cards (2) appear to be low quality and have flaws in the plastic. They have ragged plastic in the 3rd and 4th contact slots. I tend to believe that would not affect the skew.... but it could be an intermittent contact problem... I guess. Anyhow I have 2 Scandisk SD's that do not have the flaws that I am currently testing.

Last night I used a new Scandisk PC formatted card and my data DL did not have any skews or missing data for 1 sleep during 1 session.

Nord

[Nord]

Bump to the top...

Nord

[alterego61]

I updated the forum about card speed with some new data and analysis about how data is written and organized on the sd card. I'm a newbie here so I hope this link works:

viewtopic.php?p=471460#p471460

[Torrent]

Here's a graph of a recent flow skew issue on the S9, even though I have been power cycling the S9 daily,
it didn't avoid skew on several occasions.

It points to a write problem when recording hi res flow data on the S9 unit itself where
the flow data graph starts ahead of all the other indexes (or at least is not synchronized).

Torrent

[alterego61]

I'm still not seeing the skew bug on my S9 (only had it 5 days though). If anyone cares to PM me and then send me some skew files off their SD card, I'd be interested to take a look at the underlying data and then report back to the forum.

[Nord]

pm sent

[alterego61]

Thanks Nord...I'll take a look at the files and see what I can figure out.

[alterego61]

I took at look at Nord's files (skewed data) and compared them to mine (unskewed), using the downloaded EDF file reader tools. I haven't had much time to think about conclusions yet, but for the benefit of others working on this issue, here's what I found so far, in summary and in detail, with example data.

Background
The format of all 4 EDF data files (maintained in the DATALOG directory on the SD card) is that there is header information at the beginning of the file followed by detailed data. In the header there is a starting timestamp, and then all detailed data is recorded relative to the start-time.


Summary of Findings
In the unskewed data, the timestamp in the header of each of the 4 datafiles matched each other, and matched the timestamp in the summary data for the mask event that one sees in ResScan. The timestamp in each header was 1 second later than the time contained in the filename of each file (the filename timestamp).

In the skewed data, the timestamps in the headers of all 4 datafiles were different from each other, and did not match the timestamp of the summary data. The timestamp in the filenames of the DATALOG files were about two minutes later than time of the beginning of the mask event shown in ResScan (which is shown to the nearest minute). The flowdata header (BRP file) timestamp lagged its filename timestamp by 6 seconds, the event data (EVE file) by 11 seconds, the pressure/leak data (PLD file) by 17 seconds, and the oximetry data (SAD file) by 23 seconds.


Interpretation

I am fairly sure that these four files are created exactly at the time the mask event begins (i.e. when the Start/Stop button is pressed to switch on the blower), and given their names, including the timestamp, at that time. The header data for each file is almost certainly written at that time. From the unskewed files, it appears that normally the time written in the header is one second greater than the time written into the filename - that seems reasonable.

In the skewed files, there can be differences of many seconds between the timestamps – this is surely the cause of the graphs being skewed in ResScan. I am wondering whether the timestamps are written incorrectly on creation of the files, or whether they are updated incorrectly during the mask event. I suspect the former. What did surprise me was the big gap between the start time in the summary graph and those in the detail data. This may be something to do with the cumulative effect of these skews building up day after day.


Example Data

alterego61's files (unskewed data) from 13/14 April:

Start of mask event in ResScan summary graph: 22:53
(found by hovering over usage bargraph)

Time In DATALOG directory filenames: 22.53.50

e.g. 20100413_225350_XXX.edf ; where XXX = (BRP or EVE or PLD or SAD)

Time in header data of BRP file (Flow Data): 22.53.51
Time in header data of EVE file (Event/Annotation Data): 22.53.51
Time in header data of PLD file (Pressure/Leak Data): 22.53.51
Time in header data of SAD file (Saturation/Oximetry Data): 22.53.51


Nord's files from 12/13 April:

Start of Mask Event in ResScan Summary Graph: 23:44
(found by hovering over usage bargraph)

Time In DATALOG directory filenames: 23.46.21

e.g. 20100412_234621_XXX.edf ; where XXX = (BRP or EVE or PLD or SAD)

Time in header data of BRP file (Flow Data): 23.46.27
Time in header data of EVE file (Event/Annotation Data): 23.46.32
Time in header data of PLD file (Pressure/Leak Data): 23.46.38
Time in header data of SAD file (Saturation/Oximetry Data): 23.46.44

[dave21]

Here's a list of the differences I'm seeing on my graphs over the last 10 nights. This is looking at the detailed graphs at 30 seconds in ResScan.

14th April
Start - Flow starts 9 seconds early compared to other graphs
Finish - Flow ends 7 seconds early compared to other graphs

13th April
Start - Flow starts 2 seconds early compared to other graphs
Finish - Flow extends 58 seconds from all other graphs

12th April
Start - Flow starts 4 seconds early compared to other graphs
Mid - Events skewed 4-8 secnods
Finish - Flow ends 2 seconds early compared to other graphs

11th April
Start - Flow starts 7 seconds early compared to other graphs
Finish - Flow extends 54 seconds from all other graphs

10th April
Start - Flow starts 3 seconds early compared to other graphs
Mid - Events skewed 2 seconds early vs Flow
Finish - Flow ends 1 second early compared to other graphs

09th April
Start - Flow starts 4 seconds early compared to other graphs
Mid - Events skewed varying between 4 seconds later than Flow and then 2 seconds earlier than Flow
Finish - Flow ends 2 seconds early compared to other graphs

08th April
Start - Flow starts 3 seconds early compared to other graphs
Mid - Events skewed 3 seconds earlier than Flow
Finish - Flow ends 1 second early compared to other graphs

07th April
Start - Flow starts 3 seconds early compared to other graphs
Finish - Flow ends 1 second early compared to other graphs

06th April
Start - Flow starts 4 seconds early compared to other graphs
Finish - Flow ends 2 seconds early compared to other graphs

05th April - multi-mask event
Start - Flow starts 2 seconds early compared to other graphs
Finish - Flow ends on time compared to other graphs
---
Start - Flow starts 3 seconds early compared to other graphs
Finish - Flow ends 2 seconds early compared to other graphs


Notes
- Other graphs = Flow is X seconds earlier/late than Minute, Flow Limitation, Leak, Snore Index and Pressure
- Flow extends means that it extends X seconds past the Minute, Flow Limitation, Leak, Snore Index and Pressure
- No loss of data in any block
- Some events are skewed, easy to tell with Apneas, not so easy to tell with Hypopneas as you could almost say some are skewed but some are not (depends on scoring) as the Hypopnea can get slightly better or slightly worse for extended duration, so it depends at which point the S9/ResScan marks as what it thinks is the Hypopnea
- Did not power cycle S9 from 10th April
- Did not read data from 12th April to 14th April in ResScan

As it's now 15th I've now read the SD card into ResScan so I've placed it back into the S9, power cycled it and will power cycle before my next session tonight and see if the skew changes.

[Nord]

Notes
con't...
- Did not power cycle S9 from 10th April
- Did not read data from 12th April to 14th April in ResScan

As it's now 15th I've now read the SD card into ResScan so I've placed it back into the S9, power cycled it and will power cycle before my next session tonight and see if the skew changes.

Hi Dave

I thought from your "process", you were power cycling each day... was it a mistake or did you change the process ???

And on the 15th... you are now power cycling twice... is that a change ??? Are you finding immediate changes ???

BTW: I am not power cycling in my "process" except that hasn't worked out exactly because I am having other issues where I've had to move the S9 several times during 14th and 15th - I guess you could say my scientific approach has been less scientific these past 2 days and more about fixing some comfort issues... I'm only getting two of hours sleep per night right now.

Nord

[dave21]

I thought from your "process", you were power cycling each day... was it a mistake or did you change the process ???

I changed the process as I wasn't seeing large skews to see if I start to see large skews or at worst, get an hours loss of hi-res data.

And on the 15th... you are now power cycling twice... is that a change ??? Are you finding immediate changes ???

I'm going to go back and start power cycling nightly to see what the next 10 days hold.

What is strange is that the skews are very random. If you look at the 10th Flow ends 1 second early, 11th Flow extends 54 seconds, 12th Flow ends 2 seconds early.

Thanks
Dave

[Nord]

What is strange is that the skews are very random. If you look at the 10th Flow ends 1 second early, 11th Flow extends 54 seconds, 12th Flow ends 2 seconds early.

Mine have been almost always progressive... the random skews throws a wrench into memory theory. What was different between 10th, 11th and 12th ???

Was there more shut offs of S9... more Apnea Events or something different on the 11th that might suggest a reason for the extended skew ??? Something... anything ???

Nord

[dave21]

Mine have been almost always progressive... the random skews throws a wrench into memory theory. What was different between 10th, 11th and 12th ???

No difference in terms of usage. Same 16gb SDHC Class 2 SanDisk card has been used throughout for the past month. No formats or erasures of the card were required.

Was there more shut offs of S9... more Apnea Events or something different on the 11th that might suggest a reason for the extended skew ??? Something... anything ???

Each night was a single session (I've been trying to not have multi-mask sessions to ensure it's consistent). Only difference would be the time I went to bed and turned on the machine and the time I awoke and turned the machine off.

Skews aren't really noticeable until you zoom into at least the 30 seconds or 10 seconds graphs. Then you need to zoom into the start and end and each apnea/hypopnea to see if there's any kind of skew/offset.

What I notice when checking the Apneas (as this is more distinct in matching to the Flow graph) is that even if there was say an 8 second skew at the start of the flow, there isn't for most part a skew when it hits the Apnea, so almost like the skew for me on several nights resolves itself. But likewise on some nights the skew doesn't resolve itself, or the S9 / ResScan has got a little confused with what the end point of the event is. Likewise if the Flow starts 8 seconds early you don't see the same 8 second early finish point on the graph compared with other graphs. This to me suggests it's a cache problem in the S9 that sometimes it's able to keep up with the writes and sometimes it doesn't, and that possibly there's no write back cache to ensure that everything gets written in correctly vs a timely fashion. It's almost like the TCP/UDP argument where TCP network packets has a CRC Checksum on, if they don't send correctly they get resent, where as UDP is just a send and forget, great for video where you don't mind a frame drop or two.

From what I'm seeing I think the S9 is almost just chucking the data to the SD card to write without any kind of CRC checks in place to ensure that
1) The data that it asks to write does get written
2) The data that it asks to write, can be re-read and checked and is correct

If a block of data doesn't get written, it doesn't get written, so that means then there is a difference between the Hi-Resolution data vs the Detailed Resolution data, and that could be what's then creating the skew/offset. However, that's all well and good, but that wouldn't then explain how it then seems to resolve itself midpoint through the night or for the difference in the flow graph to yoyo from being starting 7 seconds early to finishing 54 seconds later than the other graphs if you take 11th April.

EDIT:

Was there ... more Apnea Events or something different on the 11th that might suggest a reason for the extended skew ??? Something... anything ???Nord

Difficult to really say, 11th I did have only Hypopneas and only 0.1, but then if you look at the other big skew on 13th then HI=0.1 too but I had OA=0.1 and CA=0.2. Don't think we can tie this up to causing the skew.
14th - OA=0.0, CA=0.0, HI=0.9
13th - OA=0.1, CA=0.2, HI=0.1
12th - OA=0.0, CA=0.0, HI=0.5
11th - OA=0.0, CA=0.0, HI=0.1
10th - OA=0.1, CA=0.0, HI=0.4
09th - OA=0.1, CA=0.0, HI=0.8
08th - OA=0.0, CA=0.0, HI=0.2

Thanks
Dave

[Nord]

I can see the UDP argument... if there is no checksum to correct... but that also implies that there is much more/ intense data at times that causes the skew. The lack of checking would allow errors but not cause them.

And if we all have the same machines (??) then why are some getting skews and other not... maybe some are not looking at the 30 or 10 second window... but Alterego is looking closely and for 5 days no skew.

Only 2 things different that I know... he is using a Class 4 card and he just got the machine... maybe faster card has other changes for S9... maybe newer machine has built in changes... cache/ supplier/ quality/ firmware upgrade/ etc.

Have you talked with ResMed about skew issue ???
My local ResMed rep has not returned my DME's calls... Los Angeles tech support seems to be staffed by people that say "just return it".

Nord

[dave21]

I can see the UDP argument... if there is no checksum to correct... but that also implies that there is much more/ intense data at times that causes the skew. The lack of checking would allow errors but not cause them.

My analysis there would be that the machine doesn't always write per second, maybe it only writes once per minute or once per 5 mins, but some minutes there's more data to write, or it could be it's doing some house cleaning. e.g. my Tivo has to do house cleaning prior to a data call when it downloads and writes out the next few days for the EPG.

And if we all have the same machines (??) then why are some getting skews and other not... maybe some are not looking at the 30 or 10 second window... but Alterego is looking closely and for 5 days no skew.

Not wanting to insult all S9 members here or any particular member, but I still think a lot of people either aren't checking thoroughly enough, or just are not 100% sure that what they see is a skew is a skew, and then decide not to mention it. Maybe it's just a select few that are seeing these problems. I've seen it both on the ResMed 1gb card and on the 16gb card. DreamDiver has also seen it on the ResMed 1gb card too from what I recall, and Uncle Bob has definitely seen it on the ResMed 1gb card.

Only 2 things different that I know... he is using a Class 4 card and he just got the machine... maybe faster card has other changes for S9... maybe newer machine has built in changes... cache/ supplier/ quality/ firmware upgrade/ etc.

We don't know what Class the ResMed cards are, there might be different batchs of SD cards (entirely plausible). For people seeing the skew we definitely have the same firmware version, I don't believe anyone without a skew has provided their firmware version to date?

Have you talked with ResMed about skew issue ???
My local ResMed rep has not returned my DME's calls... Los Angeles tech support seems to be staffed by people that say "just return it".

Not yet, I do have access to a number of people from the Product Management team and Support team at ResMed, but they seemed to be against giving me ResScan and they wanted to read the cards themselves then PDF the reports and send them. I can talk to them but I think when they find out I have ResScan there might be a lot of questions.

There isn't really any damage to not having the graphs line up by a few seconds other than it's just a nuissance that if either we or a doctor is trying to read the graph, it only makes 90% sense and not 100% sense due to a possible skew, I'm trying to see if I can get to the point where I hit the 1hr blackout window in the flow graph data, but so far that hasn't happened again.

Thanks
Dave