S9 flow skew bug... Anyone else seeing this?

[Nord]

The most popular theory right now for the source of the skew problem is removing the SD card too soon before all data have been written to it.

I found it interesting that the open specification for SD cards (http://www.sdcard.org/developers/tech/s ... r_Spec.pdf) omits the content for the section for hot insertion and removal: "This section is a blank for the Simplified Specification".

In the SanDisk product manual (http://www.cs.ucr.edu/~amitra/sdcard/Pr ... rdv1.9.pdf), the following section indicates that hot insertion/removal of the card should be handled:

"Hot insertion and removal are allowed. The SanDisk SD Card will not be damaged by inserting or removing it into the SD bus even when the power is up:
• The inserted card will be properly reset also when CLK carries a clock frequency fPP.
• Data transfer failures induced by removal/insertion should be detected by the bus master using the CRC codes that suffix every bus transaction. "

Hi alterego

That SanDisk document is exactly what I couldn't find on their Site and phoned Tech Support to find...

Reading through the CRC instructions, it seems to say that if speeds were not met by the card... that an Error message would be sent to the Host and would probably be contained in the same files that you've been looking at. It suggests it would ignore the data sent and probably just be dropped.

Have you been able to see and interpret any Error messages yet ???

Although the way that the SD card deals with errors would match the Skew that we've seen... I really think Hot insertion/ removal is the better choice. Since I and a few others are testing that theory now... we'll know soon enough. I'm 2 days in...

Nord

[alterego61]

Hi Nord, great to see someone else trying to make sense of the problem at this level. I'm still scratching my head here.

Reading through the CRC instructions, it seems to say that if speeds were not met by the card... that an Error message would be sent to the Host and would probably be contained in the same files that you've been looking at. It suggests it would ignore the data sent and probably just be dropped.
Nord

I don't draw the same conclusion as you, at least from what I have read so far. First, the rate that data is written to the card from the s9 is very slow relative to the capacity of the card. Second, even if the card was temporarily unable to keep up with the quantity of data it was being requested to write, it would signal that to the host (s9) which, if properly programmed, should be able to deal with the situation rather than writing an error to the data files or corrupting the data. I have no experience of realtime embedded system programming but I would think that this type of error-handling would be standard practice in this type of application.

Have you been able to see and interpret any Error messages yet ???

No

Although the way that the SD card deals with errors would match the Skew that we've seen...

How do you draw that conclusion?

[-SWS]

The most popular theory right now for the source of the skew problem is removing the SD card too soon before all data have been written to it.

Guys, I apologize for not having followed this issue closely. I don't have an S9... at least not yet.

So how many days worth of detailed data can be retained on the card? Can't you test the above theory by picking one or two machines that are chronic offenders---and then leaving the card in the machine for a few days before pulling data? Or at the very least wait until afternoon/evening of the same day to pull your data...

[kennethryan]

Ken, I'm still trying to get my head around how SD cards work. This description (from Wikipedia) explicitly describes how embedded systems normally interface with SD cards:

"For the most part, the lack of a complete, open SD specification mainly affects embedded systems, since desktop users generally read SD cards via USB-based card readers. These card readers present a standard USB mass storage interface to memory cards, thus separating the operating system from the details of the underlying SD interface. However, embedded systems...usually access SD cards directly, and therefore complete programming information is necessary."

Pretty much every SD slot on a PC uses a USB interface device of some sort. Usually you have a little box with a jack for a USB cable on one end, and one or more memory card slots on the other. If you were to crack open that box, you'd see the USB connector, the card guides and contacts making up the card slots, and a chip in the middle of it all. That chip is the card interface.

USB has several classes of device predefined. If you have a mouse or keyboard, it fits within the USB human interface device class. A printer fits within the USB printer class. Hard disks and CDROMs fit into the USB Storage Device class. If the manufacturer of the interface box can make their device look like one of these predefined classes, the OS will already know how to drive them. The manufacturer does not have to write a device driver - Microsoft (or Apple or Linux) has already taken care of everything for them. (Note within the class there is data the OS can read to determine what kind of storage device, what functions are supported, etc.).

The chip in the interface box takes any of several memory card types and makes them look like a generic USD Storage Device. The manufacturer of the chip does all the work of correctly managing the physical card interface.

Embedded computers such as cameras and CPAP machines try to minimize system complexity - they don't want to spend the extra chip (=cost, space, power, etc). So they spend a minimum of hardware to interface to the card (as few as 4 wires for SD), and do the detailed card control in software. That then puts the onus on the software developer to understand all the subtleties of operating the card, from signal sequences to access timeouts and so on - including accounting for card errors, users yanking the card out early, and so on. It can get quite involved to cover all the cases, and many developers miss some.

Hope this helps a bit ...

ken

[alterego61]

So how many days worth of detailed data can be retained on the card?

7

Can't you test the above theory by picking one or two machines that are chronic offenders---and then leaving the card in the machine for a few days before pulling data? Or at the very least wait until afternoon/evening of the same day to pull your data...

Yup, that's what several of us are testing

[alterego61]

Ken, thanks for the SD and USB info.

If I read you correctly, you are saying you think ResMed, rather than using a USB SD card interface that would increase cost, size and power requirements, chose a cheaper and simpler low-level interface to the SD card interface for the s9, and may have made some kind of mistake in the low-level programming?

[-SWS]

So how many days worth of detailed data can be retained on the card?

7

Can't you test the above theory by picking one or two machines that are chronic offenders---and then leaving the card in the machine for a few days before pulling data? Or at the very least wait until afternoon/evening of the same day to pull your data...

Yup, that's what several of us are testing

Thanks.

[Nord]

I don't draw the same conclusion as you, at least from what I have read so far. First, the rate that data is written to the card from the s9 is very slow relative to the capacity of the card. Second, even if the card was temporarily unable to keep up with the quantity of data it was being requested to write, it would signal that to the host (s9) which, if properly programmed, should be able to deal with the situation rather than writing an error to the data files or corrupting the data. I have no experience of realtime embedded system programming but I would think that this type of error-handling would be standard practice in this type of application.

Same document... 4.7 Section deals with Errors... just further along.

Although the way that the SD card deals with errors would match the Skew that we've seen...

How do you draw that conclusion?[/quote]

On the basis that if it checks that data to start a "read" but it does not match the checksum... for whatever reason. It will report an Error. But then it will try another "read" some ms later and if "available" it will complete the write to the block. That would be a delayed write... and if its correct... the later timestamp than it should be.

Its not my idea at all though...
DreamDiver reported that a ResMed tech reported to him that the delayed speed of CRC check on some cards could skew the data.
My theory is based one way that "might" happen.

I am only too happy to abandon that theory as I really don't have that much confidence in it. Certainly not as much as the ResMed tech...
It doesn't account for you having Skew with the much higher quality Class 4 card.

I'm partial to the Hot Insertion/ Removal theory and I can see how that early process would cause read and write problems for the card since as Ken suggests for embedded systems, its up to the software to deal with issues properly. Their expectation and testing may not include all eventualities.

Nord

[kennethryan]

Ken, thanks for the SD and USB info.

If I read you correctly, you are saying you think ResMed, rather than using a USB SD card interface that would increase cost, size and power requirements, chose a cheaper and simpler low-level interface to the SD card interface for the s9, and may have made some kind of mistake in the low-level programming?

Yes, it's very common in embedded systems like this to use the direct low-level interface. It's not a general-purpose computer; the access patterns are well known and can usually be implemented cheaply (notice cameras, MP3 players, GPS units, and other small devices nearly always require files in a certain fixed directory structure). However as I said it is possible to make a mistake coding it up (hey it happens to the best of us).

Mind you I have no actual evidence of a firmware bug - if I did this whole discussion would be over. I'm just saying it's reasonable to consider, and why.

BTW, it's not usually an issue of cost per se (except in the cheapest consumer products such as a camera). They may not have had enough I/O pins on the microcontroller IC to support a more complex interface chip, or they may not want to spend the memory for USB storage driver code (the USB stack is quite a bit more complex than the SD interface). Driving the SD card directly *is* the simpler, safer option, and the FAT filesystem likewise is popular for its simplicity.

(Boy I'd love to crack open my S9 and see what they use for a microcontroller. Not going to happen until it gets replaced by the next gee-whiz version though, or at least not unit the warranty runs out. )

[DreamDiver]

First, thank you alterego61, kennethryan and Nord for trying to explain the housekeeping thing. I'm sure it will sink in at some point. I'll need to reread, I think.

... DreamDiver reported that a ResMed tech reported to him that the delayed speed of CRC check on some cards could skew the data.
My theory is based one way that "might" happen.

This is where it pays to be clear about interpretation. I found what was said by ResMed vague. Both your and alterego61's interpretations are easily construed on first glance. The below quote is a repeat of what was said by ResMed here. Again, my emphasis in blue.

There’s a couple things going on here, but you are confusing a few things. The SD card has a house keeping algorithm that only has certain periods of time that its ‘available for communication’ which has to align with our own data write periods. This varies by mfg.

This availability can be on the order of 250ms

http://www.ultimaserial.com/sandisk.html

Note that every manufacturer can employ their own house-keeping algorithm, that will ultimately determine how often these limits are reached. This is why we expect to see difference between SD card manufacturers, although they all follow the same standard.

This “access time limit” is much bigger than the numbers that the users are quoting here – the numbers they are quoting, relate to the speed of the card - which is much faster, and is only applicable for a single block write – when the card is “available for communication”.

We can look into the file write data and see when the exact time it took to open, read, and write the data and its clearly getting worse over time. There’s also a confusion on the board where some of the users are ripping the card out of the device before its closed the session (it doesn’t make sense to write data every single second)

It could be construed that the problem is with the card or that the problem is with the timekeeping of the S9, but it is not essentially clear what is being said. Is it better for us to have cards with the 250ms or 100ms housekeeping for the S9?

On another note, just out of sheer lack of energy, I've been leaving my card in two and three days at a time without reading it. It still shows increasing skew on a daily basis, although this card isn't almost doubling every day as the card from ResMed did - it's only increasing by about six seconds per day on average. I doubt it's about yanking the card out too soon. Most of us don't get up and immediately want to read the card. The first thing is almost always the bathroom. That should be more than enough time for the card to do any left-over housekeeping.

[Nord]

It could be construed that the problem is with the card or that the problem is with the timekeeping of the S9, but it is not essentially clear what is being said. Is it better for us to have cards with the 250ms or 100ms housekeeping for the S9?

On another note, just out of sheer lack of energy, I've been leaving my card in two and three days at a time without reading it. It still shows increasing skew on a daily basis, although this card isn't almost doubling every day as the card from ResMed did - it's only increasing by about six seconds per day on average. I doubt it's about yanking the card out too soon. Most of us don't get up and immediately want to read the card. The first thing is almost always the bathroom. That should be more than enough time for the card to do any left-over housekeeping.

Hi DD

Is it possible to keep up the contact with the ResMed Tech who wrote that info ??? Can we get him to clarify their position about the skew... since you are leaving the card in the machine and still getting the "problem"... it points to the card or the firmware.

Nord

[Nord]

Still experimenting and trying to initiate Skew... in different ways

Nord

[fiberfan]

I haven't been paying attention to skew for a bit but the skew is now large enough that it shows on my normal look at the data with a 30 minute window.

Todays data had 2 mask sessions

Mask session 1
22 sec start
20 sec end

Mask session 2
25 sec start
45 sec start

I also had a minute of flow data a few minutes after the 2nd mask session with all other graphs empty.

Sounds like it's time to reformat, I will copy files first if anyone is interested in looking at them.

[alterego61]

Less than a week with a freshly formatted card and a 2 second skew has just appeared for me - flow data begins 2 secs before pressure data.

I have waited no less than 15 minutes each day before taking the SD card out of the s9 to read it, so the theory about removing the card too soon causing the problem is blown!

[alterego61]

I managed to make contact with someone at ResMed who it seems understands the issue.

I was told that the issue is with ResScan, not with the SD card or the S9 and that there is no fix ready yet, but the resolution will likely be a software patch for ResScan.

For myself, I think I will just reformat my SD card every day as a workaround until the problem is solved.