SD Card Video Tutorial

Introducing a video tutorial on how to archive data from the S9 Series SD card.
If you have an upcoming visit with your sleep doc or DME, using the steps in the video about two weeks before your visit will provide your doctor with good, unskewed high-resolution data.

I would like to note that if you are not reformatting your card on a regular basis, that's perfectly okay! You are still receiving excellent therapy from your machine. Reformatting the SD card on a regular basis does not help or hinder actual therapy. Your machine and your health are in no danger if you do not follow these steps. Sleep easily.

That said, I have a unique forum member request: If you have had your S9 for over six months and have never reformatted your card, I would be grateful to get examples of the data from your SD card to view the long-term effects of not reformatting the SD card on skew.

As previously stated, I could also use examples of (hopefully, doctor-confirmed or software-indicated) periodic breathing, cheynes stokes, mostly REM-stage apneas, more ordinary obstructive data and any other unusual or typical data that might be instructive in a video on ResScan and Encore Pro 2. (Yes, I'm also interested in data from the System One. Although card reformatting is not indicated for the System One SD card, a similar method for archiving data from the System One could be just as useful.)

Again, all real social information (names, etc.) will be removed and changed to bogus names - maybe names of famous hobbits... Ideas, anyone?

Please PM me (DreamDiver) for an email address if you are willing to help.

DreamDiver, can you figure out a place to put links to all your videos so they are always aqble to be reviewed? I am afraid to miss them as they come up in the forum if I dont check in for a couple of days. In some other groups they have a section called files and there are links maintained permanently there for VALUABLE stuff like your videos!
thanks

Thanks again. As I mentioned before I click the F11 to see your presentation in full on one pane in my laptop. To get out from it I need to click F11 again. Before I continue to study your presentation I would like to ask: I use two SD cards and switch between them almost daily into the S9. This is if I lose one or if I need to mail to my DME. During the ResScan operation to get downloading for graphs there is a question if I want to OVERWRIGHT and I say YES. What effect it has to do with "Reformatting", and what do you mean by Reformatting?

avi123 wrote:Thanks again. As I mentioned before I click the F11 to see your presentation in full on one pane in my laptop. To get out from it I need to click F11 again. Before I continue to study your presentation I would like to ask: I use two SD cards and switch between them almost daily into the S9. This is if I lose one or if I need to mail to my DME. During the ResScan operation to get downloading for graphs there is a question if I want to OVERWRIGHT and I say YES. What effect it has to do with "Reformatting", and what do you mean by Reformatting?

'Overwrite' has no effect on SD card. The data you're overwriting is the data that's already uploaded to your ResScan patient file folder. The SD card holds about seven days of high-resolution data and about thirty days of medium resolution data, as well as about a year of low resolution data and all compliance data.

For instance:
Say I have high resolution data for February 1 - 7 and Medium resolution data for January 7 through February 7 already on my computer as well as all low resolution data since I 'started' (just for this example) on October 12 last year and compliance data.

The data on my SD card that I'm about to upload contains high resolution data for February 5 - 12, Medium Resolution data for January 12 through February 12, Low resolution data through October 12 of last year and all my compliance data. When I 'overwrite' I'm removing any data from my computer that is not similar to what's in the SD card and replacing it with what's on the SD card. So, I will end up with only High-resolution data for February 5 - 12, Medium resolution data for January 12 through February 12 and all low resolution data and compliance data since I started on October 12 last year. Overwriting deletes all my saved high resolution data on my computer for February 1 - 4, Medium resolution data from January 7 - 11, and retains my low resolution data through October 12 last year as well as compliance data.

So if I want to go back and look at my flow detailed graph (high resolution data) for the February 4 or February 2 in ResScan after I "overwrite", it will be gone. If I want to look at my Apneas, or any other medium resolution data (other detailed graphs) before January 12, it will also be gone. All my statistics (low resolution data) will remain available for view through October 12, and so will compliance data.

Because I like to keep my data I generally hit 'DISCARD' so that I keep all the historic data on my machine. 'Overwrite' is 'preferred' and bolded primarily because most DME's / Sleep Labs using this software have many patients. All that data might quickly overwhelm a harddrive. For instance, in reality, starting on March 3, I will have used my machine for exactly one year. My patient file folder currently contains 3.2GB of data. That's a lot of data, and I didn't even archive on a regular basis. Now multiply that by 300 patients. You can see where this is going. They delete old data because it's not efficient use of hard-disk space, and no one has made it a priority to archive historic data. If you are not interested in keeping your historic data, there's no reason to discotinue your current practice of overwriting. If you would rather keep the historic data readily available for future reference, it's best to hit 'discard'.

Here's a fun fact: I have about 97 archived SD card zip files for the whole year from which, if I wanted to I could reconstruct the entire set of data I currently have in my ResScan patient folder, but the archives all together total only 338MB - about a tenth the size of the data in my ResScan patient folder.

About alternating cards:
By alternating cards nightly, the DME or Doctor will receive alternating nights of missing high-resolution and medium resolution data. If you want them to have all the available data, it might be better to just make a duplicate copy of the one SD card that you use continuously every night. If it's not a big deal for you or the doctor I don't see any reason why you shouldn't continue as you are doing.

It's neat that F11 does that for you. Thanks for the tip.

I probably should have posted this earlier, but...

It is not necessary to perform a format of the SD card every two weeks to avoid data skew. The following procedure avoids the data skew while also 1) avoiding the additional wear on the SD that true formatting causes and 2) maintains all data on the card:

• Perform a backup of the card every 1-2 weeks (I do this every weekend).
• Delete all files and folders from the card.
• Insert the card in the S9 and allow the machine to "format" it. At least, that is the word used by the S9. In actuality it simply puts everything except the DATALOG folder's contents back onto the machine. I do not believe that it does a traditional computer format of the card.
• Once the S9 "format" is complete, simply restore the contents of the DATALOG folder from your backup. This is where the detail and high resolution data is stored.

By following these steps, the data skew is avoided yet all of the data remains intact on the card. I don't have to plan ahead for downloads by my DME.

I am 99% convinced that the root cause of the data skew issue is not a bug in the S9 but rather a consequence of the FAT file format used on the card. Over time, files stored under the FAT format become broken up into pieces. This is called fragmentation. When fragmentation becomes bad enough, the ability to write data to the card fast enough to keep up with the data collection rate is hampered. The more fragmentation, the worse the data skew becomes. Think of it like planting tomato plants. Most people will go down one row, then move to another row and go down it. Fragmentation is like planting one plant at the end of one row, then going over to the middle of the 5th row and planting 1 plant, then going to the end of the 2nd row, then the beginning of the 5th row, etc, etc. A lot of time is wasted if you plant this way. Over time, though, because of how the FAT format operates, which was designed in the 1980's (I think), this is exactly what the S9 has to do after 1-2 weeks of usage. The procedure outlined above effectively defragments the card, thus avoid data skew.

I saw data skew during my first 2 weeks of use. After 3+ months, I have not formatted my card, preferring to follow this procedure. I haven't seen any data skew since those first 2 weeks.

Some computer savvy folks might say, then why not simply defragment the card? The problem with that is that, as idamtnboy mentioned in another thread, the files in the root directory of the SD card, mostly likely just journal.dat (maybe one other) is intentionally placed in a specific physical location on the card. Defragmentation software will relocate it. When that happens, the S9 will either give an error message, or it will prompt to "format" it.

Please don't take my word for it, though. Try it.

DreamWeaver, sorry to post this after you did the video. Had I known you were going to do this, I would have shared this earlier. Not posting it now, however, wasn't really an option for me due the unnecessary wear and steps that your tutorial recommends. I didn't post this to point out flaws - I posted it to provide a better way to accomplish the same thing. I hope you understand.

BernieRay wrote:I probably should have posted this earlier, but...

DreamWeaver, sorry to post this after you did the video. Had I known you were going to do this, I would have shared this earlier. Not posting it now, however, wasn't really an option for me due the unnecessary wear and steps that your tutorial recommends. I didn't post this to point out flaws - I posted it to provide a better way to accomplish the same thing. I hope you understand.

Doh!

I need to make that part of a future version of the video. You're absolutely right. It achieves the same purpose and is probably simpler. Thanks BernieRay.

Excellent feedback.

Send me a bill for the redo. Had I posted this back in early December to the Data Skew thread...well...that's water under the bridge.

Still, your video is an excellent tutorial. I arrived at a similar approach, but did it from scratch. Your video would have saved me some trial and error had it been around then. I'm glad that you've posted it.

I did forget one thing. At first, I tried simply deleting the DATALOG folder and copying it back. The skew still existed, probably due to fragmentation of one of the root folder files - probably STR.EDF or journal.dat. I was actually baffled as to why defragmentation or a manual copy of the root files caused problems. It was only a few days ago that idamtnboy provided the missing piece about the physical location being enforced by the S9.

If someone finds an even simpler way to do this, I'd love to hear about it.

I think I must be missing something here.

Each time I remove my S9 SD card to enter the data into ResScan, I also copy the S9 SD card data to a "Backup" folder on my hard drive overwriting/replacing previous info copied to the backup folder as necessary. Then, occasionally, I copy the info from the backup folder to a backup SD which serves as a backup as well as it allows me to take all my data to my DME so that she can update my patient file, as I did just yesterday. The backup SD holds all my info for the last five months, the time I have been using the S9, so the DME now has my complete data history in her ResScan file.

Someone tell me what's wrong with this method.

The SD card only holds

365 days of summary data
30 days of detail data
7 days of high resolution data

The only way to keep more than these limits, outside of ResScan itself, is to keep separate backups no more than 7 days apart. The biggest reason to do this is to allow re-building of your ResScan patient file, should something happen and you don't have a viable backup of your patient file.

If you simply backup on top of the previous backup, you slow lose the ability to recover the older data.

BernieRay wrote:By following these steps, the data skew is avoided yet all of the data remains intact on the card.

I just remove power from my S9 every morning and plug it back in a night, and all of the data remains intact on the card and I've never experienced any data skew issues.

BernieRay wrote:The problem with that is that, as idamtnboy mentioned in another thread, the files in the root directory of the SD card, mostly likely just journal.dat (maybe one other) is intentionally placed in a specific physical location on the card.

No, they can be anywhere in the FAT file system and will function just the same.

Niether ResScan nor the S9 have or need any knowledge of where the files are physically located on the SD card, they just use OS system calls to get to the file heirarchy and the OS file system interface handles the physical placement of the files. The "access" time of a memory storage "drive", like a SD card or USB memory stick, is essentially constant for any physical location on the "drive", so physical location is, in any case, irrelevant.

Cheers,

Bill

The power disconnect solution didn't work for me when I tried it. It would have been nice if it had.

billbolton wrote: Niether ResScan nor the S9 have or need any knowledge of where the files are physically located on the SD card, they just use OS system calls to get to the file heirarchy and the OS file system interface handles the physical placement of the files. The "access" time of a memory storage "drive", like a SD card or USB memory stick, is essentially constant for any physical location on the "drive", so physical location is, in any case, irrelevant.

Cheers,

Bill

Then your S9 behaves differently. If the root files are moved via a copy/delete/copy on mine, the S9 won't recognize it. One file, journal.dat if I remember correctly, is always placed in the last block(s) of memory on the card. The S9 is doing that deliberately via a low-level write, because, as you point out, normal file IO calls don't care nor do they provide such control. idamnboy discovered this and I've confirmed it - several times. The fact that it won't recognize it when the file isn't in the last block(s) seems to imply that the S9 is doing a direct, low-level read, probably as a step to insure the integrity of the data.

The access time is impacted by the number of jumps to non-sequential addresses. It has nothing to do with where they are located, only that they are non-sequential. (My tomato plant analogy wasn't perfect in that regard, but it does adequately describe the concept of memory space usage and the impact of non-sequential reads/writes in relation to what I was describing.) The delays aren't nearly as large as those on a hard disk, but they aren't irrelevant either. When capturing multiple data points several times a second, those delays do add up. Regardless, the steps that I outlined do avoid the data skew issues that some of us have had and keep the files from being fragmented. Until someone can prove the real cause, all we can do is make educated guesses as to the cause.

But...it looks like we have at least 3 ways to avoid skew issues, so everyone can pick which they prefer or whichever method seems to work for them.

billbolton wrote:...
I just remove power from my S9 every morning and plug it back in a night, and all of the data remains intact on the card and I've never experienced any data skew issues.
...
No, they can be anywhere in the FAT file system and will function just the same.

Niether ResScan nor the S9 have or need any knowledge of where the files are physically located on the SD card, they just use OS system calls to get to the file heirarchy and the OS file system interface handles the physical placement of the files. The "access" time of a memory storage "drive", like a SD card or USB memory stick, is essentially constant for any physical location on the "drive", so physical location is, in any case, irrelevant.

Cheers,

Bill

I have tried this method without success. Perhaps the electrons spin in a different direction down under.

Seriously, I wonder if their are slight differences by country?

By the way - journal.dat is a misleading file. It is comprised of empty space (null characters in computer terms) - it contain no useful data whatsoever. And it's size never changes. It is always 28672 bytes long, exactly 28kb. Rarely are files exactly x Kb in size without intent. Why they chose 28Kb is a mystery. Since it's date/time stamp is set when it is created and not modified, the only remaining attributes that could be useful are its presence on the card and its physical location on the card. There isn't anything useful about this file beyond that as far I can tell.

DreamDiver, how much are you going to charge me for highjacking your thread? How much for calling you DreamWeaver?

billbolton wrote:No, they can be anywhere in the FAT file system and will function just the same.

Niether ResScan nor the S9 have or need any knowledge of where the files are physically located on the SD card, they just use OS system calls to get to the file heirarchy and the OS file system interface handles the physical placement of the files. The "access" time of a memory storage "drive", like a SD card or USB memory stick, is essentially constant for any physical location on the "drive", so physical location is, in any case, irrelevant.

Cheers,

Bill

Resscan does not care where the files are on the card. You can even have 6 months of detail and hi res files on the card and Resscan will read them all. The S9 very much cares. It wants the card to be physically (electrically?) to be just so so, or it pitches a fit.

I'm skeptical of BernieRay's suggestion that fragmented files are the cause of the data skew simply because the real time to write data is so short compared to the data skew times. My card, which has not been formatted for 4 months, I think, shows no data skew for the last few days' data.

I once moved the files instead of copying them - by mistake.
I immediately copied them back onto the SD.
The S9 reformatted the card when I reinserted it.