Python EDF Library -- To read cpap machine datalog files

Slinky wrote:I have no idea how many days of Detailed data the S9s are designed to hold and transfer so you may have to run two files for a couple of days to get the drift. How many days of Detailed data does your Clinicians Manual say will be held and reported for the S9? I would imagine that it may well vary between the different S9 models.

S9 holds 7 days of detailed data.

I've seen it do weird things if you have multiple sessions in a day... and it has weird definitions of "today" and "yesterday".. I think the cutoff is noon.

Basically if I didn't "replace" it didn't ever get the 2nd session. I think what you said about losing detailed data is right, tho.

idamtnboy wrote:

ohammersmith wrote:ooo, not sure how I missed this... will have to check it out. Rescan is effing slow and EDFBrowser was confusing me... I opened the same data file multiple times and only saw an actual line on the graph the first time. Attempting to export to TXT file resulted in a 3.1 gig file before it hung up and died.

EDFBrowser opens up every time with the time and amplitude scales set to the default very high number. Every time I open it I have to change the timescale to something like 1 hr or 20 min and the amplitude to .2L/sec in order to see a line. Could this be the reason you saw the graph only the first time?
What did you use to convert to text? I've never had a hitch using edf2text.exe. If you drag the edf file icon onto the edf2text icon you'll see a command window open momentarily, and then nothing until it's finished and the extracted files show up in the folder. I always put a copy of the edf2text file in the same folder with the edf files.

But then this is on a windows machine. If you're using the Linux versions then maybe things run differently. I don't even know if there is a Linux version of edf2text.

I thought it was the scale thing, but I couldn't get a graph on any scale after the first run. Plus when I looked at the file info dialog it showed 0 samples.

I was using some text export feature of EDFBrowser itself.

My assumption has been that when I force quit the app (I'm on a Mac, btw, so neither Linux 'nor Windows) that it borked the EDF file and couldn't get anything out of it. Fortunately, I expect that kind of stuff to happen when I had to compile the software myself so I was working with a copy.

I just haven't gotten back to trying it, again.. which I may never, if there's a Python library & I just write my own report generator.

ohammersmith wrote:

Slinky wrote:I have no idea how many days of Detailed data the S9s are designed to hold and transfer so you may have to run two files for a couple of days to get the drift. How many days of Detailed data does your Clinicians Manual say will be held and reported for the S9? I would imagine that it may well vary between the different S9 models.

S9 holds 7 days of detailed data.

I've seen it do weird things if you have multiple sessions in a day... and it has weird definitions of "today" and "yesterday".. I think the cutoff is noon.

Basically if I didn't "replace" it didn't ever get the 2nd session. I think what you said about losing detailed data is right, tho.

The S9 data card holds 7 days of high resolution data, 30 days of detailed data, and 360 days of summary data. The S9 machine memory retains only summary data, and that for 360 days. The hi res data is 25 Hz, the detail data is .5 Hz. The only hi res data is flow and mask pressure.

The cutoff is noon. I don't know what it does if you have a session continuing over the noon hour, but probably it closes the first part into a file with an end time of noon and opens a new file with a start time of noon. If you sleep over the noon hour and download in the afternoon you will have two days data going into Resscan. The way I see it, it's a catch-22 when downloading, choose overwrite you have all yesterday's data, but lose the older stuff. Choose discard and you don't get the data from the time of downloading the day before and noon of the current day. Setting the read only attribute on the rlk files in the patient folder may keep Resscan from overwriting the older data, but I haven't forced a test of that.

I believe the overwrite function when downloading causes Resscan to overwrite all previously saved patient data with data from the card. I know I had some detail flow graphs in my patient data that were older than seven days and disappeared from my patient file when I downloaded with the overwrite function. I recovered the data from backup files. Since then I've always selected discard. That causes me no problem because I never have a session between when I download the night's data and noon.

idamtnboy wrote:EDFBrowser opens up every time with the time and amplitude scales set to the default very high number. Every time I open it I have to change the timescale to something like 1 hr or 20 min and the amplitude to .2L/sec in order to see a line.

Straight from the manual:

Montages can be used to store information about which signals should be shown on
the screen, what combinations (subtractions) should be made and other properties
like filters, amplitude and color. Once you have loaded a file for review and
you have added all the signals you want (and/or all the combinations/subtractions) and
you have adjusted amplitude and colors and/or filters, you can save them as a montage.
Go to Montage -> Save. A dialog will be opened where you can choose a directory
and filename to store your montage.
Next time when you load a file, you can load a montage via Montage -> Load.

You can view the properties of a montage via Montage -> View saved montages or
Montage -> View this montage.

note:
It is important that the names (labels) of the signals are correct when you use a montage
to review different files.

You can bind the F1 to F8 keys to eight different montages. This way you can quickly switch
between different montages by pressing key F1, F2, etc.
Go to Montage -> Edit key-bindings for montages.

http://www.teuniz.net/edfbrowser/EDFbro ... l#Montages

I don't even know if there is a Linux version of edf2text.

Yes there is, in EDFbrowser go to Tools -> Export EDF/BDF to ASCII

tav wrote: Straight from the manual:

Amazing what you can learn sometimes when you take the time to read the manual!! I usually do, but sure didn't this time!

I don't even know if there is a Linux version of edf2text.

Yes there is, in EDFbrowser go to Tools -> Export EDF/BDF to ASCII

I'm referring to a small stand alone command line program for windows called edf2text.exe. Is its code incorporated inside EDFBrowser? As I recall, the one time I did the text export from EDFBrowser the output was not the same.

idamtnboy wrote:I'm referring to a small stand alone command line program for windows called edf2text.exe.

Here it is: http://www.teuniz.net/edf2ascii/

idamtnboy wrote:Is its code incorporated inside EDFBrowser? As I recall, the one time I did the text export from EDFBrowser the output was not the same.

Lots of people are using this ascii-export without complains. What are your requirements?

idamtnboy wrote:

BernieRay wrote:As to the 365/30/7 day card retention scheme, I'll post some charts this evening that will show why a full year's worth of data, including HRD, will not reliably fit on a 1Gb SD card.

I'll be interested in seeing how you come up with that. My calculations, which I've posted a time or two, indicate that there will only be about 30 to 40 mb of data total for a year's worth of data on the card.

OkAy, for the third time since Christmas, I have tried to find the spreadsheet that I had put together to show this, but I don't have a clue where it's at. I never delete files and archive and backup fanatically, so I've probably just lost it in the clutter. It took several hours to put together, so I'm a bit ticked at myself. Anyway...

The bottom line is that the minimum cluster size for a file, 16 Kb if I remember correctly on the ResMed SD cards, when combined with the data files and the number of CRC files, uses up space on the card far faster than the data that those files actually contain. So, even though the CRC files only have 8 bytes of data, 16Kb of storage is used for each one, effectively wasting a huge amount of space. Why they chose 7 days for high res instead of 14 or 30, I don't know. 30 days would likely always fit. It may simply be that medical staff are unlikely to have the time, or possibly the need, to look further back.

Had Resmed elected to use a proprietary file format instead of the open format EDFs, they could easily fit a year of high res data on a 1 Gb card. Personally, I'll take the open format any day.

BernieRay wrote:The bottom line is that the minimum cluster size for a file, 16 Kb if I remember correctly on the ResMed SD cards, when combined with the data files and the number of CRC files, uses up space on the card far faster than the data that those files actually contain. So, even though the CRC files only have 8 bytes of data, 16Kb of storage is used for each one, effectively wasting a huge amount of space. Why they chose 7 days for high res instead of 14 or 30, I don't know. 30 days would likely always fit. It may simply be that medical staff are unlikely to have the time, or possibly the need, to look further back.

I just looked at my SD card with a hex editor program. The card is formatted FAT32 with a cluster size of 4k. Let's look at the 30 days of detail files. There are 4 crc files and one EVE file for each day which are almost always less than 1k. If we allocate 4k to each of them, as you rightly point out, that's 5 x 30 x 4096 = ~640,000kb, less than 1 mb. My card has 90 days in the summary file, 30 days of detail files, and 7 days of hi res files. The total card space used is 28.9 mb, or 3% of the card. WinHex says 97% of the card is free space. I don't think there is any way one could use more than about 5% of the card, or at most 10% or 100 mb.

I just went back and reread your first comment about this. I may have misread what you were saying. If we take 15 mb of hi res files and multiply it by 52 for a whole year we get about 800mb. Then add to that another 15 mb x 12 = ~180mb to account for the detail files for a month, we are right at the max for a 1 Gb card. So what you said about not being able to record a whole year of hi res data on a 1 Gb card is right. But I don't think there would ever be any particular value in doing so anyway.

Slinky wrote:You are misusing the Overwrite and Discard options. When the software asks if you want to Overwrite or Discard it gives a date. If you had downloaded data for that date before to this patient file use the Discard option. Otherwise that date's data is overwritten.
Tomorrow morning start TWO files. Download data from the card to each file. The next morning try doing one file w/the Overwrite option and the other file w/the Discard option. Assuming you are working w/the S9s instead of the S8s I have no idea how many days of Detailed data the S9s are designed to hold and transfer so you may have to run two files for a couple of days to get the drift. How many days of Detailed data does your Clinicians Manual say will be held and reported for the S9? I would imagine that it may well vary between the different S9 models.

The data Hi Res data is held only on the SD card - not on the S9. For that reason, the S9 can record and store 7 days of HD data, but the SD card must be inserted in order for this to happen. If you don't have the card inserted for any particular session, all Hi Res Data for that session is irretrievably lost because it was never stored. Also for that reason, if you stuck a new blank SD card into the S9 the next morning after removing the SD card the has the Hi Res Data, all you'll get on the second card is transfer of the Low Res data to the second SD card. The S9 does keep Low Res data on board, but not Medium Res or Hi Res data. This is where it differs from the S8. For this reason, this experiment may have a different outcome than you expect from the S8. Hitting overwrite will also remove all Hi Res data from the computer except for the last seven days. Medium Res data may remain. Losing all my flow data would make me very sad.

idamtnboy wrote:...There are 4 crc files and one EVE file for each day which are almost always less than 1k. If we allocate 4k to each of them...

Yeah, 16kb was wrong. 4kb is the min for FAT32 on a 1 Gb card. On the crc/EVE files, though, Resmed had to account for the possibility of multiple use periods per night. For folks who wake up once during the night and stop their S9, there will be 10, 15 if they wake up twice, etc. If someone averages waking up once per night, the 940 Mb of usable space is about 275 Mb short of what's needed for a full year. That is based on average data collection rates for the BRP (352 Kb/hr), PLD (40 Kb/hr), and SAD (14 Kb/hr) files. I calculated these averages using the actual amount of data contained within each file type for about 1 month of my data. Since these files contain data that is collected at a fixed rate, these averages should be good for everyone, give or take a bit for slight variations in collection rates between the individual units.

As is typical timing for myself, I spotted my lost files this morning in a directory called "Crunch". Considering that I was crunching data the name makes sense, but that's not a typical directory name that I use, thus my inability to find it earlier.

I'm curious if you have had any success reading the raw data that comes out of the MD300W1? We have been trying to understand the data that comes out of the machine. I agree the oximetry program that comes with the machine is horrible!

I have the MD300W1 Oximiter and agree that the software is barely one level above useless, but I do use it to download the data into it's .dat files. I have worked out their binary format well enough that I am now able to read these files, or at least well enough to get the header, oxygen and pulse information. I currently have C source code that I am adding with EDF and EDF+D routines to provide full data analysis compatible with my equipment (ResMed S9 Autoset and the oximiter).

I currently have much of the framework operational (cross platform for Windows, Linux, and OS-X) and am concentrating on some dynamic graphing routines to help review the kinds of data we typically get. My intention is to make this software available as open source once I have an initial version I am happy enough with. Being modular it should be very easy to add data handling for other machines and other sleep related information. My hopes is that we will be in a position to not need the software from the companies and can all have a standardized way of looking at things. I am sure there are enough people by now to know what they want to see when looking at their data.

This project sounds very promising as an alternative to ResScan. How is it coming along? Is it non-programmer user-friendly yet?

This is a really old thread, but I wanted to bump it to see if anyone can explain how your read in the .rlk data (EDF Data from the S9) into EDFBrowser.

Perrybucsdad wrote:This is a really old thread, but I wanted to bump it to see if anyone can explain how your read in the .rlk data (EDF Data from the S9) into EDFBrowser.

You don't. EDF Browser reads the edf files on the SD card.