Suggest storing data for a single-user app
I am looking for a data storage option for storing ECG (1000 samples per second) and other patient data (e.g. blood pressure, body temperature, etc. - much lower sample rate) in a storage request for my C # .net application.
I've already evaluated SQLite (which is a great option on my own), but I'm looking for some option that meets the following requirements:
-
Small storage space - ECGs are usually sampled at 1000 samples per second, and I need to store ECG data for 24 to 48 hours (82 to 162 million data samples). On SQLite, it takes up a huge amount of space.
-
I have to quickly read some of this data (from - to timestamps).
-
I should be able to modify parts of the data without having to write all the data from that point further.
I also looked at HDF5, but didn't really figure out how to use it from C # .net.
Look for practical suggestions.
Thank,
Vikram
source to share
Your use case is perfect for HDF5.
- Small storage space - ECGs are typically sampled at 1000 samples per second and I need to store ECG data for 24 to 48 hours (~ 82 to ~ 162 million data samples). On SQLite, it takes up a huge amount of space.
HDF5 provides very efficient and compact storage. Also, you can enable various compression algorithms / filters (gzip, bzip, etc.) without too many performance hits.
- I have to quickly read some of this data (from - to timestamps).
This is really the main use case for HDF5. Cutting data from a dataset can easily be done very quickly.
- I should be able to modify parts of the data without having to write all the data from now on.
It is possible to expand the dataset as well as modify the data in-place (this is not as convenient as the UPDATE statement in SQLite). However, there are some caveats regarding data deletion. (see here for more information)
If you have a lot of meta information, you might consider storing it in SQLite and plugging those recods into HDF5 files that contain raw data. Alternatively you can also store this meta information as attributes on your nodes / datasets in HDF5 and avoid using SQLite alltogether.
The only big issue / problem with HDF5 is concurrent write operations. Therefore, if you have a requirement for simultaneous writes in a single HDF5 file, it gets more complicated.
For using HDF5 in .NET, you can check out this thread .
source to share