SSD read for large file using C ++
I am working on a Windows 10 64-bit machine, 6850K processor and 64GB DDR4 RAM, and the Samsung SSD connects via M.2. I want to read a file of about 15 GB in memory. I am currently using fstream to read an entire file into an unsigned character array using a single call to its read function. However, the speeds I achieve do not reach the maximum read speed of the SSD (1500MB / s when reading an SSD around 3500MB / s).
So I was wondering if there is a faster way? Would it be faster if I made multiple read requests for small chunks? If so, what is the optimal chunk size? I have seen some people mention 4K reads in some of the previously asked questions. Does this apply in this case?
Any help is appreciated.
My code snippet looks like this
my code read looks like this
fstream myFile;
myFile.open("file", ios::binary | ios::in);
myFile.read(reinterpret_cast<char*>(buf), 14929920000LL);
where buf
is the same size as read.
source to share
To get the maximum read speed, you need to bypass the Windows disk cache. Use Windows API calls CreateFile
, ReadFile
etc. And use unbuffered reads (pass FILE_FLAG_NO_BUFFERING
to CreateFile
). This will transfer data directly from disk to the block of memory you need, without having to copy data from one memory address to another.
You need to pay close attention to the necessary memory alignment needs that can be imposed by hardware. This usually requires memory addresses to be 512 bytes aligned, but some newer hardware may require 4096 bytes and others may not be as strict. The link in the CreateFile documentation gives complete details for FILE_FLAG_NO_BUFFERING
.
source to share