How file association works
I am writing a program that needs to traverse a large 40GB file, but I only have 16GB of physical memory. A friend told me that I can use file mapping to work around this problem. I understand how to create a file mapping and read into a file map descriptor , and how a file mapping maps parts of a file in persistent memory to different chunks of virtual memory for reading.
So, if I understand this correctly, I can create a buffer, say 10gb, and read the first 10gb of the file in that buffer. But when I need to read a missed 10gb tag in a file, will the OS choose another block automatically for me or will I have to manually do it in my code?
source to share
The features you linked are not (directly) related to file mapping. They are used for regular file I / O.
To use traditional file I / O with a really large file, you can do it as described. You will open a file, create a buffer, and read a chunk of the file into the buffer. Whenever you need to access another part of the file, you read another part into the buffer.
To use file mapping, you use CreateFile, CreateFileMapping and then MapViewOfFile . You are not creating (directly) a buffer and reading part of the file. Instead, you tell the system that you want to access the range of the file as if it were a range of memory addresses. Reading and writing to these addresses becomes file I / O behind the scenes. With this approach, you still have to work in chunks. If the part of the file that you want to access is not in the range you are currently mapping, you can create a different view (and possibly close another).
But notice that I said address space is different from RAM. If you are building 64-bit Windows, you can try to map the entire 40GB file in your address space. The fact that you only have 16GB of RAM won't stop you. There may be other problems of this size, but it's not because of your RAM. If there are other problems, you'll be back to managing the file in chunks as before.
source to share