A REALLY huge 3D array in C ++
The thing is, I'm writing some kind of dumb program that takes a handcrafted BMP image of a handcrafted sound wave and converts it to real sound. It works great, but I'm looking for a good way to load the entire image into memory. I am currently using std :: vector to load a 500MB image only to see in the task manager that the process has allocated so much of my 8GB of RAM that it needs extra paging (stopped at about 6GB). I guess this is overhead ...
And here's my question: is there a more memory efficient way to allocate huge 3D arrays? And is there an efficient way to read huge file byte byte byte? (I'm writing this while waiting for it to load, it's 85% in ~ 50 minutes)
CHANGE OS - 64-bit Windows. And that's even more than 6 GB, the RAM was too small for that, so Windows started to put the array in the paging file (I made a small percentage, and at that point the allocation was about 80%).
and here's the code i am using to highlight
vector <vector<vector<char> > > raster_data; //declaration
raster_data.resize(width);
for(int i=0; i<width; i++)
{
raster_data[i].resize(height);
for(int j=0; j<height; j++)
{
raster_data[i][j].resize(3); //(24bpp=3Bpp)
}
}
so I write data not through push_back (), but by referring to the vector, like a normal array:
raster_data[i][j][k] // in a 3d for loop
source to share
What you call a 3D array is more of a 2D array containing RGB elements, and 500Mb is not really "REALLY huge".
When used std::vector
to store only 3 bytes ( char
), one should avoid:
-
std::vector
if it consumes much more than 3 bytes (you can find out how much by checking the resultsizeof(std::vector<char>)
. - the memory you allocate is fragmented
Traditionally, for your use case that loads a 2D bitmap, memory is allocated in one block and then the values are accessed using the computed index.
Overly simplistic, it would be something like:
struct rgb { // you should check that sizeof(rgb) == 3*sizeof(char), just to be sure
char channels[3];
};
std::vector<rgb> raster_data;
raster_data.resize(width*height);
// channel access :
raster_data[j*width+i][k] = ...;
You could read the entire contents of the file efficiently in a buffer like this, but you would need to consider the alignment for each line and from the top of the head, I can't tell what that is (I believe each line is aligned on a 4 byte boundary).
To learn more about BMP bitmap data processing in C / C ++, I highly recommend you take a look at Handmade Hero, which covers the topic from the beginning, I believe in this episode: https://www.youtube.com/watch ? v = ofMJUSchXwo
source to share