Memory overhead malloc () vs new []
I want to reserve a block of memory (1GB) to load data into it for analysis. Each entry is about 10k bytes, and the entries are at least 100k. I originally intended to use malloc in C ++ code, but I was advised against it.
Now, using char * block = new char [1000000000], do you need extra memory to store pointers to each of the 1,000,000,000 elements in the array? Will using char * block = malloc (1000000000 * sizeof (char)) take less extra memory to create than a new []?
My goal is to use the least amount of memory and not want to plug up writes to and from memory.
Thank:)
source to share
On my Linux machine:
Malloc
//malloc.cc
#include <cstdlib>
int main() { char* block = (char*) malloc(1000000000); }
Runtime:
$ make malloc
$ valgrind ./malloc 2>&1|grep total
==23855== total heap usage: 1 allocs, 0 frees, 1,000,000,000 bytes allocated
New
//new.cc
int main() { char* block = new char[1000000000]; }
Runtime:
$ make new
$ valgrind ./new 2>&1|grep total
==24460== total heap usage: 2 allocs, 0 frees, 1,000,072,704 bytes allocated
The 72.704B overhead remains the same for different values.
source to share
To operator delete[]
work correctly with non-PODs, the size of the array (one size_t
) is usually placed at the beginning of the entire block, and the first object at the first correctly aligned address.
For a POD operator new[]
(no initializer), it is usually the same as malloc
.
With an initializer (again, with a POD type), the results are compiler-dependent: it can loop over the elements, or reduce to memset
.
Given the large amount of memory you are going to allocate, the results malloc
depend on the runtime. Some implementations have a hard upper limit on the block size.
If you are targeting Windows, you can use VirtualAlloc
for something this size. Likewise, use mmap
on * nix.
source to share
You asked:
Now using
char * block = new char[1000000000]
, do you need additional memory to store pointers to each of the 1,000,000,000 elements in the array?
Definitely not.
From C ++ 11 standard (section 5.3.4 New)
5 When the allocated object is an array (that is, using the syntax noptr-new-declarator or a new type identifier or a type identifier denotes the type of the array), the new expression gives a pointer to the starting element (if any) of the array.
The key part of this is that you return a pointer to the original element (if any) of the array
You also asked:
Will it use
char * block = malloc(1000000000 * sizeof(char))
less memory to create thannew[]
?
The standard does not specify anything about the overhead associated with the use of distribution methods. In most implementations, the memory overhead associated with these two methods should be roughly the same, if not exactly the same. I would be surprised if this is not the case.
source to share
If you use new
to allocate a character array, you get a character array. There will be no additional pointers for each element. You just end up with a large contiguous region of memory, similar to what you would get with malloc()
.
What new
will do is allocate memory and then call the constructor, which in your case won't do anything, since it's just an array of plain old data.
I ran a quick check with Visual Studio 2013 with debug compilation and looked at the memory allocation in Windows Task Manager when I stepped over first new
and then malloc()
and the numbers looked about the same for memory allocation at each step.
With such a large area of memory, you can run into page errors as the operating system moves your large area of memory in and out as it accesses different parts of the memory area. Not sure if you can actually do anything and I'm sure it's a big concern. In part, any paging behavior will depend on the amount of physical memory you have, along with the combination of additional services and applications and their memory usage.
source to share