Node.js Buffer is not empty
When I create a lot Buffer
, they are not always empty:
for (var i = 0; i < 100; i++) {
console.log((new Buffer(30)).toString('hex'));
}
(Partial) Exit:
782668013a0000003b00000035000000b0c17900391100003c0000003d00
e4216801ffffffff000000000100000000000000000000003e0000003f00
40c27900100000000100000000000000000000000000000018c379000000
000000000000000000000000000000000000000000000000000000000000 --> Empty
000000000000000000000000000000000000000000000000000000000000 --> Empty
0000000000000000108269014000000041000000c86f79000cf679000000
6611000080c27900c0c27900040000000100000000000000d0c279000000
00000000000000005c2468014200000043000000cc6f7900002668014400
(Partial) Exit (without .toString('hex')
and only new Buffer(10)
):
<Buffer 01 00 00 00 58 db 62 00 b4 86>
<Buffer 90 b9 65 00 08 00 00 00 03 00>
<Buffer 10 ba 65 00 04 00 00 00 00 00>
<Buffer 04 00 00 00 00 00 00 00 00 00>
<Buffer 10 00 00 00 00 00 00 00 70 ba>
<Buffer 00 00 00 00 00 00 00 00 00 00> --> Empty
<Buffer 00 00 00 00 00 00 00 00 00 00> --> Empty
<Buffer ff ff ff ff ff ff ff ff 00 00>
<Buffer 00 00 00 00 0f 00 00 00 8c 6f>
<Buffer 80 ba 65 00 00 00 00 00 aa 00>
I am running node.js v0.10.33
on a Windows 7 32 bit machine.
- Is this a problem with my particular car?
- Is this a common problem (bug) of node.js?
- Is this a special composition problem (e.g. Windows-only environments)?
- Is this expected and should I clear
Buffer
before using it? - Is this confirmed?
Update: Better behavior on v0.11.14
, worst behavior onv0.8.28
source to share
TL; DR
- Not a problem with your particular machine, it is on all machines.
- Not a mistake. This behavior exists in all memory allocation libraries.
- OS independent. All operating systems behave this way.
- If you need to initialize it, then yes, clear it up using
Buffer.prototype.fill
- Yes, it is documented in the base libraries that use nodejs / webkit: see malloc (3) / stdlib
This is most likely what you would expect when working with an API that handles memory allocation, such as Buffer.
Buffer
actually uses a module smalloc
you can think of as malloc (3) / free (3) from stdlib. h
The principle of malloc is that it allocates / reserves memory for a pointer - in webkit it can look closer to calloc for ExternalArray on objects.
See http://linux.die.net/man/3/malloc
The malloc () function allocates size bytes and returns a pointer to the allocated memory. The memory is not initialized.
Memory allocation / deallocation does not handle memory initialization because it is more expensive to go through each byte and set it to 0.
Also, 0 is not always the initial value required for a byte.
And since memory allocation can return a block of memory that was previously used by some other process, it expected to have data in the newly allocated block.
A general rule of thumb is: when allocating memory (like a buffer), if you need it , it initialized use buf.fill(0)
; although this is not always necessary, as in most scenarios when you need a Buffer, you already know the length of the data, implicitly the content.
For example, when creating, new Buffer(30)
you know that your data is 30 bytes long, which means that you already have an idea of what the data to be written looks like, so in the end you will most likely find yourself writing each byte before transferring the buffer to something else, so there is no need to add an extra initialization loop that sets each byte to 0.
For buffers where you use bytes as flags / states for an object, you can only initialize them.
eg. if yours is Buffer(10)
using the first 2 bytes as flags for some state and the remaining 8 bytes for data,
buffer[0] = buffer[1] = 0
to set them to an initial value of 0 instead of initializing all 10 bytes, then write on the last 8 anyway.
One final note: if the block of memory returned by the memory allocation has not been used before, then yes, all bytes are set to 0. However, memory handling is what the OS does - using all kinds of optimizations - so it is impossible to predict which segment of memory you will get and its contents.
source to share
In the latest versions of Node (v5.10.0 +) you can use
--zero-fill-buffers
https://nodejs.org/api/buffer.html#buffer_the_zero_fill_buffers_command_line_option
source to share