Why can I access seemingly arbitrary memory addresses in Python?
While playing with strides in NumPy I realized that you can easily traverse the boundaries of arrays:
>>> import numpy as np
>>> from numpy.lib.stride_tricks import as_strided
>>> a = np.array([1], dtype=np.int8)
>>> as_strided(a, shape=(2,), strides=(1,))
array([ 1, -28], dtype=int8)
In a similar way, I can read bytes outside the array, as well as write to them. But I don't understand how this is possible. Why doesn't the operating system stop me? I seem to be able to walk at least 100KB from this array before being thrown Segmentation fault
.
The only thing I can think of is that this memory space is directly allocated by my Python process. Does NumPy do this? Is there a fixed size for this space? What other objects can there be?
source to share
There are two different memory allocators here:
- An operating system available under Unix, for example.
brk(2)
ormmap(2)
. This will usually give you exactly what you are asking for, but it is not very convenient. - A bunch of C runtime available with
malloc(3)
andfree(3)
. It may or may not immediately return the freed memory to the operating system. He may also round the allocation to the nearest page if that is more effective. This is usually done in terms of (1).
Most applications, including NumPy and Python, use (2), not (1) (or they implement their own memory allocator on top of (2)). As a result, memory that is invalid according to (2) may still operate according to (1). You only get a segfault if you break the rules of method (1). It is also possible that you are interacting with other living objects on the heap, which has a high chance that your program will misbehave even if you are not changing anything .
source to share
Python and numpy are built with C, which has no built-in memory. Memory is allocated on the heap, which is a large block of memory. Since all the objects are allocated there, the memory area is quite large and filled with any objects. Writing to this memory will likely crash your program.
source to share