Memory error in memory, 256 GB of RAM, 64-bit python and 64-bit memory. ulimit problem?
There are many numpy memory error messages on google land, but I cannot find one that solves my problem. I am running someone else's software using a high performance server with 256GB of RAM, 64-bit opensuse 13.1, 64-bit python, and 64-bit numpy (as far as I can tell). See below.
The original author is not available for help requests, so I did my best to figure out the memory size for the numpy object it is trying to create. First, here's the stack trace:
File "/home/<me>/cmsRelease/trunk/Classes/DotData.py", line 193, in __new__
DataObj = numpy.rec.fromarrays(Columns,names = names)
File "/usr/lib64/python2.7/site-packages/numpy/core/records.py", line 562, in fromarrays
_array = recarray(shape, descr)
File "/usr/lib64/python2.7/site-packages/numpy/core/records.py", line 400, in __new__
self = ndarray.__new__(subtype, shape, (record, descr), order=order)
MemoryError
I used the following for a loop to estimate the size of an object as far as I know how:
size = 0
for i in Columns: # Columns is the list passed into numpy.rec.fromarrays
size += sys.getsizeof(i)
print "Columns size: " + str(size)
Result Columns size: 12051648
. If I'm not mistaken, it's only 12MB, but it's far from 256GB anyway.
Based on this information, I suspect there is a system limit (ulimit) preventing python from accessing memory. Startup ulimit -a
reports the following (I installed ulimit -s 256000000
before running the program):
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 2065541
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 10000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 256000000
cpu time (seconds, -t) unlimited
max user processes (-u) 2065541
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Questions
- What am I missing?
- Couldn't you measure the size of the list object correctly
Columns
? - Is there any other system property that I need to set?
I would like to Memory Error
be more specific. I appreciate your help.
Supportive system information :
System memory:
> free -h
total used free shared buffers cached
Mem: 252G 1.6G 250G 4.2M 12M 98M
-/+ buffers/cache: 1.5G 250G
Swap: 2.0G 98M 1.9G
OS version:
> cat /etc/os-release
NAME=openSUSE
VERSION="13.1 (Bottle)"
VERSION_ID="13.1"
PRETTY_NAME="openSUSE 13.1 (Bottle) (x86_64)"
ID=opensuse
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:opensuse:13.1"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://opensuse.org/"
ID_LIKE="suse"
Python version:
Python 2.7.6 (default, Nov 21 2013, 15:55:38) [GCC] on linux2
>>> import platform; platform.architecture()
('64bit', 'ELF')
Multiple species version:
>>> numpy.version
<module 'numpy.version' from '/usr/lib64/python2.7/site-packages/numpy/version.pyc'>
>>> numpy.version.version
'1.7.1'
source to share