Cython extension type inheriting from int raises MemoryError
I am trying to make an extension type inherit from int or cython.int. This is necessary for me as I need to be able to use this type as an index for some lists / arrays.
Here is the code to reproduce the error on Python 2.7.9 Win32 (I am running Windows 7) with Cython v0.22 on Anaconda:
import cython
cimport cython
import sys
cdef class ExtendedInt(int):
#cdef object __weakref__ # explicitly enable weakref for this extension type, will self-destruct when it is no longer strongly referenced.
def __add__(a, b):
return a+b
# Simple test case
def main():
total_it = 1000
for i in xrange(total_it):
for j in xrange(10000000):
ExtendedInt(j)
sys.stdout.write("\rGenerating lists of ExtendedInt : %i/%i" % (i, total_it))
What happens is that if you try to create many ExtendedInt's, at some point the Python interpreter will crash with a MemoryError. With the above code snippet on my 4GB machine, it crashes on the 11th iteration, but that will depend on the specs of the machine. I tried to enable weakref, but that doesn't fix the problem.
However, if you replace the class cdef ExtendedInt (int): "just" class ExtendedInt (int) "(so it is no longer an extension type, just a Python class), then the problem does not occur, there is no memory error.
So it seems like extension types that inherit from integers are not deallocated correctly even if I include weakref . Is this an error that I have to fill in on the tracker or am I missing something?
Is this a bug with the current version of Cython? I could not find links to bugtracker ... Or am I missing something that could solve the problem?
source to share
Stefan Behnel discovered that the bug is in Python 2.7 inside intobject.c :
Your extension type automatically inherits the "int_free" slot from its base type, so the "else" case in "int_dealloc ()" will actually call "int_free ()" and add the object to the free list even though it was not exactly the same same as PyInt. Then, when creating a new ExtendedInt instance, the Python subtype intromentation code ignores the free list and instead creates a completely new object.
Thus, the free list continues to grow until it fills all memory and the process dies. I created a CPython ticket.
Here's a link to the CPython ticket .
Here's the answer from Guido Van Rossum:
The suggested solution is to require int subclasses to override tp_free.
Note that most of this code is pretty much written, assuming that subclasses are created using class statements (in regular Python module, not Cython) that take care of all these details. This does not mean that Cython is mistakenly trying to do this, but it does mean that there is not much documentation, and it also means that I don’t think you are reporting that this is a bug in CPython.
So I'm not sure if this will be fixed or not, or if Cython does a workaround. Anyway, for those facing the same problem as me, you can get around this problem by inheriting from the object and then just override magic methods for integers like int , add index , etc. like I have explained a specific example here .
source to share