What is the reason for Python memory explosion?

I have a piece of code in a larger program that looks like this (variables renamed to protect the innocent):

import numpy as np
import sympy as sym
from sympy import lambdify
from sympy.core.cache import clear_cache()
#Empty array for later use. 

myarray = [[None for i in range(N)] for j in range(N)]

for i in range(N):
    for j in range(N):
        derivative = sym.diff(somearray[i], someotherarray[j])
        numericalderivative = lambdify(variablearray, derivative)
        myarray[i][j] = numericalderivative

      

When this is done, it slowly raises the balloons to an absolutely massive amount of storage (around 32GB, possibly higher). I expect the end result to take up a lot of memory (N is about 455, so there will be 207,025 function objects in the final array, each with a significant number of terms). However, it shouldn't take up that much memory. In fact, the final memory is much less than its maximum value during the loop: more than 2 GB.

I've also tried to collect manual garbage collection at the end of the second loop:

del derivative
del numericalderivative
clear_cache()
gc.collect()

      

I used to do this with a list, but I switched to a loop to see if it would solve memory problems, for example by making it easier to insert garbage collection commands when needed. However, a change in the comprehension of the list appears to change the manifestation of the problem: RAM appears in allocated RAM in Windows, not RAM.

Possible reasons:

  • Hacking Sympy cache. Sympy is known for its bulky cache. But if that were the reason, would you surely delete the symbolic object at the end of each loop and clear Sympy's cache? This is not true.
  • Exploding Python's free list. Python can create free lists for intermediate objects. Surely deleting both objects at every step of the loop and manual garbage collection will deal with this though?
  • The list is just long. Not at all likely, though: the memory drops to a much lower value when the loop is complete / the array is executed as generated (in previous versions).

This issue occurs in Python 2.7 (Anaconda), both Windows and Linux.

What could be causing this temporary memory explosion?

+3


source to share





All Articles