Python memory problems with matplotlib.animation

I simulate diffusion in a complex system, take arbitrary patterns as a substrate and allow arbitrary creation of diffusion fronts and allow both surface reactions and deposition of new material onto the original substrates. I am still proud of the results and you can watch the films I made here for CVD and SFD deposition on particles.

CVD Movie

SFD Movie

Unfortunately I cannot create more than 50 frames because it is running out of memory. I've tried cleaning things up as much as possible while simulating, but I think I am missing something. Summarizing:

I start by creating an empty list

ims = []

      

        
        
        
      

    

Then, every time my "simulation" runs, if frame number % frame "rate" == 0

, it generates a frame that:

• displayed using plt.ion()
  
  over plt.draw()
  
  and
• uses ims.append()
  
  to add a rendered graph to an array of animated frames.

Before every frame render, I run plt.clf()

to prevent the graph from showing only with an increase in the number of overlaid graphs.

Without a step, the ims.append()

code consumes between 140 and 170 MB of RAM. At this point, 50 frames are consuming about 1.4 GB of RAM . Obviously this is very limiting. 50 frames is good, but I would like at least 350. It might not be possible with this route, but it involves using memory exclusively from the ims array of roughly 24MB per frame.

The workaround is to create a frame and convert it to a file .svg

or .png

inside a loop and save it to disk. I find this rendering process very intensive, so it often makes the code quite slow. Also, creating 350 PNG files and then manually converting them to video is pretty messy, so I'd like to somehow get it all inside the program itself.

Does anyone have an idea how to reduce memory usage in this code example without having to have to render and write every frame to disk?

In this toy code, I just used a random number generator to populate two datasets as described in the comments to speed things up.

Code:

import matplotlib.pyplot as plt
import matplotlib.animation as anim
from numpy import *
from matplotlib import *
import time

# Defines the number of frames of animation to render.
outputframes = 50

# Defines the size of the canned simulation.
nx = 800
ny = 800

# Defines the number of actual simulation timesteps
nt = 100

# This gets the number of timesteps between outputframes.
framestep = 2

# For reporting.
framenum = 0

# Creates two steps, one for the stepped simulated step,
# and one for the prior state. There are two independently
# changing materials, each of which will have half the simulation
# space containing random values here, plus 10% overlap in the
# middle.

p1 = zeros((nx, ny, 2))
p1[360:800,:,0] = random.rand(440, ny)
p2 = zeros((nx, ny, 2))
p2[0:440,:,0] = random.rand(440, ny)

# Animation colormap setup
norm = colors.Normalize(vmin=0, vmax = 1)
# And sets up two corresponding colormaps, one blue and one
# red for p1 and p2 respectively (goal is overlaid).
cmap1 = cm.Blues
cmap2 = cm.Reds

# Sets up an empty array to hold animation frames.
ims = []

# Sets up and uses ion to draw the figure without blocking.
plt.ion()
fig = plt.figure()
plt.draw()

# Run the simulation.

for t in range(nt):
    # This looks to see how far we are, and if we're at a point
    # where t is an even multiple of framestep, we should render
    # a new frame.

    if (t%framestep == 0):
        print('Frame ' + str(framenum))
        framenum = framenum + 1
        plt.clf()
        # In here I did a bunch of stuff to get special colors in
        # the colormap to get substrates and surfaces and other
        # features clearly identified. I am creating a new frame1
        # and frame2 object because in reality I will be doing a
        # log plot math to convert to the graphic frame.
        frame1 = p1[:,:,0]

        # This part is necessary in my real program because
        # I manually modify the colormap after it created
        # to include the above mentioned special colors.
        frame1_colors = cmap1(norm(frame1))

        # This is my (not quite right) attempt to do overlaid plots.
        plt.imshow(frame1_colors, alpha = 0.5)

        # Do the same for the second set of data.
        frame2 = p2[:,:,0]
        frame2_colors = cmap2(norm(frame2))

        # The goal here was to take the combined output and make
        # it into an animation frame to append to ims, the image
        # array.

        # This is where I start to run into problems. Without the
        # ims.append, the program has constant memory usage. With
        # it, I am using 1340MB by the 50th frame. This is the
        # biggest issue. Even throwing away all other simulation
        # data, this image array for animation is *enormous*.

        # With the ims.append line replaced with the plt.imshow
        # line alone, memory usage is much smaller, ranging from
        # 140-170MB depending on execution point, but relatively
        # constant.

        ims.append([plt.imshow(frame2_colors, alpha = 0.5)])
#        plt.imshow(frame2_colors, alpha = 0.5)

        # Then try to draw updating animation to show progress
        # using draw(). As best I can tell, this basically works,
        # in that the plot is displaying with all components.
        plt.draw()


    # I'll put in a timer so that this doesn't go too fast, since
    # the actual calculation is very complex.
    time.sleep(0.01)

    # Proxy for the actual calculation. Just overwrite with new
    # random data in the overlapping ranges to show some change
    # visually.
    p1[360:800,:,1] = random.rand(440, ny)
    p2[0:440,:,1] = random.rand(440, ny)

    # In this version, it is trivial, but in the real simulation
    # p1[:,:,1] does not end up equal to p1[:,:,0], so the following
    # resets the simulation for the next timestep, overwriting the
    # old values to avoid memory overflow from the p1 and p2 arrays
    # being enormous.

    # Copy new values into old values.
    p1[:,:,0] = p1[:,:,1]
    p2[:,:,0] = p2[:,:,1]

# This is just a repeat for the final frame.
plt.clf()
frame1 = p1[:,:,0]
frame1_colors = cmap1(norm(frame1))
plt.imshow(frame1_colors, alpha = 0.5)
frame2 = p2[:,:,0]
frame2_colors = cmap2(norm(frame2))

# As above, the ims.append uses tons of memory, the imshow alone works well.
ims.append([plt.imshow(frame2_colors, alpha = 0.5)])
# plt.imshow(frame2_colors, alpha = 0.5)

plt.draw()

anim = anim.ArtistAnimation(fig, ims, blit=True)
anim.save('test.mp4', fps=10, writer='avconv')