Pyramid and matplotlib - can't there be more than one kind of SVG output?
I am developing a Pyramid Python application where I intend to create more than one SVG image for statistics charts using pie charts. In my testing, I found that one SVG view works correctly, and as soon as I add the second SVG output and the second SVG image is loaded (the order in which the SVG image is loaded does not matter), whether directly through its view or through another view referencing this view, SVG images are "concatenated" into any other calls to load the SVG file. This appears to be a bug somewhere in the Python stack, as it looks like memory is not being cleared properly (primarily in the case of more than one SVG file, see further details below). Also note that after the image / page has loaded enough,TclError is encountered.
Since I have used SVG in a more verbose application with many other views, I reproduce this in a minified / minified application to show that this is not something extra that I am doing and this code is generated directly from pyramid alchemist templates and database calls not involved. The database is heavily used in my more verbose application. This app only has 3 views, where the first view is part of the original template. I also add DEBUG logging to make it clear that there is no indication that there is an internal call to another SVG view.
Some of the presentation codes are based on Matplotlib svg as a string, not a file mainly for use StringIO
. Please note that since the pie chart is needed this is the main reason my code is different from the code in the above question. I believe the problem is essentially the same whether I use StringIO
or cStringIO
. In my code, I am using cStringIO
.
The complete application code is available at: https://github.com/danielpronych/pyramidapp
PyPlot Documentation :: pyplot api
Update: code from first SVG view (curly handles and close command):
@view_config(route_name='view_test_svg')
def test_svg_view(request):
# Full module import is not allowed by Pyramid
#from pylab import *
# Do individual required imports instead
from pylab import close, figure, axes, pie, title, savefig
# For clarity, note that the above, and below, function the same
#from matplotlib.pyplot import close, figure, axes, pie, title, savefig
log.debug('In test_svg_view')
fig = figure(1, figsize=(6,6))
ax = axes([0.1, 0.1, 0.8, 0.8])
labels = ['Frogs', 'Hogs', 'Dogs', 'Logs']
fracs = [15, 30, 45, 10]
explode=(0, 0.05, 0, 0)
pie(fracs, explode=explode, labels=labels,
autopct='%1.1f%%', shadow=True, startangle=90)
title('Raining Hogs and Dogs', bbox={'facecolor':'0.8', 'pad':5})
imgdata = cStringIO.StringIO()
fig.savefig(imgdata, format='svg')
imgdata.seek(0)
svg_dta = imgdata.getvalue()
# Close the StringIO buffer
imgdata.close()
close('all')
return Response(svg_dta, content_type='image/svg+xml')
Steps for this code :
- Download http: // localhost: 6543 / test.svg
- Download http: // localhost: 6543 / test2.svg
- Download either (1) or (2) again.
pserve
will "spin" continuously unsigned from a command prompt window.
Note. Loading the same SVG view 3 times with this code also has the same result as the steps above which do not work with the code below.
Also note that the pie pie has a return; but not a "pen" per se.
- Window
-
pserve
does not respond to Control + C commands, even after waiting a minute (especially after doing the above).
Code from first SVG view (original, steps given below, from this code):
@view_config(route_name='view_test_svg')
def test_svg_view(request):
# Full module import is not allowed by Pyramid
#from pylab import *
# Do individual required imports instead
from pylab import figure, axes, pie, title, savefig
# For clarity, note that the above, and below, function the same
#from matplotlib.pyplot import figure, axes, pie, title, savefig
log.debug('In test_svg_view')
figure(1, figsize=(6,6))
ax = axes([0.1, 0.1, 0.8, 0.8])
labels = ['Frogs', 'Hogs', 'Dogs', 'Logs']
fracs = [15, 30, 45, 10]
explode=(0, 0.05, 0, 0)
pie(fracs, explode=explode, labels=labels,
autopct='%1.1f%%', shadow=True, startangle=90)
title('Raining Hogs and Dogs', bbox={'facecolor':'0.8', 'pad':5})
imgdata = cStringIO.StringIO()
savefig(imgdata, format='svg')
imgdata.seek(0)
svg_dta = imgdata.getvalue()
# Close the StringIO buffer
imgdata.close()
return Response(svg_dta, content_type='image/svg+xml')
Python version: Python 2.7.5
Python package configuration (primary packages only)
- pyramids 1.6a1-py2.7
- Matplotlib-1.4.3-win32-py2.7
Steps taken to reproduce:
- pserve pyramidapp.
Command: pserve development.ini --reload
Starting server in PID 4912.
serving on http://0.0.0.0:6543
- Download http: // localhost: 6543 / test.svg
Please note that this works correctly
DEBUG [pyramidapp.views:22][Dummy-2] In test_svg_view
- Download http: // localhost: 6543 / test2.svg
Note that this "concatenates" both SVG files together
DEBUG [pyramidapp.views:45][Dummy-3] In test2_svg_view
- Download http: // localhost: 6543 / test.svg
Note that this works exactly the same as test2.svg, with the correct title, since they are also the same length and now the images are also merged in this view
DEBUG [pyramidapp.views:22][Dummy-4] In test_svg_view
- Apply app and download only http: // localhost: 6543 / test2.svg
Note that this works fine for the first load as this view was loaded before test.svg this time
DEBUG [pyramidapp.views:45][Dummy-2] In test2_svg_view
Tracelog when using Control + C to terminate the pserve process
Error in sys.exitfunc:
Traceback (most recent call last):
File "--python_path--\lib\atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "--python_path--\lib\site-packages\matplotlib-1.4.3-py2.7-win32.egg\ma
tplotlib\_pylab_helpers.py", line 89, in destroy_all
manager.destroy()
File "--python_path--\lib\site-packages\matplotlib-1.4.3-py2.7-win32.egg\ma
tplotlib\backends\backend_tkagg.py", line 588, in destroy
self.window.destroy()
File "--python_path--\lib\lib-tk\Tkinter.py", line 1789, in destroy
for c in self.children.values(): c.destroy()
File "--python_path--\lib\lib-tk\Tkinter.py", line 2042, in destroy
self.tk.call('destroy', self._w)
_tkinter.TclError: out of stack space (infinite loop?)
^C caught in monitor process
Important: After sufficient loading of SVG images, the following occurs:
The only way to fix this currently is to restart pserve
. Also note that views such as my_view
are loaded as expected as long as SVG images are not referenced or used by those views.
Another important note if only one SVG file, i.e. http: // localhost: 6543 / test.svg , loading all the time pserve
, it seems like the image can be reloaded / refreshed (potentially) indefinitely without any obvious problems, or encounters the following:
_tkinter.TclError
TclError: out of stack space (infinite loop?)
Traceback (most recent call last)
File "--python_path--\lib\site-packages\pyramid_debugtoolbar-2.0.2-py2.7.egg\pyramid_debugtoolbar\panels
\performance.py", line 69, in noresource_timer_handler
Display the sourcecode for this frameOpen an interactive python shell in this frameresult = handler(request)
File "--python_path--\lib\site-packages\pyramid-1.6a1-py2.7.egg\pyramid\tweens.py", line 20, in excview_tween
Display the sourcecode for this frameOpen an interactive python shell in this frameresponse = handler(request)
File "--python_path--\lib\site-packages\pyramid_tm-0.11-py2.7.egg\pyramid_tm\__init__.py", line 94, in tm_tween
Display the sourcecode for this frameOpen an interactive python shell in this framereraise(*exc_info)
File "--python_path--\lib\site-packages\pyramid_tm-0.11-py2.7.egg\pyramid_tm\__init__.py", line 75, in tm_tween
Display the sourcecode for this frameOpen an interactive python shell in this frameresponse = handler(request)
File "--python_path--\lib\site-packages\pyramid-1.6a1-py2.7.egg\pyramid\router.py", line 145, in handle_request
Display the sourcecode for this frameOpen an interactive python shell in this frameview_name
File "--python_path--\lib\site-packages\pyramid-1.6a1-py2.7.egg\pyramid\view.py", line 527, in _call_view
Display the sourcecode for this frameOpen an interactive python shell in this frameresponse = view_callable
(context, request)
File "--python_path--\lib\site-packages\pyramid-1.6a1-py2.7.egg\pyramid\config\views.py", line 384, in
viewresult_to_response
Display the sourcecode for this frameOpen an interactive python shell in this frameresult = view(context,
request)
File "--python_path--\lib\site-packages\pyramid-1.6a1-py2.7.egg\pyramid\config\views.py", line 506, in
_requestonly_view
Display the sourcecode for this frameOpen an interactive python shell in this frameresponse = view(request)
File "c:\projects\python\pyramid\pyramidapp\pyramidapp\views.py", line 55, in test2_svg_view
Display the sourcecode for this frameOpen an interactive python shell in this framesavefig(imgdata,
format='svg')
File "--python_path--\lib\site-packages\matplotlib-1.4.3-py2.7-win32.egg\matplotlib\pyplot.py", line 578, in
savefig
Display the sourcecode for this frameOpen an interactive python shell in this framedraw() # need this if
'transparent=True' to reset colors
File "--python_path--\lib\site-packages\matplotlib-1.4.3-py2.7-win32.egg\matplotlib\pyplot.py", line 571, in
draw
Display the sourcecode for this frameOpen an interactive python shell in this frameget_current_fig_manager
().canvas.draw()
File "--python_path--\lib\site-packages\matplotlib-1.4.3-py2.7-win32.egg\matplotlib\backends\backend_tkagg.py",
line 350, in draw
Display the sourcecode for this frameOpen an interactive python shell in this frametkagg.blit(self._tkphoto,
self.renderer._renderer, colormode=2)
File "--python_path--\lib\site-packages\matplotlib-1.4.3-py2.7-win32.egg\matplotlib\backends\tkagg.py", line
24, in blit
Display the sourcecode for this frameOpen an interactive python shell in this frametk.call("PyAggImagePhoto",
photoimage, id(aggimage), colormode, id(bbox_array))
TclError: out of stack space (infinite loop?)
Update (September 2015): I was hoping to see another update from Sergey as his suggested solution really helped in the beginning; however, the problem was not resolved with his solution as no one happened, although I waited a long time, I found that only my solution actually solved the problem. Finally, to be clear, my solution works in web implementation and in batching, as this issue also occurs in batching.
source to share
Sergei's answer was definitely helpful. I was able to find that by adding some more snippets of code, based on parts of this example , including the operator matplotlib.use
, the memory issues seem to be resolved. I've tested this in both my test app (2 views generating an SVG file posted above) and my main app, which currently has 3 SVG views, which I will expand to more views by creating SVG files.
In my main application, I can update between all 3 existing views multiple times, without combining the SVG image I found with previous versions of this code. This parameter seems to be necessary for more concurrent purposes, so maybe other SO questions don't reflect this issue, unless I can find something else that is causing this issue. I'm not sure if this is the best answer; however, this is working correctly at the moment and I plan to monitor if any other issues are found.
@view_config(route_name='view_test_svg')
def test_svg_view(request):
from matplotlib import use as matplotlib_use
matplotlib_use("Svg")
from matplotlib.pyplot import close, figure, axes, pie, title, savefig
log.debug('In test_svg_view')
fig = figure(1, figsize=(6,6))
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])
labels = ['Frogs', 'Hogs', 'Dogs', 'Logs']
fracs = [15, 30, 45, 10]
explode=(0, 0.05, 0, 0)
pie(fracs, explode=explode, labels=labels,
autopct='%1.1f%%', shadow=True, startangle=90)
title('Raining Hogs and Dogs', bbox={'facecolor':'0.8', 'pad':5})
imgdata = cStringIO.StringIO()
savefig(imgdata, format='svg')
imgdata.seek(0)
svg_dta = imgdata.getvalue()
# Close the StringIO buffer
imgdata.close()
close('all')
return Response(svg_dta, content_type='image/svg+xml')
source to share
I'm not familiar with pylab, but it looks like the module has some kind of global state that accumulates between calls (note how you call savefig()
without passing it any handle to the diagram created by previous calls - but for some reason it should know what has been drawn, so I guess subsequent calls just accumulate in the same "buffer", which explains the behavior you are seeing.
Here is an excerpt from the docs (emphasis mine):
Hence everything in matplotlib is organized in a hierarchy. At the top of the hierarchy is the matplotlib state environment , which is provided by the matplotlib.pyplot module. At this level, simple functions are used to add plot elements (lines, images, text, etc.) to the current axes at the current value .
Note. The Pylotts state-machine environment behaves similarly to MATLAB and should be most familiar to users with MATLAB experience. The next level in the hierarchy is the first level of the object-oriented interface, in which pyplot is only used for a few functions, such as creating a shape, and the user explicitly creates and keeps track of shape and axis objects. At this level, the user uses pyplot to create shapes, and through these shapes one or more axis objects can be created. These axis objects are then used for most action plotting.
For even more control - which is needed for things like embedding matplotlib Plots in GUI applications - the piplet layer can be removed entirely, leaving a purely object-oriented approach .
This "state machine" approach might be favored for a "linear" script that handles some data and outputs, I don't see how it might work in a web environment (consider parallel requests, etc.)
I am wondering if using "raw" Matplotlib (called the "object oriented approach" in their docs) will allow you to avoid the problem - from what I see, its API is more traditional - you instantiate a Class class and call its methods for of creating your diagram, so all state is encapsulated within that instance and not persisted between requests.
Here's an example in the Matplotlib docs: agg_oo.py . Note that it completely avoids using pylab and pyplot (the latter actually implements the "state machine layer").
I would imagine the code might look something like this (untested):
from matplotlib.figure import Figure
@view_config(route_name='view_test_svg')
def test_svg_view(request):
fig = Figure(figsize=(6,6))
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])
labels = ['Frogs', 'Hogs', 'Dogs', 'Logs']
fracs = [15, 30, 45, 10]
explode=(0, 0.05, 0, 0)
ax.pie(
fracs, explode=explode, labels=labels,
autopct='%1.1f%%', shadow=True, startangle=90)
ax.set_title('Raining Hogs and Dogs', bbox={'facecolor':'0.8', 'pad':5})
imgdata = cStringIO.StringIO()
fig.savefig(imgdata, format='svg')
imgdata.seek(0)
svg_dta = imgdata.getvalue()
# Close the StringIO buffer
imgdata.close()
return Response(svg_dta, content_type='image/svg+xml')
source to share