Burst memory limit (scrapy)
I got a splash from docker. I am creating a large lua script for splash and scrapy and then it runs I see the problem:
Lua error: error in __gc metathet (/app/splash/lua_modules/sandbox.lua:189: script is using too much memory
How can I increase the memory for the burst?
+3
Verz1Lka
source
to share
1 answer
Unfortunately, as of Splash 2.3.2, there is no built-in way to increase these limits. The limitation is hardcoded here: https://github.com/scrapinghub/splash/blob/7b6612847984fc574ebbedf9c3c750180cd93813/splash/lua_modules/sandbox.lua#L176 - you can change the value and then rebuild the Docker image by running docker build -t splash .
Splash from the Docker image then running use this image instead of the image from DockerHub.
+2
Mikhail Korobov
source
to share