Clever Geek Handbook

Burst memory limit (scrapy)

I got a splash from docker. I am creating a large lua script for splash and scrapy and then it runs I see the problem:

Lua error: error in __gc metathet (/app/splash/lua_modules/sandbox.lua:189: script is using too much memory

How can I increase the memory for the burst?

+3


source to share


1 answer


Unfortunately, as of Splash 2.3.2, there is no built-in way to increase these limits. The limitation is hardcoded here: https://github.com/scrapinghub/splash/blob/7b6612847984fc574ebbedf9c3c750180cd93813/splash/lua_modules/sandbox.lua#L176 - you can change the value and then rebuild the Docker image by running docker build -t splash .

Splash from the Docker image then running use this image instead of the image from DockerHub.



+2


source






More articles:


All Articles