Clever Geek Handbook

How do I set the burst time during a burst?

I am using scrapy-splash to crawl a webpage and run a service splash on docker.

commond:

docker run -p 8050:8050 scrapinghub/splash --max-timeout 3600

But I got a 504 error.

"error": {"info": {"timeout": 30}, "description": "Timeout exceeded rendering page", "error": 504, "type": "GlobalTimeoutError"}

Although I'm trying to add splash.resource_timeout

, request:set_timeout

or SPLASH_URL = 'http://localhost:8050?timeout=1800.0'

, nothing has changed.

Thanks for the help.

+8


source to share


2 answers


I am using a package scrapy-splash

and set the timeout in a parameter args

SplashRequest

like this:

yield scrapy_splash.SplashRequest(
    url, self.parse, endpoint='execute',
    args={'lua_source': script, 'timeout': 3600})


This works for me.

+10


source


Where can we set splash.resource_timeout, request: set_timeout parameters in scrapy-splash? The above answer is the same as running docker -p 8050: 8050 scrapinghub / splash --max-timeout 3600, isn't it?



0


source






More articles:


All Articles