How to stop spider scanning and closing based on condition?
I have a spider that fetches the latest url based on a specific date range from a paginated web page. When it gets all the latest urls, my spider should be closed.
How to close a spider?
I asked the question: Power to stop the spider
But I don't like making an exception to close the spider.
Is there any other way to achieve the same?
+3
source to share
1 answer
You must use the Close Spider extension .
Spider closing conditions can be configured using the following settings:
CLOSESPIDER_TIMEOUT
CLOSESPIDER_ITEMCOUNT
CLOSESPIDER_PAGECOUNT
CLOSESPIDER_ERRORCOUNT
+2
source to share