How do I make the Mixnode scanner scan slower?
We exceeded 20 million pages per hour and I really appreciate the speed; however I am afraid that I might put too much pressure on the targeted sites, is there a way to decrease the speed at which websites are crawled?
+3
J. Dorian
source
to share
1 answer
Not sure why you want to decrease the speed, as the documentation clearly states that:
There is a minimum delay of 10 seconds between requests sent to the same site . If a website's robots.txt directives require longer latency, Mixnode will follow the length of the delay specified by the robots.txt directives.
+3
EnJayz
source
to share