Should I use BackgroundWorker or Threads when I need to clean up a website?

I am going to screen a game site for some data. I would like to be able to send multiple requests so that I can screen multiple pages at the same time. I emailed the site admin and got permission to clean up at a moderate rate (few requests per second).

As far as I know, BackgroundWorker uses a thread pool, which I think would be desirable.
Does it make sense to use BackgroundWorker for this use case, or use actual streams?

+2


source to share


2 answers


There is another construct known as ThreadPool . It might be worth using this as it will manage multiple threads for you and you can control the min / max of threads. BackgroundWorker is limited to a single thread and is best used for WinForms applications where you have background I / O and do not want to block the user interface thread.



You need to keep the page queue in order to flush and push them to the thread pool. You can still pause or restrict flows to get the level of scrapers you want. I would personally separate the parsing of the resulting page content from the actual search for the pages over HTTP. This tends to make maintenance easier and you may not need to multithreaded local files.

+5


source


Typical use BackgroundWorker

is to maintain the user interface; instead, use a thread pool to queue multiple HTTP requests / responses.



Cm. ThreadPool.QueueUserWorkItem

+2


source







All Articles