Is it more suitable for my task: background worker or thread pool?
I have a simple web application module that basically accepts requests to save a zip file to PageLoad
from a mobile client application.
Now what I want to do is unzip the file and read the file inside it and process it further .. including writing to the database.
Update: the zip file and its contents will be smaller, so the server should not be burdened with a heavy load.
Update 2: I just read about when IIS queue requests (at global / application level). Does this mean that I don't need to implement a complex request mechanism and IIS can take care of the application on its own?
Update 3: I'm looking for offloading the processing of a loaded zip not only for the sake of minimizing the overhead (from a performance standpoint), but also to avoid the problem table-locking
when a file is being processed and records are updated to the same table. In a scenario of multiple devices requesting concurrent database refresh and background task processing, you will get an exception.
At the moment, I am zeroed out on two solutions:
- To implement parallel / message queue
- To inject the file handling code into a separate tool and schedule a job on the server to inspect the unprocessed files and process them one at a time.
Slope to Queuing Mechanism
I will try to implement since it seems less config dependent. v / s manually configures the job / schedule on the server side.
So what do you guys recommend me for this purpose?
Also, after the zip file is requested and saved on the server side, then the client and server connection will be released. Not wanting to burden IIS.
Imagine a couple hundred customers requesting a page at the same time.
I haven't actually used any of these before any samples or instructions would be more appreciated.
source to share
I would suggest a queuing system.
When you have received the file, you will save the path in a synchronized thread queue. In the meantime, the background worker (or preferably another computer) will check this queue for new files and remove the entry to process it.
This way you won't run an unknown number of threads (each zip file) and can process zip files in one place. This way you can also easily move your zip processing code to another computer when the download gets too heavy. You just need to access the shared queue.
The simplest is probably to use static Queue
with an lock
-object. This is the easiest to implement and does not require external resources. But this will cause the queue to be lost when you reuse your application.
You mentioned that losing zip files is not an option, then this approach is not the best one if you don't want to rely on external resources. Depending on your load, external resources might be required - this means you download the zip file to shared memory on another machine and add a message to the queue on the other machine.
Here's an example with a local queue:
ConcurrentQueue<string> queue = new ConcurrentQueue<string>();
void GotNewZip(string pathToZip)
{
queue.Enqueue(pathToZip); // Added a new work item to the queue
}
void MethodCalledByWorker()
{
while (true)
{
if (queue.IsEmpty)
{
// Supposedly no work to be done, wait a few seconds and check again (new iteration)
Thread.Sleep(TimeSpan.FromSeconds(5));
continue;
}
string pathToZip;
if (queue.TryDequeue(out pathToZip)) // If TryDeqeue returns false, another thread dequeue the last element already
{
HandleZipFile(pathToZip);
}
}
}
This is a very crude example. Whenever the zip appears, you add the path to the queue. Meanwhile, a background worker (or several, s threadsafe example) will process one zip after another, getting the paths from the queue. Zip files will be processed in the order in which they arrive.
You need to make sure that your application is not being recycled in the meantime. But in this case, with all the resources you have on your local machine, they will be lost when your machine crashes.
source to share
I believe you are optimizing prematurely.
You mentioned stop blocking - what dB are you using? If you add new lines or update existing ones, most modern databases in most configurations will:
- use row-level locking; and
- be fast enough not to worry about the lock.
I suggest starting with a simple method
//Unzip
//Do work
//Save results to database
and get some proof, it's too slow.
source to share