Working with multiple HTTP requests
I have a program that looks at a list of applications.
Apps
--------
App1
App2
App3
Now, for each of them, I make an HTTP request to get the list of assemblies for each application as Xml.
So a query like for example
http://example.com/getapplist.do?appid=App1
gives me an answer like
<appid name="App1">
<buildid BldName="Bld3" Status="Not Ready"></buildid>
<buildid BldName="Bld2" Status="Ready"></buildid>
<buildid BldName="Bld1" Status="Ready"></buildid>
</appid>
Now I am getting Highest Build Number with Ready status and then make another web api call like
http://example.com/getapplist.do?appid=App1&bldid=Bld2
This gives me an answer like
<buildinfo appid="App1" buildid="Bld2" value="someinfo"></build>
I am passing them to internal data tables. But now this program is taking quite a long time (3 hours) as I have about 2000 applications and there are 2 web requests for each id. I tried to sort this issue using BackgroundWorker as pointed out here . I was thinking about collating all the information from the http responses into one XML and then using that XML for further processing. This raises an error,
used by another process
So my code looks like,
if (!backgroundWorker1.IsBusy)
{
for(int i = 0; i < appList.Count; i++)
{
BackgroundWorker bgw = new BackgroundWorker();
bgw.WorkerReportsProgress = true;
bgw.WorkerSupportsCancellation = true;
bgw.DoWork += new DoWorkEventHandler(bgw_DoWork);
bgw.ProgressChanged += new ProgressChangedEventHandler(bgw_ProgressChanged);
bgw.RunWorkerCompleted += new RunWorkerCompletedEventHandler(bgw_RunWorkerCompleted);
//Start The Worker
bgw.RunWorkerAsync();
}
}
And the function DoWork
fetches the tag values and puts them in XML.
What's the best way to get the app-buildinfo details into a shared file from all HTTP responses from all background workers?
source to share
HTTP requests are IO-related and asynchronous in nature, there is no reason to use background workers to accomplish what you need.
You can use async-await
which is compatible with .NET 4 via Microsoft.Bcl.Async
and HttpClient
:
private async Task ProcessAppsAsync(List<string> appList)
{
var httpClient = new HttpClient();
// This will execute your IO requests concurrently,
// no need for extra threads.
var appListTasks = appList.Select(app => httpClient.GetAsync(app.Url)).ToList();
// Wait asynchronously for all of them to finish
await Task.WhenAll(appListTasks);
// process each Task.Result and aggregate them to an xml
using (var streamWriter = new StreamWriter(@"PathToFile")
{
foreach (var appList in appListTasks)
{
await streamWriter.WriteAsync(appList.Result);
}
}
}
This way, you process all requests at the same time and process the results of all of them after they are completed.
source to share
This solution works for .Net 2.0 and above, using async methods from the WebClient class and using a counter that is decremented with the Interlocked class and a regular lock to serialize the writing of the results to a file.
var writer = XmlWriter.Create(
new FileStream("api.xml",
FileMode.Create));
writer.WriteStartElement("apps"); // root element in the xml
// lock for one write
object writeLock = new object();
// this many calls
int counter = appList.Count;
foreach (var app in appList)
{
var wc = new WebClient();
var url = String.Format(
"http://example.com/getapplist.do?appid={0}&bldid=Bld2",
app);
wc.DownloadDataCompleted += (o, args) =>
{
try
{
var xd = new XmlDocument();
xd.LoadXml(Encoding.UTF8.GetString(args.Result));
lock (writeLock)
{
xd.WriteContentTo(writer);
}
}
finally
{
// count down our counter in a thread safe manner
if (Interlocked.Decrement(ref counter) == 0)
{
// this was the last one, close nicely
writer.WriteEndElement();
writer.Close();
((IDisposable) writer).Dispose();
}
}
};
wc.DownloadDataAsync(
new Uri(url));
}
source to share