HttpClient cipher in memory usage with big response

I am working on a console application that takes a list of endpoints for video data, makes an HTTP request, and saves the result to a file. These are relatively small videos. Due to a problem beyond my control, one of the videos is very large (in 145 minutes instead of a few seconds).

The problem I see is that my memory usage increases to ~ 1GB after calling this request, and I end up with a "Task was canceled" error (presumably because the client was timed out). This is ok, I don't want this video, but as far as my allocated memory stays high no matter what I do. I want to free memory. The task manager seems to show ~ 14MB of memory usage before this call and then expires continuously after that. In VS debugger, I see a splash.

I tried to throw everything in the expression using

by re-initializing the HttpClient

on exception , manually calling GC.Collect()

with no luck. The code I'm working with looks something like this:

consumer.Received += async (model, ea) =>
{
    InitializeHttpClient(source);
    ...
    foreach(var item in queue)
    {
        await SaveFileFromEndpoint(url, fileName);
        ...
    }
}

      

and methods:

public void InitializeHttpClient(string source)
{
    ...
    _client = new HttpClient();
    ...
}

public async Task SaveFileFromEndpoint(string endpoint, string fileName)
{
    try
    {
        using (HttpResponseMessage response = await _client.GetAsync(endpoint))
        {
            if (response.IsSuccessStatusCode)
            {
                using(var content = await response.Content.ReadAsStreamAsync())
                using (var fileStream = File.Create($"{fileName}"))
                {
                    await response.Content.CopyToAsync(fileStream);
                }
            }
        }
    }
    catch (Exception ex)
    {

    }
}

      

Here is my debugger output:

burst of memory

I think I have a few questions about what I see:

  • Is memory usage really the problem?
  • Is there a way to free memory allocated by a large HTTP request?
  • Is there a way to see the length of the request content before the call is made and memory is allocated? So far, I haven't been able to find a way to find out until the actual memory is allocated.

Thanks in advance for your help!

+3


source to share


1 answer


If you use HttpClient.SendAsync(HttpRequestMessage, HttpCompletionOption)

instead GetAsync

, you can supply HttpCompletionOption.ResponseHeadersRead

, (as opposed to the default ResponseContentRead

). This means that the response stream will be sent to you before the response body is loaded (and not after it), and will require significantly less buffer to work.



+2


source







All Articles