IOException reading large file from UNC path to byte array using .NET
I am using the following code to try and read a large file (280Mb) into a byte array from a UNC path
public void ReadWholeArray(string fileName, byte[] data)
{
int offset = 0;
int remaining = data.Length;
log.Debug("ReadWholeArray");
FileStream stream = new FileStream(fileName, FileMode.Open, FileAccess.Read);
while (remaining > 0)
{
int read = stream.Read(data, offset, remaining);
if (read <= 0)
throw new EndOfStreamException
(String.Format("End of stream reached with {0} bytes left to read", remaining));
remaining -= read;
offset += read;
}
}
This results in the following error.
System.IO.IOException: Insufficient system resources exist to complete the requested
If I run this using a local path it works fine, in my test case the UNC path actually points to the local block.
Any thoughts on what's going on here?
source to share
I suspect something below down is trying to read into another buffer and only 280MB read in one pass fails. There may be more buffers in the network case than in the local case.
I would read about 64K at a time, instead of reading the entire batch in one go. This is enough to avoid too much chunking overhead, but avoid using huge buffers.
Personally, I usually only read the end of the stream and don't assume the file size will remain constant. See this question for more information.
source to share
In addition, the written code must be placed FileStream
in a block using
. Failure to utilize resources is a very likely reason for getting "Insufficient system resources":
public void ReadWholeArray(string fileName, byte[] data)
{
int offset = 0;
int remaining = data.Length;
log.Debug("ReadWholeArray");
using(FileStream stream = new FileStream(fileName, FileMode.Open, FileAccess.Read))
{
while (remaining > 0)
{
int read = stream.Read(data, offset, remaining);
if (read <= 0)
throw new EndOfStreamException
(String.Format("End of stream reached with {0} bytes left to read", remaining));
remaining -= read;
offset += read;
}
}
}
source to share
It looks like the array was not created with sufficient size. How large is the array allocated before it is accepted? Or did you assume that the Read function reallocates the data array as needed? This is not true. Edit: Er, maybe not, I just noticed the exception you got. Not sure now.
source to share