Uploading from google cloud storage is always wrong hash
I am trying to download some files from google cloud storage (log files from published google play app).
My code looks like this
Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", "my-service-account-credential.json", EnvironmentVariableTarget.Process);
StorageClient storageClient = StorageClient.Create();
var bucketName = "mybucketname";
var buckets = storageClient.GetBucket(bucketName);
var objects = storageClient.ListObjects(bucketName).ToList();
foreach (var o in objects)
{
try
{
Directory.CreateDirectory(Path.GetDirectoryName(o.Name));
using (var fs = File.Open(o.Name, FileMode.OpenOrCreate))
{
await storageClient.DownloadObjectAsync(bucketName, o.Name, fs);
}
}
catch (Exception e)
{
if (e.Message.StartsWith("Incorrect hash"))
{
continue;
}
throw;
}
}
Actually the code works fine (judging by looking at the actual downloaded file content, these are csv files). But as you can see, I followed a nasty catch / hack attempt because every file uploaded throws an exception stating that the hash is not correct. I am assuming that the client library is comparing the hash of the loaded content to the hash of the bucket and they are not identical, resulting in an exception.
The exception is:
System.IO.IOException: Incorrect hash: expected 'DXpVGw==' (base64), was '2RMrcw==' (base64)
at Google.Cloud.Storage.V1.StorageClientImpl.<DownloadObjectAsyncImpl>d__48.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at MyClass.GoogleBucket.Functions.<DownloadGoogleBucketLogs>d__1.MoveNext() in mycode.cs:line 51
So my question is, how do you load objects without getting this exception, obviously I shouldn't be doing what I did.
source to share
TL; DR: up to version 2.1.0. (Or select and create a source before that if you're desperate.)
It was difficult to understand.
The problem is that it was HttpClient
automatically unpacking data on the fly, but the hash provided by the server is for compressed content.
We have now made changes to both the REST API support library and the library Google.Cloud.Storage.V1
to intercept and hash the loaded data before decompression. The changes are merged on Github and will be in version 2.1.0, which I expect in early January.
Note that this will not fix a separate corner case where client side decompression is disabled, resulting in server side decompression, but still with a hash of the compressed content. We track it separately , but it won't affect the sample code here, as you would only see this if you created StorageService
explicitly disabled gzip support in the initializer and then created StorageClient
to port this service.
source to share