Google Cloud Storage Unexpected end of ZLIB input stream when uploading file larger than chunksize using java api

I am trying to download reviews from a play store that are available in a bucket using the java api (details at the end of the post). For browse files that are larger than the block size when compressed, I get the following exception:

java.io.EOFException: Unexpected end of ZLIB input stream
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:116)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:51)
at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
at com.google.api.client.util.IOUtils.copy(IOUtils.java:63)
at com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest(MediaHttpDownloader.java:247)
at com.google.api.client.googleapis.media.MediaHttpDownloader.download(MediaHttpDownloader.java:199)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo(AbstractGoogleClientRequest.java:553)
at com.google.api.services.storage.Storage$Objects$Get.executeMediaAndDownloadTo(Storage.java:4540)
at com.google.api.services.storage.Storage$Objects$Get$executeMediaAndDownloadTo.call(Unknown Source)

      

The code I used is:

private void saveObject(StorageObject object) {
    try {
        System.out.println("Getting object data");
        FileOutputStream fos = new FileOutputStream (new File(object.getName()));

        Storage.Objects.Get getObject =
            storage.objects().get("pubsite_prod_rev_XXXXX", object.getName());

        // setting the chunksize to a small value fails pretty much on all files
        getObject.getMediaHttpDownloader().setDirectDownloadEnabled(false).setChunkSize(MediaHttpUploader.MINIMUM_CHUNK_SIZE)

        // setting the chunksize to larger value allows me to download some of the files but fails on larger ones
        //getObject.getMediaHttpDownloader().setDirectDownloadEnabled(false).setChunkSize(MediaHttpDownloader.MAXIMUM_CHUNK_SIZE)


        getObject.executeMediaAndDownloadTo(fos);
        fos.close();
    } catch(Exception e) {
        System.err.println("Error downloading " + object.getName());
        e.printStackTrace();
    }

      

I can upload files using gsutil, so it seems like the chunking implementation is inside the MediaHttpDownloader.

I am using java 7 and the following libs

compile group: 'com.google.api-client', name: 'google-api-client', version: '1.19.0'
compile group: 'com.google.oauth-client', name: 'google-oauth-client-jetty', version: '1.19.0'
compile group: 'com.google.apis', name: 'google-api-services-storage', version: 'v1beta2-rev56-1.19.0'

      

Also I am using groovy but don't see how it will affect the problem as the error is inside google lib.

thanks for any help

Andre

+3


source to share





All Articles