Can I upload Avro files using Snappy compression to BigQuery?
I know BigQuery supports Avro file uploads, and I am able to upload Avro file to BigQuery.
Using the command below,
java -jar avro-tools-1.7.7.jar fromjson --codec snappy --schema-file SourceSchema.avsc Source.json > Output.snappy.avro
I created an Avro file with Snappy compression and is trying to upload to BigQuery, but Upload error with errors below.
Errors:
file-00000000: The Apache Avro library failed to parse file file-00000000. (error code: invalid)
Can I upload Avro files using Snappy compression in BigQuery?
source to share
BigQuery only supports the DEFLATE and Snappy algorithm for compressing Avro data blocks, from the docs ( https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro )
Avro compressed files are not supported, but compressed chunks are found. BigQuery supports DEFLATE and Snappy codecs.
source to share
BigQuery now supports Snappy. See: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro
Avro compressed files are not supported, but compressed chunks are found. BigQuery supports DEFLATE and Snappy codecs.
source to share