BigQuery Google query is too big
It has been a while since I had the opportunity to work with the GBQ pandas module, but I noticed that one of our regression tests is now failing.
The test in question is:
https://github.com/pydata/pandas/blob/master/pandas/io/tests/test_gbq.py#L254-L267
In short, the test tries to create a table with 5 columns (types: Boolean, Float, String, Integer, Timestamp) and 1000,001 rows. Inserting these lines into chunks of 10,000 lines fails with the "Request Too Large" response.
I feel like this will probably have a similar answer to Getting "Query too large" in BigQuery - but seeing how this test has worked in previous times, I'm wondering if there is a database issue that needs to be addressed. It's also possible that the API was changed when I wasn't looking!
TL; DR version: how about our insert is too big and are there any documented limits we can refer to?
source to share
The documented limits are listed here:
https://cloud.google.com/bigquery/streaming-data-into-bigquery#quota
Answer TL: DR: While BQ does not strictly enforce the maximum rows per request of 500 rows / inserts at this time, there are some other restrictions elsewhere in the API stack related to the total size of the request that prevent a call from later.
source to share