BigQuery async query task - fetch_results () method returns wrong number of values
I am writing Python code with BigQuery Client API and trying to use async request code (written everywhere as sample code) and it doesn't work when fetch_data () method is called. Python errors with error:
ValueError: Too many values to unpack
So the 3 returned values (rows, total_count, page_token) seem to be the wrong number of return values. But I can't find any documentation on what this method should return - apart from numerous code examples that only show these 3 return results.
Here is a code snippet that shows what I am doing (not including the initialization of the "client" variable or the imported libraries that happened earlier in my code).
#---> Set up and start the async query job
job_id = str(uuid.uuid4())
job = client.run_async_query(job_id, query)
job.destination = temp_tbl
job.write_disposition = 'WRITE_TRUNCATE'
job.begin()
print 'job started...'
#---> Monitor the job for completion
retry_count = 360
while retry_count > 0 and job.state != 'DONE':
print 'waiting for job to complete...'
retry_count -= 1
time.sleep(1)
job.reload()
if job.state == 'DONE':
print 'job DONE.'
page_token = None
total_count = None
rownum = 0
job_results = job.results()
while True:
# ---- Next line of code errors out...
rows, total_count, page_token = job_results.fetch_data( max_results=10, page_token=page_token )
for row in rows:
rownum += 1
print "Row number %d" % rownum
if page_token is None:
print 'end of batch.'
break
What are the specific return results that I should expect from calling the job_results.fetch_data (...) method on an asynchronous request task?
source to share
It looks like you're right! The code no longer returns these 3 parameters.
As you can see in this commit from the shared repository, fetch_data now returns an instance of the HTTPIterator class (I guess I didn't understand this before, since I have a docker image with an older version of the bigquery client where it returns 3 values).
The only way I found to return results was doing something like this:
iterator = job_results.fetch_data()
data = []
for page in iterator._page_iter(False):
data.extend([page.next() for i in range(page.num_items)])
Note that now we no longer need to manage pageTokens
, it has been automated for the most part.
[EDIT]:
I just figured out that you can get the results by doing:
results = list(job_results.fetch_data())
Now, to admit it easier, it was before!
source to share