Clever Geek Handbook

Correct way to deal with UrlFetch speed limit without random Utilities.sleep?

Our Google Sheets add-on provides users with an extensive library of functions.

The problem is that every function triggers UrlFetch. Therefore, if users drag the column down> 100 times, they will most likely see the error: "Error: Service called too many times in a short time: urlfetch".

Apparently the general solution is to add a random bit of sleep before the UrlFetch function (e.g. https://productforums.google.com/forum/#!topic/docs/_lgg9hbU6k8 ). But is there no other way to solve this? After testing with random sleep, I can increase the limit to 200 functions at a time, maximum

The main problem is that I don't know what the limitation really is. For example, is there when> 100 UrlFetch requests are immediately queued by Google, what's the speed limit? I'm trying to figure out what our options are, but not even fully get the limits!

Thanks a lot for the help, especially if you are someone from Google :).

+3


source to share


1 answer


The answer to your question "is there when 100 UrlFetch requests are immediately in the Google queue, what's the speed limit?" basically no. The limit is not 100 calls.

You will see this error ("Error: Service called too many times in a short time: urlfetch") if one of the following conditions is true:

  • 22MB of data sent or received via urlfetch per minute
  • There are 3,000 or more calls per minute.
  • ... or if you hit the daily high. call and data rates.

In your case, it sounds like you are getting an error before hitting daily max data or daily max call, so maybe data per minute: 22 MB sent or received via urlfetch per minute.



You can continually check the number of bytes you are processing with urlfetch and use them to make the function sleep for a minute if it is near the limit. However, this is a little annoying.

You might want to try to make the function more efficient so that less data is sent or fewer calls are made. How to do this depends a lot on the function, and we will need to see if the code offers specific suggestions.

You can find Google quotas here: https://cloud.google.com/appengine/docs/quotas#UrlFetch

+2


source






More articles:


All Articles