How to handle errors from ndb.put_multi
The documentation for GAE is ndp.put_multi
severely lacking. NDB Objects and Keys - Python - Google Virtual Platform shows that it returns a list of keys ( list_of_keys = ndb.put_multi(list_of_entities)
), but says nothing about failures, NDB functions don't provide much more information.
Scrolling through the code (see below) reveals that, at least for now, put_multi
just aggregates Future.get_result()
returned from an async method that delegates the put
entity code itself . The docs for NDB Future Class now indicate that a result will be returned or an exception will be thrown. I was told, however, that the result would be None
if a specific one failed put
(I cannot find authoritative documentation, but if it is something like db.get
, then it would make sense).
So it all boils down to some questions that I cannot find answers to:
- Clearly the return value is a list - it is a list with some elements, perhaps None? Or are exceptions being used instead?
- When there is a mistake, what should be shifted? Can all objects be reset (idempotent) or only those whose return value
None
(if that's even how errors are passed)? - How common are mistakes (one answer: 1/3000 )? Do they appear in magazines (because I haven't seen them)? Is there a way to reliably simulate a bug for testing?
Using a function in an open source library implies that the operation is idempotent, but that's about it. ( Other usages don't even bother checking the return value or catching exceptions .)
Data warehouse error handling does not mention anything other than exceptions.
source to share
I agree with reading the code: put_multi()
reacts to the error in the same way as put_async().get_result()
. If it put()
throws an exception, put_multi()
it will also be useless which of the multiple calls failed. I am not aware of the circumstances when I put_multi()
returned None
for some entries in the key list.
You can re-place objects that have been placed if no other user has updated those objects since the last put attempt. Objects created using system identifiers have updated keys in memory, so reinstalling them will overwrite existing objects rather than create new ones. I would not call it idempotent precisely because retrying will overwrite any updates to those objects that are being executed by other processes.
Of course, if you need more control over the outcome, you can perform this update in a transaction, although all objects must be in the same entity group for this to work with a primitive transaction. (Cross-group transactions support up to five different groups of objects.) Failure during a transaction ensures that no entity is created or updated if any of the attempts fail.
I don't know the overall error rate for update failures. These failures are likely to include conflicting bugs or hot pills (too many updates to nearby records in too short a time, resulting in a timeout), depending on the behavior of the application. All such errors are reported to the API as exceptions.
The easiest way to test error handling paths is to wrap the class Model
and override the methods in test mode. You may love and peek into the API stub used testbed
which may be able to hook into low level calls and simulate errors. (I don't think this is a feature testbed
directly, but you can use a similar technique.)
source to share