Clever Geek Handbook

Ascertaining why an object cannot be pickled

I am getting object t

,, from api type Object

. I can't uncover it, getting the error:

  File "p.py", line 55, in <module>
    pickle.dump(t, open('data.pkl', 'wb'))
  File "/usr/lib/python2.6/pickle.py", line 1362, in dump
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.6/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.6/pickle.py", line 313, in save
    (t.__name__, obj))
pickle.PicklingError: Can't pickle 'Object' object: <Object object at 0xb77b11a0>

When I do the following:

for i in dir(t): print(type(i))

I only get string objects:

<type 'str'>
<type 'str'>
<type 'str'>
...
<type 'str'>
<type 'str'>
<type 'str'>

How can I print the contents of my object Object

to understand why it cannot be pickled?

It's also possible that the object contains C pointers to QT objects, in which case it didn't make sense for me to sort the object. But again I would like to see the internal structure of the object in order to establish this.

+8


source to share


4 answers


You might want to read the python docs and check your API class afterwards Object

.



As far as the "internal structure of an object" is concerned, typically the instance attributes are stored in an attribute __dict__

(and since no class attributes are fetching you only care about the instance attributes) - but note that you will also have to recursively inspect __dict__

for each attribute.

+3


source


I would use dill

one that has tools to investigate what inside an object makes your target non-stealable. See this answer for an example: A good example of BadItem in the Dill module , and also this question for an example of real-world use of detection tools: pandas.algos._return_false raises PicklingError with dill. dump_session for CentOS .

>>> import dill
>>> x = iter([1,2,3,4])
>>> d = {'x':x}
>>> # we check for unpicklable items in d (i.e. the iterator x)
>>> dill.detect.baditems(d)
[<listiterator object at 0x10b0e48d0>]
>>> # note that nothing inside of the iterator is unpicklable!
>>> dill.detect.baditems(x)
[]

However, the most common starting point is to use trace

:



>>> dill.detect.trace(True)
>>> dill.detect.errors(d)
D2: <dict object at 0x10b8394b0>
T4: <type 'listiterator'>
PicklingError("Can't pickle <type 'listiterator'>: it not found as __builtin__.listiterator",)
>>>

dill

also has functions for keeping track of pointers and object references, so you can build a hierarchy of how objects refer to each other. See: https://github.com/uqfoundation/dill/issues/58

Alternatively, there are also: cloudpickle.py and debugpickle.py, which for the most part are no longer being developed. I am the author dill

and hope to soon unify any functionality in these codes that is missing from dill

.

+8


source


I tried Dill but that didn't explain my problem. Instead, I used the following code from https://gist.github.com/andresriancho/15b5e226de68a0c2efd0 which encountered an error in my override __getattribute__

:

def debug_pickle(instance):
  """
  :return: Which attribute from this object can't be pickled?
  """
  attribute = None

  for k, v in instance.__dict__.iteritems():
      try:
          cPickle.dumps(v)
      except:
          attribute = k
          break

  return attribute

Edit: Here's my code reproduced using pickle and cPickle:

class myDict(dict):

    def __getattribute__(self, item):
        # Try to get attribute from internal dict
        item = item.replace("_", "$")

        if item in self:
            return self[item]

        # Try super, which may leads to an AttribueError
        return super(myDict, self).__getattribute__(item)

myd = myDict()

try: 
    with open('test.pickle', 'wb') as myf:
        cPickle.dump(myd, myf, protocol=-1)
except:
    print traceback.format_exc()


try:
    with open('test.pickle', 'wb') as myf:
        pickle.dump(myd, myf, protocol=-1)
except:
    print traceback.format_exc()

Output:

Traceback (most recent call last):
File "/Users/myuser/Documents/workspace/AcceptanceTesting/ingest.py", line 35, in <module>
  cPickle.dump(myd, myf, protocol=-1)
UnpickleableError: Cannot pickle <class '__main__.myDict'> objects

Traceback (most recent call last):
File "/Users/myuser/Documents/workspace/AcceptanceTesting/ingest.py", line 42, in <module>
  pickle.dump(myd, myf, protocol=-1)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1370, in dump
  Pickler(file, protocol).dump(obj)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
  self.save(obj)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 313, in save
  (t.__name__, obj))
PicklingError: Can't pickle 'myDict' object: {}

You will see that the reason is because the attribute names are mangled __getattribute__

+1


source


Here's an extension of Alastair's solution , in Python 3.

Is he:

  • is recursive, for working with complex objects, where the problem can be multilevel.

    The output is on the form .x[i].y.z....

    so you can see which members were called to resolve the problem. With help, dict

    it just prints instead [key/val type=...]

    , as key or value problems can arise, making it difficult (but not impossible) to access a specific key or value in dict

    .

  • allows for more types, in particular list

    , tuple

    and dict

    that need to be handled separately because they have no attributes __dict__

    .

  • returns all problems, not just the first one.

def get_unpicklable(instance, exception=None, string='', first_only=True):
    """
    Recursively go through all attributes of instance and return a list of whatever
    can't be pickled.

    Set first_only to only print the first problematic element in a list, tuple or
    dict (otherwise there could be lots of duplication).
    """
    problems = []
    if isinstance(instance, tuple) or isinstance(instance, list):
        for k, v in enumerate(instance):
            try:
                pickle.dumps(v)
            except BaseException as e:
                problems.extend(get_unpicklable(v, e, string + f'[{k}]'))
                if first_only:
                    break
    elif isinstance(instance, dict):
        for k in instance:
            try:
                pickle.dumps(k)
            except BaseException as e:
                problems.extend(get_unpicklable(
                    k, e, string + f'[key type={type(k).__name__}]'
                ))
                if first_only:
                    break
        for v in instance.values():
            try:
                pickle.dumps(v)
            except BaseException as e:
                problems.extend(get_unpicklable(
                    v, e, string + f'[val type={type(v).__name__}]'
                ))
                if first_only:
                    break
    else:
        for k, v in instance.__dict__.items():
            try:
                pickle.dumps(v)
            except BaseException as e:
                problems.extend(get_unpicklable(v, e, string + '.' + k))

    # if we get here, it means pickling instance caused an exception (string is not
    # empty), yet no member was a problem (problems is empty), thus instance itself
    # is the problem.
    if string != '' and not problems:
        problems.append(
            string + f" (Type '{type(instance).__name__}' caused: {exception})"
        )

    return problems
0


source






More articles:


All Articles