How to control what is imported when unpacking a python object?
I have the following setup:
a.py
:
class A(object):
def __init__(self, name):
self.name = name
def a(self):
print('yow {}!'.format(self.name))
b.py
:
class B(object):
def __init__(self, obj):
self.obj = obj
sender.py
:
from a import A
from b import B
message = pickle.dumps(B(A('Martin')))
receiver.py
:
my_b = pickle.loads(message) my_a = my_b.obj my_a.a()
Output: yow Martin!
In sender.py
I will take apart an object b
that acts as a carrier for the object a
. Then I send this pickled object b
through RabbitMQ to another process . In receiver.py
(this is a different process) I get a message via RabbitMQ, unickle object b
and magically b
and a
automatically imported. Can I control what is imported? I would like the worker to receiver.py
consume as little memory as possible. But if modules are imported without my control, it can bloat quickly.
Can anyone explain how to sort the imported materials and what to do with them?
source to share
What kind of control is required? As you can see from the source, when you execute pickle.loads(content)
, it actually does:
def loads(str):
file = StringIO(str)
return Unpickler(file).load()
Then there is some magic. It reads a string as a file and sends its contents based on certain keys:
GLOBAL = 'c' # push self.find_class(modname, name); 2 string args
INST = 'i' # build & push class instance
Loading the function itself:
def load(self):
"""Read a pickled object representation from the open file.
Return the reconstituted object hierarchy specified in the file.
"""
...
read = self.read # self.read = file.read, which is StringIO read()
dispatch = self.dispatch
try:
while 1:
key = read(1)
dispatch[key](self) # this function call makes a future import.
except _Stop, stopinst:
return stopinst.value
Are you interested in a method find_class()
that is used in several other load functions
( load_inst()
and load_global()
):
def find_class(self, module, name):
# Subclasses may override this:
__import__(module) # straight-forward import, you can ovveride it.
mod = sys.modules[module]
klass = getattr(mod, name)
return klass
For example, the load_inst()
function:
def load_inst(self):
module = self.readline()[:-1]
name = self.readline()[:-1]
klass = self.find_class(module, name)
# Now module is imported and ready to be used:
self._instantiate(klass, self.marker())
dispatch[INST] = load_inst
So, if you want to manage namespaces or modules that can be imported, you will need to subclass Unpickler
and override find_class()
to suit your purposes. Was my answer helpful to you?
source to share
pickle
you need to import the modules a
and b
to load the classes a
and b
that are needed to reassemble your object. I took your example and just renamed a.py to aaaa.py and b.py to bbbb.py. Now if we print the message (pickled object) sender.py sends, you will see this:
ccopy_reg
_reconstructor
p0
(cbbbb
B
p1
c__builtin__
object
p2
Ntp3
Rp4
(dp5
S'obj'
p6
g0
(caaaa
A
p7
g2
Ntp8
Rp9
(dp10
S'name'
p11
S'Martin'
p12
sbsb.
You don't need to understand all of this, but notice what bbbb
, what follows b
, and aaaa
what follows a
. To tell you pickle
how to rebuild your pickled object. To load the classes, it must import the modules in which the classes are defined. If you were trying to team up with pickle machines and block modules from loading, there would be no way to rebuild your facility.
source to share