edgegraph.output.nrpickler#
Non-recursive pickler / “diller” implementation to be used for serializing edgegraph objects.
This module contains a non-recursive implementation of a pickler that is
suggested for use when serializing edgegraph objects. This overcomes a
limitation of the default builtin pickler, which struggles with highly
recursive data structures – edgegraph’s internal linkage structure is indeed
very recursive, and graphs even a fraction of the recursion limit will often cause RecursionError\ s when being
pickled. This module uses dill ([dill]) to perform pickling of more
object types than the standard pickler; but even Dill uses a recursive pickler,
so needs the special handling provided here.
Danger
Like Python’s default pickler, this module is not secure. Only pickle / unpickle data you trust.
See the warning at the top of pickle documentation for more
information!
See also
The first mention of the non-recursive pickler used here was by Daniel Darabos (cyhawk) on https://bugs.python.org/issue2480 . I stole it shamelessly, only making minor changes for Python 3.x, dill compatibility, and readability / formatting. Thank you, Daniel!
As the special handling of recursive structure is only needed on the pickling
side, not the unpickling side, this module need only be used to serialize data.
Either the standard library pickle or the third-party dill
can be used to unpickle data.
Usage of this pickler should be similar to the built-in one:
>>> from edgegraph.builder import randgraph
>>> from edgegraph.output import nrpickler
>>> import pickle
>>> graph = randgraph.randgraph(count=100)
>>> serial = nrpickler.dumps(graph)
>>> serial
b'\\x80\\x04\\x8c\\x1cedgegraph.structure.universe\\x94\\x8c\\x08...'
>>> unpacked = pickle.loads(serial) # use the regular pickle module to unpack
>>> unpacked is graph
False
>>> len(unpacked.vertices) == len(graph.vertices)
True
At this point, the unpacked object is a Universe identical in every way to
graph, except it is a different instance. All of its vertices, links, and
any attributes of have all been unpacked.
Note
If you intend to pickle / unpickle objects which use
RLock, you must implement special handling of
__getstate__ and __setstate__ (or equivalent) customization rules
which remove these locks before pickling and re-add them to your instance
after unpickling. See mishaturnbull/edgegraph#118
for some background on this and the associated PR for an example of what it
looks like (I suggest in particular the vertex.py changes as a good
example).
Ensure you call the superclass __getstate__ and __setstate__ from
yours!
Functions