Multiprocessing.manager Nested Shared Objects Doesn't Work With Queue

Python docs of the multiprocessing module state: Changed in version 3.6: Shared objects are capable of being nested. For example, a shared container object such as a shared list c

Solution 1:

The error is caused by AutoProxy currently not handling all BaseProxy arguments. There's a pull request which has not been merged yet. You either need to monkey-patch AutoProxy, or you look into multiprocessing.managers.py and apply the changes in the patch here directly to your source code.

It's really important to fix both lines in the patch to prevent a memory leak in the server process. The manager_owned-flag is used to let the BaseProxy code know, when to skip a reference increment for a proxy the manager owns himself (through nesting).

Solution 2:

If you don't want to patch the underlying python library, you can apply this patch in your own code using the following.

I've copied the change from the pull request reference by @Darkonaut and made package name modifications so it works outside the original package. This is placed at the module level of any module that uses multiprocessing.managers.

Note that the solution I referenced in comments in @Darkonaut's answer produced seg faults in my own testing, but this solution did not.

import multiprocessing.managers

defAutoProxy(token, serializer, manager=None, authkey=None,
              exposed=None, incref=True, manager_owned=False):
    '''
    Return an auto-proxy for `token`
    '''
    _Client = multiprocessing.managers.listener_client[serializer][1]

    if exposed isNone:
        conn = _Client(token.address, authkey=authkey)
        try:
            exposed = dispatch(conn, None, 'get_methods', (token,))
        finally:
            conn.close()

    if authkey isNoneand manager isnotNone:
        authkey = manager._authkey
    if authkey isNone:
        authkey = multiprocessing.process.current_process().authkey

    ProxyType = multiprocessing.managers.MakeProxyType('AutoProxy[%s]' % token.typeid, exposed)
    proxy = ProxyType(token, serializer, manager=manager, authkey=authkey,
                      incref=incref, manager_owned=manager_owned)
    proxy._isauto = Truereturn proxy

multiprocessing.managers.AutoProxy = AutoProxy