Memory Use If Multiprocessing Queue Is Not Used By Two Separate Processes
Solution 1:
I figured out what was happening, so I'll post it here for the benefit of anyone that stumbles across question.
My memory problem resulted from a numpybug in numpy version 1.16.0. Reverting to numpy version 1.13.3 resolved the problem.
To answer the basic question: No, there is no need to worry which thread/process is doing the consuming (get
) and which thread/process is doing the producing (put
) for multiprocessing queues. There is nothing special about multiprocessing queues with respect to garbage collection. As kindall explains in response to a similar question:
When there are no longer any references to an object, the memory it occupies is freed immediately and can be reused by other Python objects
I hope that helps someone. In any case, the numpy bug should be resolved in the 1.16.1 release.
Post a Comment for "Memory Use If Multiprocessing Queue Is Not Used By Two Separate Processes"