Pickling Python Function Fails With Processpoolexecutor In A Decorator

So I asked this question and tried the ProcessPoolExecutor approach. I used the decorator suggested the following way: Running Image Manipulation in run_in_executor. Adapting to mu

Solution 1:

Decorators typically produce wrapped functions that aren't easy to pickle (serialize) because they contain hidden state. When dealing with multiprocessing, you should avoid decorators and send ordinary global functions to run_in_executor. For example, you could re-write your executor decorator into a utility function:

_pool = concurrent.futures.ProcessPoolExecutor()
 
async def exec_async(fn, *args):
    loop = asyncio.get_event_loop()
    returnawaitloop.run_in_executor(_pool, fn, *args)

Instead of decorating a function with executor, you can just await it using await exec_async(some_function, arg1, arg2, ...). Likewise, you can rewrite the pil decorator into another utility:

def pil(image, transform):
    img = PILManip.pil_image(image)
    if img.format == "GIF":
        frames = []
        for frame in ImageSequence.Iterator(img):
            res_frame = transform(frame)
            frames.append(res_frame)
        return PILManip.pil_gif_save(frames), "gif"
    elif img.format in ["PNG", "JPEG"]:
        img = transform(img)
        return PILManip.pil_image_save(img), "png"
    else:
        raise BadImage("Bad Format")

The implementation of blur now becomes an ordinary function which callspil, and which can be safely passed to exec_async:

defblur(image):
    deftransform(frame):
        frame = frame.convert("RGBA")
        return frame.filter(ImageFilter.BLUR)
    return pil(image, transform)
 
@router.get("/blur/", responses=normal_response)asyncdefblur_image(url: str):
    byt = await Client.image_bytes(url)
    img, image_format = await exec_async(blur, byt)
    return Response(img.read(), media_type=f"image/{image_format}")

Note: the above code is untested.