Async List Comprehensions In Python

Is it possible to do a list comprehension in Python asynchronously? I am trying to make many calls to a web api, where the bulk of the wait time is waiting for data to come back, w

Solution 1:

Yes, it is indeed possible to fire off multiple requests and asynchronously wait for them. Because Python is traditionally a synchronous language, you have to be very careful about what libraries you use with asynchronous Python. Any library that blocks the main thread (such as requests) will break your entire asynchronicity. aiohttp is a common choice for asynchronously making web API calls in Python. What you want is to create a bunch of future objects inside a Python list and await it. A future is an object that represents a value that will eventually resolve to something.

EDIT: Since the function that actually makes the API call is synchronous and blocking and you don't have control over it, you will have to run that function in a separate thread.

import asyncio

asyncdefmain():
    loop = asyncio.get_event_loop()
    futures = [asyncio.ensure_future(loop.run_in_executor(None, get_data, data)) for data in data_name_list]
    await asyncio.gather(*futures) # wait for all the future objects to resolve# Do something with futures# ...


loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

Please note that this is Python 3.6 code and might vary significantly from 3.7+ async code