Are your Python HTTP requests slow?
Are your Python programs taking too long to run?
The Python libraries asyncio and aiohttp may be able to help you speed up your requests and your Python programs.
Let’s dive in and walk through both of these libraries.
Note: This blogpost is written with Python 3.10 in mind
What is asyncio
asyncio is a Python standard library that was introduced to the Python world with Python 3.4 (with the keywords, `await` and `async`, being introduced in Python 3.5) . This library gives developers the ability to develop and use asynchronous code. Two important keywords were introduced with the asyncio library:
- async — Used to identify code to be run asynchronously
- await — Used to pause asynchronous execution, and is used with coroutines/await-able functions.
The Event Loop is the core of the asyncio library, and is in charge of scheduling and executing asynchronous tasks. The Event Loop allows tasks to run concurrently without blocking the main program’s execution.
Coroutines can be defined as functions that can be paused and resumed. In the asyncio library, the async keyword is used to define a function as a coroutine and the await keyword is used to work with coroutines by pausing asynchronous execution).
Tasks are units of work that can be scheduled and managed by the event loop.
A Future is a result object of an asynchronous execution, which might not be completed at the current time.
A Semaphore is one of the ways to manage shared resources and control access in a concurrent environment. We will be using a Semaphore to restrict concurrent HTTP requests.
Overall, the asyncio library is a framework for asynchronous programming and concurrency in Python. It is particularly useful with I/O-bound processes (reading/writing files, HTTP requests, etc.). In I/O bound processes, you run into having the entire program blocked waiting for the running process to complete.
What is aiohttp?
aiohttp is a Python third party library that works in conjunction with the asyncio library. Specifically, the aiohttp is used for asynchronous HTTP clients and servers.
There is a lot within the aiohttp library and we will only scratch the surface of its capabilities. We will talk specifically about aiohttp Client Sessions.
requests vs. aiohttp/asyncio
We will walk through a little test of a simple HTTP GET request being made on a site numerous times. We will utilize the requests library for synchronous execution and the aiohttp/asyncio libraries for asynchronous execution.
Here is the code for the synchronous requests example:
You will need the requests library installed.
from time import perf_counter
import requests
def make_request(url: str):
# Synchronous GET request
requests.get(url=url)
# Print first letter of domain, so we know what is requested.
if "q" in url:
print("Q", flush=True, sep="", end="")
else:
print("B", flush=True, sep="", end="")
def makes_all_requests(urls: list[str]):
for url in urls:
make_request(url=url)
if __name__ == "__main__":
urls = [
"https://books.toscrape.com/",
"http://quotes.toscrape.com/",
] * 50
print("---Starting---")
start_time = perf_counter()
makes_all_requests(urls=urls)
end_time = perf_counter()
total_time = end_time - start_time
print(f"\n---Finished in: {total_time:02f} seconds---")In this example, we are timing and executing 100 synchronous GET requests on 2 different URLs (50 requests each).
There will be varying output/timing, here is an example of a run on my computer:
---Starting---
BQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQBQ
---Finished in: 21.100947 seconds---Now let’s dive into a asynchronous example and compare speeds.
Here is the code for the asyncio/aiohttp example:
You will need the aiohttp library installed.
import asyncio
from random import random
from time import perf_counter
import aiohttp
semaphore = asyncio.Semaphore(10)
async def make_request(async_session: aiohttp.ClientSession, url: str):
# Semaphore for limiting concurrent requests to 8
async with semaphore:
# Asynchronous GET request
async with async_session.get(url=url) as _response:
# avoid overpowering the URL right away by having this happen first
await asyncio.sleep(random())
# Print first letter of domain, so we know what is requested.
if "q" in url:
print("Q", flush=True, sep="", end="")
else:
print("B", flush=True, sep="", end="")
async def makes_all_requests(urls: list[str]):
# Stores all tasks that will later be used on `asyncio.gather`
async with aiohttp.ClientSession() as async_session:
tasks = []
for url in urls:
# Creates asyncio.Task that will return a future
task = asyncio.create_task(
coro=make_request(
async_session=async_session,
url=url,
)
)
tasks.append(task)
# Tasks are ran with asyncio.gather
# By setting `return_exceptions` to False, we will raise Exceptions within
# their asyncio task instance and everything will stop, by putting True, it
# will raise when `result()` is called on the future.
await asyncio.gather(*tasks, return_exceptions=False)
if __name__ == "__main__":
urls = [
"https://books.toscrape.com/",
"http://quotes.toscrape.com/",
] * 50
print("---Starting---")
start_time = perf_counter()
asyncio.run(makes_all_requests(urls=urls))
end_time = perf_counter()
total_time = end_time - start_time
print(f"\n---Finished in: {total_time:02f} seconds---")In this example, we are doing everything that the requests example did, but asynchronously.
To start, we run our make_all_requests coroutine with asyncio.run and pass through the same argument as the original version.
The make_all_requests async function is in charge of creating an aiohttp ClientSession context, and then creates individual tasks for each request and appends them to a list variable. Once we have all of the tasks ready, we run asyncio.gather, which triggers the tasks.
A semaphore is declared at the top to be 10, and means that only 10 async operations within the semaphore context can run concurrently at one time. This is particularly useful when trying to not overpower an API and exceed rate limits.
The make_request async function utilizes the semaphore in a context, and the session.get in a context, and sleeps (non-blocking) for under a second. It performs the same request and print behavior as the original version.
There will be varying output/timing, here is an example of a run on my computer:
---Starting---
BQQBQQBBBQBQQBBQQBQBBBBBBQQQBQQQQBBQBBBBQQBBQQBQQBQQBQQBBQQBQQQQBBBBBBQQBBQQBQQQBBBBQBBQBQQBQQBQQBBQ
---Finished in: 5.694053 seconds---As you can see, it takes only 5 seconds compared to the 21 seconds from the synchronous version.
In this blogpost you were introduced to the asyncio and aiohttp libraries and learned about some foundational concepts in both. In this post, we only took a glimpse at the capabilities and powers of these two libraries and asynchronous Python programming in general.
I encourage you to look further into these libraries/other asyncio-friendly libraries and continue speeding up your Python code.
Happy Coding!
