· 9 min read
Modern Python: a progressive ecosystem to build better software
How opinionated packages and frameworks are shaping software engineering in Python
Python has been my main tool of choice for many years - even though I currently don’t use it often, I still enjoy writing scripts fast to solve simple issues. Sure, there’s the easy critique: Python, as a dynamically typed language, has some downsides, specifically if you think about error handling. But this is not at all what this post is about. Python is a tool that is so flexible that it allowed me to create simple or rather complex scripts, backends and even web frontends (yes, MVC style, serving HTML). During that time, I’ve adopted many great libraries that have greatly facilitated day-to-day work and the overall developer experience. However, I still hope to see wider adoption of this expanding ecosystem. Therefore, in this post, I’d like to give a brief overview of a couple of things that very positively influenced the way I write Python today.
Types
Yes, Python is a dynamically typed language, but type hints have been supported for quite some time and they actually are really helpful, even when writing just simple scripts for CI. Sure - Python is not compiled, therefore the type hints merely communicate intent and serve as some sort of documentation - for future you, colleagues or anyone else stumbling on that code on GitHub.
But enough talking, let’s see a quick example:
# your typical Python out there in the wild
def add(left, right):
return left + right
# oh, so this is actually meant for numbers
# you can still throw in strs or dicts but the intent is clearer now
def add(left: int, right: int) -> int:
return left + right
This is obviously a very small example but I think it conveys the whole point: Types in Python do not necessarily prevent bugs, they communicate intent. You can see from type annotations what the intention of the code is without needing to document. These annotations are not just a friendly reminder for future you and your intentions back then - they also can be helpful when thinking about how to unit test and which test cases to cover (e.g. actually passing other types).
Granted, typing can sometimes be a bit convoluted, as this small, but real-life example of a signature for a callback:
class Callback(Protocol):
""" callback function signature - example see TransferJob.update_progress() """
def __call__(self, val: int, total: int = ...) -> Any:
...
You might ask yourself: when would I ever need this?
In this particular case, it’s to accept a callback to track progress on uploads and downloads:
class TransferJob:
""" object representing a single transfer (up- / download) """
progress = 0
transferred = 0
total = 0
def update_progress(self, val: int, total: int = None) -> None:
self.transferred += val
if total is not None and self.total == 0:
# only set total if present and not set
self.total = total
# you're expected to pass a function that accepts the above described signature
async def download(self, callback_fn: Callback = None, ...):
...
Having these annotations doesn’t just facilitate looking again at your pretty code from 6 months ago - it is also quite helpful if you write libraries meant to be consumed by others.
Personally, if I encounter libraries out there in the wild without this quality of life improvement, I’d really look hard for alternatives. Luckily, type annotations have become quite popular and are used in several high-profile libraries (like the ones mentioned below). I use type annotations even for the simplest scripts and I certainly can say that future me was always thankful when coming back to these.
Async
When I started using Python primarily to solve customer issues via HTTP requests and (somewhat) RESTful JSON APIs, I quickly hit a limit when running these requests sequentially. There are solutions to this out there like multi-threading - but I’ll be honest: I never gave it more than a few thoughts, because it just is not the developer experience I am after for these particular use cases. The vast majority of the tasks I solved via Python scripting were some sort of bulk operations that the product just doesn’t handle - from importing users via CSV file to bulk update a group of users, folders or whatever entity the customer needs.
This is where async Python comes in.
Essentially, async functions are resumable functions that return early when the result is not ready, so that other functions can start or continue their work. I won’t go into the implementation details of this in Python, but essentially, with a few adjustments in the syntax, you can run multiple requests in parallel.
First, let’s look at a sync sequential example:
for user in user_list:
payload = ...
client.users.create_user(user=payload)
Assuming each request takes a couple ms, this quickly adds up as the user_list gets bigger.
Now let’s see how async concurrent can speed things up:
user_reqs = []
# this time we build the requests but don't run them directly
for user in user_list:
payload = ...
# this assumes create_user is async!
user_reqs.append(client.users.create_user(user=payload))
# we set up a generator that yields a fixed batch size
BATCH_SIZE = 5
req_gen = (user_reqs[i:i + BATCH_SIZE] for i in range(0, len(user_reqs), BATCH_SIZE))
# we process 5 concurrently
for batch in req_gen:
await asyncio.gather(*batch)
This obviously omits a couple of things to simplify this (error handling etc.) but still: With a couple of lines of code you can run these requests concurrently and reduce the total duration compared to the sequential code.
Meet the ecosystem
I’ve now introduced two game changers for me that heavily influenced the code I’ve written: types and async. To be transparent though, this did not come out of nowhere: I stumbled on all of this by looking at what the community has to offer. In the following short descriptions, I want to give credit where credit is due and highlight a small selection of great libraries, frameworks or tools that are essential parts of the modern Python ecosystem.
Pydantic
Types are great - especially if you are serializing and deserializing things, specifically from web APIs. If you worked with Python and JSON before, you’re well aware of techniques to group your data e.g. using dataclasses or just plain dictionaries. Pydantic makes it really easy to define and serialize data, otherwise throwing a reasonable exception.
But enough talking, let’s look at some example code. Let’s assume we want to import a user by reading from some CSV file and need to describe the shape of a user:
from typing import Optional
from pydantic import BaseModel
# we inherit the BaseModel
class UserImport(BaseModel):
# we define mandatory fields and use types to define the expected content
first_name: str
last_name: str
email: str
# we can define optional fields like so
phone: Optional[str] = None
This makes awesome use of type annotations and helps you define your data in a declarative way. If you try parsing something else (e.g. a number instead of str for names), an appropriate error will be thrown. Pydantic is a powerful tool that I wouldn’t miss when working with JSON web APIs and need to be precise on how I want my data to look like. Apart from the great support in code editors you get when parsing the data into the correct shape, you also fail when the data is not in the described shape.
There are much more features, check out their documentation:
FastAPI
When you need to spin up a quick HTTP API using Python, chances are you end up using something like Flask. But there are many other libraries / frameworks that aim for a great developer experience and one of those that changed my perspective was FastAPI:
- it integrates Pydantic to help you build clean and well-documented APIs (using OpenAPI 3.0)
- it does not use your classic WSGI (Web Server Gateway Interface) but instead ASGI (Async Server Gateway Interface) Basically, it combines async and typed code to deliver performance, documentation out of the box and overall just a pleasant developer experience.
I’ve written a couple of such applications (e.g. to handle webhooks and process them asynchronously) and always enjoyed the simplicity of Python coupled with performant (non-blocking request handlers!) and well-documented code (types, OpenAPI).
Let’s look at a brief example:
@router.delete('/{hook_id}', response_model={}, status_code=204,
responses={401: { "model": ErrorResponse, "description": "Unauthorized."},
404: {"model": ErrorResponse, "description": "Not found"}
})
def delete_registered_hook(hook_id: UUID, x_hook_service_token: Optional[str] = Header(None), db: Session = Depends(get_db)):
if not validate_service_token(x_hook_service_token):
logging.error(f'ERROR: wrong service token at {datetime.datetime.now()} – ID: {hook_id}')
return JSONResponse(status_code=401, content={"message": "Unauthorized.", "details": "Wrong token."})
result: Hooks = get_hook(db=db, hook_id=hook_id)
if not result:
logging.error(f'ERROR: wrong hook route at {datetime.datetime.now()} – ID: {hook_id}')
return JSONResponse(status_code=404, content={"message": "Not found.", "details": f"No hook with id {hook_id} found."})
delete_hook(db=db, hook=result)
return {}
This is just a small example but it contains a lot of the great features:
- setting up a route with params is as easy as typing the params
- passing invalid data (e.g. non-valid UUID) will lead to built-in 422 errors
- you can use dependency injection to pass a db connection
- you have a declarative way to describe expected errors and the response model
Check out the features here:
FastStream
It’s been a long post already, so I’ll keep this more concise: This is the equivalent of FastAPI but for event streaming - it allows you to quickly write producers and consumers using a similar syntax as described above:
from faststream import FastStream
from faststream.nats import NatsBroker
broker = NatsBroker("nats://localhost:4222")
app = FastStream(broker)
@broker.subscriber("in-subject")
@broker.publisher("out-subject")
async def handle_msg(user: str, user_id: int) -> str:
return f"User: {user_id} - {user} registered"
It also makes use of Pydantic, helping you process the data in the shape you expect it and otherwise providing meaningful exceptions. The example above and more can be found in their documentation:
Honorable mentions
There are many more great tools out there that deserve mentioning - but to keep it short, I want to briefly mention two more projects:
- ruff - very fast Python linting
- uv - the package manager Python needed
The linter ruff makes use of Rust to outperform your common tools like flake8 - more info here: ruff
Using pip, correctly setting up virtual environments etc. can be a hassle - there are many tools and I’ve used quite a bit, but uv really shines. It significantly simplifies dependency management compared to basic Python, and not only utilizes Rust for performance but also takes inspiration from cargo, acting as a powerful package manager. More info here: uv
Conclusion
Obviously there’s a long list of great tools and libraries that shaped the community - but the selection above is my personal journey, specifically for building web-related tooling, services etc. quickly with much better developer experience than your classic Python tools. I cannot imagine writing Python code without typing anymore and think async Python is an excellent choice for I/O-bound tasks such as HTTP requests. So if you find yourself looking at your untyped, sync Flask code from years ago, maybe take a look at what’s out there - it’s really worth it.
