Python Micro-optimisations

August 1, 2026

In this post, I outline some simple micro-optimisations to be wary of when involved with large data-intensive tooling. Measurements stated here are automated and produced by the script, bin/benchmarks.py.

Environment #

• Python: 3.14.5 (CPython)
• Machine: Apple M2, macOS-26.5.2
• Last run: 2026-08-05

Summary #

Each benchmark links to its measurement below: winner is the best variant, ratio is how many times worse the worst variant fared.

Speed #

Membership test: set vs list #

A set lookup is O(1); a list scan is O(n). Here x = 999 is the worst case - the last element of the list.

lst = list(range(1000)); st = set(lst); x = 999

# x in st
x in st

# x in lst
x in lst

Dict lookup with a default (key missing) #

The same three patterns as the hit case, but now the key is absent. try/except goes from cheapest to worst by far: raising and catching KeyError creates, propagates, and discards an exception object every time. d.get costs the same whether it hits or misses, and the in check gets cheaper.

d = {'k': 1}

# try/except
try:
    v = d['missing']
except KeyError:
    v = 0

# d.get
v = d.get('missing', 0)

# in check
v = d['missing'] if 'missing' in d else 0

Converting a tuple to a list #

[*tup] and list(tup) copy the underlying pointer array in C; the comprehension pushes every element through the bytecode loop.

tup = tuple(range(1000))

# [*tup]
[*tup]

# list(tup)
list(tup)

# comprehension
[x for x in tup]

Building a string: join vs += #

join makes one pass over the parts; += re-copies the growing string on every iteration.

parts = ['abc'] * 1000

# ''.join
''.join(parts)

# += loop
s = ''
for p in parts:
    s += p

Summing: builtin sum vs manual loop #

The builtin iterates in C, skipping bytecode dispatch per element.

data = list(range(1000))

# sum(data)
sum(data)

# manual loop
total = 0
for x in data:
    total += x

Queue from the front: deque vs list #

deque.popleft is O(1); list.pop(0) shifts every remaining element.

from collections import deque; rng = range(1000)

# deque.popleft
d = deque(rng)
while d:
    d.popleft()

# list.pop(0)
l = list(rng)
while l:
    l.pop(0)

Squaring: x*x vs x**2 #

Exponentiation goes through the generic power protocol; plain multiplication does not.

# x*x (int)
x = 42
x*x

# x**2 (int)
x = 42
x**2

# x*x (float)
x = 42.5
x*x

# x**2 (float)
x = 42.5
x**2

Checking for None: is vs == #

is is a single pointer comparison; == dispatches through the __eq__ protocol.

x = None

# x is None
x is None

# x == None
x == None

# not x
not x

Dict lookup with a default #

The key is present here, so try/except costs nothing; the in check looks up the key twice, and d.get costs a method call. The ranking inverts when the key is absent - see the miss case.

d = {'k': 1}

# try/except
try:
    v = d['k']
except KeyError:
    v = 0

# d.get
v = d.get('k', 0)

# in check
v = d['k'] if 'k' in d else 0

String formatting #

f-strings are compiled to dedicated bytecode; % and .format go through runtime parsing or a method call.

name = 'x'; val = 42

# f-string
f'{name}: {val}'

# % operator
'%s: %d' % (name, val)

# .format
'{}: {}'.format(name, val)

Building a list: comprehension vs append #

Comprehension runs on a specialised bytecode path; the loop costs a method call per element.

data = list(range(1000))

# comprehension
out = [x for x in data]

# append hoisted
out = []
append = out.append
for x in data:
    append(x)

# append loop
out = []
for x in data:
    out.append(x)

join: list comprehension vs generator #

str.join needs two passes, it first drains a generator into a list - handing it a list comprehension skips the per-element next() calls. This is the speed side of the trade-off in the memory comparison.

seq = list(range(1000))

# list comp
''.join([str(x) for x in seq])

# generator
''.join(str(x) for x in seq)

Memory #

Generator vs list #

Both compute the same sum, but the list comprehension materialises all 100,000 intermediate values before summing, while the generator holds one element at a time. (The list version is typically slightly faster - this is a speed/memory trade-off.)

# generator
total = sum(x*x for x in range(100_000))

# list comp
total = sum([x*x for x in range(100_000)])