Counter, defaultdict and deque in Python

Counting things, grouping things and keeping a queue are three loops most people write by hand. The collections module already has all three.

Everything here is in the standard library.

Counting by hand

The version with dict.get is the common one.

# counting the long way
words = ['apple', 'bread', 'apple', 'milk', 'bread', 'apple']

counts = {}
for w in words:
    counts[w] = counts.get(w, 0) + 1
print(counts)

It prints:

{'apple': 3, 'bread': 2, 'milk': 1}

Counter

Counter takes any iterable and counts it. Missing keys return 0 rather than raising.

# Counter does it in one line
from collections import Counter, defaultdict, deque

c = Counter(words)
print(c)
print(c.most_common(2))
print(c['apple'], c['missing'])

It prints:

Counter({'apple': 3, 'bread': 2, 'milk': 1})
[('apple', 3), ('bread', 2)]
3 0

most_common is the reason people reach for it. Sorting a dict by value takes a line and a lambda.

Counter arithmetic

Counters add and subtract. Subtraction drops anything that reaches zero or below.

# Counter arithmetic
stock = Counter(apple=5, bread=2)
sold = Counter(apple=3, bread=1, milk=1)
print(stock - sold)
print(stock + sold)

It prints:

Counter({'apple': 2, 'bread': 1})
Counter({'apple': 8, 'bread': 3, 'milk': 1})

Handy for stock levels, diffs between two counts and simple bag of words work.

Grouping with defaultdict

defaultdict(list) calls list() for any key that does not exist, so you can append without checking first.

# grouping with defaultdict
people = [('eng', 'ada'), ('ops', 'grace'), ('eng', 'alan')]
groups = defaultdict(list)
for team, name in people:
    groups[team].append(name)
print(dict(groups))

It prints:

{'eng': ['ada', 'alan'], 'ops': ['grace']}

The defaultdict gotcha

Reading a missing key creates it. This is the behaviour that surprises people, usually when a dict has more keys than expected after a lookup loop.

# the gotcha, reading creates the key
d = defaultdict(list)
print(d['not-there'])
print(dict(d))

plain = {}
print(plain.get('not-there'))
print(plain)

It prints:

[]
{'not-there': []}
None
{}

A plain dict with .get() does not do this. If you only want a safe read, use .get().

setdefault does the same job

If you would rather not import anything, setdefault gets you there.

# setdefault does the same without a factory
grouped = {}
for team, name in people:
    grouped.setdefault(team, []).append(name)
print(grouped)

It prints:

{'eng': ['ada', 'alan'], 'ops': ['grace']}

It is slower in a tight loop, because the empty list is built on every pass whether it is needed or not.

deque for queues

Removing from the front of a list is slow, because everything after it shifts. A deque adds and removes at both ends in constant time.

# deque for queues, because list.pop(0) is slow
q = deque(['a', 'b', 'c'])
q.appendleft('start')
print(q.popleft(), list(q))

recent = deque(maxlen=3)
for item in range(6):
    recent.append(item)
print(list(recent))

It prints:

start ['a', 'b', 'c']
[3, 4, 5]

maxlen gives you a rolling window that discards the oldest item, which is a neat fit for recent items and moving averages.

What to remember

• Counter counts any iterable and gives you most_common.
• Counters support + and -.
• defaultdict creates the key on read, not only on write.
• deque is the right type when you pop from the front.

The same module also has namedtuple and ChainMap. Reading through collections once is one of the better hours you can spend with the standard library.