Iterators in Python: What a for Loop Actually Does

A for loop is three steps you could write by hand. Once you have seen them, the empty-the-second-time bug stops being a surprise and becomes obvious.

The last post ended with a zip that came back empty the second time you used it. This post is why, and it is the idea the rest of the series is built on.

A for statement is not a primitive. It is shorthand for three things.

The loop, written out by hand

names = ['ada', 'grace', 'alan']

it = iter(names)
while True:
    try:
        name = next(it)
    except StopIteration:
        break
    print(name)

It prints:

ada
grace
alan

That is the whole for statement. Call iter() on the thing. Call next() on the result until it raises StopIteration. Catch that and stop.

Nothing else is going on. There is no list being walked by index, and no length being checked.

Iterable, iterator, and the difference that matters

names = ['ada', 'grace', 'alan']

it = iter(names)
print(type(names).__name__, '->', type(it).__name__)
print('an iterator returns itself: ', iter(it) is it)
print('a list hands out a new one:  ', iter(names) is iter(names))

It prints:

list -> list_iterator
an iterator returns itself:  True
a list hands out a new one:   False

An iterable is something you can get an iterator from. An iterator is the thing that remembers where you are.

A list is an iterable, and it holds no position — every for over it starts fresh, because every for asks for a new iterator. That is the only reason you can loop over a list twice.

Position is the whole story

names = ['ada', 'grace', 'alan']
it = iter(names)

print(next(it))
for name in it:          # picks up where next() left off
    print('loop:', name)

It prints:

ada
loop: grace
loop: alan

The for did not start at the beginning, because iter(it) gave back the same half-used iterator.

Which is exactly why this happens

scores = zip(['ada', 'grace'], [90, 85])

print('first :', list(scores))
print('second:', list(scores))

It prints:

first : [('ada', 90), ('grace', 85)]
second: []

zip returns an iterator, not an iterable that can produce fresh ones. The first list() ran it to the end. The second found it already at the end, which is what “empty” means here.

map, filter, reversed, enumerate, an open file and every generator behave the same way. If you need the values twice, store them:

pairs = list(zip(['ada', 'grace'], [90, 85]))

print('first :', pairs)
print('second:', pairs)

It prints:

first : [('ada', 90), ('grace', 85)]
second: [('ada', 90), ('grace', 85)]

Writing your own

Two methods. __iter__ returns the iterator, __next__ returns the next value or raises StopIteration.

class Countdown:
    def __init__(self, n):
        self.n = n

    def __iter__(self):
        return self

    def __next__(self):
        if self.n <= 0:
            raise StopIteration
        self.n -= 1
        return self.n + 1

for i in Countdown(3):
    print(i)

c = Countdown(3)
print(list(c))
print(list(c))

It prints:

3
2
1
[3, 2, 1]
[]

The loop works. The last line shows it has the same flaw as zip: the object is its own iterator, so it carries the position itself, so it is good for one pass.

Making it reusable

Split the two jobs. The container stays still; a separate object does the walking.

class CountdownIter:
    def __init__(self, n):
        self.n = n

    def __iter__(self):
        return self

    def __next__(self):
        if self.n <= 0:
            raise StopIteration
        self.n -= 1
        return self.n + 1

class Countdown2:
    def __init__(self, n):
        self.n = n

    def __iter__(self):
        return CountdownIter(self.n)

c = Countdown2(3)
print(list(c))
print(list(c))

It prints:

[3, 2, 1]
[3, 2, 1]

That is the same arrangement a list has, and now Countdown2 behaves like one.

It is also twenty lines to count down from three. The next post gets it to three lines.

next takes a default

Useful when you want the first item and do not want to guard the empty case yourself.

it = iter(['ada'])
print(next(it, 'nobody'))
print(next(it, 'nobody'))

It prints:

ada
nobody

Without the default, that second call raises StopIteration.

What to remember

• for x in thing means: iter(thing), then next() until StopIteration.

• An iterable can produce iterators. An iterator holds the position and returns itself from __iter__.

• Anything that returns an iterator — zip, map, filter, a file, a generator — is good for exactly one pass.

• Write __iter__ and __next__ on the same object and you have made something that works once. Return a fresh iterator from __iter__ if you want it reusable.