10 Python Generator Things I Wish I Knew Way Earlier

1) Generator function VS regular function

Regular function

A regular function uses return to output something.

Generator function

A generator function uses yield instead of return. Example:

2) yield vs return

The return keyword

return outputs something from the function. After return runs, the function stops immediately, and nothing else inside the function runs. A function can only return 1 value.

The yield keyword

Like return, yield outputs something from the function. However, after yield runs, the function doesn’t stop immediately. Which means that a function can yield multiple values.

3) Generator functions do lazy loading

Lazy loading

Values are generated only when invoked.

Wrong way to use generator functions

If we directly call our generator function, it shows a generator object. Our generator object only starts to output stuff when we iterate through it.

Correct way to use generator functions

4) Benefit of using generator functions

Regular function returning a list

What happens when we call regular_function(5)

1. the full output list [0, 1, 4, 9, 16] is first evaluated

2. [0, 1, 4, 9, 16] is stored in memory

3. [0, 1, 4, 9, 16] is returned all at once using return

Generator function yielding values from similar list

What happens when we call generator_function(5)

1. the value 0 is generated, and output using yield immediately

2. the value 1 is generated, and output using yield immediately

3. the value 4 is generated, and output using yield immediately

4. the value 9 is generated, and output using yield immediately

5. the value 16 is generated, and output using yield immediately

When we use a generator function here, we do not store the entire list [0, 1, 4, 9, 16] in memory.

The caveat of using non-generator functions

Imagine we use an input size of 1000000000. Next, imagine that instead of simple numbers, each element is a object with nested fields. We don’t want to store this abomination in memory as our computer might actually run out of memory.

Benefits of using generator functions

1. Lower memory usage — not everything in an iterable is stored in memory

2. Faster startup time — not everything in an iterable has to be processed before the first processed output is yielded

3. generators can be infinite unlike regular functions

Quick Pause

Consider buying my ebook — 256 Python Things I Wish I Knew Earlier — a collection of hard-earned insights from writing 500,000+ lines of code over 10 years. Save yourself years of trial and error.

Link: https://payhip.com/b/xpRco

5) Examples of common built-in generator functions

range()

When we use range(1_000_000), Python doesn’t store 1 million numbers in memory. Instead, range is a generator function that yields each number as we invoke it.

enumerate()

When we use enumerate(mylist), we generate a (index, value) tuple for each value in mylist. But this is actually done lazily (one by one as we invoke it), and Python doesn’t store the entire tuple list in memory.

zip()

When we use zip(mylist1, mylist2), we iterate through mylist1 and mylist2 at the same time. Similarly, this is done lazily (one by one as we invoke it), and Python doesn’t store the entire tuple list in memory.

6) Generators VS Iterators

Iterator

An iterator is an object that has __iter__() and __next__(). We use iterators for lazy iteration (getting stuff only when we need it)

We can create iterator objects by calling iter() or .__iter__() on iterables (stuff that can be iterated through)

Generator

A generator is a special kind of iterator.

Either created by 1) a function containing yield or 2) a generator expression (covered later)

Conclusion

A generator is a subset of iterator.

All generators are iterators, but not all iterators are generators.

7) next() with iterators/generators

next() with iterators

next(my_iterator) returns the next value in my_iterator, and increase the internal index in my_iterator by 1.

Whenever we call next(fruit_iterator), the next element is returned. When fruit_iterator runs out of stuff to output, a StopIteration exception is raised.

next() with generators

All generators are iterators, but not all iterators are generators. So next() works the same way for generators.

How to not raise StopIteration

If you really don’t want StopIteration to be raised, we can add a default value to the second positional argument of next()

8) Generator expressions: lazy list comprehensions

List comprehensions

List comprehensions are executed eagerly (opposite of lazily) ie. as soon as we evaluate it.

Generator expressions

Generator expressions are essentially lazy list comprehensions. They are executed lazily ie only when we need it.

You can think of a generator expression as a way to write a generator in one line without having to define a function.

9) generator.send() sends a value to the generator

Using .send(), we can send values to our generator to change its behaviour.

Example run

Under normal circumstances, if we don’t send anything to our generator, n simply increases by 1.

However, if we send something to our generator using .send(), it resets n

Explaining value_from_send = yield n

When .send() doesn’t run, value_from_send defaults to None

When .send(100) runs, value_from_send is assigned to 100

What happens when .send(100) runs

1. In value_from_send = yield n, value_from_send becomes 100

2. as value_from_send is not None, n = 100

3. The next yield statement thus yields 100

An example run

10) The `yield from` keyword

yield from is used from inside a generator function.

It is used to delegate some work to another iterable or generator.

Simple example

Negative example: so that we don’t have to do this

Key benefits

• Fewer manual loops and cleaner code

• We’re better able to chain multiple generators together

Conclusion

Hopefully this was clear and easy to understand

If You Wish To Support Me As A Writer

• Buy my Ebook — 256 Python Things I Wish I Knew Earlier at https://payhip.com/b/xpRco — a collection of hard-earned insights from writing 500,000+ lines of code over 10+ years. Save yourself from years of trial and error

Thank you — these actions go a long way, and I really appreciate it.