Lecture 14 — List Comprehensions and Generators#
🎯 Learning Objectives
By the end of this lecture, you will be able to:
- ✅ Write list, dict, and set comprehensions to replace verbose loops
- ✅ Use conditional filtering inside comprehensions
- ✅ Nest comprehensions for multi-dimensional data
- ✅ Understand generators and the
yieldkeyword for memory-efficient iteration - ✅ Choose between comprehensions, generators, and traditional loops
📑 Table of Contents
- ● 💨 Lecture 14 — List Comprehensions & Generators
- 1. ▸ 💨 1. List Comprehensions
- 2. ▸ 1.1 Basic Syntax
- 3. ▸ 1.2 Filtering with Conditions
- 4. ▸ 1.3 If-Else in Comprehensions
- 5. ▸ 1.4 Nested Comprehensions
- 6. ▸ 📖 2. Dictionary & Set Comprehensions
- 7. ▸ 2.1 Dictionary Comprehensions
- 8. ▸ 2.2 Set Comprehensions
- 9. ▸ ⚙️ 3. Generator Expressions
- 10. ▸ 🔄 4. Generator Functions with yield
- 11. ▸ 🧭 5. Choosing the Right Tool
- 12. ▸ 📝 6. Practice Exercises
- 13. ▸ 🎯 Key Takeaways
for loops to process them. This lecture teaches you the Pythonic shorthand for the most common loop patterns — creating, transforming, and filtering collections in a single, readable line.
💨 1. List Comprehensions
A list comprehension is Python's most iconic feature — a concise, readable way to create a new list by applying an expression to each item in an iterable. It replaces the common pattern of "create empty list, loop, append" with a single expressive line.
1.1 Basic Syntax#
The general form is:
new_list = [expression for item in iterable]
This is equivalent to:
new_list = []
for item in iterable:
new_list.append(expression)
The comprehension is not only shorter — it’s also faster because Python optimizes it internally.
# Traditional loop approach
squares_loop = []
for x in range(10):
squares_loop.append(x ** 2)
# List comprehension — same result, one line
squares_comp = [x ** 2 for x in range(10)]
print(f"Loop: {squares_loop}")
print(f"Comprehension: {squares_comp}")
print(f"Same? {squares_loop == squares_comp}")
# More examples — the expression can be anything
import math
# String manipulation
names = ["alice", "bob", "charlie"]
capitalized = [name.capitalize() for name in names]
print(f"Capitalized: {capitalized}")
# Math operations
angles = [0, 30, 45, 60, 90]
radians = [math.radians(a) for a in angles]
sines = [round(math.sin(r), 4) for r in radians]
print(f"Sines: {sines}")
1.2 Filtering with Conditions#
Add an if clause to include only items that meet a condition:
new_list = [expression for item in iterable if condition]
This replaces the “loop + if + append” pattern that you used in Lecture 6.
# Even numbers from 0 to 19
evens = [x for x in range(20) if x % 2 == 0]
print(f"Evens: {evens}")
# Words longer than 4 characters
words = ["Python", "is", "a", "powerful", "language", "for", "science"]
long_words = [w for w in words if len(w) > 4]
print(f"Long words: {long_words}")
# Combine transformation AND filtering: squares of even numbers
even_squares = [x ** 2 for x in range(20) if x % 2 == 0]
print(f"Even squares: {even_squares}")
1.3 If-Else in Comprehensions#
When you need an else branch, the conditional goes before the for (not after):
# Filtering (no else): [expr for x in items if condition]
# Transforming (with else): [expr_true if condition else expr_false for x in items]
This is a common source of confusion — remember: if-only goes after, if-else goes before.
# Classify numbers
labels = ["even" if x % 2 == 0 else "odd" for x in range(10)]
print(labels)
# Replace negatives with 0 (clamping)
data = [4, -2, 7, -5, 3, -1, 8]
cleaned = [x if x >= 0 else 0 for x in data]
print(f"Original: {data}")
print(f"Cleaned: {cleaned}")
# Grade conversion
scores = [92, 45, 78, 88, 55, 96, 67]
grades = ["Pass" if s >= 60 else "Fail" for s in scores]
print(f"Grades: {grades}")
1.4 Nested Comprehensions#
You can nest for loops inside a comprehension. This is especially useful for working with matrices and multi-dimensional data — something you’ll encounter frequently with NumPy.
# Flatten a 2D matrix into a 1D list
matrix = [[1, 2, 3],
[4, 5, 6],
[7, 8, 9]]
flat = [x for row in matrix for x in row]
print(f"Flattened: {flat}")
# Create a multiplication table (list of lists)
table = [[i * j for j in range(1, 6)] for i in range(1, 6)]
for row in table:
print(row)
# All unique pairs (i, j) where i < j
pairs = [(i, j) for i in range(5) for j in range(5) if i < j]
print(f"Pairs: {pairs}")
# Transpose a matrix
transposed = [[row[i] for row in matrix] for i in range(3)]
print(f"Original: {matrix}")
print(f"Transposed: {transposed}")
📖 2. Dictionary & Set Comprehensions
The same comprehension syntax works for dictionaries (with {key: value}) and sets (with {value}). Recall these data structures from Lecture 5.
2.1 Dictionary Comprehensions#
new_dict = {key_expr: value_expr for item in iterable}
# Square lookup table
squares = {x: x ** 2 for x in range(11)}
print(squares)
# Word lengths
words = ["Python", "Data", "Science", "NumPy", "Pandas"]
word_lengths = {w: len(w) for w in words}
print(word_lengths)
# Invert a dictionary (swap keys and values)
original = {"a": 1, "b": 2, "c": 3}
inverted = {v: k for k, v in original.items()}
print(f"Original: {original}")
print(f"Inverted: {inverted}")
# Filter a dictionary
prices = {"apple": 1.2, "banana": 0.5, "cherry": 3.0, "date": 2.5}
expensive = {k: v for k, v in prices.items() if v > 1.5}
print(f"Expensive items: {expensive}")
2.2 Set Comprehensions#
Same syntax as list comprehensions but with curly braces {}. Duplicates are automatically removed — that’s what sets do!
# Unique lengths of words
words = ["hello", "world", "hi", "hey", "howdy", "help"]
unique_lengths = {len(w) for w in words}
print(f"Unique lengths: {unique_lengths}")
# Unique first letters
first_letters = {w[0].lower() for w in words}
print(f"First letters: {first_letters}")
⚙️ 3. Generator Expressions
A generator expression looks like a list comprehension but uses parentheses () instead of brackets. The critical difference: it produces values lazily — one at a time, on demand — instead of creating the entire collection in memory.
This matters enormously when working with large datasets. Imagine processing a 10GB file: a list comprehension would try to load everything into RAM, while a generator processes one item at a time.
# List comprehension: creates the entire list in memory
sum_list = sum([x ** 2 for x in range(1_000_000)])
# Generator expression: computes values on-the-fly
sum_gen = sum(x ** 2 for x in range(1_000_000))
print(f"Results match? {sum_list == sum_gen}")
import sys
# Memory comparison — the difference is dramatic
list_comp = [x ** 2 for x in range(100_000)]
gen_expr = (x ** 2 for x in range(100_000))
print(f"List size: {sys.getsizeof(list_comp):>10,} bytes")
print(f"Generator size: {sys.getsizeof(gen_expr):>10,} bytes")
print(f"Ratio: {sys.getsizeof(list_comp) / sys.getsizeof(gen_expr):.0f}x more memory for the list!")
🔄 4. Generator Functions with yield
A generator function uses yield instead of return. Each time you call next() on it, the function resumes from where it paused and runs until the next yield. This lets you create infinite or very large sequences without storing them in memory.
def countdown(n):
"""Count down from n to 1."""
print(f"Starting countdown from {n}")
while n > 0:
yield n # Pause here, return n, and resume on next call
n -= 1
print("Liftoff!")
# Using the generator in a for loop
for num in countdown(5):
print(num, end=" ")
print()
One of the most elegant uses of generators is creating infinite sequences. You can’t store infinity in a list, but a generator computes values on demand:
def fibonacci():
"""Infinite Fibonacci sequence generator."""
a, b = 0, 1
while True:
yield a
a, b = b, a + b
# Get the first 15 Fibonacci numbers
fib = fibonacci()
first_15 = [next(fib) for _ in range(15)]
print(f"Fibonacci: {first_15}")
def read_large_file(filepath):
"""Memory-efficient line-by-line file reader.
This pattern is essential for processing files larger than RAM —
a common situation in scientific computing and data science.
"""
with open(filepath, 'r') as f:
for line_number, line in enumerate(f, 1):
yield line_number, line.strip()
# Example: reading our sample data
# for num, line in read_large_file("sample_data.csv"):
# print(f"Line {num}: {line}")
🧭 5. Choosing the Right Tool
Situation |
Best Tool |
Why |
|---|---|---|
Transform a list into another list |
List comprehension |
Fast, readable, Pythonic |
Need unique values from a collection |
Set comprehension |
Automatic deduplication |
Build a key-value mapping |
Dict comprehension |
Concise, clear intent |
Process huge data, one pass only |
Generator expression |
Almost zero memory |
Complex logic, multiple yields, infinite sequences |
Generator function ( |
Full control over iteration |
Side effects, mutations, or I/O in the loop |
Traditional |
Most explicit and debuggable |
📝 6. Practice Exercises
Exercise 1: Using a list comprehension, create a list of all numbers from 1 to 100 that are divisible by both 3 and 5 (i.e., FizzBuzz numbers).
Exercise 2: Given the list of student dictionaries below, use a dict comprehension to create a {name: grade} mapping for students who passed (grade >= 60).
students = [
{"name": "Alice", "grade": 85},
{"name": "Bob", "grade": 42},
{"name": "Charlie", "grade": 91},
{"name": "Diana", "grade": 58},
]
Exercise 3: Write a generator function prime_numbers() that yields prime numbers indefinitely. Use it to print the first 20 primes.
Exercise 4: Use a nested comprehension to transpose the following 3x4 matrix, then flatten it:
matrix = [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
# Exercise 1: Your code here
Click to show solution
# Numbers from 1-100 divisible by both 3 and 5
fizzbuzz_numbers = [n for n in range(1, 101) if n % 3 == 0 and n % 5 == 0]
print(fizzbuzz_numbers) # [15, 30, 45, 60, 75, 90]
# Exercise 2: Your code here
Click to show solution
students = [
{"name": "Alice", "grade": 85},
{"name": "Bob", "grade": 42},
{"name": "Charlie", "grade": 91},
{"name": "Diana", "grade": 58},
]
passed = {s["name"]: s["grade"] for s in students if s["grade"] >= 60}
print(passed) # {'Alice': 85, 'Charlie': 91}
# Exercise 3: Your code here
Click to show solution
def prime_numbers():
"""Generator that yields prime numbers indefinitely."""
n = 2
while True:
is_prime = True
for d in range(2, int(n**0.5) + 1):
if n % d == 0:
is_prime = False
break
if is_prime:
yield n
n += 1
# Print first 20 primes
gen = prime_numbers()
first_20 = [next(gen) for _ in range(20)]
print(first_20)
# Exercise 4: Your code here
Click to show solution
matrix = [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
# Transpose: swap rows and columns
transposed = [[row[i] for row in matrix] for i in range(len(matrix[0]))]
print(f"Transposed: {transposed}")
# Flatten: single list of all elements
flattened = [x for row in matrix for x in row]
print(f"Flattened: {flattened}")
🎯 Key Takeaways
- List comprehensions create lists in one line:
[expr for x in iterable if cond]. - Dictionary and set comprehensions follow the same pattern:
{k: v for ...},{x for ...}. - Generator expressions use
()instead of[]and produce values lazily, saving memory. yieldturns a function into a generator — it pauses and resumes between calls.- Use generators for large datasets where loading everything into memory is impractical.
—
—