Lecture 17 — Data Formats and APIs#
🎯 Learning Objectives
- ✅ Read and write JSON data — the universal data exchange format
- ✅ Work with CSV files using Python's built-in
csvmodule - ✅ Understand YAML and TOML configuration formats
- ✅ Fetch data from web APIs using the
requestslibrary - ✅ Parse and process real-world API responses for data analysis
📑 Table of Contents
pd.read_csv() to load tabular data. Now we'll go deeper: understanding the data formats themselves and connecting to live web APIs to fetch real data programmatically.1. JSON — JavaScript Object Notation
JSON is the lingua franca of data exchange on the web. Every API you'll ever use speaks JSON. It maps directly to Python's dictionaries and lists, making it intuitive to work with.
JSON ↔ Python type mapping:| JSON | Python ||——|——–|| {} object | dict || [] array | list || "string" | str || 123 / 3.14 | int / float || true / false | True / False || null | None |
The mapping is almost one-to-one, which is why JSON feels so natural in Python. The json module in Python’s standard library handles all the conversion automatically. Let’s start with the two most common operations: serialization (Python → JSON string) and deserialization (JSON string → Python):
import json# Python dict → JSON stringdata = { "name": "Alice", "age": 28, "languages": ["Python", "R", "Julia"], "is_student": False, "address": { "city": "Dakar", "country": "Senegal" }}json_string = json.dumps(data, indent=2)print(json_string)print(f"\nType: {type(json_string)}") # str
json.dumps() converted our Python dictionary into a formatted JSON string. Notice how Python’s False became JSON’s false, and nested structures (the address dict, the languages list) are preserved perfectly. The indent=2 parameter makes the output human-readable — without it, everything would be on one line.Now let’s go the other direction — parsing a JSON string back into Python objects:
# JSON string → Python dictparsed = json.loads(json_string)print(f"Name: {parsed['name']}")print(f"First language: {parsed['languages'][0]}")print(f"City: {parsed['address']['city']}")print(f"Type: {type(parsed)}") # dict
Once parsed, the JSON data becomes regular Python objects — you navigate it with standard dictionary indexing and list slicing. This is the core workflow for working with API data: receive JSON, parse it, extract what you need.In practice, you’ll often read JSON from files rather than strings:
# Reading and writing JSON filesimport os# Write to filewith open("sample_data.json", "w") as f: json.dump(data, f, indent=2)print("Written: sample_data.json")# Read from filewith open("sample_data.json", "r") as f: loaded = json.load(f)print(f"Loaded: {loaded['name']} from {loaded['address']['city']}")# Clean upos.remove("sample_data.json")
json.dumps() / json.loads() work with strings (the 's' stands for 'string'). json.dump() / json.load() (without 's') work with files. A common source of confusion!What happens when your Python data contains types that JSON doesn’t support — like datetime objects, sets, or custom classes? By default, json.dumps() raises a TypeError. You can solve this with a custom encoder:
# Handling special cases: dates, sets, custom objectsfrom datetime import datetimeclass DateEncoder(json.JSONEncoder): """Custom encoder that handles datetime objects.""" def default(self, obj): if isinstance(obj, datetime): return obj.isoformat() return super().default(obj)event = { "name": "Python Workshop", "date": datetime(2025, 11, 15, 14, 30), "tags": ["python", "data-science"]}print(json.dumps(event, cls=DateEncoder, indent=2))
.ipynb file in a text editor!). Machine learning experiment trackers like MLflow and Weights & Biases log metrics as JSON. And virtually every web API — from weather data to genomic databases — returns JSON responses.2. CSV — Comma-Separated Values
CSV is the simplest tabular data format — just rows of values separated by commas (or other delimiters). While Pandas is the best tool for heavy CSV work, Python's built-in csv module is lightweight, dependency-free, and perfect for quick tasks.
import csv, os# Writing CSVstudents = [ {"name": "Alice", "age": 22, "grade": "A"}, {"name": "Bob", "age": 24, "grade": "B+"}, {"name": "Charlie", "age": 21, "grade": "A-"}, {"name": "Diana", "age": 23, "grade": "B"},]with open("students.csv", "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=["name", "age", "grade"]) writer.writeheader() writer.writerows(students)print("Written: students.csv")
DictWriter is especially convenient because you work with dictionaries — each row is a {column: value} mapping, so your code reads naturally. The fieldnames parameter controls the column order.Now let’s read the data back using the mirror class, DictReader:
# Reading CSVwith open("students.csv", "r") as f: reader = csv.DictReader(f) for row in reader: print(f"{row['name']:>10} | Age {row['age']} | Grade: {row['grade']}")
DictReader automatically uses the first row as column headers and returns each subsequent row as a dictionary. This is the recommended approach for most CSV work.For simpler needs (or when you want raw list access), the basic csv.reader returns each row as a list:
# Reading with the basic csv.reader (returns lists)with open("students.csv", "r") as f: reader = csv.reader(f) header = next(reader) # Skip header row print(f"Columns: {header}") for row in reader: print(row)os.remove("students.csv")
newline='' when opening CSV files for writing on Windows, otherwise you'll get blank lines between rows. The csv module handles line endings internally.csv module reads everything as strings. If you need age as an integer, you must convert it yourself: int(row['age']). This is one reason why Pandas' read_csv() is preferred for analysis — it automatically infers data types.3. Configuration Formats: YAML & TOML
While JSON is great for data exchange, it's not ideal for human-written configuration files (no comments, strict syntax). YAML and TOML are designed for exactly this use case.
# YAML example (requires: pip install pyyaml)import yamlconfig_yaml = """# Database settingsdatabase: host: localhost port: 5432 name: myapp_db# Feature flagsfeatures: dark_mode: true beta_users: - alice - bob"""config = yaml.safe_load(config_yaml)print(f"DB Host: {config['database']['host']}")print(f"Beta users: {config['features']['beta_users']}")print(f"Type: {type(config)}") # dict
YAML’s strength is readability — it uses indentation (like Python!) to represent structure, supports comments with #, and avoids the noise of curly braces and quotes. It’s the standard format for Docker Compose files, GitHub Actions workflows, and Kubernetes configurations.TOML is a newer alternative that’s become Python’s standard for project configuration:
# TOML — Python 3.11+ has it built in!# For older Python: pip install tomlitry: import tomllib # Python 3.11+except ModuleNotFoundError: import tomli as tomllib # pip install tomli# TOML is great for project configs (like pyproject.toml)toml_string = """[project]name = "my-app"version = "1.0.0"description = "A cool Python app"[project.dependencies]numpy = ">=1.24"pandas = ">=2.0"[tool.pytest]testpaths = ["tests"]"""# Parse TOML stringimport ioconfig = tomllib.load(io.BytesIO(toml_string.encode()))print(f"Project: {config['project']['name']} v{config['project']['version']}")print(f"Dependencies: {config['project']['dependencies']}")
Every Python project you’ll encounter has a pyproject.toml file — it’s where dependencies, build settings, and tool configurations live. Understanding TOML means you can confidently edit project settings and create your own Python packages.Here’s a practical guide for choosing the right format:
When to use which format:| Format | Best For | Comments? | Human-Friendly? ||——–|———-|———–|—————–|| JSON | API data, data exchange | No | Medium || CSV | Tabular data, spreadsheets | No | Yes || YAML | Config files, CI/CD | Yes | Very || TOML | Python project config | Yes | Very |
Now that you know how to read and write structured data in multiple formats, let’s tackle the most exciting part of this lecture: fetching data from the internet programmatically. In the real world, data doesn’t just sit in files on your computer — it lives on web servers, and you access it through APIs.
4. Fetching Data from Web APIs
An API (Application Programming Interface) lets your Python code talk to web services and retrieve data programmatically. Most modern APIs return JSON. The requests library makes HTTP calls simple and intuitive.
# Install if needed: pip install requestsimport requests# GET request — fetch data from a free public APIresponse = requests.get("https://jsonplaceholder.typicode.com/users/1")print(f"Status code: {response.status_code}") # 200 = successprint(f"Content type: {response.headers['content-type']}")# Parse the JSON responseuser = response.json() # Automatically converts JSON → dictprint(f"\nUser: {user['name']}")print(f"Email: {user['email']}")print(f"City: {user['address']['city']}")
Let’s break down what happened:1. requests.get(url) sent an HTTP GET request to the server (like typing the URL in a browser)2. The server responded with a status code (200 = success) and JSON data3. response.json() parsed the JSON into a Python dictionary — the same json.loads() we learned earlier, but built into the response objectThe params argument lets you add query parameters (like filters) to your request:
# Fetching a list of itemsresponse = requests.get("https://jsonplaceholder.typicode.com/posts", params={"userId": 1})posts = response.json()print(f"User 1 has {len(posts)} posts:\n")for post in posts[:3]: print(f" [{post['id']}] {post['title'][:50]}...")
200 = success, 404 = not found, 401 = unauthorized, 500 = server error. Always check response.status_code before parsing the response.In production code, you should always handle potential errors — the server might be down, the network might be slow, or the URL might be wrong. Here’s a robust pattern using the techniques from Lecture 9 (Error Handling):
# Robust API calls with error handlingdef fetch_json(url, params=None): """Fetch JSON from a URL with error handling.""" try: response = requests.get(url, params=params, timeout=10) response.raise_for_status() # Raises HTTPError for 4xx/5xx return response.json() except requests.exceptions.Timeout: print("Request timed out!") except requests.exceptions.HTTPError as e: print(f"HTTP error: {e}") except requests.exceptions.RequestException as e: print(f"Request failed: {e}") return None# Test with valid and invalid URLsdata = fetch_json("https://jsonplaceholder.typicode.com/todos/1")if data: print(f"Todo: {data['title']} (completed: {data['completed']})")fetch_json("https://jsonplaceholder.typicode.com/invalid_endpoint")
The fetch_json() helper function encapsulates all the boilerplate: timeouts, status checking, JSON parsing, and error handling. You’ll reuse this pattern in every project that talks to APIs. Notice how it returns None on failure — callers can check if data: before proceeding.
5. From API to Analysis: The Data Pipeline
The real power comes from combining APIs with the tools you've learned. Let's fetch data from an API and analyze it with Pandas.
import pandas as pd# Fetch all todos from the APItodos = fetch_json("https://jsonplaceholder.typicode.com/todos")if todos: # Convert to DataFrame df = pd.DataFrame(todos) print(f"Shape: {df.shape}") print(f"\nFirst 5 rows:") print(df.head()) print(f"\n--- Completion rate by user ---") summary = df.groupby("userId")["completed"].agg(["sum", "count"]) summary["completion_rate"] = (summary["sum"] / summary["count"] * 100).round(1) summary.columns = ["Completed", "Total", "Rate (%)"] print(summary)
In just a few lines, we went from a URL to a fully analyzed DataFrame with summary statistics. This is the data pipeline pattern you’ll use constantly:API → JSON → DataFrame → Analysis → InsightsReal-world APIs often return data in pages. Here’s how to fetch and combine multiple pages:
# Fetching multiple pages of dataall_comments = []for post_id in range(1, 6): # First 5 posts comments = fetch_json( "https://jsonplaceholder.typicode.com/comments", params={"postId": post_id} ) if comments: all_comments.extend(comments)df_comments = pd.DataFrame(all_comments)print(f"Fetched {len(df_comments)} comments across 5 posts")print(f"\nAverage comments per post: {len(df_comments) / 5:.0f}")print(f"\nSample email domains:")df_comments["domain"] = df_comments["email"].str.split("@").str[1]print(df_comments["domain"].value_counts().head())
6. Practice Exercises
Exercise 1 (JSON): Create a nested Python dictionary representing a library catalog with at least 3 books (each having title, author, year, and genres). Save it to library.json, read it back, and print all books published after 2000.Hint: Use json.dump() to write and json.load() to read. Filter with a list comprehension: [b for b in books if b["year"] > 2000].Exercise 2 (CSV): Write a list of 5 product dictionaries (with name, price, quantity) to products.csv using csv.DictWriter. Read the file back and calculate the total inventory value (sum of price × quantity for each product).Hint: Remember that csv reads everything as strings — you’ll need float(row["price"]) and int(row["quantity"]).Exercise 3 (API → Pandas): Fetch the first 10 users from https://jsonplaceholder.typicode.com/users. Create a Pandas DataFrame and find: (a) all users whose company website ends in ".org", and (b) the most common city.*Hint:* Use response.json()[:10], then filter with df[df[“website”].str.endswith(“.org”)].**Exercise 4 (Challenge):** Build a function download_dataset(url, filename)that fetches JSON from a URL, converts it to a DataFrame, saves it as both CSV and JSON files, and prints a summary (shape, columns, first 3 rows). Include proper error handling for network and parsing failures.*Hint:* Combinefetch_json(), pd.DataFrame(), df.to_csv(), and df.to_json()` in sequence.
# Exercise 1: JSON library catalog
Click to show solution
import json
catalog = {
"library": "City Central Library",
"books": [
{
"title": "Python Crash Course",
"author": "Eric Matthes",
"year": 2019,
"genres": ["Programming", "Education"],
},
{
"title": "Clean Code",
"author": "Robert C. Martin",
"year": 2008,
"genres": ["Software Engineering"],
},
{
"title": "The Pragmatic Programmer",
"author": "David Thomas",
"year": 1999,
"genres": ["Software Engineering", "Career"],
},
],
}
# Save to JSON
with open("library.json", "w") as f:
json.dump(catalog, f, indent=2)
# Read it back
with open("library.json") as f:
data = json.load(f)
# Books published after 2000
recent = [b for b in data["books"] if b["year"] > 2000]
for b in recent:
print(f"{b['title']} ({b['year']}) by {b['author']}")
# Exercise 2: CSV product inventory
Click to show solution
import csv
products = [
{"name": "Laptop", "price": 999.99, "quantity": 10},
{"name": "Mouse", "price": 29.99, "quantity": 150},
{"name": "Keyboard", "price": 59.99, "quantity": 75},
{"name": "Monitor", "price": 349.99, "quantity": 30},
{"name": "Headset", "price": 79.99, "quantity": 60},
]
# Write to CSV
with open("products.csv", "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=["name", "price", "quantity"])
writer.writeheader()
writer.writerows(products)
# Read back and calculate total inventory value
total_value = 0
with open("products.csv") as f:
reader = csv.DictReader(f)
for row in reader:
value = float(row["price"]) * int(row["quantity"])
total_value += value
print(f"{row['name']}: ${value:,.2f}")
print(f"\nTotal inventory value: ${total_value:,.2f}")
# Exercise 3: API → Pandas analysis
Click to show solution
import requests
import pandas as pd
response = requests.get("https://jsonplaceholder.typicode.com/users")
users = response.json()[:10]
df = pd.DataFrame(users)
# (a) Users whose website ends in ".org"
org_users = df[df["website"].str.endswith(".org")]
print("Users with .org websites:")
print(org_users[["name", "website"]])
# (b) Most common city
# The 'address' column contains dicts; extract city
df["city"] = df["address"].apply(lambda a: a["city"])
most_common = df["city"].value_counts().idxmax()
print(f"\nMost common city: {most_common}")
# Exercise 4: download_dataset function
Click to show solution
import requests
import pandas as pd
def download_dataset(url, filename):
"""Fetch JSON from URL, convert to DataFrame, save as CSV and JSON."""
try:
response = requests.get(url, timeout=10)
response.raise_for_status()
data = response.json()
except requests.RequestException as e:
print(f"Network error: {e}")
return
except ValueError as e:
print(f"JSON parsing error: {e}")
return
df = pd.DataFrame(data)
df.to_csv(f"{filename}.csv", index=False)
df.to_json(f"{filename}.json", orient="records", indent=2)
print(f"Shape: {df.shape}")
print(f"Columns: {list(df.columns)}")
print(f"\nFirst 3 rows:")
print(df.head(3))
# Test with JSONPlaceholder posts
download_dataset(
"https://jsonplaceholder.typicode.com/posts",
"posts_dataset"
)
🎯 Key Takeaways
- JSON is the standard format for web APIs:
json.loads()/json.dumps(). - CSV is the most common tabular format: use
pandas.read_csv()for easy loading. - YAML and TOML are human-friendly configuration formats.
- The
requestslibrary fetches data from web APIs:requests.get(url).json(). - A data pipeline flows: fetch from API → parse response → clean → analyze → visualize.
—
| ← Previous 🎀 Lecture 16: Decorators & Context Managers | 📚 | 🎉 Course Complete! |