07.2 - CSV and JSON Data Processing

Theory 25 min Intermediate

---

CSV — Comma-Separated Values

The csv module handles reading and writing CSV files properly (handles quotes, commas in values, etc.).

Reading CSV

import csv

# Basic reader
with open("students.csv", newline="", encoding="utf-8") as f:
    reader = csv.reader(f)
    header = next(reader)   # skip header row
    for row in reader:
        print(row)   # ['Alice', '92', 'Montreal']

# DictReader — rows as dicts (recommended)
with open("students.csv", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    students = list(reader)

# students = [
#   {'name': 'Alice', 'score': '92', 'city': 'Montreal'},
#   {'name': 'Bob', 'score': '87', 'city': 'Toronto'},
# ]

# Convert types
students = [
    {**row, "score": int(row["score"])}
    for row in students
]

Writing CSV

import csv

data = [
    {"name": "Alice", "score": 92, "city": "Montreal"},
    {"name": "Bob",   "score": 87, "city": "Toronto"},
]

with open("output.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "score", "city"])
    writer.writeheader()
    writer.writerows(data)

CSV Best Practices

# Always use:
# - newline="" (let csv module handle line endings)
# - encoding="utf-8"
# - DictReader/DictWriter for clarity

# Handle different delimiters
with open("data.tsv", newline="") as f:
    reader = csv.reader(f, delimiter="\t")

# Handle quoted fields with commas
# The csv module handles this automatically
# "Smith, John",30,"Montreal, QC"  → ['Smith, John', '30', 'Montreal, QC']

---

JSON — JavaScript Object Notation

JSON is the standard format for APIs and configuration files.

Reading JSON

import json

# From file
with open("config.json", encoding="utf-8") as f:
    config = json.load(f)   # load() from file

# From string
json_str = '{"name": "Alice", "age": 30}'
data = json.loads(json_str)   # loads() from string (s = string)

print(data["name"])   # "Alice"
print(type(data))     # <class 'dict'>

Writing JSON

import json

data = {
    "name": "Alice",
    "age": 30,
    "skills": ["Python", "Docker", "SQL"],
    "address": {"city": "Montreal", "country": "Canada"}
}

# To file
with open("user.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2, ensure_ascii=False)

# To string
json_str = json.dumps(data, indent=2, ensure_ascii=False)
print(json_str)

JSON Type Mapping

Python	JSON
dict	object {}
list, tuple	array []
str	string ""
int, float	number
True/False	true/false
None	null

JSON with Custom Objects

import json
from datetime import datetime

class DateTimeEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime):
            return obj.isoformat()
        return super().default(obj)

data = {"name": "Event", "time": datetime.now()}
json.dumps(data, cls=DateTimeEncoder)
# '{"name": "Event", "time": "2026-03-29T10:00:00"}'

---

YAML (via PyYAML)

pip install pyyaml

import yaml

# config.yaml
# database:
#   host: localhost
#   port: 5432
# debug: false

with open("config.yaml") as f:
    config = yaml.safe_load(f)   # always use safe_load!

config["database"]["host"]   # "localhost"

---

Key Vocabulary

Term	Definition
CSV	Comma-Separated Values — tabular data as plain text
csv.DictReader	Reads CSV rows as OrderedDicts with header keys
csv.DictWriter	Writes dicts to CSV with header
newline=""	Required for csv module — prevents double newlines
JSON	JavaScript Object Notation — human-readable data interchange
json.load(f)	Parse JSON from a file object
json.loads(s)	Parse JSON from a string
json.dump(obj, f)	Serialize to JSON in a file
json.dumps(obj)	Serialize to JSON string
indent=2	Pretty-print JSON with 2-space indentation

---

Summary

• csv.DictReader reads CSV rows as dicts — the most useful reader
• Always use newline="" in open() when working with csv module
• json.load(f) for files; json.loads(s) for strings
• json.dump(obj, f, indent=2) for pretty-printed JSON files
• Custom objects need a custom JSONEncoder for JSON serialization
• Always use yaml.safe_load() — never yaml.load() (security risk)

---

📄️ 07.1 - File I/O

Open, read, write, and append text files using Python's built-in open() function and context managers

📄️ 07.2 - CSV & JSON

Read and write CSV datasets and JSON configuration files using Python's standard library

📄️ 07.3 - Context Managers

Write custom context managers with contextlib, navigate the filesystem with pathlib, and manage resources safely

📄️ Lab - Module 07

Build a data pipeline that reads CSV, processes records, and outputs JSON reports

📄️ Quiz - Module 07

30 questions on file I/O, CSV, JSON, pathlib, and context managers