Lab 09 — Async Web Scraper with Generators

Hands-on Lab 60 min Advanced

Objectives

1. Write generator functions for URL generation
2. Use asyncio.gather for concurrent HTTP checks
3. Use itertools for batching requests
4. Process results with generator expressions
5. Compare sync vs async performance

---

Step 1 — Install Dependencies

mkdir ~/python-course/lab-09 && cd ~/python-course/lab-09
python3 -m venv venv && source venv/bin/activate
pip install aiohttp

---

Step 2 — Build the Scraper

Create url_checker.py:

#!/usr/bin/env python3
"""
Lab 09 — Async URL Checker
Uses generators, asyncio, and itertools.
"""

import asyncio
import itertools
import time
from collections import Counter


URLS = [
    "https://python.org",
    "https://docs.python.org",
    "https://pypi.org",
    "https://flask.palletsprojects.com",
    "https://pandas.pydata.org",
    "https://numpy.org",
    "https://github.com",
    "https://example.com",
    "https://httpbin.org/status/404",
    "https://httpbin.org/status/500",
]


# ── Generator Functions ────────────────────────────────────────

def url_batches(urls, batch_size=3):
    """Yield URLs in batches using itertools."""
    for batch in itertools.batched(urls, batch_size):
        yield list(batch)


def parse_status(results):
    """Generator that yields (url, status, category) tuples."""
    for url, status, elapsed in results:
        if status is None:
            category = "ERROR"
        elif status < 300:
            category = "OK"
        elif status < 400:
            category = "REDIRECT"
        elif status < 500:
            category = "CLIENT_ERROR"
        else:
            category = "SERVER_ERROR"
        yield url, status, elapsed, category


# ── Async HTTP ────────────────────────────────────────────────

async def check_url(session, url):
    """Check a single URL and return (url, status_code, elapsed)."""
    import aiohttp
    start = time.perf_counter()
    try:
        async with session.get(url, timeout=aiohttp.ClientTimeout(total=5)) as resp:
            elapsed = time.perf_counter() - start
            return url, resp.status, round(elapsed, 3)
    except Exception as e:
        elapsed = time.perf_counter() - start
        return url, None, round(elapsed, 3)


async def check_batch(urls):
    """Check a batch of URLs concurrently."""
    import aiohttp
    async with aiohttp.ClientSession() as session:
        tasks = [check_url(session, url) for url in urls]
        return await asyncio.gather(*tasks)


async def check_all_async(urls, batch_size=3):
    """Check all URLs in batches."""
    all_results = []
    for i, batch in enumerate(url_batches(urls, batch_size), 1):
        print(f"  Checking batch {i}: {len(batch)} URLs...")
        results = await check_batch(batch)
        all_results.extend(results)
    return all_results


# ── Report ─────────────────────────────────────────────────────

def print_report(results):
    parsed = list(parse_status(results))
    categories = Counter(cat for _, _, _, cat in parsed)

    print(f"\n{'='*65}")
    print(f"  {'URL':<40} {'STATUS':>8} {'TIME':>8}  CAT")
    print(f"{'─'*65}")
    for url, status, elapsed, cat in sorted(parsed, key=lambda x: x[3]):
        status_str = str(status) if status else "ERR"
        print(f"  {url[:40]:<40} {status_str:>8} {elapsed:>7.3f}s  {cat}")

    print(f"{'='*65}")
    print(f"\n  Summary:")
    for cat, count in sorted(categories.items()):
        bar = "█" * count
        print(f"    {cat:<15} {bar} ({count})")


async def main():
    print("🔍  Async URL Checker — Lab 09\n")

    start = time.perf_counter()
    results = await check_all_async(URLS, batch_size=3)
    total = time.perf_counter() - start

    print_report(results)
    print(f"\n  ⏱  Total time: {total:.2f}s for {len(URLS)} URLs\n")


if __name__ == "__main__":
    asyncio.run(main())

---

Step 3 — Run

python3 url_checker.py

Verification ✅

The 10 URLs should complete in ~3-5s (batches of 3 in parallel), not 10× single URL time.

---

Summary

• url_batches() generator uses itertools.batched to yield URL groups
• parse_status() generator transforms raw results without storing all in memory
• asyncio.gather() runs all URLs in a batch concurrently
• check_all_async() processes batches sequentially but each batch is concurrent
• Generator pipeline: url_batches → check_batch → parse_status → report