Skip to content
Scraping & Data
-
Scraping & Data
Web Scraping Basics: robots.txt and urllib, Politely
Scraping lecture 1: a local practice shop, a first urllib fetch with an honest User-Agent, robotparser traps, and the real 404/403/timeout tracebacks.
-
Scraping & Data
Scraping Behind a Login: Cookies and CSRF in Python
Scraping lecture 4: log in to your own practice site with http.cookiejar and a CSRF token, then watch the redirect, token and expiry traps fail.
-
Scraping & Data
Scraper Robustness: Layout Changes, Alarms and Retries
Layout changes make scrapers fail silently. Add schema checks, zero-result alarms, charset fallback and capped retries, measured on a local practice shop.
-
Scraping & Data
Pagination, 429 and Backoff in Python Scraping
Scraping lecture 3: follow rel=next with stop conditions, honor Retry-After on 429, use seeded backoff, and see why 16 threads were no faster than 1.
-
Scraping & Data
Scraping Change Detection: SQLite Snapshots and Hashes
Store scraped prices in SQLite, detect real changes with field hashes, make reruns idempotent and schedule cron in UTC. Storage and commit cost measured.
-
Scraping & Data
Python html.parser: Build a 40-Line HTML Extractor
Scraping lecture 2: extract products with the stdlib html.parser, fix relative URLs with urljoin, clean whitespace and entities, and watch regex fail.
↑