Work
Keller Williams agent index
A resumable crawler that turns kw.com's geography-gated agent directory into a clean, deduplicated dataset for a whole state.
- Period
- 2025
- Role
- Built and shipped solo
- Stack
- Python · Playwright · CSV/JSON export
- Result
- ~2,000 agents indexed
kw.com/agents only answers questions you ask by geography, and only a page at
a time. Getting a statewide picture meant treating the directory as a pipeline
problem rather than a scraping problem.
The shape of it
- Enumerate the ZIP codes for a state.
- Resolve each ZIP to the internal boundary ID the site actually searches on — the step that isn’t visible from the outside, and the one that makes or breaks the run.
- Paginate every result set to exhaustion.
- Deduplicate across overlapping boundaries and export.
What made it hold up
A statewide crawl runs for hours, so every stage checkpoints to disk. Interrupt it and re-run the identical command and it picks up where it stopped rather than starting over — which is the difference between a script and something you can actually operate.
A completed run indexes roughly two thousand agents.