CrawlyMcCrawlFace

What It Does

A few of my projects over the years and at present have used homegrown spiders to crawl various website collections, particularly in search of information about veg restaurants. I have broken out that code into a modular fetch and analyze architecture now that I have multiple products in need of this type of functionality.

How it works

It's a python script running on a fairly simple SQLite database. The spider respects robots.txt and doesn't scrape anywhere it's not allowed and chooses conservative limits on hitting servers. There are several tiers available for analysis depending on the complexity of the site. Content is downloaded and saved, cached for n days and custom analysis modules are run as needed to pull the data.

Built with

• Python 3.12, SQLite (WAL), httpx, curl_cffi, Playwright/Chromium, pypdf, TOML config and recipes, and a stdlib http.server dashboard.