Watchdog
An automated website monitoring system that runs scheduled crawls as containerised workloads in the cloud, keeps historical snapshots and sends alerts. Its application domain is technical SEO.
- Stage
- Delivered and handed over
- Role
- System designer and developer
Kind of work
- Python
- Docker
- Screaming Frog
- AWS orchestration
- Monitoring automation
The problem
Websites change without notice. Seeing what changed, and when, needs a regular crawl and a history to compare it with.
A monitor has a second problem of its own. When it sends nothing, that should mean nothing changed, not that it failed to run.
My role
I designed and developed the system.
Implementation
- Scheduled collection. Crawls run on a schedule, with no one starting them by hand.
- Containerised workloads. Each run is a Docker container on AWS, started by a scheduler.
- Point-in-time history. Data is kept in raw, staged and daily snapshot layers, so the state of a site on a given day can be recovered.
- Change detection and alerts. The system detects changes and sends alerts.
The design behind it has four separate stages: collection, storage, comparison and alerting. It is meant to keep collected data apart from the comparison logic, so that a result can be checked against the data it came from, and to alert only when a change meets defined criteria.
The architecture treats pipeline observability separately from monitoring the target. Execution status and stage-level checks are intended to help distinguish a successful run with no meaningful changes from a failed or incomplete run, and make failures easier to investigate.
Separating the stages is meant to let one change without the others: a different source, or a different alert channel, without rewriting storage or comparison. The delivered system monitors websites. Other recurring comparisons would be further uses of the design.
Delivery
Watchdog was used to deliver SuperScan, a specific monitoring implementation that completed a series of consecutive clean production runs before handover.