Watchdog

An automated website monitoring system that runs scheduled crawls as containerised workloads in the cloud, keeps historical snapshots and sends alerts. Its application domain is technical SEO.

Stage
Delivered and handed over
Role
System designer and developer

Kind of work

  • Python
  • Docker
  • Screaming Frog
  • AWS orchestration
  • Monitoring automation

The problem

Websites change without notice. Seeing what changed, and when, needs a regular crawl and a history to compare it with.

A monitor has a second problem of its own. When it sends nothing, that should mean nothing changed, not that it failed to run.

My role

I designed and developed the system.

Implementation

  • Scheduled collection. Crawls run on a schedule, with no one starting them by hand.
  • Containerised workloads. Each run is a Docker container on AWS, started by a scheduler.
  • Point-in-time history. Data is kept in raw, staged and daily snapshot layers, so the state of a site on a given day can be recovered.
  • Change detection and alerts. The system detects changes and sends alerts.

The design behind it has four separate stages: collection, storage, comparison and alerting. It is meant to keep collected data apart from the comparison logic, so that a result can be checked against the data it came from, and to alert only when a change meets defined criteria.

The architecture treats pipeline observability separately from monitoring the target. Execution status and stage-level checks are intended to help distinguish a successful run with no meaningful changes from a failed or incomplete run, and make failures easier to investigate.

Separating the stages is meant to let one change without the others: a different source, or a different alert channel, without rewriting storage or comparison. The delivered system monitors websites. Other recurring comparisons would be further uses of the design.

Delivery

Watchdog was used to deliver SuperScan, a specific monitoring implementation that completed a series of consecutive clean production runs before handover.