A project that scrapes websites and alerts the user of changes in those websites.
There are a few websites out there that I was interested in watching for updates. Originally, I looked for the usual ways of watching them for changes (social media, RSS, Google Alerts, etc.), but none of them really fit. I also looked into a variety of online tools that already existed, and there didn't seem to be any that fulfilled my requirements: pinging the websites on a weekly basis, giving me a list of changes, and most importantly affordable. So I decided to make my own!
Here's how it works!
The stack has two parts:
- The frontend and API: Shows the user the results of a website scrape, add and remove sites for scraping, and view changes that were detected.
- The weekly scraping process: Using an AWS Durable Function, a weekly process is kicked off to queue the user's websites and check them for changes using AI.
The Angular project (frontend), CDK project (backend), and Shared library are stored in separate folders, refer to them for more details on their structure.
Releases are made through GitHub Actions located here.
The previous version of the project used AWS Step Functions, which I converted to an AWS Durable Function. To view the old project, view the 1.5 tag.