Sitemap Alert
Project Description
The aim of the project is to allow users to monitor up to 15 websites and track what pages are added every week and report that through an e-mail.
I built a simple UI with proper error handling using Next.js and Shadcn.
How I saved 50x of the CPU Power
My job was to compare the sitemap pages of certain websites every single day and collect the new pages that were added to every website.
An average developer without knowledge of data structures would create 2 arrays of each website and compare them.
Doing so will require 10K operations to be executed if you have 100 pages. The reason is because the big O notation in this case would be O(n^2) where “n” is the number of pages on the website.
My solution was using sets instead of arrays because that operation becomes O(n) instead of O(n^2) with sets. This reduces the number of operations from 10K to 200. Thus, I saved 50 operations on the server power for every user.
E-mail Report
The report landed in spam because the client did not pay for the full service in the time of screenshot.
Outcomes and Deliverables
- Saved the client 50 times server CPU power by implementing proper data structures.
- Saved 50% of the client’s budget by deploying the application to AWS EC2 and S3 bucket instead of Vercel infrastructure.
- Enabled secure HTTPS connections by creating an SSL certificate and an NGINX reverse proxy.
- Enabled weekly emails reports by using Mailgun, Node.js and cron jobs
App Features
- Stored sitemaps files using S3 bucket.
- Deployed to EC2 behind an NGINX reverse proxy.
- Enabled server SSL certificate.
- Enabled sitemap extraction and comparison using Node.js.
- Sending weekly email reports about changes in competitors' sitemaps.
🛠️ Tech Stack
Next.js, Node.js, TypeScript, Prisma, MySQL, TailwindCSS, AWS EC2, AWS S3 Bucket, Mailgun, NGINX, cron jobs