Sitemap Alert
Project Description
I built this monitoring system during my job as a full-stack developer at Mousalli's Advertising Agency, giving him an advantage over competitors by allowing him to monitor websites and track what pages are added every week and report that through weekly e-mails.
I built a simple UI with proper error handling using Next.js and Shadcn.
How I saved 50x of the CPU Power
My job was to compare the sitemap pages of certain websites every single day and collect the new pages that were added to every website.
An average developer without knowledge of data structures would create 2 arrays of each website and compare them.
Doing so will require 10K operations to be executed if you have 100 pages. The reason is because the big O notation in this case would be O(n^2) where “n” is the number of pages on the website.
My solution was using sets instead of arrays because that operation becomes O(n) instead of O(n^2) with sets. This reduces the number of operations from 10K to 200. Thus, I saved 50 operations on the server power for every user.
E-mail Report
The report landed in spam because we were using the free version of Mailgun at the time of screenshot.
Outcomes and Deliverables
- Optimized and saved 50x of the server CPU power by implementing proper data structures.
- Allowed my boss to monitor to track competitors websites by building web crawlers for the XML sitemaps.
- Reduced deployment costs by 50% by hosting the application on AWS EC2 and S3 instead of Vercel.
- Enabled secure HTTPS connections by creating an SSL certificate and an NGINX reverse proxy.
- Enabled weekly emails reports by using Mailgun, Node.js and cron jobs
App Features
- Stored sitemaps files using S3 bucket.
- Deployed to EC2 behind an NGINX reverse proxy.
- Enabled server SSL certificate.
- Enabled sitemap extraction and comparison using Node.js.
- Sending weekly email reports about changes in competitors' sitemaps.
🛠️ Tech Stack
Next.js, Node.js, TypeScript, Prisma, MySQL, TailwindCSS, AWS EC2, AWS S3 Bucket, Mailgun, NGINX, cron jobs