Rank tracker that watches thousands of keywords while you sleep
The client's SEO team was tracking keyword positions using a mix of third-party tools and manual spot-checks. Every morning someone would pull a report, eyeball the numbers, and flag anything that looked off. The problem wasn't effort — it was coverage. They had over 8,000 keywords across dozens of projects, and the tools they were paying for either couldn't handle the volume at a reasonable price or delivered stale data from the previous day's crawl. By the time a ranking drop was noticed, the damage had already compounded for 48 hours.
batches and storage
We built a rank tracking system directly into their platform. Each project contains keyword chains — groups of related queries organized by intent, landing page, or campaign. The system sends these keywords in optimized batches to a SERP data provider, collects the results, and stores position snapshots daily. The batching logic matters because the provider charges per request and enforces rate limits. Naively sending 8,000 individual queries would blow the budget and hit the ceiling within minutes. Instead, we group keywords by search engine and region, pack them into the largest batches the API allows, and stagger requests with controlled delays. This alone cut API costs by roughly 40% compared to their previous tooling.
parsing the serp
Parsing the results reliably was more work than expected. SERP layouts aren't uniform — a branded query returns a knowledge panel and sitelinks, a local query returns a map pack, a product query returns shopping carousels. The position of a "normal" organic result shifts depending on what else the search engine decides to show. Our parser extracts the organic listings specifically, maps them against the tracked domains, and records both the absolute position and the effective visibility (accounting for SERP features that push organic results below the fold). This distinction turned out to be valuable — the team started catching cases where their position was technically stable but real visibility had dropped because a new featured snippet appeared above them.
traffic and reconciliation
The system also pulls traffic data from the search engine's analytics platform, linking keyword positions to actual click-through rates and sessions. This integration required dealing with a separate API that has its own authentication flow, rate limits, and a naming convention for properties that doesn't always match what users enter. We built a reconciliation layer that fuzzy-matches property names and caches auth tokens with automatic refresh.
reports and alerts
Daily reports run via cron in the early morning hours. For large keyword sets — some projects track 3,000+ queries — report generation can take 20 to 30 minutes. We moved this to background processing with server-sent events pushing progress updates to the dashboard, so the team can see the report building in real time rather than staring at a spinner. When a keyword drops more than five positions in a single day, the system highlights it in the report and marks it for review. Sustained declines over a week trigger a separate alert with context: which competitors moved up, whether the landing page has technical issues, and when the content was last updated.
results
The result is a system that runs autonomously and surfaces only what needs human attention. The team stopped spending mornings on data collection and started spending them on decisions. One thing we'd do differently: the initial version stored every daily snapshot as a separate row, which worked fine for months but became a query performance problem once some projects accumulated a year of history. We've since moved to a time-series-friendly schema with partitioned tables, but the migration was painful enough to be a reminder — think about data growth on day one, not month six.
Stack
Backend: Next.js 14 (Route Handlers), Prisma, PostgreSQL
Data: XMLStock API (SERP data), Yandex Metrika API (analytics)
Scheduling: Vercel Cron for daily collection and report generation
Real-time: Server-Sent Events for progress updates