Competitor monitoring APIs: alert on a change, not another copied paragraph

A daily competitor summary can repeat the same announcement for weeks. Search found it again, the model summarized it again and a person received another notification. Nothing actually changed.

A useful competitor monitoring API workflow keeps state. It knows which pages were checked, which observations changed and when a source could not be read. Summarization comes after the comparison, not before it.

Start with an approved watchlist

Choose a small set of public pricing, product, documentation and changelog pages. Record the exact URL and the question each page answers. A pricing page and a changelog have different change rules.

Do not collect private customer areas, bypass access controls or broaden the crawl automatically. Readability through an extraction provider does not grant permission to republish a competitor's content.

Our web scraping API comparisons explain the differences between page extraction, general scraping infrastructure and hosted automation. The right supplier depends on whether you need a known page, site discovery or an interactive session.

Separate discovery from observation

Exa Search can help discover announcements or documentation URLs. A new search hit is not necessarily a newly published page. Keep the source publication date, if supported, distinct from when your monitor first found it.

For a known permitted URL, Firecrawl Scrape documents page extraction:

curl --request POST 'https://api.firecrawl.dev/v2/scrape' \
  --header "Authorization: Bearer $FIRECRAWL_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{"url":"https://example.com/pricing","formats":["markdown"]}'

This is a provider request example using a placeholder page, not a successful run or an All Them APIs endpoint. Store credentials on the server and redact them from traces.

Preserve a clean baseline

The first successful fetch creates a baseline. It should not announce that every product or price on the page just launched.

Normalize navigation, cookie notices and repeated footer text before comparing snapshots. Keep the normalization version with the record. If you change that code, establish whether the resulting diff is a website change or your own processing change.

Do not normalize away the fields you care about. Currency, billing period, usage unit, geographic availability and footnotes can change the meaning of a price even when its visible number stays the same.

Classify the outcome before drafting an alert

A failed fetch is not a removed feature. Likewise, a changed headline is not proof of a strategy change. Keep the observation narrower than the interpretation.

Use the model to summarize the diff

Give the model the relevant before/after sections and source URL, not an open-ended instruction to invent a market narrative. Require it to cite the changed text and distinguish facts from possible implications.

Read the approved URL watchlist and the last successful snapshots.
Fetch only permitted pages using configured APIs and a hard request limit.
Normalize page chrome without removing price units or important footnotes.
Separate unchanged, changed, redirected and unavailable outcomes.
Summarize only material supported changes; link to the exact source.
Do not overwrite a baseline on a failed or partial fetch.
Do not infer revenue, adoption or customer loss from a page edit.
Queue the findings for review and stay quiet when nothing material changed.

Test the alert loop, not just the scraper

Use local fixtures for an unchanged page, a price-period change, a deleted paragraph, an unrelated navigation change and a failed fetch. Verify that retries do not send duplicate alerts.

For an actual evaluation, record the watchlist, schedule, provider mode, accepted changes and false alerts. This article describes the design; it does not introduce a new measured latency or accuracy ranking.

The company account-brief guide handles deeper interpretation after an observation is reviewed. Keep the monitor's job smaller: collect a defensible change, preserve its context and avoid making people read the same story twice.