Back to blog
·4 min readReddit APIs

Reddit subreddit monitoring: recover missed runs without duplicate alerts

All Them APIs

Independent API research

The difficult day for a subreddit monitor is not the day it works. It is the day it misses two runs, resumes halfway through a changing feed and announces yesterday's posts as new.

You cannot solve that with a better summarization prompt. You need state, a bounded recovery policy and an honest definition of a completed run.

Keep the watchlist explicit

Record approved public communities, the purpose of collection and the cadence. A subreddit feed and a keyword search are separate collection paths. Give each its own configuration and checkpoint.

Choose limits before the first run: pages per community, threads worth expanding, comment pages or depth per thread, retry budget and total spend. A popular discussion can have more replies than the rest of the watchlist combined.

Use a small scope for evaluation. A search provider's first page is not evidence that your collector has indexed an entire community.

Preserve both source time and observation time

Keep the post's source timestamp separate from when your monitor discovered it. Store a stable identifier and the last successful observation.

Delayed discovery is normal. A post created before your last run can enter the current sample later, while a changed ranking can resurface an old post. Sorting by new reduces one source of confusion; it does not establish complete delivery.

Use the source ID to decide whether you have seen an item before. Use observation metadata to explain when and how you encountered it.

Commit the checkpoint at the end

Each run needs an outcome: complete, partial or unavailable. Store the result pages durably before moving the successful checkpoint.

If the third page fails, preserve the accepted rows but mark the run partial. Do not move the complete-through boundary as if every required page arrived. Track failed work separately so a recovery run can resume within its allowed scope.

For each approved community:
  Read its last complete checkpoint and the current page budget.
  Collect a bounded overlap window using the provider's current schema.
  Save accepted rows by stable source ID.
  Stop on no continuation, a repeated cursor or a reached budget.
  Record incomplete pagination as partial coverage.
  Advance the checkpoint only after all required writes succeed.
  Create alert candidates only for new relevant IDs or new tracked replies.
  Keep collection failures out of the content digest.

This is an algorithm outline, not an instruction to run an unpublished local script or a record of a completed benchmark.

Handle comments as a separate queue

A post's visible comment count is not the number of comment bodies your API returned. Depth limits, collapsed replies, deleted content and additional paging can all create gaps.

For each tracked thread, retain comment IDs and parent IDs. Record the fetched depth and whether continuation remains. Expand only threads that match the monitoring question.

FetchLayer's documentation distinguishes a post response from additional comments; ScrapeBadger's Reddit reference describes depth controls. Those contracts need different adapters even when both ultimately return comments.

Do not send the same alert for each repeated parent comment. New replies should attach to the existing discussion, with enough context for a reviewer to understand what changed.

Recover with overlap and a clear ceiling

After a missed run, read an overlapping time window and deduplicate again. If the gap is larger than your recovery budget can cover, report the uncovered interval rather than silently skipping it.

Do not combine opaque continuation tokens across providers. A fallback to a different source may return a different ordering or sample. Start a separately identified collection if you need to switch, and record the coverage limitation.

For an unchanged successful run, stay quiet. For an unavailable source, retain the previous successful snapshot. Failure to read a thread is not proof that someone deleted it.

Test failures before trusting the digest

Use fixtures for duplicate IDs, a repeated cursor, a second-page timeout, a removed comment and a crash after storage but before checkpoint advancement. Verify that rerunning does not create duplicate alerts.

Then test the same permitted inputs against a small supplier shortlist. Measure complete-job time and useful unique rows, not just the first response. The pricing guide explains why extra pages and retries belong in that calculation.

Choose the brand monitoring rules or product-research rules separately from the scheduler. Reliable collection does not make every collected post relevant, and permitted access does not remove retention or deletion obligations.