Twitter Monitoring That Survives Its First Week

November 9, 202613 min readSocialAPI Engineering

Twitter Monitoring That Survives Its First Week

A monitoring script is one of those things that works perfectly for an afternoon and then quietly stops being trustworthy.

The demo is twenty lines: poll a search, print new results. What breaks it isn't the fetching — it's everything around the fetching. Within a week you've got duplicate alerts, a gap from the deploy you did on Tuesday, and a process that died at 4am without telling anyone.

This article is about those four failures specifically. Not how to query X — that's covered elsewhere — but what state you need so the thing stays correct when it's left alone.

If you're choosing between building this and buying it, the free-tier comparison is the better starting point. This assumes you've decided to build — there's a free demo endpoint if you want to check response shapes before writing any of it.

The four things that break

Failure	Symptom	What prevents it
No dedupe state	Same tweet alerts repeatedly	Persisted seen-IDs, not an in-memory set
The restart gap	Silence across a deploy, never noticed	A high-water mark, persisted per query
Silent death	No alerts — looks identical to "nothing happened"	A heartbeat that alerts on absence
Alert fatigue	Everything routed everywhere, people mute it	Severity routing decided before you build

They compound in a specific order. People fix dedupe first because it's visible and annoying, then hit the restart gap, then discover the silent death months later — usually by noticing they haven't been alerted in a while.

1. Dedupe state has to outlive the process

Almost every tutorial uses an in-memory set:

seen = set()   # dies with the process

That's correct until the first restart, at which point your whole window re-alerts. If your notifier posts to a channel, everyone sees the last fifty tweets again and learns to ignore it.

The fix is small — persist it, and bound it:

import json, os, time

STATE = "monitor_state.json"

def load_state():
    if not os.path.exists(STATE):
        return {"seen": {}, "high_water": {}}
    try:
        with open(STATE, encoding="utf-8") as f:
            return json.load(f)
    except (json.JSONDecodeError, OSError):
        # corrupt state is better discarded than crashed on —
        # you re-alert once, instead of never starting again
        return {"seen": {}, "high_water": {}}

def save_state(st):
    tmp = STATE + ".tmp"
    with open(tmp, "w", encoding="utf-8") as f:
        json.dump(st, f)
    os.replace(tmp, STATE)          # atomic; a crash mid-write can't truncate it

def prune(st, days=7):
    cutoff = time.time() - days * 86400
    st["seen"] = {k: v for k, v in st["seen"].items() if v > cutoff}

Three details that matter more than they look.

os.replace rather than writing in place. A crash during a plain write leaves a half-written file, and then the next start can't parse its own state. Write to a temp file and rename — the rename is atomic, so the state file is always either the old one or the new one.

Catching the decode error and continuing. A corrupt state file should cost you one round of duplicate alerts, not a monitor that refuses to boot. The instinct to fail loudly is wrong here: the whole point of this process is to keep running.

Pruning. Seen-IDs grow forever otherwise. Seven days is generous — anything older can't plausibly come back as "new."

2. The restart gap: dedupe is not enough

Dedupe stops you re-alerting old tweets. It does nothing about tweets posted while you were down.

This is the failure people don't notice, because its symptom is silence — and silence looks exactly like a quiet period. You deploy at 14:00, the process is down for ninety seconds, three tweets land in that window, and they're simply never alerted. Nothing in your logs says so.

What you need is a high-water mark per query, persisted, and a cold start that reaches back:

import requests

API = "https://api.socialapi.tech/v1"
HEADERS = {"X-API-Key": "your_key"}

def check(query, st, backfill=50, normal=20):
    hw = st["high_water"].get(query)
    # no high-water mark = cold start. Reach further back once,
    # so a long outage doesn't leave a permanent hole.
    limit = normal if hw else backfill

    r = requests.get(f"{API}/search/advanced",
                     params={"query": query, "limit": limit},
                     headers=HEADERS, timeout=30)
    if r.status_code == 429:
        return []                      # rate limited: come back later, don't advance state
    r.raise_for_status()

    rows = r.json().get("data") or []
    fresh, now = [], time.time()

    for t in reversed(rows):           # oldest first
        tid = t["id"]
        if tid in st["seen"]:
            continue
        if hw and tid <= hw:           # snowflake IDs sort chronologically
            continue
        st["seen"][tid] = now
        fresh.append(t)

    if rows:
        st["high_water"][query] = max(t["id"] for t in rows)
    return fresh

tid <= hw works because snowflake IDs are time-ordered. You get a chronological comparison without parsing a single timestamp — and without depending on the API returning results in any particular order. The IDs are strings here and compare correctly as long as they're the same length, which at 19 digits they are for the foreseeable future; if you want to be strict about it, compare int(tid).

Returning early on 429 without advancing the high-water mark is the subtle one. If you treat a rate limit as "no new tweets" and move the mark forward, you've just skipped a window permanently. A temporary failure must leave state untouched so the next run picks it up. Which of the three rate limits you've hit is worth knowing — the distinction is here.

3. Alerting on absence, not just presence

Everything so far alerts when something happens. Nothing yet alerts when the monitor itself stops — and that's the failure that goes undetected longest, because a dead monitor and a quiet week produce exactly the same output: nothing.

The cheap fix is a heartbeat file plus a separate check:

def heartbeat():
    with open("monitor_heartbeat", "w") as f:
        f.write(str(int(time.time())))

# cron, every 15 minutes — deliberately NOT inside the monitor process
AGE=$(( $(date +%s) - $(cat /opt/monitor/monitor_heartbeat 2>/dev/null || echo 0) ))
[ "$AGE" -gt 900 ] && curl -s -X POST "$ALERT_WEBHOOK" \
  -d "{\"text\":\"monitor stale: ${AGE}s\"}"

The check must live outside the process it's checking. A watchdog running inside the monitor dies with it, which is the one moment you needed it. Cron, systemd timer, an uptime service — anything with a separate lifecycle.

If you'd rather not run the plumbing at all, a provider-side push removes the polling loop and the heartbeat with it — the delivery guarantees move to someone else's side. The three mechanisms compared covers when that's the right call and when polling genuinely wins.

4. Routing, decided before you build

The failure here is organisational, not technical: you route everything to one channel, people mute the channel, and the monitor is now decoration.

Decide the tiers first, because they change the code:

Tier	Example	Route	Latency that matters
Urgent	Named account posts about an outage	Page someone	Seconds
Relevant	Competitor announcement, brand mention	Team channel	Minutes
Archive	Everything matching a broad term	Database only, no notification	Hours

The archive tier is the one people skip, and it's what makes the other two usable. Most matches aren't worth a notification but are worth having when you later ask "when did this start?" Write them to storage, notify on nothing, and query them when you need to. That's also what lets you tighten the urgent tier without fear of losing data — storing the stream is a different job from alerting on it.

Putting it together

def main(queries, interval=60):
    st = load_state()
    while True:
        for q in queries:
            try:
                for t in check(q, st):
                    route(q, t)                 # your tiering from above
            except requests.RequestException as e:
                print("fetch failed:", q, e)    # loop survives one bad query
        prune(st)
        save_state(st)                          # after the whole round, not per tweet
        heartbeat()
        time.sleep(interval)

Save after the round, not per tweet. Per-tweet saves multiply your disk writes for no benefit — if you crash mid-round the worst case is re-alerting that round, which dedupe handles on the next pass.

Catch per-query, not per-loop. One malformed query shouldn't stop the other five from being checked. This is the difference between a monitor that degrades and one that stops.

That's the whole thing: persisted dedupe, a high-water mark that only advances on success, an external heartbeat, and routing tiers chosen up front. Roughly eighty lines, and it's the difference between a script that demos and one you can leave running.

Our API returns the shapes used above, and rate limits are spread across an account pool rather than tied to one credential — which matters for monitoring specifically, since you're making the same request on a schedule forever. Signing up with Google includes $0.20 in free credits, no card, if you want to run this against something real before committing.

FAQ

What's the best way to monitor Twitter? For most cases, a polling loop on a schedule with persisted dedupe state. Add push delivery only when single-digit seconds genuinely matter — it's more moving parts for latency you may not need.

Why does my monitor alert the same tweet twice? Your seen-set is in memory and doesn't survive restarts, or you're deduplicating on text rather than tweet ID. Persist the IDs and key on them.

How do I stop missing tweets during a restart? Keep a high-water mark — the newest tweet ID you've processed per query — in the same persisted state. On cold start, fetch a larger window once so an outage doesn't leave a permanent gap.

What is a high-water mark? The newest item you've successfully handled. Anything at or below it is already done. For X it's a tweet ID, which works directly as a chronological marker because snowflake IDs are time-ordered.

Can I compare tweet IDs to sort chronologically? Yes — snowflake IDs embed a timestamp, so a larger ID is newer. Compare as integers, or as equal-length strings. This saves parsing timestamps entirely.

How often should I poll? Every 60 seconds covers most monitoring. The question isn't how fast you can poll but how stale an alert can be before it's useless — set the interval from that, then make sure your page size covers bursts between polls.

How do I know if my monitor died? Have it write a heartbeat timestamp, and check that file from a separate process such as cron. A dead monitor and a quiet period look identical from the outside, so you have to alert on absence explicitly.

Why should the watchdog be a separate process? Because a watchdog inside the monitor dies with the monitor — precisely when you needed it. Use cron, a systemd timer, or an external uptime service.

Should I store state in a file or a database? A JSON file is fine for one process. Move to Redis or Postgres when multiple workers share state, or when the seen-set gets large enough that rewriting the file each round is wasteful.

How do I stop the seen-ID set growing forever? Prune by age — drop entries older than about seven days. Nothing older can plausibly reappear as new.

What if the state file gets corrupted? Catch the decode error and start from empty rather than crashing. You'll re-alert one window, which is much better than a monitor that won't boot. Write state atomically (temp file plus rename) so corruption is rare in the first place.

Should I advance my high-water mark after a rate limit? No. A temporary failure must leave state untouched, or you permanently skip that window. Return early, change nothing, and retry on the next cycle.

How do I handle rate limits in a monitoring loop? Treat 429 as "come back later": skip the round without advancing state, and back off. Don't count it as an empty result.

Why do I get alerts for old tweets on first run? A cold start with a large backfill window will surface existing matches. Either seed the high-water mark without alerting on that first pass, or accept one noisy startup.

How many keywords can one monitor handle? It's a function of request budget, not code — each query is its own request per cycle. Twenty queries at 60-second intervals is 28,800 requests a day, so price that before adding queries.

Should each keyword get its own high-water mark? Yes, per query. Different queries return different result sets, and a shared mark will cause one query to skip tweets another has already advanced past.

Can I run several monitors on the same keywords? Only with shared state, or they'll duplicate alerts independently. Either one process per query set, or a shared store with atomic updates.

How do I avoid alert fatigue? Decide severity tiers before building: page, notify, or archive silently. The archive tier is what makes the other two usable — most matches are worth storing and not worth interrupting anyone.

What should I log in a monitoring loop? Per-cycle counts, fetch failures with the query that caused them, and state-save confirmations. Enough to answer "was it running, and was it finding anything" after the fact.

Should the loop crash on an API error? No. Catch per query so one bad query doesn't stop the others, log it, and continue. A monitor that degrades beats one that stops.

Is polling or streaming better for monitoring? Polling is simpler and recovers from restarts for free; streaming is lower latency but you own a connection that will drop. Many production setups run both — stream for speed, slow poll for completeness.

How do I monitor a specific account rather than a keyword? Same loop, different endpoint — a user timeline instead of a search. Dedupe, high-water mark and heartbeat are unchanged.

Do I need an X developer account for this? Not when reading public data through a third-party API; you use that provider's key. An official developer account is for write access and official endpoints.

How far back can I backfill on a cold start? As far as your page size and plan allow. Pick a window that covers your realistic worst-case outage — then it's a bounded gap rather than an open-ended one.

Can I monitor without running a server? Yes, with provider-side push into a serverless function or a tool like Zapier. You trade control over retries for not maintaining a process.

Monitoring is the workload where per-request pricing and pooled rate limits matter most, because you're making the same call on a schedule indefinitely. If you'd rather test that than take our word for it, a Google sign-up gets you $0.20 in credits with no card attached.