Caching X API Responses: TTLs by How Fast Data Changes
Caching X API Responses: TTLs by How Fast Data Changes
A cache is the cheapest optimisation available to anything that calls an API on a schedule. It's also the easiest place to introduce bugs that look like upstream problems — stale numbers in a dashboard, a thundering herd at every expiry, a cache that grows until the process dies.
The single decision that matters is the TTL, and one TTL for everything is always wrong, because the things you're caching change at wildly different rates. A tweet's text never changes. Its like count changes every few seconds. Caching both for five minutes means one is wastefully re-fetched and the other is wrong.
If you're deciding what to store long-term rather than what to cache, the schema decisions are the other half of this. There's a free demo endpoint if you want response shapes to size your entries against.
TTLs by volatility
| Response | Changes | TTL | Reasoning |
|---|---|---|---|
| Tweet text, author, created_at | Never | Hours to days | Immutable once posted |
| Engagement counts | Constantly | 1–5 min | Stale numbers are visibly wrong |
| User profile (bio, avatar) | Rarely | 1–6 hours | Edits are infrequent |
| Follower count | Slowly, and cached upstream | 15–60 min | Already cached by the platform |
| Follower/following lists | Slowly, expensive to fetch | Hours | Cost dominates freshness |
| Search results | Constantly | 1–2 min | New matches are the point |
| Trends | Every few minutes | 5 min | Matches upstream refresh |
The split that saves the most calls: cache content and counts separately. They arrive in the same response, so the instinct is to cache the response — but then you either refetch immutable text every minute, or serve stale counts for an hour.
def cache_tweet(tweet):
# immutable half: long TTL
cache.set(f"tweet:content:{tweet['id']}", {
"text": tweet["text"],
"author_id": tweet.get("author_id"),
"created_at": tweet["created_at"],
}, ttl=86400)
# volatile half: short TTL
cache.set(f"tweet:metrics:{tweet['id']}", {
"like_count": tweet.get("like_count"),
"retweet_count": tweet.get("retweet_count"),
"view_count": tweet.get("view_count"),
}, ttl=120)
Note that followers_count is already cached upstream. The number on a profile isn't live — it refreshes on the platform's own schedule. Caching it for 15 minutes adds almost no staleness beyond what's already there, which makes it unusually cheap to cache.
The stampede, which appears immediately
A cache entry expires. Twenty requests arrive in the same second. All twenty miss, all twenty call the API. You've turned one expiry into twenty requests, and under load that's worse than having no cache.
import threading, time
_locks = {}
_locks_guard = threading.Lock()
def get_or_fetch(key, fetch, ttl):
hit = cache.get(key)
if hit is not None:
return hit
with _locks_guard:
lock = _locks.setdefault(key, threading.Lock())
with lock: # only one fetch per key
hit = cache.get(key) # ← re-check: someone may have filled it
if hit is not None:
return hit
value = fetch()
cache.set(key, value, ttl=ttl)
return value
The second cache.get inside the lock is the whole point. Without it, every thread that queued on the lock performs its own fetch once it acquires — you've serialised the stampede rather than preventing it. With it, the first thread fills the cache and the rest return immediately.
For multiple processes or machines, the same pattern needs a distributed lock — SET key value NX EX 30 in Redis, with the TTL ensuring a crashed holder can't block the key forever.
Serve stale rather than fail
Once you have a cache, it can do something more useful than reduce calls: it can keep you serving when the API doesn't answer.
def get_with_fallback(key, fetch, ttl, stale_ttl=3600):
entry = cache.get_raw(key) # includes expiry metadata
if entry and not entry.expired:
return entry.value
try:
value = fetch()
cache.set(key, value, ttl=ttl)
return value
except (RateLimited, Timeout, ServiceUnavailable):
if entry and entry.age < stale_ttl:
return entry.value # stale, labelled, available
raise
Two-minute-old engagement counts beat an error page. The important part is labelling it: return the entry's age alongside the data so the UI can say "as of 2 minutes ago" rather than presenting stale numbers as current. A dashboard that silently shows old data is worse than one that admits it.
Only catch failures that are actually transient. Falling back to stale data on a 404 hides a real deletion; on a 401 it hides a broken key. Rate limits, timeouts and 5xx are the right set — the distinction is in which errors to retry.
Cache keys that don't collide
import hashlib, json
def cache_key(endpoint: str, params: dict) -> str:
# sort so {"a":1,"b":2} and {"b":2,"a":1} produce the same key
canonical = json.dumps(params, sort_keys=True, separators=(",", ":"))
digest = hashlib.sha256(canonical.encode()).hexdigest()[:16]
return f"api:{endpoint}:{digest}"
Sorting the params matters more than it looks. Dict ordering varies with construction order, so the same logical request can produce two keys — halving your hit rate for no visible reason.
Don't put raw params in the key. Usernames and queries can contain characters your cache treats specially, and a long query makes an unwieldy key. Hashing sidesteps both.
Never include the API key in the cache key, and be careful about caching per-user data under a shared key. The classic cache bug is one user seeing another's results.
Invalidate on write, not just on expiry
TTLs handle the general case. When you know something changed, don't wait:
def record_deletion(tweet_id):
cache.delete(f"tweet:content:{tweet_id}")
cache.delete(f"tweet:metrics:{tweet_id}")
cache.set(f"tweet:deleted:{tweet_id}", True, ttl=86400) # negative cache
Negative caching is the part people skip. If an account doesn't exist, caching that fact for an hour stops you re-asking every minute. Without it, your most expensive requests are often the ones that return nothing — a monitoring loop checking a deleted account hammers the API forever, and 404 has several causes worth distinguishing.
Keep negative TTLs shorter than positive ones. A suspension can be reversed, and you don't want to cache "doesn't exist" for a day.
Bound the memory
An in-process dict cache with no eviction is a memory leak with good intentions.
from cachetools import TTLCache
# bounded: oldest entries evicted at capacity
cache = TTLCache(maxsize=10_000, ttl=300)
Pick maxsize from your working set, not optimistically. If you monitor 200 accounts, 10,000 entries is ample. If you're resolving arbitrary user lookups from traffic, an unbounded cache will eventually consume everything — and the crash arrives days later, which makes it hard to connect to the cache you added.
Move to Redis when you have more than one process. Two workers with separate in-process caches have independent hit rates and can serve different values for the same key, which produces bug reports that are very hard to reproduce.
Measure it, or you're guessing
class Stats:
def __init__(self):
self.hits = self.misses = self.stale = 0
def ratio(self):
total = self.hits + self.misses
return self.hits / total if total else 0.0
A hit ratio below ~50% usually means the TTL is too short for the access pattern, or your keys are fragmenting. Both are easy to fix once you can see them, and invisible otherwise.
Log the ratio periodically alongside your API call count. The call count is what you're paying for; the ratio tells you whether the cache is the reason it changed.
Our API bills per request, so a cache with the right TTLs translates directly into a smaller bill — and the immutable/volatile split above is where most of that saving comes from. Signing up with Google includes $0.20 in free credits and no card if you want to measure the difference on a real workload.
FAQ
How long should I cache X API responses? By volatility, not by one global number. Tweet text for hours or days, engagement counts for one to five minutes, profiles for a few hours, search results for a minute or two.
Why not use one TTL for everything? Because the things you cache change at different rates. A single TTL either serves stale counts or re-fetches immutable text needlessly — usually both.
What's the highest-value caching change? Splitting each tweet into an immutable content entry with a long TTL and a volatile metrics entry with a short one. They arrive together but shouldn't expire together.
Is it safe to cache tweet text for a day? Yes. Text, author and creation time don't change after posting. Edits exist within a short window, so if that matters to you, keep the TTL under an hour for very recent posts.
How long can I cache a follower count? 15 to 60 minutes comfortably, because the number is already cached upstream. You're adding little staleness to a figure that wasn't live to begin with.
What is a cache stampede? Many requests missing the same expired key simultaneously and all calling the API. One expiry becomes dozens of requests, which under load is worse than no cache.
How do I prevent a stampede? A per-key lock, with a second cache check inside the lock. Without that re-check you serialise the stampede instead of preventing it.
Why is the double-check inside the lock necessary? Because every thread queued on the lock would otherwise fetch when it acquires. The re-check lets them see the value the first thread just stored.
How do I do this across multiple processes?
A distributed lock — SET key NX EX 30 in Redis. The TTL on the lock matters so a crashed holder can't block that key indefinitely.
Should I serve stale data when the API fails? Yes for transient failures, and label the age. Two-minute-old counts beat an error page, as long as the UI says how old they are.
Which failures should trigger a stale fallback? Rate limits, timeouts and 5xx. Not 404 — that hides a real deletion — and not 401, which hides a broken key.
How do I build cache keys safely? Sort the params, serialise canonically, and hash. Unsorted dicts produce different keys for identical requests and silently halve your hit rate.
Should the API key be part of the cache key? No. Never cache under a key containing credentials, and be careful caching per-user data under shared keys — users seeing each other's results is the classic version of this bug.
What is negative caching? Caching the fact that something doesn't exist. Without it, a loop checking a deleted account calls the API forever, and those calls return nothing.
How long should a negative cache entry live? Shorter than the positive ones — minutes, not a day. Suspensions get reversed, and you don't want to cache a nonexistent account past that.
When should I invalidate instead of waiting for expiry? Whenever you learn something changed — a detected deletion, a rename, a profile update you just fetched. TTLs are the fallback, not the only mechanism.
In-process cache or Redis? In-process for a single worker; Redis as soon as you have two. Separate per-process caches can serve different values for the same key, which is painful to debug.
How big should maxsize be? Sized to your working set. A couple hundred monitored accounts needs very little; caching arbitrary lookups from user traffic needs an explicit bound or it grows forever.
Is an unbounded dict cache really a problem? Yes — it's a memory leak. The failure arrives days later as an OOM, which makes it hard to connect back to the cache you added.
What hit ratio should I expect? It depends on access patterns, but below about 50% usually means the TTL is too short or your keys are fragmenting. Both are easy fixes once measured.
What should I log about the cache? Hits, misses, stale serves, and your total API call count. The call count is the bill; the ratio explains it.
Does caching help with rate limits? Yes, directly — a cache hit isn't a request. It's often the cheapest way to get under a limit without touching your logic, and it composes well with bounded concurrency.
Can I cache search results? Briefly, one to two minutes. New matches are the point of a search, so a long TTL defeats the purpose while still saving something on repeated identical queries.
Should I cache paginated responses? Cache per page, keyed with the cursor. Caching an assembled multi-page result means one changed page invalidates the whole thing.
What's the most common caching mistake? Caching the whole response under one TTL. It guarantees you're simultaneously too stale on counts and too eager on text, and it's invisible until someone notices a wrong number.
Split immutable from volatile, lock per key, and measure the ratio — three changes, and most of the saving comes from the first. A Google sign-up includes $0.20 in free credits and no card.