Batching X API Requests: The 14x Difference Nobody Measures
Batching X API Requests: The 14x Difference Nobody Measures
Most code that reads X data is written one item at a time, because that is how the problem is described: get this post, get that profile. The loop follows naturally, it works, and nothing about it looks wrong.
★Measured against the batched equivalent, that loop costs roughly 14 times more traffic and 14 times more wall-clock time for the same data.★
This page covers what batching actually saves, when it does not help, and ★one return-shape behaviour that will silently inflate your results★ if nobody warns you.
The measured difference
Fetching 5 posts individually versus in one batched call:
| Individual | Batched | |
|---|---|---|
| Traffic | baseline | ★14.3× less★ |
| Wall-clock | baseline | ★14.7× faster★ |
| Quota consumed | baseline | ★5× less★ |
Why the gap is so wide: each individual request repays the full cost of a round trip — connection, headers, and per-response envelope — ★for one item of payload★. Batching amortises all of that across a hundred items.
⚠️ ★Note the quota column is a different ratio from the traffic column.★ Traffic and latency scale with round trips; quota scales with how the provider prices the batch. They are separate savings, and conflating them leads to wrong capacity planning.
Batched shapes exist for posts and profiles — the two things people most often loop over.
What can be batched
Two shapes cover most real workloads:
import requests
BASE = "https://api.socialapi.tech"
KEY = "your_api_key"
HDRS = {"X-API-Key": KEY}
def posts(ids):
"""Up to 100 post IDs in one call."""
r = requests.get(f"{BASE}/v1/tweets/batch",
params={"ids": ",".join(ids)}, headers=HDRS, timeout=60)
r.raise_for_status()
return r.json()["data"]
def profiles(usernames):
"""Many profiles in one call."""
r = requests.get(f"{BASE}/v1/batch/user_info",
params={"usernames": ",".join(usernames)},
headers=HDRS, timeout=60)
r.raise_for_status()
return r.json()["data"]
# ★chunk to the batch ceiling — do not send 500 IDs and hope★
def chunked(seq, n=100):
for i in range(0, len(seq), n):
yield seq[i:i+n]
# post_ids: the list of tweet IDs you want, e.g. from a search or your own DB
post_ids = ["1234567890123456789", "1234567890123456790"]
all_posts = [p for chunk in chunked(post_ids) for p in posts(chunk)]
★The chunking helper is not optional.★ A batch endpoint has a ceiling, and sending more than it accepts fails or truncates — truncation being the dangerous one, since it looks like success.
★The return-shape trap★
Here is the behaviour that will corrupt your data if you do not handle it.
Request 50 post IDs from X's underlying batch call and you can get 59 posts back.
★The extra ones are real posts — they are the quoted and replied-to posts referenced by the ones you asked for.★ The platform includes them because they are needed to render context.
⚠️ Why this is worse than it sounds:
- Your count is wrong — 59 where you expected 50
- Your billing is wrong if you meter on returned items
- ★Your analysis silently includes posts from accounts you never asked about★
★The fix is to filter the response against the ID set you requested.★ We do this server-side, so the batch call returns exactly the posts you asked for and nothing else — but if you are calling the platform directly, this filter is yours to write.
requested = set(ids)
clean = [p for p in response if p["id"] in requested] # ★never skip this★
When batching does not help
Batching is not universally better, and three cases are worth knowing:
1. You need one item. ★A batch of one is just a request with extra syntax.★
2. The items are discovered sequentially. If ID #2 depends on the result of #1, there is no batch to form — that is a pipeline, not a bulk fetch.
3. Latency matters more than throughput. A batch returns when its slowest member does. ★For a single interactive lookup, one small request is faster than a batch that waits on 99 others.★
The rule: ★batch when you have the full list up front and want all of it. Do not batch to look clever.★
Both batched and composite shapes are documented together — the choice depends on your access pattern.
The other shape: composite calls
A different saving applies when you need several different things about one subject.
Profile plus recent posts plus account details is three calls done naively. ★A composite endpoint runs them concurrently and returns one response★ — the saving is latency rather than quota, since the work still happens.
def whois(username):
"""Profile + account details + recent posts, concurrently."""
r = requests.get(f"{BASE}/v1/user/whois",
params={"username": username, "tweet_limit": 10},
headers=HDRS, timeout=60)
r.raise_for_status()
return r.json()["data"]
★Use composites for due diligence on one account; use batches for the same field across many.★ They solve different problems and are frequently confused.
Questions people ask
How do I batch Twitter API requests? Send many IDs or usernames in one call instead of looping. ★Chunk to the endpoint's ceiling.★
How much does batching save? Measured at 14.3× traffic and 14.7× time for posts, ★plus a separate 5× quota saving★.
Why is batching so much faster? Round-trip overhead is paid once instead of per item. ★The payload was never the expensive part.★
How many items per batch? Up to 100 post IDs. ★Chunk anything larger★ — do not rely on the server to split it.
What happens if I exceed the limit? Failure or truncation. ★Truncation is the dangerous one because it looks like success.★
Why did I get more posts than I asked for? ★Quoted and replied-to posts come along for context.★ Filter against your requested ID set.
Does the extra data cost me? Not with us — we filter server-side. ★Calling the platform directly, you pay for what you did not want.★
Can I batch profile lookups? Yes — many usernames in one call, ★which is how any ranking or comparison should be built★.
Can I batch follower lists? No. ★Lists are paginated, not batched★ — different problem, different mechanism.
Can I batch searches? No. Each query is its own search. ★Batching applies to fetching known items.★
Is a batch call one request for rate limiting? Yes — which is exactly why the quota saving is separate from the traffic saving.
Should I always batch? No. ★Not for a single item, not for sequentially discovered items, and not when latency beats throughput.★
Does batching change the data? No — same fields, same freshness. ★Only the transport differs.★
What if one ID in the batch is invalid? It is absent from the response. ★Compare returned IDs against requested ones★ rather than assuming a 1:1 mapping.
How do I know which items failed? Set difference between what you asked for and what came back. ★Deleted posts look identical to invalid IDs.★
Can I mix post IDs and usernames? No — they are different endpoints with different shapes.
What is a composite endpoint? One call that runs several different lookups concurrently and returns them together.
When should I use a composite instead of a batch? ★Composite for many fields about one subject; batch for one field about many subjects.★
Does a composite save quota? Mostly latency. ★The underlying work still happens★ — it just happens in parallel.
Is batching harder to debug? Slightly — a single failure affects the whole call. ★Log the requested ID set so you can retry precisely.★
Should I retry a whole failed batch? Usually yes, with backoff. ★Splitting into singles on failure defeats the purpose★ unless you are isolating one bad ID.
Does batching help with rate limits? Substantially — what triggers limits is request count, not payload size.
How do I batch efficiently across pages? Collect IDs while paging, then batch-fetch details afterwards. ★Two phases beats interleaving.★
Can I parallelise batches? Yes, modestly. ★Concurrency multiplies request rate★, so pair it with backoff.
Does order survive batching? Do not assume it does. ★Key the response by ID★ rather than by position.
What about deleted posts in a batch? They are simply missing — what deletion means.
Is there a cost difference per item? Batched items price lower per item, which is the point.
How does this affect a large export? Substantially — export patterns are exactly where batching pays off most.
Can I batch across accounts? Profiles yes, timelines no. ★A timeline is paginated per account.★
What is the single biggest batching mistake? ★Not filtering the response against what you requested★ — it inflates counts in a way that looks plausible.
The short version
★Looping one item at a time costs about 14× more traffic and time than batching the same fetch, and the quota saving is a separate 5×.★ Most code loops because the problem was described one item at a time, not because anyone chose it.
★The trap: a batch of 50 post IDs can return 59 posts — quoted and replied-to posts arrive for context. Filter against your requested ID set, or your counts, your billing, and your analysis all drift by a plausible-looking margin.★
Our API batches posts and profiles, filters the response to exactly what you asked for, and offers composite calls when you need several different things about one account. Read-only, flat price per call.
What batching cannot do: turn a paginated list into one request, or batch a search. ★Those are different mechanisms, and treating them as batchable is where most performance work goes wrong.★
Related reading: what actually triggers rate limits · export patterns that use batching · why the numeric ID is the right key · paging follower lists.