Collecting X Media: The Four Things That Go Wrong

November 10, 202612 min readSocialAPI Engineering

Collecting X Media: The Four Things That Go Wrong

Fetching media from X looks like a solved problem: get the post, read the URL, download the file. The code is twelve lines and it works on your first test account.

Then you run it across a few thousand posts and discover four separate problems, none of which announce themselves. Files silently overwrite each other. Your GIFs are all mp4s. Every image is a quarter of the resolution it should be. And when the job dies at 70% you have no way to resume it.

This is about those four. The fetching itself is the easy part — pulling an account's media covers that, and this assumes you've got a response in hand and now need to handle it correctly.

There's a free demo endpoint if you want to inspect response shapes before writing the download logic around them.

First: photos and videos are different fields

This trips people up before anything else, because the two media types don't arrive in one array.

{
  "id": "...",
  "photos": ["https://pbs.twimg.com/media/..."],
  "videos": [
    {
      "url": "https://video.twimg.com/...mp4",
      "duration_ms": 21000,
      "thumbnail": "https://pbs.twimg.com/..."
    }
  ]
}

photos is a list of strings. videos is a list of objects. Writing one loop that treats them uniformly is the first bug, and it fails with a TypeError on the first video post — which at least is loud. The quieter version is code that only reads photos and silently collects nothing from video posts.

A post can have up to four photos, or one video, but not both. Different hosts too: images come from pbs.twimg.com, video from video.twimg.com.

1. A GIF on X is not a GIF

X stores "GIFs" as silent, looping mp4 files. There is no .gif anywhere in the pipeline.

This matters for two reasons. If you're saving files by extension, you'll write .gif files that are actually mp4 containers — and anything downstream that trusts the extension will fail to open them. And if you're building a viewer, a GIF rendered with video controls looks wrong: people expect it to loop silently on its own, not wait for a play button.

The type is on the video object, so branch on it rather than on duration or file size:

def media_kind(video):
    # animated_gif is a silent looping mp4 — not a .gif file
    return "gif" if video.get("type") == "animated_gif" else "video"

Short videos and GIFs are indistinguishable by length, which is why guessing from duration_ms doesn't work.

2. Filename collisions lose files silently

The obvious filename is the last path segment of the URL. On a single account that's fine. Across accounts it isn't, and the failure is invisible — you just end up with fewer files than posts.

import re

def safe_name(tweet_id, index, url, kind):
    ext = "mp4" if kind in ("video", "gif") else \
          (re.search(r'\.(jpg|jpeg|png|webp)', url) or ["", "jpg"])[1]
    slot = "v" if kind in ("video", "gif") else "p"
    # tweet_id + type + index is unique per media slot; ext from kind, not the URL
    return f"{tweet_id}_{slot}{index}.{ext}"

Key on the tweet ID plus the media index, not the URL. Two different posts can carry media with the same basename, and a four-photo post needs four distinct names from one tweet ID.

The p/v marker is belt-and-braces. In practice a post carries photos or one video, never both, so the index alone is already unique. The prefix costs nothing and keeps the function correct if you merge media from two sources into one directory.

Derive the extension from the media kind rather than the URL, because of the GIF problem above — a URL ending .mp4 that's actually an animated GIF should still be saved as .mp4, and an image URL with query parameters will confuse a naive rsplit('.').

3. Image size lives in the URL

X serves the same image at several resolutions, and which one you get depends on the URL you request. This is measurable — here are real byte counts for one profile image, fetched while writing this:

URL form	Bytes
`..._normal.jpg`	2,111
`..._bigger.jpg`	3,151
`..._400x400.jpg`	28,375
`....jpg` (no suffix)	32,796

The suffix is a downscale, and stripping it gives you the largest version that URL offers — a 15x size difference between _normal and bare, for the same image.

import re

def upscale_profile_image(url: str) -> str:
    # strip the _normal / _bigger / _400x400 size suffix
    return re.sub(r'_(normal|bigger|mini|\d+x\d+)(\.[a-z]{3,4})$', r'\2', url)

Two honest caveats, because this is where articles tend to overclaim:

Avatar URLs and tweet-media URLs are different paths (/profile_images/ versus /media/) and don't behave identically. The byte counts above are measured on a profile image; don't assume the same suffix list applies elsewhere without checking the URL you actually have.

Stripping a suffix doesn't create resolution that was never uploaded. You get the largest stored version, which for a small original is the same pixels at a larger file size. Verify by reading the returned image's dimensions rather than trusting the URL — the method for that is in image sizes and cropping.

4. Resumability, or you re-download everything

A media job is slow because files are large. Which means it will be interrupted — and without state, interruption means starting over.

Note where the time actually goes. The listing is a single request: the media endpoints return up to limit posts in one call, with no cursor to walk. It's the downloads that take minutes or hours, so that's what needs checkpointing.

import json, os, pathlib, requests

API, HEADERS = "https://api.socialapi.tech/v1", {"X-API-Key": "your_key"}

def save_state(state_file, done):
    tmp = state_file + ".tmp"
    with open(tmp, "w", encoding="utf-8") as f:
        json.dump({"done": sorted(done)}, f)
    os.replace(tmp, state_file)               # atomic, same as the files

def scrape_media(username, out="media", state_file="media_state.json", limit=200):
    out_dir = pathlib.Path(out) / username
    out_dir.mkdir(parents=True, exist_ok=True)

    done = set()
    if os.path.exists(state_file):
        try:
            with open(state_file, encoding="utf-8") as f:
                done = set(json.load(f).get("done", []))
        except (json.JSONDecodeError, OSError):
            pass                              # corrupt state: start over, don't crash

    # one listing call — re-running it on resume is cheap
    r = requests.get(f"{API}/user/media", params={"username": username, "limit": limit},
                     headers=HEADERS, timeout=60)
    if r.status_code == 429:
        return                                # stop cleanly, resume later
    r.raise_for_status()
    posts = r.json().get("data") or []

    for n, p in enumerate(posts, 1):
        tid = p["id"]
        for i, url in enumerate(p.get("photos") or []):
            fetch_one(url, out_dir / safe_name(tid, i, url, "photo"), done)
        for i, v in enumerate(p.get("videos") or []):
            if v.get("url"):
                kind = media_kind(v)
                fetch_one(v["url"], out_dir / safe_name(tid, i, v["url"], kind), done)
        if n % 10 == 0:
            save_state(state_file, done)      # checkpoint every 10 posts
    save_state(state_file, done)

def fetch_one(url, path, done):
    if url in done or path.exists():
        return                                # two independent skips
    tmp = path.with_suffix(path.suffix + ".part")
    try:
        with requests.get(url, stream=True, timeout=120) as resp:
            resp.raise_for_status()
            with open(tmp, "wb") as f:
                for chunk in resp.iter_content(1 << 16):
                    f.write(chunk)
        tmp.replace(path)                     # only now does it count as complete
        done.add(url)
    except requests.RequestException:
        tmp.unlink(missing_ok=True)           # never leave a truncated file

Three things in there earn their place.

Download to .part, then rename. A crash mid-download otherwise leaves a truncated file that path.exists() will happily skip forever. The rename is atomic, so a file either doesn't exist or is complete.

Checkpoint downloads, not the listing. The listing is one cheap request you can simply repeat on resume. The downloads are the expensive part, so the state file records completed URLs every few posts, written atomically like the files themselves.

Delete the partial on failure. Without this, one network blip permanently poisons that file — it exists, so the skip check passes, and you never notice it's half a JPEG.

What this can't do

Media from protected accounts — not accessible, regardless of tooling.
Deleted media — once X removes the file, the URL 404s even if you have the post.
Original uploads above what X stored — the platform re-encodes on upload; you get their version.
Bulk download without rate limits applying — media URLs are on a CDN, but the API calls that find them are still metered.

On rights: public availability and permission to republish are different things. Collecting media for analysis is a different act from reposting it, and the second one is governed by copyright, not by the API's terms. Where the line sits goes into this properly.

Our /v1/user/media returns the shapes above, with /v1/user/photos and /v1/user/videos if you want one type without filtering client-side. Signing up with Google includes $0.20 in free credits and no card, which is enough to run the script above against a real account and see where your assumptions break.

FAQ

How do I scrape images from Twitter? Fetch the posts, read the photos array, and download each URL. The work that matters is afterwards: unique filenames, atomic writes, and resumable state.

Why are photos and videos in separate fields? Because their shapes differ — photos holds URL strings, videos holds objects with url, duration_ms and thumbnail. One loop over both raises a TypeError on the first video.

Why is the GIF I downloaded an mp4? Because X stores GIFs as silent looping mp4 files. There's no .gif in the pipeline. Check the video's type field for animated_gif rather than guessing from the extension.

How do I tell a GIF from a short video? Only by the type field. Duration doesn't distinguish them — plenty of real videos are shorter than plenty of GIFs.

How do I get the full-size image instead of a thumbnail? Strip the size suffix from the URL. Measured on one profile image: _normal 2,111 bytes versus no suffix 32,796 bytes — the same image at 15x the data.

Does removing the suffix always give a higher resolution? It gives the largest version stored for that URL. If the original upload was small, you get the same pixels in a larger file. Read the returned image's dimensions to confirm rather than trusting the URL.

Do avatar URLs and tweet image URLs work the same way? They're different paths — /profile_images/ versus /media/ — and shouldn't be assumed to accept the same parameters. Test the URL form you actually have.

Why do I end up with fewer files than posts? Filename collisions. If you name files from the URL's last path segment, media from different posts overwrites each other silently. Key on tweet ID plus media index.

How many photos can one post have? Up to four, or one video, but not both. That's why the index is needed in the filename — one tweet ID can produce four files.

Which host serves the files? Images from pbs.twimg.com, video from video.twimg.com. They're a CDN, separate from the API that gave you the URLs.

Should I download through my own server or direct? Direct from the CDN. Proxying media through your backend adds bandwidth cost and a failure point for no benefit, unless you specifically need to rewrite or watermark the files.

How do I make a media job resumable? Record completed URLs in an atomically-written state file every few posts. On restart, repeat the one listing call and skip anything already done or already on disk.

Why download to a .part file first? Because a crash mid-download leaves a truncated file, and a plain exists() check will skip it forever. Writing to .part and renaming on success means a file is either absent or complete.

What should I do if a download fails halfway? Delete the partial file. Leaving it behind is worse than having nothing, because your skip logic will treat a half-written image as done.

How do I avoid re-downloading files I already have? Two independent checks: a set of completed URLs in your state file, and a filesystem existence check. Either alone has a gap; together they cover restarts and state loss.

Can I get video in a specific resolution? You get the URL the API returns. Transcoding is your side of the line — download once, then convert locally if you need other renditions.

Is there a thumbnail for videos? Yes, on the video object. It's an image URL, useful for building a grid without downloading every mp4.

How do I get only videos, or only photos? Use the dedicated endpoints rather than filtering client-side — you avoid paying for posts you then throw away.

Can I download media from a protected account? No. Protected content isn't available through any API, whatever your relationship to the account.

What happens to media from a deleted tweet? The URL stops resolving. Having the post object saved doesn't keep the file alive — if you need it, download it when you see it.

Does this need an X developer account? Not for reading public media through a third-party API — you use that provider's key. A developer account is for official API access and write operations.

How long do media URLs stay valid? Treat them as unstable. Download the bytes when you have them rather than storing URLs and fetching later.

Is scraping Twitter media legal? Reading public data and republishing someone's photo are different acts. The first is broadly treated as acceptable; the second is a copyright question independent of the API.

How do I handle rate limits in a media job? Stop cleanly when the listing call returns 429 and resume later. The media files themselves come from a CDN, so the downloads aren't what gets rate limited.

Why is my media job so much slower than a text job? Because you're moving files, not JSON. The API calls are a small fraction of wall-clock time; the downloads dominate, which is also why resumability matters more here.

Should I store media in a database or on disk? On disk or object storage, with the metadata in the database. Storing binaries as blobs makes backups and queries painful for no real gain.

Media jobs are where per-request pricing shows its shape most clearly: the API calls are cheap and the bytes are free, so what you pay for is finding the media rather than moving it. A Google sign-up comes with $0.20 in credits and no card if you want to measure that against your own account list.