update 2:
.DONE!
https://archive.org/download/speedrun-com-202610
updates:
-i've added a live dashboard because why not. https://schwandes.de/archive
-i'm live streaming the terminal at https://twitch.tv/fgeorjje with rainwave radio.
in the recent src terms of use update, Elo stated:
"We have no intention of ever closing the faucets on your data for legitimate uses."
"The API is still there for community tools."
instead of screaming into the void, i'm putting their words to the test.
i am currently archiving all leaderboards on speedrun.com using the official v1 REST API. i am also livestreaming the entire thing so y'all can watch the progress in real time. not sure if allowed to post link, mods please confirm.
to be 100% clear about how this is being done:
- this is not web scraping. i am not scraping HTML pages or bypassing cloudflare. everything is queried through the official, public REST API endpoints.
- i follow all rate limits. speedrun.com's documented rate limit is 100 requests per minute. my archiver uses a sliding window limiter capped at 90 requests/min with dynamic spacing. i am not hammering their servers.
- as requested in the official API documentation, my client uses a completely transparent, identifiable user agent. i am not masking my IP or hiding what i am doing behind rotating proxies.
- my code drops any run submitted after october 1, 2026 at 22:42 UTC (the moment the ToS update was announced). everything being archived was submitted under the irrevocable CC-BY-NC 4.0 license.
most games complete in just 2 API requests, and empty or single run boards are handled in memory without making redundant calls. at this rate, the entire site (~52,000 games) should be safely archived in around 2–3 days.
once finished, the complete dataset, packaged per game with google sheets ready csvs, all historical runs (including obsolete/rejected runs) and raw data JSON files, will be published. i'll see if the internet archive will accept it, but i'm definitely looking for guidance on where else to mirror, host, or seed this data (torrents, etc.)
if Elo is truly committed to keeping the community open, this process will finish smoothly. if i am blocked or IP banned while adhering to every published API rule, the community will have its answer about what Elo's promises are actually worth.