davidbellamy/icml2026-scraper — explained in plain English
Analysis updated 2026-05-18
Scrape an academic conference's full schedule, papers, and PDFs into one dataset.
Let an AI agent query a conference's papers and sessions instead of browsing manually.
Publish a structured, agent-friendly dataset of a conference to Hugging Face.
Adapt the same pipeline to scrape a different Whova or OpenReview based conference.
| davidbellamy/icml2026-scraper | 0xustaz/streamgate | a-bissell/unleash-lite | |
|---|---|---|---|
| Stars | 1 | 1 | 1 |
| Language | Python | Python | Python |
| Setup difficulty | moderate | hard | hard |
| Complexity | 3/5 | 4/5 | 4/5 |
| Audience | researcher | developer | researcher |
Figures from each repo's GitHub metadata at analysis time.
Requires OpenReview and conference attendee login credentials for the full paper set, plus an optional Hugging Face token to publish.
This project is a Python pipeline that collects information about an entire academic machine learning conference, ICML 2026, and turns it into a clean, organized dataset you can search and question with an AI assistant. The result, already published as a public dataset, covers 10,815 papers, including both main conference papers and workshop papers, along with titles, abstracts, author names, session times, room locations, and links to the PDFs. The pipeline works in stages. It starts by reading the conference's public calendar page to build a full multi-day schedule of events. It then visits the individual poster pages for main conference papers, which are simple enough that the code can read them directly without needing a web browser, to collect titles, authors, and abstracts. Separately, it logs into OpenReview, the platform many conferences use to manage paper submissions, to pull the accepted main conference papers along with their PDFs and author identifiers. It also logs into the conference's own attendee area to reach workshop papers that are not available through OpenReview. Finally, all of this information is organized into a flat table format, one row per paper, and packaged into files that are easy for both people and AI tools to work with. A key idea behind the project is that this same approach can work for other conferences that use similar virtual conference platforms and OpenReview, not just ICML 2026, by changing the web addresses and identifiers used to fetch the data. Each step in the pipeline can be resumed if interrupted, since it skips items already collected, and is designed to avoid overwhelming the source websites with requests. To use it, you install the required Python packages and add your own login credentials for OpenReview, the conference website, and optionally a Hugging Face account if you want to publish the resulting dataset yourself. The code itself is released under the MIT license, though the conference data it collects belongs to the original authors and the conference.
A Python pipeline that scrapes an academic ML conference into a searchable, agent-queryable dataset of papers, schedules, and PDFs.
Mainly Python. The stack also includes Python, OpenReview API, Hugging Face.
The code is MIT licensed, so you can use and modify it freely, but the conference data it produces belongs to the original authors and conference.
Setup difficulty is rated moderate, with roughly 30min to a first successful run.
Mainly researcher.
This repo across BitVibe Labs
double-check against the repo, no cap.