lancedb/liteparse-lancedb-pdf-qa — explained in plain English
Analysis updated 2026-05-18
Turn scanned or image-heavy PDF reports into a searchable local database.
Compare different retrieval strategies like text search, image search, and hybrid search on documents.
Build a question-answering pipeline that reads real page screenshots instead of only extracted text.
Benchmark parsing speed and storage cost when indexing PDF reports for search.
| lancedb/liteparse-lancedb-pdf-qa | alirezarezvani/promptor | ayushk58/crawlyx | |
|---|---|---|---|
| Stars | 9 | 9 | 9 |
| Language | Python | Python | Python |
| Last pushed | — | 2025-09-14 | — |
| Maintenance | — | Quiet | — |
| Setup difficulty | moderate | easy | easy |
| Complexity | 3/5 | 2/5 | 2/5 |
| Audience | researcher | pm founder | pm founder |
Figures from each repo's GitHub metadata at analysis time.
Requires Python 3.13+, the uv package manager, and an OpenAI API key for most steps (an offline mode is available).
This is a demo project showing how to turn messy, image heavy PDF reports, such as corporate sustainability reports, into a local, searchable database that keeps both the text and the visual layout of every page. It is meant as companion code for a blog post about building this kind of local multimodal evidence store, and it runs entirely on your own machine without sending documents to the cloud. The pipeline works in stages. First, a tool called LiteParse reads each PDF and pulls out the text, extracts any figures or charts as images, and takes a screenshot of every page, all without calling any cloud service or AI model during this parsing step. Those results are normalized into structured records covering documents, pages, text chunks, and extracted image assets, all linked together by a shared page identifier. Everything then gets stored in LanceDB, a database built for handling both text and images together, along with the numeric vector representations used for search and full details about where each record came from. The demo uses six real sustainability reports, from companies including Google, Samsung, and Nvidia, paired with fifty benchmark questions that are labeled with the specific page containing the answer. Once the data is loaded, the project supports five different ways to search it, including plain text search over chunks or full pages, image based search using CLIP for finding visually similar figures, and a hybrid mode that combines several of these approaches. There is also an optional final step where an AI agent reads the actual page screenshots stored in the database and tries to answer the benchmark questions, with a separate AI model then judging how correct those answers were. Setting the project up requires Python 3.13 or newer and the uv package manager. Most of the pipeline needs an OpenAI API key for text embeddings and for the optional answer and judging steps, though the project also includes an offline mode using a simple local hash based embedder for testing without any API key. The README includes benchmark numbers for parsing speed, storage size, and how well each retrieval mode finds the correct page for a given question.
A local demo pipeline that parses messy PDF reports into text, images, and page screenshots, storing them all in LanceDB for multimodal search and AI question answering.
Mainly Python. The stack also includes Python, LanceDB, LiteParse.
Setup difficulty is rated moderate, with roughly 1h+ to a first successful run.
Mainly researcher.
This repo across BitVibe Labs
double-check against the repo, no cap.