Catch retrieval quality regressions in a RAG pipeline before they reach production.
Compare embedding models or chunking strategies with a printed cost estimate before running.
Get a verified explanation for why a specific search query started failing after a config change.
Add a statistically gated retrieval quality check to a CI pipeline.
| skundu42/recallops | 0xallam/my-recipe | 0xhassaan/nn-from-scratch | |
|---|---|---|---|
| Stars | 0 | — | 0 |
| Language | Python | Python | Python |
| Last pushed | — | 2022-11-22 | — |
| Maintenance | — | Dormant | — |
| Setup difficulty | easy | moderate | moderate |
| Complexity | 4/5 | 2/5 | 4/5 |
| Audience | developer | general | developer |
Figures from each repo's GitHub metadata at analysis time.
Runs fully offline at no cost by default, vector database and embedding provider extras are optional.
RecallOps is a Python tool built to catch a specific problem: when teams change how they process documents for AI search systems, called RAG pipelines, like switching embedding models, chunking methods, or rerankers, the quality of search results can quietly get worse. Most tools cannot tell you exactly why that happened, only that something changed. RecallOps is built to answer that question with proof, not guesses. The core idea is verified attribution. When search quality drops, RecallOps does not just point at a likely cause. It reruns the pipeline with only one factor reverted at a time and checks whether the failing search recovers. Only if that test passes does it report the cause as verified. Anything it cannot confirm this way is labeled as a hypothesis instead, kept separate from confirmed causes. To make this possible, RecallOps stores full embeddings, not just references to them, so it can replay past searches and test changes without paying to regenerate embeddings each time. It works by versioning an entire document pipeline, including the documents, how they were split into chunks, the embeddings, and the configuration, using content based identifiers so identical inputs always produce the same result. By default RecallOps runs entirely offline at no cost, using local hash based embeddings and a built in exact search method, with no API keys or servers required. Optional extras add support for vector databases such as pgvector, Qdrant, Chroma, and LanceDB, plus embedding providers including OpenAI, Cohere, Voyage, and sentence transformers. Any operation that would cost money through an external provider prints a cost estimate first and requires explicit confirmation. It also includes a continuous integration testing feature that checks retrieval quality against a set of test queries, using statistical methods designed to avoid false alarms from random noise. The project is aimed at teams building or maintaining RAG based search systems who want a way to test changes safely and understand exactly what broke after an update, backed by evidence rather than assumptions. It is licensed under Apache 2.0.
A Python tool that tests AI search pipelines for quality regressions and proves the exact cause using verified counterfactual reruns.
Mainly Python. The stack also includes Python, pgvector, Qdrant.
Use freely for any purpose, including commercial use, as long as you keep the copyright notice.
Setup difficulty is rated easy, with roughly 5min to a first successful run.
Mainly developer.
This repo across BitVibe Labs
double-check against the repo, no cap.