ronikobrosly/rigorloop — explained in plain English
Analysis updated 2026-05-18
Build a text extraction or reformatting script from example input/output pairs, with a trustworthy accuracy estimate.
Generate a Claude Skill or AGENTS.md guidance file and validate it against held-out examples before trusting it.
Check whether a candidate solution actually generalizes instead of just overfitting to its dev examples.
Estimate the model-call cost of an agentic build run before spending the budget.
| ronikobrosly/rigorloop | lynote-ai/ai-detector-skill | rss3208/visiomaster | |
|---|---|---|---|
| Stars | 134 | 134 | 134 |
| Language | Python | Python | Python |
| Setup difficulty | moderate | easy | hard |
| Complexity | 3/5 | 2/5 | 3/5 |
| Audience | developer | general | researcher |
Figures from each repo's GitHub metadata at analysis time.
Requires the Claude CLI installed and authenticated, plus a representative set of input/output examples, ideally 100 or more.
RigorLoop is a Python framework that uses AI agents to build small solutions to data-transformation tasks, like extracting fields from messy text or reformatting one structure into another, and evaluates the result the way a careful data scientist would rather than trusting a single test run. You give it a plain-language task description, a set of example inputs and expected outputs, and a list of checks, and it repeatedly runs a strategy agent that directs several executor agents to write and refine candidate solutions. What makes RigorLoop different from a single coding-agent session is how it measures success. Your examples get split once into three separate groups: a development set the building agents see while working, a validation set used to decide which candidate is actually improving and when to stop, and a held-out test set that no agent ever sees until the very end. This split is meant to catch a solution that only looks good because it overfit to the examples it was built and checked against, and the final report shows pass rates with confidence intervals on all three splits so the score can actually be trusted. The output can take one of three forms: a plain executable Python script, a Claude-style agent skill document, or a guidance file meant to steer a coding agent, and any of these can be copied out and used on their own without RigorLoop installed. Checks that decide whether an example passes include exact string matching, matching after normalization, JSON equality, regular expression matching, numeric tolerance, a custom Python script, or having a model judge the output against a rubric. The project requires Python 3.12 or newer on Linux or macOS, along with the Claude CLI installed and authenticated, since it invokes agents through that tool. It warns that small example sets cannot prove much statistically and recommends at least a hundred examples, and it estimates how many model calls a run will cost before you commit to running it.
A Python framework that runs agentic loops to build extraction and reformatting solutions, then evaluates them on dev, validation, and held-out test splits so results are not overfit.
Mainly Python. The stack also includes Python, Claude CLI, TOML.
Setup difficulty is rated moderate, with roughly 1h+ to a first successful run.
Mainly developer.
This repo across BitVibe Labs
double-check against the repo, no cap.