git404hub

what is agent-state fr?

ovachiever/agent-state — explained in plain English

Analysis updated 2026-05-18

1PythonAudience · developerComplexity · 3/5Setup · moderate

tl;dr

A nightly benchmark that scores your AI coding assistant on real tasks through the CLIs you actually use, to detect if the model quietly got worse over time.

vibe map

mindmap
  root((repo))
    What it does
      Nightly coding battery
      Compares to gut feel
    Tech stack
      Python
      uv package manager
    Features
      18 coding tasks
      Frozen math anchor
      Blind trial period
    Metrics
      Intelligence
      Diligence
      Judgment
    Audience
      Daily CLI coding users
      Skeptics of model decay

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

what do people make with this?

VIBE 1

Run a nightly benchmark against your coding CLI to catch a model's quality quietly declining.

VIBE 2

Compare your own daily gut sense of a model's performance against a blind, scored verdict.

VIBE 3

Track five separate skills, including whether a model caves under false pushback.

what's the stack?

Pythonuv

how it stacks up fr

ovachiever/agent-state0xustaz/streamgatea-bissell/unleash-lite
Stars111
LanguagePythonPythonPython
Setup difficultymoderatehardhard
Complexity3/54/54/5
Audiencedeveloperdeveloperresearcher

Figures from each repo's GitHub metadata at analysis time.

how do i run it?

Difficulty · moderate time til it works · 30min

Requires editing a config file with binary paths and model names before the first run.

in plain english

Agent state is a tool for developers who use AI coding assistants like Claude Code or Codex every day and suspect the model quietly gets worse over time. Many people notice a coding model feels sharp when it launches, then a few weeks later starts missing instructions, leaving unfinished work, or agreeing with wrong feedback instead of pushing back. This project tries to measure that feeling instead of just trusting it. Each night, the tool runs 18 coding tasks through the actual command line tools you use, in a fresh throwaway workspace, and scores the results against roughly 300 automated checks. It tracks five things: whether the code is correct, whether the model finishes multi step edits, whether it leaves stubs or half done work, whether it can hold dozens of scattered requirements across a long brief, and whether it can tell the difference between a true bug report and a confidently wrong one. That last category also checks whether the model caves when told its correct work is wrong, and whether it catches its own overclaiming when it says a task succeeded but the checks say otherwise. Alongside the daily coding battery, a fixed set of thirty math competition problems runs against the raw provider APIs every night as a steady reference point, so a drop there points to a change in the model itself rather than a tooling update. For the first two weeks after setup, the tool's verdicts stay hidden while you log your own daily gut feeling about the model's performance. At the end, the two records are compared: if the tool's hidden verdicts matched your gut, it proves itself trustworthy going forward. If not, it automatically turns off its own schedule. Setup happens through Python's uv tool, with a config file for model names, binary paths, and dates. The project is written in Python and has one star, so it is early and largely a personal project by its sole listed contributor.

prompts (copy fr)

prompt 1
Help me install agent-state with uv and configure it to test Claude Code on my machine.
prompt 2
Explain how agent-state's pushback-hold and pushback-fix tasks catch a model caving to false feedback.
prompt 3
Walk me through setting up agent-state's two week blind trial and the burnin comparison at the end.
prompt 4
Summarize what agent-state's CUSUM slow decay alarm is measuring and why it was rebuilt.

Frequently asked questions

what is agent-state fr?

A nightly benchmark that scores your AI coding assistant on real tasks through the CLIs you actually use, to detect if the model quietly got worse over time.

What language is agent-state written in?

Mainly Python. The stack also includes Python, uv.

How hard is agent-state to set up?

Setup difficulty is rated moderate, with roughly 30min to a first successful run.

Who is agent-state for?

Mainly developer.

peek the repo → explain another one

This repo across BitVibe Labs

double-check against the repo, no cap.