emmassist-co/hidden-runner — explained in plain English
Analysis updated 2026-05-18
Run blind evaluations of a coding agent without letting it see the test answers.
Protect private benchmarks or regression suites from being read by the code they judge.
Set up maintainer-only checks that use private credentials or fixtures safely.
Get filtered, sanitized results back from a private CI environment.
| emmassist-co/hidden-runner | 00kaku/gallery-slider-block | 0xkinno/vellum | |
|---|---|---|---|
| Stars | 0 | — | 0 |
| Language | JavaScript | JavaScript | JavaScript |
| Last pushed | — | 2021-05-19 | — |
| Maintenance | — | Dormant | — |
| Setup difficulty | hard | easy | hard |
| Complexity | 4/5 | 2/5 | 4/5 |
| Audience | developer | general | developer |
Figures from each repo's GitHub metadata at analysis time.
The public CLI installs easily via npm, but the protected launcher and real hidden capability modules must be built and set up separately.
hidden-runner solves a specific trust problem that comes up when an AI coding agent works inside a repository that also contains the tests or benchmarks used to judge that agent's work. Normally, if the agent can read the repository, it can also read the answer key, the hidden tests, or the private prompts sitting right next to the code, and it might learn to satisfy those checks directly instead of actually improving the product. This tool keeps those hidden checks somewhere the agent cannot see, while still letting the agent get useful feedback. The way it works is straightforward. The agent submits its visible code to a small command line tool. That tool passes the code to a separate, protected service running as a different user account or in a separate environment. That protected service runs the real hidden evaluation, whatever it may be, and then sends back only a filtered, approved result, never the raw hidden test files, credentials, or internal file paths. The project ships as two parts. The public part is a command line client with a narrow set of commands: submitting code, checking on a job's status, retrieving the result, and listing what checks are available. The private part, which each user has to build themselves, holds the actual hidden logic, hidden test data, and rules for what parts of the result are safe to reveal. The examples included in this repository are fake sample checks meant only to demonstrate how the system works, not real hidden tests. Installing the public command line tool is done through npm, while the protected runner service needs to be built from source and set up separately, ideally under its own dedicated user account with hidden files locked down so only that account can read them. The full README is longer than what was shown.
A tool that lets a coding agent get graded on hidden tests it can never actually read or access.
Mainly JavaScript. The stack also includes JavaScript, Node.js, CLI.
Setup difficulty is rated hard, with roughly 1h+ to a first successful run.
Mainly developer.
This repo across BitVibe Labs
double-check against the repo, no cap.