mormolykos/trainproof — explained in plain English
Analysis updated 2026-05-18
Catch a diverging or NaN training run early before it burns hours of GPU time.
Check a dataset or tokenizer for quality problems before starting a training run.
Add an automatic pass or fail gate for training runs inside a CI pipeline.
Compare a new training run against a known good baseline to catch subtle problems like shuffled labels.
| mormolykos/trainproof | 0xallam/my-recipe | 0xhassaan/nn-from-scratch | |
|---|---|---|---|
| Stars | 0 | — | 0 |
| Language | Python | Python | Python |
| Last pushed | — | 2022-11-22 | — |
| Maintenance | — | Dormant | — |
| Setup difficulty | easy | moderate | moderate |
| Complexity | 3/5 | 2/5 | 4/5 |
| Audience | developer | general | developer |
Figures from each repo's GitHub metadata at analysis time.
Installs with pip install trainproof and runs directly against existing training log files or datasets.
trainproof is a command line tool that checks machine learning training runs for problems before they waste time and money on GPU computing. Instead of relying on another AI model to guess whether a training run looks healthy, it uses a fixed set of deterministic rules with clear numeric thresholds, and every finding it reports points to the specific numbers that triggered it. The tool can check several stages of the training process. Before training starts, it can look at a dataset or a tokenizer for problems such as broken audio files, mismatched transcripts, duplicate entries, or vocabulary issues that could hurt training later. During and after training, it reads the log files a training run produces and looks for signs of trouble like the loss curve diverging, exploding gradient spikes, a learning rate set to zero so nothing is actually learning, or a run that appears to stop improving. Each check returns a PASS, WARN, or FAIL verdict with cited evidence, and can be plugged into a continuous integration pipeline since it sets an exit code based on the result. The author built this after losing an eleven hour fine-tuning run to a failure that no existing tool caught, and the README backs up its claims with real training logs from five actual fine-tuning attempts, one healthy and four each broken in one specific way on purpose, so readers can see exactly which failures the rules do and do not catch. One honest limitation the README calls out directly is that a loss curve alone cannot detect a dataset where the labels have been shuffled, since the model still appears to be learning something, even though it is meaningless. To catch that kind of problem, the tool also includes a comparison mode that checks a run's numbers against a known good baseline run. This project is aimed at people who train machine learning models, especially those doing fine-tuning work, and it assumes familiarity with training logs and command line tools. It is installed with pip and is released under the MIT license, a permissive license allowing free use including commercial use.
A command line checker that catches broken machine learning training runs early, using fixed rules and cited evidence instead of another AI's guesswork.
Mainly Python. The stack also includes Python, CLI.
Released under the MIT License, which allows free use, modification, and commercial use as long as the original copyright notice is kept.
Setup difficulty is rated easy, with roughly 5min to a first successful run.
Mainly developer.
This repo across BitVibe Labs
double-check against the repo, no cap.