Run a huge open source AI language model on a single consumer PC with two GPUs
Experiment with cutting edge MoE model architectures without data center hardware
Offload some AI computation to the CPU to free up graphics card memory
Compare decode speed across different quantization formats for a given model
| giannisanni/pulsar | doggy8088/leak-hunter | sverrejb/slint-kindle-backend | |
|---|---|---|---|
| Stars | 57 | 57 | 57 |
| Language | Rust | Rust | Rust |
| Setup difficulty | hard | easy | hard |
| Complexity | 5/5 | 3/5 | 4/5 |
| Audience | developer | ops devops | developer |
Figures from each repo's GitHub metadata at analysis time.
Needs CUDA-capable GPUs and a fast NVMe drive, tuning for a given model takes experimentation.
Pulsar is a piece of software that lets very large AI language models run on ordinary consumer computers that normally would not have enough graphics card memory to hold them. It does this by keeping the parts of the model that make decisions loaded in the graphics card's fast memory, while storing the much larger set of specialized sub-models, sometimes called experts, on a regular solid state drive and streaming them in as needed, one piece at a time. It is written in Rust and CUDA and is a full rebuild of an earlier project called NeutronStar. The project currently supports nine different AI model designs, including some of the largest publicly available open models, with sizes ranging from tens of billions up to roughly a trillion parameters. On a reference machine with two consumer graphics cards worth 16 gigabytes each, a regular gaming CPU, and one fast NVMe drive, the project reports running these enormous models at speeds from a few tokens per second up to around 50, depending on the model's size and design. Beyond simple support, the README documents a long series of performance improvements and fixes: better handling of long conversations and long documents, a mode that lets the regular computer processor help calculate parts of the model instead of only relying on the graphics card, and corrections to bugs that were quietly making some of the math slightly wrong before. Each improvement is described with before and after numbers from the author's own testing setup. For someone without a technical background, the practical meaning is this: normally, running the very largest AI models requires expensive professional hardware costing many thousands of dollars. Pulsar is an attempt to make that possible on hobbyist grade hardware instead, trading raw speed for accessibility, aimed at developers and AI enthusiasts who want to experiment with cutting edge open models without buying a data center.
An inference engine that runs huge AI models on consumer GPUs by streaming most of the model from an SSD instead of requiring massive graphics memory.
Mainly Rust. The stack also includes Rust, CUDA, GGUF.
Setup difficulty is rated hard, with roughly 1h+ to a first successful run.
Mainly developer.
This repo across BitVibe Labs
double-check against the repo, no cap.