Run a coding assistant locally with a large context window for programming tasks.
Host a private chat assistant on your own machine without cloud costs.
Search Hugging Face for AI models that fit your computer's memory and graphics power.
Run large models that exceed your graphics memory using a hybrid system RAM mode.
| antonellof/rondine | 0xustaz/streamgate | a-bissell/unleash-lite | |
|---|---|---|---|
| Stars | 1 | 1 | 1 |
| Language | Python | Python | Python |
| Setup difficulty | moderate | hard | hard |
| Complexity | 4/5 | 4/5 | 4/5 |
| Audience | developer | developer | researcher |
Figures from each repo's GitHub metadata at analysis time.
Requires compatible hardware like Apple Silicon Macs or NVIDIA graphics cards to run models locally.
Rondine is a tool for running large language models on your own computer instead of through a cloud service. It figures out how much memory and graphics power your machine has, recommends models that will actually fit, downloads them, and starts a server that other software can talk to. It works on Mac computers with Apple Silicon chips, computers with NVIDIA graphics cards, and NVIDIA DGX Spark systems. The tool does not build its own engine for running models. Instead, it acts as a layer on top of three existing engines called llama.cpp, MLX-LM, and vLLM. Rondine's job is to pick the right engine for your hardware, configure it with tuned settings like batch sizes and memory limits, and handle the workflow of getting a model up and running. You start it with a single command, and it guides you through checking your system, picking a model, downloading it, and starting the server. Rondine offers profiles for different use cases. A "coding" profile picks models suited for programming tasks, sets a larger context window of 32,000 tokens, and tunes the engine for a single user. A "chat" profile uses a smaller context and settings geared toward conversation. The tool can also search Hugging Face, a popular model repository, to discover models that fit your hardware. Results are labeled to show whether they come from Rondine's curated catalog, a live search, or a hardware-targeted recommendation. For models too large to fit entirely in graphics memory, Rondine supports a hybrid mode where some data stays in system RAM while the rest goes to the graphics card. There is also an experimental mode that lets very large models read directly from a fast SSD, though this can be extremely slow. Rondine will never choose this SSD mode automatically because performance may be unusable. The project includes documentation on installation, command-line options, engine tuning, and benchmarks. On a 32GB M2 Pro Mac, one test model ran at about 66.6 tokens per second. The code is open source under the Apache 2.0 license.
A tool that runs large language models on your own computer by auto-detecting your hardware, recommending models that fit, and starting a server for other apps to use.
Mainly Python. The stack also includes Python, llama.cpp, MLX-LM.
Use freely for any purpose, including commercial use, as long as you keep the copyright notice.
Setup difficulty is rated moderate, with roughly 30min to a first successful run.
Mainly developer.
This repo across BitVibe Labs
double-check against the repo, no cap.