Soup fine-tunes an 8B model on a 4GB laptop GPU, and publishes the claims it withdrew
Soup picked up 303 stars today on GitHub trending, and the headline number is that it trained Llama-3.1-8B-Instruct on an RTX 3050 Laptop with 4GB of VRAM. Measured: 119.6 tokens per second at 3.32 GB peak with NF4. Apache-2.0, built by Alpamys Makazhan, at https://github.com/MakazhanAlpamys/Soup
The trick is layer streaming. Normal fine-tuning keeps the frozen base model resident in GPU memory, which sets the floor on what hardware you need β and for an 8B model that floor is well above a gaming laptop. Soup streams the frozen base from host RAM one decoder layer at a time instead. Only the layer being used occupies VRAM. The peak memory number stops tracking model size and starts tracking layer size, which is a much smaller and much flatter thing. The project claims bit-exact equivalence with standard resident training, meaning this is a memory schedule, not an approximation that quietly changes your results.
The interface is the other half. One command, one YAML file. Install is pip install soup-cli[train], Python 3.10 through 3.12, and it works with any HuggingFace Hub text-generation model, with explicit support for Llama, Qwen, Gemma and Mistral. The README's framing is "no SSH, no config hell," which is aimed squarely at the gap between wanting to post-train something and spending a weekend on cloud GPU plumbing before you find out whether your idea was any good.
Here's what actually makes this worth your attention over the other projects promising the same thing. The repo publishes its measurement records including null results and claims it withdrew. There's a free Colab notebook so you can reproduce the headline number yourself, and a preprint with a DOI. In a category where every README claims an impossible speedup and nobody shows the runs that didn't work, publishing your retractions is a stronger credibility signal than any benchmark table.
Caveat worth stating: streaming from host RAM costs bandwidth, so this buys you access, not speed. If you already have the VRAM, resident training will be faster. The point is the floor β an 8B post-training run is now something you can start on the laptop you already own, and find out whether the idea works before you rent anything.
← Back to all articles
The trick is layer streaming. Normal fine-tuning keeps the frozen base model resident in GPU memory, which sets the floor on what hardware you need β and for an 8B model that floor is well above a gaming laptop. Soup streams the frozen base from host RAM one decoder layer at a time instead. Only the layer being used occupies VRAM. The peak memory number stops tracking model size and starts tracking layer size, which is a much smaller and much flatter thing. The project claims bit-exact equivalence with standard resident training, meaning this is a memory schedule, not an approximation that quietly changes your results.
The interface is the other half. One command, one YAML file. Install is pip install soup-cli[train], Python 3.10 through 3.12, and it works with any HuggingFace Hub text-generation model, with explicit support for Llama, Qwen, Gemma and Mistral. The README's framing is "no SSH, no config hell," which is aimed squarely at the gap between wanting to post-train something and spending a weekend on cloud GPU plumbing before you find out whether your idea was any good.
Here's what actually makes this worth your attention over the other projects promising the same thing. The repo publishes its measurement records including null results and claims it withdrew. There's a free Colab notebook so you can reproduce the headline number yourself, and a preprint with a DOI. In a category where every README claims an impossible speedup and nobody shows the runs that didn't work, publishing your retractions is a stronger credibility signal than any benchmark table.
Caveat worth stating: streaming from host RAM costs bandwidth, so this buys you access, not speed. If you already have the VRAM, resident training will be faster. The point is the floor β an 8B post-training run is now something you can start on the laptop you already own, and find out whether the idea works before you rent anything.
Comments