Your GPUs are
underperforming.
Pool your mismatched CUDA + ROCm + Metal GPUs into one OpenAI-compatible endpoint. Run a model that fits on no single machine — then make it faster with speculative decoding.
$ tightwad start ✓ Pool 4 GPUs · 52GB VRAM ✓ Model Llama 3.3 70B ✓ Endpoint localhost:8088 → 2.2 → 4.1 tok/s · 1.86× faster*
pip install tightwad* Measured on Llama 3.1 8B → Llama 3.3 70B, 52GB pooled over WiFi, temperature=0. Output matches the target alone with greedy decoding. Results vary. Benchmark details →
Two moves. That's the whole product.
Pool your GPUs so a big model runs at all — then change one URL so it runs fast. Dead simple, both of them.
One URL change
Point your chat app at port 8088 instead of 11434. That's the entire setup from your app's perspective.
The small model is invisible
The draft model suggests tokens; the large model accepts or corrects them. Your chat app only sees the final output.
Output quality is preserved
With greedy decoding (temperature=0), output matches the target alone. With sampling, output is statistically equivalent. The large model validates the draft.
Nothing sits idle
Your main rig, the old GPU you almost sold, an Apple laptop, or a CPU-only machine can all contribute.
That's it. Pool your hardware. Change one URL. Run bigger models, faster.
Set It Up in 30 Minutes →How a small model makes a big one fast
Speculative decoding is what Google and DeepMind already use to accelerate frontier models. Tightwad puts it on your pooled hardware.
Draft
A small model proposes a batch of candidate tokens on inexpensive hardware.
Verify
The large model evaluates the candidates together in a single forward pass.
Accept
Keep matching tokens. At the first disagreement, use the large model's correction.
Stream
Send accepted tokens to your app, with output equivalent to the target alone.
Pick your mode. Stack them.
Six inference modes — pool, speculate, race, cluster, gate, distribute. Run one or run all six. Full details on each →
Combined Mode
Pool your GPUs to fit a bigger model, then speculate on top. 1.86× measured on Llama 3.3 70B across four GPUs over WiFi.
- ✓ Run models that fit nowhere else
- ✓ 2.2 → 4.1 tok/s on the 70B pool
Speculative Proxy
A small model drafts; a large model verifies. Accelerate local inference through an OpenAI-compatible proxy, with output equivalent to the target alone.
- ✓ One URL change for your app
- ✓ Live dashboard at
/dashboard
Multi-Drafter Consensus approximate mode
Race cheap drafters in parallel. When they agree, skip target verification. An opt-in, approximate mode for trading exactness for speed.
- ✓ Skip the GPU when drafters agree
- ✓ Opt-in; consensus tokens may differ from the target
RPC Cluster
Spread model layers across CUDA, ROCm, and Metal machines with llama.cpp RPC. One endpoint for your hardware, with hot-swappable models.
- ✓ Mix NVIDIA + AMD + Apple freely
- ✓ 70B+ on consumer hardware
Quality Gate
Let a fleet of inexpensive machines draft full responses. One powerful GPU reviews them, approving, correcting, or rejecting each answer.
- ✓ Put idle CPUs to work
- ✓
tightwad gate start
Swarm Transfer
Share model files directly between workers in verified pieces. Downloads resume after interruptions, with delta updates for new versions.
- ✓ Multi-source parallel download
- ✓ Delta updates for new quants
The headline number, and how we got it
Llama 3.1 8B → Llama 3.3 70B, across a 4-GPU RPC pool (52GB VRAM, over WiFi), greedy decoding (temperature=0). Throughput went from 2.2 → 4.1 tok/s — and the 70B fits on no single machine in the pool. Under greedy decoding the output is mathematically identical to running the target alone.
Other pairings, acceptance rates, and the cloud-API caveats live on the benchmarks page — including which numbers are being re-validated under the v0.5.1+ per-position verifier.
See full benchmarks →Quick Start
No Docker Compose files with 300 environment variables. No Kubernetes YAML. Just Python and one config file.
Install
$ pip install tightwad # or from source: $ git clone https://github.com/youngharold/tightwad.git $ cd tightwad && pip install .
Configure your hardware
Connect your draft and target models. Follow the setup guide →
View example configuration
proxy: host: 0.0.0.0 port: 8088 max_draft_tokens: auto # auto-tunes based on acceptance rate fallback_on_draft_failure: true draft: url: http://192.168.1.50:11434 # Your cheap GPU (Ollama) model_name: qwen3:8b backend: ollama target: url: http://192.168.1.100:11434 # Your big GPU (Ollama) model_name: qwen3:32b backend: ollama
Start it & test
$ tightwad proxy start ✓ Draft model healthy ✓ Target model healthy ✓ Proxy listening on http://localhost:8088 # Test it (drop-in for any OpenAI SDK call) $ curl http://localhost:8088/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50}' # Check live stats $ tightwad proxy status → Rounds: 34 | Draft tokens/round: 32 | Mode: greedy (exact)
Build RPC workers (CUDA — Windows/Linux)
# Or use scripts/install-worker.sh $ cmake -B build -DGGML_CUDA=ON -DGGML_RPC=ON $ cmake --build build --config Release $ build/bin/rpc-server -p 50052 # GPU 0
Configure cluster topology
Add your coordinator, workers, and model. See the cluster guide →
View example configuration
coordinator: host: 0.0.0.0 port: 8080 backend: hip # or cuda gpus: - name: "7900 XTX #0" vram_gb: 24 workers: - host: 192.168.1.100 # NVIDIA box gpus: - name: "RTX 4070 Ti Super" vram_gb: 16 rpc_port: 50052 models: llama-3.3-70b: path: /models/Llama-3.3-70B-Q4_K_M.gguf ctx_size: 8192 flash_attn: true default: true
Start the cluster
$ tightwad start ✓ Coordinator started ✓ Worker @ 192.168.1.100:50052 online ✓ Model llama-3.3-70b loaded across 52 GB VRAM # Hot-swap to a different model anytime $ tightwad swap deepseek-r1-70b # Run the benchmark $ tightwad benchmark
Want the full walkthrough on real mismatched hardware?
Homelab cluster in 30 minutes →The research behind speculative decodingThree references from Google and DeepMind. Explore the papers and methodology.
Fast Inference from Transformers via Speculative Decoding
The foundational paper. Introduces the draft-verify loop and proves output equivalence under greedy decoding.
arxiv.org/abs/2211.17192 →Accelerating Large Language Model Decoding with Speculative Sampling
Independent parallel formulation, extends the technique to stochastic sampling with the rejection-sampling trick.
arxiv.org/abs/2302.01318 →Looking Back at Speculative Decoding
Plain-English retrospective from the original authors covering production deployment, what held up, and what didn’t.
research.google →Tightwad is independent open-source software (MIT) with no affiliation, endorsement, or commercial relationship with Google, Google DeepMind, or the listed authors. Citations are nominative fair use of public academic publications.
Why not vLLM, Ollama, or plain llama.cpp?
Each is great at something. Tightwad's lane is the awkward middle: mismatched, network-separated, consumer hardware.
Honest, side-by-side — including when not to use Tightwad.
Full comparison →