Two GPUs and 128 GB of RAM? Everything the planner decides for a beginner is exposed as a knob for you: placement mode, runtime, weight and KV quants, context, tensor split, MoE expert offload, RAM pinning, draft models, and GPUs borrowed from another machine. Same free download; the door is the Setup tab.
One layout canvas decides where the chat model, draft model, image generation, image edit, video, and music servers live across your cards and system RAM. The planner prices real bytes: weights per tensor, KV per layer (windowed layers priced at their window, hybrid-attention models by their actual KV layer count), plus per-server overhead against a measured resident-VRAM baseline with a 0.95 safety factor.
Model, KV, and overhead fit the card array entirely. No RAM spill, ever. The planner reaches for this first and picks the biggest model, quant, and context that satisfy it.
Experts go to system RAM, attention and the dense layers stay on the GPU. The count is filled automatically (--n-cpu-moe), split byte-balanced across cards, with mmap chosen by RAM headroom. A 156 GB DeepSeek-class model runs on a 5090 + 3090 + 190 GB RAM this way.
Requires a runtime with --n-cpu-moe support.
A plain multi-GPU split fills the fastest card first instead of splitting evenly (-ts by layer count). On the dev rig this is the difference between the KV spilling into WDDM shared memory and not: about 30 percent at 128k context.
On one card the planner first tries to keep the chat model and the image model resident together (coexist). When they do not both fit it switches to singularity: one model at a time, the other evicted before the next loads and pinned in system RAM so the round trip is seconds. Easy Setup picks; the Setup tab lets you force either.
One model per fast GPU, the rest parked in RAM and swapped in on demand. For people who want a chat model and an image model resident at once without sharing a card.
Opt-in --split-mode tensor. CUDA only, two or more GPUs, dense models, full offload, f16 KV. The planner refuses to enable it outside those conditions rather than let it silently fall back.
Lock the model's pages in RAM (VirtualLock / mlock) so a big mmap'd model does not page out between requests. The planner picks no-mmap (fast) versus mmap per launch shape; preflight is cached so a relaunch costs about 0.1 s instead of 5.6 s.
The chat engine is a swappable runtime row. The defaults are prebuilt llama.cpp; the rest are one click to install and one dropdown to select. All of them are launched with flags the planner derives from your hardware, and you can see and override every flag. A per-model extra flags box takes anything the planner has no knob for and appends it last, so it wins.
| Runtime | Backends | Notes |
|---|---|---|
| llama.cpp (upstream) | CPU, CUDA 12, Vulkan | The default. Prebuilt per platform. Vulkan covers AMD and Intel. |
| llama.cpp (LumaByte build) | CUDA 12 | Upstream plus a small patch stack: faster mmap load (about 3x on the weight phase) and our fill-order and offload fixes as they land. Source on GitHub as luma-llamacpp. |
| ik_llama (add-on) | CUDA, CPU | The ik fork with its own quant formats (IQ*_K and friends). Installed from the Thireus prebuilt feed, ISA-matched to your CPU. Some newer llama.cpp flags are missing; the planner knows which. |
| NInfer (add-on) | CUDA | Add-on runtime with its own model format. Measured at roughly 2x llama.cpp Q6 speculative decode on the dev rig. Bench in the app, then decide. |
| MLX | Apple Silicon | mlx-lm on macOS. Models are Hugging Face repo snapshots rather than single GGUFs. |
| stable-diffusion.cpp | CPU, CUDA 12, Vulkan, Metal | Image and video engine. Per-module placement (--auto-fit) can spread a 42 GB model across two cards. No Linux CUDA prebuilt exists yet; NVIDIA Linux uses Vulkan. |
Runtimes and models live under one folder you can relocate. Easy Setup also adopts GGUFs you already have in the LM Studio folder, the Ollama blob store (named through its manifests) and the Hugging Face cache, by hardlink rather than copy. The Runtimes view is a persisted cache; if you drop a binary in by hand, use Locate to register it. Switching from another tool.
Every catalog model ships in Q4_K_M, Q6_K, and Q8_0; ik_llama adds its own formats; any single-file or sharded GGUF from Hugging Face works via the Advanced field. Sharded downloads are one job with aggregate progress and resume.
q8_0 KV is the default when it buys context. q4_0 V cache is exposed but measured unusable on Qwen 3.8; the planner guards the known-garbage combinations. Draft-model KV quant is exposed too, and on our numbers it is a loss.
Sized to what actually fits, not a fixed 4k. For Gemma-style sliding-window models the windowed layers are priced at the window and --swa-full is only used when it also fits. 128k is the sweet spot on a 5090 for the 27B class; you can push further and watch the tok/s.
Pair a draft model, or use model-free n-gram speculation with a draft cap. Measured +26 percent on edit-heavy turns for Qwen 3.8 Flash-Next with n-gram alone. MTP-grafted quants are detected and used where present.
Per-family defaults (DRY, penalties, EOG sets for known-broken quants) plus a streaming repetition monitor that aborts a verbatim loop instead of running to the token limit. All overridable.
Structured LoRA attachment (path, multiplier, high-noise flag) with curated pairs for the Turbo and Lightning models, per-model prompt profiles, and the option to pin the text encoder to CPU when VRAM is tight.
Local API in Settings serves the loaded chat model OpenAI-compatible on 127.0.0.1:8317/v1 for IDE plugins and scripts on this machine; Network Sharing exposes the same model to other machines at /sharing/llm/v1/chat/completions behind a token over TLS. A second LumaBrowser on your LAN can lend its cards to the model running on the first, over llama.cpp's RPC transport. Leases have a TTL, the lender is single-consumer, and the shared-model list is per peer. The same network-sharing layer also lets peers use your chat and image models directly, behind a PIN, over TLS.
The RPC transport itself carries no authentication and no encryption. Both switches ship off. Use it only between machines you own on a network you control; it is not safe across anything untrusted. Borrowed GPUs run the language model only, one borrower at a time, NVIDIA cards.
A fit test tells you whether a model loads. The gambit tells you whether it behaves: a fixed set of tool-calling, editing, and long-context tasks run against the model you actually launched, with the context and GPUs the run really used recorded alongside the score. It sits beside the fit test in the Setup tab and also runs from the command line with the model and context pinned.
Models that pass a fit test and then mangle tool JSON, drop arguments after one backslash, loop on a fence, or fabricate a result when a parse fails. It never fabricates a pass on a failed parse.
Score per task, wall time, decode totals for the whole turn, the GPUs and context actually used, and a health line. Two runs diff against the task intersection so an A/B includes time, not just accuracy.
Every figure on this page was measured on our dev rig: RTX 5090 (32 GB) + RTX 3090 (24 GB), 190 GB RAM, Windows. Yours will differ; run the gambit before you trust a quant.
There is no "pro" build. Install LumaBrowser, skip Easy Setup with "Customize instead", and land on the Setup tab with every flag visible.