--- title: "Performance Levers" --- # Performance Levers diffuseR runs 20B-parameter models on 16 GB consumer GPUs, and most models on no GPU at all. It does that with three independent levers: weight precision, per-component device placement, and memory residency. This vignette is the map. The machine-readable version of the same policy is `recommend()`, which inspects your VRAM, host RAM, and installed `safetensors` capabilities and returns a configuration. Today `flux_memory_profile()` delegates to it and `serve()` consults it to pick between built LTX artifacts; the FLUX-family loaders resolve their `pin` (and FLUX.1 its `text_device`) through it. For the rest of the fields it is advisory — call it and pass them to the loaders yourself. ## Lever 1: weight precision The ladder that exists in code: fp32 → bf16/fp16 → fp8 (e4m3fn) → nf4. Quantization applies to the big diffusion transformers only — VAEs, vocoders, connectors, and modulation/embedding layers always stay at 16/32 bits (they are small and precision-sensitive; each quantizer carries an exact census of which weights it may touch). | model | DiT / UNet | text encoder(s) | VAE | |---|---|---|---| | FLUX.1 (12B) | nf4, fp8 (streamed), bf16, fp32 | T5: bf16 (GPU, 14 GB+) or fp32 (CPU) · CLIP-L: fp16/fp32 | 16/32 | | FLUX.2 klein (4B) | nf4, fp8 (resident), bf16, fp32 | Qwen3-4B: bf16/fp32 | 16/32 | | Z-Image (6B) | nf4, fp8 (resident), bf16, fp32 | Qwen3-4B: bf16/fp32 | 16/32 | | LTX-2.3 (22B video) | nf4, fp8 (streamed), bf16, fp32 | Gemma3-12B: nf4, bf16, fp32 | 16/32 | | SD 2.1 / SDXL | fp16, fp32 | CLIP: fp16/fp32 | 16/32 | Two readability rules govern the ladder: - **nf4 always loads.** Its artifacts are packed uint8 plus float32 scale blocks in sub-2 GB shards — every `safetensors` build reads them. It is the default tier for the quantized families. - **fp8 needs a capable `safetensors`.** The float8 dtypes are not yet readable by the CRAN `safetensors`; `recommend()` probes the installed build and, when a card could run fp8 but the reader cannot, recommends nf4 and surfaces the suggestion in `$note` (never an error). The SD-family models ship no quantized weights: their floor is fp16, and what varies across VRAM is placement, not precision. ## Lever 2: device placement Every component takes its own device. The SD family uses explicit device maps (`auto_devices()` strategies: `full_gpu`, `unet_gpu`, `cpu_only`); the flux family and LTX use phase offloading, where each component holds the GPU only for its own phase — text encoding, denoising, decoding — and the denoiser is the sole GPU tenant during the loop. Text encoders earn special placement: FLUX.1's T5 phase-onloads in bf16 on 14 GB+ cards (its ~9.8 GB encode phase fits) and runs fp32 on the CPU below that; the Qwen3 encoders phase-onload in bf16; the Gemma3 encoder can be GPU-resident or CPU-resident with a staged swap (see below). ## Lever 3: residency From most to least VRAM: 1. **Fully GPU-resident** — everything fits, nothing moves (large cards only). 2. **Resident quantized DiT, phase-swapped everything else** — the 16 GB sweet spot: LTX nf4 and the fp8 FLUX.2/Z-Image configurations keep the DiT on the card while text encoders swap per phase. 3. **Pinned phase swap** — component weights live in page-locked ("pinned") host RAM and move to the GPU for their phase at DMA rate. Measured on a PCIe 5.0 x8 card: 25 GB/s host-to-device (11 GB of transformer re-onloads in 0.5 s), and the offload is a pointer swap back to the still-valid pinned copy — zero bytes moved, because inference never mutates weights. Page-locking costs ~0.6 s/GB once at load. Built for the LTX pipeline components, the Gemma3 encoder, and the FLUX-family transformer, VAE decoder, and text encoder(s). 4. **Pageable phase swap** — the same movement through ordinary memory, at roughly 2-16 GB/s depending on tensor layout. The fallback when pinning is opted out or page-locking fails. 5. **Streamed weights** — the bigger-than-VRAM tier: weights stay in (pinned) host RAM permanently and stream across PCIe during each forward pass, about one byte per parameter per step. LTX fp8 and FLUX.1 fp8 run this way. 6. **CPU-only** — every model runs without a GPU, at its floor precision. There is no disk tier at inference time: weights load from disk once, and the closest thing to "swap to disk" is *unpinned* host memory being paged out by the OS — which is exactly the trade `recommend()` weighs. ## Pinning: when and when not Pinned pages are unswappable — they subtract from what the OS can page out, so on small-RAM machines they convert memory pressure into process kills rather than slowdowns. `recommend()` therefore returns `pin = TRUE` only when available host RAM covers the model's estimated pinned set twice over, `FALSE` on the CPU tier (nothing stages), and `TRUE` when RAM cannot be detected, because page-locking already fails soft per component. The global switch is `options(diffuseR.pin_staging = FALSE)` — reach for it under host memory pressure, in containers with hard memory caps, or for single-generation sessions where the one-time page-lock never pays itself back. The LTX pipeline, the Gemma3 loaders, and the FLUX-family image loaders consume the decision (their `pin` argument defaults to it); the SD-family loaders place components statically, so pinning is inert for them. ## Putting it together ```r r <- recommend("ltx") # or "flux1", "flux2", "zimage", "sdxl", "sd21" r$precision # tier the card + safetensors support r$devices # per-component placement r$pin # page-lock the phase-swapped host copies? r$note # fork suggestion when fp8 wanted but unreadable ``` Treat the result as the machine's advice: pass its fields to the loaders and generators. The FLUX-family loaders' `pin` (and FLUX.1's `text_device`), `flux_memory_profile()`, and `serve()`'s LTX artifact selection consume it automatically.