No hype. Just measured numbers from a real M3 Max 36GB Mac.
| Your Situation | Does This Help? | How Much? |
|---|---|---|
| Model fits easily (<70% RAM) | No. Pure MLX is already at full speed. | 0% gain |
| Model barely fits (80-100% RAM) | Yes. Memory pressure kills 40-60% of your speed silently. | 2.4x recovery (measured) |
| Model doesn't fit (>100% RAM) | Yes. Makes impossible possible. | Slow but works (3-7 tok/s) |
| Running other apps while using LLM | Yes. Memory manager prevents the LLM from stealing your RAM. | No more beach balls |
The problem: You downloaded Mixtral-8x22B (26GB) on your 36GB Mac. It loaded fine, but generation is weirdly slow — 40 tok/s instead of the expected 100+. Activity Monitor shows "Memory Pressure" in yellow.
What's happening: macOS is compressing and swapping memory pages. Every time MLX needs a tensor, macOS has to decompress it first. This adds 0.5-2ms per tensor access, and with 32 layers x hundreds of tensors per token, it accumulates into a 60% slowdown.
What we do: Mixed precision shrinks cold expert weights from 4-bit to 2-bit (50% smaller). Since only 25% of experts are "hot" (frequently activated), this reduces the model's memory footprint by ~20%. That 20% is the difference between "barely fits" and "comfortable":
Measured on M3 Max 36GB, Qwen MoE:
0.9x RAM footprint: 43 tok/s ########............ (macOS memory pressure)
1.1x RAM footprint: 104 tok/s #################### (no pressure, full speed)
Mixed precision gives you that 20% headroom.
Result: 43 → 104 tok/s = 2.4x faster
How to use it:
# Check if you're in the danger zone
python -m mlx_flash_compress.bench_memory_pressure --tokens 50
# If you see slowdown at 0.9x-1.0x, mixed precision will help
python -m mlx_flash_compress.run --model YOUR_MODEL --mixed-precisionThe problem: DeepSeek V3 or Qwen3-235B would give you GPT-4-quality answers locally, but they're 4x larger than your RAM.
What we do: Expert weights stream from SSD with intelligent caching. The LCP (Least Critical Priority) cache keeps the hottest experts in RAM. After 20-30 tokens of warm-up, 85%+ of expert accesses are served from RAM cache instead of SSD.
Warm-up curve (measured, simulated SSD latency):
Token 0: 83.3ms (cold — all experts from SSD)
Token 8: 5.7ms (warming — 62% cache hits)
Token 24: 0.5ms (warm — 85%+ cache hits)
Token 50: 0.5ms (steady state — 95% hits)
The ISP analogy: Like how your internet starts buffering a video slowly, then plays smoothly once the cache fills. First prompt is slow, but continuing the conversation is fast because the same experts stay cached.
How to use it:
# Start the server with expert caching
python -m mlx_flash_compress.serve --port 8080 --preload
# Or with the Rust sidecar (faster memory monitoring, true SSE streaming)
cd mlx-flash-server && cargo run --release -- \
--launch-worker --preload \
--expert-dir /path/to/experts \
--cache-mb 4096 \
--port 8080The problem: You start an LLM inference and suddenly Xcode becomes sluggish, Chrome tabs crash, and Slack takes 10 seconds to open. The LLM ate all your RAM.
What we do: The memory manager monitors macOS memory pressure in real-time and automatically adjusts the cache size. When pressure rises (you open more apps), it shrinks. When pressure drops (you close apps), it grows.
Memory monitoring:
Python version: 21ms per check (subprocess calls vm_stat)
Rust sidecar: 0.1ms per check (direct Mach syscall)
Checks every 10 seconds. Auto-adjusts cache with 2GB safety margin.
How to check:
# During inference, check memory status
curl http://localhost:8080/status
# Response includes:
# {
# "memory": {"pressure": "normal", "available_gb": 8.5, "cache_budget_gb": 6.5},
# "optimization_hints": [
# {"priority": "info", "action": "expand_cache", "message": "Plenty of RAM available"}
# ]
# }
# If pressure spikes:
curl http://localhost:8080/release # triggers GPU memory pool releaseThe problem: You want the memory management and caching benefits but you use LM Studio (or Cursor, continue.dev, Aider, etc.) as your interface.
What we do: Our server speaks the OpenAI API. Point your tool at http://localhost:8080/v1 and it just works.
# Start the server
python -m mlx_flash_compress.serve --port 8080 --preload
# LM Studio: Settings → Custom Endpoint → http://localhost:8080/v1
# Cursor: Settings → Custom Model → OpenAI Compatible → http://localhost:8080/v1
# continue.dev: ~/.continue/config.json → apiBase: "http://localhost:8080/v1"
# Aider: aider --openai-api-base http://localhost:8080/v1 --model local
# Claude Code: can use as a local model endpoint for comparison testing
# Python SDK:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")
resp = client.chat.completions.create(
model="local", messages=[{"role": "user", "content": "Hello"}]
)The problem: You're building an MLX-based app and need expert weight management with caching.
What we do: The Rust cache server exposes a Unix socket API. Your Python code requests experts by (layer, expert_id), and Rust serves them from RAM cache or SSD with async prefetch.
from mlx_flash_compress.rust_bridge import RustCacheClient
# Connect to the Rust cache
client = RustCacheClient("/tmp/mlx-flash-cache.sock")
client.connect()
# Fetch experts for a layer
result = client.fetch_experts(layer=5, experts=[3, 7, 12, 45])
# Returns: {"ExpertData": {"request_id": 1, "expert_sizes": [256, 256, 256, 256]}}
# Report routing for prefetch learning
client.report_routing(layer=5, activated=[3, 7, 12, 45], token_idx=100)
# Rust prefetches layer 6's likely experts in background
# Check cache stats
# curl http://localhost:8080/cache/stats
# {"entries": 500, "hit_rate": 0.87, "bytes_used": 128000000, ...}The real-world "barely fits" demo, measured on M3 Max 36GB:
Mixtral-8x7B-Instruct-v0.1-4bit (26.5GB footprint, 97% expert weights)
1.00x footprint: 19.3 tok/s #################### (full speed)
0.90x footprint: 9.1 tok/s ######### (53% slower — memory pressure!)
With mixed prec: 19.2 tok/s #################### (full speed restored — 2.1x recovery)
Mixed precision saves 6GB by compressing cold experts from 4-bit to 2-bit. That shifts the model from 0.90x (pressure zone) to 1.17x (comfortable).
This is exactly what happens in real life: you run Mixtral on your 36GB Mac, open a few browser tabs and Slack, and suddenly inference drops from 19 to 9 tok/s. Our mixed precision eliminates that silently.
Being honest about which numbers are measured and which are projected:
| Claim | Measured? | How | Number |
|---|---|---|---|
| Memory pressure causes 2x slowdown | Yes | mx.set_memory_limit on Mixtral-8x7B | 19.3→9.1 tok/s |
| Mixed precision recovers 2.1x | Yes | Same model, footprint reduced 23% | 9.1→19.2 tok/s |
| Qwen MoE pressure cliff | Yes | mx.set_memory_limit on Qwen MoE | 104→43 tok/s (2.4x) |
| Cache warm-up: 83ms→0.5ms | Yes | LCPCache with simulated SSD latency | 41x speedup |
| Topic return is instant | Yes | Simulated routing, measured cache hits | 99.8% hit rate |
| Rust memory check is 210x faster | Yes | host_statistics64 vs subprocess | 0.1ms vs 21ms |
| 200GB model at 3-7 tok/s | Projected | Based on SSD bandwidth + cache hit rate | Not end-to-end |
| Cache hit rate >85% at steady state | Yes | Measured on Qwen MoE routing patterns | 85-95% |
| Expert interception via Rust | Partial | Unix socket protocol works, mlx-rs pending | Need Metal Toolchain |
# 1. Memory pressure analysis (THE key demo, ~2 min)
python -m mlx_flash_compress.bench_memory_pressure --tokens 50 --pressure-levels 6
# 2. Watch cache warm up in real-time (~30 sec)
python -m mlx_flash_compress.demo_warmup --topics coding writing coding math
# 3. Real model with routing capture (~1 min)
python -m mlx_flash_compress.cached_inference --tokens 80 --multi-topic
# 4. Check what models fit your hardware
python -m mlx_flash_compress.model_browser
# 5. Full pipeline with comparison
python -m mlx_flash_compress.run --model mlx-community/Qwen1.5-MoE-A2.7B-Chat-4bit --tokens 100