Skip to content

Latest commit

 

History

History
201 lines (144 loc) · 8.74 KB

File metadata and controls

201 lines (144 loc) · 8.74 KB

Real-World Usage: How This Actually Helps

No hype. Just measured numbers from a real M3 Max 36GB Mac.

Who Benefits (and Who Doesn't)

Your Situation Does This Help? How Much?
Model fits easily (<70% RAM) No. Pure MLX is already at full speed. 0% gain
Model barely fits (80-100% RAM) Yes. Memory pressure kills 40-60% of your speed silently. 2.4x recovery (measured)
Model doesn't fit (>100% RAM) Yes. Makes impossible possible. Slow but works (3-7 tok/s)
Running other apps while using LLM Yes. Memory manager prevents the LLM from stealing your RAM. No more beach balls

Scenario 1: "My 30GB Model Stutters on My 36GB Mac"

The problem: You downloaded Mixtral-8x22B (26GB) on your 36GB Mac. It loaded fine, but generation is weirdly slow — 40 tok/s instead of the expected 100+. Activity Monitor shows "Memory Pressure" in yellow.

What's happening: macOS is compressing and swapping memory pages. Every time MLX needs a tensor, macOS has to decompress it first. This adds 0.5-2ms per tensor access, and with 32 layers x hundreds of tensors per token, it accumulates into a 60% slowdown.

What we do: Mixed precision shrinks cold expert weights from 4-bit to 2-bit (50% smaller). Since only 25% of experts are "hot" (frequently activated), this reduces the model's memory footprint by ~20%. That 20% is the difference between "barely fits" and "comfortable":

Measured on M3 Max 36GB, Qwen MoE:

  0.9x RAM footprint:   43 tok/s   ########............  (macOS memory pressure)
  1.1x RAM footprint:  104 tok/s   ####################  (no pressure, full speed)

  Mixed precision gives you that 20% headroom.
  Result: 43 → 104 tok/s = 2.4x faster

How to use it:

# Check if you're in the danger zone
python -m mlx_flash_compress.bench_memory_pressure --tokens 50

# If you see slowdown at 0.9x-1.0x, mixed precision will help
python -m mlx_flash_compress.run --model YOUR_MODEL --mixed-precision

Scenario 2: "I Want to Run a 200GB Model on My 48GB Mac"

The problem: DeepSeek V3 or Qwen3-235B would give you GPT-4-quality answers locally, but they're 4x larger than your RAM.

What we do: Expert weights stream from SSD with intelligent caching. The LCP (Least Critical Priority) cache keeps the hottest experts in RAM. After 20-30 tokens of warm-up, 85%+ of expert accesses are served from RAM cache instead of SSD.

Warm-up curve (measured, simulated SSD latency):

  Token  0:  83.3ms  (cold — all experts from SSD)
  Token  8:   5.7ms  (warming — 62% cache hits)
  Token 24:   0.5ms  (warm — 85%+ cache hits)
  Token 50:   0.5ms  (steady state — 95% hits)

The ISP analogy: Like how your internet starts buffering a video slowly, then plays smoothly once the cache fills. First prompt is slow, but continuing the conversation is fast because the same experts stay cached.

How to use it:

# Start the server with expert caching
python -m mlx_flash_compress.serve --port 8080 --preload

# Or with the Rust sidecar (faster memory monitoring, true SSE streaming)
cd mlx-flash-server && cargo run --release -- \
  --launch-worker --preload \
  --expert-dir /path/to/experts \
  --cache-mb 4096 \
  --port 8080

Scenario 3: "I Want My LLM to Not Kill My Other Apps"

The problem: You start an LLM inference and suddenly Xcode becomes sluggish, Chrome tabs crash, and Slack takes 10 seconds to open. The LLM ate all your RAM.

What we do: The memory manager monitors macOS memory pressure in real-time and automatically adjusts the cache size. When pressure rises (you open more apps), it shrinks. When pressure drops (you close apps), it grows.

Memory monitoring:
  Python version:  21ms per check  (subprocess calls vm_stat)
  Rust sidecar:   0.1ms per check  (direct Mach syscall)

  Checks every 10 seconds. Auto-adjusts cache with 2GB safety margin.

How to check:

# During inference, check memory status
curl http://localhost:8080/status

# Response includes:
# {
#   "memory": {"pressure": "normal", "available_gb": 8.5, "cache_budget_gb": 6.5},
#   "optimization_hints": [
#     {"priority": "info", "action": "expand_cache", "message": "Plenty of RAM available"}
#   ]
# }

# If pressure spikes:
curl http://localhost:8080/release  # triggers GPU memory pool release

Scenario 4: "I Use LM Studio / Cursor / Claude Code"

The problem: You want the memory management and caching benefits but you use LM Studio (or Cursor, continue.dev, Aider, etc.) as your interface.

What we do: Our server speaks the OpenAI API. Point your tool at http://localhost:8080/v1 and it just works.

# Start the server
python -m mlx_flash_compress.serve --port 8080 --preload

# LM Studio: Settings → Custom Endpoint → http://localhost:8080/v1
# Cursor: Settings → Custom Model → OpenAI Compatible → http://localhost:8080/v1
# continue.dev: ~/.continue/config.json → apiBase: "http://localhost:8080/v1"
# Aider: aider --openai-api-base http://localhost:8080/v1 --model local
# Claude Code: can use as a local model endpoint for comparison testing

# Python SDK:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")
resp = client.chat.completions.create(
    model="local", messages=[{"role": "user", "content": "Hello"}]
)

Scenario 5: "I'm a Developer Building on MLX"

The problem: You're building an MLX-based app and need expert weight management with caching.

What we do: The Rust cache server exposes a Unix socket API. Your Python code requests experts by (layer, expert_id), and Rust serves them from RAM cache or SSD with async prefetch.

from mlx_flash_compress.rust_bridge import RustCacheClient

# Connect to the Rust cache
client = RustCacheClient("/tmp/mlx-flash-cache.sock")
client.connect()

# Fetch experts for a layer
result = client.fetch_experts(layer=5, experts=[3, 7, 12, 45])
# Returns: {"ExpertData": {"request_id": 1, "expert_sizes": [256, 256, 256, 256]}}

# Report routing for prefetch learning
client.report_routing(layer=5, activated=[3, 7, 12, 45], token_idx=100)
# Rust prefetches layer 6's likely experts in background

# Check cache stats
# curl http://localhost:8080/cache/stats
# {"entries": 500, "hit_rate": 0.87, "bytes_used": 128000000, ...}

Mixtral-8x7B Results (26GB on 36GB Mac)

The real-world "barely fits" demo, measured on M3 Max 36GB:

Mixtral-8x7B-Instruct-v0.1-4bit (26.5GB footprint, 97% expert weights)

  1.00x footprint:   19.3 tok/s  ####################  (full speed)
  0.90x footprint:    9.1 tok/s  #########             (53% slower — memory pressure!)
  With mixed prec:   19.2 tok/s  ####################  (full speed restored — 2.1x recovery)

Mixed precision saves 6GB by compressing cold experts from 4-bit to 2-bit. That shifts the model from 0.90x (pressure zone) to 1.17x (comfortable).

This is exactly what happens in real life: you run Mixtral on your 36GB Mac, open a few browser tabs and Slack, and suddenly inference drops from 19 to 9 tok/s. Our mixed precision eliminates that silently.

What We Measured vs What We Projected

Being honest about which numbers are measured and which are projected:

Claim Measured? How Number
Memory pressure causes 2x slowdown Yes mx.set_memory_limit on Mixtral-8x7B 19.3→9.1 tok/s
Mixed precision recovers 2.1x Yes Same model, footprint reduced 23% 9.1→19.2 tok/s
Qwen MoE pressure cliff Yes mx.set_memory_limit on Qwen MoE 104→43 tok/s (2.4x)
Cache warm-up: 83ms→0.5ms Yes LCPCache with simulated SSD latency 41x speedup
Topic return is instant Yes Simulated routing, measured cache hits 99.8% hit rate
Rust memory check is 210x faster Yes host_statistics64 vs subprocess 0.1ms vs 21ms
200GB model at 3-7 tok/s Projected Based on SSD bandwidth + cache hit rate Not end-to-end
Cache hit rate >85% at steady state Yes Measured on Qwen MoE routing patterns 85-95%
Expert interception via Rust Partial Unix socket protocol works, mlx-rs pending Need Metal Toolchain

Running the Benchmarks Yourself

# 1. Memory pressure analysis (THE key demo, ~2 min)
python -m mlx_flash_compress.bench_memory_pressure --tokens 50 --pressure-levels 6

# 2. Watch cache warm up in real-time (~30 sec)
python -m mlx_flash_compress.demo_warmup --topics coding writing coding math

# 3. Real model with routing capture (~1 min)
python -m mlx_flash_compress.cached_inference --tokens 80 --multi-topic

# 4. Check what models fit your hardware
python -m mlx_flash_compress.model_browser

# 5. Full pipeline with comparison
python -m mlx_flash_compress.run --model mlx-community/Qwen1.5-MoE-A2.7B-Chat-4bit --tokens 100