We don't make things faster. We make impossible things possible, and tight things comfortable.
Your 36GB Mac runs a 30GB model. macOS memory pressure kicks in — compression, swapping, GPU stalls. You lose 25-50% of your speed without knowing why.
We shrink the model's memory footprint by 25% using mixed precision (hot experts at 4-bit, cold at 2-bit). The model now effectively uses 22GB instead of 30GB. Memory pressure disappears. Full speed restored.
35GB model on 36GB Mac (97% RAM used):
Without us: 8.3 tok/s ████████ (OS thrashing, -40%)
With us: 13.9 tok/s ██████████████ (+67% — pressure eliminated)
A 209GB model on a 36GB Mac. Without us: OOM crash, doesn't run at all.
With us: runs at 3-7 tok/s by streaming from SSD with intelligent caching. That's ~1 word per 0.3 seconds — slow, but it works. Good enough for:
- Code review (ask a question, go make coffee, get a GPT-4-quality answer)
- Document analysis (submit a doc, wait 30 seconds for a thorough analysis)
- Research (quality matters more than speed)
209GB model on 36GB Mac:
Without us: CRASH (doesn't run)
With us: 3.7 tok/s ███████████ (slow but working)
We add zero value here. Pure MLX runs at full speed. We don't pretend otherwise.
Nobody else combines all of these on Apple Silicon:
- Task-aware caching (different tasks use different 30% of the model)
- Adaptive memory management (never harms your other apps)
- Mixed precision per-expert (hot=4bit, cold=2bit, decided at runtime)
- C GCD acceleration (Apple's native dispatch, 8x faster than Python)
- Live topic change detection (re-caches in <2 seconds)
Similar projects exist (mlx-moe, PowerInfer, flash-moe) but each solves only part of the problem. We integrate the full stack.
| What | Number | How Measured |
|---|---|---|
| MLX baseline | 115.9 tok/s | Qwen MoE, 3 runs averaged |
| Cache hit rate | 85.4% | 24 layers, 60 experts, LCP eviction |
| Cache overhead | 3.9ms/token | Fits inside 9.1ms GPU time = FREE |
| Mixed precision | 1.80x smaller | MSE 0.000059, negligible quality loss |
| Topic switch | 92% cache swap | Adaptive profiler detects in <2s |
| Memory-safe | Auto-adjusts | Monitors macOS pressure, never harms user |
| Tests | 59 passing | Covers all modules |
Running a 200GB AI model on a 36GB MacBook sounds impossible. It is — without smart caching.
We built MLX-Flash. It streams model weights from SSD with 85% cache hit rate. Result: model runs at 3-4 tok/s instead of crashing.
That's slow. But "slow" beats "doesn't run at all."
The surprising win: models that BARELY fit (90-100% of RAM). macOS memory pressure causes 25-50% speed loss. Our mixed precision shrinks the footprint 25%, eliminating pressure entirely. Full speed restored.
github.com/szibis/MLX-Flash
Running models that "almost fit" your hardware — the overlooked problem
Everyone talks about running huge models on small devices. But there's a quieter problem: models that technically fit in RAM but perform terribly because macOS memory pressure causes GPU stalls.
We built MLX-Flash to solve three problems:
- Barely fits: 35GB model on 36GB Mac → +67% speed (pressure eliminated)
- Doesn't fit: 209GB model on 36GB Mac → 3.7 tok/s (impossible → possible)
- Comfort zone: adaptive memory management that never harms your other apps
Built on 60+ research papers, measured on real hardware, 59 tests passing.
Open source: github.com/szibis/MLX-Flash