gguf-runner

Here are 3 public repositories matching this topic...

Local inference platform for K/IQ-quant GGUF models on Apple Silicon

metal inference voice-chat lora quantization mtp mlx voice-assistant inference-engine voice-ai apple-silicon openai-api llm llm-inference gguf speculative-decoding anthropic-api gguf-model-support gguf-runner

a GGUF inference runner in Rust

rust rust-lang llm llm-inference gguf gguf-runner

GGUF-Runner - Want to run LLMs locally, use this guide, and run with LLAMA.cpp

local llama gguf local-llms gguf-runner

Add a description, image, and links to the gguf-runner topic page so that developers can more easily learn about it.

To associate your repository with the gguf-runner topic, visit your repo's landing page and select "manage topics."