mistral.rs
ActiveDescription
Fast, flexible LLM inference engine built in Rust — supports multiple model architectures and quantization schemes for high-performance local LLM deployment.
Key Features
- Zero-Config Model Loading — auto-detect architecture, quantization format, and chat template from any Hugging Face model
- True Multimodality — text, vision, video, audio input, speech generation, image generation, and embeddings in one engine
- Smart Quantization — UQFF prebuilt quantizations and ISQ with automatic best-format selection at each level
- OpenAI + Anthropic Compatible Serving — single process exposes both /v1 and /v1/messages endpoints
- Built-in Web UI — served at /ui with reasoning, code execution, plots, and inline file display; edit messages to branch conversations
- Agentic Runtime — web search, local Python code execution with model feedback, session management, and custom tool hooks
Use Cases
💡 Run local LLM inference with high performance on CUDA, Metal, and CPU
💡 Deploy OpenAI/Anthropic-compatible API servers for existing applications
💡 Build agentic workflows with web search and code execution capabilities
💡 Quantize and optimize models for specific hardware with mistralrs tune
💡 Generate images, speech, and embeddings through a unified inference engine
Strengths & Limitations
✅ Strengths
- • Actively maintained, recent updates
- • High community interest (7.6k stars)
- • Permissive open-source license (MIT)
- • Established track record (2 years in production)
Categories
Quick Start
Install via curl: curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.sh | sh. Then run: mistralrs run -m Qwen/Qwen3-4B for interactive chat, or mistralrs serve -m google/gemma-4-E4B-it to start an API server with the built-in web UI at localhost:1234/ui.