KTransformers
ActiveDescription
A flexible framework for experiencing heterogeneous LLM inference and fine-tuning optimizations — run large language models efficiently on consumer hardware with kernel-level optimizations.
Key Features
- CPU-GPU heterogeneous computing for efficient LLM inference and fine-tuning
- AMX/AVX optimized kernels for INT4/INT8 quantized inference on Intel CPUs
- MoE optimization with NUMA-aware memory management for Mixture-of-Experts models
- LLaMA-Factory integration for ultra-large MoE model fine-tuning (DeepSeek-V3/R1)
- FP8 GPU kernel support achieving 3-28x speedup on single GPU with 24GB VRAM
- Multi-backend support: Intel AMX, NVIDIA CUDA, AMD ROCm, and Ascend NPU
Use Cases
💡 Running DeepSeek-R1/V3 on consumer hardware with 24GB VRAM
💡 Fine-tuning large MoE models with limited GPU memory using quantization
💡 Integrating with SGLang for production LLM serving at scale
💡 Deploying AI agents on edge devices with CPU-GPU hybrid inference
💡 Research on heterogeneous computing optimizations for LLM workloads
Strengths & Limitations
✅ Strengths
- • Actively maintained, recent updates
- • High community interest (19.5k stars)
- • Permissive open-source license (Apache-2.0)
- • Established track record (2 years in production)
Categories
Quick Start
cd kt-kernel
pip install .
# For SFT with LLaMA-Factory:
# cd /path/to/LLaMA-Factory
# pip install -e . && pip install -r requirements/ktransformers.txt