Ollama

Active

GitHub Go MIT

Description

Local LLM runner: open-source models callable as a single CLI binary.

Key Features

One-command run — `ollama run llama3` drops into a chat
Model registry — Llama, Mistral, Qwen, Gemma built-in
OpenAI compatible — Serves /v1/chat/completions
Multimodal — Supports vision models like LLaVA
Resource-aware — CPU, Metal, CUDA auto-scheduling

Use Cases

💡 Provide LLM inference backend for local agents.

💡 Run lightweight models in CI for unit tests.

💡 Run open-source models for privacy-sensitive scenarios.

Strengths & Limitations

✅ Strengths

• Actively maintained, recent updates
• High community interest (180.0k stars)
• Permissive open-source license (MIT)
• Established track record (3 years in production)

Quick Start

# Install
brew install ollama
# Start the service
ollama serve &
# Pull a model and chat
ollama pull llama3
ollama run llama3 'Describe Rust in one sentence'

Visit GitHub

Related Projects

Llamafile

25.9k · C++

Active A

Mozilla's approach to packaging LLMs as a single executable with zero dependencies.

llmsingle-filelocal +1

· Single file — Model weights plus runtime packed into one exe
· Zero deps — No Python, CUDA, or pip needed
· Cross-platform — Linux/macOS/Windows with the same experience

View full details → View on GitHub

lemmy

1.6k · TypeScript

Stale D

Lemmy is a lightweight TypeScript library that wraps LLM tool calls into a simple, consistent workflow interface, ideal for rapidly building multi-step agent tasks.

agent-toolstypescripttool-use +1

· Lightweight LLM tool wrapper in TypeScript that unifies tool calling across models
· Multi-step workflows that simplify agent task orchestration
· Easy to embed with zero dependencies, importable as a library

View on GitHub

exo

47.2k · Python

Active A+

exo is a framework that pools many consumer-grade devices into a distributed LLM inference cluster, turning Macs, PCs and Linux boxes into one GPU pool.

distributedllmcluster +2

· Consumer-device cluster — every node is a desktop or laptop, no datacenter GPUs needed
· P2P auto-discovery — devices on the same LAN join the cluster automatically
· Dynamic partitioning — models are split by layers / tensors across nodes

View full details → View on GitHub

any-llm

2.2k · Python

Active A

Mozilla's unified LLM interface to access multiple providers without switching SDKs.

llmunified-apiprovider +1

· Unified API — Write once, run across providers
· Full coverage — OpenAI, Anthropic, Mistral, Ollama, vLLM, etc.
· Observable — Call trace and token stats

View on GitHub

small-language-modelsedge-inferencefine-tuning

Agent Small-Model Finetuning and Edge Inference

Exploring how small language models are fine-tuned and deployed for agent workloads at the edge, balancing latency, cost, and accuracy for production AI agents.