HomeProjectsArticlesPK BoardAbout
Explore Projects

Portkey AI Gateway

Stale
GitHub TypeScript MIT

Description

Portkey AI Gateway is a blazing fast AI gateway with integrated guardrails, routing to 200+ LLMs with 50+ AI guardrails through a single fast and friendly API.

Key Features

  • Route to 250+ LLMs with a single OpenAI-compatible API endpoint
  • Blazing fast at under 1ms added latency with a 122kb footprint
  • Built-in automatic retries, fallbacks, and load balancing for reliability
  • 50+ AI guardrails for content filtering and safety enforcement
  • MCP Gateway for enterprise auth and observability on MCP servers
  • Deploy anywhere: Cloudflare Workers, Docker, Node.js, or Portkey Cloud

Use Cases

💡 Centralized LLM management with provider failover for production AI apps
💡 Adding guardrails and content moderation to any LLM integration
💡 Cost optimization by routing across providers based on price and performance
💡 Enterprise AI deployments requiring audit logging and compliance
💡 Multi-tenant AI platforms needing per-user rate limiting and quotas

Strengths & Limitations

✅ Strengths

  • • High community interest (12.9k stars)
  • • Permissive open-source license (MIT)
  • • Established track record (3 years in production)

Tags

gateway llm-routing guardrails ai-safety typescript

Categories

⚡ Agent Tools 🛡️ Security & Guardrails

Quick Start

npx @portkey-ai/gateway — gateway runs at http://localhost:8787/v1. Then use the Portkey Python client: client = Portkey(provider='openai', Authorization='sk-***'); client.chat.completions.create(model='gpt-4o-mini', messages=[...])
Visit GitHub Visit Website

Related Projects

UQLM

1.2k · Python
Active A+

CVS Health's open-source uncertainty quantification library for language models, providing UQ-based hallucination detection with confidence scoring and mitigation tools to identify and reduce unreliable LLM outputs.

hallucination-detectionuncertainty-quantificationllm-evaluation +2
  • · Five categories of uncertainty quantification scorers: Black-Box (consistency), White-Box (token probability), LLM-as-a-Judge, Ensemble, and Long-Text
  • · Black-Box scorers measure response consistency through multiple generations and comparisons, compatible with any LLM, off-the-shelf
  • · White-Box scorers leverage token probabilities for uncertainty estimation, single-generation scoring with minimal latency and cost
View on GitHub

Guardrails AI

7.3k · Python
Active A+

Guardrails AI adds programmable guardrails to large language models, ensuring reliability and safety through input/output validation, structured data extraction, and custom validators.

guardrailsllm-safetyvalidation +2
  • · Input/Output Guards — detect, quantify, and mitigate specific risk types in LLM applications
  • · Guardrails Hub pre-built validators — RegexMatch, CompetitorCheck, ToxicLanguage and more out of the box
  • · Pydantic structured output — enforce LLM output format via function calling or prompt optimization
View full details → View on GitHub

Related Articles

LiteLLM成本治理LLM Gateway

LiteLLM in Practice: From Cost Governance to Team API Key Engineering

A production walkthrough of LiteLLM based on official docs and its 51k-star repo: Virtual Keys, Guardrails, routing, and Postgres-backed cost governance.

Read more →
llm-gatewaymodel-routingcost-optimization

LLM Routing and Multi-Model Gateways in Practice: A Production-Grade Multi-Model Architecture

Four LLM gateways compared, with production patterns for fallback, smart routing, cost observability, and scheduling.

Read more →
Model GatewayLiteLLMFallback

Model Gateway and Routing: Production-Grade LLM Fallback Chains

A systematic deep dive into model gateway core capabilities, LiteLLM self-hosted configuration, fallback chain design, task-aware routing, cost-aware routing, A/B testing routing, rate limiting and quotas, semantic caching, Prometheus monitoring, and production-grade deployment.

Read more →