litellm

Thu, 26 Mar 2026 16:06:17 +0800

BerriAI/litellm

🚅 LiteLLM

Call 100+ LLMs in OpenAI format. [Bedrock, Azure, OpenAI, VertexAI, Anthropic, Groq, etc.]

LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier

Use LiteLLM for

LLMs - Call 100+ LLMs (Python SDK + AI Gateway)

All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.

Python SDK

`1`	`pip install litellm`

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])

# Anthropic  
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])

AI Gateway (Proxy Server)

Getting Started - E2E Tutorial - Setup virtual keys, make your first request

1
2

pip install 'litellm[proxy]'
litellm --model gpt-4o

import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

Docs: LLM Providers

Agents - Invoke A2A Agents (Python SDK + AI Gateway)

Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI

Python SDK - A2A Protocol

from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4

client = A2AClient(base_url="http://localhost:10001")

request = SendMessageRequest(
    id=str(uuid4()),
    params=MessageSendParams(
        message={
            "role": "user",
            "parts": [{"kind": "text", "text": "Hello!"}],
            "messageId": uuid4().hex,
        }
    )
)
response = await client.send_message(request)

AI Gateway (Proxy Server)

Step 1. Add your Agent to the AI Gateway

Step 2. Call Agent via A2A SDK

from a2a.client import A2ACardResolver, A2AClient
from a2a.types import MessageSendParams, SendMessageRequest
from uuid import uuid4
import httpx

base_url = "http://localhost:4000/a2a/my-agent"  # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer sk-1234"}    # LiteLLM Virtual Key

async with httpx.AsyncClient(headers=headers) as httpx_client:
    resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url)
    agent_card = await resolver.get_agent_card()
    client = A2AClient(httpx_client=httpx_client, agent_card=agent_card)

    request = SendMessageRequest(
        id=str(uuid4()),
        params=MessageSendParams(
            message={
                "role": "user",
                "parts": [{"kind": "text", "text": "Hello!"}],
                "messageId": uuid4().hex,
            }
        )
    )
    response = await client.send_message(request)

Docs: A2A Agent Gateway

MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway)

Python SDK - MCP Bridge

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm

server_params = StdioServerParameters(command="python", args=["mcp_server.py"])

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()

        # Load MCP tools in OpenAI format
        tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")

        # Use with any LiteLLM model
        response = await litellm.acompletion(
            model="gpt-4o",
            messages=[{"role": "user", "content": "What's 3 + 5?"}],
            tools=tools
        )

AI Gateway - MCP Gateway

Step 1. Add your MCP Server to the AI Gateway

Step 2. Call MCP tools via /chat/completions

curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize the latest open PR"}],
    "tools": [{
      "type": "mcp",
      "server_url": "litellm_proxy/mcp/github",
      "server_label": "github_mcp",
      "require_approval": "never"
    }]
  }'

Use with Cursor IDE

{
  "mcpServers": {
    "LiteLLM": {
      "url": "http://localhost:4000/mcp/",
      "headers": {
        "x-litellm-api-key": "Bearer sk-1234"
      }
    }
  }
}

Docs: MCP Gateway

How to use LiteLLM

You can use LiteLLM through either the Proxy Server or Python SDK. Both gives you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:

	LiteLLM AI Gateway	LiteLLM Python SDK
Use Case	Central service (LLM Gateway) to access multiple LLMs	Use LiteLLM directly in your Python code
Who Uses It?	Gen AI Enablement / ML Platform Teams	Developers building LLM projects
Key Features	Centralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and management	Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.)

LiteLLM Performance: 8ms P95 latency at 1k RPS (See benchmarks here)

Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers

Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published. More information about the release cycle here

Support for more providers. Missing a provider or LLM Platform, raise a feature request.

OSS Adopters

Netflix

Supported Providers (Website Supported Models | Docs)

Provider	`/chat/completions`	`/messages`	`/responses`	`/embeddings`	`/image/generations`	`/audio/transcriptions`	`/audio/speech`	`/moderations`	`/batches`	`/rerank`
Abliteration (`abliteration`)	✅
AI/ML API (`aiml`)	✅	✅	✅	✅	✅
AI21 (`ai21`)	✅	✅	✅
AI21 Chat (`ai21_chat`)	✅	✅	✅
Aleph Alpha	✅	✅	✅
Amazon Nova	✅	✅	✅
Anthropic (`anthropic`)	✅	✅	✅						✅
Anthropic Text (`anthropic_text`)	✅	✅	✅						✅
Anyscale	✅	✅	✅
AssemblyAI (`assemblyai`)	✅	✅	✅			✅
Auto Router (`auto_router`)	✅	✅	✅
AWS - Bedrock (`bedrock`)	✅	✅	✅	✅						✅
AWS - Sagemaker (`sagemaker`)	✅	✅	✅	✅
Azure (`azure`)	✅	✅	✅	✅	✅	✅	✅	✅	✅
Azure AI (`azure_ai`)	✅	✅	✅	✅	✅	✅	✅	✅	✅
Azure Text (`azure_text`)	✅	✅	✅			✅	✅	✅	✅
Baseten (`baseten`)	✅	✅	✅
Bytez (`bytez`)	✅	✅	✅
Cerebras (`cerebras`)	✅	✅	✅
Clarifai (`clarifai`)	✅	✅	✅
Cloudflare AI Workers (`cloudflare`)	✅	✅	✅
Codestral (`codestral`)	✅	✅	✅
Cohere (`cohere`)	✅	✅	✅	✅						✅
Cohere Chat (`cohere_chat`)	✅	✅	✅
CometAPI (`cometapi`)	✅	✅	✅	✅
CompactifAI (`compactifai`)	✅	✅	✅
Custom (`custom`)	✅	✅	✅
Custom OpenAI (`custom_openai`)	✅	✅	✅			✅	✅	✅	✅
Dashscope (`dashscope`)	✅	✅	✅
Databricks (`databricks`)	✅	✅	✅
DataRobot (`datarobot`)	✅	✅	✅
Deepgram (`deepgram`)	✅	✅	✅			✅
DeepInfra (`deepinfra`)	✅	✅	✅
Deepseek (`deepseek`)	✅	✅	✅
ElevenLabs (`elevenlabs`)	✅	✅	✅			✅	✅
Empower (`empower`)	✅	✅	✅
Fal AI (`fal_ai`)	✅	✅	✅		✅
Featherless AI (`featherless_ai`)	✅	✅	✅
Fireworks AI (`fireworks_ai`)	✅	✅	✅
FriendliAI (`friendliai`)	✅	✅	✅
Galadriel (`galadriel`)	✅	✅	✅
GitHub Copilot (`github_copilot`)	✅	✅	✅	✅
GitHub Models (`github`)	✅	✅	✅
Google - PaLM	✅	✅	✅
Google - Vertex AI (`vertex_ai`)	✅	✅	✅	✅	✅
Google AI Studio - Gemini (`gemini`)	✅	✅	✅
GradientAI (`gradient_ai`)	✅	✅	✅
Groq AI (`groq`)	✅	✅	✅
Heroku (`heroku`)	✅	✅	✅
Hosted VLLM (`hosted_vllm`)	✅	✅	✅
Huggingface (`huggingface`)	✅	✅	✅	✅						✅
Hyperbolic (`hyperbolic`)	✅	✅	✅
IBM - Watsonx.ai (`watsonx`)	✅	✅	✅	✅
Infinity (`infinity`)				✅
Jina AI (`jina_ai`)				✅
Lambda AI (`lambda_ai`)	✅	✅	✅
Lemonade (`lemonade`)	✅	✅	✅
LiteLLM Proxy (`litellm_proxy`)	✅	✅	✅	✅	✅
Llamafile (`llamafile`)	✅	✅	✅
LM Studio (`lm_studio`)	✅	✅	✅
Maritalk (`maritalk`)	✅	✅	✅
Meta - Llama API (`meta_llama`)	✅	✅	✅
Mistral AI API (`mistral`)	✅	✅	✅	✅
Moonshot (`moonshot`)	✅	✅	✅
Morph (`morph`)	✅	✅	✅
Nebius AI Studio (`nebius`)	✅	✅	✅	✅
NLP Cloud (`nlp_cloud`)	✅	✅	✅
Novita AI (`novita`)	✅	✅	✅
Nscale (`nscale`)	✅	✅	✅
Nvidia NIM (`nvidia_nim`)	✅	✅	✅
OCI (`oci`)	✅	✅	✅
Ollama (`ollama`)	✅	✅	✅	✅
Ollama Chat (`ollama_chat`)	✅	✅	✅
Oobabooga (`oobabooga`)	✅	✅	✅			✅	✅	✅	✅
OpenAI (`openai`)	✅	✅	✅	✅	✅	✅	✅	✅	✅
OpenAI-like (`openai_like`)				✅
OpenRouter (`openrouter`)	✅	✅	✅
OVHCloud AI Endpoints (`ovhcloud`)	✅	✅	✅
Perplexity AI (`perplexity`)	✅	✅	✅
Petals (`petals`)	✅	✅	✅
Predibase (`predibase`)	✅	✅	✅
Recraft (`recraft`)					✅
Replicate (`replicate`)	✅	✅	✅
Sagemaker Chat (`sagemaker_chat`)	✅	✅	✅
Sambanova (`sambanova`)	✅	✅	✅
Snowflake (`snowflake`)	✅	✅	✅
Text Completion Codestral (`text-completion-codestral`)	✅	✅	✅
Text Completion OpenAI (`text-completion-openai`)	✅	✅	✅			✅	✅	✅	✅
Together AI (`together_ai`)	✅	✅	✅
Topaz (`topaz`)	✅	✅	✅
Triton (`triton`)	✅	✅	✅
V0 (`v0`)	✅	✅	✅
Vercel AI Gateway (`vercel_ai_gateway`)	✅	✅	✅
VLLM (`vllm`)	✅	✅	✅
Volcengine (`volcengine`)	✅	✅	✅
Voyage AI (`voyage`)				✅
WandB Inference (`wandb`)	✅	✅	✅
Watsonx Text (`watsonx_text`)	✅	✅	✅
xAI (`xai`)	✅	✅	✅
Xinference (`xinference`)				✅

Read the Docs

Run in Developer mode

Services

Setup .env file in root
Run dependant services docker-compose up db prometheus

Backend

(In root) create virtual environment python -m venv .venv
Activate virtual environment source .venv/bin/activate
Install dependencies pip install -e ".[all]"
pip install prisma
prisma generate
Start proxy backend python litellm/proxy/proxy_cli.py

Frontend

Navigate to ui/litellm-dashboard
Install dependencies npm install
Run npm run dev to start the dashboard

Enterprise

For companies that need better security, user management and professional support

Talk to founders

This covers:

✅ Features under the LiteLLM Commercial License:
✅ Feature Prioritization
✅ Custom Integrations
✅ Professional Support - Dedicated discord + slack
✅ Custom SLAs
✅ Secure access with Single Sign-On

Contributing

We welcome contributions to LiteLLM! Whether you’re fixing bugs, adding features, or improving documentation, we appreciate your help.

Quick Start for Contributors

This requires poetry to be installed.

git clone https://github.com/BerriAI/litellm.git
cd litellm
make install-dev    # Install development dependencies
make format         # Format your code
make lint           # Run all linting checks
make test-unit      # Run unit tests
make format-check   # Check formatting only

For detailed contributing guidelines, see CONTRIBUTING.md.

Code Quality / Linting

LiteLLM follows the Google Python Style Guide.

Our automated checks include:

Black for code formatting
Ruff for linting and code quality
MyPy for type checking
Circular import detection
Import safety checks

All these checks must pass before your PR can be merged.

Support / talk with founders

Schedule Demo 👋
Community Discord 💭
Community Slack 💭
Our emails ✉️ ishaan@berri.ai / krrish@berri.ai

Why did we build this

Need for simplicity: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI and Cohere.

Contributors

aci

Tue, 06 May 2025 15:30:01 +0800

aipotheosis-labs/aci

ACI: Open-Source Infra to Power Unified MCP Servers

[!NOTE] This repo is for the ACI.dev platform. If you’re looking for the Unified MCP server built with ACI.dev, see aci-mcp.

ACI.dev is the open-source infrastructure layer for AI-agent tool-use. It gives your agents intent-aware access to 600+ tools with multi-tenant auth, granular permissions, and dynamic tool discovery—exposed as either direct function calls or through a Unified Model-Context-Protocol (MCP) server.

Example: Instead of writing separate OAuth flows and API clients for Google Calendar, Slack, and more, use ACI.dev to manage authentication and provide your AI agents with unified, secure function calls. Access these capabilities through our Unified MCP server or via our lightweight Python SDK, compatible with any LLM framework.

Build production-ready AI agents without the infrastructure headaches.

Join us on Discord to help shape the future of Open Source AI Infrastructure.

🌟 Star ACI.dev to stay updated on new releases!

📺 Demo Video

ACI.dev Unified MCP Server Demo

✨ Key Features

600+ Pre-built Integrations: Connect to popular services and apps in minutes.
Flexible Access Methods: Use our unified MCP server or our lightweight SDK for direct function calling.
Multi-tenant Authentication: Built-in OAuth flows and secrets management for both developers and end-users.
Enhanced Agent Reliability: Natural language permission boundaries and dynamic tool discovery.
Framework & Model Agnostic: Works with any LLM framework and agent architecture.
100% Open Source: Everything released under Apache 2.0 (backend, dev portal, integrations).

💡 Why Use ACI.dev?

ACI.dev solves your critical infrastructure challenges for production-ready AI agents:

Authentication at Scale: Connect multiple users to multiple services securely.
Discovery Without Overload: Find and use the right tools without overwhelming LLM context windows.
Natural Language Permissions: Control agent capabilities with human-readable boundaries.
Build Once, Run Anywhere: No vendor lock-in with our open source, framework-agnostic approach.

🧰 Common Use Cases

Personal Assistant Chatbots: Build chatbots that can search the web, manage calendars, send emails, interact with SaaS tools, etc.
Research Agent: Conducts research on specific topics and syncs results to other apps (e.g., Notion, Google Sheets).
Outbound Sales Agent: Automates lead generation, email outreach, and CRM updates.
Customer Support Agent: Provides answers, manages tickets, and performs actions based on customer queries.

🔗 Quick Links

Managed Service: aci.dev
Documentation: aci.dev/docs
Available Tools List: aci.dev/tools
Python SDK: github.com/aipotheosis-labs/aci-python-sdk
Unified MCP Server: github.com/aipotheosis-labs/aci-mcp
Agent Examples Built with ACI.dev: github.com/aipotheosis-labs/aci-agents
Blog: aci.dev/blog
Community: Discord | Twitter/X | LinkedIn

💻 Getting Started: Local Development

To run the full ACI.dev platform (backend server and frontend portal) locally, follow the individual README files for each component:

Backend: backend/README.md
Frontend: frontend/README.md

👋 Contributing

We welcome contributions! Please see our CONTRIBUTING.md for more information.

Integration Requests

Missing any integrations (apps or functions) you need? Please see our Integration Request Template and submit an integration request! Or, if you’re feeling adventurous, you can submit a PR to add the integration yourself!

Python-SDK on Producthunt daily

litellm

BerriAI/litellm

🚅 LiteLLM

LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier

Use LiteLLM for

Python SDK

AI Gateway (Proxy Server)

Python SDK - A2A Protocol

AI Gateway (Proxy Server)

Python SDK - MCP Bridge

AI Gateway - MCP Gateway

Use with Cursor IDE

How to use LiteLLM

OSS Adopters

Netflix

Supported Providers (Website Supported Models | Docs)

Run in Developer mode

Services

Backend

Frontend

Enterprise

Contributing

Quick Start for Contributors

Code Quality / Linting

Support / talk with founders

Why did we build this

Contributors

aci

aipotheosis-labs/aci

ACI: Open-Source Infra to Power Unified MCP Servers

📺 Demo Video

✨ Key Features

💡 Why Use ACI.dev?

🧰 Common Use Cases

🔗 Quick Links

💻 Getting Started: Local Development

👋 Contributing

Integration Requests

Star History