Skip to content

feat: add LiteLLM as AI gateway provider#323

Open
RheagalFire wants to merge 1 commit into
MagnivOrg:masterfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as AI gateway provider#323
RheagalFire wants to merge 1 commit into
MagnivOrg:masterfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire
Copy link
Copy Markdown

Summary

  • Adds LiteLLM as a tracked provider alongside OpenAI and Anthropic
  • Users can access 100+ LLM providers through PromptLayer's observability layer via pl.litellm

Motivation

PromptLayer currently wraps OpenAI and Anthropic modules for prompt tracking. Users wanting to track calls to other providers (Google Gemini, AWS Bedrock, Cohere, Mistral, etc.) have no path through PromptLayer's tracking layer. LiteLLM provides a unified interface that follows the OpenAI response format, making it a natural fit for the existing PromptLayerBase wrapper pattern.

Changes

  • promptlayer/promptlayer.py - Added litellm case to __getattr__ in both PromptLayer (sync) and AsyncPromptLayer (async) classes, wrapping the litellm module via PromptLayerBase with function_name="litellm".
  • pyproject.toml - Added litellm as optional dependency (>=1.80.0,<1.87.0) and litellm extra (pip install promptlayer[litellm]).

Tests

Live E2E against Anthropic Claude Sonnet 4.6 (via Azure AI Foundry):

>>> import litellm
>>> resp = litellm.completion(model='anthropic/claude-sonnet-4-6', messages=[...], drop_params=True)
Response: 4
Model: claude-sonnet-4-6

Risk / Compatibility

  • Additive only. Existing openai and anthropic providers untouched.
  • litellm is an optional extra - base install unaffected (pip install promptlayer[litellm] to enable).
  • LiteLLM returns OpenAI-compatible response objects, so PromptLayerBase tracking works without modification.

Example usage

Install:

pip install promptlayer[litellm]

Python usage:

from promptlayer import PromptLayer

pl = PromptLayer(api_key="pl_...")

# Use LiteLLM through PromptLayer's tracking layer
# All calls are logged to PromptLayer dashboard
response = pl.litellm.completion(
    model="anthropic/claude-sonnet-4-6",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    drop_params=True,
)
print(response.choices[0].message.content)

# Works with any of 100+ providers:
#   'anthropic/claude-sonnet-4-6'  -> Anthropic API
#   'gemini/gemini-2.0-flash'     -> Google Gemini API
#   'openai/gpt-4o'               -> OpenAI API
#   'bedrock/anthropic.claude-v2' -> AWS Bedrock

@RheagalFire
Copy link
Copy Markdown
Author

cc @Jped @abubakarsohail

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant