LLM Gateway & Routing

LLM Gateway & Routing

Configure multi-model access, fallbacks, cost optimization, and A/​B testing.

Why Use a Gateway?

Without gateway:

• Vendor lock-in (one provider)
• No fallbacks (provider down = app down)
• Hard to A/​B test models
• Scattered API keys and configs

With gateway:

• Single API for 400+ models
• Automatic fallbacks
• Easy model switching
• Unified cost tracking

Quick Decision

OpenRouter (Recommended)

Why OpenRouter

• 400+ models: OpenAI, Anthropic, Google, Meta, Mistral, and more
• Single API: One key for all providers
• Automatic fallbacks: Built-in reliability
• A/​B testing: Easy model comparison
• Cost tracking: Unified billing dashboard
• Free credits: $1 free to start

Setup

# 1. Sign up at openrouter.ai
# 2. Get API key from dashboard
# 3. Add to .env:
OPENROUTER_API_KEY=sk-or-v1-...

Basic Usage

// Using fetch
const response = await fetch('https://openrouter.ai/​api/​v1/​chat/​completions', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.OPENROUTER_API_KEY}`,
    'Content-Type': 'application/​json',
  },
  body: JSON.stringify({
    model: 'anthropic/​claude-3-5-sonnet',
    messages: [{ role: 'user', content: 'Hello!' }],
  }),
});

With Vercel AI SDK (Recommended)

import { createOpenAI } from "@ai-sdk/​openai";
import { generateText } from "ai";

const openrouter = createOpenAI({
  baseURL: "https://openrouter.ai/​api/​v1",
  apiKey: process.env.OPENROUTER_API_KEY,
});

const { text } = await generateText({
  model: openrouter("anthropic/​claude-3-5-sonnet"),
  prompt: "Explain quantum computing",
});

Model IDs

// Format: provider/​model-name
const models = {
  // Anthropic
  claude35Sonnet: "anthropic/​claude-3-5-sonnet",
  claudeHaiku: "anthropic/​claude-3-5-haiku",

  // OpenAI
  gpt4o: "openai/​gpt-4o",
  gpt4oMini: "openai/​gpt-4o-mini",

  // Google
  geminiPro: "google/​gemini-pro-1.5",
  geminiFlash: "google/​gemini-flash-1.5",

  // Meta
  llama3: "meta-llama/​llama-3.1-70b-instruct",

  // Auto (OpenRouter picks best)
  auto: "openrouter/​auto",
};

Fallback Chains

// Define fallback order
const modelChain = [
  "anthropic/​claude-3-5-sonnet",   // Primary
  "openai/​gpt-4o",                  // Fallback 1
  "google/​gemini-pro-1.5",          // Fallback 2
];

async function callWithFallback(messages: Message[]) {
  for (const model of modelChain) {
    try {
      return await openrouter.chat({ model, messages });
    } catch (error) {
      console.log(`${model} failed, trying next...`);
    }
  }
  throw new Error("All models failed");
}

Cost Routing

// Route based on query complexity
function selectModel(query: string): string {
  const complexity = analyzeComplexity(query);

  if (complexity === "simple") {
    // Simple queries → cheap model
    return "openai/​gpt-4o-mini";  // ~$0.15/​1M tokens
  } else if (complexity === "medium") {
    // Medium → balanced
    return "google/​gemini-flash-1.5";  // ~$0.075/​1M tokens
  } else {
    // Complex → best quality
    return "anthropic/​claude-3-5-sonnet";  // ~$3/​1M tokens
  }
}

function analyzeComplexity(query: string): "simple" | "medium" | "complex" {
  // Simple heuristics
  if (query.length < 50) return "simple";
  if (query.includes("explain") || query.includes("analyze")) return "complex";
  return "medium";
}

A/​B Testing

// Random assignment
function getModel(userId: string): string {
  const hash = userId.charCodeAt(0) % 100;

  if (hash < 50) {
    return "anthropic/​claude-3-5-sonnet";  // 50%
  } else {
    return "openai/​gpt-4o";  // 50%
  }
}

// Track which model was used
const model = getModel(userId);
const response = await openrouter.chat({ model, messages });
await analytics.track("llm_call", { model, userId, latency, cost });

LiteLLM (Self-Hosted)

Why LiteLLM

• Self-hosted: Full control over data
• 100+ providers: Same coverage as OpenRouter
• Load balancing: Distribute across providers
• Cost tracking: Built-in spend management
• Caching: Redis or in-memory
• Rate limiting: Per-user limits

Setup

# Install
pip install litellm[proxy]

# Run proxy
litellm --config config.yaml

# Use as OpenAI-compatible endpoint
export OPENAI_API_BASE=http://localhost:4000

Configuration

# config.yaml
model_list:
  # Claude models
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/​claude-3-5-sonnet-latest
      api_key: sk-ant-...

  # OpenAI models
  - model_name: gpt-4o
    litellm_params:
      model: openai/​gpt-4o
      api_key: sk-...

  # Load balanced (multiple providers)
  - model_name: balanced
    litellm_params:
      model: anthropic/​claude-3-5-sonnet-latest
    litellm_params:
      model: openai/​gpt-4o
    # Requests distributed across both

# General settings
general_settings:
  master_key: sk-master-...
  database_url: postgresql://...

# Routing
router_settings:
  routing_strategy: simple-shuffle  # or latency-based-routing
  num_retries: 3
  timeout: 30

# Rate limiting
litellm_settings:
  max_budget: 100  # $100/​month
  budget_duration: monthly

Fallbacks in LiteLLM

model_list:
  - model_name: primary
    litellm_params:
      model: anthropic/​claude-3-5-sonnet-latest
    fallbacks:
      - model_name: fallback-1
        litellm_params:
          model: openai/​gpt-4o
      - model_name: fallback-2
        litellm_params:
          model: google/​gemini-pro

Usage

// Use like OpenAI SDK
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:4000",
  apiKey: "sk-master-...",
});

const response = await client.chat.completions.create({
  model: "claude-sonnet",  // Maps to configured model
  messages: [{ role: "user", content: "Hello!" }],
});

Routing Strategies

1. Cost-Based Routing

const costTiers = {
  cheap: ["openai/​gpt-4o-mini", "google/​gemini-flash-1.5"],
  balanced: ["anthropic/​claude-3-5-haiku", "openai/​gpt-4o"],
  premium: ["anthropic/​claude-3-5-sonnet", "openai/​o1-preview"],
};

function routeByCost(budget: "cheap" | "balanced" | "premium"): string {
  const models = costTiers[budget];
  return models[Math.floor(Math.random() * models.length)];
}

2. Latency-Based Routing

// Track latency per model
const latencyStats: Record<string, number[]> = {};

function routeByLatency(): string {
  const avgLatencies = Object.entries(latencyStats)
    .map(([model, times]) => ({
      model,
      avg: times.reduce((a, b) => a + b, 0) / times.length,
    }))
    .sort((a, b) => a.avg - b.avg);

  return avgLatencies[0].model;
}

// Update after each call
function recordLatency(model: string, latencyMs: number) {
  if (!latencyStats[model]) latencyStats[model] = [];
  latencyStats[model].push(latencyMs);
  // Keep last 100 samples
  if (latencyStats[model].length > 100) {
    latencyStats[model].shift();
  }
}

3. Task-Based Routing

const taskModels = {
  coding: "anthropic/​claude-3-5-sonnet",  // Best for code
  reasoning: "openai/​o1-preview",          // Best for logic
  creative: "anthropic/​claude-3-5-sonnet", // Best for writing
  simple: "openai/​gpt-4o-mini",            // Cheap and fast
  multimodal: "google/​gemini-pro-1.5",     // Vision + text
};

function routeByTask(task: keyof typeof taskModels): string {
  return taskModels[task];
}

4. Hybrid Routing

interface RoutingConfig {
  task: string;
  maxCost: number;
  maxLatency: number;
}

function hybridRoute(config: RoutingConfig): string {
  // Filter by cost
  const affordable = models.filter(m => m.cost <= config.maxCost);

  // Filter by latency
  const fast = affordable.filter(m => m.avgLatency <= config.maxLatency);

  // Select best for task
  const taskScores = fast.map(m => ({
    model: m.id,
    score: getTaskScore(m.id, config.task),
  }));

  return taskScores.sort((a, b) => b.score - a.score)[0].model;
}

Best Practices

1. Always Have Fallbacks

// Bad: Single point of failure
const response = await openai.chat({ model: "gpt-4o", messages });

// Good: Fallback chain
const models = ["gpt-4o", "claude-3-5-sonnet", "gemini-pro"];
for (const model of models) {
  try {
    return await gateway.chat({ model, messages });
  } catch (e) {
    continue;
  }
}

2. Pin Model Versions

// Bad: Model can change
const model = "gpt-4";

// Good: Pinned version
const model = "openai/​gpt-4-0125-preview";

3. Track Costs

// Log every call
async function trackedCall(model: string, messages: Message[]) {
  const start = Date.now();
  const response = await gateway.chat({ model, messages });
  const latency = Date.now() - start;

  await analytics.track("llm_call", {
    model,
    inputTokens: response.usage.prompt_tokens,
    outputTokens: response.usage.completion_tokens,
    cost: calculateCost(model, response.usage),
    latency,
  });

  return response;
}

4. Set Token Limits

// Prevent runaway costs
const response = await gateway.chat({
  model,
  messages,
  max_tokens: 500,  // Limit output length
});

5. Use Caching

// LiteLLM caching
litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: localhost
    port: 6379
    ttl: 3600  # 1 hour

References

• references/​openrouter-guide.md - OpenRouter deep dive
• references/​litellm-guide.md - LiteLLM self-hosting
• references/​routing-strategies.md - Advanced routing patterns
• references/​alternatives.md - Helicone, Portkey, etc.

Templates

• templates/​openrouter-config.ts - TypeScript OpenRouter setup
• templates/​litellm-config.yaml - LiteLLM proxy config
• templates/​fallback-chain.ts - Fallback implementation