Multi Llm Providers

Google

• gemini-2.0-flash, gemini-1.5-pro, gemini-1.5-flash, gemini-pro

Cohere

• command-r-plus, command-r, command-light, command

Ollama (Local)

• llama3.2, llama3.1, mistral, mixtral
• codellama, phi-4, deepseek-coder

RuVector (Custom)

• Vector-optimized provider for embeddings and search

Provider Manager

import { createProviderManager } from '@claude-flow/providers';

const manager = createProviderManager({
  providers: [
    { provider: 'anthropic', apiKey: '...', model: 'claude-3-5-sonnet-latest' },
    { provider: 'openai', apiKey: '...', model: 'gpt-4o' },
    { provider: 'ollama', apiUrl: 'http://localhost:11434', model: 'llama3.2' },
  ],
  defaultProvider: 'anthropic',
  loadBalancing: {
    enabled: true,
    strategy: 'cost-based',  // 'round-robin' | 'least-loaded' | 'latency-based' | 'cost-based'
  },
  fallback: {
    enabled: true,
    maxAttempts: 3,
  },
  cache: {
    enabled: true,
    ttl: 3600,
    maxSize: 1000,
  },
  costOptimization: {
    enabled: true,
    maxCostPerRequest: 0.50,
  },
});

Provider Interface

Every provider implements ILLMProvider:

interface ILLMProvider extends EventEmitter {
  readonly name: LLMProvider;
  readonly capabilities: ProviderCapabilities;

  initialize(): Promise<void>;
  complete(request: LLMRequest): Promise<LLMResponse>;
  streamComplete(request: LLMRequest): AsyncIterable<LLMStreamEvent>;

  listModels(): Promise<LLMModel[]>;
  getModelInfo(model: LLMModel): Promise<ModelInfo>;
  validateModel(model: LLMModel): boolean;

  healthCheck(): Promise<HealthCheckResult>;
  getStatus(): ProviderStatus;

  estimateCost(request: LLMRequest): Promise<CostEstimate>;
  getUsage(period?: UsagePeriod): Promise<UsageStats>;

  destroy(): void;
}

Making Requests

Completion

const response = await manager.complete({
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'Hello!' },
  ],
  model: 'claude-3-5-sonnet-latest',
  temperature: 0.7,
  maxTokens: 1024,
});

// response.content     — Generated text
// response.usage       — { promptTokens, completionTokens, totalTokens }
// response.cost        — { promptCost, completionCost, totalCost, currency }
// response.latency     — Response time in ms
// response.finishReason — 'stop' | 'length' | 'tool_calls' | 'content_filter'

Streaming

for await (const event of manager.streamComplete(request)) {
  switch (event.type) {
    case 'content': console.log(event.delta?.content); break;
    case 'tool_call': /* handle tool call */ break;
    case 'done': /* completion finished */ break;
    case 'error': /* handle error */ break;
  }
}

Tool Calling

const response = await manager.complete({
  messages: [...],
  tools: [{
    type: 'function',
    function: {
      name: 'get_weather',
      description: 'Get current weather',
      parameters: {
        type: 'object',
        properties: { location: { type: 'string' } },
        required: ['location'],
      },
    },
  }],
  toolChoice: 'auto',  // 'auto' | 'none' | 'required' | { function: { name } }
});

Cost Constraints

const response = await manager.complete({
  messages: [...],
  costConstraints: {
    maxCost: 0.10,
    preferredModels: ['claude-3-haiku-20240307', 'gpt-4o-mini'],
  },
});

Provider Capabilities

interface ProviderCapabilities {
  supportedModels: LLMModel[];
  maxContextLength: Record<string, number>;
  maxOutputTokens: Record<string, number>;
  supportsStreaming: boolean;
  supportsToolCalling: boolean;
  supportsSystemMessages: boolean;
  supportsVision: boolean;
  supportsAudio: boolean;
  supportsFineTuning: boolean;
  supportsEmbeddings: boolean;
  supportsBatching: boolean;
  rateLimit?: { requestsPerMinute, tokensPerMinute, concurrentRequests };
  pricing: Record<string, { promptCostPer1k, completionCostPer1k, currency }>;
}

Cost Estimation

const estimate = await provider.estimateCost(request);
// { estimatedPromptTokens, estimatedCompletionTokens, estimatedTotalTokens,
//   estimatedCost: { prompt, completion, total, currency }, confidence }

Usage Tracking

const usage = await provider.getUsage('day');
// { period, requests, tokens, cost, errors, averageLatency, modelBreakdown }

Error Handling

Typed errors for precise error handling:

• LLMProviderError — Base error with code, provider, statusCode, retryable
• RateLimitError — 429 with retryAfter duration
• AuthenticationError — 401, not retryable
• ModelNotFoundError — 404, not retryable
• ProviderUnavailableError — 503, retryable

import { isRateLimitError, isLLMProviderError } from '@claude-flow/providers';

try {
  await provider.complete(request);
} catch (error) {
  if (isRateLimitError(error)) {
    await sleep(error.retryAfter * 1000);
  }
}

Base Provider Features

The BaseProvider abstract class provides all providers with:

• Circuit breaker protection — Prevents cascading failures
• Health monitoring — Periodic health checks
• Cost tracking — Per-request cost calculation
• Request metrics — Latency, throughput, error rates
• Automatic retries — Configurable retry with backoff

Load Balancing Strategies