Huggingface Inference

Free models (rate-limited):

• mistralai/Mistral-7B-Instruct-v0.3 (7B, instruction-tuned)
• meta-llama/Llama-2-7b-chat-hf (7B, chat)
• codellama/CodeLlama-7b-hf (code generation)

For higher throughput or larger models, configure dedicated Inference Endpoints.

Parameters

You can specify in your prompt:

• Model ID (required unless using default)
• Task type (text-generation, feature-extraction, sentence-similarity)
• Parameters: max_tokens, temperature, top_p

Example with parameters: "Using mistralai/Mistral-7B-Instruct-v0.3, with temperature=0.7 and max_tokens=500, generate: 'Write a haiku about code'"

Rate Limits

Free Inference API: ~ few requests per minute. For production use:

1. Deploy dedicated Inference Endpoint
2. Set HF_INFERENCE_ENDPOINT environment variable to point to your endpoint
3. Token authentication required for gated models

Environment

• HF_TOKEN: Hugging Face access token (required for gated models)
• HF_INFERENCE_ENDPOINT: Optional custom endpoint URL (defaults to api inference)

Error Handling

• 429: Rate limit exceeded, retry with exponential backoff
• 403: Model gated, check HF_TOKEN permissions
• 404: Model ID not found or not supported for this task