Rocm Vllm Deployment

Not required for testing: The skill will proceed without these set:

• HF_TOKEN: Optional — public models work without it; gated models fail at download with clear error
• HF_HOME: Optional — defaults to /root/.cache/huggingface/hub

Environment Variable Detection

Priority Order:

1. Explicit parameter (highest) — Provided in task/request (e.g., hf_token: "xxx")
2. Environment variable — Already set in shell or from parent process
3. ~/.bash_profile — Source to load variables
4. Default value (lowest) — HF_HOME defaults to /root/.cache/huggingface/hub

Philosophy: Fail fast for configuration errors, fail at download time for authentication errors.

Helper Scripts

Location: <skill-dir>/scripts/

check-env.sh

Validate and load environment variables before deployment.

Usage:

# Basic check (HF_TOKEN optional, HF_HOME optional with default)
./scripts/check-env.sh

# Strict mode (HF_HOME required, fails if not set)
./scripts/check-env.sh --strict

# Quiet mode (minimal output, for automation)
./scripts/check-env.sh --quiet

# Test with environment variables
HF_TOKEN="hf_xxx" HF_HOME="/models" ./scripts/check-env.sh

Exit Codes:

Note: This script is optional. You can also directly run source ~/.bash_profile.

generate-report.sh

Generate human-readable deployment report after successful deployment.

Usage:

./scripts/generate-report.sh <model-id> <container-name> <port> <status> [model-load-time] [memory-used]

# Example:
./scripts/generate-report.sh \
  "Qwen-Qwen3-0.6B" \
  "vllm-qwen3-0-6b" \
  "8001" \
  "✅ Success" \
  "3.6" \
  "1.2"

Parameters:

Output: $HOME/vllm-compose/<model-id>/DEPLOYMENT_REPORT.md

Exit Codes:

Integration: This script is automatically called in Phase 7 of the deployment workflow.

Input Schema

Output Structure

All deployment artifacts MUST be saved to:

$HOME/vllm-compose/<model-id-slash-to-dash>/

Convert model ID to directory name by replacing / with -:

• openai/gpt-oss-20b → $HOME/vllm-compose/openai-gpt-oss-20b/
• Qwen/Qwen3-Coder-Next-FP8 → $HOME/vllm-compose/Qwen-Qwen3-Coder-Next-FP8/

Per-model directory structure:

$HOME/vllm-compose/<model-id>/
├── deployment.log          # Full deployment logs (stdout + stderr)
├── test-results.json       # Functional test results (JSON format)
├── docker-compose.yml      # Generated Docker Compose file
├── .env                    # HF_TOKEN environment (chmod 600, optional)
└── DEPLOYMENT_REPORT.md    # Human-readable deployment summary

File requirements:

• deployment.log — Capture ALL container logs during deployment
• test-results.json — Save API response from functional test request
• DEPLOYMENT_REPORT.md — Generated in Phase 7
• All three files MUST exist before marking deployment as complete

Execution Workflow

Phase 0: Environment Check & Auto-Repair

Step 0.1: Load Environment Variables

# Source ~/.bash_profile to load HF_HOME and HF_TOKEN
source ~/.bash_profile

# If HF_HOME is not defined, it defaults to /root/.cache/huggingface/hub

If HF_HOME is not defined in ~/.bash_profile, it defaults to /root/.cache/huggingface/hub.

Step 0.2: Create Output Directory

• Create: $HOME/vllm-compose/<model-id>/

Step 0.3: Initialize Logging

• All output → $HOME/vllm-compose/<model-id>/deployment.log

Step 0.4: System Checks

• Detect OS and package manager
• Check Python, pip, huggingface_hub
• Check Docker, docker compose
• Check ROCm tools (rocm-smi/amd-smi)
• Check GPU access (/dev/kfd, /dev/dri)
• Check disk space (20GB minimum)

Phase 1: Model Download

Use HF_HOME from Phase 0 (environment variable or default):

# Download model to HF_HOME
huggingface-cli download <model_id> --local-dir "$HF_HOME/hub/models--<org>--<model>"

# Or use snapshot_download via Python:
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='<model_id>', cache_dir='$HF_HOME')"

Authentication Handling:

On Authentication Failure:

echo "ERROR: Model download failed - authentication required"
echo "This model requires a valid HF_TOKEN."
echo ""
echo "Please add to ~/.bash_profile:"
echo "  export HF_TOKEN=\"hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\""
echo "Then run: source ~/.bash_profile"
exit 1

• Locate model path in HF cache: $HF_HOME/hub/models--<org>--<model-name>/
• Log download progress to deployment.log

Phase 2: Model Parameter Detection

• Read config.json from model
• Auto-detect: max_model_len, hidden_size, num_attention_heads, num_hidden_layers, vocab_size, dtype
• Validate TP size divides attention heads
• Estimate VRAM requirement

Phase 3: Docker Compose Configuration

Generate files in output directory:

• docker-compose.yml → $HOME/vllm-compose/<model-id>/docker-compose.yml

  • Mount HF_HOME as volume (read-only for models)
  • NO hardcoded tokens in compose file

• .env → $HOME/vllm-compose/<model-id>/.env (optional)

  • Contains: HF_TOKEN=<value>
  • Permissions: chmod 600
  • Only created if user explicitly requests persistent token storage

Volume mount example: