Production-ready vLLM deployment on AMD ROCm GPUs. Combines environment auto-check, model parameter detection, Docker Compose deployment, health verification, and functional testing with comprehensive logging and security best practices.
Production-ready automation for deploying vLLM inference services on AMD ROCm GPUs using Docker Compose.
Recommended (for production): Add to ~/.bash_profile:
# HuggingFace authentication token (required for gated models)
export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
# Model cache directory (optional)
export HF_HOME="$HOME/models"
# Apply changes
source ~/.bash_profile
Not required for testing: The skill will proceed without these set:
/root/.cache/huggingface/hubPriority Order:
hf_token: "xxx")/root/.cache/huggingface/hub| Variable | Required | If Missing |
|---|---|---|
HF_TOKEN | Conditional | Continue without token (public models work; gated models fail at download with clear error) |
HF_HOME | No | Warning + Default — Use /root/.cache/huggingface/hub |
Philosophy: Fail fast for configuration errors, fail at download time for authentication errors.
Location: <skill-dir>/scripts/
Validate and load environment variables before deployment.
Usage:
# Basic check (HF_TOKEN optional, HF_HOME optional with default)
./scripts/check-env.sh
# Strict mode (HF_HOME required, fails if not set)
./scripts/check-env.sh --strict
# Quiet mode (minimal output, for automation)
./scripts/check-env.sh --quiet
# Test with environment variables
HF_TOKEN="hf_xxx" HF_HOME="/models" ./scripts/check-env.sh
Exit Codes:
| Code | Meaning |
|---|---|
| 0 | Environment check completed (variables loaded or defaulted) |
| 2 | Critical error (e.g., cannot source ~/.bash_profile) |
Note: This script is optional. You can also directly run source ~/.bash_profile.
Generate human-readable deployment report after successful deployment.
Usage:
./scripts/generate-report.sh <model-id> <container-name> <port> <status> [model-load-time] [memory-used]
# Example:
./scripts/generate-report.sh \
"Qwen-Qwen3-0.6B" \
"vllm-qwen3-0-6b" \
"8001" \
"✅ Success" \
"3.6" \
"1.2"
Parameters:
| Parameter | Required | Description |
|---|---|---|
model-id | Yes | Model ID (with / replaced by -) |
container-name | Yes | Docker container name |
port | Yes | Host port for API endpoint |
status | Yes | Deployment status (e.g., "✅ Success") |
model-load-time | No | Model loading time in seconds |
memory-used | No | Memory consumption in GiB |
Output: $HOME/vllm-compose/<model-id>/DEPLOYMENT_REPORT.md
Exit Codes:
| Code | Meaning |
|---|---|
| 0 | Report generated successfully |
| 1 | Missing required parameters |
| 2 | Output directory not found |
Integration: This script is automatically called in Phase 7 of the deployment workflow.
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
| model_id | String | Yes | - | HuggingFace model ID |
| docker_image | String | No | rocm/vllm-dev:nightly | vLLM Docker image |
| tensor_parallel_size | Integer | No | 1 | Number of GPUs |
| port | Integer | No | 9999 | API server port |
| hf_home | String | No | ${HF_HOME} or /root/.cache/huggingface/hub | Model cache directory |
| hf_token | Secret | Conditional | ${HF_TOKEN} | HuggingFace token (optional for public models, required for gated models) |
| max_model_len | Integer | No | Auto-detect | Maximum sequence length |
| gpu_memory_utilization | Float | No | 0.85 | GPU memory utilization |
| auto_install | Boolean | No | true | Auto-install dependencies |
| log_level | String | No | INFO | Logging verbosity |
All deployment artifacts MUST be saved to:
$HOME/vllm-compose/<model-id-slash-to-dash>/
Convert model ID to directory name by replacing / with -:
openai/gpt-oss-20b → $HOME/vllm-compose/openai-gpt-oss-20b/Qwen/Qwen3-Coder-Next-FP8 → $HOME/vllm-compose/Qwen-Qwen3-Coder-Next-FP8/Per-model directory structure:
$HOME/vllm-compose/<model-id>/
├── deployment.log # Full deployment logs (stdout + stderr)
├── test-results.json # Functional test results (JSON format)
├── docker-compose.yml # Generated Docker Compose file
├── .env # HF_TOKEN environment (chmod 600, optional)
└── DEPLOYMENT_REPORT.md # Human-readable deployment summary
File requirements:
deployment.log — Capture ALL container logs during deploymenttest-results.json — Save API response from functional test requestDEPLOYMENT_REPORT.md — Generated in Phase 7Step 0.1: Load Environment Variables
# Source ~/.bash_profile to load HF_HOME and HF_TOKEN
source ~/.bash_profile
# If HF_HOME is not defined, it defaults to /root/.cache/huggingface/hub
If HF_HOME is not defined in ~/.bash_profile, it defaults to /root/.cache/huggingface/hub.
Step 0.2: Create Output Directory
$HOME/vllm-compose/<model-id>/Step 0.3: Initialize Logging
$HOME/vllm-compose/<model-id>/deployment.logStep 0.4: System Checks
Use HF_HOME from Phase 0 (environment variable or default):
# Download model to HF_HOME
huggingface-cli download <model_id> --local-dir "$HF_HOME/hub/models--<org>--<model>"
# Or use snapshot_download via Python:
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='<model_id>', cache_dir='$HF_HOME')"
Authentication Handling:
| Scenario | Behavior |
|---|---|
| Public model + no token | ✅ Download succeeds |
| Public model + token provided | ✅ Download succeeds |
| Gated model + no token | ❌ Download fails with "authentication required" error |
| Gated model + invalid token | ❌ Download fails with "invalid token" error |
| Gated model + valid token | ✅ Download succeeds |
On Authentication Failure:
echo "ERROR: Model download failed - authentication required"
echo "This model requires a valid HF_TOKEN."
echo ""
echo "Please add to ~/.bash_profile:"
echo " export HF_TOKEN=\"hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\""
echo "Then run: source ~/.bash_profile"
exit 1
$HF_HOME/hub/models--<org>--<model-name>/deployment.logGenerate files in output directory:
docker-compose.yml → $HOME/vllm-compose/<model-id>/docker-compose.yml
.env → $HOME/vllm-compose/<model-id>/.env (optional)
HF_TOKEN=<value>chmod 600Volume mount example: