Jetson Deploy

Domain Principles Table

Knowledge Base Lookups

Use search_knowledge (grounded-code-mcp) to ground decisions in authoritative references.

Protocol: Search edge_ai and robotics collections for Jetson and TensorRT guidance. Search automation for containerization and fleet deployment context. Always cite the source path from KB results.

Workflow

Deployment Pipeline

┌──────────────────────────────────────────────────────────────────────┐
│                                                                      │
│   ┌───────┐    ┌─────────────┐    ┌─────────┐    ┌──────────┐       │
│   │ SETUP │───>│ CONTAINERIZE│───>│ CONVERT │───>│ OPTIMIZE │       │
│   └───────┘    └─────────────┘    └─────────┘    └──────────┘       │
│                                                       │              │
│                                                       v              │
│                                   ┌────────┐    ┌───────────┐       │
│                                   │ DEPLOY │<───│ BENCHMARK │       │
│                                   └────────┘    └───────────┘       │
│                                       │                              │
│                                       └──── (iterate if needed) ──┐ │
│                                                                   │ │
│                                   ┌──────────┐                    │ │
│                                   │ OPTIMIZE │<───────────────────┘ │
│                                   └──────────┘                      │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

Pre-Flight Checklist

Before beginning any deployment workflow, verify:

PRE-FLIGHT VERIFICATION
┌──────────────────────────────────────────────────────────────────┐
│ □ JetPack version confirmed (cat /etc/nv_tegra_release)         │
│ □ L4T version matches expected (dpkg -l nvidia-l4t-core)        │
│ □ CUDA version confirmed (nvcc --version)                       │
│ □ TensorRT version confirmed (dpkg -l tensorrt)                 │
│ □ Available disk space > 10GB (df -h)                           │
│ □ Docker runtime is nvidia (docker info | grep -i runtime)      │
│ □ Power mode is set (sudo nvpmodel -q)                          │
│ □ Fan mode is set (sudo jetson_clocks --show)                   │
│ □ Network access for container pulls (if needed)                │
│ □ Model files are accessible on device                          │
└──────────────────────────────────────────────────────────────────┘

If ANY checkbox is unchecked → STOP. Resolve before proceeding.

Step 1: SETUP

Objective: Confirm the Jetson device is properly configured for deployment.

Actions:

1. SSH into the Jetson device or open local terminal
2. Run cat /etc/nv_tegra_release to confirm L4T version
3. Run sudo nvpmodel -q to check current power mode
4. Run sudo nvpmodel -m <MODE> to set target power mode (0=MAXN, 1=15W, 2=7W for Orin Nano)
5. Run sudo jetson_clocks to lock clock frequencies for consistent benchmarking
6. Install jtop: sudo pip3 install jetson-stats
7. Verify Docker nvidia runtime: docker run --rm --runtime nvidia --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi

Exit Criteria:

• JetPack version documented
• Power mode set and confirmed
• Docker nvidia runtime functional
• jtop installed and running

Step 2: CONTAINERIZE

Objective: Build a reproducible container environment using jetson-containers.

Actions:

1. Clone jetson-containers: git clone https://github.com/dusty-nv/jetson-containers
2. Select appropriate base image for JetPack version
3. Define container requirements (TensorRT, OpenCV, model framework)
4. Build container using jetson-containers build or docker build
5. Test container starts with GPU access
6. Mount model directory and data directory as volumes

Exit Criteria:

• Container builds without errors
• Container runs with --runtime nvidia
• GPU is accessible inside container (python3 -c "import tensorrt; print(tensorrt.__version__)")
• Model files accessible via volume mount

Step 3: CONVERT

Objective: Convert model from training format to TensorRT engine.

Actions:

1. Export model to ONNX format (if not already ONNX)
2. Validate ONNX model: python3 -c "import onnx; model = onnx.load('model.onnx'); onnx.checker.check_model(model)"
3. Convert ONNX to TensorRT engine using trtexec or Python API
4. Specify target precision (FP16 default for Orin Nano)
5. Handle dynamic shapes if needed
6. Verify engine loads and produces output

Exit Criteria:

• TensorRT engine file created (.engine or .trt)
• Engine loads without errors
• Engine produces output on sample input
• Output shape matches expected dimensions

Step 4: OPTIMIZE

Objective: Tune the TensorRT engine and pipeline for target performance.

Actions:

1. Profile with trtexec: trtexec --loadEngine=model.engine --iterations=100 --avgRuns=50
2. Try INT8 quantization if FP16 does not meet latency target (requires calibration)
3. Experiment with workspace size: --memPoolSize=workspace:1024MiB
4. Optimize input preprocessing (use GPU-accelerated resize, normalize)
5. Add CUDA streams for async execution if pipeline allows

Exit Criteria:

• Latency meets target at specified power mode
• Memory usage leaves headroom for OS and other processes
• No thermal throttling under sustained load

Step 5: BENCHMARK

Objective: Produce reliable, reproducible performance measurements.

Actions:

1. Set power mode explicitly: sudo nvpmodel -m <MODE>
2. Lock clocks: sudo jetson_clocks
3. Run warm-up iterations (50+) before measuring
4. Record latency (mean, P50, P95, P99) across 1000+ iterations
5. Monitor with tegrastats during benchmark: tegrastats --interval 1000
6. Record GPU utilization, memory usage, temperature, power draw
7. Validate accuracy against reference outputs

Exit Criteria:

• Benchmark results documented with power mode and clock state
• Latency distribution captured (not just mean)
• Thermal behavior stable (no throttling during measurement)
• Accuracy validated against golden reference

Step 6: DEPLOY

Objective: Finalize the deployment for production operation.

Actions:

1. Create Docker Compose or systemd service for auto-start
2. Configure restart policies for resilience
3. Set up logging and monitoring (tegrastats, jtop, application logs)
4. Configure watchdog for thermal protection
5. Validate end-to-end pipeline with production data
6. Document deployment configuration

Exit Criteria:

• Application starts automatically on boot
• Restart policy handles crashes gracefully
• Monitoring is active and accessible
• End-to-end pipeline validated with real data

State Block Format

Maintain state across conversation turns using this block:

<jetson-deploy-state>