Nomad

Allocations

# List all allocations for a job
nomad job allocs <job>

# Inspect a specific allocation
nomad alloc status <alloc-id>

# Stream stdout logs for an allocation
nomad alloc logs <alloc-id>

# Stream stderr logs for an allocation
nomad alloc logs -stderr <alloc-id>

# Follow (tail) stdout logs
nomad alloc logs -f <alloc-id>

# Follow stderr
nomad alloc logs -f -stderr <alloc-id>

# Follow logs for a specific task within an allocation
nomad alloc logs -f <alloc-id> <task-name>

# Execute a command inside a running allocation
nomad alloc exec -task <task-name> <alloc-id> /bin/sh

# Execute a one-off command
nomad alloc exec -task <task-name> <alloc-id> env

Nodes & Cluster

# List all client nodes
nomad node status

# Inspect a specific node
nomad node status <node-id>

# Check cluster health (Raft peers)
nomad operator raft list-peers

# Drain a node before maintenance (ONLY when explicitly instructed)
nomad node drain -enable -deadline 10m <node-id>

# Disable drain after maintenance (ONLY when explicitly instructed)
nomad node drain -disable <node-id>

# Check ACL token (verify auth is working)
nomad acl token self

Runbooks

Debug Failed Allocation

When a job allocation is in failed or lost state:

1. List jobs to find the affected job:

   nomad status
   

2. Get job detail and look for failed allocations:

   nomad status <job>
   

3. Inspect the failing allocation (note the alloc ID from step 2):

   nomad alloc status <alloc-id>
   

4. Check the Recent Events section in alloc status output for error messages (OOM kill, port conflict, image pull failure, etc.).

5. Read stderr logs for the task:

   nomad alloc logs -stderr <alloc-id>
   

6. If the container exited immediately, also read stdout:

   nomad alloc logs <alloc-id>
   

7. Cross-reference with the node the alloc ran on:

   nomad node status <node-id>
   

Rolling Deploy

When deploying a new version of a job:

1. Edit the job spec with the new version or image tag.

2. Plan the job to see what will change (this is always safe):

   nomad job plan <job.nomad.hcl>
   

3. Review the diff output — confirm the only changes are the expected ones.

4. When explicitly instructed to apply, run the job:

   nomad job run <job.nomad.hcl>
   

5. Watch the deployment status:

   nomad status <job>
   

6. Follow logs during rollout:

   nomad alloc logs -f <new-alloc-id>
   

7. If a deployment gets stuck, check the deployment detail:

   nomad job deployments <job>
   

Drain Node

Before taking a node offline for maintenance (only when explicitly instructed):

1. Identify the node ID:

   nomad node status
   

2. Inspect the node to see currently running allocations:

   nomad node status <node-id>
   

3. Enable drain with a deadline (allocations migrate to other nodes):

   nomad node drain -enable -deadline 10m <node-id>
   

4. Monitor until the node shows ineligible and all allocs have migrated:

   nomad node status <node-id>
   

5. Perform maintenance on the node.

6. Re-enable the node after maintenance:

   nomad node drain -disable <node-id>
   

Read Allocation Logs

When a developer asks for logs from a running or recently-stopped service:

1. Find the job:

   nomad status <job>
   

2. List allocations and find the relevant one (most recent, running status):

   nomad job allocs <job>
   

3. For real-time log tailing:

   nomad alloc logs -f <alloc-id>
   

4. For stderr (crash logs, panic output):

   nomad alloc logs -f -stderr <alloc-id>
   

5. If the job has multiple tasks (e.g. sidecar), specify the task:

   nomad alloc logs -f <alloc-id> <task-name>
   

Safety

Never run the following without an explicit instruction from the user:

• nomad job run — deploys or updates a running job
• nomad job stop — stops and removes a job
• nomad system gc — forces garbage collection of stopped allocations and jobs
• nomad node drain -enable — drains a client node

Read-only commands (nomad status, nomad alloc status, nomad alloc logs, nomad job plan, nomad operator raft list-peers) are always safe to run.