Debug Kubernetes deployment issues - pods, services, configs, images, and ArgoCD sync problems
Use this skill when the user reports Kubernetes deployment issues, pod failures, or service connectivity problems.
CrashLoopBackOff, ImagePullBackOff, ErrImagePull, CreateContainerConfigError, Pending state$DEV_SSH<service>-dev (e.g., auth-dev, order-dev)gitops/apps/<service>/base/apps/<service>/overlays/dev/docker build, docker-compose, or make docker-build. All image building happens in GitLab CI.Edit code → commit → push → wait for CI pipeline to build image
Update gitops (if needed) → commit → push → ArgoCD hard-refresh → verify
Edit gitops file → commit → push → ArgoCD hard-refresh → verify
# Pod status
$DEV_SSH "kubectl get pods -n <service>-dev"
# Events (best first signal)
$DEV_SSH "kubectl get events -n <service>-dev --sort-by=.metadata.creationTimestamp | tail -20"
# Logs
$DEV_SSH "kubectl logs -n <service>-dev -l app.kubernetes.io/name=<service> --tail=80 2>&1"
# Previous crash logs
$DEV_SSH "kubectl logs -n <service>-dev -l app.kubernetes.io/name=<service> --previous --tail=50 2>&1"
# Describe pod (for CreateContainerConfigError, probe failures)
$DEV_SSH "kubectl describe pod -n <service>-dev -l app.kubernetes.io/name=<service> 2>&1 | tail -40"
Edit the relevant file(s) in gitops/apps/<service>/.
GitOps File Map:
| What to fix | File |
|---|---|
| Image tag | 🚫 NEVER modify manually (Handled by CI/CD) |
| Env vars / config | overlays/dev/configmap.yaml |
| Secrets (DB password, keys) | overlays/dev/secrets.yaml |
| Ports, probes, Dapr annotations | base/deployment.yaml or base/worker-deployment.yaml |
| Migration job | base/migration-job.yaml |
| Kustomize resource list | overlays/dev/kustomization.yaml → resources: |
Infrastructure endpoints (dev cluster):
PostgreSQL : postgresql.infrastructure.svc.cluster.local:5432
Redis : redis.infrastructure.svc.cluster.local:6379
Consul : consul.infrastructure.svc.cluster.local:8500
Secrets pattern — if secretRef: name: <service>-secrets is in deployment but missing:
# overlays/dev/secrets.yaml
apiVersion: v1