Kubernetes Deployment

etcd

• Distributed key-value store - Stores all cluster state
• Consistency - Ensures data consistency across control plane
• Durability - Persists cluster configuration and state
• Watch notifications - Notifies API server of state changes

Controller Manager (kube-controller-manager)

• Runs controllers - Replication, endpoints, namespace, service accounts
• Reconciliation loop - Continuously compares desired vs current state
• Resource lifecycle - Manages creation, updates, and deletion
• Leader election - Ensures only one active controller in HA setups

Scheduler (kube-scheduler)

• Pod placement - Selects appropriate nodes for pod deployment
• Resource optimization - Considers resource requirements and constraints
• Scheduling policies - Applies affinity, taints, and tolerations
• Node selection - Evaluates nodes based on filters and priorities

kubelet (Node Component)

• Pod execution - Runs containers on nodes as instructed
• Health reporting - Reports pod and node status to API server
• Current state reporting - Updates API server with actual state
• API server communication - Receives pod specifications and reports status

Reconciliation Loop Process

The reconciliation loop is the core mechanism that ensures desired state matches actual state:

1. Desired State Creation: User creates resources via kubectl (pods, deployments, services)
2. State Persistence: API server stores desired state in etcd
3. Watch Trigger: Controllers and scheduler watch for relevant changes
4. Current State Assessment: Components assess current cluster state
5. Gap Analysis: Compare desired vs current state
6. Action Execution: Take actions to align current state with desired state
7. Status Update: Report new state back to API server
8. Loop Continuation: Repeat continuously to maintain consistency

Component Collaboration Flow

User Action → API Server → etcd (Store Desired State)
      ↓
Scheduler Watches → Finds Suitable Node → Updates Pod Spec
      ↓
Kubelet Watches → Receives Pod Assignment → Starts Containers
      ↓
Controller Watches → Monitors Actual State → Adjusts as Needed
      ↓
Status Updates → API Server → etcd (Store Current State)

Detailed Reconciliation Process

The reconciliation loop follows this detailed sequence:

1. User Action: A user creates/modifies a resource (e.g., Deployment) via kubectl or API call
2. API Server Storage: The API server validates the request and stores the desired state in etcd
3. Controller Monitoring: Controllers continuously watch for changes in the desired state through the API server
4. Actual State Assessment: Controllers compare the desired state with the actual state of the cluster
5. Action Execution: When discrepancies are detected, controllers take corrective actions
6. State Updates: As actions are executed, the actual state changes, which is reflected in etcd
7. Continuous Loop: The process repeats continuously to ensure the actual state matches the desired state

Key Principles of Reconciliation

• Eventual Consistency: The system continuously works toward the desired state, eventually converging
• Idempotency: Actions can be safely repeated without causing unintended side effects
• Distributed Coordination: Multiple controllers operate independently while coordinating through etcd
• Watch Mechanism: Controllers use efficient watch mechanisms rather than constant polling
• Rate Limiting: Controllers implement backoff strategies to prevent overwhelming the system

Failure Handling in Reconciliation

• Self-Healing: When a Pod crashes, the ReplicaSet controller creates a new one to maintain the desired replica count
• Node Failures: The Node Controller detects failed nodes and marks their Pods as needing replacement
• Retry Logic: Failed operations are retried with exponential backoff to handle transient issues
• Leader Election: Critical controllers support leader election to ensure only one instance makes decisions at a time

Example: Pod Creation Reconciliation

1. kubectl create -f pod.yaml → API Server receives pod specification
2. API Server saves → Pod object stored in etcd with nodeName=""
3. Scheduler watches → Detects unscheduled pod, selects node based on resource requirements, constraints, and policies
4. Scheduler updates → Pod object in etcd with nodeName="node-1"
5. Kubelet watches → Detects pod assigned to its node through the API server
6. Kubelet executes → Starts container using container runtime and reports status back
7. Kubelet reports → Updates pod status to Running and sends health information
8. API Server syncs → Updates etcd with new status
9. Controllers monitor → Continuously check that the pod is running as expected
10. Reconciliation loop continues → If pod fails, controllers create a replacement to maintain desired state

Mermaid Sequence Diagram: K8s System Flow

Here's a visualization of the interactions between components during the reconciliation process:

%%{init:{"theme":"neutral"}}%%
sequenceDiagram
    actor me
    participant apiSrv as control plane<br><br>api-server
    participant etcd as control plane<br><br>etcd datastore
    participant cntrlMgr as control plane<br><br>controller<br>manager
    participant sched as control plane<br><br>scheduler
    participant kubelet as node<br><br>kubelet
    participant container as node<br><br>container<br>runtime
    me->>apiSrv: 1. kubectl create -f pod.yaml
    apiSrv-->>etcd: 2. save new state
    cntrlMgr->>apiSrv: 3. check for changes
    sched->>apiSrv: 4. watch for unassigned pods(s)
    apiSrv->>sched: 5. notify about pod w nodename=""
    sched->>apiSrv: 6. assign pod to node
    apiSrv-->>etcd: 7. save new state
    kubelet->>apiSrv: 8. look for newly assigned pod(s)
    apiSrv->>kubelet: 9. bind pod to node
    kubelet->>container: 10. start container
    kubelet->>apiSrv: 11. update pod status
    apiSrv-->>etcd: 12. save new state

When to Use This Skill

Use this skill when you need to:

1. Deploy containerized applications to Kubernetes clusters
2. Scale applications horizontally with Horizontal Pod Autoscaling
3. Configure resource limits and requests for optimal performance
4. Implement health checks and readiness probes
5. Apply security best practices and Pod Security Standards
6. Set up network policies and RBAC controls
7. Configure Ingress for external access
8. Implement production-ready deployment patterns
9. Understand and troubleshoot reconciliation loop behavior
10. Optimize control plane component configurations

Prerequisites Validation

Before using this skill, verify your Kubernetes setup:

# Check kubectl version
kubectl version --client

# Verify cluster access and endpoints
kubectl cluster-info

# Check available nodes and their status
kubectl get nodes

# Test basic functionality
kubectl run test-pod --image=nginx --dry-run=client -o yaml

Cluster Verification Commands

Use these commands to verify cluster connectivity and health:

kubectl cluster-info

Displays the addresses of the control plane and services labeled with kubernetes.io/cluster-service=true. This command is useful for quickly getting an overview of your cluster's essential service endpoints.

# Display cluster information
kubectl cluster-info

# For more detailed debugging and diagnosis of cluster issues
kubectl cluster-info dump

# Get cluster info with specific output format
kubectl cluster-info --output wide

kubectl get nodes

Checks the status of all nodes in the cluster to ensure they are healthy and ready to accept workloads.

# List all nodes and their status
kubectl get nodes

# Get detailed information about nodes
kubectl get nodes -o wide

# Check nodes with labels
kubectl get nodes --show-labels

# Get specific node information
kubectl describe node <node-name>

# Check nodes with specific label selectors
kubectl get nodes -l <label-key>=<label-value>

# Monitor nodes in real-time
kubectl get nodes --watch

Context Management Commands

Use these commands to manage cluster contexts and kubeconfig:

Current Context Management

# Display the current context
kubectl config current-context

# List all available contexts
kubectl config get-contexts

# Switch to a specific context
kubectl config use-context <context-name>

# Get current context with namespace
kubectl config view --minify

Kubeconfig Management

# View merged kubeconfig settings
kubectl config view

# View raw kubeconfig with certificate data
kubectl config view --raw

# Get specific information using jsonpath
kubectl config view -o jsonpath='{.users[*].name}'  # List all users
kubectl config view -o jsonpath='{.clusters[*].name}'  # List all clusters
kubectl config view -o jsonpath='{.contexts[*].name}'  # List all contexts

# Set namespace for current context
kubectl config set-context --current --namespace=<namespace-name>

# Create a new context with specific cluster and user
kubectl config set-context <context-name> --cluster=<cluster-name> --user=<user-name> --namespace=<namespace-name>

# Rename current context
kubectl config rename-context <old-name> <new-name>

# Delete a context
kubectl config delete-context <context-name>

# Set cluster information
kubectl config set-cluster <cluster-name> --server=<server-url> --certificate-authority=<ca-file-path>

# Set user credentials
kubectl config set-credentials <user-name> --token=<token>  # For token-based auth
kubectl config set-credentials <user-name> --client-certificate=<cert-file> --client-key=<key-file>  # For client cert auth

# Unset configuration elements
kubectl config unset users.<user-name>
kubectl config unset clusters.<cluster-name>
kubectl config unset contexts.<context-name>

Managing Multiple Kubeconfig Files

You can work with multiple kubeconfig files by setting the KUBECONFIG environment variable:

# Append to existing KUBECONFIG
export KUBECONFIG="${KUBECONFIG}:${HOME}/.kube/config:${HOME}/.kube/additional-config"

# Use specific kubeconfig file
kubectl --kubeconfig=/path/to/kubeconfig get nodes

# Temporarily use different kubeconfig
KUBECONFIG=/path/to/alternative/config kubectl get pods

Common Context Management Scenarios

Switching Between Environments

# Switch between development and production contexts
kubectl config use-context development
kubectl config use-context production

# Create aliases for quick switching (bash/zsh)
alias kx='kubectl config use-context'
alias kn='kubectl config set-context --current --namespace'

# Example usage:
kx dev-cluster  # Switch to dev cluster
kn my-namespace  # Set namespace for current context

Troubleshooting Context Issues

# Check if current context is properly configured
kubectl cluster-info
kubectl get nodes

# If getting connection errors, verify current context
kubectl config current-context

# View detailed config for troubleshooting
kubectl config view --minify --output yaml

# Reset current context if corrupted
kubectl config use-context <valid-context-name>

Pod Manifest Structure and Resource Management

Basic Pod Manifest Structure

A Kubernetes Pod manifest follows this structure:

apiVersion: v1  # API version for Pods