Deep Dive:

Containers use two Linux kernel features to work:

• Namespaces — Give each container its own isolated view of the system (process tree, network, filesystem, users). A container can't see other containers or the host processes.
• Cgroups (Control Groups) — Limit how much CPU, memory, disk I/O, and network a container can use. Prevents one container from starving others.

Unlike Virtual Machines, containers share the host OS kernel. This makes them:

• Start in milliseconds (VMs take minutes)
• Use MBs of memory (VMs use GBs)
• Near-native performance (no hypervisor overhead)

A container image is built in layers. Each instruction in a Dockerfile (FROM, COPY, RUN) creates a layer. Layers are cached and shared — if 10 services use the same base image, that layer is stored only once.

Deep Dive:

Docker consists of several components:

• Docker CLI — Command-line interface (docker build, docker run, docker push)
• Docker Daemon (dockerd) — Background service that manages images and containers
• containerd — The actual container runtime that Docker uses internally
• Dockerfile — A text file with step-by-step instructions to build an image

# Example Dockerfile
FROM node:20-alpine          # Base image
WORKDIR /app                 # Set working directory
COPY package*.json ./        # Copy dependency files
RUN npm ci --production      # Install dependencies
COPY . .                     # Copy application code
EXPOSE 3000                  # Document the port
CMD ["node", "server.js"]    # Start command

Important distinction: Kubernetes dropped Docker as its container runtime in v1.24. But this does NOT mean Docker is dead. Docker is still the standard tool for building images. K8s just uses containerd directly to run them (cutting out the Docker daemon middleman). Your Docker-built images work perfectly on K8s.

Deep Dive:

Kubernetes was designed by Google based on 15 years of running production workloads on their internal system called Borg. Open-sourced in 2014, now maintained by the CNCF.

What Kubernetes actually does:

• Scheduling — Decides which machine runs which container based on resource needs, constraints, and policies
• Self-healing — Restarts crashed containers, replaces unresponsive Pods, kills containers failing health checks
• Scaling — Horizontally (more replicas) or vertically (more CPU/memory), automatically or manually
• Service discovery & load balancing — Gives Pods DNS names, distributes traffic
• Rolling updates & rollbacks — Deploy new versions with zero downtime, roll back if something breaks
• Secret & config management — Inject configuration and credentials without baking them into images
• Storage orchestration — Automatically attach cloud disks, NFS, or other storage to Pods

The Declarative Model — the most important concept:

You tell K8s what you want (desired state in YAML), not how to do it. K8s continuously reconciles actual state with desired state. If you say "I want 5 replicas" and one crashes, K8s creates a new one automatically. This reconciliation loop runs forever.

Deep Dive:

A cluster has two types of machines:

Control Plane Nodes (Masters):

• Run the "brain" of Kubernetes — API Server, etcd, Scheduler, Controller Manager
• Typically 3 or 5 nodes for high availability (odd number needed for etcd quorum)
• Should NOT run application workloads in production
• In managed K8s (EKS, GKE, AKS), the cloud provider manages these entirely

Worker Nodes:

• Run your actual application Pods
• Each runs: kubelet (agent), kube-proxy (networking), container runtime (containerd)
• Can be physical servers, VMs, or cloud instances
• Can be added/removed dynamically (auto-scaling)
• Can have different sizes (mix of large and small instances)

Deep Dive:

Every worker node runs three essential components:

• kubelet — The agent that communicates with the control plane. It receives Pod specifications and ensures the described containers are running and healthy.
• kube-proxy — Maintains network rules for Service routing. Implements iptables or IPVS rules.
• Container runtime — The software that actually runs containers (containerd or CRI-O).

Node lifecycle:

• Registration — Node joins the cluster and registers with the API server
• Heartbeat — kubelet sends heartbeats via Lease objects in kube-node-lease namespace
• NotReady — If heartbeats stop, the node is marked NotReady after 40s (default)
• Eviction — After 5 minutes of NotReady, Pods are rescheduled to other nodes
• Drain — kubectl drain gracefully evicts all Pods before maintenance
• Cordon — kubectl cordon marks a node as unschedulable without evicting existing Pods

Deep Dive:

Key properties:

• Shared network — All containers in a Pod share one IP address and port space. They talk to each other via localhost
• Shared storage — Containers can mount the same volumes to share files
• Co-scheduled — All containers in a Pod always run on the same node
• Ephemeral — Pods are disposable. Created, destroyed, and replaced, never "repaired"

Multi-container patterns:

• Sidecar — Helper container (Envoy proxy, log shipper, monitoring agent)
• Init Container — Runs before the main container to set up prerequisites
• Ambassador — Proxy for outbound connections
• Adapter — Transforms output (log format conversion)

Deep Dive:

Default namespaces: default, kube-system, kube-public, kube-node-lease.

What namespaces scope: Pods, Services, Deployments, ConfigMaps, Secrets, Roles, ServiceAccounts, PVCs.

What namespaces DON'T scope (cluster-wide): Nodes, PersistentVolumes, ClusterRoles, StorageClasses, Namespaces themselves.

Common strategies: per-team (team-payments), per-app (app-checkout), per-environment (staging), or hybrid.

Governance tools: ResourceQuota (cap resources), LimitRange (set defaults), NetworkPolicy (firewall), RBAC (access control).

Deep Dive:

Labels are the glue of K8s. A Service finds Pods, a Deployment manages ReplicaSets, NetworkPolicies target Pods — all through label selectors.

metadata:
  labels:
    app: payment-service
    team: payments
    env: production

Selector types: Equality-based (app = my-api) and Set-based (env in (production, staging)).

Annotations store larger metadata: build timestamps, Git SHAs, monitoring config (prometheus.io/scrape: "true").

# Viewing
kubectl get pods -A                       # All namespaces
kubectl describe pod my-pod               # Detailed info
# Debugging
kubectl logs my-pod -f                    # Stream logs
kubectl exec -it my-pod -- /bin/sh        # Shell in
kubectl debug -it my-pod --image=busybox  # Ephemeral debug
# Managing
kubectl apply -f manifest.yaml            # Create/update
kubectl rollout undo deploy/my-app        # Rollback
kubectl scale deploy my-app --replicas=5  # Scale
# Networking
kubectl port-forward svc/my-app 8080:80   # Local tunnel
# Context
kubectl config use-context prod           # Switch cluster

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: payment-api
  template:
    metadata:
      labels:
        app: payment-api
    spec:
      containers:
      - name: api
        image: payment-api:2.1.0
        resources:
          requests: { cpu: "250m", memory: "256Mi" }
          limits: { cpu: "1", memory: "1Gi" }

Rolling update: New ReplicaSet created → new Pods start → pass readiness probes → old Pods terminated. maxSurge: 1 = at most 4 total. maxUnavailable: 0 = always 3 ready.

Rollback: kubectl rollout undo deployment/payment-api (K8s keeps 10 revisions by default).

What makes it special:

• Stable Pod names — mysql-0, mysql-1, mysql-2
• Stable DNS — mysql-0.mysql-headless.default.svc.cluster.local
• Persistent storage — Each Pod gets its own PVC that survives restarts
• Ordered operations — Created 0→1→2, deleted 2→1→0

Requires a headless Service (clusterIP: None) and volumeClaimTemplates for per-Pod storage.

What runs as DaemonSets: Log collection (Fluent Bit), monitoring (Node Exporter, Datadog), network plugins (Calico, Cilium), storage drivers (CSI), security agents (Falco).

Use nodeSelector or tolerations to restrict to specific node types (e.g., GPU nodes only).

Key settings: backoffLimit (retries), activeDeadlineSeconds (timeout), concurrencyPolicy (Allow/Forbid/Replace for CronJobs), restartPolicy (Never or OnFailure).

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"           # 2 AM daily
  concurrencyPolicy: Forbid       # Skip if previous still running
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: report
            image: report-gen:1.0

Deployments create new ReplicaSets on updates and scale the old one down. Old ReplicaSets are kept (default 10) for rollback capability.

Types: ClusterIP (internal, default), NodePort (expose on node ports 30000-32767), LoadBalancer (cloud LB, costs money), ExternalName (CNAME alias), Headless (clusterIP: None, returns Pod IPs directly for StatefulSets).

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
spec:
  ingressClassName: nginx
  tls:
  - hosts: [api.myapp.com]
    secretName: tls-secret
  rules:
  - host: api.myapp.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service: { name: api-v1, port: { number: 80 } }
      - path: /
        pathType: Prefix
        backend:
          service: { name: frontend, port: { number: 80 } }

Controllers: NGINX (most popular), Traefik, HAProxy, AWS ALB, Istio Gateway.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payment-route
spec:
  parentRefs:
  - name: production-gateway
  hostnames: ["api.myapp.com"]
  rules:
  - matches:
    - path: { type: PathPrefix, value: /payments }
    backendRefs:
    - name: payment-v2
      port: 80
      weight: 90          # 90% to v2
    - name: payment-v3
      port: 80
      weight: 10          # 10% canary

DNS format: my-service.my-namespace.svc.cluster.local (or just my-service within same namespace).

kube-proxy modes: iptables (O(n), default), IPVS (O(1), better at scale), eBPF/Cilium (highest performance, replaces kube-proxy).

East-West = service-to-service within cluster. North-South = external traffic.

Major plugins:

• Cilium — eBPF-based, highest performance, built-in observability (Hubble), service mesh. The rising star. CNCF graduated
• Calico — Most widely deployed. BGP routing, excellent Network Policy support
• Flannel — Simple overlay, NO Network Policy support. Dev only
• AWS VPC CNI — Real VPC IPs on EKS. Limited by ENI capacity per node

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-frontend
spec:
  podSelector:
    matchLabels: { app: api }
  policyTypes: [Ingress]
  ingress:
  - from:
    - podSelector:
        matchLabels: { app: frontend }
    ports:
    - port: 8080

Capabilities: mTLS (zero-trust), retries/timeouts/circuit breaking, traffic splitting (canary), request-level metrics and traces, fine-grained AuthorizationPolicies.

Options: Istio (most features, CNCF graduated, Ambient mode = sidecar-less), Linkerd (simpler, Rust proxy), Cilium (eBPF-based, no sidecars).

When to adopt: 10+ microservices AND you need mandatory mTLS, advanced traffic management, or request-level observability. Don't adopt for 2-3 services.

Volume-mounted ConfigMaps support hot-reload (~60s). Env vars do NOT — need Pod restart. immutable: true for better performance at scale. 1 MB size limit.

Types: Opaque, TLS, dockerconfigjson, service-account-token, basic-auth.

Access Modes: RWO (single node), ROX (read-only many), RWX (read-write many, needs EFS/NFS), RWOP (single Pod).

Reclaim: Delete (default) or Retain (keep data). VolumeSnapshots for backups.

QoS Classes: Guaranteed (requests==limits, highest priority), Burstable (requests<limits), BestEffort (none set, first evicted).

Units: CPU: 1=1 core, 100m=0.1 core. Memory: 128Mi, 1Gi.

Hard vs Soft: requiredDuring... (must) vs preferredDuring... (try).

Effects: NoSchedule, PreferNoSchedule, NoExecute (evict existing).

v2 supports multiple metrics. behavior section controls scaling speed (scale up fast, down slow to prevent flapping). Requires Metrics Server.

# KEDA: Scale Kafka consumer to zero
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
spec:
  scaleTargetRef: { name: kafka-consumer }
  minReplicaCount: 0
  maxReplicaCount: 50
  triggers:
  - type: kafka
    metadata:
      topic: orders
      lagThreshold: "100"

Applies to voluntary disruptions (drain, autoscaler), NOT involuntary (node crash, OOM).

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: dev
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
---
kind: RoleBinding
metadata:
  name: dev-team-reader
  namespace: dev
subjects:
- kind: Group
  name: dev-team
roleRef:
  kind: Role
  name: pod-reader

Test: kubectl auth can-i create pods --namespace dev

Pod Security Admission enforces PSS per namespace via labels: enforce, audit, warn.

# Kyverno: Block :latest tag
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-tag
    match:
      any:
      - resources: { kinds: [Pod] }
    validate:
      message: "Using ':latest' is not allowed."
      pattern:
        spec:
          containers:
          - image: "!*:latest"

Key sources: kube-state-metrics (K8s object state), Node Exporter (node hardware), Metrics Server (HPA/kubectl top), app /metrics endpoints.

# PromQL examples
rate(http_requests_total{status="500"}[5m])     # Error rate
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))  # P99
sum by (service)(rate(http_requests_total[5m]))  # RPS per service

Long-term: Thanos or Grafana Mimir for multi-cluster, long-retention storage.

Loki (Grafana) = label-based, cost-effective, Grafana integration. Elasticsearch = full-text search, powerful but heavy. Loki increasingly replacing EFK.

How: Request gets trace ID → each service creates a span → propagated via headers → assembled into full trace.

Backends: Jaeger (CNCF), Grafana Tempo, Zipkin. OTel Collector = vendor-neutral pipeline.

Methods: HTTP GET, TCP Socket, gRPC, Exec command.

Four Golden Signals: Latency, Traffic, Errors, Saturation.

SLO 99.9% = 43 min downtime/month. When error budget exhausted → freeze features, fix reliability.

Pros: Atomic switchover, instant rollback. Cons: Double resources, DB schema compatibility needed.

# Argo Rollouts canary
spec:
  strategy:
    canary:
      steps:
      - setWeight: 5
      - pause: {duration: 5m}
      - analysis:
          templates: [{ templateName: success-rate }]
      - setWeight: 25
      - pause: {duration: 10m}
      - setWeight: 100

Argo CD: UI, SSO, ApplicationSets, auto-sync, self-healing. Flux: Toolkit approach, deeper Helm/Kustomize integration, no built-in UI.

Use Helm for: third-party packages, complex parameterization. Use Kustomize for: your own apps, simple environment overrides. Many teams use both.

API Server: Auth → AuthZ → Admission → Validation → Persist. Stateless, run multiple replicas.

etcd: Raft consensus, 3 or 5 nodes. Fast SSDs required. Losing etcd without backup = losing everything.

Scheduler: Filter (which nodes CAN) → Score (which is BEST). Considers resources, affinity, taints, topology.

Controller Manager: Runs Deployment, ReplicaSet, Node, Job, Endpoint controllers. Watch → Compare → Act.

Management: Cluster API (declarative lifecycle), Rancher (UI), Argo CD ApplicationSets (multi-cluster deploys).

Managed: EKS (AWS, most popular), GKE (Google, most mature), AKS (Azure, free control plane).

Self-managed: kubeadm (official), k3s (edge/IoT, lightweight), RKE2 (FIPS, government), OpenShift (enterprise).

Examples: Certificate (cert-manager), VirtualService (Istio), PostgresCluster (CloudNativePG).

A CRD alone just stores data. A Custom Controller watches it and takes action = the Operator pattern.

Major Operators: cert-manager (TLS), Prometheus Operator (monitoring), CloudNativePG (PostgreSQL), Strimzi (Kafka), Crossplane (cloud infra), External Secrets Operator (secrets sync).

Build with: Kubebuilder (Go), Operator SDK (Go/Ansible/Helm), Metacontroller (any language).

Core pattern: Watch → Compare desired vs actual → Act. Runs continuously. Idempotent.

Tools: Backstage (CNCF portal), Crossplane (cloud infra as CRDs), Tilt/Skaffold (inner-loop dev).

Tools: Kubecost, OpenCost (CNCF), Goldilocks (VPA dashboard). Chargeback via labels attributes costs to teams.

RTO = max downtime. RPO = max data loss. Lower values = higher cost (multi-region).

Patterns: Multi-AZ (minimum), Active-Active (lowest RTO), Active-Passive (simpler).

Team Topologies: Platform team (builds K8s platform), Stream-aligned (product features), Enabling (adoption help).

TCO: Compute + Storage + Networking + Licensing + People (biggest cost) + Opportunity cost.

Zero Trust stack: mTLS (service mesh), least-privilege RBAC, default-deny NetworkPolicies, PSS Restricted, short-lived credentials (Vault/IRSA), signed images only.