Actionable rules for designing, deploying, operating, and scaling containerized workloads on Kubernetes with GitOps, IaC, and CNCF–standard tooling.
Your Kubernetes manifests are scattered across wikis, Slack threads, and outdated runbooks. Your team burns hours debugging failed deployments, tracking down resource bottlenecks, and fixing security vulnerabilities that could have been prevented. Every production incident traces back to the same root cause: inconsistent, undocumented Kubernetes configurations that nobody fully understands.
Modern applications demand orchestration that scales from zero to millions of requests without manual intervention. Yet most teams are still:
The promise of Kubernetes was supposed to solve these problems, not create new ones.
These Kubernetes Container Orchestration Rules transform your cluster from a collection of YAML files into a self-healing, security-hardened, cost-optimized platform. Instead of fighting Kubernetes, you'll have a system that automatically handles the complexity while you focus on shipping features.
This isn't another generic Kubernetes tutorial. These rules codify battle-tested patterns from production environments running thousands of workloads, distilled into actionable configurations that prevent the most common failure modes.
Before: Your payment API goes down because the health check was pointing to / instead of /health, and Kubernetes kept routing traffic to failing pods for 3 minutes.
After: Every deployment includes properly configured startup, readiness, and liveness probes that catch failures in seconds:
livenessProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 3
periodSeconds: 10
Your mean time to recovery drops from minutes to seconds because Kubernetes automatically handles the failure detection and traffic routing.
Before: Your team over-provisions everything "to be safe," wasting 40% of your cloud budget, or under-provisions and suffers random OOMKills that crash customer-facing services.
After: Right-sized resource requests and limits based on actual usage patterns, with automatic scaling that prevents both waste and outages:
resources:
requests:
memory: "256Mi"
cpu: "200m"
limits:
memory: "256Mi" # equal to request for guaranteed QoS
Combined with HPA targeting 70% CPU utilization, your workloads scale precisely when needed while maintaining cost efficiency.
Before: Your cluster runs everything as root with unrestricted network access because "we'll harden it later" (but later never comes).
After: Security-by-default configurations that require zero-trust networking and least-privilege access:
Before: Production differs from staging because someone kubectl applyed a quick fix directly to the cluster, and now nobody knows what the actual state should be.
After: Every change flows through Git with automated validation, and Argo CD ensures your cluster state matches your repository with self-healing enabled.
The Old Way (3-4 hours):
The New Way (30 minutes):
helm create my-service with your standardized chart templatekubeconform and kubectl diff in CIThe Old Way:
The New Way:
1. Establish GitOps Workflow
# Install Argo CD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
2. Implement Validation Pipeline
Add to your .github/workflows/validate.yml:
- name: Validate Kubernetes manifests
run: |
kubeconform -strict -summary manifests/*.yaml
kubectl diff -f manifests/
3. Set Resource Policies Deploy pod disruption budgets and resource quotas to prevent resource exhaustion.
1. Enable Pod Security Standards
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
2. Implement Network Policies Start with deny-all and whitelist required communications.
3. External Secrets Integration Replace Kubernetes Secrets with AWS Secrets Manager or HashiCorp Vault integration.
1. Deploy Monitoring Stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus prometheus-community/kube-prometheus-stack
2. Configure Cost Monitoring Install Kubecost for resource usage visibility and budget alerts.
3. Enable Autoscaling Deploy HPA and VPA for automatic resource optimization.
1. Service Mesh Integration Implement Istio for advanced traffic management and security policies.
2. Advanced Deployment Strategies Set up Argo Rollouts for canary and blue-green deployments.
3. Multi-Cluster Management Extend patterns to staging and development clusters.
Your team stops dreading Kubernetes deployments and starts treating them as routine operations. New engineers can contribute to infrastructure within days instead of months because everything is documented in code and follows consistent patterns.
Once you've implemented these foundational rules, you'll be ready for advanced patterns:
These rules aren't just configurations—they're the foundation for a platform that scales with your team and business. Start with the validation pipeline and GitOps workflow, then gradually implement security hardening and observability. Within a month, you'll have transformed from Kubernetes survivors to Kubernetes masters.
Your production environment will thank you, your team will thank you, and your on-call rotation will finally get some sleep.
You are an expert in Kubernetes, Docker, Helm, Istio, Argo CD, Prometheus/Grafana, Terraform, and GitOps workflows.
# Key Principles
- Treat the **cluster as cattle**, not pets—everything is reproducible from code.
- Prefer **declarative manifests** stored in Git; never mutate resources manually in production.
- Apply **least-privilege** security (RBAC, network policies, PodSecurity standards).
- Keep workloads **stateless by default**; state lives in managed services or CSI volumes.
- Design for **horizontal scalability** first; vertical only when unavoidable.
- Automate everything: provisioning, deployments, policy checks, and roll-backs.
- Observe, measure, and improve: metrics, logs, traces, cost, and security.
# YAML / Kubernetes Manifests
- File naming: `<workload>-<env>.yaml` (e.g., `payment-api-prod.yaml`).
- One **kind** per file; order: `apiVersion`, `kind`, `metadata`, `spec`.
- Use two-space indentation; avoid tabs.
- All resource values use SI units (`500Mi`, `200m`).
- Always specify `metadata.name`, `metadata.namespace`, and descriptive labels:
```yaml
metadata:
name: payment-api
namespace: prod
labels:
app.kubernetes.io/name: payment-api
app.kubernetes.io/component: backend
app.kubernetes.io/environment: prod
```
- Keep comments actionable; remove obsolete TODOs before merging to `main`.
- Helm: values keys are lower-kebab-case, defaulted to sane minimums.
# Error Handling & Validation
- **Readiness/Liveness/Startup probes**: required for every Deployment, StatefulSet, Job >5 min run-time.
```yaml
livenessProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 3
periodSeconds: 10
```
- Validate manifests with `kubeconform` (strict), then `kubectl diff` in CI.
- Enable **pod-disruption-budgets (PDB)** for HA workloads: `minAvailable >= 50%`.
- Detect unhealthy nodes/pods via Prometheus alerts (`KubeNodeNotReady`, `KubePodCrashLooping`).
- Use `Retry` & `CircuitBreaker` at service mesh level (Istio) instead of app layer retries when possible.
# Language-Specific Rules — Container Specs
- Base images: `scratch` or `distroless` when possible; multi-stage builds to shrink image <100 MB.
- Tag images immutably (`<git-sha>`), never `latest` in production.
- Set **resources**:
```yaml
resources:
requests:
memory: "256Mi"
cpu: "200m"
limits:
memory: "256Mi" # equal to request for guaranteed QoS
```
- Use `envFrom` + Secrets/ConfigMaps; never hard-code credentials.
# Framework-Specific Rules
## Kubernetes Core
- Namespaces map to lifecycle environments (`dev`, `stg`, `prod`) or teams.
- Enforce **RBAC**: service accounts per app, roles per namespace, cluster roles only for infra.
- NetworkPolicies: default **deny-all**; whitelist ingress/egress per app.
- Storage: define `StorageClass` per performance tier; RWX only when needed.
- Deployments: always use `strategy.rollingUpdate` with `maxUnavailable: 25%`, `maxSurge: 25%`.
- StatefulSets: set `podManagementPolicy: Parallel` for fast bootstrap when ordering unneeded.
- CronJobs: `concurrencyPolicy: Forbid` unless duplicates are safe.
## Helm Charts
- Keep chart version aligned to app semver (`appVersion`); bump chart minor on values schema change.
- Expose resource, HPA, and ingress configs in `values.yaml`; everything else opinionated.
- Lint with `helm lint --strict` + chart-testing in PR.
## Istio (Service Mesh)
- Use `PeerAuthentication` with `mtls: STRICT` cluster-wide.
- Define `DestinationRule` + `VirtualService` per service; canary via `subset` traffic splits.
- Enable `EnvoyFilter` for rate limiting only when gateway cannot.
## Argo CD / GitOps
- One application per namespace; auto-sync with `prune: true` and `selfHeal: true`.
- Use `app.kubernetes.io/managed-by=argocd` label for traceability.
# Testing & CI/CD
- PR Pipeline order: `yamllint` ➜ `kubeconform` ➜ Helm template dry-run ➜ `kubectl kustomize` ➜ integration tests ➜ Docker build ➜ push ➜ Argo CD auto-deploy to `dev`.
- Use Kind or k3d for local smoke tests.
- Canary or blue/green via Argo Rollouts with success metrics (`p95_latency < 300 ms`, `error_rate < 1%`).
# Performance & Scaling
- **Horizontal Pod Autoscaler**: target 70% CPU or custom Prometheus metrics; min 2 replicas.
- **Vertical Pod Autoscaler** in “recommendation” mode first; switch to “auto” after 2 weeks stable.
- Enable **Cluster Autoscaler** or Karpenter with node pools `{memory, cpu, arch}`-optimized.
- Run `kubecost` for cost allocation; alert when namespace > budget by 10%.
# Security
- Require `runAsNonRoot: true`, `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`.
- Scan images with Trivy on every push and in-cluster via `trivy-operator`.
- Enforce admission controls via Kyverno/Opa Gatekeeper: prohibit plaintext Secrets, require labels, resource limits.
- Store secrets in external manager (AWS Secrets Manager, Vault) and sync via CSI Secrets Store.
- Patch clusters monthly; operate on **N-1** version at most.
# Observability
- Prometheus Operator + Grafana dashboards: node, pod, container, ingress, mesh.
- Loki + Promtail for logs; retain 30 days.
- Jaeger for traces; sample 1% prod, 100% dev.
- Alerting rules: severity labels (`critical`, `warning`) and SLO-based alerts only.
# Infrastructure as Code (Terraform)
- Define kubernetes cluster, node pools, and IAM via Terraform modules.
- Separate state files per environment; store in remote backend with locking (e.g., S3 + DynamoDB).
- Use `terraform validate`, `tflint`, `tfsec` in CI.
# Common Pitfalls & How to Avoid Them
- Setting CPU limits low causes throttling ➜ set **requests**, avoid limits unless multi-tenant burst prevention.
- Missing probes leads to stuck pods ➜ always define health checks.
- Pulling large images delays scaling ➜ use minimal images and image caching layer.
- Over-granting cluster-admin roles ➜ restrict RBAC, audit regularly.
# Directory Layout (mono-repo example)
```
├─ apps/
│ ├─ payment-api/
│ │ ├─ chart/ # Helm chart
│ │ ├─ overlays/
│ │ │ ├─ dev/
│ │ │ └─ prod/
│ │ └─ Dockerfile
├─ infra/
│ ├─ terraform/
│ └─ argocd-apps/
└─ .github/workflows/
```
# Summary Checklist (pre-merge)
- [ ] Manifests validated & linted
- [ ] Resources right-sized
- [ ] Probes defined
- [ ] RBAC least privilege
- [ ] NetworkPolicy applied
- [ ] Secrets externalized
- [ ] Images scanned & immutable tags
- [ ] CI pipeline green
- [ ] Argo CD sync OK in `dev` environment