Actionable rules for designing, coding, deploying and operating secure, container-native microservices.
You're tired of microservices that work perfectly in development but crumble under production load. Services that can't scale, can't be monitored, and turn your deployment pipeline into a nightmare. This comprehensive ruleset transforms how you architect, code, and deploy microservices—creating systems that actually survive real-world conditions.
Most microservices implementations fail spectacularly in production. You've probably seen these patterns:
These aren't edge cases—they're the norm when microservices lack proper architectural discipline.
This ruleset gives you battle-tested patterns for building microservices that actually work in production. You get comprehensive coverage across the entire stack: Java/Spring Boot, Node.js, Python, Docker, Kubernetes, service mesh, and observability.
What makes this different: Every rule is designed for real production environments with proper security, monitoring, and operational requirements built-in from day one.
# Before: Service updates break consumers
- Update service code
- Deploy and hope nothing breaks
- Fix integration issues in production
# After: Contract-first development
- Define API changes in OpenAPI spec
- Run contract tests to verify compatibility
- Deploy with confidence knowing integrations work
// Every request gets end-to-end traceability
{
"timestamp": "2024-01-15T10:30:45.123Z",
"service": "billing-service",
"traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
"spanId": "00f067aa0ba902b7",
"level": "INFO",
"message": "Invoice created successfully",
"userId": "user-123",
"invoiceId": "inv-456"
}
# Automated canary deployment with SLO monitoring
helm upgrade billing-service --set image.tag=abc123 --set traffic.canary=10
# System automatically promotes or rolls back based on error rates
Java/Spring Boot Services:
// Standard controller pattern with proper validation
@RestController
@RequestMapping("/invoices")
public class InvoiceController {
private final InvoiceService service;
public InvoiceController(InvoiceService service) {
this.service = service;
}
@PostMapping
public ResponseEntity<InvoiceDto> create(
@Valid @RequestBody CreateInvoiceDto req,
@RequestHeader("Idempotency-Key") String idemKey) {
return ResponseEntity.status(HttpStatus.CREATED)
.body(service.create(req, idemKey));
}
}
Container Configuration:
# Multi-stage build with distroless final image
FROM eclipse-temurin:17-jdk-alpine AS build
WORKDIR /app
COPY . .
RUN ./gradlew bootJar
FROM gcr.io/distroless/java17:nonroot
COPY --from=build /app/build/libs/*.jar /app.jar
HEALTHCHECK --interval=30s --timeout=3s --start-period=60s \
CMD curl -f http://localhost:8080/actuator/health || exit 1
ENTRYPOINT ["java", "-jar", "/app.jar"]
# Production-ready deployment with proper resource limits
apiVersion: apps/v1
kind: Deployment
metadata:
name: billing-service
labels:
app.kubernetes.io/name: billing-service
version: v1.2.3
spec:
replicas: 3
template:
spec:
containers:
- name: billing-service
image: ghcr.io/company/billing-service:abc123
resources:
requests:
cpu: 200m
memory: 512Mi
limits:
cpu: 1000m
memory: 1Gi
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
initialDelaySeconds: 30
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
periodSeconds: 5
# Istio configuration for secure service communication
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
spec:
mtls:
mode: STRICT
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: billing-service
spec:
http:
- match:
- headers:
canary:
exact: "true"
route:
- destination:
host: billing-service
subset: canary
weight: 100
- route:
- destination:
host: billing-service
subset: stable
weight: 90
- destination:
host: billing-service
subset: canary
weight: 10
# GitHub Actions workflow for automated deployment
name: Deploy Microservice
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Run Tests
run: |
./gradlew test
./gradlew contractTest
- name: Security Scan
run: ./gradlew dependencyCheckAnalyze
- name: Build and Push
run: |
docker build -t ghcr.io/company/billing-service:${{ github.sha }} .
docker push ghcr.io/company/billing-service:${{ github.sha }}
- name: Deploy Canary
run: |
helm upgrade billing-service ./charts/billing-service \
--set image.tag=${{ github.sha }} \
--set traffic.canary=10
Your microservices architecture doesn't have to be fragile. These rules give you the foundation for services that scale, deploy reliably, and operate smoothly in production. Stop fighting fires and start building systems that work.
You are an expert in container-native, cloud-first microservices built with Java (Spring Boot), Node.js and Python, packaged in Docker, orchestrated by Kubernetes and serviced by Istio/Linkerd in AWS.
Key Principles
- Single-responsibility services: one bounded business capability per service.
- Stateless at runtime; state is externalized (DB, cache, object store).
- Contract first: APIs defined in OpenAPI/Swagger, committed before code.
- Secure-by-design: threat-model every service; implement least-privilege IAM.
- Automate everything: Git-centric trunk-based development, 100 % IaC.
- Observability is mandatory: exhaustive logging, metrics and traces emitted as first-class features.
- Fail fast & heal fast: health probes, circuit breakers, retries, bulkheads.
Java (Spring Boot)
- Use Spring Boot 3 (Java 17+) with spring-cloud-starter.
- Package each service as a bootable fat JAR, then containerise using multi-stage Docker build (jlink → distroless). No JPA entities in API layer.
- Naming: *<domain>-service* artifactId, package root *com.company.<domain>*.
- Use `@RestController` + constructor injection; prohibit `@Autowired` on fields.
- Expose DTOs (record) separate from JPA entities. Map with MapStruct.
- Validation: annotate DTOs with `javax.validation` and trigger via `@Valid`.
- Use Reactor/Project Loom virtual threads for non-blocking I/O.
- Disable default Tomcat session; enforce stateless JWT auth filter.
Node.js (TypeScript)
- Use ES2022 + ECMAScript modules.
- Structure: `src/handlers`, `src/routes`, `src/services`, `src/lib`.
- Enforce `strictNullChecks` & `noImplicitAny`. Interface contracts mirror OpenAPI.
- Always return Promises; top-level `await` in ESM modules.
- Never mutate `req` or `res` objects outside middleware.
Python (FastAPI)
- Python 3.11+, `pydantic` models for request/response schemas.
- Dependency-inject via FastAPI’s `Depends`. Avoid global state.
- Uvicorn with `--workers $CPU` and `--loop uvloop`.
Error Handling and Validation
- Validate inbound data at the edge (controller/handler). Reject known-bad fast.
- Standard error envelope:
```json
{"errorId":"uuid","code":"<UPPER_SNAKE>","message":"<user-friendly>","details":{}}
```
- Throw typed exceptions (`NotFoundException`, `DomainViolation`) that map to HTTP codes via global exception mapper.
- Never leak stack traces; attach `errorId` for correlation.
- Use idempotency keys on POST endpoints that create resources.
Framework-Specific Rules
Spring Cloud / Netflix OSS
- Use Spring Cloud LoadBalancer + Kubernetes service DNS for discovery; no Eureka in-cluster.
- Resilience4j for circuit breaking, retries, rate limiting.
- Externalize configuration via Spring Cloud Config → backed by Git.
Istio / Linkerd (Service Mesh)
- mTLS STRICT globally; declare `PeerAuthentication` per-namespace when downgrading.
- Use `VirtualService` for canary (weight-based) and `DestinationRule` for subsets.
- Propagate `x-request-id`, `traceparent`, `baggage` headers end-to-end.
Docker & Kubernetes
- One service → one container → one Helm chart.
- Base images: distroless or official `*-slim` only, pinned digest.
- HEALTHCHECK in Dockerfile mapping to `/actuator/health` or `/health`.
- Resource requests: CPU = 100 m + (0.1 × peak), Memory = 1.5× heap.
- Readiness probe must test downstream dependencies.
- Use `Deployment` + `HPA` (CPU & custom latency metric).
- Labels: `app.kubernetes.io/name`, `version`, `env` (dev|stg|prd).
Security
- Centralized OIDC identity provider issues short-lived JWTs; validate with JWKS.
- Use OAuth 2.0 client-credentials for service-to-service; scope = service name.
- Secrets via Kubernetes Secrets sealed by SealedSecrets or AWS Secrets Manager.
- Enforce TLS 1.2+ everywhere, disable plaintext NodePort.
- Perform dependency scanning (OWASP Dependency-Check, npm audit, pip-audit) in CI.
CI/CD
- Trigger pipeline on merge to `main`; all artefacts versioned with Git SHA.
- Stages: lint → unit tests → contract tests → SCA → build → integration tests → security tests → push → deploy to dev → smoke → promote.
- Use Helm Chart Museum + `helm diff` for pull-request previews.
- Canary deploy: 10 % traffic for 15 min with SLO error budget ≤ 0.1 %.
- Rollback policy: `kubectl rollout undo` automated via pipeline on SLO breach.
Testing
- Unit: ≥ 90 % coverage; mock network boundaries.
- Contract: Pact V4 between consumer and provider, verified in pipeline.
- Integration: docker-compose or Testcontainers spin-up real deps.
- End-to-end: Cypress/Postman collection hitting staging.
- Chaos: inject latency & pod kills with Litmus/Gremlin weekly.
Observability & Performance
- Logs: JSON lines, RFC3339 timestamps, service, version, traceId.
- Metrics: RED method (Rate/Errors/Duration) per endpoint, exported via Prometheus.
- Traces: OpenTelemetry SDK; sample 100 % on staging, 5 % on prod.
- Alerting: 3 Golden Signals (latency, error rate, saturation) with SLOs.
- JFR (Java), Clinic.js (Node) & Py-spy (Python) profiles in canary.
Naming Conventions
- Repos: `<domain>-service` (e.g., billing-service).
- Branches: `feat/<ticket>`, `fix/<ticket>`, `ops/<infra>`.
- Docker images: `ghcr.io/org/<service>:<sha>`.
- Kubernetes namespaces: `<team>-<env>`.
Common Pitfalls & Guards
- Do not share database schemas between services; use events for sync.
- Avoid synchronous request chains > 3 hops; prefer async messaging (Kafka/SQS).
- Never cache JWTs; rely on expiry and refresh flows.
- Ensure graceful shutdown: 30 s `preStop` sleep; catch SIGTERM and drain.
Ready-to-Use Examples
- Spring Boot controller skeleton:
```java
@RestController
@RequestMapping("/invoices")
public class InvoiceController {
private final InvoiceService service;
public InvoiceController(InvoiceService service) { this.service = service; }
@PostMapping
public ResponseEntity<InvoiceDto> create(@Valid @RequestBody CreateInvoiceDto req,
@RequestHeader("Idempotency-Key") String idemKey) {
return ResponseEntity.status(HttpStatus.CREATED)
.body(service.create(req, idemKey));
}
}
```
- Kubernetes readiness probe:
```yaml
livenessProbe:
httpGet: { path: "/actuator/health/liveness", port: 8080 }
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet: { path: "/actuator/health/readiness", port: 8080 }
periodSeconds: 5
```
Follow these rules to deliver scalable, secure, and maintainable microservices that can be confidently operated in production.