Chaos Engineering Architecture Diagrams: Resilience Testing Patterns (2026)
How to diagram chaos engineering systems: experiment lifecycle, blast radius controls, LitmusChaos and AWS FIS architectures, GameDay pipelines, and steady-state hypothesis testing — with AI prompt templates.
R
Ryan·Senior AI Engineer
Chaos engineering architecture diagrams document one of the most counterintuitive practices in platform engineering: deliberately injecting failures to find weaknesses before production incidents do. Netflix popularized chaos engineering with Chaos Monkey in 2011; in 2026 it is a standard discipline at hyperscalers, fintech platforms, and any organization with SLO commitments. Architecture diagrams for chaos engineering must show the full experiment lifecycle — hypothesis definition, blast radius scoping, fault injection, steady-state monitoring, and automatic abort — not just "we killed some pods." This guide covers how to diagram chaos engineering platforms and the architectural patterns that make resilience testing safe and repeatable.
The chaos experiment lifecycle
Every chaos engineering architecture diagram should show the experiment lifecycle as a control loop. The five phases map to distinct architectural components:
- 1. Steady-state hypothesis: Define what "normal" looks like before injecting any fault. Steady-state metrics are typically SLO signals: p99 latency < 200ms, error rate < 0.1%, success rate > 99.9%. These become the abort thresholds. Diagram the hypothesis as a pre-condition check against your observability stack (Datadog, Prometheus, Grafana) before the experiment starts.
- 2. Blast radius definition: Scope the experiment to a specific namespace, deployment, AZ, or percentage of traffic. Never run chaos experiments without explicit blast radius controls. Diagram the blast radius selector as a separate component that gates the fault injector — the injector cannot proceed without a valid scope.
- 3. Fault injection: The actual fault: pod kill, CPU stress, memory pressure, network partition, disk full, DNS failure, latency injection. Each fault type requires a different injection mechanism — shown as distinct components in the architecture (chaos agent, OS-level stress tool, iptables rule, tc-netem).
- 4. Continuous monitoring: During the experiment, a monitoring loop compares live metrics against the steady-state thresholds. If any SLO breaches, the automatic abort fires. Show this as a feedback loop with a separate abort controller component.
- 5. Rollback and cleanup: Whether the experiment completes normally or aborts, the rollback phase must be shown: chaos agent removed, iptables rules flushed, pod count restored. Diagram the rollback path as guaranteed — it runs even if the experiment controller crashes (dead-man's switch pattern).
Core chaos engineering platforms and their architectures
- LitmusChaos (CNCF graduated): The de-facto Kubernetes-native chaos platform. Architecture: ChaosCenter (control plane, React UI + GraphQL API) → ChaosOperator (watches ChaosEngine CRDs) → ChaosRunner (Kubernetes Job) → ChaosExperiment (pulls fault injection logic from ChaosHub). Agents deployed per cluster; ChaosCenter can manage multiple clusters from a single control plane.
- AWS Fault Injection Service (FIS): Managed chaos platform for AWS workloads. FIS experiment templates define: target resources (by tag, ARN, filter), actions (aws:ec2:terminate-instances, aws:ecs:stop-task, aws:rds:failover-db-cluster, aws:network:disrupt-connectivity), stop conditions (CloudWatch alarm). No agents required — FIS uses IAM permissions to invoke AWS APIs directly. Architecture shows the IAM role trust relationship as the blast radius control.
- Gremlin: Commercial chaos platform with agents deployable as Kubernetes DaemonSets, EC2 packages, or Docker containers. Offers CPU attack, memory attack, packet loss, latency, DNS failure, shutdown, disk fill. Architecture: Gremlin Control Plane (SaaS) → Gremlin Daemon (agent on each node) ← Team API credentials. Blast radius controlled via target tags and team-level permissions.
- Netflix Chaos Monkey (open source): The original — randomly terminates VM instances in production during business hours. Modern implementation uses Spinnaker integration. Architecture is intentionally simple: a scheduler that calls AWS EC2 TerminateInstances on randomly selected instances in ASGs tagged for chaos. Its simplicity is a design statement — if your system cannot survive random instance termination, it is not resilient.
- Chaos Mesh: CNCF chaos platform with broad fault types including pod faults, network faults (bandwidth limiting, packet corruption, DNS errors), stress testing, kernel faults, and time skew injection. Architecture: Chaos Dashboard → Chaos Controller Manager → CRDs (PodChaos, NetworkChaos, StressChaos) → chaos-daemon DaemonSet on each node.
Prompt examples for chaos engineering architecture diagrams
LitmusChaos multi-cluster architecture
"LitmusChaos architecture for a multi-cluster Kubernetes platform. ChaosCenter (control plane): deployed in management cluster (EKS), consists of auth server, GraphQL server, frontend (React), MongoDB (Atlas). Two target clusters (EKS us-east-1 prod, EKS eu-west-1 prod), each with: ChaosOperator (watches ChaosEngine CRDs), chaos-agent (connects back to ChaosCenter via WebSocket, no inbound port needed). Experiment flow: SRE defines ChaosEngine CR via ChaosCenter UI → ChaosOperator creates ChaosRunner Job → Job pulls ChaosExperiment from ChaosHub (GitHub-backed experiment library) → injects fault (pod-delete: kills 50% of pods in payment-service namespace) → monitors Prometheus steady-state: error_rate < 0.5% → auto-abort if threshold exceeded → cleanup. Show approval gate: experiments targeting prod require Slack bot approval (Litmus Slack integration) before execution. Annotate blast radius scoping: namespace selector = {payment-service}, pod label = {app: payment-api}, percentage = 50."
AWS FIS experiment for AZ failure simulation
"AWS Fault Injection Service experiment architecture for AZ failure simulation on a multi-AZ e-commerce platform. System under test: ECS Fargate services (api, checkout, inventory) across 3 AZs (us-east-1a/b/c), RDS Aurora Multi-AZ, ElastiCache Redis cluster mode, ALB. FIS experiment template: target = EC2 instances in AZ us-east-1b tagged {chaos-eligible: true}. Actions: (1) aws:ec2:stop-instances on all us-east-1b instances, (2) aws:ecs:stop-task for all tasks in us-east-1b, (3) aws:elasticloadbalancing:deregister-targets for us-east-1b targets. Duration: 10 minutes. Stop conditions: CloudWatch alarm {error_rate_high} (error rate > 1%). IAM role: FISRole with policies: ec2:StopInstances, ecs:StopTask, elasticloadbalancing:DeregisterTargets scoped to tagged resources only. Expected outcome: ALB routes to remaining 2 AZs within 30 seconds, RDS Aurora fails over to standby in us-east-1c within 60 seconds. Monitoring: Datadog dashboard pinned to experiment window showing 5xx rate, checkout success rate, DB connection pool."
Network partition chaos with Chaos Mesh
"Chaos Mesh network partition experiment for a microservices payment system. Services: payment-api → fraud-check → ledger → notification. Experiment: inject a network partition between payment-api and fraud-check (NetworkChaos CR: action: partition, direction: both, target: pods with label app=fraud-check). Expected behavior: payment-api should use circuit breaker (Resilience4j, threshold: 5 failures in 10s → open circuit) and fall back to async fraud check via SQS queue. Steady-state hypothesis: payment success rate > 95% even during fraud-check partition (async fallback). Chaos Mesh architecture: Chaos Dashboard (frontend) + Chaos Controller Manager (watches NetworkChaos CRDs) + chaos-daemon DaemonSet (uses tc-netem to inject network rules on target pod veth interfaces). Show the network path: chaos-daemon adds iptables DROP rule on the pod's network namespace for traffic to fraud-check pods. Abort condition: payment success rate drops below 90% (Prometheus alert). Experiment duration: 5 minutes."
GameDay pipeline with automated chaos in CI/CD
"Automated chaos engineering pipeline integrated with CI/CD for a SaaS platform. Pipeline stages in GitHub Actions: (1) Deploy to staging environment (ArgoCD sync), (2) Run smoke tests (k6 load test for 5 minutes at 50% prod traffic), (3) Trigger chaos experiment via LitmusChaos API (pod-delete on 30% of api pods, duration 2 minutes), (4) k6 continues running during chaos — chaos test passes if p99 latency stays under 300ms and error rate stays under 0.5%, (5) If chaos test fails → notify Slack #platform-reliability channel, block prod deploy, open Jira incident ticket automatically, (6) If chaos test passes → deploy to production. Show: GitHub Actions workflow calling LitmusChaos GraphQL API to create ChaosEngine CR, polling experiment status, fetching results. Annotate: chaos runs only on staging, not prod, to enable fast iteration. Blast radius: staging namespace, labeled pods only. Add weekly GameDay trigger: GitHub Actions cron job every Tuesday at 10 AM UTC runs a more severe experiment (full AZ simulation via FIS on staging AWS account)."
Chaos engineering fault types and injection methods
| Fault type | Injection mechanism | What it tests | Tool support |
|---|---|---|---|
| Pod / instance kill | kubectl delete pod, EC2 terminate | Auto-healing, restart policies | Chaos Monkey, LitmusChaos, FIS |
| Network latency | tc-netem (100–500ms delay) | Timeouts, circuit breakers | Chaos Mesh, Gremlin, tc |
| Network partition | iptables DROP rules on veth | Fallbacks, async queues | Chaos Mesh, Pumba, FIS |
| CPU stress | stress-ng, cgroups | Noisy neighbor, resource limits | Gremlin, LitmusChaos StressChaos |
| DNS failure | CoreDNS rule injection | DNS caching, retry logic | Chaos Mesh DNSChaos, Gremlin |
| AZ failure | Stop all instances/tasks in AZ | Multi-AZ failover, RDS replica | AWS FIS, manual scripting |
Architecture checklist for safe chaos engineering
- Blast radius controls: Every chaos platform must have explicit scope limiters — namespace selectors, resource tags, percentage caps. Diagram these as mandatory gates, not optional parameters. An experiment without blast radius controls is an incident.
- Automatic abort conditions: Show the monitoring feedback loop explicitly. Abort conditions should trigger on SLO signals, not just on experiment timeout. The abort must roll back all injected faults immediately.
- Dead-man's switch: If the chaos controller itself crashes during an experiment, a dead-man's switch must clean up. Diagram this as a separate watchdog process that reverts injected faults if it loses heartbeat from the controller for >30 seconds.
- Pre-experiment checklist: Show the pre-flight check as a formal gate: on-call engineer acknowledged, no active incidents, relevant stakeholders notified, experiment is scoped to non-prod or a chaos-eligible subset of prod.
- Findings tracking: Chaos engineering produces findings — weaknesses in retry logic, missing circuit breakers, insufficient timeouts. Show the feedback loop from experiment results to a findings tracker (Jira, Linear) and into the next sprint's backlog. A chaos program without findings tracking is theater.
Related guides: disaster recovery architecture diagrams, observability architecture diagrams, Kubernetes architecture diagrams, SRE architecture diagrams, and microservices architecture diagrams.
Ready to try it yourself?
Start Creating - Free