Terraform composition
environment→modules→AWS resources
The environment folder is the composition root. It wires module outputs and variables together for dev, staging, or prod.
Visual infrastructure guide
terraform-platform-infra-live
This repo is the live Terraform composition for the Enterprise Data Platform. It builds the AWS network, lake storage, IAM, processing, orchestration, serving, analytics backend, frontends, and monitoring resources for each environment.
Module dependency map
What Terraform builds, and why order matters
Foundation firstNetworking, buckets, KMS, IAM roles, and Glue databases are shared prerequisites. Most later modules cannot even be planned correctly without their outputs.
Default dev pathProcessing, Step Functions, Analytics Agent, and Monitoring are the normal short-session setup. It keeps the platform usable without running every expensive component.
Optional modulesIngestion, Redshift serving, and MWAA are enabled when needed. They are dashed because they are intentionally not always on in dev.
Network topology
What lives in public vs private subnets
Public means edge onlyThe ALB receives browser traffic. The NAT Gateway lets private workloads call AWS APIs or Claude, but it does not allow inbound access.
Private means protected computeECS, Glue, RDS, DMS, and MWAA live in private subnets. They communicate through VPC routes, security groups, IAM, and S3 endpoints.
S3 endpoint reduces exposureGlue, DMS, and ECS can reach S3 over the AWS private network instead of crossing the public internet.
Data and serving architecture
How infrastructure supports the full analytics platform
Infrastructure is the stageThis repo does not generate data or run analytics code itself. It creates the AWS services those repos need to run safely.
Two orchestration choicesStep Functions is the default fast path. MWAA is the optional Airflow UI path for visual DAG demonstrations.
Serving boundaryThe infrastructure creates ECS, ALB, ECR, and monitoring for the analytics service; the detailed frontend behavior belongs in the analytics-agent repo.
Session modes
What is active in dev and why
Default is practicalMost sessions only need the foundation, Step Functions, agent serving, and monitoring. That keeps startup faster and cost lower.
Ingestion is specialRDS and DMS cost money while running, so ingestion is only enabled when you need fresh CDC from PostgreSQL.
MWAA is for visual orchestrationAirflow is useful for demos and DAG inspection, but Step Functions is the lean default pipeline path.
Concept map
The infrastructure ideas this repo teaches
Least privilege by layer
KMS→IAM roles→scoped buckets
Roles are created for the service that needs them. Glue, DMS, MWAA, Redshift, and ECS do not all get the same permissions.
Private compute
private subnets→S3 endpoint→NAT outbound
The platform exposes only what must be public. Most services have no public IP and reach S3 privately.
Orchestration modes
Step FunctionsorMWAA
Both paths run the same data pipeline. Step Functions is lean; MWAA provides the Airflow UI and DAG graph.
Serving choices
FastAPI→StreamlitHTMLSlack
The frontends should be presentation layers. The Analytics Agent backend remains the governed query brain.
Observability
logs→metrics→alarms
The monitoring module makes failures visible: pipeline errors, ECS health, ALB problems, and stale Silver data.
Environment mental model
Same code, different environment values
dev
- Fast iteration and demos
- Short-lived sessions to save cost
- Ingestion and MWAA enabled only when needed
staging
- Production-like validation
- Stronger availability settings
- Useful before promoting infra changes
prod
- Same module structure
- Production-grade variables
- Designed to avoid console-created drift