EDP platform-analytics-web
Enterprise data platform walkthrough

From source changes to governed analytics answers.

This page explains the full public demo: controlled deployment, OLTP changes, CDC into the lake, Bronze to Silver to Gold modeling, orchestration, stakeholder access, and the Analytics Agent request loop. The HTML dashboard is one entry point into that wider platform.

Terraform session CDC pipeline Bronze Silver Gold dbt marts Claude on AWS Stakeholder apps
Frontend position

The HTML dashboard is a peer frontend, not a fork of the agent.

Three entry points, one analytics backend The frontend can change independently because the backend API contract stays stable. Streamlit UI agent repo ↗ HTML dashboard web repo ↗ custom UX surface Slack MCP gateway repo ↗ FastAPI Agent analytics-agent repo ↗ guardrails, charts, reports Claude API SQL + insight Athena queries Gold S3 Audit logs + artifacts Gold S3 dbt marts same API
Separate frontendThis repo changes the browser experience without editing the Python agent service.
Shared backendStreamlit, HTML, and Slack all rely on the same governed SQL and audit behavior.
Demo valueThe HTML surface is easier to polish for stakeholders because layout, colors, tables, and exports are fully controlled.
Platform Topology

Frontend surfaces, analytics ownership, and governed Gold access.

Platform topology

HTML, Streamlit, and Slack hand off to one analytics service, which reads curated Gold marts through Athena.

Topology

One governed analytics core

Three entry points share one control point for SQL, audit evidence, and governed Gold access.
HTML Streamlit Slack
Request flow

The browser stays same-origin while the server proxies API calls.

HTML dashboard runtime path The Node server serves static assets and forwards `/api/*` to the existing backend, stripping the `/api` prefix. Browser ALB HTML service FastAPI Claude/Athena Gold/Audit 1. Open app 2. Serve index.html, CSS, JS 3. Dashboard renders in browser 4. POST /api/ask/stream 5. Proxy to /ask/stream 6. SQL, insight, verdict 7. Query and audit 8. Render insight, KPI cards, chart, table, SQL, details, CSV/PDF exports
Same-origin browserThe browser talks to `/api/*` on the same host, so the UI avoids CORS changes in the backend.
Thin serverThe Node server serves files, proxies requests, and exposes `/healthz`; business logic remains in FastAPI.
Rich client renderingThe JavaScript layer formats KPIs, tables, charts, CSV downloads, PDFs, session history, and engineer logs.
API contract

The frontend is designed around the existing agent endpoints.

Frontend contract with the analytics agent The UI does not invent new backend semantics. It presents the existing API results in a more controlled stakeholder layout. Browser actions check backend status ask business question download log / PDF / CSV Existing FastAPI endpoints GET /health GET /examples POST /ask/stream POST /report/pdf GET /engineer-log Rendered outputs health badge and session stats insight, KPIs, chart, table SQL, verdict, cost, exports
Contract firstThe frontend is replaceable because it depends on documented API endpoints, not internal Python objects.
Presentation layerKPI extraction and CSV creation happen in the browser from returned rows and columns.
Audit visibilityEngineer logs and details remain accessible for troubleshooting without exposing implementation complexity up front.
Deployment path

The deploy workflow builds a container and updates a separate ECS service.

Code-to-ECS deployment The workflow authenticates with GitHub OIDC, pushes a Node image to ECR, then updates the analytics-web ECS service. Manual deployweb repo ↗ OIDC roleinfra repo ↗ Docker buildweb repo ↗ Amazon ECRsha + latest tags Task definitioninfra repo ↗ ECS serviceinfra repo ↗ ALBinfra ↗
Separate serviceThe HTML dashboard can be deployed independently from the existing agent service and Streamlit UI.
Small runtimeThe container runs Node, serves `src/`, and proxies `/api/*` to `BACKEND_URL`.
Rollback shapeBecause images are tagged by commit SHA, ECS can move back to a previous task definition if needed.
Platform walkthrough

End-to-end architecture diagrams for the public demo page.

These frames follow the full data journey: controlled deployment, OLTP source modeling, I/U/D change capture, Bronze evidence, Silver reconciliation, star schema modeling, Gold marts, orchestration, stakeholder access, agent reasoning, and service permissions.

1 2 3 4 5 6 7 8 9 10

Walkthrough stages

The public demo progresses from setup, through Bronze-to-Gold processing, into stakeholder-facing delivery.

Source and apps Bronze raw data Silver cleaned data Gold analytics Orchestration and AI Stakeholder access Security and monitoring
1

Session start and controlled deployment

The session workflow applies infrastructure, prepares source data, deploys artifacts, runs the pipeline, exposes the apps, and tears down runtime resources after the demonstration.

Next
Session workflow: repos, cloud resources, data pipeline, apps, teardown GitHub Actions coordinates the public demo while cloud resource identifiers stay out of logs. Source repos infra live ↗ simulator ↗ glue ↗ dbt ↗ GitHub Actions OIDC session 1. Terraform apply network, lake, IAM, runtime ↗ 2. Source ready simulator with RDS/DMS ↗ 3. Artifacts Glue ↗ dbt ↗ 4. Run pipeline session orchestrator ↗ 5. Entry points agent ↗ web ↗ Slack ↗ 6. Destroy runtime removed after session Demo access Streamlit UI HTML web app Slack gateway end of session The important idea: source code drives a repeatable session, and runtime resources are removed when the demo ends.
2

OLTP source model and business changes

OLTP means the operational database that accepts day-to-day transactions. The simulator creates inserts, updates, and deletes across six PostgreSQL tables.

PrevNext
Source model: six OLTP tables capture the business as it happens I/U/D means Insert, Update, Delete. CDC means Change Data Capture, the process that records those changes without polling every table. Simulator events platform-cdc-simulator ↗ I: insertnew customer, new order U: updatestatus, payment, shipment D: deletecancellations or cleanup PostgreSQL OLTP schema Reference tables describe people and products. Transaction tables record orders and their lifecycle. customerscustomer_id productsproduct_id orderscustomer_id order_itemsorder_id, product_id paymentsorder_id shipmentsorder_id writes business rows places contains appears in paid via ships as This is the operational starting point: small, normalized tables optimized for transactions, not analytics.
3

CDC into Bronze: full load plus I/U/D files

DMS reads the PostgreSQL write-ahead log as a replication client. Every insert, update, and delete becomes raw, append-only evidence in Bronze.

PrevNext
Bronze ingestion: CDC preserves every source change Bronze is immutable: files are added, not rewritten. That makes it the audit trail for the platform. PostgreSQL source 6 OLTP tablescustomers to shipments WAL streamordered source changes I/U/D AWS DMS terraform-platform-infra-live ↗ Full loadexisting rows once CDC streamnew changes after load S3 Bronze raw zone LOAD fileinitial snapshot op Iinsert op Uupdate op Ddelete Each record carries operation and DMS timestamp replication snapshot files change files Permission boundary: DMS can write to Bronze and use KMS encryption. It does not write Silver or Gold.
4

Silver CDC reconciliation and quality gates

Bronze can contain many versions of the same business entity. Glue keeps the latest valid row, removes deletes from current-state Silver, and sends failures to quarantine.

PrevNext
Silver reconciliation: latest clean state from raw CDC history Nothing is silently dropped. Good rows go to Silver, failed rows go to quarantine with a reason. Bronze records customer_id = 42 I at 10:00 U at 10:05 U at 10:12 D at 10:20 if deleted Glue PySpark logic platform-glue-jobs ↗ 1. group by business key 2. sort by DMS timestamp descending 3. keep rank 1, filter op D, validate S3 Silver one current row per key Quarantine failed row plus reason Metrics freshness row counts read raw passes fails Permission boundary: Glue can read Bronze, write Silver and quarantine, update Catalog, publish metrics, and use KMS.
5

Silver star schema modeling

Silver converts operational tables into analytics-friendly dimensions and facts. Dimensions describe who and what. Facts record measurable business events.

PrevNext
Star schema: business-friendly model for Athena and dbt The model keeps the source relationships, but presents them as reusable analytical tables. fact_orders one row per order partitioned by order year/month dim_customer who placed the order dim_product what was bought fact_order_items products per order fact_payments amount and status fact_shipments carrier and delivery Silver model boundary platform-glue-jobs ↗ 2 dimensions plus 4 fact tables, registered in Glue Catalog and stored as Parquet in S3. customer_id order_id product_id through order items order_id order_id
6

Gold modeling with dbt and Athena

Gold tables are curated answer tables for common business questions. dbt builds them from Silver, tests them, and publishes metadata the agent uses to explain answers.

PrevNext
Gold: each mart is built to answer a business question Silver is a reusable model. Gold is the answer layer: pre-modeled tables for revenue, customers, products, payments, and delivery. S3 Silverstar schema dbt projectbusiness logic run_dbt job platform-dbt-analytics ↗ dbt deps freshness dbt run row count dbt test docs S3 Gold: business question answer tables Gold models in platform-dbt-analytics ↗ monthly_revenue_trendHow is revenue trending? revenue_by_countryWhich countries drive revenue? customer_segmentsWho are our customer groups? payment_method_performanceHow do payment types perform? top_selling_productsWhat products sell most? category_performanceWhich categories perform best? carrier_delivery_performanceHow well do carriers deliver? dbt metadata for the agentmanifest, catalog, table descriptions Permission boundary: the dbt job reads Silver and dbt project files, then writes Gold answer tables and dbt metadata.
7

Orchestration paths

Both orchestrators run the same logical pipeline. Step Functions starts AWS jobs directly for quick sessions. MWAA expresses the same stages as Airflow tasks.

PrevNext
Orchestration: one pipeline, two runners The ordered work is the same. Step Functions is compact for short sessions; MWAA shows the same stages as an Airflow DAG. Session Startorchestrator ↗ Same logical pipeline stages Both runners clean Bronze, refresh the Catalog, build Gold, validate rows, run tests, and publish metadata. 6 Silver jobsglue repo ↗ row-count checkinfra repo ↗ Catalog crawlerinfra repo ↗ freshness gatedbt source test dbt rundbt repo ↗ Gold checksdbt tests ↗ Step Functions default runnerdefined in infra repo ↗ MWAA Airflow runnerAirflow DAG repo ↗ Runner choice changes how the stages start, not what the stages do. Catalog crawlerupdates Glue Catalog for Athena/dbt ↗ row-count checkfails empty, missing, or sharply changed tables ↗
8

Stakeholder access and runtime layout

There are three entry points, but one analytics owner. Web and Slack services collect questions, then the Analytics Agent performs the governed Gold query.

PrevNext
Access: three entry points, one governed analytics backend The entry-point services are deliberately thin. They pass questions to the agent instead of reading lake data. Browser useropens app Slack userasks in Slack ALBbrowser entry Analytics Webplatform-analytics-web ↗ Streamlit UIplatform-analytics-agent ↗ Slack gatewayslack gateway repo ↗ Analytics Agent FastAPI 8080 ↗ only service that queries Gold Gold martsdbt repo ↗ HTML Streamlit Slack app event proxied question local call agent API read only Permission boundary: web, Streamlit, and Slack collect questions. The Analytics Agent owns the governed Gold query.

Stakeholder access

Browser, Streamlit, and Slack all hand the same business question to one governed analytics service.

Access Root

Stakeholder questions

Questions enter through browser or Slack, then converge on one analytics owner and one governed data path.
User Web Agent
9

Analytics Agent request loop

This full sequence shows the browser path, four Claude calls, SQL guardrails, one correction loop, Athena execution, audit writes, and the returned answer.

PrevNext
Agent request loop: browser question to audited answer Streamlit and FastAPI run in the same ECS task, so the UI-to-backend hop is local inside the container. Stakeholder AWS Load Balancer Streamlit UI ↗ FastAPI backend ↗ Claude on AWS Amazon Athena S3 logs Streamlit and FastAPI are inside the same ECS task 1 opens app in browser 2 forwards request to UI 3 browser shows text box and Submit button 4 types question and clicks Submit 5 forwards the question 6 sends question internally, no network hop 7 Call 1: generate SQL 8 SQL query 9 validate SQL guardrails 10 Call 2: infer SQL intent, SQL only 11 inferred SQL intent 12 Call 3: compare original question vs inferred intent 13 yes/no verdict plus discrepancy detail Retry loop, only when the verdict says the SQL missed the question 14 regenerate SQL with correction detail 15 corrected SQL query 16 validate corrected SQL 17-18: re-infer corrected SQL intent for audit 19 run query against Gold tables 20 results: rows and columns 21 Call 4: generate insight and chart title 22 2-3 sentence insight and chart title 23 write audit log and engineer log 24 returns insight, chart, cost, verdict 25 renders answer, chart, and cost in browser
10

Least privilege by service

Each box says what that service is allowed to read, write, or start. Anything not listed is intentionally outside that service boundary.

PrevTop
IAM: permissions are separated by service responsibility Authentication identifies the caller. Authorization is the list of allowed actions shown in each box. Source simulator ECSplatform-cdc-simulator ↗Can connect only to source RDS DMSterraform-platform-infra-live ↗Can write raw CDC files only to Bronze Glue Silver jobsplatform-glue-jobs ↗Can write Silver and Quarantine run_dbt Glue jobplatform-dbt-analytics ↗Can write Gold tables and dbt metadata Step Functionsterraform-platform-infra-live ↗Can invoke row-count Lambda MWAA Airflowplatform-orchestration-mwaa-airflow ↗Can start Glue, crawler, and Athena Athena workgroupCan read Catalog table definitionsCan write query results to results path Analytics Agentplatform-analytics-agent ↗Can write audit evidence and charts Claude Platform on AWSAgent can call configured workspaceUsed for SQL, intent check, insight Analytics Webplatform-analytics-web ↗Can proxy browser calls to the agent Slack gatewayplatform-slack-mcp-gateway ↗Can call the Analytics Agent API Redshift servingterraform-platform-infra-live ↗Can read table definitions from Catalog Public logsShow summaries, not identifiers or URLs DestroyRuntime services removed after session

Least-privilege map

Each service stops at a specific boundary across ingestion, orchestration, query execution, and public entry points.

Least Privilege

Service boundaries by responsibility

Pipeline services, orchestration layers, query surfaces, and public entry points each keep a separate trust boundary.
Reads Writes Starts

Repository mental model

Static UI → /api proxy → FastAPI agent

The repo is a presentation layer. It should stay thin and let the existing backend own analytics logic.

Why this can coexist with Streamlit

Streamlit + HTML → same API

Both frontends can evolve separately because neither changes the backend contract.

Visual README for platform-analytics-web. The markdown README remains the exact command and operations reference.