EDP platform-dbt-analytics
dbt analytics modeling layer

Silver tables become business-ready Gold marts.

This repository owns the SQL modeling layer. dbt reads validated Silver facts and dimensions, builds staging and intermediate views, persists seven Gold mart tables, runs data quality gates, and publishes artifacts the analytics agent uses as schema context.

dbt Athena DuckDB local target Gold marts manifest.json catalog.json
Full platform context

dbt is the semantic bridge between clean data and business answers.

Silver-to-Gold semantic modeling path dbt starts after Glue. Its job is to turn clean operational Silver tables into documented, tested, business-facing Gold marts. Silver inputs facts + dimensions from Glue fact_orders fact_payments dim_customer Staging models rename, cast, document stg_* views source cleanup Business logic joins, metrics, marts dbt + Athena model DAG tests + docs Gold outputs analytics-ready marts gold_customer_lifetime_value gold_sales_daily gold_payment_method_revenue dbt artifacts manifest.json + catalog.json Analytics agent context schema names and columns read Silver transform write Gold
Business semanticsGlue produces clean operational tables; dbt gives those tables business meaning through joins, aggregations, tests, and documentation.
No loading stepAthena queries Parquet directly in S3, so dbt SQL reads Silver and writes Gold without moving data into a warehouse cluster.
Agent contextdbt artifacts describe models and columns, giving the analytics agent safer schema context when generating SQL.
Model DAG

Fifteen models build in layers: source cleanup, reusable joins, final marts.

dbt dependency graph The graph is easiest to read as three business paths after staging: orders, products, and delivery. Silver sources Staging views Reusable joins Gold marts customers products orders order_items payments shipments stg_customers stg_products stg_orders stg_order_items stg_payments stg_shipments int_orders customer + order payment + shipment int_product_sales items + product staging model inputs Order and revenue marts monthly_revenue_trend revenue_by_country payment_methods customer_segments Product marts category_performance top_selling_products Delivery mart carrier_delivery_performance
Staging layerStaging views standardize names, casts, casing, and partition columns without introducing business aggregation.
Intermediate layerReusable joins keep mart SQL smaller and make the order and product analysis paths easier to reason about.
Mart layerSeven persisted Gold tables answer finance, product, customer, and operations questions without recomputing heavy joins each time.
Gold outputs

Each mart is shaped around a question stakeholders actually ask.

Gold mart catalogue The marts are the contract between SQL modeling and the analytics experiences. Finance monthly_revenue_trendrevenue over time revenue_by_countrymarket performance payment_method_perfsuccess and failure rates Product category_performancecategory and brand revenue top_selling_productsranked product revenue Customer customer_segmentslifetime value and activity bands Operations carrier_delivery_perfdelivery reliability and speed Analytics Agent reads these Gold tables Streamlit · HTML dashboard · Slack MCP
Finance martsThese answer revenue, geography, and payment performance questions directly, which is why the agent can produce business-facing answers quickly.
Product and customer martsProduct rankings and customer segments are pre-modeled instead of derived ad hoc from raw line items every time.
Operations martCarrier performance keeps delivery reliability visible beside revenue and customer value, rounding out the demo story.
Data quality gates

Gold is only refreshed after freshness, row-count, and dbt tests pass.

Live pipeline gate sequence When a gate fails, the old trusted Gold tables remain in place and the pipeline stops before publishing new artifacts. Silver ready Glue output exists D2 freshness source:silver tests dbt run build model DAG D3 row counts Gold vs Silver tolerance dbt test schema + accepted values Gold publish tables + artifacts Stop path: do not refresh Gold, do not upload new agent context The previous successful Gold layer remains queryable while Silver or SQL issues are fixed. fail fail fail
Freshness before modelingThe project checks that Silver reached the expected March 2026 cutoff before running Gold SQL.
Row-count reconciliationGold staging counts are compared with CloudWatch metrics published by Glue, avoiding expensive Athena count scans.
Model testsPrimary keys, accepted values, relationships, and positive monetary values are validated before artifacts are handed to the agent.
Local and AWS execution

The same dbt project runs cheaply on DuckDB and validates against Athena in dev.

Two execution paths, one model graph DuckDB makes PR feedback cheap. Athena validates the same project against live dev data before Gold is trusted. Local and CI path Parquet fixturesdata/silver DuckDB targetno AWS needed dbt run + testseconds in CI AWS dev path Silver CatalogGlue metadata Athena targetSQL engine Gold S3 tablesIceberg marts Same SQL models
DuckDB pathCI can lint and run the dbt project against local Parquet fixtures without AWS credentials or Athena cost.
Athena pathThe deploy workflow validates against real dev Silver tables, then writes Gold Parquet/Iceberg output to S3.
Same projectThe profile changes the engine and source location, but the model graph remains the same.
Deployment and artifacts

dbt changes are uploaded to S3 and picked up by orchestration at runtime.

Deployment flow The dbt project is not baked into MWAA plugins.zip. It is synced to S3 and downloaded by workers at runtime. GitHub Actionsmanual deploy sync projectmodels, macros, profiles run dbt in devdbt deps/run/test MWAA S3 bucketdbt project path Gold S3verified mart tables MWAA workerdownloads project Bronze metadatamanifest + catalog Agentschema context
Fast model rolloutUploading to S3 avoids a slow MWAA environment update for every SQL change.
Runtime pickupThe Airflow DAG downloads the latest dbt project when the Gold task starts, so orchestration uses the current model files.
Artifact handoffManifest and catalog artifacts make model names, columns, and descriptions available to the analytics agent.

Repository mental model

Silver → staging → intermediate → Gold marts

The repo is where clean operational data becomes durable analytics products.

Why marts are tables

Run aggregation once → Serve fast queries

Pre-computed Gold tables keep BI and agent queries from re-running heavy joins on every request.

Visual README for platform-dbt-analytics. The markdown README remains the exact command and operations reference.