At DataBolt, we've worked with dozens of organizations scaling their dbt projects from a handful of models to thousands. The transition from small-scale to enterprise-level transformations exposes challenges that aren't immediately obvious when you're just getting started. Let me share the patterns that separate successful large-scale dbt implementations from those that become unmaintainable bottlenecks.
The Architecture Foundation: Layer Your Models Deliberately
The most critical decision you'll make is how to structure your project layers. While dbt's documentation suggests staging, intermediate, and mart layers, at scale you need more nuance.
Here's the approach that has served us best:
- Staging models: One-to-one with source tables, minimal transformation (renaming, casting, basic deduplication only)
- Intermediate models: Business logic lives here, organized by domain (finance, product, marketing)
- Mart models: Final, denormalized tables optimized for consumption
- Metrics models: If using dbt metrics or aggregates that power BI tools
The key insight: intermediate models should be organized by business domain, not by technical function. Create subdirectories like intermediate/finance/, intermediate/product/, and intermediate/customer/. This makes ownership clearer and reduces cognitive load when navigating hundreds of models.
Incremental Models: Your Performance Lifeline (and Potential Headache)
Once your data volumes reach meaningful scale, incremental models become non-negotiable. But they're also where many teams shoot themselves in the foot.
The Golden Rules for Incremental Models
First, always include a full-refresh strategy. We've seen too many cases where incremental logic had bugs that went undetected for months, corrupting downstream models. Your orchestration should full-refresh critical incremental models on a weekly or monthly cadence.
{{ config(
materialized='incremental',
unique_key='event_id',
on_schema_change='fail',
incremental_strategy='merge'
) }}Second, be explicit about your incremental strategy. For most data warehouses, we recommend:
- Snowflake/Databricks: Use
mergewith a unique key for updates,appendfor insert-only event data - BigQuery: Leverage
insert_overwritewith partition-level replacements when possible - Redshift: Stick with
delete+insertpatterns for smaller incrementals
Third, set on_schema_change='fail' in production. Schema changes in incremental models are dangerous and should be intentional migrations, not surprises.
Cultivate a Robust Testing Culture
At small scale, you can manually verify model outputs. At large scale, comprehensive testing is the only path to reliability.
Layer Your Testing Strategy
We recommend a four-tier approach:
1. Source freshness and integrity: Define freshness expectations for all sources. This catches upstream pipeline failures before they cascade.
sources:
- name: raw_events
freshness:
warn_after: {count: 2, period: hour}
error_after: {count: 6, period: hour}2. Contract tests on staging: Test data types, null expectations, and accepted values on every staging model. This is your defense perimeter.
- name: stg_orders
columns:
- name: order_id
tests:
- unique
- not_null
- name: status
tests:
- accepted_values:
values: ['pending', 'confirmed', 'shipped', 'delivered', 'cancelled']3. Business logic tests on intermediates: Custom data tests that validate business rules. For example, ensuring revenue calculations never produce negative values, or that customer lifecycles follow valid state transitions.
4. Relationship and integrity tests on marts: Verify referential integrity and cross-model consistency in your final outputs.
Here's an opinionated take: generic tests are necessary but insufficient. Every project with more than 100 models should have at least a dozen custom data tests that encode domain-specific business logic. These catch the subtle bugs that matter most to your stakeholders.
Performance Optimization: Beyond Basic Materialization
Large-scale dbt projects can easily balloon into multi-hour run times. Here's how to keep things snappy.
Leverage Warehouse-Specific Features
Don't treat your warehouse as a generic SQL engine. Use its optimization features:
- Snowflake: Set clustering keys on high-cardinality columns you frequently filter by. Configure appropriate warehouse sizes for different model groups using dbt-snowflake-utils macros.
- BigQuery: Partition and cluster your tables. For large fact tables, partitioning by date and clustering by frequently-filtered dimensions can reduce query costs by 90%.
- Databricks: Use Delta Lake features like Z-ordering and liquid clustering for your largest tables.
{{ config(
materialized='incremental',
partition_by={'field': 'event_date', 'data_type': 'date'},
cluster_by=['user_id', 'event_type']
) }}Parallelize Strategically with Run Groups
Use dbt's selector syntax and tags to create logical run groups. We typically structure this as:
- Hourly: Critical event processing and real-time metrics
- Daily: Core business models and dashboards
- Weekly: Heavy aggregations and historical analyses
Tag your models appropriately and run them on different schedules. This prevents long-running analytical models from delaying time-sensitive operational reports.
Documentation: Make It Discoverable and Maintained
Documentation at scale is about discoverability, not just completeness. A well-documented column on a model nobody can find is useless.
Our recommended practices:
- Document at the schema level first: Use
docsblocks for reusable column descriptions. Define business terms once in.mdfiles and reference them across models. - Embed data lineage explicitly: In model descriptions, explain what business question this model answers and which teams/dashboards consume it.
- Use meta fields for ownership: Tag models with owner information so people know who to ask questions.
models:
- name: fct_orders
description: |
Daily order facts with customer and product dimensions.
Powers the Revenue Dashboard (Tableau) and supports Finance reporting.
meta:
owner: "data-platform-team"
slack_channel: "#data-orders"Controversial opinion: Don't aim for 100% documentation coverage. Focus on documenting your mart layer completely and your intermediate layer selectively. Staging models should have brief descriptions, but documenting every column is often redundant with source system documentation.
Modularization: Packages and Project Structure
Once you exceed 300-400 models, consider splitting your monolithic dbt project.
We've successfully implemented two patterns:
Pattern 1: Domain-based packages - Separate projects for major business domains (customer, product, finance) that compile independently but can reference each other via ref. Use dbt's project dependencies feature.
Pattern 2: Layer-based separation - One project for staging/intermediate, another for marts and metrics. The marts project depends on the transformations project.
Both work. Choose based on your team structure. If you have domain-specific data teams, go with Pattern 1. If you have a centralized data platform team with embedded analysts, Pattern 2 often maps better to deployment boundaries.
CI/CD: Protect Production Ruthlessly
For large-scale dbt projects, CI is non-negotiable. At minimum:
- Run
dbt compileanddbt parseon every PR to catch syntax errors - Run
dbt test -m state:modified+to test only changed models and their children - Use slim CI to build only modified models in a dev schema, dramatically reducing CI time
- Enforce documentation checks: fail CI if new models lack descriptions
For production deployments, implement a blue/green or canary pattern where possible. Build models in a staging schema, run full test suites, then swap/promote to production. This prevents partially-completed runs from breaking downstream consumers.
Monitoring and Observability
Finally, instrument your dbt runs. At scale, you need to know:
- Which models are taking progressively longer to run (performance degradation)
- Which models fail most frequently (reliability issues)
- Which models are actually being queried downstream (to identify candidates for deprecation)
Tools like dbt Cloud, Elementary, or re_data can provide this. If you're self-hosted, export run results to a monitoring table and build alerts around it.
The Bottom Line
Scaling dbt successfully requires treating it as a software engineering discipline, not just SQL scripting. Layer your architecture thoughtfully, test comprehensively, optimize deliberately, and monitor continuously. The teams that do this build data platforms that remain maintainable and performant even with thousands of models and dozens of contributors.
The practices above aren't theoretical—they're distilled from real implementations processing billions of rows daily. Start implementing them before you feel the pain, and your future self will thank you.