When your dbt project grows from dozens to hundreds of models, from a handful of analysts to cross-functional teams, and from gigabytes to terabytes of data, the practices that worked at small scale start breaking down. We've seen teams struggle with 8-hour transformation runs, cryptic dependency chains, and models that mysteriously break in production.

After working with organizations processing billions of rows daily through dbt, we've identified patterns that separate smoothly-running large-scale projects from those that become maintenance nightmares. Let's dive into the practices that actually matter when your dbt project grows up.

Project Structure: Think in Layers, Not Folders

The most common mistake we see in growing dbt projects is organizing models by business domain too early. While it seems intuitive to create folders like marketing/, finance/, and product/, this approach breaks down when models need to reference each other across domains—which happens constantly in real analytics work.

Instead, structure your project in transformation layers that mirror your data's journey from raw to refined:

This layered approach makes dependencies flow in one direction—staging feeds intermediate, intermediate feeds marts. When someone asks "why is this model slow?", you can trace the dependency chain logically rather than jumping randomly between domain folders.

Naming Conventions That Scale

Enforce strict naming conventions from day one. We recommend:

The double underscore between source and entity in staging models is deliberate—it makes clear what system the data came from at a glance, which becomes invaluable when you're integrating 15+ data sources.

Materialization Strategy: Default to Views, Upgrade Deliberately

Here's an unpopular opinion: most teams over-materialize their models. The default impulse when a query is slow is to materialize it as a table, but this creates a cascade of problems—longer build times, stale data issues, and increased warehouse costs.

Our recommended approach:

For truly large datasets, incremental models are your friend, but they come with complexity. Only use incremental materialization when:

When you do go incremental, always include a full-refresh cadence (weekly or monthly) to catch any rows that slipped through your incremental logic.

Performance Optimization: Profile Before You Optimize

We've seen data teams spend weeks optimizing models that run in 30 seconds while ignoring the 4-hour model because "it's always been slow." Don't guess—measure.

Use dbt's run_results.json artifact to identify your actual bottlenecks. Build a simple dashboard tracking model execution times over time. Focus your optimization efforts on:

  1. Models that run longest in absolute terms
  2. Models that are called frequently and whose delays cascade to many downstream models
  3. Models whose performance is degrading over time

Common Performance Wins

Once you've identified your bottlenecks, try these strategies in order:

Filter early, filter often: Push WHERE clauses as far upstream as possible. If you only need the last 2 years of data, filter in staging, not in your final mart.

Be selective with joins: Only join the columns you actually need. Instead of SELECT a.*, b.* FROM table_a a JOIN table_b b, explicitly list required columns. This matters enormously with wide tables.

Use CTEs wisely: Common table expressions make queries readable, but some warehouses (looking at you, Redshift) don't optimize them well. If a CTE is referenced multiple times, consider materializing it as an intermediate model.

Leverage snapshots for slowly changing dimensions: If you're repeatedly querying historical states of data, dbt snapshots are more efficient than maintaining handrolled SCD2 logic.

Testing and Documentation: The Unglamorous Essentials

In small projects, you can remember what each model does and manually verify outputs. At scale, this is impossible. Testing and documentation aren't nice-to-haves—they're the foundation of maintainability.

Testing Philosophy

Implement tests at boundaries where data quality issues are most likely to enter your system:

Don't just use dbt's built-in tests. Create custom schema tests for domain-specific logic. For example, if customer lifetime value should never exceed $1M in your business, codify that as a test.

Use data quality tools like dbt-expectations or elementary-data for more sophisticated testing patterns. Row-count anomaly detection has caught more issues in our projects than any other single test type.

Documentation That Actually Helps

Auto-generated documentation is better than nothing, but barely. Useful documentation answers:

Use dbt's meta fields to store additional context like SLA requirements, downstream dependencies outside dbt, or data classification for compliance purposes.

Orchestration and CI/CD: Slim Builds Change Everything

Once your project has 200+ models, running a full dbt build on every commit becomes untenable. This is where dbt's state-based selection (slim CI) becomes critical.

Configure your CI pipeline to only run and test modified models and their downstream dependencies:

dbt build --select state:modified+ --state ./prod-state --defer

This command compares your current branch to production state, identifies what changed, and only builds those models plus anything downstream. What was a 45-minute CI run becomes 3 minutes.

For production orchestration, consider running different model groups on different schedules. Not everything needs to run hourly. Separate your models into:

Tag models appropriately and schedule them based on actual business requirements, not just "run everything every night."

Managing Complexity: Keep Models Simple

The most elegant dbt projects we've seen share a common trait: individual models are boring. The complexity lives in the composition of simple models, not in 500-line SQL statements with nested CTEs eight levels deep.

If a model file exceeds 200 lines or has more than 5 CTEs, it's almost certainly doing too much. Break it apart into intermediate models. Yes, this creates more files, but it also creates:

Each model should do one thing well. This is the Unix philosophy applied to data transformation, and it scales beautifully.

Final Thoughts

Running dbt at scale isn't fundamentally different from running it at small scale—it just punishes bad practices harder. The shortcuts that work fine with 20 models become critical failures at 500 models. The good news is that the practices that enable scale also make projects more maintainable, more collaborative, and more reliable at any size.

Start with solid foundations: layered project structure, deliberate materialization choices, comprehensive testing, and clear documentation. Add sophisticated optimization and orchestration as you need them, not before.

The teams that succeed at scale are those that treat their dbt project as a software engineering project, not just a collection of SQL files. Version control everything, review all changes, test rigorously, and refactor ruthlessly. Your future self—and your teammates—will thank you.