CI/CD Pipelines for Data Engineers
Data pipelines are production software. Here's how to build CI/CD that catches bad transforms before they corrupt dashboards: testing strategy, environment promotion, slim runs, and rollback patterns.
The Augmented Dev ยท Archive
Dynamics 365, Power Platform, AI, system design, and web development โ filter by theme or search across every post.
Newsletter
One email per post. No spam, no tracking pixels, unsubscribe anytime.
4 of 225 articles tagged data-engineering
Data pipelines are production software. Here's how to build CI/CD that catches bad transforms before they corrupt dashboards: testing strategy, environment promotion, slim runs, and rollback patterns.
Why column-oriented databases run analytical queries 100x faster than row-oriented ones โ covering physical layout, compression algorithms, vectorized execution, and predicate pushdown with concrete examples.
Hard-won lessons for structuring dbt projects that scale: layer conventions, testing strategies, slim CI runs, incremental models, and documentation that doesn't rot.
A practical walkthrough of Snowflake's four security layers โ role hierarchy, column-level masking policies, row access policies, and environment isolation โ with real SQL examples.