CI/CD Pipelines for Data Engineers
Data pipelines are production software. Here's how to build CI/CD that catches bad transforms before they corrupt dashboards: testing strategy, environment promotion, slim runs, and rollback patterns.
The Augmented Dev ยท Archive
Dynamics 365, Power Platform, AI, system design, and web development โ filter by theme or search across every post.
Newsletter
One email per post. No spam, no tracking pixels, unsubscribe anytime.
169 articles
Data pipelines are production software. Here's how to build CI/CD that catches bad transforms before they corrupt dashboards: testing strategy, environment promotion, slim runs, and rollback patterns.
Why column-oriented databases run analytical queries 100x faster than row-oriented ones โ covering physical layout, compression algorithms, vectorized execution, and predicate pushdown with concrete examples.
Hard-won lessons for structuring dbt projects that scale: layer conventions, testing strategies, slim CI runs, incremental models, and documentation that doesn't rot.
A deep dive into how Kafka distributes work across consumers, why rebalancing stalls your pipeline, and how to choose an offset commit strategy that matches your delivery guarantee requirements.
Why PostgreSQL's connection model breaks under load, how PgBouncer fixes it, and how to configure transaction-mode pooling without getting bitten by prepared statements or advisory locks.
What Dataverse elastic tables are, how TTL, partitioning, and JSON columns work, and how to decide between elastic and standard tables.
How the D365 F&O Feature management workspace works, what feature states mean for ALM, and how to ship your own X++ features behind flags.
How Power Automate retry policies work, which errors they cover, how to configure them, and how to keep retries from duplicating side effects.
Why dual writes lose events, how the transactional outbox pattern fixes it, and the relay, ordering, and cleanup decisions that make it production-ready.
Signing, retries, ordering, and dead-lettering: the design decisions that separate reliable webhook delivery from silent event loss.
Shipping code and releasing a feature are different events that most teams accidentally fuse together. Feature flags split them โ and unlock trunk-based development, safe rollouts, and instant rollback.
Shipping LLM features on vibes works until a prompt tweak silently breaks ten other cases. Here is how to build evals that catch regressions before your users do.
Asking a model to 'return JSON' and parsing the result is how you get 3 a.m. pages. Tool schemas, constrained decoding, and validation turn a probabilistic text generator into a reliable API.
A practical guide to designing branching Business Process Flows in model-driven Dynamics 365 apps, with stage-gating, automation hooks, the storage model, and hard limits.
Data entities look like simple tables until you need a nested import, a derived column, or a field with no storage. Knowing the three advanced shapes is what makes integrations actually work.
Most RAG systems that retrieve bad context aren't failing at embeddings or reranking โ they're failing at chunking. How you split documents quietly decides what your model can ever find.
A fast producer and a slow consumer is a recipe for an out-of-memory crash. Backpressure is the discipline of letting the slow part tell the fast part to wait. Here is how to design it in.
How the Dataverse event pipeline actually works: pre-validation, pre-operation, post-operation stages, the transaction boundary, entity images, depth, and registration guidance.
More connections is not more throughput. Past a point, adding connections makes your database slower. Here is how pools actually work and how to size one without guessing.
Collections feel fast and convenient, so makers overuse them โ and ship apps that lose data and break on refresh. Here is when to hold data in memory and when it belongs in Dataverse.
Modern client-side scripting for model-driven apps using formContext over the deprecated Xrm.Page, with handler wiring, async Xrm.WebApi patterns, and maintainability tips.
A data-modeling guide to Dataverse column types: choice vs multi-select choices vs lookup vs polymorphic customer columns, local vs global choices, status reason, and a decision checklist.
How does a database survive a power cut mid-write without corrupting your data? The answer is a deceptively simple rule: log the change before you apply it. Here is why WAL is everywhere.
How synchronous and asynchronous classic Dataverse workflows differ in execution, rollback, ordering, and performance, and where Power Automate now fits in.
The hardest part of React Server Components isn't the syntax โ it's knowing which kind of component you're writing and why. Here is the mental model that makes the boundary obvious.
A Bloom filter answers 'have I seen this?' using a tiny fraction of the memory a real set would need. The price is false positives โ and understanding that trade is the whole skill.
Construct multipart batch payloads, use changesets for transactional all-or-nothing writes, reference records across requests, and handle service protection limits.
Transient failures are normal in any flow that calls an API. The difference between a flow that self-heals and one that pages you at 2 a.m. is how you handle retries and the failures that stick.
A single ACID transaction can't span order, payment, and inventory services. Sagas chain local transactions with compensating actions to keep distributed state consistent โ eventually.
Add fields, computed columns, and validation to standard D365 Finance & Operations data entities the upgrade-safe way โ with X++ examples and the staging-table traps to avoid.
The like button that responds before the server confirms feels instant. The trick is updating the UI first and reconciling later โ and handling the rollback so you never mislead the user.
When to use Chain of Command and when to use pre/post event handlers in Dynamics 365 Finance & Operations โ with X++ examples, a decision table, and the gotchas that trip up teams.
When a downstream service slows down, naive retries turn one sick dependency into a system-wide outage. A circuit breaker fails fast, sheds load, and gives the dependency room to recover.
A synchronous plug-in runs inside the user's transaction with a hard time limit. One slow query or external call can break saves for everyone. Here is how to keep plug-ins fast and safe.
Add buttons, write Power Fx visibility rules, target grid form and subgrid locations, pass the selected record, and know when you still need JavaScript libraries.
Enable auditing at org, table, and column level, read audit history, manage retention and storage cost, and lock down sensitive fields with column security profiles.
A practical guide to the Electronic Reporting (ER) framework in D365 Finance โ data models, model mappings, and format configurations to produce custom files without X++.
You wrote to the database and then published an event โ and the publish failed. Now your systems disagree. The outbox pattern makes the write and the event atomic without a two-phase commit.
A search box that fires a request on every keystroke, a scroll handler running 60 times a second โ these are debounce and throttle problems. The concepts are simple; React makes them subtly tricky.
A practical guide to the lead lifecycle, what qualify creates, the Lead-to-Opportunity BPF, duplicate detection, lead scoring, and rep adoption tips.
Default dimensions, ledger dimensions, account structures, and the DimensionAttributeValueCombination table explained โ with X++ patterns for reading and building dimensions in D365 Finance.
When you add a node to a sharded cache, naive hashing remaps almost every key. Consistent hashing moves only a fraction. Here is the mechanism, the virtual-node fix, and the traps.
Native N:N relationships are quick to set up and impossible to extend. A manual junction table is more work and far more flexible. Choosing wrong means a painful migration later.
A practical primer on case creation, the case resolution BPF, public and private queues, queue items, routing rules, SLAs, and parent-child cases.
How the number sequence framework works in D365 Finance & Operations โ continuous vs non-continuous, preallocation, cleanup, and X++ patterns for generating numbers on custom tables.
How the role-based security model in D365 Finance & Operations actually works โ privileges, duties, roles, segregation of duties, and how to extend security without overlayering.
A clear mental model for solutions, the dev-test-prod ALM flow, managed properties, solution layers, layering holes, segmentation, and patches.
Move files in and out of D365 Finance & Operations reliably using the recurring integrations REST API โ enqueue, dequeue, acknowledge, and the status polling patterns that keep data flowing.
How to build extensible data security policies in D365 Finance & Operations to filter records by user context โ with a worked example, query constructs, and the performance pitfalls to avoid.
Practical D365 Finance and Operations LCS guide for environment tiers, database refreshes, bacpac exports, restores, and sandbox governance.
How to build a Power Pages site over Dataverse โ pages, basic vs advanced forms, lists, table permissions, web roles, anonymous vs authenticated access, and Liquid.
How to ground a Copilot Studio agent on Dataverse and other knowledge sources, keep answers cited and security-trimmed, test for hallucination, and choose tools vs knowledge.
Practical D365 F&O X++ event handler guide for table CRUD events, form data source events, ordering caveats, and when to use CoC.
A practical guide to Dataverse business units, owner vs access vs group teams, ownership and access levels, hierarchy security, and modeling org structures without BU sprawl.
Build and deploy D365 Finance and Operations packages with Azure DevOps, hosted build pipelines, LCS Asset Library, and release safeguards.
Practical design guidance for model-driven app views, charts, and dashboards: view types, filtering and sorting, chart aggregation, interactive dashboards, performance, and security.
Design resilient D365 Finance and Operations OData integrations with 429 handling, Retry-After, batching, backoff, jitter, and DMF fallbacks.
Step-by-step D365 Finance and Operations workflow framework guide covering categories, types, approvals, event handlers, and submit code.
How to scale D365 F&O batch: batch groups bound to servers, max threads per AOS, splitting work into parallel tasks with SysOperation, task dependencies, and troubleshooting.
Clear X++ guide to TempDB vs InMemory temporary tables in D365 F&O, with joins, indexes, set-based operations, and performance tradeoffs.
Compare dual-write, virtual entities, OData, and DMF for Finance and Operations to Dataverse integration with latency and failure tradeoffs.
A hands-on guide to building approval workflows in Dynamics 365 Finance & Operations using the workflow editor, with conditional routing, escalation, and delegation.
Learn when to use Dataverse low-code plug-ins with Power Fx for transactional server-side logic, and when C# plug-ins or flows still win.
How to build financial statements in Dynamics 365 Finance using row, column, tree and report definitions, with practical tips on dimension sets and performance.
Build a first Dataverse C# plug-in with IPlugin, pipeline stages, Target handling, tracing, registration, and production-safe practices.
When to use query-based versus Report Data Provider SSRS reports in Dynamics 365 F&O, the full component stack, and an X++ sketch of an RDP class and data contract.
A practical Dataverse logic placement guide comparing business rules, Power Fx, JavaScript form scripts, and C# plug-ins for scale.
Configure Copilot in Dynamics 365 Sales with practical guidance on summaries, email drafting, meeting prep, data readiness, and governance.
A practical breakdown of inventory costing in Dynamics 365 Supply Chain Management with item model groups, physical vs financial updates, inventory close, and a FIFO example.
Design scalable model-driven app forms with tabs, sections, quick views, business rules, form scripts, security, and performance patterns.
Use Dataverse alternate keys and Web API upsert to build idempotent integrations that match business identifiers, avoid duplicates, and simplify retries.
Compare Dataverse calculated, rollup, and formula columns with practical examples, platform limits, timing tradeoffs, and when plug-ins are safer.
Secure Power Pages data with Dataverse table permissions, access scopes, web roles, column profiles, and testing practices that prevent data leaks.
Configure Dynamics 365 Customer Service SLAs, KPI timers, business hours, pause rules, Power Automate actions, and entitlements without surprises.
Choose OData or FetchXML for Dataverse Web API queries with practical examples for select, filter, joins, aggregation, paging, and performance.
Choose between canvas apps, model-driven apps, and Power Pages using a practical matrix for UX, data, licensing, governance, and audience fit.
Learn how Power Apps delegation works, why blue warnings matter, and the Dataverse patterns that keep large-table queries accurate and fast.
Build a first Power Apps PCF control with the CLI, manifest, TypeScript lifecycle, local testing, solution packaging, and safe deployment steps.
Design offline-capable Power Apps for field teams using canvas app caching, queue-and-sync patterns, conflict handling, and mobile offline profiles.
Use Power Fx named formulas and With to replace noisy OnStart logic, reduce repeated lookups, and make canvas apps faster and easier to maintain.
Design a Power Platform environment strategy with dev, test, prod, managed solutions, pipelines, DLP, routing, and governance that scales for teams.
Practical guide to Power Platform DLP policies, connector groups, environment scope, custom connectors, endpoint rules, and flow breakage diagnosis.
Understand Managed Environments in Power Platform, including governance features, licensing, security controls, insights, trade-offs, and rollout timing.
Build a first Copilot Studio agent with topics, generative answers, actions, authentication, testing, publishing, and Teams or website channels.
Embed Power BI content in canvas apps, pass app selections into report filters, read Power BI selections back, and handle licensing and performance.
Learn responsive canvas app layout with horizontal and vertical containers, flexible sizing, breakpoints, scaling settings, accessibility, and tab order.
Memorize these Power Automate expressions to build faster flows, handle nulls, shape arrays, format dates, reduce action count, and debug WDL.
Learn when to call the Dataverse Web API from Power Automate for actions, functions, batch requests, impersonation, row association, and binds.
Use child flows in Power Automate to centralize reusable logic, pass typed inputs and outputs, handle errors, reduce duplication, and improve ALM.
Speed up Power Automate Apply to each loops with concurrency, Filter array, Select, OData filters, pagination strategy, lower action counts, and safer runs.
Master Power Automate Parse JSON with generated schemas, nullable fields, required properties, arrays, nested access, and practical error fixes.
Build your first Power Automate Desktop RPA bot with selectors, variables, loops, Excel data, cloud triggers, licensing, and reliable handoffs.
Use Power Automate trigger conditions to prevent noisy runs, save quota, filter SharePoint changes, skip system updates, and debug triggers.
Implement try-catch-finally error handling in Power Automate using Scopes, run-after settings, result outputs, retries, alerts, and Terminate.
Speed up bulk SharePoint creates and updates in Power Automate by calling the $batch API with changesets, chunking, and per-item error handling.
Create Teams adaptive cards from Power Automate, collect inputs, capture responders, update cards, validate replies, and choose cards wisely.
Use connection references and environment variables to make Power Automate solution deployments predictable across dev, test, and production.
Learn how Power Automate pagination, thresholds, filters, and batching keep large Dataverse and SharePoint datasets reliable.
Secure Power Automate secrets with Azure Key Vault, secure inputs and outputs, least privilege, rotation, and safer ALM patterns.
How to expose Dynamics 365 Finance and Operations to Claude through MCP โ the architecture, the data entity choices, the auth model, and the operations that should never be one prompt away.
A practical guide to the design decisions that determine whether an event-driven system stays maintainable or quietly rots โ delivery guarantees, ordering, idempotency, schema evolution, and the outbox.
Build resilient Power Automate integrations with Azure Service Bus queues, topics, sessions, duplicate detection, and dead-letter handling.
Master Dynamics 365 Customer Service unified routing with skills-based assignment, capacity profiles, and overflow rules to cut handle time and boost CSAT.
Walk through the architecture of a production search system โ inverted indexes, BM25 ranking, vector embeddings, and hybrid retrieval for 50M documents at 10K QPS.
An introduction to the Model Context Protocol and how the Dataverse MCP server lets Claude read and write business data through natural language.
PPR was the experimental flag in Next.js 15; in 16 it's been folded into Cache Components. Learn how static shells and dynamic holes work, how to migrate incrementally, and how to measure the LCP gains.
Instrument your .NET services once with OpenTelemetry and ship traces to Jaeger, Grafana Tempo, or Azure Monitor โ all without changing application code.
Power Platform Pipelines let teams promote solutions from dev to test to prod with built-in approvals and validations โ no Azure DevOps required.
A hands-on example of using the Dataverse MCP server with Claude to query tables, create records, and update data in Dynamics 365 through natural language.
Use Claude Code hooks to enforce policy at the moment actions happen โ before edits, before tool calls, on session start โ without relying on pre-commit hooks everyone learns to bypass.
From zero to a working Model Context Protocol server in about 100 lines โ tools, resources, a local client test, and the traps that will bite you on day one.
A precise look at Claude's prompt caching โ how the cache hit math actually works, the 5-minute and 1-hour TTL variants, and the subtle mistakes that silently disable it.
A practical walkthrough of Snowflake's four security layers โ role hierarchy, column-level masking policies, row access policies, and environment isolation โ with real SQL examples.
The standout features from Microsoft's 2026 Release Wave 1 for Dynamics 365 and Power Platform, and what they mean for your projects.
The tool-use features of the Claude API look straightforward in the docs. These are the production patterns that separate a demo from a system that handles real traffic โ parallel tool calls, retry loops, error recovery, and tool_choice tactics.
The Claude Agent SDK is the fastest path to a capable coding-style agent. The raw messages API is the fastest path to understanding what your agent is doing. Here's when each is the right tool.
A system design deep dive into building a notification platform that handles push, email, SMS, and in-app notifications at scale โ covering architecture, priority queues, fan-out strategies, rate limiting, and delivery tracking.
Retrieval-augmented generation demos look great. Production RAG is a different engineering problem โ chunking, hybrid retrieval, reranking, evaluation, and the failure modes nobody mentions at conferences.
A deep dive into the three dominant API paradigms โ REST, GraphQL, and gRPC โ covering design principles, pagination strategies, versioning, authentication patterns, and practical guidance on choosing the right one for your system.
A practical guide to the features that changed how we build React apps โ Server Components, the new compiler, and the patterns that stuck.
A comprehensive guide to event-driven architecture โ covering pub/sub, event sourcing, CQRS, saga patterns, message broker trade-offs, and the hard lessons teams learn in production.
A practical comparison of the vector databases people actually deploy in 2026 โ and an honest look at when a vector database is the wrong tool for the job.
A deep dive into caching patterns that power the world's fastest systems โ from cache-aside and write-through to multi-level architectures, stampede prevention, and real-world eviction strategies at scale.
Idempotency keys are what separate a payment system that double-charges during a retry from one that doesn't. The mechanism looks simple and has five subtle failure modes you need to know about.
Cut through the hype and understand when a monolith, modular monolith, or microservices architecture is the right fit. Covers service boundaries, data ownership, communication patterns, and practical migration strategies.
Understand how database indexes work under the hood, from B+ tree internals to composite index design. Learn to read EXPLAIN plans, avoid common anti-patterns, and make informed trade-offs between read performance and write amplification.
LLM-powered pipelines break in the same ways old monoliths did โ when you stack synchronous calls. Queues, events, and the patterns that make a six-step AI pipeline survive a partial outage.
A complete system design walkthrough for building real-time chat at scale โ covering WebSocket management, message delivery guarantees, presence systems, group chat fan-out, and end-to-end encryption.
A deep dive into the CAP theorem โ what it really means for distributed systems, why you can only pick two guarantees, and how real databases like Cassandra, MongoDB, and DynamoDB make their trade-offs.
Three patterns for multi-tenant data isolation in SaaS, the trade-offs between cost, blast radius, and compliance, and a migration path from one to another when you outgrow your first choice.
A hands-on tutorial for creating custom Copilot plugins in Dynamics 365 Sales โ from defining a custom API to registering a plugin that Copilot can invoke, with real examples and permission gotchas.
A complete walkthrough of designing a URL shortener service โ covering hashing strategies, database schema, caching, analytics, and scaling to billions of redirects.
A practical breakdown of the Sales Qualification Agent, Case Management Agent, and the governance framework you need before deploying them.
A step-by-step guide to building a live-updating dashboard using Next.js API routes, Server-Sent Events, and Supabase Realtime โ with reconnection handling and smooth UI transitions.
How the Accounting Rules Engine, multi-entity journals, flexible hierarchies, and Copilot in D365 Finance 10.0.46+ transform period close for multi-company environments.
Set up automated tests for your canvas apps using Power Apps Test Engine โ from first test case to CI/CD pipeline integration.
A field-tested guide to dual-write configuration between D365 Finance & Operations and Dataverse โ covering initial sync failures, conflict resolution, performance tuning, and when to use alternatives like virtual entities or data export service.
Most alerting setups produce noise that teams learn to ignore. Here is how to design alerts around the four golden signals, set thresholds that mean something, and build a system where every alert is worth waking someone up for.
A first-person account of a production outage caused by skipping staging. The small fix, the cascading failure, the 3am scramble, and the process changes that followed.
A detailed comparison of Azure Functions and Azure Container Apps โ pricing, cold starts, scaling, runtime support, and a decision flowchart for picking the right one.
How to build a deployment pipeline that swaps staging and production slots on Azure App Service โ with smoke tests, warm-up, rollback strategy, and a real YAML pipeline you can steal.
ARM templates are painful. Bicep fixes the developer experience with cleaner syntax, modules, and first-class Azure CLI integration. Here is how to migrate and start using it in your CI/CD pipelines.
Why every public API needs rate limiting, the four main algorithms compared, implementation in Next.js with Redis, proper HTTP headers, and Azure API Management policies.
A practical comparison of session-based auth, JWT auth, and OAuth 2.0 / OIDC โ how each works, when to use them, security pitfalls, and recommended patterns for SPAs, server apps, and mobile.
A clear-eyed look at Command Query Responsibility Segregation โ what it actually is, how to implement it in .NET, when it pays off, and when it's unnecessary complexity.
A practical guide to breaking apart a monolith into microservices โ identifying boundaries, splitting the database, handling data consistency, and avoiding the traps that turn your migration into a distributed monolith.
How to diagnose and fix slow X++ queries in D365 Finance & Operations using Trace Parser, execution plans, proper index design, join selection, and set-based operations โ with real before/after examples.
How to use virtual entities in D365 Finance & Operations to surface external data from APIs and databases without importing, syncing, or paying the ETL tax โ with a real-world supplier data example.
Stop copying .ps1 files into shared folders. Learn how to package PowerShell scripts into proper modules with versioning, publish them to a private feed, and distribute them to your team like a professional.
A practical guide to data migration using the Data Management Framework in D365 Finance & Operations โ covering entity selection, staging tables, transformation patterns, error handling, and performance tuning from real enterprise migrations.
Concrete, measurable patterns to speed up slow Power Apps canvas apps โ with before/after examples for each optimization.
Advanced Power Fx patterns that separate beginners from power users โ With, Sequence, ParseJSON, error handling, regex, and more.
Build a full round-trip integration between Power Automate and D365 Finance & Operations โ trigger on business events, transform data, and write back via OData.
How to use Chain of Responsibility (CoR) to extend standard X++ classes in D365 Finance & Operations โ with code samples for wrapping methods, accessing protected members, and avoiding common pitfalls.
How to build, activate, and consume Business Events and Data Events in Dynamics 365 Finance & Operations with X++ code samples and integration patterns.
How Dataverse security roles actually work, common mistakes that leave data exposed, and the layered model you should be using in 2026.
How to build batch jobs using the SysOperation framework in D365 Finance & Operations โ contract, controller, and service classes with X++ code samples.
Go beyond basic approvals โ build a multi-level approval chain with rich Adaptive Cards, timeout handling, escalation logic, and full audit trails in Power Automate.
Build a custom connector from an OpenAPI spec, configure authentication, handle pagination, and share it across your organization โ step by step.
Build resilient flows with scope-based try-catch-finally patterns, configure run after, and structured error logging in 2026.
A step-by-step guide to diagnosing and resolving NuGet package source errors in Visual Studio and dotnet CLI.
How to configure concurrency limits on scheduled and trigger-based flows to prevent data corruption and race conditions.
Reliable techniques for detecting empty date fields in Dataverse and SharePoint connectors within Power Automate flows.
How to use environment variables, connection references, and solution-aware configuration for clean ALM in Power Platform.
How to distinguish between create and update operations in Dataverse-triggered flows and implement conditional logic accordingly.
Three practical techniques for deduplicating array data in Power Automate, from simple expressions to XPath tricks.
A practical reference for solution component types, their IDs, and how to work with solution XML in Dynamics 365 and Power Platform.
Practical expressions and defensive patterns to prevent null field errors from crashing your Power Automate flows at runtime.
How to resolve execution policy errors when running .ps1 scripts, and why you should care about script signing in 2026.