Evaluating LLM Applications: Getting Past 'It Looks Good to Me'
Shipping LLM features on vibes works until a prompt tweak silently breaks ten other cases. Here is how to build evals that catch regressions before your users do.
The Augmented Dev ยท Archive
Dynamics 365, Power Platform, AI, system design, and web development โ filter by theme or search across every post.
Newsletter
One email per post. No spam, no tracking pixels, unsubscribe anytime.
12 of 325 articles tagged ai
Shipping LLM features on vibes works until a prompt tweak silently breaks ten other cases. Here is how to build evals that catch regressions before your users do.
Asking a model to 'return JSON' and parsing the result is how you get 3 a.m. pages. Tool schemas, constrained decoding, and validation turn a probabilistic text generator into a reliable API.
Most RAG systems that retrieve bad context aren't failing at embeddings or reranking โ they're failing at chunking. How you split documents quietly decides what your model can ever find.
How to expose Dynamics 365 Finance and Operations to Claude through MCP โ the architecture, the data entity choices, the auth model, and the operations that should never be one prompt away.
Use Claude Code hooks to enforce policy at the moment actions happen โ before edits, before tool calls, on session start โ without relying on pre-commit hooks everyone learns to bypass.
From zero to a working Model Context Protocol server in about 100 lines โ tools, resources, a local client test, and the traps that will bite you on day one.
A precise look at Claude's prompt caching โ how the cache hit math actually works, the 5-minute and 1-hour TTL variants, and the subtle mistakes that silently disable it.
The tool-use features of the Claude API look straightforward in the docs. These are the production patterns that separate a demo from a system that handles real traffic โ parallel tool calls, retry loops, error recovery, and tool_choice tactics.
The Claude Agent SDK is the fastest path to a capable coding-style agent. The raw messages API is the fastest path to understanding what your agent is doing. Here's when each is the right tool.
Retrieval-augmented generation demos look great. Production RAG is a different engineering problem โ chunking, hybrid retrieval, reranking, evaluation, and the failure modes nobody mentions at conferences.
A practical comparison of the vector databases people actually deploy in 2026 โ and an honest look at when a vector database is the wrong tool for the job.
LLM-powered pipelines break in the same ways old monoliths did โ when you stack synchronous calls. Queues, events, and the patterns that make a six-step AI pipeline survive a partial outage.