Agentic AI Mastery Get the full course →

They teach you to build an agent.
This teaches you to ship one that survives production.

Evaluation, observability, reliability, security, and governance for real agentic systems — the last mile academic and edtech courses skip. Three modules free.

Get the full course →Start free →

What you'll actually build

Evaluation & TDD

Most courses teach you to build an agent — not to know whether it's actually good, or to catch regressions before users do.

What you build: A production eval harness: golden datasets, an eval-gate in CI, and TDD for agents (Module 3.1).

The artifact: A golden-set pass-rate report that blocks a bad deploy.

Real-world anchor: the instructor's PACCA healthcare platform ships behind a 100-case golden set at a 100% eval-gate pass.

Observability & Reliability

A service fails loudly with a 500. An agent fails silently — it keeps generating tokens, confidently wrong.

What you build: Distributed tracing across LLM calls, tools, and sub-agents; reliability and failure-recovery patterns (Modules 3.2–3.3, 2.4).

The artifact: An observability trace + a multi-agent topology diagram pinpointing where a silent failure propagated.

Covers single-agent, orchestrator-workers, and decentralized handoff topologies most curricula never reach.

Security, Governance & Cost

Before an agent touches production it has to be secure, governed, and cost-controlled — the part academic courses skip.

What you build: OWASP-LLM-Top-10 threat modeling, guardrails, governance artifacts, and cost/latency optimization (Modules 3.4–3.7).

The artifact: A threat model + guardrail spec and a cost-latency curve for the reference system.

Real-world anchor: the instructor's CRISP-AG framework governs enterprise agentic AI to ISO 42001 / NIST AI RMF.

A taste of the rigor · FA-1.1 — Fix These Prompts

For each broken prompt, classify its failure mode (from the taxonomy in Segment 1) and sketch a fix. Reveal the answer to self-check, then continue to the next one.

Broken prompt: Tell me about the customer.

Input: a 3-paragraph customer complaint email.

The full curriculum — 18 modules, ~150 hours

Three modules are free below. The rest — retrieval, agentic orchestration, and production engineering — is the paid depth.

M0: Foundations On-Ramp

0.1 How LLMs Actually Behave
Tokens, sampling, context, and why output is probabilistic — the mental model everything else rests on.
0.2 Python-for-AI, SDKs & Structured Output
Reproducible environments, the provider-neutral seam, and schema-valid structured output.

M1: Retrieval & Grounding

1.1 Prompting & Structured Output for Reliability
Systematic prompting and a prompt-evaluation harness that scores variants on a labeled set.
1.2 Tool Use / Function Calling
The tool-use loop, typed tool design, and a multi-tool agent built once raw and once with a framework.
8h · Pro
1.3 Embeddings & RAG Fundamentals
Build the ingest → chunk → embed → store → retrieve → generate pipeline from scratch.
9h · Pro
1.4 Advanced & Agentic RAG + RAG Evaluation
Re-ranking, hybrid search, agentic retrieval, and measuring quality with RAGAS.
11h · Pro

M2: Agentic Orchestration

2.1 The Agent Loop: Reflection, ReAct, Plan-Execute
Build a ReAct loop from scratch, add reflection, and know when an agent is the wrong tool.
9h · Pro
2.2 Memory & State
Short- and long-term memory, episodic/semantic/procedural types, and context-budget management.
9h · Pro
2.3 Workflow Patterns
Prompt chaining, routing, parallelization, evaluator-optimizer, and orchestrator-workers.
8h · Pro
2.4 Multi-Agent Systems & Topologies
Single-loop vs manager vs decentralized handoffs, shared state, and error propagation.
10h · Pro
2.5 Interoperability: MCP, A2A, Contracts & HITL
Build an MCP server, write orchestration contracts, and gate high-risk actions with human-in-the-loop.
9h · Pro

M3: Production Engineering

3.1 Evaluation & Test-Driven Development for Agents
Offline/online eval, LLM-as-judge grader design, regression-in-CI, and TDD for agents.
11h · Pro
3.2 Observability & Monitoring
OpenTelemetry tracing for every LLM/tool/handoff, plus cost/latency/drift dashboards.
8h · Pro
3.3 Reliability & Failure Recovery
A failure-mode taxonomy, retries, circuit breakers, checkpointing, escalation, and chaos drills.
9h · Pro
3.4 Security & Guardrails (OWASP LLM Top 10)
Exploit and defend direct and indirect prompt injection; layered guardrails and red-teaming.
9h · Pro
3.5 Governance & Responsible AI
Turn NIST AI RMF, ISO/IEC 42001, and the EU AI Act into reviewable governance artifacts.
7h · Pro
3.6 Synthetic Data Generation
Generate eval sets, adversarial edge cases, and privacy-safe demo data.
7h · Pro
3.7 Cost & Performance Optimization
Caching, batching, model routing, and token budgeting against explicit SLOs.
7h · Pro

Who's teaching this

David Reed — Head of AI/ML & Agentic Delivery — PhD CS, MBA, Wharton fellow

Proven pedagogy
  • 61.4% FAANG placement rate (vs ~5% industry baseline)
  • 22% performance lift across 400+ MAANG instructors
Real production work
  • PACCA — production multi-agent healthcare AI, eval-gated, HIPAA / FDA-SaMD
  • CRISP-AG — enterprise agentic-AI governance framework (ISO 42001 / NIST AI RMF)
  • ALCA & LPA — edtech agents with 20–90% measured time savings

Inventor on the Amazon recommendation-engine patent (US 6,850,988).

Get the full course → See the portfolio

Get the module drops + a free production-readiness checklist