Agentic AI Mastery · M1.1 — Prompting & Structured Output

M1.1 — Prompting & Structured Output

The discipline of making model output reliable — measured with numbers, not vibes.

🟪 Hiring signal: Prompt engineering (req-006) and structured output / JSON schema (req-026) each score 3/3 and appear at high frequency across harvested 2026 JDs. This module is where you stop guessing at prompts and start engineering them.

career-forge.org | v1.0.0 | 2026-06-14
M1.1 · The Two Engineers

The Problem You Will Solve

Two engineers are debugging the same flaky AI feature.

  • Engineer A adds exclamation marks to the system prompt — "answer accurately!!!" — re-runs it once, and says "I think that helped."
  • Engineer B runs a 30-example labeled eval before and after every change, and reports the delta in accuracy.

Engineer A is guessing. Engineer B is engineering.

🟪 Hiring signal: A prompt is a software artifact — it needs version control, regression tests, and a change log. That habit is what req-006 actually screens for.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Why This Module → Career Anchor

Requirement JD frequency M1.1 coverage
Prompt engineering (req-006) High Score 3 — practiced + measured in labs
Structured output / Pydantic / JSON schema (req-026) High Score 3 — schema as a reliability tool
Systematic LLM evaluation High Score 2 — A/B harness, confusion matrix

🟩 Production tip: Citing context outperforms adding exclamation marks because it exploits how transformer attention works — not because it is polite. Every rung of the ladder has a measurable effect on a labeled set.

🟪 Archetype strongest fit: AI Engineer — practical fluency across the LLM ecosystem (the GitLab Senior AI Engineer JD names exactly this).

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Before We Begin — What You Need

You MUST already own (from M0.2):

  • Structured output: you can enforce a JSON schema with Pydantic + instructor, and you know the difference between prompt-only JSON and schema-enforced output.
  • Client SDK loop: you can make a complete litellm.completion() call and inspect the response.
  • .env hygiene: you have a working local environment with secrets loaded from a .env file — no hardcoded keys.
  • Tooling installed: litellm, pydantic, instructor, plus basic json and jsonlines familiarity.

🟨 Gotcha — Missing these? Do not proceed. Return to M0.2 (Python SDKs & Structured Output) first — this module builds directly on its llm.py seam.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Bridge — Where M1.1 Fits

The curriculum arc: Foundation → Retrieval → Orchestration → Production

M0.2 Structured Output  ──►  M1.1 (you are here)  ──►  M1.2 Tool Use
        ↑                            ↑
  Output is well-formed      Reasoning is reliably correct
  (schema enforcement)       (measured on a labeled set)

M0.2 showed you how to make the model's output structurally trustworthy. M1.1 teaches you to make the model's reasoning reliably correct — and to measure that reliability with numbers, not vibes.

🟦 Connecting concept: Tool descriptions in M1.2 are prompts. The engineering discipline you build here applies directly to them.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

New Terms in This Module

Term Definition
Prompt ladder Anthropic's ordered hierarchy of prompt techniques, from clear-and-direct up to grounding constraints — applied lowest-rung-first.
A/B eval Comparing two prompt versions against the same labeled dataset to measure which performs better.
Prompt registry A versioned store of prompt templates with metadata: version, author, eval score, date.
Confusion matrix For classification: true/false positives and negatives per class — assesses quality beyond a single accuracy number.

🟦 Pre-training principle: These four terms recur in every slide that follows. Meeting them now keeps them from adding cognitive load mid-explanation.

career-forge.org | v1.0.0 | 2026-06-14
M1.1 · Concept 1 — The Ladder

Concept 1 — Anthropic's Prompt-Engineering Ladder

Ordered by impact-per-effort. Apply from the bottom up — only add complexity when lower rungs fail.

Seven-rung prompt-engineering ladder: clear and direct, assign a role, separate data from instructions, give examples, step-by-step reasoning, prefill, avoid hallucination

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Concept 1 (cont.) — Reading the Ladder

The rungs are levers, not tips — each has a measurable effect on a labeled eval set:

  • Rungs 1–3 (clear, role, delimited data): cheapest, highest impact. Most reliability problems are solved here.
  • Rungs 4–5 (few-shot, chain-of-thought): add tokens and latency; reserve for hard or ambiguous cases.
  • Rungs 6–7 (prefill/schema, grounding constraint): lock the format and constrain invention — rung 6 is the M0.2 schema technique, re-cast as a reliability lever.

🟩 Production tip: Each rung adds cost and latency. Use the lowest rung that achieves your quality target — and prove it on the labeled set.

🟨 Gotcha — Enthusiasm-as-Signal: "answer accurately!!!" adds no structural signal. The model attends to content, not exclamation marks.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Concept 2 — Structured Output as a Reliability Tool

Structured output is not just machine-parseability — it is a reliability technique. Forcing {"category": "...", "reason": "..."} eliminates whole failure modes:

  • Hedging — "well, it depends..." disappears when a schema demands a category.
  • Format drift — every call returns the same shape, so downstream code never branches on prose.
  • Decision-dodging — the model cannot "wriggle out" of committing to a value.

🟩 Production tip: Every LLM call that feeds a downstream system should have a schema. No schema = no contract = no composability. Combine rung 6 (schema) with rung 3 (cite context) for extraction tasks.

🟥 Anti-pattern — Prompt-only JSON at rung 1: asking for JSON in plain text without schema enforcement. It parses most of the time — and the failures land in production.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Concept 2 (cont.) — Schema-Enforced Classification

Rung 6 in code: a tool schema makes category a typed enum the model must fill.

TOOL = {
    "type": "function",
    "function": {
        "name": "classify",
        "parameters": {
            "type": "object",
            "properties": {
                "category": {"type": "string",
                             "enum": ["REFUND_ELIGIBLE", "EXCHANGE_ONLY",
                                      "INSUFFICIENT_INFO"]},
                "reason": {"type": "string"},
            },
            "required": ["category", "reason"],
        },
    },
}

🟦 Concept: The enum constraint is the reliability lever — the model cannot return a category outside your three valid values.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Concept 3 — Systematic A/B Eval on a Labeled Set

Intuition-driven prompting is a dead end: you iterate once, it seems better, you ship — then discover the gain was on the three cases you tested while untested ones regressed.

The workflow (same rigor as software testing):

  1. Build a dataset — 20–50 examples with ground-truth labels (start at 30).
  2. Define a metric — exact-match for classification; a confusion matrix for which class fails.
  3. Run variants — execute Prompt A and Prompt B against the full set.
  4. Report win-rates — "B beats A on 15/30, ties on 10, loses on 5."

🟨 Gotcha — Sample size: 30 examples catch gross failures; 300+ catch subtle regressions. Start at 30 and expand — never treat 30 as a production guarantee.

career-forge.org | v1.0.0 | 2026-06-14
M1.1 · Concept 3 — The Harness

Concept 3 (cont.) — The A/B Harness

Same dataset, two prompts, one comparison. This harness is the MP-1 seed.

A/B prompt-eval harness: one labeled dataset feeds Prompt A and Prompt B; each produces scores and a confusion matrix; comparing the delta decides whether to ship B or reject the change

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Concept 4 — Prompt Versioning & Regression

Prompts are code. They have versions, they have tests, and they can regress. A change that fixes three cases while breaking one existing case is a regression — you only catch it against a fixed labeled set.

Minimal implementation:

  • Store prompts as versioned constants or YAML, not inline strings. Each has an ID, a version, an eval score, and a change description.
  • Regression gate: if a new version scores lower than the previous one on the golden set → reject.
  • CI integration: run the eval harness on every prompt change in the repository.

🟩 Production tip: Treat prompt changes like code changes — they deserve PR review and automated tests. GitHub Copilot's team maintains a benchmark suite of completions that prompt changes must not regress on.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Check Your Reasoning — Before We Code

Formative Check 1 of 2 | Bloom level: Understand | ~8 minutes

This is not a grade. It is a mirror. Answer confidently and correctly, and you are ready for the worked example.

Q: Your system prompt currently says "Please answer accurately and helpfully!!!" A teammate suggests replacing it with "You are a senior support specialist. Cite the policy provided, then think step by step." Without running an eval, which change is more likely to improve reliability — and why?

  • A) The exclamation-mark version — more emphasis signals importance.
  • B) The teammate's version — it applies rungs 2, 3, and 5, which change how the model attends to the input.
  • C) They are equivalent; only the model version matters.

🟩 Take the full check inside the platform — it grades you live and explains every distractor.

▶ Open Check 1 (Understand) — multiple-choice + short answer · ~8 min

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Worked Example — Climbing the Ladder on a Refund Classifier

The task: classify a customer return request as REFUND_ELIGIBLE, EXCHANGE_ONLY, or INSUFFICIENT_INFO, scored against a labeled set.

We climb the ladder rung by rung, recording the score at each step:

  1. Rung 1 — bare prompt. Baseline: does it even commit to a category?
  2. Rung 3 — delimited policy + output format. Ground the decision in the policy.
  3. Rung 5 — chain-of-thought. "Think step by step before the final JSON."
  4. Rung 6 — schema enforcement. Force the typed enum so the category is always valid.

🟩 What to observe: each rung produces more reliable, more structured output — and the labeled set tells you how much more.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Code Walk — Rung 1 → Rung 3

MODEL = os.getenv("LLM_MODEL", "claude-sonnet-4-6")

def complete(content):
    r = litellm.completion(
        model=MODEL, temperature=0, max_tokens=512,
        messages=[{"role": "user", "content": content}],
    )
    return r.choices[0].message.content.strip()

# Rung 1: bare prompt — no policy, no format
print(complete(CUSTOMER_MSG))

# Rung 3: delimited policy + explicit output format
prompt3 = f"""
<policy>{POLICY}</policy>
<customer_message>{CUSTOMER_MSG}</customer_message>
Reply as JSON, based only on the policy: {{"category": "...", "reason": "..."}}
"""
print(complete(prompt3))

🟦 Concept: rung 3's delimiters mark where policy ends and the request begins — a boundary that also guards against injection.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Code Walk — Rung 5 → Rung 6

# Rung 5: chain-of-thought — append a reasoning instruction
prompt5 = prompt3 + "\n\nThink step by step before giving your final JSON."
print(complete(prompt5))

# Rung 6: schema enforcement — the category MUST be a valid enum
r = litellm.completion(
    model=MODEL,
    messages=[{"role": "user", "content": prompt3}],
    tools=[TOOL],
    tool_choice={"type": "function", "function": {"name": "classify"}},
    temperature=0, max_tokens=512,
)
args = r.choices[0].message.tool_calls[0].function.arguments
print(json.dumps(json.loads(args), indent=2))

🟩 Production tip: rung 6 with tool_choice forced is the cross-provider equivalent of Anthropic prefill — the model must call your tool and match the schema.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Results — The Trade-off Is Visible

Climbing the ladder on the labeled refund set (illustrative — your numbers come from your own run):

Rung Technique Accuracy Relative cost/call
1 Bare prompt 2/5 1.0×
3 Delimited policy + format 4/5 1.1×
5 + Chain-of-thought 4/5 1.8×
6 + Schema enforcement 5/5 1.2×

🟦 Concept: rung 3 bought the biggest jump for almost no cost. Rung 5 added cost without moving accuracy here — so on this task, you would not ship it. Rung 6 locked the format for a small cost.

🟨 Gotcha: these numbers are from one small set. Re-run on 30+ before drawing conclusions — the win-rate on five cases is not significant.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Apply It — Plan Your Next Three Changes

Formative Check 2 of 2 | Bloom level: Apply | ~10 minutes

You have a prompt that classifies customer support tickets into 5 categories. Current accuracy on your 30-example set is 73%.

Q: Describe the exact sequence of 3 changes you would try next, in ladder order, and how you would determine whether each one actually improved things.

A strong answer:

  • Apply rungs in order — try rung 3 (cite/ delimit) before reaching for rung 5 or 7.
  • For each change: run the A/B harness, compute the delta, and read the confusion matrix to see which category improved.
  • If the delta is below your threshold → discard the change and keep the prior version.

🟩 Type your plan below — the platform grades it against a root-cause / method / measurement rubric.

▶ Open Check 2 (Apply) — natural-language plan · up to 3 attempts · ~10 min

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Lab Brief — Your Turn

HW-1.1: Take the provided 30-example task; write 3 prompt variants; report win-rates with a confusion matrix.

What you build:

  • 3 prompt variants in your registry (may start from the lab variants, but must differ meaningfully).
  • A 3×3 confusion matrix per variant — REFUND_ELIGIBLE × EXCHANGE_ONLY × INSUFFICIENT_INFO.
  • A results table: variant | accuracy | most-common error type.

"Done" looks like:

  1. All 3 variants run on the full 30-example set — not just 5.
  2. Confusion matrices are numerically correct.
  3. The best variant reaches ≥ 75% accuracy, and your write-up explains why it wins.

🟦 Stretch: add a simple LLM-judge that scores the reason field separately from the category, and compare the two metrics.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

How to Submit

Two supported paths — pick one:

1. Platform (primary): submit through the in-page lab UI; the AI grader scores against the rubric and returns line-level feedback in about 30 seconds.

▶ Submit your lab — AI-graded

2. Production track (GitHub fork): fork drdgreed/career-foundry, add your solution under submissions/<your-handle>/m1-1/, and open a PR. The same grader runs in CI as a PR comment.

🟩 Production tip: The CI grader on your fork is the same harness the platform runs — your prompt eval is gated exactly the way a production LLM team would gate it.

Roadmap: see ROADMAP.md for cohort support channels and the CI-grader rollout.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

What You Build — MP-1 Prompt-Eval Harness

By the end of the lab you have a standalone CLI tool — the MP-1 seed:

  • Reads a JSONL dataset (one {id, input, expected} per line).
  • Runs ≥ 3 prompt variants from a versioned registry.
  • Outputs a scored comparison table (exact-match, optional LLM-judge).
  • Saves results to JSONL for regression tracking.
python mp1_harness.py --dataset data/returns_eval.jsonl --output results/run_001.jsonl

🟪 Portfolio signal: this harness is reused and upgraded in M3.1 to become your full agent-evaluation framework. It is the first piece of evaluation infrastructure in your portfolio.

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Reflection Prompt

Take 5 minutes before you start HW-1.1. Write your answers in a text file.

1. Which rung produced the biggest accuracy jump — and what does that reveal about your baseline's errors?

If rung 3 (cite context) moved the number most, your baseline was likely hallucinating policy details it was never given.

2. In your current or target role, which AI features would benefit most from a versioned prompt registry and regression tests — and who would own it?

Name a concrete feature and a concrete owner.

3. What is the minimum eval-set size you would accept before deploying a prompt to production? Write your reasoning.

Defend the number — 30 is a floor, not a finish line.

📖 Full reflection page: Open on career-forge.org (offline mirror).

career-forge.org | v1.0.0 | 2026-06-14
Agentic AI Mastery · M1.1 — Prompting & Structured Output

Self-Assessment Rubric

Score yourself 0–3 on each criterion. Threshold to advance to M1.2: 11/15.

Criterion 0 — Not yet 1 — Emerging 2 — Proficient 3 — Exemplary
Ladder application Can name rungs Applies one or two Climbs in order on a real task Justifies each rung by measured impact
Structured output as reliability Sees it as formatting Uses a schema Combines schema + cite-context Explains failure modes it removes
A/B eval discipline Eyeballs outputs Runs on a few cases Runs full 30 + win-rate Reports confusion matrix per class
Versioning & regression Inline prompt strings Stores versions Regression gate on golden set Gate wired into CI
Reasoning quality "It felt better" States a delta Attributes delta to a rung Explains why via attention/grounding

📊 Self-assessment page: Open on career-forge.org (offline mirror).

Next module: M1.2 — Tool Use & Function Calling. You will give the model actions, not just text. Tool descriptions are prompts — the discipline you just built decides whether the model calls the right tool at the right time.

career-forge.org | v1.0.0 | 2026-06-14

ACT 1: HOOK & ANCHOR — Slides 1–3

_header: "" _footer: ""

_notes: (1) KEY CONCEPTS: Welcome to M1.1, the first module of Milestone 1 (Retrieval & Grounding). You arrive here with structured output from M0.2: you can enforce a Pydantic schema and make a complete client-SDK call. M1.1 takes the next step — it treats prompting as an engineering discipline with measurable outcomes. The central thesis: "Write a better prompt" is not a strategy; "Measure output quality on a 30-example labeled set, identify failure categories, iterate, and report win-rates" is. Two real, high-frequency hiring requirements anchor the module: req-006 (prompt engineering) and req-026 (structured output / Pydantic / JSON schema), each scored 3/3 in the coverage matrix. (2) EMPHASIZE: This module is not a list of prompt tricks. It is a method — the same measure-don't-guess discipline that scales into the full evaluation stack in M3.1. The prompt-eval harness you build here (MP-1) is the seed of that stack. (3) MISCONCEPTION: The most common pre-module misconception is the Enthusiasm-as-Signal Misconception — the belief that tone or emphasis ("answer accurately!!!") functions in a prompt the way emphasis works in human writing. We name and correct it in Act 3. (4) REFERENCES: - [Coverage Matrix — req-006 prompt engineering, req-026 structured output](https://github.com/drdgreed/career-foundry) - [Anthropic Prompt Engineering Guide](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering)

_notes: (1) KEY CONCEPTS: This contrast frames the whole module. Most engineers who ship LLM features that "work most of the time" are Engineer A — they change a prompt, eyeball one or two outputs, and ship. The hidden cost: the improvement was on the cases they happened to test, while untested edge cases silently regressed. Prompts also degrade on their own as providers update models upstream — what worked last quarter may quietly fail this quarter. Engineer B's labeled-eval discipline is the only way to detect either failure mode. (2) EMPHASIZE: Prompt engineering without measurement is just intuition with extra steps. Every untested change to a prompt is a latent bug waiting to reach production. (3) MISCONCEPTION: "More forceful language makes the model comply." Tone is not a structural signal to a transformer. Naming this now sets up the ladder in Act 3. (4) REFERENCES: - [Anthropic — Building Effective Agents (2024-12)](https://www.anthropic.com/research/building-effective-agents) - [GitHub Copilot prompt-benchmark discipline](https://github.com/drdgreed/career-foundry)

_notes: (1) KEY CONCEPTS: These numbers come from the coverage matrix — real harvested 2026 job descriptions, not estimates. Prompt engineering (req-006) and structured output (req-026) both score 3/3 at high frequency. Unlike M0.1, this module produces a portfolio artifact: the prompt-eval harness (MP-1). That is why the coverage scores here reach 3, not 1. (2) EMPHASIZE: "Practical fluency across the LLM ecosystem" is the literal phrasing of the GitLab Senior AI Engineer JD — and it is exactly what a versioned prompt registry plus an eval harness demonstrates. (3) MISCONCEPTION: A learner may assume structured output is "just an API parameter." It is a reliability technique — we make that case in Concept 2. (4) REFERENCES: - [Coverage Matrix — req-006, req-026](https://github.com/drdgreed/career-foundry) - [Anthropic Prompt Engineering Guide](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering)

ACT 2: PREREQS & BRIDGE — Slides 4–6

_notes: (1) KEY CONCEPTS: M1.1 assumes M0.2 is complete. The hard dependency is structured output: you must already be able to enforce a Pydantic schema and distinguish prompt-only JSON from schema-enforced output. If that is shaky, the rung-6 material (schema as a reliability tool) will not land. The labs reuse the litellm seam from M0.2 verbatim, so a working .env and a known-good completion() call are non-negotiable gates. (2) EMPHASIZE: These prerequisites are binary gates. If you cannot make a complete schema-enforced call before this module, the worked example will be confusing rather than illuminating. (3) MISCONCEPTION: "I know prompting already." Knowing prompt tricks is not the same as being able to measure a prompt change on a labeled set — which is the actual M1.1 skill. (4) REFERENCES: - [litellm docs](https://docs.litellm.ai) - [python-dotenv README](https://github.com/theskumar/python-dotenv)

_notes: (1) KEY CONCEPTS: M0.2 contained variance at the output boundary — schema enforcement means a malformed answer can no longer become invalid JSON. M1.1 attacks the next layer: the content of the answer. A schema guarantees you get a category back; it does not guarantee the category is correct. That correctness is what a labeled eval measures. The bridge to M1.2 is concrete: a tool description is a prompt, and its quality determines whether the model calls the right tool at the right time. ALT TEXT: A linear diagram showing M0.2 Structured Output → M1.1 → M1.2 Tool Use. An upward arrow under M0.2 reads "Output is well-formed (schema enforcement)"; an upward arrow under M1.1 reads "Reasoning is reliably correct (measured on a labeled set)." (2) EMPHASIZE: Schema enforcement and correctness are different problems. M0.2 solved the first; M1.1 solves the second. (3) MISCONCEPTION: Some learners expect a "prompt cookbook." M1.1 is a method, not a recipe list — the ladder is ordered by measured impact-per-effort. (4) REFERENCES: - [Curriculum spec — Module M1.1](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [Anthropic Prompt Engineering Guide](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering)

_notes: (1) KEY CONCEPTS: Four terms anchor the module. The prompt ladder is the conceptual spine. A/B eval is the measurement method. The prompt registry is the artifact that makes prompts versionable like code. The confusion matrix is the diagnostic that tells you which class your prompt is getting wrong — not just that accuracy dropped. You do not need to memorize these now; meeting them once gives you a hook for the detailed treatment in Act 3. (2) EMPHASIZE: "Confusion matrix" is the most important entry — a single accuracy number hides which category is failing. The matrix shows you exactly where to aim the next prompt change. (3) MISCONCEPTION: Learners often treat accuracy as the only metric. Accuracy alone can mask a prompt that is great on three classes and useless on a fourth. (4) REFERENCES: - [Anthropic Prompt Engineering Guide](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering) - [Mayer 2009 — Multimedia Learning, Pre-training principle](https://www.hartford.edu/faculty-staff/faculty/fcld/_files/12%20Principles%20of%20Multimedia%20Learning.pdf)

ACT 3: CONCEPT BUILD-UP — Slides 7–14

_notes: (1) KEY CONCEPTS: The ladder is the conceptual core of the module. It is ordered by impact-per-effort, and the rule is to climb from the bottom: only add a higher rung when a lower one fails to hit your quality target. Rung 1 — be clear and direct ("Classify this as REFUND, BILLING, or TECHNICAL" beats "help with this"). Rung 2 — assign a role, narrowing the output distribution toward your domain. Rung 3 — separate data from instructions with delimiters or XML tags; this is also a security property, preventing instruction injection from user content. Rung 4 — give one to three well-chosen few-shot examples. Rung 5 — ask for step-by-step reasoning (chain-of-thought) for multi-step or ambiguous tasks. Rung 6 — prefill or speak for the model to force the output format (links straight to M0.2 schema enforcement). Rung 7 — explicitly constrain against hallucination by grounding ("if it is not in the policy, say you don't have enough information"). The canonical lesson fact (Q-M1-01): citing context beats exclamation marks because it exploits attention, not social cues. ALT TEXT: A left-to-right flowchart of seven nodes labeled rung 1 through rung 7: be clear and direct; assign a role; separate data from instructions; give examples (few-shot); step-by-step (chain-of-thought); prefill / speak for model; avoid hallucination. The lower rungs are styled green, the middle purple, the top two yellow. (2) EMPHASIZE: Each rung adds cost and latency. Use the lowest rung that achieves your quality target — do not jump straight to chain-of-thought. (3) MISCONCEPTION: The Sky-High-Rung Misconception — reaching for rung 5 or 7 before trying rung 3. Cheaper rungs often close most of the gap. (4) REFERENCES: - [Anthropic Prompt Engineering Guide](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering) - [Lesson M1.1 — Segment 1, the ladder](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output)

_notes: (1) KEY CONCEPTS: This slide reframes the ladder as a cost ladder. Rungs 1 through 3 are nearly free in tokens and latency and resolve the majority of reliability problems — clear instructions, a role to narrow the distribution, and delimited data to prevent injection. Rungs 4 and 5 cost real tokens (few-shot examples and chain-of-thought reasoning both expand the prompt and the output), so reserve them for cases where the cheaper rungs leave measurable gaps. Rungs 6 and 7 are about locking the contract: rung 6 is exactly the M0.2 schema enforcement re-cast as a reliability lever, and rung 7 is an explicit grounding constraint against hallucination. (2) EMPHASIZE: The discipline is "lowest rung that hits target, proven on the labeled set" — not "every rung, always." (3) MISCONCEPTION: The Enthusiasm-as-Signal Misconception — tone and emphasis in a prompt do not function like emphasis in human writing. The transformer attends to content structure, not punctuation intensity. (4) REFERENCES: - [Anthropic Prompt Engineering Guide](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering) - [Anthropic — Building Effective Agents (2024-12)](https://www.anthropic.com/research/building-effective-agents)

_notes: (1) KEY CONCEPTS: This connects rung 6 of the ladder back to M0.2 and reframes it. In M0.2, schema enforcement was about getting parseable output. Here it is about reliability: when you force the model to emit a typed object instead of prose, you delete entire failure modes at once. Hedging ("it depends") cannot survive a required enum field. Format drift cannot happen when the shape is fixed. And the model cannot dodge a decision when the schema demands a value. The strongest combination for extraction tasks is rung 6 (schema) plus rung 3 (cite context): the schema fixes the shape, the citation grounds the content. (2) EMPHASIZE: No schema = no contract = no composability. Any call whose output feeds another system must have a schema. (3) MISCONCEPTION: The anti-pattern is prompt-only JSON at rung 1 — "reply as JSON" with no schema enforcement. It works in testing and fails on the long tail. The cross-provider fix is a tool schema with forced tool choice, equivalent to Anthropic prefill in reliability. (4) REFERENCES: - [Lesson M1.1 — Segment 2, structured output as reliability](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [litellm tool / function calling docs](https://docs.litellm.ai)

_notes: (1) KEY CONCEPTS: This is the concrete form of rung 6. The tool schema declares a function named classify whose category parameter is an enum restricted to exactly three values. When you call the model with this tool and force tool choice, the model cannot return a category outside that set, and the reason field is required. This is the cross-provider equivalent of Anthropic prefill: instead of partially filling the assistant turn, you constrain the output to match a schema. The reliability win is structural — invalid categories become impossible, not just unlikely. (2) EMPHASIZE: The enum is doing the heavy lifting. A free-text category field would let the model invent "MAYBE_REFUND" — the enum forbids it. (3) MISCONCEPTION: "A schema slows me down." The schema is defined once and reused across every call; the marginal cost is near zero and it removes a class of downstream bugs. (4) REFERENCES: - [Lesson M1.1 — Lab 2, structured-output reliability](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [litellm tool calling](https://docs.litellm.ai)

_notes: (1) KEY CONCEPTS: This is the measure-don't-guess core. The failure mode it prevents is the classic one: a prompt change that fixes the handful of cases you eyeballed while silently breaking others you did not check. The fix is the same as in software engineering — a fixed labeled dataset and a reproducible metric. Build 20 to 50 examples with diverse inputs and balanced failure modes (start at 30, the statistical floor for detecting meaningful differences). Use exact-match for classification, and a confusion matrix to see which class is failing. Run both variants on the full set and report win-rates, not anecdotes. The labeled set doubles as your regression dataset — it guards future changes against breaking existing behavior. (2) EMPHASIZE: Report win-rates over the whole set, not impressions from a few outputs. "Better on 2 of the 3 I looked at" is not a result. (3) MISCONCEPTION: The 30-is-enough fallacy (Q-M1-02). Thirty catches gross failures; production reliability needs 300-plus. The lesson teaches you to start at 30 and expand — not to stop there. (4) REFERENCES: - [Lesson M1.1 — Segment 3, dataset/metric/A-B](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [Anthropic — Building Effective Agents (2024-12)](https://www.anthropic.com/research/building-effective-agents)

_notes: (1) KEY CONCEPTS: This diagram is the architecture of the harness you build in Lab 3 and ship as MP-1. A single labeled dataset feeds both Prompt A and Prompt B; each variant is run across all examples; each produces a scores object and a confusion matrix; the comparison step computes the delta and the win-rate. The decision is binary against the golden set: if B beats A, ship B and bump its version in the registry; if B regresses, reject the change and keep A. This is exactly the regression gate that production LLM teams (e.g., GitHub Copilot) run on every prompt change. ALT TEXT: A left-to-right flowchart. A green "Labeled dataset (30 examples + ground truth)" node fans out to two purple nodes "Prompt A" and "Prompt B"; each leads to a teal scores node; both scores nodes feed a yellow decision diamond "Compare delta / win-rate"; the diamond branches green to "Ship B, bump version" and red to "Reject change, keep A." (2) EMPHASIZE: The harness is reusable infrastructure, not throwaway lab code. It is upgraded into the full agent-evaluation framework in M3.1. (3) MISCONCEPTION: "I'll compare prompts by reading outputs side by side." That does not scale and is not reproducible. The harness gives you a number and a regression target. (4) REFERENCES: - [Lesson M1.1 — Lab 3, A/B eval harness (MP-1 seed)](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [Career-Forge repo](https://github.com/drdgreed/career-foundry)

_notes: (1) KEY CONCEPTS: This is the operationalization of everything prior. Prompts get the same treatment as code: stored in a versioned registry (Python constants or YAML, never scattered inline strings), each entry carrying an ID, version, eval score, and change note. When you change a prompt, you bump its version and run the eval suite. The regression gate is the key control: if the new version scores lower than the previous one on the golden set, you reject it before it ships. Wire that gate into CI so it runs automatically on every prompt change. This is precisely what GitHub Copilot's team does — a benchmark suite of thousands of completions that new prompt changes must not regress on. A 2% gain on one slice that ships without checking the other 98% is not progress. (2) EMPHASIZE: A regression is a fix that breaks something that previously worked. Without a fixed labeled set, you cannot detect one — you only see the cases you happened to retest. (3) MISCONCEPTION: "Versioning prompts is overkill for a small project." The moment you have more than one prompt and more than one person, an unversioned prompt is an untracked dependency. (4) REFERENCES: - [Lesson M1.1 — Segment 4, versioning & regression](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [GitHub Copilot — benchmark-gated prompt changes](https://github.com/drdgreed/career-foundry)

ACT 4: FORMATIVE CHECK #1 — Slide 15 (Understand level)

_notes: (1) KEY CONCEPTS: Pause before code. The intended answer is B — the teammate's version applies rung 2 (persona), rung 3 (cite context), and rung 5 (chain-of-thought), each of which changes how the transformer attends to the input. Exclamation marks add no structural signal. But the question deliberately contains a trap in the phrase "without running an eval": you cannot know for certain which is better without measuring it. The complete answer is "B is more likely on first principles, AND you must measure it on the labeled set to be sure." This is the Understand-level check before the Apply-level worked example. (2) EMPHASIZE: The right answer has two halves — the ladder-based reasoning (why B) and the humility (measure it anyway). A learner who gives only the first half has missed the module's thesis. (3) MISCONCEPTION: The Enthusiasm-as-Signal Misconception — distractor A. Tone and emphasis in a prompt do not function like emphasis in human writing. (4) REFERENCES: - [Anderson & Krathwohl 2001 — Bloom's Revised Taxonomy, Understand level](https://quincycollege.edu/wp-content/uploads/Anderson-and-Krathwohl_Revised-Blooms-Taxonomy.pdf) - [Anthropic Prompt Engineering Guide](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering)

ACT 5: WORKED EXAMPLE + LIVE CODE — Slides 16–20

_notes: (1) KEY CONCEPTS: This worked example mirrors Lab 1 of the lesson. We take a single concrete task — refund-request classification with three categories — and climb the ladder one rung at a time, recording the score after each change. Rung 1 is the bare prompt: we check whether the model even identifies refund eligibility. Rung 3 adds a delimited policy block and an explicit output format, grounding the decision in the actual return policy. Rung 5 adds chain-of-thought. Rung 6 enforces the schema so the category is always one of the three valid values. The point is not the absolute scores but the deltas: each rung's contribution is measurable on the labeled set. (2) EMPHASIZE: We climb in ladder order and measure after each rung. That ordering is the discipline — it tells you which lever actually moved the number. (3) MISCONCEPTION: The Sky-High-Rung Misconception — skipping to chain-of-thought or schema before checking what rung 3 alone buys you. Often delimiting the data and citing the policy closes most of the gap. (4) REFERENCES: - [Lesson M1.1 — Lab 1, progressive prompt improvement](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [Anthropic Prompt Engineering Guide](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering)

_notes: (1) KEY CONCEPTS: This is the rung 1 to rung 3 transition in code. The complete() helper is the same litellm seam from M0.2, pinned to temperature 0 for a deterministic policy decision. Rung 1 sends the bare customer message — no policy, no format. Rung 3 wraps the policy and the customer message in XML-style delimiters and demands a JSON shape "based only on the policy." Two things change at once: the model is grounded in the actual policy text, and the delimiters create an unambiguous boundary between trusted instructions and untrusted user content — an injection guard, not just formatting. (2) EMPHASIZE: Delimiting data is both a quality lever and a security control. "Based only on the policy" plus delimiters is the single highest-leverage cheap change. (3) MISCONCEPTION: "Delimiters are cosmetic." They materially reduce instruction-injection risk from user-supplied content and sharpen the model's grounding. (4) REFERENCES: - [Lesson M1.1 — Lab 1 code](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [litellm completion docs](https://docs.litellm.ai/docs/completion/input)

_notes: (1) KEY CONCEPTS: This completes the climb. Rung 5 simply appends "think step by step before the final JSON" to the rung-3 prompt — chain-of-thought as a one-line change. Rung 6 switches from free-text output to a forced tool call: we pass the TOOL schema and set tool_choice to require the classify function, so the model must return arguments that match the schema, and we parse them from tool_calls. Forcing tool_choice is the cross-provider equivalent of Anthropic prefill — it guarantees both that the tool is called and that the category is a valid enum value. (2) EMPHASIZE: Rung 5 is a one-line prompt edit; rung 6 is a structural change to the call. Measure both — chain-of-thought sometimes helps and sometimes only adds latency, which is exactly why you check the number. (3) MISCONCEPTION: "Chain-of-thought always improves accuracy." Not for simple, unambiguous classification — sometimes it only adds tokens and latency. The labeled set is how you decide. (4) REFERENCES: - [Lesson M1.1 — Lab 1, rungs 5–6](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [litellm tool calling](https://docs.litellm.ai)

_notes: (1) KEY CONCEPTS: The results table makes the trade-off concrete. The numbers are illustrative — the point is the shape of the curve, which learners reproduce on their own runs. Rung 3 (delimited policy plus format) produces the largest accuracy jump at almost no extra cost, which is the typical pattern: cheap structural rungs do most of the work. Rung 5 (chain-of-thought) adds substantial cost per call but, on this simple classification task, does not move accuracy — so you would not ship it here. Rung 6 (schema enforcement) recovers the last case and locks the format for a small cost. The lesson is to read accuracy and cost together and ship the cheapest rung that hits your target. (2) EMPHASIZE: A rung that adds cost without adding accuracy is a rung you do not ship. The table shows exactly that for rung 5 on this task — and only the eval reveals it. (3) MISCONCEPTION: Drawing conclusions from five examples. Five cases cannot support a significance claim — this is why the homework expands to 30 and beyond. (4) REFERENCES: - [Lesson M1.1 — Lab 1 "what to observe"](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [Anthropic — Building Effective Agents (2024-12)](https://www.anthropic.com/research/building-effective-agents)

ACT 6: FORMATIVE CHECK #2 — Slide 21 (Apply level)

_notes: (1) KEY CONCEPTS: This is the Apply-level check. The learner must produce a concrete plan, not recognize a definition. The strong answer applies the ladder in order — try rung 3 (delimit and cite the policy) before jumping to chain-of-thought or constitutional prompts — and pairs every change with measurement: run the A/B harness, compute the delta, and inspect the confusion matrix to see which of the five categories actually improved. The decision rule is explicit: if the delta is below threshold, discard the change. This rehearses the full measure-don't-guess loop on a fresh scenario. (2) EMPHASIZE: Ladder order plus per-change measurement plus a discard threshold. A plan missing any of the three is incomplete. (3) MISCONCEPTION: The Sky-High-Rung Misconception — jumping straight to chain-of-thought or constitutional prompts without first trying the cheaper context-citation or few-shot rungs. (4) REFERENCES: - [Anderson & Krathwohl 2001 — Bloom's Revised Taxonomy, Apply level](https://quincycollege.edu/wp-content/uploads/Anderson-and-Krathwohl_Revised-Blooms-Taxonomy.pdf) - [Lesson M1.1 — Formative check FA-1.1](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output)

ACT 7: PRACTICE BRIEF — Slides 22–24

_notes: (1) KEY CONCEPTS: This is the graded homework, HW-1.1. The deliverable is three meaningfully different prompt variants run on the full provided 30-example dataset, each scored with a 3-by-3 confusion matrix across the three categories, plus a results table reporting accuracy and the most-common error type per variant. The acceptance bar: all three variants run on all 30 (not a subset), the confusion matrices are numerically correct, the best variant reaches at least 75% accuracy, and — most importantly — the write-up explains why the best variant wins, not just that it does. The stretch goal introduces an LLM-judge that grades the reason field separately, foreshadowing the judge work in M3.1. If accuracy stays under 80% on the best prompt, add 20 more examples and identify the failure mode. (2) EMPHASIZE: "Explain why it wins" is the highest-value part. Producing the table mechanically without the explanation misses the learning objective. (3) MISCONCEPTION: Running only the 5 lab cases instead of all 30. Five cases cannot support a confusion matrix or a significance claim. (4) REFERENCES: - [Lesson M1.1 — Homework HW-1.1](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [litellm docs](https://docs.litellm.ai)

_notes: (1) KEY CONCEPTS: Two submission paths, both fully supported. The platform path is primary: the in-page lab UI routes your submission to the AI grader, which scores against the rubric and returns line-level feedback quickly. The production-track path is the GitHub fork: fork the public career-foundry repo, add your solution under a per-handle submissions directory, and open a PR — the identical grader runs in CI and posts results as a PR comment. The dual-track design is deliberate: it mirrors how a real LLM team gates prompt changes behind an automated eval, and it gives learners a portfolio-visible artifact. (2) EMPHASIZE: Both paths run the same grader. The fork path is not a second-class option — it is the production-realistic one and produces a public PR you can show in interviews. (3) MISCONCEPTION: "The two paths grade differently." They share one harness; the score is the same regardless of path. (4) REFERENCES: - [Career-Forge platform — M1.1](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [drdgreed/career-foundry repo](https://github.com/drdgreed/career-foundry)

_notes: (1) KEY CONCEPTS: MP-1 is the milestone artifact. It combines Lab 3 (the A/B harness) and Lab 4 (the versioned registry) into one standalone CLI: it reads a JSONL dataset, runs at least three registered prompt variants, outputs a scored comparison table (exact-match plus an optional LLM-judge score), and saves the per-row results to JSONL so they can be diffed for regressions. The invocation shown is the canonical interface. This is not throwaway lab code — it is reused and upgraded in M3.1 into the full agent-evaluation framework, which is why it is worth writing cleanly now. (2) EMPHASIZE: MP-1 is portfolio-grade evaluation infrastructure. Hiring managers screening for req-006 and req-026 want to see exactly this: a learner who builds the harness, not one who eyeballs outputs. (3) MISCONCEPTION: "It's just a script." The JSONL output is the regression record; without it, you have a one-off comparison, not infrastructure. (4) REFERENCES: - [Lesson M1.1 — Mini-Project MP-1](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [drdgreed/career-foundry repo](https://github.com/drdgreed/career-foundry)

ACT 8: REFLECTION & BRIDGE — Slides 25–26

_notes: (1) KEY CONCEPTS: These three reflection questions close the module and are meant to be written, not just thought. Question 1 turns the worked example into diagnosis: the rung that moved the number most tells you what kind of error your baseline was making — a big jump at rung 3 implies the baseline was inventing ungrounded policy details. Question 2 is the professional-transfer question: name a real feature in your work that needs a versioned registry and regression tests, and name who would own it. Question 3 forces a defensible position on eval-set size, reinforcing that 30 is a starting floor, not a production guarantee. Writing the answers surfaces remaining gaps before the lab embeds them in code. (2) EMPHASIZE: The reflection is not complete until written. Question 2's "who owns it" is the one most learners skip and the one that most reflects production maturity. (3) MISCONCEPTION: Treating reflection as optional throat-clearing. Catching a misconception in five minutes of reflection is far cheaper than catching it in two hours of debugging. (4) REFERENCES: - [Kolb 1984 — Experiential Learning, reflective observation](https://citt.it.ufl.edu/resources/course-development/the-learning-process/types-of-learners/kolbs-four-stages-of-learning/) - [Lesson M1.1 — reflection prompts](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output)

_notes: (1) KEY CONCEPTS: This five-criterion rubric lets learners self-score 0 to 3 before submitting, with 11 of 15 as the advance threshold to M1.2. The criteria map directly to the module's objectives: ladder application (mLO-1.1a), structured output as a reliability tool (mLO-1.1a), A/B eval discipline (mLO-1.1b/c), versioning and regression (mLO-1.1c), and reasoning quality (the through-line). Exemplary on each is defined behaviorally — e.g., not just running the eval but reporting a per-class confusion matrix, not just storing versions but wiring the regression gate into CI. The bridge to M1.2 is explicit: tool descriptions are prompts, so this exact discipline transfers to function calling. (2) EMPHASIZE: The advance threshold is 11/15 — proficiency on most criteria with at least some exemplary work. A learner stuck at 1s on eval discipline should redo the harness before advancing. (3) MISCONCEPTION: Self-scoring generously. The behavioral descriptors are deliberately concrete so "it felt better" cannot be scored as proficient reasoning. (4) REFERENCES: - [Lesson M1.1 — enabling objectives mLO-1.1a/b/c](https://career-forge.org/agentic-ai-mastery/module/m1-1-prompting-structured-output) - [Anderson & Krathwohl 2001 — Bloom's Revised Taxonomy](https://quincycollege.edu/wp-content/uploads/Anderson-and-Krathwohl_Revised-Blooms-Taxonomy.pdf)