Prompting & Structured Output for Reliability

Systematic prompting and a prompt-evaluation harness that scores variants on a labeled set.

📊 Open the slide deck →

M1.1 — Prompting & Structured Output for Reliability

Module: M1.1 | Milestone: 1 (Retrieval & Grounding) Effort: 7h | Builds toward: MP-1 (prompt-eval harness), PP-1

📊 Slide deck available — Open the presenter-ready slides for this module (or download PDF / PPTX).

Enabling Learning Objectives

Prerequisites

• Completed M0.2 — Pydantic structured output, the llm.py seam, .env setup
• litellm, pydantic, instructor installed
• Basic familiarity with json, csv/jsonlines in Python

Overview

Prompting is not a soft skill — it is an engineering discipline with measurable outcomes. “Write a better prompt” is not a strategy. “Measure output quality on a 30-example labeled set, identify failure categories, iterate, and report win-rates” is.

This lesson teaches Anthropic’s prompt-engineering ladder: seven techniques in order of impact, from the most fundamental (be clear and direct) to the most powerful (chain-of-thought). More importantly, it teaches systematic iteration: you will build a dataset, define a metric, run variants, and report results. This is the same discipline that scales to multi-agent evaluation in M3.1 — the prompt-eval harness you build here (MP-1) is the seed of your full evaluation stack.

For the COC reference system, this lesson addresses the core reliability challenge: a customer-facing assistant that inconsistently refuses legitimate refunds or invents policy details is a liability. Prompt engineering is the first defense; structured output is the second.

Segment 1 — Anthropic’s Prompt-Engineering Ladder

Concept

The ladder is ordered by impact-per-effort. Apply techniques from the bottom up — only add complexity when lower rungs fail:

1. Be clear and direct. State exactly what you want. “Classify this as REFUND, BILLING, or TECHNICAL” beats “help with this message.”
2. Assign a role. “You are a customer-support specialist for a home-goods retailer” narrows the model’s output distribution toward your domain.
3. Separate data from instructions. Use XML tags or delimiters to make the boundary unambiguous: <customer_message>...</customer_message>. This prevents instruction-injection from user content (a security property, not just style).
4. Give examples (few-shot). One well-chosen example often doubles reliability on hard cases. Three examples with diversity across edge cases is usually sufficient; beyond that, returns diminish.
5. Ask for step-by-step reasoning (chain-of-thought). “Think step by step before giving your final answer” improves accuracy on multi-step or ambiguous tasks. Critical for policy interpretation.
6. Prefill / speak for the model. Starting the assistant turn forces the model’s output into a specific format: {"eligible": locks in JSON output without a JSON mode.
7. Avoid hallucination explicitly. “If the information is not in the provided policy document, say ‘I don’t have enough information to answer this.’” Grounding constraints reduce hallucination — but only if you state them.

These are not tips — they are levers. Each one has a measurable effect on a labeled evaluation set.

Common Prompt Failure Modes

Before you can fix a prompt, you have to name what is wrong with it. These five failure modes account for the majority of unreliable prompts — and each maps directly to the ladder rung that fixes it. You will classify prompts against this taxonomy in the formative check (FA-1.1) and throughout the course.

Diagnosis comes first, then the fix: classify the failure, then apply the matching rung.

Lab 1 — Progressive Prompt Improvement

Goal: Apply the ladder incrementally and observe quality improvement on a fixed test case. Files: lessons/m11/lab1_ladder.py

# lab1_ladder.py
# Install: uv pip install litellm python-dotenv pydantic
import os, json
from dotenv import load_dotenv
load_dotenv()

import litellm

MODEL = os.getenv("LLM_MODEL", "claude-sonnet-4-6")

def complete(messages, system=None, temperature=0, max_tokens=512):
    msgs = []
    if system:
        msgs.append({"role": "system", "content": system})
    msgs.extend(messages)
    r = litellm.completion(model=MODEL, messages=msgs,
                           temperature=temperature, max_tokens=max_tokens)
    return r.choices[0].message.content.strip()

CUSTOMER_MSG = (
    "I ordered a blender (order #99988) on December 1st 2023 and it arrived broken. "
    "I want a replacement or a refund. My email is customer@email.com."
)

POLICY = """
RETURN POLICY:
- Physical damage on delivery: full refund within 60 days, no questions asked.
- Changed mind: refund within 30 days if item is unopened.
- Used items: no refund, exchange only.
"""

# --- Rung 1: bare prompt ---
print("=== Rung 1: Bare ===")
print(complete([{"role": "user", "content": CUSTOMER_MSG}]))

# --- Rung 2: role + clear instruction ---
print("\n=== Rung 2: Role + Instruction ===")
print(complete(
    [{"role": "user", "content": CUSTOMER_MSG}],
    system="You are a customer-support specialist for HomeGoods Co. "
           "Classify the customer's request and state whether it is eligible for a refund."
))

# --- Rung 3: delimited data, explicit output format ---
print("\n=== Rung 3: Delimiters + Output Format ===")
prompt3 = f"""
<policy>{POLICY}</policy>
<customer_message>{CUSTOMER_MSG}</customer_message>

Based only on the policy above, determine:
1. Category: REFUND_ELIGIBLE | EXCHANGE_ONLY | INSUFFICIENT_INFO
2. Reason: one sentence citing the policy clause
Reply as JSON: {{"category": "...", "reason": "..."}}
"""
print(complete([{"role": "user", "content": prompt3}]))

# --- Rung 4: chain-of-thought ---
print("\n=== Rung 4: Chain-of-Thought ===")
prompt4 = prompt3 + "\n\nThink step by step before giving your final JSON."
print(complete([{"role": "user", "content": prompt4}]))

# --- Rung 5: prefill (speaks for model, forces JSON start) ---
# Note: prefill is Anthropic-specific. With litellm, we use tool schema instead.
print("\n=== Rung 5: Tool Schema (enforced structured output) ===")
tool = {
    "type": "function",
    "function": {
        "name": "classify_request",
        "parameters": {
            "type": "object",
            "properties": {
                "category": {"type": "string",
                             "enum": ["REFUND_ELIGIBLE", "EXCHANGE_ONLY",
                                      "INSUFFICIENT_INFO"]},
                "reason": {"type": "string"},
                "confidence": {"type": "number", "minimum": 0, "maximum": 1},
            },
            "required": ["category", "reason", "confidence"],
        },
    },
}
r = litellm.completion(
    model=MODEL,
    messages=[{"role": "user", "content": prompt3}],
    tools=[tool],
    tool_choice={"type": "function", "function": {"name": "classify_request"}},
    temperature=0, max_tokens=512,
)
print(json.dumps(json.loads(
    r.choices[0].message.tool_calls[0].function.arguments), indent=2))

What to observe: Each rung produces more reliable, more structured output. Track: does Rung 1 even identify refund eligibility? Does Rung 3 cite the policy? Does Rung 5 force the right type on category?

Segment 2 — Structured Output as a Reliability Tool; Prefill

Concept

Structured output is not just about machine-parseability — it is a reliability technique. When you force the model to produce {"eligible": true, "reason": "..."} instead of prose, you eliminate entire failure modes: hedging (“well, it depends…”), irrelevant elaboration, format inconsistency, and the model “wriggling out” of a decision.

Prefill (Anthropic-specific) is the most direct version of this: you partially fill the assistant turn before generation, and the model completes it. {"eligible": forces the output to be valid JSON from the first token. This works because the model is completing a sequence — you are constraining the sequence’s beginning.

For cross-provider code (the course default), use tool schemas with tool_choice="required" instead: the model must call your tool, and the arguments must match your schema. This is equivalent to prefill in terms of reliability.

Speaking for the model at the prompt level — ending your instructions with “Your response:” or structuring the conversation so the model’s first token is naturally the start of your desired format — is the lower-tech version and works when tool schemas are not available.

The lesson is: every LLM call that feeds a downstream system should have a schema. No schema = no contract = no composability.

Lab 2 — Structured Output Reliability Benchmark

Goal: Measure how often four prompt variants produce parseable, correct-category output on 10 test cases. Files: lessons/m11/lab2_reliability.py

# lab2_reliability.py
import os, json
from dotenv import load_dotenv
load_dotenv()

import litellm
from pydantic import BaseModel
from typing import Literal

MODEL = os.getenv("LLM_MODEL", "claude-sonnet-4-6")

class Classification(BaseModel):
    category: Literal["REFUND_ELIGIBLE", "EXCHANGE_ONLY", "INSUFFICIENT_INFO"]
    reason: str

POLICY = """
RETURN POLICY:
- Physical damage on delivery: full refund within 60 days.
- Changed mind: refund within 30 days if item is unopened.
- Used items: no refund, exchange only.
- Missing receipt: insufficient info.
"""

TEST_CASES = [
    # (message, expected_category)
    ("Blender arrived smashed. Order was yesterday.", "REFUND_ELIGIBLE"),
    ("I changed my mind about the coffee maker. Still sealed.", "REFUND_ELIGIBLE"),
    ("I used the blender for 3 months and it broke.", "EXCHANGE_ONLY"),
    ("Not happy with purchase.", "INSUFFICIENT_INFO"),
    ("Item came with missing parts. Ordered 2 weeks ago.", "REFUND_ELIGIBLE"),
]

TOOL = {
    "type": "function",
    "function": {
        "name": "classify",
        "parameters": {
            "type": "object",
            "properties": {
                "category": {"type": "string",
                             "enum": ["REFUND_ELIGIBLE", "EXCHANGE_ONLY",
                                      "INSUFFICIENT_INFO"]},
                "reason": {"type": "string"},
            },
            "required": ["category", "reason"],
        },
    },
}

def classify(msg: str, use_policy: bool = True, use_tool: bool = True):
    content = (f"<policy>{POLICY}</policy>\n" if use_policy else "") + \
              f"<message>{msg}</message>\nClassify the return request."
    kw = {}
    if use_tool:
        kw["tools"] = [TOOL]
        kw["tool_choice"] = {"type": "function", "function": {"name": "classify"}}
    r = litellm.completion(
        model=MODEL,
        messages=[{"role": "user", "content": content}],
        temperature=0, max_tokens=256, **kw
    )
    if use_tool:
        raw = r.choices[0].message.tool_calls[0].function.arguments
        return Classification(**json.loads(raw))
    # free-form fallback
    return r.choices[0].message.content.strip()

for policy_flag, tool_flag, label in [
    (False, False, "No policy, no schema"),
    (True,  False, "Policy, no schema"),
    (True,  True,  "Policy + schema"),
]:
    correct = 0
    for msg, expected in TEST_CASES:
        try:
            result = classify(msg, use_policy=policy_flag, use_tool=tool_flag)
            if tool_flag:
                got = result.category
            else:
                got = result.upper()
            if expected in got:
                correct += 1
        except Exception:
            pass
    print(f"{label:30s}  accuracy={correct}/{len(TEST_CASES)}")

What to observe: The “No policy, no schema” variant will likely get several cases wrong. “Policy + schema” should be near 5/5. The delta between them is the empirical value of structured prompting.

Segment 3 — Systematic Prompt Iteration: Dataset, Metric, A/B

Concept

Intuition-driven prompting is a dead end. You iterate once, it seems better, you ship it — and then you discover the improvement was on the three cases you tested, while the edge cases you did not test regressed. The engineering solution is the same as software testing: a labeled dataset and a reproducible metric.

The workflow:

1. Build a dataset. Collect 20–50 examples with ground-truth labels. For COC, this means refund-eligibility cases with known correct decisions. For generation tasks, it means example inputs with human-rated reference outputs.
2. Define a metric. Exact-match for classification tasks (is the category correct?). BLEU/ROUGE for text generation (usually too coarse). LLM-as-judge with a rubric for open-ended outputs (you will build this in M3.1, but a simpler version suffices now).
3. Run variants. Execute both variants on the full dataset.
4. Report win-rates. “Variant B beats Variant A on 15/30 examples (50%), ties on 10, loses on 5.”

This “measure, don’t guess” discipline is what separates prompt engineering from prompt guessing. It also creates the regression dataset: the labeled set becomes a guard against future changes breaking existing behavior.

Prompt versioning is a natural extension: store each prompt variant with a version ID in a YAML or Python constant. When you change a prompt, bump the version and run the eval suite. If the new version regresses on previously-passing cases, you see it before it ships.

Lab 3 — A/B Prompt Evaluation Harness (Mini-Project MP-1 Seed)

Goal: Implement a harness that runs N prompt variants on a JSONL dataset and outputs a scored comparison table. Files: lessons/m11/lab3_eval_harness.py

# lab3_eval_harness.py — MP-1 seed
# Install: uv pip install litellm python-dotenv pydantic
import os, json, time
from pathlib import Path
from dotenv import load_dotenv
load_dotenv()

import litellm

MODEL = os.getenv("LLM_MODEL", "claude-sonnet-4-6")

# ---- Dataset ----------------------------------------------------------------
# Format: {"input": str, "expected": str, "id": str}
DATASET = [
    {"id": "c01", "input": "Blender arrived smashed. Order yesterday.",
     "expected": "REFUND_ELIGIBLE"},
    {"id": "c02", "input": "Changed mind, still sealed, ordered 2 weeks ago.",
     "expected": "REFUND_ELIGIBLE"},
    {"id": "c03", "input": "Used it for 3 months and the motor died.",
     "expected": "EXCHANGE_ONLY"},
    {"id": "c04", "input": "Not satisfied.",
     "expected": "INSUFFICIENT_INFO"},
    {"id": "c05", "input": "Package opened but item was defective out of box.",
     "expected": "REFUND_ELIGIBLE"},
    {"id": "c06", "input": "I've had it for 6 months.",
     "expected": "INSUFFICIENT_INFO"},
]

POLICY = """
RETURN POLICY:
- Physical damage on delivery: full refund within 60 days.
- Changed mind: refund within 30 days if item is unopened.
- Used items: no refund, exchange only.
- Missing receipt / insufficient info: ask for more information.
"""

# ---- Prompt variants --------------------------------------------------------
VARIANTS = {
    "v1_bare": lambda item: (
        f"What is the return category for: {item['input']}\n"
        "Answer with one of: REFUND_ELIGIBLE, EXCHANGE_ONLY, INSUFFICIENT_INFO"
    ),
    "v2_policy": lambda item: (
        f"<policy>{POLICY}</policy>\n"
        f"<request>{item['input']}</request>\n"
        "Reply with exactly one of: REFUND_ELIGIBLE, EXCHANGE_ONLY, INSUFFICIENT_INFO"
    ),
    "v3_cot_policy": lambda item: (
        f"<policy>{POLICY}</policy>\n"
        f"<request>{item['input']}</request>\n"
        "Think step by step, then reply with exactly one of: "
        "REFUND_ELIGIBLE, EXCHANGE_ONLY, INSUFFICIENT_INFO"
    ),
}

# ---- Evaluation -------------------------------------------------------------
VALID_CATS = {"REFUND_ELIGIBLE", "EXCHANGE_ONLY", "INSUFFICIENT_INFO"}

def extract_category(text: str) -> str:
    """Extract the classification from free-form model output."""
    for cat in VALID_CATS:
        if cat in text.upper():
            return cat
    return "UNPARSEABLE"

def run_variant(variant_fn, dataset, pause_s=0.5):
    results = []
    for item in dataset:
        prompt = variant_fn(item)
        try:
            r = litellm.completion(
                model=MODEL,
                messages=[{"role": "user", "content": prompt}],
                temperature=0, max_tokens=256,
            )
            raw = r.choices[0].message.content.strip()
            got = extract_category(raw)
        except Exception as e:
            got = f"ERROR:{e}"
        results.append({
            "id": item["id"],
            "expected": item["expected"],
            "got": got,
            "correct": got == item["expected"],
        })
        time.sleep(pause_s)  # simple rate-limit guard
    return results

# ---- Run & Report -----------------------------------------------------------
print(f"{'Variant':<20} {'Accuracy':>10} {'Failures'}")
print("-" * 55)
for variant_name, variant_fn in VARIANTS.items():
    results = run_variant(variant_fn, DATASET)
    correct  = sum(r["correct"] for r in results)
    failures = [r["id"] for r in results if not r["correct"]]
    print(f"{variant_name:<20} {correct}/{len(DATASET):>3}       {failures}")

print("\nMP-1 note: add --output flag and JSONL export to save results for regression.")

What to observe: v3_cot_policy should outperform v1_bare. Record which test case IDs fail per variant — those are your regression targets.

Segment 4 — Prompt Versioning and Regression

Concept

Prompts are code. They have versions, they have tests, and they can regress. A prompt change that fixes three cases while breaking one existing case is a regression — but you will only know if you are running against a fixed labeled set.

The minimal implementation:

• Store prompt templates as versioned Python constants or YAML files (not inline strings).
• The eval harness takes a version ID and a dataset path; outputs a scores file.
• Add the eval run to CI: pytest tests/test_prompts.py --live passes only if accuracy ≥ threshold.

This is exactly what GitHub Copilot and production LLM products do — Copilot’s team has a benchmark suite of completions that new prompt changes must not regress on. The harness you are building here (MP-1) is a simplified version of that discipline.

For the COC project, you will version the classification prompt, the response-drafting prompt, and the refund-eligibility prompt separately. Each gets its own labeled eval set. When you improve one, you run all three to check for crosstalk.

Lab 4 — Versioned Prompt Registry

Goal: Store two prompt versions in a registry; run the eval harness on both and compare. Files: lessons/m11/lab4_versioned.py

# lab4_versioned.py
import os
from dotenv import load_dotenv
load_dotenv()

import litellm

MODEL = os.getenv("LLM_MODEL", "claude-sonnet-4-6")

POLICY = """
RETURN POLICY: physical damage on delivery → full refund within 60 days.
Changed mind → refund within 30 days if unopened. Used items → exchange only.
"""

# ---- Prompt registry (in prod, store in YAML/DB) ----------------------------
PROMPT_REGISTRY = {
    "classify_v1": {
        "version": "1.0",
        "system": "You are a returns specialist.",
        "user_template": "Classify this request: {input}\n"
                         "Reply: REFUND_ELIGIBLE | EXCHANGE_ONLY | INSUFFICIENT_INFO",
    },
    "classify_v2": {
        "version": "2.0",
        "system": "You are a returns specialist. Apply policy exactly as written.",
        "user_template": (
            f"<policy>{POLICY}</policy>\n"
            "<request>{input}</request>\n"
            "Reply with exactly one token: REFUND_ELIGIBLE | EXCHANGE_ONLY | INSUFFICIENT_INFO"
        ),
    },
}

def run_prompt(prompt_id: str, input_text: str) -> str:
    spec = PROMPT_REGISTRY[prompt_id]
    user = spec["user_template"].format(input=input_text)
    r = litellm.completion(
        model=MODEL,
        messages=[
            {"role": "system", "content": spec["system"]},
            {"role": "user",   "content": user},
        ],
        temperature=0, max_tokens=64,
    )
    return r.choices[0].message.content.strip()

test_input = "I bought this toaster 2 months ago and used it every day. It broke."
for pid in PROMPT_REGISTRY:
    result = run_prompt(pid, test_input)
    version = PROMPT_REGISTRY[pid]["version"]
    print(f"[{pid} v{version}] → {result}")

What to observe: Both prompts return a classification. With the versioned registry, you can diff prompt text, track which version shipped when, and run regression tests against any version.

Mini-Project: MP-1 — Prompt-Eval Harness

Brief: Combine Labs 3 and 4 into a standalone CLI tool:

• Reads a JSONL dataset (one {id, input, expected} per line)
• Runs ≥3 prompt variants from a registry
• Outputs a scored comparison table (exact-match + optional LLM-judge score)
• Saves results to a JSONL file for regression tracking

You will invoke the harness from the command line like this:

python mp1_harness.py --dataset data/returns_eval.jsonl --output results/run_001.jsonl

This harness is reused and upgraded in M3.1 to become your full agent evaluation framework.

Industry Example

IE-PROMPT-GitHubCopilot — GitHub Copilot

GitHub Copilot’s system prompt includes repository context (file tree, recent edits, open tabs), language and framework detected from imports, and explicit instructions about completion style. Small changes to which context is included and how it is formatted produce large behavioral deltas in completion quality — the engineering team maintains a benchmark suite of thousands of completions to detect regressions when prompt changes ship.

The lesson: prompts are not static configuration. They are the primary lever for behavioral change in an LLM product, and they need the same rigor as code changes — versioning, testing, and regression monitoring. A 2% improvement on a benchmark that ships without a regression check on the other 98% is not progress.

Architectural takeaway: Every production LLM feature needs a labeled eval set that runs in CI. The prompt-eval harness you build in MP-1 is that infrastructure.

Formative Check (FA-1.1)

FA-1.1 — Fix these prompts. Below are five broken prompts. For each one, classify its failure mode using the taxonomy from Segment 1, then sketch a fix. Reveal the intended answer to self-check before moving to the next — diagnosis first, then the matching rung.

Homework (HW-1.1)

HW-1.1 — Take a provided 30-example task, write 3 prompt variants, and report win-rates with a confusion matrix per variant.

What you’ll build (deliverables)

Submit these four files:

Scaffold & input

The starter hw11_eval.py, requirements.txt, and the labeled dataset are provided with this module (download from the module resources):

• Input dataset: hw11_returns.jsonl — 30 customer messages, each with a ground-truth category (REFUND_ELIGIBLE | EXCHANGE_ONLY | INSUFFICIENT_INFO).

What is a confusion matrix?

A confusion matrix is a square table that shows where a classifier is right and wrong. Rows are the true category, columns are the predicted category; cell (row i, col j) counts the examples whose true label was i but the model predicted j. The diagonal holds the correct predictions; every off-diagonal cell is a specific kind of error — and which cell is large tells you the model’s actual confusion.

                      PREDICTED
                 REFUND  EXCHANGE  INSUFFICIENT
TRUE  REFUND        8        1            1
      EXCHANGE      2        7            1
      INSUFFICIENT  0        1            8

Read it like this: accuracy = diagonal ÷ total = (8 + 7 + 8) ÷ 30 = 77%. The largest off-diagonal cell (2 EXCHANGE messages predicted REFUND) is the model’s most-common error — exactly the “most-common error type” your write-up needs. Reference: scikit-learn confusion_matrix.

Output

• 3 meaningfully-different prompt variants in your registry
• A 3×3 confusion matrix per variant (REFUND_ELIGIBLE × EXCHANGE_ONLY × INSUFFICIENT_INFO)
• A markdown table: variant | accuracy | most-common error type

Acceptance criteria

1. All 3 variants run on the full 30-example set (not just 5).
2. Confusion matrices are numerically correct (each row sums to that class’s example count).
3. The best variant achieves ≥ 75% accuracy.
4. The write-up explains why the best variant outperforms — using the confusion matrix, not just the headline number.

Suggested test:

# tests/test_hw11.py
import json, pathlib

def test_results_file_has_90_rows():
    """3 variants × 30 examples = 90 result rows"""
    rows = [json.loads(l) for l in pathlib.Path("hw11_results.jsonl").open()]
    assert len(rows) == 90

def test_best_variant_accuracy():
    rows = [json.loads(l) for l in pathlib.Path("hw11_results.jsonl").open()]
    from collections import defaultdict
    by_variant = defaultdict(list)
    for r in rows:
        by_variant[r["variant"]].append(r["correct"])
    best = max(by_variant, key=lambda v: sum(by_variant[v]))
    best_acc = sum(by_variant[best]) / len(by_variant[best])
    assert best_acc >= 0.75, f"Best variant accuracy {best_acc:.2%} < 75%"

Stretch 🔵 — Add a simple LLM-judge grader that scores the model’s reason field (is it citing the correct policy clause?) separately from the category. Compare the two metrics.

What You Built

• lab1_ladder.py — progressive prompt improvement through 5 rungs of the Anthropic ladder (→ mLO-1.1a)
• lab2_reliability.py — structured output reliability benchmark: free-form vs policy vs schema (→ mLO-1.1a)
• lab3_eval_harness.py — A/B prompt evaluation harness on a labeled dataset (→ mLO-1.1b/mLO-1.1c)
• lab4_versioned.py — versioned prompt registry with regression-ready structure (→ mLO-1.1c)
• Completed HW-1.1 — 3-variant eval on 30 examples with confusion matrices (→ mLO-1.1b)
• MP-1 seed: eval harness pattern that scales to full agent evaluation in M3.1

Links & References

• Master course spec: §4 Module M1.1
• Anthropic Prompt Engineering Guide: https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering
• litellm docs: https://docs.litellm.ai
• Next lesson: M1.2 — Tool Use / Function Calling