Python-for-AI, SDKs & Structured Output

Reproducible environments, the provider-neutral seam, and schema-valid structured output.

📊 Open the slide deck →

M0.2 — Python-for-AI, SDKs & Structured Output

Module: M0.2 | Milestone: 0 (Foundations On-Ramp) Effort: 8h | Builds toward: SA-M0 (foundations gate), HW-0.2

📊 Slide deck available — Open the presenter-ready slides for this module (or download PDF / PPTX).

Enabling Learning Objectives

Prerequisites

• Completed M0.1 — understand LLM probabilistic output, finish_reason, and basic API calls
• Python 3.11+; uv installed (pip install uv or brew install uv)
• At least one provider API key

Overview

The number-one reason learners stall in agent courses is environment friction: broken dependencies, hardcoded keys, “it works on my machine” problems. This lesson solves that once, correctly, so you never debug pip again while trying to learn architecture.

Beyond setup, this lesson teaches the most important primitive for building reliable agents: structured output. An LLM call that returns free-form prose is hard to compose. An LLM call that returns a Pydantic-validated object with typed fields — and raises loudly when it does not — is a function. Functions compose. Agents are composed of functions.

You will implement structured output three ways across two labs — prompt-based JSON and a typed tool schema in Lab 2, then Pydantic-with-instructor in Lab 3 — compare their reliability, and add a validation-and-retry loop. By the end, you will also understand the key distinction that frames the rest of the course: a client SDK gives you raw API access and you own the control flow, while an agent SDK manages the loop for you. Both have their place; knowing the boundary is what lets you choose.

Segment 1 — Reproducible Environment and Secret Hygiene

Concept

Professional Python-for-AI projects require four things to be reproducible and safe:

1. Isolated dependency environment. Use uv (fastest) or venv. Never install AI dependencies globally — version conflicts between litellm, langchain, and their transitive dependencies are a constant source of pain.

2. Secrets in .env, never in code. API keys committed to git are rotated within minutes by automated scanners, and your billing will reflect it. The pattern: .env (gitignored, real keys), .env.example (committed, placeholder values).

3. requirements.txt or pyproject.toml. Pin your dependencies. litellm==1.30.2 today may break on 1.31.0 tomorrow. Lock it.

4. A Makefile or justfile with setup, test, run targets. Reduces every project entry point to a single, documented command. This matters more when you hand the project to a hiring manager or co-worker.

The course scaffold enforces these non-negotiables in every project. The llm.py seam (single file, all model calls go through it) is the sixth: it means you can swap providers by changing one env var, not hunting through twenty files.

Lab 1 — Project Scaffold Setup

Goal: Stand up a minimal, correctly configured project that makes an authenticated API call. Files: lessons/m02/setup.sh, lessons/m02/.env.example, lessons/m02/lab1_env.py

# lab1_env.py
# Install: uv pip install litellm python-dotenv
# Run:     python lab1_env.py
import os
from dotenv import load_dotenv

load_dotenv()  # reads .env in the current directory

# Fail loudly if the key isn't set — better than a cryptic auth error
api_key = os.getenv("ANTHROPIC_API_KEY") or os.getenv("OPENAI_API_KEY")
if not api_key:
    raise EnvironmentError(
        "No API key found. Copy .env.example to .env and add your key."
    )

model = os.getenv("LLM_MODEL", "claude-sonnet-4-6")
print(f"Using model: {model}")

import litellm

response = litellm.completion(
    model=model,
    messages=[{"role": "user", "content": "Reply with exactly: READY"}],
    max_tokens=10,
    temperature=0,
)
print(f"API responded: {response.choices[0].message.content.strip()}")
print(f"Tokens used: {response.usage.total_tokens}")

The matching .env.example:

# .env.example — copy to .env and fill in real values. NEVER commit .env.
ANTHROPIC_API_KEY=sk-ant-REPLACE_ME
# OR:
OPENAI_API_KEY=sk-REPLACE_ME
LLM_MODEL=claude-sonnet-4-6
LLM_MODEL_SMALL=claude-haiku-4-5

What to observe: The script prints the model name and “READY”. If the key is missing, it raises a clear error rather than a cryptic HTTP 401 buried in a stack trace.

Variation: Deliberately remove the .env file and observe the error message. Then try an invalid key and observe the AuthenticationError from litellm.

Segment 2 — Structured Output: JSON Mode and Tool Schemas

Concept

Free-form text output from an LLM is not composable. If you ask “What is the refund eligibility?” and get back a paragraph, your downstream code cannot reliably extract the boolean decision, the reason, and the confidence score. Structured output solves this by constraining the model to emit machine-parseable data.

Three approaches, in order of reliability:

1. Prompt-based JSON: ask the model to respond in JSON. Works, but the model may wrap the JSON in prose, add markdown fences, or produce invalid JSON. Requires manual parsing and error handling.

2. JSON mode / response format: providers offer a native mode (response_format={"type": "json_object"} on OpenAI; Anthropic achieves it via prefill or tool schemas) that guarantees valid JSON syntax — but not your specific schema.

3. Tool/function schema: define a function with a typed schema; force the model to “call” it. The response is structured to the schema and you get the arguments as a dict. This is the most reliable approach for extracting specific fields.

The trade-off: stricter structured output = fewer creative failures but occasionally the model refuses or produces shallow answers when its “natural” output would be better. For extraction tasks (contact info, refund eligibility, intent classification) — use strict schemas. For generation tasks (draft a response) — use loose or no structure.

Lab 2 — Two Structured-Output Approaches Side-by-Side

Goal: Extract the same information two ways — prompt-based JSON and a typed tool schema — and compare their reliability and error surface. (The third rung, Pydantic with instructor, is built in Lab 3.) Files: lessons/m02/lab2_structured.py

# lab2_structured.py
# Install: uv pip install litellm python-dotenv pydantic
import os, json
from dotenv import load_dotenv
load_dotenv()

import litellm

MODEL = os.getenv("LLM_MODEL", "claude-sonnet-4-6")

def complete(messages, model=None, **kw):
    return litellm.completion(model=model or MODEL, messages=messages, **kw)

CUSTOMER_TEXT = """
Hi, I'm Sarah Johnson. I bought a coffee maker (order #12345) on January 5th 2024.
It stopped working after 2 weeks. I want a full refund. My email is sarah@example.com.
"""

# --- Approach 1: Prompt-based JSON (fragile) ---
print("=== Approach 1: Prompt JSON ===")
r1 = complete([
    {"role": "user", "content": (
        f"Extract: name, email, order_id, issue from this message. "
        f"Reply with ONLY a JSON object.\n\n{CUSTOMER_TEXT}"
    )}
], temperature=0, max_tokens=256)
raw = r1.choices[0].message.content.strip()
# Strip markdown fences if present
if raw.startswith("```"):
    raw = raw.split("```")[1]
    if raw.startswith("json"):
        raw = raw[4:]
try:
    data1 = json.loads(raw)
    print(json.dumps(data1, indent=2))
except json.JSONDecodeError as e:
    print(f"PARSE ERROR: {e}\nRaw: {raw}")

# --- Approach 2: Tool schema (reliable) ---
print("\n=== Approach 2: Tool Schema ===")
EXTRACT_TOOL = {
    "type": "function",
    "function": {
        "name": "extract_customer_request",
        "description": "Extract structured fields from a customer message.",
        "parameters": {
            "type": "object",
            "properties": {
                "customer_name":  {"type": "string"},
                "customer_email": {"type": "string"},
                "order_id":       {"type": "string"},
                "issue_summary":  {"type": "string"},
                "refund_requested": {"type": "boolean"},
            },
            "required": ["customer_name", "customer_email", "order_id",
                         "issue_summary", "refund_requested"],
        },
    }
}
r2 = complete(
    [{"role": "user", "content": f"Extract from this message:\n\n{CUSTOMER_TEXT}"}],
    tools=[EXTRACT_TOOL],
    tool_choice={"type": "function", "function": {"name": "extract_customer_request"}},
    temperature=0,
    max_tokens=512,
)
tool_call = r2.choices[0].message.tool_calls[0]
data2 = json.loads(tool_call.function.arguments)
print(json.dumps(data2, indent=2))

What to observe: Approach 1 may occasionally produce output wrapped in markdown; Approach 2 reliably returns a dict matching the schema. Both should extract the same values — but Approach 2 raises a KeyError if the schema is violated rather than silently passing bad data downstream.

Segment 3 — Pydantic Validation and the Retry-on-Invalid Loop

Concept

Tool schemas guarantee valid JSON syntax and field presence, but not semantic validity. A model might return customer_email: "not provided" for a string field — syntactically valid, semantically wrong. Pydantic adds the semantic layer: typed fields, validators, and clear error messages.

The pattern for production-grade structured extraction is:

1. Ask the model to fill a tool schema.
2. Parse the JSON arguments into a Pydantic model.
3. If Pydantic raises ValidationError, feed the error back to the model and ask it to fix the output.
4. Retry up to N times. If still invalid, raise or return a default.

This retry loop is a micro-instance of the evaluator-optimizer pattern (M2.3). The same principle — ask the LLM, check the result, repair if needed — scales up to entire agent outputs.

The instructor library automates this pattern: it wraps litellm/OpenAI/Anthropic clients and returns validated Pydantic objects directly, handling the retry loop internally. Using it in M0.2 means you can focus on schema design rather than boilerplate.

Lab 3 — Pydantic Extraction with Validation + Retry

Goal: Extract structured data into a Pydantic model; demonstrate the repair loop on invalid output. Files: lessons/m02/lab3_pydantic.py

# lab3_pydantic.py
# Install: uv pip install litellm python-dotenv pydantic instructor
import os
from dotenv import load_dotenv
load_dotenv()

from pydantic import BaseModel, field_validator
from typing import Optional
import instructor
import litellm

MODEL = os.getenv("LLM_MODEL", "claude-sonnet-4-6")

# --- Pydantic schema ---
class CustomerRequest(BaseModel):
    customer_name: str
    customer_email: str
    order_id: str
    issue_summary: str
    refund_requested: bool

    @field_validator("customer_email")
    @classmethod
    def email_looks_valid(cls, v: str) -> str:
        if "@" not in v:
            raise ValueError(f"'{v}' does not look like an email address")
        return v.lower().strip()

    @field_validator("order_id")
    @classmethod
    def order_id_not_empty(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("order_id cannot be empty")
        return v.strip()

# --- Use instructor for auto-retry on validation failure ---
# instructor patches the litellm client to return Pydantic objects
client = instructor.from_litellm(litellm.completion)

CUSTOMER_TEXT = """
Hi, I'm Sarah Johnson. I bought a coffee maker (order #12345) on January 5th 2024.
It stopped working after 2 weeks. I want a full refund. My email is sarah@example.com.
"""

try:
    result: CustomerRequest = client.chat.completions.create(
        model=MODEL,
        response_model=CustomerRequest,
        messages=[{
            "role": "user",
            "content": f"Extract all fields from this customer message:\n\n{CUSTOMER_TEXT}"
        }],
        max_retries=3,   # instructor will retry on ValidationError
        temperature=0,
        max_tokens=512,
    )
    print("Extracted successfully:")
    print(result.model_dump_json(indent=2))
except Exception as e:
    print(f"Extraction failed after retries: {e}")

# --- Manual repair loop (without instructor, for understanding) ---
print("\n=== Manual repair loop ===")
import json

def extract_with_retry(text: str, max_retries: int = 3) -> CustomerRequest:
    import litellm as ll
    TOOL = {
        "type": "function",
        "function": {
            "name": "extract_customer_request",
            "parameters": CustomerRequest.model_json_schema(),
        }
    }
    messages = [{"role": "user",
                 "content": f"Extract from this message:\n\n{text}"}]
    for attempt in range(max_retries):
        r = ll.completion(
            model=MODEL,
            messages=messages,
            tools=[TOOL],
            tool_choice={"type": "function",
                         "function": {"name": "extract_customer_request"}},
            temperature=0,
            max_tokens=512,
        )
        raw = r.choices[0].message.tool_calls[0].function.arguments
        try:
            return CustomerRequest(**json.loads(raw))
        except Exception as e:
            messages.append({"role": "assistant", "content": None,
                             "tool_calls": r.choices[0].message.tool_calls})
            messages.append({"role": "tool",
                             "tool_call_id": r.choices[0].message.tool_calls[0].id,
                             "content": f"Validation error: {e}. Please fix and call again."})
    raise RuntimeError(f"Could not extract valid data after {max_retries} retries")

result2 = extract_with_retry(CUSTOMER_TEXT)
print(result2.model_dump_json(indent=2))

What to observe: The instructor-based version is concise and handles retries automatically. The manual loop shows what happens under the hood — understanding both is important when instructor does not support your provider/format.

Segment 4 — Client SDK vs Agent SDK: The Fundamental Distinction

Concept

Before you pick a framework, understand what you are choosing between:

Client SDK (e.g., anthropic, openai, litellm):

• You call the API and get a response.
• You write the while loop, the if finish_reason == "tool_use", the message accumulation, the tool dispatch.
• Full control; maximum transparency; more code.
• Good for: simple single-turn calls, custom loops, learning what is actually happening.

Agent SDK / framework (e.g., LangGraph, smolagents, Anthropic’s Agent SDK, AutoGen):

• The framework manages the loop — the perceive→reason→act→observe cycle is baked in.
• You define tools, agents, and transitions; the framework calls the model and routes the results.
• Less code for common patterns; harder to debug when behavior is unexpected.
• Good for: multi-step agent loops, multi-agent orchestration, stateful workflows.

The course teaches both (MLO-12: comparative judgment). You will build the raw loop in M2.1, then re-implement it in LangGraph, then write five sentences on what the framework hid. That exercise is the one that builds real architectural judgment.

For now: every lab in Milestones 0 and 1 uses the client SDK (litellm) directly. You will graduate to frameworks only after you understand what they abstract away.

Lab 4 — Side-by-Side: Client SDK vs a Minimal Agent SDK

Goal: Implement the same “answer a question, retry once if empty” task using raw litellm and using a minimal agent abstraction. Files: lessons/m02/lab4_sdk_comparison.py

# lab4_sdk_comparison.py
# Install: uv pip install litellm python-dotenv smolagents
import os
from dotenv import load_dotenv
load_dotenv()

import litellm

MODEL = os.getenv("LLM_MODEL", "claude-sonnet-4-6")
TASK  = "What are the top 3 benefits of offering free returns in e-commerce? Reply in bullet points."

# ============================================================
# Version A: Raw client SDK — you own every line of the loop
# ============================================================
print("=== Version A: Client SDK ===")

def answer_with_retry_client(task: str, max_retries: int = 2) -> str:
    messages = [{"role": "user", "content": task}]
    for attempt in range(max_retries):
        r = litellm.completion(
            model=MODEL,
            messages=messages,
            temperature=0,
            max_tokens=512,
        )
        content = r.choices[0].message.content.strip()
        if content:
            return content
        # Empty response — add observation and retry
        messages.append({"role": "assistant", "content": content})
        messages.append({"role": "user",
                         "content": "Your response was empty. Please answer the question."})
    return "No answer after retries."

print(answer_with_retry_client(TASK))

# ============================================================
# Version B: smolagents CodeAgent — the loop is managed for you
# ============================================================
# NOTE: smolagents is optional; skip if not installed.
try:
    from smolagents import LiteLLMModel, CodeAgent

    print("\n=== Version B: smolagents CodeAgent ===")
    llm = LiteLLMModel(model_id=MODEL)
    agent = CodeAgent(tools=[], model=llm)
    result = agent.run(TASK)
    print(result)
    print("\n--- What smolagents hid ---")
    print("• The while-loop that checks stop reason")
    print("• Message accumulation across turns")
    print("• Tool dispatch (none here, but configured via tools=[])")
    print("• Step logging and execution state")
except ImportError:
    print("\nsmolagents not installed — run: uv pip install smolagents")
    print("Key point: a framework wraps the same litellm call in a managed loop.")

What to observe: Both produce the same output. Version A is ~15 lines of explicit control flow. Version B is 3 lines — but those 3 lines hide the loop, the message accumulation, and the retry logic. When something goes wrong in Version B, you need to understand Version A to debug it.

Industry Example

IE-STRUCTURED-Stripe — Invoice Line-Item Extraction

Extracting structured data from invoice images and PDFs is a canonical use case for LLM-as-a-typed-function: given an unstructured invoice document, produce a typed Invoice object with line items, totals, vendor name, and payment terms — validated against a schema before any downstream processing.

The engineering insight is that the LLM call becomes a transformation function: unstructured text → validated typed object. Every downstream system (billing logic, accounting, reconciliation) can depend on the typed contract rather than parsing prose. Validation and retry are not optional extras — they are what makes the function trustworthy enough to run in a payment context.

Architectural takeaway: Treat every LLM call that feeds downstream code as a typed function with a schema, a validator, and a failure mode. Free-form output is only acceptable at the user-facing surface.

Formative Check (FA-0.2)

FA-0.2 — Pick the right structured-output technique. The lab is to build extract_contact(text) -> Contact that returns a Pydantic-validated object or fails loudly. Below, each requirement of that lab is broken out: for each one, classify which technique from this lesson’s reliability ladder (Segments 2–3) is the right tool, then reveal the intended answer to self-check before moving on — design decision first, then the matching rung.

Homework (HW-0.2)

HW-0.2 — Build an extract_contact(text) -> Contact function with a Pydantic schema + validation + one retry on invalid JSON.

Specification

Input: Free-form text strings containing contact information (some well-formed, some missing fields).

Output: A module extract_contact.py containing:

• Contact(BaseModel) with at minimum: name: str, email: str, phone: Optional[str], company: Optional[str]
• extract_contact(text: str) -> Contact — raises ExtractionError after 1 retry if a valid object cannot be produced

Acceptance criteria:

1. Returns a valid Contact for the well-formed inputs in the test suite.
2. Raises ExtractionError (not a generic exception) when the model cannot find required fields after retry.
3. email is lowercase-stripped before returning.
4. The function works without modifying any environment variables (reads from .env).

Suggested test:

# tests/test_hw02.py
import pytest
from extract_contact import extract_contact, ExtractionError

GOOD  = "Hi, I'm Maria Santos. Email me at maria@example.com. Phone: 555-1234."
NO_EMAIL = "This is Bob. He works at Acme. No contact info provided."

def test_good_extraction():
    c = extract_contact(GOOD)
    assert c.name.lower() in {"maria santos", "maria"}
    assert "maria" in c.email
    assert c.email == c.email.lower()

def test_missing_email_raises():
    with pytest.raises(ExtractionError):
        extract_contact(NO_EMAIL)

Stretch 🔵 — Add a confidence: float field (0–1) to Contact and have the model self-report confidence; log cases where confidence < 0.7 to a separate file.

Catch-up 🟡 — Use the instructor library directly (Lab 3 style) and add the ExtractionError wrapper around its ValidationError.

What You Built

• lab1_env.py — reproducible project scaffold with .env secret loading and authenticated first API call (→ mLO-0.2a)
• lab2_structured.py — two structured-output approaches (prompt JSON, tool schema) side-by-side (→ mLO-0.2b)
• lab3_pydantic.py — Pydantic extraction with validation and the repair loop, using both instructor and manual retry (→ mLO-0.2b)
• lab4_sdk_comparison.py — same task implemented with raw litellm vs smolagents; explicit comparison of what the framework hides (→ mLO-0.2c)
• Completed HW-0.2 — production-grade extract_contact function with schema, validation, and retry (→ mLO-0.2a/mLO-0.2b)

Links & References

• Master course spec: §3 Module M0.2
• litellm docs: https://docs.litellm.ai
• Pydantic docs: https://docs.pydantic.dev
• instructor library: https://python.useinstructor.com
• smolagents: https://huggingface.co/docs/smolagents
• Next lesson: M1.1 — Prompting & Structured Output for Reliability