Agentic AI Mastery Get the full course →

How LLMs Actually Behave

Tokens, sampling, context, and why output is probabilistic — the mental model everything else rests on.

📊 Open the slide deck →

M0.1 — How LLMs Actually Behave: The Mental Model

Module: M0.1 | Milestone: 0 (Foundations On-Ramp) Effort: 7h | Builds toward: SA-M0 (foundations gate)

📊 Slide deck available — Open the presenter-ready slides for this module (or download PDF / PPTX).

Enabling Learning Objectives

IDObjectiveBloom
mLO-0.1aExplain tokens, context windows, sampling (temperature/top-p), and why LLM output is probabilistic, not rule-based.Understand
mLO-0.1bDescribe what a model can/can’t know: training cutoff, no native tool access, hallucination as a failure mode.Understand
mLO-0.1cCall a chat model via API, control determinism, and read stop_reason/usage.Apply

Prerequisites

Overview

Every failure mode in an agent system — hallucination, unbounded loops, excessive cost, brittle outputs — traces back to a misunderstanding of how the underlying model actually works. Most developers carry a mental model borrowed from deterministic software: same input → same output, the system “knows” things, it has persistent state. None of those are true of an LLM.

This lesson installs the correct mental model before you write a single line of agent code. You will learn what a token is and why it matters for cost and context limits, why the same prompt produces different outputs at different temperatures, why the model has a knowledge cutoff and no native tools, and how hallucination is a structural property of next-token prediction rather than a bug to be patched. You will finish by making your first authenticated API call and reading the response metadata that every subsequent lesson depends on.

The COC reference system — the Customer Operations Copilot you will build across the course — handles consequential actions like issuing refunds. Understanding that its core component is a probabilistic text predictor is what motivates every guardrail, eval, and human-in-the-loop gate you will add later.

Segment 1 — Tokenization, Context Windows, and Cost

Concept

A language model does not see characters or words — it sees tokens, which are roughly 3–4 characters of English text (subword units determined by the model’s tokenizer). This has three immediate practical consequences:

  1. Cost is per-token, not per-character or per-request. A 1,000-word document is roughly 1,300 tokens. Multiply by your per-token rate and suddenly the economics of RAG, long system prompts, and multi-turn conversations become concrete.

  2. The context window is a fixed token budget — every token of input (system prompt + history + retrieved context + user message) plus every token of output must fit within it. Exceeding the window either truncates your input silently or raises an error, depending on the provider. Current frontier models offer windows on the order of 200K tokens (provider- and tier-dependent, with some long-context tiers going higher). Big windows feel unlimited until you’re running a multi-agent loop with multiple retrieved documents per turn.

  3. Token counting is a first-class engineering concern. You need to be able to count tokens before sending a request to stay within budget, estimate cost before deploying a feature, and implement context compaction when history grows (M2.2).

The tiktoken library handles OpenAI tokenizers; Anthropic exposes anthropic.count_tokens(). For provider-neutral code, litellm.token_counter() wraps both.

Lab 1 — Measure What You Send

Goal: Count tokens in a message before sending it, inspect usage in the response, and calculate the per-request cost. Files: lessons/m01/lab1_tokens.py

# lab1_tokens.py
# Install: uv pip install litellm python-dotenv
import os, json
from dotenv import load_dotenv
load_dotenv()

import litellm

MODEL = os.getenv("LLM_MODEL", "claude-sonnet-4-6")

def complete(messages, model=None, **kw):
    """llm.py seam — the only place the course calls a provider."""
    return litellm.completion(model=model or MODEL, messages=messages, **kw)

# --- 1. Count tokens BEFORE the call ---
messages = [
    {"role": "system", "content": "You are a helpful customer-support assistant."},
    {"role": "user",   "content": "What is your return policy for electronics?"},
]

input_tokens = litellm.token_counter(model=MODEL, messages=messages)
print(f"Input tokens (estimated): {input_tokens}")

# --- 2. Make the call ---
response = complete(messages, max_tokens=256)

# --- 3. Inspect usage ---
usage = response.usage
print(f"Input tokens  (actual):   {usage.prompt_tokens}")
print(f"Output tokens:            {usage.completion_tokens}")
print(f"Total tokens:             {usage.total_tokens}")
print(f"Stop reason:              {response.choices[0].finish_reason}")

# --- 4. Estimate cost (rough; real rates vary) ---
# Claude Sonnet input ~$3 / 1M tokens, output ~$15 / 1M tokens (illustrative)
INPUT_RATE  = 3.00 / 1_000_000
OUTPUT_RATE = 15.00 / 1_000_000
cost = usage.prompt_tokens * INPUT_RATE + usage.completion_tokens * OUTPUT_RATE
print(f"Estimated cost:           ${cost:.6f}")
print(f"\nModel reply:\n{response.choices[0].message.content}")

What to observe: The estimated pre-call count should be close (within a few tokens) to the actual prompt_tokens. Note that the model’s reply is short but output tokens cost ~5× more per token than input tokens — this ratio shapes every optimization in M3.7.

Variation: Replace the user message with a 500-word paste of product documentation and observe the token jump. Then try max_tokens=10 and see finish_reason become "length" instead of "end_turn" or "stop".

Segment 2 — Sampling, Temperature, and Probabilistic Output

Concept

When a model generates its next token, it produces a probability distribution over its entire vocabulary — tens of thousands of candidates. Temperature scales that distribution before sampling:

Top-p (nucleus sampling) is an alternative: instead of scaling all logits, truncate to the smallest set of tokens whose cumulative probability exceeds p, then sample from that set. top_p=0.9 means “only sample from the 90% of the probability mass.” In practice, most production systems use temperature OR top-p, not both.

The critical mental model upgrade: the model is not “looking up” an answer. It is predicting what token most plausibly follows your input, based on patterns seen during training. The same prompt at temperature=1 will produce different outputs on different calls — not because the model is broken, but because that IS the model. Reliability engineering for agents (M3.3) and structured output (M0.2, M1.1) are both responses to this fundamental property.

For COC, a refund-decision agent cannot run at high temperature — a different token choice could mean approving a refund that should be denied. Customer-facing drafts might legitimately vary. Temperature is a design decision, not a knob to ignore.

Lab 2 — Temperature Variance Experiment

Goal: Empirically observe how temperature affects output variance across 5 runs of the same prompt. Files: lessons/m01/lab2_temperature.py

# lab2_temperature.py
import os, collections
from dotenv import load_dotenv
load_dotenv()

import litellm

MODEL = os.getenv("LLM_MODEL", "claude-sonnet-4-6")

def complete(messages, temperature=0.0, max_tokens=64):
    return litellm.completion(
        model=MODEL,
        messages=messages,
        temperature=temperature,
        max_tokens=max_tokens,
    )

PROMPT = [
    {"role": "user", "content": (
        "A customer paid $49.99 for a blender on 2024-01-15. "
        "They want a refund. Our policy allows refunds within 30 days. "
        "Today is 2024-01-20. Should the refund be approved? "
        "Reply with exactly one word: YES or NO."
    )}
]

for temp in [0.0, 0.5, 1.0, 1.5]:
    outputs = []
    for _ in range(5):
        r = complete(PROMPT, temperature=temp)
        outputs.append(r.choices[0].message.content.strip())
    counts = collections.Counter(outputs)
    print(f"temp={temp:.1f}  responses={outputs}  unique={len(counts)}")

What to observe: At temperature=0, all 5 calls should return the same answer. As temperature rises, you will see increasing variance — eventually getting wrong answers (“MAYBE”, “Yes, but…”, etc.) even though the policy is unambiguous. This is why structured output + low temperature is mandatory for consequential decisions.

Variation: Change the prompt to “Write a one-sentence tagline for our blender” and repeat. Now high temperature is desirable.

Segment 3 — Knowledge Cutoff, No Native Tools, and Hallucination

Concept

The model’s weights were frozen at a training cutoff date — typically 6–18 months before the model was released. It has no access to events, prices, order statuses, or documents created after that date. More importantly, even within its training window, it does not “know” your company’s internal policies, your customer’s order history, or your current inventory — that information simply was not in its training data.

This has two design implications that motivate almost everything in Milestones 1 and 2:

  1. RAG (M1): ground the model’s outputs in retrieved, current, company-specific documents.
  2. Tools (M1.2): give the model callable functions to fetch live data (order status, weather, stock price).

Hallucination is not a bug — it is the expected behavior of a next-token predictor operating outside its training distribution. When asked a question it cannot answer from its weights, it generates the most plausible-sounding token sequence, which may be confident and wrong. Grounding (RAG + tools) and output verification (M1.4, M3.1) are the engineering responses. The model has no way to distinguish “I know this” from “I’m guessing this convincingly.”

There is also no persistent state between API calls. Each call is stateless — the model has no memory of previous conversations unless you explicitly include prior messages in the messages array. Memory management (M2.2) is the engineering layer that creates the illusion of continuity.

Lab 3 — Observing Hallucination and Statelessness

Goal: Deliberately trigger hallucination and demonstrate inter-call statelessness. Files: lessons/m01/lab3_hallucination.py

# lab3_hallucination.py
import os
from dotenv import load_dotenv
load_dotenv()

import litellm

MODEL = os.getenv("LLM_MODEL", "claude-sonnet-4-6")

def chat(messages, temperature=0.0, max_tokens=200):
    r = litellm.completion(model=MODEL, messages=messages,
                           temperature=temperature, max_tokens=max_tokens)
    return r.choices[0].message.content.strip()

# --- Demonstrate hallucination on an unknowable fact ---
print("=== Hallucination demo ===")
reply = chat([{"role": "user", "content":
    "What was the exact refund rate for Acme Corp's electronics category in Q3 2024?"}])
print(reply)
print()
# Expected: confident-sounding but fabricated numbers, OR a refusal.
# Either outcome confirms the point: the model either hallucinates or refuses.

# --- Demonstrate statelessness ---
print("=== Statelessness demo ===")
# Call 1
call1 = [{"role": "user", "content": "My name is Alex and I love hiking."}]
r1 = chat(call1)
print(f"Call 1 reply: {r1}")

# Call 2 — new call, NO history included
call2 = [{"role": "user", "content": "What is my name?"}]
r2 = chat(call2)
print(f"Call 2 reply (no history): {r2}")
# Model won't know the name — it was in a different call.

# Call 3 — same exchange, history included
call3 = [
    {"role": "user",      "content": "My name is Alex and I love hiking."},
    {"role": "assistant", "content": r1},
    {"role": "user",      "content": "What is my name?"},
]
r3 = chat(call3)
print(f"Call 3 reply (history included): {r3}")
# Now the model knows — because we included it in the messages array.

What to observe: Call 2 will produce something like “I don’t have access to your name” or make one up. Call 3 will correctly say “Alex”. The only difference is the messages array — this is the entire mechanism by which agents maintain apparent memory.

Segment 4 — Reading stop_reason and Usage for Loop Control

Concept

Every completion response includes metadata that agent loops depend on:

An agent loop that ignores finish_reason will silently truncate output and treat partial JSON as valid — a common source of hard-to-debug failures. In M1.2 you will build the tool-use loop; in M3.3 you will add the reliability layer. Both depend on reading these fields correctly.

Lab 4 — Loop-Aware Response Handling

Goal: Write a response-handling wrapper that raises on unexpected stop reasons. Files: lessons/m01/lab4_stop_reason.py

# lab4_stop_reason.py
import os
from dotenv import load_dotenv
load_dotenv()

import litellm

MODEL = os.getenv("LLM_MODEL", "claude-sonnet-4-6")

class UnexpectedStopReason(Exception):
    pass

def complete_safe(messages, max_tokens=512, model=None):
    """
    A stop-reason-aware wrapper around litellm.completion.
    Raises on truncation or content filter.
    Returns (content, usage) on success.
    """
    response = litellm.completion(
        model=model or MODEL,
        messages=messages,
        max_tokens=max_tokens,
    )
    choice = response.choices[0]
    reason = choice.finish_reason

    if reason == "length":
        raise UnexpectedStopReason(
            f"Output truncated at {max_tokens} tokens. "
            "Increase max_tokens or shorten your prompt."
        )
    if reason == "content_filter":
        raise UnexpectedStopReason("Response blocked by content filter.")
    # "stop" / "end_turn" = normal; "tool_calls"/"tool_use" handled by caller
    return choice.message.content, response.usage

# --- Test: normal completion ---
try:
    content, usage = complete_safe(
        [{"role": "user", "content": "Summarize the concept of RAG in two sentences."}],
        max_tokens=512,
    )
    print(f"OK — {usage.completion_tokens} output tokens")
    print(content)
except UnexpectedStopReason as e:
    print(f"Error: {e}")

# --- Test: force truncation ---
print("\n--- Forcing truncation ---")
try:
    content, usage = complete_safe(
        [{"role": "user", "content": "Write a 500-word essay on customer service."}],
        max_tokens=10,   # deliberately tiny
    )
    print(f"OK — {usage.completion_tokens} output tokens")
except UnexpectedStopReason as e:
    print(f"Caught expected error: {e}")

What to observe: The normal call succeeds and prints the summary. The truncated call raises UnexpectedStopReason with a helpful message. This pattern — fail loudly rather than silently — is a core engineering discipline for every agent loop you write.

Industry Example

IE-MENTALMODEL-Klarna — Klarna’s Customer-Service Assistant

Klarna deployed an LLM-based assistant that handled two-thirds of customer-service chats within its first month — the equivalent of 700 full-time agents. The system worked because it was grounded in live order data (tools), constrained to approved policy text (RAG), and equipped with an escalation path to human agents for cases outside the policy envelope.

What made it production-viable was precisely the engineering that wraps the probabilistic core: structured output so the model’s decisions are machine-readable, guardrails so it cannot invent policy, and human escalation so uncertain cases do not get confidently wrong answers.

Architectural takeaway: A bare LLM at the center of a customer-support workflow is dangerous. The probabilistic core becomes reliable only when it is bounded by grounding, structured output, and escalation — the three themes that run through all of Milestones 1–2.

Formative Check (FA-0.1)

FA-0.1 — Predict the behavior. Below are ten statements about how LLMs actually behave. For each one, decide True or False using the mental model from Segments 1–4 before running any code, then reveal the answer to self-check — predict first, then verify against the explanation.

Homework (HW-0.1)

HW-0.1 — Write three prompts at temp=0 and temp=1, run each 5×, tabulate variance; one-paragraph explanation.

Specification

Input: Three prompts of your choosing:

Output — two artifacts derived from one source of truth:

  1. hw01_results.csv — the raw log, one row per API call, with these exact columns:

    ColumnTypeMeaning
    typestringOne of factual, creative, ambiguous
    temperaturefloat (0.0 or 1.0)The sampling temperature for that call
    runint (1–5)Which of the 5 repetitions this row is
    responsestringThe model’s reply, verbatim (no paraphrasing)

    3 prompt types × 2 temperatures × 5 runs = 30 rows (plus a header).

  2. A derived markdown table (computed from the CSV), grouping by type + temperature and reporting the number of unique outputs:

    TypeTempRun 1Run 2Run 3Run 4Run 5Unique outputs
    factual0.0YESYESYESYESYES1
    factual1.0YESYESNOYESYES2
    …

Plus a paragraph explaining why variance differs across prompt types and temperatures.

Acceptance criteria:

  1. hw01_results.csv exists with exactly 30 data rows and the columns type, temperature, run, response.
  2. The factual prompt at temperature=0.0 yields a single unique response across all 5 runs.
  3. The derived markdown table is populated with actual response values (not paraphrases) and a correct Unique outputs count per type/temperature group.
  4. The paragraph correctly attributes variance differences to temperature AND prompt ambiguity.

Suggested test:

# tests/test_hw01.py
import csv, pathlib

CSV_PATH = pathlib.Path("hw01_results.csv")
EXPECTED_COLS = {"type", "temperature", "run", "response"}

def _load_rows():
    assert CSV_PATH.exists(), "hw01_results.csv not found"
    rows = list(csv.DictReader(CSV_PATH.open()))
    assert EXPECTED_COLS.issubset(rows[0].keys()), (
        f"CSV must have columns {EXPECTED_COLS}, got {set(rows[0].keys())}")
    return rows

def test_csv_has_30_rows():
    """3 prompts × 2 temps × 5 runs = 30 rows."""
    rows = _load_rows()
    assert len(rows) == 30, f"Expected 30 rows, got {len(rows)}"

def test_factual_temp0_is_consistent():
    """All 5 temp=0 runs on the factual prompt should return the same answer."""
    rows = _load_rows()
    factual_t0 = [r["response"] for r in rows
                  if r["type"] == "factual" and r["temperature"] == "0.0"]
    assert len(factual_t0) == 5, "Expected 5 factual/temp=0 runs"
    assert len(set(factual_t0)) == 1, "Factual prompt at temp=0 should be consistent"

Catch-up 🟡 — Start with lab2_temperature.py and extend it to loop over 3 prompts instead of 1, writing each call to hw01_results.csv with csv.DictWriter(fieldnames=["type", "temperature", "run", "response"]), then derive the markdown table from that CSV.

What You Built

Links & References

This is 1 of 18 modules.

Get retrieval, agentic orchestration, and production engineering — with graded projects and a capstone.

Get the full course →