Practical 2: the running coach

Technique 1: one function for every model call

Problem it solves: change the model, count tokens or enforce a budget once, not everywhere.

MODEL = "openai/qwen3-14b"

def call_llm(messages: list[dict], temperature: float = 0.2) -> tuple[str, Any]:
    """Send messages to the model and return (reply_text, usage)."""
    response = completion(model=MODEL, api_base=os.environ["OPENAI_BASE_URL"],
                          api_key=os.environ["OPENAI_API_KEY"],
                          messages=messages, temperature=temperature)
    text = response.choices[0].message.content or ""
    return text, response.usage

The or "" guards against content being None. The coach uses temperature 0.2 for consistent advice.

Technique 2: thinking, and keeping it out of the history

Problem it solves: use extra reasoning only when it’s worth the cost.

Qwen3 is hybrid: thinking is off by default on the course server and switched on per request:

extra_body={"chat_template_kwargs": {"enable_thinking": thinking}}
reasoning = getattr(message, "reasoning_content", None)   # None if there isn't any

For “Hello!” thinking is wasted; for “Is 20 km a week enough for a 10 km race?” it helps.

Why not store the reasoning in the history? It’s often far longer than the answer, and the whole history is resent every turn, so it would multiply the cost of every later call. call_llm returns only message.content, and .strip() removes the blank lines left behind.

Technique 3: the token budget

Problem it solves: stops a bug from spending the course allowance (700,000 tokens per hour, 2,000,000 per day).

TOKENS_USED = 0
TOKEN_BUDGET = 100_000

class TokenBudgetExceeded(Exception): ...

def call_llm(...):
    global TOKENS_USED
    if TOKENS_USED >= TOKEN_BUDGET:
        raise TokenBudgetExceeded(f"This run has used its budget of {TOKEN_BUDGET} tokens.")
    response = completion(...)
    TOKENS_USED += response.usage.total_tokens

The overshoot. The check happens before each call, using the total so far. If every call uses \(c\) tokens and the budget is \(B\):

\[\text{calls made} = \left\lceil \frac{B}{c} \right\rceil, \qquad \text{tokens used} = c\left\lceil \frac{B}{c} \right\rceil < B + c.\]

A run can overshoot by less than one call’s worth.

Technique 4: settings in config.toml

Problem it solves: change settings without editing code, and record which settings gave a result.

# config.toml. Secrets (API key, server URL) stay in environment variables.
[model]
name = "openai/qwen3-14b"
temperature = 0.2
thinking = false
token_budget = 100_000   # per script run

CONFIG_FILE = Path(__file__).with_name("config.toml")   # next to config.py, wherever you run from
CONFIG = tomllib.loads(CONFIG_FILE.read_text(encoding="utf-8"))

What goes where: values you change between runs (model, temperature, thinking, budget) go in config.toml; secrets never do (environment variables); logic stays in code. The file is read once, at import, so restart Python after editing it.

Technique 5: the user profile

Problem it solves: makes advice specific to the runner, by putting facts in the system prompt.

def build_system_prompt(profile: dict) -> str:
    return f"""You are a supportive running coach.
Give short, practical advice (at most 5 sentences).

Athlete profile:
{json.dumps(profile, indent=2)}
"""

Who decides what the model sees? The application. Changing current_weekly_km from 25 to 5 should change the advice. Changing race_date tells you little, because the model doesn’t know today’s date unless you put it in the prompt too.

In lecture terms the profile is explicit, inspectable state, a bit like a deductive agent’s \(\Delta\). But the LLM doesn’t reason soundly over it: it can still contradict the profile, so inspect the output.

Technique 6: the trimmed conversation window

Problem it solves: keeps each request’s cost bounded, at the price of forgetting.

class Conversation:
    def __init__(self, system_prompt: str, max_messages: int = 10) -> None:
        self.system_prompt = system_prompt
        self.max_messages = max_messages
        self.history: list[dict] = []

    def get_messages(self) -> list[dict]:
        recent = self.history[-self.max_messages:]
        return [{"role": "system", "content": self.system_prompt}] + recent

Why is the system prompt outside the window? It holds the role, the rules and the profile. If it were trimmed the coach would turn into a generic assistant after a few turns.

The maths. At turn \(t\) (after add_user) the history holds \(2t - 1\) messages. The turn-1 message is still sent iff

\[2t - 1 \le k \iff t \le \frac{k + 1}{2}.\]

Without trimming, the prompt grows every turn (quadratic total, as in week 1). With a window it grows until the window is full, then stays roughly flat at about \(s + k\bar m\) tokens.

Save and resume: load session.json at start-up if it exists, and conv.save(SESSION_FILE) before break. The system prompt isn’t saved, because it’s rebuilt from the current profile each run.

Connecting it to the lecture

Practice prompt

I'm studying CS5940 Intelligent Agents. The week 2 practical builds a running coach in Python with LiteLLM: a call_llm(messages, temperature) function, an optional Qwen3 thinking mode, a TOKEN_BUDGET checked before each call, settings in config.toml, a JSON user profile in the system prompt, and a Conversation class that sends only the last max_messages history messages plus the system prompt. I want practice on:
- why every call goes through one function, and what "or ''" and type hints are for
- when to switch thinking on, and why reasoning text is kept out of the history
- token budget arithmetic: number of calls ceil(B/c), tokens used, which call raises, and the overshoot < one call
- what belongs in config.toml vs environment variables vs code
- why the profile goes in the system prompt, and why changing race_date tells you little
- the trimmed window: the turn-1 message is sent iff 2t - 1 <= k; how many messages get_messages() returns; why the system prompt is never trimmed
- save/resume with session.json
- linking the coach to the lecture (explicit state, no commitments, no planning)

Please:
1. Use plain, everyday language and explain any term in one short sentence.
2. Give me ONE question at a time, easy first, then harder; include code-reading and arithmetic questions.
3. Wait for my answer, then mark it and show a model answer with working.
4. If I get it wrong, explain the mistake and give me a similar question.
5. Every 5 questions, summarise my strengths and what to practise.