Start here
CS5940 Model Map
One page per kind of agent, showing what problem it solves, which techniques live inside it, and how it answers the same design questions as the others. Start here when the lectures feel like a parade of acronyms.
The pattern behind the module
The CS5940 lectures teach architectures: ways of building the inside of an agent. Every one of them is an answer to the same few questions from week 1: what can it observe, what does it remember, what is its goal, how does it decide, what can it do?
- A deliberative agent keeps an explicit model of the world and reasons about it: prove what to do, commit to intentions, plan.
- A reactive agent skips the reasoning and maps what it senses straight to an action, by priority.
- A hybrid agent stacks both in layers.
- A learning agent (week 4) works out what to do from rewards, by planning over a known model or by trial and error.
- The practicals build an LLM-based agent from scratch, one layer of scaffold at a time.
And the same tension runs through all of them: react in time to a changing world, or pursue goals systematically? Each architecture strikes a different balance.
Pick a page
W1
Start here: what is an agent?
Definitions, agent vs workflow, the delegation problem, six design questions, five traditions.
W1Environment properties
Observable, deterministic, episodic, static, discrete, single-agent: and what each one demands.
W1The formal agent model
Runs, see / next / action, expected utility, achievement and maintenance tasks. The maths of week 1.
W1Practical 1: the chat loop
The model forgets everything: your messages list is the memory. Why tokens grow quadratically.
W2Deductive reasoning agents
Choose actions by proving them. The vacuum world, the closed world, calculative rationality.
W2Teleo-reactive programs
An ordered list of rules, re-checked every cycle, that pushes towards a goal.
W2Agent0 (agent-oriented programming)
Program agents in beliefs and commitments. Commitment rules, REQUEST and INFORM.
W2BDI and practical reasoning
Beliefs, desires, intentions. Deliberation decides what; means-ends reasoning decides how.
W2STRIPS planning
Pre, add and delete lists in the blocks world. A plan is a path through states.
W2Practical 2: the running coach
call_llm, token budgets, config files, a user profile, and a trimmed conversation window.
W3The BDI control loop
Cycles 1 to 4: options, filter, plan, and how long to stay committed to ends and means.
W3Intention reconsideration
Cycle 5, when reconsidering is worth it, and bold vs cautious agents (Kinny and Georgeff).
W3AgentSpeak(L)
BDI as a programming language: events, plans, the applicable unifier, intention stacks.
W3Subsumption (Brooks)
No symbols, no reasoning: layered behaviours, lowest wins. The Mars explorer and its crumbs.
W3Hybrid architectures
Reflexes and planning in layers. Horizontal vs vertical, InteRRaP, TouringMachines.
W4Markov decision processes
States, actions, noisy transitions, rewards, γ. The Markov property and the 4 × 3 gridworld.
W4Return, value and Bellman
Discounted return, V and Q, and "reward now + value of what comes next".
W4Value iteration
Plan with a known model: sweep the Bellman update until it settles, then read off the policy.
W4Q-learning
Learn action values from experience alone: TD error, learning rate, off-policy, bootstrapping.
W4Exploration vs exploitation
Why a greedy learner gets stuck. ε-greedy, decaying ε, optimism, Thompson sampling.
W4POMDPs and belief states
When the agent can’t see the state: a belief over states, updated by a Bayes filter.
W4When RL goes wrong
Reward hacking (the boat race), exploding tables, distribution shift.
Each architecture against the design questions
| Architecture | What it remembers | Where the goal lives | How it decides | Weak spot |
|---|---|---|---|---|
| Deductive agent | a database of logical facts \(\Delta\) | nowhere explicit (in the designer’s head) | prove \(Do(\alpha)\) from rules \(\rho\) | slow; calculative rationality |
| Teleo-reactive program | nothing beyond the current percept | implicit, the program is written for it | first true rule in an ordered list | can’t reason about its goal |
| Agent0 | beliefs, capabilities, commitments | implicit: honour commitments | commitment rules on incoming messages | no planning; can’t ask for help |
| BDI + STRIPS | beliefs, desires, intentions | explicit intentions | deliberate (what), then plan (how) | deliberation takes time |
| BDI control loop + reconsideration | \(B\), \(D\), \(I\), current plan | explicit intentions | options, filter, plan; replan if unsound; reconsider when cheap test says so | choosing how bold to be |
| AgentSpeak(L) | belief base, events, intention stacks | goals inside plans and events | pick an applicable plan from a library | only handles what someone wrote a plan for |
| Subsumption | nothing | implicit in the behaviours | the highest-priority behaviour that fires | no long-term view; hard to engineer |
| Hybrid (InteRRaP, TouringMachines) | a model per layer | upper layers | reflex first; escalate up layers when needed | complexity |
| Value iteration | a value for every state, \(V(s)\) | the reward function \(R\) | best action by one Bellman backup over the known model | needs \(P\) and \(R\); tables only |
| Q-learning | a table \(Q(s, a)\) learned from experience | the reward function \(R\) | \(\arg\max_a Q(s, a)\), exploring ε of the time | slow to learn; exploits a bad reward |
| LLM practicals | the messages list your code keeps |
the system prompt | model inference inside your loop | the model is stateless and can be confidently wrong |
Techniques that keep coming back
| Technique | What problem it solves | Appears in |
|---|---|---|
| state vs observation | two situations look the same, so the agent needs memory | formal model, BDI beliefs, the practicals’ history |
see / next / action |
the general shape of an agent with memory | formal model, deductive agents, subsumption |
| closed world + negation as failure | “not provable” means false | deductive agents, STRIPS, AgentSpeak contexts |
| first applicable rule wins | fast, predictable choice when several rules match | deductive agents (order of \(Ac\)), teleo-reactive, AgentSpeak’s \(S_O\), subsumption |
| commitment | stop re-deciding everything | Agent0 commitments, BDI intentions, commitment strategies |
| meta-level control | decide whether to think | reconsideration, hybrid layers |
| plan libraries vs planning from scratch | speed vs flexibility | STRIPS, AgentSpeak, LLM-generated plans |
| resending context | the LLM remembers nothing between calls | both practicals |
| Bellman backup | value now from the value of what comes next | value iteration, Q-learning, policy evaluation |
| belief state | can’t see the true state, so carry a summary of the history | POMDPs, BDI beliefs, the practicals’ messages list |
| counting things | how hard a design is | \(\lvert Ac\rvert^{\lvert E\rvert}\) agents, 22 blocks states, \(m^n\) vs \(m^2(n-1)\), tokens per turn, Q-table sizes \(2^n \lvert A\rvert\) |
How the weeks build on each other
- Week 1 sets the stage: what counts as an agent, why delegating to one is risky, how to describe its environment, and the maths of runs and utility. The practical shows that an LLM on its own remembers nothing.
- Week 2 builds the first real architectures, all symbolic: prove what to do (deductive), follow ordered rules towards a goal (teleo-reactive), program in beliefs and commitments (Agent0), and BDI with STRIPS planning. The practical adds budgets, settings and a trimmed memory.
- Week 3 asks what happens when the world changes while the agent is thinking. BDI answers with commitment and controlled reconsideration, and gets a language (AgentSpeak). Brooks answers by not thinking at all (subsumption). Hybrids do both, in layers.
- Week 4 makes “choosing well” mathematical. An MDP describes a world with random outcomes and rewards; the value of a state counts everything that might follow. With a known model the agent can plan (value iteration); without one it can learn (Q-learning), as long as it explores. Then the assumptions break: the agent can’t see the state (POMDPs and beliefs), or the reward isn’t what we meant (reward hacking).
How to use these pages
- Read the green “why this exists” box first on every page. That’s the context the lectures skip.
- Purple boxes are plain-English word lists. Yellow boxes are worked examples to copy out by hand. Red boxes flag slide typos and points the slides compress.
- Turn on Try-first (top bar) to hide every worked example until you’ve had a go on paper.
- Press Mark as understood on each technique. Your progress shows in the sidebar, on these tiles and on the module picker (saved in this browser only).
- Play with the interactive demos: step the vacuum and corridor robots, stack blocks with STRIPS, run an AgentSpeak trace, pit bold against cautious agents, run value iteration on the gridworld, train a Q-learner, and more.
- Each page ends with a practice prompt. Click “Copy prompt” and paste it into any AI chat to get questions one at a time, with model answers after you reply.
- Each page says which parts of Wooldridge are on the slides and which you can skip. In the scanned PDF, add 21 to the book’s page number.
Built from the week 1 to 4 slides and practical pages, and the CS5940 study guides. Wooldridge has no chapter list for this module, so the reading advice is matched against the slides. Pages for later weeks can be added in new w5, w6… folders.