// Project brief

AI Agentic — Build Aria

Python · FastAPI · OpenAI · LangChain · RAG · MCP · Agents

Join the team building Aria, an AI support agent. Ship real tickets that grow her from raw LLM calls into a full agentic system — LangChain, tools, memory, RAG, multi-agent hand-off, MCP, and evaluation — through PRs, CI, and code review, earning verified story points.

Work items

Story points

118

Est. effort

~59h

Top difficulty

7/10

● 14 storys● 5 tasks● 5 bugs

// What you'll learn, in order

Each layer builds on the one before it. You never meet a concept before the thing it depends on.

01Talking to a model4
02Frameworks and model choice4
03Making it an agent (tools)5
04Memory3
05Retrieval (RAG)4
06Multiple agents2
07External tools via MCP1
08Evaluation1

// How it works

01
Enroll
Get your own private repo, generated from the project blueprint.
02
Ship work items
Pick up bugs, tasks, and stories — one PR each, run through CI and AI code review.
03
Earn story points
Merge to bank verified points and unlock the next item on the backlog.

// The backlog

24work items, unlocked in order as you ship. Open any of them to see exactly what you'd be doing — nothing here is hidden until you sign up.

1 · Talking to a model

4 items · 11 SP

TASK-001
Wire Aria to the OpenAI chat API
Aria's backend and chat UI exist, but she can't actually talk yet — the model call is stubbed out. This is the very first thing that makes the product real.
How to do this ▾
S · 2SP
Why it matters
Aria's backend and chat UI exist, but she can't actually talk yet — the model call is stubbed out. This is the very first thing that makes the product real.
What you do
OpenAIChatModel.complete() raises NotImplementedError. Implement it: call the OpenAI Chat Completions API with self.model and the given messages, and return the assistant's reply text.

Learn: how a chat completion request is shaped (model + list of role/content messages) and where the reply lives in the response (choices[0].message.content).
Hints
Implement in app/adapters/openai_llm.py. Use self._get_client().chat.completions.create(...). The fake client in tests/conftest.py mimics the real response shape, so don't hard-code any provider details beyond the call.
Done when
complete(messages) returns the assistant's text

The configured self.model and the messages are passed to the API

The OpenAI client is only imported/constructed lazily (tests inject a fake)

Tests pass (tests/test_wi_001.py)
STORY-002
Assemble the chat prompt (system + history + user)
Aria needs a consistent personality and must remember the current conversation. Right now every turn is sent with no system prompt and no history.
How to do this ▾
S · 2SP
Why it matters
Aria needs a consistent personality and must remember the current conversation. Right now every turn is sent with no system prompt and no history.
What you do
Implement build_messages(history, user_message) in app/chat.py. It must return the message list sent to the model: the system prompt first, then prior history, then the new user message.

Learn: the role/content message format and why the system message shapes every reply.
Hints
SYSTEM_PROMPT is already defined in app/chat.py. History items are already {role, content} dicts.
Done when
Returns [system, ...history, user] in that order

The system message uses SYSTEM_PROMPT

Works with empty history

Tests pass (tests/test_wi_002.py)
BUG-003
Streaming parser returns raw SSE lines instead of text
We turned on streaming so replies appear token-by-token, but the UI is printing raw `data: {...}` gibberish. A customer sent a screenshot — embarrassing.
How to do this ▾
S · 2SP
Why it matters
We turned on streaming so replies appear token-by-token, but the UI is printing raw data: {...} gibberish. A customer sent a screenshot — embarrassing.
What you do
parse_stream_chunk(line) in app/chat.py returns the whole line. It should parse one Server-Sent-Events line and return just the incremental text (choices[0].delta.content), or None for the [DONE] sentinel, blank lines, and comment (:) lines.

Learn: how streaming responses arrive as SSE and how to extract deltas safely.
Hints
Lines look like data: {"choices":[{"delta":{"content":"Hi"}}]}. Parse JSON after the data: prefix; guard with try/except.
Done when
Extracts delta.content from a data: {...} line

Returns None for [DONE], blank, and :-comment lines

Never raises on malformed lines

Tests pass (tests/test_wi_003.py)
TASK-004
Add retries, timeout, and token-usage logging
The OpenAI API occasionally rate-limits or times out, and finance wants to know what we're spending. One flaky call shouldn't break a customer chat.
How to do this ▾
M · 5SP
Why it matters
The OpenAI API occasionally rate-limits or times out, and finance wants to know what we're spending. One flaky call shouldn't break a customer chat.
What you do
Wrap the model call so it (1) times out after a configurable number of seconds, (2) retries transient failures with exponential backoff (max 3 attempts), and (3) records total_tokens from each response via an injected usage_sink.

Learn: production concerns around LLM calls — reliability and cost observability.
Hints
Keep retry/backoff logic testable: inject a sleep function and a usage_sink callback so tests assert calls without real waiting. See app/adapters/openai_llm.py and the Retry seam in app/interfaces.py.
Done when
Transient errors retry up to 3 times with backoff, then re-raise

A non-transient error is not retried

Each successful call reports total_tokens to the usage sink

Timeout is passed to the client

Tests pass (tests/test_wi_004.py)

2 · Frameworks and model choice

4 items · 15 SP

STORY-005
Introduce a LangChain chat-model wrapper
We're standardizing on LangChain so we can swap models, add tools, and chain steps without rewriting Aria each time. First step: run a chat turn through LangChain.
How to do this ▾
M · 5SP
Why it matters
We're standardizing on LangChain so we can swap models, add tools, and chain steps without rewriting Aria each time. First step: run a chat turn through LangChain.
What you do
Add a LangChainChatModel adapter that satisfies the same ChatModel interface but drives a LangChain chat model under the hood. complete(messages) must convert our {role, content} dicts into LangChain messages, invoke the model, and return the text.

Learn: LangChain's message types (System/Human/AI) and the .invoke() call.
Hints
Create app/adapters/langchain_model.py. Map system|user|assistant to SystemMessage|HumanMessage|AIMessage. The fake in conftest returns a message with a .content attribute.
Done when
LangChainChatModel implements the ChatModel interface

role dicts map correctly to System/Human/AI messages

Returns the reply text

LangChain is imported lazily; tests inject a fake chat model

Tests pass (tests/test_wi_005.py)
TASK-006
Model registry — choose the model by name
Different jobs want different models: a cheap fast one for chit-chat, a stronger one for hard tickets. Ops wants to switch via config, not code.
How to do this ▾
S · 3SP
Why it matters
Different jobs want different models: a cheap fast one for chit-chat, a stronger one for hard tickets. Ops wants to switch via config, not code.
What you do
Implement get_model(name, **overrides) that returns a configured ChatModel for a known key (e.g. "fast", "smart"), applying temperature/model-id defaults from a registry and letting callers override them. Unknown names raise a clear error.

Learn: decoupling model choice from model use.
Hints
Put the registry in app/models_registry.py. Return the interface type, not a concrete client, so the agent code stays provider-agnostic.
Done when
Known keys return a model configured with the right model-id and defaults

Overrides (e.g. temperature) win over defaults

Unknown key raises ValueError naming the key

Tests pass (tests/test_wi_006.py)
STORY-007
Prompt templates via LangChain
Support answers should follow a consistent structure (greeting, answer, next step). Hard-coded f-strings are getting messy and hard to reuse.
How to do this ▾
M · 5SP
Why it matters
Support answers should follow a consistent structure (greeting, answer, next step). Hard-coded f-strings are getting messy and hard to reuse.
What you do
Introduce a reusable prompt template for Aria's answers with named variables (e.g. customer_name, question, tone). Provide render_prompt(**vars) that fills the template and returns the message list ready for the model.

Learn: ChatPromptTemplate, variable substitution, and why templates beat string concatenation.
Hints
Create app/prompts.py. You can use LangChain's ChatPromptTemplate (lazy import) or a thin wrapper — tests check behavior, not the library.
Done when
Template exposes named variables and fills them correctly

Missing a required variable raises a clear error

render_prompt returns messages usable by ChatModel.complete

Tests pass (tests/test_wi_007.py)
BUG-008
Temperature/config isn't reaching the model
QA set temperature to 0 for deterministic tests, but answers still vary wildly. The config is being dropped somewhere between the registry and the API call.
How to do this ▾
S · 2SP
Why it matters
QA set temperature to 0 for deterministic tests, but answers still vary wildly. The config is being dropped somewhere between the registry and the API call.
What you do
Trace why temperature (and other generation config) set on the model isn't applied to the actual API call. Fix it so configured parameters are forwarded on every complete().

Learn: how config flows from construction to the request, and the danger of silently dropping kwargs.
Hints
Look at how the adapter stores config vs. what it passes to .create()/.invoke(). The bug is intentionally planted in the adapter wiring.
Done when
Configured temperature (and other params) reach the underlying call

A per-call override still works

Tests pass (tests/test_wi_008.py)

3 · Making it an agent (tools)

5 items · 24 SP

STORY-009
Build the agent with create_agent
Aria should do more than answer — she should decide when to look things up or take an action. This is the leap from chatbot to agent.
How to do this ▾
L · 8SP
Why it matters
Aria should do more than answer — she should decide when to look things up or take an action. This is the leap from chatbot to agent.
What you do
Assemble Aria as an agent using create_agent: give it the chat model, the system prompt, and an (initially empty) tool list, and expose run(user_message) that returns the agent's final answer.

Learn: the agent loop — the model reasons, optionally calls tools, then answers.
Hints
Create app/agent/core.py. Use LangChain/LangGraph create_agent. Keep the model and tools injectable so the fake model can drive a deterministic loop in tests.
Done when
build_agent(model, tools) returns an agent exposing run()

With no tools, it answers directly

The system prompt is applied

create_agent / LangGraph imported lazily; tests inject a fake model

Tests pass (tests/test_wi_009.py)
TASK-010
Add a calculator tool
Customers ask things like 'I bought 3 seats at $12/mo for 8 months — what's my total?'. LLMs are unreliable at arithmetic; give Aria a real calculator.
How to do this ▾
S · 3SP
Why it matters
Customers ask things like 'I bought 3 seats at $12/mo for 8 months — what's my total?'. LLMs are unreliable at arithmetic; give Aria a real calculator.
What you do
Implement a calculator tool (safe arithmetic evaluation) that satisfies the Tool interface and register it with the agent. Reject anything that isn't a math expression.

Learn: what a tool is — name, description the model reads, and a run(input) function — and why tools beat asking the LLM to compute.
Hints
Create app/agent/tools/calculator.py. Do NOT use bare eval; parse with ast or a whitelist. Implement the Tool protocol from app/interfaces.py.
Done when
calculator.run("3*12*8") returns 288

Non-math / unsafe input raises or returns a clear error, never executes arbitrary code

Tool has a name and a model-facing description

Tests pass (tests/test_wi_010.py)
STORY-011
Add a knowledge-base lookup tool
Aria keeps guessing at plan limits and refund policy. Give her a tool that looks up authoritative answers from our internal KB.
How to do this ▾
M · 5SP
Why it matters
Aria keeps guessing at plan limits and refund policy. Give her a tool that looks up authoritative answers from our internal KB.
What you do
Implement a kb_lookup tool that takes a query and returns matching help-center entries from an injected knowledge source. Register it with the agent so the model can call it when a customer asks about policy.

Learn: tools that fetch data, and dependency-injecting the data source so it's testable.
Hints
Create app/agent/tools/kb_lookup.py. Depend on a KnowledgeSource interface; the real one can be backed by the RAG store built in Epic 5.
Done when
kb_lookup.run(query) returns relevant entries from the injected source

Returns a clear 'no match' result when nothing is found

Source is injected (fake KB in tests)

Tests pass (tests/test_wi_011.py)
BUG-012
Agent loops forever when a tool errors
A customer chat hung and ran up a huge token bill: a failing tool made Aria retry the same call over and over until the request timed out.
How to do this ▾
M · 5SP
Why it matters
A customer chat hung and ran up a huge token bill: a failing tool made Aria retry the same call over and over until the request timed out.
What you do
Reproduce: when a tool raises, the agent re-invokes it endlessly instead of recovering. Add a max-iteration / step guard and make tool errors surface to the model as a handled result so it can apologize or try another path.

Learn: agent safety — bounding the loop and handling tool failures gracefully.
Hints
In app/agent/core.py, set/enforce a recursion or step limit and wrap tool execution. The planted bug is an unbounded loop with swallowed exceptions.
Done when
A persistently failing tool no longer loops indefinitely

The agent stops after a bounded number of steps

A tool error is passed back as an observation, not a crash

Tests pass (tests/test_wi_012.py)
TASK-013
Validate tool inputs
The model sometimes calls tools with the wrong argument shape (a string where a number is expected). We want clean, predictable failures, not stack traces in the chat.
How to do this ▾
S · 3SP
Why it matters
The model sometimes calls tools with the wrong argument shape (a string where a number is expected). We want clean, predictable failures, not stack traces in the chat.
What you do
Add input schemas/validation to the tools so bad arguments produce a structured error the agent can read, instead of raising raw exceptions.

Learn: typed tool inputs and why validation makes agents robust.
Hints
Use pydantic models or explicit checks in each tool's run. Keep the error message model-readable.
Done when
Each tool validates its input and returns a structured error on bad input

Valid input behaves as before

Tests pass (tests/test_wi_013.py)

4 · Memory

3 items · 18 SP

STORY-014
Short-term conversation memory (sliding window)
Aria forgets what the customer said two messages ago. She needs to hold the current conversation — but not the entire history forever.
How to do this ▾
M · 5SP
Why it matters
Aria forgets what the customer said two messages ago. She needs to hold the current conversation — but not the entire history forever.
What you do
Implement a windowed conversation memory: append user/assistant turns and, when building the prompt, include only the last N turns.

Learn: short-term memory as bounded context, and why we can't just send everything.
Hints
Create app/memory/short_term.py implementing a Memory interface (add, context). Window size is configurable.
Done when
Turns are stored and returned in order

Only the last N turns are included in the prompt

Adding beyond N drops the oldest

Tests pass (tests/test_wi_014.py)
STORY-015
Long-term memory across sessions
Returning customers hate re-explaining themselves. Aria should recall durable facts ('prefers email', 'on the Pro plan') even in a brand-new session.
How to do this ▾
L · 8SP
Why it matters
Returning customers hate re-explaining themselves. Aria should recall durable facts ('prefers email', 'on the Pro plan') even in a brand-new session.
What you do
Add a long-term memory store that persists facts keyed by user, and can be queried to enrich the prompt for a new session. Support writing a fact, reading a user's facts, and retrieving the most relevant ones for a query.

Learn: the difference between short-term (this chat) and long-term (across chats) memory, and how retrieval selects what's relevant.
Hints
Create app/memory/long_term.py behind a LongTermStore interface. The real backend can reuse the Epic 5 vector store; tests use an in-memory fake.
Done when
Facts persist per user via an injected store

Reading returns a user's facts; unknown user returns empty

Relevance query returns the most relevant facts first

Store is injected (fake in tests)

Tests pass (tests/test_wi_015.py)
BUG-016
Memory grows unbounded and blows the context limit
Long chats started failing with a context-length error, and costs spiked — we were shoving the entire transcript into every request.
How to do this ▾
M · 5SP
Why it matters
Long chats started failing with a context-length error, and costs spiked — we were shoving the entire transcript into every request.
What you do
The prompt builder concatenates all memory with no bound, eventually exceeding the model's context window. Fix it so the assembled context stays within a token budget (trim/summarize oldest content), while keeping the system prompt and latest user message.

Learn: token budgeting and graceful context trimming.
Hints
Inject a token-counter function so tests are deterministic. The planted bug is an unbounded join in the prompt assembly.
Done when
Assembled context never exceeds the configured token budget

System prompt and newest user message are always kept

Oldest turns are trimmed (or summarized) first

Tests pass (tests/test_wi_016.py)

5 · Retrieval (RAG)

4 items · 21 SP

STORY-017
Ingest and chunk docs into embeddings
Our help center has hundreds of articles. To answer from them, Aria first needs them chunked and embedded into a vector store.
How to do this ▾
L · 8SP
Why it matters
Our help center has hundreds of articles. To answer from them, Aria first needs them chunked and embedded into a vector store.
What you do
Implement ingestion: split documents into overlapping chunks, embed each chunk via an injected Embedder, and upsert them into an injected vector store with metadata (source, title).

Learn: the RAG pipeline's first half — chunking, embeddings, and vector storage.
Hints
Create app/rag/ingest.py. Use the Embedder interface and a VectorStore interface (real impl: Chroma, lazy import). Chunk on characters/tokens with overlap.
Done when
Documents are split into chunks with configurable size/overlap

Each chunk is embedded and stored with metadata

Re-ingesting the same doc doesn't duplicate (idempotent by id)

Embedder and store are injected (fakes in tests)

Tests pass (tests/test_wi_017.py)
STORY-018
Retrieve the top-k relevant chunks
Ingestion is done; now Aria needs to actually find the right passages for a question before answering.
How to do this ▾
M · 5SP
Why it matters
Ingestion is done; now Aria needs to actually find the right passages for a question before answering.
What you do
Implement retrieve(query, k): embed the query and return the k most similar chunks from the vector store, each with its similarity score and metadata.

Learn: semantic search — embedding the query and ranking by similarity.
Hints
Create app/rag/retrieve.py. Reuse the Embedder and VectorStore interfaces from ingestion.
Done when
Returns k results ranked by similarity (most similar first)

Each result carries its text, score, and metadata

k is respected; fewer available returns all

Tests pass (tests/test_wi_018.py)
STORY-019
Answer with retrieved context and citations
Support answers must be grounded and auditable — customers (and auditors) want to see which article an answer came from.
How to do this ▾
M · 5SP
Why it matters
Support answers must be grounded and auditable — customers (and auditors) want to see which article an answer came from.
What you do
Combine retrieval with generation: build a prompt that includes the retrieved chunks as context, ask the model to answer using only that context, and return the answer plus the list of cited sources. If nothing relevant is found, say so instead of hallucinating.

Learn: the RAG payoff — grounding answers and citing sources.
Hints
Create app/rag/answer.py. Compose retrieve + ChatModel. Instruct the model to answer only from context; return {answer, sources}.
Done when
Answer is generated from retrieved context

Cited sources (from chunk metadata) are returned

Empty/irrelevant retrieval yields a safe 'I don't know' response, not a guess

Tests pass (tests/test_wi_019.py)
BUG-020
Retrieval returns duplicates and poor ranking
Answers cite the same article three times and miss the obviously-relevant one. Retrieval quality is hurting trust.
How to do this ▾
S · 3SP
Why it matters
Answers cite the same article three times and miss the obviously-relevant one. Retrieval quality is hurting trust.
What you do
Fix two planted issues: near-duplicate chunks aren't deduped, and results aren't actually sorted by score before truncation to k. Ensure results are ranked correctly and de-duplicated by source/content.

Learn: post-retrieval hygiene — sorting and dedup materially change answer quality.
Hints
The bug is in app/rag/retrieve.py: truncation happens before sorting, and there's no dedup pass.
Done when
Results are sorted by score before taking top-k

Duplicate chunks (same content/source) are collapsed

Correct k distinct results returned

Tests pass (tests/test_wi_020.py)

6 · Multiple agents

2 items · 13 SP

STORY-021
Supervisor routes to specialist agents
One do-everything prompt is getting unwieldy. We want a supervisor that reads the request and routes it to the right specialist (billing vs. technical).
How to do this ▾
L · 8SP
Why it matters
One do-everything prompt is getting unwieldy. We want a supervisor that reads the request and routes it to the right specialist (billing vs. technical).
What you do
Build a supervisor agent that classifies an incoming message and delegates to one of several sub-agents, then returns the specialist's answer. The routing decision must be inspectable.

Learn: the multi-agent / supervisor pattern and why decomposition beats one giant agent.
Hints
Create app/agent/supervisor.py. Model sub-agents as objects with a run(); the supervisor picks by classification (LLM or rules) — keep the classifier injectable.
Done when
Supervisor selects the correct specialist for representative inputs

The chosen specialist's answer is returned

Routing decision is exposed (e.g. in the result)

Sub-agents are injected (fakes in tests)

Tests pass (tests/test_wi_021.py)
STORY-022
Specialist hand-off with shared context
A billing chat sometimes turns technical mid-way. The customer shouldn't have to repeat themselves when Aria hands off to another specialist.
How to do this ▾
M · 5SP
Why it matters
A billing chat sometimes turns technical mid-way. The customer shouldn't have to repeat themselves when Aria hands off to another specialist.
What you do
Support hand-off: a specialist can pass control (with the conversation context and a reason) to another specialist, whose answer is returned to the user. Prevent infinite hand-off ping-pong.

Learn: stateful hand-off between agents and guarding against delegation loops.
Hints
Extend app/agent/supervisor.py. Track a hand-off count and cap it; carry the memory/context object across the hand-off.
Done when
A specialist can hand off to another with context preserved

The final specialist's answer is returned

Hand-off is bounded (no infinite loops)

Tests pass (tests/test_wi_022.py)

7 · External tools via MCP

1 item · 8 SP

STORY-023
Expose an MCP server's tools to the agent
Other teams publish capabilities (order lookup, ticket creation) as MCP servers. Aria should use them as tools without us re-implementing each integration.
How to do this ▾
L · 8SP
Why it matters
Other teams publish capabilities (order lookup, ticket creation) as MCP servers. Aria should use them as tools without us re-implementing each integration.
What you do
Connect to an MCP server, list its tools, and adapt them into our Tool interface so the agent can call them like any other tool. Handle connection and tool-call errors cleanly.

Learn: what MCP is (a standard protocol for exposing tools/resources) and how to bridge external tools into an agent.
Hints
Create app/agent/mcp_tools.py. Use langchain-mcp-adapters / the MCP client (lazy import). Wrap each discovered tool so run() calls the MCP session; tests inject a fake session listing 1–2 tools.
Done when
MCP tools are discovered and wrapped as Tools

The agent can invoke a wrapped MCP tool and get its result

Connection/tool errors are handled, not crashed

MCP client is injected (fake session in tests)

Tests pass (tests/test_wi_023.py)

8 · Evaluation

1 item · 8 SP

STORY-024
Evaluation harness with prompt versioning
Before we change Aria's prompt, we need proof it's actually better. We want to version prompts and score them against a fixed set of cases so we ship improvements, not regressions.
How to do this ▾
L · 8SP
Why it matters
Before we change Aria's prompt, we need proof it's actually better. We want to version prompts and score them against a fixed set of cases so we ship improvements, not regressions.
What you do
Build a small eval harness: run a set of test cases (input + expected/graded criteria) against a named prompt version, score each with graders (exact/keyword/LLM-as-judge behind an interface), and produce an aggregate score per prompt version so two versions can be compared.

Learn: offline evaluation, prompt versioning, and data-driven prompt iteration — the loop that keeps an AI product improving.
Hints
Create app/eval/harness.py and store prompt versions in app/prompts.py (or app/eval/prompts/). Keep graders behind an interface so an LLM-judge can be faked. Emit a small report object (per-case + aggregate).
Done when
Prompts are addressable by version; a run records which version it used

Each case is graded and an aggregate score is produced

Two prompt versions can be scored and compared on the same cases

Grader/model are injected (deterministic fakes in tests)

Tests pass (tests/test_wi_024.py)