RAG and Agents Practice Exercises

You can explain RAG. You can sketch an agent loop. But when a real question lands on your desk, do you know whether to retrieve, route, or just answer? These exercises close that gap between familiarity and judgment.

The drills below assume you have already built a basic RAG pipeline and a simple agent. If those builds are fresh in your memory, great. If they are fuzzy, skim your earlier work first. These exercises are not about writing more code from scratch. They are about making better decisions with the code you already have.

How to Use These Exercises

Each exercise follows the same deliberate-practice pattern: read the scenario, commit to an answer, then compare against the solution. Do not read the solution first. The learning happens when you commit to a wrong answer and discover why it is wrong.

You will need a lightweight setup: a notebook or script with a vector store and an agent framework you have used before. Nothing production-grade. The point is to run scenarios, inspect output, and treat mistakes as evidence about what your system is actually doing.

Some exercises ask you to classify or plan rather than run code. Do those on paper first. The judgment matters more than the execution.

Exercise 1: Choose the Right Architecture

Goal: Decide whether a plain LLM, a RAG pipeline, or an agent fits a given question.

Here are four realistic questions. For each one, choose your tool and write one sentence explaining why.

1. "What is the capital of France?"
2. "What was our company's revenue in the most recent quarter?"
3. "Compare our current leave policy with the one from last year and tell me what changed for new employees."
4. "Summarize the key arguments for and against remote work."

Think it through before reading the solution.

Solution:

Question 1 is a plain LLM question. The answer is stable, public, and almost certainly in the model's training data. Retrieval adds latency without adding accuracy.

Question 2 needs retrieval. The figure is private, recent, and stored somewhere the model has never seen. You must pull the current number from your own data. Whether that means RAG over documents or a structured database lookup depends on where the number lives—the point is that the model cannot answer from memory.

Question 3 needs an agent. You cannot answer it with one retrieval pass. You need to find the current policy, find the previous policy, compare them, and reason about what changed for a specific group. That is multi-step work with decisions about what to look up next.

Question 4 is trickier. A plain LLM can produce a reasonable summary from memory. But if you want the summary grounded in your collected sources, RAG is the better choice. The decision depends on whether you need the model's general knowledge or your specific documents.

The decision rule: Ask three questions before you pick a tool. Is the information private or recent enough that the model cannot know it? If yes, you need retrieval. Can the answer be found in one lookup, or does it require connecting pieces across sources? If the path is known in advance and fixed, a RAG pipeline may be enough. If the path depends on what you find—where you search next changes based on what the first search returns—you need an agent.

Common mistake: Reaching for RAG or agents when a plain prompt would do. Retrieval is context selection, not a magic upgrade. And an agent is not justified by having multiple retrieval steps alone. If you know the exact sequence in advance, a fixed workflow is simpler, cheaper, and easier to debug. An agent earns its complexity when tool selection or iteration must happen dynamically.

Note: My rule of thumb is simple. Static knowledge the model already has? Plain LLM. Information that exists somewhere in your data, with a known lookup path? RAG. A question that requires connecting information across sources or deciding what to look up next based on what you find? Agent.

Exercise 2: Trace a Broken Retrieval

Goal: Diagnose why a RAG pipeline returns poor answers.

Your pipeline retrieves chunks and generates an answer. The symptom: the answer looks generic, as if the model ignored the retrieved context entirely. When you inspect the retrieved chunks, they seem only loosely related to the question.

Your job is not to guess a culprit. Your job is to trace the pipeline from query to retrieved chunks to final prompt, and identify which check would reveal the failure.

Here is what you know:

• The chunks were created with a fixed chunk size of 500 characters, with no overlap.
• The index was built three months ago with embedding model A.
• Your query script currently loads embedding model B.
• Top-k is set to 3.
• The generation prompt says: "Answer the user's question using the context below."

Trace the failure. Where do you look first, and what would each check tell you?

Solution:

Start with the cheapest check: print the retrieved chunks and read them. Ask whether a human given those chunks could answer the question. If not, the problem is upstream of the model. The model can only work with the evidence it receives.

The most likely culprit is the embedding mismatch. You indexed with model A and you are querying with model B. Different embedding models map text into different vector spaces. The query vectors and the document vectors are not comparable, so your similarity search returns chunks that are only loosely related to the question. This is a severe failure mode, and it is invisible unless you check which embedding model each stage uses.

If the embeddings match, the next suspect is chunking. With 500-character chunks and no overlap, a relevant sentence can sit in a chunk dominated by unrelated content. The embedding for that chunk points somewhere in the middle, and the model cannot tell which part matters.

Top-k at 3 is low, but low top-k produces missing evidence, not loosely related evidence. If the retriever found the right neighborhood, you would see relevant chunks that simply are not enough. The symptom you described—chunks that are only vaguely on topic—points to the retriever never finding the right neighborhood at all.

The prompt is the last place to look. A weak prompt can make the model ignore good context, but it cannot make the retriever return bad chunks. Check the prompt only after you have confirmed the retrieved evidence is actually relevant.

Common mistake: Debugging the prompt when the retriever is feeding the model garbage. Think of retrieval as giving the model a limited desk. Every irrelevant chunk takes space away from the evidence it actually needs. Inspect the chunks before you blame the model.

Exercise 3: Route the Query

Goal: Decide how an agent should route different questions across tools.

You are building a customer support agent for a software company. It has four tools: vector search over product documentation, a structured lookup for account details, web search, and a direct answer capability.

Assign each query to the right tool and explain your reasoning:

1. "How do I reset my password?"
2. "What is my current plan and billing cycle?"
3. "Is there a known issue with the latest version of your API client?"
4. "What is the weather in Berlin?"

Solution:

Query 1 goes to vector search over documentation. The answer is a procedure described in your docs.

Query 2 goes to the structured lookup. Account details live in a database, not in documentation. Vector search would fail here because this is a precise record lookup, not a semantic match.

Query 3 is the interesting one. Start with documentation search for known, documented issues. If the docs do not cover it, escalate to web search—the issue may be recent, reported by other users, or fixed in a release note that has not reached your docs yet. The order matters: your documentation is authoritative for your product, while web search is discovery. Treat web results as leads to verify, not as automatic truth.

Query 4 should trigger clarification. You have no weather tool. A well-designed agent should say so rather than invent a forecast.

The source-authority rule: For each claim in an answer, identify which source is authoritative. Internal documentation owns your product's behavior. Structured data owns account records. Web search can point you to information, but it does not make that information true. When sources conflict, the agent should surface the conflict rather than quietly pick a winner.

Common mistake: Giving an agent too many tools and letting it misroute, or too few and forcing wrong answers. The agent's system prompt and tool descriptions shape routing decisions. If you describe a tool vaguely, the agent will use it vaguely. If you let web search answer questions your documentation should own, you will get plausible-sounding but wrong answers.

Exercise 4: Check the Grounding

Goal: Identify where an agent's answer drifts from retrieved evidence.

Here is a short agent transcript. The agent retrieved two passages from a company's refund policy and produced this answer.

Retrieved passage 1: "Customers may request a full refund within 30 days of the original purchase date. Refunds are processed within 5-7 business days after approval."

Retrieved passage 2: "Reseller purchases are handled under the partner program. Contact your reseller for return instructions."

Agent's answer: "Our refund policy allows full refunds within 30 days of purchase. Customers who purchased through a reseller may be eligible for extended coverage under the partner program. Refunds are processed within 5-7 business days after approval."

For each sentence in the answer, identify the exact passage that supports it—or mark it as unsupported.

Solution:

Sentence 1 is grounded. Passage 1 states the 30-day full refund window directly.

Sentence 2 is a hallucination risk. Passage 2 says reseller purchases are handled under the partner program and directs customers to contact their reseller. It says nothing about extended coverage. The agent inferred that "partner program" implies extended coverage, then stated that inference as policy. That is invented. The grounded version would say: "Reseller purchases are handled under the partner program—contact your reseller for return instructions."

Sentence 3 is grounded. Passage 1 states the 5-7 business day processing window.

The lesson: Retrieval does not guarantee grounding. The model can retrieve the right document and still connect the wrong dots. Inference is not evidence. When you check an agent's answer, you should be able to point to the exact passage that supports each claim. If you cannot, the claim is unsupported—regardless of how plausible it sounds.

Warning: Do not trust the final answer just because the pipeline retrieved something. Check that the answer actually uses the retrieved evidence. If the agent is unsure whether its claim is supported, it should re-search or admit the information is missing rather than answer from memory.

Exercise 5: Design the Agent Loop

Goal: Plan the steps an agent should take for a multi-hop question.

Here is the scenario: a user asks, "Does our parental leave policy cover employees in our new Berlin office?"

The agent has access to: the company policy document, an employee handbook, and a list of office locations with local employment regulations.

Write the sequence of actions the agent should take. What does it retrieve first? What does it check? When does it re-search? When does it stop?

Solution:

A naive single-retrieval pass would search for "parental leave Berlin" and hope the answer appears in one chunk. It probably will not, because the answer requires connecting two facts: what the policy says and whether it applies to Berlin.

A better agent loop looks like this:

1. Retrieve the parental leave policy section.
2. Check whether the policy mentions geographic scope or exclusions.
3. If the policy does not mention Berlin, retrieve the office locations document to confirm Berlin is a company office.
4. Retrieve local employment regulations for Berlin to check whether local law overrides the policy.
5. Synthesize an answer that connects all three sources.
6. Stop when the evidence is sufficient to answer the question.

The agent decomposes the question, retrieves what it needs for each part, and verifies that the pieces fit together.

The stopping rule: The loop ends when the agent has enough evidence to answer each part of the question, or when it has exhausted reasonable attempts. If the policy document does not mention Berlin and the office list does not include Berlin, the agent should stop and report what it found and what is missing—not speculate about coverage. If local regulations conflict with company policy, the agent should surface the conflict and flag the question for human review rather than pick a winner.

Common mistake: Building an unbounded loop that re-searches forever. Every iteration costs latency and tokens. Cap the loop. If the agent has not found sufficient evidence after a few attempts, it should stop and report what it found and what is missing.

What These Exercises Reveal

If you worked through all five exercises, you have practiced the skills that separate people who understand RAG and agents from people who can build them:

• Choosing the simplest workflow that has the evidence and control the task needs
• Tracing retrieval failures through observable pipeline evidence
• Routing queries to the right tools and knowing which source is authoritative
• Verifying that each claim in an answer is grounded in a specific passage
• Bounding agent loops so they stop when the evidence is sufficient

These exercises are not about plumbing. They are about judgment. The code is the easy part. Knowing when to retrieve, when to route, when to verify, and when to stop is the durable skill.

Here is your next experiment. Take one of your earlier builds and run it against a fixed set of five questions. For each run, record four things: the retrieved chunks, the tool the agent selected, the final answer, and whether each claim in that answer points to a retrieved passage. Then change exactly one variable—chunk size, top-k, a tool description, or the grounding instruction in your prompt. Run the same five questions again and compare the records.

Change one variable per run. If you change chunk size and prompt wording at the same time, you will not know which change caused the difference. The output is your evidence. Read it, adjust, and run again.

That practice loop—build, test, break, inspect, revise—is the skill that will carry you further than any single architecture.