Answers From YOUR Documents — With Citations | Agentic AI 101 #14, by The Hot Path on YouTube. 11 min 14 s.
- Video length
- 11 min 14 s
- Time to do it
- 2 hours
- Difficulty
- Medium
- Steps
- 8
What you need
Steps
Start from the fact that the model knows nothing about your documents. RAG fetches the right small piece at question time and hands it over, instead of trying to teach the model your files.
Split each document into chunks of a few hundred tokens, with a small overlap so sentences are not cut in half. Chunk size changes answer quality more than most settings.
Run every chunk through an embedding model and store the resulting vectors in a vector store. Text with similar meaning lands close together even when the words differ.
Add keyword or BM25 search next to the vector search. Meaning-based search alone misses exact things like error codes and version numbers, so combine both result lists.
Pass the top 20 or so candidates through a reranker and keep only the best 3 to 5. This is the step most people skip and the biggest quality win.
Write a grounding prompt that says answer only from the supplied context, cite the source for each claim, and say you do not know when the context does not cover it.
When an answer is wrong, print the retrieved chunks and their scores before changing the prompt. Bad retrieval is almost always the cause, not the model.
Ask a real question and click each citation to confirm it jumps to the exact line in the source. Also ask something your documents do not cover and check that it refuses.
Tips and common mistakes
Do not paste whole documents into the prompt: it costs more, often does not fit, and answer quality drops anyway.
Keep chunks large enough to hold a complete idea but small enough that the top few fit your context window.
Log the retrieved chunks and scores for every query so you can see why an answer went wrong.
A refusal is a good result when the sources really do not cover the question.
Questions
Do I need to fine-tune the model on my documents?
No. RAG leaves the model alone and supplies the relevant text at question time. That is cheaper and easier to update when documents change.
What if the answer is not in my documents?
A grounded system should say so. Instruct it to refuse when the retrieved context does not contain the answer, and test that case yourself.
Why does the system cite the wrong page?
The citation comes from the retrieved chunk, so a wrong citation means the wrong chunk was fetched. Check your chunking first, then add reranking.
This guide was drafted from the video's details and general task knowledge. It has not been checked against the full video. Check the video and relevant product instructions before following a procedure. Report a correction.
