Building a Local RAG (Notebook + API)

This guide walks you through building a fully local, CPU-only Retrieval-Augmented Generation (RAG) system from a PDF on disk to a working API you can query. Everything runs on your own machine: no API keys, no cloud costs, no data leaving your laptop. Kindly note that this is implementation assumes you have little resources i.e no GPU enable machine else lucky you! In this guide we will use Ollama for running local models, and yes you can embrace other tools however for this we will use Ollama

The Core Cycle

• Ingest = load the document, split it into chunks, embed the chunks, store them in a vector database.
• Query = take a question, retrieve the most relevant chunks, hand them to an LLM along with the question, and return the generated answer.

Start Here

Set-Up

Python, virtual Environment

• Download Python from here
• We create virtual environment with the aim of isolating your python projects. There other techniques of isolationg your projects:
  • Docker
  • Kubernetes

creating a virtual Environment:

• create folder > open the folder on visual studio
• open terminal via visual studio
• run this: python -m venv env , kindly note env is the name of your environment
• to activate the environment:

  • on windows:

    • approach one:

      • open command prompt on vs code
      • navigate to envrionment folder i.e env > open the Scripts
      • find activate.bat file
      • click...hold and drag to the terminal space
      • press enter

    • approach 2:

      • open command prompt on vs code
      • run this: .\env\Scripts\activate.bat
      • press enter

  • for linux/ macos: - open terminal on vs code - run: source env/bin/activate

Finally:

• Create a .gitignore file : it makes sure that you only version control what is meant to be version controlled

kindly make sure you have your virtual environment is activated if so kindly run the following on your terminal:

• For Ollama models:

ollama pull nomic-embed-text # meant for the embedding model

• For the python libraries:

pip install fastapi uvicorn
pip install "unstructured[pdf]" langchain langchain_community langchain-text-splitters chromadb
pip install langchain-ollama langchain-chroma

Note:

• chromadb meant for the vector DB(store vector representations of our document(s))
• langchain-text-splitters # this will help us get the text splitter for chunking
• langchain langchain_community => our core framework/ library to build AI applications in particular LLM and Agentic based applications
• since we are focusing on pdfs alone you can download the necessary library to suppor that using pip install "unstructured[pdf]" else go for this(quit large) pip install "unstructured[all-docs]" it will have the capacity to handle many file types

API

Install FastAPI create project: uv init {app_name} --bare, our app name is api add fastAPI: uv add "fastapi[standard]" cd api && uv add "fastapi[standard]"