What this builds
This guide builds a small retrieval-augmented question-answering tool. It splits your text files into pieces, turns each piece into an embedding, and stores them in a file. To answer a question it embeds the question, finds the most similar pieces and gives them to a local model with instructions to answer only from them. Nothing leaves your server.
It uses plain Python with no extra packages, which keeps it easy to read and fine for a few hundred pages. For larger collections, move the vectors into Qdrant or pgvector (see the vector database guide). A small local model will sometimes miss details or make mistakes, so treat answers as pointers and check the sources it prints.
What you need
Ollama running on the server with an embedding model and a chat model, Python 3 and a folder of plain text or Markdown files. Convert PDFs to text first, for example with pdftotext from the poppler-utils package.
ollama pull nomic-embed-text
ollama pull llama3.2:3b
python3 --version
mkdir -p ~/docqa/docs && cd ~/docqasudo apt install poppler-utils
pdftotext report.pdf docs/report.txtsudo dnf install poppler-utilsWrite the indexer
The script reads each text file in docs, cuts it into overlapping chunks of about 800 characters, asks Ollama for an embedding of each chunk and saves everything to index.json. Re-run it whenever the documents change.
cat > ~/docqa/index.py <<'EOF'
import json, pathlib, urllib.request
OLLAMA = "http://127.0.0.1:11434"
EMBED_MODEL = "nomic-embed-text"
SIZE, OVERLAP = 800, 150
def embed(text):
req = urllib.request.Request(OLLAMA + "/api/embed",
data=json.dumps({"model": EMBED_MODEL, "input": text}).encode(),
headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=120) as r:
return json.load(r)["embeddings"][0]
def chunks(text):
step = SIZE - OVERLAP
for start in range(0, max(len(text), 1), step):
piece = text[start:start + SIZE].strip()
if piece:
yield piece
index = []
for path in sorted(pathlib.Path("docs").rglob("*")):
if path.suffix.lower() not in (".txt", ".md"):
continue
text = path.read_text(encoding="utf-8", errors="ignore")
for n, piece in enumerate(chunks(text)):
index.append({"file": str(path), "chunk": n, "text": piece, "vector": embed(piece)})
print("indexed", path, flush=True)
pathlib.Path("index.json").write_text(json.dumps(index))
print(len(index), "chunks saved")
EOF
python3 -m py_compile ~/docqa/index.py && echo "Syntax OK"Write the question script
This script embeds the question, ranks every chunk by cosine similarity, takes the best four, and asks the chat model to answer using only that context. It prints the answer followed by the files and chunk numbers that were used, so you can check them.
cat > ~/docqa/ask.py <<'EOF'
import json, math, sys, urllib.request
OLLAMA = "http://127.0.0.1:11434"
EMBED_MODEL, CHAT_MODEL, TOP_K = "nomic-embed-text", "llama3.2:3b", 4
def post(path, payload):
req = urllib.request.Request(OLLAMA + path, data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=300) as r:
return json.load(r)
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)) or 1)
question = " ".join(sys.argv[1:]).strip()
if not question:
sys.exit("usage: python3 ask.py your question here")
index = json.load(open("index.json"))
qvec = post("/api/embed", {"model": EMBED_MODEL, "input": question})["embeddings"][0]
best = sorted(index, key=lambda c: cosine(qvec, c["vector"]), reverse=True)[:TOP_K]
context = "\n\n---\n\n".join(c["text"] for c in best)
reply = post("/api/chat", {"model": CHAT_MODEL, "stream": False, "messages": [
{"role": "system", "content": "Answer the question using only the context. If the context does not contain the answer, say you do not know."},
{"role": "user", "content": "Context:\n" + context + "\n\nQuestion: " + question}]})
print(reply["message"]["content"])
print("\nSources:")
for c in best:
print(" -", c["file"], "chunk", c["chunk"])
EOF
python3 -m py_compile ~/docqa/ask.py && echo "Syntax OK"Run it on a few documents
Put a few text files in docs, build the index and ask a question you can verify. Indexing time grows with the amount of text, so try a small set first and time it. The first answer is slow while the chat model loads.
cd ~/docqa
cp /etc/os-release docs/os-release.txt
time python3 index.py
python3 ask.py "What operating system is this server running?"indexed docs/os-release.txt
1 chunks saved
Based on the provided context, the operating system is Ubuntu.
Sources:
- docs/os-release.txt chunk 0
Indexed chunks: 1 embedding dimensions: 768Improve the results
Retrieval quality matters more than the model. If answers miss information, try smaller chunks for facts and larger ones for narrative text, retrieve more chunks, or clean the text before indexing, because navigation menus and page numbers pollute search. Keep the instruction to answer only from the context, and check the source list for every answer.
If answers are too slow, use a smaller chat model, fewer chunks or shorter chunks. The prompt length affects speed because the model must read the whole context before answering.
Keep documents private
The index file contains your text and its vectors, so protect it like the documents. Keep the folder readable only by you, do not publish the service without a login, and delete the index when you no longer need it. The tools make requests only to Ollama on this server, which you can verify by reading the scripts.
chmod 700 ~/docqa
chmod 600 ~/docqa/index.json
sudo ss -lntp | grep 11434Official documentation
These upstream references provide more detail on the commands and workflows in this guide.