What this builds
This guide runs a small web service that accepts a question and returns an answer from a model served by Ollama on the same server. It uses only Python's standard library, so there is nothing to install beyond Python itself. The service listens on 127.0.0.1, runs as its own user and restarts if it fails. Put it behind the reverse proxy and add authentication before any public use.
You need Ollama running with a small model (see the Ollama guide) and Python 3, which both Ubuntu 24.04 and AlmaLinux 9 include.
python3 --version
curl -s http://127.0.0.1:11434/api/tags | head -c 200Create the user and folder
Create a system user for the service and a folder for its code that root owns, so that the service cannot change its own program.
sudo useradd --system --home-dir /opt/chatbot --shell /usr/sbin/nologin chatbot
sudo mkdir -p /opt/chatbotWrite the service
The program reads a JSON body, asks Ollama for a short answer, and returns it. It limits the size of the question, applies a timeout to the model call and a system prompt that keeps answers short. Adapt the model name to the one you pulled. The bind address is the loopback interface on purpose.
sudo tee /opt/chatbot/app.py >/dev/null <<'EOF'
import json, os, urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
OLLAMA = os.environ.get("OLLAMA_URL", "http://127.0.0.1:11434/api/chat")
MODEL = os.environ.get("MODEL", "llama3.2:3b")
PORT = int(os.environ.get("PORT", "8000"))
MAX_QUESTION = 2000
class Handler(BaseHTTPRequestHandler):
def _send(self, code, payload):
body = json.dumps(payload).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_POST(self):
if self.path != "/ask":
return self._send(404, {"error": "not found"})
try:
length = int(self.headers.get("Content-Length", 0))
if length < 1 or length > 10000:
return self._send(413, {"error": "request too large"})
value = json.loads(self.rfile.read(length))["question"]
if not isinstance(value, str) or not value.strip():
return self._send(400, {"error": "question must be a non-empty string"})
if len(value.strip()) > MAX_QUESTION:
return self._send(400, {"error": "question is too long"})
question = value.strip()
except Exception:
return self._send(400, {"error": "send JSON like {\"question\": \"...\"}"})
req = urllib.request.Request(OLLAMA, data=json.dumps({
"model": MODEL, "stream": False,
"options": {"num_predict": 300},
"messages": [
{"role": "system", "content": "You are a concise, helpful assistant. Answer in a few sentences."},
{"role": "user", "content": question}]}).encode(),
headers={"Content-Type": "application/json"})
try:
with urllib.request.urlopen(req, timeout=120) as r:
answer = json.load(r)["message"]["content"]
except Exception as exc:
print("model call failed:", exc, flush=True)
return self._send(502, {"error": "model unavailable"})
print("answered a question of", len(question), "characters", flush=True)
self._send(200, {"answer": answer})
def log_message(self, *args):
pass
ThreadingHTTPServer(("127.0.0.1", PORT), Handler).serve_forever()
EOF
sudo chown -R root:root /opt/chatbot
python3 -m py_compile /opt/chatbot/app.py && echo "Syntax OK"Create the unit file
The unit runs the program as the chatbot user. The restart policy brings it back after a crash, and the sandboxing options remove most privileges it does not need. The service needs to reach only the local network, so you could also add IPAddressAllow rules if you want to be stricter.
sudo tee /etc/systemd/system/chatbot.service >/dev/null <<'EOF'
[Unit]
Description=Small AI chatbot service
After=network.target ollama.service
[Service]
User=chatbot
Group=chatbot
WorkingDirectory=/opt/chatbot
Environment=MODEL=llama3.2:3b
ExecStart=/usr/bin/python3 /opt/chatbot/app.py
Restart=on-failure
RestartSec=3
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now chatbot
systemctl status chatbot --no-pagerAsk a question
Call the service from the server. The first request is slower because Ollama has to load the model into memory. A tens-of-seconds first answer on a small plan is normal, and later requests answer faster while the model stays loaded.
curl -s -m 150 http://127.0.0.1:8000/ask -H "Content-Type: application/json" -d '{"question":"What is a reverse proxy?"}'Add limits before anyone else can use it
This service has no authentication and no rate limit. Anyone who can reach it can use your CPU. Keep it on localhost and call it from your own application. If you publish it, put Nginx in front with HTTPS and basic authentication or another login, as the security guide describes, and add rate limiting there. Also limit concurrency: a small server handles one or two generations at a time.
limit_req_zone $binary_remote_addr zone=chat:10m rate=10r/m;
# inside the location block
limit_req zone=chat burst=5 nodelay;/ask HTTP 400 send JSON like {"question": "..."}
/ask HTTP 400 question must be a non-empty string
/ask HTTP 400 question must be a non-empty string
/ask HTTP 400 question is too long
/ask HTTP 413 request too largeRead the logs and update
Everything the program prints goes to the journal. The program logs lengths, not the questions, so you can count usage without storing people's text. To change the code, edit the file and restart. Test with curl after each change.
sudo journalctl -u chatbot -f
sudo systemctl restart chatbotOfficial documentation
These upstream references provide more detail on the commands and workflows in this guide.