Local LLM · pgvector · building a RAG chatbot
Connect LM Studio, pgvector, Gemini, and browser ONNX inference in eight steps, from retrieval to model-delivery cost.
- Difficulty
- Advanced
- Lessons
- 8
Sometimes a single ChatGPT call is not enough. Internal docs, personal notes, data you cannot send outside. RAG (Retrieval Augmented Generation) lets an LLM answer only from materials you hand-pick.
Who it's for
- Engineers running LLMs on local GPUs or on-prem without sending data out
- Anyone who wants a chatbot that answers with citations from their own documents
- People wanting a single track covering embeddings, vector search, and prompt design
What you can do afterwards
- Run Gemma / Llama family models locally with LM Studio
- Store embeddings in PostgreSQL + pgvector with HNSW indexes
- Build a minimal FastAPI + LangChain pipeline (retrieve → prompt → generate)
- Swap Gemini and local LLMs freely
- Control system prompts, few-shot, and output schemas
- Design WebGPU, WASM, and server fallbacks with a model-transfer budget
Flow
Operating retrieval-augmented generation
Start with local models and embeddings so data stays within the controlled boundary.
Connect vector search and evidence assembly into a reproducible RAG pipeline.
Keep cloud models disabled by default and enable them only after quality, budget, and transfer gates pass.
Compare prompts, SaaS, and browser inference by latency, cost, and privacy.
The first half (1–4) covers retrieval mechanics. The second half (5–8) makes operational choices about models, prompts, execution location, and transfer cost.
Steps
- Why local LLMs · getting started with LM Studio — OpenAI-compatible endpoint · swapping models · VRAM
- Embeddings — text to vectors — the math behind semantic search · 768 dims
- pgvector + HNSW setup — install · index choice · cosine vs dot product
- RAG pipeline — chunking · retrieve · top-k · rerank · prompt injection
- Gemini · OpenAI-compatible APIs — switching local ↔ cloud · cost · latency
- Prompt design — system prompts · few-shot · output schemas · hallucination
- NotebookLM vs your own RAG — SaaS RAG comparison; choosing the right tool per slot
- Browser ONNX inference — WebGPU/WASM capability, lazy model loading, server fallback
Prerequisites — python-data-pipeline + Python 3.13+ + uv + PostgreSQL 15+ + LM Studio.