Local LLM · pgvector · building a RAG chatbot

Sometimes a single ChatGPT call is not enough. Internal docs, personal notes, data you cannot send outside. RAG (Retrieval Augmented Generation) lets an LLM answer only from materials you hand-pick.

Who it's for

• Engineers running LLMs on local GPUs or on-prem without sending data out
• Anyone who wants a chatbot that answers with citations from their own documents
• People wanting a single track covering embeddings, vector search, and prompt design

What you can do afterwards

• Run Gemma / Llama family models locally with LM Studio
• Store embeddings in PostgreSQL + pgvector with HNSW indexes
• Build a minimal FastAPI + LangChain pipeline (retrieve → prompt → generate)
• Swap Gemini and local LLMs freely
• Control system prompts, few-shot, and output schemas
• Design WebGPU, WASM, and server fallbacks with a model-transfer budget

Flow

Operating retrieval-augmented generation

1Local foundation

Start with local models and embeddings so data stays within the controlled boundary.

2Retrieval store

Connect vector search and evidence assembly into a reproducible RAG pipeline.

3Cost transition

Keep cloud models disabled by default and enable them only after quality, budget, and transfer gates pass.

4Execution choice

Compare prompts, SaaS, and browser inference by latency, cost, and privacy.

The first half (1–4) covers retrieval mechanics. The second half (5–8) makes operational choices about models, prompts, execution location, and transfer cost.

Steps

1. Why local LLMs · getting started with LM Studio — OpenAI-compatible endpoint · swapping models · VRAM
2. Embeddings — text to vectors — the math behind semantic search · 768 dims
3. pgvector + HNSW setup — install · index choice · cosine vs dot product
4. RAG pipeline — chunking · retrieve · top-k · rerank · prompt injection
5. Gemini · OpenAI-compatible APIs — switching local ↔ cloud · cost · latency
6. Prompt design — system prompts · few-shot · output schemas · hallucination
7. NotebookLM vs your own RAG — SaaS RAG comparison; choosing the right tool per slot
8. Browser ONNX inference — WebGPU/WASM capability, lazy model loading, server fallback

Prerequisites — python-data-pipeline + Python 3.13+ + uv + PostgreSQL 15+ + LM Studio.