Building a Personal AI Brain: Qdrant, Embeddings, and the Illusion of Memory
Vector databases sound magical until you actually use one. The story of my Qdrant setup, the grocery list that broke cosine similarity, and the domain filters that fixed it.
I was debugging my Qdrant setup on a Saturday afternoon, feeling very clever about my self-hosted AI memory system, when I asked it something reasonable: "What's the status of my recent project follow-ups?" I got back a note I'd stored about needing to pick up eggs, Greek yogurt, and that specific brand of oat milk my wife insists on.
Not a hallucination. Not a bug. A mathematically perfect cosine similarity match. The grocery list had used words like "items," "pending," "need to check," and "follow up" — the exact same vocabulary fingerprint as a project status note. The embedding model found patterns. It found its nearest neighbor in vector space. It returned results with full confidence.
It was correct by the only measure it knew. That's the moment the illusion broke for me — and where this post starts.
embed-text
What I Actually Use This For
Before getting into what went wrong, here's why I built it: context initialization. When a new session starts, the first thing it does is query the knowledge base for relevant context based on what I'm working on that day. That 5-second query replaces 10 minutes of me re-explaining project history, decisions made, things tried, things abandoned.
Running in my system: Qdrant on the NAS, the nomic-embed-text model for generating embeddings (runs locally on Ollama — no API calls, no cost), two collections: a personal knowledge base with thousands of points of session notes, task outcomes, and project summaries, and a research archive for reading snapshots. Operating budget: modest monthly power beyond hardware already running.
The collection gets populated automatically. After each session, a script runs in the background. It pulls key facts and decisions from the session transcript, formats them into clean structured notes, generates embeddings via nomic-embed-text, and upserts them to Qdrant. No manual curation step. The curation happens upstream — during the session, when I'm explicit about what I decided and why. Garbage in, grocery lists out. Good structured notes in, useful memory out.
AI doesn't remember. It retrieves. The quality of retrieval determines the quality of the memory illusion.
At thousands of points, it's genuinely useful. I can ask about past architecture decisions and get back the actual reasoning from months ago. That's the memory illusion working as intended.
How It Works (The Two-Sentence Version)
An embedding model converts text into a list of numbers — a vector — where similar-meaning text produces similar vectors. Vector search finds the stored vectors nearest to your query vector. That's it. It's a nearest-neighbors problem, not memory. The system doesn't "know" your work tasks any more than Google "remembers" your search history — it finds what's mathematically most similar to what you asked.
Cosine similarity measures the angle between two vectors — how similar their direction is in high-dimensional space. It's excellent at capturing semantic similarity. It will also confidently return a grocery list when you ask about project follow-ups if those two things happen to live near each other in the model's concept space. Similarity is not relevance. This distinction breaks systems that don't account for it.
The Chunk Size Problem Nobody Tells You About
When you index documents for vector search, you split them into chunks first. The size of those chunks is one of the most consequential decisions you'll make — and almost nothing online tells you how to actually choose.
Too small: a 64-token chunk might be a single sentence. "The project is on hold." Nearly meaningless when retrieved — on hold because of what? Which project? Who decided? Too large: a 2048-token chunk contains 15 different topics. The embedding tries to represent all of them and ends up representing none of them well. The retrieval signal dilutes across everything crammed into the chunk.
For factual notes and project summaries: 256–512 tokens with 10% overlap. For conversational content: 512–1024 tokens. The overlap is not optional — it prevents relevant context from being split across chunk boundaries and lost. My knowledge base collection settled on 384 tokens with 15% overlap after testing with actual queries. Your content type will produce different numbers. Test with your real data, not synthetic examples. There is no universal answer and anyone who gives you one hasn't tested it.
The Fix: Domain Filtering Before Vector Search
The grocery list incident had a specific root cause: the knowledge base was a single flat namespace. Personal reminders, grocery lists, infrastructure decisions, and project notes all lived in the same vector space and competed on the same similarity metric. A grocery list that used task-tracking language was indistinguishable from a work note at query time.
The fix was adding a domain metadata field to every stored point — values like jobs, infra, personal, research — and rewriting all queries to filter by domain before running the vector search. Hybrid retrieval: metadata filter first, then vector search inside that filtered subset. The false positive rate dropped to near zero.
Before: Query "status of recent project tasks" against the full collection. Results ranked by cosine similarity across all points. Grocery list wins on vocabulary overlap.
After: Query pre-filtered to domain = "jobs", then vector search within that subset. Only job-related notes are in the candidate pool. The grocery list isn't even evaluated. The correct result returns in the top 3 every time.
The lesson isn't that vector search is flawed — it's that semantic similarity across unrelated domains is the wrong retrieval problem to solve. Namespace your data. Filter before you search. The vector similarity does its job well when it's operating on a coherent domain, not a soup of everything you've ever written down.
They show you happy-path retrieval. "Ask about Paris, get the Paris document." Nobody shows you the grocery list problem — where similarity is high but relevance is zero. They don't cover the metadata filter pattern that fixes it. They don't mention that your embedding model's concept space was learned from internet text, meaning domain-specific vocabulary (SAP transaction codes, internal project names) may not embed well without fine-tuning. They don't tell you that nomic-embed-text outperforms OpenAI's ada-002 on several benchmarks and runs locally for free. Read the paper, not just the tutorial.
The Honest Summary
Qdrant is excellent software. The self-hosted setup on my NAS runs reliably, costs nothing extra, and has made every session more productive by eliminating the context re-establishment overhead that was silently eating 10-15 minutes of every conversation.
But the grocery list was the most useful thing that happened to this project. It broke my mental model before I'd built a production pipeline on a false assumption — that vector similarity equals relevance. Add a domain field. Filter before you search. Be specific in what you store. Do those three things and the memory illusion works well enough to be genuinely useful.
Don't skip them and you'll get grocery lists. Which, in retrospect, is a pretty good outcome for catching a fundamental design flaw early.
Stay with us · decision
Do You Trust Your AI Memory System?
Have you ever encountered a situation where an AI system misunderstood your query or provided incorrect information? What steps do you take to ensure the accuracy and reliability of your AI systems?
No account needed — pick a take, then keep reading. We rotate these prompts so each piece feels like a conversation, not a clone.
