I Built an AI Orchestration System in My Basement. Here's What Actually Worked.
Spoiler: it wasn't LangChain. Five pipelines running 24/7 on a mid-range GPU — what actually worked, what broke, and the framework that survived.
I bought a mid-range GPU "just to try something." That's the lie I told myself. It was convincing enough that I believed it until the box arrived, the GPU went into the PCIe slot, and I ran my first ollama pull llama3.1:8b. The model downloaded. I ran a prompt. It responded. It was 11pm on a Tuesday.
I stayed up until 4am talking to it. Not because it was impressive — it was a local 8B model, so it was wrong about things with confidence, which is a different kind of impressive — but because something clicked. This thing was running on my hardware. My own rack. My GPU. Not billing me per token. I was hooked in a way I couldn't explain to my wife the next morning.
Six months later I have five pipelines running 24/7. Here's what I learned, what I threw away, and the thing the tutorials never tell you.
Every Framework I Tried Failed the Same Way
I spent the first two months doing what everyone does: LangChain first, then AutoGen, then a custom Python orchestrator I wrote myself, then Flowise, then CrewAI. Each one had a working demo in a day. Each one fell apart in production for exactly the same reason.
LangChain was the worst. I had a RAG pipeline working fine for about a week. Then it started returning inconsistent results I couldn't diagnose. I opened the debugger and found myself reading LangChain source code, not my own code. The chain had swallowed the actual error somewhere inside an abstraction. I had 47 Python packages installed. I couldn't tell you which 12 were doing anything. Threw it out after three weeks.
AutoGen was more interesting but the same problem at a different layer. I set up three agents — a planner, an executor, a critic — and it worked great on simple tasks. Then the planner and critic disagreed on something. I had no idea what messages had passed between them or why the conversation had gone sideways. There's no execution trace. There's no "here is what each agent said and what it decided." Threw it out after two weeks.
My own custom Python framework lasted a month before I realized I was just rebuilding n8n, badly. Error handling, retries, scheduling, logging — I was writing infrastructure code instead of solving my actual problem. That one hurt more because I'd convinced myself that controlling the code meant I understood the system.
n8n Worked on the First Try
I was already using n8n for basic automations — nothing fancy, just some webhook-to-Slack stuff. I needed a scheduled ingestion workflow that would fetch external records, filter them against qualification gates, and push results to my phone via ntfy. I built it in n8n because it was already there. In one of the nodes I added an HTTP call to the Ollama REST endpoint to classify each record.
It worked. The workflow showed me every node. The Ollama response was JSON I could inspect right there in the execution log. When it broke, I could see exactly which node had failed, what the input was, what came back. I fixed it in four minutes.
I rebuilt everything after that.
n8n handles orchestration flow. Ollama handles inference. HTTP calls in between. That's the whole architecture.
The listing evaluation pipeline was first. Then the NAS health monitor. Then the handoff system where I write task markdown files and agents execute the steps via SSH while I sleep. Then the memory layer. The activity feed. All of it is n8n workflows making HTTP calls to Ollama, with results stored in Postgres or SQLite. There is almost no custom Python left — the Python that exists is for ingestion that needs real session management, not because the orchestration required it.
What's Actually Running
Five pipelines. All running on self-hosted infrastructure with GPU-backed Ollama via Docker and a rotating model pool.
Listing evaluator. Runs every three hours. Ingests external records, filters against configurable qualification gates, sends each passing record to qwen2.5-coder:14b for relevance scoring, and creates Jira tickets for high-confidence matches. A human reviews before any irreversible action. This is the pipeline that made the whole project worth it — it surfaces signal I would have missed manually.
NAS health monitor. Every 15 minutes, checks disk temps, pool status, Docker container health, GPU utilization. If anything looks off, ntfy alert to my phone. llama3.1:8b handles the classification here — it's fast and light on GPU memory.
Handoff executor. I write a task markdown file describing what needs to happen. n8n picks it up, sends it to deepseek-r1:14b to reason through the steps, and executes each step via SSH on the NAS. The model choice matters here — deepseek-r1 thinks in chains before answering, which makes it better at multi-step task decomposition than the others.
Memory system. After each session, a workflow pulls the transcript, sends it to qwen2.5-coder:14b to extract learnings, and writes them into Qdrant as vector embeddings. Retrieval runs through LiteLLM so I can swap the embedding model without touching the workflow.
Activity feed. Aggregates events from all of the above into the orchestration dashboard in real time. Mostly just a WebSocket drain into a Postgres table, but it means I can see at a glance what the system has been doing while I was at work.
The stack, actually: n8n for workflow scheduling and data flow, Ollama for inference (qwen2.5-coder:14b for code + classification, deepseek-r1:14b for reasoning, llama3.1:8b for bulk fast tasks), LiteLLM as a unified proxy so model changes don't require touching workflows, Qdrant for vector memory, ntfy for alerts, Postgres for state.
LangChain RAG pipeline: three weeks, worked for one week, broke mysteriously, spent more time reading framework source than my own code. Out.
AutoGen multi-agent setup: planner + executor + critic. Great demos. When the agents disagreed on something real, there was no trace of why. No visibility into the conversation. Out after two weeks.
Custom Python orchestrator: I wrote this myself and it lasted a month. I was proud of it until I realized I had written a bad version of n8n with worse error handling and no UI. Out.
Flowise: Looked promising. Hit a wall the moment I needed any logic that wasn't in the node library. Turns out "low-code" means "easy until it doesn't apply." Out.
CrewAI: Tried it briefly. Same visibility problem as AutoGen. Could not figure out what was happening inside an agent run without reading docs I didn't write. Out.
Every one of these is on disk somewhere. The directories are still there. I keep them as a reminder that frameworks are not the point.
What Nobody Tells You About Model Selection
Every tutorial says "just use llama3" and moves on. That's fine for a demo. In a real system, running one model for everything is slow, memory-hungry, and often wrong.
Fast bulk model (llama3.1:8b): classification, yes/no decisions, quick extraction. Smaller 8B models fit comfortably on local GPU memory and respond in under a second. Use it for anything where you're processing a list.
Code/technical model (qwen2.5-coder:14b): code generation, structured JSON output, anything technical. This is my default. It's consistently better than the non-code-tuned 14B models on every task I've thrown at it.
Reasoning model (deepseek-r1:14b): multi-step planning, anything that needs to think before answering. Slower, but the chain-of-thought output is useful — I log it and sometimes the reasoning trace tells me something the final answer doesn't.
LiteLLM sits in front of all of these. n8n workflows call a local LiteLLM endpoint with a model name. If I want to swap one model for something else next month, I change one config line in LiteLLM, not every workflow.
The Actual Lesson
The system running now shares almost nothing with week one. Week one had LangChain, a different database schema, a logging approach I thought was clever, and a custom orchestrator that seemed smart. The current system works because I rebuilt it five times. Each rebuild I learned one thing and threw out two.
The move that changed everything wasn't a technical decision — it was accepting that I needed to see everything. Every input. Every output. Every failure. Any tool that hid those things from me was the wrong tool, regardless of how good its GitHub stars were.
If you want to start: pull Ollama, run llama3.1:8b, call it with curl, look at the raw JSON. Build three things that call it directly before you install any framework. You will understand more in three days of that than in three weeks of documentation.
The 4am conversations with the local model are still happening. Now they happen automatically while I sleep. That is a better outcome than I had any right to expect from "just trying something."
If you want to start: pull Ollama, run llama3.1:8b, call it with curl, look at the raw JSON. Build three things that call it directly before you install any framework.
Your move: Pick one repetitive task you're currently doing manually — monitoring a dashboard, checking a log file, routing an email. Build it as an n8n workflow with a single Ollama HTTP call. Don't install a framework. Don't write a Python script. Just a curl to localhost:11434 inside a workflow node. If it works in 20 minutes, you know more than you did after a month of LangChain. If it doesn't, you can see exactly where it broke. That visibility alone is worth more than every abstraction layer in PyPI.
Stay with us · challenge
What Framework Will You Choose?
Which AI orchestration framework do you think will be the most reliable for your next project? Share your choice and why.
No account needed — pick a take, then keep reading. We rotate these prompts so each piece feels like a conversation, not a clone.
