Building an AI Agent Knowledge Base: MCP + RAG + Vector DB for Extended Local Memory

Introduction

Every AI agent has the same fundamental limitation: it forgets. Each conversation starts from zero. The research it did yesterday, the preferences it learned, the documents it analyzed — all gone when the session ends. Context windows are getting larger (Gemini's 2M tokens, Claude's 200K), but they're still finite, expensive to fill, and ephemeral.

What if your AI agent could build a permanent, searchable knowledge base from everything it encounters? Every web page it researches, every PDF it reads, every conversation insight — indexed, embedded, and retrievable in milliseconds. That's what we're building in this guide.

The stack is three technologies working together:

• MCP (Model Context Protocol) — Anthropic's open standard that lets AI agents connect to external tools and data sources through a universal interface
• RAG (Retrieval-Augmented Generation) — The technique of finding relevant information and injecting it into the AI's prompt before it answers
• Vector Databases — Purpose-built storage that understands semantic similarity, not just keyword matching

By the end, you'll have a working system where your AI agent can store anything it learns and recall it when needed — across sessions, across topics, growing smarter over time.

The Problem: AI Amnesia

Let's be concrete about what we're solving. Consider an AI agent like OpenClaw running Claude:

• Monday: You ask it to research vector databases. It reads 15 articles, compares 8 products, writes a summary. Brilliant work.
• Wednesday: You ask "what did you find about Qdrant vs ChromaDB?" — blank stare. New session. All gone.
• Friday: You ask it to write a blog post about vector databases. It starts from scratch, potentially reaching different conclusions than Monday.

The agent has no persistent memory. It's like having an incredibly smart colleague who gets amnesia every time they leave the room. File-based memory (like markdown notes) helps, but it doesn't scale — you can't semantic-search through 10,000 markdown files efficiently.

MCP: The Universal Plug for AI Agents

What Is MCP?

The Model Context Protocol (MCP) is an open standard released by Anthropic in November 2024. Think of it as USB-C for AI — a single, universal interface that connects any AI model to any external tool or data source.^[1]

Before MCP, connecting an AI agent to, say, a database required building a custom integration for each combination of AI model and data source. If you had 5 AI models and 10 data sources, that's 50 custom integrations. MCP collapses this to 5 + 10 = 15: each model implements MCP once, each data source implements MCP once, and they all work together.

How MCP Works

MCP uses a client-server architecture built on JSON-RPC 2.0, inspired by the Language Server Protocol (LSP) that powers IDE features like autocomplete:^[2]

• MCP Client — Lives inside the AI application (Claude Desktop, OpenClaw, Cursor, etc.). It discovers and calls tools exposed by servers.
• MCP Server — A lightweight process that exposes specific capabilities. It can provide tools (functions the AI can call), resources (data the AI can read), and prompts (templates for common operations).
• Transport — Communication happens over stdio (local processes) or SSE/HTTP (remote servers).

When Claude Desktop connects to an MCP server, it discovers what tools are available and can call them as needed during conversation. The AI decides when to use a tool based on the user's question — it's not hardcoded.

Why Anthropic Created MCP

The origin story is pragmatic. Developer David Soria Parra was frustrated with constantly copying context between tools and AI assistants.^[3] Every integration was a one-off. MCP emerged as the universal solution — build the connector once, use it everywhere. In December 2025, Anthropic donated MCP to the Agentic AI Foundation (AAIF) under the Linux Foundation, co-founded with Block and OpenAI, signaling it's becoming an industry standard, not just an Anthropic project.^[2]

MCP vs Traditional API Integrations

Existing MCP Servers for Knowledge & Memory

The MCP ecosystem already has servers relevant to building a knowledge base:

• mcp-server-qdrant — Official Qdrant MCP server. Store and retrieve semantic memories. Configure collection name, embedding model, and search parameters. Drop-in knowledge base.^[4]
• rag-mcp-server (PyPI) — Generic RAG server supporting multiple embedding models including multilingual options. Initialize a knowledge base from a directory, then search it.^[5]
• mcp-server-filesystem — Read/write local files. The simplest "memory" — save notes as files. Limited to keyword search.
• postgres-mcp — Full PostgreSQL access via MCP. Combined with pgvector, this becomes a knowledge base with SQL power.^[6]
• knowledge-rag (Lobehub) — Document ingestion + vector search with configurable chunking.

RAG: How Retrieval-Augmented Generation Works

RAG is the technique that makes this all work. Instead of hoping the AI "knows" something from training, you find relevant information and inject it into the prompt before the AI generates a response. It's like giving the AI an open-book exam instead of a closed-book one.

The RAG pipeline has two phases:

Ingestion Pipeline (Offline)

1. Ingest — Load documents (PDFs, web pages, markdown, tweets, code files)
2. Chunk — Split documents into smaller pieces (typically 200–1000 tokens each)
3. Embed — Convert each chunk into a high-dimensional vector using an embedding model
4. Store — Save vectors + original text + metadata in a vector database

Query Pipeline (Online)

1. Embed query — Convert the user's question into a vector using the same embedding model
2. Search — Find the most similar vectors in the database (cosine similarity / dot product)
3. Rerank (optional) — Use a cross-encoder to re-score results for better precision
4. Augment — Inject the top-k retrieved chunks into the LLM prompt as context
5. Generate — The LLM answers using both its training knowledge and the retrieved context

Chunking Strategies

How you split documents determines retrieval quality more than almost any other decision. Get this wrong and your agent retrieves garbage; get it right and it's magic.

Embedding Models

The embedding model converts text into vectors. The quality of these vectors directly determines retrieval accuracy. Here's the current landscape:

Retrieval Strategies & Reranking

Similarity Search

The simplest approach: embed the query, find the K nearest vectors by cosine similarity. Fast, works well for straightforward questions. This is what 90% of RAG implementations use.

Hybrid Search (BM25 + Vector)

Combines traditional keyword matching (BM25/TF-IDF) with semantic vector search. This catches cases where the user asks for a specific term that semantic search might miss. For example, searching for "pgvector" — keyword search finds exact matches, while vector search finds conceptually similar content about "PostgreSQL vector extension."

Weaviate and Qdrant have hybrid search built-in. For others, you implement it by running both searches and merging results with reciprocal rank fusion (RRF).

Reranking

After initial retrieval (say, top 20 results), a cross-encoder reranker scores each result against the original query more carefully. This is slower but dramatically improves precision. Popular rerankers:

• Cohere Rerank — API-based, easy to use, excellent quality
• bge-reranker-v2-m3 — Open-source, run locally
• Pinecone Rerank — Built into Pinecone's pipeline

Vector Database Comparison

This is the most-asked question in the RAG space. Here's an honest comparison based on real-world usage, community sentiment, and benchmarks:

Which Should You Pick?

The Complete Architecture

Here's what we're building — an AI agent (OpenClaw/Claude) with persistent, searchable memory that grows over time:

Data Pipeline: Ingest → Chunk → Embed → Store

When the agent encounters useful information (articles, docs, conversations), it calls the MCP server's ingest tool:

1. Ingest: Accept raw text, URL, file path, or HTML
2. Parse: Extract clean text (strip HTML, extract PDF text)
3. Chunk: Split into ~512-token pieces with overlap
4. Embed: Send chunks to embedding API, get vectors back
5. Store: Upsert vectors + text + metadata into vector DB

Query Pipeline: Question → Retrieve → Augment

When the agent needs to answer a question, it calls the search tool:

1. Embed query: Convert the question to a vector
2. Search: Find top-K similar chunks (K=5-10)
3. Filter: Apply metadata filters (date range, source type)
4. Return: Send relevant chunks back to the agent as context

Getting Started: The Simplest Setup

Let's build the simplest possible knowledge base: ChromaDB + OpenAI embeddings + a custom MCP server. Total setup time: ~15 minutes.

Prerequisites

• Python 3.10+
• An OpenAI API key (for embeddings)
• Claude Desktop, OpenClaw, or any MCP-compatible client

Step 1: Install Dependencies

pip install chromadb openai mcp[cli] beautifulsoup4 pypdf2

Step 2: Create the MCP Server

This server exposes three tools: ingest_text, ingest_url, and search.

Complete Code: Knowledge Base MCP Server

# knowledge_mcp_server.py
from mcp.server import Server
from mcp.server.stdio import stdio_server
from mcp.types import Tool, TextContent
import chromadb
from openai import OpenAI
import json, hashlib, re
from bs4 import BeautifulSoup
import urllib.request

# --- Configuration ---
EMBEDDING_MODEL = "text-embedding-3-small"
COLLECTION_NAME = "knowledge_base"
CHUNK_SIZE = 512       # tokens (roughly 4 chars per token)
CHUNK_OVERLAP = 50     # token overlap between chunks
TOP_K = 8              # number of results to return

# --- Init ---
openai_client = OpenAI()  # uses OPENAI_API_KEY env var
chroma_client = chromadb.PersistentClient(path="./knowledge_db")
collection = chroma_client.get_or_create_collection(
    name=COLLECTION_NAME,
    metadata={"hnsw:space": "cosine"}
)

server = Server("knowledge-base")

def chunk_text(text: str, chunk_size: int = CHUNK_SIZE,
               overlap: int = CHUNK_OVERLAP) -> list[str]:
    """Split text into overlapping chunks by approximate token count."""
    words = text.split()
    tokens_per_chunk = chunk_size * 4  # rough char estimate
    chars_overlap = overlap * 4
    chunks = []
    start = 0
    text_clean = " ".join(words)
    while start < len(text_clean):
        end = start + tokens_per_chunk
        chunk = text_clean[start:end]
        if chunk.strip():
            chunks.append(chunk.strip())
        start = end - chars_overlap
    return chunks if chunks else [text_clean[:tokens_per_chunk]]

def get_embeddings(texts: list[str]) -> list[list[float]]:
    """Get embeddings from OpenAI."""
    response = openai_client.embeddings.create(
        model=EMBEDDING_MODEL,
        input=texts
    )
    return [e.embedding for e in response.data]

def ingest(text: str, source: str = "manual",
           url: str = "", metadata: dict = None):
    """Chunk, embed, and store text."""
    chunks = chunk_text(text)
    embeddings = get_embeddings(chunks)
    ids = [hashlib.md5(
        (c + source).encode()).hexdigest() for c in chunks]
    metadatas = [
        {"source": source, "url": url,
         "chunk_index": i, "total_chunks": len(chunks),
         **(metadata or {})}
        for i in range(len(chunks))
    ]
    collection.upsert(
        ids=ids, embeddings=embeddings,
        documents=chunks, metadatas=metadatas
    )
    return len(chunks)

@server.list_tools()
async def list_tools():
    return [
        Tool(name="ingest_text",
             description="Store text in the knowledge base "
                         "for future retrieval.",
             inputSchema={
                 "type": "object",
                 "properties": {
                     "text": {"type": "string",
                              "description": "Text to store"},
                     "source": {"type": "string",
                                "description": "Source label"},
                     "url": {"type": "string",
                             "description": "Source URL"}
                 },
                 "required": ["text"]
             }),
        Tool(name="ingest_url",
             description="Fetch a URL, extract text, and store "
                         "in the knowledge base.",
             inputSchema={
                 "type": "object",
                 "properties": {
                     "url": {"type": "string",
                             "description": "URL to fetch"}
                 },
                 "required": ["url"]
             }),
        Tool(name="search",
             description="Search the knowledge base for "
                         "information relevant to a query.",
             inputSchema={
                 "type": "object",
                 "properties": {
                     "query": {"type": "string",
                               "description": "Search query"},
                     "n_results": {"type": "integer",
                                   "description": "Max results",
                                   "default": TOP_K}
                 },
                 "required": ["query"]
             }),
    ]

@server.call_tool()
async def call_tool(name: str, arguments: dict):
    if name == "ingest_text":
        n = ingest(arguments["text"],
                   arguments.get("source", "manual"),
                   arguments.get("url", ""))
        return [TextContent(type="text",
                text=f"Stored {n} chunks in knowledge base.")]

    elif name == "ingest_url":
        url = arguments["url"]
        req = urllib.request.Request(url, headers={
            "User-Agent": "KnowledgeBot/1.0"})
        html = urllib.request.urlopen(req).read().decode()
        soup = BeautifulSoup(html, "html.parser")
        for tag in soup(["script", "style", "nav", "footer"]):
            tag.decompose()
        text = soup.get_text(separator="\n", strip=True)
        n = ingest(text, source="web", url=url)
        return [TextContent(type="text",
                text=f"Fetched and stored {n} chunks from {url}")]

    elif name == "search":
        query = arguments["query"]
        n_results = arguments.get("n_results", TOP_K)
        q_embedding = get_embeddings([query])[0]
        results = collection.query(
            query_embeddings=[q_embedding],
            n_results=n_results,
            include=["documents", "metadatas", "distances"]
        )
        output = []
        for i, doc in enumerate(results["documents"][0]):
            meta = results["metadatas"][0][i]
            dist = results["distances"][0][i]
            relevance = round(1 - dist, 3)
            output.append(
                f"[{relevance}] {doc[:500]}\n"
                f"  Source: {meta.get('source', '?')} "
                f"| URL: {meta.get('url', 'N/A')}")
        return [TextContent(type="text",
                text="\n\n".join(output) if output
                     else "No relevant results found.")]

async def main():
    async with stdio_server() as (read, write):
        await server.run(read, write,
                         server.create_initialization_options())

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

Step 3: Configure Your AI Client

Add to Claude Desktop's claude_desktop_config.json:

{
  "mcpServers": {
    "knowledge-base": {
      "command": "python",
      "args": ["knowledge_mcp_server.py"],
      "env": {
        "OPENAI_API_KEY": "sk-..."
      }
    }
  }
}

For OpenClaw, add to your MCP configuration:

# In your OpenClaw config
mcp:
  servers:
    knowledge-base:
      command: python
      args: ["/path/to/knowledge_mcp_server.py"]
      env:
        OPENAI_API_KEY: "sk-..."

Step 4: Use It

Now your agent has three new capabilities:

• "Store this article about Kubernetes security" → Agent calls ingest_url
• "Remember that pgvector supports HNSW indexes" → Agent calls ingest_text
• "What do we know about vector database performance?" → Agent calls search, gets relevant chunks, incorporates them into its response

Ingesting Different Data Types

# HTML/Web pages — handled by ingest_url
# For PDFs:
from pypdf2 import PdfReader
reader = PdfReader("document.pdf")
text = "\n".join(page.extract_text() for page in reader.pages)
ingest(text, source="pdf", url="document.pdf")

# For Markdown files:
with open("notes.md") as f:
    ingest(f.read(), source="markdown", url="notes.md")

# For tweets/posts (pass as text):
ingest("Thread by @karpathy: The hottest new programming "
       "language is English...",
       source="twitter", url="https://x.com/karpathy/...")

RAG Pitfalls & How to Avoid Them

1. Chunk Size Mismatch

Too small (50 tokens): Chunks lack context. "It costs $20/month" — what costs $20/month? Too large (2000 tokens): Chunks contain too many topics, reducing precision. Sweet spot: 256–512 tokens for most content.

2. Irrelevant Retrieval

The top-K results look relevant by embedding similarity but don't actually answer the question. Fix: Add a relevance threshold (drop results below 0.7 cosine similarity). Add reranking.

3. Hallucination from Bad Context

The AI gets retrieved chunks that are tangentially related and confabulates an answer that sounds right but isn't grounded. Fix: Instruct the model to cite which retrieved chunk supports each claim. Use "answer only from the provided context" system prompts.

4. Stale Data

Your knowledge base has outdated information but no way to know. Fix: Store ingestion timestamps as metadata. Let the agent filter by recency. Implement a refresh/re-ingest pipeline for important sources.

5. Embedding Model Mismatch

You embedded documents with model A and query with model B. Vectors live in incompatible spaces. Fix: Never mix embedding models. If you switch models, re-embed everything.

What the Community Says

Hacker News

The HN community has been actively discussing these technologies:

• "Forget Vector Databases: RAG with Just SQL and LLM" — RisingWave's argument that you can do RAG with SQL streaming, no vector DB needed. Interesting but niche.^[9]
• "I accidentally built SQLite for AI memory (Memvid)" — A project storing memories as video-encoded vectors. Creative approach to the persistence problem.^[10]
• "Clamp — Git-like version control for RAG vector databases" — Versioning for your knowledge base, solving the "oops I corrupted my embeddings" problem.
• MCP discussions are exploding — MindsDB, PostgreSQL, and dozens of other tools adding MCP support as it becomes the de facto standard.

Reddit (r/MachineLearning)

The consensus on vector database choice from the ML community:^[8]

• "Qdrant replaced Milvus and I've had no issues" — common sentiment
• "FAISS is NOT a vector database. It's a lower-level index" — important distinction
• "Pinecone has vendor lock-in; I tend to avoid these" — preference for open-source
• ChromaDB getting better but still seen as prototyping-only by many

GitHub Trending

The major RAG frameworks being used in production:

• LlamaIndex — Purpose-built for RAG. Best for connecting LLMs to custom data. Excellent MCP integration.
• LangChain — General-purpose LLM framework. Larger ecosystem but more complexity. Good for chains of operations.
• Haystack (deepset) — Production-focused. Strong on evaluation and pipeline composition.
• RAGFlow — Open-source RAG engine with built-in document processing and chunking.