Keyword search fails when AI agents need to recall context. Vector search finds what matters even when the words do not match. Here's how I use it.
My AI agents used to forget everything between sessions. Every conversation started from zero. The agent that helped draft a client proposal on Monday had no idea the proposal existed by Wednesday.
The fix was memory. But not the kind you might think.
I tried keyword-based memory first. Store facts as text, search by matching words. It worked until it did not. An agent looking for “client budget concerns” would not find a memory stored as “the prospect said the pricing was too high.” Same concept. Different words. Zero results.
This is the problem vector search solves.
Traditional keyword search matches exact words. You search for “budget” and you find documents containing the word “budget.” If the document says “pricing” instead, you miss it.
Vector search works differently. It converts text into numerical representations called embeddings. These embeddings capture meaning, not just words. “Budget concerns” and “pricing was too high” end up near each other in the embedding space because they mean similar things.
When an agent searches memory, it converts the query into an embedding and finds the stored memories with the closest embeddings. The words do not need to match. The meaning does.
For agent memory, this is the difference between useful recall and a glorified text file.
The exact architecture, memory layers, and delegation patterns I use to run 50 agents across two businesses.
Get the AI Agent Blueprint →My memory system uses PostgreSQL with the pgvector extension. Here is why this combination works well.
PostgreSQL is already there. I use Postgres for everything in Ayla OS: task logs, agent state, configuration, scheduling. Adding vector search to an existing database is simpler than running a separate vector database.
pgvector is lightweight. It adds vector data types and similarity search to Postgres without requiring a new service, a new deployment, or a new set of credentials. One extension, one command to enable it.
The embedding model runs locally. I use nomic-embed-text for generating embeddings. It runs on my server, costs nothing per query, and produces embeddings that are good enough for memory retrieval. No API calls, no per-token billing, no rate limits.
The flow is straightforward. When an agent produces a result or learns something worth remembering, the memory is converted to an embedding and stored in Postgres. When an agent needs to recall context, it converts the query to an embedding and searches for the nearest matches.
Not every memory layer needs vector search. My system has six layers, and vector search powers the three where it matters most.
Layer 3: Agent memories. These are facts, decisions, and context that agents store during their work. “Client X prefers informal communication.” “The content calendar runs on a Monday/Wednesday/Friday cadence.” “Last deployment broke the staging environment.” Vector search finds these even when the query uses different phrasing than the original memory.
Layer 5: Document index. I have hundreds of internal documents: project specs, meeting notes, process documentation. Rather than expecting agents to know which document contains what, the entire corpus is embedded and searchable. An agent asking about “how the cron system handles failures” finds the relevant section of the cron documentation even if it never mentions the word “failure.”
Layer 6: Deep document store. For longer documents, research papers, and reference material. The embeddings here are chunked (broken into smaller pieces before embedding) so that a search returns the relevant section, not the entire document.
The other three layers use exact matching or are always loaded in context. Vector search is overkill for a configuration file that is always present or a session state document that is read in full.
Pinecone, Weaviate, Qdrant. There are purpose-built vector databases. Why did I stick with pgvector in Postgres?
Operational simplicity. Every new database is a new thing to manage: backups, monitoring, updates, credentials, networking. I already manage Postgres. Adding pgvector is one extension, not one infrastructure component.
Joins and filters. My queries often combine vector similarity with traditional filters. “Find memories similar to X that were created in the last 7 days by Agent Y.” In Postgres, that is a single query. In a dedicated vector database, you need to manage the metadata filtering separately or accept limitations.
Good enough performance. At my scale (tens of thousands of memories, not millions), pgvector is fast. Queries return in milliseconds. If I were running a consumer product with millions of vectors and thousands of concurrent searches, a dedicated vector database might make sense. I am running an internal system for 50 agents. Postgres handles it without breaking a sweat.
The tradeoff is that pgvector lacks some advanced features of dedicated vector databases: approximate nearest neighbors at massive scale, built-in clustering, real-time indexing for high-write workloads. None of those matter for my use case.
The difference was immediate and measurable.
Recall accuracy jumped. Before vector search, agents found relevant memories about 40% of the time when using keyword search. After vector search, that number went above 85%. The gap was entirely due to vocabulary mismatch.
Agent context improved. When agents can recall relevant prior context, their outputs are better. A content agent that remembers “the last post on this topic used an operational example” produces something complementary rather than repetitive. A business development agent that remembers “this client had concerns about timeline” addresses those concerns proactively.
Fewer repeated mistakes. I store failure reports in memory. When an agent encounters a similar situation, vector search surfaces the previous failure and the resolution. Instead of making the same mistake twice, the agent avoids it.
Cross-agent knowledge sharing. When one agent learns something, all agents can find it through vector search. The research agent discovers a fact about a client, and the content agent later surfaces that fact when writing about the client’s industry. No explicit handoff required. The memory layer handles it.
If you are building agents and they have any form of memory, vector search is worth implementing early.
Step 1. Pick your embedding model. For local use, nomic-embed-text or similar open models work well. For API-based, OpenAI’s embedding models are a common choice.
Step 2. Add pgvector to your Postgres database (or use any vector-capable store you already run).
Step 3. Embed your most important memories. Start with facts and context that agents reference frequently. You do not need to embed everything on day one.
Step 4. Replace keyword searches with vector similarity searches. The API is nearly identical: instead of a WHERE clause with text matching, you use a distance function with an embedding.
Step 5. Measure recall. Run queries you know should return specific memories and check whether they do. Adjust your embedding chunking and similarity thresholds based on the results.
The blueprint covers the full memory architecture, including which layers use vector search and how the embedding pipeline works.
Memory is what separates a demo from a system. Vector search is what makes memory actually work.
Subscribe to The Signal
One issue per week. The biggest AI build of the week, the top five videos worth your time, and an operator's read on what actually matters. For builders who ship, not dreamers who scroll.
No spam. Unsubscribe anytime. Just The Signal.