Memory Systems for Conversational AI: Benchmark Report December 28, 2025 Executive Summary This benchmark evaluates 9 memory systems (with Cognee tested across 4 retrieval modes) for business assistant platforms. The goal: identify the most effective approach for building low-latency, accurate, and cost-efficient memory architecture in conversational AI systems managing business context across multiple chat sessions. Key Findings 1. Structured Memory (LLM extraction + full-context retrieval) outperforms other approaches Achieves 92.2% retrieval accuracy with 3.1% hallucination rate Handles both direct fact lookups and indirect/inferential queries No vector search overhead; all extracted facts returned within token budget 2. High accuracy alone doesn't mean production-ready Systems achieving 98%+ accuracy were excluded due to latency (4.7s) or cost (111K tokens/query) Production requires balancing accuracy, latency, cost, AND query type handling 3. Vector search struggles with indirect queries Systems using semantic similarity matching dropped 20% accuracy on inferential questions "How should we adjust spending?" doesn't embed similarly to "Q1 target: 100M" 4. Knowledge graph architectures fail for business conversations Multiple graph-based systems achieved <15% accuracy Graph extraction loses narrative context and specific details users ask about All Systems by Accuracy Note on "No Answer" (%): Exact percentages tracked for top-performing systems (Rows 1-4). Approximate ranges (~) shown for mid-tier systems based on architectural patterns. Cognee variants show higher no-answer rates due to knowledge graph extraction limitations. --- 1. The Memory Problem in Conversational AI 1.1 Why Memory Matters Large Language Models (LLMs) are stateless by design. Each API call starts fresh: the model has no inherent knowledge of previous conversations, user preferences, or business context. This creates a fundamental challenge for conversational AI assistants that need to: Remember user preferences across sessions ("I prefer concise responses") Track business facts that evolve over time ("Q1 target changed from 100M to 125M") Maintain context across multiple conversation threads Recall specific details from weeks or months ago Without memory, every conversation starts from zero. Users must repeat themselves, and the AI cannot build a coherent understanding of the business it serves. 1.2 The Multi-Context Challenge Business assistant platforms typically support multiple chat contexts: General Chat: Open-ended conversations where users discuss any topic (company updates, personal preferences, strategic planning) Focused Tasks: Goal-oriented chats for specific projects (financial analysis, document drafting, research) The problem: Information shared in General Chat should be available in Focused Tasks, and vice versa. A user discussing Q1 targets with their main assistant should have that context automatically available when working on financial projections in a separate task. 1.3 Current Industry Approaches Memory systems for conversational AI generally fall into five architectural patterns: Each approach makes different trade-offs between accuracy, latency, cost, and scalability. This benchmark empirically tests these trade-offs. --- 2. Our Benchmark This benchmark compares 9 memory systems using synthetic datasets modeled on realistic business conversations. This section covers the dataset, systems tested, evaluation metrics, and benchmark process. 2.1 Dataset Profile Dataset Design Note: The benchmark was originally designed with multiple companies to evaluate cross-company memory isolation. However, testing revealed this approach was redundant: the core challenge lies in managing memory complexity within a single context. The design was simplified to one company while maintaining evaluation rigor by increasing chat depth (96 conversations) and fact evolution (77 facts change over time). This change had no measurable impact on benchmark results, as memory system performance is driven by context size and fact mutation, not company count. Understanding Evolved Facts: The dataset includes 163 unique fact IDs, but 77 of these facts evolve over time (e.g., "Q1 target" changes from 100M to 125M in a later month). This creates 240 total fact instances. Systems must: 1. Store the current value (not the outdated one) 2. Handle contradictions when facts update 3. Return the most recent value when queried This is critical for the "Evolved Accuracy" metric: systems that don't handle updates correctly score lower. 2.2 Memory Systems Evaluated Cognee Retrieval Modes Tested: Cognee is a single knowledge graph system that supports multiple retrieval strategies. We tested four modes: Note: Cognee also supports CYPHER mode which was not tested. 2.3 Evaluation Metrics How Hallucination Was Calculated: Hallucination is measured as a percentage of total answers generated and is independent from accuracy. They measure different things: Accuracy: Does the answer contain the correct fact from memory? Hallucination: Does the answer invent facts NOT present in retrieved memory? These metrics are orthogonal: an answer can be accurate AND hallucinated (correct fact + invented details), inaccurate without hallucination (missing fact, no inventions), or accurate without hallucination (correct fact, no extras). They do not sum to 100% because they measure different failure modes, not complementary outcomes. 2.4 Benchmark Process Phase 1: Ingestion 1. Reset adapter (clear all memories) 2. Ingest 96 chat conversations chronologically 3. Each chat averages 44.9 turns (~2,826 tokens) 4. Measure: Ingestion latency, memories extracted Phase 2: Retrieval & Evaluation 1. Issue 192 evaluation questions 2. Retrieve memories for each question 3. Generate answers from retrieved context 4. Evaluate using LLM-as-judge against ground truth Phase 3: Scoring Answers classified as: Correct: Fully matches ground truth Partially Correct: Contains relevant but incomplete information Incorrect: Wrong information No Answer: System admits lack of information Hallucination: Fabricates information not in memories --- 3. Memory Implementation Patterns This section explains the fundamentally different architectural approaches tested in this benchmark. 3.1 Pattern A: Raw Context (Full Chat History) Why it works: No extraction = no information loss. Modern LLMs excel at finding specific facts in long context. Why it fails at scale: Token costs grow linearly with conversation history. This represents a realistic business scenario where: New facts are continuously added (e.g., "Hired new sales manager") Retained facts persist across months (e.g., "Company founded in 1978") Evolved facts update over time (e.g., "Q1 target changed from 100M to 125M") As conversation history grows, token costs grow linearly. This makes raw context impractical for production despite its accuracy advantage. 3.2 Pattern B: LLM Extraction + Vector Search Ingestion Phase: Retrieval Phase: Why it works: Semantic search finds relevant facts quickly. Memory merging handles contradictions by comparing new memories against existing ones. Trade-offs: 1. Indirect query weakness: Vector search struggles when query keywords don't match stored facts. "How should we adjust spending?" doesn't embed similarly to "Q1 target: 100M SEK", even though they're logically related. 2. Context quality matters: The LLM receives raw text chunks, not coherent answers. Well-structured facts are easier for the LLM to reason about than fragmented conversation snippets. 3.3 Pattern C: Structured Memory (LLM Extraction + Full Context) Ingestion Phase: Retrieval Phase: How it works: LLM extraction compresses conversations into structured facts Merge step handles contradictions and deduplication No vector search; all facts returned within token budget LLM performs inference over complete context Trade-offs: Extraction adds processing time during ingestion Fixed token budget may prioritize recent facts over older ones Requires tuning of merge strategy for optimal deduplication 3.4 Pattern D: Agent-Centric Memory Ingestion Phase: Retrieval Phase (Multi-Step): How it works: Agent autonomously decides how to search memory Can take multiple reasoning steps to find information Searches, analyzes results, re-searches if needed Composes final answer based on multi-step reasoning Trade-offs: Multi-step reasoning adds latency to response generation Agent behavior can be less predictable than direct retrieval Requires careful tuning to prevent infinite search loops External service dependency (cloud-based agent) 3.5 Pattern E: Knowledge Graph Retrieval Ingestion Phase: Retrieval Phase: Key Lesson: Graph returns entities but loses narrative context Works well for simple facts, poorly for nuanced queries Entity extraction can lose temporal and causal relationships Complex business narratives may fragment across many nodes Best suited for: User preferences and settings (discrete facts) Highly structured domains with clear relationships Hybrid approaches combined with other retrieval methods 4. Results 4.1 Performance Overview (Categorized) Results are grouped by suitability for production use. --- ✅ PRODUCTION RECOMMENDED (>89% Evolved Accuracy, <5% Hallucination) These systems meet all production requirements. --- ⚠️ HIGH ACCURACY BUT IMPRACTICAL Excellent accuracy but excluded from production recommendation due to latency or cost. --- 🔍 POTENTIAL CANDIDATES (65-85% Accuracy) Worth investigating with tuning, but below target accuracy. ❌ NOT VIABLE (<50% Accuracy or High Hallucination) Failed to meet minimum accuracy thresholds or have unacceptable hallucination rates. 4.2 Complete Ranking by Accuracy Note on "No Answer" (%): Exact percentages tracked for top-performing systems (Rows 1-4). Approximate ranges (~) shown for mid-tier systems based on architectural patterns. Cognee variants show higher no-answer rates due to knowledge graph extraction limitations. 4.3 Score Distribution Analysis Production Tier (>85% Evolved Accuracy) Failing Tier (<20% Evolved Accuracy) --- 5. Query Type Impact: The Indirect Benchmark 5.1 Why Query Type Matters Not all user queries are direct fact lookups. In production, users often ask questions that require inference: The indirect query requires the system to: 1. Understand the implicit need for "Q1 target" information 2. Retrieve relevant facts without exact keyword matches 3. Synthesize an answer that applies the retrieved context 5.2 Indirect Benchmark Methodology We ran a separate benchmark with 192 indirect questions against 3 top-performing adapters to test this hypothesis. Example Question Transformation: Direct: "What is the maximum response time for P1 bugs?" Indirect: "How quickly should our team react to critical production issues?" 5.3 Indirect Benchmark Results Critical Finding: Semantic Memory's accuracy drops 20 percentage points on indirect queries. 5.4 Why Vector Search Struggles with Indirect Queries Architecture: Semantic Memory uses vector embeddings to match queries to stored memories. The Problem: "adjust spending" and "Q1 target: 100M SEK" have low semantic similarity despite being logically related. Vector search finds semantically similar text, not logically connected facts. 5.5 Why Structured Memory Holds Steady Architecture: Structured Memory returns all extracted facts within the token budget, letting the LLM perform inference. Why It Works: No retrieval bottleneck. The LLM sees all relevant facts and can perform the inference step that indirect queries require. 5.6 Architectural Implications Recommendation: For production systems with mixed query types, prefer Structured Memory. While it uses more tokens per query (higher cost), it has no memory search latency and avoids the 20% accuracy drop on indirect queries that vector search suffers from. --- 6. Deep Analysis: Why Systems Succeed or Fail 6.1 Why Full Chat History Dominates (98.4% Accuracy) Architecture: Stores raw conversation text without extraction. At query time, passes the entire conversation history to the LLM's long-context window. Why It Works: 1. Zero Information Loss: Every word from original chats preserved 2. LLM Strength: Modern LLMs excel at "needle-in-haystack" retrieval 3. No Extraction Errors: Avoids losing critical details during summarization Why It's Impractical for Production: 111,183 tokens per query (prohibitively expensive) As conversation history grows, costs scale linearly For 1M queries/month: ~$33K in token costs (vs ~$550 for extraction-based approaches) Actual Response Example (Correct): 6.2 Why Agent-Centric Memory Achieves 98.4% (But Is Excluded) Architecture: Agent-centric memory where each context gets a dedicated agent with archival storage. Conversations are stored as passages, and the agent uses internal tools to search its memory. Why It Works: 1. Agent Autonomy: Agent decides how to search and combine information 2. Passage-Based Storage: Individual conversation turns are independently searchable 3. Multi-Step Reasoning: Agent can take up to 5 steps to find and compose answers 4. Context Awareness: Agent maintains understanding across the entire conversation archive Performance Characteristics: 4,713ms latency: Higher due to multi-step agent reasoning 52 tokens/query: Extremely efficient context usage 5.2% hallucination: Slightly higher due to agent composition Why It's Excluded from Production Recommendation: Agent-centric architectures can achieve excellent accuracy, but the latency trade-off makes them unsuitable for real-time conversational AI where users expect sub-second responses. When Agent-Centric Makes Sense: Batch processing where latency doesn't matter Complex research queries requiring multi-step reasoning Applications where 5+ second response time is acceptable 6.3 Why Structured Memory Achieves 92.2% (The Recommended Approach) Architecture: LLM-based extraction with intelligent memory merging. During ingestion, an LLM extracts structured facts from conversations and merges them with existing facts (handling updates, deduplication, and compression). At query time, all facts are returned within a 5K token budget. Why It Works Well: LLM extraction captures key business facts in structured format Intelligent merging handles contradictions (e.g., "Q1 target changed from 100M to 125M") Token budget ensures consistent retrieval performance No vector search overhead; direct fact retrieval Handles indirect queries well (+1.6% on indirect benchmark) Why It Misses 8%: Extraction sometimes loses nuanced details Aggressive merging can simplify multi-part facts Token limit forces prioritization of recent facts Why It's the Recommended Solution: Actual Response Example (Partially Correct): Future Evolution Opportunities: This architecture can be evolved to address the 8% accuracy gap: 1. Tiered Memory: Maintain a "deep memory" (unlimited) and a "fast memory" (5K token budget). Query-time retrieval pulls from fast memory; periodic reviews compress deep memory into fast memory. 2. Review Agents: Since extraction happens between sessions (not blocking user responses), quality can be improved with review agents that: Validate extracted facts against source conversations Ensure numerical details (like "60%") are preserved Merge conflicting facts intelligently 3. Adaptive Token Budget: Dynamically expand the token budget for complex queries while keeping it small for simple lookups. 6.4 Why Vector Search Achieves 89.1% (The Cost-Optimized Alternative) Architecture: LLM extraction + vector search. Extracts memories from conversations and stores them with embeddings for semantic search. Why It Works: 1. LLM Extraction: Captures key facts during ingestion 2. Vector Search: Semantic similarity enables flexible retrieval 3. Memory Merging: Handles contradictions by updating existing memories Performance Characteristics: 339ms latency: Fastest among high-accuracy systems 171 tokens/query: Very efficient 2.1% hallucination: Lowest among all top performers Critical Limitation: Indirect Query Weakness Vector search drops 20.3 percentage points on indirect queries (89.1% → 68.8%). This is the most significant accuracy drop among tested systems. When to Use Vector Search: ✅ Latency-critical applications requiring <500ms response ✅ Direct fact lookup use cases ❌ Avoid when users ask inferential questions Actual Response Example (Correct): Future Improvement Opportunities: The 20% accuracy drop on indirect queries is addressable with refinements: 1. Query Expansion via LLM Rewriting: Transform indirect queries into explicit search terms before embedding Query: "How should we adjust spending?" → Expanded: "Q1 target, burn rate, runway, budget allocation" Trade-off: Adds latency (~50-100ms) but could recover 10-15% accuracy on indirect queries 2. Hybrid Retrieval: Combine vector search with Structured Memory fallback Fast path: Direct queries use vector search (339ms) Slow path: Indirect queries detected and answered with Structured Memory Result: Recover indirect accuracy without sacrificing direct query latency 3. Improved Embedding Models: Emerging embedding models show better semantic understanding, so indirect query performance should naturally improve 6.5 Why Knowledge Graph Systems Fail (<15% Accuracy) Multiple knowledge graph-based systems (Zep, Mem0, Vertex AI Memory Bank) achieved <15% accuracy. The failure modes are consistent: Root Cause: Over-Summarization and Context Loss Knowledge graphs extract "salient facts" rather than storing raw text. The retrieved facts are too abstract: generic statements that lose the specific details needed to answer questions. Concrete Example: Why These Systems Fail for Business Conversations: 1. 92-95% "No Answer" Rate: Systems return "I do not have that information" despite ingesting all 4,314 conversation turns 2. Selective Extraction: Knowledge graphs extract "salient facts" but lose precision; specific percentages, dates, and version names don't survive 3. Architecture Mismatch: These systems are designed for user preferences and behavioral patterns, not precise business fact retrieval Where Knowledge Graphs May Excel: User preference learning and behavioral patterns Long-term context consolidation Highly structured domains with discrete, simple facts 6.6 Why Pre-Computed Summaries Have 12% Hallucination Rate One tested approach (Cognee Summaries) achieved 25% accuracy with 12% hallucination, the highest hallucination rate in the benchmark. Actual Hallucination Example: Why Hallucinations Occur: 1. Graph Context Loss: Graph structure loses narrative details 2. LLM Interpolation: When facts are missing, LLM fills gaps 3. Category Confusion: Similar-sounding roles get conflated Lesson: Pre-computed summaries are fundamentally unsuited for precise fact retrieval. They compress away the very details that users ask about. --- 7. Ingestion & Cost Analysis 7.1 Ingestion Performance Key Insight: Vector search approaches have higher ingestion time (99.5s/chat) due to per-turn extraction and memory merging. However, this processing can happen asynchronously after the chat session ends, so it doesn't impact user-facing latency. 7.2 Memory Density Analysis The "Memories per Chat" column refers to the average number of distinct memory items returned to the user for each chat upon retrieval. This indicates how much actionable information a user receives in response to a typical query, after system filtering or aggregation. 7.3 Production Cost Projection What "100K queries/month" means: 100,000 user queries across all contexts per month Across a typical business with 5-10 active users Distributed across General Chat and Focused Tasks Represents ~3,300 queries per user per month or ~110 per day Each query triggers one memory retrieval cycle Ingestion cost is a one-time per conversation amortized monthly \Infra cost assumes dedicated Vector Search node ($0.10/hr × 720 hrs = $72/mo). In practice, shared compute reduces this significantly. --- 8. Requirements Compliance Assessment 8.1 Evaluation Criteria This benchmark assesses memory systems against the following key criteria: 8.2 Results: System Compliance Legend: ✅ Meets/Exceeds | ⚠️ Marginal | ❌ Does Not Meet --- 9. Recommendation Primary Recommendation: Structured Memory Structured Memory achieves 92.2% accuracy with 3.1% hallucination—meeting all production requirements while avoiding the architectural trade-offs that limit other approaches. The benchmark revealed that high accuracy alone doesn't equal production-readiness: systems scoring 98%+ were excluded due to latency or cost constraints. Structured Memory succeeds because it handles both direct fact lookups and indirect inferential queries equally well (+1.6% on indirect benchmark), a critical capability that vector search approaches lack (-20% on indirect queries). By extracting and merging facts post-session, it adds no latency to user responses while maintaining robust contradiction handling as business facts evolve over time. Alternative: Vector Search (Cost-Optimized) For scenarios where token cost is the primary constraint: Strengths: ✅ 171 tokens/query: 8x fewer tokens than Structured Memory ✅ 2.1% hallucination: lowest overall (47% better than Structured memory) ✅ 89.1% accuracy: above 85% threshold Trade-offs: ⚠️ +339ms search latency (vector search overhead) ❌ -20.3% on indirect queries (significant risk for inferential questions) When to Use: High query volume (>3,800 queries/scenario), break-even point vs. Structured Memory Users primarily ask direct fact-lookup questions The 20% accuracy drop on indirect queries is acceptable Not Recommended for Production --- 10. Conclusion This comprehensive benchmark of 9 memory systems (12 configurations including Cognee variants) reveals stark differences in performance and critical insights about query type impact. Key Findings 1. Structured Memory is the Recommended Solution 92.2% accuracy with 3.1% hallucination Handles both direct and indirect queries well (+1.6% on indirect benchmark) No external dependencies, balanced cost Extraction happens post-session, no impact on query latency 2. High Accuracy ≠ Production Ready Agent-centric achieves 98.4% accuracy but 4.7s latency makes it impractical Full Chat History achieves 98.4% but 111K tokens/query is cost-prohibitive Production requires balancing accuracy, latency, cost, AND query type handling 3. Query Type Matters Vector search drops 20% on indirect queries despite lower token cost Vector search struggles with inferential questions Full-context approaches handle both query types 4. Knowledge Graphs Fail for Business Conversations Multiple graph-based systems achieve <15% accuracy Root cause: Over-summarization and context loss during extraction Graph architectures lose the narrative details users ask about 5. Context Presentation Format Matters How retrieved content is formatted significantly impacts LLM answer quality Raw chunks (fragmented conversation snippets) are harder for LLMs to reason about Structured facts (well-formatted, deduplicated) produce better answers Final Recommendation Deploy Structured Memory for production. It offers the best balance of accuracy, hallucination control, and robustness to different query types, with no vector search overhead. Vector search is a viable alternative only when: 1. Query volume exceeds 3,800 queries per context (break-even point due to higher ingestion costs) 2. Users primarily ask direct fact-lookup questions 3. The 20% accuracy drop on indirect queries is acceptable 4. The +339ms search latency is acceptable --- Future Outlook Why Structured Memory Is Well-Positioned: The industry trends strongly favor Structured Memory: Token costs continue to decline: LLM pricing has decreased 50-90% over the past 18 months, making Structured Memory's higher token cost less of a concern Context windows are expanding: models now support 100K+ tokens natively, and LLMs are increasingly good at reasoning over long text No retrieval bottleneck: Unlike vector search, Structured Memory doesn't suffer from semantic mismatch on indirect queries As these trends continue, Structured Memory's current trade-off (more tokens per query) becomes increasingly negligible while its accuracy advantage remains. Hybrid Exploration: This benchmark evaluated pure implementations. A hybrid approach combining vector search efficiency with Structured Memory robustness merits investigation: Using vector search for fast and frequent queries, when memory bank is large and chat context grows Falling back to Structured Memory for complex or rare queries Combining both for real-time contextual improvements Such a hybrid may deliver the best of both worlds (low latency and high accuracy) while remaining operationally simple. --- 11. Appendices Appendix A: Dataset Details Company: Synthetic small business (~10 employees, B2B industrial sector) Product: Flagship hardware product with software component Scenarios: 12 monthly executive summaries (January–December 2024) Fact Categories: Company profile, People, Time-sensitive, Operating details, Preferences Language: Swedish (testing multi-language capability) Appendix B: Question Types Direct Benchmark (192 questions): Indirect Benchmark (192 questions, 3 adapters): The indirect benchmark tests whether systems can retrieve facts when queries don't explicitly mention them. Appendix C: Test Environment Evaluation Model: gemini-flash-latest (for LLM-as-judge) Total Evaluation Tokens: 523,660 input / 21,955 output Benchmark Duration: ~4 hours total across all systems Appendix D: Tested Systems Reference