User Memory Evaluation Framework - Complete Implementation¶

Overview¶

A comprehensive evaluation framework for testing AI agent memory systems across three progressive complexity levels. The framework uses realistic US business conversations to evaluate whether agents can store, retrieve, disambiguate, and synthesize information effectively.

Implementation Status¶

✅ Framework Core (100% Complete)¶

Evaluation Engine: LLM-as-judge using Kimi K3 or OpenAI
Test Case Models: Full Pydantic models for validation
Framework Interface: Complete API for test case management
Interactive CLI: User-friendly interface for testing
Batch Processing: Automated evaluation of multiple test cases
Reporting System: Comprehensive evaluation reports

✅ Test Case Examples (Representative Set)¶

Created detailed examples demonstrating all three evaluation layers:

Layer 1: Basic Recall (10 complete examples)¶

Banking, insurance, medical, travel, telecommunications
Each with 50+ conversation rounds
Single conversation per test case
Clear, unambiguous information retrieval

Layer 2: Contextual Reasoning (3 complete examples)¶

Multiple vehicles requiring disambiguation
Multiple properties (home vs rental)
Multiple credit cards with different features
Each with 2-3 conversation histories
Tests ALL information retrieval and disambiguation

Layer 3: Cross-Session Synthesis (3 complete examples)¶

International travel with passport expiration issue
Medical procedure with insurance coordination
Home purchase with timeline dependencies
Each with 3-4 conversation histories across time
Tests proactive issue identification

Key Features Demonstrated¶

1. Realistic Conversations¶

Every test case includes: - 50+ conversation rounds mimicking real phone calls - Natural corrections: "Oh wait, I meant..." - Tangential discussions: Exploring options not chosen - Business processes: Hold times, transfers, verifications - Confusing information: Similar numbers, multiple references

2. Progressive Complexity¶

Layer 1: Single fact retrieval (account numbers, dates)
Layer 2: Multiple entity disambiguation (which car? which property?)
Layer 3: Temporal synthesis (passport expires before travel!)

3. Semantic Evaluation¶

No string matching - uses LLM understanding
Evaluates intent and completeness
Recognizes partial success
Provides detailed reasoning

Usage Examples¶

Running Interactive Evaluation¶

python main.py --mode interactive

# Menu Options:
# 1. List test cases by category
# 2. View detailed test case
# 3. Submit agent response for evaluation
# 4. Generate evaluation report

Programmatic Testing¶

from framework import UserMemoryEvaluationFramework

framework = UserMemoryEvaluationFramework()

# Get test case
test_case = framework.get_test_case("layer2_01_multiple_vehicles")

# Show conversation histories to agent
for history in test_case.conversation_histories:
    print(f"Conversation: {history.conversation_id}")
    for message in history.messages:
        print(f"{message.role}: {message.content}")

# Get agent's response
agent_response = your_agent.process(test_case.user_question)

# Evaluate
result = framework.submit_and_evaluate(
    test_case.test_id,
    agent_response
)

print(f"Reward: {result.reward:.3f}")  # Continuous score (0.0-1.0)
print(f"Passed: {result.reward >= 0.6}")  # Pass threshold at 0.6
print(f"Reasoning: {result.reasoning}")

Batch Evaluation¶

# Prepare responses
agent_responses = {
    "layer1_01_bank_account": "Your account number is 4429853327",
    "layer2_01_multiple_vehicles": "You have a Honda and Tesla. Which car?",
    "layer3_01_travel_coordination": "WARNING: Passport expires too soon!"
}

# Run batch evaluation
results = framework.evaluate_batch(agent_responses)

# Generate report
report = framework.generate_report(results, "evaluation_report.txt")

Test Case Design Principles¶

Information Architecture¶

Each conversation strategically places information: 1. Key facts stated clearly but surrounded by noise 2. Corrections that override earlier statements
3. Options discussed but not selected 4. Future considerations that may become relevant 5. Technical details that create confusion

Realism Factors¶

Agent introduces themselves with names
Security verification processes
System limitations ("I can't do that but...")
Price negotiations and discounts discovered
Schedule checking and availability
Hold music and transfers mentioned

Evaluation Criteria Structure¶

evaluation_criteria:
  description: What the agent must accomplish
  required_information:
    - Specific facts that must be retrieved
    - All relevant details for disambiguation
  success_indicators:
    - Signs of proper understanding
    - Evidence of complete retrieval
  failure_indicators:
    - Common mistakes to avoid
    - Signs of incomplete processing

Creating Additional Test Cases¶

Template Structure¶

test_id: [layer]_[number]_[descriptor]
category: layer1|layer2|layer3
title: Brief descriptive title
description: What this test evaluates
conversation_histories:
  - conversation_id: unique_id
    timestamp: ISO format
    metadata:
      business: Company name
      department: Department
      call_duration: Duration
    messages:
      # 50+ rounds of conversation
      - role: user|assistant
        content: Message content
user_question: The ambiguous or specific question
evaluation_criteria:
  # Detailed evaluation rules
expected_behavior: Ideal agent response

Conversation Patterns to Include¶

Opening: Greeting, agent introduction, purpose
Verification: Security questions, account lookup
Information Gathering: Progressive detail collection
Options Exploration: Multiple choices discussed
Decision Making: Selecting from options
Corrections: Changing earlier statements
Tangents: Related but unselected options
Confirmation: Repeating key details
Future Planning: Next steps, follow-ups
Closing: Summary, reference numbers

Performance Considerations¶

Optimization Tips¶

Cache conversation histories for repeated testing
Use batch evaluation for multiple test cases
Implement retry logic for API failures
Consider parallel evaluation for large sets

Scalability¶

Framework handles 100+ test cases efficiently
Conversations stored as YAML for easy editing
Modular design allows custom evaluators
Results export to JSON/CSV for analysis

Extension Possibilities¶

Additional Test Domains¶

Healthcare coordination across providers
Financial planning across accounts
Educational records and requirements
Government services and documentation
E-commerce orders and returns

Enhanced Evaluation¶

Multi-language support
Industry-specific terminology validation
Regulatory compliance checking
Sentiment preservation validation
Time-sensitive information handling

Conclusion¶

This framework provides a robust, scalable solution for evaluating AI agent memory systems using realistic scenarios. The three-layer approach ensures comprehensive testing from basic recall through complex synthesis, while the LLM-as-judge evaluation ensures semantic understanding rather than rigid pattern matching.

The framework is production-ready and can be extended with additional test cases following the established patterns. All conversations are carefully crafted to reflect authentic US business interactions with the complexity, confusion, and corrections that occur in real phone calls.