Retrieval Pipeline Integration¶

This project uses the existing retrieval pipeline service instead of directly embedding FAISS or BM25 libraries.

Architecture¶

┌─────────────────────────────────┐
│   User Memory RAG Agent          │
│                                  │
│  - Chunks conversations          │
│  - Prepares documents            │
│  - Manages local chunk storage   │
└────────────┬────────────────────┘
             │
             │ HTTP API
             ▼
┌─────────────────────────────────┐
│   Retrieval Pipeline Service     │
│   (Port 4242)                    │
│                                  │
│  - Dense indexing (FAISS)        │
│  - Sparse indexing (BM25)        │
│  - Hybrid search                 │
│  - Reranking                     │
└─────────────────────────────────┘

Setup¶

1. Start the Retrieval Pipeline¶

The retrieval pipeline must be running before using this system:

cd projects/week3/retrieval-pipeline
python api_server.py

This will start the retrieval pipeline service on http://localhost:4242

2. Install Dependencies¶

This project no longer requires FAISS or BM25 directly:

pip install -r requirements.txt

Required packages: - openai - For LLM interactions - requests - For communicating with retrieval pipeline - pyyaml - For loading test cases - rich - For terminal UI - python-dotenv - For environment variables

3. Configure Environment¶

Create a .env file with your API keys:

# LLM Provider (at least one required)
KIMI_API_KEY=your_kimi_api_key
OPENAI_API_KEY=your_openai_api_key  # Optional, for other providers

# Configuration
LLM_PROVIDER=kimi
INDEX_MODE=hybrid

How It Works¶

Document Indexing¶

Documents are sent to the retrieval pipeline in this format:

{
    "text": "Document content to index",
    "metadata": {
        "doc_id": "unique_identifier",
        "test_id": "test_case_id",
        "conversation_id": "conv_123",
        # ... other metadata
    }
}

The retrieval pipeline: 1. Generates embeddings for dense search 2. Builds BM25 index for sparse search 3. Returns a generated doc_id which we map to our chunk IDs

Search Process¶

Query Submission: Sends search query to retrieval pipeline
Retrieval: Pipeline performs dense/sparse/hybrid search
ID Resolution: Maps returned doc_ids back to our chunk IDs
Result Construction: Builds SearchResult objects with local chunks

API Endpoints Used¶

GET /health - Check if service is available
POST /clear - Clear existing index
POST /index - Index a single document
POST /search - Search indexed documents

Testing¶

Quick Test¶

Run the pipeline integration test:

python test_pipeline.py

This verifies: - Retrieval pipeline connectivity - Document indexing - Search functionality

Startup Test¶

Test system initialization:

python test_startup.py

Full Demo¶

Run the interactive demo:

python main.py --mode demo

Or use the interactive interface:

python main.py

Troubleshooting¶

"Retrieval pipeline not available"¶

Solution: Start the retrieval pipeline service:

cd projects/week3/retrieval-pipeline
python api_server.py

"422 Unprocessable Entity" errors¶

Cause: Document format mismatch Solution: Ensure documents have text field at root level, not in a documents array

"Chunk not found in local storage"¶

Cause: Doc ID mapping issue Solution: The system now handles this automatically by: - Storing doc_id mappings during indexing - Checking metadata in search results - Using fallback to mapped IDs

Key Changes from Direct FAISS/BM25¶

No Direct Index Management: The retrieval pipeline handles all indexing
HTTP Communication: All operations go through REST API
Doc ID Mapping: We maintain mapping between our chunk IDs and pipeline's generated IDs
Simplified Dependencies: No need for faiss-cpu, rank-bm25, or nltk
Service Dependency: Requires retrieval pipeline to be running

Performance Considerations¶

Latency: HTTP overhead adds ~10-50ms per operation
Batch Operations: Documents are indexed one at a time (pipeline limitation)
Caching: Local chunk storage reduces retrieval overhead
Scalability: Retrieval pipeline can be scaled independently

Future Enhancements¶

Batch Indexing: Add batch endpoint to retrieval pipeline
Persistent Mapping: Save doc_id mappings to disk
Connection Pooling: Reuse HTTP connections
Retry Logic: Add exponential backoff for failures
Async Operations: Use async HTTP client for better performance