跳转至

Structured Index Implementation Guide

Overview

This project implements two advanced document indexing approaches for handling large technical documentation:

  1. RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval)
  2. GraphRAG (Graph-based Retrieval Augmented Generation)

Both approaches are designed to handle complex technical documentation like the Intel® 64 and IA-32 Architectures Software Developer's Manual (5000+ pages).

Architecture

RAPTOR Tree-Based Index

RAPTOR creates a hierarchical tree structure through recursive summarization:

Document
    ↓
[Chunks] → [Embeddings] → [Clusters]
    ↓           ↓              ↓
Level 0: Leaf nodes (original chunks with summaries)
    ↓
Level 1: Parent nodes (cluster summaries)
    ↓
Level 2: Higher-level summaries
    ↓
Root: Top-level abstraction

Key Features: - Multi-level abstraction hierarchy - Gaussian Mixture Model clustering - Recursive summarization at each level - Cross-level search capability

GraphRAG Knowledge Graph

GraphRAG builds a knowledge graph with entities and relationships:

Document
    ↓
[Chunks] → [Entity Extraction] → [Relationship Discovery]
    ↓              ↓                        ↓
Entities ←→ Relationships → Knowledge Graph
    ↓
[Community Detection]
    ↓
Community Summaries
    ↓
Hierarchical Communities

Key Features: - LLM-based entity and relationship extraction - Community detection (Leiden/Louvain algorithms) - Hierarchical community summarization - Graph-based search across entities and communities

Implementation Details

Core Components

  1. Document Processor (document_processor.py)
  2. Handles multiple file formats (PDF, TXT, MD, HTML)
  3. Optimized for technical documentation
  4. Special handling for Intel manual format

  5. Table extraction and formatting

  6. RAPTOR Indexer (raptor_indexer.py)

  7. Text chunking with configurable overlap
  8. Embedding generation using sentence transformers
  9. GMM clustering for node grouping
  10. Recursive summarization using LLMs

  11. Tree persistence and loading

  12. GraphRAG Indexer (graphrag_indexer.py)

  13. Entity extraction focused on technical concepts
  14. Relationship discovery between entities
  15. NetworkX graph construction
  16. Community detection and summarization

  17. Graph persistence and querying

  18. API Service (api_service.py)

  19. RESTful API using FastAPI
  20. Asynchronous processing for large documents
  21. Support for file uploads
  22. Unified interface for both indexing approaches
  23. Real-time status and statistics

Processing Pipeline

Building Indexes

  1. Document Processing 

    processor = DocumentProcessor()
text = await processor.process_file(Path("intel_manual.pdf"))

  2. RAPTOR Indexing 

    raptor = RaptorIndexer(config)
raptor.build_index(text)  # Creates tree structure
raptor.save_index()       # Persists to disk

  3. GraphRAG Indexing 

    graphrag = GraphRAGIndexer(config)
graphrag.build_knowledge_graph(text)  # Extract entities
graphrag.detect_communities()          # Find communities
graphrag.hierarchical_summarization()  # Create hierarchies
graphrag.save_index()                  # Persist graph

Querying

  1. RAPTOR Search
  2. Creates query embedding
  3. Searches across all tree levels
  4. Returns nodes with different abstraction levels

  5. Includes level-specific summaries

  6. GraphRAG Search

  7. Supports entity, community, or hybrid search
  8. Returns entities with relationships
  9. Includes community summaries
  10. Provides graph context

API Endpoints

Endpoint Method Purpose
/build POST Build index from text/file
/upload POST Upload and index document
/query POST Query indexes
/status GET Check index status
/statistics GET Get index statistics
/indexes DELETE Clear indexes

Integration with Agentic RAG

The structured indexes integrate seamlessly with the Agentic RAG system:

  1. Configuration (agentic-rag/config.py) 

    KnowledgeBaseType.RAPTOR   # Tree-based backend
KnowledgeBaseType.GRAPHRAG # Graph-based backend

  2. Tool Integration (agentic-rag/tools.py)

  3. _search_raptor(): Queries RAPTOR API
  4. _search_graphrag(): Queries GraphRAG API

  5. Unified search interface for agents

  6. Agent Usage 

    config.knowledge_base.type = KnowledgeBaseType.RAPTOR
agent = AgenticRAG(config)
response = agent.query("What are x86 registers?")

Usage Examples

Command Line Interface

# Build both indexes
python main.py build intel_manual.pdf --type both

# Query RAPTOR
python main.py query "MOV instruction syntax" --type raptor

# Query GraphRAG
python main.py query "CPU register relationships" --type graphrag

# Start API server
python main.py serve

Python API

from config import get_raptor_config, get_graphrag_config
from raptor_indexer import RaptorIndexer
from graphrag_indexer import GraphRAGIndexer

# RAPTOR Example
raptor_config = get_raptor_config()
raptor = RaptorIndexer(raptor_config)
raptor.build_index(document_text)
results = raptor.search("SSE instructions", top_k=5)

# GraphRAG Example
graphrag_config = get_graphrag_config()
graphrag = GraphRAGIndexer(graphrag_config)
graphrag.build_knowledge_graph(document_text)
results = graphrag.search("instruction relationships", top_k=5)

HTTP API

# Build index
curl -X POST http://localhost:4242/build \
  -H "Content-Type: application/json" \
  -d '{"file_path": "intel_manual.pdf", "index_type": "both"}'

# Query
curl -X POST http://localhost:4242/query \
  -H "Content-Type: application/json" \
  -d '{"query": "vector instructions", "index_type": "hybrid"}'

Performance Considerations

RAPTOR

  • Indexing Time: O(n log n) for clustering
  • Memory: Stores embeddings for all nodes
  • Query Time: Fast similarity search
  • Best For: Hierarchical information, long documents

GraphRAG

  • Indexing Time: O(n²) for relationship extraction
  • Memory: Graph structure can be large
  • Query Time: Graph traversal overhead
  • Best For: Complex relationships, entity-centric queries

Configuration Options

RAPTOR Settings

chunk_size: 1000          # Words per chunk
chunk_overlap: 200        # Overlap between chunks
tree_depth: 3            # Maximum tree levels
summarization_length: 200 # Summary word count

GraphRAG Settings

chunk_size: 1200              # Words per chunk
max_knowledge_triples: 10     # Triples per chunk
community_detection: "leiden" # Algorithm choice
summarization_model: "gpt-5.6-luna"

Extending the System

Adding New Document Types

  1. Extend DocumentProcessor with new format handlers
  2. Add format-specific extraction logic
  3. Update supported_formats dictionary

Custom Entity Extraction

  1. Modify prompt in extract_entities_relationships()
  2. Add domain-specific entity types
  3. Customize relationship types

Alternative Clustering

  1. Replace GMM in RAPTOR with other algorithms
  2. Implement custom similarity metrics
  3. Add dimensionality reduction options

Graph Algorithms

  1. Add new community detection algorithms
  2. Implement graph embedding techniques
  3. Add path-finding for relationship queries

Troubleshooting

Common Issues

  1. Out of Memory
  2. Reduce chunk_size
  3. Process documents in sections

  4. Use smaller embedding models

  5. Slow Indexing

  6. Use faster/smaller LLMs
  7. Reduce tree_depth or max_triples

  8. Enable caching

  9. Poor Search Results

  10. Adjust chunk_size and overlap
  11. Fine-tune clustering parameters

  12. Improve entity extraction prompts

  13. API Errors

  14. Check API keys in .env
  15. Monitor rate limits
  16. Verify index exists before querying

Future Enhancements

  1. Hybrid Indexing: Combine RAPTOR and GraphRAG
  2. Incremental Updates: Add documents without rebuilding
  3. Multi-modal Support: Handle images and tables
  4. Cross-lingual: Support multiple languages
  5. Active Learning: Improve extraction with feedback
  6. Distributed Processing: Scale to larger documents
  7. Query Optimization: Cache frequent queries
  8. Visualization: Interactive graph/tree exploration

References