跳转至

Multimodal Agent with Multiple Extraction Techniques

An educational agent framework that compares different multimodal content extraction techniques across multiple AI providers (Gemini, OpenAI, Doubao).

Features

Three Extraction Modes

  1. Native Multimodality: Direct processing using model's built-in multimodal capabilities
  2. Gemini 2.5 Pro: Documents (PDF), Images, Audio
  3. GPT-5/GPT-4o: Images (OpenAI multimodal format)

  4. Doubao 1.6: Images (OpenAI multimodal format)

  5. Extract to Text: Convert multimodal content to text first, then process

  6. PDF: OCR using Gemini or GPT-5
  7. Image: Generate descriptions using GPT-5 or Doubao 1.6

  8. Audio: Transcription using Whisper API or Gemini

  9. Multimodal Analysis Tools: Additive functionality for follow-up questions

  10. Image analysis tool (GPT-5/Doubao 1.6)
  11. Audio analysis tool (Gemini 2.5 Pro)
  12. PDF analysis tool (Gemini 2.5 Pro)

Architecture

The agent follows a modular architecture with clear separation of concerns:

MultimodalAgent
├── Configuration (config.py)
│   ├── Model configurations
│   ├── API key management
│   └── Extraction mode settings
├── Agent Core (agent.py)
│   ├── Message handling (OpenAI format)
│   ├── Conversation history
│   ├── Mode-specific processing
│   └── Streaming responses
└── Multimodal Tools
    ├── Image analysis
    ├── Audio analysis
    └── PDF analysis

Installation

  1. Clone the repository and navigate to the project directory: 

    cd chapter3/multimodal-agent

  2. Install dependencies: 

    pip install -r requirements.txt

  3. Configure API keys: 

    cp env.example .env
# Edit .env with your API keys

  4. Load environment variables: 

    export $(cat .env | xargs)

Quick Offline Start (No API Key)

Generate a bundled multimodal sample — a chart-bearing report — so you can run Experiment 3-7 end to end. The exact quarterly figures live only in the chart's bars, not in the surrounding text, which is what makes the three-paradigm trade-off measurable.

# Offline: creates test_files/sample_chart.png and test_files/sample_report.pdf
python create_sample.py           # or: python demo.py --generate-sample

Then compare the three extraction paradigms on the same file + question (requires a vision API key such as OpenAI or Gemini):

python demo.py \
  --file test_files/sample_chart.png \
  --query "Which quarter had the highest revenue, and what was the exact value?" \
  --model gpt-5.6-luna

All CLIs expose a Chinese --help (python demo.py --help, python main.py --help, python create_sample.py --help).

Usage

Interactive Chat Mode

Start an interactive session with the agent:

python main.py --interactive

Available commands in interactive mode: - /file <path> - Load a multimodal file - /mode <native|extract_to_text> - Switch extraction mode - /model <model_name> - Switch model - /tools <on|off> - Enable/disable multimodal tools - /history - Show conversation history - /clear - Clear conversation history - /quit - Exit

Process Single File

Process a file with a specific query:

# Native mode (default)
python main.py --file document.pdf --query "What is the main topic?"

# Extract to text mode
python main.py --mode extract_to_text --file image.jpg --query "Describe this image"

# With multimodal tools enabled
python main.py --tools --mode extract_to_text --file audio.mp3 --query "What's the content?"

Programmatic Usage

import asyncio
from agent import MultimodalAgent, MultimodalContent
from config import ExtractionMode

async def example():
    # Initialize agent
    agent = MultimodalAgent(
        model="gemini-3.5-flash",
        mode=ExtractionMode.NATIVE,
        enable_tools=True
    )

    # Process a PDF
    content = MultimodalContent(
        type="pdf",
        path="document.pdf"
    )

    result = await agent.process_multimodal_content(
        content,
        "Summarize this document"
    )
    print(result)

    # Chat with streaming
    async for chunk in agent.chat("Tell me more about the key points", stream=True):
        print(chunk, end="", flush=True)

asyncio.run(example())

Comparing Extraction Techniques

Demo Script

Run the comprehensive comparison demo:

# Compare extraction modes for a specific file (flags form)
python demo.py --file document.pdf --query "What are the key findings?" --model gpt-5.6-luna

# Backward-compatible positional form still works
python demo.py document.pdf "What are the key findings?"

# Save the full transcript, and skip the cross-model pass
python demo.py --file test_files/sample_chart.png \
  --query "Which quarter had the highest revenue?" \
  --model gpt-5.6-luna --skip-model-comparison --output result.txt

# This will run:
# 1. Native multimodal mode
# 2. Extract to text mode
# 3. Extract to text with tools
# 4. Comparison across different models (unless --skip-model-comparison)

Demo CLI flags:

Flag Description
--file / positional file Multimodal file to process (image / PDF / audio)
--query / positional query Question to ask about the file
--model Model for native/extract modes (default: gemini-3.5-flash)
--skip-model-comparison Only run the three-paradigm comparison
--generate-sample Offline: create the bundled chart sample, then exit
--output, -o Also write the full transcript to a file

Mode Comparison

Mode Advantages Disadvantages Best For
Native - Preserves full context
- Better visual understanding
- Direct processing		 - Limited model support
- Higher token usage		 Complex documents with mixed content
Extract to Text - Works with all models
- Lower token usage
- Can cache extractions		 - Loses visual context
- Two-step process		 Text-heavy documents, cost optimization
With Tools - Best of both worlds
- Follow-up questions
- Selective deep analysis		 - More complex setup
- Multiple API calls		 Interactive sessions, detailed Q&A

Supported File Types

Documents

  • PDF files (up to 1000 pages)
  • Best with Gemini 2.5 Pro native mode

Images

  • JPEG, PNG, GIF, BMP, WebP
  • Supported by all models

Audio

  • MP3, WAV, M4A, FLAC, AAC, OGG
  • Transcription via Whisper or Gemini

Model Capabilities

Model Native PDF Native Image Native Audio Extract to Text Tools Support
Gemini 2.5 Pro ✅ ✅ ✅ ✅ ✅
GPT-5/GPT-4o ❌ ✅ ❌ ✅ ✅
Doubao 1.6 ❌ ✅ ❌ ✅ ✅

API Configuration

Required API Keys

  1. Google Gemini: Required for PDF/Audio native processing
  2. Get key at: https://makersuite.google.com/app/apikey

  3. Set: GOOGLE_API_KEY (or GEMINI_API_KEY — both are read)

  4. OpenAI: Required for GPT models and Whisper

  5. Get key at: https://platform.openai.com/api-keys

  6. Set: OPENAI_API_KEY

  7. Doubao: Required for Doubao model

  8. Get key at: https://console.volcengine.com/
  9. Set: DOUBAO_API_KEY (or ARK_API_KEY — both are read)

File Size Limits

  • PDF: 20MB
  • Images: 20MB
  • Audio: 25MB

Testing

Run the test suite:

python test_multimodal.py

Examples

Example 1: Analyzing a Research Paper

# Native mode for best understanding
agent = MultimodalAgent(
    model="gemini-3.5-flash",
    mode=ExtractionMode.NATIVE
)

content = MultimodalContent(type="pdf", path="research_paper.pdf")
summary = await agent.process_multimodal_content(
    content,
    "What are the main contributions and findings?"
)

Example 2: Processing Images with Follow-up

# Extract to text with tools for follow-up questions
agent = MultimodalAgent(
    model="gpt-5.6-luna",
    mode=ExtractionMode.EXTRACT_TO_TEXT,
    enable_tools=True
)

# Initial processing
await agent.chat("I have an image at photo.jpg", MultimodalContent(type="image", path="photo.jpg"))

# Follow-up using tools
await agent.chat("What objects are in the background?")
await agent.chat("What's the color scheme?")

Example 3: Audio Transcription and Analysis

# Using Whisper for transcription
agent = MultimodalAgent(
    model="gpt-5.6-luna",
    mode=ExtractionMode.EXTRACT_TO_TEXT
)

content = MultimodalContent(type="audio", path="interview.mp3")
transcript = await agent._extract_audio_to_text(content)
print(f"Transcript: {transcript}")

# Analyze the transcript
analysis = await agent._answer_with_context(
    transcript,
    "What are the key points discussed?"
)

Best Practices

  1. Mode Selection:
  2. Use Native mode when visual/audio understanding is crucial
  3. Use Extract to Text for cost optimization and caching

  4. Enable tools for interactive sessions with follow-ups

  5. Model Selection:

  6. Gemini 2.5 Pro: Best for PDFs and audio
  7. GPT-4o/GPT-5: Best for complex image understanding

  8. Doubao 1.6: Alternative for image processing

  9. Performance Optimization:

  10. Cache extracted text for repeated queries
  11. Use streaming for better user experience

  12. Process files under size limits

  13. Error Handling:

  14. Always validate file existence and type
  15. Check API key configuration before processing
  16. Handle rate limits and API errors gracefully

Troubleshooting

Common Issues

  1. API Key Errors:
  2. Ensure all required API keys are set in .env

  3. Check API key validity and quota

  4. File Processing Errors:

  5. Verify file format is supported
  6. Check file size is within limits

  7. Ensure file path is correct

  8. Model Compatibility:

  9. Not all models support all content types natively
  10. Use Extract to Text mode for unsupported combinations

Architecture Details

Message Format

The agent uses OpenAI-compatible message format:

{
    "role": "user" | "assistant" | "system" | "tool",
    "content": "message text" | [{"type": "text", "text": "..."}, ...],
    "tool_calls": [...],  # Optional
    "tool_call_id": "...",  # For tool responses
}

Tool Calling Format

Tools follow OpenAI function calling specification:

{
    "type": "function",
    "function": {
        "name": "analyze_image",
        "description": "...",
        "parameters": {
            "type": "object",
            "properties": {...},
            "required": [...]
        }
    }
}

Streaming Implementation

The agent supports streaming responses for better UX: - Gemini: Native streaming API - OpenAI/Doubao: Stream via chat completions API - Tool results are streamed as they complete

Contributing

This is an educational project demonstrating multimodal AI capabilities. Contributions should focus on: - Adding new extraction techniques - Improving model comparisons - Enhancing documentation - Adding more test cases

License

MIT License - See LICENSE file for details

OpenRouter 通用回退 / Universal OpenRouter fallback

This experiment now supports a universal OpenRouter fallback for its chat LLM.

  • If the primary provider key (e.g. MOONSHOT_API_KEY / KIMI_API_KEY / OPENAI_API_KEY / DOUBAO_API_KEY …) is present, behavior is unchanged.
  • Else if OPENROUTER_API_KEY is set, the chat LLM is automatically routed through OpenRouter (https://openrouter.ai/api/v1). Model names are mapped automatically: gpt-*/o1-* → openai/…, claude-* → anthropic/claude-opus-4.8, ids already containing / are kept as-is, and other provider-native ids (e.g. kimi-k3, doubao-*) fall back to openai/gpt-5.6-luna. Set OPENROUTER_MODEL to force a specific OpenRouter model id.
  • Else a clear error lists the accepted keys.

Add OPENROUTER_API_KEY=... to your .env (see env.example) to enable it.

Note: image analysis and text chat route through OpenRouter (vision-capable default openai/gpt-5.6-luna). Audio transcription (Whisper) and native-PDF extraction still require a direct OpenAI/Gemini key.