English¶

Experiment 7-16 reproduction anchor¶

This directory is the book-owned training guide. The executable GAIA/MCP environment and training integration are in bojieli/AWorld at chapter7/AWorld, verified at commit a52d61d6d483e66b22ef16970eae5bbf4f4ab2ec. The veRL backend is bojieli/verl at chapter7/verl, verified at 1593fc3a8cf894debdc3dece2a23ed739c282789.

git clone https://github.com/bojieli/AWorld.git chapter7/AWorld
git -C chapter7/AWorld checkout --detach a52d61d6d483e66b22ef16970eae5bbf4f4ab2ec
git clone https://github.com/bojieli/verl.git chapter7/verl
git -C chapter7/verl checkout --detach 1593fc3a8cf894debdc3dece2a23ed739c282789

cd chapter7/AWorld/env
bash run-local.sh
# After dataset/reward configuration, in a second shell:
cd chapter7/AWorld/train/examples/train_gaia_with_aworld_verl
bash run.sh

This is an open-ended training experiment with no manuscript baseline. The commands were inspected, not run, during this documentation audit.

# AWorld Train *A "Learning from Practice" Training Framework for Agentic AI* [![License: MIT][license-image]][license-url] [![Paper](https://img.shields.io/badge/arXiv-2508.20404-b31b1b.svg)](https://arxiv.org/abs/2508.20404)

Table of Contents¶

Introduction
Important Note on This Educational Experiment
Core Features
GAIA Environment Tool Ecosystem
Web Interaction Tools
Document Processing Tools
Multimedia Processing Tools
Intelligent Reasoning Tools
Code Execution Tools
File System Tools
Excel Processing Tools
Knowledge Retrieval Tools
Tool Statistics Summary
Core Architecture
Quick Start
Install the Training Framework
Configure the GAIA Environment
Build a Custom Agent
Prepare for Training
Start Training
Latest Optimizations
Troubleshooting
Performance Benchmarks
Advanced Topics
Citation
Community & Support

Introduction¶

AWorld Train is an open-source training framework implementing the "Learning from Practice" paradigm, specifically designed for Agentic AI. According to the AWorld paper, building high-performance Agent systems requires three core elements:

Algorithm: Learning mechanisms that enable the Agent to adapt and improve from environment interactions
Environment: Complex interaction scenarios providing rich feedback and diverse challenges
Priors: The foundational capabilities of current LLMs in areas like reasoning, mathematics, and vision

AWorld Train addresses the core bottleneck of traditional methods—inefficient experience generation—through a distributed architecture. On the GAIA benchmark, we achieved a 14.6x speedup in data collection, making large-scale reinforcement learning training feasible.

⚠️ Important Note on This Educational Experiment¶

GAIA (General AI Assistants Benchmark) is one of the most challenging benchmarks for evaluating Agent capabilities and a competitive arena for SOTA (State-of-the-Art) Agent systems. According to the paper:

Data Scarcity: The GAIA validation set contains only 165 questions, and the test set has approximately 300 questions, far fewer than the data volume typically required for RL training
Computational Resource Requirements: The Qwen3-32B-AWorld model from the paper required training on 2 clusters of 8x A100 GPUs for several days to achieve 32.23% performance, which is still far from SOTA performance (over 80%)
Task Complexity: GAIA questions involve multi-modal understanding, multi-step reasoning, tool chain calls, etc., requiring an average of 10-20 interaction rounds to complete

Therefore, this project adopts an education-friendly configuration, using Qwen3-4B-Thinking-2507 as the base model for faster training.

The goals of this project are: - ✅ Demonstrate the complete "Learning from Practice" training pipeline - ✅ Understand the Agent-Environment interaction mechanism - ✅ Practice applying RL algorithms (PPO/GRPO) in Agent training

Core Features¶

⚡ Efficient Concurrency: Distributed task execution, 14.6x data collection acceleration
🔌 Framework Agnostic: Supports mainstream RL frameworks like VeRL, OpenRLHF, AReaL, SWIFT
🛠️ Tool Ecosystem: Built-in 26 MCP servers providing 126 tool functions, covering search, browser, code execution, multi-modal processing, etc.
📊 Long Context: Supports 131K token contexts for handling complex multi-turn interactions
🎯 SOTA Performance: Qwen3-32B-AWorld achieves 32.23% pass@1 on the GAIA test set

GAIA Environment Tool Ecosystem¶

According to the paper and MCP Server implementation, AWorld provides comprehensive tool support for GAIA tasks, totaling 26 MCP servers and 126 tool functions. Below is the complete list of tools by category:

🌐 Web Interaction Tools (3 Servers, 27 Documented Tools)¶

1. Google Search Server (`googlesearch-server`)¶

search_google: Perform web searches using the Google Custom Search API
get_search_capabilities: Retrieve search service capability information

Typical Applications: Querying real-time information, fact-checking, starting point for multi-hop reasoning

2. Browser Use Server (`browser-server`)¶

browser_use: LLM-based intelligent browser automation (using the browser-use library)
get_browser_capabilities: Retrieve browser automation capabilities

Features: - Automatic handling of bot detection and CAPTCHAs - Supports form filling, file downloads, content extraction - Integrates visual understanding and memory functions

3. Playwright Server (`ms-playwright`)¶

Provides 23 fine-grained browser control tools: - Navigation: browser_navigate, browser_navigate_back - Interaction: browser_click, browser_type, browser_hover, browser_drag, browser_select_option - Forms: browser_fill_form, browser_file_upload - Debugging: browser_console_messages, browser_network_requests, browser_take_screenshot - Management: browser_close, browser_resize, browser_tabs, browser_handle_dialog - Execution: browser_evaluate, browser_press_key, browser_wait_for - Snapshots: browser_snapshot, browser_install

Comparison: browser-server offers high-level automation, while ms-playwright provides fine-grained control

📄 Document Processing Tools (5 Servers, 9 Documented Tools)¶

4. Documents CSV Server (`documents-csv-server`)¶

extract_csv_content: Extract and analyze CSV file content (supports Markdown/JSON output formats)
list_supported_formats: List supported CSV formats

5. Documents DOCX Server (`documents-docx-server`)¶

extract_docx_content: Extract Word document content (including text, tables, images)
list_supported_formats: List supported DOCX formats

6. Documents PPTX Server (`documents-pptx-server`)¶

extract_pptx_content: Extract PowerPoint content (slide text, notes, layout)
list_supported_formats: List supported PPTX formats

7. Documents PDF Server (`documents-pdf-server`)¶

convert_document_to_markdown: Convert PDF to Markdown (preserving structure and formatting)

8. Documents TXT Server (`documents-txt-server`)¶

extract_text_content: Extract plain text file content
list_supported_formats: List supported text encodings

GAIA Application Scenario: Processing attached files (70% of GAIA dataset questions include document attachments)

🎥 Multimedia Processing Tools (3 Servers, 12 Tools)¶

9. Media Audio Server (`media-audio-server`)¶

transcribe_audio: Speech-to-text (supports Whisper API)
extract_audio_metadata: Extract audio metadata (duration, bitrate, sample rate)
trim_audio: Trim audio segments
list_supported_formats: List supported audio formats (MP3, WAV, M4A, etc.)

10. Media Image Server (`media-image-server`)¶

extract_text_ocr: OCR text recognition (based on Tesseract/Cloud Vision API)
analyze_image_ai: AI image analysis (scene recognition, object detection, description generation)
get_image_metadata: Extract image metadata (dimensions, EXIF, capture time)

11. Media Video Server (`media-video-server`)¶

analyze_video: Video content analysis (scene segmentation, keyframe extraction)
summarize_video: Video summary generation
extract_keyframes: Extract keyframe images

Supplementary: Standalone Multimedia Tools¶

Audio Server (audio-server): mcp_transcribe_audio - Advanced speech transcription
Image Server (image-server): mcp_image_recognition - Image recognition and classification

GAIA Application: Approximately 40% of GAIA questions involve image, audio, or video analysis

💡 Intelligent Reasoning Tools (3 Servers, 6 Tools)¶

12. Intelligence Code Server (`intelligence-code-server`)¶

generate_python_code: Generate and validate Python code (for mathematical calculations, data processing)
get_reasoning_capabilities: Retrieve code generation capability information

13. Intelligence Think Server (`intelligence-think-server`)¶

complex_problem_reasoning: Complex problem reasoning (mathematical proofs, algorithm design, logic puzzles)
get_reasoning_capabilities: Retrieve reasoning capability information

Features: Invokes more powerful reasoning models (e.g., GPT-4o, Claude 3.7 Sonnet) for deep thinking

14. Intelligence Guard Server (`intelligence-guard-server`)¶

guarding_reasoning_process: Reasoning process protection and validation (prevents hallucinations, checks logical consistency)
get_guarding_capabilities: Retrieve guarding capability information

Paper Highlight: These "thinking tools" enable smaller models to leverage the reasoning capabilities of larger models, allowing them to "stand on the shoulders of giants"

💻 Code Execution Tools (3 Servers, 15 Documented Tools)¶

15. Terminal Server (`terminal-server`)¶

execute_command: Execute terminal commands (Python, bash, system commands)
get_command_history: Retrieve command execution history
get_terminal_capabilities: Retrieve terminal capability information

Security Features: Command whitelist, timeout control, output truncation

16. E2B Code Server (`e2b-code-server`)¶

e2b_upload_file: Upload files to the sandbox
e2b_run_code: Execute code in an isolated sandbox (supports Python, JavaScript, multiple languages)

Advantage: Completely isolated execution environment, preventing malicious code from affecting the main system

17. Terminal Controller (`terminal-controller`)¶

Provides 10 advanced terminal management tools: - execute_command, get_command_history, get_current_directory, change_directory - list_directory, write_file, read_file - insert_file_content, delete_file_content, update_file_content

Difference: terminal-server focuses on command execution, while terminal-controller provides file system management.

📂 File System Tools (1 Server, 14 Tools)¶

18. Filesystem Server (`filesystem-server`)¶

Complete file operation capabilities: - Reading: read_file, read_text_file, read_media_file, read_multiple_files - Writing: write_file, edit_file - Management: create_directory, move_file, get_file_info - Search: search_files, list_directory, list_directory_with_sizes, directory_tree - Permissions: list_allowed_directories - Lists allowed directories for access

GAIA Application: Access dataset attachments (in the /root/workspace/gaia_dataset/ directory)

📊 Excel Processing Tools (1 Server, 29 Tools)¶

19. Excel Server (`excel`)¶

Provides enterprise-grade Excel operations:

Data Operations (9): - read_data_from_excel, write_data_to_excel - insert_rows, insert_columns, delete_sheet_rows, delete_sheet_columns - copy_range, delete_range, validate_excel_range

Workbook/Sheet Management (7): - create_workbook, create_worksheet, copy_worksheet - delete_worksheet, rename_worksheet, get_workbook_metadata

Advanced Features (13): - apply_formula, validate_formula_syntax - format_range, create_chart, create_pivot_table, create_table - merge_cells, unmerge_cells, get_merged_cells - get_data_validation_info

GAIA Typical Tasks: Analyze complex Excel data tables, compute statistical metrics, generate charts

🔍 Knowledge Retrieval Tools (3 Servers, 15 Documented Tools)¶

20. Wikipedia Server (`wiki-server`)¶

search_wikipedia: Search Wikipedia entries
get_article_content: Get full article content
get_article_summary: Get article summary
get_article_categories: Get article categories
get_article_links: Get article links
get_article_history: Get article history (for time-sensitive questions)
get_wikipedia_capabilities: Get Wikipedia service capabilities

Special Features: Supports multiple languages, historical version queries (GAIA has questions like "population data for a certain month and year")

21. ArXiv Server (`parxiv-server`)¶

search_papers: Search arXiv papers
get_paper_details: Get detailed paper information (abstract, authors, citations)
download_paper: Download paper PDF
get_arxiv_capabilities: Get arXiv service capabilities
get_categories: Get arXiv category list

22. Wayback Machine Server (`wayback-server`)¶

list_archived_versions: List archived versions of a webpage
get_archived_content: Get webpage content at a specific point in time
get_wayback_capabilities: Get Wayback Machine capabilities

GAIA Application: Answer historical queries like "information on a certain website in 2015"

📥 Other Utility Tools (3 Servers, at Least 4 Documented Tools)¶

23. Download Server (`download-server`)¶

download_file: Download a network file to local storage
get_download_capabilities: Get download service capabilities

24. Read Web Server (`readweb-server`)¶

Provides webpage content reading capabilities (specific tools defined by MCP configuration)

25. Google Search Alternative (`google-search`)¶

search: Simplified search interface
read_webpage: Read webpage content

Tool Statistics Summary¶

Category	Servers	Tools	Key Capabilities
Web Interaction	3	27 documented	Search, intelligent browsing, fine-grained control
Document Processing	5	9 documented	CSV, Word, PPT, PDF, TXT
Multimedia	5	12	Audio transcription, OCR, image/video analysis
Intelligent Reasoning	3	6	Code generation, complex reasoning, verification
Code Execution	3	15 documented	Terminal commands, sandbox execution, file management
File System	1	14	Complete file operation capabilities
Excel	1	29	Enterprise-grade spreadsheet processing
Knowledge Retrieval	3	15	Wikipedia, ArXiv, historical web pages
Other	3	At least 4 documented	File download, webpage reading

Counts in this table reflect the interfaces explicitly enumerated above. The upstream MCP configuration is the source of truth for complete totals, especially for servers whose individual interfaces are not listed here.

Tool Call Examples (from Training Logs)¶

# Google Search Example
Tool call: aworld-mcp__search_google
Tool args: {"query": "Wyoming population 2020", "num_results": 5}
Result: {"success": true, "results": [{"title": "Wyoming - Census Bureau", "snippet": "576,851..."}]}

# File System Example
Tool call: aworld-mcp__list_directory
Tool args: {"path": "/root/workspace/gaia_dataset/2023/test"}
Result: ["[FILE] 021a5339-...-bd9b-9368b3efda7a.pdf", "[FILE] 03c577c9-...-f8f598de14c1.mp3", ...]

# CSV Processing Example (will error if tabulate dependency is missing)
Tool call: aworld-mcp__extract_csv_content
Tool args: {"file_path": "/root/workspace/gaia_dataset/2023/test/52e8ce1c-...-67d1648779b9.csv"}
Error: "CSV extraction failed: Missing optional dependency 'tabulate'"

# Wikipedia History Query Example
Tool call: aworld-mcp__get_article_history
Tool args: {"title": "Cat", "date": "20191231", "language": "en"}
Result: {...historical Wikipedia content...}

Core Architecture¶

AWorld Train uses a four-stage training pipeline:

Architecture

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│ Environment │───▶│    Agent    │───▶│   Adapter   │───▶│   Training  │
│   Setup     │    │Construction │    │   Layer     │    │  Framework  │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
     (MCP)           (AWorld)            (VeRL)           (PPO/GRPO)

Environment Setup: Deploy GAIA MCP Server, providing 20+ tool capabilities
Agent Construction: Implement a custom AgentLoop, define decision logic
Adapter Integration: Unify interfaces, connect to the RL training framework
Training Execution: Configure reward functions and hyperparameters, start the training task

Quick Start¶

System Requirements¶

Component	Requirement
Operating System	Linux (recommended) / macOS / Windows
Hardware	Minimum 4 CPU cores + 8GB RAM Training recommended: 8x A100/H100 GPU
Software	Docker, NVIDIA Driver, CUDA 12.1+

Install Training Framework¶

Using VeRL as an example, the installation steps are as follows:

# Run this block from any directory after setting the absolute book-repository path.
export BOOK_ROOT=/absolute/path/to/ai-agent-book

# 1. Install system dependencies (Ubuntu/Debian)
sudo apt-get update
sudo apt-get install -y build-essential git wget

# 2. Install CUDA Toolkit (matching your GPU Driver)
# Reference: https://developer.nvidia.com/cuda-downloads

# 3. Install PyTorch (matching CUDA version)
pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu121

# 4. Clone the book's verified AWorld fork to the canonical companion path
git clone https://github.com/bojieli/AWorld.git "${BOOK_ROOT}/chapter7/AWorld"
git -C "${BOOK_ROOT}/chapter7/AWorld" checkout --detach a52d61d6d483e66b22ef16970eae5bbf4f4ab2ec

# 5. Clone/install the verified VeRL backend
git clone https://github.com/bojieli/verl.git "${BOOK_ROOT}/chapter7/verl"
git -C "${BOOK_ROOT}/chapter7/verl" checkout --detach 1593fc3a8cf894debdc3dece2a23ed739c282789
pip install -e "${BOOK_ROOT}/chapter7/verl"

Important Note: Some dependencies of VeRL need to be compiled in a CUDA environment. Please ensure steps 1-2 are completed first.

Configure GAIA Environment¶

The GAIA environment is deployed via Docker and provides MCP (Model Context Protocol) services.

1. Download the Dataset¶

# Download the GAIA dataset from Hugging Face
cd ${BOOK_ROOT}/chapter7/AWorld/env/gaia-mcp-server/docker
mkdir -p gaia_dataset

# Use Hugging Face CLI to download (requires pip install huggingface_hub first)
huggingface-cli download gaia-benchmark/GAIA --repo-type dataset --local-dir gaia_dataset

2. Configure Environment Variables¶

cd ${BOOK_ROOT}/chapter7/AWorld/env/gaia-mcp-server/mcp_servers
cp .env_template .env

# Edit the .env file to configure necessary API keys
vim .env

Example .env file (partial fields):

# OpenAI API (for intelligence-code-server, etc.)
OPENAI_API_KEY=your-openai-api-key

# Google Search API (for googlesearch-server)
GOOGLE_API_KEY=your-google-api-key
GOOGLE_CSE_ID=your-search-engine-id

# E2B API (for code execution sandbox)
E2B_API_KEY=your-e2b-api-key

3. Start the MCP Server¶

cd ${BOOK_ROOT}/chapter7/AWorld/env
bash run-local.sh

Once started successfully, you will see output similar to:

Starting services...
DISPLAY=:0
2025-10-06 05:20:42,368 - __main__ - INFO - Starting MCP Server Proxy...
2025-10-06 05:20:42,373 - mcp_server_proxy.mcp_server_proxy - INFO - Added MCP server executor: googlesearch-server
...
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

The MCP Server provides two services: - MCP Interface: http://localhost:8080/mcp (Agent tool calls) - VNC Interface: http://localhost:5901/vnc.html?autoconnect=true (visual debugging)

4. Verify the Environment¶

# Set environment variables
export MCP_SERVER_URL=http://localhost:8080/mcp

# Test connection (Python)
python3 << EOF
from train.adapter.verl.common import get_agent_tool_env_and_servers

config, servers = get_agent_tool_env_and_servers()
print(f"Available servers: {len(servers)}")
print(f"Sample servers: {servers[:5]}")
EOF

Expected output:

Available servers: 20
Sample servers: ['readweb-server', 'browser-server', 'documents-csv-server', ...]

Building a Custom Agent¶

Implementing AgentLoop¶

The core of creating a custom Agent is to inherit AworldAgentLoop and implement the build_agents() method. Below is a complete example of a GAIA Agent:

# train/examples/train_gaia_with_aworld_verl/custom_agent_loop.py

from aworld.agents.llm_agent import Agent
from aworld.config import AgentConfig
from train.adapter.verl.aworld_agent_loop import AworldAgentLoop
from train.adapter.verl.common import get_agent_tool_env_and_servers

class GaiaAgentLoop(AworldAgentLoop):
    """Custom Agent Loop for GAIA tasks"""

    async def build_agents(self):
        # Get MCP environment configuration and list of available servers
        gaia_env_config, gaia_env_servers = get_agent_tool_env_and_servers()

        # Build Agent instance
        return Agent(
            conf=AgentConfig(
                # LLM server address is dynamically managed by VeRL
                llm_base_url=await self.get_llm_server_address(),
                llm_model_name=await self.get_llm_server_model_name(),
                llm_api_key="dummy",  # No real API key needed for VeRL internal communication
            ),
            name="gaia_super_agent",

            # System prompt (defines Agent role and capabilities)
            system_prompt="""You are a helpful AI assistant specialized in solving complex tasks.
You have access to various tools including web search, code execution, and file analysis.
When given a task, break it down into steps and use available tools systematically.
Always provide your final answer in <answer>...</answer> tags.""",

            # MCP tool configuration
            mcp_config=gaia_env_config,
            mcp_servers=gaia_env_servers,
        )

Configuring agent.yaml¶

Register your AgentLoop in train/examples/train_gaia_with_aworld_verl/agent.yaml:

- name: gaia_agent
  _target_: train.examples.train_gaia_with_aworld_verl.custom_agent_loop.GaiaAgentLoop

Advanced Scenarios¶

Multi-Agent System¶

from aworld.swarms.swarm import Swarm

class MultiAgentLoop(AworldAgentLoop):
    def build_agents(self):
        config, servers = get_agent_tool_env_and_servers()

        # Create specialized Agents
        researcher = Agent(
            conf=AgentConfig(...),
            name="researcher",
            system_prompt="You are a research specialist...",
            mcp_servers=["googlesearch-server", "wiki-server"]
        )

        coder = Agent(
            conf=AgentConfig(...),
            name="coder",
            system_prompt="You are a coding expert...",
            mcp_servers=["e2b-code-server", "terminal-server"]
        )

        # Build Swarm
        return Swarm(
            agents=[researcher, coder],
            coordinator=researcher  # Main coordinating Agent
        )

Preparing for Training¶

1. Prepare the Dataset¶

Run the dataset generation script to convert GAIA data into training format:

cd ${BOOK_ROOT}/chapter7/AWorld/train/examples/train_gaia_with_aworld_verl/gaia_datasets

python create_dataset.py \
  --dataset_path ${BOOK_ROOT}/chapter7/AWorld/env/gaia-mcp-server/docker/gaia_dataset \
  --output_dir ~/datasets \
  --train_size 300 \
  --test_size 100

This will generate: - ~/datasets/train.parquet: 300 training samples - ~/datasets/test.parquet: 100 test samples

Data format (Parquet):

Field	Type	Description
`prompt`	List[Dict]	Formatted chat messages `[{"role": "user", "content": "..."}]`
`data_source`	str	Data source identifier `"gaia"`
`ability`	str	Ability category `"agi"`
`reward_model`	Dict	Reward configuration `{"style": "GAIA", "ground_truth": "..."}`
`extra_info`	Dict	Additional metadata (task_id, level, etc.)
`agent_name`	str	Target Agent name

2. Configure the Reward Function¶

Define the reward logic in train/examples/train_gaia_with_aworld_verl/metrics/gaia_reward_function.py:

import re
from aworld.logs.util import logger

def gaia_reward_func(data_source, solution_str, ground_truth, extra_info=None):
    """
    Reward function for GAIA tasks

    Args:
        data_source: Data source identifier
        solution_str: Complete solution generated by the Agent
        ground_truth: Ground truth answer
        extra_info: Additional information (e.g., task_id, level)

    Returns:
        float: Reward value (0.0 or 1.0)
    """
    # Extract content from <answer>...</answer> tags in solution_str
    pattern = r'<answer>(.*?)</answer>'
    match = re.search(pattern, solution_str, re.DOTALL | re.MULTILINE)

    if not match:
        logger.warning("No answer tag found in solution")
        return 0.0

    answer = match.group(1).strip()
    logger.info(f"Extracted answer: {answer}, Ground truth: {ground_truth}")

    # Use GAIA standard scorer (supports numbers, lists, strings)
    if question_scorer(answer, ground_truth):
        return 1.0
    else:
        return 0.0

def question_scorer(model_answer: str, ground_truth: str) -> bool:
    """GAIA standard scoring logic (detailed implementation omitted)"""
    # Supports number comparison, list comparison, string normalization comparison
    # See full code for details
    ...

3. Configure the Training Script¶

Edit run.sh to configure key parameters:

#!/usr/bin/env bash
set -xeuo pipefail

# ============ Cluster Topology ============
export GPUS_PER_NODE=${GPUS_PER_NODE:-8}
export NNODES=${NNODES:-1}

# ============ Model and Data ============
model_path=${model_path:-Qwen/Qwen3-4B-Thinking-2507}
train_files=$DATA_ROOT/datasets/train.parquet
test_files=$DATA_ROOT/datasets/test.parquet

# ============ Custom Configuration ============
path_to_train="/root/AWorld/train"
agent_loop_config_path=${path_to_train}/examples/train_gaia_with_aworld_verl/agent.yaml
reward_fn_file_path=${path_to_train}/examples/train_gaia_with_aworld_verl/metrics/gaia_reward_function.py
reward_fn_name=gaia_reward_func

# ============ Training Hyperparameters ============
# PPO Algorithm Configuration
adv_estimator=grpo              # Use Group Relative Policy Optimization
clip_ratio_low=0.2              # PPO clip lower bound
clip_ratio_high=0.28            # PPO clip upper bound
actor_lr=1e-6                   # Actor learning rate

# Long Context Configuration (AWorld Latest Optimization)
max_turns=32                    # Maximum number of interaction turns (increased from 8 to 32)
max_prompt_length=4096          # Maximum prompt length (increased from 1024 to 4096)
max_response_length=4096        # Maximum response length (increased from 2048 to 4096)

# Batch Configuration
train_batch_size=32             # Training batch size (increased from 1 to 32)
ppo_mini_batch_size=8           # PPO mini-batch size (4 gradient updates)
n_resp_per_prompt=16            # Sample 16 responses per prompt (increased from 1)
n_resp_per_prompt_val=16        # Number of samples for validation

# ============ MCP Server ============
export MCP_SERVER_URL=${MCP_SERVER_URL:-http://localhost:8080/mcp}

# ============ Performance Optimization ============
export VLLM_USE_V1=1                      # Use vLLM v1 engine
export VLLM_ATTENTION_BACKEND=FLASH_ATTN  # FlashAttention-2
infer_tp=1                                # Tensor Parallel size
train_sp=1                                # Sequence Parallel size
offload=true                              # Offload parameters to CPU

# ============ VeRL Training Command ============
python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=$adv_estimator \
    data.train_files="['$train_files']" \
    data.val_files="['$test_files']" \
    data.return_raw_chat=true \
    data.train_batch_size=$train_batch_size \
    data.max_prompt_length=$max_prompt_length \
    data.max_response_length=$max_response_length \
    actor_rollout_ref.model.path="$model_path" \
    actor_rollout_ref.rollout.multi_turn.max_user_turns=$max_turns \
    actor_rollout_ref.rollout.multi_turn.max_assistant_turns=$max_turns \
    actor_rollout_ref.rollout.max_model_len=131072 \
    actor_rollout_ref.rollout.max_num_batched_tokens=131072 \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.9 \
    actor_rollout_ref.rollout.agent.agent_loop_config_path=$agent_loop_config_path \
    custom_reward_function.path="${reward_fn_file_path}" \
    custom_reward_function.name="${reward_fn_name}" \
    trainer.logger=['console','wandb'] \
    trainer.experiment_name=aworld_train_qwen3_4b \
    trainer.save_freq=5 \
    trainer.test_freq=5 \
    +trainer.num_steps=300

Launching Training¶

Single-Node Training¶

cd ${BOOK_ROOT}/chapter7/AWorld/train/examples/train_gaia_with_aworld_verl

# Launch training (8 GPUs)
export DATA_ROOT=~/datasets
export GPUS_PER_NODE=8
bash run.sh

Multi-Node Training (Slurm)¶

# Submit Slurm job (2 nodes, 8 GPUs per node)
sbatch <<EOF
#!/bin/bash
#SBATCH --job-name=aworld-train
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --time=48:00:00

export DATA_ROOT=/path/to/datasets
export NNODES=2
export GPUS_PER_NODE=8

srun bash run.sh
EOF

Training Monitoring¶

The training process supports multiple logging backends:

# 1. Console Output
# View training metrics in real-time (loss, reward, KL divergence, etc.)

# 2. WandB Visualization (Recommended)
# Visit https://wandb.ai/<your-project>/aworld_train_qwen3_4b

# 3. TensorBoard
tensorboard --logdir ~/datasets/checkpoint/aworld_train_qwen3_4b

Key Monitoring Metrics:

Metric	Description	Target Value
`reward/mean`	Average reward	Gradually increase to 0.3+
`reward/max`	Maximum reward	Reach 1.0
`actor/loss`	Actor loss	Steadily decrease
`rollout/response_length`	Response length	Adjust based on task
`rollout/num_turns`	Average number of turns	Efficient tool use (5-15 turns)

Latest Optimizations¶

Based on the latest code changes (commit a52d61d6), we have implemented the following key optimizations:

1. Long Context Support (verl_provider.py)¶

# New dynamic max_model_len parameter configuration
self.max_model_len = params.get("max_model_len", 24576)

Impact: Supports context windows of up to 131K tokens, enabling complex multi-turn dialogues and large-scale tool call histories.

2. Data Format Optimization (create_dataset.py)¶

# Prompt formatted as a list of chat messages (adapting to VeRL return_raw_chat mode)
rl_dataset["prompt"].append([{"role": "user", "content": data["Question"]}])

Impact: Seamlessly integrates with VeRL's chat template system, avoiding format conversion overhead.

3. Hyperparameter Tuning (run.sh)¶

Parameter	Old Value	New Value	Improvement
`max_turns`	8	32	4x interaction depth
`max_prompt_length`	1024	4096	4x input capacity
`max_response_length`	2048	4096	2x output capacity
`train_batch_size`	1	32	32x training efficiency
`n_resp_per_prompt`	1	16	16x sample diversity

Impact: - Deeper Reasoning: Allows the agent to perform longer tool chains and reasoning - More Efficient Training: Larger batches accelerate convergence, diverse sampling improves generalization - More Stable Optimization: ppo_mini_batch_size=8 enables 4 gradient updates, balancing training stability and efficiency

4. Memory Optimization¶

# vLLM Configuration
actor_rollout_ref.rollout.max_model_len=131072           # Model context length
actor_rollout_ref.rollout.max_num_batched_tokens=131072  # Batched token count
actor_rollout_ref.rollout.gpu_memory_utilization=0.9     # GPU memory utilization

Impact: Supports concurrent inference of 32K+ tokens on A100 80GB, fully utilizing GPU resources.

5. New Qwen3-30B-A3B Training Script¶

# run_qwen3_30b_a3b.sh
infer_tp=4   # Tensor Parallel (Inference)
train_sp=8   # Sequence Parallel (Training)

Impact: Supports training larger models, leveraging model parallelism to overcome single-GPU limitations.

Troubleshooting¶

Agent Training Process Output Examples¶

✅ Normal Inference Flow¶

(AgentLoopWorker pid=448354)   [agent] Content: Okay, let's see. So the user is asking about the population difference between the two states that have both Carl's Jr. and Hardee's fast food restaurants...
(AgentLoopWorker pid=448354)   [agent] Tool call: aworld-mcp__search_google - ID: chatcmpl-tool-94b30baa
(AgentLoopWorker pid=448354)   [agent] Tool args: {"query": "Wyoming population 2020", "num_results": 5}
(AgentLoopWorker pid=448354)   [agent] Content: ["{\"success\": true, \"message\": {\"query\": \"Wyoming population 2020\", \"results\": [{\"title\": \"Wyoming - Census Bureau Profile\", \"snippet\": \"576,851. The Total Population for Wyoming is 576,851...\"...}

Explanation: The agent correctly calls the tool and receives results; the inference chain is complete.

✅ Successful File Listing¶

(AgentLoopWorker pid=448358)   [agent] Content: ["[FILE] 021a5339-744f-42b7-bd9b-9368b3efda7a.pdf\n[FILE] 03c577c9-4227-48a9-9b75-f8f598de14c1.mp3\n[FILE] 063800f6-8832-4856-972b-17b877612533.png\n..."]
(AgentLoopWorker pid=448358)   [agent] Content: Okay, let's try to figure out how many horror titles are overdue based on the inventory file...

Explanation: The file system tool works correctly; the agent can access dataset files.

GAIA MCP Server Output Example#### ✅ Normal Startup Output¶

$ docker logs gaia-mcp-server-gaia-mcp-server-1 -f

Starting services...
DISPLAY=:0
2025-10-06 05:20:42,368 - __main__ - INFO - Starting MCP Server Proxy...
2025-10-06 05:20:42,370 - mcp_server_proxy.mcp_server_proxy - INFO - Loaded MCP tool schema: mcp_tool_schema=
  readweb-server:
  browser-server:
    - get_browser_capabilities
    - browser_use
  documents-csv-server:
    - extract_csv_content
    - list_supported_formats
  googlesearch-server:
    - search_google
    - get_search_capabilities
  ...
2025-10-06 05:20:42,373 - mcp_server_proxy.mcp_server_proxy - INFO - Added MCP server executor: googlesearch-server
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

Checkpoints: - ✅ Starting MCP Server Proxy appears - ✅ 20+ tool servers loaded (Added MCP server executor) - ✅ Uvicorn listening on port 8080

✅ Normal Request During Training¶

INFO:     208.64.254.164:36416 - "POST /mcp HTTP/1.1" 200 OK
2025-10-06 05:09:23,880 - mcp.server.lowlevel.server - INFO - Processing request of type CallToolRequest
2025-10-06 05:09:27,006 - mcp.server.lowlevel.server - INFO - Processing request of type CallToolRequest
[10/06/25 05:09:27] INFO     🔍 Searching Google for: 'Speaker of the House that passed act...'
                    INFO     ✅ Found 5 results in 0.42s

Explanation: The agent is calling the Google search tool normally, with fast request-response times (< 1s).

❌ Common Errors and Solutions¶

1. CSV Extraction Failed (Missing Dependency)¶

ERROR    CSV extraction failed: Missing optional dependency 'tabulate'.
         Use pip or conda to install tabulate.

Cause: The to_markdown() function in pandas requires the tabulate library.

Solution:

# Enter the MCP Server Docker container
docker exec -it gaia-mcp-server-gaia-mcp-server-1 bash

# Install the missing dependency
cd /app/mcp_servers/documents_server
source .venv/bin/activate
pip install tabulate

# Restart the container
exit
docker restart gaia-mcp-server-gaia-mcp-server-1

2. OpenAI API Key Error¶

WARNING  coding failed: Error code: 401 - {'error': {'message': 'Incorrect API key provided: [redacted-openrouter-key]', 'type': 'invalid_request_error', 'code': 'invalid_api_key'}}

Cause: Tools like intelligence-code-server require an OpenAI API key to generate code.

Solution:

# Method 1: Configure a real OpenAI API Key
vim ${BOOK_ROOT}/chapter7/AWorld/env/gaia-mcp-server/mcp_servers/.env
# Add: OPENAI_API_KEY=your-openai-api-key

# Method 2: Use a compatible service like OpenRouter
# Configure in .env:
LLM_BASE_URL=https://openrouter.ai/api/v1
OPENAI_API_KEY=your-openrouter-api-key

# Restart the service
cd ${BOOK_ROOT}/chapter7/AWorld/env
bash run-local.sh

3. Wikipedia API 429 Rate Limiting¶

ERROR    Wikipedia summary retrieval error:
         requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Cause: The Wikipedia API is rate-limiting (429 Too Many Requests), but the Python library does not handle it gracefully.

Solution:

# Method 1: Reduce concurrent requests (adjust training parameters)
train_batch_size=16  # Reduce from 32 to 16
n_resp_per_prompt=8  # Reduce from 16 to 8

# Method 2: Use a proxy or switch Wikipedia mirror
# Configure in wiki_server/.env:
WIKIPEDIA_BASE_URL=https://en.wikipedia.org/w/api.php

# Method 3: Add request retry logic (requires code modification)
# Add exponential backoff in wiki_server/src/wiki.py

4. Tool Execution Timeout¶

2025-10-06 05:21:08,548 - mcp_server_proxy.mcp_server_executor - INFO - Starting tool server browser-server...
[No response after 10 seconds]

Cause: The browser tool (Playwright) is slow to start or has insufficient resources.

Solution:

# Increase timeout configuration
vim ${BOOK_ROOT}/chapter7/AWorld/env/gaia-mcp-server/mcp_servers/.env
# Add:
TOOL_EXECUTION_TIMEOUT=120  # Increase from default 60s to 120s

# Warm up the browser environment
docker exec -it gaia-mcp-server-gaia-mcp-server-1 bash
cd /app/mcp_servers/browser_server
uv run python -c "from playwright.sync_api import sync_playwright; sync_playwright().start()"

5. vLLM OOM (Out of Memory)¶

RuntimeError: CUDA out of memory. Tried to allocate 20.00 GiB (GPU 0; 79.35 GiB total capacity; 75.12 GiB already allocated; 2.31 GiB free; 78.90 GiB reserved in total by PyTorch)

Solution:

# Method 1: Reduce batch size
train_batch_size=16             # Reduce from 32
ppo_mini_batch_size=4          # Reduce from 8

# Method 2: Reduce GPU memory utilization
actor_rollout_ref.rollout.gpu_memory_utilization=0.75  # Reduce from 0.9

# Method 3: Enable CPU offload
actor_rollout_ref.actor.fsdp_config.param_offload=true
actor_rollout_ref.actor.fsdp_config.optimizer_offload=true

# Method 4: Use larger Tensor Parallel
infer_tp=4  # Increase from 1 to 4 (requires 4 GPUs)

Debugging Tips¶

1. Enable Verbose Logging¶

# VeRL training logs
export RAY_LOGGING_LEVEL=DEBUG
export HYDRA_FULL_ERROR=1

# MCP Server logs
docker logs -f gaia-mcp-server-gaia-mcp-server-1 --tail 100

# vLLM inference logs
export VLLM_LOGGING_LEVEL=DEBUG

2. Step-by-Step Agent Debugging¶

# test_agent_debug.py
from train.examples.train_gaia_with_aworld_verl.custom_agent_loop import GaiaAgentLoop

# Create AgentLoop (without starting training)
loop = GaiaAgentLoop()
agent = loop.build_agents()

# Test a single question
response = agent.chat("What is the capital of France?")
print(response)

3. Check Tool Availability¶

# Test MCP Server health status
curl http://localhost:8080/health

# List all tools
curl -X POST http://localhost:8080/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc": "2.0", "method": "tools/list", "params": {}, "id": 1}'

Performance Benchmarks¶

GAIA Test Set Results (Based on Paper)¶

Model	Pass@1	Data Collection Speedup	Training Time
GPT-4o (Baseline)	27.91%	-	-
DeepSeek-V3	31.89%	-	-
Qwen3-32B-AWorld	32.23%	14.6x	48h (8x A100)

Key Findings: - Through the "learning from practice" paradigm, a 32B parameter model surpasses GPT-4o and DeepSeek-V3 - Distributed experience generation reduces data collection time from 7 days to 12 hours - End-to-end training (SFT + PPO) improves performance on the GAIA validation set by over 15 percentage points

Hardware Performance¶

Configuration	Throughput	GPU Utilization	Memory Usage
1x A100 80GB (TP=1)	120 tokens/s	85%	72GB
4x A100 80GB (TP=4)	450 tokens/s	92%	68GB/GPU
8x A100 80GB (FSDP+TP)	850 tokens/s	95%	70GB/GPU

Optimization Recommendations: - Small models (< 8B): Single GPU training, infer_tp=1, train_sp=1 - Medium models (8-30B): Use TP=4 for faster inference, infer_tp=4 - Large models (> 30B): Combine TP and SP, infer_tp=4, train_sp=8

Advanced Topics¶

Custom Reward Functions¶

Beyond GAIA's binary reward, you can implement more complex reward shaping:

def dense_reward_func(data_source, solution_str, ground_truth, extra_info=None):
    """Dense reward function (considering intermediate steps)"""
    reward = 0.0

    # 1. Tool usage reward (encourages exploration)
    num_tool_calls = solution_str.count("Tool call:")
    reward += min(num_tool_calls * 0.1, 0.5)  # Max 0.5 points

    # 2. Reasoning quality reward (based on CoT)
    if "<think>" in solution_str and "</think>" in solution_str:
        reward += 0.2  # Reward for having a reasoning process

    # 3. Final answer reward (main score)
    if question_scorer(extract_answer(solution_str), ground_truth):
        reward += 1.0

    # 4. Efficiency penalty (avoid excessive tool calls)
    if num_tool_calls > 15:
        reward -= 0.2

    return reward

Multi-task Training¶

# gaia_datasets/create_multitask_dataset.py
def create_multitask_dataset():
    datasets = []

    # Task 1: GAIA
    gaia_ds = load_gaia_dataset(...)
    gaia_ds['task_type'] = 'gaia'
    datasets.append(gaia_ds)

    # Task 2: Code Execution
    code_ds = load_code_dataset(...)
    code_ds['task_type'] = 'code'
    datasets.append(code_ds)

    # Task 3: Web Navigation
    web_ds = load_webarena_dataset(...)
    web_ds['task_type'] = 'web'
    datasets.append(web_ds)

    # Mixed sampling
    return pd.concat(datasets).sample(frac=1.0)

Citation¶

If you use AWorld Train in your research, please cite our paper:

@article{yu2025aworld,
  title={AWorld: Orchestrating the Training Recipe for Agentic AI},
  author={Yu, Chengyue and Lu, Siyuan and Zhuang, Chenyi and Wang, Dong and others},
  journal={arXiv preprint arXiv:2508.20404},
  year={2025}
}

Community & Support¶

GitHub Issues: https://github.com/inclusionAI/AWorld/issues
Discord: https://discord.gg/aworld
Paper: https://arxiv.org/abs/2508.20404
Official Documentation: https://inclusionai.github.io/AWorld/

**AWorld Train** — Let Your Agent Learn from Practice Made with ❤️ by [Inclusion AI](https://github.com/inclusionAI)

中文¶

# AWorld Train *面向 Agentic AI 的"从实践中学习"训练框架* [![License: MIT][license-image]][license-url] [![Paper](https://img.shields.io/badge/arXiv-2508.20404-b31b1b.svg)](https://arxiv.org/abs/2508.20404)

目录¶

简介
关于本教育性实验的重要说明
核心特性
GAIA 环境工具生态
Web 交互工具
文档处理工具
多媒体处理工具
智能推理工具
代码执行工具
文件系统工具
Excel 处理工具
知识检索工具
工具统计总结
核心架构
快速开始
安装训练框架
配置 GAIA 环境
构建自定义 Agent
准备训练
启动训练
最新优化
故障排查
性能基准
进阶主题
引用
社区与支持

简介¶

AWorld Train 是实现 "从实践中学习"（Learning from Practice） 范式的开源训练框架，专门为 Agentic AI 设计。根据 AWorld 论文，构建高性能 Agent 系统需要三个核心要素：

算法（Algorithm）：使 Agent 能够从环境交互中适应和改进的学习机制
环境（Environment）：提供丰富反馈和多样化挑战的复杂交互场景
先验（Priors）：当前大模型在推理、数学、视觉等领域的基础能力

AWorld Train 通过分布式架构解决了传统方法的核心瓶颈——经验生成效率低下。在 GAIA 基准测试中，我们将数据收集速度提升了 14.6 倍，使得大规模强化学习训练变得可行。

⚠️ 关于本教育性实验的重要说明¶

GAIA（General AI Assistants Benchmark） 是目前最具挑战性的 Agent 能力评测基准之一，也是 SOTA（State-of-the-Art）Agent 系统的竞技场。根据论文所示：

数据稀缺性：GAIA validation set 仅包含 165 个问题，test set 约 300 个问题，远少于传统 RL 训练所需的数据量
计算资源需求：论文中的 Qwen3-32B-AWorld 模型需要在 2 台 8x A100 GPU 集群上训练多天才能达到 32.23% 的性能，而这距离 SOTA 性能（80% 以上）还非常远
任务复杂度：GAIA 问题涉及多模态理解、多步推理、工具链调用等，平均需要 10-20 轮交互才能完成

因此，本项目采用了教育友好的配置，使用 Qwen3-4B-Thinking-2507 作为基座模型，训练速度较快。

本项目的目标是： - ✅ 演示完整的 "从实践中学习" 训练流程 - ✅ 理解 Agent-Environment 交互机制 - ✅ 实践 RL 算法（PPO/GRPO）在 Agent 训练中的应用

核心特性¶

⚡ 高效并发：分布式任务执行，14.6x 数据收集加速
🔌 框架无关：支持 VeRL、OpenRLHF、AReaL、SWIFT 等主流 RL 框架
🛠️ 工具生态：内置 26 个 MCP 服务器，提供 126 个工具函数，涵盖搜索、浏览器、代码执行、多模态处理等
📊 长上下文：支持 131K tokens 上下文，处理复杂多轮交互
🎯 SOTA 性能：Qwen3-32B-AWorld 在 GAIA 测试集达到 32.23% pass@1

GAIA 环境工具生态¶

根据论文和 MCP Server 实现，AWorld 为 GAIA 任务提供了全面的工具支持，共计 26 个 MCP 服务器，126 个工具函数。以下是按类别的完整工具列表：

🌐 Web 交互工具（3 服务器，9 工具）¶

1. Google Search Server (`googlesearch-server`)¶

search_google: 使用 Google Custom Search API 进行网络搜索
get_search_capabilities: 获取搜索服务能力信息

典型应用：查询实时信息、事实核查、多跳推理的起点

2. Browser Use Server (`browser-server`)¶

browser_use: 基于 LLM 的智能浏览器自动化（使用 browser-use 库）
get_browser_capabilities: 获取浏览器自动化能力

特性： - 自动处理机器人检测和验证码 - 支持表单填写、文件下载、内容提取 - 集成视觉理解和记忆功能

3. Playwright Server (`ms-playwright`)¶

提供 23 个精细化浏览器控制工具： - 导航：browser_navigate, browser_navigate_back - 交互：browser_click, browser_type, browser_hover, browser_drag, browser_select_option - 表单：browser_fill_form, browser_file_upload - 调试：browser_console_messages, browser_network_requests, browser_take_screenshot - 管理：browser_close, browser_resize, browser_tabs, browser_handle_dialog - 执行：browser_evaluate, browser_press_key, browser_wait_for - 快照：browser_snapshot, browser_install

对比：browser-server 提供高级自动化，ms-playwright 提供细粒度控制

📄 文档处理工具（5 服务器，12 工具）¶

4. Documents CSV Server (`documents-csv-server`)¶

extract_csv_content: 提取和分析 CSV 文件内容（支持 Markdown/JSON 格式输出）
list_supported_formats: 列出支持的 CSV 格式

5. Documents DOCX Server (`documents-docx-server`)¶

extract_docx_content: 提取 Word 文档内容（包括文本、表格、图片）
list_supported_formats: 列出支持的 DOCX 格式

6. Documents PPTX Server (`documents-pptx-server`)¶

extract_pptx_content: 提取 PowerPoint 内容（幻灯片文本、注释、布局）
list_supported_formats: 列出支持的 PPTX 格式

7. Documents PDF Server (`documents-pdf-server`)¶

convert_document_to_markdown: 将 PDF 转换为 Markdown（保留结构和格式）

8. Documents TXT Server (`documents-txt-server`)¶

extract_text_content: 提取纯文本文件内容
list_supported_formats: 列出支持的文本编码

GAIA 应用场景：处理附件文件（GAIA 数据集 70% 的问题包含文档附件）

🎥 多媒体处理工具（3 服务器，12 工具）¶

9. Media Audio Server (`media-audio-server`)¶

transcribe_audio: 语音转文字（支持 Whisper API）
extract_audio_metadata: 提取音频元数据（时长、比特率、采样率）
trim_audio: 裁剪音频片段
list_supported_formats: 列出支持的音频格式（MP3、WAV、M4A 等）

10. Media Image Server (`media-image-server`)¶

extract_text_ocr: OCR 文字识别（基于 Tesseract/Cloud Vision API）
analyze_image_ai: AI 图像分析（场景识别、对象检测、描述生成）
get_image_metadata: 提取图像元数据（尺寸、EXIF、拍摄时间）

11. Media Video Server (`media-video-server`)¶

analyze_video: 视频内容分析（场景分割、关键帧提取）
summarize_video: 视频摘要生成
extract_keyframes: 提取关键帧图像

补充：独立多媒体工具¶

Audio Server (audio-server): mcp_transcribe_audio - 高级语音转写
Image Server (image-server): mcp_image_recognition - 图像识别和分类

GAIA 应用：约 40% 的 GAIA 问题涉及图片、音频或视频分析

💡 智能推理工具（3 服务器，6 工具）¶

12. Intelligence Code Server (`intelligence-code-server`)¶

generate_python_code: 生成和验证 Python 代码（用于数学计算、数据处理）
get_reasoning_capabilities: 获取代码生成能力信息

13. Intelligence Think Server (`intelligence-think-server`)¶

complex_problem_reasoning: 复杂问题推理（数学证明、算法设计、逻辑谜题）
get_reasoning_capabilities: 获取推理能力信息

特点：调用更强大的推理模型（如 GPT-4o、Claude 3.7 Sonnet）进行深度思考

14. Intelligence Guard Server (`intelligence-guard-server`)¶

guarding_reasoning_process: 推理过程保护和验证（防止幻觉、检查逻辑一致性）
get_guarding_capabilities: 获取保护能力信息

论文亮点：这些"思考工具"使小模型能够调用大模型的推理能力，实现"站在巨人的肩膀上"

💻 代码执行工具（3 服务器，24 工具）¶

15. Terminal Server (`terminal-server`)¶

execute_command: 执行终端命令（Python、bash、系统命令）
get_command_history: 获取命令执行历史
get_terminal_capabilities: 获取终端能力信息

安全特性：命令白名单、超时控制、输出截断

16. E2B Code Server (`e2b-code-server`)¶

e2b_upload_file: 上传文件到沙箱
e2b_run_code: 在隔离沙箱中执行代码（支持 Python、JavaScript、多种语言）

优势：完全隔离的执行环境，防止恶意代码影响主系统

17. Terminal Controller (`terminal-controller`)¶

提供 10 个高级终端管理工具： - execute_command, get_command_history, get_current_directory, change_directory - list_directory, write_file, read_file - insert_file_content, delete_file_content, update_file_content

区别：terminal-server 专注命令执行，terminal-controller 提供文件系统管理

📂 文件系统工具（1 服务器，14 工具）¶

18. Filesystem Server (`filesystem-server`)¶

完整的文件操作能力： - 读取：read_file, read_text_file, read_media_file, read_multiple_files - 写入：write_file, edit_file - 管理：create_directory, move_file, get_file_info - 搜索：search_files, list_directory, list_directory_with_sizes, directory_tree - 权限：list_allowed_directories - 列出允许访问的目录

GAIA 应用：访问数据集附件（/root/workspace/gaia_dataset/ 目录）

📊 Excel 处理工具（1 服务器，29 工具）¶

19. Excel Server (`excel`)¶

提供企业级 Excel 操作能力：

数据操作（9 个）： - read_data_from_excel, write_data_to_excel - insert_rows, insert_columns, delete_sheet_rows, delete_sheet_columns - copy_range, delete_range, validate_excel_range

工作簿/表管理（7 个）： - create_workbook, create_worksheet, copy_worksheet - delete_worksheet, rename_worksheet, get_workbook_metadata

高级功能（13 个）： - apply_formula, validate_formula_syntax - format_range, create_chart, create_pivot_table, create_table - merge_cells, unmerge_cells, get_merged_cells - get_data_validation_info

GAIA 典型任务：分析复杂的 Excel 数据表、计算统计指标、生成图表

🔍 知识检索工具（3 服务器，11 工具）¶

20. Wikipedia Server (`wiki-server`)¶

search_wikipedia: 搜索维基百科词条
get_article_content: 获取完整文章内容
get_article_summary: 获取文章摘要
get_article_categories: 获取文章分类
get_article_links: 获取文章链接
get_article_history: 获取文章历史版本（用于时间敏感问题）
get_wikipedia_capabilities: 获取 Wikipedia 服务能力

特色功能：支持多语言、历史版本查询（GAIA 中有"某年某月的人口数据"类问题）

21. ArXiv Server (`parxiv-server`)¶

search_papers: 搜索 arXiv 论文
get_paper_details: 获取论文详细信息（摘要、作者、引用）
download_paper: 下载论文 PDF
get_arxiv_capabilities: 获取 arXiv 服务能力
get_categories: 获取 arXiv 分类列表

22. Wayback Machine Server (`wayback-server`)¶

list_archived_versions: 列出网页的历史存档版本
get_archived_content: 获取特定时间点的网页内容
get_wayback_capabilities: 获取 Wayback Machine 能力

GAIA 应用：回答"2015 年某网站上的信息"这类历史查询问题

📥 其他实用工具（3 服务器，3 工具）¶

23. Download Server (`download-server`)¶

download_file: 下载网络文件到本地
get_download_capabilities: 获取下载服务能力

24. Read Web Server (`readweb-server`)¶

提供网页内容读取能力（具体工具由 MCP 配置定义）

25. Google Search Alternative (`google-search`)¶

search: 简化的搜索接口
read_webpage: 读取网页内容

工具统计总结¶

类别	服务器数	工具数	关键能力
Web 交互	3	32	搜索、智能浏览、精细控制
文档处理	5	12	CSV、Word、PPT、PDF、TXT
多媒体	5	14	音频转写、OCR、图像/视频分析
智能推理	3	6	代码生成、复杂推理、验证
代码执行	3	36	终端命令、沙箱执行、文件管理
文件系统	1	14	完整文件操作能力
Excel	1	29	企业级表格处理
知识检索	3	11	Wikipedia、ArXiv、历史网页
其他	2	3	文件下载、网页读取
总计	26	126	涵盖 GAIA 所需的全部能力

工具调用示例（来自训练日志）¶

# Google 搜索示例
Tool call: aworld-mcp__search_google
Tool args: {"query": "Wyoming population 2020", "num_results": 5}
Result: {"success": true, "results": [{"title": "Wyoming - Census Bureau", "snippet": "576,851..."}]}

# 文件系统示例
Tool call: aworld-mcp__list_directory
Tool args: {"path": "/root/workspace/gaia_dataset/2023/test"}
Result: ["[FILE] 021a5339-...-bd9b-9368b3efda7a.pdf", "[FILE] 03c577c9-...-f8f598de14c1.mp3", ...]

# CSV 处理示例（缺少 tabulate 依赖时会报错）
Tool call: aworld-mcp__extract_csv_content
Tool args: {"file_path": "/root/workspace/gaia_dataset/2023/test/52e8ce1c-...-67d1648779b9.csv"}
Error: "CSV extraction failed: Missing optional dependency 'tabulate'"

# Wikipedia 历史查询示例
Tool call: aworld-mcp__get_article_history
Tool args: {"title": "Cat", "date": "20191231", "language": "en"}
Result: {...historical Wikipedia content...}

核心架构¶

AWorld Train 采用四阶段训练流水线：

Architecture

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│ Environment │───▶│    Agent    │───▶│   Adapter   │───▶│   Training  │
│   Setup     │    │Construction │    │   Layer     │    │  Framework  │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
     (MCP)           (AWorld)            (VeRL)           (PPO/GRPO)

环境配置：部署 GAIA MCP Server，提供 20+ 工具能力
Agent 构建：实现自定义 AgentLoop，定义决策逻辑
适配器集成：统一接口，对接 RL 训练框架
训练执行：配置奖励函数和超参数，启动训练任务

快速开始¶

系统要求¶

组件	要求
操作系统	Linux (推荐) / macOS / Windows
硬件	最低 4 CPU 核心 + 8GB RAM 训练推荐 8x A100/H100 GPU
软件	Docker, NVIDIA Driver, CUDA 12.1+

安装训练框架¶

以 VeRL 为例，安装步骤如下：

# 可在任意目录运行；先设置本书仓库绝对路径。
export BOOK_ROOT=/absolute/path/to/ai-agent-book

# 1. 安装系统依赖（Ubuntu/Debian）
sudo apt-get update
sudo apt-get install -y build-essential git wget

# 2. 安装 CUDA Toolkit（匹配你的 GPU Driver）
# 参考：https://developer.nvidia.com/cuda-downloads

# 3. 安装 PyTorch（匹配 CUDA 版本）
pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu121

# 4. 将本书已核验的 AWorld fork 克隆到规范配套路径
git clone https://github.com/bojieli/AWorld.git "${BOOK_ROOT}/chapter7/AWorld"
git -C "${BOOK_ROOT}/chapter7/AWorld" checkout --detach a52d61d6d483e66b22ef16970eae5bbf4f4ab2ec

# 5. 克隆并安装已核验的 VeRL 后端
git clone https://github.com/bojieli/verl.git "${BOOK_ROOT}/chapter7/verl"
git -C "${BOOK_ROOT}/chapter7/verl" checkout --detach 1593fc3a8cf894debdc3dece2a23ed739c282789
pip install -e "${BOOK_ROOT}/chapter7/verl"

重要提示：VeRL 的某些依赖需要在 CUDA 环境下编译，请确保先完成步骤 1-2。

配置 GAIA 环境¶

GAIA 环境通过 Docker 部署，提供 MCP（Model Context Protocol）服务。

1. 下载数据集¶

# 从 Hugging Face 下载 GAIA 数据集
cd ${BOOK_ROOT}/chapter7/AWorld/env/gaia-mcp-server/docker
mkdir -p gaia_dataset

# 使用 Hugging Face CLI 下载（需要先 pip install huggingface_hub）
huggingface-cli download gaia-benchmark/GAIA --repo-type dataset --local-dir gaia_dataset

2. 配置环境变量¶

cd ${BOOK_ROOT}/chapter7/AWorld/env/gaia-mcp-server/mcp_servers
cp .env_template .env

# 编辑 .env 文件，配置必要的 API 密钥
vim .env

.env 文件示例（部分字段）：

# OpenAI API（用于 intelligence-code-server 等）
OPENAI_API_KEY=your-openai-api-key

# Google Search API（用于 googlesearch-server）
GOOGLE_API_KEY=your-google-api-key
GOOGLE_CSE_ID=your-search-engine-id

# E2B API（用于代码执行沙箱）
E2B_API_KEY=your-e2b-api-key

3. 启动 MCP Server¶

cd ${BOOK_ROOT}/chapter7/AWorld/env
bash run-local.sh

启动成功后，你将看到类似输出：

Starting services...
DISPLAY=:0
2025-10-06 05:20:42,368 - __main__ - INFO - Starting MCP Server Proxy...
2025-10-06 05:20:42,373 - mcp_server_proxy.mcp_server_proxy - INFO - Added MCP server executor: googlesearch-server
...
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

MCP Server 提供两个服务： - MCP 接口：http://localhost:8080/mcp（Agent 工具调用） - VNC 界面：http://localhost:5901/vnc.html?autoconnect=true（可视化调试）

4. 验证环境¶

# 设置环境变量
export MCP_SERVER_URL=http://localhost:8080/mcp

# 测试连接（Python）
python3 << EOF
from train.adapter.verl.common import get_agent_tool_env_and_servers

config, servers = get_agent_tool_env_and_servers()
print(f"Available servers: {len(servers)}")
print(f"Sample servers: {servers[:5]}")
EOF

预期输出：

Available servers: 20
Sample servers: ['readweb-server', 'browser-server', 'documents-csv-server', ...]

构建自定义 Agent¶

实现 AgentLoop¶

创建自定义 Agent 的核心是继承 AworldAgentLoop 并实现 build_agents() 方法。以下是 GAIA Agent 的完整示例：

# train/examples/train_gaia_with_aworld_verl/custom_agent_loop.py

from aworld.agents.llm_agent import Agent
from aworld.config import AgentConfig
from train.adapter.verl.aworld_agent_loop import AworldAgentLoop
from train.adapter.verl.common import get_agent_tool_env_and_servers

class GaiaAgentLoop(AworldAgentLoop):
    """GAIA 任务的自定义 Agent Loop"""

    def build_agents(self):
        # 获取 MCP 环境配置和可用服务列表
        gaia_env_config, gaia_env_servers = get_agent_tool_env_and_servers()

        # 构建 Agent 实例
        return Agent(
            conf=AgentConfig(
                # LLM 服务地址由 VeRL 动态管理
                llm_base_url=self.get_llm_server_address(),
                llm_model_name=self.get_llm_server_model_name(),
                llm_api_key="dummy",  # VeRL 内部通信不需要真实 API Key
            ),
            name="gaia_super_agent",

            # 系统提示（定义 Agent 角色和能力）
            system_prompt="""You are a helpful AI assistant specialized in solving complex tasks.
You have access to various tools including web search, code execution, and file analysis.
When given a task, break it down into steps and use available tools systematically.
Always provide your final answer in <answer>...</answer> tags.""",

            # MCP 工具配置
            mcp_config=gaia_env_config,
            mcp_servers=gaia_env_servers,
        )

配置 agent.yaml¶

在 train/examples/train_gaia_with_aworld_verl/agent.yaml 中注册你的 AgentLoop：

- name: gaia_agent
  _target_: train.examples.train_gaia_with_aworld_verl.custom_agent_loop.GaiaAgentLoop

高级场景¶

多 Agent 系统¶

from aworld.swarms.swarm import Swarm

class MultiAgentLoop(AworldAgentLoop):
    def build_agents(self):
        config, servers = get_agent_tool_env_and_servers()

        # 创建专业化 Agent
        researcher = Agent(
            conf=AgentConfig(...),
            name="researcher",
            system_prompt="You are a research specialist...",
            mcp_servers=["googlesearch-server", "wiki-server"]
        )

        coder = Agent(
            conf=AgentConfig(...),
            name="coder",
            system_prompt="You are a coding expert...",
            mcp_servers=["e2b-code-server", "terminal-server"]
        )

        # 构建 Swarm
        return Swarm(
            agents=[researcher, coder],
            coordinator=researcher  # 主协调 Agent
        )

准备训练¶

1. 准备数据集¶

运行数据集生成脚本，将 GAIA 数据转换为训练格式：

cd ${BOOK_ROOT}/chapter7/AWorld/train/examples/train_gaia_with_aworld_verl/gaia_datasets

python create_dataset.py \
  --dataset_path ${BOOK_ROOT}/chapter7/AWorld/env/gaia-mcp-server/docker/gaia_dataset \
  --output_dir ~/datasets \
  --train_size 300 \
  --test_size 100

这将生成： - ~/datasets/train.parquet：300 条训练样本 - ~/datasets/test.parquet：100 条测试样本

数据格式（Parquet）：

字段	类型	说明
`prompt`	List[Dict]	格式化的聊天消息 `[{"role": "user", "content": "..."}]`
`data_source`	str	数据来源标识 `"gaia"`
`ability`	str	能力类别 `"agi"`
`reward_model`	Dict	奖励配置 `{"style": "GAIA", "ground_truth": "..."}`
`extra_info`	Dict	额外元数据（task_id, level 等）
`agent_name`	str	目标 Agent 名称

2. 配置奖励函数¶

在 train/examples/train_gaia_with_aworld_verl/metrics/gaia_reward_function.py 中定义奖励逻辑：

import re
from aworld.logs.util import logger

def gaia_reward_func(data_source, solution_str, ground_truth, extra_info=None):
    """
    GAIA 任务的奖励函数

    Args:
        data_source: 数据来源标识
        solution_str: Agent 生成的完整解答
        ground_truth: 标准答案
        extra_info: 额外信息（如 task_id, level）

    Returns:
        float: 奖励值（0.0 或 1.0）
    """
    # 从 solution_str 中提取 <answer>...</answer> 标签内容
    pattern = r'<answer>(.*?)</answer>'
    match = re.search(pattern, solution_str, re.DOTALL | re.MULTILINE)

    if not match:
        logger.warning("No answer tag found in solution")
        return 0.0

    answer = match.group(1).strip()
    logger.info(f"Extracted answer: {answer}, Ground truth: {ground_truth}")

    # 使用 GAIA 标准评分器（支持数字、列表、字符串）
    if question_scorer(answer, ground_truth):
        return 1.0
    else:
        return 0.0

def question_scorer(model_answer: str, ground_truth: str) -> bool:
    """GAIA 标准评分逻辑（省略详细实现）"""
    # 支持数字比较、列表比较、字符串归一化比较
    # 详见完整代码
    ...

3. 配置训练脚本¶

编辑 run.sh，配置关键参数：

#!/usr/bin/env bash
set -xeuo pipefail

# ============ 集群拓扑 ============
export GPUS_PER_NODE=${GPUS_PER_NODE:-8}
export NNODES=${NNODES:-1}

# ============ 模型和数据 ============
model_path=${model_path:-Qwen/Qwen3-4B-Thinking-2507}
train_files=$DATA_ROOT/datasets/train.parquet
test_files=$DATA_ROOT/datasets/test.parquet

# ============ 自定义配置 ============
path_to_train="/root/AWorld/train"
agent_loop_config_path=${path_to_train}/examples/train_gaia_with_aworld_verl/agent.yaml
reward_fn_file_path=${path_to_train}/examples/train_gaia_with_aworld_verl/metrics/gaia_reward_function.py
reward_fn_name=gaia_reward_func

# ============ 训练超参数 ============
# PPO 算法配置
adv_estimator=grpo              # 使用 Group Relative Policy Optimization
clip_ratio_low=0.2              # PPO clip 下界
clip_ratio_high=0.28            # PPO clip 上界
actor_lr=1e-6                   # Actor 学习率

# 长上下文配置（AWorld 最新优化）
max_turns=32                    # 最大交互轮数（从 8 提升到 32）
max_prompt_length=4096          # 提示最大长度（从 1024 提升到 4096）
max_response_length=4096        # 响应最大长度（从 2048 提升到 4096）

# 批次配置
train_batch_size=32             # 训练批次大小（从 1 提升到 32）
ppo_mini_batch_size=8           # PPO mini-batch 大小（4 个梯度更新）
n_resp_per_prompt=16            # 每个提示采样 16 个响应（从 1 提升）
n_resp_per_prompt_val=16        # 验证时采样数

# ============ MCP Server ============
export MCP_SERVER_URL=${MCP_SERVER_URL:-http://localhost:8080/mcp}

# ============ 性能优化 ============
export VLLM_USE_V1=1                      # 使用 vLLM v1 引擎
export VLLM_ATTENTION_BACKEND=FLASH_ATTN  # FlashAttention-2
infer_tp=1                                # Tensor Parallel 大小
train_sp=1                                # Sequence Parallel 大小
offload=true                              # 参数卸载到 CPU

# ============ VeRL 训练命令 ============
python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=$adv_estimator \
    data.train_files="['$train_files']" \
    data.val_files="['$test_files']" \
    data.return_raw_chat=true \
    data.train_batch_size=$train_batch_size \
    data.max_prompt_length=$max_prompt_length \
    data.max_response_length=$max_response_length \
    actor_rollout_ref.model.path="$model_path" \
    actor_rollout_ref.rollout.multi_turn.max_user_turns=$max_turns \
    actor_rollout_ref.rollout.multi_turn.max_assistant_turns=$max_turns \
    actor_rollout_ref.rollout.max_model_len=131072 \
    actor_rollout_ref.rollout.max_num_batched_tokens=131072 \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.9 \
    actor_rollout_ref.rollout.agent.agent_loop_config_path=$agent_loop_config_path \
    custom_reward_function.path="${reward_fn_file_path}" \
    custom_reward_function.name="${reward_fn_name}" \
    trainer.logger=['console','wandb'] \
    trainer.experiment_name=aworld_train_qwen3_4b \
    trainer.save_freq=5 \
    trainer.test_freq=5 \
    +trainer.num_steps=300

启动训练¶

单机训练¶

cd ${BOOK_ROOT}/chapter7/AWorld/train/examples/train_gaia_with_aworld_verl

# 启动训练（8卡 GPU）
export DATA_ROOT=~/datasets
export GPUS_PER_NODE=8
bash run.sh

多机训练（Slurm）¶

# 提交 Slurm 作业（2 节点，每节点 8 卡）
sbatch <<EOF
#!/bin/bash
#SBATCH --job-name=aworld-train
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --time=48:00:00

export DATA_ROOT=/path/to/datasets
export NNODES=2
export GPUS_PER_NODE=8

srun bash run.sh
EOF

训练监控¶

训练过程支持多种日志后端：

# 1. 控制台输出
# 实时查看训练指标（loss, reward, KL divergence 等）

# 2. WandB 可视化（推荐）
# 访问 https://wandb.ai/<your-project>/aworld_train_qwen3_4b

# 3. TensorBoard
tensorboard --logdir ~/datasets/checkpoint/aworld_train_qwen3_4b

关键监控指标：

指标	说明	目标值
`reward/mean`	平均奖励	逐步上升至 0.3+
`reward/max`	最大奖励	达到 1.0
`actor/loss`	Actor 损失	稳定下降
`rollout/response_length`	响应长度	根据任务调整
`rollout/num_turns`	平均轮数	高效利用工具（5-15 轮）

故障排查¶

Agent 训练过程输出示例¶

✅ 正常推理流程¶

(AgentLoopWorker pid=448354)   [agent] Content: Okay, let's see. So the user is asking about the population difference between the two states that have both Carl's Jr. and Hardee's fast food restaurants...
(AgentLoopWorker pid=448354)   [agent] Tool call: aworld-mcp__search_google - ID: chatcmpl-tool-94b30baa
(AgentLoopWorker pid=448354)   [agent] Tool args: {"query": "Wyoming population 2020", "num_results": 5}
(AgentLoopWorker pid=448354)   [agent] Content: ["{\"success\": true, \"message\": {\"query\": \"Wyoming population 2020\", \"results\": [{\"title\": \"Wyoming - Census Bureau Profile\", \"snippet\": \"576,851. The Total Population for Wyoming is 576,851...\"...}

说明：Agent 正确调用工具并接收结果，推理链路完整。

✅ 文件列表成功¶

(AgentLoopWorker pid=448358)   [agent] Content: ["[FILE] 021a5339-744f-42b7-bd9b-9368b3efda7a.pdf\n[FILE] 03c577c9-4227-48a9-9b75-f8f598de14c1.mp3\n[FILE] 063800f6-8832-4856-972b-17b877612533.png\n..."]
(AgentLoopWorker pid=448358)   [agent] Content: Okay, let's try to figure out how many horror titles are overdue based on the inventory file...

说明：文件系统工具正常工作，Agent 能够访问数据集文件。

GAIA MCP Server 输出示例¶

✅ 正常启动输出¶

$ docker logs gaia-mcp-server-gaia-mcp-server-1 -f

Starting services...
DISPLAY=:0
2025-10-06 05:20:42,368 - __main__ - INFO - Starting MCP Server Proxy...
2025-10-06 05:20:42,370 - mcp_server_proxy.mcp_server_proxy - INFO - Loaded MCP tool schema: mcp_tool_schema=
  readweb-server:
  browser-server:
    - get_browser_capabilities
    - browser_use
  documents-csv-server:
    - extract_csv_content
    - list_supported_formats
  googlesearch-server:
    - search_google
    - get_search_capabilities
  ...
2025-10-06 05:20:42,373 - mcp_server_proxy.mcp_server_proxy - INFO - Added MCP server executor: googlesearch-server
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

检查点： - ✅ Starting MCP Server Proxy 出现 - ✅ 20+ 工具服务器被加载（Added MCP server executor） - ✅ Uvicorn 在 8080 端口监听

✅ 训练时正常请求¶

INFO:     208.64.254.164:36416 - "POST /mcp HTTP/1.1" 200 OK
2025-10-06 05:09:23,880 - mcp.server.lowlevel.server - INFO - Processing request of type CallToolRequest
2025-10-06 05:09:27,006 - mcp.server.lowlevel.server - INFO - Processing request of type CallToolRequest
[10/06/25 05:09:27] INFO     🔍 Searching Google for: 'Speaker of the House that passed act...'
                    INFO     ✅ Found 5 results in 0.42s

说明：Agent 正常调用 Google 搜索工具，请求响应快速（< 1s）。

❌ 常见错误及解决方案¶

1. CSV 提取失败（缺少依赖）¶

ERROR    CSV extraction failed: Missing optional dependency 'tabulate'.
         Use pip or conda to install tabulate.

原因：pandas 的 to_markdown() 功能需要 tabulate 库。

解决方案：

# 进入 MCP Server Docker 容器
docker exec -it gaia-mcp-server-gaia-mcp-server-1 bash

# 安装缺失依赖
cd /app/mcp_servers/documents_server
source .venv/bin/activate
pip install tabulate

# 重启容器
exit
docker restart gaia-mcp-server-gaia-mcp-server-1

2. OpenAI API 密钥错误¶

WARNING  coding failed: Error code: 401 - {'error': {'message': 'Incorrect API key provided: [redacted-openrouter-key]', 'type': 'invalid_request_error', 'code': 'invalid_api_key'}}

原因：intelligence-code-server 等工具需要 OpenAI API 密钥生成代码。

解决方案：

# 方法 1：配置真实 OpenAI API Key
vim ${BOOK_ROOT}/chapter7/AWorld/env/gaia-mcp-server/mcp_servers/.env
# 添加：OPENAI_API_KEY=your-openai-api-key

# 方法 2：使用 OpenRouter 等兼容服务
# .env 中配置：
LLM_BASE_URL=https://openrouter.ai/api/v1
OPENAI_API_KEY=your-openrouter-api-key

# 重启服务
cd ${BOOK_ROOT}/chapter7/AWorld/env
bash run-local.sh

3. Wikipedia API 429 限流¶

ERROR    Wikipedia summary retrieval error:
         requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

原因：Wikipedia API 限流（429 Too Many Requests），但 Python 库没有正确处理。

解决方案：

# 方法 1：降低并发请求（调整训练参数）
train_batch_size=16  # 从 32 降低到 16
n_resp_per_prompt=8  # 从 16 降低到 8

# 方法 2：使用代理或切换 Wikipedia 镜像
# 在 wiki_server/.env 中配置：
WIKIPEDIA_BASE_URL=https://en.wikipedia.org/w/api.php

# 方法 3：添加请求重试逻辑（需修改代码）
# 在 wiki_server/src/wiki.py 中添加 exponential backoff

4. 工具执行超时¶

2025-10-06 05:21:08,548 - mcp_server_proxy.mcp_server_executor - INFO - Starting tool server browser-server...
[10 秒后无响应]

原因：浏览器工具（Playwright）启动慢或资源不足。

解决方案：

# 增加超时配置
vim ${BOOK_ROOT}/chapter7/AWorld/env/gaia-mcp-server/mcp_servers/.env
# 添加：
TOOL_EXECUTION_TIMEOUT=120  # 从默认 60s 增加到 120s

# 预热浏览器环境
docker exec -it gaia-mcp-server-gaia-mcp-server-1 bash
cd /app/mcp_servers/browser_server
uv run python -c "from playwright.sync_api import sync_playwright; sync_playwright().start()"

5. vLLM OOM（显存不足）¶

RuntimeError: CUDA out of memory. Tried to allocate 20.00 GiB (GPU 0; 79.35 GiB total capacity; 75.12 GiB already allocated; 2.31 GiB free; 78.90 GiB reserved in total by PyTorch)

解决方案：

# 方法 1：降低批次大小
train_batch_size=16             # 从 32 降低
ppo_mini_batch_size=4          # 从 8 降低

# 方法 2：降低 GPU 内存利用率
actor_rollout_ref.rollout.gpu_memory_utilization=0.75  # 从 0.9 降低

# 方法 3：启用 CPU offload
actor_rollout_ref.actor.fsdp_config.param_offload=true
actor_rollout_ref.actor.fsdp_config.optimizer_offload=true

# 方法 4：使用更大的 Tensor Parallel
infer_tp=4  # 从 1 增加到 4（需要 4 卡）

调试技巧¶

1. 启用详细日志¶

# VeRL 训练日志
export RAY_LOGGING_LEVEL=DEBUG
export HYDRA_FULL_ERROR=1

# MCP Server 日志
docker logs -f gaia-mcp-server-gaia-mcp-server-1 --tail 100

# vLLM 推理日志
export VLLM_LOGGING_LEVEL=DEBUG

2. 单步调试 Agent¶

# test_agent_debug.py
from train.examples.train_gaia_with_aworld_verl.custom_agent_loop import GaiaAgentLoop

# 创建 AgentLoop（不启动训练）
loop = GaiaAgentLoop()
agent = loop.build_agents()

# 测试单个问题
response = agent.chat("What is the capital of France?")
print(response)

3. 检查工具可用性¶

# 测试 MCP Server 健康状态
curl http://localhost:8080/health

# 列出所有工具
curl -X POST http://localhost:8080/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc": "2.0", "method": "tools/list", "params": {}, "id": 1}'

性能基准¶

GAIA 测试集结果（根据论文）¶

模型	Pass@1	数据收集加速	训练时间
GPT-4o (Baseline)	27.91%	-	-
DeepSeek-V3	31.89%	-	-
Qwen3-32B-AWorld	32.23%	14.6x	48h (8x A100)

关键发现： - 通过"从实践中学习"范式，32B 参数模型超越了 GPT-4o 和 DeepSeek-V3 - 分布式经验生成将数据收集时间从 7 天缩短至 12 小时 - 端到端训练（SFT + PPO）在 GAIA 验证集上提升 15+ 个百分点

硬件性能¶

配置	Throughput	GPU 利用率	内存占用
1x A100 80GB (TP=1)	120 tokens/s	85%	72GB
4x A100 80GB (TP=4)	450 tokens/s	92%	68GB/GPU
8x A100 80GB (FSDP+TP)	850 tokens/s	95%	70GB/GPU

优化建议： - 小模型（< 8B）：单卡训练，infer_tp=1, train_sp=1 - 中等模型（8-30B）：使用 TP=4 加速推理，infer_tp=4 - 大模型（> 30B）：组合使用 TP 和 SP，infer_tp=4, train_sp=8

进阶主题¶

自定义奖励函数¶

除了 GAIA 的二元奖励，你还可以实现更复杂的奖励塑形：

def dense_reward_func(data_source, solution_str, ground_truth, extra_info=None):
    """密集奖励函数（考虑中间步骤）"""
    reward = 0.0

    # 1. 工具使用奖励（鼓励探索）
    num_tool_calls = solution_str.count("Tool call:")
    reward += min(num_tool_calls * 0.1, 0.5)  # 最多 0.5 分

    # 2. 推理质量奖励（基于 CoT）
    if "<think>" in solution_str and "</think>" in solution_str:
        reward += 0.2  # 有思考过程

    # 3. 最终答案奖励（主要分数）
    if question_scorer(extract_answer(solution_str), ground_truth):
        reward += 1.0

    # 4. 效率惩罚（避免过度工具调用）
    if num_tool_calls > 15:
        reward -= 0.2

    return reward

多任务训练¶

# gaia_datasets/create_multitask_dataset.py
def create_multitask_dataset():
    datasets = []

    # 任务 1：GAIA
    gaia_ds = load_gaia_dataset(...)
    gaia_ds['task_type'] = 'gaia'
    datasets.append(gaia_ds)

    # 任务 2：Code Execution
    code_ds = load_code_dataset(...)
    code_ds['task_type'] = 'code'
    datasets.append(code_ds)

    # 任务 3：Web Navigation
    web_ds = load_webarena_dataset(...)
    web_ds['task_type'] = 'web'
    datasets.append(web_ds)

    # 混合采样
    return pd.concat(datasets).sample(frac=1.0)

引用¶

如果你在研究中使用了 AWorld Train，请引用我们的论文：

@article{yu2025aworld,
  title={AWorld: Orchestrating the Training Recipe for Agentic AI},
  author={Yu, Chengyue and Lu, Siyuan and Zhuang, Chenyi and Wang, Dong and others},
  journal={arXiv preprint arXiv:2508.20404},
  year={2025}
}

社区与支持¶

GitHub Issues: https://github.com/inclusionAI/AWorld/issues
Discord: https://discord.gg/aworld
论文: https://arxiv.org/abs/2508.20404
官方文档: https://inclusionai.github.io/AWorld/

**AWorld Train** — 让你的 Agent 从实践中学习 Made with ❤️ by [Inclusion AI](https://github.com/inclusionAI)

参数	旧值	新值	提升
`max_turns`	8	32	4x 交互深度
`max_prompt_length`	1024	4096	4x 输入容量
`max_response_length`	2048	4096	2x 输出容量
`train_batch_size`	1	32	32x 训练效率
`n_resp_per_prompt`	1	16	16x 样本多样性

English¶

Experiment 7-16 reproduction anchor¶

Table of Contents¶

Introduction¶

⚠️ Important Note on This Educational Experiment¶

Core Features¶

GAIA Environment Tool Ecosystem¶

🌐 Web Interaction Tools (3 Servers, 27 Documented Tools)¶

1. Google Search Server (googlesearch-server)¶

2. Browser Use Server (browser-server)¶

3. Playwright Server (ms-playwright)¶

📄 Document Processing Tools (5 Servers, 9 Documented Tools)¶

4. Documents CSV Server (documents-csv-server)¶

5. Documents DOCX Server (documents-docx-server)¶

6. Documents PPTX Server (documents-pptx-server)¶

7. Documents PDF Server (documents-pdf-server)¶

8. Documents TXT Server (documents-txt-server)¶

🎥 Multimedia Processing Tools (3 Servers, 12 Tools)¶

9. Media Audio Server (media-audio-server)¶

10. Media Image Server (media-image-server)¶

11. Media Video Server (media-video-server)¶

Supplementary: Standalone Multimedia Tools¶

💡 Intelligent Reasoning Tools (3 Servers, 6 Tools)¶

12. Intelligence Code Server (intelligence-code-server)¶

13. Intelligence Think Server (intelligence-think-server)¶

14. Intelligence Guard Server (intelligence-guard-server)¶

💻 Code Execution Tools (3 Servers, 15 Documented Tools)¶

15. Terminal Server (terminal-server)¶

16. E2B Code Server (e2b-code-server)¶

17. Terminal Controller (terminal-controller)¶

📂 File System Tools (1 Server, 14 Tools)¶

18. Filesystem Server (filesystem-server)¶

📊 Excel Processing Tools (1 Server, 29 Tools)¶

19. Excel Server (excel)¶

🔍 Knowledge Retrieval Tools (3 Servers, 15 Documented Tools)¶

20. Wikipedia Server (wiki-server)¶

21. ArXiv Server (parxiv-server)¶

22. Wayback Machine Server (wayback-server)¶

📥 Other Utility Tools (3 Servers, at Least 4 Documented Tools)¶

23. Download Server (download-server)¶

24. Read Web Server (readweb-server)¶

25. Google Search Alternative (google-search)¶

Tool Statistics Summary¶

Tool Call Examples (from Training Logs)¶

Core Architecture¶

Quick Start¶

System Requirements¶

Install Training Framework¶

Configure GAIA Environment¶

1. Download the Dataset¶

2. Configure Environment Variables¶

3. Start the MCP Server¶

4. Verify the Environment¶

Building a Custom Agent¶

Implementing AgentLoop¶

Configuring agent.yaml¶

Advanced Scenarios¶

Multi-Agent System¶

Preparing for Training¶

1. Prepare the Dataset¶

2. Configure the Reward Function¶

3. Configure the Training Script¶

Launching Training¶

Single-Node Training¶

Multi-Node Training (Slurm)¶

Training Monitoring¶

Latest Optimizations¶

1. Long Context Support (verl_provider.py)¶

2. Data Format Optimization (create_dataset.py)¶

3. Hyperparameter Tuning (run.sh)¶

4. Memory Optimization¶

5. New Qwen3-30B-A3B Training Script¶

Troubleshooting¶

Agent Training Process Output Examples¶

✅ Normal Inference Flow¶

✅ Successful File Listing¶

GAIA MCP Server Output Example#### ✅ Normal Startup Output¶

✅ Normal Request During Training¶

❌ Common Errors and Solutions¶

1. CSV Extraction Failed (Missing Dependency)¶

1. Google Search Server (`googlesearch-server`)¶

2. Browser Use Server (`browser-server`)¶

3. Playwright Server (`ms-playwright`)¶

4. Documents CSV Server (`documents-csv-server`)¶

5. Documents DOCX Server (`documents-docx-server`)¶

6. Documents PPTX Server (`documents-pptx-server`)¶

7. Documents PDF Server (`documents-pdf-server`)¶

8. Documents TXT Server (`documents-txt-server`)¶

9. Media Audio Server (`media-audio-server`)¶

10. Media Image Server (`media-image-server`)¶

11. Media Video Server (`media-video-server`)¶

12. Intelligence Code Server (`intelligence-code-server`)¶

13. Intelligence Think Server (`intelligence-think-server`)¶

14. Intelligence Guard Server (`intelligence-guard-server`)¶

15. Terminal Server (`terminal-server`)¶

16. E2B Code Server (`e2b-code-server`)¶

17. Terminal Controller (`terminal-controller`)¶

18. Filesystem Server (`filesystem-server`)¶

19. Excel Server (`excel`)¶

20. Wikipedia Server (`wiki-server`)¶

21. ArXiv Server (`parxiv-server`)¶

22. Wayback Machine Server (`wayback-server`)¶

23. Download Server (`download-server`)¶

24. Read Web Server (`readweb-server`)¶

25. Google Search Alternative (`google-search`)¶

1. Google Search Server (`googlesearch-server`)¶

2. Browser Use Server (`browser-server`)¶

3. Playwright Server (`ms-playwright`)¶

4. Documents CSV Server (`documents-csv-server`)¶

5. Documents DOCX Server (`documents-docx-server`)¶

6. Documents PPTX Server (`documents-pptx-server`)¶

7. Documents PDF Server (`documents-pdf-server`)¶

8. Documents TXT Server (`documents-txt-server`)¶

9. Media Audio Server (`media-audio-server`)¶

10. Media Image Server (`media-image-server`)¶

11. Media Video Server (`media-video-server`)¶

12. Intelligence Code Server (`intelligence-code-server`)¶

13. Intelligence Think Server (`intelligence-think-server`)¶

14. Intelligence Guard Server (`intelligence-guard-server`)¶

15. Terminal Server (`terminal-server`)¶

16. E2B Code Server (`e2b-code-server`)¶

17. Terminal Controller (`terminal-controller`)¶

18. Filesystem Server (`filesystem-server`)¶

19. Excel Server (`excel`)¶

20. Wikipedia Server (`wiki-server`)¶

21. ArXiv Server (`parxiv-server`)¶

22. Wayback Machine Server (`wayback-server`)¶

23. Download Server (`download-server`)¶

24. Read Web Server (`readweb-server`)¶

25. Google Search Alternative (`google-search`)¶