Experiment 7-12 reproduction anchor¶

Experiment 7-12 is the V-IRL-L/VL track inside SFTvsRL, not a separate repository named SpatialReasoning. This directory is the book-owned explanation; executable code is bojieli/SFTvsRL at chapter7/SFTvsRL, verified at commit fef0a4a3367260a0934be1e40b01e4021698e023.

git clone https://github.com/bojieli/SFTvsRL.git chapter7/SFTvsRL
git -C chapter7/SFTvsRL checkout --detach fef0a4a3367260a0934be1e40b01e4021698e023
cd chapter7/SFTvsRL
pip install -r requirements.txt
pip install -e gym
bash scripts/virl_training/vl_train.sh
bash scripts/virl_evaluation/vl_indist_eval.sh
bash scripts/virl_evaluation/vl_rule_ood_eval.sh
bash scripts/virl_evaluation/vl_visual_ood_eval.sh

The same checkout supplies Experiment 7-11 through scripts/gp_training/ and scripts/gp_evaluation/. These commands were statically verified but not executed during this audit.

English¶

This document provides a detailed introduction to the design, implementation, and evaluation methods of the Vision-Language Navigation Reinforcement Learning experiment based on the V-IRL platform.

V-IRL (Virtual Intelligence in Real Life) is an open-source platform for building and testing virtual agents, enabling agents to interact in virtual real-world environments using real geospatial data and street view imagery.

1. Experiment Overview¶

This experiment implements the Vision-Language Navigation task based on the V-IRL (Virtual Intelligence in Real Life) platform, aiming to verify the advantages of Reinforcement Learning (RL) over Supervised Fine-Tuning (SFT) in visual generalization capability.

V-IRL Platform Introduction: - Platform Positioning: An open-source platform for building and testing virtual agents, enabling agents to perceive, reason, and act in virtual but realistic environments - Core Features: - Utilizes real-world geospatial data and street view imagery - Supports city-wide navigation and task execution globally - Provides rich sensory inputs (vision, geolocation, place information, etc.) - Supports multiple task types (navigation, place recommendation, urban planning, collaboration, etc.) - Technical Foundation: Based on Google Maps Platform's Street View and geospatial APIs

Core Findings of This Experiment: - RL-trained models can generalize to visually out-of-distribution (OOD) environments - SFT-trained models tend to memorize training data and generalize poorly in OOD scenarios - RL improves the model's underlying visual recognition capability through outcome-based reward

Paper Source: SFT Memorizes, RL Generalizes Code Repository: - Recommended (my fork version)：bojieli/SFTvsRL ⭐ - Official version：LeslieTrue/SFTvsRL

2. Experiment Objectives¶

2.1 Research Questions¶

Answer the following core questions: 1. Visual Generalization: Can a Vision-Language Model transfer navigation skills learned in one city (NYC) to a city with a completely different visual appearance (San Francisco)? 2. Training Method Comparison: Which method, SFT or RL, learns more generalizable visual representations? 3. Source of RL Advantage: Why does RL improve visual generalization? Does it enhance underlying visual recognition?

2.2 Experiment Design Principles¶

Controlled Variables: - Same base model (Llama-3.2-11B-Vision) - Same task environment (V-IRL platform navigation task) - Same evaluation metrics (per-step accuracy, success rate)

Independent Variables: - Training method: SFT vs RL (PPO) - Visual environment: In-Distribution (NYC) vs Out-of-Distribution (San Francisco) - Action space: Absolute directions vs Relative directions

3. Experiment Results¶

3.1 Main Results¶

According to the paper's Figure 1 and experimental results:

Metric	SFT	RL (PPO)	Improvement
In-Distribution Per-Step Accuracy	~85%	~90%	+5%
Rule OOD Per-Step Accuracy	~15%	~70%	+55%
Visual OOD Generalization	Failed (<10%)	Successful (~60%)	+50%
V-IRL Mini Benchmark	44.0%	77.8%	+33.8%

3.2 Key Findings¶

RL Achieves Visual Generalization:
In the San Francisco (visual OOD) environment, the RL model maintains ~60% per-step accuracy
The SFT model drops to <10% in the same environment, close to random guessing
RL Improves Visual Recognition Capability:
Ablation experiments show that RL-trained models improve accuracy on the card recognition task (GeneralPoints-VL)
This indicates that RL not only learns navigation strategies but also improves the underlying visual encoder
Necessity of SFT:
Training RL directly from the base model fails (cannot output structured JSON)
SFT acts as a "format teacher" to stabilize output format, enabling effective RL training

4. Environment Setup¶

4.1 System Requirements¶

# Hardware Requirements
- GPU: 8×H100/H800/A100 (80GB) for training
- Memory: 1000GB RAM for training
- Storage: ~500GB for NYC route data + street views + checkpoints

# Software Environment
- Python: 3.13.0

4.2 Installation Steps¶

# 1. Clone the verified fork into the book's canonical external-checkout path
export BOOK_ROOT=/absolute/path/to/ai-agent-book
git clone https://github.com/bojieli/SFTvsRL.git "${BOOK_ROOT}/chapter7/SFTvsRL"
git -C "${BOOK_ROOT}/chapter7/SFTvsRL" checkout --detach fef0a4a3367260a0934be1e40b01e4021698e023
cd "${BOOK_ROOT}/chapter7/SFTvsRL"

# 2. Create conda environment
conda create -n SFTvsRL python==3.13 -y
conda activate SFTvsRL

# 3. Install dependencies
pip install -r requirements.txt

# 4. Install gym environments
cd gym
pip install -e .
cd ..

# 5. Download data from HuggingFace
huggingface-cli download tianzhechu/SFTvsRL_Data --local-dir ./data

# 6. Login to wandb (create an account on wandb.ai to obtain API key)
wandb login

4.3 Official Code Bugs and Fixes¶

Problem Description:

The official LeslieTrue/SFTvsRL repository has a serious bug that prevents saving checkpoints after training:

# rl/trainer/base_trainer.py (official version, line 38)
def __init__(self, ..., save_every=None, ...):
    ...
    self.save_ckpt = save_ckpt
    self.save_every = None  # ❌ BUG: Hardcoded to None, ignores configuration parameter!

Bug Impact:

# In the training loop
for update in range(self.num_updates):
    if self.save_ckpt:
        save_model = (update + 1) % self.save_every == 0  # ❌ TypeError!
        # Because self.save_every = None, modulo operation fails

Even with save_every: 1 set in the configuration file, training will throw an error:

TypeError: unsupported operand type(s) for %: 'int' and 'NoneType'

Fix:

The bojieli/SFTvsRL fork fixes this issue:

# rl/trainer/base_trainer.py (fixed, line 38)
def __init__(self, ..., save_every=None, ...):
    ...
    self.save_ckpt = save_ckpt
    self.save_every = save_every  # ✅ Correctly uses the configuration parameter

Fixed Files:

rl/trainer/base_trainer.py: Fixed save_every parameter passing
rl/configs/llama_virl_vl.yaml: Added missing save_every: 1 configuration

Why is this fix needed?

After 15 hours of training:
- Official version: ❌ Cannot save checkpoint, losing training progress
- Fixed version: ✓ Successfully saves checkpoint, enabling evaluation and continued training

4.4 Data Download and Path Configuration¶

4.4.1 Download Data¶

# 1. Create data directory
mkdir -p /root/SFTvsRL_Data
cd /root/SFTvsRL_Data

# 2. Download VIRL data (from HuggingFace)
huggingface-cli download tianzhechu/SFTvsRL_Data \
    --include "VIRL_routes/*" \
    --local-dir .

# 3. Unzip data
cd VIRL_routes
unzip nyc_1k_routes.zip
unzip VLN_mini.zip  # For Visual OOD evaluation

# 4. Verify directory structure
ls -la nyc_1k_routes/
# Should see:
# - route_infos.json
# - gps_pano_mapping.pkl
# - street_views/

4.4.2 Final Directory Structure¶

/root/SFTvsRL_Data/
└── VIRL_routes/
    ├── nyc_1k_routes/              # NYC training data
    │   ├── route_infos.json        # Route information
    │   ├── gps_pano_mapping.pkl    # GPS to panorama ID mapping
    │   └── street_views/           # Street view image directory
    │       ├── pano_XXX_h000.jpg
    │       ├── pano_XXX_h090.jpg
    │       └── ...
    ├── VLN_mini/                   # San Francisco OOD data
    │   ├── route_infos.json
    │   ├── gps_pano_mapping.pkl
    │   └── street_views/
    └── ...

4.4.3 Configure Training Script Paths (Optional)¶

If you do not have access to /root, configure the paths in scripts/virl_training/vl_train.sh:

BASE_DIR="/root/SFTvsRL_Data/VIRL_routes"

4.5 Start Training¶

After completing data download and configuration, you can directly run training:

# 1. Enter the code directory
cd /root/SFTvsRL

# 2. Activate environment
conda activate SFTvsRL

# 3. Start training (using the pre-configured script)
bash scripts/virl_training/vl_train.sh

The training script automatically uses the following paths (configured in the script):

BASE_DIR="/root/SFTvsRL_Data/VIRL_routes"
ROUTE_INFO="${BASE_DIR}/nyc_1k_routes/route_infos.json"
GPS_TO_PANO="${BASE_DIR}/nyc_1k_routes/gps_pano_mapping.pkl"
STREETVIEWS="${BASE_DIR}/nyc_1k_routes/street_views/"

Expected Output:

Parsed instruction: ['First, turn right to face north.', ...]
Collecting Trajectories: 100%|████████| 256/256 [40:25<00:00]
PPO Training Epoch 0/4: 100%|████████| 256/256 [05:12<00:00]
PPO Training Epoch 1/4: 100%|████████| 256/256 [05:10<00:00]
...
Saving checkpoint to: train_ckpt/virl_vl/checkpoint-epoch-4/

Check the checkpoint after training:

ls -lh train_ckpt/virl_vl/
# Should see (if save_every=5):
# checkpoint-epoch-4/
# checkpoint-epoch-9/
# checkpoint-epoch-14/

5. Task Definition¶

Task: The agent (VLM) must navigate to a target location in real-world street scenes based on natural language instructions.

Input: 1. Global Instruction: A complete description of the navigation route.

1. First, turn left to face south.
2. Move forward until you reach next intersection where Battery Park is nearby.
3. Turn right to face west.
4. Move forward until you reach destination.

Visual Observation: A 2×2 grid of street view images (4 directions).

┌─────────┬─────────┐
│ Front   │ Right   │
├─────────┼─────────┤
│ Back    │ Left    │
└─────────┴─────────┘

Each image: 640×640 pixels
Total image size: 1280×1280 pixels (with 5px separator)
Actual model input: 2405×2405 pixels (1200×2 + 5)

History Sequence (Observation-Action Sequence):

O_0: "No landmarks nearby; You observe an intersection"
A_0: "turn_direction(south)"
O_1: "Battery Park on your right; No intersection"
A_1: "forward()"
...

Output: A structured JSON action.

{
  "current observation": "Battery Park on your right; You observe an intersection",
  "current instruction": "Turn right to face west",
  "action": "turn_direction(west)"
}

5.2 Success Conditions¶

An episode is successful if: 1. ✓ The correct destination is reached (executing stop() at the correct location). 2. ✓ Each waypoint is executed correctly within the allowed number of verification attempts (default 2). 3. ✓ The maximum number of steps is not exceeded.

6. Action Space¶

6.1 Absolute Action Space¶

The default action space used during training:

ACTION_SPACE = [
    "forward()",                    # Move forward one step
    "turn_direction(north)",        # Turn to face north (0°)
    "turn_direction(northeast)",    # Turn to face northeast (45°)
    "turn_direction(east)",         # Turn to face east (90°)
    "turn_direction(southeast)",    # Turn to face southeast (135°)
    "turn_direction(south)",        # Turn to face south (180°)
    "turn_direction(southwest)",    # Turn to face southwest (225°)
    "turn_direction(west)",         # Turn to face west (270°)
    "turn_direction(northwest)",    # Turn to face northwest (315°)
    "stop()"                        # Stop (destination reached)
]

Features: - Uses absolute direction terms (compass directions). - Independent of the agent's current heading. - Aligns with natural human navigation habits.

6.2 Relative Action Space¶

The action space used for Rule OOD evaluation:

ACTION_SPACE_RELATIVE = [
    "forward()",                      # Move forward one step
    "turn_direction(left)",           # Turn left (~-90°)
    "turn_direction(right)",          # Turn right (~+90°)
    "turn_direction(slightly left)",  # Turn slightly left (-45° to 0°)
    "turn_direction(slightly right)", # Turn slightly right (0° to +45°)
    "stop()"                          # Stop
]

Features: - Uses relative directions (relative to the current heading). - Tests the model's generalization ability to different instruction formats. - Action semantics are completely different, but the task objective is the same.

6.3 Action Space Configuration¶

Set in the configuration file:

# rl/configs/llama_virl_vl.yaml
env_config:
  absolute_action: true  # True: Absolute, False: Relative

Override in training/evaluation scripts:

# Training with absolute actions
--env_config.absolute_action=True

# Evaluation with relative actions (Rule OOD)
--env_config.absolute_action=False

7. RL Environment Details¶

7.1 Environment Class Structure¶

VIRL uses the OpenAI Gym interface:

class NavigationEnvironment(gym.Env):
    """
    V-IRL Navigation Environment

    Main Components:
    - Platform: Google Street View interface
    - Ground Truth Rail: Precomputed correct path
    - Verification System: Action verification and feedback mechanism
    """

    def __init__(self,
        route_info_path,      # Path to route data
        resolution=1200,      # Image resolution
        verify_iter=2,        # Number of attempts per waypoint
        absolute_action=True, # Action space type
        relocation=True,      # GPS relocation to nearest panorama point
        drop_rate=0.5,        # Drop rate for interpolated points between waypoints
        ...
    )

7.2 Ground Truth Rail¶

The environment precomputes a "rail" (reference trajectory) containing:

Dense waypoints:
Original route: 5-10 intersections
After interpolation: one point every 5-10 meters
Total: ~20-40 waypoints per route

Each waypoint contains:

waypoint = {
    'geocode': [40.758, -73.985],           # GPS coordinates
    'heading': 180,                          # Heading (degrees)
    'gt_action': 'turn_direction(south)',   # Correct action
    'observation': 'Battery Park on right', # Landmark description
    'intersection_observation': 'You observe an intersection',
    'instruction': 'Turn left to face south',  # Current instruction to execute
    'instruction_idx': 1                     # Instruction index
}

7.3 Verification Mechanism¶

Multiple Attempt Mechanism (verify_iter=2):

# Step 1: Agent outputs an action
agent_action = model.generate(obs, instruction, history)

# Step 2: Compare with ground truth
if agent_action == gt_action:
    reward = +1  # CORRECT_ACTION
    move_to_next_waypoint()
    remaining_attempts = verify_iter  # Reset attempt count
else:
    reward = -1  # INCORRECT_ACTION
    remaining_attempts -= 1

    if remaining_attempts > 0:
        # Stay at current position, provide feedback, allow retry
        feedback = f"Incorrect action. Expected {gt_action}"
        stay_at_current_position()
    else:
        # Attempts exhausted, force move to next waypoint (penalty)
        reward = -1
        force_move_to_next_waypoint()

Reward Function:

REWARD_FN_VIRL = {
    "CORRECT_ACTION": +1,           # Action is correct
    "INCORRECT_ACTION": -1,         # Action is incorrect
    "INCORRECT_OBS": -1.5,          # Observation description error (incorrectly detecting intersection)
    "INCORRECT_INSTRUCTION": -1.75  # Instruction understanding error
}

7.4 Episode Termination Conditions¶

An episode ends when any of the following occurs:

Success: done=True, is_success=True
Executes stop() at the correct location.
Failure: truncated=True, is_success=False
Exceeds the maximum number of steps (determined by rail length, typically 20-40 steps).
Forced Advance: is_success=False
Attempts for a waypoint are exhausted, forced to advance.
Episode continues but is marked as a failure.

7.5 Environment Configuration¶

# rl/configs/llama_virl_vl.yaml
env_config:
  id: 'gym_virl/Navigation-v0'
  route_info_path: "..."           # Route data
  resolution: 1200                 # Street view image resolution
  verify_iter: 2                   # Verification attempt count
  absolute_action: true            # Action space type
  relocation: true                 # GPS relocation
  drop_rate: 0.5                   # Waypoint sampling rate
  straight_line_length: 5          # Number of interpolation points between two intersections

  platform_cfg:
    STREET_VIEW:
      SIZE: [640, 640]             # Single street view size
      HEADING: 0                   # Default heading
      PITCH: 0                     # Pitch angle
      FOV: 90                      # Field of view
      SOURCE: outdoor              # Street view source

    OFFLINE:
      ENABLED: True                # Use offline cache
      PANORAMA_DIR: "..."          # Street view image directory
      GPS_TO_PANO_PATH: "..."      # GPS to panorama ID mapping
      MAPPING_RADIUS: 20           # Relocation search radius (meters)

8. Trajectory Generation Mechanism¶

8.1 Trajectory Collection During Training¶

Collect 256 steps per update (not episodes):

def collect_trajectories(self):
    """
    Collect 256 environment interaction steps
    May span multiple episodes (routes)
    """
    obs, info = self.env.reset()  # Initialize first route

    for step in range(256):
        # 1. Construct prompt
        prompt = format_prompt(
            global_instruction=info['global_instruction'],
            obs_act_seq=info['obs_act_seq'],
            current_obs=obs
        )

        # 2. Model generates action (inference)
        with torch.no_grad():
            # Process 4 street view images
            obs_image = convert_to_2x2_grid(obs)  # [2405, 2405, 3]

            # VLM forward pass
            values, io_dict, output_text, action_log_prob, action_tokens = \
                actor_critic.act_oneline(
                    inputs=(obs_image, prompt),
                    temperature=0.2,
                    max_new_tokens=512
                )

            # Parse JSON output
            action = parse_json(output_text)['action']

        # 3. Execute action
        obs_next, reward, done, truncated, info = env.step(output_text)

        # 4. Store in rollout buffer
        rollouts.insert(
            obs={"image": obs, "io_dict": io_dict},
            action_log_prob=action_log_prob,
            value=values,
            reward=reward,
            mask=1-int(done or truncated)
        )

        running_reward += reward

        # 5. Episode management
        if done or truncated:
            # Current episode ends, start new episode
            log_episode_reward(running_reward)
            running_reward = 0
            obs, info = env.reset()  # Load new route
        else:
            obs = obs_next

    return rollouts  # Data for 256 steps

Key Points: - 256 steps may contain 10-15 complete episodes (depending on route length) - The last episode may be incomplete (truncated) - All data is saved for PPO training

8.2 Multi-Episode Trajectory Example¶

Update 1: Collect 256 steps
├─ Episode 1 (Route A, 14 steps): Success ✓
│  └─ Steps 0-13: [turn_direction(south), forward(), ..., stop()]
│
├─ Episode 2 (Route B, 22 steps): Success ✓
│  └─ Steps 14-35: [...]
│
├─ Episode 3 (Route C, 18 steps): Failed ✗
│  └─ Steps 36-53: [...] (exceeded attempts at waypoint 12)
│
├─ Episode 4 (Route D, 16 steps): Success ✓
│  └─ Steps 54-69: [...]
│
├─ ...
│
└─ Episode N (Route X, partial): Truncated
   └─ Steps 240-255: [...] (episode incomplete, but data still used for training)

8.3 LLM Input and Output Examples¶

LLM Input Example:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

<|image|>
[Task Description]
You are an expert in navigation. You will receive a sequence of instructions to follow while observing your surrounding street views. You
are also provided with your observation and action history in text. Your goal is to first analyze the instruction and identify the next sentence to be executed.
Then, you need to provide the action to be taken based on the current observation and instruction.

[Instruction]
1. First, turn left to face northeast.
2. Move forward until you reach next intersection where Battery Playscape is on your right behind.
3. Turn right to face north.
4. Move forward until you reach next intersection.
5. Turn slightly left to face northwest.
6. Move forward until you reach next intersection.
7. Turn left to face north.
8. Move forward until you reach next intersection.
9. Turn right to face southeast.
10. Move forward until you reach next intersection.
11. Turn right to face south.
12. Move forward until you reach destination where The destination Cafe De Novo is on your right.


[Observation format]
You observe a 2x2 grid of streetview images with the following headings:
[front, right
 back, left]
You need to identify if any of the landmarks in the instruction are visible in the street view grid.

[Action space]
"forward()": indicates moving forward one step
"turn_direction(x)": indicates adjust the ego agent direction towards x direction. x could be any following 8 directions ['north', 'northeast', 'east', 'southeast', 'south', 'southwest', 'west', 'northwest']
"stop()": indicates the navigation is finished.

[Observations and actions sequence]
O_1: No landmarks nearby;
A_1: turn_direction(northeast)
O_2: No landmarks nearby;
A_2: forward()
O_3: No landmarks nearby;
A_3: forward()
O_4: Battery Playscape is on your right behind; You observe an intersection
A_4: turn_direction(north)
O_5: No landmark nearby; You observe an intersection
A_5: turn_direction(northwest)
O_6: No landmarks nearby;
A_6: forward()
O_7: No landmarks nearby;
A_7: forward()
O_8: No landmarks nearby;
A_8: forward()
O_9: No landmark nearby; You observe an intersection
A_9: turn_direction(north)
O_10: No landmarks nearby;
A_10: forward()
O_11: No landmarks nearby;
A_11: forward()
O_12: No landmarks nearby;
A_12: forward()
O_13: You observe an image of 4 views; You observe an intersection
A_13:


[Output]
{
  "current observation": latest observation from the street view grid,
  "current instruction": analyze the full instruction and identify the sentence to be executed,
  "action": the action to be taken chosen from the action space,
}
<|eot_id|><|start_header_id|>assistant<|end_header_id|>

LLM Output Example:

{
  "current observation": "No landmark nearby; You observe an intersection",
  "current instruction": "Turn right to face southeast.",
  "action": "turn_direction(southeast)",
}

9. Dataset¶

9.1 Training Dataset¶

NYC 1K Routes:

Data Source: Collected via Google Maps API
Number of Routes: 1,000
Coverage Area: Manhattan, Brooklyn, Queens, New York City
Total Waypoints: ~20,000-30,000
Street View Images: ~100,000 (640×640, 4 directions/location)

Data Structure:

// route_infos.json
[
  [  // Route list
    {      "route_id": "nyc_001",
      "start_place": {
        "name": "Times Square",
        "geocode": [40.758, -73.985],
        "relocated_geocode": [40.7580, -73.9855]
      },
      "dest_place": {
        "name": "Central Park South",
        "geocode": [40.767, -73.979]
      },
      "init_heading": 0,
      "milestone_info": "Turn left to face south. Move forward...",
      "route_results": {
        "geocode_list": [[40.760, -73.984], ...],
        "landmark_list": ["Battery Park", "Plaza Hotel", ...]
      }
    },
    ...
  ],
  1000  // Total number of routes
]

Street View Cache:

nyc_1k_routes/street_views/
├─ pano_XXX_h000.jpg  # Heading 0° (Front)
├─ pano_XXX_h090.jpg  # Heading 90° (Right)
├─ pano_XXX_h180.jpg  # Heading 180° (Back)
└─ pano_XXX_h270.jpg  # Heading 270° (Left)

GPS Mapping:

# gps_pano_mapping.pkl
{
    (40.758, -73.985): "pano_ABC123",  # GPS -> Panorama ID
    (40.759, -73.984): "pano_DEF456",
    ...
}

9.2 Evaluation Datasets¶

Dataset	Type	Routes	Data Path	Purpose
NYC Test (In-Dist)	In-Distribution	48	`/root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/`	Test training distribution performance
NYC Test (Rule OOD)	Rule OOD	48	`/root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/`	Test relative action generalization
SF Routes (Visual OOD)	Visual OOD	18	`/root/SFTvsRL_Data/VIRL_routes/VLN_mini/`	Test visual environment generalization

Visual OOD Data Characteristics (San Francisco, VLN_mini): - Different architectural styles (Victorian vs Modern) - Different terrain (Hills vs Flat) - Different color distributions (Pastel houses vs Glass buildings) - Different landmark types (Cable cars, Golden Gate vs Yellow cabs, Statue of Liberty)

9.3 Data Statistics¶

NYC 1K Routes (Training + In-Dist/Rule OOD Evaluation):
- Location: /root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/
- Number of Routes: 1,000
- Street View Images: ~100,000
- Data Size: ~30-40 GB

VLN_mini (Visual OOD Evaluation):
- Location: /root/SFTvsRL_Data/VIRL_routes/VLN_mini/
- Number of Routes: 18 (San Francisco)
- Street View Images: ~2,000
- Data Size: ~1-2 GB

10. Vision Input Processing¶

10.1 Why Use 4-Direction Images Instead of Panoramas?¶

Problem: The Google Street View API provides 360° panoramas (equirectangular panorama). Why convert them into 4 static images?

Design Motivation¶

Computational Efficiency
Panorama: 2048×1024 or larger (~2-6 MB/image)
4 static images: 4 × 640×640 = ~1.2 MB
Storage and loading speed improved by 2-5x
Model Input Constraints
Vision Transformers are sensitive to input dimensions
Processing a 2×2 grid (2405×2405) is more aligned with standard model training than a panorama (2048×1024)
The 2×2 grid is nearly square, reducing padding and distortion
Task Relevance
Human navigation also focuses on the four cardinal directions: front, back, left, right
No need to look up or down (pitch = 0°)
The 4 directions contain all the information needed for navigation
Data Augmentation Flexibility
Each direction can be processed independently
Easily extensible to different heading configurations
Facilitates attention visualization (which direction is more important)
Consistency with V-IRL Original Design
The V-IRL paper (Yang et al., 2024) originally used the 4-direction design
Maintaining consistency facilitates comparison of experimental results

Technical Comparison¶

Approach	Panorama	4 Static Images
Resolution	2048×1024	4 × 640×640
File Size	2-6 MB	1.2 MB
FOV Coverage	360° × 180°	4 × 90° = 360° (Horizontal)
Processing Complexity	Requires equirectangular projection handling	Direct use
Model Adaptation	Requires special handling	Standard 2D CNN/ViT
Storage Cost	High (100K × 6MB = 600GB)	Low (100K × 1.2MB = 120GB)

Disadvantages of Panoramas¶

Distortion:
Equirectangular projection is severely distorted near the poles
Requires special preprocessing or model adaptation
Information Redundancy:
Sky and ground occupy many pixels but have low information value
Navigation primarily focuses on horizontal landmarks
Computational Overhead:
Larger images require more GPU memory
Training and inference speed decrease significantly

10.2 Street View Image Acquisition¶

def _get_visual_observation(self):
    """
    Get 4-direction street view images for the current location

    Returns:
        np.array: RGB image of shape [2405, 2405, 3]
    """
    # 1. Get 4 images from the Platform
    image_list = self.platform.get_all_streetview_from_geocode(
        geocode=self.current_geocode,
        cur_heading=self.current_heading
    )
    # image_list = [front, right, back, left] (each 640×640)

    # 2. Resize to a uniform resolution
    resized_images = [
        image.resize((self.resolution, self.resolution))  # 1200×1200
        for image in image_list
    ]

    # 3. Concatenate into a 2×2 grid
    line_width = 5  # Black separator line
    canvas = Image.new('RGB',
                       (self.resolution * 2 + line_width,   # 2405
                        self.resolution * 2 + line_width),  # 2405
                       (0, 0, 0))  # Black background

    # Place images:
    # [0,0] -> (0, 0)           Front
    # [1,0] -> (1205, 0)        Right
    # [0,1] -> (0, 1205)        Back
    # [1,1] -> (1205, 1205)     Left
    for i, image in enumerate(resized_images):
        x = (i % 2) * (self.resolution + line_width)
        y = (i // 2) * (self.resolution + line_width)
        canvas.paste(image, (x, y))

    return np.array(canvas)  # [2405, 2405, 3]

10.3 Visual Input Diagram¶

2×2 Street View Grid (2405 × 2405 pixels)

┌─────────────────────┬─────────────────────┐
│                     │                     │
│    Front View       │    Right View       │
│    (1200×1200)      │    (1200×1200)      │
│                     │                     │
│    Heading: 0°      │    Heading: 90°     │
│                     │                     │
├─────────────────────┼─────────────────────┤
│                     │                     │
│    Back View        │    Left View        │
│    (1200×1200)      │    (1200×1200)      │
│                     │                     │
│    Heading: 180°    │    Heading: 270°    │
│                     │                     │
└─────────────────────┴─────────────────────┘

Black separator line: 5 pixels

10.4 Llama-3.2-Vision Image Processing¶

def formulate_payload(self, question, obs=None):
    """
    Construct the input format for Llama-3.2-Vision

    Args:
        question: Text prompt
        obs: PIL.Image or np.array
    """
    self.payload = [
        {
            "role": "user",
            "content": [{"type": "text", "text": question}]
        }
    ]

    if obs is not None:
        # Convert to PIL Image
        if isinstance(obs, np.ndarray):
            obs = Image.fromarray(obs)

        # Insert at the beginning of content (Llama format requirement)
        self.payload[0]['content'].insert(0, {
            "type": "image",
            "image": obs
        })

def process_input(self, obs, prompt):
    """
    Process input using the processor
    """
    # 1. Apply chat template
    input_text = self.processor.apply_chat_template(
        self.payload,
        add_generation_prompt=True
    )

    # 2. Process image + text
    inputs = self.processor(
        obs,           # PIL Image
        input_text,    # Formatted prompt
        return_tensors="pt",
        add_special_tokens=False
    ).to(self.model.device)

    # inputs = {
    #     'input_ids': tensor([[..., image_tokens, ..., text_tokens]]),
    #     'attention_mask': tensor([[1, 1, ..., 1]]),
    #     'pixel_values': tensor([[[...]]]),  # Processed image features
    #     'cross_attention_mask': tensor([[[...]]])
    # }

    return inputs

10.5 Vision Encoder Processing Flow¶

Input Image (2405×2405×3)
    ↓
Llama-3.2-Vision Processor
    ↓
├─ Image Processor:
│  ├─ Resize to model input size
│  ├─ Normalize (mean=[0.48145466, 0.4578275, 0.40821073])
│  └─ Convert to tensor
│
└─ Vision Encoder (CLIP-based):
   ├─ Patch Embedding (16×16 patches)
   ├─ Vision Transformer Layers
   └─ Output: Visual tokens (sequence length ~1000)
       ↓
Cross-Attention with Language Model
    ↓
Language Decoder generates action JSON

# Image acquisition configuration
platform_cfg:
  STREET_VIEW:
    SIZE: [640, 640]        # Single raw image dimensions
    FOV: 90                 # Field of view (degrees)
    PITCH: 0                # Pitch angle (horizontal)
    SOURCE: outdoor         # Outdoor street view

# Environment configuration
env_config:
  resolution: 1200          # Resized dimension per image
  # Final input: 2×1200 + 5 = 2405 pixels

# Model configuration (Llama-3.2-Vision built-in)
model:
  vision_encoder:
    image_size: 560         # Model input size (auto-resized)
    patch_size: 14          # Patch embedding size
    hidden_size: 1280       # Vision hidden dimension

11. RL Training Process¶

11.1 Training Flow Overview¶

SFT Initialization
    ↓
┌────────────────────────────────────────┐
│   RL Training Loop (15 Updates)        │
│                                        │
│  For update in [0, 1, ..., 14]:       │
│                                        │
│  ┌──────────────────────────────────┐ │
│  │  Phase 1: Rollout (256 steps)   │ │
│  │  ├─ Multiple episodes            │ │
│  │  ├─ Collect (obs, action, reward)│ │
│  │  └─ Compute value predictions    │ │
│  └──────────────────────────────────┘ │
│            ↓                           │
│  ┌──────────────────────────────────┐ │
│  │  Phase 2: PPO Training (4 epochs)│ │
│  │  ├─ Compute advantages (GAE)     │ │
│  │  ├─ 4 epochs × 256 samples       │ │
│  │  ├─ Update value network         │ │
│  │  └─ Update learning rate         │ │
│  └──────────────────────────────────┘ │
│                                        │
└────────────────────────────────────────┘
    ↓
Save Final Checkpoint

11.2 PPO (Proximal Policy Optimization) Algorithm Principle¶

PPO is an on-policy reinforcement learning algorithm proposed by OpenAI in 2017. It ensures training stability by limiting the magnitude of policy updates and is currently one of the most popular RL algorithms.

11.2.1 Core Idea¶

Problem Background:

In policy gradient methods, we aim to maximize the expected return:

\[J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)]\]

Traditional Policy Gradient methods (e.g., REINFORCE) update the policy directly using gradient ascent, but suffer from two issues: 1. Low sample efficiency: Collected data can only be used once 2. Training instability: Large policy updates can lead to performance collapse

PPO's Solution:

PPO enables data reuse through importance sampling while using a clipping mechanism to limit the magnitude of policy updates, balancing sample efficiency and training stability.

11.2.2 Importance Sampling¶

Core question: How to use data collected by the old policy $\pi_{\theta_{old}}$ to update the new policy $\pi_\theta$?

Importance Sampling Formula:

\[\mathbb{E}_{a \sim \pi_{\theta_{old}}}[f(a)] = \mathbb{E}_{a \sim \pi_\theta}\left[\frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)} f(a)\right]\]

Here, $\frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)}$ is called the importance ratio.

Application in RL:

We can use trajectories collected by the old policy to estimate the expected return of the new policy:

\[J(\theta) \approx \sum_{t} r_t \cdot \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}\]

Problem: When $\pi_\theta$ diverges significantly from $\pi_{\theta_{old}}$, the variance of the importance ratio explodes, leading to training instability.

11.2.3 PPO-Clip Objective Function¶

PPO uses a clipping mechanism to constrain the range of the importance ratio, preventing overly large policy updates.

Define ratio:

\[r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}\]

PPO-Clip Objective Function:

\[L^{CLIP}(\theta) = \mathbb{E}_t\left[\min\left(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t\right)\right]\]

Where: - $\hat{A}_t$: advantage function, indicating how much better the action is compared to the average - $\epsilon$: clip parameter ($\epsilon=0.1$ in this experiment) - $\text{clip}(r, 1-\epsilon, 1+\epsilon)$: constrains $r$ to the range $[0.9, 1.1]$

Intuitive Understanding:

When Advantage > 0 (good action):
If $r_t > 1.1$: clipping limits increase, preventing over-optimism
If $0.9 < r_t < 1.1$: normal update
If $r_t < 0.9$: no penalty (probability has already decreased)
When Advantage < 0 (bad action):
If $r_t < 0.9$: clipping limits decrease, preventing over-pessimism
If $0.9 < r_t < 1.1$: normal update
If $r_t > 1.1$: no reward (probability has already increased)

Mathematical Expression:

\[ L^{CLIP}(\theta) = \mathbb{E}_t\left[\min\left( \begin{cases} r_t \hat{A}_t & \text{if } \hat{A}_t \geq 0 \\ \text{clip}(r_t, 1-\epsilon, 1+\epsilon) \hat{A}_t & \text{if } \hat{A}_t < 0 \end{cases} \right)\right] \]

This ensures the policy does not deviate too far from the old policy, thereby maintaining training stability.

11.2.4 Advantage Calculation (GAE)¶

Advantage Function Definition:

\[A^\pi(s_t, a_t) = Q^\pi(s_t, a_t) - V^\pi(s_t)\]

It measures how good it is to take action $a_t$ in state $s_t$ compared to the average.

Problem: We do not have the true $Q$ and $V$; they need to be estimated.

Generalized Advantage Estimation (GAE):

GAE is an advantage estimation method that balances bias and variance:

\[\hat{A}_t^{GAE(\gamma, \lambda)} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}\]

Where: - $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$: TD error (temporal difference error) - $\gamma$: discount factor ($\gamma=0.9$ in this experiment) - $\lambda$: GAE parameter ($\lambda=0.95$ in this experiment)

Recursive Calculation (backwards):

\[\hat{A}_t = \delta_t + (\gamma \lambda) \hat{A}_{t+1}\]

Intuitive Understanding:

$\lambda=0$: looks at only one-step TD error, low variance but high bias
$\lambda=1$: looks at the full return, low bias but high variance
$\lambda=0.95$: balances both, the best choice in practice

Return Calculation:

\[\hat{R}_t = \hat{A}_t + V(s_t)\]

This $\hat{R}_t$ is used to train the value network.

11.2.5 Value Function Loss¶

In addition to the policy loss, PPO also needs to train the value network to estimate state values:

\[L^{VF}(\theta) = \mathbb{E}_t\left[\max\left((V_\theta(s_t) - \hat{R}_t)^2, (\bar{V}_t - \hat{R}_t)^2\right)\right]\]

Where: - $V_\theta(s_t)$: current value estimate - $\hat{R}_t$: target return (computed from GAE) - $\bar{V}_t = V_{\theta_{old}}(s_t) + \text{clip}(V_\theta(s_t) - V_{\theta_{old}}(s_t), -\epsilon, \epsilon)$: clipped value

Purpose of Clipping:

Prevents the value function from updating too aggressively, similar stability considerations as policy clipping.

11.2.6 Entropy Bonus¶

To encourage exploration, PPO adds an entropy bonus:

\[L^{ENT}(\theta) = \mathbb{E}_t[H(\pi_\theta(\cdot|s_t))]\]

Where $H$ is the entropy of the policy distribution. Higher entropy means the policy is more random and exploratory.

11.2.7 Total Loss Function¶

The final loss function for PPO is a weighted sum of three terms:$$L(\theta) = -L^{CLIP}(\theta) + c_1 L^{VF}(\theta) - c_2 L^{ENT}(\theta)$$

In this experiment: - $c_1 = 0.5$: value loss coefficient - $c_2 = 0.01$: entropy coefficient

Training Process:

Collect 256-step trajectory data (using $\pi_{\theta_{old}}$)
Compute advantage and return for all time steps
Train for 4 epochs on these 256 samples:
Each epoch iterates over all samples
Compute loss and update parameters for each sample
Update once after accumulating gradients for 128 steps
Update $\theta_{old} \leftarrow \theta$, proceed to the next round

11.2.8 Why is PPO Suitable for VLM Training?¶

Stability: The clipping mechanism prevents large updates, protecting expensive pretrained models
Sample Efficiency: Each batch of data can be trained for multiple epochs (4 epochs in this experiment)
Simplicity: Compared to TRPO, PPO does not require complex second-order optimization
Scalability: Easily integrates with distributed training frameworks like DeepSpeed

11.2.9 PPO Configuration in This Experiment¶

ppo_config:
  clip_param: 0.1           # ε = 0.1, limits ratio to [0.9, 1.1]
  ppo_epoch: 4              # Train 4 epochs per batch of data
  mini_batch_size: 1        # Train sample by sample (due to high VLM memory usage)
  value_loss_coef: 0.5      # c_1, value loss weight
  entropy_coef: 0.01        # c_2, entropy bonus weight
  max_grad_norm: 0.01       # Gradient clipping threshold (very small for stability)

11.3 Detailed PPO Code Implementation¶

11.3.1 Advantage Calculation (GAE) Code¶

def compute_returns(self, next_value, gamma=0.9, gae_lambda=0.95):
    """
    Use Generalized Advantage Estimation (GAE)
    to compute return and advantage for each step

    Args:
        next_value: value prediction for the last step
        gamma: discount factor
        gae_lambda: GAE λ parameter
    """
    self.value_preds[-1] = next_value
    gae = 0

    # Compute backwards
    for step in reversed(range(self.num_steps)):  # 255 → 0
        # TD error
        delta = (self.rewards[step] +
                gamma * self.value_preds[step + 1] * self.masks[step + 1] -
                self.value_preds[step])

        # GAE accumulation
        gae = delta + gamma * gae_lambda * self.masks[step + 1] * gae

        # Return = Advantage + Value
        self.returns[step] = gae + self.value_preds[step]

    # Normalize advantages (for training stability)
    advantages = self.returns[:-1] - self.value_preds[:-1]
    advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-5)

    return advantages

11.3.2 PPO Loss Calculation Code¶

def ppo_update(self, rollouts):
    """
    PPO training: 4 epochs × 256 samples

    Each epoch iterates over all 256 samples (mini_batch_size=1)
    """
    advantages = rollouts.compute_returns(next_value)
    grad_accum_steps = 128
    optimizer.zero_grad()

    for epoch in range(4):  # ppo_epoch = 4
        for sample_idx in range(256):  # num_steps = 256
            # 1. Get sample
            obs_batch = rollouts.obs[sample_idx]
            old_action_log_prob = rollouts.action_log_probs[sample_idx]
            return_batch = rollouts.returns[sample_idx]
            value_pred_old = rollouts.value_preds[sample_idx]
            advantage = advantages[sample_idx]

            # 2. Re-evaluate (with gradients)
            new_value, new_action_log_prob, entropy = actor_critic.evaluate_actions(
                **obs_batch['io_dict']
            )
            entropy_loss = -entropy.mean()

            # 3. Compute probability ratio
            ratio = torch.exp(new_action_log_prob - old_action_log_prob)

            # 4. Policy Loss (Clipped Surrogate Objective)
            surr1 = ratio * advantage
            surr2 = torch.clamp(ratio, 1.0 - clip_param, 1.0 + clip_param) * advantage

            # Ratio clipping protection (prevent gradient explosion)
            if torch.any(ratio > 10):
                policy_loss = -surr2.mean()
            else:
                policy_loss = -torch.min(surr1, surr2).mean()

            # 5. Value Loss (Clipped)
            value_pred_clipped = (value_pred_old +
                torch.clamp(new_value - value_pred_old,
                           -clip_param, clip_param))

            value_losses = (new_value - return_batch).pow(2)
            value_losses_clipped = (value_pred_clipped - return_batch).pow(2)
            value_loss = 0.5 * torch.max(value_losses,
                                        value_losses_clipped).mean()

            # 6. Total Loss
            loss = (value_loss * value_loss_coef +  # 0.5
                   policy_loss +
                   entropy_loss * entropy_coef)      # 0.01

            # 7. Backward & Update
            accelerator.backward(loss / grad_accum_steps)

            should_step = ((sample_idx + 1) % grad_accum_steps == 0 or
                           sample_idx + 1 == 256)
            if should_step:
                accelerator.clip_grad_norm_(
                    actor_critic.parameters(),
                    max_grad_norm  # 0.01
                )
                optimizer.step()
                optimizer.zero_grad()

11.4 Model Architecture¶

class VLMValue(nn.Module):
    """
    Value Network: Used to estimate state value
    """
    def __init__(self, base):
        super().__init__()
        self.base = base  # Llama-3.2-11B-Vision (frozen generation path)

        # 3-layer MLP value head
        self.value_head = nn.Sequential(
            nn.Linear(4096, 1024),
            nn.ReLU(),
            nn.Linear(1024, 512),
            nn.ReLU(),
            nn.Linear(512, 1)
        ).to(base.device, dtype=torch.bfloat16)

    def forward(self, inputs):
        # Forward pass to get hidden states
        outputs = self.base(**inputs, output_hidden_states=True)
        hidden_states = outputs.hidden_states  # All layers

        # Use the last token of the last layer
        last_hidden = hidden_states[-1][:, -1]  # [batch, 4096]

        # Value prediction
        values = self.value_head(last_hidden)  # [batch, 1]
        return values

class VLMPolicy(nn.Module):
    """
    Policy Network: Wraps Value Network + Generation
    """
    def __init__(self, tokenizer, value_model, generation_config):
        super().__init__()
        self.tokenizer = tokenizer
        self.value_model = value_model
        self.base = value_model.base
        self.temperature = generation_config.temperature
        self.max_new_tokens = generation_config.max_new_tokens

    def act_oneline(self, inputs, obs=None):
        """
        Generate action (inference, no gradients)

        Returns:
            values: state value estimates
            io_dict: input/output dictionary (for subsequent training)
            output_text: decoded JSON text
            action_log_prob: action log probability
        """
        with torch.no_grad():
            # 1. Generation
            outputs = self.base.generate(
                **inputs,
                max_new_tokens=self.max_new_tokens,
                temperature=self.temperature,
                output_scores=True,
                output_hidden_states=True,
                return_dict_in_generate=True
            )

            output_ids = outputs['sequences'][:, inputs['input_ids'].shape[1]:]
            output_text = self.tokenizer.decode(output_ids[0],
                                               skip_special_tokens=True)

            # 2. Concatenate input + output
            cated_io = torch.cat((inputs['input_ids'], output_ids), dim=1)

            # 3. Prepare inputs for evaluation
            io_dict = self._prepare_io_dict(inputs, output_ids)

            # 4. Evaluate to get value & log_prob
            values, sum_log_prob, action_tokens_log_prob = \
                self.evaluate(**io_dict, inference=True)

        return values, io_dict, output_text, sum_log_prob, action_tokens_log_prob

    def evaluate_actions(self, **io_dict):
        """
        Re-evaluate actions (training, with gradients)

        Returns:
            values: new value estimates
            action_log_probs: new log probabilities
            entropy: policy distribution entropy (for the exploration bonus)
        """
        # 1. Forward pass with gradients
        outputs = self.base(
            **io_dict['new_inputs'],
            output_hidden_states=True
        )

        # 2. Compute value
        hidden_states = outputs.hidden_states[-1][:, -1]
        values = self.value_model.value_head(hidden_states)

        # 3. Compute log probabilities
        logits = outputs.logits
        output_ids = io_dict['io_pair'][1]  # Generated tokens

        action_log_probs = self._compute_log_probs(logits, output_ids)

        # 4. Compute policy entropy over the generated tokens
        dist = torch.distributions.Categorical(logits=logits)
        entropy = dist.entropy().mean()

        return values, action_log_probs, entropy

11.5 DeepSpeed ZeRO Distributed Training Principles¶

This experiment uses DeepSpeed ZeRO Stage 2 for distributed training across 8 GPUs. DeepSpeed is a deep learning optimization library developed by Microsoft, with ZeRO (Zero Redundancy Optimizer) as its core technology.

11.5.1 Why DeepSpeed?¶

Memory Challenges:

Training the 11B-parameter Llama-3.2-Vision model faces significant memory pressure:

Model Parameters:
- Base Model: 11B × 2 bytes (bf16) = 22 GB
- Value Head: 4096×1024 + 1024×512 + 512×1 ≈ 5M × 2 = 10 MB

Optimizer States (Adam):
- Momentum: 11B × 4 bytes (fp32) = 44 GB
- Variance: 11B × 4 bytes (fp32) = 44 GB

Gradients:
- Gradients: 11B × 2 bytes (bf16) = 22 GB

Activations:
- Forward pass: ~20-40 GB (depends on batch size)
- Backward pass: ~20-40 GB

Total: ~170-190 GB

A single H800 GPU has only 80 GB of memory, insufficient to hold the complete training state!

Problems with Traditional Data Parallelism:

Traditional DDP (Distributed Data Parallel) stores a complete model replica on each GPU:

GPU 0: [Full Model + Full Optimizer State + Gradients] = 170 GB ✗
GPU 1: [Full Model + Full Optimizer State + Gradients] = 170 GB ✗
...
GPU 7: [Full Model + Full Optimizer State + Gradients] = 170 GB ✗

Memory Redundancy Issue: - Identical optimizer states stored across 8 GPUs (8 × 88 GB = 704 GB redundant!) - Each GPU still requires 170 GB, exceeding single-card capacity

11.5.2 Core Idea of ZeRO¶

Zero Redundancy Optimizer reduces memory usage by eliminating redundancy:

Core Principle: Partitioning + Communication

Instead of storing redundant data:
1. Partition data across different GPUs
2. Reconstruct complete data via communication (All-Gather) when needed
3. Trade communication time for memory space

Three Stages of ZeRO:

Stage	Partitioned Content	Memory Savings	Communication Overhead
ZeRO-1	Optimizer States	~4×	Low
ZeRO-2	+ Gradients	~8×	Medium
ZeRO-3	+ Parameters	~64×	High

This experiment uses ZeRO Stage 2, balancing memory savings and communication overhead.

11.5.3 Detailed Mechanism of ZeRO Stage 2¶

Memory Partitioning Strategy:

# 8 GPUs, each GPU stores only 1/8 of optimizer states and gradients

GPU 0:
  - Full Model Parameters (22 GB)
  - Optimizer States [0:N/8] (44/8 = 5.5 GB)
  - Gradients [0:N/8] (22/8 = 2.75 GB)
  - Activations (~20 GB)
  → Total: ~50 GB ✓

GPU 1:
  - Full Model Parameters (22 GB)
  - Optimizer States [N/8:2N/8] (5.5 GB)
  - Gradients [N/8:2N/8] (2.75 GB)
  - Activations (~20 GB)
  → Total: ~50 GB ✓

... (GPU 2-7 similar)

Training Workflow:

Forward Pass

Each GPU computes independently:
- Input: its own batch (total batch / 8)
- Uses: full model parameters (same on all GPUs)
- Output: its own loss and activations

Backward Pass

Each GPU computes gradients independently:
GPU i: ∂L/∂θ (full gradients)

Then Reduce-Scatter:
GPU i: keep only ∂L/∂θ[i×N/8:(i+1)×N/8]
→ Each GPU stores only 1/8 of the gradients

Optimizer Step

Each GPU updates its partition of optimizer states using its gradient
partition. Model parameters remain fully replicated on every GPU, and the
updated parameter values are synchronized across the data-parallel group.

Key Communication Operations:

Reduce-Scatter (Gradient Aggregation + Partitioning)

Input: Full gradients on each GPU
Operation: Sum and partition
Output: Each GPU gets 1/8 of the aggregated gradients

Time Complexity: O(N/P) where P=8

Parameter Synchronization

ZeRO-2 does not partition model parameters. Each GPU retains a complete
parameter replica while optimizer states and gradients are sharded.

11.5.4 Optimizer Offload Mechanism¶

This experiment also uses CPU Offloading:

deepspeed_config:
  offload_optimizer_device: cpu    # Offload optimizer states to CPU
  offload_param_device: none       # Do not offload parameters

How It Works:

During training:
1. Optimizer States are stored in CPU memory (cheap and large capacity)
2. When an update is needed, they are transferred to the GPU for computation
3. After the update is complete, they are transferred back to the CPU

Memory Distribution:
GPU: Model Parameters (22 GB) + Gradients (2.75 GB) + Activations (20 GB) ≈ 45 GB ✓
CPU: Optimizer States (5.5 GB per GPU) → Does not occupy GPU memory

Trade-offs: - ✓ Saves GPU memory: ~5.5 GB per GPU - ✗ Increases training time: CPU-GPU transfer overhead (~10-15%)

For the 11B model, this trade-off is worthwhile as it prevents OOM (Out of Memory).

11.5.5 Gradient Accumulation¶

In conjunction with ZeRO, this experiment uses gradient accumulation over 128 steps:

grad_accum_steps: 128

Purpose: Simulate a larger batch size

Actual Workflow:
for i in range(128):
    # Forward & Backward (do not update parameters)
    loss = model(batch_i)
    loss.backward()  # Accumulate gradients

# Update only after 128 accumulations
optimizer.step()  # Use accumulated gradients
optimizer.zero_grad()

Effective Batch Size:

Per-GPU Batch Size: 1
Num GPUs: 8
Grad Accum Steps: 128

Effective Batch Size = 1 × 8 × 128 = 1024

Why Gradient Accumulation?

Memory Constraints: Batch size = 1 is the maximum the VLM can handle
Training Stability: Large batch size (1024) helps stabilize RL training
Sample Efficiency: PPO requires a sufficiently large batch to estimate advantage

11.5.6 Mixed Precision Training (BF16)¶

mixed_precision: bf16
downcast_bf16: 'yes'

BFloat16 vs Float32:

Type	Bits	Exponent Bits	Mantissa Bits	Range	Precision
FP32	32	8	23	±3.4×10³⁸	High
BF16	16	8	7	±3.4×10³⁸	Medium
FP16	16	5	10	±6.5×10⁴	Medium

Advantages of BF16: - ✓ Memory halved: 44 GB → 22 GB - ✓ Computation accelerated: ~2× on H800 - ✓ Large dynamic range: Same as FP32 (prevents overflow) - ✓ No loss scaling needed (simpler than FP16)

Training Flow:

# Forward & Backward in BF16
with autocast(dtype=torch.bfloat16):
    output = model(input)
    loss = criterion(output, target)

loss.backward()  # Gradients in BF16

# Optimizer in FP32 (Master Weights)
optimizer.step()  # Update FP32 parameters
model.to(torch.bfloat16)  # Convert back to BF16 for next forward pass

11.5.7 Actual Memory Usage Analysis¶

Single GPU Memory Distribution (ZeRO-2 + Offload + BF16):

Model Parameters (BF16): 11B × 2 bytes = 22 GB
Gradients (BF16, 1/8): 11B × 2 / 8 = 2.75 GB
Activations (BF16): ~15-20 GB
Optimizer States (CPU Offload): 0 GB (on CPU)
Temporary Buffers: ~5 GB

Total per GPU: ~45-50 GB / 80 GB = 56-62% Utilization ✓

Comparing Different Configurations:

Configuration	Memory Usage	Feasible
Single GPU, No Optimization	170 GB	✗ OOM
DDP, 8 GPUs	170 GB each	✗ OOM
ZeRO-2, 8 GPUs	55 GB each	✓
ZeRO-2 + Offload	50 GB each	✓
ZeRO-3	30 GB each	✓ (but slower communication)

Communication Overhead:

Reduce-Scatter: ~0.1 seconds per step
All-Gather: ~0.1 seconds per step
Total Overhead: ~10-15% of training time
Trade-off: Acceptable (compared to being unable to train)

11.5.8 DeepSpeed Configuration Details¶

# scripts/config_zero2_8gpu.yaml

compute_environment: LOCAL_MACHINE
distributed_type: DEEPSPEED

# Core Configuration
deepspeed_config:
  zero_stage: 2                      # ZeRO Stage 2
  offload_optimizer_device: cpu      # Optimizer → CPU
  offload_param_device: none         # Parameters stay on GPU
  zero3_init_flag: false             # Do not use ZeRO-3 initialization
  overlap_comm: false                # Do not overlap communication with computation (more stable)

# Precision Configuration
mixed_precision: bf16                # BFloat16 Mixed Precision
downcast_bf16: 'yes'                 # Automatic conversion

# Distributed Configuration
num_machines: 1                      # Single node
num_processes: 8                     # 8 GPUs
rdzv_backend: static                 # Static topology (no dynamic joining)
same_network: true                   # Same network (low latency)

Why Not ZeRO-3?

ZeRO-3 saves more memory but has higher communication overhead:

Stage	Communications/Step	Data Volume/Step	Training Speed
ZeRO-2	2	22 GB	1.0×
ZeRO-3	4	44 GB	0.6×

For an 11B model, ZeRO-2 is sufficient, and the extra overhead of ZeRO-3 is unnecessary.

11.5.9 Actual Training Performance¶

Throughput:

Rollout Phase (256 steps):
- Time: ~40 minutes
- Speed: 6.4 steps/min
- Bottleneck: Environment interaction + Model inference

PPO Training Phase (4 epochs × 256 samples):
- Time: ~20 minutes
- Speed: 51.2 samples/min
- Bottleneck: Gradient computation + Communication

Total per Update: ~60 minutes
Total Training (15 updates): ~15 hours

Scalability Analysis:

Number of GPUs	Theoretical Speedup	Actual Speedup	Efficiency
1	1.0×	1.0×	100%
2	2.0×	1.8×	90%
4	4.0×	3.4×	85%
8	8.0×	6.2×	78%

The efficiency loss is mainly due to communication overhead, but a 6× speedup is still achieved.

11.6 Training Monitoring Metrics¶

Key Metrics Logged to WandB:

wandb.log({
    # Training Progress
    'total_num_steps': total_steps,
    'compute_tokens': token_count,

    # Loss
    'value_loss': value_loss,
    'action_loss': policy_loss,
    'dist_entropy': entropy,

    # Reward Statistics
    'reward.mean': rewards.mean(),
    'reward.std': rewards.std(),
    'reward.max': rewards.max(),
    'reward.min': rewards.min(),

    # Value Statistics
    'value.mean': values.mean(),
    'value.std': values.std(),

    # Return Statistics
    'return.mean': returns.mean(),
    'return.std': returns.std(),

    # Episode Statistics
    'episode_rewards.mean': np.mean(episode_rewards),
    'success_rate': success_rate,
    'per_step_accuracy': step_accuracy
})

12. Hyperparameter Configuration¶

12.1 Core Hyperparameter Overview¶

Category	Parameter	Value	Description
Training Scale	`num_updates`	15	Total training rounds
	`num_steps`	256	Steps collected per round
	`ppo_epoch`	4	PPO training epochs
	`grad_accum_steps`	128	Gradient accumulation steps
Learning Rate	`init_lr`	1e-7	Initial learning rate
	`lr_max_steps`	20	Total steps for LR scheduler
	`end_lr`	1e-9	Final learning rate
PPO	`clip_param`	0.1	PPO clip range
	`value_loss_coef`	0.5	Value loss coefficient
	`entropy_coef`	0.01	Entropy coefficient
	`max_grad_norm`	0.01	Gradient clipping threshold
GAE	`gamma`	0.9	Discount factor
	`gae_lambda`	0.95	GAE λ
Environment	`verify_iter`	2	Number of verification attempts
	`resolution`	1200	Image resolution
Generation	`temperature`	0.2	Generation temperature
	`max_new_tokens`	512	Maximum generation length

12.2 Complete Configuration File¶

# rl/configs/llama_virl_vl.yaml

trainer: LlamaTrainer

# Gradient Accumulation Configuration
grad_accum_steps: 128

# Optimizer Configuration
optimizer_config:
  init_lr: !!float 1e-6      # Will be overridden by script to 1e-7
  eps: !!float 1e-7
  weight_decay: 0
  lr_max_steps: 100           # Will be overridden by script to 20
  end_lr: !!float 1e-9

# PPO Configuration
ppo_config:
  clip_param: 0.1             # ε in PPO clip
  ppo_epoch: 4                # PPO training epochs per round
  mini_batch_size: 1          # Batch size
  value_loss_coef: 0.5        # Value loss weight
  entropy_coef: 0.01          # Entropy bonus weight
  max_grad_norm: 0.01         # Gradient clipping

# Return Computation Configuration
compute_return_kwargs:
  use_gae: true               # Use GAE
  gamma: 0.9                  # Discount factor γ
  gae_lambda: 0.95            # GAE λ
  use_proper_time_limits: False

# Training Configuration
report_to: wandb              # Log to WandB
run_name: "virl_vl_training"
num_steps: 512                # Will be overridden by script to 256
num_processes: 1
num_updates: 20               # Will be overridden by script to 15

# Environment Configuration
env_config:
  id: 'gym_virl/Navigation-v0'
  route_info_path: ""
  resolution: 1200
  verify_iter: 2
  absolute_action: true
  relocation: true
  drop_rate: 0.5
  straight_line_length: 5

  platform_cfg:
    STREET_VIEW:
      SIZE: [640, 640]
      HEADING: 0
      PITCH: 0
      FOV: 90
      SOURCE: outdoor

    OFFLINE:
      ENABLED: True
      PANORAMA_DIR: ""
      GPS_TO_PANO_PATH: ""
      MAPPING_RADIUS: 20

  platform_save_dir: "./logs/"

# Model Configuration
model: llama
model_path: ""

# Prompt Configuration
prompt_config:
  relocation: true
  use_vision: true
  use_language: false
  enable_verification: true
  prompt_vision: ["Q_VIRL_VL"]
  pattern_vision: ["action"]

# Generation Configuration
generation_config:
  temperature: 0.2
  max_tokens: 300
  max_new_tokens: 512
  thought_prob_coef: 0.5
  num_beams: 1

# Output Configuration
output_dir: logs/train.jsonl
seed: 42
save_ckpt: False
save_every: 1

12.3 Training Script Override Parameters¶

# scripts/virl_training/vl_train.sh

# Training Parameters
LR=1e-7
save_model=True
save_every=5  # Save every 5 updates
CKPT_NAME="tianzhechu/VIRL-VL-Init"
PORT=$((RANDOM % 10000 + 1000))

# Data Path (using absolute paths)
BASE_DIR="/root/SFTvsRL_Data/VIRL_routes"
ROUTE_INFO="${BASE_DIR}/nyc_1k_routes/route_infos.json"
GPS_TO_PANO="${BASE_DIR}/nyc_1k_routes/gps_pano_mapping.pkl"
STREETVIEWS="${BASE_DIR}/nyc_1k_routes/street_views/"

# Start training
DS_SKIP_CUDA_CHECK=1 TOKENIZERS_PARALLELISM=false \
    accelerate launch \
    --config_file scripts/config_zero2_8gpu.yaml \
    --main_process_port ${PORT} -m rl.launcher \
    -f rl/configs/llama_virl_vl.yaml \
    --output_dir=train_ckpt/virl_vl/ \
    --optimizer_config.init_lr=${LR} \
    --optimizer_config.lr_max_steps=20 \
    --prompt_config.enable_verification=True \
    --num_updates=15 \
    --num_steps=256 \
    --model_path=${CKPT_NAME} \
    --save_ckpt=${save_model} \
    --save_every=${save_every} \
    --env_config.route_info_path=${ROUTE_INFO} \
    --env_config.platform_cfg.OFFLINE.PANORAMA_DIR=${STREETVIEWS} \
    --env_config.platform_cfg.OFFLINE.GPS_TO_PANO_PATH=${GPS_TO_PANO}

12.4 DeepSpeed ZeRO-2 Configuration¶

# scripts/config_zero2_8gpu.yaml

compute_environment: LOCAL_MACHINE

deepspeed_config:
  offload_optimizer_device: cpu    # Offload optimizer to CPU
  offload_param_device: none       # Do not offload parameters
  zero3_init_flag: false
  zero_stage: 2                    # ZeRO Stage 2
  overlap_comm: false

distributed_type: DEEPSPEED
downcast_bf16: 'yes'               # BF16 mixed precision
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 8                   # 8 GPUs
rdzv_backend: static
same_network: true
use_cpu: false

12.5 Key Hyperparameter Explanations¶

Why is lr=1e-7 so small? - RL training requires stability; a large learning rate can cause policy collapse - The value network is initialized from scratch and needs careful training - Paper experiments verify: 1e-7 > 1e-6 (more stable)

Why is lr_max_steps=20 so short? - With only 15 updates, a 20-step LR schedule covers the entire training - lr_scheduler.step() is called after each update - Actual: ~1.3 updates per LR step (20/15)

Why is max_grad_norm=0.01 so small? - RL training is prone to gradient explosion (importance sampling) - Strict gradient clipping ensures training stability - Paper ablation studies confirm this value is optimal

Why verify_iter=2? - Balances exploration and efficiency: 2 attempts are sufficient for learning - Too many attempts → slow training, information redundancy - Too few attempts → difficulty recovering from errors, unstable training

13. Evaluation Methods¶

13.1 Evaluation Pipeline¶

# 1. In-Distribution Evaluation
bash scripts/virl_evaluation/vl_indist_eval.sh

# 2. Rule OOD Evaluation
bash scripts/virl_evaluation/vl_rule_ood_eval.sh

# 3. Visual OOD Evaluation
bash scripts/virl_evaluation/vl_visual_ood_eval.sh

13.2 Evaluation Configuration Comparison¶

Configuration	In-Dist	Rule OOD	Visual OOD
`num_traj`	48	48	18
`absolute_action`	True	False	True
`route_info_path`	NYC routes	NYC routes	SF routes
`verify_iter`	2	2	2
Number of GPUs	1	1	1

13.3 Evaluation Metrics¶

13.3.1 Per-Step Accuracy¶

Definition: Accuracy of individual actions

per_step_accuracy = (
    num_correct_actions / total_actions
) × 100%

Example:

Route 1: 15 steps, 13 correct → 13/15 = 86.7%
Route 2: 20 steps, 18 correct → 18/20 = 90.0%
...
Route 48: 12 steps, 10 correct → 10/12 = 83.3%

Overall Per-Step Accuracy =
    (13 + 18 + ... + 10) / (15 + 20 + ... + 12) = 87.5%

13.3.2 Success Rate¶

Definition: Complete route success rate

success_rate = (
    num_successful_routes / total_routes
) × 100%

Success Conditions: 1. Reaches the correct destination 2. All waypoints are passed within the allowed number of attempts 3. Does not exceed the maximum number of steps

Example:

48 routes:
- 35 routes: Success ✓
- 10 routes: Partial failure (some waypoint errors) ✗
- 3 routes: Complete failure (did not reach destination) ✗

Success Rate = 35 / 48 = 72.9%

13.3.3 Other Metrics¶

metrics = {
    'mean_reward': Average total reward per route,
    'std_reward': Standard deviation of reward,
    'mean_steps': Average number of steps,
    'mean_verification_steps': Average number of verification steps (including retries)
}

13.4 Evaluation Output Example¶

// logs/virl_vl_indist_verify_2/virl_vl_indist.jsonl

// Route 1, Step 0
{"sample_id": 0, "veri_step": 0, "output": "{\"action\": \"turn_direction(south)\"}", "reward": 1, "info": {...}}

// Route 1, Step 1
{"sample_id": 0, "veri_step": 1, "output": "{\"action\": \"forward()\"}", "reward": 1, "info": {...}}

// ... more steps ...

// Route 1, Final step
{"sample_id": 0, "veri_step": 14, "output": "{\"action\": \"stop()\"}", "reward": 1, "info": {...}}

// Route 1 Summary
{"Success": true, "sample_id": 0, "output": "{\"action\": \"stop()\"}", "reward": 15, "info": {...}}
{"Split": "===================="}

// Route 2, Step 0 (Failed attempt)
{"sample_id": 1, "veri_step": 0, "output": "{\"action\": \"turn_direction(north)\"}", "reward": -1, "info": {"Verify Info": "Incorrect action..."}}

// Route 2, Step 1 (Retry, Success)
{"sample_id": 1, "veri_step": 1, "output": "{\"action\": \"turn_direction(south)\"}", "reward": 1, "info": {...}}

// ... 47 more routes ...

// Overall Statistics
{
  "mean_reward": 12.5,
  "std_reward": 3.2,
  "success_rate": 0.729,
  "per_step_accuracy": 0.875,
  "mean_steps": 14.2,
  "mean_verification_steps": 1.15
}

13.5 Detailed Evaluation Script¶

#!/bin/bash
# scripts/virl_evaluation/vl_indist_eval.sh

VITER=2                    # Number of verification attempts
ENABLE=True                # Enable verification mechanism
ABS=True                   # Use absolute action space
NUM_TRAJ=48                # Evaluate 48 routes
CKPT_NAME="train_ckpt/virl_vl/checkpoint-epoch-14"  # Trained checkpoint
OUTPUT_FOLDER="logs/virl_vl_indist_verify_${VITER}"
PORT=$((RANDOM % 10000 + 2000))

# Data paths (using absolute paths)
BASE_DIR="/root/SFTvsRL_Data/VIRL_routes"
ROUTE_INFO="${BASE_DIR}/nyc_1k_routes/route_infos.json"
GPS_TO_PANO="${BASE_DIR}/nyc_1k_routes/gps_pano_mapping.pkl"
STREETVIEWS="${BASE_DIR}/nyc_1k_routes/street_views/"

# Evaluate using 1 GPU
DS_SKIP_CUDA_CHECK=1 accelerate launch \
    --config_file scripts/config_zero2_1gpu.yaml \
    --main_process_port ${PORT} \
    -m evaluation.launcher \
    -f evaluation/configs/llama_virl_vl.yaml \
    --model_path=${CKPT_NAME} \
    --output_dir=${OUTPUT_FOLDER}/virl_vl_indist.jsonl \
    --env_config.route_info_path=${ROUTE_INFO} \
    --env_config.platform_cfg.OFFLINE.PANORAMA_DIR=${STREETVIEWS} \
    --env_config.platform_cfg.OFFLINE.GPS_TO_PANO_PATH=${GPS_TO_PANO} \
    --prompt_config.enable_verification=${ENABLE} \
    --env_config.verify_iter=${VITER} \
    --env_config.absolute_action=${ABS} \
    --num_traj=${NUM_TRAJ}

14. Experimental Results Analysis¶

14.1 Key Findings¶

Based on the paper's Figure 1 and experimental data:

14.1.1 In-Distribution Performance¶

Model	Per-Step Accuracy	Success Rate
SFT	~85%	~60%
RL (PPO)	~90%	~75%

Conclusion: RL outperforms SFT on the training distribution as well (+5% step accuracy)

14.1.2 Rule OOD Generalization¶

Model	In-Dist	Rule OOD	Generalization Gap
SFT	85%	15%	-70%
RL	90%	70%	-20%

Key Findings: - SFT collapses on Rule OOD (from 85% → 15%), indicating memorization of training data - RL maintains 70% accuracy, indicating learning of generalizable rules

14.1.3 Visual OOD Generalization¶

Model	NYC (In-Dist)	SF (Visual OOD)	Generalization Gap
SFT	85%	<10%	-75%
RL	90%	~60%	-30%

Key Findings: - SFT nearly fails in a different city, indicating overfitting to visual features - RL maintains 60% accuracy in SF, indicating learning of transferable visual representations

14.2 Why Does RL Generalize?¶

The paper analyzes the generalization mechanism of RL through ablation experiments:

14.2.1 The Role of Outcome-based Reward¶

Experiment Design: - RL-Process: Feedback given at each step (+1/-1) - RL-Outcome: Reward given only at episode end

Results: - RL-Outcome achieves higher accuracy on visual recognition (GeneralPoints-VL) - This indicates that outcome-based reward forces the model to learn better visual representations

Principle:

Process Reward:
  Step 1: +1 (correct action, but maybe wrong reasoning)
  → Model may rely on shortcuts (e.g., memorized patterns)

Outcome Reward:
  All steps: 0, 0, 0, ..., +10 (final success)
  → Model must learn end-to-end reasoning, including visual understanding

14.2.2 SFT as Format Teacher¶

Experiment: Train RL directly from the base model

Result: Failure (see Figure 20 of the paper) - Model cannot output structured JSON - Generates lengthy code snippets - Fails to converge

Conclusion: - SFT stabilizes output format ("format teacher") - RL builds strategy and generalization ability on top of this - SFT + RL is the optimal combination

14.2.3 Training Curve Comparison¶

Per-Step Accuracy over Training

SFT:
  Update 0-5:   Rapid rise (0% → 80%)
  Update 5-10:  Continued rise (80% → 85%)
  Update 10-20: Overfitting begins (85% → 85%)

  Rule OOD: Continuous decline (80% → 15%)
  → Memorizes training rules

RL:
  Update 0-5:   Steady rise (85% → 88%)
  Update 5-10:  Continued rise (88% → 90%)
  Update 10-15: Remains stable (90% → 90%)

  Rule OOD: Synchronous rise (60% → 70%)
  → Learns generalizable rules

14.3 V-IRL Mini Benchmark SOTA¶

The paper achieves SOTA on the official V-IRL benchmark:

Method	Success Rate
GPT-4V (Yang et al., 2024)	44.0%
RL (Ours)	77.8%
Improvement	+33.8%

Explanation: - Multi-round RL training significantly improves navigation ability - RL's generalization advantage is particularly evident in complex real-world environments

14.4 Failure Case Analysis¶

The paper provides two categories of failure cases:

14.4.1 RL Failure Without SFT Initialization¶

Problem: Direct RL training produces unstructured output

Example Output:
"To solve this problem, we can use a brute force approach
by generating all possible combinations... [generates Python code]"

Reason: The base model has not undergone instruction fine-tuning and does not understand the task format

14.4.2 RL Failure from Overfitted Checkpoint¶

Problem: Starting RL from a severely overfitted SFT checkpoint

Example:
  Rule: Relative actions
  Model Output: "turn_direction(northwest)"  # Still using absolute!

Reason: SFT overfitting is too deep; RL cannot correct it

Implications: - The training extent of SFT and RL needs to be balanced - SFT should not be trained for too long (to avoid overfitting) - Paper suggests: Train SFT only until a reasonable output format is achieved

15. References¶

15.1 Papers¶

Main Paper: Chu, T., Zhai, Y., Yang, J., et al. (2025). SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training. ICML 2025. arXiv:2501.17161
V-IRL Environment: Yang, J., et al. (2024). V-IRL: Grounding Virtual Intelligence in Real Life. V-IRL Platform
RL4VLM: Zhai, Y., et al. (2024). Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning. RL4VLM

15.2 Code Repositories¶

Recommended to use (): bojieli/SFTvsRL ⭐
Official Implementation: LeslieTrue/SFTvsRL
Project Page: https://tianzhechu.com/SFTvsRL
Dataset: HuggingFace - SFTvsRL_Data
Model Checkpoints: HuggingFace - SFTvsRL Models

⚠️ Note: Please use the bojieli fork version, which fixes a critical bug in the official version that prevents checkpoint saving (see Section 4.3 for details).

Llama-3.2-Vision: Dubey, A., et al. (2024). The Llama 3 Herd of Models. Meta AI.
PPO: Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347
GAE: Schulman, J., et al. (2016). High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016.

Appendix¶

A. Frequently Asked Questions¶

Q0: Training error TypeError: unsupported operand type(s) for %: 'int' and 'NoneType'? - This is a bug in the official code! - Solution: Use the fixed fork/version recorded at the top of this guide

git -C "${BOOK_ROOT}/chapter7/SFTvsRL" rev-parse HEAD
# Expected: fef0a4a3367260a0934be1e40b01e4021698e023

- Detailed explanation in Section 4.3

Q1: Why is RL training so slow? - Requires online interaction with the environment (256 steps × 15 updates = 3,840 interactions) - Each step requires image loading + model inference (~2-3 seconds/step) - PPO training requires 4 epochs × 256 samples (~1 hour/update)

Q2: Can I train with fewer GPUs? - Theoretically yes, but you need to adjust grad_accum_steps to maintain the effective batch size - 8 GPUs → 4 GPUs: double grad_accum_steps (128 → 256) - Training time will increase significantly

Q3: How to reproduce the paper's results? 1. Use the provided SFT checkpoint (tianzhechu/VIRL-VL-Init) 2. Train strictly with the hyperparameter configuration for 15 updates 3. Test on the same evaluation set (NYC 48 routes, SF 18 routes)

Q4: Why is SFT initialization necessary? - Stabilizes output format (JSON structure) - Provides basic instruction-following ability - Accelerates RL convergence

B. Training Checklist¶

Confirm the following before running training:

[ ] Use the fixed code (chapter7/SFTvsRL at fef0a4a3367260a0934be1e40b01e4021698e023) ⭐
[ ] Verify the bug is fixed (grep "self.save_every = save_every" rl/trainer/base_trainer.py)
[ ] Install all dependencies (pip install -r requirements.txt && cd gym && pip install -e .)
[ ] Download and extract the dataset to /root/SFTvsRL_Data/VIRL_routes/
[ ] Extract nyc_1k_routes.zip
[ ] Extract VLN_mini.zip (for Visual OOD evaluation)
[ ] Verify files exist: route_infos.json, gps_pano_mapping.pkl, street_views/
[ ] Download the SFT checkpoint (tianzhechu/VIRL-VL-Init)
[ ] Confirm data paths are correct:
[ ] ROUTE_INFO="/root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/route_infos.json"
[ ] GPS_TO_PANO="/root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/gps_pano_mapping.pkl"
[ ] STREETVIEWS="/root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/street_views/"
[ ] Check GPU count and memory (8×80GB)
[ ] Configure WandB API key (wandb login)

C. Evaluation Checklist¶

Confirm the following before running evaluation:

[ ] Training is complete and checkpoints are saved (train_ckpt/virl_vl/checkpoint-epoch-*)
[ ] Dataset is downloaded and extracted:
[ ] In-Dist & Rule OOD: /root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/
[ ] Visual OOD: /root/SFTvsRL_Data/VIRL_routes/VLN_mini/
[ ] Modify the evaluation script:
[ ] Update CKPT_NAME="train_ckpt/virl_vl/checkpoint-epoch-14"
[ ] Confirm BASE_DIR="/root/SFTvsRL_Data/VIRL_routes"
[ ] Select the correct ROUTE_INFO path based on the evaluation type

中文¶

本文档详细介绍基于 V-IRL 平台的视觉-语言导航强化学习实验的设计、实现和评估方法。

V-IRL (Virtual Intelligence in Real Life) 是一个用于构建和测试虚拟智能体的开源平台，使智能体能够利用真实的地理空间数据和街景图像在虚拟的真实世界环境中交互。

目录¶

1. 实验概述
2. 实验目的
3. 实验效果
4. 环境设置
5. 任务定义
6. Action Space（动作空间）
7. RL 环境详解
8. Trajectory 生成机制
9. 数据集
10. Vision 输入处理
11. RL 训练过程
12. 超参数配置
13. 评估方法
14. 实验结果分析
15. 参考文献

1. 实验概述¶

本实验基于 V-IRL (Virtual Intelligence in Real Life) 平台实现视觉-语言导航任务，旨在验证强化学习（RL）相比监督微调（SFT）在视觉泛化能力上的优势。

V-IRL 平台简介： - 平台定位：用于构建和测试虚拟智能体的开源平台，使智能体能够在虚拟但真实的环境中感知、思考和行动 - 核心特性： - 利用真实世界的地理空间数据和街景图像 - 支持全球范围内的城市导航和任务执行 - 提供丰富的感官输入（视觉、地理位置、地点信息等） - 支持多种任务类型（导航、地点推荐、城市规划、协作等） - 技术基础：基于 Google Maps Platform 的街景和地理空间 API

本实验核心发现： - RL 训练的模型能够泛化到视觉上分布外（OOD）的环境 - SFT 训练的模型倾向于记忆训练数据，在 OOD 场景下泛化能力差 - RL 通过 outcome-based reward 提升了模型的底层视觉识别能力

论文出处：SFT Memorizes, RL Generalizes
代码仓库： - 推荐使用（我的 fork 版本）：bojieli/SFTvsRL ⭐ - 官方版本：LeslieTrue/SFTvsRL

2. 实验目的¶

2.1 研究问题¶

回答以下核心问题： 1. 视觉泛化：Vision-Language Model 能否将在某个城市（NYC）学到的导航能力迁移到视觉外观完全不同的城市（San Francisco）？ 2. 训练方法对比：SFT vs RL 哪种方法能更好地学习可泛化的视觉表征？ 3. RL 优势来源：RL 为何能提升视觉泛化能力？是否改善了底层的视觉识别？

2.2 实验设计原理¶

控制变量： - 使用相同的 base model（Llama-3.2-11B-Vision） - 使用相同的任务环境（V-IRL 平台的导航任务） - 使用相同的评估指标（per-step accuracy, success rate）

自变量： - 训练方法：SFT vs RL（PPO） - 视觉环境：In-Distribution (NYC) vs Out-of-Distribution (San Francisco) - 动作空间：Absolute directions vs Relative directions

3. 实验效果¶

3.1 主要成果¶

根据论文 Figure 1 和实验结果：

指标	SFT	RL (PPO)	提升
In-Distribution Per-Step Accuracy	~85%	~90%	+5%
Rule OOD Per-Step Accuracy	~15%	~70%	+55%
Visual OOD Generalization	失败 (<10%)	成功 (~60%)	+50%
V-IRL Mini Benchmark	44.0%	77.8%	+33.8%

3.2 关键发现¶

RL 实现视觉泛化：
在 San Francisco（视觉 OOD）环境下，RL 模型保持 ~60% per-step accuracy
SFT 模型在同样环境下降至 <10%，接近随机猜测
RL 提升视觉识别能力：
通过消融实验发现，RL 训练后的模型在卡片识别（GeneralPoints-VL）任务上准确率提升
说明 RL 不仅学习导航策略，还改善了底层视觉编码器
SFT 的必要性：
直接用 RL 从 base model 训练会失败（无法输出结构化 JSON）
SFT 作为 "format teacher" 稳定输出格式，使 RL 能够有效训练

4. 环境设置¶

4.1 系统要求¶

# 硬件要求
- GPU: 8×H100/H800/A100 (80GB) for training
- Memory: 1000GB RAM for training
- Storage: ~500GB for NYC route data + street views + checkpoints

# 软件环境
- Python: 3.13.0

4.2 安装步骤¶

# 1. 将已核验 fork 克隆到本书规范外部 checkout 路径
export BOOK_ROOT=/absolute/path/to/ai-agent-book
git clone https://github.com/bojieli/SFTvsRL.git "${BOOK_ROOT}/chapter7/SFTvsRL"
git -C "${BOOK_ROOT}/chapter7/SFTvsRL" checkout --detach fef0a4a3367260a0934be1e40b01e4021698e023
cd "${BOOK_ROOT}/chapter7/SFTvsRL"

# 2. Create conda environment
conda create -n SFTvsRL python==3.13 -y
conda activate SFTvsRL

# 3. Install dependencies
pip install -r requirements.txt

# 4. Install gym environments
cd gym
pip install -e .
cd ..

# 5. Download data from HuggingFace
huggingface-cli download tianzhechu/SFTvsRL_Data --local-dir ./data

# 6. Login to wandb (create an account on wandb.ai to obtain API key)
wandb login

4.3 官方代码的 Bug 与修复¶

问题描述：

官方 LeslieTrue/SFTvsRL 仓库存在一个严重的 bug，导致训练完成后无法保存 checkpoint：

# rl/trainer/base_trainer.py (官方版本，第 38 行)
def __init__(self, ..., save_every=None, ...):
    ...
    self.save_ckpt = save_ckpt
    self.save_every = None  # ❌ BUG: 硬编码为 None，忽略配置参数！

Bug 影响：

# 训练循环中
for update in range(self.num_updates):
    if self.save_ckpt:
        save_model = (update + 1) % self.save_every == 0  # ❌ TypeError!
        # 因为 self.save_every = None，无法进行模运算

即使在配置文件中设置了 save_every: 1，训练时也会报错：

TypeError: unsupported operand type(s) for %: 'int' and 'NoneType'

修复方案：

bojieli/SFTvsRL fork 修复了这个问题：

# rl/trainer/base_trainer.py (修复后，第 38 行)
def __init__(self, ..., save_every=None, ...):
    ...
    self.save_ckpt = save_ckpt
    self.save_every = save_every  # ✅ 正确使用配置参数

修复的文件：

rl/trainer/base_trainer.py：修复 save_every 参数传递
rl/configs/llama_virl_vl.yaml：添加缺失的 save_every: 1 配置

为什么需要这个修复？

训练 15 小时后：
- 官方版本：❌ 无法保存 checkpoint，损失训练进度
- 修复版本：✓ 成功保存 checkpoint，可以进行评估和继续训练

4.4 数据下载与路径配置¶

4.4.1 下载数据¶

# 1. 创建数据目录
mkdir -p /root/SFTvsRL_Data
cd /root/SFTvsRL_Data

# 2. 下载 VIRL 数据（从 HuggingFace）
huggingface-cli download tianzhechu/SFTvsRL_Data \
    --include "VIRL_routes/*" \
    --local-dir .

# 3. 解压数据
cd VIRL_routes
unzip nyc_1k_routes.zip
unzip VLN_mini.zip  # 用于 Visual OOD 评估

# 4. 验证目录结构
ls -la nyc_1k_routes/
# 应该看到：
# - route_infos.json
# - gps_pano_mapping.pkl
# - street_views/

4.4.2 最终目录结构¶

/root/SFTvsRL_Data/
└── VIRL_routes/
    ├── nyc_1k_routes/              # NYC 训练数据
    │   ├── route_infos.json        # 路线信息
    │   ├── gps_pano_mapping.pkl    # GPS 到全景 ID 映射
    │   └── street_views/           # 街景图片目录
    │       ├── pano_XXX_h000.jpg
    │       ├── pano_XXX_h090.jpg
    │       └── ...
    ├── VLN_mini/                   # San Francisco OOD 数据
    │   ├── route_infos.json
    │   ├── gps_pano_mapping.pkl
    │   └── street_views/
    └── ...

4.4.3 配置训练脚本路径（可选）¶

如果你没有权限访问 /root，请在 scripts/virl_training/vl_train.sh 中配置路径：

BASE_DIR="/root/SFTvsRL_Data/VIRL_routes"

4.5 开始训练¶

完成数据下载和配置后，可以直接运行训练：

# 1. 进入代码目录
cd /root/SFTvsRL

# 2. 激活环境
conda activate SFTvsRL

# 3. 启动训练（使用已配置好的脚本）
bash scripts/virl_training/vl_train.sh

训练脚本自动使用以下路径（已在脚本中配置）：

BASE_DIR="/root/SFTvsRL_Data/VIRL_routes"
ROUTE_INFO="${BASE_DIR}/nyc_1k_routes/route_infos.json"
GPS_TO_PANO="${BASE_DIR}/nyc_1k_routes/gps_pano_mapping.pkl"
STREETVIEWS="${BASE_DIR}/nyc_1k_routes/street_views/"

预期输出：

Parsed instruction: ['First, turn right to face north.', ...]
Collecting Trajectories: 100%|████████| 256/256 [40:25<00:00]
PPO Training Epoch 0/4: 100%|████████| 256/256 [05:12<00:00]
PPO Training Epoch 1/4: 100%|████████| 256/256 [05:10<00:00]
...
Saving checkpoint to: train_ckpt/virl_vl/checkpoint-epoch-4/

训练完成后检查 checkpoint：

ls -lh train_ckpt/virl_vl/
# 应该看到（如果 save_every=5）：
# checkpoint-epoch-4/
# checkpoint-epoch-9/
# checkpoint-epoch-14/

5. 任务定义¶

5.1 导航任务描述¶

任务：智能体（VLM）需要根据自然语言指令，在真实世界街景中导航到目标地点。

输入： 1. 全局指令（Global Instruction）：完整的导航路线描述

1. First, turn left to face south.
2. Move forward until you reach next intersection where Battery Park is nearby.
3. Turn right to face west.
4. Move forward until you reach destination.

视觉观察（Visual Observation）：2×2 街景图片网格（4个方向）

┌─────────┬─────────┐
│ Front   │ Right   │
├─────────┼─────────┤
│ Back    │ Left    │
└─────────┴─────────┘

每张图片：640×640 pixels
总图片尺寸：1280×1280 pixels (with 5px separator)
实际输入模型：2405×2405 pixels (1200×2 + 5)

历史序列（Observation-Action Sequence）：

O_0: "No landmarks nearby; You observe an intersection"
A_0: "turn_direction(south)"
O_1: "Battery Park on your right; No intersection"
A_1: "forward()"
...

输出：结构化 JSON 格式的动作

{
  "current observation": "Battery Park on your right; You observe an intersection",
  "current instruction": "Turn right to face west",
  "action": "turn_direction(west)"
}

5.2 成功条件¶

一个 episode 成功需满足： 1. ✓ 到达正确目的地（执行 stop() 在正确位置） 2. ✓ 每个 waypoint 在允许的验证次数内（默认 2 次）执行正确动作 3. ✓ 未超过最大步数限制

6. Action Space（动作空间）¶

6.1 Absolute Action Space（绝对动作空间）¶

训练时使用的默认动作空间：

ACTION_SPACE = [
    "forward()",                    # 向前移动一步
    "turn_direction(north)",        # 转向正北（0°）
    "turn_direction(northeast)",    # 转向东北（45°）
    "turn_direction(east)",         # 转向正东（90°）
    "turn_direction(southeast)",    # 转向东南（135°）
    "turn_direction(south)",        # 转向正南（180°）
    "turn_direction(southwest)",    # 转向西南（225°）
    "turn_direction(west)",         # 转向正西（270°）
    "turn_direction(northwest)",    # 转向西北（315°）
    "stop()"                        # 停止（到达目的地）
]

特点： - 使用绝对方位词（罗盘方向） - 与智能体当前朝向无关 - 符合人类自然导航习惯

6.2 Relative Action Space（相对动作空间）¶

用于 Rule OOD 评估的动作空间：

ACTION_SPACE_RELATIVE = [
    "forward()",                      # 向前移动一步
    "turn_direction(left)",           # 左转（~-90°）
    "turn_direction(right)",          # 右转（~+90°）
    "turn_direction(slightly left)",  # 微左转（-45° to 0°）
    "turn_direction(slightly right)", # 微右转（0° to +45°）
    "stop()"                          # 停止
]

特点： - 使用相对方向（相对于当前朝向） - 测试模型对不同指令格式的泛化能力 - 动作语义完全不同，但任务目标相同

6.3 动作空间配置¶

在配置文件中设置：

# rl/configs/llama_virl_vl.yaml
env_config:
  absolute_action: true  # True: Absolute, False: Relative

在训练/评估脚本中覆盖：

# Training with absolute actions
--env_config.absolute_action=True

# Evaluation with relative actions (Rule OOD)
--env_config.absolute_action=False

7. RL 环境详解¶

7.1 环境类结构¶

VIRL 使用 OpenAI Gym 接口实现：

class NavigationEnvironment(gym.Env):
    """
    V-IRL 导航环境

    主要组件：
    - Platform: Google Street View 接口
    - Ground Truth Rail: 预计算的正确路径
    - Verification System: 动作验证与反馈机制
    """

    def __init__(self, 
        route_info_path,      # 路线数据路径
        resolution=1200,      # 图片分辨率
        verify_iter=2,        # 每个 waypoint 的尝试次数
        absolute_action=True, # 动作空间类型
        relocation=True,      # GPS 重定位到最近的全景点
        drop_rate=0.5,        # waypoint 之间插值点的丢弃率
        ...
    )

7.2 Ground Truth Rail（参考轨迹）¶

环境预先计算一条 "rail"（参考轨迹），包含：

密集 waypoints：
原始路线：5-10 个交叉口
插值后：每 5-10 米一个点
总计：~20-40 个 waypoints per route

每个 waypoint 包含：

waypoint = {
    'geocode': [40.758, -73.985],           # GPS 坐标
    'heading': 180,                          # 朝向（度数）
    'gt_action': 'turn_direction(south)',   # 正确动作
    'observation': 'Battery Park on right', # 地标描述
    'intersection_observation': 'You observe an intersection',
    'instruction': 'Turn left to face south',  # 当前执行的指令
    'instruction_idx': 1                     # 指令索引
}

7.3 Verification Mechanism（验证机制）¶

多次尝试机制（verify_iter=2）：

# 步骤 1: 智能体输出动作
agent_action = model.generate(obs, instruction, history)

# 步骤 2: 与 ground truth 比较
if agent_action == gt_action:
    reward = +1  # CORRECT_ACTION
    move_to_next_waypoint()
    remaining_attempts = verify_iter  # 重置尝试次数
else:
    reward = -1  # INCORRECT_ACTION
    remaining_attempts -= 1

    if remaining_attempts > 0:
        # 保持在当前位置，给予反馈，允许重试
        feedback = f"Incorrect action. Expected {gt_action}"
        stay_at_current_position()
    else:
        # 尝试次数用尽，强制移动到下一个 waypoint（惩罚）
        reward = -1
        force_move_to_next_waypoint()

Reward Function：

REWARD_FN_VIRL = {
    "CORRECT_ACTION": +1,           # 动作正确
    "INCORRECT_ACTION": -1,         # 动作错误
    "INCORRECT_OBS": -1.5,          # 观察描述错误（错误检测路口）
    "INCORRECT_INSTRUCTION": -1.75  # 指令理解错误
}

7.4 Episode 终止条件¶

Episode 结束于以下任一情况：

成功：done=True, is_success=True
在正确位置执行 stop()
失败：truncated=True, is_success=False
超过最大步数（由 rail 长度决定，通常 20-40 步）
强制前进：is_success=False
某个 waypoint 尝试次数用尽，被迫前进
Episode 继续但标记为失败

7.5 环境配置¶

# rl/configs/llama_virl_vl.yaml
env_config:
  id: 'gym_virl/Navigation-v0'
  route_info_path: "..."           # 路线数据
  resolution: 1200                 # 街景图片分辨率
  verify_iter: 2                   # 验证尝试次数
  absolute_action: true            # 动作空间类型
  relocation: true                 # GPS 重定位
  drop_rate: 0.5                   # waypoint 采样率
  straight_line_length: 5          # 两个交叉口间插值点数

  platform_cfg:
    STREET_VIEW:
      SIZE: [640, 640]             # 单张街景尺寸
      HEADING: 0                   # 默认朝向
      PITCH: 0                     # 俯仰角
      FOV: 90                      # 视场角
      SOURCE: outdoor              # 街景来源

    OFFLINE:
      ENABLED: True                # 使用离线缓存
      PANORAMA_DIR: "..."          # 街景图片目录
      GPS_TO_PANO_PATH: "..."      # GPS 到全景 ID 映射
      MAPPING_RADIUS: 20           # 重定位搜索半径（米）

8. Trajectory 生成机制¶

8.1 训练时的 Trajectory 收集¶

每个 update 收集 256 steps（非 episodes）：

def collect_trajectories(self):
    """
    收集 256 个环境交互步骤
    可能横跨多个 episodes（routes）
    """
    obs, info = self.env.reset()  # 初始化第一条路线

    for step in range(256):
        # 1. 构造 prompt
        prompt = format_prompt(
            global_instruction=info['global_instruction'],
            obs_act_seq=info['obs_act_seq'],
            current_obs=obs
        )

        # 2. 模型生成动作（inference）
        with torch.no_grad():
            # 处理 4 张街景图片
            obs_image = convert_to_2x2_grid(obs)  # [2405, 2405, 3]

            # VLM 前向传播
            values, io_dict, output_text, action_log_prob = \
                actor_critic.act_oneline(
                    inputs=(obs_image, prompt),
                    temperature=0.2,
                    max_new_tokens=512
                )

            # 解析 JSON 输出
            action = parse_json(output_text)['action']

        # 3. 执行动作
        obs_next, reward, done, truncated, info = env.step(output_text)

        # 4. 存储到 rollout buffer
        rollouts.insert(
            obs={"image": obs, "io_dict": io_dict},
            action_log_prob=action_log_prob,
            value=values,
            reward=reward,
            mask=1-done
        )

        running_reward += reward

        # 5. Episode 管理
        if done or truncated:
            # 当前 episode 结束，开始新 episode
            log_episode_reward(running_reward)
            running_reward = 0
            obs, info = env.reset()  # 加载新路线
        else:
            obs = obs_next

    return rollouts  # 256 步的数据

关键点： - 256 steps 可能包含 10-15 个完整 episodes（取决于路线长度） - 最后一个 episode 可能未完成（truncated） - 所有数据都保存用于 PPO 训练

8.2 Multi-Episode Trajectory 示例¶

Update 1: 收集 256 steps
├─ Episode 1 (Route A, 14 steps): Success ✓
│  └─ Steps 0-13: [turn_direction(south), forward(), ..., stop()]
│
├─ Episode 2 (Route B, 22 steps): Success ✓
│  └─ Steps 14-35: [...]
│
├─ Episode 3 (Route C, 18 steps): Failed ✗
│  └─ Steps 36-53: [...] (exceeded attempts at waypoint 12)
│
├─ Episode 4 (Route D, 16 steps): Success ✓
│  └─ Steps 54-69: [...]
│
├─ ...
│
└─ Episode N (Route X, partial): Truncated
   └─ Steps 240-255: [...] (episode未完成，但数据仍用于训练)

8.3 LLM 输入输出示例¶

LLM 输入示例：

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

<|image|>
[Task Description]
You are an expert in navigation. You will receive a sequence of instructions to follow while observing your surrounding stree tviews. You
are also provided with your observation and action history in text. Your goal is to first analyze the instruction and identify the next sentence to be executed.
Then, you need to provide the action to be taken based on the current observation and instruction.

[Instruction]
1. First, turn left to face northeast.
2. Move forward until you reach next intersection where Battery Playscape is on your right behind.
3. Turn right to face north.
4. Move forward until you reach next intersection.
5. Turn slightly left to face northwest.
6. Move forward until you reach next intersection.
7. Turn left to face north.
8. Move forward until you reach next intersection.
9. Turn right to face southeast.
10. Move forward until you reach next intersection.
11. Turn right to face south.
12. Move forward until you reach destination where The destination Cafe De Novo is on your right.


[Observation format]
You observe a 2x2 grid of streetview images with the following headings:
[front, right
 back, left]
You need to identify if any of the landmarks in the instruction are visible in the street view grid.

[Action space]
"forward()": indicates moving forward one step
"turn_direction(x)": indicates adjust the ego agent direction towards x direction. x could be any following 8 directions ['north', 'northeast', 'east', 'southeast', 'south', 'southwest', 'west', 'northwest']
"stop()": indicates the navigation is finished.

[Observations and actions sequence]
O_1: No landmarks nearby;
A_1: turn_direction(northeast)
O_2: No landmarks nearby;
A_2: forward()
O_3: No landmarks nearby;
A_3: forward()
O_4: Battery Playscape is on your right behind; You observe an intersection
A_4: turn_direction(north)
O_5: No landmark nearby; You observe an intersection
A_5: turn_direction(northwest)
O_6: No landmarks nearby;
A_6: forward()
O_7: No landmarks nearby;
A_7: forward()
O_8: No landmarks nearby;
A_8: forward()
O_9: No landmark nearby; You observe an intersection
A_9: turn_direction(north)
O_10: No landmarks nearby;
A_10: forward()
O_11: No landmarks nearby;
A_11: forward()
O_12: No landmarks nearby;
A_12: forward()
O_13: You observe an image of 4 views; You observe an intersection
A_13:


[Output]
{
  "current observation": latest observation from the street view grid,
  "current instruction": analyze the full instruction and identify the sentence to be executed,
  "action": the action to be taken chosen from the action space,
}
<|eot_id|><|start_header_id|>assistant<|end_header_id|>

LLM 输出示例：

{
  "current observation": "No landmark nearby; You observe an intersection",
  "current instruction": "Turn right to face southeast.",
  "action": "turn_direction(southeast)",
}

9. 数据集¶

9.1 训练数据集¶

NYC 1K Routes：

数据来源：Google Maps API 采集
路线数量：1,000 条
覆盖区域：纽约市 Manhattan, Brooklyn, Queens
总 waypoints：~20,000-30,000 个
街景图片：~100,000 张（640×640, 4 方向/位置）

数据结构：

// route_infos.json
[
  [  // 路线列表
    {
      "route_id": "nyc_001",
      "start_place": {
        "name": "Times Square",
        "geocode": [40.758, -73.985],
        "relocated_geocode": [40.7580, -73.9855]
      },
      "dest_place": {
        "name": "Central Park South",
        "geocode": [40.767, -73.979]
      },
      "init_heading": 0,
      "milestone_info": "Turn left to face south. Move forward...",
      "route_results": {
        "geocode_list": [[40.760, -73.984], ...],
        "landmark_list": ["Battery Park", "Plaza Hotel", ...]
      }
    },
    ...
  ],
  1000  // 路线总数
]

Street View 缓存：

nyc_1k_routes/street_views/
├─ pano_XXX_h000.jpg  # Heading 0° (Front)
├─ pano_XXX_h090.jpg  # Heading 90° (Right)
├─ pano_XXX_h180.jpg  # Heading 180° (Back)
└─ pano_XXX_h270.jpg  # Heading 270° (Left)

GPS 映射：

# gps_pano_mapping.pkl
{
    (40.758, -73.985): "pano_ABC123",  # GPS -> Panorama ID
    (40.759, -73.984): "pano_DEF456",
    ...
}

9.2 评估数据集¶

数据集	类型	路线数	数据路径	用途
NYC Test (In-Dist)	In-Distribution	48	`/root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/`	测试训练分布性能
NYC Test (Rule OOD)	Rule OOD	48	`/root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/`	测试相对动作泛化
SF Routes (Visual OOD)	Visual OOD	18	`/root/SFTvsRL_Data/VIRL_routes/VLN_mini/`	测试视觉环境泛化

Visual OOD 数据特点（San Francisco，VLN_mini）： - 不同建筑风格（Victorian vs Modern） - 不同地形（Hills vs Flat） - 不同颜色分布（Pastel houses vs Glass buildings） - 不同地标类型（Cable cars, Golden Gate vs Yellow cabs, Statue of Liberty）

9.3 数据统计¶

NYC 1K Routes (训练 + In-Dist/Rule OOD 评估):
- 位置: /root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/
- 路线数: 1,000 条
- 街景图片: ~100,000 张
- 数据大小: ~30-40 GB

VLN_mini (Visual OOD 评估):
- 位置: /root/SFTvsRL_Data/VIRL_routes/VLN_mini/
- 路线数: 18 条（San Francisco）
- 街景图片: ~2,000 张
- 数据大小: ~1-2 GB

10. Vision 输入处理¶

10.1 为什么使用 4 方向图片而非全景图？¶

问题：Google Street View API 提供 360° 全景图（equirectangular panorama），为什么要转换成 4 个方向的静态图片？

设计动机¶

计算效率
全景图：2048×1024 或更大（~2-6 MB/张）
4 张静态图：4 × 640×640 = ~1.2 MB
存储与加载速度提升 2-5 倍
模型输入限制
Vision Transformer 对输入尺寸敏感
处理 2×2 网格（2405×2405）比处理全景图（2048×1024）更符合模型训练习惯
2×2 网格接近正方形，减少 padding 和变形
任务相关性
人类导航时也是关注前后左右四个方向
不需要看头顶和脚下（pitch = 0°）
4 方向已经包含导航所需的全部信息
数据增强灵活性
可以单独处理每个方向
易于扩展到不同的 heading 配置
便于实现 attention visualization（哪个方向更重要）
与 V-IRL 原始设计一致
V-IRL 论文（Yang et al., 2024）原本就使用 4 方向设计
保持一致性便于对比实验结果

技术对比¶

方案	全景图	4 方向静态图
分辨率	2048×1024	4 × 640×640
文件大小	2-6 MB	1.2 MB
FOV 覆盖	360° × 180°	4 × 90° = 360° (水平)
处理复杂度	需要 equirectangular 投影处理	直接使用
模型适配	需要特殊处理	标准 2D CNN/ViT
存储成本	高 (100K × 6MB = 600GB)	低 (100K × 1.2MB = 120GB)

全景图的缺点¶

Distortion（畸变）：
Equirectangular 投影在极点附近严重变形
需要特殊的预处理或模型适配
信息冗余：
天空和地面占据大量像素但信息价值低
导航主要关注水平方向的地标
计算开销：
更大的图片需要更多 GPU 内存
训练和推理速度显著下降

10.2 街景图片获取¶

def _get_visual_observation(self):
    """
    获取当前位置的 4 方向街景图片

    Returns:
        np.array: [2405, 2405, 3] 的 RGB 图片
    """
    # 1. 从 Platform 获取 4 张图片
    image_list = self.platform.get_all_streetview_from_geocode(
        geocode=self.current_geocode,
        cur_heading=self.current_heading
    )
    # image_list = [front, right, back, left] (each 640×640)

    # 2. 调整为统一分辨率
    resized_images = [
        image.resize((self.resolution, self.resolution))  # 1200×1200
        for image in image_list
    ]

    # 3. 拼接为 2×2 网格
    line_width = 5  # 黑色分隔线
    canvas = Image.new('RGB', 
                       (self.resolution * 2 + line_width,   # 2405
                        self.resolution * 2 + line_width),  # 2405
                       (0, 0, 0))  # 黑色背景

    # 放置图片：
    # [0,0] -> (0, 0)           Front
    # [1,0] -> (1205, 0)        Right
    # [0,1] -> (0, 1205)        Back
    # [1,1] -> (1205, 1205)     Left
    for i, image in enumerate(resized_images):
        x = (i % 2) * (self.resolution + line_width)
        y = (i // 2) * (self.resolution + line_width)
        canvas.paste(image, (x, y))

    return np.array(canvas)  # [2405, 2405, 3]

10.3 视觉输入示意图¶

2×2 Street View Grid (2405 × 2405 pixels)

┌─────────────────────┬─────────────────────┐
│                     │                     │
│    Front View       │    Right View       │
│    (1200×1200)      │    (1200×1200)      │
│                     │                     │
│    Heading: 0°      │    Heading: 90°     │
│                     │                     │
├─────────────────────┼─────────────────────┤
│                     │                     │
│    Back View        │    Left View        │
│    (1200×1200)      │    (1200×1200)      │
│                     │                     │
│    Heading: 180°    │    Heading: 270°    │
│                     │                     │
└─────────────────────┴─────────────────────┘

黑色分隔线：5 pixels

10.4 Llama-3.2-Vision 图片处理¶

def formulate_payload(self, question, obs=None):
    """
    构造 Llama-3.2-Vision 的输入格式

    Args:
        question: 文本 prompt
        obs: PIL.Image or np.array
    """
    self.payload = [
        {
            "role": "user",
            "content": [{"type": "text", "text": question}]
        }
    ]

    if obs is not None:
        # 转换为 PIL Image
        if isinstance(obs, np.ndarray):
            obs = Image.fromarray(obs)

        # 插入到 content 最前面（Llama 格式要求）
        self.payload[0]['content'].insert(0, {
            "type": "image", 
            "image": obs
        })

def process_input(self, obs, prompt):
    """
    使用 processor 处理输入
    """
    # 1. Apply chat template
    input_text = self.processor.apply_chat_template(
        self.payload, 
        add_generation_prompt=True
    )

    # 2. Process image + text
    inputs = self.processor(
        obs,           # PIL Image
        input_text,    # Formatted prompt
        return_tensors="pt",
        add_special_tokens=False
    ).to(self.model.device)

    # inputs = {
    #     'input_ids': tensor([[..., image_tokens, ..., text_tokens]]),
    #     'attention_mask': tensor([[1, 1, ..., 1]]),
    #     'pixel_values': tensor([[[...]]]),  # 处理后的图片特征
    #     'cross_attention_mask': tensor([[[...]]])
    # }

    return inputs

10.5 Vision Encoder 处理流程¶

Input Image (2405×2405×3)
    ↓
Llama-3.2-Vision Processor
    ↓
├─ Image Processor:
│  ├─ Resize to model input size
│  ├─ Normalize (mean=[0.48145466, 0.4578275, 0.40821073])
│  └─ Convert to tensor
│
└─ Vision Encoder (CLIP-based):
   ├─ Patch Embedding (16×16 patches)
   ├─ Vision Transformer Layers
   └─ Output: Visual tokens (sequence length ~1000)
       ↓
Cross-Attention with Language Model
    ↓
Language Decoder generates action JSON

10.6 Vision 相关超参数¶

# 图片获取配置
platform_cfg:
  STREET_VIEW:
    SIZE: [640, 640]        # 单张原始图片尺寸
    FOV: 90                 # 视场角（degrees）
    PITCH: 0                # 俯仰角（水平）
    SOURCE: outdoor         # 室外街景

# 环境配置
env_config:
  resolution: 1200          # 每张图片调整后的尺寸
  # 最终输入：2×1200 + 5 = 2405 pixels

# 模型配置（Llama-3.2-Vision 内置）
model:
  vision_encoder:
    image_size: 560         # 模型输入尺寸（自动调整）
    patch_size: 14          # Patch embedding size
    hidden_size: 1280       # Vision hidden dimension

11. RL 训练过程¶

11.1 训练流程概览¶

SFT 初始化
    ↓
┌────────────────────────────────────────┐
│   RL Training Loop (15 Updates)        │
│                                        │
│  For update in [0, 1, ..., 14]:       │
│                                        │
│  ┌──────────────────────────────────┐ │
│  │  Phase 1: Rollout (256 steps)   │ │
│  │  ├─ 多个 episodes                │ │
│  │  ├─ 收集 (obs, action, reward)   │ │
│  │  └─ 计算 value predictions       │ │
│  └──────────────────────────────────┘ │
│            ↓                           │
│  ┌──────────────────────────────────┐ │
│  │  Phase 2: PPO Training (4 epochs)│ │
│  │  ├─ Compute advantages (GAE)     │ │
│  │  ├─ 4 epochs × 256 samples       │ │
│  │  ├─ Update value network         │ │
│  │  └─ Update learning rate         │ │
│  └──────────────────────────────────┘ │
│                                        │
└────────────────────────────────────────┘
    ↓
保存 Final Checkpoint

11.2 PPO (Proximal Policy Optimization) 算法原理¶

PPO 是一种 on-policy 强化学习算法，由 OpenAI 在 2017 年提出。它通过限制策略更新的幅度来保证训练稳定性，是目前最流行的 RL 算法之一。

11.2.1 核心思想¶

问题背景：

在策略梯度方法中，我们希望最大化期望回报：

\[J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)]\]

传统的 Policy Gradient 方法（如 REINFORCE）直接用梯度上升更新策略，但存在两个问题： 1. 样本效率低：每次收集的数据只能用一次 2. 训练不稳定：大的策略更新可能导致性能崩溃

PPO 的解决方案：

PPO 通过引入 importance sampling 实现数据复用，同时使用 clipping 机制限制策略更新幅度，在样本效率和训练稳定性之间取得平衡。

11.2.2 Importance Sampling（重要性采样）¶

核心问题：如何用旧策略 $\pi_{\theta_{old}}$ 采集的数据来更新新策略 $\pi_\theta$？

Importance Sampling 公式：

\[\mathbb{E}_{a \sim \pi_{\theta_{old}}}[f(a)] = \mathbb{E}_{a \sim \pi_\theta}\left[\frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)} f(a)\right]\]

其中，$\frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)}$ 称为 importance ratio（重要性比率）。

在 RL 中的应用：

我们可以用旧策略收集的轨迹来估计新策略的期望回报：

\[J(\theta) \approx \sum_{t} r_t \cdot \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}\]

问题：当 $\pi_\theta$ 与 $\pi_{\theta_{old}}$ 差异过大时，importance ratio 方差爆炸，导致训练不稳定。

11.2.3 PPO-Clip 目标函数¶

PPO 通过 clipping 机制限制 importance ratio 的范围，防止策略更新过大。

定义 ratio：

\[r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}\]

PPO-Clip 目标函数：

\[L^{CLIP}(\theta) = \mathbb{E}_t\left[\min\left(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t\right)\right]\]

其中： - $\hat{A}_t$：advantage（优势函数），表示该动作比平均水平好多少 - $\epsilon$：clip 参数（本实验中 $\epsilon=0.1$） - $\text{clip}(r, 1-\epsilon, 1+\epsilon)$：将 $r$ 限制在 $[0.9, 1.1]$ 范围内

直觉理解：

当 Advantage > 0（好的动作）：
如果 $r_t > 1.1$：clip 限制增长，防止过度乐观
如果 $0.9 < r_t < 1.1$：正常更新
如果 $r_t < 0.9$：不惩罚（已经降低了概率）
当 Advantage < 0（坏的动作）：
如果 $r_t < 0.9$：clip 限制下降，防止过度悲观
如果 $0.9 < r_t < 1.1$：正常更新
如果 $r_t > 1.1$：不奖励（已经提高了概率）

数学表达：

\[ L^{CLIP}(\theta) = \mathbb{E}_t\left[\min\left( \begin{cases} r_t \hat{A}_t & \text{if } \hat{A}_t \geq 0 \\ \text{clip}(r_t, 1-\epsilon, 1+\epsilon) \hat{A}_t & \text{if } \hat{A}_t < 0 \end{cases} \right)\right] \]

这保证了策略不会偏离旧策略太远，从而维持训练稳定性。

11.2.4 Advantage 计算（GAE）¶

Advantage 函数定义：

\[A^\pi(s_t, a_t) = Q^\pi(s_t, a_t) - V^\pi(s_t)\]

它衡量在状态 $s_t$ 执行动作 $a_t$ 相比平均水平的优劣。

问题：我们没有真实的 $Q$ 和 $V$，需要估计。

Generalized Advantage Estimation (GAE)：

GAE 是一种平衡偏差（bias）和方差（variance）的 advantage 估计方法：

\[\hat{A}_t^{GAE(\gamma, \lambda)} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}\]

其中： - $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$：TD error（时序差分误差） - $\gamma$：折扣因子（本实验 $\gamma=0.9$） - $\lambda$：GAE 参数（本实验 $\lambda=0.95$）

递归计算（从后往前）：

\[\hat{A}_t = \delta_t + (\gamma \lambda) \hat{A}_{t+1}\]

直觉理解：

$\lambda=0$：只看一步 TD error，低方差但高偏差
$\lambda=1$：看完整回报，低偏差但高方差
$\lambda=0.95$：平衡两者，是实践中的最佳选择

Return 计算：

\[\hat{R}_t = \hat{A}_t + V(s_t)\]

这个 $\hat{R}_t$ 用于训练 value network。

11.2.5 Value Function Loss¶

除了策略损失，PPO 还需要训练 value network 来估计状态价值：

\[L^{VF}(\theta) = \mathbb{E}_t\left[\max\left((V_\theta(s_t) - \hat{R}_t)^2, (\bar{V}_t - \hat{R}_t)^2\right)\right]\]

其中： - $V_\theta(s_t)$：当前 value 估计 - $\hat{R}_t$：目标 return（从 GAE 计算） - $\bar{V}_t = V_{\theta_{old}}(s_t) + \text{clip}(V_\theta(s_t) - V_{\theta_{old}}(s_t), -\epsilon, \epsilon)$：clipped value

Clipping 的作用：

防止 value function 更新过大，与 policy clipping 类似的稳定性考虑。

11.2.6 Entropy Bonus¶

为了鼓励探索，PPO 添加 entropy bonus：

\[L^{ENT}(\theta) = \mathbb{E}_t[H(\pi_\theta(\cdot|s_t))]\]

其中 $H$ 是策略分布的熵（entropy）。熵越大，策略越随机，探索性越强。

11.2.7 总损失函数¶

PPO 的最终损失函数是三项的加权和：

\[L(\theta) = -L^{CLIP}(\theta) + c_1 L^{VF}(\theta) - c_2 L^{ENT}(\theta)\]

在本实验中： - $c_1 = 0.5$：value loss 系数 - $c_2 = 0.01$：entropy 系数

训练过程：

收集 256 步轨迹数据（使用 $\pi_{\theta_{old}}$）
计算所有时间步的 advantage 和 return
对这 256 个样本训练 4 个 epochs：
每个 epoch 遍历所有样本
每个样本计算损失并更新参数
梯度累积 128 步后更新一次
更新 $\theta_{old} \leftarrow \theta$，进入下一轮

11.2.8 为什么 PPO 适合 VLM 训练？¶

稳定性：Clipping 机制防止大幅更新，保护昂贵的预训练模型
样本效率：每批数据可以训练多个 epochs（本实验 4 epochs）
简单性：相比 TRPO，PPO 不需要复杂的二阶优化
可扩展性：易于与 DeepSpeed 等分布式训练框架结合

11.2.9 本实验中的 PPO 配置¶

ppo_config:
  clip_param: 0.1           # ε = 0.1，限制 ratio 在 [0.9, 1.1]
  ppo_epoch: 4              # 每批数据训练 4 个 epochs
  mini_batch_size: 1        # 逐样本训练（因为 VLM 内存占用大）
  value_loss_coef: 0.5      # c_1，value loss 权重
  entropy_coef: 0.01        # c_2，entropy bonus 权重
  max_grad_norm: 0.01       # 梯度裁剪阈值（非常小，保证稳定）

11.3 PPO 代码实现详解¶

11.3.1 Advantage 计算（GAE）代码¶

def compute_returns(self, next_value, gamma=0.9, gae_lambda=0.95):
    """
    使用 Generalized Advantage Estimation (GAE)
    计算每步的 return 和 advantage

    Args:
        next_value: 最后一步的 value prediction
        gamma: 折扣因子
        gae_lambda: GAE λ 参数
    """
    self.value_preds[-1] = next_value
    gae = 0

    # 从后向前计算
    for step in reversed(range(self.num_steps)):  # 255 → 0
        # TD error
        delta = (self.rewards[step] + 
                gamma * self.value_preds[step + 1] * self.masks[step + 1] -
                self.value_preds[step])

        # GAE 累积
        gae = delta + gamma * gae_lambda * self.masks[step + 1] * gae

        # Return = Advantage + Value
        self.returns[step] = gae + self.value_preds[step]

    # 标准化 advantages（用于训练稳定）
    advantages = self.returns[:-1] - self.value_preds[:-1]
    advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-5)

    return advantages

11.3.2 PPO Loss 计算代码¶

def ppo_update(self, rollouts, next_value):
    """
    PPO 训练：4 epochs × 256 samples

    每个 epoch 遍历所有 256 个样本（mini_batch_size=1）
    """
    advantages = compute_advantages(rollouts)

    for epoch in range(4):  # ppo_epoch = 4
        for sample_idx in range(256):  # num_steps = 256
            # 1. 获取样本
            obs_batch = rollouts.obs[sample_idx]
            old_action_log_prob = rollouts.action_log_probs[sample_idx]
            return_batch = rollouts.returns[sample_idx]
            value_pred_old = rollouts.value_preds[sample_idx]
            advantage = advantages[sample_idx]

            # 2. 重新评估（带梯度）
            new_value, new_action_log_prob, entropy = actor_critic.evaluate_actions(
                **obs_batch['io_dict']
            )
            entropy_loss = -entropy.mean()

            # 3. Compute probability ratio
            ratio = torch.exp(new_action_log_prob - old_action_log_prob)

            # 4. Policy Loss (Clipped Surrogate Objective)
            surr1 = ratio * advantage
            surr2 = torch.clamp(ratio, 1.0 - clip_param, 1.0 + clip_param) * advantage

            # Ratio clipping protection (防止梯度爆炸)
            if torch.any(ratio > 10):
                policy_loss = -surr2.mean()
            else:
                policy_loss = -torch.min(surr1, surr2).mean()

            # 5. Value Loss (Clipped)
            value_pred_clipped = (value_pred_old + 
                torch.clamp(new_value - value_pred_old, 
                           -clip_param, clip_param))

            value_losses = (new_value - return_batch).pow(2)
            value_losses_clipped = (value_pred_clipped - return_batch).pow(2)
            value_loss = 0.5 * torch.max(value_losses, 
                                        value_losses_clipped).mean()

            # 6. Total Loss
            loss = (value_loss * value_loss_coef +  # 0.5
                   policy_loss +
                   entropy_loss * entropy_coef)      # 0.01

            # 7. Backward & Update
            accelerator.backward(loss)

            if accelerator.sync_gradients:
                accelerator.clip_grad_norm_(
                    actor_critic.parameters(),
                    max_grad_norm  # 0.01
                )

            optimizer.step()
            optimizer.zero_grad()

11.4 模型架构¶

class VLMValue(nn.Module):
    """
    Value Network: 用于估计状态价值
    """
    def __init__(self, base):
        super().__init__()
        self.base = base  # Llama-3.2-11B-Vision (冻结 generation 路径)

        # 3-layer MLP value head
        self.value_head = nn.Sequential(
            nn.Linear(4096, 1024),
            nn.ReLU(),
            nn.Linear(1024, 512),
            nn.ReLU(),
            nn.Linear(512, 1)
        ).to(base.device, dtype=torch.bfloat16)

    def forward(self, inputs):
        # 前向传播获取 hidden states
        outputs = self.base(**inputs, output_hidden_states=True)
        hidden_states = outputs.hidden_states  # All layers

        # 使用最后一层的最后一个 token
        last_hidden = hidden_states[-1][:, -1]  # [batch, 4096]

        # Value prediction
        values = self.value_head(last_hidden)  # [batch, 1]
        return values

class VLMPolicy(nn.Module):
    """
    Policy Network: 包装 Value Network + Generation
    """
    def __init__(self, tokenizer, value_model, generation_config):
        super().__init__()
        self.tokenizer = tokenizer
        self.value_model = value_model
        self.base = value_model.base
        self.temperature = generation_config.temperature
        self.max_new_tokens = generation_config.max_new_tokens

    def act_oneline(self, inputs, obs=None):
        """
        生成动作（inference，无梯度）

        Returns:
            values: 状态价值估计
            io_dict: 输入输出字典（用于后续训练）
            output_text: 解码后的 JSON 文本
            action_log_prob: 动作对数概率
        """
        with torch.no_grad():
            # 1. Generation
            outputs = self.base.generate(
                **inputs,
                max_new_tokens=self.max_new_tokens,
                temperature=self.temperature,
                output_scores=True,
                output_hidden_states=True,
                return_dict_in_generate=True
            )

            output_ids = outputs['sequences'][:, inputs['input_ids'].shape[1]:]
            output_text = self.tokenizer.decode(output_ids[0], 
                                               skip_special_tokens=True)

            # 2. 拼接 input + output
            cated_io = torch.cat((inputs['input_ids'], output_ids), dim=1)

            # 3. Prepare inputs for evaluation
            io_dict = self._prepare_io_dict(inputs, output_ids)

            # 4. Evaluate to get value & log_prob
            values, sum_log_prob, action_tokens_log_prob = \
                self.evaluate(**io_dict, inference=True)

        return values, io_dict, output_text, sum_log_prob, action_tokens_log_prob

    def evaluate_actions(self, **io_dict):
        """
        重新评估动作（training，带梯度）

        Returns:
            values: 新的 value 估计
            action_log_probs: 新的 log 概率
            entropy: 策略分布的熵（用于 exploration bonus）
        """
        # 1. Forward pass with gradients
        outputs = self.base(
            **io_dict['new_inputs'],
            output_hidden_states=True
        )

        # 2. Compute value
        hidden_states = outputs.hidden_states[-1][:, -1]
        values = self.value_model.value_head(hidden_states)

        # 3. Compute log probabilities
        logits = outputs.logits
        output_ids = io_dict['io_pair'][1]  # Generated tokens

        action_log_probs = self._compute_log_probs(logits, output_ids)

        # 4. 计算生成 token 上的策略熵
        dist = torch.distributions.Categorical(logits=logits)
        entropy = dist.entropy().mean()

        return values, action_log_probs, entropy

11.5 DeepSpeed ZeRO 分布式训练原理¶

本实验使用 DeepSpeed ZeRO Stage 2 在 8 张 GPU 上进行分布式训练。DeepSpeed 是微软开发的深度学习优化库，ZeRO（Zero Redundancy Optimizer）是其核心技术。

11.5.1 为什么需要 DeepSpeed？¶

内存挑战：

训练 11B 参数的 Llama-3.2-Vision 模型面临巨大的内存压力：

模型参数：
- Base Model: 11B × 2 bytes (bf16) = 22 GB
- Value Head: 4096×1024 + 1024×512 + 512×1 ≈ 5M × 2 = 10 MB

优化器状态（Adam）：
- Momentum: 11B × 4 bytes (fp32) = 44 GB
- Variance: 11B × 4 bytes (fp32) = 44 GB

梯度：
- Gradients: 11B × 2 bytes (bf16) = 22 GB

激活值（Activations）：
- Forward pass: ~20-40 GB (取决于 batch size)
- Backward pass: ~20-40 GB

总计：~170-190 GB

单张 H800 GPU 只有 80 GB 内存，无法容纳完整的训练状态！

传统数据并行的问题：

传统 DDP（Distributed Data Parallel）在每张 GPU 上保存完整的模型副本：

GPU 0: [完整模型 + 完整优化器状态 + 梯度] = 170 GB ✗
GPU 1: [完整模型 + 完整优化器状态 + 梯度] = 170 GB ✗
...
GPU 7: [完整模型 + 完整优化器状态 + 梯度] = 170 GB ✗

内存冗余问题： - 8 张 GPU 上存储相同的 optimizer states（8 × 88 GB = 704 GB 冗余！） - 每张 GPU 仍然需要 170 GB，超出单卡容量

11.5.2 ZeRO 核心思想¶

Zero Redundancy Optimizer 通过消除冗余来减少内存占用：

核心原理：分片存储（Partitioning）+ 通信重建（Communication）

不存储冗余数据，而是：
1. 将数据分片存储在不同 GPU 上
2. 需要时通过通信（All-Gather）重建完整数据
3. 用通信时间换内存空间

ZeRO 的三个阶段：

Stage	分片内容	内存节省	通信开销
ZeRO-1	Optimizer States	~4×	低
ZeRO-2	+ Gradients	~8×	中等
ZeRO-3	+ Parameters	~64×	高

本实验使用 ZeRO Stage 2，在内存节省和通信开销间取得平衡。

11.5.3 ZeRO Stage 2 详细机制¶

内存分片策略：

# 8 张 GPU，每张 GPU 只存储 1/8 的优化器状态和梯度

GPU 0: 
  - 完整模型参数 (22 GB)
  - Optimizer States [0:N/8] (44/8 = 5.5 GB)
  - Gradients [0:N/8] (22/8 = 2.75 GB)
  - Activations (~20 GB)
  → Total: ~50 GB ✓

GPU 1:
  - 完整模型参数 (22 GB)
  - Optimizer States [N/8:2N/8] (5.5 GB)
  - Gradients [N/8:2N/8] (2.75 GB)
  - Activations (~20 GB)
  → Total: ~50 GB ✓

... (GPU 2-7 类似)

训练流程：

Forward Pass（前向传播）

每张 GPU 独立计算：
- 输入：各自的 batch（总 batch / 8）
- 使用：完整模型参数（所有 GPU 相同）
- 输出：各自的 loss 和 activations

Backward Pass（反向传播）

每张 GPU 独立计算梯度：
GPU i: ∂L/∂θ (完整梯度)

然后 Reduce-Scatter：
GPU i: 只保留 ∂L/∂θ[i×N/8:(i+1)×N/8]
→ 每张 GPU 只存储 1/8 的梯度

Optimizer Step（参数更新）

每张 GPU 更新自己负责的参数分片：
GPU 0: θ[0:N/8] ← θ[0:N/8] - lr × ∂L/∂θ[0:N/8]
GPU 1: θ[N/8:2N/8] ← θ[N/8:2N/8] - lr × ∂L/∂θ[N/8:2N/8]
...

然后 All-Gather：
所有 GPU 广播自己更新的参数，重建完整模型

关键通信操作：

Reduce-Scatter（梯度聚合 + 分片）

输入：每张 GPU 的完整梯度
操作：求和并分片
输出：每张 GPU 得到 1/8 的聚合梯度

时间复杂度：O(N/P) where P=8

All-Gather（参数重建）

输入：每张 GPU 的 1/8 参数
操作：收集并广播
输出：每张 GPU 得到完整参数

时间复杂度：O(N/P)

11.5.4 Optimizer Offload 机制¶

本实验还使用了 CPU Offloading：

deepspeed_config:
  offload_optimizer_device: cpu    # Optimizer 状态卸载到 CPU
  offload_param_device: none       # 参数不卸载

工作原理：

训练时：
1. Optimizer States 存储在 CPU 内存（便宜且大容量）
2. 需要更新时，传输到 GPU 计算
3. 更新完成后，传回 CPU

内存分布：
GPU: 模型参数 (22 GB) + 梯度 (2.75 GB) + Activations (20 GB) ≈ 45 GB ✓
CPU: Optimizer States (5.5 GB per GPU) → 不占用 GPU 内存

权衡： - ✓ 节省 GPU 内存：~5.5 GB per GPU - ✗ 增加训练时间：CPU-GPU 传输开销（~10-15%）

对于 11B 模型，这个权衡是值得的，因为避免了 OOM（Out of Memory）。

11.5.5 梯度累积（Gradient Accumulation）¶

配合 ZeRO，本实验使用 梯度累积 128 步：

grad_accum_steps: 128

目的：模拟更大的 batch size

实际流程：
for i in range(128):
    # Forward & Backward（不更新参数）
    loss = model(batch_i)
    loss.backward()  # 梯度累积

# 累积 128 次后才更新
optimizer.step()  # 使用累积的梯度
optimizer.zero_grad()

等效 Batch Size：

Per-GPU Batch Size: 1
Num GPUs: 8
Grad Accum Steps: 128

Effective Batch Size = 1 × 8 × 128 = 1024

为什么需要梯度累积？

内存限制：Batch size = 1 是 VLM 能承受的最大值
训练稳定性：大 batch size（1024）有助于 RL 训练稳定
样本效率：PPO 需要足够大的 batch 来估计 advantage

11.5.6 混合精度训练（BF16）¶

mixed_precision: bf16
downcast_bf16: 'yes'

BFloat16 vs Float32：

类型	位数	指数位	尾数位	范围	精度
FP32	32	8	23	±3.4×10³⁸	高
BF16	16	8	7	±3.4×10³⁸	中
FP16	16	5	10	±6.5×10⁴	中

BF16 优势： - ✓ 内存减半：22 GB → 11 GB - ✓ 计算加速：~2× on H800 - ✓ 动态范围大：与 FP32 相同（防止 overflow） - ✓ 无需 loss scaling（比 FP16 简单）

训练流程：

# Forward & Backward in BF16
with autocast(dtype=torch.bfloat16):
    output = model(input)
    loss = criterion(output, target)

loss.backward()  # 梯度 in BF16

# Optimizer in FP32（Master Weights）
optimizer.step()  # 更新 FP32 参数
model.to(torch.bfloat16)  # 转回 BF16 供下次前向

11.5.7 实际内存占用分析¶

单 GPU 内存分布（ZeRO-2 + Offload + BF16）：

模型参数（BF16）: 11B × 2 bytes = 22 GB
Gradients（BF16, 1/8）: 11B × 2 / 8 = 2.75 GB
Activations（BF16）: ~15-20 GB
Optimizer States（CPU Offload）: 0 GB (在 CPU 上)
临时缓冲区: ~5 GB

Total per GPU: ~45-50 GB / 80 GB = 56-62% 占用率 ✓

对比不同配置：

配置	内存占用	是否可行
单 GPU，无优化	170 GB	✗ OOM
DDP，8 GPU	170 GB each	✗ OOM
ZeRO-2，8 GPU	55 GB each	✓
ZeRO-2 + Offload	50 GB each	✓
ZeRO-3	30 GB each	✓（但通信慢）

通信开销：

Reduce-Scatter: 每步 ~0.1 秒
All-Gather: 每步 ~0.1 秒
总开销: ~10-15% 训练时间
权衡: 可接受（相比无法训练）

11.5.8 DeepSpeed 配置详解¶

# scripts/config_zero2_8gpu.yaml

compute_environment: LOCAL_MACHINE
distributed_type: DEEPSPEED

# 核心配置
deepspeed_config:
  zero_stage: 2                      # ZeRO Stage 2
  offload_optimizer_device: cpu      # Optimizer → CPU
  offload_param_device: none         # 参数保留在 GPU
  zero3_init_flag: false             # 不使用 ZeRO-3 初始化
  overlap_comm: false                # 不重叠通信与计算（更稳定）

# 精度配置
mixed_precision: bf16                # BFloat16 混合精度
downcast_bf16: 'yes'                 # 自动转换

# 分布式配置
num_machines: 1                      # 单节点
num_processes: 8                     # 8 GPUs
rdzv_backend: static                 # 静态拓扑（无动态加入）
same_network: true                   # 同一网络（低延迟）

为什么不用 ZeRO-3？

ZeRO-3 节省更多内存但通信开销大：

Stage	通信次数/步	通信量/步	训练速度
ZeRO-2	2 次	22 GB	1.0×
ZeRO-3	4 次	44 GB	0.6×

对于 11B 模型，ZeRO-2 已经足够，无需 ZeRO-3 的额外开销。

11.5.9 实际训练性能¶

吞吐量：

Rollout Phase（256 steps）:
- 时间：~40 分钟
- 速度：6.4 steps/min
- 瓶颈：环境交互 + 模型推理

PPO Training Phase（4 epochs × 256 samples）:
- 时间：~20 分钟  
- 速度：51.2 samples/min
- 瓶颈：梯度计算 + 通信

Total per Update: ~60 分钟
Total Training (15 updates): ~15 小时

扩展性分析：

GPU 数量	理论加速比	实际加速比	效率
1	1.0×	1.0×	100%
2	2.0×	1.8×	90%
4	4.0×	3.4×	85%
8	8.0×	6.2×	78%

效率损失主要来自通信开销，但仍然实现了 6× 加速。

11.6 训练监控指标¶

WandB 记录的关键指标：

wandb.log({
    # 训练进度
    'total_num_steps': total_steps,
    'compute_tokens': token_count,

    # Loss
    'value_loss': value_loss,
    'action_loss': policy_loss,
    'dist_entropy': entropy,

    # Reward 统计
    'reward.mean': rewards.mean(),
    'reward.std': rewards.std(),
    'reward.max': rewards.max(),
    'reward.min': rewards.min(),

    # Value 统计
    'value.mean': values.mean(),
    'value.std': values.std(),

    # Return 统计
    'return.mean': returns.mean(),
    'return.std': returns.std(),

    # Episode 统计
    'episode_rewards.mean': np.mean(episode_rewards),
    'success_rate': success_rate,
    'per_step_accuracy': step_accuracy
})

12. 超参数配置¶

12.1 核心超参数总览¶

类别	参数	值	说明
训练规模	`num_updates`	15	总训练轮数
	`num_steps`	256	每轮收集步数
	`ppo_epoch`	4	PPO 训练 epochs
	`grad_accum_steps`	128	梯度累积步数
学习率	`init_lr`	1e-7	初始学习率
	`lr_max_steps`	20	LR 调度器总步数
	`end_lr`	1e-9	最终学习率
PPO	`clip_param`	0.1	PPO clip 范围
	`value_loss_coef`	0.5	Value loss 系数
	`entropy_coef`	0.01	Entropy 系数
	`max_grad_norm`	0.01	梯度裁剪阈值
GAE	`gamma`	0.9	折扣因子
	`gae_lambda`	0.95	GAE λ
环境	`verify_iter`	2	验证尝试次数
	`resolution`	1200	图片分辨率
生成	`temperature`	0.2	生成温度
	`max_new_tokens`	512	最大生成长度

12.2 完整配置文件¶

# rl/configs/llama_virl_vl.yaml

trainer: LlamaTrainer

# 梯度累积配置
grad_accum_steps: 128

# 优化器配置
optimizer_config:
  init_lr: !!float 1e-6      # 会被脚本覆盖为 1e-7
  eps: !!float 1e-7
  weight_decay: 0
  lr_max_steps: 100           # 会被脚本覆盖为 20
  end_lr: !!float 1e-9

# PPO 配置
ppo_config:
  clip_param: 0.1             # ε in PPO clip
  ppo_epoch: 4                # 每轮 PPO 训练 epochs
  mini_batch_size: 1          # 批量大小
  value_loss_coef: 0.5        # Value loss 权重
  entropy_coef: 0.01          # Entropy bonus 权重
  max_grad_norm: 0.01         # 梯度裁剪

# Return 计算配置
compute_return_kwargs:
  use_gae: true               # 使用 GAE
  gamma: 0.9                  # 折扣因子 γ
  gae_lambda: 0.95            # GAE λ
  use_proper_time_limits: False

# 训练配置
report_to: wandb              # 记录到 WandB
run_name: "virl_vl_training"
num_steps: 512                # 会被脚本覆盖为 256
num_processes: 1
num_updates: 20               # 会被脚本覆盖为 15

# 环境配置
env_config:
  id: 'gym_virl/Navigation-v0'
  route_info_path: ""
  resolution: 1200
  verify_iter: 2
  absolute_action: true
  relocation: true
  drop_rate: 0.5
  straight_line_length: 5

  platform_cfg:
    STREET_VIEW:
      SIZE: [640, 640]
      HEADING: 0
      PITCH: 0
      FOV: 90
      SOURCE: outdoor

    OFFLINE:
      ENABLED: True
      PANORAMA_DIR: ""
      GPS_TO_PANO_PATH: ""
      MAPPING_RADIUS: 20

  platform_save_dir: "./logs/"

# 模型配置
model: llama
model_path: ""

# Prompt 配置
prompt_config:
  relocation: true
  use_vision: true
  use_language: false
  enable_verification: true
  prompt_vision: ["Q_VIRL_VL"]
  pattern_vision: ["action"]

# 生成配置
generation_config:
  temperature: 0.2
  max_tokens: 300
  max_new_tokens: 512
  thought_prob_coef: 0.5
  num_beams: 1

# 输出配置
output_dir: logs/train.jsonl
seed: 42
save_ckpt: False
save_every: 1

12.3 训练脚本覆盖参数¶

# scripts/virl_training/vl_train.sh

# 训练参数
LR=1e-7
save_model=True
save_every=5  # 每 5 个 updates 保存一次
CKPT_NAME="tianzhechu/VIRL-VL-Init"
PORT=$((RANDOM % 10000 + 1000))

# 数据路径（使用绝对路径）
BASE_DIR="/root/SFTvsRL_Data/VIRL_routes"
ROUTE_INFO="${BASE_DIR}/nyc_1k_routes/route_infos.json"
GPS_TO_PANO="${BASE_DIR}/nyc_1k_routes/gps_pano_mapping.pkl"
STREETVIEWS="${BASE_DIR}/nyc_1k_routes/street_views/"

# 启动训练
DS_SKIP_CUDA_CHECK=1 TOKENIZERS_PARALLELISM=false \
    accelerate launch \
    --config_file scripts/config_zero2_8gpu.yaml \
    --main_process_port ${PORT} -m rl.launcher \
    -f rl/configs/llama_virl_vl.yaml \
    --output_dir=train_ckpt/virl_vl/ \
    --optimizer_config.init_lr=${LR} \
    --optimizer_config.lr_max_steps=20 \
    --prompt_config.enable_verification=True \
    --num_updates=15 \
    --num_steps=256 \
    --model_path=${CKPT_NAME} \
    --save_ckpt=${save_model} \
    --save_every=${save_every} \
    --env_config.route_info_path=${ROUTE_INFO} \
    --env_config.platform_cfg.OFFLINE.PANORAMA_DIR=${STREETVIEWS} \
    --env_config.platform_cfg.OFFLINE.GPS_TO_PANO_PATH=${GPS_TO_PANO}

12.4 DeepSpeed ZeRO-2 配置¶

# scripts/config_zero2_8gpu.yaml

compute_environment: LOCAL_MACHINE

deepspeed_config:
  offload_optimizer_device: cpu    # Optimizer offload 到 CPU
  offload_param_device: none       # 参数不 offload
  zero3_init_flag: false
  zero_stage: 2                    # ZeRO Stage 2
  overlap_comm: false

distributed_type: DEEPSPEED
downcast_bf16: 'yes'               # BF16 混合精度
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 8                   # 8 GPUs
rdzv_backend: static
same_network: true
use_cpu: false

12.5 关键超参数解释¶

为什么 lr=1e-7 这么小？ - RL 训练需要稳定性，大学习率会导致 policy collapse - Value network 从零初始化，需要谨慎训练 - 论文实验验证：1e-7 > 1e-6 (更稳定)

为什么 lr_max_steps=20 这么短？ - 只有 15 个 updates，20 步 LR 调度覆盖整个训练 - 每个 update 结束后调用 lr_scheduler.step() - 实际：~1.3 updates per LR step (20/15)

为什么 max_grad_norm=0.01 这么小？ - RL 训练容易出现梯度爆炸（importance sampling） - 严格的梯度裁剪保证训练稳定性 - 论文消融实验验证此值最优

为什么 verify_iter=2？ - 平衡探索与效率：2 次尝试足够学习 - 过多尝试 → 训练慢，信息冗余 - 过少尝试 → 难以恢复错误，训练不稳定

13. 评估方法¶

13.1 评估流程¶

# 1. In-Distribution Evaluation
bash scripts/virl_evaluation/vl_indist_eval.sh

# 2. Rule OOD Evaluation  
bash scripts/virl_evaluation/vl_rule_ood_eval.sh

# 3. Visual OOD Evaluation
bash scripts/virl_evaluation/vl_visual_ood_eval.sh

13.2 评估配置对比¶

配置项	In-Dist	Rule OOD	Visual OOD
`num_traj`	48	48	18
`absolute_action`	True	False	True
`route_info_path`	NYC routes	NYC routes	SF routes
`verify_iter`	2	2	2
GPU 数量	1	1	1

13.3 评估指标¶

13.3.1 Per-Step Accuracy¶

定义：单步动作正确率

per_step_accuracy = (
    num_correct_actions / total_actions
) × 100%

示例：

Route 1: 15 steps, 13 correct → 13/15 = 86.7%
Route 2: 20 steps, 18 correct → 18/20 = 90.0%
...
Route 48: 12 steps, 10 correct → 10/12 = 83.3%

Overall Per-Step Accuracy = 
    (13 + 18 + ... + 10) / (15 + 20 + ... + 12) = 87.5%

13.3.2 Success Rate¶

定义：完整路线成功率

success_rate = (
    num_successful_routes / total_routes
) × 100%

成功条件： 1. 到达正确目的地 2. 所有 waypoints 在允许尝试次数内通过 3. 未超过最大步数

示例：

48 routes:
- 35 routes: 成功 ✓
- 10 routes: 部分失败（某些 waypoint 错误）✗
- 3 routes: 完全失败（未到达目的地）✗

Success Rate = 35 / 48 = 72.9%

13.3.3 其他指标¶

metrics = {
    'mean_reward': 平均每条路线的总奖励,
    'std_reward': 奖励标准差,
    'mean_steps': 平均步数,
    'mean_verification_steps': 平均验证步数（包括重试）
}

13.4 评估输出示例¶

// logs/virl_vl_indist_verify_2/virl_vl_indist.jsonl

// Route 1, Step 0
{"sample_id": 0, "veri_step": 0, "output": "{\"action\": \"turn_direction(south)\"}", "reward": 1, "info": {...}}

// Route 1, Step 1
{"sample_id": 0, "veri_step": 1, "output": "{\"action\": \"forward()\"}", "reward": 1, "info": {...}}

// ... more steps ...

// Route 1, Final step
{"sample_id": 0, "veri_step": 14, "output": "{\"action\": \"stop()\"}", "reward": 1, "info": {...}}

// Route 1 Summary
{"Success": true, "sample_id": 0, "output": "{\"action\": \"stop()\"}", "reward": 15, "info": {...}}
{"Split": "===================="}

// Route 2, Step 0 (Failed attempt)
{"sample_id": 1, "veri_step": 0, "output": "{\"action\": \"turn_direction(north)\"}", "reward": -1, "info": {"Verify Info": "Incorrect action..."}}

// Route 2, Step 1 (Retry, Success)
{"sample_id": 1, "veri_step": 1, "output": "{\"action\": \"turn_direction(south)\"}", "reward": 1, "info": {...}}

// ... 47 more routes ...

// Overall Statistics
{
  "mean_reward": 12.5,
  "std_reward": 3.2,
  "success_rate": 0.729,
  "per_step_accuracy": 0.875,
  "mean_steps": 14.2,
  "mean_verification_steps": 1.15
}

13.5 评估脚本详解¶

#!/bin/bash
# scripts/virl_evaluation/vl_indist_eval.sh

VITER=2                    # 验证尝试次数
ENABLE=True                # 启用验证机制
ABS=True                   # 使用绝对动作空间
NUM_TRAJ=48                # 评估 48 条路线
CKPT_NAME="train_ckpt/virl_vl/checkpoint-epoch-14"  # 训练后的 checkpoint
OUTPUT_FOLDER="logs/virl_vl_indist_verify_${VITER}"
PORT=$((RANDOM % 10000 + 2000))

# 数据路径（使用绝对路径）
BASE_DIR="/root/SFTvsRL_Data/VIRL_routes"
ROUTE_INFO="${BASE_DIR}/nyc_1k_routes/route_infos.json"
GPS_TO_PANO="${BASE_DIR}/nyc_1k_routes/gps_pano_mapping.pkl"
STREETVIEWS="${BASE_DIR}/nyc_1k_routes/street_views/"

# 使用 1 GPU 进行评估
DS_SKIP_CUDA_CHECK=1 accelerate launch \
    --config_file scripts/config_zero2_1gpu.yaml \
    --main_process_port ${PORT} \
    -m evaluation.launcher \
    -f evaluation/configs/llama_virl_vl.yaml \
    --model_path=${CKPT_NAME} \
    --output_dir=${OUTPUT_FOLDER}/virl_vl_indist.jsonl \
    --env_config.route_info_path=${ROUTE_INFO} \
    --env_config.platform_cfg.OFFLINE.PANORAMA_DIR=${STREETVIEWS} \
    --env_config.platform_cfg.OFFLINE.GPS_TO_PANO_PATH=${GPS_TO_PANO} \
    --prompt_config.enable_verification=${ENABLE} \
    --env_config.verify_iter=${VITER} \
    --env_config.absolute_action=${ABS} \
    --num_traj=${NUM_TRAJ}

14. 实验结果分析¶

14.1 核心发现¶

根据论文 Figure 1 和实验数据：

14.1.1 In-Distribution 性能¶

模型	Per-Step Accuracy	Success Rate
SFT	~85%	~60%
RL (PPO)	~90%	~75%

结论：RL 在训练分布上也优于 SFT（+5% step accuracy）

14.1.2 Rule OOD 泛化¶

模型	In-Dist	Rule OOD	Generalization Gap
SFT	85%	15%	-70%
RL	90%	70%	-20%

关键发现： - SFT 在 Rule OOD 上崩溃（从 85% → 15%），说明记忆训练数据 - RL 保持 70% 准确率，说明学习了可泛化的规则

14.1.3 Visual OOD 泛化¶

模型	NYC (In-Dist)	SF (Visual OOD)	Generalization Gap
SFT	85%	<10%	-75%
RL	90%	~60%	-30%

关键发现： - SFT 在不同城市几乎失败，说明过度拟合视觉特征 - RL 在 SF 保持 60% 准确率，说明学习了可迁移的视觉表征

14.2 为什么 RL 能泛化？¶

论文通过消融实验分析了 RL 的泛化机制：

14.2.1 Outcome-based Reward 的作用¶

实验设计： - RL-Process: 每步给予反馈（+1/-1） - RL-Outcome: 只在 episode 结束给予奖励

结果： - RL-Outcome 在视觉识别（GeneralPoints-VL）上准确率更高 - 说明 outcome-based reward 迫使模型学习更好的视觉表征

原理：

Process Reward:
  Step 1: +1 (correct action, but maybe wrong reasoning)
  → 模型可能依赖 shortcuts（如记忆模式）

Outcome Reward:
  All steps: 0, 0, 0, ..., +10 (final success)
  → 模型必须学习端到端推理，包括视觉理解

14.2.2 SFT 作为 Format Teacher¶

实验：直接用 RL 从 base model 训练

结果：失败（见论文 Figure 20） - 模型无法输出结构化 JSON - 生成冗长的代码片段 - 无法收敛

结论： - SFT 稳定输出格式（"format teacher"） - RL 在此基础上学习策略和泛化能力 - SFT + RL 是最优组合

14.2.3 训练曲线对比¶

Per-Step Accuracy over Training

SFT:
  Update 0-5:   快速上升 (0% → 80%)
  Update 5-10:  继续上升 (80% → 85%)
  Update 10-20: 过拟合开始 (85% → 85%)

  Rule OOD: 持续下降 (80% → 15%)
  → 记忆训练规则

RL:
  Update 0-5:   稳定上升 (85% → 88%)
  Update 5-10:  继续上升 (88% → 90%)
  Update 10-15: 保持稳定 (90% → 90%)

  Rule OOD: 同步上升 (60% → 70%)
  → 学习可泛化规则

14.3 V-IRL Mini Benchmark SOTA¶

论文在 V-IRL 官方 benchmark 上达到 SOTA：

方法	Success Rate
GPT-4V (Yang et al., 2024)	44.0%
RL (Ours)	77.8%
提升	+33.8%

说明： - 多轮 RL 训练显著提升导航能力 - RL 的泛化优势在复杂真实环境中尤为明显

14.4 失败案例分析¶

论文提供了两类失败案例：

14.4.1 无 SFT 初始化的 RL 失败¶

问题：直接 RL 训练生成非结构化输出

示例输出：
"To solve this problem, we can use a brute force approach 
by generating all possible combinations... [生成 Python 代码]"

原因：Base model 未经过指令微调，不理解任务格式

14.4.2 过拟合 checkpoint 的 RL 失败¶

问题：从严重过拟合的 SFT checkpoint 开始 RL

示例：
  Rule: Relative actions
  Model Output: "turn_direction(northwest)"  # Still using absolute!

原因：SFT 过拟合太深，RL 无法纠正

启示： - 需要平衡 SFT 和 RL 的训练程度 - SFT 不宜训练过久（避免过拟合） - 论文建议：SFT 训练到合理格式输出即可

15. 参考文献¶

15.1 论文¶

主论文：Chu, T., Zhai, Y., Yang, J., et al. (2025). SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training. ICML 2025. arXiv:2501.17161
V-IRL 环境：Yang, J., et al. (2024). V-IRL: Grounding Virtual Intelligence in Real Life. V-IRL Platform
RL4VLM：Zhai, Y., et al. (2024). Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning. RL4VLM

15.2 代码仓库¶

推荐使用（）：bojieli/SFTvsRL ⭐
官方实现：LeslieTrue/SFTvsRL
项目主页：https://tianzhechu.com/SFTvsRL
数据集：HuggingFace - SFTvsRL_Data
模型 Checkpoints：HuggingFace - SFTvsRL Models

⚠️ 注意：请使用 bojieli fork 版本，它修复了官方版本中导致无法保存 checkpoint 的严重 bug（详见 4.3 节）。

15.3 相关工作¶

Llama-3.2-Vision：Dubey, A., et al. (2024). The Llama 3 Herd of Models. Meta AI.
PPO：Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347
GAE：Schulman, J., et al. (2016). High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016.

附录¶

A. 常见问题¶

Q0: 训练报错 TypeError: unsupported operand type(s) for %: 'int' and 'NoneType'？ - 这是官方代码的 bug！ - 解决方案：使用本指南开头固定的 fork/版本

git -C "${BOOK_ROOT}/chapter7/SFTvsRL" rev-parse HEAD
# 预期：fef0a4a3367260a0934be1e40b01e4021698e023

- 详细说明见 4.3 节

Q1: 为什么 RL 训练这么慢？ - 需要在线与环境交互（256 steps × 15 updates = 3,840 interactions） - 每步需要图片加载 + 模型推理（~2-3 秒/step） - PPO 训练需要 4 epochs × 256 samples（~1 小时/update）

Q2: 可以用更少的 GPUs 训练吗？ - 理论上可以，但需调整 grad_accum_steps 保持有效 batch size - 8 GPUs → 4 GPUs：grad_accum_steps 翻倍（128 → 256） - 训练时间会显著增加

Q3: 如何复现论文结果？ 1. 使用提供的 SFT checkpoint（tianzhechu/VIRL-VL-Init） 2. 严格按照超参数配置训练 15 updates 3. 在同样的评估集上测试（NYC 48 routes, SF 18 routes）

Q4: 为什么需要 SFT 初始化？ - 稳定输出格式（JSON 结构） - 提供基础指令跟随能力 - 加速 RL 收敛

B. 训练 Checklist¶

运行训练前确认：

[ ] 使用修复后的代码（chapter7/SFTvsRL，提交 fef0a4a3367260a0934be1e40b01e4021698e023）⭐
[ ] 验证 bug 已修复（grep "self.save_every = save_every" rl/trainer/base_trainer.py）
[ ] 安装所有依赖（pip install -r requirements.txt && cd gym && pip install -e .）
[ ] 下载并解压数据集到 /root/SFTvsRL_Data/VIRL_routes/
[ ] 解压 nyc_1k_routes.zip
[ ] 解压 VLN_mini.zip（用于 Visual OOD 评估）
[ ] 验证文件存在：route_infos.json, gps_pano_mapping.pkl, street_views/
[ ] 下载 SFT checkpoint（tianzhechu/VIRL-VL-Init）
[ ] 确认数据路径正确：
[ ] ROUTE_INFO="/root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/route_infos.json"
[ ] GPS_TO_PANO="/root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/gps_pano_mapping.pkl"
[ ] STREETVIEWS="/root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/street_views/"
[ ] 检查 GPU 数量和内存（8×80GB）
[ ] 配置 WandB API key（wandb login）

C. 评估 Checklist¶

运行评估前确认：

[ ] 训练完成并保存了 checkpoint（train_ckpt/virl_vl/checkpoint-epoch-*）
[ ] 数据集已下载并解压：
[ ] In-Dist & Rule OOD: /root/SFTvsRL_Data/VIRL_routes/nyc_1k_routes/
[ ] Visual OOD: /root/SFTvsRL_Data/VIRL_routes/VLN_mini/
[ ] 修改评估脚本：
[ ] 更新 CKPT_NAME="train_ckpt/virl_vl/checkpoint-epoch-14"
[ ] 确认 BASE_DIR="/root/SFTvsRL_Data/VIRL_routes"
[ ] 根据评估类型选择正确的 ROUTE_INFO 路径

Experiment 7-12 reproduction anchor¶

English¶

VIRL-VL: Vision-Language Navigation with Reinforcement Learning¶

Table of Contents¶

1. Experiment Overview¶

2. Experiment Objectives¶

2.1 Research Questions¶

2.2 Experiment Design Principles¶

3. Experiment Results¶

3.1 Main Results¶

3.2 Key Findings¶

4. Environment Setup¶

4.1 System Requirements¶

4.2 Installation Steps¶

4.3 Official Code Bugs and Fixes¶

4.4 Data Download and Path Configuration¶

4.4.1 Download Data¶

4.4.2 Final Directory Structure¶

4.4.3 Configure Training Script Paths (Optional)¶

4.5 Start Training¶

5. Task Definition¶

5.1 Navigation Task Description¶

5.2 Success Conditions¶

6. Action Space¶

6.1 Absolute Action Space¶

6.2 Relative Action Space¶

6.3 Action Space Configuration¶

7. RL Environment Details¶

7.1 Environment Class Structure¶

7.2 Ground Truth Rail¶

7.3 Verification Mechanism¶

7.4 Episode Termination Conditions¶

7.5 Environment Configuration¶

8. Trajectory Generation Mechanism¶

8.1 Trajectory Collection During Training¶

8.2 Multi-Episode Trajectory Example¶

8.3 LLM Input and Output Examples¶

9. Dataset¶

9.1 Training Dataset¶

9.2 Evaluation Datasets¶

9.3 Data Statistics¶

10. Vision Input Processing¶

10.1 Why Use 4-Direction Images Instead of Panoramas?¶

Design Motivation¶

Technical Comparison¶

Disadvantages of Panoramas¶

10.2 Street View Image Acquisition¶

10.3 Visual Input Diagram¶

10.4 Llama-3.2-Vision Image Processing¶

10.5 Vision Encoder Processing Flow¶

10.6 Vision-Related Hyperparameters¶

11. RL Training Process¶

11.1 Training Flow Overview¶

11.2 PPO (Proximal Policy Optimization) Algorithm Principle¶

11.2.1 Core Idea¶

11.2.2 Importance Sampling¶

11.2.3 PPO-Clip Objective Function¶

11.2.4 Advantage Calculation (GAE)¶

11.2.5 Value Function Loss¶

11.2.6 Entropy Bonus¶

11.2.7 Total Loss Function¶

11.2.8 Why is PPO Suitable for VLM Training?¶

11.2.9 PPO Configuration in This Experiment¶

11.3 Detailed PPO Code Implementation¶

11.3.1 Advantage Calculation (GAE) Code¶

11.3.2 PPO Loss Calculation Code¶

11.4 Model Architecture¶

11.5 DeepSpeed ZeRO Distributed Training Principles¶

11.5.1 Why DeepSpeed?¶

11.5.2 Core Idea of ZeRO¶

11.5.3 Detailed Mechanism of ZeRO Stage 2¶

11.5.4 Optimizer Offload Mechanism¶

11.5.5 Gradient Accumulation¶

11.5.6 Mixed Precision Training (BF16)¶

11.5.7 Actual Memory Usage Analysis¶

11.5.8 DeepSpeed Configuration Details¶

11.5.9 Actual Training Performance¶

11.6 Training Monitoring Metrics¶

12. Hyperparameter Configuration¶

12.1 Core Hyperparameter Overview¶