veRL Training Process: Complete System Analysis¶

Based on: SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning (2025)

Table of Contents¶

Overall Architecture
Training Configuration
Complete Training Loop
Phase 1: Rollout with Dynamic Sampling
Phase 2: Outcome Reward Modeling
Phase 3: Advantage Estimation (GRPO)
Phase 4: Actor Update (PPO with Clip Higher)
Phase 5: Validation
Data Flow and Synchronization
Performance Analysis
Key Paper Insights

Overall Architecture¶

System Components¶

┌─────────────────────────────────────────────────────────────────┐
│                         Ray Framework                            │
│  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐   │
│  │ ActorRollout   │  │    Critic      │  │   RefPolicy    │   │
│  │   Worker       │  │    Worker      │  │    Worker      │   │
│  │  (8 GPUs)      │  │  (8 GPUs)      │  │  (8 GPUs)      │   │
│  └────────────────┘  └────────────────┘  └────────────────┘   │
│         ↓                    ↓                    ↓             │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │            Shared GPU Resource Pool                       │  │
│  │         [GPU 0, GPU 1, ..., GPU 7]                       │  │
│  └──────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                              ↓
                    ┌─────────────────┐
                    │  RayTrainer     │
                    │  (Main Process) │
                    └─────────────────┘

Key Components (from main_ppo.py:159-173):

role_worker_mapping = {
    Role.ActorRollout: ray.remote(RobActorRolloutRefWorker),  # VLA model for rollout & training
    Role.Critic: ray.remote(CriticWorker),                     # Value network (GAE only)
    Role.RefPolicy: ray.remote(RobActorRolloutRefWorker)      # Frozen reference policy (KL penalty)
}

resource_pool_spec = {
    'global_pool': [8] * 1,  # 8 GPUs on 1 node
}

mapping = {
    Role.ActorRollout: 'global_pool',
    Role.Critic: 'global_pool',
    Role.RefPolicy: 'global_pool',
}

Worker Responsibilities: 1. ActorRollout Worker: - Generate rollouts in environments - Compute log probabilities - Update actor policy (backpropagation) - Compute entropy for exploration

Critic Worker (optional, only for GAE):
Estimate value function V(s)
Used for advantage estimation
RefPolicy Worker (optional, only if kl_coef > 0):
Compute reference log probabilities from frozen SFT model
Used for KL penalty regularization

Training Configuration¶

From run_openvla_oft_rl_twin2.sh:

Dataset Configuration¶

data.task_suite_name=robotwin2_beat_block_hammer
data.num_trials_per_task=1000           # 1000 task instances
data.n_samples=8                        # 8 rollouts per task instance
data.train_batch_size=64                # 64 task instances per training step
data.val_batch_size=256                 # 256 for validation
data.oversample_factor=1                # No oversampling

Model Configuration¶

actor_rollout_ref.model.path=$SFT_MODEL_PATH              # OpenVLA-7B SFT model
actor_rollout_ref.model.vla=openvla-oft
actor_rollout_ref.model.action_token_len=14               # 14 tokens per action
actor_rollout_ref.model.action_chunks_len=25              # 25 action chunks per VLA call

Training Configuration¶

# Optimization
actor_rollout_ref.actor.optim.lr=5e-6
actor_rollout_ref.actor.ppo_mini_batch_size=128           # Mini-batch size for PPO updates
actor_rollout_ref.actor.ppo_micro_batch_size=8            # Micro-batch size (= num_gpus)

# PPO Hyperparameters  
actor_rollout_ref.actor.clip_ratio_high=0.28             # PPO clip ratio upper bound
actor_rollout_ref.actor.clip_ratio_low=0.2               # PPO clip ratio lower bound
actor_rollout_ref.actor.entropy_coeff=0.0                # No entropy bonus
actor_rollout_ref.actor.grad_clip=1                      # Gradient clipping

# Advantage Estimation
algorithm.adv_estimator=grpo                              # GRPO (Group Relative Policy Optimization)
algorithm.kl_ctrl.kl_coef=0.00                           # No KL penalty (RefPolicy disabled)

# Training Schedule
trainer.total_epochs=100                                  # 100 epochs
trainer.save_freq=20                                      # Save checkpoint every 20 steps
trainer.test_freq=4                                       # Validate every 4 steps
trainer.val_before_train=True                             # Run validation before training

Computed Training Parameters¶

IMPORTANT: Terminology Clarification

There are THREE different types of "steps" in this system:

Training Steps (global_steps): Main training loop iterations
This is what we mean by "1500 steps"
One training step = collect rollouts + update policy
Environment Steps: Physical actions executed in simulation
Maximum 200 per rollout for beat_block_hammer task
These are the actual robot control steps
VLA Inference Steps: Number of VLA model forward passes
200 env steps ÷ 25 action_chunks_len = 8 VLA calls per rollout
Each VLA call generates 25 action chunks

Dataset Size (from rob_dataset.py:171-197):

# For RoboTwin 2.0, loads pre-collected feasible seeds
train_dataset_size = num_trials_per_task = 1000 task instances

# Training dataloader
train_batch_size = 64 * oversample_factor = 64
shuffle = True
drop_last = True

# Training steps per epoch (NOT environment steps!)
steps_per_epoch = train_dataset_size // train_batch_size 
                = 1000 // 64 = 15 training steps (last 40 samples dropped)

Total Training Steps (from ray_trainer.py:329-334):

total_training_steps = len(train_dataloader) * total_epochs
                     = 15 * 100 = 1500 training steps (per config)

⚠️ INCONSISTENCY ALERT: The paper (SimpleVLA-RL, Figure 3) shows training curves up to 300 RL Training Steps, not 1500!

This means either: 1. Paper uses fewer epochs: 300 / 15 = 20 epochs instead of 100 2. Config is for extended training: The provided config may be for experiments beyond the paper 3. Partial results shown: Paper may show early results from longer runs

Most likely: The paper experiments used trainer.total_epochs=20 (300 steps), while the provided config has trainer.total_epochs=100 for extended training.

Complete Training Loop¶

High-Level Structure (from `ray_trainer.py:469-717`)¶

def fit(self):
    global_steps = 0

    # ========== Initial Validation (Optional) ==========
    if val_before_train:
        val_metrics = self._validate(global_steps=0)
        # Runs 256 rollouts without training

    # ========== Main Training Loop ==========
    for epoch in range(total_epochs):  # 100 epochs
        self.train_dataloader.start_new_epoch()

        while True:  # Inner loop: process all batches in epoch
            # ------------------ Step 1: Rollout ------------------
            # Collect batch_size * n_samples = 64 * 8 = 512 rollouts
            valid_batch = []
            while len(valid_batch) < batch_size * n_samples:
                batch_dict = train_dataloader.get_next_batch()  # Get 64 task instances

                # Generate 8 rollouts per task instance
                gen_batch_output = actor_rollout_wg.generate_sequences(batch_dict)

                # Verify success and compute raw scores
                scores_tensor = reward_fn.verify(gen_batch_output)

                # Filter by accuracy if enabled
                filtered_batch = self.filter(scores_tensor, gen_batch_output, n_samples=8)

                valid_batch.append(filtered_batch)

            # ------------------ Step 2: Reference Policy (Optional) ------------------
            if use_reference_policy:
                ref_log_prob = ref_policy_wg.compute_ref_log_prob(valid_batch)
                valid_batch.union(ref_log_prob)

            # ------------------ Step 3: Reward Computation ------------------
            reward_tensor_dict, reward_metrics = reward_fn(valid_batch)
            valid_batch['token_level_scores'] = reward_tensor_dict['all']

            # Apply KL penalty if using reference policy
            valid_batch, kl_metrics = apply_kl_penalty(valid_batch, kl_ctrl)

            # ------------------ Step 4: Advantage Estimation ------------------
            valid_batch = compute_advantage(valid_batch, gamma, lam, adv_estimator='grpo')

            # ------------------ Step 5: Actor Update (Backpropagation) ------------------
            if global_steps >= critic_warmup:
                actor_output = actor_rollout_wg.update_actor(valid_batch)
                entropy_output = actor_rollout_wg.compute_entropy(valid_batch)

            # ------------------ Step 6: Validation ------------------
            if (global_steps + 1) % test_freq == 0:  # Every 4 steps
                val_metrics = self._validate(global_steps)

            # ------------------ Step 7: Checkpoint Saving ------------------
            if (global_steps + 1) % save_freq == 0:  # Every 20 steps
                actor_rollout_wg.save_checkpoint(checkpoint_path)

            global_steps += 1

            if len(valid_batch) == 0:  # Epoch exhausted
                break

    # ========== Final Validation ==========
    val_metrics = self._validate(global_steps=global_steps)

Training Summary¶

Metric	Value	Calculation	Notes
Epochs (Config File)	100	`trainer.total_epochs=100`	Extended training
Epochs (Paper)	20	Inferred from Figure 3	Actual paper experiments
Training Steps per Epoch	15	`1000 tasks // 64 batch_size`	Main loop iterations
Total Training Steps (Config)	1,500	`15 × 100 epochs`	If using config as-is
Total Training Steps (Paper)	300	`15 × 20 epochs`	Shown in Figure 3
Rollouts per Training Step	512	`64 tasks × 8 samples`	-
Total Rollouts (Paper)	153,600	`300 × 512 rollouts`	For 300 steps
Env Steps per Rollout	~50-200	Varies by task success	Physical robot actions
VLA Calls per Rollout	~2-8	`env_steps / 25`	Model forward passes
Total Env Steps (Paper)	~15M	`153.6k × 100 avg`	For 300 steps
Total VLA Inferences (Paper)	~600k	`153.6k × 4 avg`	For 300 steps
Validations (Paper)	75	`300 // 4 test_freq`	Every 4 training steps
Checkpoints (Paper)	15	`300 // 20 save_freq`	Every 20 training steps

Phase 1: Rollout¶

Rollout Process (from `ray_trainer.py:513-556`)¶

# 1. Load task batch from dataset
batch_dict = train_dataloader.get_next_batch()
# Returns: {
#   'task_suite_name': ['robotwin2_beat_block_hammer'] * 64,
#   'task_id': [-1] * 64,
#   'trial_id': [seed_1, seed_2, ..., seed_64],  # From pre-collected feasible seeds
#   'trial_seed': [seed_1, seed_2, ..., seed_64],
# }

# 2. Prepare generation batch
gen_batch = DataProto.from_single_dict(batch_dict)
gen_batch.select(batch_keys=['task_id', 'trial_id', 'trial_seed'])

# 3. Expand to n_samples copies (for GRPO/RLOO)
batch_lst = [[batch[i] for _ in range(n_samples)] for i in range(len(batch))]
# Creates 64 × 8 = 512 copies

# 4. Set generation config
gen_batch.meta_info = {
    'eos_token_id': tokenizer.eos_token_id,
    'n_samples': 8,  # Number of samples per task
    'pad_token_id': tokenizer.pad_token_id,
}

# 5. Generate rollouts (distributed across 8 GPUs)
gen_batch_output = actor_rollout_wg.generate_sequences(gen_batch)

Inside `generate_sequences` (from `rob_rollout.py:495-507`)¶

def generate_sequences(self, prompts):
    batch_size = prompts.batch.batch_size[0]  # 512

    # Use different batch sizes for training vs validation
    if prompts.meta_info.get('n_samples') is None:
        micro_batch_size = val_micro_batch_size  # 8 (validation)
    else:
        micro_batch_size = config.micro_batch_size  # 1 (training)

    num_chunks = max(batch_size // micro_batch_size, 1)  # 512 // 1 = 512 chunks
    batch_prompts = prompts.chunk(chunks=num_chunks)

    # Process each chunk sequentially (but distributed across GPUs by Ray)
    output = [self._generate_minibatch(p) for p in batch_prompts]
    output = DataProto.concat(output)

    return output

Detailed Rollout Execution (from `rob_rollout.py:609-775`)¶

For each minibatch (1 task instance with n_samples=8):

def _generate_minibatch_robotwin(prompts):
    # Extract task information
    batch_size = 8  # n_samples
    task_id = [-1] * 8
    trial_id = [same_seed] * 8  # All 8 samples use same seed
    trial_seed = [same_seed] * 8
    task_suite_name = ['robotwin2_beat_block_hammer'] * 8

    # ========== 1. Initialize Environments (Parallel) ==========
    env_wrappers = []
    for idx in range(8):
        wrapper = RobotwinEnvWrapper(task_name, trial_id[idx], trial_seed[idx], ...)
        env_wrappers.append(wrapper)

    # Initialize all 8 environments in parallel using ThreadPoolExecutor
    init_futures = [thread_pool.submit(w.initialize) for w in env_wrappers]
    wait_for_completion(init_futures)  # ~5-15 seconds (SAPIEN scene creation)

    # ========== 2. Collect Initial Observations ==========
    inputs = []
    task_descriptions = []
    for wrapper in env_wrappers:
        obs = wrapper.get_obs()  # Get RGB + proprioception
        instruction = wrapper.get_instruction()
        inputs.append(obs_to_input(obs))
        task_descriptions.append(instruction)

    # ========== 3. Main Rollout Loop ==========
    max_steps = 200  # Maximum ENVIRONMENT steps for beat_block_hammer
                     # This is NOT the same as training steps!
                     # These are physical robot control steps
    step = 0  # Current environment step counter
    vla_history = []

    while step < max_steps:
        active_indices = [i for i, w in enumerate(env_wrappers) if w.active]

        if len(active_indices) == 0:
            break  # All environments done

        # --- 3.1 Process VLA Input ---
        vla_input = process_input(inputs, task_descriptions)
        # Converts images to tensors, tokenizes text, normalizes proprio
        # Transfers to GPU, pads sequences
        # Shape: {
        #   'input_ids': (8, seq_len),
        #   'pixel_values': (8, num_images*patches, vision_dim),
        #   'proprio': (8, 14),
        #   'attention_mask': (8, seq_len),
        # }

        # --- 3.2 VLA Inference (GPU) ---
        vla_output = _generate_one_step(vla_input)
        # Forward pass through OpenVLA model:
        # - Vision encoder: Extract image features
        # - LLM: Generate action tokens (14 tokens × 25 chunks = 350 tokens)
        # - Action decoder: Convert tokens to continuous actions
        # Output: actions shape (8, 25, 14)
        #   - 25 action chunks
        #   - 14 dimensions (7 DOF × 2 arms)

        # --- 3.3 Execute Actions (CPU, Parallel) ---
        step_futures = []
        for idx in active_indices:
            future = thread_pool.submit(env_wrappers[idx].step, vla_output['action'][idx])
            step_futures.append((idx, future))

        # Wait for all environments to finish executing 25 action chunks
        new_inputs = []
        for idx, future in step_futures:
            obs, done = future.result()  # ~1500ms for 25 steps
            new_inputs.append(obs_to_input(obs))
            if done:
                env_wrappers[idx].active = False

        # --- 3.4 Store Trajectory Data ---
        vla_history.append({
            'responses': vla_output['responses'],        # (8, 25*14) action tokens
            'input_ids': vla_output['input_ids'],        # (8, seq_len) input tokens
            'attention_mask': vla_output['attention_mask'],
            'pixel_values': vla_output['pixel_values'],
            'action': vla_output['action'],              # (8, 25, 14) continuous actions
            'proprio': vla_output['proprio'],            # (8, 14) proprioception
            'step': step
        })

        inputs = new_inputs
        step += 25  # Advance by action_chunks_len (environment steps, not training steps!)
                    # After this loop, step will be at most 200 (max_steps)

    # ========== 4. Environment Cleanup ==========
    cleanup_futures = [thread_pool.submit(w.close) for w in env_wrappers]
    wait_for_completion(cleanup_futures)
    torch.cuda.empty_cache()

    # ========== 5. Prepare Output Batch ==========
    output_batch = prepare_output_batch(vla_history, task_records, batch_size=8)
    # Concatenates all timesteps: shape (8, num_steps, ...)
    # Adds metadata: complete (bool), finish_step (int)

    return output_batch

Rollout Output Format¶

output_batch = {
    'responses': Tensor(8, num_steps, 350),        # Action tokens (25*14)
    'input_ids': Tensor(8, num_steps, seq_len),    # Input token IDs
    'attention_mask': Tensor(8, num_steps, seq_len),
    'pixel_values': Tensor(8, num_steps, patches, dim),
    'proprio': Tensor(8, num_steps, 14),           # Proprioception
    'complete': Tensor(8),                          # Success flags
    'finish_step': Tensor(8),                       # Number of steps executed
}

Rollout Timing Breakdown¶

For 1 task instance with 8 samples:

Phase	Duration	Notes
Environment Init	5-15s	SAPIEN scene creation (parallel)
VLA Inference (×8 steps)	~2.4s	300ms × 8, GPU 100%
Action Execution (×8 steps)	~12s	1500ms × 8, CPU-bound
Environment Cleanup	1-3s	Close scenes, save videos
Total	20-30s	Per minibatch

For full training step (512 rollouts = 64 tasks × 8 samples): - With 8 GPUs processing in parallel: ~30-40 seconds - Ray schedules 64 tasks across 8 workers dynamically

Phase 2: Reward Computation¶

2.1 Verification (from `ray_trainer.py:559-570`)¶

# Called during rollout collection
scores_tensor, reward_metrics, format_metrics, _ = reward_fn.verify(roll_batch)

Verification Logic (from `main_ppo.py:42-60`)¶

def verify(self, data):
    completes = data.batch['complete'].tolist()  # Success flags from environments
    batch_size = data.batch['responses'].size(0)  # 512

    # Convert boolean success to float rewards
    score = [float(item) for item in completes]  # [0.0, 1.0, 0.0, ...]

    # Format correctness (always 1.0 for VLA)
    format = [1.0] * batch_size

    # Store in batch
    data.batch['acc'] = torch.tensor(score, dtype=torch.float32)
    data.batch['format_correctness'] = torch.tensor(format, dtype=torch.float32)

    # Compute metrics
    reward_metrics = {'all': data.batch['acc'].mean().item()}  # Average success rate
    format_metrics = {'all': 1.0}

    return score, reward_metrics, format_metrics, reward_format_metrics

2.2 Filtering (from `ray_trainer.py:573-578`)¶

if config.data.filter_accuracy:
    # Filter rollouts by average success rate per task
    # Keeps tasks with success rate in [accuracy_lower_bound, accuracy_upper_bound]
    # Default: [0.1, 0.9]
    filtered_batch = self.filter(scores_tensor, roll_batch, n_samples=8)

Dynamic Sampling (Paper Section 3.3)¶

Key Insight from Paper: This implements "Dynamic Sampling" from the paper, which ensures non-zero advantage estimates and stable gradients by filtering out groups where all trajectories either succeed or fail.

Mathematical Formulation (Equation 10 from paper):

0 < |{traj_i(a_i, s_i) | is_successful(traj_i(a_i, s_i))}| < G

Where G is the group size (n_samples = 8).

Implementation (from ray_trainer.py:755-816):

def filter(reward_tensor, batch, n_samples=8):
    # Reshape to (num_tasks, n_samples)
    reward_matrix = reward_tensor.sum(-1).reshape(-1, n_samples)  # (64, 8)

    # Compute average success rate per task group
    acc_tensor = torch.mean(reward_matrix, dim=-1)  # (64,)
    # Example: [0.0, 0.25, 0.5, 0.75, 1.0, ...]

    # Dynamic Sampling: Filter by accuracy bounds
    # Keeps only groups with MIXED outcomes (not all success or all failure)
    acc_mask = (acc_tensor >= 0.1) & (acc_tensor <= 0.9)
    # 0.1 threshold: excludes all-failure groups (0.0)
    # 0.9 threshold: excludes all-success groups (1.0)

    # Expand mask to cover all samples in kept groups
    final_mask = acc_mask.repeat_interleave(n_samples)  # (64*8,)

    # Apply mask
    filtered_batch = batch.slice(final_mask)

    return filtered_batch

Dynamic Sampling Benefits (from paper): 1. Non-zero advantages: Groups with all success/failure have zero variance, leading to zero advantages 2. Stable gradients: Mixed outcomes provide meaningful learning signal 3. Curriculum learning: Naturally focuses on tasks of appropriate difficulty 4. Proven effective in LLM RL: Based on recent LLM RL research (Yu et al., 2025; Cui et al., 2025a)

Filtering Effect: - Original batch: 64 tasks × 8 samples = 512 rollouts - After Dynamic Sampling: ~30-50 tasks × 8 samples = ~240-400 rollouts (varies) - Excluded: ~20% all-failure groups + ~10% all-success groups - Remaining: ~70% mixed-outcome groups for effective training

2.3 Outcome Reward Modeling (Paper Section 3.2)¶

Key Insight from Paper: SimpleVLA-RL uses outcome-only rewards (success/failure) without hand-crafted process rewards. This is inspired by LLM RL success (DeepSeek-R1) and addresses the scalability challenges of designing dense rewards for every task.

2.3.1 Reward Shaping (from `ray_trainer.py:631-647`)¶

def reward_fn(batch):
    # Initialize reward tensors
    reward_tensor = torch.zeros_like(batch['responses'], dtype=torch.float32)  # (N, num_steps, 350)
    verifier_reward = torch.zeros_like(batch['responses'], dtype=torch.float32)

    # Reshape to (N, num_steps * 350)
    reward_tensor = reward_tensor.reshape((reward_tensor.shape[0], -1))
    verifier_reward = verifier_reward.reshape((verifier_reward.shape[0], -1))

    # Compute valid response length
    valid_response_length = batch['finish_step'] * action_token_len  # finish_step × 14

    # Place reward at the end of trajectory (sparse reward)
    verifier_score = batch['acc'].cpu().numpy().tolist()  # [0.0, 1.0, 0.0, ...]
    for i in range(verifier_reward.shape[0]):
        verifier_reward[i, valid_response_length[i] - 1] += verifier_score[i]

    reward_tensor_dict = {'gt_scores': verifier_reward}

    # Apply reward coefficient
    if config.verifier.reward_coef != 0:
        reward_tensor += config.verifier.reward_coef * reward_tensor_dict['gt_scores']

    reward_tensor_dict['all'] = reward_tensor

    return reward_tensor_dict, reward_metrics

Reward Structure:

Trajectory:  [a_0, a_1, a_2, ..., a_T]
Rewards:     [0.0, 0.0, 0.0, ..., r_T]  # Sparse reward at end
                                  ↑
                                  Success (1.0) or Failure (0.0)

Phase 3: Advantage Estimation (GRPO)¶

Algorithm: Group Relative Policy Optimization (GRPO)

Key Insight from Paper: GRPO is specifically designed for outcome-based rewards in RL. Instead of traditional GAE which requires value functions and dense rewards, GRPO uses relative comparisons within groups of trajectories from the same task.

3.1 KL Penalty (Optional) (from `ray_trainer.py:642-647`)¶

batch, kl_metrics = apply_kl_penalty(
    batch, 
    kl_ctrl=self.kl_ctrl,
    kl_penalty='kl',
    action_token_len=14,
    action_chunks_len=25
)

KL Penalty Logic (from `ray_trainer.py:85-120`)¶

def apply_kl_penalty(data, kl_ctrl, kl_penalty='kl', action_token_len=14, action_chunks_len=25):
    responses = data.batch['responses']  # (N, num_steps, 350)

    # Compute trajectory length
    traj_length = responses.size(1) * action_chunks_len  # num_steps × 25
    token_level_scores = data.batch['token_level_scores']  # Raw rewards

    # Compute response mask (valid tokens)
    finish_step = data.batch['finish_step'] * action_token_len  # finish_step × 14
    steps = torch.arange(traj_length * action_token_len)
    steps_expanded = steps.unsqueeze(0).expand(batch_size, -1)
    response_mask = steps_expanded < finish_step.unsqueeze(1)

    # Compute KL divergence if reference policy is used
    if 'ref_log_prob' in data.batch.keys():
        # KL = log π(a|s) - log π_ref(a|s)
        kld = old_log_probs - ref_log_prob  # (N, response_length)
        kld = kld * response_mask
        beta = kl_ctrl.value  # KL coefficient (adaptive or fixed)
    else:
        beta = 0
        kld = torch.zeros_like(response_mask, dtype=torch.float32)

    # Apply KL penalty to rewards
    token_level_rewards = token_level_scores - beta * kld

    # Update KL controller
    current_kl = masked_mean(kld, mask=response_mask, axis=-1).mean().item()
    kl_ctrl.update(current_kl=current_kl, n_steps=batch_size)

    data.batch['token_level_rewards'] = token_level_rewards

    metrics = {'critic/kl': current_kl, 'critic/kl_coeff': beta}
    return data, metrics

3.2 Advantage Computation (from `ray_trainer.py:649-655`)¶

batch = compute_advantage(
    batch,
    gamma=1.0,                         # config.algorithm.gamma (not shown in script)
    lam=0.95,                          # config.algorithm.lam (not shown in script)
    adv_estimator='grpo',              # Group Relative Policy Optimization
    config=config
)

GRPO Advantage Estimation (from `ray_trainer.py:160-173`)¶

def compute_advantage(data, gamma, lam, adv_estimator='grpo', config):
    responses = data.batch['responses']
    response_length = responses.size(1) * responses.size(2)  # num_steps × 350

    # Compute response mask
    finish_step = data.batch['finish_step'] * action_token_len
    steps = torch.arange(response_length)
    steps_expanded = steps.unsqueeze(0).expand(batch_size, -1)
    response_mask = steps_expanded < finish_step.unsqueeze(1)

    token_level_rewards = data.batch['token_level_rewards']
    index = data.non_tensor_batch['uid']  # Unique ID per task

    # GRPO: Group samples by task, use mean as baseline
    advantages, returns = core_algos.compute_grpo_outcome_advantage(
        token_level_rewards=token_level_rewards,
        eos_mask=response_mask,
        index=index  # Groups samples from same task
    )

    data.batch['advantages'] = advantages
    data.batch['returns'] = returns

    return data

GRPO Algorithm (Paper Section 2.3 & `verl/trainer/ppo/core_algos.py`)¶

Paper Definition: GRPO is an outcome-based advantage estimation method that groups trajectories from the same prompt/task and uses the group mean as a baseline.

Why GRPO for VLA? 1. No value function needed: Unlike GAE, doesn't require training a separate critic 2. Works with sparse rewards: Designed for outcome-only rewards (success/failure) 3. Relative learning: Encourages better actions relative to other attempts at the same task 4. Variance reduction: Group mean provides better baseline than zero

def compute_grpo_outcome_advantage(token_level_rewards, eos_mask, index):
    """
    Group Relative Policy Optimization (GRPO)

    Paper Formula (Section 2.3):
    A_i = R_i - (1/N) Σ_{j=1}^N R_j

    Where:
    - R_i: Total return for trajectory i
    - N: Number of samples per task group (n_samples = 8)
    - A_i: Advantage (how much better than group average)

    Process:
    1. Compute total return for each sample: R_i = Σ rewards
    2. Group samples by task (using unique ID)
    3. Compute group baseline: b = mean(R_1, ..., R_N)
    4. Compute advantages: A_i = R_i - b

    This encourages relative improvement within each task group.
    """
    batch_size = token_level_rewards.size(0)

    # Compute total returns (sum of rewards over trajectory)
    returns = (token_level_rewards * eos_mask).sum(-1)  # (batch_size,)

    # Group by task index (uid)
    unique_indices = torch.unique(index)
    advantages = torch.zeros_like(returns)

    for idx in unique_indices:
        mask = (index == idx)  # All 8 samples from same task
        group_returns = returns[mask]  # N=8 samples

        # Compute group baseline (mean return across 8 samples)
        baseline = group_returns.mean()

        # Compute relative advantages (how much better than average)
        group_advantages = group_returns - baseline
        advantages[mask] = group_advantages

    # Broadcast advantages to all timesteps
    advantages_broadcast = advantages.unsqueeze(-1).expand_as(token_level_rewards)
    advantages_broadcast = advantages_broadcast * eos_mask

    returns_broadcast = returns.unsqueeze(-1).expand_as(token_level_rewards)
    returns_broadcast = returns_broadcast * eos_mask

    return advantages_broadcast, returns_broadcast

GRPO Example:

Task 1 (8 samples):
- Returns: [0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0]
- Baseline: mean = 0.5
- Advantages: [-0.5, 0.5, -0.5, 0.5, -0.5, 0.5, 0.5, -0.5]

Task 2 (8 samples):
- Returns: [0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0]
- Baseline: mean = 0.25
- Advantages: [-0.25, -0.25, -0.25, -0.25, 0.75, 0.75, -0.25, -0.25]

Phase 4: Actor Update (PPO with Clip Higher)¶

Algorithm: Proximal Policy Optimization (PPO) with modified clipping range

Key Insight from Paper (Section 3.3 - "Clipping Higher"): - Standard PPO clips ratio to [1-ε, 1+ε] typically [0.8, 1.2] - SimpleVLA-RL increases upper bound to 1.28: [0.8, 1.28] - Rationale: Higher upper bound allows more exploration by not restricting probability increase of low-probability actions - Inspired by: DAPO (Yu et al., 2025) for LLM RL - Result: Shown in Figure 3(b) - ~10% improvement over baseline by step 300

4.1 Update Trigger (from `ray_trainer.py:660-671`)¶

if global_steps >= critic_warmup:  # Default: critic_warmup = 0
    # Update actor using PPO
    batch.meta_info['is_filtered'] = True
    batch.meta_info['train_mode'] = False

    actor_output = actor_rollout_wg.update_actor(batch)

    # Compute entropy for exploration
    entropy_output = actor_rollout_wg.compute_entropy(data=batch)

4.2 PPO Update (in ActorRolloutRefWorker)¶

def update_actor(self, batch):
    """
    Proximal Policy Optimization (PPO) update

    Batch contains:
    - responses: (N, num_steps, 350) action tokens
    - old_log_probs: log π_old(a|s)
    - advantages: A(s, a)
    - returns: R(s, a)
    - attention_mask: valid token mask
    """

    # ========== PPO Configuration ==========
    ppo_mini_batch_size = 128   # Mini-batch size for PPO updates
    ppo_micro_batch_size = 8    # Micro-batch size (= num_gpus)
    clip_ratio_low = 0.2        # PPO clip lower bound (1 - 0.2 = 0.8)
    clip_ratio_high = 0.28      # PPO clip upper bound (1 + 0.28 = 1.28) ← CLIP HIGHER!
    num_ppo_epochs = 1          # Number of PPO epochs (default: 1)

    # Note: clip_ratio_high = 0.28 means clipping at 1.28 (not 1.2)
    # This is the "Clip Higher" enhancement from the paper

    # ========== Prepare Data ==========
    batch_size = len(batch)  # After filtering: ~240-400 rollouts

    # Shuffle data for mini-batch SGD
    indices = torch.randperm(batch_size)
    shuffled_batch = batch.slice(indices)

    # ========== Mini-Batch Loop ==========
    num_mini_batches = batch_size // ppo_mini_batch_size  # ~2-3 mini-batches

    metrics = defaultdict(list)

    for mini_batch_idx in range(num_mini_batches):
        start = mini_batch_idx * ppo_mini_batch_size
        end = start + ppo_mini_batch_size
        mini_batch = shuffled_batch.slice(range(start, end))

        # ========== Micro-Batch Loop (for memory efficiency) ==========
        num_micro_batches = ppo_mini_batch_size // ppo_micro_batch_size  # 128 // 8 = 16

        accumulated_loss = 0.0

        for micro_batch_idx in range(num_micro_batches):
            micro_start = micro_batch_idx * ppo_micro_batch_size
            micro_end = micro_start + ppo_micro_batch_size
            micro_batch = mini_batch.slice(range(micro_start, micro_end))

            # --- Forward Pass ---
            with torch.autocast(device_type='cuda', dtype=torch.bfloat16):
                output = model(
                    input_ids=micro_batch['input_ids'],
                    pixel_values=micro_batch['pixel_values'],
                    proprio=micro_batch['proprio'],
                    attention_mask=micro_batch['attention_mask'],
                    labels=micro_batch['responses'],  # Teacher forcing
                )

                logits = output.logits  # (micro_batch_size, seq_len, vocab_size)

                # Compute log probabilities
                log_probs = F.log_softmax(logits, dim=-1)

                # Gather log probs for actual actions
                action_log_probs = torch.gather(
                    log_probs, 
                    dim=-1, 
                    index=micro_batch['responses'].unsqueeze(-1)
                ).squeeze(-1)  # (micro_batch_size, seq_len)

                # Get old log probs from rollout
                old_log_probs = micro_batch['old_log_probs']

                # Compute probability ratio: π(a|s) / π_old(a|s)
                ratio = torch.exp(action_log_probs - old_log_probs)

                # Get advantages
                advantages = micro_batch['advantages']

                # Normalize advantages (per micro-batch)
                advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)

                # --- PPO Clipped Objective with "Clip Higher" ---
                # Standard PPO: L_CLIP = min(ratio * A, clip(ratio, 0.8, 1.2) * A)
                # SimpleVLA-RL: L_CLIP = min(ratio * A, clip(ratio, 0.8, 1.28) * A)
                # The 1.28 upper bound allows more exploration!
                clipped_ratio = torch.clamp(ratio, 1 - clip_ratio_low, 1 + clip_ratio_high)
                # = torch.clamp(ratio, 0.8, 1.28)

                loss_unclipped = ratio * advantages
                loss_clipped = clipped_ratio * advantages

                policy_loss = -torch.min(loss_unclipped, loss_clipped)

                # Apply mask to only valid tokens
                response_mask = micro_batch['attention_mask'][:, -action_log_probs.size(1):]
                policy_loss = (policy_loss * response_mask).sum() / response_mask.sum()

                # Value loss (if using critic)
                # value_loss = F.mse_loss(values, returns)

                # Total loss
                loss = policy_loss  # + value_coef * value_loss

                # Normalize by num_micro_batches for gradient accumulation
                loss = loss / num_micro_batches

            # --- Backward Pass ---
            loss.backward()

            accumulated_loss += loss.item()

            # Store metrics
            metrics['actor/policy_loss'].append(policy_loss.item())
            metrics['actor/ratio_mean'].append(ratio.mean().item())
            metrics['actor/ratio_max'].append(ratio.max().item())
            metrics['actor/ratio_min'].append(ratio.min().item())
            metrics['actor/clip_fraction'].append(
                ((ratio < 1 - clip_ratio_low) | (ratio > 1 + clip_ratio_high)).float().mean().item()
            )

        # --- Gradient Clipping and Optimizer Step (after accumulating micro-batches) ---
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

        optimizer.step()
        optimizer.zero_grad()

    # Average metrics
    for key in metrics:
        metrics[key] = np.mean(metrics[key])

    return DataProto(meta_info={'metrics': metrics})

PPO Update Summary¶

Parameter	Value	Notes
Batch Size	~240-400	After accuracy filtering
Mini-Batch Size	128	For PPO updates
Micro-Batch Size	8	For memory efficiency
Num Mini-Batches	~2-3	240-400 // 128
Num Micro-Batches	16	128 // 8
Clip Ratio	[0.2, 0.28]	PPO clipping range
Learning Rate	5e-6	Adam optimizer
Gradient Clipping	1.0	Max norm

Timing: - Forward + Backward per micro-batch: ~500ms - Total per mini-batch: 16 × 500ms = 8s - Total per step: 2-3 × 8s = ~16-24s

4.3 Entropy Computation (from `ray_trainer.py:666`)¶

entropy_output = actor_rollout_wg.compute_entropy(data=batch)

Purpose: Measure policy entropy for exploration monitoring

def compute_entropy(data):
    # Forward pass to get logits
    logits = model(...)

    # Compute entropy: H = -Σ p(a) log p(a)
    probs = F.softmax(logits, dim=-1)
    log_probs = F.log_softmax(logits, dim=-1)
    entropy = -(probs * log_probs).sum(-1)  # (batch_size, seq_len)

    # Average over valid tokens
    entropy_mean = (entropy * response_mask).sum() / response_mask.sum()

    return DataProto(meta_info={'metrics': {'actor/entropy': entropy_mean.item()}})

Phase 5: Validation¶

5.1 Validation Trigger (from `ray_trainer.py:673-679`)¶

if (global_steps + 1) % test_freq == 0:  # Every 4 steps
    val_metrics = self._validate(global_steps=global_steps+1)
    val_metrics = {f'val/{key}': val for key, val in val_metrics.items()}
    logger.log(data=val_metrics, step=global_steps)

5.2 Validation Process (from `ray_trainer.py:336-391`)¶

def _validate(self, global_steps=0):
    reward_tensor_lst = []
    data_source_lst = []

    # Loop over validation dataloader
    for test_data in val_dataloader:  # 256 / 256 = 1 batch
        test_batch = DataProto.from_single_dict(test_data)

        # Set validation config
        test_batch.meta_info = {
            'eos_token_id': tokenizer.eos_token_id,
            'pad_token_id': tokenizer.pad_token_id,
            'recompute_log_prob': False,
            'do_sample': False,           # Greedy sampling for validation
            'validate': True,              # Validation mode flag
            'global_steps': global_steps   # For video naming
        }

        # Generate rollouts (no n_samples, single rollout per task)
        test_output_gen_batch = actor_rollout_wg.generate_sequences(test_batch)
        # Uses val_micro_batch_size = 8 (larger than training micro_batch_size = 1)

        test_batch = test_batch.union(test_output_gen_batch)

        # Evaluate using reward function
        verifier_score, reward_metrics, format_metrics, _ = val_reward_fn.verify(test_batch)
        reward_tensor = torch.tensor(verifier_score, dtype=torch.float32).unsqueeze(-1)

        # Log metrics
        for k, v in reward_metrics.items():
            metric_dict['test_reward/' + k] = v

        reward_tensor_lst.append(reward_tensor)
        data_source_lst.append(test_batch.non_tensor_batch['data_source'])

    # Aggregate results
    reward_tensor = torch.cat(reward_tensor_lst, dim=0).sum(-1).cpu()  # (256,)
    data_sources = np.concatenate(data_source_lst, axis=0)

    # Compute metrics by data source
    data_source_reward = {}
    for i in range(reward_tensor.shape[0]):
        data_source = data_sources[i]
        if data_source not in data_source_reward:
            data_source_reward[data_source] = []
        data_source_reward[data_source].append(reward_tensor[i].item())

    metric_dict = {}
    for data_source, rewards in data_source_reward.items():
        metric_dict[f'test_score/{data_source}'] = np.mean(rewards)

    metric_dict[f'test_score/all'] = reward_tensor.mean().item()

    return metric_dict

Validation Dataset (from `rob_dataset.py:136-209`)¶

# For validation
val_dataset = Robotwin_Dataset(
    task_suite_name='robotwin2_beat_block_hammer',
    num_trials_per_task=128,  # Overridden for validation
    train_val='valid'
)

# Loads two types of seeds:
# 1. IID seeds: From training distribution (128 seeds)
# 2. OOD seeds: From eval distribution (128 seeds)
# Total: 256 task instances

# Data source labels:
# - 'robotwin2_beat_block_hammer_train_iid'
# - 'robotwin2_beat_block_hammer_eval_ood'

Validation Metrics¶

{
    'val/test_score/robotwin2_beat_block_hammer_train_iid': 0.65,  # IID success rate
    'val/test_score/robotwin2_beat_block_hammer_eval_ood': 0.42,   # OOD success rate
    'val/test_score/all': 0.535,                                   # Overall success rate
}

Validation Timing: - 256 rollouts with val_micro_batch_size=8 - Distributed across 8 GPUs: 256 / 8 = 32 rollouts per GPU - Per rollout: ~20-30s - Total: ~30-40s (parallel execution)

Data Flow and Synchronization¶

Overall Data Flow¶

┌──────────────────────────────────────────────────────────────────┐
│                      Training Step i                              │
└──────────────────────────────────────────────────────────────────┘
                              │
                              ↓
   ┌──────────────────────────────────────────────────────────┐
   │  1. Dataset Sampling                                      │
   │     - Sample 64 task instances from dataset              │
   │     - Each with unique feasible seed                     │
   │     - Expand to 64 × 8 = 512 copies for n_samples       │
   └──────────────────────────────────────────────────────────┘
                              │
                              ↓
   ┌──────────────────────────────────────────────────────────┐
   │  2. Rollout (Distributed across 8 GPUs via Ray)          │
   │     - Each GPU processes ~64 rollouts (8 tasks × 8)      │
   │     - Initialize environments                            │
   │     - VLA inference + action execution loop              │
   │     - Collect trajectory data                            │
   │     - Output: (512, num_steps, ...)                      │
   └──────────────────────────────────────────────────────────┘
                              │
                              ↓
   ┌──────────────────────────────────────────────────────────┐
   │  3. Verification & Filtering (Main Process)              │
   │     - Compute success flags                              │
   │     - Filter by accuracy: keep 10%-90% success rate      │
   │     - Output: (~300, num_steps, ...)                     │
   └──────────────────────────────────────────────────────────┘
                              │
                              ↓
   ┌──────────────────────────────────────────────────────────┐
   │  4. Reference Policy (Optional, Distributed)             │
   │     - Compute log π_ref(a|s) for KL penalty              │
   │     - Only if kl_coef > 0                                │
   └──────────────────────────────────────────────────────────┘
                              │
                              ↓
   ┌──────────────────────────────────────────────────────────┐
   │  5. Reward Computation (Main Process)                    │
   │     - Apply reward shaping (sparse reward at end)        │
   │     - Apply KL penalty if using ref policy               │
   │     - Output: token_level_rewards                        │
   └──────────────────────────────────────────────────────────┘
                              │
                              ↓
   ┌──────────────────────────────────────────────────────────┐
   │  6. Advantage Estimation (Main Process)                  │
   │     - GRPO: Group samples by task                        │
   │     - Compute advantages = returns - group_mean          │
   │     - Output: advantages, returns                        │
   └──────────────────────────────────────────────────────────┘
                              │
                              ↓
   ┌──────────────────────────────────────────────────────────┐
   │  7. Actor Update (Distributed across 8 GPUs)             │
   │     - PPO mini-batch loop (2-3 mini-batches)            │
   │     - Micro-batch loop (16 micro-batches per mini-batch)│
   │     - Forward + backward + gradient accumulation         │
   │     - Gradient clipping + optimizer step                 │
   │     - Output: actor_metrics                              │
   └──────────────────────────────────────────────────────────┘
                              │
                              ↓
   ┌──────────────────────────────────────────────────────────┐
   │  8. Entropy Computation (Distributed)                    │
   │     - Measure policy entropy for exploration monitoring  │
   │     - Output: entropy_metrics                            │
   └──────────────────────────────────────────────────────────┘
                              │
                              ↓
   ┌──────────────────────────────────────────────────────────┐
   │  9. Validation (Every 4 steps, Distributed)              │
   │     - 256 rollouts (greedy sampling)                     │
   │     - Compute success rates (IID and OOD)                │
   │     - Output: val_metrics                                │
   └──────────────────────────────────────────────────────────┘
                              │
                              ↓
   ┌──────────────────────────────────────────────────────────┐
   │  10. Checkpoint Saving (Every 20 steps)                  │
   │      - Save actor model weights                          │
   │      - Save critic model weights (if GAE)                │
   │      - Save optimizer state                              │
   └──────────────────────────────────────────────────────────┘
                              │
                              ↓
                      global_steps += 1

Ray Distributed Execution¶

# Worker allocation (8 GPUs)
actor_rollout_wg.world_size = 8   # 8 workers, 1 per GPU
actor_rollout_wg.local_size = 8   # All on same node

# When calling actor_rollout_wg.generate_sequences(batch)
# Ray automatically distributes workload:

batch_size = 512  # 64 tasks × 8 samples
world_size = 8
micro_batch_size = 1

# Each GPU processes:
local_batch_size = batch_size // world_size = 512 // 8 = 64 rollouts

# Ray scheduling:
# GPU 0: processes rollouts [0:64]
# GPU 1: processes rollouts [64:128]
# GPU 2: processes rollouts [128:192]
# ...
# GPU 7: processes rollouts [448:512]

# All GPUs run in parallel, synchronization at end

FSDP Model Sharding¶

actor_rollout_ref.actor.fsdp_config.param_offload=False
actor_rollout_ref.actor.fsdp_config.grad_offload=True
actor_rollout_ref.actor.fsdp_config.optimizer_offload=True

FSDP Strategy: - Param Offload: Disabled → Parameters stay on GPU - Grad Offload: Enabled → Gradients offloaded to CPU after backward - Optimizer Offload: Enabled → Optimizer states kept on CPU

Model Sharding:

OpenVLA-7B Model:
├── Vision Backbone (400M-700M params)  → Sharded across 8 GPUs
├── Projector (8M params)               → Sharded across 8 GPUs
├── LLM (7B params)                     → Sharded across 8 GPUs
└── Total: ~7.5B params                 → ~940MB per GPU (in bfloat16)

Performance Analysis¶

Training Step Timing Breakdown¶

ORIGINAL ESTIMATE (WRONG!):

Phase	Estimated Duration	GPU Util	Notes
1. Rollout	30-40s	15%	Bottleneck: CPU physics simulation
2. Verification & Filtering	1-2s	0%	CPU operations
3. Reference Policy	2-3s	80%	If kl_coef > 0 (disabled in config)
4. Reward Computation	<1s	0%	CPU tensor operations
5. Advantage Estimation	<1s	0%	CPU tensor operations
6. Actor Update	16-24s	90%	PPO mini-batch loop
7. Entropy Computation	1-2s	80%	Additional forward pass
8. Logging	<1s	0%	Write metrics to WandB
Total per step	50-70s	30-40%	Average GPU utilization

ACTUAL REALITY (from user logs):

Phase	Actual Duration	GPU Util	Notes
1. Rollout	1070-1100s (~18 min)	15-20%	Collects ~896 rollouts (to get 512 after filtering)
2. Verification & Filtering	4-5s	0%	Filters 896 → 512 rollouts
3. Reference Policy	0s	0%	Disabled (kl_coef=0)
4. Reward Computation	<1s	0%	CPU tensor operations
5. Advantage Estimation	0.14-0.20s	0%	GRPO computation
6. Actor Update (PPO)	140-145s (~2.4 min)	90%	4 mini-batches, dataloader_length=4
7. Entropy Computation	<1s	80%	Included in actor update
8. Logging	<1s	0%	Write metrics to WandB
Total per training step	1220-1300s (~20-22 min)	20%	Average GPU utilization

Additional Operations: - Validation (every 4 training steps): +30-40s (256 rollouts) - Checkpoint Saving (every 20 training steps): +10-20s

Why Rollout is So Slow: - Each rollout takes ~1.2 seconds (includes env init, 8 VLA calls, 200 env steps, cleanup) - Need to collect 896 rollouts to get 512 valid ones after filtering (75% efficiency) - 896 × 1.2s = 1075 seconds ≈ 18 minutes

Full Training Timeline¶

PAPER EXPERIMENTS (300 steps, 20 epochs):

Total Training Duration:
= 300 training_steps × 1230 seconds/step
= 369,000 seconds
= 102.5 hours
≈ 4.3 days (with 8 GPUs running continuously)

IF USING FULL CONFIG (1500 steps, 100 epochs):

Total Training Duration:
= 1500 training_steps × 1230 seconds/step
= 1,845,000 seconds
= 512.5 hours
≈ 21.4 days (with 8 GPUs running continuously)

Rollout Statistics¶

For Paper (300 steps): | Metric | Value | |--------|-------| | Total Rollouts | 153,600 | | Successful Rollouts | ~60,000-80,000 (depends on learning) | | Failed Rollouts | ~70,000-90,000 | | Filtered Out | ~40,000 (accuracy filtering) | | Used for Training | ~110,000-120,000 |

For Full Config (1500 steps): | Metric | Value | |--------|-------| | Total Rollouts | 768,000 | | Successful Rollouts | ~300,000-400,000 (depends on learning) | | Failed Rollouts | ~400,000-500,000 | | Filtered Out | ~200,000 (accuracy filtering) | | Used for Training | ~500,000-600,000 |

GPU Memory Usage¶

Per GPU (Rollout Phase):
├── Model Parameters (sharded): ~940MB
├── Activation Memory: ~1.5GB
├── Environment Buffers: ~500MB
├── KV Cache: ~2GB
└── Total: ~5GB / 40GB (12.5%)

Per GPU (Training Phase):
├── Model Parameters (sharded): ~940MB
├── Gradients: ~940MB
├── Activation Memory: ~3GB
├── Optimizer States (offloaded to CPU): 0MB
└── Total: ~5GB / 40GB (12.5%)

Peak Memory: ~6GB / 40GB (15%)

Key Paper Insights¶

What Makes SimpleVLA-RL Work?¶

Based on SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning, the framework achieves state-of-the-art performance through three key enhancements:

1. Dynamic Sampling (Section 3.3, Equation 10)¶

Problem: Groups with all success or all failure have zero advantage variance, leading to unstable gradients.

Solution: During rollout collection, exclude groups where all trajectories either succeed or fail:

0 < |{traj_i | is_successful(traj_i)}| < G

Implementation: data.accuracy_lower_bound=0.1 and data.accuracy_upper_bound=0.9

Result: ~15% improvement over baseline (Figure 3a)

Benefit: - Non-zero advantages ensure meaningful learning signal - Natural curriculum learning (focuses on appropriate difficulty) - Proven effective in LLM RL (Yu et al., 2025; Cui et al., 2025a)

2. Clipping Higher (Section 3.3)¶

Problem: Standard PPO's symmetric clipping [0.8, 1.2] restricts exploration by limiting probability increases.

Solution: Modify clipping range to [0.8, 1.28] (asymmetric):

clip_ratio_high = 0.28  # Upper bound at 1.28 (not 1.2)
clip_ratio_low = 0.2    # Lower bound at 0.8

Implementation: actor_rollout_ref.actor.clip_ratio_high=0.28

Result: ~10% improvement over baseline (Figure 3b)

Benefit: - Allows larger probability increases for low-probability actions - Encourages exploration of new action patterns - Inspired by DAPO (Yu et al., 2025) for LLM RL

3. Higher Rollout Temperature (Section 3.3)¶

Problem: Low temperature (1.0) during rollout leads to deterministic, repetitive trajectories.

Solution: Increase sampling temperature from 1.0 to 1.6 during rollout phase:

actor_rollout_ref.rollout.temperature=1.6

Implementation: Used during generate_action_verl() in rollout, not during training

Result: ~15% improvement over baseline (Figure 3c)

Benefit: - Generates diverse trajectories for better exploration - Critical for discovering new successful strategies - Widely used in recent LLM RL work (Liu et al., 2025c; An et al., 2025)

Novel Phenomenon: "Pushcut"¶

Discovery (Section 6.1): During RL training, the policy discovered a previously unseen action pattern called "pushcut": - SFT model only learns to grasp-and-lift objects - After RL, model learns to push objects toward target while moving - This "sliding" strategy is more efficient than pure lifting - Not seen in any training demonstrations!

Significance: - Shows RL enables emergence of novel, optimal strategies - Goes beyond imitation of human demonstrations - Similar to AlphaGo discovering unconventional moves

Performance Results (from Paper)¶

LIBERO Benchmark (Figure 3): - Baseline SFT: ~60% success rate (Long) - With all enhancements: ~90% success rate (Long) at 300 steps - +30 percentage points improvement

RoboTwin 2.0 Benchmark (Table 1): - Outperforms π₀ (previous SOTA) on multiple tasks - Example: beat_block_hammer task success rate improvement

Real-World Experiments (Section 5.3): - Successfully generalizes to real robot hardware - Demonstrates sim-to-real transfer capability

Why RL Works for VLA (Paper's Main Argument)¶

Parallel to LLM RL Success: 1. DeepSeek-R1 showed RL dramatically improves step-by-step reasoning 2. VLA = step-by-step action planning (analogous to CoT reasoning) 3. RL enables learning from outcome rewards only (no hand-crafted process rewards)

Key Advantages: 1. Addresses Data Scarcity: No need for massive human demonstration data 2. Improves Generalization: RL naturally explores beyond training distribution 3. Discovers Novel Patterns: Like "pushcut" - goes beyond imitation 4. Scalable: Only requires environment simulator, not human operators

Summary¶

Key Characteristics of veRL Training¶

GRPO + PPO Pipeline: Separates advantage estimation (GRPO) from policy update (PPO)
GRPO works with sparse outcome rewards
PPO provides stable policy updates
Three Critical Enhancements: Dynamic Sampling + Clip Higher + Higher Temperature
Each provides ~10-15% improvement independently
Combined effect: ~30% improvement over baseline SFT
Outcome-Only Rewards: No hand-crafted process rewards
Success/failure at trajectory end
Inspired by LLM RL (DeepSeek-R1)
Scalable to new tasks
Exploration-Focused: Multiple mechanisms encourage exploration
Dynamic sampling focuses on learnable tasks
Clip higher allows larger policy updates
High temperature generates diverse rollouts
300-Step Training: Paper shows convergence at 300 training steps (~4.3 days)
Much faster than expected for robotic RL
Config has 1500 steps for extended experiments

System Bottlenecks¶

Rollout is 87.8% of time: Physics simulation dominates
Each training step: ~18 minutes rollout, ~2.4 minutes training
Need ~896 rollouts to get 512 valid (after filtering)
Dynamic Sampling reduces effective batch size: ~30% overhead
Required for stable training
Worthwhile tradeoff for better learning
Low GPU utilization during rollout: ~15-20%
VLA inference only 300ms vs 1500ms physics
Fundamental limitation of CPU-based simulators

Recommendations for Improvement¶

Based on Paper: 1. Tune the three key hyperparameters: - data.accuracy_lower_bound and data.accuracy_upper_bound (Dynamic Sampling) - actor_rollout_ref.actor.clip_ratio_high (Clip Higher) - actor_rollout_ref.rollout.temperature (Exploration)

Task-Specific Tuning:
Harder tasks may need wider accuracy bounds (0.05-0.95)
Easier tasks may need higher temperature (1.8-2.0)

Based on Implementation Analysis: 1. Increase Rollout Throughput: - Use GPU-accelerated simulator (Isaac Gym/Sim) - Reduce action chunk length (25 → 10-15) - Pipeline CPU/GPU work

Optimize Training Efficiency:
Increase n_samples for better GRPO baselines
Use multiple PPO epochs if overfitting is not a concern
Adjust filtering to keep more data if collection is expensive
Monitor Key Metrics:
actor/pg_clipfrac: Should be 2-5% (too high = policy changing too fast)
critic/advantages/mean: Should be near 0 (GRPO working correctly)
Dynamic sampling efficiency: Aim for 70-80% retention rate

veRL Training Process: Complete System Analysis¶

Table of Contents¶

Overall Architecture¶

System Components¶

Training Configuration¶

Dataset Configuration¶

Model Configuration¶

Training Configuration¶

Computed Training Parameters¶

Complete Training Loop¶

High-Level Structure (from ray_trainer.py:469-717)¶

Training Summary¶

Phase 1: Rollout¶

Rollout Process (from ray_trainer.py:513-556)¶

Inside generate_sequences (from rob_rollout.py:495-507)¶

Detailed Rollout Execution (from rob_rollout.py:609-775)¶

Rollout Output Format¶

Rollout Timing Breakdown¶

Phase 2: Reward Computation¶

2.1 Verification (from ray_trainer.py:559-570)¶

Verification Logic (from main_ppo.py:42-60)¶

2.2 Filtering (from ray_trainer.py:573-578)¶

Dynamic Sampling (Paper Section 3.3)¶

2.3 Outcome Reward Modeling (Paper Section 3.2)¶

2.3.1 Reward Shaping (from ray_trainer.py:631-647)¶

Phase 3: Advantage Estimation (GRPO)¶

3.1 KL Penalty (Optional) (from ray_trainer.py:642-647)¶

KL Penalty Logic (from ray_trainer.py:85-120)¶

3.2 Advantage Computation (from ray_trainer.py:649-655)¶

GRPO Advantage Estimation (from ray_trainer.py:160-173)¶

GRPO Algorithm (Paper Section 2.3 & verl/trainer/ppo/core_algos.py)¶

Phase 4: Actor Update (PPO with Clip Higher)¶

4.1 Update Trigger (from ray_trainer.py:660-671)¶

4.2 PPO Update (in ActorRolloutRefWorker)¶

PPO Update Summary¶

4.3 Entropy Computation (from ray_trainer.py:666)¶

Phase 5: Validation¶

5.1 Validation Trigger (from ray_trainer.py:673-679)¶

5.2 Validation Process (from ray_trainer.py:336-391)¶

Validation Dataset (from rob_dataset.py:136-209)¶

Validation Metrics¶

Data Flow and Synchronization¶

Overall Data Flow¶

Ray Distributed Execution¶

FSDP Model Sharding¶

Performance Analysis¶

Training Step Timing Breakdown¶

Full Training Timeline¶

Rollout Statistics¶

GPU Memory Usage¶

Key Paper Insights¶

What Makes SimpleVLA-RL Work?¶

1. Dynamic Sampling (Section 3.3, Equation 10)¶

2. Clipping Higher (Section 3.3)¶

3. Higher Rollout Temperature (Section 3.3)¶

Novel Phenomenon: "Pushcut"¶

Performance Results (from Paper)¶

Why RL Works for VLA (Paper's Main Argument)¶

Summary¶

Key Characteristics of veRL Training¶

System Bottlenecks¶

Recommendations for Improvement¶

High-Level Structure (from `ray_trainer.py:469-717`)¶

Rollout Process (from `ray_trainer.py:513-556`)¶

Inside `generate_sequences` (from `rob_rollout.py:495-507`)¶

Detailed Rollout Execution (from `rob_rollout.py:609-775`)¶

2.1 Verification (from `ray_trainer.py:559-570`)¶

Verification Logic (from `main_ppo.py:42-60`)¶

2.2 Filtering (from `ray_trainer.py:573-578`)¶

2.3.1 Reward Shaping (from `ray_trainer.py:631-647`)¶

3.1 KL Penalty (Optional) (from `ray_trainer.py:642-647`)¶

KL Penalty Logic (from `ray_trainer.py:85-120`)¶

3.2 Advantage Computation (from `ray_trainer.py:649-655`)¶

GRPO Advantage Estimation (from `ray_trainer.py:160-173`)¶

GRPO Algorithm (Paper Section 2.3 & `verl/trainer/ppo/core_algos.py`)¶

4.1 Update Trigger (from `ray_trainer.py:660-671`)¶

4.3 Entropy Computation (from `ray_trainer.py:666`)¶

5.1 Validation Trigger (from `ray_trainer.py:673-679`)¶

5.2 Validation Process (from `ray_trainer.py:336-391`)¶

Validation Dataset (from `rob_dataset.py:136-209`)¶