PPO LunarLander flip, recover and land agent
This repository contains a Stable-Baselines3 PPO
actor-critic agent trained on a customised
LunarLander-v3 environment.
Learned task
The curriculum teaches one policy to:
- complete a full rotation in a fixed direction;
- recover upright and arrest angular motion;
- enter the landing zone;
- reduce descent speed and land safely.
Reward configuration version: v5-soft-touchdown-refinement.
Changes in this upload
- Continued from the selected Phase D checkpoint.
- Added an altitude-dependent vertical-speed target.
- Added a near-ground quadratic descent-overspeed penalty.
- Added a dedicated in-zone crash penalty.
- Reduced risky late horizontal corrections using a deadband.
Reward design
The shaped reward includes:
- one-off rotation-progress and flip-completion rewards;
- an upright post-flip recovery reward;
- horizontal guidance towards the landing zone;
- an altitude-dependent vertical-speed target;
- attitude and angular-speed control;
- a near-ground descent-overspeed penalty;
- distinct penalties for off-zone landings and in-zone crashes.
| Parameter | Value |
|---|---|
required_rotations | 1 |
rotation_direction | 1 |
upright_tolerance_radians | 0.3 |
recovery_angular_velocity_tolerance | 0.3 |
pre_flip_original_reward_weight | 0.15 |
post_flip_original_reward_weight | 3 |
pre_flip_center_weight | 25 |
pre_flip_horizontal_speed_weight | 15 |
pre_flip_shaping_weight | 1 |
pre_flip_shaping_clip | 10 |
flip_corridor_half_width | 0.35 |
flip_entry_penalty_weight | 600 |
rotation_progress_bonus | 300 |
flip_completion_bonus | 500 |
recovery_bonus | 250 |
flip_landing_bonus | 2000 |
landing_without_flip_penalty | 600 |
no_flip_terminal_penalty | 500 |
failed_landing_penalty | 900 |
outside_zone_landing_penalty | 1200 |
in_zone_crash_penalty | 1500 |
post_flip_shaping_weight | 2 |
post_flip_shaping_gamma | 0.999 |
post_flip_shaping_clip | 30 |
post_flip_center_weight | 100 |
post_flip_horizontal_speed_weight | 50 |
post_flip_vertical_speed_weight | 100 |
post_flip_angle_weight | 70 |
post_flip_angular_speed_weight | 40 |
post_flip_leg_contact_weight | 30 |
landing_zone_half_width | 0.2 |
post_flip_zone_excess_weight | 180 |
post_flip_target_vx_gain | 0.6 |
post_flip_max_target_vx | 0.4 |
post_flip_horizontal_deadband | 0.08 |
post_flip_target_vy_high | -0.45 |
post_flip_target_vy_near_ground | -0.12 |
near_ground_height | 0.6 |
safe_touchdown_vertical_speed | 0.14 |
near_ground_overspeed_weight | 120 |
Evaluation
Deterministic evaluation over 100 fixed-seed episodes:
| Metric | Value |
|---|---|
| Mean shaped reward | -413.81 |
| Mean original reward | -671.00 |
| Full-rotation rate | 15.0% |
| Recovery rate | 9.0% |
| Recovery given a flip | 60.0% |
| Safe-landing rate | 0.0% |
| Flip-and-land rate | 0.0% |
| Terminal in-zone rate | 44.0% |
| In-zone crash rate | 13.0% |
Architecture
- Algorithm: PPO
- Policy: MLP actor-critic
- Actor hidden layers:
[128, 128] - Critic hidden layers:
[128, 128] - Observation dimensions:
11 - Discrete actions:
4
Training configuration
| Parameter | Value |
|---|---|
| Phase timesteps | 10000000 |
| Parallel environments | 32 |
| Learning rate | 5e-05 |
| Rollout steps per environment | 1024 |
| Batch size | 64 |
| Optimisation epochs | 4 |
| Gamma | 0.999 |
| GAE lambda | 0.98 |
| Entropy coefficient | 0.005 |
| PPO clip range | 0.1 |
| Training seed | 42 |
Replay
- Seed:
20000 - Original reward:
-669.61 - Shaped reward:
-294.96 - Rotations completed:
1.00 - Flip completed:
False - Recovery completed:
False - Landed safely:
False - Outcome:
episode_ended_without_flip
Repository files
ppo-LunarLander-v3-flip-128x128.zip: selected PPO modelflip_landing_reward_wrapper.py: custom environment wrappertraining_config.json: PPO training settingsreward_config.json: reward configurationepisode_results.csv: fixed-seed evaluation episodesresults.json: machine-readable evaluation summaryconfig.json: compact model metadatareplay.gif: model-card previewreplay.mp4: full replay