PPO Booster Lander — True 3D
Reinforcement learning — teacher/student transfer to 3D • 2026
The 2D booster-landing controller's true-3D successor: the student policy is warm-started by imitating the frozen 2D teacher's throttle/pitch behavior on planar states, then trained through a 3D curriculum to fly with a real quaternion orientation, angular velocity, and a 4th control axis (roll) the 2D policy never had.
Problem
The 2D booster lander only ever had to solve a planar problem — one horizontal axis, one tilt angle, two actions. Extending that to a real rocket means a full rigid-body state (3D position, 3D velocity, orientation, angular velocity) and a 4th control input (roll), where the 2D policy's throttle/gimbal instincts are a reasonable starting point but not a solution.
Teacher/student transfer
Rather than training the 3D policy from scratch, the strongest 2D PPO checkpoint became a frozen teacher. A new student policy — 14 observations in, 4 actions out — is warm-started to imitate the teacher's throttle and pitch behavior on planar states (where the 3D task degenerates to the 2D one), while its lateral gimbal and roll outputs start near zero and are learned from scratch. From there PPO takes over entirely; the teacher is only a warm start, not a runtime dependency.
True 3D state and control
Observation (14): position (x, y, z), velocity (vx, vy, vz), orientation as a unit quaternion (qx, qy, qz, qw), angular velocity (wx, wy, wz), and fuel. Action (4): throttle, pitch gimbal, lateral/yaw gimbal, and roll — all raw in [-1, 1], with throttle remapped internally to [0, 1]. Orientation dynamics run through a rate-controller (target angular rates driven by the gimbal/roll commands, integrated through Euler angles) rather than the simplified single-axis tilt the 2D environment used.
Curriculum
Success-gated PPO curriculum over difficulty — 0.00 (planar, teacher-compatible) → 0.05 → 0.10 → 0.20 → 0.35 — where each stage only advances once the policy clears a required deterministic success rate at the current difficulty. The checkpoint running below (student_before_residual_transfer) is the output of that curriculum, ahead of a planned residual-transfer refinement pass.
Try it
Click in the sky to spawn the booster in full 3D — camera angle, difficulty, and initial position/velocity/orientation/spin are all adjustable live, with quaternion and angular-velocity telemetry alongside the render.
Status
This is an active project — the model and simulator here will keep updating as the residual-transfer stage and further curriculum progress land.