Tools: Reinforcement Learning ML

OpenAI Gym

OpenAI Gym is an open source Python library providing environment for reinforcement learning algorithm. It will gradually transfer to Gymnasium library, I will try the Gymnasium in the future.

Stable Baseline3 is a set of reliable implementations of reinforcement learning algorithms in PyTorch. To support Gym version >= 0.26, you will need the pull request here.

In this project, I followed the tutorial to implement DQN (Deep Reinforcement Learning) and DDPG (Deep Deterministic Policy Gradient) in different environment.

highway-fast-v0

Code

The first task is on a highway environment, the car needs to avoid crashing into other cars by switching to different lanes smoothly.

If we let the car idle, it will crash.

After using DQN training for 50000 steps, the best model success rate is 0.84 out of 25 tests.

The learning curve and the evaluation result.

PandaReach-v2

Code

The second task is the localization of a Panda robot arm. Hindsight Experience Replay Buffer is added to help the robot learn sparse reward better.

Let’s first move the arm using intuition.

After using DDPG training for 300000 steps, the final success rate is 0.97 out of 30 tests.

There are still vibration after reaching the goal, future work would focus on imposing speed limit or combine traditional control to improve the performance and increase safety.