Policy gradient methods are an appealing approach in reinforcement learningbecause they directly optimize the cumulative reward and can straightforwardlybe used with nonlinear function approximators such as neural networks. The twomain challenges are the large number of samples typically required, and thedifficulty of obtaining stable and steady improvement despite thenonstationarity of the incoming data. We address the first challenge by usingvalue functions to substantially reduce the variance of policy gradientestimates at the cost of some bias, with an exponentially-weighted estimator ofthe advantage function that is analogous to TD(lambda). We address the secondchallenge by using trust region optimization procedure for both the policy andthe value function, which are represented by neural networks.  Our approach yields strong empirical results on highly challenging 3Dlocomotion tasks, learning running gaits for bipedal and quadrupedal simulatedrobots, and learning a policy for getting the biped to stand up from startingout lying on the ground. In contrast to a body of prior work that useshand-crafted policy representations, our neural network policies map directlyfrom raw kinematics to joint torques. Our algorithm is fully model-free, andthe amount of simulated experience required for the learning tasks on 3D bipedscorresponds to 1-2 weeks of real time.

High-Dimensional Continuous Control Using Generalized Advantage Estimation

Scan to connect with one of our mobile apps

Coinbase Wallet app

Coinbase app

Or try the Coinbase Wallet browser extension