Model-Based Reinforcement Learning With Probabilistic Ensemble Terminal Critics for Data-Efficient Control Applications

Abstract

This article proposes a data-efficient model-based reinforcement learning (RL) algorithm empowered by reliable future reward estimates achieved through a confidence-based probabilistic ensemble terminal critics (PETC). The proposed algorithm utilizes a model-predictive controller to choose an action that optimizes the sum of the near and distant future rewards for a given current state. Near future rewards with high confidence are determined directly from trained deterministic dynamics and reward models. Distant future rewards beyond these horizons are meticulously assessed using the proposed confidence-based PETC, which minimizes estimation errors inherent in the distant future and quantifies uncertainty confidence. Through such confidence-based guided actions, the proposed approach is expected to operate in a reliable, explainable, and data-efficient manner, consistently guiding the system to an optimal trajectory. A comparison with the existing state-of-the-art RL algorithms for eight DeepMind Control Suite tasks confirms the superior data efficiency of the proposed approach, which achieves an average cumulative reward of 761.2 in merely 500K steps, whereas the other algorithms score below 700.0. The proposed algorithm is also successfully applied to two real-world control applications, namely single- and double-cartpole swing-up tasks.