Model-Based Reinforcement Learning for Environments with Delayed Feedback

Abstract

In recent years, reinforcement learning (RL) has made significant improvements in complex games, continuous control tasks, and real-world control tasks. However, there are several obstacles to adapting RL to the real world. One of the difficulties is a signal delay. In real world tasks, signal delay can occur in many cases and is often not avoidable. The mismatch between delayed observation and true observation causes performance degradation. To overcome this issue, we propose a Model-Based State Estimation (MBSE) algorithm that estimates the true feedback from the delayed feedback. We tested our algorithm on MuJoCo control tasks and compared it with other algorithms in a delayed feedback environment, and our algorithm showed significant performance improvement.