Deep Learning based User Adaptive Hand Gesture Recognition using 60 GHz Radar

Abstract

Hand gesture recognition has been in the spotlight as a new input method for a new human-computer interaction (HCI). Accordingly, various research has recently been attempted to develop a system with high gesture recognition accuracy through the combination of a small radar device and deep learning. However, the hand gesture recognition system using radar proposed in the existing studies suffers from a common problem. Although the hand gesture recognition system proposed in each study showed high accuracy, there is a serious problem that the accuracy of the result inference is significantly degraded in the actual use environment. This critical problem arises from the discrepancy between the gesture data learned by each system and the actual input data. The most important factor causing this discrepancy is behavioral habits that are subtly different for each user. That is, even if different users make the same gesture, the speed, direction, radius, and angle of the gesture are all different for each user.

In this dissertation, we propose a user adaptive hand gesture recognition system that can guarantee high generalization performance. The proposed system introduces a domain adaptation algorithm to reduce the discrepancy between the training data and the actual input data. Among the various domain adaptation algorithms, an adversarial learning approach is used to prevent the classifier from distinguishing whether the input data belongs to the training data set (source domain) or the test data set (target domain). Therefore, the accuracy of gesture classification is improved for any input data, and as a result, high generalization performance is guaranteed although is a fine difference in gestures between users. The domain discriminator playing domain adaptation role is included in the deep learning network empirically designed to have optimal classification performance for the source domain data and adjusts the feature space of the input data.

In addition, we analyze the effects of the hardware characteristics of the radar devices and input data types on gesture inference results. The number of transmit/receive antennas and the frame rate of the radar are the biggest factors determining the amount of gesture information in the collected signal. Depending on the processing of the collected signal, it can be transformed into various input data types such as Doppler spectrogram, Range-Doppler map (RDM), and motion profile. For real-time gesture recognition, a deep learning network suitable for each input data type must be designed. Therefore, we empirically implement an optimal deep learning network for each radar characteristic and input data type.

For the performance evaluation, we used 60 GHz frequency modulated continuous wave (FMCW) radar, and performed gesture recognition experiments in real environments. As a result, the proposed system outperforms existing hand gesture recognition systems in terms of generalization performance.