Multi-modal Convolutional Neural Networks for Human Activity Recognition

Abstract

Human activity recognition involves classifying times series data, measured at inertial sensors such as accelerometers or gyroscopes, into one of pre-defined actions. Recently, convolutional neural network (CNN) has established itself as a powerful technique for human activity recognition, where convolution and pooling operations are applied along the temporal dimension of sensor signals. In most of existing work, 1D convolution operation is applied to individual univariate time series and capture local dependency over time in series of observations measured at inertial sensors, while multi-sensors or multi-modality yield multivariate time series. I present a CNN with 2D kernels in both convolutional and pooling layers, to capture local dependency along both temporal and spatial domains, i also propose multi-modal CNN referred to as CNN-pf and CNN-pff, where 2D convolution and pooling employing both partial weight sharing and full weight sharing for our CNN models in such a way that modality specific characteristics as well as common characteristics across modalities are learned from multi-modal (or multi-sensor) data and are eventually aggregated in upper layers. Experiments on benchmark datasets demonstrate the small parameter number and high performance of our CNN models, compared to state of the arts methods.