Unified Online Multiple Object Tracking Framework Using Versatile Affinity Network and Event-Aware Loss

Abstract

Multiple object tracking (MOT) is a core problem in computer vision and appears in various fields, such as video surveillance, human computer interaction, and autonomous driving. In recent years, tracking-by-detection has become the most popular MOT method. The main operation in tracking-by-detection is the computation of affinity scores between targets and detection candidates. Deep convolutional neural networks (CNNs) have been successfully applied to tracking-by-detection MOT methods, particularly in the use of deep features, for accurate candidate associations by learning similarity measures between the features. Despite benefits of CNNs, MOT methods still suffer from abnormal events, leading to numerous false positives (FPs), false negatives (FNs), and identity switches (IDs). In several methods, the deep neural network is trained to prevent these abnormal events: however, existing methods only train the subnetworks of the MOT framework, and the training loss and samples cannot directly reduce the occurrence of such abnormal events. Even if each part of an MOT network is trained effectively, the trained network is not guaranteed to be optimized for a real MOT environment. This drawback comes from the fact that most MOT methods train their subnetworks without using a unified network architecture and a unified loss. This dissertation proposes a unified MOT framework to address the problems caused by separately trained subnetworks and indirect loss for training MOT networks. We propose a versatile affinity network (VAN) that can perform the entire MOT process in a single network including target specific SOT, affinity computation between target and candidates, and decision of tracking termination. This process is performed by making the SOT prediction scores compatible with the affinity values between targets and candidates. We train the VAN in an end-to-end manner by using event-aware sampling that is designed to reduce the potential error caused by FNs, FPs, and identity switching. The proposed VAN significantly reduce the effort required to tune hyperparameters and did not require handcrafted rules for integrating the SOT and the affinity network. We implemented the VAN using two baselines with different candidate classification methods to demonstrate the effects of the proposed VAN. We also conducted extensive experiments including ablation studies on three public benchmark datasets: MOT2015, MOT2016, and MOT2017. We improved MOT performances by 1.5% compared with baseline in terms of MOT accuracy (MOTA) on MOT2015 dataset, 1.3% on MOT2016 dataset, and 1.7% on MOT2017 dataset. The experimental results verify that the proposed method successfully improves the MOT performances compared with that of baseline methods, and outperforms recent state-of-the-art MOT methods in terms of several MOT metrics including MOTA, identity F1 score (IDF1), and FP. We propose an end-to-end online MOT framework using an event-aware loss in order to control occurrences of abnormal events in an online MOT situation and compel the tracker to take appropriate actions under the occurrences of abnormal events. We analyze the entire cases of assignment between the target and candidate, and categorize the cases into meaningful events. The proposed event-aware loss is back-propagated from the association layer to the feature extraction layer. We conducted several experiments including ablation studies on various public MOT challenge benchmark datasets. We improved MOT performances by 0.9% compared with baseline in terms of MOTA on MOT2015 dataset, 1.1% on MOT2016 dataset, and 0.9% on MOT2017 dataset. The experimental results verify that the proposed method successfully improves the MOT performances compared with recent state-of-the-art MOT methods and each event affecting the MOT measure can be controlled using the proposed event-aware loss. We propose a unified online MOT framework combining the versatile affinity network and the event-aware loss. The proposed framework can be successfully applied to existing multiple object tracker, and it can be trained in end-to-end manner to improve overall MOT performances. We conducted several experiments on three MOT benchmark datasets to verify the improvement of MOT performances compared with baseline tracker. We improved MOT performances by 2.1% compared with baseline in terms of MOTA on MOT2015 dataset, 1.5% on MOT2016 dataset, and 2.0% on MOT2017 dataset. The experimental results verify that we successfully perform the entire MOT tasks using the proposed unified MOT framework.