MDARTS: Multi-objective Differentiable Neural Architecture Search

Abstract

In this thesis, we present a differentiable neural architecture search (NAS) method that takes into account two competing objectives, quality of result (QoR) and quality of service (QoS) with hardware design constraints. NAS research has recently received a lot of attention due to its ability to automatically find architecture candidates that can outperform handcrafted ones. However, the NAS approach which complies with actual HW design constraints has been underexplored. A naive NAS approach for this would be to optimize a combination of two criteria of QoR and QoS, but we first identify that the simple extension of the prior art often yields degenerated architectures, and suffers from a sensitive hyperparameter tuning. Instead, we propose to formulate it as a differentiable multiobjective optimization, called MDARTS. MDARTS has an affordable search time and can find Pareto front. We also identify the problematic gap between all the existing differentiable NAS results and those final post-processed architectures, where soft connections are binarized. This gap leads to performance degradation when being deployed. To mitigate this gap, we propose a separation loss that discourages indefinite connections of components by implicitly minimizing entropy. In our experiment, we show that MDARTS is able to find the architectures that have lower error than the state-of-the-art (2.35 % and 14.99 % of top-1 test error on CIFAR-10 and CIFAR-100, respectively) with an affordable latency for hardware. Also, we found several architecture candidates that place even closer to Pareto front than the ones obtained from the state-of-the-art NAS methods. We also demonstrate MDARTS can complete a single search process within eight GPU-hours on both CIFAR-10 and CIFAR-100.