Semi-Ensemble: A Simple Approach to Over-parameterized Model Interpolation Pohang University of Science and Technology

Abstract

In this thesis, I propose a new and simple model merging method motivated by the mechanism of model ensemble. Previous alignment-based model merging algo- rithms align the unique and dissimilar neurons, which should be preserved to mimic performance of model ensemble. Therefore, I propose Semi-Ensemble, which takes advantage of the extended parameter space to preserve different neurons without inter- polating them. Semi-Ensemble can generate various degrees of over-parameterization, having model merging and model ensemble as special cases, and efficiently imitate characteristics of ensembled prediction such as calibration score. By carefully con- structing the extended joint parameter space, the interpolated model can strike better trade-off between the total number of parameters and model accuracy.