Learning Visual Correspondence

Abstract

This dissertation presents a comprehensive study on advancing visual correspondence in computer vision, focusing on overcoming the challenges posed by large intra-class variations and the limitations of existing feature representation techniques. The research navigates through the complexities of establishing accurate correspondences between different instances of the same object categories, crucial for a wide range of applications including object recognition, image retrieval, 3D reconstruction, few-shot learning, geometric shape assembly and beyond. Central to the dissertation is the exploration in leveraging the strengths of deep neural networks, moving beyond traditional approaches that primarily depend on a single convolutional layer output. This involves a detailed examination of multi-layer representations, which provide a more nuanced understanding of images by capturing both fine-grained details and broader contexts, thus addressing the challenge of local ambiguities and enhancing the robustness of feature matching in complex visual environments. An integral part of the research is the development of efficient matching algorithms that achieve real-time performance. These algorithms are designed to regularize and optimize the process of establishing dense correspondences, ensuring both speed and accuracy in various matching scenarios. The dissertation also introduces a novel dataset, enriching the field with a diverse range of image pairs and comprehensive annotations, facilitating a deeper analysis and evaluation of semantic correspondence methods. A significant contribution of the dissertation is in the realm of geometric matching, where it introduces a new perspective on convolutional matching techniques. This involves extending the concept of Hough transform to high-dimensional convolutions, offering an innovative approach to handle the complexities of non-rigid matching scenarios. The proposed methods are not only compatible with various neural network architectures but also enhance the robustness against background clutter and provide flexibility in dealing with diverse image transformations. Furthermore, we apply our proposed neural matching components for visual correspondence to address challenges of its high-level applications such as few-shot segmentation and geometric shape assembly; the research on few-shot learning delves into the creation of advanced networks by integrating multi-level correlations and high-dimensional convolutions to capture a rich set of correspondences across multiple visual aspects, thereby mirroring the human visual system’s ability for object generalization and balancing computational efficiency with precise mask prediction. For geometric assembly, this dissertation also presents an efficient matching layers that effectively approximate high-order convolutions with sub-quadratic complexity, advancing state of the arts in terms of both efficiency and efficacy. Overall, the proposed methods represent a unique leap forward, addressing complex challenges in visual correspondence and its applications. These contributions signify a substantial advancement in the field of image matching, pushing the boundaries of current technologies and laying the groundwork for future explorations.