Improving Text-to-Image Generation by Discriminator with Recaption Ability

Abstract

Text-to-image synthesis aims to generate a photo-realistic image from a given natural language description. A text description, unlike a label condition, includes many constraints which make the synthesis task challenging. Although significant progress has been made in generating visually realistic images using Generative Adversarial Networks (GANs), current text-to-image synthesis models ignore some text constraints. In this paper, we address the text-image consistency problem by adopting image captioning task. Image captioning is an inversion problem of text-to-image synthesis so it works to keep cycle consistency. To this end, we propose a Recaptioning Discriminator (RecapD) which not only computes the adversarial logits but also redescribes the input image. The RecapD internally has captioning model which is trained with the discriminator. Therefore, RecapD is more efficient than adopting an extra pre-trained captioning model. Furthermore, RecapD encourages the generator to produce a realistic and text-aligned image for good redescription by using the internal captioning model. Experiments on the MS-COCO dataset show the superiority of our proposed method compared to recent text-to-image synthesis models. Ablation study demonstrates the effectiveness of the proposed RecapD. We use FID to measure image quality, and R-precision to evaluate text-image consistency. The RecapD significantly improves performance of both image quality and text-image consistency.