Transfer Learning of Automatic Post-Editing with Cross-lingual Language Model

Abstract

In this thesis, we present a new approach to Automatic Post-Editing (APE) that uses the cross-lingual language model (XLM) as a pre-trained model for the English-German language pair. To initialize our APE model's encoder, we use the weights of a pre-trained XLM that has been jointly trained with a translation language modeling (TLM) objective by using parallel corpora. Besides, we experimented with various decoder-initialization methods such as using the weights of pre-trained models and tying parameters. Experimental results imply that training a model with both a masked language modeling (MLM) and a TLM objective is more effective than training the same model with an MLM objective only. In experiments conducted on the WMT16, '17, '18 test datasets, Our APE results showed significant improvements over the MT outputs in translation quality.