LXMERT swMATH ID: 42494 Software Authors: Hao Tan, Mohit Bansal Description: LXMERT: Learning Cross-Modality Encoder Representations from Transformers. Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pre-trained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre-trained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR2, and improve the previous best result by 22 Homepage: https://arxiv.org/abs/1908.07490 Source Code: https://github.com/airsplay/lxmert Dependencies: Python Related Software: GloVe; ViLBERT; BLEU; Faster R-CNN; Flickr30K; XLNet; BabyWalk; AllenAct; VirtualHome; HRL4IN; CHALET; SAPIEN; Habitat; TorchCraft; ALFRED; MINOS; SQIL; TEACh; EvolveGraph; iGibson Cited in: 2 Publications all top 5 Cited by 9 Authors 1 Caglayan, Ozan 1 Francis, Jonathan 1 Haralampieva, Veneta 1 Kitamura, Nariaki 1 Labelle, Felix 1 Lu, Xiaopeng 1 Navarro, Ingrid 1 Oh, Jean 1 Specia, Lucia Cited in 1 Serial 2 The Journal of Artificial Intelligence Research (JAIR) Cited in 1 Field 2 Computer science (68-XX) Citations by Year