swMATH ID: 42494
Software Authors: Hao Tan, Mohit Bansal
Description: LXMERT: Learning Cross-Modality Encoder Representations from Transformers. Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pre-trained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre-trained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR2, and improve the previous best result by 22
Homepage: https://arxiv.org/abs/1908.07490
Source Code:  https://github.com/airsplay/lxmert
Dependencies: Python
Related Software: GloVe; ViLBERT; BLEU; Faster R-CNN; Flickr30K; XLNet; BabyWalk; AllenAct; VirtualHome; HRL4IN; CHALET; SAPIEN; Habitat; TorchCraft; ALFRED; MINOS; SQIL; TEACh; EvolveGraph; iGibson
Cited in: 2 Publications

Citations by Year