XLNet swMATH ID: 31118 Software Authors: Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le Description: XLNet: Generalized Autoregressive Pretraining for Language Understanding. With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, XLNet outperforms BERT on 20 tasks, often by a large margin, and achieves state-of-the-art results on 18 tasks including question answering, natural language inference, sentiment analysis, and document ranking. Homepage: https://arxiv.org/abs/1906.08237 Source Code: https://github.com/zihangdai/xlnet Keywords: arXiv_cs.CL; Machine Learning; arXiv_cs.LG; BERT; Autoregressive Pretraining; Language Understanding; Transformer-XL Related Software: BERT; RoBERTa; GloVe; ALBERT; word2vec; Tensor2Tensor; BLEU; SQuAD; Transformers; Adam; Python; GPT-3; GLUE; BERTScore; Rouge; DistilBERT; SuperGLUE; ELECTRA; Transformer-XL; SBERT Cited in: 13 Publications Standard Articles 1 Publication describing the Software Year XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le 2019 all top 5 Cited by 53 Authors 1 Aghahadi, Zeinab 1 Baldwin, Timothy 1 Bravo, Cristián 1 Camacho-Collados, José 1 Chen, Lihan 1 Chen, Yunwen 1 Cruz, Meenalosini Vimal 1 Esuli, Andrea 1 Francis, Jonathan 1 Hamilton, William L. 1 Jia, Chenyan 1 Jin, Ke 1 Jorge, Alípio Mário 1 Ju, Yuqi 1 Kitamura, Nariaki 1 Koto, Fajri 1 Labelle, Felix 1 Lau, Jey Han 1 Lavrač, Nada 1 Li, Haonan 1 Li, Zuchao 1 Liang, Jiaqing 1 Liu, Jingping 1 Liu, Ruibo 1 Loureiro, Daniel 1 Lu, Xiaopeng 1 Malagò, Luigi 1 Martinc, Matej 1 Moreo, Alejandro 1 Mues, Christophe 1 Namburu, Anupama 1 Navarro, Ingrid 1 Oh, Jean 1 P., Mangalraj 1 Pollak, Senja 1 R., Nandha Kumar 1 S., Sudhakar Ilango 1 Sebastiani, Fabrizio 1 Sethuraman, Sibi Chakkaravarthy 1 Škrlj, Blaž 1 Stevenson, Matthew 1 Talebpour, Alireza 1 Tripathy, Jatin Karthik 1 Vijayakumar, Vaidehi 1 Volpi, Riccardo 1 Vosoughi, Soroush 1 Wang, Chao 1 Wei, Jason Z. 1 Xiao, Yanghua 1 Xu, Guangxuan 1 Zhang, Zhisong 1 Zhao, Hai 1 Zhou, Junru all top 5 Cited in 9 Serials 3 Artificial Intelligence 3 The Journal of Artificial Intelligence Research (JAIR) 1 Machine Learning 1 European Journal of Operational Research 1 Journal of Applied Non-Classical Logics 1 Data Mining and Knowledge Discovery 1 Synthesis Lectures on Artificial Intelligence and Machine Learning 1 Computer Science Review 1 Information Geometry Cited in 5 Fields 11 Computer science (68-XX) 2 Statistics (62-XX) 1 Mathematical logic and foundations (03-XX) 1 Differential geometry (53-XX) 1 Game theory, economics, finance, and other social and behavioral sciences (91-XX) Citations by Year