FacebookAI/xlm-roberta-base · Hugging Face

Overview

XLM-RoBERTa (XLM-R) is a cross-lingual extension of RoBERTa, introduced by Facebook AI in the paper "Unsupervised Cross-lingual Representation Learning at Scale" (Conneau et al., 2020). The model is pre-trained on 2.5 terabytes of filtered CommonCrawl data covering 100 languages, using a masked language modeling (MLM) objective where 15% of tokens are randomly masked and predicted. XLM-R achieves state-of-the-art results on cross-lingual benchmarks such as XNLI (cross-lingual natural language inference) and MLQA (multilingual question answering). The base version has 278 million parameters, 12 layers, 768 hidden size, and 12 attention heads. It supports input sequences up to 512 tokens and uses a SentencePiece (Unigram) tokenizer with a vocabulary of 250k tokens. The model is available under the MIT license and can be deployed via the Hugging Face Transformers library in PyTorch, TensorFlow, JAX, and ONNX formats.

Capabilities

• Multilingual Understanding: Supports 100 languages, including high-resource languages (e.g., English, Spanish, Chinese) and low-resource languages (e.g., Swahili, Urdu).
• Cross-lingual Transfer: Can be fine-tuned on one language and applied to others without additional training, enabling zero-shot cross-lingual transfer.
• Fill-Mask Task: Predicts masked tokens in a sentence, allowing language modeling and feature extraction.
• Tokenization: Uses SentencePiece (Unigram) tokenizer with a vocabulary of 250k tokens, which handles subword segmentation across languages.
• Architecture: Base version has 12 layers, 768 hidden size, 12 attention heads; large version has 24 layers, 1024 hidden size, 16 attention heads (559M parameters).
• Input Length: Supports sequences up to 512 tokens.
• Formats: Available in PyTorch, TensorFlow, JAX, ONNX, and SafeTensors formats.

Use cases

• Cross-lingual Text Classification: Sentiment analysis, topic labeling, or intent detection across multiple languages.
• Named Entity Recognition (NER): Identify entities in multilingual text.
• Question Answering: Fine-tune on SQuAD or MLQA for multilingual QA.
• Machine Translation Quality Estimation: Assess translation quality without parallel data.
• Zero-shot Cross-lingual Transfer: Train on English data, deploy on other languages.
• Feature Extraction: Generate multilingual embeddings for downstream tasks.
• Sentence Similarity: Compute cross-lingual sentence embeddings for retrieval or clustering.

License & deployment

• License: MIT License (as per Hugging Face model card).
• Deployment: Available via Hugging Face Transformers library. Can be deployed on CPU/GPU with PyTorch, TensorFlow, or JAX. ONNX and SafeTensors formats are supported for optimized inference.
• Official Repository: FacebookAI/xlm-roberta-base
• Hardware Considerations: The base model requires approximately 1.1GB of memory (FP32). For inference, a GPU is recommended for speed, but CPU inference is possible. The large model requires more memory (~2.2GB).
• Runtime: Inference time varies; on a modern GPU, processing a batch of 8 sequences of length 128 takes ~0.1 seconds for the base model.

Alternatives

• mBERT: Multilingual BERT with 110 languages, smaller vocabulary (110k tokens), but lower performance on low-resource languages.
• XLM: Cross-lingual Language Model by Facebook, predecessor to XLM-R, with 15 languages.
• DistilBERT Multilingual: Distilled version of mBERT, faster but lower accuracy.
• LaBSE: Language-agnostic BERT Sentence Embedding, optimized for sentence similarity.
• mT5: Multilingual T5, encoder-decoder architecture, supports 101 languages.
• RemBERT: Multilingual model with improved cross-lingual transfer.

FAQ

Q: What is the difference between XLM-R and XLM? A: XLM-R is an improved version of XLM, pre-trained on much more data (2.5TB vs 2.5GB) and covering 100 languages instead of 15. XLM-R uses RoBERTa's training dynamics and achieves significantly better cross-lingual performance.

Q: Can XLM-R be used for languages not in its 100-language set? A: No, the model is limited to the 100 languages it was pre-trained on. For other languages, consider using a model like mT5 or training a custom model.

Q: What is the recommended fine-tuning approach? A: Fine-tune on a downstream task (e.g., classification, NER) using the Hugging Face Trainer API. Use a learning rate of 2e-5 to 5e-5, batch size of 16-32, and 2-5 epochs. For cross-lingual transfer, fine-tune on one language and evaluate on others.

Q: Is XLM-R available in ONNX format? A: Yes, the model is available in ONNX format for optimized inference. You can also convert PyTorch models to ONNX using the Hugging Face Optimum library.

Q: What are the hardware requirements for running XLM-R? A: The base model requires about 1.1GB of RAM for inference (FP32). For training, a GPU with at least 8GB VRAM is recommended. The large model requires more resources.

Q: How does XLM-R compare to GPT-3 for multilingual tasks? A: XLM-R is specifically designed for cross-lingual understanding and is more efficient for tasks like classification and NER. GPT-3 is a generative model and may require more data and compute for fine-tuning.

Q: Can XLM-R be used for text generation? A: No, XLM-R is an encoder-only model designed for understanding tasks. For generation, consider encoder-decoder models like mT5 or BART.

Q: What is the license for XLM-R? A: The model is released under the MIT License, allowing commercial use, modification, and distribution.

Q: Are there any safety considerations? A: Like all large language models, XLM-R may reflect biases present in its training data. Users should evaluate and mitigate biases for their specific use case. The model card does not provide specific safety evaluations.

Q: How do I cite XLM-R? A: Cite the paper: Conneau, A., et al. (2020). Unsupervised Cross-lingual Representation Learning at Scale. arXiv:1911.02116.