Pre-trained transformer-based language models for Sundanese

• Main Authors: Lucky, H. (Author), Suhartono, D. (Author), Wongso, W. (Author)
• Format: Article
• Language: English
• Published: Springer Science and Business Media Deutschland GmbH 2022
• Subjects: Low-resource Language; Natural Language Understanding; Sundanese Language; Transformers
• Online Access: View Fulltext in Publisher

 The Sundanese language has over 32 million speakers worldwide, but the language has reaped little to no benefits from the recent advances in natural language understanding. Like other low-resource languages, the only alternative is to fine-tune existing multilingual models. In this paper, we pre-trained three monolingual Transformer-based language models on Sundanese data. When evaluated on a downstream text classification task, we found that most of our monolingual models outperformed larger multilingual models despite the smaller overall pre-training data. In the subsequent analyses, our models benefited strongly from the Sundanese pre-training corpus size and do not exhibit socially biased behavior. We released our models for other researchers and practitioners to use. © 2022, The Author(s).