Publicación:
Language identification with scarce data: A case study from Peru
Language identification with scarce data: A case study from Peru
No hay miniatura disponible
Fecha
2018
Autores
Espichán-Linares A.
Oncevay-Marcos A.
Título de la revista
Revista ISSN
Título del volumen
Editor
Springer Verlag
Proyectos de investigación
Unidades organizativas
Número de la revista
Abstracto
Language identification is an elemental task in natural language processing, where corpus-based methods reign the state-of-the-art results in multi-lingual setups. However, there is a need to extend this application to other scenarios with scarce data and multiple classes to face, analyzing which of the most well-known methods is the best fit. In this way, Peru offers a great challenge as a multi-cultural and linguistic country. Therefore, this study focuses in three steps: (1) to build from scratch a digital annotated corpus for 49 Peruvian indigenous languages and dialects, (2) to fit both standard and deep learning approaches for language identification, and (3) to statistically compare the results obtained. The standard model outperforms the deep learning one as it was expected, with 95.9% in average precision, and both corpus and model will be advantageous inputs for more complex tasks in the future.
Descripción
Palabras clave
The standard model,
Deep learning,
Information management,
Linguistics,
Natural language processing systems,
Best fit,
Complex task,
Corpus-based methods,
Language identification,
Learning approach,
Multiple class,
State of the art,
Big data