IRISA is proud to be involved in the creation of EuroBERT, the most powerful multilingual encoder model for information retrieval, classification and quality estimation (for summarisation and machine translation) in 15 languages.
Caio Corro, an associate professor at INSA Rennes and a researcher in the LinkMedia research team, collaborated on the creation of EuroBERT.
Intelligence Artificielle et NLP : la suite d'encodeurs EuroBERT franchit un nouveau cap
Trained on 5,000 billion tokens, this suite offers sovereign, open-source models delivering the best text representation performance for European languages, as well as for tasks related to mathematics and coding.
The collaboration between CentraleSupélec’s MICS laboratory, Diabolocom, Artefact, and Unbabel, supported by the technological expertise of AMD and CINES, has resulted in the release of the most advanced multilingual text representation model. This model serves as a fundamental building block for information retrieval (RAG), classification, and quality estimation (summarization, translation).
These types of models are essential in natural language processing (NLP) and have been among the most downloaded on Hugging Face for many years. Their ability to accurately capture the meaning and context of sentences, offering a refined and in-depth linguistic understanding, is crucial for the development of advanced artificial intelligence applications. The new EuroBERT model is available as of March 10, 2025, under the Apache 2.0 license on the Hugging Face platform.
The research project was led by Nicolas Boizard, a Cifre PhD candidate at Diabolocom, with major contributions from Hippolyte Gisserot-Boukhlef, a Cifre PhD candidate at Artefact, and Duarte Alves, a PhD candidate at Instituto Superior Técnico (IST). It builds upon research initiated by Pierre Colombo, Associate Professor at CentraleSupélec, and was conducted under the supervision of Céline Hudelot, Director of MICS, and André Martins, Associate Professor at IST. The results are detailed in a paper published on arXiv on March 10, 2025: https://arxiv.org/abs/2503.05500
A new technological leap in text encoding
EuroBERT stands out from currently available encoders in five key ways:
- It is sovereign and fully open-source, including both its source code and datasets.
- It supports 8 major European languages as well as 7 of the most widely spoken non-European languages.
- Trained on 5 trillion tokens, twice the amount of data used for standard encoders or generative models such as Llama 2 (2 trillion tokens), EuroBERT offers optimal capabilities without additional usage costs.
- The EuroBERT family provides the best foundation for information retrieval (RAG), classification, and quality estimation (summarization, translation).
- It excels in previously underexplored areas such as mathematical data processing and programming languages.
It is available in three model sizes (210M, 610M, and 2.1B), offering an optimal balance between speed, quality, and cost, tailored to the needs of enterprise users.
EuroBERT is thus transforming natural language processing applications based on sentence representations, such as text analysis, information retrieval, classification, and information extraction.
The strength and added value of collaborative research
As with the CroissantLLM and EuroLLM models published on Hugging Face in 2024, these scientific advancements were made possible through a close and dynamic public-private collaboration rooted in the Paris-Saclay ecosystem and extended across Europe. The teams from MICS, IST, Diabolocom, Artefact, and Unbabel worked together within the framework of three ongoing PhD projects, supported by the French supercomputer Adastra, powered by AMD Instinct™ Accelerators and AMD EPYC™ processors.
...
Read the full press release on Artefact website (in english & french)
Lire l'entièreté du communiqué de presse sur le site de CentraleSupélec (in french)