📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project trained a large Italian-language LLM from scratch but scored only 4.9% on the Italian school exam benchmark. This highlights challenges in scaling language models for country-specific knowledge.
Italy’s Minerva-3B, a large language model trained from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, revealing significant challenges in achieving deep country-specific knowledge through scale alone.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s National Research Council and CINECA, trained the model on a substantial dataset of 2.5 trillion tokens, roughly half of which was Italian language data. Despite this large-scale effort and open publication of weights and data, the model’s performance on the INVALSI exam was near chance, a stark contrast to its impressive technical benchmarks.
Researchers concluded that while dataset composition and scale are important, they are not sufficient alone to develop models capable of handling complex, country-specific academic tasks. The results suggest that even significant investments in native-language data and parameters may not guarantee the desired depth of knowledge, raising questions about the optimal scale and approach for sovereign-language models.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Jetson AGX Orin 64GB Developer Kit 275 Tops, with 1TB SSD,8MP USB Camera, AI Embedded Development Provides AI Large Models
AGX Orin 64GB Development Kit makes it easy to get started with AGX Orin. Its compact size, rich…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
country-specific language AI models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications of Minerva’s Low Academic Benchmark Score
The results from Minerva-3B challenge assumptions that larger, native-language models automatically translate into better country-specific knowledge. This has broad implications for European sovereign AI strategies, indicating that scale alone may not be enough and that more targeted or different approaches might be necessary to achieve meaningful results in national languages and contexts.
The findings also highlight the importance of understanding the limits of current scaling strategies, especially when public discourse often equates larger models with better performance. Policymakers and researchers must reconsider the investment levels and methodologies needed to develop truly effective country-specific AI systems.
Background on European Sovereign LLM Development
Italy’s Minerva project emerged as a major effort to build a European sovereign language model from scratch, utilizing Italy’s national supercomputing infrastructure and a large dataset of 2.5 trillion tokens, half of which was Italian. Unlike other approaches such as Portugal’s AMÁLIA, which layered specialization onto a multilingual foundation, Minerva was trained entirely from scratch, aiming for a model with strong Italian language capabilities.
Previous European projects have debated whether continuation pre-training or training from scratch offers better results. Minerva’s approach was to train from scratch, and its performance on technical benchmarks was promising. However, the low score on the INVALSI exam exposes a gap between technical proficiency and real-world academic understanding, raising questions about the effectiveness of scale versus targeted data and training strategies.
“While dataset size and parameters are important, they are not sufficient for handling complex language tasks in specific national contexts.”
— Research team member, Orlando et al.
Unresolved Questions About Model Scaling and Knowledge Depth
It remains unclear whether increasing the size of the dataset or the number of parameters beyond current levels would significantly improve Minerva’s performance on complex academic tasks. The specific factors limiting the model’s understanding of country-specific content are still under investigation, and ongoing research aims to clarify whether different training methodologies could yield better results.
Next Steps for European Sovereign Language Models
The research team plans to continue refining Minerva, including experiments with different training regimens and data compositions. Further evaluations on diverse benchmarks will help determine whether increased scaling or alternative approaches are necessary to achieve deeper country-specific knowledge. Policymakers and AI developers are expected to reassess investment strategies based on these emerging insights.
Key Questions
Why did Minerva-3B perform poorly on the Italian school exam?
The low score suggests that despite large-scale training, the model lacks sufficient depth in country-specific academic knowledge, indicating that scale alone is not enough to master complex content.
Does this mean training from scratch is ineffective for language models?
Not necessarily. The results highlight that scale is important but must be complemented by targeted data, training strategies, and possibly different architectures to achieve desired performance levels.
What are the implications for European AI strategies?
The findings suggest that European projects may need to reconsider investment levels and methodologies, focusing more on quality and specificity of data rather than scale alone to develop effective country-specific models.
Will future models overcome these limitations?
Ongoing research aims to explore alternative training approaches, larger datasets, and different architectures, which may improve performance in complex, country-specific tasks.
Source: ThorstenMeyerAI.com
