🔍 Read the full analysis: What My 71-Configuration AI Test Means For My News Pipeline on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Thorsten Meyer reports that a test of 71 model configurations for his news-writing pipeline favored GPT-6 Astra for quality but GPT-6 Luna for production cost. He also says two fallback models produced invented quotes or unusable output and were removed. The results come from a limited test and are not an independent comparison of all AI models.
AI writer Thorsten Meyer says a test of 71 model configurations for his news pipeline led him to use GPT-6 Luna for routine production, despite GPT-6 Astra receiving the highest quality scores. He also removed two fallback models after his judges flagged invented quotes in one model’s articles and invalid output from another. The findings concern his own workflow, not a general ranking of AI systems.
Meyer says he tested configurations between September 8 and October 13, 2026, using the production prompt and two real news sources: a 6,340-character product filing and a 619-character political news item. Each configuration generated an English and a German article. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, assessed source fidelity, fabrication, clarity about uncertainty, news value, structure, polish and German language quality.
In the standard test, GPT-6 Astra scored 4.00 out of 5 in both languages and received a perfect source-fidelity score. In a follow-up run involving 12 articles, it scored 3.93, with a reported 95% bootstrap interval of 3.83 to 4.02. Meyer says Astra cost about $0.12 per article and took around 50 seconds. GPT-6 Luna scored 3.67 in English and 3.68 in German in the standard test; in the robustness run it scored 3.67, with source fidelity of 4.71. Meyer reported a cost of about $0.0012 per article and an average time of 17 seconds.
Meyer says the cost and quality trade-off led him to choose Luna for routine work. He also tested Luna at six reasoning-effort settings. The scores varied, but he judged the differences too small to be reliable; the maximum setting cost about 12 times as much and took about 12 times as long as the no-effort setting, without a clear quality gain. He says all Luna articles in the test produced valid output after the repair step used by his pipeline.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Cost, Quality and Editorial Risk
The results illustrate a practical decision for publishers using automated drafting: the highest-scoring model may not be a cost-effective choice for every article. In Meyer’s test, Astra’s reported price was roughly 100 times Luna’s, while Luna still produced scores he considered suitable for routine production. That conclusion depends on his quality threshold, sources and editing workflow; another publisher may reach a different result.
The fallback findings point to a separate risk. Meyer says GPT-4.1 nano scored 1.62 in English and 1.68 in German, with source fidelity of 1.00, and that judges identified invented quotes. He says the locally run gpt-oss-120b produced zero valid articles in four attempts and generated named quotations not present in the source. Such failures matter because a backup system can still publish misleading material if the pipeline treats its output as acceptable. Meyer says both models were removed from the route on October 7.
These are reported results from one operator’s evaluation, not evidence that the same models will perform identically on other beats, prompts or source material. The test nonetheless shows why a newsroom should evaluate the whole production route, including failover behavior, rather than select a model solely from a public leaderboard.
As an affiliate, we earn on qualifying purchases.
How Meyer Built the Test
Meyer says he designed the evaluation to resemble his daily work: each model received a real source and the same production prompt, with English and German editions required. He included API, subscription, free-tier and locally run open-weight configurations. The two sources differed sharply in length, and he says the shorter political item better reflects the thin source material common in his workflow.
The first comparison used two articles per configuration, so Meyer set a rule that score differences below about 0.3 points could be noise. He then ran the strongest configurations on 10 additional articles, mostly based on short sources, and treated that robustness run as the more reliable ranking. He says an article counted only if it produced valid output after the same repair step used in production.
The test also found that coding performance did not predict news-writing performance. Meyer reports that Claude Opus 5.5 scored 3.46 in English and 3.39 in German in the initial writing test, and 3.52 in the robustness run. Those figures were below Luna’s robustness score, despite Opus being one of the judges and, according to Meyer, a strong coding model. The comparison is specific to his article-writing task.
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Results
The source report does not provide the full article set, judge-level scores, or a complete configuration-by-configuration results table. It is also not clear from the supplied material how the two judges’ assessments were combined, whether they were blind to model identity, or how much human editing followed the pipeline’s repair step. Those details would help readers assess reproducibility and the practical quality of published output.
The initial test used only two sources per configuration, and even the later robustness run comprised 12 articles for the strongest configurations. Meyer acknowledges that small score gaps may reflect noise. His account does not establish how the models would perform across a broader range of topics, languages, breaking-news conditions or changes in prompts and model versions. The claimed cost figures also describe his setup and should not be treated as universal prices.
The report says GPT-6 Luna now runs first, but the supplied text ends before specifying the complete replacement or failover arrangement. It is not clear which backup, if any, now handles requests when Luna is unavailable or returns invalid output.
AI language model for news writing
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Pipeline After October 7
Meyer says GPT-4.1 nano and gpt-oss-120b were removed from the news route on October 7, and that Luna is now the production model. The source account does not give a date for a further test or describe a finalized backup chain. Any later assessment would need to check whether Luna’s quality holds across a larger and more varied stream of sources, and whether the revised route rejects unsupported quotations and malformed output before publication.
For readers and newsroom operators, the immediate takeaway is limited but concrete: Meyer’s reported test changed his own model choice and failover setup. The next useful evidence would be results from continued production monitoring, including error rates, human correction needs, costs and performance on short or rapidly changing news sources.
cost-effective AI writing assistant
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
Which model did Meyer choose for routine news production?
Meyer says GPT-6 Luna now runs in production because it scored below Astra but cost far less per article in his test.
Which model received the highest quality score?
GPT-6 Astra scored 4.00 out of 5 in both languages in the standard test and 3.93 across 12 articles in the robustness run, according to Meyer.
Why were two fallback models removed?
Meyer says GPT-4.1 nano produced invented quotations flagged by judges, while gpt-oss-120b produced no valid articles in four attempts and also generated unsupported named quotes.
Does the test show that Luna is the best news-writing model?
No. It reports results for Meyer’s prompt, sources, languages and evaluation process. The findings do not establish how the models would rank in other newsroom workflows.
Did higher reasoning effort improve Luna’s articles?
Meyer says score differences across six effort settings were within the noise of the test. The maximum setting took and cost about 12 times as much as the no-effort setting without a clear gain.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
