Apertus 1.5 - first impressions from using Switzerland’s updated AI model

On July 24, the Swiss AI Initiative — EPFL, ETH Zurich, and CSCS — released Apertus 1.5. By July 25, we had already deployed it to oss.zuericitygpt.ch. In this blogpost we will take a look at our first impressions testing Apertus 1.5 on the OSS version of ZueriCityGPT.

It was not just the open-source side that upgraded. The production system at zuericitygpt.ch switched from GPT-4o-mini to GPT-5.6 Luna — the cost-efficient tier in OpenAI's latest model family. So in the following, we will take a look at the effect of two simultaneous model upgrades on ZueriCityGPT, tested against the same knowledge base.

What is new in each model

Apertus 1.5 adds 2 trillion tokens of continued pretraining to the 70B model. For the first time, the model accepts images alongside text, and audio input is available experimentally. Instruction following — a clear weak spot in 1.0 — was a focus of the update. The model remains fully open-source under Apache 2.0, trained on the Alps supercomputer at CSCS.

GPT-5.6 Luna is the smallest model in OpenAI's GPT-5.6 family (Sol / Terra / Luna), released on July 9, 2026. It supports a million-token context window and is designed for high-volume, latency-sensitive workloads — the profile of a public-sector chatbot like ZüriCityGPT. The production system ran on GPT-5.4-nano for most of July before switching to Luna on July 31; our controlled test on August 7 ran against Luna.

A disclaimer about the comparison

An important point needs to be reiterated, as it was often misunderstood in the first Apertus release: it is not possible to directly compare the performance of a Large Language Model (LLM) like Apertus to an integrated solution like Luna, which does not provide access directly to the closed LLM inside of it. The software stack that is run by OpenAI to optimize interactions with their models is proprietary, whereas Apertus is running on an open-source stack that you can deploy in the cloud or on-premises.

Similarly, while ZueriCityGPT has an OSS model version, not all source code of the underlying system LiipGPT has been open-sourced at this point. LiipGPT ships with a lot of features that a sophisticated Retrieval-Augmented-Generation (RAG)-system needs in order to provide you with accurate answers based on a curated knowledge base.

Both models run on the same LiipGPT platform and the same knowledge base of official content from stadt-zuerich.ch. Note that the OSS version also uses different models for embedding (Qwen3-Embedding-0.6B) and reranking (BGE-Reranker), which can affect answer quality. Neither system uses web search — all answers are generated exclusively from documents retrieved from the stadt-zuerich.ch knowledge base via RAG.

The OSS version of ZüriCityGPT will add 16k tokens from the chunks into the prompt while the Luna version adds 32k tokens. This is because the price of Apertus tokens is roughly 4 times higher than Luna model tokens. We therefore actively choose to constrain the amount of tokens used for cost-control and to achieve a similar cost to question ratio.

The numbers

We have two data sources: historical quality metrics from real user traffic (measured automatically by the LiipGPT platform), and a controlled test where we asked 20 identical questions to both systems on the same day.

A note on how to read the comparisons: throughout this post, we make two kinds of comparison. Before → after compares the same system across its model upgrade (Apertus 1.0 → 1.5, or GPT-4o-mini → GPT-5.6 Luna). Head-to-head compares the two systems against each other on the same question, same day. We always name the specific models so it is clear which comparison we are making.

Historical metrics — real user traffic

Over the past months, the LiipGPT platform has scored every answer using our three established metrics:

• Match Rate (MR: is the answer acceptable?)
• Match Rate +1 (MR+1: is the answer actually good?)
• Faithfulness (Faith: does the answer stick to the source documents?).

In the following, we aggregate each model's entire deployment period. This gives the clearest before-and-after picture.

Apertus 1.5 made the bigger relative leap: Match Rate +1 jumped from 40% to 59% — a 19 percentage point improvement. GPT-5.6 Luna went from 70% to 87%, a 16 percentage point jump. The absolute gap narrowed only slightly, from 30 points to 27.

Controlled test — 20 identical questions

We repeated the exact same 20 questions from the original blog post, asking them on both systems on August 7, 2026. One person asked all 20 questions on each system — this is not a user study but a controlled comparison isolating model changes from knowledge-base changes.

Apertus 1.5 improved its average score from 4.40 to 4.70. GPT-5.6 Luna came in at 5.15.

How did the three key metrics change?

Our automated scoring scale runs from 1 (wrong or refuses to answer) through 3 (partially correct) to 6 (excellent). We define "acceptable" as any score above 2 — the answer is at least partially useful. "Good" means above 3 — the answer is substantively correct and helpful, not just borderline.

*The faithfulness delta for Luna is a measurement artifact — see the discussion in "What we learned" below.

Apertus 1.5 gives fewer acceptable answers (85% vs 100%) but more good ones (75% vs 65%). This is the "honest refusal" pattern: the model no longer produces borderline answers that scrape past the threshold. When it answers, it answers well; when it cannot, it says so. The Apertus team confirmed this matches their own testing with OR-Bench, a benchmark designed to measure over-refusal behaviour. GPT-5.6 Luna shows a milder version of the same trend.

In a head-to-head battle, GPT-5.6 Luna won 6 questions, Apertus 1.5 won 2, and 12 were ties. Compare that to the original: GPT-4o-mini won 9, Apertus 1.0 won 3, with 8 ties. Ties grew from 8 to 12 — meaning Apertus now matches the production model on most questions.

Matched questions — real users, same question, both systems

Twenty-five real user questions were asked on both systems during the Apertus 1.5 period. The questions from different users who but we chose to simulate them on both systems to achieve a comparison.

On matched questions, the gap in average score is just 0.16 points (4.92 vs 5.08). Match Rate: 96% vs 100%. Match Rate +1: 76% vs 92%. Faithfulness: 0.59 vs 0.83.

The example questions revisited

In the original blog, we highlighted specific scenarios to make the numbers tangible. Here is how they changed — plus two new ones that show the practical impact of both upgrades.

"Est-ce que tu parles français?" — both answer now

This was the most visible failure in the original comparison. When asked "Do you speak French?", GPT-4o-mini responded fluently in French. Apertus 1.0 declined, saying it could not answer questions about its own linguistic capabilities.

Apertus 1.5 now answers: "Oui, je parle français. Je suis à votre disposition pour vous aider dans la langue que vous préférez." The IFEval improvement that the Apertus team targeted has real practical impact.

Population — improved but still off

In the original test, GPT-4o-mini returned Zurich's current population (452,421), while Apertus gave a 2013 figure. Apertus 1.5 now returns a more up-to-date number but uses the wrong date. GPT-5.6 Luna cites the correct combination of time and population. Apertus 1.5 plays a part in this, but retrieval logs tell a different story: the OSS embedding model (Qwen3-Embedding-0.6B) ranks old statistical PDFs from 2010–2013 above current population pages, and because those PDFs are large, they fill the smaller context window (16k vs production) before up-to-date sources are included. The production system's embedding model (OpenAI text-embedding-3-small) retrieves the current data as its top result.

Waste dump — the tables turned

This was Apertus 1.0's showcase win in the original blog: scoring 6/6 where GPT-4o-mini scored 3/6. In 1.5, the roles reversed. GPT-5.6 Luna now delivers a comprehensive answer. Apertus 1.5 asks the user to specify what kind of waste they mean. This is the clearest example of Apertus 1.5's new pattern: better instruction following sometimes means it asks clarifying questions rather than making assumptions.

"Who is the mayor?" — the honest refusal in practice

This question illustrates why acceptable answers dropped from 100% to 85%. GPT-5.6 Luna correctly answered "Raphael Golta". Apertus 1.5 refused: "The provided information does not contain the name of the current mayor of the city." Rather than guessing like 1.0 did (which returned the outdated name "Corine Mauch"), 1.5 admits the gap. This is a retrieval issue, likely amplified by the smaller OSS embedding model.

"was gibts neues im zoo?" — Apertus finds its voice in German

Apertus 1.0 scored 3/6 — a vague, generic answer. Apertus 1.5 now delivers a detailed German response covering the Lewa Savanne, the Kaeng Krachan Elefantenpark, and the Zooseilbahn controversy. It scores 5/6, matching GPT-5.6 Luna.

This is notable because the knowledge base is predominantly German, and zoo news is the kind of practical, frequently-asked city question where a chatbot needs to perform. Apertus 1.0 struggled to synthesize multiple retrieved documents into a coherent answer; 1.5 does this naturally. The improvement is not about finding the right documents — the RAG pipeline retrieved them before, too — but about what the model does with them once retrieved.

What we learned

The quality gap is closing, from both sides. Apertus 1.5 narrowed the gap substantially — but both models got better at the same time. On matched real-user questions, the average score difference is just 0.16 points. On the controlled test, the gap went from 0.50 (4.40 vs 4.90) to 0.45 (4.70 vs 5.15).

Instruction following improved — with a tradeoff. Apertus 1.0 had a clear weakness in instruction following (IFEval 44%). In 1.5, the French language question works, the identity answer is clean, and the English responses are more natural. But the same improvement created a new pattern: when retrieved documents do not clearly contain the answer, the model now refuses rather than attempting to synthesize. This dropped the controlled-test Match Rate from 100% to 85%.

The English/German gap persists — but it is not the model. We asked "wie alt ist die stadt zürich?" in German and "How old is the city of Zurich?" in English. Apertus 1.5 scored 6/6 on the German version and 2/6 on the English version. The gap most likely comes from the OSS embedding model struggling with cross-lingual retrieval against the predominantly German knowledge base.

Faithfulness requires careful measurement. The faithfulness numbers initially included refusal answers, which led to low scores because the refusal obviously cannot be grounded in facts. On that basis, GPT-5.6 Luna scores 0.84 — essentially identical to GPT-4o-mini's 0.83. Apertus improved slightly (0.53 to 0.56 in production traffic, 0.60 to 0.61 in the controlled test). "Faithfulness" is our platform's metric for how well the answer sticks to the retrieved sources — it is not a term the Apertus team uses internally, but they confirmed that general accuracy improvement and alignment work are ongoing goals. The gap between Apertus and the production system (0.56 vs 0.84) depends as much on how the RAG pipeline surfaces and frames source material as on the model itself.

Both systems benefit equally from knowledge base updates. The knowledge base now correctly reflects that Raphael Golta replaced Corine Mauch as Stadtpräsident in May 2026. Both models handle this correctly — confirming that the LiipGPT platform's RAG pipeline works as designed: model-agnostic, with the knowledge base as the single source of truth.

Looking ahead

Compared to our recent test of Apertus 1.0, the picture has shifted meaningfully. Back then, Apertus was an experiment with many limitations. Apertus 1.5 seems to handle many questions at the same level as GPT-5.6 Luna. The cases where it still falls short (faithfulness, cross-lingual retrieval) are increasingly about other open-source components involved, less on the model itself. But in our test field of RAG application, the cost ratio is important. If you compare the token cost of GPT-5.6 Luna on Azure with the token cost of Apertus 1.5 70B on Infomaniak, the Apertus model will cost you roughly 4 times as much as the GPT model. Obviously there are other advantages like enhanced data sovereignty and higher ethical standards in training the model but still, being able to run Apertus 1.5 in sizes between 8B and 70B would help achieve a better cost to value ratio in our case.

We are sharing these results with the Apertus team and the wider community for review. As before, our aim is not to provide a scientific benchmark but a practical report from a production deployment. Consider that these are just our first impressions, and we are looking forward to further test Apertus with other RAG-deployments and in more use case scenarios. If you would like to test how Apertus 1.5 works when refining brand text, our latest TextMate has the model available as well.

The question from our last post — "how much quality gap are you willing to accept for full sovereignty?" — now has a different answer. On most questions, the answer quality is equivalent. The remaining gap is in faithfulness and source attribution — Apertus is improving, and GPT-5.6 Luna maintains the same level as its predecessor once measurement artifacts are accounted for. Closing that gap will require improvements to the open-source RAG stack (embedding, reranking) as well as finding a good mix between model size and speed / price and might be less focused on the language model itself.

Please do not hesitate to contact me if you have any questions about Apertus or AI solutions in general; I’d be delighted to discuss this with you.

Acknowledgements

Thank you to the Swiss AI Initiative — EPFL, ETH Zurich, and CSCS — for continuing to develop Apertus as a public good. Thank you Oleg Lavrovsky and Martin Renou for reviewing a draft of this post and providing feedback from the Apertus team's perspective.

Thank you Christian Stocker and the LiipGPT team for deploying the update within 24 hours of release.

Thank you Public AI and Infomaniak for hosting Apertus inference for us.

Parts of this analysis were prepared with the help of Claude. The data was collected and scored automatically by the LiipGPT platform.