Since September 2025, we have been running Apertus — the Swiss open-source language model developed by EPFL, ETH Zurich and the Swiss National Supercomputing Centre (CSCS) — on ZüriCityGPT, our reference implementation of a City AI assistant. Eight months later, I want to share what we have learned and what we are curious about for the upcoming Apertus 1.5 release.

A brief history of LiipGPT and ZüriCityGPT

2023, Chregu released the first version of LiipGPT. 2024, we were able to use only open-source models on ZüriCityGPT. 2025, we switched the open-source version to the newly released Apertus model. In this blog post, we talk about the insights gained since then.

Setting up the experiment

LiipGPT, our generative AI framework, has a pluggable model layer. That means we can swap the underlying language model without changing anything else — same Retrieval-Augmented Generation (RAG) pipeline, same knowledge base of official content from stadt-zuerich.ch. This gave us a straightforward way to compare models in a real deployment.

The production system at zuericitygpt.ch runs GPT-4o-mini on Azure Europe. Alongside it, we run an open-source variant at oss.zuericitygpt.ch where in the past we compared Llama and Mixtral models. We switched the open-source variant to Apertus-70B when it was released in September 2025, then moved to Infomaniak-hosted Apertus in April 2026. (To be clear: ZüriCityGPT itself and the LiipGPT platform are not open source — but the underlying language model on the OSS variant is.)

We run Apertus as the 70-billion-parameter model. It was trained from scratch on the Alps supercomputer using Swiss hydroelectricity, with 15 trillion tokens of training data spanning over 1,800 languages. It is fully open-source under an Apache 2.0 licence.

Over the last eight months, we collected over 39,000 conversations on the production side and 3,400 on the open-source side. Quality is measured using three metrics: how often an answer is acceptable (Match Rate), how often it is actually good (Match Rate +1), and how faithfully it reflects the source documents (Faithfulness). These are metrics our LiipGPT team has established as practical product metrics, not scientific benchmarks — the data is calculated real-time and asynchronously by the LiipGPT platform while users ask different questions over time, evolving website content, and model endpoint changes. Parts of the analysis were prepared with the help of Claude. Our aim is not to provide a scientific study but more a practical report. I share these results with the Apertus community because real-world deployment data of this open-source model is rare — and we think it's useful.

What we observed

Both models get the job done. On the basic question — does the user get an acceptable answer? — both models perform similarly. GPT-4o-mini scores 81%, Apertus 82% on historically matched questions. In a controlled test with 20 identical questions asked at the same moment, both hit 100%. Both models generally work as expected.

The gap shows up in answer quality. When we raise the bar to "is this answer actually good?", GPT-4o-mini scores 72% and Apertus 55%. That is a noticeable difference, and it matters for users who expect precise, well-structured responses.

Faithfulness is more nuanced than we expected. GPT-4o-mini averages 0.79, Apertus 0.50 on our faithfulness metric. But these numbers need context. We noticed something we started calling the "faithfulness paradox": when a model says "I cannot answer this," you might expect high faithfulness — no hallucination, after all. In practice, these evasive answers score low because the model is failing to use information that is available. A model that engages with its sources and cites them scores higher, even though it is making more verifiable claims. Both behaviours have tradeoffs, and both showed up across the two systems.

Sometimes Apertus gave the better answer. The most striking example was the waste dump question: "What are the rules for using the Zurich waste dump?" GPT-4o-mini responded cautiously, saying it could not provide specific rules. Apertus gave a detailed, structured answer with recycling centre details, phone numbers, and cited sources — scoring 6/6 on quality where GPT-4o-mini scored 3/6. When the knowledge base contains a clear, structured answer, Apertus can be surprisingly good at extracting and presenting it.

That said, this willingness to commit cuts both ways. In cases where the source material is ambiguous, Apertus can feel more helpful while being less strictly grounded — and for public-sector chatbots, traceability matters as much as helpfulness.

GPT-4o-mini is more concise and better at synthesis. Apertus tends to produce longer, more verbose answers. In a public-service chatbot, that is a real UX issue — users want a direct answer with clear next steps, not three paragraphs of context.

GPT-4o-mini also handles factual freshness better: it returned Zurich's current population (452,421), while Apertus fell back to a 2013 figure. When information has changed or the answer requires connecting multiple sources, GPT-4o-mini felt more reliable.

Language switching is a weak spot. When we asked "Est-ce que tu parles français?", GPT-4o-mini responded fluently in French. Apertus declined, saying it could not answer questions about its own linguistic capabilities. In more recent tests we saw that Apertus followed the language instructions in better ways, so we are looking forward to see this further improve.

The Apertus-70B IFEval score of 44% (vs 58% for comparable models) confirms that instruction following is an area where the model has room to grow.

The trend is positive. In October 2025, Apertus achieved a Match Rate +1 of 59%. By June 2026, it had climbed to 63% — the best month on record. That is not a dramatic leap, but the direction is consistent and it came without a major model update. Meanwhile, GPT-4o-mini has been stable at 63–83% throughout. The gap is narrowing.

What we are looking forward to

The Apertus team is preparing version 1.5, with improvements expected in multimodal support, agentic capabilities, tool calling, enhanced reasoning, and better training for regional Swiss languages. We plan to update ZüriCityGPT's open-source instance as soon as it becomes available and continue our quality monitoring.

What encourages us most is that the collaboration is flowing in both directions. After our aiLights talk, the Apertus team shared our findings with their staff. This close feedback loop between research and practice is exactly what makes the Swiss AI ecosystem worth investing in.

My practical advice, for anyone considering sovereign AI: start with the use case, then choose the model. If your priority is maximum answer quality today, more popular models are usually the stronger choice. If your priority is data sovereignty, transparency, and independence — and your knowledge base is well-curated — Apertus already delivers. The question is not "sovereign AI or quality" but "how much quality gap are you willing to accept for full sovereignty?" And that gap is closing.

Check the Slides and Recording for further details.

Acknowledgements

I would like to thank the ETH Zurich, EPFL, and the wider Swiss AI Initiative community for making Apertus available as a public good. As Martin Jaggi from EPFL put it: "We aim to provide a blueprint for how a trustworthy, sovereign, and inclusive AI model can be developed." (source) Eight months in, we can see that blueprint taking shape and are looking forward to test the 1.5 model in practice as well.

• Thank you Christian Stocker and the LiipGPT team for your dedication to build a practical, experiment-driven and scalable solution for AI chat and search.

• Thank you Sabine Wildemann and the aiLights team for hosting our talk.

• Thank you Oleg Lavrovsky and the Apertus community for your feedback and support.

• Thank you Public AI and our partner Infomaniak for providing the infrastructure.

Build with Apertus: join Hack Apertus

Want to get hands-on with sovereign Swiss AI? Hack Apertus is a two-stage open-source hackathon series where teams build on Apertus to develop blueprints for sovereign infrastructure across the public and private sector. Liip is a partner — we are excited to see what the community builds.

Sign up at hackapertus.ch.