A chatbot is only as good as the content it is based on. Outdated pages and contradictions can sit unnoticed on a website for years. A chatbot exposes them without mercy, because it reads everything and cites what it finds.
So we built a tool that checks the content before the bot answers from it.
When a chatbot says something wrong, the problem is rarely the model. Usually it is on page 1,243.
A chatbot is a very attentive reader
Take Acme Inc, a fictional Zurich automation company with a small website: 25 pages, one team, a handful of services, some news and events, the usual legal pages. The home page says "founded in 1987". The About page says "since our founding in 1994". Jonas Brunner is CFO on the team page, but a news item from May congratulates him as Head of Robotics Engineering. The response time in the predictive maintenance subscription is four hours, or 24, depending on which page is open.
Each of these is a small error. Nobody reads both pages in the same sitting, so they go unnoticed and never get fixed. A chatbot does read both, and it answers one way or the other depending on the question.
On a 25-page site, someone would still catch this. Our clients run websites with 15,000+ pages, grown over ten or more years and maintained by dozens of editorial teams and hundreds of editors. Nobody there knows what the website as a whole claims, and no human can guarantee quality at that scale. So we need to automate content quality, which means first defining what "good quality" actually is.
"good" content
We split "quality" into three layers, in the order a machine can check them: objective facts first, then the reader's judgment, then the context of the system.
In short:
- Content quality asks whether the content is right (correctness, completeness, currency, consistency).
- Communicative quality asks whether it reaches the reader (comprehensibility, relevance, structure, tone).
- Functional quality asks whether it works in the system (findability, accessibility, credibility, maintainability).
Some of this remains editorial judgment. But a surprisingly large part of this list can be checked by a machine.
Three questions
From the list we picked the three points that hurt most on large websites:
- Consistency: where does the content contradict itself? Opening hours, fees, deadlines, responsibilities, numbers. The same fact often lives in three places, and at least one is outdated.
- Currency: what has expired? Content whose facts have run out: a registration deadline from last year, a date that has passed.
- Relevance: do users ask things the content does not answer at all? This is the most invisible gap. Nobody complains about a page that does not exist.
A manual content audit answers these questions for a sample, once, and is outdated by the time it is finished. We wanted something that checks the whole site and can be repeated.
Content Quality Analysis
We built a module in LiipGPT that we call Content Quality Analysis, CQA for short, "c’est quoi?" when we feel fancy. It crawls a website and runs three checks (one per question above). The result is a prioritised list of concrete findings with sources, which a person confirms or dismisses.
1. The contradiction finder
The naive approach would compare every page with every other page. At 1,691 pages that gives about 1.4 million pairs, and most of them have nothing to do with each other. So we work in stages:
- Extract claims. A language model pulls the checkable facts out of every page: "founding year: 1987", "response time predictive maintenance: 4 hours", "Jonas Brunner: CFO". Each claim gets a type and an entity.
- Find candidates. Claims are embedded as vectors. We only look at pairs that are semantically close, regardless of which section of the website they live in.
- Filter cheaply, verify expensively. A small model discards pairs that are obviously harmless. A strong model judges what remains against the full pages, three times independently. Only what at least two of three runs confirm makes it into the report.
The last stage is deliberately the expensive one. We would rather have few findings that hold than many the team has to re-check by hand.
For Acme Inc the funnel looks like this: 25 pages, 178 claims, 78 candidate pairs, three confirmed contradictions.
On a real site with around 1,700 pages, 8,680 claims produced almost 38 million theoretical pairs, and seven confirmed contradictions remained at the end.
2. Outdated content
The second check is less spectacular and still often the biggest surprise. CQA looks at the age of the content per section and searches for deadlines that have already passed: "register by 31 March 2025", "valid until end of 2024". A multilingual date parser tells a deadline apart from a narrative date ("in 2019 we ...").
On Acme's site, six of the 25 pages have not been touched in over three years, four of them for more than five, among them the privacy policy from 2019 and a COVID notice from March 2020. And the site still carries deadlines that have passed: an apprenticeship intake that closed in March 2024 and the COVID notice's "by appointment only until 30 April 2020". On the real site mentioned above, 37 percent of pages had not changed in over two years. Old pages aren't necessarily wrong, but they show the team where to look first.
3. Intent alignment
The third check needs a running chatbot, because it uses the real questions people ask. CQA groups these questions by intent, names each group ("order spare parts", "prices for maintenance contracts") and has a model judge whether the existing content answers the group, answers it partially, or not at all.
For Acme Inc we set up a LiipGPT chatbot and asked it 1,240 questions of the kind customers and prospects would ask. The result: 18 intents, 11 answered, 5 partial, 2 not at all. Partial, for example: "how long do spare parts take to arrive?" The spare-parts page explains how to order and says nothing about delivery times. Not answered at all: "do you work outside Europe?" and "do you offer leasing or financing?". The website answers neither, though both come up often.
On a large public-sector chatbot the same picture looked like this: 107,000 questions, 111 intents, the 50 most frequent judged: 22 answered, 21 partial, 7 not at all. The biggest gap was banal and expensive at the same time. People wanted an e-mail address and did not get one in 63% of the cases, because there is none in the content.
Actionable items for the team
A work list instead of an audit. Instead of saying "content quality is 6.4 out of 10" it returns an actual Todo List: "these seven places contradict each other, these twelve deadlines have expired, the audience asks about these two topics and the website does not answer them". Every finding has a URL. It can be fixed this afternoon.
Sorted by owner. Findings can be filtered by website section and, once the mapping is in place, by the organisational unit responsible. Communications sees its findings, HR sees its own.
Repeatable. An analysis can be run again after every relaunch, or simply every quarter. Fixed findings disappear, new ones show up, and the chatbot gets better without anyone touching the model, prompts or configuration.
What it is not
CQA is not a spell checker and does not rewrite text. It does not replace an editorial concept and does not judge whether a text is complete or comprehensible. That remains human work, and findability is better served by SEO tooling. And "confirmed" means confirmed three times by a language model, not by a person. The final call on whether 1987 or 1994 is right is made by someone at Acme. The tool only makes sure the question gets asked.
Have a look at page 1,243
Anyone planning a chatbot, running one and wondering why it sometimes claims things that "aren't quite right", or simply asking which of their pages need work first: we are happy to talk.