What the search agent searches with
Our legal assistant has no embeddings and no vector database. A model searches with words, in Postgres, and a small classifier makes the quick calls around it. This is the search engine underneath, from how a passage is stemmed to how a statistics table becomes something it can quote.
Our legal assistant used to answer from the ten passages whose embeddings sat nearest the question, and it kept missing the one section that mattered. Now a model searches the documents itself, reads what it finds, and every quote it gives is checked word for word against the source. This is what it searches with. The short answer is Postgres full-text search, a few hundred lines of Go around it, and a small classifier that makes the decisions not worth a model call.
- 0model calls to ingesta document is split on its headings and indexed
- 11stemmerschosen per document, from the text itself
- 2passages per documentin one search, so one long act cannot fill the page
- 9toolssix for documents, three for official statistics
A passage is a few paragraphs under a heading#
Nothing in the ingest calls a model. A file arrives already turned into Markdown by the frontend, it is split on its headings, and each piece becomes a passage with the headings above it kept as its path. The law comes from Lovdata's open data, one passage per section. That makes ingesting fast and free, and the same document always gives the same passages.
Each document gets a stemmer, and the stemmer is picked from the text without asking a model. The first 6,000 characters are scanned for small words that are common in one language and rare in its neighbours: "ikke" and "og" for Norwegian, "och" and "inte" for Swedish, "af" and "hvad" for Danish. The language with the most hits wins, but only with at least five of them and a lead of two. Norwegian and Danish share most of their small words, and a near tie is left undecided. An undecided document is indexed as written. That loses the inflections, but it never mangles a word.
Postgres ships stemmers for Norwegian, Swedish, Danish, English, German, Finnish, Dutch, French, Spanish, Italian and Portuguese, and any of them can be set by hand on a collection. Changing it rebuilds the index from the stored text, so nothing has to be imported again.
A search is any of the words, in every language at once#
A search box usually wants all of its words. Ours wants any of them. The agent writes queries of four or five words, and a passage that carries four of the five is worth reading. The ranking puts the one with all five first.
The query is stemmed once for every language present in the collections being searched, and the literal words are added on top, so "oppsigelse" matches both its stemmed form and itself. Before stemming, the small words of every language are removed. That was a bug before it was a rule. An English stemmer handed "er", "på" and "med" keeps them as words, and a Norwegian question then matched nearly every passage in the English collection next door.
The score is Postgres' own cover-density rank, plus a fixed 0.3 when the words appear in the order they were asked. That bonus is what separates the passage that states a thing from the one that only mentions its parts.
One document may give at most two passages to a search. A common word's best passages tend to sit in a few long documents, and without the cap an act like the Working Environment Act fills every slot. To keep the cap from returning fewer hits than were asked for, the query first pulls twelve times as many rows, up to 400, and the cap is applied to those. When the agent searches inside documents it has named, the cap goes up to 40, because then a long document is exactly what it asked for.
Packs carry the vocabulary#
A collection with settings is called a pack. There are three kinds: documents someone uploaded, Norwegian law from Lovdata, and official statistics. A pack has its stemmer, a description, groups of synonyms and an optional dictionary. A pack that is switched off is not searched even if a question names it.
The synonyms widen every search quietly. A query containing any word of a group also searches the others, while the phrase bonus still reads the query as it was typed. The Lovdata pack has a group for "sparken", "oppsigelse" and "avskjed", so the everyday word finds the sections that use the formal ones. The dictionary is for the agent. Each entry has the everyday word, the words the documents use instead, the different things the word can mean, a few ready-made searches and the rule an answer about it usually turns on. The related_terms tool reads it, and it is only offered when a pack in scope has a dictionary, so the model never learns of a tool that can only return nothing.
When there is no dictionary entry, the documents name their own concepts. discover_terms runs the agent's query for 40 passages, up to six from each document, and counts the words that appear in at least two of them. Words the agent asked for, short words and stopwords are dropped. Each remaining word is scored by how much more common it is in the hits than in the whole collection:
The top twelve go back to the agent. A word that is everywhere in the hits and rare in the rest of the collection is probably the documents' own name for what the agent is looking for. Neither tool's output may be quoted. They only give the agent better words to search with.
A classifier makes the quick calls#
Some decisions come up on every question and are not worth a round with a large model. For those we use Jev, a small custom classifier. It answers multiple-choice questions in a few hundred milliseconds. Each answer is acted on only when Jev is sure enough, and when it isn't, or isn't reachable, the agent does what it would have done anyway.
How sure Jev must be before its call is used
Before the first model call, Jev decides whether a chat message stands on its own or is a follow-up to rewrite, which pack it is about from the packs' descriptions, and what the first move should be: search, find a named document, define a term, or say the question is out of scope.
Then the question is searched before the model sees it, as a whole and as up to three separate questions inside it. The results go in as a note in the conversation, not as tool calls the model never made. A made-up call carries none of the provider's own state, such as a Gemini thought signature, and a provider that checks for it rejects the whole conversation.
Every passage a search returns is read by Jev beside the question, twelve at a time, and labelled relevant, partial or irrelevant. A passage Jev is at least 90% sure is irrelevant is left out of what the model is shown, with a line saying so. Input tokens are most of a round's time, so this is where the classifier saves the most. If every hit looks irrelevant, the model sees them all, because a search that found nothing useful is worth knowing about too.
Sending less#
Three searches with different words often return the same passages. A passage that has been shown once is not sent again in that run, and a document's address and language go only with its first appearance. Every copy would otherwise be paid for again in each later round.
Two nudges keep the agent from stopping early. When it has found four or more documents and not opened some of them, it is told their titles while it is still searching, because the rule is often in one document and the exception in another. And before a "not settled" answer is served, the agent is shown once more the lines it already read that mention what the question asks about. A model that gives up has often read past the deciding sentence.
Statistics become documents#
When the statistics pack is switched on, the agent gets three more tools. find_statistics searches the tables of SSB, Eurostat, the OECD, the World Bank, the IMF, Norges Bank, the UN's SDG database, Our World in Data, the Offshore Directorate, NVE and the company register. describe_statistics lists a table's dimensions and their codes. get_statistics fetches only the cells the agent asked for.
What it fetches is stored as a document in the statistics collection, one line per figure, like "Bergen · 2024: 5821". From then on it is a document like any other. A figure in an answer is quoted from that line and checked word for word against it, the same way a sentence from the law is. Each question fetches its own tables with its own filters, so an old fetch for a different question can't answer this one. If the agent keeps searching and describing tables without fetching a figure, it is told that a title is not a number.
What it adds up to#
The search engine is the boring part, on purpose. Postgres does the matching and the ranking. The Go around it decides what goes into a query, how many passages one document may give, and what the model doesn't need to read twice. The classifier only takes things away, and only when it is sure. Every claim in the answer is still checked by a program against the full text of the source it names.
We still haven't run the old benchmark against this system, so there is no score here yet. When we do, it will be the next piece.