Skip to content

What we have learned so far

The research’s first measurements — and six things we learned before it began

Round 1’s models and its benchmark were released openly on 12 November 2026: how to use them comes first, just below. The rest of the first half of this page is the research’s results, the most recent first. From its eighth week, her larger model taught to talk — how natural it sounds, read blind by AI readers, how it answers from records, worked maths examples, reading much longer texts — and why the work on natural conversation is in English only. From its seventh week, every model the research has trained, run itself or read so far on one benchmark, with no model named, and the conversations she learned from, read against a quality bar. From its sixth week, her larger model, trained, set beside her main model on the same tests, and then taught to read further back in a text. From its fourth week, her main run, finished: what training measured, and what it has not measured yet — with her first evaluation as a base model and a test of adding code and maths to her reading. From its third, what she would read — how much pet care, how much Romanian — and how long that main run would be. From its second, how Gaia’s vocabulary was chosen, with the measurements behind the choice, and what the same measure shows across the languages people in the UK live in. The second half is older: patterns from building and running an assistant inside pet-care software used by licensed businesses, where a confident wrong answer reaches somebody with an animal in front of them. Several were arrived at after a first attempt that measured worse, and they hold whatever the research concludes.

Released openly · 12 November 2026

Use Gaia: Round 1’s models and its benchmark

On 12 November 2026 Round 1 was released openly on Hugging Face, by ProjectGaiaUK (opens in a new tab): her chat model and the base model it was made from, under Apache-2.0, each with its model card and the data card, and the benchmark they were measured on, with every system’s scores. Anybody may download them, run them, test them and build on them. Read each card’s limits first.

  • The chat model

    gaia-3b-chat

    A three-billion-parameter English and Romanian conversational model, trained from nothing by Project Gaia. It answers from numbered records placed in its prompt, cites them, and says — in English and in Romanian — when they do not answer. Round 1, version 1.0, chosen on 12 November 2026.

    For: Research on answering from records and saying when they do not answer; building on with your own records and checks.

    Its limit: A research model, not safe to answer pet-care questions unguarded: run it behind a retrieval guard and a veterinary gate, never as a veterinary adviser.

    Licence: Apache-2.0

  • The base model

    gaia-3b-base-128k

    The three-billion-parameter base model the chat model was made from: trained from nothing on openly licensed English, Romanian, pet-care, code and maths text, then taught to read up to 131,072 tokens at a time.

    For: Reproducing the chat model’s fine-tunes, and making other fine-tunes of your own.

    Its limit: It continues text: it was never taught to converse, follow instructions or stop, and carries no safety behaviour.

    Licence: Apache-2.0

  • The benchmark

    gaia-records-bench

    Gaia Records Benchmark, v0.1 — pet care and animal law, English–Romanian

    2,307 held-out items in three sets, English and Romanian, each answered from numbered records: does the answer cite the right record, carry its cautions and its vet referral, and say when the records do not answer? With a scorer that needs no GPU, the judged pass’s prompt for an openly licensed AI model, and every system’s scores.

    For: Testing any model, your own included, on answering from records and on saying when they do not answer.

    Its limit: It measures answers against the records only — not whether advice is right beyond them — and its Romanian set is small and legal in register.

    Licence: items under their sources’ licences (Open Government Licence v3.0 for the English, CC BY 4.0 for the Romanian); code Apache-2.0

Both models are published as safetensors in bfloat16, ready for the usual open-source tools; there is no GGUF file yet — one follows once the conversion tool reads this vocabulary. The data card, naming every dataset and its licence, ships with both models.

The chat model was chosen on measurements made by machines, an openly licensed AI judge and AI readers — never confirmed by people. No person has read its answers.

How to cite

The models: gaia-3b-chat and gaia-3b-base-128k, Project Gaia Round 1, version 1.0, released 12 November 2026, Apache-2.0. Project Gaia, MyPetSitter Ltd. https://huggingface.co/ProjectGaiaUK (opens in a new tab)

The benchmark: Gaia Records Benchmark: pet care and animal law, English–Romanian, version 0.1.0, released 12 November 2026. Items under the Open Government Licence v3.0 (English) and CC BY 4.0 (Romanian); code Apache-2.0. Project Gaia, MyPetSitter Ltd. https://huggingface.co/datasets/ProjectGaiaUK/gaia-records-bench (opens in a new tab)

The authors and BibTeX for both are on the model cards, the organisation’s page and the benchmark’s CITATION.cff (opens in a new tab).

The computing time: each card carries the award’s acknowledgement, as at the end of the research below — UKRI AI Research Resource (AIRR) Gateway · project 0261-1548-0489-1 · 4,000 GPU hours of computing time. The award supports the research. It is not an endorsement of any product. Work that uses them should cite McIntosh-Smith, S., Alam S. R. and Woods, C. (2024). “Isambard-AI: a leadership class supercomputer optimised specifically for Artificial Intelligence”. https://doi.org/10.48550/arXiv.2410.11199 (opens in a new tab)

From the research · Round 1, week eight

Taught to talk: natural-sounding, partly coherent, and still weak at reasoning

Her larger model was taught to hold a conversation, in four rounds, and then given worked maths examples. AI readers read its replies blind, beside those of an openly licensed model of four billion parameters, and picked the one that sounded most like a real person. In plain words, three things came out. It sounds natural: in single replies the readers picked it a little more often than that open model, in English and in Romanian, and over several turns far more often. It is only partly coherent: over several turns it holds together less well than the open model — about 3.2 against 3.8 on a scale of 1 to 5. And it is weak at reasoning: it gets about one grade-school maths word problem in 11 right. Every reading of how it sounds was made by AI readers, not by people; nothing here says how safe or how right its pet-care answers are, and the project uses it to answer nobody. On 11 November a blend of that model with a round that raised its Romanian became her chat model. It scored level with the model it was made from on maths, checked by a script, and AI readers found it level in English and over several turns and ahead in Romanian openings on a first reading, less clearly on a second. Later that day a lesson in saying when the records do not answer was added to the blend, again with no further training, and that became her chat model: on the first evaluation pass it said so 21 times in 28, where the blend never did. The blend is kept as it was, to go back to. The next morning the same lesson, taught again with Romanian added, took its place: her chat model now says so in English and in Romanian, in the language of the question (R24).

Measured on Isambard-AI and on our own machines from 8 to 11 November 2026: her larger model taught to talk, given maths worked examples, blended with a round that raised its Romanian, then given a lesson in saying when the records do not answer. Every reading of how she sounds was made by AI readers, not yet by people. Nothing here says how safe or how right her pet-care answers are, and she answers nobody.

  • 5,066of our own conversations in what her chat model learned fromamong 107,739; each set of ours read by AI readers against the bar before it was used
  • 9.3%of grade-school maths word problems right, after the maths worked examples3.9% before them, of 1,319; her chat model now 7.7%, the model it replaced 7.6%, the blend 9.0%
  • 0.935the share of hidden numbers found in a 131,072-token text200 fresh tries, her chat model of 10 November
  • 3,485.5GPU hours used of the 4,000514.5 left on 11 November, 130 of them a reserve kept back

Taught to talk

After its first round, AI readers picked her reply as the one most like a person’s on 80% of openings — the 4B open model on 18%, her base model on 0%

The first round learned from 102,673 conversations from openly licensed sets, most of them written by people, and our own answer set — none of the conversations we generated. Then our own generated conversations began to pass the bar: four sets of them, each read by two AI readers (70 of 80, 72 of 80, 68 of 80 and 70 of 80 read good), joined the next three rounds, until 5,066 of the 107,739 conversations were ours. Over every batch read, the recipe that made most of them reads 83% good (332 of 400, five batches): a set passes partly by the luck of its sample, so the pooled figure is the recipe’s own.

A preference step — learning which of two replies people preferred, from 10,253 pairs that people ranked — ranked held-out pairs the people’s way only 59.3% and 59.7% of the time, barely better than a coin. Round 4, without it, was chosen on 8 November: the AI readers preferred it in English openings, the one place its interval cleared the others’ (only just), and leaned the same way over several turns; in Romanian the three were level.

The first round: which opening reply sounds most like a real person?Picked by four AI readers, blind, on 60 held-out openings: the share of the openings each was picked on, ties aside. Read on 8 November.
  • Her larger model, first round — English (37)81%
  • Her larger model, first round — Romanian (23)78%
  • The 4B open model — English (37)16%
  • The 4B open model — Romanian (23)22%
  • Her base model, never taught to talk — English (37)0%
  • Her base model, never taught to talk — Romanian (23)0%
The choice: round 4, and the two rounds with a preference stepPicked by eight AI readers, blind, on 8 November; the line across each bar is its 95% interval. Round 4 was chosen.
  • English openings (37) · Round 446%, 95% interval 30 to 62%
  • English openings (37) · Round 3 with the preference step16%, 95% interval 5 to 30%
  • English openings (37) · Round 4 with the preference step16%, 95% interval 5 to 30%
  • Romanian openings (100) · Round 433%, 95% interval 24 to 42%
  • Romanian openings (100) · Round 3 with the preference step33%, 95% interval 24 to 42%
  • Romanian openings (100) · Round 4 with the preference step32%, 95% interval 24 to 41%
  • Several turns (50) · Round 442%, 95% interval 28 to 56%
  • Several turns (50) · Round 3 with the preference step24%, 95% interval 12 to 36%
  • Several turns (50) · Round 4 with the preference step24%, 95% interval 14 to 36%

How we know

AI readers — fresh instances of an AI model, each given only the brief and its own items — saw the replies with the systems hidden and the order shuffled, and picked the one that sounded most like a real person talking, or called a tie. Long replies were cut short for the reading. Every figure here is an AI model’s judgement; no person has read these replies yet. The same family of AI model set the bar our own conversations passed, so part of round 4’s lead may be the reader’s taste.

Answering from records

Given the right record, her chat model of 10 November cites it 30 times in 31; when no record answers the question, it never says so — 0 times in 28

This was the first evaluation pass on held-out pet-care questions. Given the record that answers, her chat model of 10 November cites it, and carries its caution and its vet referral nearly every time. But when none of the records in front of it answers the question, it never says so: it answers from a neighbouring record in a confident voice. Read in full by an AI reader, her round-4 model once turned guidance about horses into guidance about a cat. The 4B open model, as released, said the records did not answer 14 times in 28.

So she must not answer pet-care questions on her own: something outside the model has to decide first that a record holds the answer. The blend that became her chat model on 11 November (below) went through this pass the same day and never said so either — 0 times in 28. The lesson added to it since says so most of the time, not every time (R23, below).

Answering pet-care questions from the records it is givenHer chat model of 10 November on 90 held-out items from 31 questions, all in English, scored by a script on the words of each answer. The last bar is the 4B open model as released, on the same items.
  • Cites the right record placed in front of it (of 31)30 of 31
  • Carries the record’s caution (of 14)13 of 14
  • Carries the record’s vet referral (of 12)12 of 12
  • Says that no record answers, when none does (of 28)0 of 28
  • The 4B open model, as released: says that no record answers (of 28)14 of 28

How we know

A script scored every answer on its words: whether it names the right record, carries the record’s caution and vet referral, and says that the records do not answer when they do not. An AI reader then read in full the 28 answers of her round-4 model where no record held the answer. The questions are the titles of the records’ own sections, and whether the advice is right is not measured. No person has read these answers.

Maths worked examples

Worked maths examples, written by an openly licensed model and checked by running them, raised the grade-school maths she gets right from 3.9% to 9.3% — still weak

A worked example is a problem, the steps to its answer, and the answer. An openly licensed model wrote them — maths and code only, never pet care — and every one was checked by running it: a maths answer had to agree with a program written without seeing it, and a piece of code had to pass its own tests and a second set written without seeing the code. Of 20,400 planned, 10,800 were kept (6,343 maths, 4,457 code); AI readers found 150 of 160 maths and 146 of 160 code examples fit to teach from.

Taught on all of them, she got more maths right and less code: the code examples taught her to write a numbered plan before the code — 493 of 500 answers in one programming test opened with “1.” — and at this size the plan is often wrong. Taught on the maths alone, she kept the maths gain and lost no code, and that model became her chat model on 10 November, until a blend of it took its place the next day (below). That model had two known costs: it quotes the record word for word a little less often (19 times in 31, against 23), and its Romanian fell. Its maths lessons had replayed only 1,377 Romanian conversations, where round 4 had learned from 14,363 (below).

Grade-school maths word problems: the share right1,319 problems from a published test set, one answer each, checked by a script; the line across each bar is its 95% interval. Measured on 10 November.
  • Before the worked examples (round 4)3.9%, 95% interval 3.0 to 5.1%
  • All the worked examples, twice through8.9%, 95% interval 7.4 to 10.5%
  • All the worked examples, once through7.0%, 95% interval 5.8 to 8.6%
  • The maths examples only — chosen 10 November9.3%, 95% interval 7.9 to 11.0%
Short programming problems: the share right164 problems from a published test set; each answer’s code was run against the problem’s own tests. The line across each bar is its 95% interval.
  • Before the worked examples (round 4)10.4%, 95% interval 6.6 to 16.0%
  • All the worked examples, twice through7.3%, 95% interval 4.2 to 12.3%
  • All the worked examples, once through7.9%, 95% interval 4.7 to 13.1%
  • The maths examples only — chosen 10 November11.6%, 95% interval 7.5 to 17.4%

How we know

Every answer was checked by running it, not by reading it: one answer per problem, the most likely each time. The published test sets were kept out of the worked examples — anything near one of their problems was dropped, and none of the prompts that wrote them came near one. The judgement that the examples were fit to teach from is AI readers’, not people’s.

How she sounds, read blind

Her replies sound more like a person’s than the 4B open model’s, in English and in Romanian; over several turns she holds together less well — 3.17 of 5 against its 3.84

Natural-sounding: in a single reply the AI readers picked her as the most like a person a little more often than the 4B open model, in both languages — though inside the intervals — and over several turns far more often (38% against 14%). Partly coherent: over several turns the open model holds together best, clear of both of hers — and is picked least often as sounding like a person. Her chat model of 10 November and her round-4 model are level in English openings and over several turns: their intervals overlap, and a second reading of the same conversations, with another of her models as the third, put them the other way round. In Romanian openings her round-4 model was picked 10 points more often: the measured cost of the maths lessons.

Opening replies: which sounds most like a real person?Every opening of the held-out set, 100 in English and 100 in Romanian, picked by ten AI readers, blind, on 11 November; the line across each bar is its 95% interval.
  • English openings (100) · Her chat model of 10 November32%, 95% interval 23 to 41%
  • English openings (100) · Her round-4 model35%, 95% interval 25 to 44%
  • English openings (100) · The 4B open model28%, 95% interval 19 to 37%
  • Romanian openings (100) · Her chat model of 10 November32%, 95% interval 24 to 42%
  • Romanian openings (100) · Her round-4 model42%, 95% interval 32 to 52%
  • Romanian openings (100) · The 4B open model26%, 95% interval 18 to 35%
Several turns: which sounds most like a real person?100 conversations of several turns, in English only — no Romanian test conversation of several turns is left. Picked by five AI readers, blind.
  • Her chat model of 10 November38%, 95% interval 29 to 47%
  • Her round-4 model47%, 95% interval 37 to 57%
  • The 4B open model14%, 95% interval 8 to 21%
Several turns: how well each holds together, from 1 to 5The same 100 conversations, each system’s replies scored from 1 to 5 by the same AI readers. The axis starts at 1, the bottom of the scale, not zero.
  • Her chat model of 10 November3.17, 95% interval 2.95 to 3.38
  • Her round-4 model3.13, 95% interval 2.91 to 3.35
  • The 4B open model3.84, 95% interval 3.70 to 3.98

How we know

AI readers — fresh instances of an AI model, each given only the brief and its own items, never which reply came from which model — read every item. Readings of only twenty conversations of several turns had pointed the other way and reversed when a hundred were read, so only the larger readings are shown. Every figure is an AI model’s judgement; no person has read these replies yet.

Reading further still

She reads 131,072 tokens at a time, and the lessons in talking kept it: her chat model of 10 November found a number hidden in such a text 93.5% of the time

Two more stages, begun on 4 November 2026, taught her larger model to read further back: texts of 32,768 tokens, then 131,072. After the first, a number hidden in a 131,072-token text was found only when it sat at the very end; after the second, almost anywhere but there — where her base model carries on writing the text instead of answering. Loss on short held-back text hardly moved (English 2.175 to 2.178, Romanian 2.347 to 2.352).

Every model taught to talk kept it, and found the number more often than the base model — partly because the lessons taught them to answer a question at all. Their loss on plain text rose a little with each lesson (English, at 2,048 tokens: 2.18 for the base, 2.24 after round 4, 2.32 for her chat model of 10 November): they now talk like an assistant rather than carry on a text. The blend that replaced it kept it too: on the first, smaller set of hidden numbers it found 0.90, where the model it was made from found 0.89. The blend with a lesson added (R23, below) found 0.84 on that set; on new hidden numbers her chat model now finds fewer than the blend, a real cost (R24).

A number hidden in a 131,072-token text: the share found20 tries in each place, after the first of the two stages (texts of 32,768 tokens) and after the second (131,072). Measured on 6 November.
  • After the first stage
  • After the second
  • At the startafter the first stage 0.00, after the second 0.90
  • 25% of the way inafter the first stage 0.00, after the second 1.00
  • 50% of the way inafter the first stage 0.00, after the second 1.00
  • 75% of the way inafter the first stage 0.00, after the second 0.85
  • At the very endafter the first stage 0.80, after the second 0.20
Every model taught to talk, on new hidden numbers in 131,072-token texts40 new hidden numbers at each of five places, the same ones for every model, measured on 11 November. Beside each: the share found at the very end.
  • Her base model, after the longer reading0.840, at the very end, 0.275
  • First conversation round0.950, at the very end, 0.800
  • Round 40.915, at the very end, 0.700
  • Her chat model of 10 November0.935, at the very end, 0.775
  • All worked examples, once through0.935, at the very end, 0.775

How we know

Counted by a script, no reader: a number is hidden at the start, a quarter, half and three quarters of the way in, and at the very end of a long text, and the model is asked for it. Her base model was measured with one number format inside the model and her chat models with another; the base, her round-4 model and her chat model of 10 November were measured again the other way, and nothing moved by more than 0.001 in loss, or by more than one number in a hundred at this length. A hidden number is a test of finding, not of understanding.

Romanian

Two Romanian rounds traded English for Romanian and were not kept; a blend of the second with the model they started from was — and the work on natural conversation is in English only

We stopped making more Romanian conversations because the cost was too high for this project. The open models we tried could not write natural Romanian to our standard, and reaching it would have needed native speakers to write or correct conversations at scale. So the work on natural conversation is in English only. Gaia’s Romanian comes from its main training and from existing open conversation sets, and its Romanian results are reported separately.

The maths lessons of her chat model of 10 November replayed only 1,377 Romanian conversations, where round 4 had learned from 14,363. Two rounds gave it every Romanian conversation of round 4’s set again — none of them new, and nothing made for these rounds — with English and maths beside them. Each raised Romanian (26% to 37% and 26% to 41%) and lowered English openings and longer conversations; maths held. A round was to be kept only if its Romanian gain cleared the intervals and nothing else fell, so neither was kept.

Then her chat model of 10 November was blended with the second round — 70% of one and 30% of the other, with no further training. Against the model it was made from, on the same Romanian openings and read by AI readers, the blend came out +18 points (95% interval +3 to +33) on a first reading (41% against 23%, intervals apart); on a second, +9 points (95% interval −7 to +25); together, +13.5 points (95% interval −1.5 to +28.5) — an interval that includes no change. English openings (16% against 22%), several turns (27% against 31%) and maths (9.0% against 9.3%) stayed inside their intervals. By the same rule, read on the first reading, the blend was kept: on 11 November it became her chat model, with the smaller second reading recorded beside it, and the model it was made from is kept as it was. Later that day a lesson in saying when the records do not answer was added to it and took its place (below); the blend is kept as it was, to go back to.

Picked as sounding most like a person: her chat model of 10 November, and the same model after each Romanian roundEach round read three-way with her chat model of 10 November and her round-4 model by 15 AI readers, blind, on 100 items of each kind, on 11 November. The ring is that model in that reading; the dot is the Romanian round.
  • Her chat model of 10 November
  • After the Romanian round
  • Romanian openings — the first Romanian roundher chat model of 10 November 26%, after the round 37%
  • English openings — the first Romanian roundher chat model of 10 November 27%, after the round 18%
  • Several turns (English) — the first Romanian roundher chat model of 10 November 40%, after the round 26%
  • Romanian openings — the second Romanian roundher chat model of 10 November 26%, after the round 41%
  • English openings — the second Romanian roundher chat model of 10 November 28%, after the round 20%
  • Several turns (English) — the second Romanian roundher chat model of 10 November 42%, after the round 25%

How we know

The Romanian results this week are the openings: every conversation of several turns, and every pet-care question above, is in English only. Every reading was made by AI readers, not by people.

Saying when the records do not answer

A lesson added to the blend made it say when the records in front of it do not answer — 21 times in 28 on the first pass, where the blend never did

The blend that became her chat model on 11 November went through the first evaluation pass the same day and, like the models before it, never said when the records in front of it did not answer — 0 times in 28. A copy of her round-4 model had already been taught to say so, in one fixed sentence written by us; the questions of this pass were kept out of what it learned from. The change that lesson made to its weights was added to the blend, with no further training, and the result was measured exactly as the blend had been, against it.

It now says so 21 times in 28, and 76.2% of the time over three more runs. On every measure with an interval it stayed within chance of the blend. The records it cites, and their cautions and vet referrals, are within one item of the blend’s. Grade-school maths came out 7.6% against the blend’s 9.0%, and code a little lower too, all inside the intervals. To the AI readers it was +4 points in English openings, +7 over several turns and −10 points (95% interval −26 to +6) in Romanian openings, the weakest — all inside their intervals. In ordinary chat with no records, 627 conversations of the chat test, it stopped cleanly 97.6% of the time against 97.0%, repeated itself less, and said the records do not answer only twice, both in long documents. So, by the rule set before it was measured, it became her chat model the same day, until R24’s model took its place the next morning. The blend is kept as it was, and going back to it is one switch.

It has a known cost. Its loss on plain held-back text rose by about a tenth (English 2.307 to 2.414, pet-care pages 1.615 to 1.765): it predicts ordinary text a little less well, as each lesson in talking has done (R21). That will be written on its model card.

On a number hidden in a 131,072-token text it found 0.84 where the blend found 0.90, with 20 hidden numbers in each place. The check on new hidden numbers has since been made: on the same 200 it found 170 where the blend found 189 (a machine count) — a real cost, not chance. And on seven of the 28 it did not say so: she must still not answer pet-care questions on her own, and the project uses her to answer nobody.

The first pass is small. The evaluation that is published has since run on a larger held-out set: 641 questions and 1,904 items, made from 66 pages of UK government guidance under the Open Government Licence, all in English, with a Romanian held-out set beside it. A script scores the answers, and an openly licensed AI model then judges them by their meaning — an AI model’s judgement, never a person’s. What it found on saying so is the next finding.

When none of the records answers the question: does it say so?The first evaluation pass’s 28 held-out items where no record holds the answer, all in English, scored by a script on the words of each answer, on 11 November. The last bar is the 4B open model as released, on the same items (R18).
  • The blend with the lesson added (of 28)21 of 28
  • The blend it was made from (of 28)0 of 28
  • The 4B open model, as released (of 28)14 of 28
Picked as sounding most like a person: the blend, and the blend after the lessonRead three-way with the blend and her round-4 model by 15 AI readers, blind, on 100 items of each kind, on 11 November. The ring is the blend; the dot is the blend with the lesson added.
  • The blend it was made from
  • The blend with the lesson added
  • Romanian openings (100)the blend 39%, with the lesson 29%
  • English openings (100)the blend 27%, with the lesson 31%
  • Several turns (English) (100)the blend 34%, with the lesson 41%

How we know

A script scored every answer on its words, as in R18; code was checked by running it and maths by its final number, both by a script, on the same problems for both models. AI readers — 15 fresh instances of an AI model, each given only the brief and its own part, never which reply came from which model — read the replies. Every figure is a machine count or an AI model’s judgement; no person has read these replies yet, and nothing here says how safe or how right its pet-care answers are.

Saying so in Romanian too

Her chat model now says when the records do not answer in Romanian too, and in Romanian — on 88% of the Romanian questions no record answers, where the model it replaced said so on 24%, and only in English

The model of R23 said so in English. On Romanian questions where no record held the answer it said so far less often, and in English when it did; and on 29 of 104 Romanian items it answered as if it were the person asking. So the lesson was taught again with 326 Romanian rows added, built from 43 openly licensed pages registered before they were fetched and from our answer set’s Romanian examples, with one fixed Romanian sentence beside the English one, both written by us. Its change was added to the blend exactly as before, with no further training, and measured against the model it would replace, on held-out questions.

On the Romanian held-out set it now says so on 88% of the questions no record answers, every time in Romanian; the model it replaced said so on 24%, every time in English, and the blend never did. It no longer answers a Romanian question as if it were the person asking: 0 times in 104, where the model it replaced did so 29 times. In English it says so on 84% of the larger set’s questions no record answers, against 83% — unchanged, +1.1 points (95% interval −1.1 to +3.4). An openly licensed AI model judged 88% of its English answers correct where a record did answer, against 81% for the model it replaced — an AI model’s judgement, never a person’s. By the rule set before it was measured, it became her chat model on 12 November; the model it replaced is kept as it was, and going back to it is one switch.

On the long-context check that hides a number deep in a text of about 100,000 words, the chosen model finds it 179 times in 200, where the model before the records-decline work finds it 189 times in 200, on the same 200 hidden numbers (a machine count). That is a real cost of teaching it to say when its records do not answer — in English and Romanian — and it is stated on the model card.

The model it replaced found 170 of the same 200: this one gives back about half of that loss, and the rest is real. It still does not say so on every question no record answers, and it sometimes says so where a record does: she must still not answer pet-care questions on her own, and the project uses her to answer nobody.

When none of the records answers the question: does it say so?The held-out items where no record holds the answer — 99 Romanian, from openly licensed pages registered before they were fetched, and 622 English, from the larger set — scored by a script on the words of each answer, on 12 November.
  • Romanian (99) · Her chat model now, in Romanian88%
  • Romanian (99) · The model it replaced (R23), in English24%
  • Romanian (99) · The blend, before the lesson0%
  • English (622) · Her chat model now84%
  • English (622) · The model it replaced (R23)83%
  • English (622) · The blend, before the lesson2%
A number hidden in a 131,072-token text: found, of the same 200The same 200 new hidden numbers for every model, at five depths; each one found or not, counted by a script, on 12 November.
  • Her chat model now179 of 200
  • The model it replaced (R23)170 of 200
  • The blend, before the lesson189 of 200

How we know

A script scored every answer on its words, and counted every hidden number found. The Romanian held-out set comes from openly licensed pages registered before they were fetched, with the questions near anything a model learned from taken out. An openly licensed AI model judged the answers by their meaning — an AI model’s judgement, never a person’s. No person has read these replies yet, and nothing here says how safe or how right its pet-care answers are.

Cheaper per answer

Cheaper per answer, with a trade

On one modern GPU the chosen model answers about 74 benchmark questions a second, about £0.01 per 1,000 answers at the award’s own valuation of a GPU hour. A commercial model called through its API on the same questions costs about £1.94 per 1,000 at its list prices: about 180 times more at full load, and a rented GPU pays for itself only above about 1,050 answers an hour.

On the same questions the chosen model says when its records do not answer far more often (0.84 against 0.58 in English, 0.88 against 0.65 in Romanian). The commercial model is more often right where the records do answer (0.93 against 0.88, judged by an AI model) and mixes up species less often.

How we know

Machine counts and an AI judge’s verdicts; no person read these.

What comes next

The first evaluation pass ran on all four systems, the fourth — a large hosted model — last. What it found was worked on: her chat model now says when no record answers, most of the time, in English and in Romanian, without losing its conversation (above). The evaluation that is published ran on the larger held-out set: four systems answered the same questions, compared on accuracy, knowing when to refer to a vet, and cost. On 12 November 2026 its items, its scorer and every system’s scores were released openly with her models (Use Gaia, at the top of this page). Next comes the white paper, revised with the results. She is a research model: she must still not answer pet-care questions on her own.

From the research · Round 1, week seven

Every model the research trained, ran or read, measured — and conversations read against a bar

Before her larger model was taught to hold a conversation from conversations we made ourselves, they were made again, quality first, and read one by one against a bar — by an AI model, not by a person. This week also puts every model the research has trained, run itself or read so far on one benchmark: Gaia’s own, the openly licensed models that wrote and checked the conversations, and the ones we only read. Not on it: the hosted models that carry the conversation in the research workspace — phrasing her next question, planning the interview and judging its replies — which the older half of this page is about. No model is named here. Each is described by what it is, its size and its licence, with every measurement, how many it rests on, and the day it was taken.

Measured on Isambard-AI and on our own machines, and read against a quality bar, up to 7 November 2026: the conversations we make for her larger model and the open sets it learns from, and every model the research has trained, run itself or read. By then her main model had had one fine-tune, on answering from records, and none had been taught everyday conversation — nothing here says how well she converses.

  • 26entries on the benchmarkten for Gaia’s own models, every other model unnamed
  • 80%the best English read for the gate32 of 40 conversations, read by an AI model; the bar is 85%
  • 47.5%the best Romanian read for the gate19 of 40; on 7 November no source had reached 85%
  • 85%the bar our own conversations must reach, in English and in Romanianread by an AI reader on a sample of each set we make; a set that falls short is made again

Conversations, read against the bar

In English, 86% read good at the strict check; in Romanian no model has reached 85% yet — so none of our own Romanian conversations is used until one does

These are everyday conversations between a person and Gaia — the weather forecast, calling a plumber, a trip to the mountains — with every one that touches animals or pet care taken out. They teach how people talk, never what to say about pet care. Each was written by an openly licensed model, run by us, then checked by machine before an AI model read it.

In English, open model A writes warm, short and natural conversation: read at the strictest check, 43 of 50 were good. In Romanian the slips are small — a verb form, an agreement, a missing accent — and they are exactly what a native speaker hears at once. The best any model managed on its own, even at the strictest check, was 66%. Open model B writes the most natural Romanian of all and keeps the most, 60% read good; an automatic correction fixed 6 of its conversations and damaged 6 good ones.

So none of our own Romanian conversations is used yet. In both languages an AI reader reads a sample of every set we make and a strict automatic judge checks every conversation; a set of ours is used only when its sample reads 85% good or better, and one that falls short is made again with another openly licensed model.

Read good by an AI model — the share of conversationsEach bar is a reading by an AI model, not by a person, of conversations that had already passed every automatic check, read on or before 6 November. The gold line is the bar every set must reach before it is used: 85%.
  • English · Open model A, round 180%, 32 of 40 read good
  • English · Open model A, round 2, the strict cut86%, 43 of 50 read good
  • Romanian · Open model A, round 141%, 13 of 32 read good
  • Romanian · Open model A, round 245%, 18 of 40 read good
  • Romanian · Open model A, round 2, steadier sampling61%, 19 of 31 read good
  • Romanian · Open model D, pilot≈19%, about 4 of 21 read good
  • Romanian · Open model B, pilot60%, 24 of 40 read good
  • Romanian · Open model B, pilot, the strict cut66%, 19 of 29 read good
  • Romanian · Open model B, pilot, then corrected automatically60%, 24 of 40 read good

How we know

An AI model read each set, one conversation at a time, after the automatic checks: the topic screen, the rules, and a second model that must quote the exact words at fault. No person has read these sets yet. Each bar rests on 21, 29, 31, 32, 40 and 50 conversations — enough to show a direction, not to rank two models closely.

Conversations, read for the gate

On 7 November, the best English read for the gate was 80% read good, the best Romanian 47.5% — that day no source had reached 85%, so none of our own conversations was used yet

On 7 November the kept conversations of each source were read for the gate: 40 a language, each already through the topic screen, every rule, the model’s own check and the strict automatic judge. A source passes only when 85% or more of them read good.

In English, open model I wrote the best: 75–80% read good across its newer settings, within each other’s sampling error and below the bar. Open model A, whose first pilots read well, kept 14% of its first large English batch, and 42.5% of those read good: a pilot is not a batch. Its own first large batch, made with its best setting, read 67.5% good across two readers (28 and 26 of 40); with its pilot, 86 of 120. In Romanian the best was 47.5%, from open model I, thinking before each reply.

The models’ own checks did not separate good conversations from poor ones. Of what the open models’ Romanian checks passed, 56–58% read good, against 52% of all 143 read. A third model asked for evidence lifted the weakest sets — English 42.5% to 52%, Romanian 35% to 50% — but lowered open model I’s best English, 80% to 77%; open model I judging its own set gave no lift (77.5% to 76%).

So that day no source of ours had passed, and none of our own conversations was used. A source that fell short was made again — and the next day four English sources of ours passed (week eight, above). No Romanian source ever did.

Read good by an AI model, for the gate — each source’s sampleEach bar is a reading by an AI model, not by a person, of a seeded sample of one source’s kept conversations, read on 7 November. A source passes only at 85% or more of at least 40 read. A bar “after a third judge” counts only what that judge also passed, so it rests on fewer than 40.
  • English · Open model A, first large English batch42.5%, 17 of 40 read good
  • English · Open model I, 3b prompts65%, 26 of 40 read good
  • English · Open model I, 3b prompts, after a third judge71%, 22 of 31 read good
  • English · Open model I, thinking before each reply80%, 32 of 40 read good
  • English · Open model I, 3c prompts77.5%, 31 of 40 read good
  • English · Open model I, thinking and 3c prompts75%, 30 of 40 read good
  • English · Open model I, English wave 2, two readers67.5%, 54 of 80 read good
  • Romanian · Open model B, a steadier temperature35%, 14 of 40 read good
  • Romanian · Open model I, 3b prompts25%, 10 of 40 read good
  • Romanian · Open model I, 3b prompts, after a third judge35%, 8 of 23 read good
  • Romanian · Open model I, thinking before each reply47.5%, 19 of 40 read good

How we know

An AI model read each sample, one conversation at a time, after the automatic checks: the topic screen, the rules, the model’s own check and a second model that must quote the exact words at fault. No person has read these samples. Each bar rests on 23, 31, 40 and 80 conversations — enough to show a direction, not to rank two settings closely.

What her first conversation round learned from

102,673 conversations: open sets and our own answer set — none of our own generated conversations, because none had passed the gate when it was built

Most of the set was written by people: 82,965 conversations from 12 openly licensed sets. 18,862 come from two openly licensed research sets written by a model — everyday conversation only, with every one that touches animals or pet care taken out — and 846 from our own answer set, written by people from open records.

After it, a preference step learned which of two replies people preferred, from 10,253 pairs that people ranked. None of our own generated conversations is in this set; they joined the later rounds as they passed (week eight, above).

The conversations by who wrote them — the share of the setEach bar is one kind of conversation in the set her larger model’s first conversation round learned from, as a share of its 102,673; the number beside it is how many conversations.
  • People-written open conversation sets82,965, 82,965 conversations from 12 sets
  • Open model-written research sets, pet care screened out18,862, 18,862 conversations from two sets
  • Our answer set, written by people from open records846, 846 conversations from one set

How we know

Every set was checked for licence, toxicity, duplicates and overlap with the tests before it was used. 88,309 conversations are in English and 14,364 in Romanian.

The benchmark: every model the research trained, ran or read, and what it knows

26 entries in five groups. Every model outside Gaia’s own is given a letter or a number instead of its name; the comparison is about what each did on our work, not about who made it. Many figures rest on a few dozen readings: they show a direction, not a ranking.

Gaia’s own models

10 on the benchmark

Trained from nothing with our own code, data and vocabulary: base models, the main model’s first fine-tune on answering from records, and her larger model taught to talk. Every reading of how they talk was made by AI readers, never yet by people.

Her larger model

Gaia’s own, about three billion parameters, trained from nothing

Licence: ours, released openly on 12 November 2026 under Apache-2.0

What it knows

A base model: it predicts English, Romanian, pet-care pages, code and maths better than her main model, and does far better on simple arithmetic word problems. It now reads 131,072 tokens at a time and finds a number planted in them most of the time, missing mostly at the very end, where it continues the text instead of answering. As a base model it answers nobody; her chat models, the next five cards, were taught from it.

Measured

Bits per byte on held-back English · Romanian (lower is better)
0.7082 · 0.8094
n 4,514,099 and 4,785,192 tokens · 29 Oct 2026
Bits per byte on held-back pet care (partly seen in training)
0.4760 — about 58% of those tokens fell inside its reading
n 7,999,999 tokens · 29 Oct 2026
Reasoning check, four options (chance 25%)
71.6% right (501)
n 700 problems · 29 Oct 2026
6 more measurements
Picking the right record on its own (first place)
60.2% — word overlap alone: 69.3%
n 231 questions, 439 records · 29 Oct 2026
A number hidden in a 8,192-token text, share found
0.89 (at the very end: 0.45)
n 20 tries at each of 5 depths · 29 Oct 2026
GPU hours, main run
1706.9 on the machine’s record
29 Oct 2026
A number hidden in a 131,072-token text, share found (fresh keys)
0.84 (at the very end: 0.275)
n 40 tries at each of 5 depths · 11 Nov 2026
Held-back loss at 2,048 tokens after the 128k phase, English · Romanian · pet care
2.1784 · 2.352 · 1.434 (after the 32k phase 2.1753 · 2.3474 · 1.4361)
6 Nov 2026
How well it answers, how safely
Not measured — a base model; its chat models are measured below

Good at

Predicting the text it was built for; arithmetic word problems; long texts.

Not for

Answering anybody; picking the right record on its own.

Her chat model (the one chosen)

Gaia’s own: her blend with a Romanian round, with a lesson in saying when the records do not answer — in English and in Romanian — added, and no further training

Licence: ours, released openly on 12 November 2026 under Apache-2.0

What it knows

Made on 12 November from her blend with a Romanian round, exactly as the model before it was: the change that teaching a copy of her round-4 chat model to say when the records do not answer had made to its weights was added to the blend, with no further training. This time the lesson’s data gained Romanian rows, built from openly licensed pages registered before they were fetched. It says so on most of the questions no record answers, in English and in Romanian, and in Romanian it says so in Romanian; it no longer answers a Romanian question as if it were the person asking. It finds fewer numbers hidden deep in a long text than the blend: a real cost, stated on its model card. No person has read it yet.

Measured

Answering from records: says when no record answers — English, the larger held-out set
0.838 — the model it replaced 0.826; on the same items +0.011 [−0.011, +0.034]
n 622 held-out items where no record answers, English; string-level · 12 Nov 2026
Answering from records in Romanian: says when no record answers, and in which language
0.879, every time in Romanian — the model it replaced 0.242, every time in English
n 99 held-out Romanian items where no record answers; string-level · 12 Nov 2026
Romanian: answers as if it were the person asking
0 — the model it replaced 29
n 104 Romanian items where no record answers · 12 Nov 2026
8 more measurements
Says the records do not answer where one does: English · Romanian
69 of 1,282 · 6 of 224 — the model it replaced 67 · 3
n 1,282 English and 224 Romanian answerable items; string-level · 12 Nov 2026
Answerable items judged correct · wrong by an open AI judge model: English · Romanian
0.875 · 0.032 and 0.935 · 0.028 — the model it replaced 0.814 · 0.051 and 0.850 · 0.065
n 1,282 English answerable items, and the answerable items of the clean Romanian set; majority of three judge runs · 12 Nov 2026
Answering from records, the first pass: says when no record answers
0.821 — the model it replaced 0.750
n 28 of 90 held-out items, from 31 questions, English; string-level · 12 Nov 2026
Maths word problems (GSM8K) · code (HumanEval · MBPP), greedy, one answer each
7.7% · 7.9% · 11.4% — the model it replaced: 7.6% · 10.4% · 12.4%; within their intervals
n 1,319, 164 and 500 problems · 12 Nov 2026
Romanian openings · English openings · several turns (English), picked as sounding most like a person, blind, by AI readers, against her blend with a Romanian round
+2 [−14, +18] · +14 [0, +29] · +7 [−9, +23] points on the same items
n 100 items of each kind · 12 Nov 2026
Coherence over several turns, 1–5, by AI readers (English only)
3.02 — her blend with a Romanian round 3.22
n 100 conversations · 12 Nov 2026
A number hidden in a 131,072-token text, found (the same new hidden numbers for every model)
179 of 200 — the model it replaced 170, her blend with a Romanian round 189; on the same numbers +9 [+4, +15] against the model it replaced and −10 [−17, −4] against the blend (exact paired tests, p 0.004 and 0.006): a real cost of saying when the records do not answer
n 200 new hidden numbers, 40 at each of 5 depths · 12 Nov 2026
How safe its pet-care answers are; how people read it
Not measured — no person has read it; released openly on 12 November 2026, and the project uses it to answer nobody

Good at

Everyday conversation; saying when the records in front of it do not answer, in English and in Romanian, most of the time; finding a fact in a long text.

Not for

Answering pet-care questions on its own; arithmetic.

Her first model taught to say so

Gaia’s own: her blend with a Romanian round, with a lesson in saying when the records do not answer added, and no further training — her chat model from 11 to 12 November, kept to go back to

Licence: ours, kept for the research, not released

What it knows

Made on 11 November from her blend with a Romanian round: the change that teaching a copy of her round-4 chat model to say when the records do not answer had made to its weights was added to the blend, with no further training. On the first evaluation pass it says so on most of the questions no record answers, where the blend never did. Its other answers from records were within one item of the blend’s; its maths and code, and how AI readers found it sound, were within chance of the blend’s, Romanian openings the weakest. Its loss on plain text rose, and it finds fewer numbers hidden deep in a long text than the blend. In Romanian it said so in English, and sometimes answered as if it were the person asking, so on 12 November the model with the lesson in both languages took its place (the card before). No person has read it yet.

Measured

Answering from records: says when no record answers
21 of 28 (over three more runs 0.762) — the model it was made from: 0 of 28
n 28 of 90 held-out items, from 31 questions, English; string-level · 11 Nov 2026
Answering from records: cites the record found · end to end · carries the caution · carries the vet referral · stops cleanly
0.962 · 0.806 · 0.929 · 1.000 · 0.978 — the model it was made from: 1.000 · 0.839 · 0.929 · 1.000 · 0.978
n 90 held-out items from 31 questions, English; string-level · 11 Nov 2026
Maths word problems (GSM8K) · code (HumanEval · MBPP), greedy, one answer each
7.6% · 10.4% · 12.4% — the model it was made from: 9.0% · 12.2% · 13.6%; paired on the same problems −1.4 [−3.1, +0.4], −1.8 [−4.9, +1.2], −1.2 [−3.6, +1.4] points
n 1,319, 164 and 500 problems · 11 Nov 2026
9 more measurements
Romanian openings picked as sounding most like a person, blind, by AI readers
29% [20–38] — the model it was made from 39% [29–49]; on the same openings −10 [−26, +6] points
n 100 Romanian openings · 11 Nov 2026
English openings · several turns (English) picked as sounding most like a person, blind, by AI readers
31% · 41% — the model it was made from 27% · 34%; on the same items +4 [−11, +18] and +7 [−10, +24] points
n 100 openings and 100 conversations · 11 Nov 2026
Coherence over several turns, 1–5, by AI readers (English only)
3.11 — the model it was made from 3.25
n 100 conversations · 11 Nov 2026
Ordinary chat with no records: stops cleanly · repetition · says the records do not answer
97.6% · 0.014 · 2 times, both in long-document items — the model it was made from: 97.0% · 0.033 · 0
n 627 conversations · 11 Nov 2026
A number hidden in a 131,072-token text, share found
0.84 — the model it was made from 0.90
n 20 tries at each of 5 depths · 11 Nov 2026
A number hidden in a 131,072-token text, found (the same new hidden numbers for every model)
170 of 200 — the model it was made from 189; on the same numbers −19 [−27, −11] (an exact paired test, p < 0.0001): a real cost
n 200 new hidden numbers, 40 at each of 5 depths · 12 Nov 2026
Held-back loss at 2,048 tokens, English · pet care · maths
2.414 · 1.765 · 1.440 — the model it was made from 2.307 · 1.615 · 1.348: higher, the known cost
11 Nov 2026
Answering from records on the larger held-out set: says when no record answers — English · Romanian
0.826 · 0.242, every Romanian one in English
n 622 English and 99 Romanian held-out items where no record answers; string-level · 12 Nov 2026
How safe its pet-care answers are; how people read it
Not measured — no person has read it, and it answers nobody

Good at

Everyday conversation; saying when the records in front of it do not answer, most of the time; finding a fact in a long text.

Not for

Answering pet-care questions on its own; arithmetic.

Her blend with a Romanian round

Gaia’s own: her chat model of 10 November blended with a Romanian round, with no further training — her chat model for part of 11 November, kept to go back to

Licence: ours, kept for the research, not released

What it knows

Made on 11 November by averaging the weights of her chat model of 10 November (the next card) and a round that had raised its Romanian, seven parts to three, with no further training. Read blind by AI readers, it was picked as sounding most like a person more often than the model it was made from in Romanian openings — clearly on a first reading, less so on a second — and level with it in English and over several turns; maths, code and long reading held. On the first evaluation pass it never said when no record answered, so the same day a lesson in saying so was added to it (the card before), and it is kept to go back to. No person has read it yet.

Measured

Maths word problems (GSM8K), greedy, one answer each
9.0% [7.6–10.7] — the model it was blended from: 9.3% [7.9–11.0]
n 1,319 problems · 11 Nov 2026
Code (HumanEval · MBPP), greedy, one answer each
12.2% · 13.6% — the model it was blended from: 11.6% · 13.8%
n 164 and 500 problems · 11 Nov 2026
Romanian openings picked as sounding most like a person, blind, by AI readers
41% [32–50] — the model it was blended from 23% [15–31]; on the same openings +18 [+3, +33] points, a second reading +9 [−7, +25], the two pooled +13.5 [−1.5, +28.5]
n 100 Romanian openings, read twice · 11 Nov 2026
6 more measurements
English openings · several turns (English) picked as sounding most like a person, blind, by AI readers
16% · 27% — the model it was blended from 22% · 31%, inside the intervals
n 100 openings and 100 conversations · 11 Nov 2026
Coherence over several turns, 1–5, by AI readers (English only)
3.24 — the model it was blended from 3.13
n 100 conversations · 11 Nov 2026
A number hidden in a 131,072-token text, share found
0.90 — the model it was blended from 0.89
n 20 tries at each of 5 depths · 11 Nov 2026
Held-back loss at 2,048 tokens, English · Romanian · pet care
2.307 · 2.609 · 1.615 — the model it was blended from 2.322 · 2.623 · 1.638
11 Nov 2026
Answering from records: says when no record answers · cites the record found · end to end · carries the caution · carries the vet referral
0 of 28 · 1.000 · 0.839 · 0.929 · 1.000
n 90 held-out items from 31 questions, English; string-level · 11 Nov 2026
How safe its pet-care answers are; how people read it
Not measured — no person has read it, and it answers nobody

Good at

Everyday conversation; finding a fact in a long text.

Not for

Answering pet-care questions on its own; arithmetic.

Her chat model of 10 November

Gaia’s own, her larger model taught to talk, then given maths worked examples — chosen on 10 November, and on 11 November the base of the blends that followed it

Licence: ours, kept for the research, not released

What it knows

Talks naturally in a single reply: in a blind reading by AI readers it was picked as sounding most like a person more often than an open model of four billion parameters, in English and in Romanian. Partly coherent over several turns, below that open model. Weak at reasoning: it gets about one grade-school maths word problem in eleven right. With the right record in front of it, it cites it; when no record answers the question it never says so, so it is not safe to answer pet care on its own. No person has read it yet.

Measured

Maths word problems (GSM8K), greedy, one answer each
9.3% [7.9–11.0] — round 4 before the maths: 3.9% [3.0–5.1]
n 1,319 problems · 10 Nov 2026
Code (HumanEval · MBPP), greedy, one answer each
11.6% · 13.8% — round 4: 10.4% · 11.6%
n 164 and 500 problems · 10 Nov 2026
Picked as sounding most like a person, blind, by AI readers: English · Romanian openings
32% · 32% — round 4 35% · 42%; the 4B open model 28% · 26%
n 100 openings in each language · 11 Nov 2026
5 more measurements
Picked as sounding most like a person over several turns, blind, by AI readers (English only)
38% — round 4 47%; the 4B open model 14%
n 100 conversations · 11 Nov 2026
Coherence over several turns, 1–5, by AI readers (English only)
3.17 — round 4 3.13; the 4B open model 3.84
n 100 conversations · 11 Nov 2026
A number hidden in a 131,072-token text, share found (fresh keys)
0.935 (at the very end: 0.775)
n 40 tries at each of 5 depths · 11 Nov 2026
Answering from records: cites the right record placed in front of it · says when no record answers
30 of 31 · 0 of 28
n 90 held-out items from 31 questions, English; string-level · 10 Nov 2026
How safe its pet-care answers are; how people read it
Not measured — no person has read it, and it answers nobody

Good at

Everyday conversation; finding a fact in a long text; carrying a record placed in front of it.

Not for

Answering pet-care questions on its own; arithmetic.

Her other chat models

Gaia’s own: every other conversation fine-tune of her larger model, measured and kept

Licence: ours

What it knows

Four rounds of teaching her larger model to talk, a preference step after three of them, variants with maths and code worked examples, two rounds that tried to give back its Romanian, and two copies taught to say when the records do not answer — the second one’s change is inside her chat model now. The first round, from open conversation sets and our answer set with none of the conversations we generated, was already picked as sounding most like a person far more often than the base model; later rounds added our own generated conversations that passed the bar. Worked examples with code lowered its code; the Romanian rounds raised Romanian and lowered English and longer conversations, so none of them was chosen.

Measured

Our own conversations in the training set, rounds 1 · 2 · 3 · 4
0 · 1,041 · 3,747 · 5,066 (of 102,673 · 103,714 · 106,420 · 107,739)
n 4 training sets · 8 Nov 2026
Round 1 against the base and the 4B open model: openings picked as most like a person, blind, by AI readers
80% — the base 0%, the open model 18%
n 60 openings (37 English, 23 Romanian) · 8 Nov 2026
Round 4 against the two preference-tuned models: English · Romanian openings picked, blind, by AI readers
46% · 33% — the others 16% and 16% · 33% and 32%
n 37 English and 100 Romanian openings · 8 Nov 2026
4 more measurements
Maths word problems (GSM8K) after the worked examples: all, 2 epochs · all, 1 epoch · maths only
8.9% · 7.0% · 9.3% — round 4 3.9%
n 1,319 problems · 10 Nov 2026
Code (HumanEval) after the worked examples: all, 2 epochs · all, 1 epoch · maths only
7.3% · 7.9% · 11.6% — round 4 10.4%
n 164 problems · 10 Nov 2026
Romanian openings picked as most like a person, blind, by AI readers: the first · the second Romanian round
37% · 41% — her chat model of 10 November 26% · 26%
n 100 Romanian openings in each reading · 11 Nov 2026
A number hidden in a 131,072-token text, share found (fresh keys): round 1 · round 4 · all, 1 epoch
0.95 · 0.915 · 0.935
n 40 tries at each of 5 depths, each · 11 Nov 2026

Good at

Showing what each change did, every one measured on the same sets.

Not for

Use — none of them is the chosen model.

Her main model

Gaia’s own, about 1.2 billion parameters, trained from nothing

Licence: ours, kept for the research, not released

What it knows

A base model, fluent in English and pet-care text and weaker in Romanian. It read no code or maths, and is barely above chance on arithmetic. It has had one fine-tune, on answering from the records it is given: on string-level checks, over items that include ones it trained on, it now stops cleanly, cites the right record and carries a due hedge or vet referral nearly every time. It has not been taught everyday conversation.

Measured

Bits per byte on held-back English · Romanian (lower is better)
0.7583 · 0.8783
n 4,514,099 and 4,785,192 tokens · 18 Oct 2026
Bits per byte on held-back pet care (partly seen in training)
0.6005 — about 40% of those tokens fell inside its reading
n 7,999,999 tokens · 18 Oct 2026
Reasoning check, four options (chance 25%)
28.7% right (201)
n 700 problems · 18 Oct 2026
6 more measurements
Picking the right record on its own (first place)
62.3%
n 231 questions, 439 records · 18 Oct 2026
GPU hours, training
373.6 on the machine’s record
17 Oct 2026
First fine-tune, base → fine-tuned: stopped cleanly · cites the right record · end to end
0% → 98.4% · 0.3% → 97.4% · 0.5% → 82.7%
n 3,592 items, some of them trained on; string-level · 7 Nov 2026
First fine-tune, base → fine-tuned: carries a due hedge · a due vet referral
36.8% → 98.1% · 31.7% → 98.3%
n 3,592 items, some of them trained on; string-level · 7 Nov 2026
First fine-tune, after epoch 1 · epoch 2 · the end: cites the right record — end to end
91.3% · 95.9% · 97.4% — 73.7% · 77.7% · 82.7%
n 3,592 items, some of them trained on; string-level · 7 Nov 2026
How it converses
Not measured — not taught everyday conversation

Good at

Fluent English and pet-care text for its size.

Not for

Arithmetic; answering anybody.

The test models (160 million parameters)

Gaia’s own small test models, eleven of them

Licence: ours

What it knows

Nothing a person could use. Each answered one design question: how much pet care and Romanian to read, whether to add code and maths, how to stretch the context.

Measured

Loss on held-back pet care, no pet care → 15% (English level)
2.763 → 2.559 (English 3.018 → 3.015)
n 4 million tokens per set, 6 models · 6 Oct 2026
Reasoning check across the code-and-maths arms
20.7–25.6% — chance is 25%
n 700 problems, 5 models · 18 Oct 2026
GPU hours, both tests
49.87
n 10 models · 18 Oct 2026

Good at

Settling a design question cheaply.

Not for

Saying anything about answers — far too small.

The vocabulary test models (43 million parameters)

Gaia’s own smallest test models, eight of them

Licence: ours

What it knows

Nothing a person could use. They showed that the vocabulary we chose matches a larger one once both are given the same computing time.

Measured

Bits per byte, equal tokens: 16,384 · 32,768 · 49,152 pieces
1.1321 · 1.1151 · 1.1070
n 2 seeds each · 28 Sept 2026
Bits per byte, equal compute, 32,768 pieces
1.1079 (49,152: 1.1070)
n 1 seed · 28 Sept 2026

Good at

Choosing the vocabulary.

Not for

Anything else.

The first smoke-test models

Gaia’s own toy models, a few million parameters

Licence: ours

What it knows

Nothing useful: they proved the code, the machine and resuming a run from where it stopped.

Measured

Four GPUs, first run on the machine
197 million tokens in 2 min 46 s of wall time; 1.6–2.0 million a second across the node while training
n 1 run · 27 Sept 2026

Good at

Proving the pipeline.

Not for

Anything else.

Open models that wrote or corrected conversations

6 on the benchmark

Open-weights models we ran ourselves to write everyday two-person conversations, pet care screened out.

Open model A

A very large mixture-of-experts model (over 200 billion parameters)

Licence: Apache-2.0

What it knows

Natural, warm, short English conversation, read good in more than four of five in the first pilots — but 42.5% on its first large English batch. Fluent Romanian with small grammar slips, never reaching our bar. Left alone it invents a life of its own and local facts, and as a judge of its own writing it agreed not at all with a reference reading made by an AI model.

Measured

Big run (grid 2): kept after every check
18,730 of 40,000 (46.8%; English 10,834, Romanian 7,896)
n 40,000 · 5 Nov 2026
English, read by an AI model, round 1
32 of 40 read good (80%)
n 40 conversations · 6 Nov 2026
English, read by an AI model, at the strict cut
43 of 50 read good (86%)
n 50 conversations · 6 Nov 2026
8 more measurements
Romanian, read by an AI model — best of three tries
19 of 31 read good (61%)
n 31 conversations · 6 Nov 2026
As a judge of its own writing: agreement with a reference read by an AI model (κ, English · Romanian)
0.00 · 0.004
n 80 conversations · 5 Nov 2026
As a judge of its own writing: mean score · passed (the trial)
9.77 · 299 of 300
n 300 conversations · 5 Nov 2026
Correcting another model’s Romanian (read by an AI model)
24 of 40 read good before and after: 6 fixed, 6 damaged
n 40 conversations · 6 Nov 2026
English wave 1 (the 3b prompts): kept after every check
2,131 of 15,000 (14%)
n 15,000 · 7 Nov 2026
English wave 1, read by an AI model for the gate
17 of 40 read good (42.5%)
n 40 conversations · 7 Nov 2026
As a third judge asked for evidence: read good before → after its filter
the weakest sets lifted — English 42.5% → 52%, Romanian 35% → 50% — but open model I’s English with thinking lowered, 80% → 77%
n 40 conversations a set · 7 Nov 2026
Its Romanian check, against the AI reader’s labels: share of what it passed that read good · share read good overall
56–58% · 52% (the three open models’ own checks) — they do not separate good from poor
n 143 Romanian conversations · 7 Nov 2026

Good at

Everyday English conversation.

Not for

Romanian to our bar; judging its own writing.

Open model B

A very large mixture-of-experts model (over 300 billion parameters), run across two nodes

Licence: MIT (as its model card states)

What it knows

The most natural, shortest and least invented Romanian of every model we tried, and it keeps more than twice as much. Its faults are small grammar slips, and the second judge flags it as talking as if it were a person in about four conversations in ten. At a steadier temperature its Romanian read 35% good. No model has reached our bar in Romanian so far.

Measured

Romanian kept after every check, pilot
73 of 200 (37%)
n 200 · 6 Nov 2026
Romanian, read by an AI model
24 of 40 read good (60%)
n 40 conversations · 6 Nov 2026
Romanian, read by an AI model, strict cut
19 of 29 read good (66%)
n 29 conversations · 6 Nov 2026
7 more measurements
Big run: kept after every check (today’s rules)
410 of 1,800 (22.8%)
n 1,800 · 6 Nov 2026
Second judge’s flags on the big run: talks as a person · invented facts · bad Romanian
39.4% · 5.7% · 4.5%
n 1,800 · 6 Nov 2026
Romanian kept after every check and the strict judge cut, in batches
465 in 10 batches of 50 — none used yet
n 465 · 6 Nov 2026
GPU hours, pilot · big run
about 1.1 · about 2.5
6 Nov 2026
Romanian at a steadier temperature: kept after every check
135 of 500 (27%)
n 500 · 7 Nov 2026
Romanian at a steadier temperature, read by an AI model for the gate
14 of 40 read good (35%)
n 40 conversations · 7 Nov 2026
Its Romanian check, against the AI reader’s labels: share of what it passed that read good · share read good overall
56–58% · 52% (the three open models’ own checks) — they do not separate good from poor
n 143 Romanian conversations · 7 Nov 2026

Good at

Natural Romanian conversation.

Not for

Romanian to our bar so far.

Open model C

A mid-sized mixture-of-experts model (10–50 billion parameters), of open model A’s family

Licence: Apache-2.0

What it knows

Fast and cheap — about five times the throughput. Without our prompt it answered with emoji and headed lists. We chose against it on quality, which is what is scarce, not computing time.

Measured

Conversations per GPU hour, pilot
10,693 (the very large model: 2,013)
n 300 · 3 Nov 2026
Quality, read
Not measured — not recorded

Good at

Speed.

Not for

Our conversations.

Open model D

A mid-sized dense model (10–50 billion parameters) trained for European languages

Licence: Apache-2.0

What it knows

Clean Romanian grammar, but it talks like a service desk and makes little real small talk. It could not judge our conversations, and when asked to correct them it fixed the commonest slips while changing or deleting correct words.

Measured

As a judge, scoring 1–10: agreement with a reference read by an AI model (κ, English · Romanian)
0.01 · 0.004 — 9s across the board, no flag ever set
n 74 conversations · 5 Nov 2026
As a judge, quoting the fault, against a reference read by an AI model: κ · accuracy (English)
0.28 · 0.65 (Romanian κ 0.04)
n 74 conversations · 5 Nov 2026
Writing Romanian: kept · read good by an AI model
32 of 200 · about 4 of 21 read good (≈19%)
n 200 written, 21 read · 6 Nov 2026
2 more measurements
Correcting Romanian: poor conversations that became good (read by an AI model)
2 of 19 (41% → about 47%), with errors added
n 32 conversations · 6 Nov 2026
As a Romanian checker: passed good · passed poor (as an AI model read them)
7 of 13 · 10 of 19 — it does not separate them
n 32 conversations · 6 Nov 2026

Good at

Clean Romanian grammar.

Not for

Judging, checking or correcting our conversations.

Open model I

An openly licensed model, a newer version of open model B

Licence: MIT

What it knows

The best English of the conversations read for the gate: 75–80% read good across its three newer settings, within each other’s sampling error and below our bar. Its Romanian read 25% good, and 47.5% when it thinks before each reply. As a judge of its own set it gave no lift, and its Romanian check does not separate good conversations from poor ones.

Measured

English kept after every check: 3b prompts · thinking · 3c prompts · thinking and 3c
100 of 300 (33%) · 120 of 300 (40%) · 95 of 300 (32%) · 107 of 300 (36%)
n 300 a set · 7 Nov 2026
Romanian kept after every check: 3b prompts · thinking
108 of 500 (22%) · 73 of 200 (37%)
n 500 and 200 · 7 Nov 2026
English wave 2 (thinking, the 3b prompts): kept after every check
2,121 of 6,000 (35%)
n 6,000 · 7 Nov 2026
11 more measurements
English, read by an AI model for the gate — wave 2, by two readers
54 of 80 read good (67.5%): 28 and 26 of 40 each; with its pilot 86 of 120 (72%)
n 120 conversations · 7 Nov 2026
English, read by an AI model for the gate — 3b prompts
26 of 40 read good (65%)
n 40 conversations · 7 Nov 2026
English, read by an AI model for the gate — 3b prompts, after a third judge
22 of 31 read good (71%)
n 31 conversations · 7 Nov 2026
English, read by an AI model for the gate — thinking before each reply
32 of 40 read good (80%)
n 40 conversations · 7 Nov 2026
English, read by an AI model for the gate — 3c prompts
31 of 40 read good (77.5%)
n 40 conversations · 7 Nov 2026
English, read by an AI model for the gate — thinking and 3c prompts
30 of 40 read good (75%)
n 40 conversations · 7 Nov 2026
Romanian, read by an AI model for the gate — 3b prompts
10 of 40 read good (25%)
n 40 conversations · 7 Nov 2026
Romanian, read by an AI model for the gate — 3b prompts, after a third judge
8 of 23 read good (35%)
n 23 conversations · 7 Nov 2026
Romanian, read by an AI model for the gate — thinking before each reply
19 of 40 read good (47.5%)
n 40 conversations · 7 Nov 2026
As a judge of its own set (the 3c prompts): read good before → after its filter
77.5% → 76% — no lift
n 40 conversations · 7 Nov 2026
Its Romanian check, against the AI reader’s labels: share of what it passed that read good · share read good overall
56–58% · 52% (the three open models’ own checks) — they do not separate good from poor
n 143 Romanian conversations · 7 Nov 2026

Good at

English conversation, thinking before each reply.

Not for

Romanian to our bar; judging its own writing.

Open model J

A very large mixture-of-experts model (about 400 billion parameters, 17 billion active)

Licence: Apache-2.0

What it knows

Nothing yet: three starts failed on our machine’s serving software, so it has not written a conversation.

Measured

Starts on our machine
Not measured — three starts failed on the machine’s serving software
7 Nov 2026

Good at

Not measured.

Not for

Not measured.

Open models and classifiers that checked them

5 on the benchmark

A judge that must quote each fault, and small classifiers in the screen.

Open model E

A mid-sized dense model (10–50 billion parameters), run as text only

Licence: Apache-2.0

What it knows

It agrees with a reference reading, made by an AI model, only when it must copy out the exact words at fault; scoring conversations from 1 to 10 failed for every judge we tried. The best open judge we have for English, and not yet calibrated for Romanian.

Measured

Quoting the fault, against a reference read by an AI model: κ · accuracy (English)
0.49 · 0.75 — kept 24, of which 18 read good
n 40 conversations · 5 Nov 2026
Quoting the fault, against a reference read by an AI model: κ (Romanian)
0.08 — the reference has only 2 good of 34
n 34 conversations · 5 Nov 2026
Conversations it dropped in the big run
9,356 of 40,000
n 40,000 · 5 Nov 2026

Good at

Finding the fault in an English conversation.

Not for

Romanian, until calibrated; scores out of ten.

Classifier 1

A small multilingual classifier (under one billion parameters)

Licence: MIT

What it knows

It tells pet-care talk from everything else in English, and cannot in Romanian — so Romanian is screened by a word list of our own.

Measured

Share scoring above 0.5: conversations flagged by keywords · clean ones (English)
0.60 · 0.01
n 80 flagged and 160 clean per source · 5 Nov 2026
The same in Romanian
0.03 · 0.00 — it does not read Romanian
n 80 flagged and 160 clean per source · 5 Nov 2026
English conversations it dropped from a set
110 of 5,084
n 5,084 · 5 Nov 2026

Good at

Catching pet care in English.

Not for

Romanian.

Classifier 2

A smaller multilingual classifier, tried for speed

Licence: MIT

What it knows

It scored flagged and clean conversations alike in both languages, so it was not used.

Measured

Share above 0.5, flagged · clean: English, then Romanian
0.30 · 0.25, then 0.57 · 0.62
n 80 flagged and 160 clean per source · 5 Nov 2026

Good at

Speed.

Not for

Telling pet care from anything else.

Classifier 3

A small English toxicity classifier

Licence: Apache-2.0

What it knows

It separates insults from ordinary English cleanly.

Measured

Score on an ordinary sentence · on an insult
0.001 · 0.994
n 2 sentences · 3 Nov 2026
English texts above 0.5 in a conversation set
1,616 of 591,498
n 591,498 English texts · 3 Nov 2026

Good at

English toxicity.

Not for

Any other language.

Classifier 4

A small multilingual toxicity classifier

Licence: Apache-2.0

What it knows

It catches real insults in Romanian, but scores ordinary casual Romanian high, so it is used only at a high cut.

Measured

Score on a real insult
0.994
n 1 message · 5 Nov 2026
Score on ordinary casual Romanian, a message
0.2–0.8
n 150 conversations (the trial’s Romanian) · 5 Nov 2026
Benign Romanian conversations dropped at a 0.5 cut · at 0.9
55 of 150 · 2 of 150
n 150 · 5 Nov 2026
1 more measurement
Romanian texts above 0.5 in a conversation set
81 of 39,260
n 39,260 Romanian texts · 3 Nov 2026

Good at

Catching real insults.

Not for

A low cut in Romanian.

The open model in the four-system comparison

1 on the benchmark

Arm 2 is the open model as released; arm 3 is the same model fine-tuned on the same answer set as Gaia’s main model. All answer the same items, marked by the same string-level checks — and the items include ones arm 3 and Gaia’s main model trained on.

The 4B open model

A 4B open model — arms 2 and 3 of the four-system comparison: as released, and fine-tuned on our answer set

Licence: Apache-2.0

What it knows

As released it finds the right record almost always, but leaves about one answer in five empty and carries a due hedge or vet referral only about half the time. Fine-tuned on our answer set it leaves none empty and carries every due hedge and referral — on string-level checks, over items that include ones it trained on.

Measured

Arm 2, as released: stopped cleanly · empty answers · cites the right record · cites only it · end to end
99.8% · 20.4% · 98.1% · 89.7% · 84.0%
n 3,592 items, some of them trained on; string-level · 7 Nov 2026
Arm 2, as released: carries a due hedge · a due vet referral
57.5% · 48.5%
n 3,592 items, some of them trained on; string-level · 7 Nov 2026
Arm 3, fine-tuned: stopped cleanly · empty answers · cites the right record · cites only it · end to end
98.8% · 0% · 99.6% · 99.6% · 84.9%
n 3,592 items, some of them trained on; string-level · 7 Nov 2026
2 more measurements
Arm 3, fine-tuned: carries a due hedge · a due vet referral
100% · 100%
n 3,592 items, some of them trained on; string-level · 7 Nov 2026
Judged by concept, not by string
Not measured — the judged pass has not run

Good at

Finding and citing the right record, as released.

Not for

Hedging and referring to a vet, unless fine-tuned.

Read, never run — dataset authors, vocabularies, set aside

4 on the benchmark

Models whose writing we screened, whose vocabularies we measured, or that we considered and did not run.

Open model F

A mid-sized model (10–50 billion parameters) that wrote an openly licensed Romanian dataset

Licence: Apache-2.0

What it knows

Fluent Romanian, but its set is one-turn documents, not conversations: an AI model read few of them as good conversation; the judge’s rubric passed most, but with its length rule almost none — one reason a reading, not a rubric, decides.

Measured

Read good by an AI model
3 of 20 (15%)
n 20 · 5 Nov 2026
Passed by the second judge
92.7%
n 300 · 5 Nov 2026
Passed by the second judge, with its length rule
1%
n 300 · 5 Nov 2026

Good at

Fluent Romanian.

Not for

Conversation.

Open model G

A mid-sized model (10–50 billion parameters) that wrote an openly licensed English dataset of conversations between invented personas

Licence: CC BY 4.0 (the dataset)

What it knows

In every reply the assistant has a human life of its own — which is exactly what we teach Gaia not to claim.

Measured

Read good by an AI model
0 of 20
n 20 · 5 Nov 2026
Passed by the second judge
5.7%
n 300 · 5 Nov 2026

Not for

Teaching an assistant to talk as itself.

Published vocabularies

The vocabularies of today’s frontier and open models — measured, the models never run

Licence: each its own published licence

What it knows

Cheap for English; every one charges Romanian half again to nearly twice as many tokens.

Measured

Romanian against English on real web text, three vocabularies (ours: 1.082×)
1.52× – 1.819×
n 20 MB held-out text · 28 Sept 2026
Romanian against English on translated sentences, seven vocabularies (ours: 1.19×)
1.58× – 1.88×
n 1,012 sentences · 29 Sept 2026

Good at

English.

Not for

Romanian.

Open model H

A very large open model, considered for Romanian and not run

Licence: MIT

What it knows

Not run: it would have needed three nodes of the machine, and open model B fitted on two.

Measured

Any measurement
Not measured — it would need three nodes

From the research · Round 1, week six

A larger model, trained — and reading four times further back

The week-five decision was taken on 18 October, against the conditions set before the round began, and its reasons are on the home page’s timeline: they were met, so a larger model was attempted. About three billion parameters, trained from nothing across two nodes at once, on a reading chosen by one more test — how much code and maths to add (week four, below). It finished on 29 October, and the same day it was trained on to read four times further back in a text.

Measured on Isambard-AI from 18 to 29 October 2026: Gaia’s larger model, trained from nothing, then trained on to read longer texts. Its answers have not been evaluated yet — nothing here says how well it answers, how safe its answers are, or whether what it says about animals is right.

  • 152,600learning steps — every one planned150 billion tokens read, 18 October to 29 October
  • 183,289tokens a second, the medianacross two nodes — 98.9% of the speed measured first
  • 1.896loss on held-back text at the end3.258 at the first check — lower is better
  • 1,706.9GPU hours on the machine’s recordthe largest stage of the round so far

What training measured

Loss on held-back text fell from 3.258 to 1.896 — on English, Romanian, pet care, maths and code

It read 150 billion tokens: English 60.5%, Romanian 8.5%, pet care 11%, code 10% and maths 10% — the shape the code-and-maths test chose. As for her main run, the last fifth of the run lowered the learning rate, and that is where most of the late fall came. At the end, on the mixed held-back text, it was roughly as unsure of each next piece as a choice among 7 equally likely ones. That row does not compare with her main model’s: this model’s held-back mix includes code and maths, which are easier to predict. The fair comparison is the next result.

What a loss is not, here as before: a score for whether what it writes is true, safe, or what a vet would say.

Loss on held-back text — lower is betterThe trainer’s own estimate at the first check, after 1,000 steps, and at the end, after 152,600. The axis starts at 0.50, not zero. The rows do not rank one another: each is a different kind of text.
  • At the first check (step 1,000)
  • At the end of the run
  • All held-back textfirst check 3.258, end of the run 1.896
  • Englishfirst check 3.620, end of the run 2.225
  • Romanianfirst check 4.296, end of the run 2.391
  • Pet care — flattered, see belowfirst check 3.142, end of the run 1.520
  • Mathsfirst check 2.554, end of the run 1.380
  • Code (Python)first check 1.264, end of the run 0.637

How we know

These are the trainer’s own estimates: at every check it scored a random sample of held-back text from each kind of reading, so neighbouring checks move by one to three hundredths from chance alone. The next result is the measurement — one script, the same files, both models.

Beside her main model

On text neither model read, the larger one needs 6.6% fewer bits for English and 7.8% fewer for Romanian — and a reasoning check went from 28.7% to 71.6%

Measured the same way on the same held-back text, the larger model predicts English and Romanian better: on text neither model ever read, it needs 6.6% fewer bits for each byte of English and 7.8% fewer for Romanian.

The pet-care row looks like the largest change, and it is the least clean. The held-back pet-care pages were kept out of both models’ pet-care reading — but they are also pages of the general English both read, and about 40% of them fell inside the main model’s reading and about 58% inside the larger one’s. So the pet-care figures flatter both, the larger more; how much of that change is better prediction is not measured yet.

On the reasoning check — sums, food over a number of days, the price of a run of walks, nights of boarding — her main model was barely above chance and the larger one answered 71.6% right. This pair cannot say why: the larger model is bigger, and it also read code and maths that the main model never saw.

One thing did not improve. Asked to pick, on its own, the right record among all 439 of ours for each of 231 questions, the larger model put it first for 139 of them and the main model for 144 — a difference within chance — and both did worse than simply matching words, which found 160. This tests the model alone; the research’s own retrieval is a separate method, built for the job.

Bits per byte on held-back text — lower is betterThe same files for both models: her main model, scored on 18 October, and the larger one, on 29 October. Bits per byte counts per byte of text, so it does not depend on how the text is cut into pieces. The axis starts at 0.40, not zero.
  • Her main model
  • Her larger model
  • English — never readmain model 0.758, larger model 0.708
  • Romanian — never readmain model 0.878, larger model 0.809
  • Pet care — flattered, see abovemain model 0.601, larger model 0.476
A reasoning check — the share answered right700 small arithmetic and pet-care word problems, each with four answers to choose from. The gold line is chance: one in four, 25%.
  • Her main model28.7%
  • Her larger model71.6%

How we know

One script, unchanged, scored both models on the same files, and every count the two runs share is identical — checked rather than assumed. English and Romanian are the fair comparison: neither model was trained on that text. Each model was trained once, with one seed.

Reading further back

Trained on at four times the length, it finds a number hidden in a long text — and lost nothing at the old length

The finished larger model was trained on for 3.0 billion more tokens, read 8,192 at a time instead of 2,048. Before it, the model was lost beyond its old length: the further into a long text, the worse it predicted, and in a text longer than 2,048 tokens it never found the hidden number at all. After it, every part of a long text is predicted about as well as the start, and the hidden number is found almost everywhere — the one weak place is the very end of the longest text, 9 times in 20, which the next stage’s test will look into.

Reading further cost nothing at the old length: on every kind of held-back text, its loss at 2,048 tokens is slightly lower after the stage than before, by 0.004 to 0.022.

A number hidden in a long text: the share found20 tries in each place, at the start, 25% of the way in, 50% of the way in, 75% of the way in and the very end, averaged. Before this stage the model had only ever read 2,048 tokens at a time.
  • Before this stage
  • After it
  • A text of 2,048 tokensbefore 0.90, after 1.00
  • A text of 4,096 tokensbefore 0.00, after 0.99
  • A text of 8,192 tokensbefore 0.00, after 0.89
English: loss along a long text — lower is betterHeld-back long English documents, scored in pieces of 2,048 tokens from the start. The axis starts at 2.00, not zero.
  • Before this stage
  • After it
  • Tokens 0–2kbefore 2.26, after 2.24
  • Tokens 2–4kbefore 5.17, after 2.15
  • Tokens 4–6kbefore 6.36, after 2.15
  • Tokens 6–8kbefore 6.87, after 2.19

How we know

Before and after were scored by one script on held-back long texts that no training read. Each place in the hidden-number test is 20 tries, so one try moves it by 0.05; one run. A hidden number is a test of finding, not of understanding.

From the research · Round 1, week four

Her main run is finished — what it measured, and what it did not

With her vocabulary and her reading chosen, Gaia’s own model was trained from nothing: 1.2 billion parameters, one run, reading the chosen mix from its first step to its last. It finished on 17 October. For the first time, the figures on this page are about Gaia herself rather than small test models — and what training measures is one thing only: how well she predicts text.

Measured on Isambard-AI from 13 to 17 October 2026: Gaia’s own main model, trained from nothing on the chosen mix. Her answers have not been evaluated yet — nothing here says how well she answers, how safe her answers are, or whether what she says about animals is right.

  • 89,000learning steps — every one planned70 billion tokens read, 13 October to 17 October
  • 373.6GPU hours it used4% under the 389 planned
  • 216,971tokens a second, the median9% faster than the timing run
  • 2.311loss on held-back text at the end3.466 at the first check — lower is better

What training measured

Loss on held-back text fell from 3.466 to 2.311 — a measure of prediction, not of knowledge

Loss is how surprised a model is by each next piece of text, averaged over text it was never trained on; lower means it predicts that kind of text better. At the end of her run, on the mixed held-back text, Gaia was roughly as unsure of each next piece as a choice among 10 equally likely ones. At the first check it was a choice among 32.

What a loss is not: a score for whether what she writes is true, safe, or what a vet would say. A model can predict text well and still state wrong things fluently — which is why Gaia is built to look facts up rather than rely on what she memorised, and why nothing on this page is a claim about her answers. That needs the evaluation the research was designed around — accuracy, knowing when to refer to a vet, and cost, on the same questions as three other systems — and it has not run yet.

The rows do not say which language she is best at. Pet-care pages are a narrower kind of text than the general web, and Romanian is cut into more, shorter pieces than English, so a lower number on one row than another does not mean better. Nor do they compare with the test models’ losses below: smaller models, scored on different text. One thing they do show. Each pet-care page was read about 2.5 times, which the reading test never tried, and loss on the held-back pet-care pages kept falling through the run, to its lowest at the last check — where reading the same pages too often would have shown first.

A correction, found on 29 October: those held-back pet-care pages were kept out of her pet-care reading, but they are also pages of the general English she read, and about 40% of them fell inside it. So her pet-care figures flatter her a little. Her English and Romanian were never read.

Loss on held-back text — lower is betterLoss per token at the first check, after 1,000 steps, and at the end, after 89,000. The axis starts at 1.50, not zero. The four rows do not rank one another: each is a different kind of text.
  • At the first check (step 1,000)
  • At the end of the run
  • All held-back textfirst check 3.466, end of the run 2.311
  • Englishfirst check 3.486, end of the run 2.374
  • Romanianfirst check 4.153, end of the run 2.585
  • Pet carefirst check 3.013, end of the run 1.857

How we know

Every check scored text set aside before training, by a hash of each page’s identity, and never read by the run. A check reads a random sample of that text, so neighbouring checks move by about a hundredth from chance alone. The whole run used one vocabulary, so its own checks compare with each other; the last fifth of it lowered the learning rate, as planned, and most of the late fall came there.

What she reads, again

Adding code and maths cost English a little and barely moved pet care — and made our pet-care answers less surprising to the test models

Her larger model was to read more than her main one, so the next question was whether some of it should be code and maths — text that rewards careful, step-by-step prediction. Five small test models answered it, scored on 18 October. Code was learned at once: with any code in the reading, the test models predicted code at under half the bits per byte they needed without it — 1.59–1.61 without, 0.58–0.62 with. Maths was learned mostly from maths.

The cost fell where it was expected. English paid a little, in proportion to the English the code and maths replaced — about 1.1% for the mix with both. Pet care moved by less than 1%. And the measure closest to the task moved the right way: our own written pet-care answers became a little less surprising in every mix with code, most of all with code and maths together — about 2.8%. One run of each, on small models, so this is a direction, not a proof. That mix — 10% code and 10% maths — is what her larger model read.

A reasoning check was scored too, and says nothing at this size: every test model, with or without code and maths, answered 20.7–25.6% right where guessing gives 25%.

Our written pet-care answers: how surprising each test model finds them — lower is betterBits per byte on the answers to our 329 written pet-care questions. The axis starts at 1.50, not zero: the differences are small against the whole.
  • No code — her main run’s mix1.560
  • No code — the same mix, another seed1.545
  • Code 10%1.529
  • Code 20%1.523
  • Code 10% and maths 10% (chosen)1.509
Maths text: how well each test model predicts it — lower is betterBits per byte on held-back maths pages. The test model that read maths predicts it far better than the rest.
  • No code — her main run’s mix1.283
  • No code — the same mix, another seed1.307
  • Code 10%1.213
  • Code 20%1.198
  • Code 10% and maths 10% (chosen)0.965

How we know

Five test models of 160 million parameters read 3 billion tokens each, identical but for the mix; code and maths were taken out of the English share only, so each differs from the first in one thing. The first mix was trained twice, with two seeds: a difference smaller than the one between those two is not one we read meaning into. All were scored by one script on the same held-back text, in bits per byte. The stage used 13.77 GPU hours.

Her first evaluation

Her first evaluation as a base model: she predicts held-back text clearly better than the test models did — and on reasoning she is barely above chance

The day after her main run finished, the finished model — before any teaching — was put through three tests. On held-back English and Romanian, text she never read, she needs 22.6–25.4% fewer bits for each byte than the test model that read the same mix. On 700 small arithmetic and pet-care word problems with four answers each, she answered 28.7% right, where guessing gives 25%: a base model this size barely does them.

And asked to pick, on her own, the right record for each of 231 questions, she put it first 62.3% of the time — far above guessing, and below simply matching words. That is the shape the research expected: the model learns how a good answer reads, and a method built for the job finds the facts.

Bits per byte on held-back text — lower is betterThe test model that read the same mix as her main run (160 million parameters), and her main model, on the same kinds of held-back text — English and Romanian, which neither read. The axis starts at 0.70, not zero.
  • The test model
  • Her main model
  • English — never readtest model 0.980, main model 0.758
  • Romanian — never readtest model 1.178, main model 0.878
Picking the right record on her own — the share put first231 of our questions, each with one right record among all 439. Guessing would put it first less than once in a hundred.
  • Her main model, on its own62.3%
  • Matching words, no model at all69.3%

How we know

One script scored the finished base model the day after training ended, on text set aside before training. The questions in the retrieval test were written from the records, so they share words with them: matching words is the floor to beat there, not a weak baseline. None of this is the evaluation of her answers, which comes after she is taught to converse.

From the research · Round 1, week three

What she reads: chosen the same way

With the vocabulary chosen, the next question was what Gaia should read, and in what proportions. Her main run reads general English, some Romanian and pet-care text. Too little pet care and she is a general model; too much Romanian and her English pays for it. So the mix was measured before it was chosen.

Measured on Isambard-AI on 5 and 6 October 2026, on small test models reading general English and Romanian web text and the pet-care pages of the same web text — not Gaia herself. Her main run read the mix chosen here, and finished on 17 October 2026.

  • 6small test models160 million parameters each, the chosen vocabulary
  • 3 billiontokens read by eachthe same reading in six different mixes
  • 36.1GPU hours for the stageof the 90 budgeted for it
  • 15% · 10%pet care · Romanian — the mix chosenRomanian as a share of the general reading

What she reads

Pet-care reading at 15% made the test models clearly better at pet care, and cost their English nothing

Six small test models of 160 million parameters, built with the chosen vocabulary, each read 3 billion tokens. They were identical except for the mix: pet-care text at 0%, 5% or 15% of everything read, and Romanian at 10% or 30% of the general reading. The pet-care text is the pet-care pages of the same openly licensed English web text the rest of the reading comes from — 1,007,890 pages, 1.28 billion tokens — with every page that held one of our test questions removed (1,366 pages), so no test can have been read in advance.

Each model was then scored by loss: how surprised it is by each next piece of text, lower meaning it predicts that kind of text better. At 15% pet care, pet-care loss fell from 2.763 to 2.559, while English stayed level (3.018 → 3.015). Romanian at 30% made the models better at Romanian — its loss fell by about 0.23 — but worse at English, by about 0.06, and at pet care, by 0.04 to 0.05, whatever the pet-care share.

So Gaia’s main run read 15% pet care, with Romanian at 10% of the general reading (8.5% of everything she reads). The project is for UK pet care; Romanian stays in her reading so that the language result can be measured, and 10% is enough for that — the vocabulary below had already brought Romanian’s extra cost down to 8%.

Pet-care text: how well each test model predicts it — lower is betterLoss per token on 4 million held-back tokens of pet-care text. Each line runs from Romanian at 30% of the general reading to 10%, the share chosen. The axis starts at 2.50, not zero: the differences are small against the whole, and they are what decided the mix.
  • Romanian at 30%
  • Romanian at 10% (chosen)
  • No pet careRomanian at 30% 2.816, Romanian at 10% 2.763
  • 5% pet careRomanian at 30% 2.696, Romanian at 10% 2.647
  • 15% pet care (chosen)Romanian at 30% 2.595, Romanian at 10% 2.559
English text: the same models — lower is betterLoss per token on 4 million held-back tokens of English. The axis starts at 3.00, not zero. Down the rows, English barely moves with the pet-care share; along each line, taking Romanian from 30% to 10% gives English back about 0.06.
  • Romanian at 30%
  • Romanian at 10% (chosen)
  • No pet careRomanian at 30% 3.073, Romanian at 10% 3.018
  • 5% pet careRomanian at 30% 3.077, Romanian at 10% 3.008
  • 15% pet care (chosen)Romanian at 30% 3.076, Romanian at 10% 3.015

How we know

Every run used the same vocabulary, the same model shape and the same number of tokens; only the mix changed. All six were scored afterwards by one script on the same held-back text — pages set aside by a hash of each page’s identity and never read by any run — and because they share one vocabulary their losses compare directly. Each mix was run once, so a difference of a few thousandths between two rows is not one we read meaning into; the pet-care gain is many times that. The stage used 36.1 GPU hours of the 90 budgeted for it.

The plan

Her main run was sized from a measured speed, not a guess

Before her main run started, it was timed. Her main model — 1.2 billion parameters — ran for a few minutes on one node of four GPUs, reading the chosen mix, and read 199,864 tokens a second. The run was sized from that measurement: about 70 billion tokens and about 389 GPU hours, with some time held back for evaluation and checkpoints.

That is about three times what the usual rule for training a model of this size efficiently would give it — about twenty tokens for every parameter. It is past that on purpose: a small model that will be asked many questions is worth training for longer.

Her pet-care reading for that run came from a second, larger set: the same rule, plus a page must name an animal people keep at least three times, read from a larger sample of the same openly licensed web text — 3,339,103 pages, 4.19 billion tokens. At 15% of the run she read each pet-care page about 2.5 times, not about eight: loosening the rule instead was measured, and it found human medicine rather than more pets.

  • 1.2 billionparameters in her main modelthe size the 32,768-piece vocabulary was chosen for
  • 199,864tokens a second, measuredone node of four GPUs, reading the chosen mix
  • 70 billiontokens planned for her to readsized to fit the run’s share of the computing time
  • 389GPU hours planned for itwith some time held back for evaluation

How we know

The speed was read after the model had compiled and warmed up, on the configuration the main run then used; two larger batch sizes were tried and do not fit in the GPUs’ memory. These figures were its plan. What it actually read and used is in week four, above.

From the research · Round 1, week two

Her vocabulary is chosen: the first measurements of the research

Before a model can learn anything it needs a vocabulary — the pieces text is cut into before the model reads it. Round 1 trains Gaia on English and Romanian, so the first question was how many pieces to have, and whether Romanian would pay more than English to say the same thing. The award asks us to reduce that cost and to report it language by language.

Measured on Isambard-AI on 28 September 2026, on small test models reading general English and Romanian web text — not Gaia herself, whose main run came later.

  • 8small test models43 million parameters each, before the vocabulary
  • 1 billiontokens read by eachone shared English and Romanian stream
  • 9.8GPU hours for the stageof the 60 budgeted for it
  • 32,768tokens — the vocabulary chosenfinished three days ahead of plan

Language fairness

An English-first vocabulary charges Romanian almost twice. Ours charges it 8% more.

We built three candidate vocabularies — 16,384, 32,768 and 49,152 pieces — from equal amounts of English and Romanian, and measured each on 20 MB of each language it had never seen. Romanian needs 1.07× to 1.10× the tokens of English at every size, and every Romanian letter with a diacritic — ă, â, î, ș, ț — comes back exactly as it went in. An English-only vocabulary of the same sizes charges Romanian 1.80–1.86×.

Why Romanian? Because of the numbers, not the people behind the project. After English and Polish, Romanian is the most common main language in England and Wales — 472,000 people in the 2021 Census, up from 68,000 in 2011, the largest rise of any language. Measured on the same 1,012 professionally translated sentences in 32 of the UK’s languages — not the web text of this result’s chart, and seven vocabularies rather than three — the published vocabularies of today’s frontier models charge Romanian 1.6 to 1.9 times English. It is not the most charged — the heaviest tax falls on South Asian languages (below). Romanian and Polish, the two largest communities, are charged almost exactly the same; Round 1 needed one of them, and Romanian is a language the project can check closely. Source: Office for National Statistics, Census 2021 (opens in a new tab).

How many tokens Romanian needs for every token English needsMeasured on text each vocabulary never saw. The gold line is 1.00× — Romanian costing exactly what English costs.
  • Range across a group
  • The vocabulary chosen
  • Our other candidates
  • 1.00× — no extra cost
  • An English-only vocabulary, same three sizes1.80–1.86×
  • Three widely used published vocabularies1.52–1.82×
  • Ours — 16,3841.10×
  • Ours — 32,768 (chosen)1.08×
  • Ours — 49,1521.07×

How we know

The cost is small but not zero, and we report both sides of it: making room for Romanian costs English about 7% more tokens at the smallest size (4.07 characters a token instead of 4.35). Every vocabulary was measured with one counting function, on text held out from every training sample, and the published vocabularies were fetched at exact, pinned versions.

The UK’s languages

Across 32 of the UK’s languages, the heaviest tax falls on South Asian languages — Malayalam needs 6.31× the tokens of English on average

On 29 September we counted the tokens the same 1,012 sentences need in each of the 30 most common main languages in England and Wales after English, from the 2021 Census, and in Welsh and Scottish Gaelic — the same meaning in every language, professionally translated. Seven published vocabularies used by today’s frontier and open models read them. Kurdish was measured as its two main varieties, Sorani and Kurmanji — separate translations, each in its own script — so the chart has 33 rows for 32 languages.

Averaged across the seven, the most charged are Malayalam (6.31×), Panjabi (6.16×), Gujarati (5.46×), Tamil (4.82×) and Bengali (3.77×). Romanian is 24th of 33 at 1.78×, and Polish — the largest community — 1.76×. On each of these five, the gap between the fairest and the least fair of the seven is wider than the average itself: the same Malayalam sentences need from 1.85× to 10.82× the tokens of English, against an average of 6.31×, depending only on which vocabulary reads them.

How many tokens each language needs for every token English needsThe bar is the average of seven published vocabularies used by today’s frontier and open models; the thin line runs from the fairest of them to the least fair. Sorted from most to least charged, Romanian highlighted. The gold line is 1.00× — no extra cost.
  • Average of the seven
  • Fairest to least fair
  • Romanian
  • 1.00× — no extra cost
  • Malayalam6.31×, from 1.85× to 10.82× across the seven vocabularies
  • Panjabi6.16×, from 2.62× to 10.40× across the seven vocabularies
  • Gujarati5.46×, from 1.79× to 8.63× across the seven vocabularies
  • Tamil4.82×, from 1.47× to 7.65× across the seven vocabularies
  • Bengali3.77×, from 1.21× to 5.80× across the seven vocabularies
  • Kurdish (Sorani)3.52×, from 2.27× to 4.80× across the seven vocabularies
  • Greek3.51×, from 2.14× to 5.25× across the seven vocabularies
  • Nepali3.19×, from 1.53× to 4.76× across the seven vocabularies
  • Hindi3.13×, from 1.31× to 4.76× across the seven vocabularies
  • Urdu2.91×, from 1.52× to 4.34× across the seven vocabularies
  • Pashto2.80×, from 1.71× to 4.30× across the seven vocabularies
  • Scottish Gaelic2.32×, from 1.99× to 2.42× across the seven vocabularies
  • Persian2.31×, from 1.46× to 4.03× across the seven vocabularies
  • Albanian2.16×, from 1.83× to 2.37× across the seven vocabularies
  • Kurdish (Kurmanji)2.12×, from 1.97× to 2.20× across the seven vocabularies
  • Somali2.09×, from 1.72× to 2.23× across the seven vocabularies
  • Welsh2.05×, from 1.74× to 2.18× across the seven vocabularies
  • Arabic2.04×, from 1.38× to 3.47× across the seven vocabularies
  • Bulgarian2.02×, from 1.62× to 2.63× across the seven vocabularies
  • Hungarian1.98×, from 1.68× to 2.15× across the seven vocabularies
  • Tagalog1.97×, from 1.69× to 2.10× across the seven vocabularies
  • Lithuanian1.95×, from 1.67× to 2.19× across the seven vocabularies
  • Slovak1.88×, from 1.62× to 2.13× across the seven vocabularies
  • Romanian1.78×, from 1.58× to 1.88× across the seven vocabularies
  • Polish1.76×, from 1.51× to 1.91× across the seven vocabularies
  • Russian1.72×, from 1.37× to 2.46× across the seven vocabularies
  • Turkish1.72×, from 1.37× to 2.22× across the seven vocabularies
  • Italian1.56×, from 1.38× to 1.64× across the seven vocabularies
  • French1.53×, from 1.37× to 1.61× across the seven vocabularies
  • Spanish1.47×, from 1.30× to 1.57× across the seven vocabularies
  • Portuguese1.41×, from 1.23× to 1.55× across the seven vocabularies
  • Cantonese1.39×, from 1.03× to 2.09× across the seven vocabularies
  • Chinese (all other Chinese)1.29×, from 0.94× to 1.87× across the seven vocabularies

The sentences are FLORES-200’s devtest set — the same 1,012 sentences professionally translated into every language (Meta AI, CC-BY-SA 4.0). Languages and speakers: Office for National Statistics, Census 2021; Welsh and Scottish Gaelic were added as languages of the UK. Chinese (all other Chinese) is the census’s own category, “All other Chinese”: Chinese other than Cantonese Chinese (its own row, Cantonese) and Mandarin Chinese (30,825 people, not among the 30), which the census counts on their own. It is measured as written simplified Chinese.

How we know

Our own vocabulary is not the answer to this chart. Built from equal English and Romanian, it brings Romanian to 1.19× on these sentences — below all seven — but it has few pieces for any other script. On every other language it charges more than every one of the seven, and far more on other scripts: a median 1.60 more tokens for every English token than the least fair of them, against 0.15 on the other Latin-script languages. Malayalam needs 13.98×, where the seven charge 1.85–10.82×. It is a fix for Romanian only, and Round 1 claims nothing more. On these translated sentences it charges Romanian 1.19× rather than the 1.08× measured on web text above: a different kind of text gives different figures, so every comparison here is made within one text. Each vocabulary counted every sentence on its own, with no special tokens, and each was fixed to an exact file: a pinned revision, a named encoding checked against its own fingerprint, or the hash of our own file.

Measurement

A cost measured on one sentence looked half again as large

The quick way to measure this cost is on a single sentence. On one sentence, an English-only vocabulary appears to charge Romanian 2.90×; on 20 MB of real text the same vocabulary charges 1.86×. One sentence is not a sample of a language.

And a ratio can fall for the wrong reason. A vocabulary built from English children’s stories looked fairer on web text than on its own — only because its English collapsed there, from 4.70 characters a token to 3.09. A ratio shrinks when English gets worse as surely as when Romanian gets better, so we report both languages, never the ratio alone.

The same vocabularies, measured two waysTokens Romanian needs for every token English needs. The line runs from the one-sentence figure to the real-text figure.
  • Measured on one sentence
  • Measured on 20 MB of real text
  • English-only vocabulary, 32,768one sentence 2.90×, real text 1.86×
  • Ours, 32,768one sentence 1.13×, real text 1.08×

How we know

This is why every figure on this page is measured on held-out text — files no vocabulary and no model ever trained on — and why the one-sentence figures are shown beside the real ones rather than quietly replaced.

The choice

At equal compute, the middle vocabulary matches the largest — so we chose 32,768

Eight small test models, identical except for their vocabulary, each read one billion tokens of the same English and Romanian stream and were scored on the same held-out text in bits per byte — the only fair unit, because a bigger vocabulary makes fewer, harder predictions over the same sentence. At equal tokens the largest vocabulary won.

But equal tokens is not equal work. Given the extra steps to spend exactly what the largest spent, 32,768 came within a thousandth of it — about half the seed-to-seed noise — and the smallest stayed clearly behind.

How well each test model learned — lower is betterBits per byte on held-out text. The axis starts at 1.10, not zero: these differences are small, and the shaded band shows how far the same model moves from one random seed to the next.
  • Equal tokens — two seeds each
  • Equal compute — one seed each
  • Seed-to-seed noise
  • 16,384equal tokens 1.1321, equal compute 1.1155
  • 32,768 (chosen)equal tokens 1.1151, equal compute 1.1079
  • 49,152 (the reference)equal tokens and equal compute: 1.1070
Work per token while learningMillions of operations for every token read. The output layer runs on every token and grows with the vocabulary: the largest does 31% more work per token than the smallest.
  • 16,384402M
  • 32,768 (chosen)465M
  • 49,152528M

How we know

Why 32,768: as good as the largest once the work is equal, an embedding a third smaller at the size Gaia will be trained at, and the size her one-billion-parameter configuration was already built for. The equal-compute runs are one seed each, so they place the vocabularies rather than prove an order — and we say so.

Method

On a small model, an equal-token comparison flatters a big vocabulary

A comparison that gives every vocabulary the same number of tokens quietly gives the biggest one more compute — and on a small model, a lot more. At the size of our test models the output layer is a large share of every step; at the size Gaia will be trained at, it is a small one. A vocabulary chosen on a small model at equal tokens would have been chosen partly for the work it was given.

Share of each learning step spent in the output layerThe output layer’s cost grows with the vocabulary, not with the rest of the model — so its share shrinks as the model grows.
  • At 50 million parameters (the test models)16–36%
  • At 1 billion parametersabout 5%

How we know

A second reading of the plan whose only job was to find its flaw caught the equal-compute assumption before a single GPU hour was spent. The whole stage then cost 9.8 GPU hours of the 60 budgeted for it, and finished three days early.

Acknowledgement

The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR). Isambard-AI is operated by the University of Bristol and is funded by the UK Government’s Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023]. UKRI AI Research Resource (AIRR) Gateway · project 0261-1548-0489-1 · 4,000 GPU hours of computing time. The award supports the research. It is not an endorsement of any product.

Cite: McIntosh-Smith, S., Alam S. R. and Woods, C. (2024). “Isambard-AI: a leadership class supercomputer optimised specifically for Artificial Intelligence”. https://doi.org/10.48550/arXiv.2410.11199

Before the research

Six things we learned building a careful AI

Referral is built in, not asked for

Routing a clinical question to a veterinary surgeon is part of what the system can do, rather than an instruction the model is asked to obey. There is no confidence threshold to misconfigure and no prompt that can talk it out of it.

How we know

A model told only what it must NOT do refuses. The veterinary rule had to be rewritten to say what it MAY do — several possible causes, never one; what a person can safely check; and above all which signs change the urgency — before it gave a useful answer at all.

Facts are retrieved, never memorised

The model learns the shape of a good answer — name the source, give the date it was checked, say which authority decides, defer to it. The facts themselves are fetched each time.

How we know

Guidance changes without telling anybody. A model that memorised last year’s rules is wrong this year and nothing in its tone says so. Separating the shape from the facts is what makes a stale answer impossible rather than unlikely.

An explicit state for having nothing to answer from

Three states, not two: grounded, needs-context, and no-evidence — decided before a word is written, and each one gets different instructions.

How we know

Every earlier guard treated the symptom, catching a recycled answer after the model had already written one. None told it, in advance, that it was empty-handed. We also tried a confidence threshold and abandoned it: measured on real questions, answerable and unanswerable ones overlapped completely, so no threshold separated them.

An unsafe action is absent, not forbidden

Anything that cannot be taken back — sending a message in somebody’s name, changing a record, cancelling a booking — is not in the set of things the model may choose from. It is not listed and then discouraged; it is not there. The assistant drafts, and a person presses send.

How we know

A thing that cannot be chosen cannot be chosen by mistake. A prompt that discourages an action leaves the failure available on the day the request is worded slightly differently, and the wording that finds it will not be one anybody tested. Every exception argued for so far has been refused on the same ground: an exception is what every later exception cites.

A question about dogs is not answered about cats

Where a question names one species, records about another are demoted rather than excluded — because if the only record covering a question is filed under the wrong animal, a demoted answer is worse and a removed one is nothing.

How we know

The first setting was far too harsh, and our own test corpus could not see it: every question in it had been written against the record it belonged to, so no question asked about the wrong animal. A probe set built to pit records against each other found 75 queries where the rule changed the answer — and 58 of those changes were for the worse. The finding is about the measurement, not the number: a rule that arbitrates between two things cannot be tested by questions authored one at a time.

A rule the model keeps breaking is fixed by the shape of its answer, not by a bigger model

When a rule must hold every time — ask one question, never pass judgement on the person’s practice — it is written into the structure of the reply: the one question sits in a field of its own and is checked on its own, before anybody sees it.

How we know

Asked for in plain instructions, our small model broke the one-question rule on most replays of the same conversation turn. A large general model, at roughly fifteen times the price per reply, made the same kind of mistake. Giving the small model more thinking time helped only a little, for about two and a half times the cost. Moving the question into its own checked field took the small, cheap model from about one reply in four passing to eight in eight on that turn. It was measured on one conversation turn, replayed, and we say so — but the direction was not close.

Why this is on a research site

The patterns hold whichever way the research goes

The research question is whether a small, specialised model can be more reliable than a large general one on UK pet-care rules. Wherever that line falls, we have said in advance that we will publish it — finding the boundary is a result, and it is the one the next small company asking this question actually needs.

None of the patterns above depend on that answer. They are about how a system around a model should be built when a wrong answer has a cost — which is true of a large model, a small one, and whatever comes next. If they are useful to somebody building in another regulated field, they have done their job.

The measurements at the top of this page are the research’s first. The rest will be added at the top, newest first, as they are made, whichever way they land.

This is taking longer than usual — still connecting.