How we trained Barbet for a 1M-token context

Barbet is a base language model with approximately 1.1 billion parameters. It learns by reading text, predicting what comes next, and adjusting its internal parameters using prediction errors. It has not been trained as a chat assistant; this report evaluates its text predictions.

The preceding text available for a prediction is the model’s “context.” In this article, 1M means an input length of 1,048,576 tokens. A token is a unit into which the model divides text, not necessarily a word or character. The length of one input and the total amount of training data processed are different quantities.

This report covers the August 2026 Barbet release and excludes later experiments.

Continuing from 256K to 1M

The June article described Barbet trained at lengths up to 256K. Its experimental 1M configuration changed how the model represents text positions, allowing the program to attempt longer inputs. Barbet had not yet been trained on 1M-token sequences.

We continued training the same Barbet, starting at the shorter length of 64K and increasing to 128K, 256K, 512K, and 1M. At the 1M stage, each training sequence contained the full 1,048,576 tokens. The model predicted each next token, and prediction errors across the sequence updated its parameters.

We also changed its attention layers, which let the model draw on earlier text. Some layers could originally read directly only from the nearest 8K tokens. We progressively converted them to read the full preceding text, then added one more such layer. More layers could access distant information directly, at a higher computational cost. The appendix gives the full architecture and position settings.

The data included exercises in finding information, ordering events, tracking changing values, and following linked relationships. Key records appeared at different positions, so practice was not concentrated only at the beginning or end.

These exercises still used pretraining: prediction errors were counted across the whole text, not only the final answer. This release did not use instruction tuning, dialogue or human-preference training, or an external-memory module.

Finding information and using it are separate tasks

Consider an inventory record:

Warehouse A held 3 boxes.

Later, Warehouse A received 2 more boxes.

This is an illustration, not an actual test item. To continue “The number of boxes received was…”, the model only needs to find 2. To continue “Warehouse A now holds…”, it needs to combine the records and calculate 5.

When a long stretch of text separates the records, an error could mean the model failed to find the numbers or found them but could not calculate with them. Distinguishing those problems requires separate tests of retrieval and calculation.

We therefore interleaved shorter exercises, from 512 to 8K tokens. Some placed records nearby for calculation practice. Others separated the records for practice in retrieving the original values and calculating with them. Providing that practice did not establish that the model had learned the tasks; separate tests were still needed. Full 1M training remained part of this work, and the model’s maximum context was not reduced to 8K.

The final stage used 8K texts for 768 parameter updates, processing about 100 million tokens. We then reloaded the saved model and tested it again at the full 1M length. Ending with shorter exercises did not remove the need for a long-context test.

Continued training from the original Barbet to this release processed approximately 5.52 billion tokens, including 3.93 billion in sequences exactly 1M tokens long. These are processed-token counts and may include repeated data; they exclude Barbet’s original pretraining. The appendix lists totals by sequence length and the training settings.

How do we check whether earlier information helped?

Completing a computation on a 1M-token input establishes that the length runs. To check whether its information helps, we give the same Barbet two versions: one retains the correct key records; the other removes or corrupts them. The two texts have the same length and similar structure.

Using the inventory illustration, we ask whether the records of 3 boxes and 2 arrivals make “5 boxes” more likely than when those records are missing. This paired test directly checks a base model’s text predictions, without requiring prior training in conversation or instruction following.

However, the scoring program already knows that “5 boxes” is the answer. If the answer contains several tokens, the program supplies the correct preceding tokens before scoring each next one. The model does not generate the entire answer on its own.

Even if “5 boxes” becomes more likely, the model could still prefer a different answer. The test tells us whether the correct information helps, not how often the model would generate the correct answer.

Results: six families benefit; combining records does not pass

After reloading, Barbet completed scoring at exactly 1M tokens for all seven task families, with 20 matched examples per family. It did not run out of memory or produce invalid scores.

We first calculate each question’s score difference with and without the key information, then average those differences within each family. Six families showed a benefit from the correct information, including after accounting for sample uncertainty. With only 20 pairs per family, this does not guarantee improvement on every question or at every position. Full scores and statistical methods are in the appendix.

Helpful information in six families cannot be converted into an accuracy of six-sevenths. Combining records to calculate a result showed only a small mean improvement, which could not rule out no improvement. In that family, when key information appeared around 30% and 50% of the way through the text, retaining it made the mean score worse.

A separate 8K test distinguished copying a retrieved value from calculating a result and passed only three of six checks. For example, the model passed the test of copying distant values but not the test of combining them in a calculation. Some calculations also failed with shorter texts, so distance alone cannot explain every failure.

We checked existing text-prediction ability with 6,955 examples covering code, English, Japanese and Korean, math, multilingual text, and Chinese. None of the six scores worsened relative to Barbet before continued training. This result applies to the tested text, not every application, factual accuracy, or safety behavior.

What does “stable 1M” mean in this report?

The project calls this release “stable 1M” because full 1M-token computation completes, distant information helps in six task families, and the tested base-language scores do not regress. This is the project’s release standard, not a guarantee of understanding arbitrary long documents.

The training path used full long texts, interleaved shorter retrieval and calculation exercises, and evaluation after reloading. Architecture, data, and training volume all changed without separate comparisons isolating each factor. This record cannot establish which change contributed most or guarantee the same result from repeating the recipe.

These results do not establish reliable answer generation about arbitrary 1M-token texts, 2M capability, or overall parity with larger models. To establish calculation ability, the next test must let Barbet continue from the records, generate the combined total or updated value itself, and check each answer.

Technical appendix

Expand the sections below for training settings and full scores. The fixed revision link and file hashes in the last section identify the specific model files.

[G,S,S,M]×7 → [G,G,S,M]×7 → [G,G,G,M]×7 → [G,G,G,M]×7+G

{
  "max_position_embeddings": 1048576,
  "rope_theta": 10000000.0,
  "rope_scaling": null
}

Distant-evidence effect = target NLL without evidence − target NLL with evidence

Saved-model directory tree
0380aee774af7f1e81c8e4b99d016171b07490999a117f66653d4c520b580cf6

model.safetensors
05376dde654c9beb8154e1e997d0765ce4e236e51d4de99f38ad2e20fd730000

config.json
968e293a32e225a993e9a0e503ed4f41f0d3c69e7d7b03c85078497935b48d91