How Barbet 1B Base reached a native, evidence-backed 1M context

The original June article carefully separated “256K native training” from “1M RoPE extrapolation.” That was the correct description of the weights at the time. Two months later, the weights, architecture, and evaluation evidence have changed, giving 1M a narrower but substantially stronger meaning.

Here, 1M means a context length of 1,048,576 tokens in one sequence. It is not a training-token budget. The figure 3,925,868,544 exact-1M physical training tokens refers to the cumulative number of tokens processed across 1M-length training sequences; the two quantities should not be conflated.

A falsifiable definition of 1M

Setting max_position_embeddings to 1,048,576 only makes the program accept a larger range of position indices. It does not show that the weights were updated at that length, or that evidence near token 900K will affect a later prediction.

For this release, stable 1M consists of three conditions that can fail independently:

1. Physical execution: a freshly loaded checkpoint completes exact 1,048,576-token forward scoring without OOM, NaN, or non-finite target NLL.
2. Distant-evidence use: a long context containing the correct evidence raises the likelihood of the correct continuation relative to a length- and surface-matched context in which that evidence is removed or corrupted.
3. Base-capability retention: continued long-context pretraining does not materially damage the original model’s Chinese, multilingual, English, math, and code modeling ability.

The first tests runtime, the second capability, and the third the cost of acquiring that capability. We use the name stable 1M only when all three hold.

The starting point was still Barbet

The training lineage began from the native Role-B checkpoint with the following layer pattern:

[G, S, S, M] × 7

G denotes global attention, S an 8K sliding-attention layer, and M Mamba2. After converting the original Hugging Face checkpoint to native Megatron Role-B, selected raw-FP32 logits at 4K and 8K were exact. This fixed a trustworthy starting point: later changes could be attributed to continued pretraining and architectural evolution rather than conversion drift.

The project also explored KDA, but pure-KDA did not retain enough of the original capability and was excluded from the release lineage. The final checkpoint contains no QSA, external-memory controller, or other test-time memory module. PangolinTokenizer, the 1,536 hidden size, seven Mamba2 blocks, the tied embedding and LM head, and the all-token causal language-model objective all remain part of Barbet.

Shortening the path between distant tokens

The original architecture had one global-attention layer per four-layer block. At a 1M context, the intervening sliding-window layers lengthen the path through which distant tokens can influence one another. During training, the two sliding slots were progressively converted to global attention, followed by an additional global layer that was first materialized as an identity mapping:

[G,S,S,M]×7
  → [G,G,S,M]×7
  → [G,G,G,M]×7
  → [G,G,G,M]×7+G

The final insertion itself consumed zero training tokens. The new path received acquisition and exact-1M training only afterward. Materialization reset the local iteration counter, so iter7008 in the checkpoint name is not the number of updates in its entire ancestry. Measured from Role-B, the selected release lineage contains 42 accepted optimizer stages and 11,715 updates.

The architecture evolved, but the checkpoint was not replaced by another pretrained model family. The principal change was the capacity for long-range token mixing.

Training did not simply march toward longer sequences

The selected-lineage ledger after Role-B records 5,516,558,336 physical training tokens. Of these, 22 stages and 3,744 updates used exact 1,048,576-token sequences, accounting for 3,925,868,544 tokens.

This ledger includes only stages in the selected ancestry that completed optimizer updates. Pre-forward failures, evaluation, export, reload, identity materialization, and branches not selected for release are excluded.

The historical R2 repository separately documented a roughly 150B-token, three-phase recipe budget spanning 8K general pretraining, Traditional-Chinese midtraining, and extension from 32K to 256K. The historical records do not bind every completed phase to the released Role-B checkpoint in one completion receipt. We therefore describe 150B as a documented recipe budget, not an exact checkpoint-bound consumed total.

Establish the length, then isolate the failure

The selected parent progressed through 64K, 128K, 256K, 512K, and an initial exact-1M stage. This established the hardware and distributed execution path while introducing retrieval, variable binding, ordering, multi-key tasks, short compositional chains, and varied evidence depths.

Diagnostics then exposed a more specific weakness: the model could sometimes copy a value from far away but could not reliably use that same value for aggregation or a state update. Adding only more 1M examples would confound three explanations. The model might fail to find the evidence, lack the local operator, or fail to bind an acquired operator to distant evidence.

Later stages therefore returned to 512, 1K, 2K, 4K, and 8K to test local computation, literal remote copy, and remote binding separately before another exact-1M closeout. These shorter stages isolated and repaired capabilities; they did not reduce the model’s maximum context back to 8K.

The final consolidation stage ran from immutable iter6240 to iter7008:

The data mixture was 25% replay, 50% strict-digit remote compute binding, 12.5% local sampler repair, and 12.5% literal remote 8K. The objective was ordinary all-token causal next-token cross-entropy, with no answer-only loss, teacher logits, distillation, SFT, or RLHF. All 768 updates completed with no skipped update, NaN, OOM, or fatal error.

Evaluating a base model as a base model

Barbet is a causal base model with no chat template or instruction tuning. Making conversational exact match the main gate would add prompt format, decoding policy, and instruction following to the measurement, obscuring the pretraining question: did the model use distant information?

Each exact-1M evaluation example therefore has two length- and surface-matched contexts. The coherent context retains the correct distant evidence; the ablated context removes or corrupts it. Teacher forcing then measures target-only negative log-likelihood for the correct continuation:

evidence benefit = ablated target NLL − coherent target NLL

A positive value means the correct evidence increased the model probability of the correct continuation. Each task family contains 20 matched rows, and 2,000 paired-bootstrap resamples estimate the 95% confidence interval of the mean effect. The frozen release gate was fixed before the terminal run: all 140 scores had to be finite, at least five of seven task families needed a confidence-interval lower bound above zero, and all six BPB-retention buckets had to pass.

Exact-1M results

Fresh-loaded iter7008 completed all 140 rows across eight shards. Six task families passed:

Aggregation had a slightly positive mean, but its interval crossed zero. Its mean effects were also negative when evidence appeared at 30% and 50% depth. We retain this result as not proven rather than hiding it behind the overall 6/7 outcome.

Retaining base-model capability

The retention suite contains 6,955 examples and 3,000,079 model tokens. It compares iter7008 with Role-B using bits per UTF-8 byte (BPB; lower is better):

None of the six buckets regressed relative to Role-B. This supports frozen likelihood retention; it does not cover every downstream benchmark, factual behavior, or safety property.

Why not require all seven tasks?

At roughly 1.119B parameters, this release asks whether the model shows reproducible 1M distant-evidence effects across several task families without trading away its original language-modeling ability. Requiring every task, position, and surface form to succeed would test universal robustness, a broader claim than this release makes.

The 5/7 threshold was frozen before final training and was not adjusted after seeing the result. Finite exact-1M scores, task-level statistical effects, and BPB retention remained hard requirements. Free generation, 7/7, positive effects at every evidence depth, and 6/6 strict-8K operator cells remained research diagnostics. The final strict-8K result was 3/6, another indication that operator transfer is incomplete.

This calibration avoids two extremes: claiming million-token understanding from runtime alone, and requiring a billion-parameter base model to solve every long-context problem with universal robustness in one release. The failed cells remain visible.

What the evidence does and does not support

The evidence supports the following statement:

final-global-barbet-iter7008-stable-1m-v11 is a causal base model with approximately 1.119B parameters and a native 1,048,576-token runtime. Across six frozen exact-1M likelihood task families, correct distant evidence produces a reproducible positive effect on the correct continuation. All six base-model BPB-retention buckets also pass.

This experiment does not establish:

• complete understanding of arbitrary million-token documents;
• reliable aggregation or uniform performance at every evidence depth;
• equivalence between teacher-forced likelihood and dependable free generation;
• instruction following, dialogue quality, factuality, or safety alignment;
• a native 2M context;
• overall capability comparable with large frontier foundation models.

These boundaries do not weaken the six positive task results. They identify the work that remains.

Checkpoint and verifiable identities

The release checkpoint is final-global-barbet-iter7008-stable-1m-v11. At the time of publication, the Hugging Face repository remains access-controlled: OpenFormosa/barbet-1b-base. Public code and model documentation are in OpenFormosa/Barbet.

The release configuration is:

{
  "max_position_embeddings": 1048576,
  "rope_theta": 10000000.0,
  "rope_scaling": null
}

The configs/barbet_1b/ and configs/barbet_1b_1m/ directories in the GitHub repository retain the original R2 legacy presets for reproducing earlier research. Iter7008 should be loaded with the config.json shipped alongside its Hugging Face weights.

The conclusion of Barbet’s 1M effort is specific. Running a long sequence is only the starting point. Correct distant evidence must change the probability of the correct continuation, and continued pretraining must preserve the model’s earlier language-modeling ability. Iter7008 closes that evidence chain for six task families. Aggregation remains the next unresolved capability.