Barbet 1B Base: a hybrid decoder-only language model for Traditional Chinese

Key points

• A hybrid decoder architecture — a four-layer repeating motif (global attention → sliding-window attention → sliding-window attention → Mamba sequence mixer) that keeps a periodic global information channel, a low-cost local window, and an alternative sequence-mixing path all at once.
• 256K native, 1M extrapolation — progressively trained to a 262,144-token native context, with a RoPE linear-extrapolation configuration to 1M offered for research; the two are kept strictly distinct.
• A stable tokenizer contract — a fixed PangolinTokenizer throughout, giving high compression efficiency for Traditional Chinese at both training and inference time.
• Leading cross-model evaluation — on the TAIDE-14 task set, Barbet reaches 0.7488 bits/byte, first among comparable billion-parameter public models, and ranks first on 10 of 14 tasks.

Why build a Traditional-Chinese base model?

The quality of a large language model is not determined by parameter count and data scale alone. Tokenizer design, the data-processing pipeline, the training schedule, the long-context extension strategy, and even the correctness of model conversion and the evaluation protocol — every link shapes the final result. For Traditional Chinese the challenges are more concrete: a model must balance general multilingual ability, Traditional-Chinese contextual understanding, long-document processing, tokenizer coverage, and deployment cost.

The goal of Barbet 1B Base is direct: to build a reproducible, convertible, long-context-extensible billion-parameter base language model as a starting point for Traditional-Chinese, multilingual, and long-context research.

This article introduces, in order, the model’s design motivation, architectural choices, tokenizer contract, training pipeline, long-context extension strategy, and preliminary evaluation results.

The model at a glance

Barbet 1B Base is a pure decoder-only causal language model — not an instruction-tuned conversational assistant. Its intended uses include: Traditional-Chinese base-model research; multilingual pretraining and tokenizer-behavior research; long-context retrieval and extrapolation-behavior analysis; hybrid-attention architecture research; and downstream task fine-tuning (speech recognition, speech synthesis, optical character recognition, and so on).

Core specifications:

The model should not be used directly in scenarios requiring safety alignment, factual guarantees, or high-stakes decisions.

Architecture: trade-offs in a hybrid decoder

Why not pure global attention?

For a model that must process 256K tokens, running global attention at every layer makes compute grow quadratically with sequence length. But discarding global attention entirely would cost the model its ability to exchange information across an entire document. Barbet’s answer is a hybrid: most layers use a low-cost local mechanism, while at fixed intervals one global-attention layer is kept, ensuring information can still flow across the whole sequence.

A four-layer repeating hybrid pattern

Barbet uses a fixed four-layer repeating pattern:

Global attention → Sliding-window attention → Sliding-window attention → Mamba sequence mixer

The full allocation across the 28 layers:

This design gives the model three properties at once: global-attention layers provide an information channel spanning the whole sequence; sliding-window layers handle local context efficiently with a fixed 8192-token window; and Mamba layers provide a sequence-mixing mechanism distinct from attention, with different state behavior during step-by-step decoding.

Inside each decoder block

The compute flow of each logical decoder block is:

RMSNorm → sequence mixer → residual → RMSNorm → SwiGLU feed-forward → residual

The sequence mixer is, depending on the layer index, global attention, sliding-window attention, or a Mamba mixer.

Attention-layer configuration

All attention layers use grouped-query attention (GQA): 16 query heads share 2 key/value heads, cutting the KV-cache memory requirement to one-eighth. Other key settings: head dimension 128; RMSNorm applied to queries and keys; rotary position embeddings (RoPE, base frequency 10,000,000); dropout zero.

Global-attention layers perform standard causal attention. Sliding-window layers restrict each query to attend to only the most recent 8192 tokens. The two alternate, concentrating most layers’ compute pressure in a local window while global information is refreshed once every four layers.

Mamba sequence-mixer layers

Key Mamba parameters: state dimension 64, convolution kernel width 4, expansion factor 2, head dimension 128, group count 2.

When loaded in HuggingFace, if mamba_ssm is installed the model uses the fused Mamba2 scan path. Without that package, it falls back to a pure-PyTorch reference implementation. The latter is mainly for portability testing and debugging, and is not guaranteed to be identical to the Megatron training path.

Multi-step prediction auxiliary objective

During training, in addition to the standard next-token prediction loss, a multi-step prediction auxiliary loss is added. The model also predicts the tokens at positions t+2 (weight 0.2) and t+3 (weight 0.1). The model exported to HuggingFace contains only the main causal-LM path; the multi-step prediction heads are training-only components and are not used for generation.

Parameter rebalancing: why tie the embeddings?

Traditional-Chinese and multilingual tokenizers usually have large vocabularies. If the output projection does not share weights with the embedding layer, a billion-parameter model allocates too large a fraction of its parameters to the vocabulary, squeezing the capacity of the model body. By tying the embedding layer to the LM output head, Barbet invests the saved parameter budget into a deeper model body, gaining more modeling capacity at a comparable parameter count. This is the core of the R2 parameter-rebalancing strategy.

Tokenizer: the PangolinTokenizer contract

The model uses a fixed PangolinTokenizer; the tokenizer is not retrained during the training pipeline. The core numbers of the tokenizer contract:

Special-token ID assignments: <unk> = 114688, <s> = 114689, </s> = 114690, <pad> = 114691.

The tokenizer does not automatically insert beginning- or end-of-sequence markers during training. Each document is appended a single </s> as an end-of-document marker by Megatron preprocessing. Conditioning markers (when present in the text) must remain a single token ID and must not be split. Training and validation use the same tokenizer contract to avoid tokenization drift.

Before formal data construction, the following must be verified: the correctness of the BOS / EOS / PAD / UNK token IDs; the single-ID nature of reserved special tokens; effective-vocabulary coverage; that the Megatron-padded vocabulary size matches the checkpoint weight shapes; that no BOS or EOS marker is implicitly inserted; and tokenizer-version immutability.

Data: governance principles and processing pipeline

Abstract categories of the training corpus

Because the training-data sources are not public, this article describes only the data types and governance principles. After de-identification, the training corpus can be divided into the following categories:

Data-governance gates

Before entering the pretraining pipeline, training data must pass the following gates: a usage-license gate (excluding data that does not meet internal usage policy); a license gate (excluding data whose licensing status is not permitted); a contamination gate (reducing overlap between validation and training data); exact deduplication (removing exactly duplicate documents); near-deduplication (removing or down-weighting near-duplicate content); quality filtering (removing low-information-density, garbled, or template-polluted content); boilerplate filtering (removing navigation bars, headers and footers, and repeated disclaimers); token accounting (using the actual tokenizer to compute the token budget); and train/validation split separation.

Metadata conditioning markers

The model is trained with metadata conditioning markers, letting it observe, during training, abstract markers related to language, register, domain, or data attributes. The format can be abstracted as:

<conditioning marker>
<source marker>
document body
</s>

The purpose of the conditioning markers is to provide a learnable conditioning signal that helps the model distinguish language, register, and domain, and to offer an optional steering interface at inference time. To avoid an inference-time mismatch, the training recipe includes an unmarked cooldown stage, so the model can still generate stably without a metadata prefix.

Training: a three-stage pipeline

Overall schedule

Barbet 1B Base uses a three-stage training pipeline, with a total token budget of about 150 billion tokens:

Stage 1: general pretraining (build general language ability)
Stage 2: Traditional-Chinese mid-training (shift the model distribution toward the Traditional-Chinese context)
Stage 3: progressive long-context extension (expand from 8K up to 256K)

Per-stage specifications

Shared hyperparameters

The optimizer is AdamW (β₁ = 0.9, β₂ = 0.95, ε = 1e-8). Weight decay 0.1, gradient clipping 1.0, precision bf16. Training uses Transformer Engine; FP8 and quantization-aware training are not enabled.

Stage 1: general pretraining

The core goal of Stage 1 is to build a stable language-model foundation. This stage uses large-scale de-identified general corpora, with a small amount of target-language replay to reduce the risk that some token embeddings become too cold before the subsequent mid-training.

Validation in this stage focuses on: whether the training loss decreases smoothly; whether numerical anomalies appear; whether the tokenizer and end-of-document marker pipeline is correct; whether throughput meets expectations; and whether checkpoints can be resumed normally. Stage 1 is not the main Traditional-Chinese-improvement stage, so Traditional-Chinese validation metrics serve only as a trend reference.

P1b is an unmarked cooldown. After cooldown, the model maintains stable generation behavior under natural prompts with no metadata prefix.

Stage 2: Traditional-Chinese mid-training

Stage 2 shifts the model distribution toward Traditional Chinese and the local context. This stage’s data goes through stricter deduplication, quality filtering, and replay mixing. The main goals: lower Traditional-Chinese validation loss; raise localized probability-probe scores; keep general language ability from degrading substantially; and provide a stable checkpoint for the subsequent long-context extension.

P2b is an unmarked cooldown after mid-training. The learning rate continues to decay from the end of mid-training, giving the model a more stable language distribution before entering long-context training.

Stage 3: progressive long-context extension

Stage 3 progressively extends the sequence length from 8K to 256K:

8K → 32K → 64K → 128K → 256K

This stage has seven core principles. The RoPE base frequency stays at 10,000,000. The sliding-window layers keep their 8192 window. Global-attention layers retain full causal attention. The proportion of long-document data rises with each stage. Short-context replay is kept to avoid short-text regression. Each stage continues from the previous stage’s checkpoint. Promotion gates are decided by long-document perplexity and the magnitude of short-context regression.

Context parallelism increases with sequence length: 4 at the 32K stage, 4 at 64K, 8 at 128K, and 16 at 256K. Long-context training also enables full uniform recomputation to relieve activation-memory pressure.

Long context: from native 256K to 1M extrapolation

The native 256K training target

Barbet 1B Base’s native training-target context length is 256K tokens. The model is configured as:

{
  "max_position_embeddings": 262144,
  "rope_theta": 10000000,
  "sliding_window_size": 8192
}

Long-context ability is supported jointly by several design choices: progressive context training; high-base-frequency rotary position embeddings; one global-attention layer every four layers; local sliding windows; the Mamba sequence mixer; a long-document training-data mixture; and short-context replay for regression control.

The 1M research extrapolation configuration

The 1M configuration is an inference-time linear RoPE extrapolation, not native training:

{
  "max_position_embeddings": 1048576,
  "rope_scaling": {
    "type": "linear",
    "factor": 4.0,
    "original_context_length": 262144
  }
}

256K is the native pretraining context. 1M is a RoPE-extrapolation configuration of the same weights. RoPE introduces no learnable position parameters, so the weight shapes are compatible. But an actual 1M prefill requires an optimized long-context runtime; under a naive implementation, global-attention layers remain quadratic.

This distinction is critical. Even if needle-in-a-haystack tests partially succeed under the 1M extrapolation configuration, one cannot claim the model reliably understands the full 1M context.

Evaluation

Evaluation design principles

Barbet 1B Base uses multi-layer validation. The purpose of each validation layer:

How to compare across tokenizers correctly

Token perplexity across different tokenizers is not directly comparable. Barbet uses byte-normalized loss (bits per byte, BPB) as the cross-tokenizer-comparable metric. This ensures models with different vocabulary sizes are compared on a fair basis.

Probability probing

Probability probing uses paired-continuation scoring. For each probe item, the model computes the average log-probability of a positive continuation and a negative continuation; if the positive is higher than the negative, it counts as correct. This evaluation can output paired accuracy, per-category accuracy, the positive–negative gap, and a stagewise regression check.

Results

The results show several key points. The model is stable in HuggingFace loading and conversion correctness. Its greedy-decoding behavior is very close to the Megatron checkpoint. It has long-context retrieval ability at the 256K native context. It retains partial retrieval ability under the 512K and 1M extrapolation configurations. On byte-normalized loss, it can serve as a follow-up research checkpoint for a Traditional-Chinese / multilingual base model.

A caveat on long-context evaluation

Needle-in-a-haystack-style tests mainly measure exact retrieval. They cannot prove that the model fully understands 1M tokens, has robust multi-hop reasoning, can stably handle all long-text tasks, nor can they equate 1M extrapolation with native 1M training. This article therefore strictly distinguishes “native-context evaluation” from “extrapolation-context evaluation” in all results.

TAIDE-14 cross-model comparison

To compare fairly across models of the same scale, we evaluate Barbet 1B Base against three billion-parameter public models — openbmb/MiniCPM5-1B-Base, meta-llama/Llama-3.2-1B, and LiquidAI/LFM2.5-1.2B-Instruct — on the TAIDE-14 task set (taide/TAIDE-14-tasks; 14 tasks, 140 samples total) using byte-normalized loss (bits per byte, BPB). No samples were truncated for any model. BPB is the negative log-likelihood of the target tokens divided by the UTF-8 byte count of the evaluation text; it removes the effect of differing tokenizer vocabulary sizes and tokenization granularity on perplexity, making cross-model comparison meaningful.

Under the response_only protocol (scoring the loss of the positive response text only, excluding the prompt prefix), the overall ranking is:

Barbet leads MiniCPM5 (0.7577) at 0.7488 bits/byte, a margin of 0.0089; the gap to Llama-3.2-1B is 0.1702. Note the tokens/byte column: PangolinTokenizer splits each byte into 0.2164 tokens — the lowest of the four models — meaning higher compression efficiency for Traditional Chinese, with each token carrying more information on average.

Per task, Barbet ranks first on 10 of the 14 tasks:

The four tasks where Barbet ranks second (writing, letter writing, recommendation, advice giving) are all won by MiniCPM5, but most gaps are within 0.02 bits/byte. To be clear, what is compared here is base-model language-modeling efficiency, not post-alignment assistant performance.

Design trade-offs and reflections

The design of Barbet 1B Base reflects several practical trade-offs for billion-parameter models in Traditional-Chinese, multilingual, and long-context scenarios.

First, reallocating vocabulary parameters. Traditional-Chinese and multilingual tokenizers usually have large vocabularies. Tying the embedding layer to the output head and investing the saved parameters into a deeper model body is a direct way to raise model capacity under a fixed parameter budget.

Second, cost control through hybrid attention. Pure global attention is too costly at 256K sequence length. By interleaving global, sliding-window, and Mamba layers, the model retains a periodic global information channel while confining most layers’ compute pressure to a local window. During step-by-step decoding, Mamba layers have state behavior distinct from an attention KV cache, providing additional sequence-mixing diversity.

Third, the pragmatic choice of progressive long-context extension. Direct native 1M training carries too much cost and stability risk. Progressive extension gives each stage a clear promotion gate. The 1M configuration is clearly positioned as an extrapolation add-on, not a native-training claim. This distinction is essential to avoid over-claiming.

Fourth, engineering transparency. The technical report fully discloses the pipeline, data governance, tokenizer contract, training recipe, evaluation protocol, and model card. This lets later researchers reproduce the process, verify conversion correctness, and build downstream fine-tuning on top of it.

Safety, ethics, and limitations

Barbet 1B Base is a base model that has not been instruction-tuned or safety-aligned. The model may exhibit: hallucination; harmful continuations; prompt drift; repetition; unsafe generation; memorization-like behavior; biased or culturally insensitive continuations; and an inability to refuse harmful instructions.

Therefore, any user-facing application requires additional instruction tuning, safety tuning, red-teaming, and policy enforcement. Direct deployment as a conversational assistant is not recommended, nor is use in medical, legal, financial, or other domains requiring professional guarantees.

The 1M extrapolation configuration must also not be mislabeled as native 1M pretraining. Even if needle-in-a-haystack partially succeeds under the 1M configuration, it does not mean the model can reliably understand the full 1M context.

Conclusion

Barbet 1B Base is a billion-parameter hybrid causal language model produced by the Open Formosa training stack. The model supports context up to 1M tokens (256K native training length, RoPE-extrapolated to 1M), uses an interleaved design of global attention, sliding-window attention, and a Mamba sequence mixer, and shifts the parameter cost of a large vocabulary into a deeper model body through embedding tying. The training pipeline spans general pretraining, Traditional-Chinese mid-training, and progressive long-context extension, with a fixed tokenizer contract, strict data governance, stable training gates, multi-layer evaluation, and Megatron-to-HuggingFace conversion verification.

The model should be regarded as a research base model. It is not a conversational assistant, and it should not be claimed to be a native 1M pretrained model. Future work includes: instruction tuning; safety alignment; quantization-aware-training experiments; byte-normalized evaluation on more held-out sets; long-context reasoning tests; and stricter conversion-consistency and inference-runtime verification.

@techreport{barbet1bbase2026,
  title     = {Barbet 1B Base: A Hybrid Decoder-Only Causal Language Model
               for Traditional Chinese, Multilingual Pretraining,
               and Long-Context Modeling},
  author    = {Open Formosa / Barbet Contributors},
  year      = {2026},
  institution = {Open Formosa},
  note      = {Training data sources are not disclosed.}
}