BlueMagpie-TTS: Taiwanese-accent, Chinese–English code-switching speech synthesis

Key points

• Keep the acoustics, swap the brain — retain VoxCPM’s pretrained acoustic stack as a whole, and replace only the text-semantic model that “decides what to say” with Barbet, joined by a bridge module.
• Built for the Taiwanese context — it targets two often-overlooked needs at once, Taiwanese-accent Chinese and Chinese–English mixing (code-switching), so a single model handles local accent and code-switching naturally.
• Listenable and verifiable — quality is measured by a closed loop: synthesize with TTS → transcribe back with Breeze-ASR-25 → compare character by character. The interactive demo below lets you hear it and see the transcription yourself.
• Honest boundaries — it is not a review-free, production-grade system; output can still be wrong, and reference audio and speaker vectors must be authorized before they can be used for synthesis or distribution.

Listen first

Calling a speech model “good” means nothing — you have to hear it. The sentences below come from a 500-sentence set of “hard” Chinese sentences, deliberately seeded with English words, abbreviations, numbers, and proper nouns: exactly where everyday Taiwanese speech applications tend to break.

Our evaluation is a closed loop: hand the text to BlueMagpie-TTS to synthesize speech, hand the speech to Taiwan’s Breeze-ASR-25 speech-recognition model to transcribe it back to text, and compare character by character. How far the two differ is the character error rate (CER).

The last sentence is itself about how TTS tends to slur “open” and “AI” into one strange sound — and the model reads it correctly, with Breeze-ASR-25 cleanly recovering “OpenAI”, “TTS”, and “AI”. That is exactly the code-switching problem this model is meant to solve.

Want to try it yourself? Open the live demo (Hugging Face Space), drop in a Chinese–English mixed sentence, and hear the result in real time.

It still makes mistakes

Defining the boundaries matters as much as defining the uses. The same hard test set also contains cases the model does not handle well. In the line below, LLM is not pronounced cleanly enough, and ASR hears it as “LOL and” — the boundaries of English abbreviations like this are still one of the model’s weak spots.

1. What this is

1.1 Why build this model

Speech applications in Taiwan have two often-overlooked needs: a Taiwanese accent, and Chinese–English mixing.

Take code-switching first. A single utterance can contain Chinese, English words, abbreviations, and proper nouns at once — this is the norm in Taiwan, but a hard problem for speech synthesis. Most off-the-shelf TTS models do well on pure Chinese or pure English, but stumble at the boundaries of code-switching.

Now the accent. Most models’ Chinese leans toward other Mandarin accents, and does not sound the way people in Taiwan speak.

BlueMagpie-TTS targets both at once. Its goal is simple: let a single model naturally handle Taiwanese-accent Chinese and Chinese–English mixed speech generation.

1.2 What it can do

The model supports three usage scenarios, plus a streaming mode.

The most common is the first: feed in some text, get back speech. Everything else is optional advanced control.

1.3 What it cannot do

Defining the scope is as important as defining the use. Keep a few bottom lines in mind before using it.

First, it is not a review-free, production-grade system. Generated speech can be wrong, and without human review it should not be used directly for real-world notifications or public playback.

Second, authorization is a hard rule. The bundled Hung-yi Lee speaker vector is authorized and can be used directly as an example. But to clone anyone else’s voice, or to use any other speaker vector, you must first obtain that person’s permission. Speaker-vector tables and the synthesized audio must not be distributed publicly without authorization.

1.4 Where the name comes from

The full project name is OpenFormosa Blue Magpie TTS. “Blue Magpie” is taken from the Taiwan Blue Magpie (Urocissa caerulea). Choosing it as the mark has three layers of meaning: the Taiwan Blue Magpie has a loud, highly recognizable call, echoing the heart of TTS — turning text into sound; its signature long tail brings a visual sense of flow and extension; and OpenFormosa (Formosa) points to the project’s footing in Taiwan and its focus on Taiwanese Mandarin.

2. What the model looks like

2.1 The core idea

A typical TTS model is one solid block: text goes in, speech comes out, and everything in between is trained together.

BlueMagpie-TTS takes a different path. It keeps an already-trained acoustic stack with good audio quality intact as a whole, and swaps out only the brain responsible for “deciding what to say,” replacing it with Barbet.

The benefit is direct. Barbet brings text understanding and prosody planning; the acoustic stack retains the pronunciation detail it has already accumulated. Each does its own job.

2.2 Two off-the-shelf parts

BlueMagpie-TTS does not reinvent the wheel; it combines two off-the-shelf parts.

Barbet is the text-semantic language model, from OpenFormosa/Barbet. It is installed automatically from GitHub when you install this project.

The acoustic module is taken from VoxCPM2 (OpenBMB, Apache-2.0). It is already bundled in the project (under bluemagpie/_vendor/) and needs no separate installation.

The two have incompatible internal formats. The bridge module’s job is to translate one side’s output into a form the other understands, so the interfaces connect.

3. Installation

Clone the project, then install in editable mode. The dependent Barbet package is installed automatically.

git clone https://github.com/OpenFormosa/BlueMagpie-TTS
cd BlueMagpie-TTS
pip install -e .

To save the synthesized audio as .wav, also install soundfile:

pip install soundfile

4. How to use it

This section is the heart of it. Below is how to load the model, synthesize speech, clone a voice, specify a speaker, and stream output.

4.1 Load the model

Download from Hugging Face and load:

import os
from huggingface_hub import snapshot_download
from transformers import PreTrainedTokenizerFast
from bluemagpie import BlueMagpieModel

model_dir = snapshot_download("OpenFormosa/BlueMagpie-TTS", token=True)
# Load the tokenizer straight from tokenizer.json; compatible with newer transformers (5.x)
tokenizer = PreTrainedTokenizerFast(tokenizer_file=os.path.join(model_dir, "tokenizer.json"))
model = BlueMagpieModel.from_local(model_dir, tokenizer=tokenizer, training=False, device="cuda")

If you already have a local copy of the model files, point model_dir at that directory; everything else is the same. device can be "cuda" or "cpu", and is chosen automatically if unset. Always use training=False for inference.

4.2 Basic synthesis: text to speech

The most basic use: give some text, get back speech. target_text can mix Chinese and English directly — the model handles the switching itself.

import soundfile as sf

audio = model.generate(
    target_text="這是 AI TTS code switching 測試。",
    cfg_value=2.8,
    inference_timesteps=9,
    max_len=2000,
    retry_badcase=True,
)
sf.write("output.wav", audio.squeeze().cpu().numpy(), model.sample_rate)

This uses the recommended parameters (cfg_value=2.8, inference_timesteps=9). The code’s original defaults are actually 2.0 and 10, but the former is recommended; see the parameter table below for why.

4.3 Voice cloning: imitate a speaker from a reference clip

Given a reference_wav_path, the output imitates the speaker timbre of that clip.

audio = model.generate(
    target_text="今天的會議改到下午三點。",
    reference_wav_path="speaker.wav",
    cfg_value=2.8,
    inference_timesteps=9,
)

A reminder, again: only use voices you have the right to synthesize.

4.4 Specified speaker: control timbre with a speaker vector

The model ships with Prof. Hung-yi Lee’s speaker vector as an example, used with his permission, stored at checkpoints/hung_yi_lee_speaker_centroids.pt in the model directory. Load the vector table, pull the vector for speaker ID hung_yi_lee, and pass it via speaker_centroid to set the timbre.

import os
import torch

centroids = torch.load(
    os.path.join(model_dir, "checkpoints", "hung_yi_lee_speaker_centroids.pt"),
    map_location="cpu",
    weights_only=True,
)
speaker_centroid = centroids["centroids"][centroids["speaker_ids"].index("hung_yi_lee")]

audio = model.generate(
    target_text="今天天氣真好。",
    speaker_centroid=speaker_centroid,   # or pass your own authorized speaker vector
    cfg_value=2.8,
    inference_timesteps=9,
)

4.5 Streaming output

When you need to play while synthesizing, use generate_streaming. It is a generator that returns audio chunks one at a time.

chunks = []
for chunk in model.generate_streaming(target_text="今天天氣真好。"):
    chunks.append(chunk)
    # play or write out the chunk in real time here

Note: automatic retry (retry_badcase) is not supported in streaming mode.

4.6 Four input modes

The features above are really different parameter combinations of the same generate interface. The table summarizes the four modes.

4.7 Tuning the common parameters

Understand these few parameters and you can strike a balance between “stable” and “natural.”

The “Recommended” column is the officially tuned setting, also recorded in generation_defaults in config.json. This set was found using 500 hard Chinese sentences (the same batch demonstrated at the top of this article), together with Taiwan’s Breeze-ASR-25 speech-recognition model, optimizing normalized CER.

4.8 A few practical tips

For long text, split it into sentence-sized pieces, synthesize each, then concatenate the waveforms — adding a little fade in/out at the seams if needed. For more continuity, pass a short authorized clip from the previous segment as a prompt when synthesizing the next.

It runs without a GPU. Set device to "cpu"; it is slower, but short sentences synthesize in tens of seconds. Output is 48 kHz mono.

If you do not pass a tokenizer and rely on auto-loading, it may fail to load under transformers 5.x, or only error with “No tokenizer attached” when you call generate(). Load straight from tokenizer.json and pass it in, as in the example above, to avoid this.

5. How well it works

On the internal test set, BlueMagpie-TTS performs as follows. Lower is better.

Relative to the original reference model, character error rate drops by about 58.0% and word error rate by about 63.9%.

On generation speed, the median real-time factor is 4.748 and the maximum 5.288 (seconds of audio produced per second of compute; higher is faster).

One last emphasis: all the numbers above come from an internal test set, not a public benchmark, and are used only for internal judgment of model quality.

Appendix: files, license, and links

Files included in the model

License

The code is Apache-2.0. The model weights are under an “other” license with usage restrictions; the key point is that reference audio and speaker vectors must be authorized and consented to before they can be used for synthesis or distribution.

Links

• Live demo: https://huggingface.co/spaces/voidful/BlueMagpie-TTS-Demo
• Model: https://huggingface.co/OpenFormosa/BlueMagpie-TTS
• Code: https://github.com/OpenFormosa/BlueMagpie-TTS
• Text model, Barbet: https://github.com/OpenFormosa/Barbet
• Acoustic module, VoxCPM: https://github.com/OpenBMB/VoxCPM

Conclusion

BlueMagpie-TTS’s design fits in one sentence: keep the acoustic stack that works (VoxCPM), and swap out only the brain that decides “what to say” (Barbet). The former preserves pronunciation quality; the latter brings the ability to handle Taiwanese accent and Chinese–English mixing; the two are joined by a bridge module. For users, only two things matter: feed in text to synthesize, and give an authorized reference clip or speaker vector when you need a specific voice. Everything else is optional advanced control.