Supervision order changes the trade-off
Synthetic speech improves content accuracy but can weaken speaker similarity; real speech can restore it.
LOW-RESOURCE SPEECH SYNTHESIS · RESEARCH PROJECT
From Reliable Text
to Real Voices
Trust-Aware Progressive Adaptation for Low-Resource TTS
1 Beijing Logic Intelligence Technology 2 University of Washington
3 Beijing University of Posts and Telecommunications
4 University of California, USA 5 Northwestern University, USA
01Learn the languageSynthetic text–speech pairs
02Recover the voiceReliability-weighted real speech
2 languages. 2 backbones.One progressive adaptation framework.
01 / OVERVIEW
Learn pronunciation.
Keep the speaker.
The challenge of low-resource TTS
Synthetic speech offers pronunciation supervision. Real recordings bring natural prosody and diverse voices. Their order—and the trust placed in each transcript—shape what a model learns.
Adapting a multilingual TTS model to an underrepresented language requires accurate pronunciation and control over an unseen reference speaker. Synthetic-only adaptation can improve content accuracy while reducing speaker similarity. Real-speech adaptation can restore similarity, but noisy ASR transcripts may erode those pronunciation gains.
We first adapt on synthetic pairs (S-stage), then on real recordings (R-stage). Agreement between two ASR systems determines how much each real example contributes to training.
Read the abstract
Low-resource text-to-speech (TTS) adaptation is constrained by scarce paired data and costly manual transcription. Existing fixed-voice TTS systems can provide relatively accurate pronunciation, but their synthetic speech offers limited speaker diversity and may exhibit flat prosody. Real recordings provide natural prosody and diverse voices, yet their automatic speech recognition (ASR) pseudo-labels may contain transcription errors. We find that supervision order affects content accuracy and speaker similarity. We propose trust-aware progressive adaptation: synthetic-to-real adaptation first establishes text–speech correspondences, then restores reference-speaker control using real speech. Transcript-agreement weighting uses agreement between two fixed ASR systems as a proxy for pseudo-label reliability to limit noisy supervision. Experiments with FireRedTTS3 on Burmese and Lao and OmniVoice on Burmese show improved content accuracy with high naturalness and competitive speaker similarity. Jointly considering supervision order and pseudo-label reliability when combining synthetic and real speech offers a practical path to zero-shot voice cloning in low-resource languages with less manual transcription. Audio demos are available at the project page.
Trust-aware progressive adaptation
Synthetic-to-real adaptation uses transcript-agreement weighting on pseudo-labeled real speech to limit noisy supervision.
Content gains and reference conditioning
Independent ASR corroborates gains across languages and backbones; FireRedTTS3 interventions probe acoustic history.
02 / AUDIO DEMOS
Hear the difference.
Compare the same target text across adaptation stages. Listen for pronunciation, naturalness, and how closely the generated voice matches the reference.
One example per language, selected by the highest cubic overall listening score in the available collection. Each comparison uses the same target text and reference voice. View full evaluation results ↓
03 / METHOD
Two stages. Complementary supervision.
Establish target-language text–speech correspondences first. Then recover reference-speaker control with real recordings, reducing the influence of uncertain transcripts.
Adapt with synthetic pairs
A fixed teacher synthesizes target-language text. Quality filtering and same-voice reference pairing create synthetic supervision. Adapt the base model with unit sample weights.
Measure character disagreement
Gemini 2.5 Flash supplies the primary text; Omnilingual ASR supplies the comparison. Normalize both, then compute code-point edit distance relative to the primary transcript length.
dᵢ = ED(N(x̃ᵢ), N(A₂(yᵢ))) / max(1, |N(x̃ᵢ)|)
Adapt with weighted real speech
Use cubic decay with a 0.10 floor. Scale each complete sample loss, then average by the number of examples. The ASR scoring is offline and adds no ASR computation at inference.
wᵢ = max(0.10, [1 − min(dᵢ, 1)]³)
Agreement is a proxy for reliability, not verified correctness: both ASR systems can share errors. Normalization uses Unicode NFC and removes whitespace and punctuation/symbol/control categories, without case, numeral, or Zawgyi conversion. Empty primary labels are excluded; a successful empty secondary transcript gives d = 1 and is distinct from an ASR failure. No transcript fusion or correction is performed. Backbone architectures and objectives are retained; optimizers and schedules restart between stages. FireRedTTS3 weights the complete flow + 0.1 × stop loss; OmniVoice weights masked acoustic-token prediction.
Data and training protocol Paper §3.1 +
Real recordings also undergo duration, single-speaker, audio-quality, and transcript-validity screening in our released video pipeline before agreement weighting.
Burmese
S: 160,918 Azure Nilar/Thiha pairs (291.148 h). R: 48,043 DVB pairs (92.263 h). S/R training uses 53,640 / 16,015 updates.
Lao
S: 119,246 pairs (200.004 h). R: 26,004 pairs (45.117 h). S/R training uses 39,749 / 8,668 updates.
Controlled adaptation
One device, effective batch 3, AdamW, 3% warmup, cosine decay, S/R learning rates 2 × 10⁻⁶ / 10⁻⁶. Dev120 selects γ = 3 and a 0.10 floor by three-seed mean CER subject to SIM-O ≥ 0.6970.
04 / EXPERIMENTAL RESULTS
Content accuracy meets voice similarity.
The cubic S→R method achieves the highest reported H and naturalness MOS in all three evaluated language–backbone groups. These are numerical rankings, not evidence of significance. Adapted CER and SIM-O report mean ± sample SD over three training seeds.
OMNIVOICE · BURMESE6.85%
CER after cubic S→R
10.35% for OmniVoice Base · 3.50-point reduction
FIREREDTTS3 · BURMESE0.6997
Speaker similarity after cubic S→R
0.4513 after S-stage · Table 1
FIREREDTTS3 · LAO3.97/5
Naturalness MOS after cubic S→R
3.39 after S-stage · Table 1
| Adaptation strategy | CER (%) ↓ | SIM-O ↑ | H ↑ | MOS [95% CI] ↑ | Independent-ASR CER (%) ↓ |
|---|
How to read: CER is character error rate (lower is better). SIM-O measures speaker similarity. H is the equal-weight harmonic mean of character accuracy and SIM-O on a 0–100 scale. MOS measures naturalness on a 1–5 scale. MOS brackets give approximate 95% crossed-bootstrap confidence intervals. Bold marks the numerical best in each column. Independent-ASR CER uses the same audio and target texts, scored by recognizers excluded from labeling, weighting, filtering, and selection. Compare CER within each language and scorer.
External Burmese systems Table 2 +
| System | CER (%) ↓ | SIM-O ↑ |
|---|---|---|
| Fish Audio S2-Pro | 83.89 | 0.6376 |
| MMS-TTS-mya * | 20.49 | 0.1029 |
| F5-Myanmar-TTS v2 | 35.24 | 0.5863 |
| IMS-Toucan | 95.40 | 0.3298 |
| mmSpeech Tacotron * | 78.70 | 0.0914 |
| OmniVoice Base | 10.35 | 0.7278 |
| FireRedTTS3 cubic | 16.90 | 0.6997 |
| OmniVoice cubic | 6.85 | 0.7098 |
* Fixed-voice models: SIM-O is descriptive. All outputs remain scored, including 210/400 mmSpeech Tacotron outputs reaching its 12.54 s limit and IMS-Toucan outputs with Burmese phoneme warnings. The selected audio examples above include FireRedTTS3 cubic and OmniVoice Base.
What makes the weighting work? Table 3 +
Burmese FireRedTTS3 controls from S: mean ± sample SD over three training seeds. Shuffle results are averaged within each seed.
Keep the recordings. Change the labels.
On 2,921 matched FLEURS recordings (11.578 h), official text reduces CER by 8.22 points versus ASR pseudo-labels at uniform weight 1.0.
| Setting | CER (%) ↓ | SIM-O ↑ |
|---|---|---|
| S starting point (Table 1a) | 17.43 ± 0.45 | 0.4513 ± 0.0041 |
| Pseudo-labels · w = 0.5 | 21.17 ± 0.38 | 0.6345 ± 0.0024 |
| Pseudo-labels · w = 1.0 | 23.67 ± 0.19 | 0.6487 ± 0.0009 |
| Official text · w = 1.0 | 15.45 ± 0.15 | 0.6342 ± 0.0012 |
Assignment matters beyond loss scale.
Cubic weighting outperforms shuffled weights and equal-mean uniform weighting under matched broadcast training conditions.
| Setting | CER (%) ↓ | SIM-O ↑ |
|---|---|---|
| Uniform 1.0 (Table 1a) | 23.27 ± 0.15 | 0.6942 ± 0.0014 |
| Uniform 0.5 | 20.91 ± 0.37 | 0.6996 ± 0.0005 |
| Linear · γ = 1 | 23.25 ± 0.49 | 0.6956 ± 0.0030 |
| Cubic · globally shuffled | 21.34 ± 0.52 | 0.6984 ± 0.0010 |
| Cubic · duration-stratified shuffle | 21.13 ± 0.22 | 0.6988 ± 0.0032 |
| Uniform · equal mean | 21.09 ± 0.55 | 0.6993 ± 0.0008 |
| Cubic · γ = 3 | 16.90 ± 0.26 | 0.6997 ± 0.0016 |
On the 2,921-utterance Burmese FLEURS label-control subset, transcript disagreement correlates with label CER against official text (Spearman ρ = 0.560; Pearson r = 0.546). Mean label CER is 5.94% at d < 0.1 (n = 971), versus 32.30% at d ≥ 0.6 (n = 286). On the full Burmese FLEURS training split, both ASRs produce identical but incorrect normalized transcripts for 3.2% of utterances.
Mixtures, reference conditioning & acoustic history Paper §4 +
Progressive adaptation
The real-example ratio sweep uses 0/10/25/50/75/90/100% real data. Synthetic replay restores less similarity than real-only continuation. Exposure-matched joint training from Base yields 20.160% CER / 0.65120 SIM-O.
Reference response
Embedding-switch response across 16 texts is 1.3147 / 0.9602 / 1.2723 for Base / S / historical uniform 0.5 S→R. Matched-reference SIM-O is 0.7804 / 0.5672 / 0.7625. Real adaptation restores response weakened by S; cubic was not tested in this intervention.
Acoustic history
Recorded rather than generated prefixes improve suffix SIM-O by 0.0739 / 0.0211 for S / historical uniform 0.5 S→R over eight eligible pairs. Gains occur in 7/8 pairs for each model; paired 95% intervals are [0.0225, 0.1204] / [0.0025, 0.0374]. These case-conditioned results show history dependence, not naturally accumulating voice drift. Cubic was not tested.
No corresponding intervention or ratio-sweep audio is available in the supplied collection. These results are reported from the manuscript and are not inferred from the selected listening clips.
Interpreting MOS. Paired cubic-minus-uniform-0.5 differences are +0.210 [0.025, 0.392] for FireRedTTS3/Burmese, +0.130 [−0.047, 0.315] for FireRedTTS3/Lao, and +0.090 [−0.052, 0.230] for OmniVoice/Burmese. Only the first marginal 95% interval excludes zero. These exploratory comparisons are unadjusted for multiplicity; the paper does not claim consistent MOS improvement across settings.
Evaluation scope. Burmese uses Common400 for CER and Clone300 for speaker similarity; Lao uses FLEURS404 with one reference. Adapted CER/SIM-O use seeds 42/17/73. Per language, 20 listeners rated 30 matched text–reference conditions per system (600 ratings/system). MOS uses 10,000 crossed-bootstrap replicates over listeners and conditions; listening audio uses seed-42 checkpoints. Audited Clone300 speakers are disjoint from adaptation speakers; audio near-duplicate auditing remains incomplete. The two selected examples illustrate synthesis quality; the tables report full evaluation results.
05 / RESOURCES & CITATION
Explore the research.
Read the manuscript, explore the reference implementation, listen to the examples, or cite this work.
Paper PDFLatest manuscript · 5 pages · updated 23 Sep 2026arXiv preprintarXiv:2609.25951 · v1, 22 Sep 2026Selected audio2 selected texts · 11 comparison WAV filesDemo metadataText, language, model labels & file hashesGitHub repositoryTraining, preprocessing & website source · private repositoryModel weights · Hugging FaceOmniVoice Burmese · cubic · native routed LoRA & inference guide
BIBTEX
@misc{lu2026reliable,
title = {From Reliable Text to Real Voices:
Trust-Aware Progressive Adaptation
for Low-Resource TTS},
author = {Lu, Jiayi and Geng, Yizhong and
Yang, Jinghan and Jiang, Tianhan and An, Boxun and Gao, Yingming and Li, Ya},
year = {2026},
eprint = {2609.25951},
archivePrefix = {arXiv},
primaryClass = {cs.SD},
url = {https://arxiv.org/abs/2609.25951}
}arXiv v1 was submitted 22 Sep 2026. The repository PDF includes the later 23 Sep revisions.