Khmer Speech Recognition
Fine-tuning Whisper for Khmer, and why the data was the hard part.
Phanith LIM
Whisper transcribes English well. On Khmer it mostly guesses.
So I fine-tuned it: custom scripts, a dataset I put together, and a training pipeline that runs end to end. Almost all of the effort went into the data, and almost none into the model.
Why Khmer is awkward
Khmer runs words together. No spaces between them, and almost no punctuation. A sentence arrives as one unbroken string of characters.
That breaks an assumption most speech pipelines quietly make, which is that you can line words up with sounds because you can see where the words are. Here you cannot, so the normalization step below carries weight that it would not carry in English.
Cutting audio into training pairs
Whisper learns from short clips paired with the exact text spoken in them. My recordings were long, with a single transcript covering the whole file, so something has to decide where each word falls in time. That is forced alignment.
Two paths run in parallel and meet at the aligner.
The transcript side
- Start with the raw transcript.
- Normalize it, fixing punctuation, casing and formatting.
- Romanize the Khmer script into Latin characters.
- Run uroman normalization so the text matches what the model expects.
- Tokenize it into a token sequence.
The audio side
- Start with the waveform.
- Run it through the acoustic model.
- Read off the emission probabilities, one distribution over tokens per frame.
The aligner takes the token sequence and those probabilities and works out which token was being spoken in which frame. What comes back is every token with a timestamp and a confidence score.
The timestamps give the cut points. The confidence scores are the useful part: a clip the aligner is unsure about is usually a clip where the transcript and the audio disagree, and those are worth dropping before they teach the model something wrong.
Trying it out
The fine-tuned model runs in a small demo that takes a microphone or an uploaded file and returns the transcription. It is the screenshot at the top of this post.
The video walks through the results:
Khmer speech recognition is still far behind English, and a small model trained on a small dataset is not going to close that. What this does show is that the gap is mostly a data problem, and data problems are the kind you can work at.
Resources
- phanithlim/fine-tuning-whisper, the scripts and the pipeline
- Models and datasets on Hugging Face
- Report & Slide