Skip to main content

iApp ASR Pro Evaluation on Mozilla Common Voice 17.0

· 3 min read
Kobkrit Viriyayudhakorn
Kobkrit Viriyayudhakorn
CEO @ iApp Technology

iApp ASR preview

This report presents the evaluation results of the automatic speech recognition system (iApp ASR Pro) on the Mozilla Common Voice 17.0 dataset, measured as the share of words and characters transcribed correctly.

Dataset and Evaluation Metrics​

This evaluation was conducted on the Mozilla Common Voice 17.0 dataset, a widely used dataset containing diverse audio samples. It is used to test ASR accuracy. We used Word Error Rate (WER) and Character Error Rate (CER) as the main metrics for evaluating each ASR service.

WER and CER Calculation​

We used the jiwer library to calculate WER and CER, which are standard tools for ASR evaluation. Each metric is described as follows:

  • Word Error Rate (WER): Measures the error rate at the word level, calculated from the sum of word substitutions, deletions, and insertions, divided by the total number of words in the reference transcript.

  • Character Error Rate (CER): Measures the error rate at the character level, calculated in the same way as WER but at the character level, providing a more detailed accuracy measurement.

WER and CER calculation was performed using jiwer according to the following steps:

  1. Adjust the reference transcript and the transcript obtained from the ASR to be in the same format (e.g., lowercase, remove punctuation).

  2. Calculate WER and CER by comparing the results from each ASR with the reference transcript.

Accuracy Evaluation Table​

Measured on the Common Voice 17.0 Thai test set: 11,042 samples the models had not seen, with men, women and children among the speakers.

MeasureiApp ASR BaseiApp ASR Pro
Word error rate14.52%, so 85.48% of words correct8.01%, so 91.99% of words correct
Character error rate5.87%, so 94.13% of characters correct2.19%, so 97.81% of characters correct
Response time, typical0.3 sslower than Base; the accuracy is what it buys

Important Information:

  • iApp ASR Pro gets 91.99% of words right (word error rate 8.01%) and 97.81% of characters right (character error rate 2.19%).

  • iApp ASR Base gets 85.48% of words right (word error rate 14.52%) and 94.13% of characters right, and answers in 0.3 s in typical use.

Conclusion​

iApp ASR Pro trades speed for accuracy and gets 91.99% of Thai words right on this test set. iApp ASR Base answers in 0.3 s in typical use and gets 85.48% of words right. Both tiers take the same request and return the same response, so you can choose per call.

You can test iApp ASR PRO at https://iapp.co.th/docs/speech/speech-to-text/thai