lingvo.tasks.asr.tools.simple_wer module

Stand-alone script to evalute the word error rate (WER) for ASR tasks.

THIS SCRIPT IS NO LONGER SUPPORTED. PLEASE USE simple_wer_v2.py INSTEAD.

Tensorflow and Lingvo are not required to run this script.

Example of Usage:

python simple_wer.py file_hypothesis file_reference
python simple_wer.py file_hypothesis file_reference diagnosis_html

where file_hypothesis is the file name for hypothesis text and file_reference is the file name for reference text. diagnosis_html (optional) is the html filename to diagnose the errors.

Or you can use this file as a library, and call either of the following:

  • ComputeWER(hyp, ref) compute WER for one pair of hypothesis/reference

  • AverageWERs(hyps, refs) average WER for a list of hypotheses/references

Note to evaluate the ASR, we consider the following pre-processing:

  • change transcripts to lower-case

  • remove punctuation: " , . ! ? (  ) [ ]

  • remove extra empty spaces

lingvo.tasks.asr.tools.simple_wer.ComputeEditDistanceMatrix(hs, rs)[source]

Compute edit distance between two list of strings.

Parameters

  • hs – the list of words in the hypothesis sentence

  • rs – the list of words in the reference sentence

Returns

Edit distance matrix (in the format of list of lists), where the first index is the reference and the second index is the hypothesis.

lingvo.tasks.asr.tools.simple_wer.PreprocessTxtBeforeWER(txt)[source]

Preprocess text before WER caculation.

lingvo.tasks.asr.tools.simple_wer._GenerateAlignedHtml(hyp, ref, err_type)[source]

Generate a html element to highlight the difference between hyp and ref.

Parameters

  • hyp – Hypothesis string.

  • ref – Reference string.

  • err_type – one of ‘none’, ‘sub’, ‘del’, ‘ins’.

Returns

a html string where disagreements are highlighted.

  • hyp highlighted in green, and marked with <del> </del>

  • ref highlighted in yellow

lingvo.tasks.asr.tools.simple_wer.GenerateSummaryFromErrs(nref, errs)[source]

Generate strings to summarize word errors.

Parameters

  • nref – integer of total words in references

  • errs – dict of three types of errors. e.g. {‘sub’:10, ‘ins’: 15, ‘del’: 3}

Returns

  • string summarizing total error, total word, WER,

  • string breaking down three errors: deleting, insertion, substitute

Return type

Two strings

lingvo.tasks.asr.tools.simple_wer.ComputeWER(hyp, ref, diagnosis=False)[source]

Computes WER for ASR by ignoring diff of punctuation, space, captions.

Parameters

  • hyp – Hypothesis string.

  • ref – Reference string.

  • diagnosis (optional) – whether to generate diagnosis str (in html format)

Returns

  • dict of three types of errors. e.g. {'sub':0, 'ins': 0, 'del': 0}

  • num of reference words, integer

  • aligned html string for diagnois (empty if diagnosis = False)

Return type

A tuple of 3 elements

lingvo.tasks.asr.tools.simple_wer.AverageWERs(hyps, refs, verbose=True, diagnosis=False)[source]

Computes average WER from a list of references/hypotheses.

Parameters

  • hyps – list of hypothesis strings.

  • refs – list of reference strings.

  • verbose – optional (default True)

  • diagnosis (optional) – whether to generate list of diagnosis html

Returns

  • dict of three types of errors. e.g. {'sub':0, 'ins': 0, 'del': 0}

  • num of reference words, integer

  • list of aligned html string for diagnosis (empty if diagnosis = False)

Return type

A tuple of 3 elements

lingvo.tasks.asr.tools.simple_wer.main(argv)[source]