Voice quality is one of the main characteristics of speech transmission systems. When analyzing voice quality one must not only consider audio signal degradation caused by transmission over telecom channels, but also specifics of speaker’s voice, conditions of listener’s hearing and variation of these parameters in time.

The most known methods for quality evaluation of voice transmission systems were developed by Telecommunication Standardization Sector of International Telecommunications Union (ITU-T) in the middle of 90-s. Results of this work are presented in Recommendation P.800 (P.830) «Methods for subjective determination of transmission quality» [1, 2]. This document describes conditions for voice quality testing, audio contents, scoring and methods to evaluate results. Typically “Methods for subjective determination of transmission quality” are used to obtain mean subjective quality score according to five-digit scale (Mean Opinion Score – MOS).

Unfortunately P.800 recommendation tests may lead to ambiguous results. Recommendation is warning about comparing MOS scores received under different conditions and consider such approach incorrect. Besides that preforming tests according to P.800 takes a lot of time and requires a lot of testers involved in the process.

In order to move from subjective (MOS) scores to objective ones and to automate the quality measurement, ITU-T has developed the P.861 recommendation, which is based on low level quantitative measurements [3]. Recommendation P.861 is a follow-up of PSQM method (Perceptual Speech Quality Measurement), developed by KPN Research and devoted to objective analysis of speech codecs performance with a low level of degradation.

However, it is impossible to utilize PSQM for evaluation of work of a real communication system because the method does not consider all the important factors influencing human perception. Among these factors are delay, jitter, packet loss as well as signal level clipping.

In February 2001 ITU-T has issued another recommendation ITU-T P.862 [4], which describes a more advanced algorithm for voice quality testing – PESQ (Perceptual Evaluation of Speech Quality). The algorithm includes level and time aligning, human perception and cognitive modeling. Due to these additional operations the approach considers signal amplification/ attenuation in a communication system, time delays and jitter as well as spectrum bands, which are the most significant for human perception. Based on cognitive modeling PESQ also recalculates objective quality score into MOS values.

A disadvantage of PESQ as well as other similar algorithm is the fact that they are based on comparing of two signals: original and transmitted through a communication system. This approach may create a range of difficulties connected with setting and preforming voice quality testing. One requires to arrange signal recording on both sides of the telecommunication system as well as records transmission to the test system. Besides this real time quality monitoring in such approach appears quite difficult as well.

In order to solve the challenging issues mentioned above ITU-T has developed a new recommendation P.563 [5] introduced in May 2004. This recommendation determines algorithm for evaluating speech quality by listening to communication sessions. The algorithm takes into account single-side distortions, speech trunk parameters, noise and speech naturalness. Developers of P.563 call attention that P.563 does not provide overall quality estimation of speech transmission. Distortions driven by delays, echo, loss of loudness and everything related to two-sided interaction cannot be taken into consideration by this method.

It’s widely thought that P.563 provides a high level of correlation between automated and expert quality scores. However, simple tests based on ITU-T sound database for codec testing [6] may raise some doubts about the consistence of the algorithm provided together with its description.

Table.1. Comparison between results of P.563 and expert estimations

The problem discovered in the distributed P.563 algorithm implementation required development of an alternative solution. Further down one can find one of possible solutions that is implemented in Sevana NIQA (Non-Intrusive Quality Analyzer).

NIQA’s (Non-Intrusive Quality Analyzer) approach is based on a database of trained etalons called associations. Each association corresponds to a group of files that have close expert estimations of sound quality and common set of reasons for sound quality degradation. For each association NIQA calculates and stores a distribution of parameters’ values.

Basic algorithm showing how NIQA obtains sound quality scores is represented on the picture below.



NIQA - non-intrusive voice quality testing software

When loading sound signal the system excludes all fragments with low energy level (according to threshold). The excluded fragments correspond to “absolute silence” and are considered irrelevant for obtaining sound quality score.

At the next phase the signal is split into frames used in voice activity detection algorithm (VAD). The system calculates energy values for each frame what increases accuracy of VAD. With the help of VAD algorithm the signal divides to active and inactive components that are processed separately. The system builds level histograms for both active and inactive signal components.

By discrete cosine transform (DCT) the system obtains signal spectrum and checks the active components frames for DTMF presence and then excludes the frames that are similar to DTMF from further processing.

Next stage applies the first level of psycho-acoustic model to the signal spectrum. This model checks different types of masking (including pre-masking and post-masking). According to clear peaks of spectrum energy the system splits the signal into tone and noise components.

Second level of psycho-acoustic model performs energy normalization of the signal – energy levels are transformed into loudness levels at 1kHz. Third level of psycho-acoustic model transforms loudness levels into several detectable grades of loudness that allow to ignore sound signal changes, which are not recognized by human ear.

The next step is to split signal spectrum into bands that are critical to human ear perception and calculate parameters both on and out of the bands. Based on the computed signal parameters the system selects most similar associations from the database and performs matching. According to selected associations the system determines how much each of them influence the overall quality and then generates the final voice quality score as a combination of scores for selected associations and according to correspondent weights.

Sevana NIQA Testing and Evaluation

Sevana NIQA has been tested utilizing the same ITU-T speech database that is used for conformance testing of P.563 algorithm. In the tests we used a total of 376 English language recordings. All recordings were sorted into 4 groups depending on their MOS scores (represented in the documentation attached to the sound database). For all groups of recordings we determined average expert scores and average NIQA scores (Table 2). In order to illustrate comparison with P.563 we also calculated average errors for P.563 and NIQA scores for the same tests.

Table.2. Comparison of NIQA scores against expert estimations

The results clearly show that NIQA allows receiving much higher accuracy between generated quality scores and expert estimations than P.563. NIQA scores are less precise only for records with very low MOS scores (in the range from 1 to 2). In all other cases NIQA provides 2-3 times higher quality scores precision compared to MOS values.

References

1. Methods for subjective determination of transmission quality // ITU-T Recommendation P.800 / http://www.itu.int/rec/T-REC-P.800/en

2. Subjective performance assessment of telephone-band and wideband digital codecs // ITU-T Recommendation P.830 / http://www.itu.int/rec/T-REC-P.830/en

3. Objective quality measurement of telephone-band (300-3400 Hz) speech codecs // ITU-T Recommendation P.861 / http://www.itu.int/rec/T-REC-P.861/en

4. Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs // ITU-T Recommendation P.862 / http://www.itu.int/rec/T-REC-P.862/en

5. Single-ended method for objective speech quality assessment in narrow-band telephony applications // ITU-T Recommendation P.563 / http://www.itu.int/rec/T-REC-P.563-200405-I/en

6. ITU-T coded-speech database // Supplement 23 to ITU-T P-series Recommendations / http://www.itu.int/rec/T-REC-P.Sup23-199802-I/en

Copied from: http://www.sevana.fi/non-intrusive-voice-quality-testing-software.php

Non-intrusive voice qualiy testing does not require a reference signal and can work with real communications data in real time.

Non-intrusive techniques provide a possibility to test and monitor a greater amount of communications and therefore obtain a quite reliable information about the networks (VoIP, PSTN, GSM, CDMA, LTE) quality.

Today Sevana announces availability of new software for voice quality testing for customers’ and partners’ evaluation. Be among the first to test the new approach to voice quality testing and monitoring!

This software is delivered only upon request.