We decided to use one of the most popular free SIP softphones – pjsip (www.pjsip.org). This is a cute, light, but powerful tool that can do the two main things required for creating a VQM system:

• functionality to make SIP calls – obviously all SIP phones have this functionality
• ability to play and record audio files

If you have a SIP software phone that supports these two features (and most likely any of them does) then by using Sevana’s AQuA or NIQA product you can setup a simple Voice Quality Monitoring (VQM) within a couple of minutes.

First of all you need to have a SIP account (although calling to an IP address is also possible). We have used free sip accounts provided by www.realsip.com. Then you should create a configuration file for your SIP-phone that will serve as an answering machine. This file may be as simple as the following:

# we don’t want the host’s audio device

#–null-audio

# SIP parameters

–realm realsip.com

–registrar sip:realsip.com # DNS SRV, or FQDN

–id sip:sevana@realsip.com

–username XXXXXX

–password YYYYYY

# default of 55 will be rejected as being too short by sipX

–reg-timeout 3600

# auto-answer all calls with “200 OK”

–auto-answer 200

# limit call duration – this maybe actually important if you like to automatically hang up after the test call is

# finished

#–duration 20

# automatically loop incoming RTP to outgoing RTP – maybe useful sometimes, but not this time

#–auto-loop

# mix WAV file into the audio stream

–play-file AE1F5901.wav # This is the audio that is going to be played to the calling party

# we would recommnd using Sevana speech model file that AQuA can generate, but this time we have

# chosen a sample test audio in French

# This command tells the softphone to record incoming calls into call.wav file stored in the same folder

–rec-file call.wav

# This is important command to tell the softphone that audio sampling rate should be 8kHz, because NIQA

# product can test only speech signals at 8kHz (although AQuA can test any type of audio: voice, HD Voice

# and even HD Audio)

–clock-rate=8000

# This command tells the system to automatically play the audio file we set (in this case AE1F5901.wav)

–auto-play

# And this command enables recording of incoming call

–auto-rec

# These are two important commands that set level of details in the log file (3 is just what we need, but you

# can check pjsip manual for other options), and the call log will be stored in log.txt – perfect!

–log-level=3

–log-file=log.txt

Amazingly, but for a simple voice quality monitoring solution the server part is pretty much ready! Let’s configure PJSIP for the calling party:

# SIP parameters

–realm realsip.com

–registrar sip:realsip.com # DNS SRV, or FQDN

–id sip:sevanaoy@realsip.com

–username XXXXXXX

–password YYYYYYY

# default of 55 will be rejected as being too short by sipX

–reg-timeout 3600

# limit call duration – alright, we want to hangup after 20 seconds

–duration 20

# mix WAV file into the audio stream

# Note, this is another audio that wll be played to our voice quality monitoring “server”

–play-file test.wav

# And this call.wav will be stored on the client side

–rec-file call.wav

–clock-rate=8000

–auto-play

–auto-rec

–log-level=3

–log-file=log.txt

Now let’s go for the first voice quality test! Yes, it’s that simple:

On the server side run:

pjsua –config-file=config.cfg

Wait till the server boots and switches to waiting for a call status. And then on the client side issue the command:

pjsua –config-file=config.cfg sip:sevana@realsip.com

You will see how the client will make a call, and the server will respond (even just by changes in the command line windows of the server and the client). Then after 20 seconds the calling party will hangup like it was written in the configuration file. Let’s see what we have got…

Client side:

We have a log.txt file containing important data about VoIP call parameters:

[DISCONNCTD] To: sip:sevana@realsip.com;tag=5eca32938acd491c88739797ad3a3d09

Call time: 00h:00m:20s, 1st res in 1191 ms, conn in 1193ms

SRTP status: Not active Crypto-suite: (null)

#0 speex @16KHz, sendrecv, peer=192.168.0.190:4000

RX pt=103, stat last update: 00h:00m:00.931s ago

total 685pkt 45.3KB (72.7KB +IP hdr) @avg=17.9Kbps/28.7Kbps

pkt loss=0 (0.0%), discrd=0 (0.0%), dup=0 (0.0%), reord=0 (0.0%)

(msec) min avg max last dev

loss period: 0.000 0.000 0.000 0.000 0.000

jitter : 0.000 17.254 57.000 19.250 5.127

TX pt=103, ptime=20ms, stat last update: 00h:00m:01.941s ago

total 700pkt 46.4KB (74.4KB +IP hdr) @avg 18.3Kbps/29.4Kbps

pkt loss=1 (0.1%), dup=0 (0.0%), reorder=0 (0.0%)

(msec) min avg max last dev

loss period: 20.000 20.000 20.000 20.000 0.000

jitter : 0.000 22.700 35.437 21.750 12.653

RTT msec : 2.563 5.744 12.999 3.082 4.237

And we also have call.wav, which contains the audio recorded on the server side.

Now it’s time to use Sevana NIQA to obtain a MOS score of the call:

niqa -rdf TstBase.nbf -gqa call.wav

And the result is:

Sevana NonIntrusive Audio Quality Analyzer – NIQA v.1.1.1.24.

Copyright (c) 2010 by Sevana Oy, Finland. All rights reserved.

—————————————————————

Database loaded!

Quality of file ‘C:\NIQA\SIP\call.wav’ is 3.337179.

Used next Asins: ‘FFr4′

Quality score calculated!

Wow! NIQA not only provided the MOS score (MOS = 3.34), but was also able to recognize that it was a Female voice speaking in French (Ffr4). MOS score is pretty good, but the most important thing is that we can now create the first record of voice quality monitoring:

And another important characteristic: RTT msec : 2.563 5.744 12.999 3.082 4.237

So, we know quite a lot about the VoIP conditions of the incoming call as well as that the call quality was good (MOS is quite high).

Server side:

Here we also have a log.txt file containing the same important data about VoIP call parameters:

[DISCONNCTD] To: <sip:sevanaoy@realsip.com>;tag=6878cb877e224f89bbad8cbcb66df63b

Call time: 00h:00m:20s, 1st res in 79 ms, conn in 297ms

SRTP status: Not active Crypto-suite: (null)

#0 speex @16KHz, sendrecv, peer=192.168.0.167:4000

RX pt=103, stat last update: 00h:00m:01.781s ago

total 690pkt 45.7KB (73.3KB +IP hdr) @avg=18.0Kbps/28.9Kbps

pkt loss=0 (0.0%), discrd=0 (0.0%), dup=0 (0.0%), reord=0 (0.0%)

(msec) min avg max last dev

loss period: 0.000 0.000 0.000 0.000 0.000

jitter : 0.562 23.396 221.500 221.500 6.725

TX pt=103, ptime=20ms, stat last update: 00h:00m:04.953s ago

total 700pkt 46.4KB (74.4KB +IP hdr) @avg 18.3Kbps/29.4Kbps

pkt loss=0 (0.0%), dup=0 (0.0%), reorder=0 (0.0%)

(msec) min avg max last dev

loss period: 0.000 0.000 0.000 0.000 0.000

jitter : 16.125 18.021 19.437 19.437 1.393

RTT msec : 2.853 13.575 34.667 2.853 14.91

Now the same procedure with the call.wav file that was created on the server side:

niqa -rdf TstBase.nbf -gqa call.wav

And the result is:

Sevana NonIntrusive Audio Quality Analyzer – NIQA v.1.1.1.24.

Copyright (c) 2010 by Sevana Oy, Finland. All rights reserved.

—————————————————————

Database loaded!

Quality of file ‘C:\NIQA\SIP\call.wav’ is 3.329050.

Used next Asins: ‘FFr4′

Quality score calculated!

Alright, now we can build the same call quality monitoring record as we did for the calling party:

Finally: RTT msec : 2.853 13.575 34.667 2.853 14.915

Calling party and called party voice quality records:

Now we can compare two records and evaluate what this gives to us:

Calling party:

Called party:

As one can see there are not many differences and the quality score is quite good and stable both for transmitted and received audio, now how can we use these records for our QoS analysis?

QoS Monitoring Solution

As long as the quality remains quite high this analysis and call quality data stored in a database (like MySQL f.e.) is not that useful, but how to detect that the quality went down? Only by R-value calculation is not the best approach, and therefore we suggest the following case for your consideration:

1. QoS monitoring system based on obtaining MOS scores provided by Sevana NIQA and VoIP parameters is running and storing call quality records for statistics

1. The system always monitors MOS value as the key voice quality indicator
2. MOS score drops down, let’s say below 2
3. The system searching for a call quality record with the highest MOS score and matches its parameters against parameters of the “bad call”.
4. The system immediately will be able to indicate what are the reasons for the quality loss, because it can compare all main network parameters for high and low MOS scores.
5. The system will be able to continously provide speech quality scores for all calls thus allowing to visualize what trunks, routes, destinations are a matter of lower quality

What’s the catch?

Well, there is no catch… Well, there is always a catch, but this time it’s very simple:

We hope we have shown with this small howto a simple approach that will allow anybody to build his own QoS monitoring solution based just on a software SIP phone allowing to monitor… not, not the voice or speech quality of VoIP calls, but

1. how many times your VoIP customers are happy about your service
2. what makes your customers unhappy in your service
3. where is the problem that makes your customers unhappy
4. where to search for the problem origin

And if you doubt that we can help really anybody to enable his own QoS monitoring, just think of answering a couple of questions:

• Is it expensive is loosing customers due to making them unhappy of making calls in your system?
• Is it important to be sure that your Service Level Agreement (SLA) is always valid?
• Do you know enough about Sevana NIQA?
• Have you contacted Sevana concerning using NIQA for your OoS system?

If you don’t know answer even to just one of the questions, please contact us: give us a call, send an email, we’ll get back to you immediately and we are sure you will be pleased with what our voice quality assessment software can do to keep your customers happy and you aware of having control over your VoIP system.

We are sure that NIQA is just what you need…

…because:

• Available for evaluation
• Strong competitor for ITU P.563 / P.564
• Ability to be trained to detect reasons for quality loss
• Ability to be trained for customer specific needs
• Multi-platform
• High performance
• Outstanding pricing
• Available as online service

Typically voice quality in a network would suffer from:

– Noise
– Silence
– Low bit-rate encoding
– Network errors (both in mobile and packet-switched)
– Delays, Echo, Jitter, etc
– Handsets/terminals

Speech quality however corresponds to the clarity or clearness of speech delivered from a speaker to a listener, but when talking about speech quality in packet-switched networks we’ll find lots of factors that affect the quality:

– Vocoder selection
– Front-end and signal level clippings
– Signal loss and packet loss
– Signal gain/ attenuation
– Echo cancellation in double talk
– Signal noise from equipment and disturbances from analog networks

And what is quality of speech in the system for a voice service provider? Well, that’s kind of a key issue, because…

- Unhappy customers stop using the service
- Continuously unhappy customers move to competitors and spread negative word of mouth
- If the goal is to obtain new and retain existing customers a service provider requires:
– Effective means to test and monitor quality of voice services
– Ability to receive voice quality metrics for end-users

So, hopefully all mentioned above clearly shows that transparency and visibility required for effective service quality management is an absolute mus for a voice service provider, which has only one point when thinking of what is voice quality – it is happy or unhappy customers, competitive or failed business, growth or a downswing…

Click here to request Sevana NIQA evaluation

Follow us on twitter

Today we start a set of posts related to the so called non-intrusive (no need for a reference file) single-ended speech/voice qualiy assessment, and our main target will be to explain more about how this can be done in VoIP networks. As old followers know we have intrusive (PESQ-like) voice and audio quality anlysis tool AQuA that already has several happy customers, but in March 2010 we have released for evaluation Sevana NIQA – a non-intrusive voice quality testing software that we particulary target at VoIP. Our goal is to provide VoIP community (and in particular Asterisk users) with a full set of voice quality assessment tools: intrusive and non-intrusive, and today we are going to saysome words ot the latter one.

We would dare to think that integration of commercial and enterprise voice services into packet-switched networks significanlty cuts costs and brings value. Most likely there won’t be any objections here… However, customer expectations of service quality are always the key issue of any voice service and the only way to win and retain customers is to make sure the quality of service meets customers’ expectations. Customers of any voice service are used to the quality of circuit-switched networks.

When integrated into packet-switched networks voice services QoS sustainability requires much more attention:

• Control packet loss, jitter and delay in complex network solutions
• Minimize quality impairments for vocoders, VAD, jitter buffers, echo cancellers and other signal processing involved

At this point one should realize that it’s essential to have knowledge of service quality delivered to the customers, don’t you think so? If you do, the we meet in the next post related to the subject of “What is voice quality?”. Stay in touch!

Click here to request Sevana NIQA evaluation

Follow us on twitter

Voice quality is one of the main characteristics of speech transmission systems. When analyzing voice quality one must not only consider audio signal degradation caused by transmission over telecom channels, but also specifics of speaker’s voice, conditions of listener’s hearing and variation of these parameters in time.

The most known methods for quality evaluation of voice transmission systems were developed by Telecommunication Standardization Sector of International Telecommunications Union (ITU-T) in the middle of 90-s. Results of this work are presented in Recommendation P.800 (P.830) «Methods for subjective determination of transmission quality» [1, 2]. This document describes conditions for voice quality testing, audio contents, scoring and methods to evaluate results. Typically “Methods for subjective determination of transmission quality” are used to obtain mean subjective quality score according to five-digit scale (Mean Opinion Score – MOS).

Unfortunately P.800 recommendation tests may lead to ambiguous results. Recommendation is warning about comparing MOS scores received under different conditions and consider such approach incorrect. Besides that preforming tests according to P.800 takes a lot of time and requires a lot of testers involved in the process.

In order to move from subjective (MOS) scores to objective ones and to automate the quality measurement, ITU-T has developed the P.861 recommendation, which is based on low level quantitative measurements [3]. Recommendation P.861 is a follow-up of PSQM method (Perceptual Speech Quality Measurement), developed by KPN Research and devoted to objective analysis of speech codecs performance with a low level of degradation.

However, it is impossible to utilize PSQM for evaluation of work of a real communication system because the method does not consider all the important factors influencing human perception. Among these factors are delay, jitter, packet loss as well as signal level clipping.

In February 2001 ITU-T has issued another recommendation ITU-T P.862 [4], which describes a more advanced algorithm for voice quality testing – PESQ (Perceptual Evaluation of Speech Quality). The algorithm includes level and time aligning, human perception and cognitive modeling. Due to these additional operations the approach considers signal amplification/ attenuation in a communication system, time delays and jitter as well as spectrum bands, which are the most significant for human perception. Based on cognitive modeling PESQ also recalculates objective quality score into MOS values.

A disadvantage of PESQ as well as other similar algorithm is the fact that they are based on comparing of two signals: original and transmitted through a communication system. This approach may create a range of difficulties connected with setting and preforming voice quality testing. One requires to arrange signal recording on both sides of the telecommunication system as well as records transmission to the test system. Besides this real time quality monitoring in such approach appears quite difficult as well.

In order to solve the challenging issues mentioned above ITU-T has developed a new recommendation P.563 [5] introduced in May 2004. This recommendation determines algorithm for evaluating speech quality by listening to communication sessions. The algorithm takes into account single-side distortions, speech trunk parameters, noise and speech naturalness. Developers of P.563 call attention that P.563 does not provide overall quality estimation of speech transmission. Distortions driven by delays, echo, loss of loudness and everything related to two-sided interaction cannot be taken into consideration by this method.

It’s widely thought that P.563 provides a high level of correlation between automated and expert quality scores. However, simple tests based on ITU-T sound database for codec testing [6] may raise some doubts about the consistence of the algorithm provided together with its description.

Table.1. Comparison between results of P.563 and expert estimations

The problem discovered in the distributed P.563 algorithm implementation required development of an alternative solution. Further down one can find one of possible solutions that is implemented in Sevana NIQA (Non-Intrusive Quality Analyzer).

NIQA’s (Non-Intrusive Quality Analyzer) approach is based on a database of trained etalons called associations. Each association corresponds to a group of files that have close expert estimations of sound quality and common set of reasons for sound quality degradation. For each association NIQA calculates and stores a distribution of parameters’ values.

Basic algorithm showing how NIQA obtains sound quality scores is represented on the picture below.



NIQA - non-intrusive voice quality testing software

When loading sound signal the system excludes all fragments with low energy level (according to threshold). The excluded fragments correspond to “absolute silence” and are considered irrelevant for obtaining sound quality score.

At the next phase the signal is split into frames used in voice activity detection algorithm (VAD). The system calculates energy values for each frame what increases accuracy of VAD. With the help of VAD algorithm the signal divides to active and inactive components that are processed separately. The system builds level histograms for both active and inactive signal components.

By discrete cosine transform (DCT) the system obtains signal spectrum and checks the active components frames for DTMF presence and then excludes the frames that are similar to DTMF from further processing.

Next stage applies the first level of psycho-acoustic model to the signal spectrum. This model checks different types of masking (including pre-masking and post-masking). According to clear peaks of spectrum energy the system splits the signal into tone and noise components.

Second level of psycho-acoustic model performs energy normalization of the signal – energy levels are transformed into loudness levels at 1kHz. Third level of psycho-acoustic model transforms loudness levels into several detectable grades of loudness that allow to ignore sound signal changes, which are not recognized by human ear.

The next step is to split signal spectrum into bands that are critical to human ear perception and calculate parameters both on and out of the bands. Based on the computed signal parameters the system selects most similar associations from the database and performs matching. According to selected associations the system determines how much each of them influence the overall quality and then generates the final voice quality score as a combination of scores for selected associations and according to correspondent weights.

Sevana NIQA Testing and Evaluation

Sevana NIQA has been tested utilizing the same ITU-T speech database that is used for conformance testing of P.563 algorithm. In the tests we used a total of 376 English language recordings. All recordings were sorted into 4 groups depending on their MOS scores (represented in the documentation attached to the sound database). For all groups of recordings we determined average expert scores and average NIQA scores (Table 2). In order to illustrate comparison with P.563 we also calculated average errors for P.563 and NIQA scores for the same tests.

Table.2. Comparison of NIQA scores against expert estimations

The results clearly show that NIQA allows receiving much higher accuracy between generated quality scores and expert estimations than P.563. NIQA scores are less precise only for records with very low MOS scores (in the range from 1 to 2). In all other cases NIQA provides 2-3 times higher quality scores precision compared to MOS values.

References

1. Methods for subjective determination of transmission quality // ITU-T Recommendation P.800 / http://www.itu.int/rec/T-REC-P.800/en

2. Subjective performance assessment of telephone-band and wideband digital codecs // ITU-T Recommendation P.830 / http://www.itu.int/rec/T-REC-P.830/en

3. Objective quality measurement of telephone-band (300-3400 Hz) speech codecs // ITU-T Recommendation P.861 / http://www.itu.int/rec/T-REC-P.861/en

4. Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs // ITU-T Recommendation P.862 / http://www.itu.int/rec/T-REC-P.862/en

5. Single-ended method for objective speech quality assessment in narrow-band telephony applications // ITU-T Recommendation P.563 / http://www.itu.int/rec/T-REC-P.563-200405-I/en

6. ITU-T coded-speech database // Supplement 23 to ITU-T P-series Recommendations / http://www.itu.int/rec/T-REC-P.Sup23-199802-I/en

Copied from: http://www.sevana.fi/non-intrusive-voice-quality-testing-software.php

Non-intrusive voice qualiy testing does not require a reference signal and can work with real communications data in real time.

Non-intrusive techniques provide a possibility to test and monitor a greater amount of communications and therefore obtain a quite reliable information about the networks (VoIP, PSTN, GSM, CDMA, LTE) quality.

Today Sevana announces availability of new software for voice quality testing for customers’ and partners’ evaluation. Be among the first to test the new approach to voice quality testing and monitoring!

This software is delivered only upon request.