2017│11│21 – The P3 Mobile Network Test in Spain
For the third time in a row, the consulting, engineering and testing company P3 communications and connect magazine have examined the Spanish mobile networks. As all of Spain’s four mobile operators have worked hard to improve regarding performance and coverage, the P3 connect Mobile Benchmark Spain promises interesting results. Which network is the best choice for voice and for data communications? And how does our new crowdsourced operational excellence score affect the results?
Results in a nutshell
Thanks to the carefully designed methodology, based on comprehensive drivetests and, for the first time, an additional crowdsourced assessment of data service availability (see page 8), P3‘s network benchmarks are highly objective, and have been widely accepted as authoritative. This year, the drive tests covered 17 of the largest cities in Spain, each with more than 100,000 inhabitants. Our measurement cars also visited smaller towns and drove on connecting roads and motorways. The areas in which we tested account for more than 11.5 million people, or almost 25 per cent of the Spanish population.
P3‘s rigorous measurements included the use of up-to-date LTE Cat 9 as well as VoLTE-capable smartphones for the tests. Also, we constantly readjust the thresholds of our evaluation. With steps like these, we reflect the latest technical developments in the mobile networks and once more emphasise the scope of our benchmarking: How do the mobile networks perform at the edge of what is technically feasible – and to what extent do customers benefit from these capabilities? In order to provide valid answers to these questions, we have used the most comprehensive mobile plans available from each operator.
Substantial improvements in the Spanish networks, Vodafone once again clear winner
Almost all Spanish operators improved their performance levels in comparison to our 2016 mobile network benchmark. For the third time in a row, the overall winner is Vodafone with the grade “very good“, showing clear improvements in the data discipline and similar voice results as in 2016. A reduction in the total voice points can be explained with the addition of the crowd score resulting in adjustments in the maximum achievable points in the other disciplines.
Movistar comes in second with the overall grade “good“. In comparison to 2016 and also to its competitors Vodafone and Orange, Movistar falls a little behind in the voice category, but can make up for this with strong data results. Orange still ranks third and achieves the grade “good“, as in 2016, but manages to reduce the gap to Movistar thus showing a clear overall improvement. The smallest Spanish operator Yoigo ranks last again, but achieves considerable improvements, particularly in the data category. This results in the rating “satisfactory“, which is a full grade above its last year‘s result.
Movistar is the brand name that the Spanish telecommunications company Telefónica uses for the mobile network in its home market. Telefónica S.A. itself is one of the largest telco companies in the world. The operator is present in 21 countries with a total of 127,000 employees and achieved worldwide revenues of over €52 billion in 2016.
While the company introduced the Movistar brand in Latin American countries in 2005, it has been active in Spain since the launch of GSM services back in 1995. Today, Movistar is the largest mobile operator in Spain with about 15.3 million subscribers, which equals a market share of roughly 32 per cent. It offers GSM service at 900 and 1800 MHz, UMTS/3G at 900 and 2100 MHz and LTE at 800, 1800 and 2600 MHz.
Since the end of 2014, Movistar has supported 4G+ carrier aggregation with maximum speeds of 150 Mbps. The operator claims to reach 89 per cent of the Spanish population with its 4G network.
A close look at Spain’s networks
P3 communications GmbH, based in Aachen, Germany, is a world leader in mobile network testing. It is part of the P3 group, with over 3000 employees worldwide and a turnover of more than €300 million. P3 is partnering with the international telecommunications magazine connect, which has more than 20 years of editorial expertise and is one of the leading test authorities in Europe for telecommunications products and services.
Together, P3 and connect have been conducting the most important network benchmark test in Germany for 15 years, extending it to Austria and Switzerland in 2009. Since 2014, the range of public benchmarks has continuously been expanded, today covering Spain, the Netherlands, Sweden, the UK and Australia.
In 2016 alone, P3 compiled more than 60,000 measurement hours in 65 countries across five continents, with its test vehicles covering more than one million kilometres. As the de-facto industry standard, the P3 benchmarking methodology focuses on customer-perceived network quality – examining voice telephony that makes up 38.8 per cent of the result, data connectivity that contributes 58.2 per cent as well as operational excellence, currently accounting for three per cent of the total result. P3‘s network benchmarks are widely accepted as a completely objective authority.
New Crowd Score confirms the established ranking order
In order to also reflect the operational excellence of the Spanish networks, we have added a crowd-sourcing component to our methodology (see page 8). But even these additional results only underline the familiar ranking order in Spain – see the details for yourself on the following pages.
On their tour through Spain, P3‘s four test cars visited 17 of the largest Spanish cities and many smaller towns and also covered the connecting roads. For the voice rating, each car carried eight Samsung Galaxy S7 smartphones that permanently called each other. The connected testing equipment registered success ratios, setup times and speech quality. In order to simulate normal smartphone usage, data transfers took place in the background of the test calls.
Vodafone ranks best in voice and overall, Orange is second in the voice discipline
At the time of testing, Orange was the only operator that offered VoLTE (Voice over LTE) to its customers. This resulted in fastest call setup times in all categories. However, Vodafone delivered the best speech quality and also showed excellent success ratios especially in big cities and smaller towns.
In terms of call success ratios, Movistar follows closely behind Vodafone and takes the lead on connecting roads. When analysing speech quality, Movistar is on par with Orange in big cities and slightly ahead of Orange in towns and on the roads. In the overall voice ranking, however, Orange comes in second after Vodafone due to the significantly faster call setup times, relegating Movistar to the third rank.
Yoigo‘s results are particularly interesting, bearing in mind, that the smallest contender recently switched from Movistar to Orange for national roaming. Yoigo still scores clearly behind its competitors. In comparison to the previous year, Yoigos voice results declined somewhat in the bigger cities and considerably in smaller towns, while they noticeably improved on the connecting roads.
Voice results at a glance
Vodafone delivers the best results in all tested voice categories. Orange achieves a second rank, probably due to its unique VoLTE offering. Movistar had the best success ratios on connecting roads. Yoigo ranks last with mixed results compared to the previous year.
An additional important aspect of mobile service quality – complementing performance and measured values – is the actual availability of mobile connectivity to the customers. Obviously, even the best performing network is only of limited benefit to its users, if it is frequently impaired by outages or disruptions. Therefore, P3 has been looking into additional methods for the quantitative determination of network availability: collecting data via crowdsourcing. This method must however not be confused with the drivetests described on the previous pages. We are convinced that crowdsourcing can significantly enhance the aspects of benchmarking: Drivetesting has obvious advantages as a very controlled environment, while crowdsourcing accelerates when looking for longer time periods or geography beyond defined test routes. So, when it comes to diagnosing the sheer availability of the respective mobile networks, a crowdsourcing approach can provide additional insights. Therefore, P3 has developed an app-based crowdsourcing mechanism in order to assess how a large number of mobile customers experience the availability of their mobile network. We call this aspect “operational excellence”.
The P3 connect Mobile Benchmark in Spain is one of the first occasions where we expand our scoring scheme with the results of this crowdsourced investigation. As we have considered the results from August, September and October 2017, and each month is represented by a maximum of ten achievable points, in the benchmark at hand, the so called “crowd-score” contributes up to 30 points to the total result.
The detailed methodology of our analysis and the calculation of the resulting points is described on page 11 of this report. As a consequence of this addition, the P3 connect Mobile Benchmark is the only mobile network test which combines the two aspects drivetesting and crowdsourcing, thus providing the most comprehensive view on network performance.
Crowdsourcing shows Spanish networks are highly reliable
The in-depth analysis of our crowdsourcing data gathered in the three months preceding and including our measurement campaign in Spain, shows that the Spanish networks are all in all very stable and reliable. As degradations observed in the night hours between 0.00 a.m. and 6.00 a.m. are not accounted for, we did not register any incidents for Movistar and O2 during the observation period.
In the relevant period, Vodafone only suffered a one hour service degradation in the evening of October 2nd. According to our scoring principle, this one-hour degradation costs one point – so Vodafone scored a total of 29 out of 30 possible crowd score points.
In the Yoigo network, we registered an incident that occurred in the early morning of August 16th. As we only count degradations after 6.00 a.m., this also led to a take-off of one point. For this reason, also Yoigo scored 29 out of 30 possible points in total in the operational excellence category.