2017│11│21 – The P3 Mobile Network Test in Spain
For the third time in a row, the consulting, engineering and testing company P3 communications and connect magazine have examined the Spanish mobile networks. As all of Spain’s four mobile operators have worked hard to improve regarding performance and coverage, the P3 connect Mobile Benchmark Spain promises interesting results. Which network is the best choice for voice and for data communications? And how does our new crowdsourced operational excellence score affect the results?
Results in a nutshell
Thanks to the carefully designed methodology, based on comprehensive drivetests and, for the first time, an additional crowdsourced assessment of data service availability (see page 8), P3‘s network benchmarks are highly objective, and have been widely accepted as authoritative. This year, the drive tests covered 17 of the largest cities in Spain, each with more than 100,000 inhabitants. Our measurement cars also visited smaller towns and drove on connecting roads and motorways. The areas in which we tested account for more than 11.5 million people, or almost 25 per cent of the Spanish population.
P3‘s rigorous measurements included the use of up-to-date LTE Cat 9 as well as VoLTE-capable smartphones for the tests. Also, we constantly readjust the thresholds of our evaluation. With steps like these, we reflect the latest technical developments in the mobile networks and once more emphasise the scope of our benchmarking: How do the mobile networks perform at the edge of what is technically feasible – and to what extent do customers benefit from these capabilities? In order to provide valid answers to these questions, we have used the most comprehensive mobile plans available from each operator.
Substantial improvements in the Spanish networks, Vodafone once again clear winner
Almost all Spanish operators improved their performance levels in comparison to our 2016 mobile network benchmark. For the third time in a row, the overall winner is Vodafone with the grade “very good“, showing clear improvements in the data discipline and similar voice results as in 2016. A reduction in the total voice points can be explained with the addition of the crowd score resulting in adjustments in the maximum achievable points in the other disciplines.
Movistar comes in second with the overall grade “good“. In comparison to 2016 and also to its competitors Vodafone and Orange, Movistar falls a little behind in the voice category, but can make up for this with strong data results. Orange still ranks third and achieves the grade “good“, as in 2016, but manages to reduce the gap to Movistar thus showing a clear overall improvement. The smallest Spanish operator Yoigo ranks last again, but achieves considerable improvements, particularly in the data category. This results in the rating “satisfactory“, which is a full grade above its last year‘s result.
Movistar is the brand name that the Spanish telecommunications company Telefónica uses for the mobile network in its home market. Telefónica S.A. itself is one of the largest telco companies in the world. The operator is present in 21 countries with a total of 127,000 employees and achieved worldwide revenues of over €52 billion in 2016.
While the company introduced the Movistar brand in Latin American countries in 2005, it has been active in Spain since the launch of GSM services back in 1995. Today, Movistar is the largest mobile operator in Spain with about 15.3 million subscribers, which equals a market share of roughly 32 per cent. It offers GSM service at 900 and 1800 MHz, UMTS/3G at 900 and 2100 MHz and LTE at 800, 1800 and 2600 MHz.
Since the end of 2014, Movistar has supported 4G+ carrier aggregation with maximum speeds of 150 Mbps. The operator claims to reach 89 per cent of the Spanish population with its 4G network.
Vodafone España has been present on the Spanish mobile communications market since the year 2000. Then, the British Vodafone Group acquired Airtel Móviles which had operated in Spain since 1994. In Spain, Vodafone now reports 14.4 million mobile customers, adding up to a market share of about 30 per cent and making Vodafone the second largest operator in the country with a narrow lead over its competitor Orange.
In the fiscal year 2016/ 2017, Vodafone Spain achieved revenues of €4.5 billion which contributes about nine per cent to the whole Vodafone Group‘s financial result.
Vodafone‘s mobile network in Spain offers GSM service at 900 and 1800 MHz, UMTS/3G at 900 and 2100 MHz and LTE at 800, 1800, 2100 and 2600 MHz. The Vodafone 4G network in Spain supports LTE carrier aggregation (“4G+“) with maximum downlink speeds of 300 Mbps.
Vodafone España claims to offer the best LTE coverage in Spain, reaching approximately 94 per cent of the Spanish population.
Orange España is the brand name of France Telecom‘s mobile network in Spain. It has been operating under this name since 2006. Previously, the network was known as “Amena“ – this brand name lives on in Orange Spain‘s portfolio as a low-cost offer that is only available on the internet. Also, its network serves a number of mobile virtual network operators such as MasMovil, Carrefour Móvil and others. With 14.1 million customers, Orange is the third largest Spanish mobile operator with a market share of about 29 per cent. In the fiscal year 2016, Orange Spain achieved a revenue of €5 billion which contributed 12 per cent to the whole Orange Group‘s results.
Orange Spain has deployed 2G networks at 900 and 1800 MHz, 3G networks at 900 and 2100 MHz and 4G at 800, 1800 and 2600 MHz. The operator claims that its 4G network reaches 93 per cent of the Spanish population. Also, Orange is the first Spanish operator to offer VoLTE to its 4G customers.
Yoigo was the latest mobile operator to enter the Spanish market. Founded in the year 2000 under the name Xfera, the company started its actual operation in 2006, offering only a UMTS/3G network at 2100 MHz. At this time, the Swedish telecommunications company TeliaSonera acquired the majority of shares and rebranded the network as “Yoigo“. This name was supposed to reflect the simplicity and ease in rates as well as in the use of the service. In June 2016, the former MVNO (mobile virtual network operator) Másmóvil bought the company.
Yoigo had a national roaming agreement with Movistar until the end of 2016. Since January 2017, Yoigo customers freely roam in the 2G and 3G networks of Orange at locations without Yoigo coverage.
The current customer base of Yoigo is 4.2 million subscribers, which equals a market share of 9 per cent.
Today, Yoigo operates 3G at 2100 MHz as well as 4G at 1800 MHz. The operator currently claims a LTE coverage of approximately 89 per cent of the Spanish population.
A close look at Spain’s networks
P3 communications GmbH, based in Aachen, Germany, is a world leader in mobile network testing. It is part of the P3 group, with over 3000 employees worldwide and a turnover of more than €300 million. P3 is partnering with the international telecommunications magazine connect, which has more than 20 years of editorial expertise and is one of the leading test authorities in Europe for telecommunications products and services.
Together, P3 and connect have been conducting the most important network benchmark test in Germany for 15 years, extending it to Austria and Switzerland in 2009. Since 2014, the range of public benchmarks has continuously been expanded, today covering Spain, the Netherlands, Sweden, the UK and Australia.
In 2016 alone, P3 compiled more than 60,000 measurement hours in 65 countries across five continents, with its test vehicles covering more than one million kilometres. As the de-facto industry standard, the P3 benchmarking methodology focuses on customer-perceived network quality – examining voice telephony that makes up 38.8 per cent of the result, data connectivity that contributes 58.2 per cent as well as operational excellence, currently accounting for three per cent of the total result. P3‘s network benchmarks are widely accepted as a completely objective authority.
New Crowd Score confirms the established ranking order
In order to also reflect the operational excellence of the Spanish networks, we have added a crowd-sourcing component to our methodology (see page 8). But even these additional results only underline the familiar ranking order in Spain – see the details for yourself on the following pages.
On their tour through Spain, P3‘s four test cars visited 17 of the largest Spanish cities and many smaller towns and also covered the connecting roads. For the voice rating, each car carried eight Samsung Galaxy S7 smartphones that permanently called each other. The connected testing equipment registered success ratios, setup times and speech quality. In order to simulate normal smartphone usage, data transfers took place in the background of the test calls.
Vodafone ranks best in voice and overall, Orange is second in the voice discipline
At the time of testing, Orange was the only operator that offered VoLTE (Voice over LTE) to its customers. This resulted in fastest call setup times in all categories. However, Vodafone delivered the best speech quality and also showed excellent success ratios especially in big cities and smaller towns.
In terms of call success ratios, Movistar follows closely behind Vodafone and takes the lead on connecting roads. When analysing speech quality, Movistar is on par with Orange in big cities and slightly ahead of Orange in towns and on the roads. In the overall voice ranking, however, Orange comes in second after Vodafone due to the significantly faster call setup times, relegating Movistar to the third rank.
Yoigo‘s results are particularly interesting, bearing in mind, that the smallest contender recently switched from Movistar to Orange for national roaming. Yoigo still scores clearly behind its competitors. In comparison to the previous year, Yoigos voice results declined somewhat in the bigger cities and considerably in smaller towns, while they noticeably improved on the connecting roads.
Voice results at a glance
Vodafone delivers the best results in all tested voice categories. Orange achieves a second rank, probably due to its unique VoLTE offering. Movistar had the best success ratios on connecting roads. Yoigo ranks last with mixed results compared to the previous year.
The volume of mobile data downloads and uploads is exponentially growing. 4G/LTE currently is the best technology to cope with these increasing demands, while all Spanish 4G networks have increased their coverage in terms of the population. While Movistar, Orange and Vodafone also compete about who delivers the highest data rates, Yoigo is still mainly concentrating on expanding its LTE footprint. According to its own claims (also see page 3), the smallest Spanish contender has now reached 89 per cent 4G coverage of the Spanish population.
P3‘s testing rewards coverage and stability as well as fast data rates. The benchmarking of web-page downloads as well as file downloads and uploads is testing the maximum throughputs available to customers. At the same time, it assesses the networks‘ availability and stability by examining success ratios.
In order to assess typical performance as well as peak speeds, we determined two values: the minimum data rate that is available in 90 per cent of the cases, and additionally the peak data rate that is surpassed in 10 per cent of the cases.
P3‘s approach for YouTube testing recognises that the popular video service uses adaptive bit rates. This method strives for a better user experience, subordinating pixel resolution to stable playback. As a consequence, besides success ratios, start times and the absence of interruptions, we have added the average video resolution as another important performance indicator.
Vodafone leads in data in the cities, but faces strong competitors – including Yoigo
The results show that all operators offer a mature level of LTE coverage in the big cities. Here, all four Spanish networks show excellent success ratios. All four candidates reach their best performance values in this environment, but in contrast to the previous year, the gap between big cities and smaller towns has narrowed.
In the web-page category examined in the cities, Orange comes in closely behind Vodafone and with a narrow lead over Movistar. However, on the whole Movistar achieves the second rank in the “data in cities“ category – mainly due to higher data rates achieved in the file downloads and uploads.
Yoigo ranks behind the other three contenders also in this category – but presents its strongest data results in the cities. With high success ratios and data rates not far behind the middle field, Yoigo is a serious alternative for data communications in the cities.
All operators have improved in smaller towns
In the smaller towns, the four Spanish operators show similar results and a similar ranking order as in the bigger cities. The best news, however, is that the reliability and performance of each Spanish mobile network has clearly improved over last year‘s results. Another nice observation are the excellent YouTube results of all Spanish networks that we determined in the towns as well as in cities and even on the roads. Watching video streams on the go in Spain in 2017 is a far more pleasant experience than it used to be one year ago.
Yoigo shows biggest improvement in data
What was true for the smaller towns, also applies to the connecting roads: The ranking order in the data disciplines remains the same in all tested categories. And, in comparison to 2016‘s results, all operators clearly managed to improve their scores. This is especially applicable for Yoigo, which only achieved 41 per cent of the possible points in this category in the previous year, and improved its score to a respectable 74 per cent this time. This is a truly remarkable result for Spain’s smallest contender.
Overall, Movistar ranks second in data discipline
In the overall examination, Movistar ranks second in the data discipline after a strong Vodafone – but also Orange is following at a not too far distance.
An additional important aspect of mobile service quality – complementing performance and measured values – is the actual availability of mobile connectivity to the customers. Obviously, even the best performing network is only of limited benefit to its users, if it is frequently impaired by outages or disruptions. Therefore, P3 has been looking into additional methods for the quantitative determination of network availability: collecting data via crowdsourcing. This method must however not be confused with the drivetests described on the previous pages. We are convinced that crowdsourcing can significantly enhance the aspects of benchmarking: Drivetesting has obvious advantages as a very controlled environment, while crowdsourcing accelerates when looking for longer time periods or geography beyond defined test routes. So, when it comes to diagnosing the sheer availability of the respective mobile networks, a crowdsourcing approach can provide additional insights. Therefore, P3 has developed an app-based crowdsourcing mechanism in order to assess how a large number of mobile customers experience the availability of their mobile network. We call this aspect “operational excellence”.
The P3 connect Mobile Benchmark in Spain is one of the first occasions where we expand our scoring scheme with the results of this crowdsourced investigation. As we have considered the results from August, September and October 2017, and each month is represented by a maximum of ten achievable points, in the benchmark at hand, the so called “crowd-score” contributes up to 30 points to the total result.
The detailed methodology of our analysis and the calculation of the resulting points is described on page 11 of this report. As a consequence of this addition, the P3 connect Mobile Benchmark is the only mobile network test which combines the two aspects drivetesting and crowdsourcing, thus providing the most comprehensive view on network performance.
Crowdsourcing shows Spanish networks are highly reliable
The in-depth analysis of our crowdsourcing data gathered in the three months preceding and including our measurement campaign in Spain, shows that the Spanish networks are all in all very stable and reliable. As degradations observed in the night hours between 0.00 a.m. and 6.00 a.m. are not accounted for, we did not register any incidents for Movistar and O2 during the observation period.
In the relevant period, Vodafone only suffered a one hour service degradation in the evening of October 2nd. According to our scoring principle, this one-hour degradation costs one point – so Vodafone scored a total of 29 out of 30 possible crowd score points.
In the Yoigo network, we registered an incident that occurred in the early morning of August 16th. As we only count degradations after 6.00 a.m., this also led to a take-off of one point. For this reason, also Yoigo scored 29 out of 30 possible points in total in the operational excellence category.
While these reductions of service availability were certainly annoying to the customers of the affected networks, they only have a limited impact to the overall results and did not change the actual ranking in Spain. However, for next year, we plan to consider a larger number of months and will increase the share that our crowd score has in the total result.
Operational excellence at a glance
Considering August, September and October of 2017, we could determine a one-hour degradation in the Vodafone network and an incident in the Yoigo network that due to our nightly hold-off interval only counts as one hour. For Movistar and Orange, we did not observe any relevant incidents. As we considered a three-month period, which contributes 30 to the total maximum of 1000 achievable points, both Vodafone and Yoigo score 29 out of 30 points, while Movistar and Orange both score the full 30 points.