2018│12│20 – The 2018 Mobile Network Test in the United Kingdom

Download the report as PDF here

THE 2018 MOBILE NETWORK TEST IN THE UNITED KINGDOM

FOR THE FIFTH TIME IN A ROW, THE BENCHMARKING EXPERT UMLAUT AND THE ACKNOWLEDGED TELECOMMUNICATIONS MAGAZINE CONNECT HAVE CONDUCTED THEIR MOBILE NETWORK BENCHMARK IN THE UK.  IN A MARKET CHARACTERISED BY ITS CONTINUOUS UPGRADING OF 4G NETWORKS, GROWING DATA RATES AND ENHANCEMENTS SUCH AS VOICE OVER LTE, WE WANT TO FIND OUT: WHICH OPERATOR OFFERS THE BEST MOBILE NETWORK?

Results in a nutshell

THE OVERALL RANKING OF THIS YEAR’S UMLAUT´S CONNECT MOBILE BENCHMARK IN THE UK LOOKS FAMILIAR. BUT IN THE DETAILED CATEGORIES, THE OPERATORS SHOW VARYING RESULTS – IMPROVING IN SOME AREAS, BUT LOSING GROUND IN OTHERS.

Umlaut‘s network benchmarks are widely accepted as being highly objective and defining the de-facto  industry standard The carefully designed methodology of the 2018 benchmark in the UK combines drivetests and walktests for executing detailed voice and data measure­ments under controlled circumstances with a sophisticated crowdsourcing approach. This provides profound insights into the overall coverage of voice, data and 4G services, real-world User Download Speeds and Data Service Availability. For the first time in the UK, this year we have extended our walktest to also cover railway connections. Umlaut‘s holistic approach to network benchmarking includes both drivetest and walktest measurements as well as crowdsourcing. The drive and walktests allow for evaluating the cutting edge of the networks‘ capabilities. Crowdsourcing unveils the service quality, performance and coverage actually experienced by the users. We have thoroughly weighted these components in order to give a realistic and authoritative assessment of the rated networks‘ true potential and performance.

EE TAKES THE OVERALL LEAD, VODAFONE ALSO STRONG IN CITIES AND ON THE ROADS. THREE OFFERS BEST DATA PER-FORMANCE IN TOWNS, O2 RANKS SECOND IN CROWD SCORE.

In the voice category, EE is clearly leading, showing  strong results in all tested scenarios and delivering excellent speech quality due to its use of the modern EVS (Enhanced Voice Services) codec. Vodafone offers excellent success ratios for voice calls in the city walktests together with EE. Three ranks third and O2 last, with both offering good voice results in the towns. In the data tests, EE shows the best results as well. This is applicable for the city drivetests, on the roads and – at a lower level – on railways. In the city walktests, EE and Vodafone are on a par, with the other two lagging behind. Three shows the best data performance in towns. In the crowd score, EE takes the lead in all categories, particulary in the crowdsourced quality. It did not suffer any service degradations from May to October 2018. O2 ranks second in the crowdsourcing and Vodafone third, only one point behind O2. Three follows at some distance on the last rank in the crowd discipline but proves to be also very reliable with only hour of service degradation in the observation period. All UK operators show considerable room for improvements on the railways.

TEST ROUTE

THE UK’s OPERATORS

EACH OF THE FOUR MOBILE NETWORKS COMPETING IN THE UK CLAIMS TO HAVE GROWING SUBSCRIBER NUMBNERS. EE AND O2 ARE THE LARGEST PLAYERS, FOLLOWED BY VODAFONE, WITH THE SMALLER THREE ATTACKING WITH AGGRESSIVE TARIFFS.

ee_gruen_weiss

EE

With approximately 30 million customers, EE (formerly Everything Everywhere) is the biggest mobile network operator in the UK. Since January 2016, EE has been part of the British Telecom Group. EE started offering a 4G service in October 2012.Regarding their 4G ­coverage, EE has chosen to report geographic coverage instead of population coverage from now on. In this context, they refer to the Connected Nations ­Report by UK’s ­regulator ­Ofcom and thus quote to ­currently offer 84 per cent 4G geographic coverage. EE operates its 4G network at 800 MHz, 1800 MHz and 2600 MHz. Additionally, it operates  2G at 1800 MHz and 3G at 2100 MHz which also reach a population coverage in the 98 to 99 per cent range. EE operates a growing number LTE Advanced network cells that support speeds of up to 450 Mbps under the name “4GEE”. Also, EE offers Voice over LTE (VoLTE) in most of its 4G network. All plans come with 4G at no extra cost.

logo_vodafone-300x217

Vodafone

Vodafone UK is part of the Vodafone Group which is also headquartered in the UK. The Vodafone Group owns and operates networks in 21 countries. Vodafone UK launched 4G/LTE in 2013. With around 17 million mobile subscribers, Vodafone is the third largest mobile network in the UK after EE and O2.  In June 2012, Vodafone and O2 signed a deal to “pool” their network technologies, creating a single national grid of 18,500 transmitter sites. Both networks however announced they would continue to use their own independent spectrum. Vodafone operates 4G/LTE at 800, 1800, 2100  and 2600 MHz and claims to cover 99 per cent of the UK population. Additionally, Vodafone offers 2G at 900 and 1800 MHz plus 3G at 900 and 2100 MHz.

With “Carrier Aggregation”, Vodafone is upgrading its 4G network to “4G+”, offering up to 1 Gbps – as well as Voice over LTE (VoLTE) telephony.

3

Three

Three UK is a subsidiary of Hutchison Whampoa and launched its mobile service in the UK in 2003. As a relatively young operator Three started as a 3G-only network supplemented by 2G via national roaming. In December 2013, Three began to roll out its 4G/LTE service and expanded it rapidly all over the UK.  With about 10.1 million customers Three is the smallest mobile network operator in the UK but claims to carry over 36 per cent of the nation‘s data traffic. Offering the cheapest price for 4G and unlimited data plans (excluding tethering) may well support this claim. In addition to 1800 MHz, Three offers 4G also at 800 MHz as well as 3G on 2100 MHz. The company currently claims to cover 99 per cent of the UK‘s population with at least 3G. According to their own account, abut 84 per cent of Three’s customer base is using 4G. The operator has also started to deploy Voice over LTE (VoLTE) in most of its 4G network.

.

o2

O2

O2 claims to be the second largest mobile network operator in the UK with approxi­mately 25 million customers. Formerly a subsidiary of British Telecom, O2 plc was purchased by the Spanish telecommunications com­pany Telefónica in 2006. Today, the company also owns half of the mobile virtual network operator Tesco Mobile which operates on the O2 network in the UK. O2 started providing its 4G service in August 2013 and has expanded this service across the UK since. In November 2018, O2 claimed to cover approximately 99 per cent of the UK population with its 4G service. O2 operates its 4G network mainly on 800 MHz with ­limited additional 1800 and 2100 MHz coverage in metropolitan areas. Additionally, O2 provides 2G on 900 and 1800 MHz and 3G on 900 and 2100 MHz. Like the other UK opera­tors, O2 has rolled out Voice over LTE (VoLTE) in most of its 4G network.

A CLOSE LOOK AT UK NETWORKS

WITH THE ADDITION OF CROWDSOURCING, UMLAUT AND CONNECT HAVE EXTENDED THEIR SCOPE IN ORDER TO PRESENT AN EVEN MORE COMPLETE LOOK AT NETWORK QUALITY AND PERFORMANCE.

Umlaut, headquartered in Aachen, Germany, is a world leader in mobile network testing. The company has over 4,100 employees worldwide and a turnover of more than 350 million Euros. Umlaut is partnering with the international telecommunications magazine connect, which has 25 years of editorial expertise and is one of the leading test authorities in Europe for telecommunications products and services. Together, umlaut and connect have been conducting the most important network benchmark test in Germany for more than 15 years, extending it to Austria and Switzerland since 2009.

Starting in 2014, umlaut has also been conducting benchmarks in the UK and Australia, expanding its public mobile network tests to the Netherlands, Spain and Sweden in 2016 as well as examining many other mobile networks all over the world including those in the USA and Singapore. The 2018 umlaut connect Mobile Benchmark in the UK consists of drivetests and walktests conducted from October 30 to December 3, 2018. Four drive test cars together covered about 11,750 kilometres, visiting 22 cities and 35 towns. Additionally, two walktest teams visited ten cities and travelled on trains between them. The test areas account for approximately 16.6 million people, or about 26.2 per cent of the total population of the United Kingdom.

 

“The UK’s networks improved overall on last year. We adapt our testing methodology every year to keep close to technological advances as well as to the actual experience of endusers. This year, we have added the railway measurements into our scope for the umlaut connect Mobile Benchmark in the UK and derived some interesting conclusions. For next year, we expect the first developments in 5G to come.”


Hakan Ekmen, CEO of umlaut communications GmbH

 

VOICE

MANY CUSTOMERS USE VOICE SERVICES LESS INTENSELY THAN DATA. HOWEVER, WHEN ACTUALLY TAKING OR PLACING A PHONE CALL, THEY EXPECT RELIABLE CONNECTIONS. HOW DO THE UK NETWORKS MANAGE TO FULLFIL THESE EXPECTATIONS?

EE LEADS IN VOICE, SHOWING VERY GOOD RESULTS IN CITIES, TOWNS AND ON ROADS. IN TRAINS, THE RESULTS ARE IN NEED OF IMPROVEMENT.

All four operators in the UK now support Voice over LTE (VoLTE). VoLTE transmits voice calls as data packets over a 4G connection. This way, the otherwise necessary “circuit-switched fallback“, which forces smartphones to switch back to 3G in order to take or place a phone call, can be avoided. Also, VoLTE supports better audio codecs providing operators with the opportunity to deliver higher speech quality to their customers. For the voice rating, each drivetest car and each walktest team carried one Samsung Galaxy S8 smartphone per operator. The phones in the cars called a counterpart in one of the other cars. The phones carried by the walktest teams called a stationary counterpart. The connected testing equipment registered success ratios, call setup times and speech quality. In order to simulate normal smartphone usage, data transfers took place in the background of the test calls.

EE LEADS IN THE BIG CITY DRIVETESTS
In the voice drivetests conducted in 22 UK cities, EE achieves the highest score. It offers the highest call success ratios, shortest call setup times and best speech quality. Vodafone ranks second due to slightly lower KPIs in all these ca­tegories. Three follows on the third rank, being on par with Vodafone in terms of success ratios and speech quality, but showing longer setup times. O2 ranks last.

EE AHEAD IN BIG CITY WALKTESTS
The walktest teams generally captured a higher performance level for voice in the cities than the drive­test teams – except for O2 which reaches a degree of fullfillment of 75 per cent in both disciplines. EE shows an impressive success ratio of 100 per cent and also very good results for the other KPIs. The ranking is the same as in the drivetests. O2 offers nice speech quality, but lags behind in the other KPIs.

EE HAS ALSO BEST RESULTS IN TOWNS
The results of the drivetests leading through 35 towns accounts for the same ranking as in the cities. EE is leading once again, followed by Vodafone, and Three is close behind. O2 ranks last. Interestingly, Three and O2 achieve slightly better results in the smaller towns than in the big cities. Also, it is pleasing to see that the performance of all four candidates in the towns went up compared to the previous year.

EE STREETS AHEAD ON ROADS
When it comes to conducting voice calls on the roads, EE is again in the lead. Vodafone ranks recond, with Three and O2 following at some distance. Again, compared to last year’s results, all four UK operators managed to increase their performance in this discipline. With call success ratios aorund 97 per cent, even the weaker contenders offer an acceptable level of voice service.

EE LEADS IN OVERALL WEAK RESULTS ON BRITISH RAILWAYS
For the first time in the UK, this year’s umlaut connect Mobile Benchmark also comprises walktests conducted while travelling on British trains. However, their results are quite sobering: Even EE as the strongest candidate only achieves 48 per cent of the possible points in this discipline. Vodafone follows with a meagre 29, Three at 27 and O2 at 24 per cent. The main problem is the very limited call success ratios – our tests results indicate that EE customers only stand a 87 per cent to get through when trying to conduct a phone call while travelling on a British train. This chance drops to 81 per cent in Three’s and O2’s networks and to even 78 per cent at Vodafone. The latter at least offers better call setup times when such a call attempt actually succeeds.

VOICE RESULTS AT A GLANCE
In the voice category, EE is clearly leading the field. It shows strong results in all aggregations. As EE mainly uses the modern EVS codec, it delivers excellent speech quality. Vodafone follows on the second rank, offering excellent success ratios in the city walktests together with EE. Three ranks third, but is in general on a par with Vodafone in success ratios, lagging behind mainly in call setup times. O2 comes in last, but offers good results especially in smaller towns. The newly added railway measurements reveal big challenges for all operators.

DATA

WITH THE VOLUME OF TRANSMITTED DATA ­PERMANENTLY GROWING, DATA CONNECTIVITY CONSTANTLY BECOMES MORE IMPORTANT. WHICH OPERATOR IN THE UK ­MANAGES BEST TO MEET THE INCREASING DEMAND?

EE TAKES THE OVERALL LEAD IN THE DATA DISCIPLINE AS WELL. IN THE CITY WALKTESTS, VODAFONE IS ON A PAR WITH EE. THREE SCORES BEST IN TOWNS.

Data connectivity is the most prestigious discipline in our benchmark and also in the operators’ marketing. All four UK networks claim to cover a large part of the population with LTE services – the claimed percentages range in the high nineties. Three states a combined 3G/4G data coverage of 99 per cent of the population and specifies that about 84 per cent of its 10.1 million customers are using 4G (also see page 3). All four operators are continuing to spend millions and billions on upgrading and expanding their networks to meet the growing demand. However, the four operators pursue different rollout strategies. EE and Vodafone chase each other with continually growing data rates that currently go up to 1 Gbps, based on the so-called carrier aggregation (the combination of multiple carrier frequencies). In contrast, O2 and Three stick around a solid 100 Mbps and mainly focus on enlarging their 4G footprints.

UMLAUT’S SCORING REWARDS HIGH PERFORMANCE AS WELL AS THE NETWORKS’ AVAILABILITY AND STABILITY
In order to assess the performance and reliability of data connections, each of our four drivetest cars and also the walktest team carried one Samsung Galaxy S8 per operator. Supporting the LTE category 16, these smartphones were able to benefit from carrier aggregation up to 1 Gbps. Umlaut‘s testing considers such fast throughputs as well as the networks‘ availability and stability.

The benchmarking of webpage downloads as well as file downloads and uploads rewards fast throughputs. At the same time, P3 assesses the networks‘ availabilty and stability by examining success ratios. In order to assess typical performance as well as peak speeds, we determined the minimum data rates that are availability in 90 per cent of the cases plus the peak data rates that would be surpassed in 10 per cent of the cases. YouTube playouts concentrate on reliability aspects. As YouTube streams videos at adaptive bitrates, the average value of the received video resolution is another important performance indicator.

EE TAKES THE LEAD IN CITY DRIVETESTS
In the scoring of their performances determined during the drivetests conducted in 22 UK cities, EE takes the lead, followed by a strong performance by Vodafone. These two operators overall deliver quite high success ratios, convincing download speeds and Youtube playback performance. O2 and Three fall behind, with O2 scoring slightly better than Three in this category.

EE AND VODAFONE ON A PAR  IN BIG CITY WALKTESTS
In the data results of the walktests that were conducted in ten large cities of the UK, EE and Vodafone achieve the same score. Looking closer at the single KPIs, both are on a par in the web tests, EE is a little ahead in the file transfer tests and Vodafone lead regarding Youtube playback. The gap between the leading duo and the other contenders is a little more pronounced, with O2 falling behind distinctly in this category. Among other weak points, O2 shows noticeably low success ratios when it comes to playing back Youtube videos. In terms of using data connectivity while walking in the big cities, there is clearly a two-tier system in the UK.

THREE IS DATA CHAMPION IN TOWNS
Providing data connectivity in towns seems to be the domain of Three. Here the smallest UK operator even outperforms the strong contenders EE and Vodafone. O2 ranks last, offering the same overall performance as in the larger cities.

EE RANKS FIRST ON THE ROADS, BUT GOOD RESULTS FOR ALL CONTENDERS
The scores achieved by the UK operators on the connecting roads is altogether pleasant. EE leads, but the other three contenders show also good results. More good news: All four operators show better data performances on the roads compared to their results from last year.

EE BEST ON RAILWAYS, BUT OVERALL MUCH ROOM FOR IMPROVEMENT
While there is good news concerning data performance on the roads, the same can unfortunately not be said about the railways. Only EE shows relatively acceptable performance in this discipline. Three ranks second, Vodafone third and O2 last – each at a distinct distance to their competitors.

DATA RESULTS AT A GLANCE
EE shows the best data performance and reliability in the city drivetests, on the roads and – at a lower level – on railways. In the city walktests, EE and Vodafone are on a par, with the other two operators distinctly lagging behind. Three offered the best data performance in towns. Alongside the other operators, O2 achieves relatively good results on the roads. All UK operators show considerable room for improvements on the railways.

CROWD

IN THIS YEAR, THE RESULTS OF CROWDSOURCING ANALYSES ARE PART OF THE TOTAL SCORE FOR THE FIRST TIME. 268,000 USERS IN THE UK HAVE CONTRIBUTED TO THE DATA GATHERING THAT TOOK PLACE FROM AUGUST TO OCTOBER, 2018.

EE TAKES THE LEAD IN CROWDSOURCED QUALITY AND OVERALL IN ALL CROWDSOURCED CATEGORIES.

While the drivetests and walktests determine the peak performance of the examined networks, crowdsourcing can add important dimensions such as time, geography or variety in devices and tariff plans – if done in the right way. For the collection of crowd data, P3 has integrated a background diagnosis pro­cesses into more than 800 diverse Android apps. If one of these applications is installed on the enduser’s phone and the user authorizes the background analysis, data collection takes place 24/7, 365 days a year. Reports are generated for every quarter of an hour and sent daily to P3‘s cloud servers. Such reports generate just a small number of bytes per message and do not include any personal user data. A more detailed description of our crowdsourcing methodology can be found on page 15 of this report. Based on the total population count of 66 million people, one of 246 inhabitants of the United Kingdom has contributed to the gathering of 2.99 billion single samples of crowd data. The considered test area represents 99 per cent of the built-up area of the UK.

EE LEADS THE FIELD REGARDING CROWDSOURCED COVERAGE
All UK operators show a good score level for their voice and data coverage. In the Quality of Voice Coverage, EE, Vodafone and O2 are more or less on a par, In the Quality of Data Coverage (which considers 3G plus 4G), Three takes a narrow lead. The Quality of 4G Coverage leaves some room for improvement. In particular, Three falls behind competition regarding 4G coverage for both footprint (Test Area Coverage) as well as the likelihood of being able to actually use 4G service (4G Quality of Coverage). Looking at all three coverage KPIs combined, EE leads the field.

ASSESSING COVERAGE BASED ON CROWD KPIS
Our coverage metrics correnspond to the results of our drivetests and walktests. However, it is no surprise that the crowdsourced KPIs for voice, data and 4G coverage deviate to a certain extent from the population coverage values stated by the operators: P3’s gathering of crowd data reflects where people actually are and move as opposed to their places of residence and working. Furthermore, our crowdsourcing also comprises indoor or other disadvantageous reception situations, while operators commonly base their claims on outdoor reception only.

EE SHOWs FASTEST DOWNLOAD SPEEDS IN TOP 10 PER CENT OF EVALUATION AREAS
In the assessment of download speeds available to the users, EE shows the best results for the top 10 per cent of Evaluation Areas. In this category as well as in the assessment of average download speeds, Three scores ahead of Vodafone and O2. Generally, it must be taken into consideration that a part of the actual user base probably experiences data speed limitations caused by their mobile tariffs.

EE DID NOT SHOW ANY SERVICE DEGRADA­TIONS FROM MAY 2018 TO OCTOBER 2018
Other than the rest of the crowd KPIs, our examination of Data Service Availability covers six months (May to ­October 2018). EE scores best in this consideration again, showing no service degradations in the observation period. Three ranks second with a one hour outage in July. Vodafone suffered two outages – one in August and one in October. O2 ranks last in this category, because our examination revealed three outages in July, August and October.

 

CROWD RESULTS AT A GLANCE
EE takes the lead in the crowdsourced quality, and overall in all crowdsourced categories. It offers the highest data rates and did not suffer any service degradations from May to October 2018. O2 ranks second in all crowdsourced categories, and Vodafone third, only one point behind O2. Three follows at some distance on the last rank in this discipline.

LONDON

TRADITIONALLY, UMLAUT AND CONNECT TAKE A CLOSER LOOK AT THE UK‘S CAPITAL TO SEE HOW THE OPERATORS COVER THIS LIVELY CENTRE OF BUSINESS, POLITICS AND SOCIETY.

Being by far the most densely populated area in the UK, London is an especially demanding terrain for deploying and maintaining a mobile network. This is why we regularly take a closer look to see how coverage in the capital compares to the rest of the United Kingdom. Therefore we have filtered the results of the voice and data drive and walktests as well as the crowd results obtained in the London area from the nationwide values. As the city scores neither contain the results of the drivetests conducted in the smaller towns or on the connecting roads nor the walktest results from the trains, we have adapted the maximum achievable points accordingly. So while the partial scores for voice, data and crowd can be compared with the nationwide results, this should not be done with the points for the drivetests and walktest.

EE LEADS IN LONDON TOO, VODAFONE RANKS SECOND AND SCORES SLIGHTLY BETTER THAN IN THE WHOLE UK
The overall winner in London is EE – much the same as in the nationwide assessment. In the capital, this operator achieves somewhat higher scores than in the overall results. Interestingly, its drivetest results are slightly lower than in the whole country assessement. In the walktests, EE scores slightly better in the capital or, in the case of the voice measurements, it achieves the same score of a full 100 per cent. The crowd results in London are also a little higher than the nationwide score. EE achieves the same 100 per cent score as all over the UK. Vodafone ranks second in London as well, but achieves better scores in London for the voice drivetest as well as for the data drive and walktest results compared to its nationwide score. Also, its crowd score for London is considerably higher than all over the country. The scores of the voice walktests conducted in London fall a little behind the nationwide average.

O2 OVERTAKES THREE IN LONDON, BOTH ARE CLOSE TOGETHER
In contrast to the nationwide ranking, O2 achieves the third rank in London and overall scores two points ahead of Three. In comparison to the whole UK, O2 in London shows particularly better results in the voice drive and walktests and in the crowd score, but falls behind in the data drivetests and walktests. Notably, Three‘s results in London are behind the nationwide average with the only exception of the crowd score.

LONDON RESULTS AT A GLANCE
As in the nationalwide results, EE is also the winner in London. It achieves better or the same results than in the whole UK, particularly in the walktests. The drivetests conducted in London show weaker results – which is also true for Vodafone and Three, as well as O2’s data performance. Still, Vodafone reaches a strong second rank in the capital. O2 overtakes Three in London, with both contenders scoring close together.

 

UK’s LARGEST CITIES

FOR THE INHABITANTS OF OTHER LARGE CITIES IN THE UK, IT IS INTERESTING TO SEE HOW THE DIFFERENT OPERATORS PERFORM IN THEIR AREAS. THEREFORE, WE PERFORMED ADDITIONAL ANALYSES FOR NINE LARGE CITIES ALL OVER THE UNITED KINGDOM.

As interesting as the focus on the densely populated London area is, the inhabitants of other large UK cities and the capitals of the other countries besides England have their own perspective on network performance and availability. In order to also provide valuable insights for their inhabitants, we have additionally analysed the performances of the four operators in nine large UK cities – specifically Belfast, Birmingham, Bristol, Cardiff, Edinburgh, Glasgow, Liverpool, Manchester and Sheffield.

SAME RANKING AS NATIONWIDE IN BELFAST, LIVERPOOL AND SHEFFIELD: EE LEADS, VODAFONE SECOND
The same ranking as in the nationwide assessment can be seen in Belfast, Liverpool and Sheffield. Here, EE is the clear leader, followed by Vodafone. Three ranks third, and O2 last. While in Belfast and Sheffield this ranking is quite distinct, all four contenders score quite close together in Liverpool. Here, even Three and O2 show comparably strong results.

VODAFONE OVERTAKES EE IN BIRMINGHAM, BRISTOL, CARDIFF, GLASGOW AND MANCHESTER
In Birmingham, Bristol, Cardiff and Manchester, the fight for the top rank is close, but Vodafone manages to overtake a still strong EE in these cities by a close margin. In Birmingham, Vodafone is stronger in voice than EE, in Cardiff and Glasgow stronger in data. In Bristol and Manchester, Vodafone outranks the nationwide winner in both disciplines. Their crowd scores are in most cases close together. In Birmingham and Manchester, O2 shows stronger results than Three. In Bristol, Cardiff and Glasgow it is just the other way round.

EE LEADS IN EDINBURGH, BUT O2 RANKS SECOND, THREE THIRD AND VODAFONE LAST
In Edinburgh, it is no surprise that EE ranked first, but the order of the other contenders is interesting. In the capital of Scotland, O2 shows a particularly strong performance and ranks second due to especially good data results. The crowd scores confirm the leading position of EE, the other three operators rank close together in this perspective. Three even manages to overtake Vodafone which can be explained by Vodafone‘s eye-catchingly weak data results in this city.

 

METHODOLOGY

The umlaut connect Mobile Benchmark in the UK comprises of the results of extensive voice and data drivetests and walktests as well as a sophisticated crowdsourcing approach.

DRIVETESTS AND WALKTESTS
The drivetests and walktests in the UK took place from October 30 to December 3, 2018 All samples were collected during the day, between 8.00 a.m. and 10.00 p.m. The network tests covered inner-city, outer metropolitan and sub­urban areas. Measurements were also taken in smaller towns and on the connecting highways. The four measurement cars together covered about 3,750 kilometres in the cities, about 1,700 km in towns and about 6,300 km on the roads – resulting in a total of 11,750 kilometres. The combination of test areas has been selected to provide representative test results across the UK population. The areas selected for the 2018 test account for approximately 16.6 million people, or roughly 26.2 per cent of the total population of the UK. The drivetests covered 22 cities and 35 towns. Additionally, two teams conducted walktests in 10 cities and also on railway journeys between them. The routes are shown on page 1, all visited cities and towns  are listed in the box on the right.  The four drivetest cars as well as the battery-powered backpacks of the walktest teams were equipped with arrays of Samsung Galaxy S8 smartphones for the ­simultaneous measurement of voice and data services.

VOICE TESTING
One smartphone per operator in each car was used for the voice tests, setting up test calls from one car to another. The walktest team also carried one smartphone per operator for the voice tests. In this case, the smartphones called a stationary counterpart. The audio quality of the calls was evaluated using the HD-voice capable and ITU standardised POLQA wideband algorithm. All smartphones used for the voice tests were set to VoLTE preferred mode. In networks or areas where this modern 4G-based voice technology was not available, they would perform a fallback to 3G or 2G.

As a new KPI in 2018, we assess the so-called P90 value for call setup times. P90 values specify the threshold in a statistical distribution, below which 90 per cent of the gathered values are ranging. In order to account for typical smartphone use during the voice tests, background data traffic was generated through random injection of small amounts of HTTP traffic. The voice scores account for 34 per cent of the total results.

DATA TESTING
Data performance was measured by using three more Galaxy S8 in each car or per walktest team – one per operator. Their radio access technology was set to LTE preferred mode.
For the web tests, they accessed web pages according to the widely recognised Alexa ranking In addition, the static Kepler test web page as specified by ETSI (European Telecommunications Standards Institute) was used. In order to test the data service performance, files of 3 MB and 1 MB for download and upload were transferred from or to a test server located on the Internet. In addition, the peak data performance was tested in uplink and downlink directions by assessing the amount of data that was transferred within a seven seconds time period. The evaluation of YouTube playback takes into account that YouTube dynamically adapts the video resolution to the available bandwidth. So, in addition to success ratios, start times and playouts without interruptions, the measurements also determined average video resolution.

All the tests were conducted with the best performing mobile plan available from each operator. Data scores account for 51 per cent of the total results.

CROWDSOURCING

Additionally, umlaut conducted crowd-based analyses of the UK networks which contribute 15 per cent to the end result. They are based on data that were gathered in August, September and October, 2018. For the collection of crowd data, umlaut has integrated a background diagnosis processes into 800+ diverse Android apps. If one of these applications is installed on the end-user’s phone and the user authorizes the background analysis, data collection takes place 24/7, 365 days a year. Reports are generated for every quarter of an hour and sent daily to umlaut‘s cloud servers. Such reports contain just a small number of bytes per message and do not include any personal user data. Interested parties can deliberately take part in the data gathering with the specific ”U get“ app (see box below on the right).

NETWORK COVERAGE
For the assessment of network coverage, umlaut lays a grid of 2 by 2 kilometres over the whole test area. The “evaluation areas“ generated this way are then sub-divided into 16 smaller tiles. To ensure statistical relevance, umlaut requires a certain number of users and measurement values per operator for each tile and each evaluation area. If these thresholds are not met by one of the operators, this part of the map will not be considered in the assess­ment for the sake of fairness. “Quality of Coverage“ reveals whether voice and data services actually work in an evaluation area. Umlaut does this because not in each area that allegedly provides network reception, mobile services can actually be used. We specify these values for the coverage of voice services (3G and 4G combined), data (3G and 4G combined) and 4G only.

viating from the other crowd score elements, Data Service Availability is rated based on a six-month observation period – in this case from May to Oct 2018.

DATA THROUGHPUTS
Additionally, umlaut investigates the data rates that were actually available to each user. For this purpose, we determine the best obtained data rate for each user during the evaluation period and then calculate their average value. In addition, we determine the so-called P90 values for the top throughput of each evaluation area as well as of each user‘s best throughput. P90 values specify the threshold in a statistical distribution, below which 90 per cent of the gathered values are ranging and depict how fast the network is under favorable conditions.

DATA SERVICE AVAILABILITY
Formerly called “operational excellence“, this parameter indicates the number of outages or service degradations – events where data connectivity is impacted by a number of cases that significantly exceeds the expectation level. To judge this, the algorithm looks at a sliding window around the hour of interest. This ensures that we only consider actual degradations as opposed to a simple loss of network coverage due to prolonged indoor stays or similar reasons. In order to ensure statistical relevance, each operator must have sufficient statistics for trend and noise analyses per each evaluated hour. The exact number depends on the market size and number of operators. A valid assessment month must comprise of at least 90 per cent of valid assessment hours. Deviating from the other crowd score elements, Data Service Availability is rated based on a six-month observation period – in this case from May to Oct 2018.

CONCLUSION

EE IS THE WINNER WHILST VODAFONE ACHIEVED A STRONG SECOND PLACE RANK. THREE SHOWS SOME SPECIFIC STRENGTHS BUT FELL BEHIND IN THE OVERALL SCORING, WHICH IS SIMILAR FOR VODAFONE. O2 PERFORMED SIMILARLY TO LAST YEAR.

EEs confirms its top rank from previous years and wins the umlaut connect Mobile Benchmark in the UK for the fifth time in a row (in 2016, EE and Vodafone won together). Compared to last year’s results, EE improves significantly in the voice discipline, but achieves a slightly decreased data score. In the now extended crowd analyses, EE is also the winner. Vodafone achieves a good second rank, showing especially good results in the big cities and on the roads. In comparison to the previous year, Vodafone managed to improve in the voice results, but lost some ground in the data scores. The same is also true for Three, which fell back to the grade “satisfactory”. A remarkable strength of this smallest operator in the UK is that it achieved the best data score in towns. Like all UK ­operators, Three was also able to increase its performance on the roads for both voice and data. O2 ranks last. Showing more or less the same performance than last year costs some points, because the requirements of our testing methodology increase year over year. In the crowdsourced assessment, O2 achieved the second-best results, one point ahead of Vodafone. However, our newly added railway measurements reveal big challenges for all UK operators.

ee_gruen_weiss

EE

The overall winner of the 2018 P3 connect Mobile Benchmark in the UK is EE. This operator clearly leads in all tested disciplines including the now extended crowd score and also leads the field in most of the single categories of our assessment. EE’s strong voice results are based on its use of the modern EVS codec.

logo_vodafone-300x217

Vodafone

Vodafone achieves an overall good second rank. This operator scores particularly strong in the big cities and on the connecting roads. In the data walktests in big cities, Vodafone is on a par with EE. In the crowd analyses, Vodafone achieves an overall good result, scoring one point behind O2.

3

Three

The smallest UK operator ranks third overall, but scores best in the data drivetests in smaller towns. Like all UK operators, Three increases its performance on the roads. Also, it shows a good Data Service availability with just one hour of degradation in the observation period. A decrease in its data scores may be a result of owning the least amount of spectrum in the UK.

o2

O2

The third largest operator in the UK all in all scores the same performance as last year – which due to our increasing requirements leads to a net loss in points. However, in our extended crowd score, O2 ranks second. Like Three, O2 offers good voice results particularly in smaller towns. As its competitors, O2 also improved on the roads.

Last year’s report can be downloaded here