Download the report as PDF here
2018│04│10 – Public benchmark report in the Netherlands
The benchmarking expert P3 has been testing the Dutch mobile networks since 2015, showing always excellent and continuously increasing network performance in the Netherlands. With significant changes in the market due to recent mergers, our fourth mobile network test once more promises an exciting race. How did the four independent candidates perform this time?
RESULTS IN A NUTSHELL
P3‘s network benchmarks are widely accepted as the de-facto industry standard and for being highly objective. The carefully designed metho- dology of our 2018 benchmark in the Nether- lands combines two cars conducting drivetests in 21 Dutch cities and on the connecting roads between them with walktests in seven larger cities and tests on Dutch railways. The areas in which we have tested account for more than 5.7 million people, or for about 34 per cent of the Dutch population.
P3‘s measurement included the use of up-to- date LTE Cat 9 as well as VoLTE-capable smart- phones for the tests. Also, we constantly readjust the thresholds of our evaluation with steps like these we take the latest technical developments of the mobile networks into account. The scope of our benchmark is to show how the mobile networks perform at the edge of what is techni- cally feasible – and to what extent do customers benefit from these capabilities? In order to pro- vide valid assessments, we have also used the most comprehensive mobile plans available from each operator.
T-Mobile,Vodafone and KPN have improved compared to their 2017 scores.
As in the two previous years, the overall winner is T-Mobile. This time, the winner even achieves an impressive 977 points – the highest score which has ever been achieved in a P3 connect Mobile Benchmark – and the grade „outstanding“.
Vodafone follows closely behind achieving the same grade. KPN ranks third with the overall grade “very good”. All three operators were able to improve clearly compared to their 2017 re- sults, with Vodafone making the biggest leap (+31 points), followed by T-Mobile (+21 points) and KPN (+10 points). Only Tele2 has lost some ground (-11 points) compared to its 2017 score.
T-Mobile maintains the lead in almost every service and category. However, all in all T-Mobile, Vodafone and KPN show excellent results. KPN has lost points because of slightly inferior speech quality and longer call setup times. All Dutch ope- rators show a strong performance for all data services on connecting roads as well as in trains.
In order to assess the performance and reliability of data connections, our two drivetest cars carried two smartphones per operator for the data tests: One Samsung Galaxy S7 and one Sony Xperia XZ. This choice was made in order to take the variations into account that result from the interaction of different smartphone types with the different networks. The walktest team carried one Samsung Galaxy S7 per operator. Supporting the LTE category 9, both smartphone types were able to benefit from the so-called aggregation of three LTE carrier frequencies which the Dutch operators offer under the name of 4G+ and which could theoretically transmit up to 450 Mbps. P3‘s testing considers fast throughputs as well as the networks‘ availability and stability. Web page and file downloads or file uploads reward fast speeds, while recording success ratios and assessing YouTube playouts concentrate on reliability aspects. As YouTube streams videos at adaptive bitrates, the average value of the received video resolution is another important performance indicator.
Overall high data performance in large Dutch cities
In the drivetests conducted in the cities, T-Mobile is ahead with 97 per cent of the achievable points, but
its competitors follow closely. Vodafone and KPN both score 95 per cent, while Tele2 scores 93 per cent of possible points in this category
T-Mobile and Vodafone ahead in data walktests
The data walktests conducted in seven large Dutch cities (see page 6 for the complete list) yield very similar results. In this category, however, T-Mobile and Vodafone share the top position, and KPN falls marginally behind. Tele2 scores weaker in this discipline, due to slightly lower success ratios. But even the smallest Dutch network still achieves 88 per cent of the possible points.
T-Mobile leads in drivetests in smaller towns, KPN, Vodafone and Tele2 follow closely
In the drivetests performed in the smaller Dutch towns, T-Mobile achieves the highest score and KPN ranks second, while Vodafone and Tele2 follow closely behind. This ranking order can be seen in measurement values like the speeds of file downloads and uploads. All four operators achieve excellent results when it comes to accessing YouTube while driving in smaller towns.
All Dutch operators share very strong results on connecting roads
A similar ranking can be observed on the connecting roads. T-Mobile also leads in this discipline with 99 per cent of the possible points. KPN ranks second with 98 per cent, while Vodafone and Tele2 share the third rank with 97 per cent of the achievable points. These results are rather sensational, attesting that Dutch drivers can ex- pect top mobile connectivity on the roads of their country.
T-Mobile and KPN achieve highest scores for data connectivity on Dutch railways
Our test of data connections in Dutch trains again demonstrates the high quality of mobile coverage in the Netherlands. T-Mobile leads the field and even scores higher than in the voice discipline. KPN ranks second but scores a little lower than with its voice connections on trains. Vodafone achieves the third and Tele2 the fourth rank in this discipline. Again, compared to the results in other European trains, the Dutch operators show excellent performances for data connections in trains.
Data results at a Glance
T-Mobile also leads the data discipline of this test. Although KPN has lost some points in comparison to its 2017 data results, the largest Dutch operator still ranks second in the disci- pline. Vodafone achieves the third rank but shows the biggest improvement compared to last year’s data results. Tele2 scores last in this discipline and falls some points behind its 2017 data results, still achieving a very good overall performance in the data discipline.
In 2000, the German Deutsche Telekom bought a minority of the Dutch mobile network operator Ben, which two years later was extended to a 100 per cent acquisition. In 2003, Ben was renamed T-Mobile Netherlands, with the brand “Ben“ becoming a “no-frills“ offer within its portfolio. In 2007, T-Mobile Netherlands additionally acquired the ope- rator Orange. At the end of its fiscal year 2017, T-Mobile Netherlands reported 3.85 million customers and a revenue of 1.4 billion Euros which equals a mobile market share of about 21 per cent. In December 2017, the company announced to acquire Tele 2.
T-Mobile Netherlands operates 2G mostly at 1800 MHz, 3G at 900 and 2100 MHz and 4G/LTE at 900, 1800, 2100 and 2600 MHz. The company claims a 4G coverage of more than 99 per cent of the Dutch population. With 4G+, T-Mobile network offers a maximum download speed of 300 Mbit/s. VoLTE was introduced in October 2017.
The P3 connect Mobile Benchmark in the Netherlands took place from February 16th until March 10th, 2018. All samples were collected between 8am and 10pm. The network tests covered 21 cities, of which 16 are counting more than 100 000 inhabitants, and 31 smaller towns. Furthermore, our test routes included 3,600 kilometres of connecting roads. Additionally, a walktest team visited seven larger cities. The team took the trains between the cities and also into more rural areas of the country, performing voice and data tests on the railways as well.
This combination of test areas has been carefully selected to provide a significant series of test results covering the Dutch population. The areas chosen for the 2018 test account for about 5.8 million people, or 34 per cent of the total Dutch population.
P3 conducted the tests with two drive-test cars, equipped with arrays of Samsung Galaxy S7 Cat 9 smart- phones (voice) as well as a mixed allocation of Samsung Galaxy S7 and Sony Xperia XZ Cat 9 smartphones (data) for the simultaneous measurement of voice and data services.
Two smartphones per operator in each car were used for the voice tests, setting up test calls from one car to another. The walktest team carried one smartphone per operator for the voice tests. In this case, the smartphones called a stationary counterpart. The audio quality of the transmitted speech samples was evaluated using the HD-voice capa- ble and ITU standardised so-called POLQA wideband algorithm. All Dutch operators offer 4G capable subscrip- tions. In total, two voice channels were in operation. Both smartphones establishing the first voice test channel were set to 4G preferred mode forcing the network to perform a fallback to UMTS in LTE areas. On the second channel, both smartphones were set to VoLTE-preferred to simulate customer behaviour with the latest technology. The smartphones used in the walktests were all set to VoLTE-preferred.
In order to account for typical smartphone-use scenarios during the voice tests, background data traffic was generated in a controlled way through random injection of small amounts of HTTP traffic. The voice test scores account for 40 per cent of the total benchmark results.
Data performance was measured by using four smartphones in each car – one per operator. One car was equipped with four Samsung Galaxy S7 while the other car was carrying four Sony Xperia XZ in order to respect the variable data performance of different smart phones in different networks. In order to further reflect the customer experience, the radio access technology was set to LTE preferred mode. The web tests accessed web pages according to the widely recognised Alexa ranking. In addition, the static “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) was used.
In order to test the data service performance, files of 3 MB and 1 MB for download and upload were transferred from or to a test server located on the Internet. In addition, the peak data performance was tested in uplink and downlink directions by assessing the amount of data that was transferred within a seven seconds time period. Another discipline was the playback of YouTube videos. It took into account that YouTube dynamically adapts the video resolution to the available band- width. So, in addition to success ratios, start times and playouts without interruptions, YouTube measurements also determined the average video resolution.
All the tests were conducted with the best-performing mobile plan available from each operator. Data scores account for 60 per cent of the total results.
Routes and samples
In the selected 21 cities and 31 smaller towns, the cars had to follow predefined routes. Altogether, the two test cars covered more than 6,420 kilometres, of which approximately 2,820 km led through the cities and towns, while 3,600 km covered connecting roads.
Performance indicators and rating
The score weighting reflects both the geographical distribution of the Dutch population and the ranking of usage scenarios. Therefore, 600 of the total of 1000 maximum points were assigned to the cities – 450 maximum points for the drivetest results (voice: max. 180 p., data: max. 270 p.) and 150 for walktests results (voice: max. 60 p., data: max. 90 p.). For the towns, a maximum of 200 points is available (voice: max. 80 p., data: max. 120 p.) and for the connecting roads a maximum of 125 points (voice: max. 50 p., data: max. 75 p.). The tests conducted in trains account for another 75 points (voice: max. 30 p., data: max. 45 p.) The tables on page 2 and page 11 of this report show the percentage of maximum points that each operator has achieved in each discipline.
T-Mobile wins again. Vodafone shows also “outstanding“ results. A very good KPN ranks third, and Tele2 fourth – but still with a “very good“ score.
For the third time in a row, T-Mobile is the clear winner of the P3 connect Mobile Benchmark
in the Netherlands. Its impressive 977 total points is the highest score ever achieved in any P3 connect benchmark. As the distinct leader in the data category and being on par with Vodafone in the voice discipline, T-Mobile‘s grade “outstanding“ is well deserved. If this ope- rator succeeds in integrating the frequencies and base stations of its projected subsidiary Tele2, it will be excellently positioned for the future.
The pursuer Vodafone also shows an “out- standing“ performance in this year‘s bench- mark. It achieves the second rank especially due to excellent voice results which are the same a those of test winner T-Mobile, while in the data discipline, Vodafone is almost at the same level as the third-ranking KPN.
KPN is strong in the data discipline, where it ranks second behind test winner T-Mobile. But the Dutch market leader loses some points in the voice category due to slightly lower speech quality and longer call setup times. However, all in all KPN still achieves a very good result.
Tele2, the smallest Dutch operator and des- tined to be owned by T-Mobile, achieves the fourth rank. Due to its concentration on LTE/4G, it achieves a decent score in the data discipli- ne. Its score for data connectivity on the roads is particularly strong. However, Tele2‘s voice score shows some room for improvement. It will be interesting to see how the merger with T-Mobile will affect this operator‘s future results.
When comparing the results of this year‘s mobile network benchmark in the Netherlands to those from recent tests in other European countries, Dutch operators are among the top tiers. Even the fourth-ranking Tele2 scores at a level that would make it a suitable candidate for a winning position in countries with overall weaker mobile networks like Spain or the UK.
The third largest Dutch operator wins the 2018 benchmark with a distinct lead in the data category and a voice result on a par with second ranking Vodafone. The ope- rator earns the grade “outstanding“. Above that, T-Mobile was able to improve on last year’s results both in the voice and in the data discipline
With an “outstanding“ overall performance, Vodafone well earns the second rank. In the voice discipline it is on a par with T-Mobile, in the data category it scores similar to KPN. In both disciplines, Vodafone shows a distinct improvement over its scores from last year’s benchmark
With strong data results and a still very good voice score, the largest Dutch operator ranks third. Compared to its 2017 scores, KPN has improved in the voice disci- pline, but loses some points in the data category. However, all in all this operator again achieves a very good result.
The Netherlands’ smallest operator scores last and loses points compared to its score in our 2017 benchmark both in the voice and data categories. However, Tele2’s results are still very competitive and deserve the overall grade “very good“. In the future, this operator will be part of T-Mobile.
In the near future, we plan to complement our measurements with an additional assessment of service availability. For the Netherlands, we present this crowdsourced approach as a case study this year– it will become part of the overall scoring next year.
An additional important aspect of mobile service quality – above performance and measured values – is the actual availability of the mobile networks. Obviously, even the best performing network is only of limited benefit to its users, if it is frequently impaired by outages or disruptions. Therefore, P3 has been looking into additional methods for the quantitative determination of net- work availability, collecting data via crowdsourcing. This method must however not be confused with the drivetests described on the previous pages. We are convinced that crowdsourcing will further enhance our benchmarks. Drivetesting has advantages as a very controlled environment. Crowdsourcing enables statements about network availability on a larger scale in terms of time and geography. However, when it comes to diagnose the sheer availability of the respective mobile networks, a crowdsourcing approach can provide additional in- sights. Therefore, P3 has developed an app-based crowdsourcing mechanism in order to assess how a large number of mobile customers experience the availability of their mobile network. We call this aspect “operational excellence“.
In the future, we envision this consideration to become part of the overall scoring of our mobile network tests. But as we have been conducting this method in the Netherlands only for a couple of months and have not yet reached statistically firm numbers of users for all tested networks within the months considered, we have decided to present the results as a case study this year. So, the resulting observations are not yet included in the score of our network test. Nonetheless, in next year‘s P3 connect Mobile Benchmark in the Netherlands, we expect our crowdsourcing results to become a part of the overall test score. The P3 connect Mobile Benchmark will then be the only mobile network test which combine drivetesting and crowdsourcing, thus delivering the most comprehensive view on network performance. As a matter of fact, we envision to even extend the scope of our crowdsourcing results. In addition to reliability, also coverage reach and quality as well as users‘ best throughput (average and top values) will become part of the crowd score.
Crowdsourcing shows: Dutch networks very reliable
For this case study, we have taken a closer look at the data network availability in the Netherlands for the months preceding and including our measurement tours – specifically December 2017, January 2018 and February 2018. An in-depth analysis of our crowdsourcing data shows that the Dutch networks are all in all very stable and reliable. The only degradation that we could actually observe, happened on February 22nd at 13:00 h in the Vodafone network. This operator suffered an observable service degradation on this day that lasted no more than one hour.
This one hour of limited availability costs Vodafone one point in the simulated crowd score for February. KPN, T-Mobile und Tele2 did not suffer any obser- vable degradations in the period under consideration. This minor reduction of service availability would have had only a limited impact to the overall result even if we already had included them into our scoring. Even when taking into account the conversion of maximum achievable points to a lower total in order to “make room“ for the crowd- sourcing points, the overall ranking would not have changed and Vodafone would have only suffered a loss of one point.
However, with close point distances between the winning contenders as they are typical in highly competitive markets with extremely high perfor- mance levels such as the Netherlands, multiple or prolonged degradations clearly would have the potential to alter the final ranking order of our benchmark. We are already excited to put this new component of our testing procedures officially into effect starting with the P3 connect Mobile Benchmark in the Netherlands next year.