Download the report as PDF here

2018│04│10 – Public benchmark report in the Netherlands

The benchmarking expert P3 has been testing the Dutch mobile networks since 2015, showing always excellent and continuously increasing network performance in the Netherlands. With significant changes in the market due to recent mergers, our fourth mobile network test once more promises an exciting race. How did the four independent candidates perform this time?

RESULTS IN A NUTSHELL

P3‘s network benchmarks are widely accepted as the de-facto industry standard and for being highly objective. The carefully designed metho- dology of our 2018 benchmark in the Nether- lands combines two cars conducting drivetests in 21 Dutch cities and on the connecting roads between them with walktests in seven larger cities and tests on Dutch railways. The areas in which we have tested account for more than 5.7 million people, or for about 34 per cent of the Dutch population.
P3‘s measurement included the use of up-to- date LTE Cat 9 as well as VoLTE-capable smart- phones for the tests. Also, we constantly readjust the thresholds of our evaluation with steps like these we take the latest technical developments of the mobile networks into account. The scope of our benchmark is to show how the mobile networks perform at the edge of what is techni- cally feasible – and to what extent do customers benefit from these capabilities? In order to pro- vide valid assessments, we have also used the most comprehensive mobile plans available from each operator.

T-Mobile,Vodafone and KPN have improved compared to their 2017 scores.

As in the two previous years, the overall winner is T-Mobile. This time, the winner even achieves an impressive 977 points – the highest score which has ever been achieved in a P3 connect Mobile Benchmark – and the grade „outstanding“.
Vodafone follows closely behind achieving the same grade. KPN ranks third with the overall grade “very good”. All three operators were able to improve clearly compared to their 2017 re- sults, with Vodafone making the biggest leap (+31 points), followed by T-Mobile (+21 points) and KPN (+10 points). Only Tele2 has lost some ground (-11 points) compared to its 2017 score.
T-Mobile maintains the lead in almost every service and category. However, all in all T-Mobile, Vodafone and KPN show excellent results. KPN has lost points because of slightly inferior speech quality and longer call setup times. All Dutch ope- rators show a strong performance for all data services on connecting roads as well as in trains.

VOICE

Voice services may be less important today – however, customers expect reliable connections when talking on the phone. How do the Dutch networks fulfil these expectations?
As the last Dutch operator, T-Mobile introduced Voice over LTE (VoLTE) in October 2017. So, today all four contenders support this useful technology. VoLTE transmits voice calls as data packets over the 4G connection. This way, the formerly necessary “circuit-switched fallback“, which forced smartphones to switch back to 3G or 2G in order to take or place a phone call, can be avoided. Also, VoLTE codecs potentially support a wider audio bandwidth providing operators with the opportunity to deliver higher speech quality to their customers.
On their tour through the Netherlands, P3‘s two test cars visited 21 cities, of which 16 are counting over 100000 inhabitants, as well as 31 towns. Also, the cars covered about 3 600 km of connecting roads. Additionally, a walktest team conducted tests in seven cities. For the voice rating, each car carried a total of eight Samsung Galaxy S7 smartphones (two per ope- rator). The walktest team carried a total of four Galaxy S7 – one per operator. The phones in the cars always called a counterpart in the other car. The phones carried by the walktest teams called a stationary counterpart. The connected testing equipment registered success ratios, setup times and speech quality. In order to simulate normal smartphone usage, data transfers took place in the background of the test calls

T-Mobile and Vodafone on par in voice, KPN and Tele2 a little behind

In the voice tests conducted in cities, towns and on connecting roads, it was a neck-and-neck race between T-Mobile and Vodafone. In the drivetests executed in the larger Dutch cities Vodafone scored slightly ahead of T-Mobile. For the KPIs call setup time and speech quality, both contenders achieve the maximum number of
available point, so the race in this sector is decided by the call success ratios. With exactly 99.91 per cent, Vodafone scores slightly higher than T-Mobile with 99.88 per cent. In the smaller towns, their scores are on a par. On the connecting roads, T-Mobile scored slightly ahead of Vodafone.
So, in the intermediate result of the voice discipline T-Mobile and Vodafone are on a par, both scoring 396 points.

Compared to the top two, the drive test re- sults of KPN and Tele2 in cities, towns and on connecting roads falls a little behind due to slight- ly lower call success ratios and speech quality values as well as longer call setup times

Walktests conducted in seven cities show excellent results of all networks

For the first time, a walktest has been conduc- ted as part of the P3 connect Mobile Benchmark in the Netherlands. The walktest team visited Almere, Amsterdam, Den Haag, Nijmegen, Rotterdam, Tilburg and Utrecht. The results of their measurements show high success ratios and short call setup times for T-Mobile, Vodafone and KPN. In this discipline, KPN joined the top two contenders – with T-Mobile scoring 100 per cent of the available points in the category, while Vodafone and KPN scored 99 per cent.
Tele2 follows at a little distance, but still achieves 92 per cent of the possible points.
Examining the speech quality observed in our test calls, there is a more distinct ranking order: T-Mobile scores highest, followed closely by Vodafone. Then, there is a little gap with KPN reaching the third and Tele2 reaching the fourth rank. Still, our results point out that smartphone users walking in the large Dutch cities can expect an excellent voice performance from any of their mobile operators.

Voice results in Dutch trains among the best in Europe

In 2018, P3 connect Mobile Benchmark in the Netherlands for the first time also includes tests conducted in railways. In order to perform them, the walktest team took trains to travel between the seven cities listed above. Additionally, they took trains to the very north and the very south of the country (see map on the right-hand side). It is no surprise that the measurement values gathered in the challenging railway scenario
fall a little behind compared to the drivetests and the walktests in the cities. Still, the Dutch networks achieve very good results in this discipline. Here, KPN is slightly ahead, while Vodafone and T-Mobile follow at close distance and Tele2 with a little wider gap. The overall re- sults of the train tests however rank the voice connectivity in Dutch trains among the Euro- pean top group. Similar ratings were achieved in Switzerland. Compared to Germany and Austria, the performance in Dutch trains is by far better.

Voice results at a Glance

T-Mobile and Vodafone share the win in the voice discipline. KPN also achieves very good results and was clearly able to improve over its 2017 voice score. Slightly inferior speech quality and longer call setup times prevent an even better score for KPN. Tele2 falls a little behind but still achieves decent voice results.

DATA

In order to assess the performance and reliability of data connections, our two drivetest cars carried two smartphones per operator for the data tests: One Samsung Galaxy S7 and one Sony Xperia XZ. This choice was made in order to take the variations into account that result from the interaction of different smartphone types with the different networks. The walktest team carried one Samsung Galaxy S7 per operator. Supporting the LTE category 9, both smartphone types were able to benefit from the so-called aggregation of three LTE carrier frequencies which the Dutch operators offer under the name of 4G+ and which could theoretically transmit up to 450 Mbps. P3‘s testing considers fast throughputs as well as the networks‘ availability and stability. Web page and file downloads or file uploads reward fast speeds, while recording success ratios and assessing YouTube playouts concentrate on reliability aspects. As YouTube streams videos at adaptive bitrates, the average value of the received video resolution is another important performance indicator.

Overall high data performance in large Dutch cities

In the drivetests conducted in the cities, T-Mobile is ahead with 97 per cent of the achievable points, but
its competitors follow closely. Vodafone and KPN both score 95 per cent, while Tele2 scores 93 per cent of possible points in this category

T-Mobile and Vodafone ahead in data walktests

The data walktests conducted in seven large Dutch cities (see page 6 for the complete list) yield very similar results. In this category, however, T-Mobile and Vodafone share the top position, and KPN falls marginally behind. Tele2 scores weaker in this discipline, due to slightly lower success ratios. But even the smallest Dutch network still achieves 88 per cent of the possible points.

T-Mobile leads in drivetests in smaller towns, KPN, Vodafone and Tele2 follow closely

In the drivetests performed in the smaller Dutch towns, T-Mobile achieves the highest score and KPN ranks second, while Vodafone and Tele2 follow closely behind. This ranking order can be seen in measurement values like the speeds of file downloads and uploads. All four operators achieve excellent results when it comes to accessing YouTube while driving in smaller towns.

All Dutch operators share very strong results on connecting roads

A similar ranking can be observed on the connecting roads. T-Mobile also leads in this discipline with 99 per cent of the possible points. KPN ranks second with 98 per cent, while Vodafone and Tele2 share the third rank with 97 per cent of the achievable points. These results are rather sensational, attesting that Dutch drivers can ex- pect top mobile connectivity on the roads of their country.

T-Mobile and KPN achieve highest scores for data connectivity on Dutch railways

Our test of data connections in Dutch trains again demonstrates the high quality of mobile coverage in the Netherlands. T-Mobile leads the field and even scores higher than in the voice discipline. KPN ranks second but scores a little lower than with its voice connections on trains. Vodafone achieves the third and Tele2 the fourth rank in this discipline. Again, compared to the results in other European trains, the Dutch operators show excellent performances for data connections in trains.

Data results at a Glance

T-Mobile also leads the data discipline of this test. Although KPN has lost some points in comparison to its 2017 data results, the largest Dutch operator still ranks second in the disci- pline. Vodafone achieves the third rank but shows the biggest improvement compared to last year’s data results. Tele2 scores last in this discipline and falls some points behind its 2017 data results, still achieving a very good overall performance in the data discipline.

T-Mobile

In 2000, the German Deutsche Telekom bought a minority of the Dutch mobile network operator Ben, which two years later was extended to a 100 per cent acquisition. In 2003, Ben was renamed T-Mobile Netherlands, with the brand “Ben“ becoming a “no-frills“ offer within its portfolio. In 2007, T-Mobile Netherlands additionally acquired the ope- rator Orange. At the end of its fiscal year 2017, T-Mobile Netherlands reported 3.85 million customers and a revenue of 1.4 billion Euros which equals a mobile market share of about 21 per cent. In December 2017, the company announced to acquire Tele 2.
T-Mobile Netherlands operates 2G mostly at 1800 MHz, 3G at 900 and 2100 MHz and 4G/LTE at 900, 1800, 2100 and 2600 MHz. The company claims a 4G coverage of more than 99 per cent of the Dutch population. With 4G+, T-Mobile network offers a maximum download speed of 300 Mbit/s. VoLTE was introduced in October 2017.

Vodafone

The Dutch subsidiary of the international Vodafone Group acquired the formerly independent operator Libertel in 2003, forming Vodafone Netherlands. In 2016, it merged with the cable and fibre operator Ziggo. Today, 50 per cent of the joint company VodafoneZiggo is owned by the Vodafone Group and another 50 per cent by Liberty Global. VodafoneZiggo is the second largest mobile opera- tor in the Netherlands with around 5 million mobile custo- mers equalling a mobile market share of about 28 per cent. The company also specifies nearly 4 million video, over 3 million fixed broadband and about 2.5 million fixed telephony subscribers. For the calendar year 2017, VodafoneZiggo announced a mobile revenue of 4 billion Euros.
The company operates 2G networks at 900 and 1800 MHz, 3G at 2100 MHz and 4G/ LTE at 800, 1800, 2100 and 2600 MHz. Vodafone claims to offer more than 98 per cent co- verage with 4G (up to 150 Mbit/s). In many Dutch regions the company offers “4G+“ up to 250 Mbit/s. VoLTE was introduced in November 2016.

KPN

The Koninklijke PTT Neder- land N.V. emerged from the privatisation of the formerly state-owned PTT in 1998. With about 8 million mobile customers, KPN had a market share of about 44 per cent in the Netherlands at the end of 2017. With these numbers, the company is by far the lar- gest Dutch mobile operator. For the fiscal year 2017, the company reported a total revenue of 6.5 billion Euros.
The company focuses on marketing its KPN brand, however with Simyo, Telfort and Ortel it also has offerings in the “no-frills“ segment. KPN operates 2G/GSM at 900 and 1800 MHz, 3G/UMTS at 900 and 2100 MHz and 4G/LTE at 800, 1800, 2100 and 2600 MHz. The company claims a 99 per cent 4G availa- bility all over the country. Where KPN already offers carrier aggregation, the theoretical maximum speed of its LTE network is 225 Mbit/s. KPN markets this configuration as “4G+“. KPN has offered VoLTE since October 2016.

Tele2

The Swedish telecommunications operator Tele2 acquired the former Versatel N.V. in 2005 and merged it with its existing Dutch operations.
Originally acting as a MVNO (mobile virtual network operator), Tele2 nowadays operates its own infrastructure and has a portfolio of fixed telephony, data, internet and mobile telephony products.
At the end of its fiscal year 2017, Tele2 reported more than 1.2 million mobile customers which equals a market share of about 5 per cent.
As a result of the announced merger with T-Mobile NL, Tele2 will hold a minority stake of 25 per cent in the combined company. At the time of writing this report, Tele2 operated its own network and T-Mobile NL announced to continue to use the Tele2 brand in the market.
Tele2 operates its own LTE network at 800 and 2600 MHz and offers roaming in the T-Mobile 2G and 3G networks to its customers. Tele2‘s 4G network claims a coverage of 97 per cent of the country and offers carrier aggregation („4G+“) up to 225 Mbit/s as well as VoLTE.

METHODOLOGY

The P3 connect Mobile Benchmark in the Netherlands took place from February 16th until March 10th, 2018. All samples were collected between 8am and 10pm. The network tests covered 21 cities, of which 16 are counting more than 100 000 inhabitants, and 31 smaller towns. Furthermore, our test routes included 3,600 kilometres of connecting roads. Additionally, a walktest team visited seven larger cities. The team took the trains between the cities and also into more rural areas of the country, performing voice and data tests on the railways as well.

This combination of test areas has been carefully selected to provide a significant series of test results covering the Dutch population. The areas chosen for the 2018 test account for about 5.8 million people, or 34 per cent of the total Dutch population.

P3 conducted the tests with two drive-test cars, equipped with arrays of Samsung Galaxy S7 Cat 9 smart- phones (voice) as well as a mixed allocation of Samsung Galaxy S7 and Sony Xperia XZ Cat 9 smartphones (data) for the simultaneous measurement of voice and data services.

Voice testing

Two smartphones per operator in each car were used for the voice tests, setting up test calls from one car to another. The walktest team carried one smartphone per operator for the voice tests. In this case, the smartphones called a stationary counterpart. The audio quality of the transmitted speech samples was evaluated using the HD-voice capa- ble and ITU standardised so-called POLQA wideband algorithm. All Dutch operators offer 4G capable subscrip- tions. In total, two voice channels were in operation. Both smartphones establishing the first voice test channel were set to 4G preferred mode forcing the network to perform a fallback to UMTS in LTE areas. On the second channel, both smartphones were set to VoLTE-preferred to simulate customer behaviour with the latest technology. The smartphones used in the walktests were all set to VoLTE-preferred.
In order to account for typical smartphone-use scenarios during the voice tests, background data traffic was generated in a controlled way through random injection of small amounts of HTTP traffic. The voice test scores account for 40 per cent of the total benchmark results.

Data testing

Data performance was measured by using four smartphones in each car – one per operator. One car was equipped with four Samsung Galaxy S7 while the other car was carrying four Sony Xperia XZ in order to respect the variable data performance of different smart phones in different networks. In order to further reflect the customer experience, the radio access technology was set to LTE preferred mode. The web tests accessed web pages according to the widely recognised Alexa ranking. In addition, the static “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) was used.

In order to test the data service performance, files of 3 MB and 1 MB for download and upload were transferred from or to a test server located on the Internet. In addition, the peak data performance was tested in uplink and downlink directions by assessing the amount of data that was transferred within a seven seconds time period. Another discipline was the playback of YouTube videos. It took into account that YouTube dynamically adapts the video resolution to the available band- width. So, in addition to success ratios, start times and playouts without interruptions, YouTube measurements also determined the average video resolution.

All the tests were conducted with the best-performing mobile plan available from each operator. Data scores account for 60 per cent of the total results.

Routes and samples

In the selected 21 cities and 31 smaller towns, the cars had to follow predefined routes. Altogether, the two test cars covered more than 6,420 kilometres, of which approximately 2,820 km led through the cities and towns, while 3,600 km covered connecting roads.

Performance indicators and rating

The score weighting reflects both the geographical distribution of the Dutch population and the ranking of usage scenarios. Therefore, 600 of the total of 1000 maximum points were assigned to the cities – 450 maximum points for the drivetest results (voice: max. 180 p., data: max. 270 p.) and 150 for walktests results (voice: max. 60 p., data: max. 90 p.). For the  towns, a maximum of 200 points is available (voice: max. 80 p., data: max. 120 p.) and for the connecting roads a maximum of 125 points (voice: max. 50 p., data: max. 75 p.). The tests conducted in trains account for another 75 points (voice: max. 30 p., data: max. 45 p.) The tables on page 2 and page 11 of this report show the percentage of maximum points that each operator has achieved in each discipline.

CONCLUSION

T-Mobile wins again. Vodafone shows also “outstanding“ results. A very good KPN ranks third, and Tele2 fourth – but still with a “very good“ score.

For the third time in a row, T-Mobile is the clear winner of the P3 connect Mobile Benchmark
in the Netherlands. Its impressive 977 total points is the highest score ever achieved in any P3 connect benchmark. As the distinct leader in the data category and being on par with Vodafone in the voice discipline, T-Mobile‘s grade “outstanding“ is well deserved. If this ope- rator succeeds in integrating the frequencies and base stations of its projected subsidiary Tele2, it will be excellently positioned for the future.
The pursuer Vodafone also shows an “out- standing“ performance in this year‘s bench- mark. It achieves the second rank especially due to excellent voice results which are the same a those of test winner T-Mobile, while in the data discipline, Vodafone is almost at the same level as the third-ranking KPN.
KPN is strong in the data discipline, where it ranks second behind test winner T-Mobile. But the Dutch market leader loses some points in the voice category due to slightly lower speech quality and longer call setup times. However, all in all KPN still achieves a very good result.
Tele2, the smallest Dutch operator and des- tined to be owned by T-Mobile, achieves the fourth rank. Due to its concentration on LTE/4G, it achieves a decent score in the data discipli- ne. Its score for data connectivity on the roads is particularly strong. However, Tele2‘s voice score shows some room for improvement. It will be interesting to see how the merger with T-Mobile will affect this operator‘s future results.

When comparing the results of this year‘s mobile network benchmark in the Netherlands to those from recent tests in other European countries, Dutch operators are among the top tiers. Even the fourth-ranking Tele2 scores at a level that would make it a suitable candidate for a winning position in countries with overall weaker mobile networks like Spain or the UK.

1 T-Mobile

The third largest Dutch operator wins the 2018 benchmark with a distinct lead in the data category and a voice result on a par with second ranking Vodafone. The ope- rator earns the grade “outstanding“. Above that, T-Mobile was able to improve on last year’s results both in the voice and in the data discipline

2 Vodafone

With an “outstanding“ overall performance, Vodafone well earns the second rank. In the voice discipline it is on a par with T-Mobile, in the data category it scores similar to KPN. In both disciplines, Vodafone shows a distinct improvement over its scores from last year’s benchmark

3 KPN

With strong data results and a still very good voice score, the largest Dutch operator ranks third. Compared to its 2017 scores, KPN has improved in the voice disci- pline, but loses some points in the data category. However, all in all this operator again achieves a very good result.

4 Tele2

The Netherlands’ smallest operator scores last and loses points compared to its score in our 2017 benchmark both in the voice and data categories. However, Tele2’s results are still very competitive and deserve the overall grade “very good“. In the future, this operator will be part of T-Mobile.

OPERATIONAL EXCELLENCE

In the near future, we plan to complement our measurements with an additional assessment of service availability. For the Netherlands, we present this crowdsourced approach as a case study this year– it will become part of the overall scoring next year.

An additional important aspect of mobile service quality – above performance and measured values – is the actual availability of the mobile networks. Obviously, even the best performing network is only of limited benefit to its users, if it is frequently impaired by outages or disruptions. Therefore, P3 has been looking into additional methods for the quantitative determination of net- work availability, collecting data via crowdsourcing. This method must however not be confused with the drivetests described on the previous pages. We are convinced that crowdsourcing will further enhance our benchmarks. Drivetesting has advantages as a very controlled environment. Crowdsourcing enables statements about network availability on a larger scale in terms of time and geography. However, when it comes to diagnose the sheer availability of the respective mobile networks, a crowdsourcing approach can provide additional in- sights. Therefore, P3 has developed an app-based crowdsourcing mechanism in order to assess how a large number of mobile customers experience the availability of their mobile network. We call this aspect “operational excellence“.

In the future, we envision this consideration to become part of the overall scoring of our mobile network tests. But as we have been conducting this method in the Netherlands only for a couple of months and have not yet reached statistically firm numbers of users for all tested networks within the months considered, we have decided to present the results as a case study this year. So, the resulting observations are not yet included in the score of our network test. Nonetheless, in next year‘s P3 connect Mobile Benchmark in the Netherlands, we expect our crowdsourcing results to become a part of the overall test score. The P3 connect Mobile Benchmark will then be the only mobile network test which combine drivetesting and crowdsourcing, thus delivering the most comprehensive view on network performance. As a matter of fact, we envision to even extend the scope of our crowdsourcing results. In addition to reliability, also coverage reach and quality as well as users‘ best throughput (average and top values) will become part of the crowd score.

Crowdsourcing shows: Dutch networks very reliable

For this case study, we have taken a closer look at the data network availability in the Netherlands for the months preceding and including our measurement tours – specifically December 2017, January 2018 and February 2018. An in-depth analysis of our crowdsourcing data shows that the Dutch networks are all in all very stable and reliable. The only degradation that we could actually observe, happened on February 22nd at 13:00 h in the Vodafone network. This operator suffered an observable service degradation on this day that lasted no more than one hour.

This one hour of limited availability costs Vodafone one point in the simulated crowd score for February. KPN, T-Mobile und Tele2 did not suffer any obser- vable degradations in the period under consideration. This minor reduction of service availability would have had only a limited impact to the overall   result even if we already had included them into our scoring. Even when taking into account the conversion of maximum achievable points to a lower total in order to “make room“ for the crowd- sourcing points, the overall ranking would not  have changed and Vodafone would have only suffered a loss of one point.

However, with close point distances between the winning contenders as they are typical in highly competitive markets with extremely high perfor- mance levels such as the Netherlands, multiple or prolonged degradations clearly would have the potential to alter the final ranking order of our benchmark. We are already excited to put this new component of our testing procedures officially into effect starting with the P3 connect Mobile Benchmark in the Netherlands next year.

FINAL RESULTS