P3 MOBILE BENCHMARK USA 2018

VERIZON ACHIEVES ‘BEST IN TEST’ AND WINS THE P3 MOBILE BENCHMARK USA

The international benchmarking expert P3 has been testing the performance of cellular networks around the globe for more than 15 years. Since the beginning of this year, P3 has also analyzed US carriers’ performance and published a series of reports each examining a different aspect culminating in the present report. The “P3 Mobile Benchmark USA” determines the “BEST in test” carrier in the US.
All P3 reports reveal deep insights into the mobile communication quality of the four big carriers’ networks. While the first three reports focused on distinct quality aspects and referred to specific environments – “Connectivity on Interstates” is a P3 report on cellular coverage on the interstates, “US State Connectivity” is a P3 report on cellular coverage in the individual US states, “Data Speed Report” is a P3 report about data speed comparison in the US markets, Metropolitan Statistical Areas (MSA) – the present report draws a nationwide holistic picture of the quality of service of the US mobile network carriers. The question is: Which carrier achieves the best test results for mobile voice and data in the US?

In order to answer this question, we have looked at the three different aspects that matter the most for consumers: getting good coverage (Coverage), experiencing decent speed (Speed) and no suffering from temporary network outages or severe service degradations (Data Service Availability).
The present analysis is based on a unique crowed source data set of more than 34 billion measurement (34,000,000,000) samples collected on 2.3 million consumers’ smartphones during real-life usage, thus representing the nationwide consumer experience of mobile networks in the United States. With this footprint, 1 out of every 141 inhabitants is covered. Different performance disciplines and facets are used to calculate the P3 Score. The P3 benchmarking and scoring methodology is globally regarded as the de-facto industry standard for benchmarking mobile networks in a scientific, yet intuitive way. So, which carrier achieves the highest P3 Score and can claim to be “BEST in test”?

RESULTS IN A NUTSHELL

TO WIN THE P3 MOBILE BENCHMARK AND DESERVE THE “BEST IN TEST” ACCOLADE THE NETWORKS MUST PROVE THEMSELVES IN THREE DISCIPLINES – COVERAGE, DATA SPEED AND DATA SERVICE AVAILABILITY.

First, coverage for both voice and data services determines half of the overall scoring of 1,000 points. Here, we consider six different aspects of coverage from the consumers’ perspective. Second, the Data Speed determining 20% of the overall score considers the diverse facets of speed as provided by the networks and experienced by the users. And, third, the continuity of the networks’ service delivery is included with 30% of the overall score. Here, network outages or service degradations are analyzed for evaluating the Data Service Availability. Looking at the total score from all disciplines, the P3 Mobile Benchmark in the United States has a clear winner: Verizon. AT&T follows in second place. Both networks rank high with more than 75% of the achievable P3 Score. T-Mobile and Sprint are ranked third and fourth with just above 60% of the achievable points.

Verizon is BEST in test outperforming the other networks with overall 812 points. Especially in the Coverage discipline, Verizon is ahead with 438 out of 500 points which is about 17% better than second placed AT&T. Furthermore, Verizon has reached the best User Download Speed rating.

AT&T sticks out with the best rated Data Service Availability and reaches the maximum achievable score in this discipline with 300 points. This means that during the whole observation period – March to August ’18 – AT&T did not show a single significant data network outage. In contrast, Verizon had a couple of observable service degradations and, in this discipline, scores behind T-Mobile and Sprint.

In the overall ranking, T-Mobile is in third place beating the fourth ranked Sprint, particularly with a higher rating for Coverage. Conversely, Sprint stands out with second best rated Data Speed, directly behind Verizon and significantly ahead of both T-Mobile and AT&T.

P3 crowdsourcing measurement footprint in the United States.

“VERIZON ACHIEVES THE BEST IN TEST AWARD.”

“VERIZON DOMINATES IN COVERAGE.”

Total achieved nationwide P3 Score

A CLOSER LOOK AT THE US MOBILE NETWORKS

COVERAGE

The coverage provided to smartphone users today is a composition of different technologies, partly layered and covering the same area simultaneously. Currently there is still a mixture of so-called 2G technologies like CDMA or GSM, also 3G and 4G, which is also called LTE, up and running. On top of that, what technology consumers are actually served with may depend on the kind of service they are using. While Verizon, AT&T and T-Mobile have rolled out voice call capability throughout their 4G networks long ago, Sprint has announced an upgrade to their 4G network in the near future.

Sprint currently handles voice calls with the 1X CDMA part of their network. However, data services are usually provided with the LTE parts of the networks by all carriers, partly supplemented by legacy 2G or 3G technology.

So, the coverage users experience does not only depend on where they are and which network they use but also on the type of device and whether they are on a phone call or using cellular network data.

The comparison of network coverage in the P3 benchmark consequently requires a detailed consideration of multiple factors. For voice, data and LTE, the scoring is applied to two properties each that characterize the coverage. First, the footprint, i.e. the availability of a service in the test area and second the quality of the coverage expressed as the share of users actually connected to the service in that area.

4G LTE COVERAGE

Looking at the 4G LTE coverage results it becomes clear that Verizon has the lead in LTE coverage. AT&T customers are much less likely to get LTE than Verizon customers. Even T-Mobile users are more likely to experience LTE than AT&T users although AT&T has a significantly larger LTE footprint than T-Mobile. AT&T still makes significant use of its 3G layer.

COVERAGE FOR DATA SERVICES

Carriers use both 3G and 4G technologies to deliver high speed data services. 2G technologies are effectively not capable of supporting data connection with sufficient speed. The coverage for data services assesses the portion of the test area where network coverage was sufficient for a good data experience (i.e. 3G or 4G) and the respective quality of that coverage.

Again, Verizon scores best, closely followed by AT&T benefiting from a significant portion of users served with 3G data. The networks reach 89% and 87% of the achievable maximum score, respectively. Data services in both Verizon and T-Mobile are almost entirely handled on 4G LTE. In contrast, Sprint users are more often served with 3G than users in any other network. However, due to a good quality of coverage, Sprint scores only about 2% worse than T-Mobile with respect to coverage for data services.

COVERAGE FOR VOICE SERVICE

T-Mobile and AT&T use a combination of 3G and 4G technologies to deliver voice services. Sprint announced it was launching Voice-over-LTE (VoLTE) in late 2018 but currently carries the voice traffic on its 1X CDMA network. Verizon has converted into an LTE-only network and provides all voice and data services over the same network. It is worth mentioning that the four big US carriers also have adopted WiFi calling which is especially useful when connected to WiFi in areas with weak carrier coverage. However, this technology currently does not support mobility and is not considered in the scoring analysis.
Looking at the coverage metrics for voice services, there are similarities to the data coverage. All carriers benefit from their 3G/4G LTE footprint that also determined the coverage for data services. All carriers except AT&T can provide even better voice coverage compared to data by adding 2G coverage in some cases. Like for all other coverage aspects, Verizon again shows the best values for both test area coverage and quality of coverage.
Overall, Verizon achieves 87.6% of the achievable maximum score in the coverage discipline, followed by AT&T with 74.6%, T-Mobile with 52.8% and Sprint with 46.6%.

DATA SPEED

“SPRINT CONVINCES WITH SECOND RATED USER DOWNLOAD SPEED.“

USER DOWNLOAD SPEED

In the same way as the different facets of coverage find their way in to the scoring, also the user download speed deserves a diligent consideration. The P3 scoring methodology looks at the real-life data traffic speed and thus analyses what really happens on the consumers’ smartphones when they use any of their apps. Compared to old fashioned speed tests that are based on downloading bulks of artificial data, the analysis of real life traffic for doing a meaningful speed benchmark is much harder, but it is worth it. A proper cleansing and smart aggregation of the collected data allows a nuanced view on different facets of speed.
The P3 scoring methodology considers download speed from two different angles. First, it considers what the network is capable of, i.e. the speed users can get regardless of any possible limitations. That is called the Network Top Speed. Second, it conversely considers the User Speed Experience determined by the average and top speed results across the whole consumer base.

Overall, Verizon achieves the highest User Download Speed score with Network Top Speed of 70.7 Mbit/s, the fastest speed amongst all carriers. Here, Verizon benefits from its lead in 4G LTE coverage. Remarkably enough, Sprint comes in on second place in the User Download Speed discipline. Although Sprint does not reach the Network Top Speed of the other carriers, it is obviously still capable of providing the best User Speed Experience. That demonstrates how the user’s perspective on speed can be different from the network’s perspective and that it is worth looking at both sides of the coin. Obviously, consumers on Sprint face less limitations in their real-life smartphone usage than users in the other networks. Nevertheless, also Verizon customers enjoy a 5 Mbit/s average User Speed Experience compared to 4.2 Mbit/s and 4.6 Mbit/s for AT&T and T-Mobile customers, respectively.
T-Mobile also does quite well in the user download speed discipline. Though only ranked third, the network convinces with the second-best Network Top Speed of 59.4 Mit/s and very good top speed results for User Speed Experience of 12 Mbit/s.
Only AT&T has little to stick out with in this discipline. Despite the unchallenged second rank in coverage the carrier ranks fourth with respect to user download speed. The larger 4G LTE footprint puts AT&T ahead of Sprint with respect to Network Top Speed but the other carriers are ahead of AT&T regarding the User Speed Experience.

DATA SERVICE AVAILABILITY

Consumers can enjoy their network’s coverage and speed only if the carrier manages to operate the network without service interruptions. Therefore, the P3 Score also considers the Data Service Availability as a third discipline in the benchmark. Sophisticated statistics on the huge amount of crowd sourced consumer data are used to determine network wide anomalies. This way, the duration of service degradations is recorded. The Data Service Availability analysis reflects how excellent operators perform their network operations.
For the benchmark, the operational excellence is scored for a period of six consecutive months, from March until August 2018. For each month, the total duration in terms of clock hours affected by significant data service degradations is recorded. Furthermore, the number of days that have been affected by these service degradations is also examined to distinguish series of short outages from few long ones. The months are scored separately, so the carriers had to keep the networks stable for a long period to gain high scores for the Data Service Availability.
AT&T is the model student in this discipline. During the whole observation period, AT&T did not show a single significant degradation in Data Service Availability and consequently is awarded with the maximum achievable score in this discipline.
Verizon customers suffered from degradations especially in April and June which overall put Verizon on the 4th position. Even the fact that no degradations were recorded in March and July did not suffice to top any of the competitors.
T-Mobile ranks second, despite a bad month of May where a total of 10 clock hours with degraded Data Service Availability was recorded. But apart from that, four out of six months without noticeably reduced Data Service Availability is an acknowledgement of a good T-Mobile network operations.
Sprint ranks third, just slightly behind T-Mobile, with service degradations observed on four days within three out of the six months.

“AT&T IMPRESSES WITH BEST DATA SERVICE AVAILABILITY.”