2017|12|04 – THE 2017 P3 CONNECT MOBILE BENCHMARK IN AUSTRALIA
Download the report as PDF here
The benchmarking expert P3 communications has been testing the mobile network performance in Australia since 2014. Now, for its fourth mobile network test in Australia, P3 teams up with the internationally acknowledged telecommunications magazine connect. This combines P3’s longtime network benchmarking expertise with decades of connect’s
editorial experience. Together, we have carefully examined the performance and quality of the mobile networks in Australia.
It was a neck-and-neck race of the two largest Australian operators. This year, Optus takes the “best in test“ accolade from an also “very good“ Telstra.Vodafone confirms its position from the previous test, receiving the grade “good“ and catching up further.
Results in a nutshell
P3‘s network benchmarks are widely accepted for setting industry standards as well as being highly objective. The carefully designed methodology scheduled four drivetest cars to visit nine large Australian cities and 19 smaller towns as well as the connecting roads between them. In addition, two teams conducted walktests in four large cities. The areas in which we tested account for more than 13.7 million people, or about 61 per cent of Australia‘s population.
Another example for P3‘s great attention to detail is the use of up-to-date LTE “Cat 9“ as well as VoLTE-capable smartphones for the tests. They reflect the latest technical developments in the mobile networks and once more emphasise the scope of our benchmarking: How do the mobile networks perform at the edge of what is technically feasible – and to what extent do customers benefit from these capabilities? In order to provide valid answers to these questions, we have also used the most comprehensive mobile plans available from each operator. Also, we constantly readjust the thresholds of our evaluation to represent the latest technological advancements.
The pleasant results of P3‘s previous mobile network benchmarks in Australia have set high expectations. In fact, all three operators were able to meet them – Optus and Telstra managed to distinctly improve their scores, and Vodafone confirmed its position by gaining five points over last year‘s result.
Optus wins due to lead in voice, Telstra scores best in data
It was a neck-and-neck race of the two biggest Australian operators. Ultimately, Optus won with an advance of three points over Telstra. Both receive the grade “very good“. Optus secures its overall win with a clear lead in the voice category, while Telstra scores best in the data discipline. Vodafone ranks third, scoring behind the leading two both in voice and data. Still, Vodafone performs decently and gains the overall grade “good“.
The Australian mobile communications market is served by three operators that clearly differ in terms of market shares. All three of them offer the latest mobile technologies like LTE-Advanced and VoLTE.
According to its own statement, Telstra currently has 17.4 million mobile subscribers. This makes this operator market leader in Australia.
The history of Telstra dates far back: The company originated as part of the Postmaster-General‘s Department and was fully privatised by 2006. The last remaining government share was effectively sold in 2011. Today, Telstra offers voice, mobile and internet access, as well as Pay TV and other entertainment services to Australian customers.
The operator provides 3G (UMTS) services at 850 and 2100 MHz and a 4G (LTE including LTE-Advanced) network on the 700, 900, 1800 and 2600 MHz frequency bands. Like its competitors, Telstra has been continually increasing its LTE coverage and offerings, now providing LTE-Advanced under the name “4GX“ and claiming a 4G coverage of 99 per cent of the Australian population.
With reportedly 9.8 million mobile subscribers, Singtel Optus is the second largest mobile operator in Australia.
The company was originally founded as AusSat in 1981 and later changed its name to “Cable & Wireless Optus“. Since 2001, it is a wholly owned subsidiary of the Singapore-based SingTel. The company offers its mobile services mainly under the brand name Optus, but also maintains a number of subsidiary brands, such as Virgin Mobile Australia. Its offerings span fixed and mobile telephony and data networks, broadband access for both consumers and business customers, cable TV and satellite services.
Optus provides a 3G network at 900 and 2100 MHz and 4G/LTE at the 700, 1800, 2100, 2300 and 2600 MHz bands. Its 4G offerings include LTE-Advanced under the brand name “4G Plus“. As of November 2017, the company claims to cover 96 per cent of Australians with its 4G/LTE service.
Vodafone Australia quotes a subscriber base of approx. 7 million mobile customers. This makes Vodafone Australia‘s third largest mobile operator.
The current company was formed in a merger of Vodafone Australia and Hutchison 3G Australia (and its former “3“ network) in 2009. As an indication to its two owners each holding a 50 per cent share, the company is often referred to as “VHA“ (Vodafone-Hutchison Australia). Today, it offers a full suite of mobile and fixed networks and access products.
The joint company offers 2G at 900 MHz, 3G at 900 and 2100 MHz as well as 4G at 850, 1800 and 2100 MHz. Upgrading and expanding its 4G service, Vodafone enabled carrier aggregation on all of their 4G bands which is part of the LTE-Advanced specification.
The company claims to cover “over 22 million” Australians but doesn’t break out a specific percentage. Put into relation to the total number of 24.3 million inhabitants of the country, this would equal at least91 per cent of the population.
CLOSER LOOK AT AUSTRALIA’S NETWORKS
Over the past years, all operators in Australia have shown continuous improvements which reflected in our network tests. We are happy to report the trend persists in 2017.
The German company P3 is a world leader in mobile network testing. It is part of the P3 group, with over 3000 employees worldwide and a turnover of more than 300 million Euros (468 million Australian Dollars).
In 2017, P3 runs its authoritative mobile network benchmark in Australia for the fourth time. In 2017, this is realised for the first time together with the international telecommunications magazine connect for its network test in Australia. The magazine has more than 20 years of editorial experience and is one of the leading test authorities for telecommunications products and services in the world. Together, P3 and connect have been conducting mobile network benchmarks in Europe for more than 15 years.
In 2016 alone, P3 compiled more than 60,000 measurement hours in 65 countries across five continents, with its test vehicles covering almost one million kilometres.
As the de-facto industry standard, the P3 benchmarking methodology focuses on customer-perceived network quality – examining both voice telephony as well as data connectivity. P3‘s network benchmarks are widely accepted as an objective authority.
Clear improvements in 2017
As in the years before, we have taken great effort to provide an objective view on the performance and quality of the Australian mobile networks. Without trying to get ahead of the detailed results, we are happy to assert that all three Australian operators were able to improve their scores over the previous test.
While data communications are ever more prevalent, customers still expect excellent voice quality. How do the Australian mobile networks perform in this discipline?
In the communications habits of many users, speaking on the phone has become less important in comparison to internet-based communications like instant messaging or e-mailing. But telephony is still a key feature of every mobile network – and in those cases when people use it, they naturally expect top reliability and a high speech quality.
Australian networks are well positioned for fulfilling these expectations, as all three operators have deployed Voice over LTE (VoLTE). This modern technology helps to better coordinate voice and data communications in the data-centric LTE networks and avoids kludges like the “circuit-switched fallback“ that forces a smartphone to switch to a 2G or 3G connection while conducting phone calls.
Voice: Optus is the champion, Telstra overall second, Vodafone strong in big cities
The drivetests and walktests in the big cities show a high level of reliability and performance in all three networks. Optus is a little ahead in the drivetest scenarios, while both Optus and Vodafone lead by a small margin ahead of Telstra in the walktests.
The ranking order becomes a little more distinct in the smaller towns. Here, Optus leads the field, followed at a small distance by Telstra and, again with a little gap, by Vodafone. The results here are still good. As can be expected, they drop considerably on the roads: Here, Optus and Telstra are still performing quite well, while the score of Vodafone drops to a comparably meagre 49 per cent of the total achievable points.
VOICE results At a Glance
Overall winner Optus also scores highest in the voice category. In the big cities, Vodafone and Telstra are following closely. In contrast, in the smaller towns and on the connecting roads, the ranking order is clear: Optus ranks first, Telstra second and Vodafone third.
With data volumes growing rapidly and tight limitations on available bandwidth, all operators face challenges in providing a satisfactory user-experience. Which Australian contender manages to best meet the growing demand?
According to the Australian Bureau of Statistics, the volume of data downloaded via mobile devices from May to June 2017 was 175,076 Terabytes. This represents a 19.9 per cent increase compared to the same period of the preceding year. For the 26.3 million mobile subscribers of the country, this equates to an average of 2.2 GB of data downloaded per subscriber per month. These numbers not only prove that mobile internet usage is hugely popular in Australia, but also that the typical user behaviour puts a heavy load on the mobile networks and poses a considerable challenge to all Australian mobile operators. Telstra, Optus and Vodafone are continuing to spend billions of dollars on upgrading and expanding their networks to meet these demands.
A good mobile internet experience demands quick loading times. This includes the download of different types of web-pages, downloads and uploads of large files as well as the increasingly important reception of video streams from services like Youtube. In order to check the reliability and performance of data connectivity in the Australian networks, one Samsung Galaxy S7 per operator was installed in each of our four test cars and in the backpacks carried by each walktest team.
The phones constantly performed a suite of tests. For most of these checks, success ratios and average session times were logged. For downloads and uploads we also recorded average throughputs. In order to assess typical performance as well as peak speeds, we determined the minimum data rates that are available in 90 per cent of the cases plus the peak data rates that would be surpassed in 10 per cent of the cases.
When testing the Youtube performance, besides success ratios, start times and the absence of interruptions, the average received video resolution is another important performance indicator.
All three Australian operators show strong performance in cities
As already observed in the voice discipline, all three Australian operators are strong in the big cities. In the overall results of our drivetests, Telstra is in the lead, followed at short distances by Vodafone and Optus. In the overall result of our walktests conducted in the cities, Vodafone takes the lead, closely followed by Telstra, while Optus ranks third at a distance of three per cent.
A first indication to Optus‘ victory this year showed when the drivetest cars examined the smaller towns. Here, Optus takes the lead, closely ahead of Telstra. Vodafone, that fell back in the voice tests in smaller towns and on the connecting roads, gives a similar picture in the data tests: The distance between the second-ranked Telstra and Vodafone on the third rank is a considerable ten per cent of the maximum points in this category.
Still, a closer analysis of the measurement values shows the comforting result that even when data rates drop, the success ratios are still comparably good – except for the results of video playback on roads in the Vodafone network. But even in the networks with weaker performance values, customers can still expect mostly reasonable connectivity.
Telstra leads in data results on connecting roads, Optus also strong, Vodafone falls behind
On the connecting roads between the cities and towns, Telstra scores a little better than the also strong Optus. Vodafone once more falls behind. In this case, this backlog is not only affected by throughput rates, but substantiates in clearly reduced success ratios too.
For mobile users who regularly need a good connectivity when they are travelling by car between Australian cities and towns, our results indicate advantages for Telstra and Optus.
Data results at a Glance
Telstra is the category winner in the data tests, followed by an also strong Optus. Overall in this discipline, Vodafone falls somewhat behind. However, this result is only due to Vodafone’s limited data performance in smaller towns and particularly on connecting roads. Here, Telstra and Optus achieved clearly better results. In the big cities, all three contenders are very strong.
AUSTRALIA’S FIVE LARGEST CITIES
For the inhabitants of the densely populated urban areas in Australia, it is particularly interesting to see how the different operators perform in their home cities. Therefore, we performed individual analyses for the five largest Australian metropolises.
More than 14 million Australians live in one of the five largest urban areas of the country – which is a whopping 60 per cent of the total population.
Therefore, and in order to support these users by giving them detailed test results about their local mobile network situation, we have taken a closer look at the results in Sydney, Melbourne, Brisbane, Perth and Adelaide. With the exception of Perth, the test teams conducted drivetests as well as walktests in these metropolitan areas. In order to facilitate an easier comparison of the individual results, we have converted the achieved points into percentages – compensating the fact, that the absence of a walktest result would otherwise reduce the maximum number of possible points.
As we have already stated, the operators all in all perform strongly in the bigger cities. So, the differences in their results are not huge – but still worth a closer examination.
Sydney: Telstra leads overall and in data, Optus strongest in voice
In Australia‘s biggest city, the market leader Telstra is closely ahead of its opponents Optus and Vodafone. Telstra scores particularly strong in the data tests, where Vodafone ranks second and Optus third. Optus makes up leeway in the voice discipline where it closely leads ahead of Telstra and Vodafone who turn out to be equally strong in telephony. Neither operator is a bad pick for smartphone users in Sydney, but our results allow them to make their choice according to their individual requirements.
Melbourne: Optus scores highest in all disciplines
Optus achieves the highest score in the Melbourne area according to our test results. This operator scores best in voice, in data and thus also in the overall evaluation. Telstra ranks second in voice and overall, while Vodafone is slightly ahead in data.
Adelaide: Local champion Vodafone
In Adelaide, Vodafone takes the lead both regarding the voice score as well as the data score. Overall and in voice, Optus ranks second here, while Telstra achieves the second-best score in the data discipline, but overall ranks third.
Brisbane: Telstra ahead, Optus behind
Telstra achieves the best overall test result in Brisbane and scores especially high in the data tests. In the voice discipline, the differences between all three contenders are small, whereas in data, Optus ranks clearly behind the other two.
Also in Perth: Local champion Vodafone
Perth is the second area where Vodafone is ahead of its competitors both in voice and data. Telstra follows closely with almost equal scores, while Optus falls a little behind with limited data results.
The methodology of the P3 connect Mobile Benchmark is the result of P3’s many years of experience. It was carefully designed to evaluate and objectively compare the performance and service quality of Australia’s mobile networks from the users’ perspective.
The P3 connect Mobile Benchmark in Australia took place from October 11th to October 24th, 2017. All samples were collected between 8 am and 10 pm. The network tests covered nine larger cities with more than 100 000 inhabitants each. Additionally, our test routes led through 19 smaller towns as well as the connecting roads between them. This combination of test areas had been carefully selected to provide a significant series of test results covering the Australian population. The areas chosen for the 2017 test account for approximately 13.7 million people, or 61 per cent of the total Australian population.
P3 conducted the tests with four drive-test cars, equipped with arrays of Samsung Galaxy S7 Cat 9 smartphones for the simultaneous measurement of cellular voice and cellular data services. Additionally, two teams conducted walktests in four large Australian cities.
Two smartphones per operator in each car were used for the voice tests, resulting in a total of six voice test devices per car or 24 devices in total. The walktest teams also carried two smartphones per operator in their backpacks. All phones were constantly setting up test calls between each other – from car to car and from each walking staff member to a stationary counterpart. The audio quality of the transmitted speech samples was evaluated using the HD-voice capable and ITU standardized so-called POLQA wideband algorithm. All Australian operators offer 4G capable subscriptions. To take the high share of LTE offerings into account, speech samples were acquired partly in 4G preferred to 3G preferred mode and partly in 4G preferred to 4G preferred mode. As a consequence, in 4G networks with VoLTE support, the phones would prefer this voice mode. If 4G (or VoLTE respectively) was not available, they would need to to switch (“fall back”) to 2G or 3G for the voice calls (so called “circuit-switched fall back“ or CSFB).
In order to account for typical smartphone use scenarios during the voice tests, background data traffic was generated in a controlled way through random injection of small amounts of HTTP traffic. The voice test scores account for 40 per cent of the total benchmark results.
Data performance was measured using one additional Samsung Galaxy S7 per operator in each car – resulting in three more smartphones per car or a total of 12 devices in total for the data drivetest. For the data walktests, each of the two teams carried one Galaxy S7 per operator. So, in total, the drivetest cars carried 12 devices for the data tests and the walktest teams carried six devices for the data tests. For all of them, the radio access technology was set to LTE preferred mode.
The web tests accessed the ten most popular web pages in Australia according to the widely recognized Alexa ranking. In addition, the static “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) was used. In order to test the data service performance, files of 3MB and 1MB for download and upload respectively were transferred from or to a test server located on the Internet. Furthermore, the peak data performance was tested in uplink and downlink directions by assessing the amount of data that was transferred within a seven seconds time period.
Another discipline was the playback of Youtube videos. It took into account that Youtube dynamically adapts the video resolution to the available bandwidth. So, in addition to success ratios, start times and playouts without interruptions, Youtube measurements also determined the average video resolution.
All the tests were conducted with the best-performing mobile plan available from each operator. Data scores account for 60 per cent of the total results.
The test routes are shown on page 1. In the 9 big cities and 19 smaller towns, the cars followed predefined routes. Altogether, the four test cars covered approximately 12 400 kilometres, of which approx. 4700 km led through big cities and 1300 km through the smaller towns, while 6400 km were covered on connecting roads and crossing the Outback. In order to cover both the east and the west coast, the drivetest teams flew from Brisbane to Perth, subsequently taking cars from there which also carried the measurement equipment.
During all of the drivetests, two cars were present in the same cities, but on different routes to avoid any interference of one car’s measurement by the other car’s.
The walktests were conducted in four large Australian cities. For this effort, the teams measured in so-called “areas of interest“ with a distinctive user frequency like central business districts, selected tourist hotspots and major shopping malls in Sydney, Melbourne, Brisbane and Adelaide. The teams even tested on public transport in each city, such as buses or metro train lines.
Performance indicators and rating
The score weighting reflects both the geographical distribution of Australia‘s population and the ranking of usage scenarios. Therefore, 600 of the total of 1000 maximum points were assigned to the cities – 240 maximum points refer to the voice results and 360 maximum points reflect the data results. For the towns and the roads, a maximum of 200 points each is available. In both categories, the possible maximum is 80 points in the voice, and 120 points in the data category. The score breakdown shown below depicts the distribution of the maximum points over all the drivetest and walktest categories. In the tables on page 2 and page 11 of this report, you can find the percentage of maximum points that each operator has achieved in each discipline.