2017|12|04 – THE 2017 P3 CONNECT MOBILE BENCHMARK IN AUSTRALIA
The benchmarking expert P3 communications has been testing the mobile network performance in Australia since 2014. Now, for its fourth mobile network test in Australia, P3 teams up with the internationally acknowledged telecommunications magazine connect. This combines P3’s longtime network benchmarking expertise with decades of connect’s
editorial experience. Together, we have carefully examined the performance and quality of the mobile networks in Australia.
It was a neck-and-neck race of the two largest Australian operators. This year, Optus takes the “best in test“ accolade from an also “very good“ Telstra.Vodafone confirms its position from the previous test, receiving the grade “good“ and catching up further.
Results in a nutshell
P3‘s network benchmarks are widely accepted for setting industry standards as well as being highly objective. The carefully designed methodology scheduled four drivetest cars to visit nine large Australian cities and 19 smaller towns as well as the connecting roads between them. In addition, two teams conducted walktests in four large cities. The areas in which we tested account for more than 13.7 million people, or about 61 per cent of Australia‘s population.
Another example for P3‘s great attention to detail is the use of up-to-date LTE “Cat 9“ as well as VoLTE-capable smartphones for the tests. They reflect the latest technical developments in the mobile networks and once more emphasise the scope of our benchmarking: How do the mobile networks perform at the edge of what is technically feasible – and to what extent do customers benefit from these capabilities? In order to provide valid answers to these questions, we have also used the most comprehensive mobile plans available from each operator. Also, we constantly readjust the thresholds of our evaluation to represent the latest technological advancements.
The pleasant results of P3‘s previous mobile network benchmarks in Australia have set high expectations. In fact, all three operators were able to meet them – Optus and Telstra managed to distinctly improve their scores, and Vodafone confirmed its position by gaining five points over last year‘s result.
Optus wins due to lead in voice, Telstra scores best in data
It was a neck-and-neck race of the two biggest Australian operators. Ultimately, Optus won with an advance of three points over Telstra. Both receive the grade “very good“. Optus secures its overall win with a clear lead in the voice category, while Telstra scores best in the data discipline. Vodafone ranks third, scoring behind the leading two both in voice and data. Still, Vodafone performs decently and gains the overall grade “good“.
The Australian mobile communications market is served by three operators that clearly differ in terms of market shares. All three of them offer the latest mobile technologies like LTE-Advanced and VoLTE.
According to its own statement, Telstra currently has 17.4 million mobile subscribers. This makes this operator market leader in Australia.
The history of Telstra dates far back: The company originated as part of the Postmaster-General‘s Department and was fully privatised by 2006. The last remaining government share was effectively sold in 2011. Today, Telstra offers voice, mobile and internet access, as well as Pay TV and other entertainment services to Australian customers.
The operator provides 3G (UMTS) services at 850 and 2100 MHz and a 4G (LTE including LTE-Advanced) network on the 700, 900, 1800 and 2600 MHz frequency bands. Like its competitors, Telstra has been continually increasing its LTE coverage and offerings, now providing LTE-Advanced under the name “4GX“ and claiming a 4G coverage of 99 per cent of the Australian population.
CLOSER LOOK AT AUSTRALIA’S NETWORKS
Over the past years, all operators in Australia have shown continuous improvements which reflected in our network tests. We are happy to report the trend persists in 2017.
The German company P3 is a world leader in mobile network testing. It is part of the P3 group, with over 3000 employees worldwide and a turnover of more than 300 million Euros (468 million Australian Dollars).
In 2017, P3 runs its authoritative mobile network benchmark in Australia for the fourth time. In 2017, this is realised for the first time together with the international telecommunications magazine connect for its network test in Australia. The magazine has more than 20 years of editorial experience and is one of the leading test authorities for telecommunications products and services in the world. Together, P3 and connect have been conducting mobile network benchmarks in Europe for more than 15 years.
While data communications are ever more prevalent, customers still expect excellent voice quality. How do the Australian mobile networks perform in this discipline?
In the communications habits of many users, speaking on the phone has become less important in comparison to internet-based communications like instant messaging or e-mailing. But telephony is still a key feature of every mobile network – and in those cases when people use it, they naturally expect top reliability and a high speech quality.
Australian networks are well positioned for fulfilling these expectations, as all three operators have deployed Voice over LTE (VoLTE). This modern technology helps to better coordinate voice and data communications in the data-centric LTE networks and avoids kludges like the “circuit-switched fallback“ that forces a smartphone to switch to a 2G or 3G connection while conducting phone calls.
Voice: Optus is the champion, Telstra overall second, Vodafone strong in big cities
The drivetests and walktests in the big cities show a high level of reliability and performance in all three networks. Optus is a little ahead in the drivetest scenarios, while both Optus and Vodafone lead by a small margin ahead of Telstra in the walktests.
The ranking order becomes a little more distinct in the smaller towns. Here, Optus leads the field, followed at a small distance by Telstra and, again with a little gap, by Vodafone. The results here are still good. As can be expected, they drop considerably on the roads: Here, Optus and Telstra are still performing quite well, while the score of Vodafone drops to a comparably meagre 49 per cent of the total achievable points.
VOICE results At a Glance
Overall winner Optus also scores highest in the voice category. In the big cities, Vodafone and Telstra are following closely. In contrast, in the smaller towns and on the connecting roads, the ranking order is clear: Optus ranks first, Telstra second and Vodafone third.
With data volumes growing rapidly and tight limitations on available bandwidth, all operators face challenges in providing a satisfactory user-experience. Which Australian contender manages to best meet the growing demand?
According to the Australian Bureau of Statistics, the volume of data downloaded via mobile devices from May to June 2017 was 175,076 Terabytes. This represents a 19.9 per cent increase compared to the same period of the preceding year. For the 26.3 million mobile subscribers of the country, this equates to an average of 2.2 GB of data downloaded per subscriber per month. These numbers not only prove that mobile internet usage is hugely popular in Australia, but also that the typical user behaviour puts a heavy load on the mobile networks and poses a considerable challenge to all Australian mobile operators. Telstra, Optus and Vodafone are continuing to spend billions of dollars on upgrading and expanding their networks to meet these demands.
A good mobile internet experience demands quick loading times. This includes the download of different types of web-pages, downloads and uploads of large files as well as the increasingly important reception of video streams from services like Youtube. In order to check the reliability and performance of data connectivity in the Australian networks, one Samsung Galaxy S7 per operator was installed in each of our four test cars and in the backpacks carried by each walktest team.
The phones constantly performed a suite of tests. For most of these checks, success ratios and average session times were logged. For downloads and uploads we also recorded average throughputs. In order to assess typical performance as well as peak speeds, we determined the minimum data rates that are available in 90 per cent of the cases plus the peak data rates that would be surpassed in 10 per cent of the cases.
AUSTRALIA’S FIVE LARGEST CITIES
Sydney: Telstra leads overall and in data, Optus strongest in voice
In Australia‘s biggest city, the market leader Telstra is closely ahead of its opponents Optus and Vodafone. Telstra scores particularly strong in the data tests, where Vodafone ranks second and Optus third. Optus makes up leeway in the voice discipline where it closely leads ahead of Telstra and Vodafone who turn out to be equally strong in telephony. Neither operator is a bad pick for smartphone users in Sydney, but our results allow them to make their choice according to their individual requirements.
The methodology of the P3 connect Mobile Benchmark is the result of P3’s many years of experience. It was carefully designed to evaluate and objectively compare the performance and service quality of Australia’s mobile networks from the users’ perspective.
The P3 connect Mobile Benchmark in Australia took place from October 11th to October 24th, 2017. All samples were collected between 8 am and 10 pm. The network tests covered nine larger cities with more than 100 000 inhabitants each. Additionally, our test routes led through 19 smaller towns as well as the connecting roads between them. This combination of test areas had been carefully selected to provide a significant series of test results covering the Australian population. The areas chosen for the 2017 test account for approximately 13.7 million people, or 61 per cent of the total Australian population.
P3 conducted the tests with four drive-test cars, equipped with arrays of Samsung Galaxy S7 Cat 9 smartphones for the simultaneous measurement of cellular voice and cellular data services. Additionally, two teams conducted walktests in four large Australian cities.
Two smartphones per operator in each car were used for the voice tests, resulting in a total of six voice test devices per car or 24 devices in total. The walktest teams also carried two smartphones per operator in their backpacks. All phones were constantly setting up test calls between each other – from car to car and from each walking staff member to a stationary counterpart. The audio quality of the transmitted speech samples was evaluated using the HD-voice capable and ITU standardized so-called POLQA wideband algorithm. All Australian operators offer 4G capable subscriptions. To take the high share of LTE offerings into account, speech samples were acquired partly in 4G preferred to 3G preferred mode and partly in 4G preferred to 4G preferred mode. As a consequence, in 4G networks with VoLTE support, the phones would prefer this voice mode. If 4G (or VoLTE respectively) was not available, they would need to to switch (“fall back”) to 2G or 3G for the voice calls (so called “circuit-switched fall back“ or CSFB).
In order to account for typical smartphone use scenarios during the voice tests, background data traffic was generated in a controlled way through random injection of small amounts of HTTP traffic. The voice test scores account for 40 per cent of the total benchmark results.
Data performance was measured using one additional Samsung Galaxy S7 per operator in each car – resulting in three more smartphones per car or a total of 12 devices in total for the data drivetest. For the data walktests, each of the two teams carried one Galaxy S7 per operator. So, in total, the drivetest cars carried 12 devices for the data tests and the walktest teams carried six devices for the data tests. For all of them, the radio access technology was set to LTE preferred mode.
The web tests accessed the ten most popular web pages in Australia according to the widely recognized Alexa ranking. In addition, the static “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) was used. In order to test the data service performance, files of 3MB and 1MB for download and upload respectively were transferred from or to a test server located on the Internet. Furthermore, the peak data performance was tested in uplink and downlink directions by assessing the amount of data that was transferred within a seven seconds time period.
Another discipline was the playback of Youtube videos. It took into account that Youtube dynamically adapts the video resolution to the available bandwidth. So, in addition to success ratios, start times and playouts without interruptions, Youtube measurements also determined the average video resolution.
All the tests were conducted with the best-performing mobile plan available from each operator. Data scores account for 60 per cent of the total results.
The test routes are shown on page 1. In the 9 big cities and 19 smaller towns, the cars followed predefined routes. Altogether, the four test cars covered approximately 12 400 kilometres, of which approx. 4700 km led through big cities and 1300 km through the smaller towns, while 6400 km were covered on connecting roads and crossing the Outback. In order to cover both the east and the west coast, the drivetest teams flew from Brisbane to Perth, subsequently taking cars from there which also carried the measurement equipment.
During all of the drivetests, two cars were present in the same cities, but on different routes to avoid any interference of one car’s measurement by the other car’s.
The walktests were conducted in four large Australian cities. For this effort, the teams measured in so-called “areas of interest“ with a distinctive user frequency like central business districts, selected tourist hotspots and major shopping malls in Sydney, Melbourne, Brisbane and Adelaide. The teams even tested on public transport in each city, such as buses or metro train lines.