The device and its interaction with the network impact all aspects of the user’s experience of mobile services. The device is the lens through which the customer perceives both the network and the service. In this sense, the device is the experience. As competition to attract and retain subscribers grows increasingly intense, it becomes a top priority to understand the challenges communication service providers (CSPs) and device manufacturers face in a fast-moving market, where competitive differentiation can make all the difference. Key considerations include:
CSPs need to safeguard against poorly performing devices to protect their brand
CSPs incur significant costs for poorly performing devices through customer support, returns, and most damagingly, from churn
CSPs need consistent experience across the entire device portfolio
Inconsistent testing procedures and lack of knowledge of the target network leave uncertainty that bands and band combinations are tested properly
Assessment of the network alone doesn’t quantify the end user experience
It all boils down to answering the question: does a device deliver a great experience and how is that determined? The simple answer is that mobile devices must be independently certified before their deployment on the network. This benefits key players in the 5G ecosystem in the following ways:
Service providers assure new mobile devices meet operator performance standards before going to customers on their network and raise the bar on overall user experience by comparing performance of devices across suppliers and enabling improvements over time to optimize the overall user experience
Device manufacturers improve device quality, identify issues prior to market release and reduce device returns and NTFs, while accelerating time to market and acceptance with appropriate R&D/Pre-testing
Chipset vendors improve chipset quality and gain critical tools for measuring performance of new technologies (e.g., for 5G data or HD/4K video)
The “how” of this certification process is the critical element of assuring success. A number of considerations should be kept in mind:
Support complex validation criteria. Devices perform differently depending on a complex set of factors including radio environment and band, specific network and device configuration settings, chipsets and other components, and the overall device design. The bottom line: the only way to know which device models deliver a superior user experience is to evaluate the device in the actual live network, and in some cases complement it with lab tests, to help identify the root cause of performance shortcomings.
Measure against gold standard reference handsets and performance databases. Define performance relative to reference devices and to the operator’s handset portfolio. Here, proven test methodologies and established best practices should be applied by a vendor-neutral industry expert in mobile device testing, with specialized expertise in 5G.
Focus on end user experience. To ensure user experience on a mobile device before launch, mobile quality of experience (QoE) testing is required to measure the perceived user experience on new devices. Parameters of testing should cover video quality, voice quality, data performance, application performance, and location accuracy – to predict real-world performance and acceptance. This must be performed in multiple locations and conditions: venues, urban and rural – driving, walking, and stationary. Device performance results must be verifiable and repeatable, and subject to independent audits.
How can mobile device certification be achieved with actionable results?
Once the objectives above are incorporated into a testing approach, executing on the strategy involves a number of key considerations. Your mobile device testing certification solution should not just measure the experience – it should also provide critical data for triaging the root causes of poor user experience. Analytics based on IP and RF data help isolate root causes impacting the device, network, or device-network interaction. As the act of logging on the device can sometimes impact user experience KPIs, a mix of logged and non-logged test cases gives you the balance you need between real-world assessment and actionable results.
Using in-network data helps operators triage device versus network problems to facilitate measurable improvement in key QoE metrics such as speech quality and throughput to foster a consistent and optimal experience across the device portfolio.
Testing criteria for definitive device certification
A suitableshould encompass an approach that evaluates the key factors impacting the user experience of calls: video, voice, and data for 5G, 4G LTE, Wi-Fi, and 3G services. The experience metrics should include call setup time, call setup success and retention, video smoothness, video MOS, speech quality, conversational speech delay, web browsing speed, file download speed, latency, and more. In greater detail, these testing criteria should cover the following areas:
Video calling in stationary and mobile environments
OTT video streaming by popular services
Branded video applications (chat or streaming)
Buffering, freezing, over-compression/MOS, time-to-first frame, frame statistics
Smart capture and processing to filter out CGI, graphical tickers, black bars that can impact scoring, region of interest
Voice experience (VoNR, VoLTE, VoWiFi, 3G)
Speech quality in stationary and mobile environments
Call initiation and retention
Speech and calling during handover
Speech and calling during simultaneous voice and data service usage
Network edge retention
Speech quality by codec
Speech quality in noise
Acoustic speech quality and loudness (Handset, HHHF, Bluetooth)
Data experience (5G, LTE)
Multiple protocols to cover routing priority differences
Control the device-side client and the cloud/edge server
Run both “typical user” and “fill-the-pipe” tests
Cover both stationary and mobility scenarios
Fingerprint the resource assignment to ensure fair comparison
File transfer speed, streaming data speed (UDP), web browsing speed in stationary and mobile environments, file transfer during simultaneous voice and data service usage
Assess how devices deliver today’s growing list of low latency-dependent services
Throughput, latency, jitter, loss
Control and user plane location
Lat/Lon and z-axis
Accuracy, time to fix and yield
Applying proven best practices for device certification
A suitable certification test strategy should also follow proven best practices which include:
Measure what matters. Functional testing is increasingly insufficient for assuring user experience (UX); combined live network and lab evaluation of UX is a better approach.
Stress the device like a user. Measurements of UX are only valid if they reflect real user behavior. Implement multi-service use cases to stress the device like a user.
Ensure repeatability. Factors impacting UX are highly variable. Each UX category has a different variability. If you don’t take enough samples, results won’t be repeatable.
Assess from the top down. A lot of time and money can be spent gathering tons of data you don’t use. Broad statistical assessments with focused drilldown into problems saves time and accelerates triage and isolation of UX issues to network, device, and service.
Replicate together. Devices are going global with fewer regional variations, but carrier-specific requirements and band combinations remain critical. Consistently replicate tests around the world, close to each team in the lab and live network.
Relative, not just absolute. All devices are not created equal, with a wide variance in UX across models. It’s a competitive marketplace. Showing rank by UX category is a powerful motivator. Consider relative performance, not just absolute.
Choosing the right certification partner
When selecting a device certification partner, keep your priorities in mind. These likely include protecting your brand and finding customer impacting problems before launch to ensure a positive user experience. It’s essential to avoid customer support issues, device returns and churn costs down the line. Finding and fixing issues earlier in the launch process requires consistent test methodologies and trusted results. Build your program to quantify the experience on your network.
When selecting a device certification partner, keep your priorities in mind. These likely include protecting your brand and finding customer impacting problems before launch to ensure a positive user experience.
Experience and best practices
A qualified device certification partner will take advantage of methods and best practices learned from an extensive legacy of QoE programs. These best practices must apply for evaluating and comparing user experience across devices and work for any device and any network technology from 5G to VoLTE, to Wi-Fi, to 3G. Most importantly, the best practices must include a robust statistical approach that ensures the best and worst performing devices are reliably identified.
Your partner should have established trust-based working relationships with global operators, OEMs, and chipset vendors. The partner should also consult with new suppliers to coach them through your process.
Innovation and investment
Your partner should continually invest in technology and solutions to assess what matters today and tomorrow to address elements of emerging demands. Examples include the evolution to 5G and mmWave testing, as well as market-leading OTT video, cloud gaming and mobile edge computing (MEC) solutions. Their focus should be on helping customers improve the mobile quality of experience and maintaining a commitment to continuous improvement and investment in new approaches to QoE test solutions such as remote execution, automation, and monitoring. This includes proactive monitoring of devices in the field, all of which improves device and service time to market and reduces costs while increasing effectiveness.
With these capabilities and qualifications in place, all of which Spirent possesses, service providers, device OEMs, and chipset vendors can be confident their mobile device certification program is reliable to support their device launches now and into the future.