Conformity Magazine

See this article in our Digital Edition
Download a pdf of this article (right click to save the pdf)

The use of Wi-Fi connectivity in non-PC based mobile devices, such as MP3 players, dual-mode cellular and Wi-Fi VoIP phones, video games, printers, smartphones, PDAs and televisions, is rapidly growing. In fact, International Data Corporation (IDC) states that shipments of these devices, commonly know as application-specific devices (ASDs), have grown by more than 50% in the past year, and nearly 300 million devices are forecast to sell in 2010 (see Figure 1).

Figure 1: Wi-Fi-enabled ASD shipment forecast

The industry, however, is quickly discovering the complexity involved in incorporating Wi-Fi into a product design, and the difficulties of testing connectivity. These challenges include the complexity of configuring and using Wi-Fi, the cost of adding the hardware, and the difficulty of integrating and testing Wi-Fi-enabled devices. For companies, test labs, and industry groups like the Wi-Fi Alliance who are testing the interoperability of these new products, this becomes a challenge of scope that needs to be addressed.

This article will discuss the need for scalable test methodologies and tools to support interoperability testing of Wi-Fi enabled devices. It will also outline the Wi-Fi Alliance’s approach for station testing, and highlight how this approach improves the flexibility and scalability of the certification program to support non-PC devices. Finally, the article will discuss some of the challenges that design, development and quality assurance engineers may encounter while implementing this new method, and provide an overview of the certification testing process using the new approach.

The Need for a Scalable Test Methodology
When the original IEEE 802.11 standard was specified in 1997, the unique requirements of ASDs were not considered. The traditional Wi-Fi interoperability test methodology, developed around PC-centric devices, utilizes a standard PC keyboard interface, “off-the-shelf” traffic generators and performance measurement tools, and established high-throughput benchmarks for practical demonstration of device interoperability. Unlike PC-centric devices, ASDs are not networked in the traditional sense, as they lack a common interface for control, operation or testing, and typically do not support “off-the-shelf” test tools like traffic generators and performance measurement software.

As a result of these differences, the traditional Wi-Fi test methodology does not adequately accommodate ASD testing. Efficient ASD testing requires a more flexible methodology capable of accommodating unique user interfaces, such as a keypad or touch screen, different operating systems and varied security mechanisms. Another consideration in testing ASDs is that application processors are typically less powerful, which limits the overall performance capabilities when compared with a traditional PC.

As ASDs such as game consoles, cell phones, digital picture frames and MP3 players with embedded Wi-Fi started emerging in the market, interoperability testing of these devices necessitated the development (by the device manufacturer) and approval (by the Wi-Fi Alliance) of a manual, customized test plan for each new product – a costly and complex process that did not provide the scalability needed to efficiently test the increasing number of ASDs. It quickly became apparent that without a new scalable methodology to streamline ASD interoperability certification, device testing expenses and time-to-market would increase, resulting in higher product costs, lower product quality, delayed device shipments and slower market growth.

The Wi-Fi Alliance Test Engine Methodology
The Wi-Fi Alliance, a global, non-profit industry association that promotes wireless technologies, provides its members with testing and certification programs based on the IEEE 802.11 specification. A critical step in bringing Wi-Fi products to market, Wi-Fi Alliance certification validates the interoperability of Wi-Fi devices from different manufacturers, ensuring an optimal user experience. Since the testing program began in March of 2000, more than 4,000 products have been WI-FI CERTIFIED™. Growth in the diversity of Wi-Fi-enabled products, including consumer electronics and dual-mode mobile phones, have led to the certification of more than 1,000 Wi-Fi products over approximately the past year and a half¹.

The Wi-Fi Alliance recognized the challenges of testing the exploding product category of ASDs and introduced the Wi-Fi Alliance Test Engine methodology in an effort to address them. The Test Engine methodology allows station tests to be streamlined for any type of product. The key component is a framework for client testing that is extensible to any Wi-Fi client, and which does not compromise the integrity of the comprehensive Wi-Fi testing processes.

In defining the Test Engine methodology, the Wi-Fi Alliance ASD technical task group proposed a framework referred to as the Wi-Fi Alliance Test Engine, or simply the Test Engine. Its three key components include:

Station configuration and control: The methodology defines a standard API through a relay that allows vendors to design stations that can be configured for the tests and then stimulated to perform the actions required. Because the devices being tested lack standard user interfaces such as a keyboard or mouse, the interface is defined independent of the vendor’s implementation.
Test traffic generation: A large part of interoperability testing requires the transmission and reception of traffic by the ASD as a way to measure throughput, connectivity and functionality. The ASD initiates or terminates this traffic, and the Test Engine methodology defines and provides a traffic generation element for integration into the ASD. The traffic generator is specifically designed for devices with limited memory and processing power.
Test results measurement: The Test Engine uses the ADEPT-WFA tool (from Azimuth Systems) and test automation to enable traffic analysis and reporting that is independent of the capabilities of the ASD. The methodology uses intelligent capture and analysis techniques to validate connectivity, performance and functional testing.

In addressing these areas of station testing, the Test Engine provides a scalable approach that is suitable for the growing variety of Wi-Fi enabled products. The resulting test solution, required to certify products using the Wi-Fi Alliance Test Engine methodology, includes five key components (see Figure 2):

DUT software: The Wi-Fi traffic generator (WTG) software and device under test (DUT) agent reside on the ASD. The WTG software generates the Wi-Fi traffic used in the tests, and the DUT agent software enables communications to configure and control the ASD.
Control PC software: This relay control software implements the parser for the control API to convert commands to the ASD’s native control protocol, command set and physical interface (often USB).
Capture engine: The capture engine provides an independent mechanism to capture accurate, time-stamped promiscuous wireless and wired traffic for results analysis.
Test Engine management tool: The management tool controls and configures the ASD via the control PC and executes the test plan. In addition, the management tool controls the other test bed equipment.
Alliance test bed: The test bed is made up of clients and access points used as reference devices for interoperability, helping vendors verify products in a multi-vendor marketplace.

Figure 2: Key components of the Wi-Fi Alliance Test Engine methodology

The Wi-Fi Alliance provides sample DUT software and control PC software in source code form to its members. Designed to prepare ASDs for Wi-Fi certification, this software is designed to be easy to port and use in these unique environments. The Test Engine management tool automates the entire test plan for Wi-Fi Alliance certification.

Implementing the Test Engine
Porting the Wi-Fi Alliance software for configuration and traffic generation is the first step toward certifying a non PC-based device with embedded Wi-Fi connectivity. The Wi-Fi Alliance provides the reference implementation source code of these components to assist in installing the software on the device. The Wi-Fi traffic generator is small-footprint code that is easily portable to any operating system.

The Test Engine configuration software standardizes programmatic control of the ASD, including the security modes WPA-PSK and WPA2-PSK and different extensible authentication protocol (EAP) types if the ASD supports enterprise authentication.

Each device may also have a unique method for sending configuration commands, such as USB, COM, IrDA and Ethernet. The Test Engine methodology’s concept of a DUT control agent (DUT-CA) abstracts the control interface, allowing the user to implement configuration commands over any interface supported by the ASD. The DUT-CA approach is also a secure approach, allowing the user’s intellectual property, the control interface and enabling commands, to be kept secret.

The DUT-CA receives each command in the standard format specified by the test engine – essentially ASCII characters sent over a TCP socket. Each command is then translated into proprietary ASD-specific commands and executed, after which the DUT-CA may return a result.

To cause the ASD to perform an action, the test manager issues an appropriate command. This command passes through the control network and arrives at the PC that runs the DUT control agent, which ultimately controls the ASD. The ASD then executes the command and performs the action required for the test (see Figure 3).

Figure 3: Depiction of communication between Test Manager and DUT

While this architecture was originally designed for certification testing of ASDs, it is important to recognize that vendors can also use these tools for performance and certification testing of both ASDs and traditional Wi-Fi devices.

Certification Testing using the New Approach
The Test Engine architecture automates test plans through scripts and libraries that allow vendors to:

Operate the ASD through the API defined in the Test Engine methodology;
Generate traffic from the ASD through the wireless traffic generator;
Capture the resultant activity;
Analyze the results and determine whether the device has passed or failed the test.

Many of the certification test cases require traffic capture and analysis of packets to and from the ASD to determine test results. By capturing packets on its 802.11 and Ethernet interfaces, and performing careful analysis on those packets, the capture engine is used to:

Verify the number of test packets successfully sent and received by the ASD during the test;
Measure the throughput performance of the data transfers;
Validate that the content of the packets is in accordance with test plan requirements.

The automation control automates configuration of the ASD, traffic generation from the ASD and captures test packets using the capture agent. The script automatically determines the throughput achieved for all the traffic tests and determines if those test cases have passed or failed.

Performance Testing for ASDs
The challenges faced by Wi-Fi Alliance certification that are addressed using the Test Engine methodology are no different for a vendor validating the ability of a phone to deliver good voice quality, or a picture frame to download a file in a reasonable amount of time. The Test Engine methodology can be extended to a broad range of wireless functional, interoperability and performance testing of non-PC devices.

(An example of such testing is the author company’s FMC Performance Test Suite. This test script includes a series of benchmark tests used to measure and compare the voice and data performance of dual mode (Wi-Fi/Cellular-enabled). The FMC Performance tests uses the standard Test Engine methodology, coupled with extensions for both voice and data functionality, to automate and run voice quality, data throughput, roaming, Wi-Fi/Cellular network handover and battery life tests showing the universal nature of the test engine architecture.)

The ability to automate the device for testing and the ability to control the activity, for example, data transfers such as Web browsing or voice calls, is central to manufacturers and service providers being able to efficiently test and certify device performance and network compatibility.

Summary
As the number of products with embedded Wi-Fi connectivity continues to grow, the industry is discovering the complexity involved in incorporating Wi-Fi into a product design. The Wi-Fi Alliance recognized the challenges of testing this exploding product category and introduced the new Wi-Fi Alliance Test Engine methodology in an effort to address them and provide vendors with a scalable testing approach. The benefits of this new methodology include:

A standard approach that simplifies device implementation and can be used by different vendor platforms;
A standard platform to run the certification test, which can reduce Wi-Fi certification test costs;
An approach which is scalable and can be used to test interoperability, performance and functionality of diverse vendor solutions;
Fully automated certification testing that allows multiple SKUs to be tested repeatably in unattended mode.

The net result is a methodology that is cost effective and efficient for vendors and test labs to implement. The new approach meets the challenges of testing the growing number of Wi-Fi enabled devices and will enable continued proliferation of interoperable, Wi-Fi Certified products.

Graham Celine is a senior director of marketing with Azimuth Systems, and can be reached at graham_celine@azimuthsystems.com.

Notes

“More Than 4,000 Products Wi-Fi CERTIFIED™ by the Wi-Fi Alliance®” Wi-Fi Alliance, November 12, 2007