Endorsements
Motorola - Access Products
The GPON Broadband Access Group of Motorola, located in Lowell, MA, develops equipment that provides Fiber to the Premises (FTTP) broadband services to residential homes and commercial subscribers. This optical network equipment resides in both the head-end central offices as well as termination points located in each home or commercial office.
Motorola engineers are developing two complex FPGAs for the next generation GPON equipment. These FPGAs implement the intelligent packet processing and forwarding functions that are needed to deliver end-to-end ultra-high broadband services. This Motorola team is made up of experienced ASIC and FPGA design and verification engineers who have used and developed several successful methods to track their status and testing over the past twenty years.
As they began the work to design and verify the GPON FPGAs, the team knew that they wanted to develop or license a high quality system of real-time test and status tracking tools. Given their past experience, the bar was high. The system needed to facilitate transparency and collaboration within the team, and it had to fit seamlessly into their existing blend of tools, scripts, and methodology.
The combination of constrained random testing and automated regression scripts with large server farms create the ability to run hundreds--sometimes thousands--of tests per engineer, every night. While this amount of testing is necessary to verify the rich functionality of these complex FPGAs, generating and executing all these tests also results in a large number of results that need to be inspected and failures that need to be analyzed every morning. This can make for a daunting backlog of engineering teams some mornings.
Motorola needed the ability to track every test result, characterize it, and catalog it automatically. For every reported failure, teams need to do a root cause analysis to determine why the test failed and if the failure represents a bug to be fixed. Engineers are hesitant to file bugs until they are sure there is a real problem with the design. Test scripts report failures for many reasons, including non-design bugs. Server timeouts, license failures, script errors and test bench errors can create clutter in the results that make real design bugs harder to spot. Filtering and classifying all of these failure types, as well as tracking the historical behavior of tests, can be a daunting task when hundreds of tests are run each night per engineer.
The DV Notebook allowed the GPON team to create all of the custom web based dashboards and automated result tracking that they needed, without the schedule or budget impact of an in-house development.
To ensure the quality of their product, the engineers at Motorola needed to be able to thoroughly track and resolve each and every testing failure reported by their verification environment. Any manual process of engineers taking notes, sending email, and using spreadsheets would not be efficient enough to manage the immense quantity of result data. As a small team, they could not afford anything that might be unwieldy, prone to error, or time consuming. They needed an automated solution that would integrate directly with their existing, in-house, multi-vendor tools environment.
Whether the goal is an overall picture of health and progress, or one-click access to a detailed log or wave files, the DV Notebook provided the flexibility to present the engineer with the exact information needed to analyze problems and to make progress. The GPON engineers use DV Notebook dashboards to categorize and sort test results using graphs, charts, heat maps, and interactive result tagging pages. The resulting system tracks verification progress and shows real-time status to all team members in any web browser. The web-based dashboards allow each engineer and manager to drill down to the level of detail they need for the task at hand.
Motorola engineers on the GPON project defined a deployment for the DV Notebook that allowed them to track, tag and comment on every test result. These tags feed back into high-level views of debug activity that is visible to the whole team from via their web browsers, keeping everyone in sync. On prior projects, designers had kept detailed test tracking notes on each RTL failure using spreadsheets. However, manually cutting and pasting test results from emails and text editors into the spreadsheet had proven to be time consuming and error prone. In addition, these files were not easily viewed by the rest of the team, so status meetings were needed to report on progress. The DV Notebook solved these problems.
The GPON team also recommended specific visualizations for their project data. Engineers had a picture in their minds of how FPGA projects should evolve. The advantage to the DV Notebook is that now they can see that activity on the screen, point to it, and discuss it. The insight that one or two engineers might have had about the quirks and trends in test results are now up on the screen, so that the team can agree on where they are and what each of them should concentrate on.
"As the three project leads, Ted, Ben, and I are keeping track of our nightly regressions primarily through the Achilles tools now. The DV Notebook, and the heat maps in particular, save us a great amount of time every morning. Everyone?s testing is on a single heat map. It tells us where we are and what is the most critical thing to do next. I can start with a overall summary of the past month's test results; and then start to drill down to gather more and more information until I am finally viewing the exact log and wave files of a test that I want to debug. The DV Notebook doesn?t let ANY failure fall through the cracks." - Kirk Anderson, Motorola, Inc.
The engineers at Motorola have reduced the amount of time manually managing the results of their nightly and weekend regression from 3-4 hours down to less than an hour! They have reduced the amount of time managing regression results by 50%, saving at least 2 hours a day. They can quickly resolve trivial failures in order to focus their analysis on more critical errors. This buys the team time to continue developing new testing features to ensure completed testing coverage of the design.
The DV Notebook allows engineers to track hundreds and even thousands of tests per engineer every day and to record detailed comments and insights as verification progresses. Without the benefits of the DV Notebook, engineers must either develop an in-house result tracking system, or manually triage errors and results, using spreadsheets or spiral notebooks. These options are time consuming and do not provide full exposure of the information to the rest of the team.
UNH-IOL
The University of New Hampshire Interoperability Laboratory
(UNH-IOL)
uses all three products offered by Achilles Test Systems to perform high-volume testing of DSL equipment.
They were able to use the DV Notebook, the DV Notebook SF, and the QA Notebook to conduct highly efficient automated
testing in conformance with the DSL Forum TR-67 and TR-100 test plans.
They found this solution was more flexible and adapted better to change and also provided better storage efficiency
than their existing database solution.
Read the following case study for more details.
Case Study: Achilles Provides Agile Test Generation and Dynamic Management
Customer: UNH-IOL
Introduction: Millions of people rely on DSL every day. The
DSL Testing Consortium
at UNH-IOL is committed to testing ADSL, ADSL2, ADSL2plus, VDSL, and VDSL2 products. The consortium brings together members
and industry to foster interoperability among DSL products and increase the market feasibility of DSL technology.
Situation: The DSL Testing Consortium within IOL provides world-class automation tools to vendors to ensure
interoperability between devices. IOL's testing solution,
LASI,
is a leading edge suite that incorporates DSLAM test beds, Line Simulators,
Waveform Generators, and Traffic Analyzers. One of the common advantages to both LASI and the Achilles QA Notebook
is computerized data acquisition and automated testing control of the DUT and testing devices in parallel. IOL
requires 24-hour testing and provides identical testing conditions; no matter when or how many times a device is tested.
The testing environment remains consistent. This precision affords regression testing of new hardware and firmware versions
by revealing any changes in the device's connectivity.
Problem: To meet anticipated and emerging needs for interoperability verification, IOL found the specific implementation
of the current solution to be too rigid and inflexible. One of the biggest problems IOL faced when executing Broadband Forum
TR-67 and
TR-100 test
plans is in adapting detailed test sequences dictated by the standards body to the specific capabilities of particular DLSAMs
and DSL Modems. Even more demanding is the need to create custom parameter sweeps to instantly meet the needs or requests of
IOL's customers.
LASI has a very powerful test sequence generation, but it is built into the same code-base as the hardware drivers.
Making changes in the test parameters was time consuming.
In other words, test sequences were being generated and run at the same time. To maintain this system, the software needed
to grow larger and more complex for each new testing sequence and each hardware type that was tested.
IOL needed a more dynamic solution.
Solution: The Achilles Test Notebook products were chosen for this application because they allow the IOL to decouple the
generation of test parameters from the actual execution of the test.
"One specific example of the many benefits that this system provides us is the ability to separate the generation of tests
from the actual execution of the tests. This flexibility is very powerful and provides great benefit to us. The Achilles
Test System tools provide the organization, agility and dynamic management needed to accomplish our vision of where testing is
going. It allowed us to add a high degree of intelligence to our testbench. We have been able to create a system
quickly and at a very low cost, which greatly improves our current testing process," said Jason Walls, Senior Technologist
at UNH-IOL.
Conclusion:
The three major tools of the Achilles Test Suite work together in a seamless fashion and provide the flexibility and adaptability
to automate the process AND meet UNH's diverse set of needs in an efficient and timely fashion.
Configuration:
| Processor: | Intel Core 2 Duo, 2.13GHz |
| Memory: | 2GB DDR2-667MHz |
| Hard Drive: | 160GB 7200 rpm |
| Operating System: | RedHat Enterprise Linux |
Software:
Achilles Test Systems: DV, SF and QA Notebook