One to rule them all

A quest to unify the different automation frameworks and testing workflows throughout the company.

Maxlinear has done a number of acquisitions in recent years, this vastly expanded the product portfolio spread across different sectors. As a semiconductor company adequate testing of these products is very important. The increased complexity of these products and the ever-increasing importance of the embedded firmware drives the need for solid test automation and continuous integration strategies.

The existing automation frameworks were reviewed, the requirements for the new unified framework were collected and the new architecture was forged. A modular python framework emerged to keep it lean while able to address a wide range of needs. Execution and result representation is handled by the Robot Framework, while test case management is provided by the JIRA add-on XRay.

One framework was very complete feature wise but was a purpose-built monolith for a single product line, hence a modular approach is desired. Creating python packages for each module in the architecture enables the use of standard python package tools and infrastructure. Every test project can then mix and match their modules to their specific needs while boosting reuse of the framework’s core modules.

Another framework was a C/C++ built GUI with a library of macro functions, while it enabled non-programmers to write their test cases it proved tough to maintain with only little people capable of expanding and fixing issues. By adopting python this threshold is lower and the same user experience was ensured by a custom PyCharm IDE plugin.

A different framework, again a C/C++ built GUI specifically developed for one product family, provided a nice interface to the different setups, their serial consoles, measurement equipment, etc. and enables test scripts to be executed. While it suffers the same drawbacks as the other C/C++ frameworks regarding maintenance it also turned out to be an obstacle in regression testing. It was designed to be operated by a human, not by some script needed for automated continuous integration.

All of the existing frameworks used spreadsheets in some form or another to manage the test cases, this does, however, fall short when processing large result sets or you want to extract complex metrics. A feature-rich add-on for JIRA, XRay, was selected and integrated into the framework enabling a unified workflow for test development, result reporting, and management.

Some projects have been switched to the new architecture and a lot of them will follow as the new framework matures and the feature set increases. For some projects they coexist, as the new framework fills the gap of features the legacy frameworks do not have. This even being an option shows how this new modular framework can actually rule them all.