Migrating the Magento 2 test suite to Magento

One of the biggest weaknesses of Magento – if it’s possible to talk about weaknesses at all compared to other shop systems – is undoubtedly its lack of a PHPUnit test suite similar to the one that will be introduced with Magento 2. As Magento is currently unable to name a final completion date for Magento 2, Magento will almost certainly remain in use for projects for longer than originally expected. It seems that Magento has also reached this same conclusion because major advances have been realised over Versions CE 1.7.x and EE 1.12.x regarding testability with PHPUnit.

Probably every developer who has ever worked with Magento would agree that a test suite to automate checks of Magento’s basic functionality in the form of unit and integration tests is sorely missing. What’s more, own developments also need to be testable of course, both as standalones and interacting with the basic functionality. If we take a closer look at the Magento 2 test suite, Magento’s strategic direction for the next few years is clear. In view of the fact that quality is a key criterion for assessing Solution Partners, Magento will in future supply its development collaborators with the test suite and the Magento Automated Testing Guide – two tools to help achieve this vital quality boost and push up the Magento partner rating.

In this article, Tim Wagner of TechDivision describes the steps required to migrate the test suite from Magento 2 to Magento and provides examples to illustrate various applications as well as the suite’s potential for integration into a build and deployment process.

Possible test methods

The test suite unites the four most popular test methods. Unit, integration, and performance tests are shipped with the suite, which also includes tests for the static code analysis. Apart from the performance tests, all of the above are based on Sebastian Bergmann’s PHPUnit. This article is mainly concerned with the unit and integration tests, which are the most important for the majority of developers. By combining these two test methods, the manufacturer is aiming to significantly enhance the quality – and hence maintainability – of Magento projects in the medium to long term. I won’t go into the details of the static code analysis here because they can be separately debated in connection with the build and deployment process, and at the migration time the analysis’ implementation by Magento was only very rudimentary. I also won’t pay any attention to the performance tests because they have to be implemented with JMeter and are likewise called separately.

Porting of the test suite

Based on our build and deployment process, in which the developer has the option of automatically creating a new Magento instance at any time using the build tool – in this case ANT – one of the chief objectives of the porting, aside from a marked quality improvement, was to give developers immediate access to the tests whenever they create a new development instance and enable these tests to be seamlessly integrated into all of the development processes. The porting took place in three steps starting in April 2012.

Step 1: Availability of the test suite as an extension

In the first step, the Magento 2 sources valid at the time of porting were cloned from the Git repository. The test suite was then built on as a separate extension, meaning first that it can be installed in any Magento shop as of Version CE 1.7.x/EE 1.12.x and second that an installation package extended with the test suite can be created by an automated process for each new Magento version and made available on a build server.

As shown in Illustration 1, in addition to the necessary configuration and the helper classes in the app directory, the extension thus includes the bootstrapper (labelled 1 in the image below), the dev directory (2) copied from the Magento 2 sources, and the lib and downloader directories (5 and 3).

The lib directory contains several classes that are specific to Magento 2 but are needed by the test suite for the initialisation. The downloader directory has files belonging to older Magento versions that are required for the downloader. Although the downloader is no longer shipped with current Magento versions, it is essential to generate the code coverage report correctly. In some files Magento still erroneously references PEAR classes (4), even though they are not actually needed anymore for operation. Unfortunately, the static code analysis with PHPUnit returns a fatal error if these classes are not found in the downloader directory. The PHPUnit check for the sources when the code coverage report is generated could also be optionally dispensed with using the whitelist in phpunit.xml.dist.

Step 2: Adaptation of the test suite to Magento

In the second step, the test suite shipped with Magento 2 had to be adapted to Magento. At the start of the migration we specified Magento CE 1.6.x/EE 1.11.x as the target version. However, we soon realised that modifications to the Magento core classes would be inevitable in order to use the test framework. In Version CE 1.7.x/EE 1.12.x and higher, the necessary changes have been made by Magento directly in the core, so that no further adaptations are needed for the test suite to be usable.

Test rewrites

The above-mentioned necessary changes included rewriting the bootstrapper, which is something we’ll discuss later, as well as adaptations to the tests themselves. The latter proved to be far more complicated owing to restrictions in Magento.

The architecture – mainly the design and layout – has been altered, meaning that a lot of the Magento 2 tests couldn’t be simply replicated, for instance because files and directories just don’t exist anymore and methods have changed or been added for the first time.

One of Magento’s not-so-nice restrictions was revealed when we attempted to rewrite the tests. The annotations for creating test data (see the "Creation of test data" section) in Magento 2 can be specified either on the class level or on the method level. However, when they were used with Magento, problems were frequently encountered with inconsistent test data, presumably because the database layer in Magento 2 had been changed. The test cases concerned had to be rewritten accordingly during the migration, in other words the annotations had to be set to method level if not disabled altogether.

The affected tests – currently around 630 – were not deleted but skipped, and we plan to test and rewrite them in due course. Moreover, 27 test cases were declared by Magento as incomplete at the time of porting. Where appropriate, these will also be completed in future versions of the test suite.

Adaptation of the bootstrapper

The test suite must be initialised by means of a bootstrapper in order to run the unit and integration tests. As shown in Listing 1, the bootstrapper is registered in PHPUnit’s configuration file phpunit.xml.dist; it initialises the runtime environment in which the test suite is executed.

With regard to the unit tests, all this means is that the include paths are set, the autoloader initialised, and the temporary files directory emptied.

Considerably more work is involved when it comes to the integration tests because a Magento sandbox based on the current shop configuration has to be created along with a test database. Magento also enhances PHPUnit with extra profiling options and a few observers that make it much easier for developers to write tests.

In other words, the additional annotations provided by the test suite

• @magentoAppIsolation
• @magentoDataFixture
• @magentoConfigFixture

support the developer with functions for creating test and configuration data automatically and running individual tests in an isolated (reset) instance.

Sandbox tests

The test suite creates a sandbox before running the tests. All global configuration files as well as all configuration files contained in the extensions are copied to a temporary sandbox directory underneath the test suite. During the bootstrapping process the shop is initialised on the basis of this directory structure. Since the tests demand adaptations to the configuration files, this method ensures that no changes are necessary to the shop in order to run the tests, which can take place completely isolated and without any side-effects from the live system.

Local vs distribution testing

During the implementation phase it became obvious that running every single test on every local build isn’t viable in practice because it would take too long. On a fully featured development machine the test suite alone, including the code coverage report, already takes more than fifteen minutes without the tests for the various extensions. In addition to the default configuration file phpunit.xml.dist, which executes all tests (distribution testing), there is also a configuration file called phpunit.xml.local for local development or for creating new package versions; this second file only runs the tests contained in the namespace of the current extension (local testing). If the extension has the name TechDivision_GermanTax, for example, and the tests are called using the phpunit.xml.local configuration file, all tests underneath the dev/tests/integration/testsuite/TechDivision directory will be executed and the code coverage report will only be generated for files with the relevant extension; depending on the number of tests, this can add up to a huge time saving.

Step 3: Integration into the development process

The third and last step of the porting phase was probably the biggest challenge of all: the integration into the development process. Both the test suite and the tests are shipped with Magento 2 as standard while the Magento sources are supplied without the test suite, of course. Magento packages for local development have been extended with the test suite to make the integration as straightforward and seamless as possible. For one thing, the tests can be started from the local development environment anytime by specifying an ANT target and for another, developers can be certain during the build process that the package won’t be created until all tests have been successfully completed. By calling the various ANT targets, they can decide spontaneously which test cases – namely unit and/or integration tests – should be executed. With the help of the build properties they can also choose whether to run only the tests for the extension currently under development or the core tests as well.

Tools like Phing or, in our case, ANT can be selected for the build and deployment process. The targets shown in Listing 2 are provided for running the tests.

As the name implies, the ANT target phpunit-run-integration-tests is used to run the integration tests in the test suite. A total of 1,950 tests are executed in the present version, although – as mentioned earlier – 628 of these are skipped and 27 are incomplete owing to incompatibilities with Magento. The current test suite with 2,990 assertions covers approximately 11% of all Magento core classes, i.e. almost 90% of the Magento source text is not covered by tests.

These targets can be executed either direct from the local development environment or automatically by the CI server during the build. The build properties (see Listing 3) let the developer configure whether or not tests should be run during the build process, and if so which ones (distribution or local). The database for running the integration tests is also configurable; a separate, empty database is created as the default option.

Illustration 2 shows the output in the command line after running the integration tests in the current version of the test suite.

If the tests are executed successfully, the package is created and automatically saved in the PEAR channel. Of course, there’s no compulsion to use PEAR for package management though it has served us as a very efficient solution over the last few years. Composer is now being tested as an alternative.

To make sure every extension is executable both as a standalone and interacting with all other extensions in the framework of a project, a separate job is created in the CI server for each extension and project. If the developer pushes a change into the extension’s Git repository, the CI server automatically starts the build process, executes the extension-specific test suite, and – assuming it is successful – builds a new PEAR package with the latest version and places it on the channel server. The newest version of all extensions is installed and all tests – both core and extension – run before a project is released. The new release is not enabled for deployment until the project has passed all of the tests.

The Scrum Master and Product Owner are able to keep an eye on the various metrics, as well as the number of unit and integration tests created, via the Jenkins front end during the sprint (Illustration 3). Following the sprint, the tests ensure that the required quality is maintained simply and effectively even if the extension or the shop is developed further.

Creation of test data

PHPUnit offers running a test with different data structures using so-called data providers as a standard option. A method that returns an array or an iterator with the data to be tested can be declared for this purpose by means of the @dataProvider annotation. In this case, the signature of the method to be tested must contain the corresponding parameters depending on the data structure, as shown in Listing 4.

However, test data that can’t be presented easily using a CSV file or an array is almost always necessary, especially to run the integration tests. As mentioned earlier, the test suite gets around this problem by providing functions that enable the data per test to be created conveniently, then automatically deleted again afterwards, through additional annotations on the method level

• @magentoDataFixture
• @magentoConfigFixture

These ensure that the database is reset to its original state before each test starts. To make this as efficient as possible, the data that is specified by means of the annotations is created in the framework of a transaction; the test is subsequently run and the transaction reset with a rollback. In other words, the database is not modified at any time because the transaction is never actually committed.

As shown in Listing 5, a relative path must be specified to the file for the @magentoDataFixture annotation, starting from the root directory of the test suite concerned, e.g.

Listing 6 shows how the developer can create test data within these scripts similar to the implementation within a model.

Problems encountered with the test suite

Numerous problems that would never normally be noticed were revealed when the test suite was used in the daily development process. It became clear during the rewrites that the suite’s developers had not yet scrutinised every aspect of testing, at least not by the time of porting. I’d now like to discuss what I consider to be the most serious problems linked to the porting process.

Separation of installation and data scripts

For instance, in many cases the installation and data scripts are not separately saved in accordance with Magento’s own instructions, namely installation scripts in the sql directory and data scripts in the data directory. If they are not kept separate in this way, errors occur when the test database is created by the test suite because in the first step Magento only creates the tables but does not import any data. If this separation rule is ignored when an extension in which the data must be linked to the tax table is installed, constraint problems are inevitable because no tax data exists at this point in time precisely because of this separation.

Simply separating installation from data scripts goes some distance toward avoiding problems of this kind but is not enough on its own. It is equally imperative to save dependencies to other extensions, as well as to core modules, in the relevant configuration file under app/etc/modules. These dependencies determine the order in which the installation and data scripts are run when the test database is installed by the test suite. Without them, there is no way of telling whether a required table and its data really do exist before the installation or data script is run.

Localisations

Another problem arises when projects are localised. As shown in Listing 7, the test suite created by Magento for Magento 2 assumes that the test database is localised for the US. Several of the tests involve checks for fixed values, such as prices in USD.

Since extensions such as TechDivision_GermanTax modify the database localisation accordingly by means of the installation and data scripts, these tests are bound to fail. Listing 8 shows how to solve this problem by first localising the fixed values based on the locale configured in the shop concerned and then running the test.

Integration of third-party extensions

In addition to the dependencies of extensions, the implementation revealed many other situations in which the Magento core tests fail after integrating third-party extensions, because by overwriting the core classes they change the original functionality. There are basically three possible ways to solve this problem.

The first and best alternative would of course be to nip the problem in the bud by completely dispensing with rewrites.

The second option, which should be preferred whenever a rewrite is unavoidable, allows the developer to configure an extension so that it has to be enabled in the Control Panel. This stops the core tests from failing because the extension must first be enabled by means of the @magentoConfigFixture annotation.

The third and last method lets developers substitute the additional @magentoRewriteTestMethod annotation for one core test. The DocBlock method shown in Listing 9, for instance, is substituted for the core test called MageTest::testGetModel().

Dependencies of extensions

Localisations and third-party extensions should on no account be underestimated; at the same time, however, the problems associated with them are relatively easy to work around. The dependencies of extensions represent a far more complicated challenge when the test suite is executed.

Dependencies to the Magento core only play a minor role, of course (see Separation of installation and data scripts). On the other hand, if an extension has one or more dependencies to other or third-party extensions, these dependencies must likewise exist when the test suite is called. Although Magento incorporates a mechanism for telling an extension that it is dependent on one or more other extensions, dependent extensions are not automatically installed together with the main extension. Here, the solution is determined by the developer’s build and package management tools, i.e. ANT and PEAR in our case. The dependencies defined for each PEAR package guarantee the availability of all other required packages whenever an extension is installed.

Each time one of the ANT targets (see Step 3: Integration into the development process) is called in order to run the test suite, a new Magento instance is automatically created (if one does not already exist) and a separate PEAR channel initialised for every instance. A PEAR package is then created locally based on the latest version of the sources and installed using the above-mentioned PEAR channel. If other packages are also needed, they are installed at the same time with the help of the dependencies defined in the PEAR package. A consistent test environment which ensures that all sources necessary to run the tests exist in the test instance is thus built prior to calling the test suite.

You may be asking why we also integrate third-party extensions in our build and deployment process in the framework of projects. The answer to that question is that third-party extensions are often resorted to in a project for reasons of efficiency and cost, although their functionality nearly always has to be extended. In the majority of cases another, dependent extension is implemented too to prevent problems from occurring when the main extension is updated. As described above, the third-party extension must be installed as well in order to use the test suite, yet this is only possible if the extension is simultaneously integrated in the developer’s own build and deployment process.

Conclusion

In spite of the still relatively low test coverage, the test suite’s implementation has already helped generate a significant improvement in quality. A whole series of problems that are normally concealed were identified and eliminated during the migration. As a result of implementing and working with it daily, combined with the decision to switch to SCRUM, the developers were forced virtually overnight to adopt TDD as a development approach. Although this meant extra work in the first step in the form of inception training, workshops and writing tests, it was very soon evident that quality and hence customer satisfaction rise proportionally to the additional effort.

The latest version of the test suite can be downloaded from TechDivision GmbH’s Github repository and used free of charge in your own projects. Since a large number of tests are still disabled for compatibility reasons, we are naturally grateful for any support from the community as we endeavor to port all meaningful tests from Magento 2 and achieve the maximum possible test coverage.

Tim Wagner is head of design and development at TechDivision GmbH, which has been a Magento Gold Partner since its inception