Magento Magento Functional Testing Framework

Good day, today we look at testing with  Magento Functional Testing Framework (MFTF), a powerful native tool that can help you facilitate testing.

So what is this framework and what does it provide?

What is Magento 2 MFTF?

MFTF is a set of ready-made tests that check if a Magento 2 system functionality runs as expected. In case the default functionality was expanded, you need to cover the new features with autotests, and notably, even manual testers can do it. The thing is, MFTF only uses XML for writing test cases, no PHP involved. PHP phase comes later when XML test cases are auto-converted into PHP scripts and tested in Codeception, Selenium or Allure. These are the tools MFTF employs.

This autoconversion spares QA specialists the trouble of scripting and enables them to write test cases on the fly. Besides, the framework provides a set of functions, and the existing tests can be used as a viable template for writing new tests. However, to use the whole range of functions MFTF offers, you should configure it correctly.

Planning to work with MFTF or just started testing with it? Here at Amasty, we have decided to share some useful tips from our  MFTF experience. They will help you avoid stumbling blocks and make a fast and efficient start.

So let’s begin.

Prep steps

Though the framework is relatively easy to use, you should tread carefully from the very start. Remember that small mistakes and even typos you make at the beginning may cost you dear later.

So here are the 3 preparatory steps for seamless testing:

Step #1

Testing a Magento 2.3 system, you should store the test cases in the module folder, directory Test/Mftf.

Step #2

Apart from the directory that stores your test cases, you should set up such subdirectories as ActionGroup, Data, Page, Section and other in the MFTF directory. These will store the data structures necessary to run the tests.

Step #3

You should set noNamespaceSchemaLocation correctly. For Magento 2.3 test cases, it should start with urn:magento:mftf. Otherwise your test won’t start. Please, see the examples of incorrect and correct implementations below.

Simple prep steps complete, you are ready to proceed with writing test cases. And there comes another surprise: annotation section. In MFTF test annotations are of special importance.

Mind annotation section

Annotations explain each test case (or a group of test cases) in MFTF. Annotations should be easily understandable, as you will have to submit test reports to other team members,  and they should see what a test case does exactly. So here are best practices for writing annotations. Stories and Title should be unique to help you and other team members understand what the test case checks. For the same reason, TestCaseId should also be unique.

• Make sure the Value field in a Group annotation is meaningful. Value permits to run a separate Group of tests independently. Relying on our testing practice, we recommend that each test case covers the following Groups. (Note that one test can employ several Groups.):
  • VendorName Group for all your tests.
  • Module Group that corresponds to the name of the module folder and unites all tests belonging to this module. For example, the group of Order Attributes module is called Orderattr.
  • Magento functionality Group (optional). This group includes test cases that cover native Magento features, such as Checkout.
  • You can also single out separate logical groups of tests within some module or new custom features. This was the case with testing Improved Layered Navigation module when we introduced several groups of test cases: for filters (FiltersILN), for brands (ShopByBrandILN), for SEO (SeoILN), and more.

Here are the bad and the good example of the Value field.

A reasonably descriptive annotation will help the whole project team to locate the detected issues swiftly. But good annotations are not the only aspect ensuring a smooth start.

Testing with MFTF: cleansing

It takes ages to find the necessary object in an old cluttered house, and the same holds true for testing, both manual and automated. A tidy testing environment is one of the keys to fast and efficient testing. So how to ensure it in MFTF?

First of all, upon completion, a test case should clean up the data that may hamper repeating it or running other tests.

Example:  A test sets up a field in ?heckout, and in case this field is empty, you can’t place an order. Logically, this will affect the tests related to order placement functionality.

Remember that “cleaning up” is not always about deleting the data. You can simply render a field necessary in the first test optional in the second test. To do so, go to Magento or module settings.

The data a test creates should be unique. This applies to IDs, URLs, keys, names, etc.

Summing up

In this post, we’ve decided to share some ideas on making the start with MFTF painless and swift. So here’s a short sum-up based on our practice:

• Prep steps: Do invest some time in the correct naming and storing of your test cases.

• Informative annotation section: Make annotations to your test cases reasonably detailed. This will ensure that the whole team understand what a test case is about.

• Tidy test environment: Clotting your test environment with unnecessary data will only make your test cases stumble. So clean it up timely.

This neat checklist helps us set off to testing quickly.

Maybe you have something to add? Please, leave your comments.

For more specific questions arising in the process of testing, you can consult a comprehensive MFTF guide.

magento magento2-functional-testing-framework