Abstract

QIMAone team is working with agile scrum methodology to develop a SaaS solution that aims to ease the Quality Control inspections and audits within the supply chain market. QIMAone solution comes with a lot of technical and functional challenges to be taken into consideration.

QA team mission is to ensure the quality of the delivery of each release.

To succeed in our mission, we are often invited to take a step back and have a wider view on our core activities as a QA team. Therefore we think about strategy, best practices, continuous improvement and industrialization. This allows us to define what we do, where we are, where we want to go and how.

Among responsibilities of a Quality Assurance Automation Eng. (TAE) we can list :

• To design.
• To develop.
• To implement.
• To maintain a test automation solution (TAS).

In the meantime, and within each solution, some tasks are repeated, similar questions need to be answered and identical issues need to be treated and prioritized. These reoccurring concepts, steps and approaches in automating testing are generating the core need of establishing a Test Automation Architecture (TAA).

In this article we are going to present the Test Automation Architecture of the Test Automation Solution provided by the QIMAone QA team. It will help us having an overview of the actual state of play so we can evaluate it and identify potential improvements. In addition, it will help you getting inspired if you are working on a solution with a similar dimension.

Introduction

a. The team

QIMAone is developed today by 3 agile teams. Each team have its own specific targets and is responsible for a specific functional scope of the application. In parallel they are all working to build the same solution so they also collaborate together to achieve some common goals. The composition of each team is as follows  :

• Product Manager
• 2 or 3 Front-end developer
• 2 or 3 Back-end developer
• 1 UX-UI
• 1 QA

In addition, a TechOps team and a Data team are operating closely with the 3 agile teams.

The developpers were the ones who implemented the first cypress automated tests before having a QA team back in 2019. In 2020, the QA team started to be hired and the initiated automation solution was fully handed back to the QA team.

b. Method / process :

As mentioned earlier, the teams are working together in Agile Scrum methodology and are splitting their milestones into 2 weeks sprints. In each sprint, 2 production releases are delivered.

A release is deployed in production every Wednesday. Thus, testing campaigns are also scheduled to be done each week on different environments before the production deployment date.

Check this article to understand the detailed QIMAone release process

c. Environments :

Let's go into the stages of the release process. This will help us later understanding how automation is integrated in the DevOps environment, therefore apprehend the TAA.

During its journey towards the production, the developed features of each team are handled by the developers at first on their local environnements. Then they are deployed for the first time on a sandbox environnement automatically created by a workflow of the CICD upon a pull request creation.  At this stage we have a first cloud accessible environment containing the newly developed feature.

After the approval of the pull request, it is merged and closed. Once the PR is closed, the corresponding sandbox environment is automatically deleted and the feature goes to another CICD workflow that will deploy it on the integration environment. On this environment we have all merged features available.

Every Thursday, a release is created (to be deployed next Wednesday in production environment). This release includes all the features that were merged into integration environment during the week. Once the release is successfully created, it is deployed on two environments, the demo environment and the QA environement.
The demo environment is reserved to the demonstrations to the clients while the QA environment is reserved to the full non regression automated testing.

Once the non regression automated tests are validated on the QA environment, the deployment of the release on the preproduction environment is approved.

After the preproduction, comes the production deployment.

Now that we discovered the release process and the different environments, let's see within all that which and when testing activities are involved. Hence, understanding the TAA will be obvious.

Test activities and Test Automation Solution :

Test activities in QIMAone starts at the early beggining of the process. Just after the definition of the requirements, the test design is started before the development itself, the other test activities continues in parallel with the release process until the deployment on production.

a.  Test Creation

Starting by the test generation, we design test cases manually. We do it frequently through 3 amigos meetings to make sure we are covering the wider perimeter of the feature within our scenarios. Therefore, they are written in gherkin. Sometimes we use the format of steps/actions/expected results.

Once the tests are created, they are manually executed on sandbox and integration environments essentially, then they are executed potentially on other environments.

b.  Test Definition

The generated scenarios are also in most of the time destinated to automation. Every sprint, new features are developed and bugs are fixed. This adds directly to QA backlog new tests to be automated. Hence the TAS is growing day after day and sprint after sprint.

We automate our tests by developing effective, reusable and maintainable test script for automation. To guarantee those three characteristics we are creating internal libraries with different commands and functions that can be reused for different test case scripts. Our targeted locators (elements on which we perform an action on the application using our automated solution) are either stored in json files per page or in classes related a functional scope.

Given the context described above we try to define our tests in a way that can be used in multiple environments. Test data are also defined within the automated scripts, we either use data dedicated to automation or we CRUD data using api for the specific test.

c.  Test Execution

The execution of the automated tests is a whole process. We distinguish 2 types of execution. The Manual execution and the automated execution.

First, the manual execution, which is mainly done on improvements and new features, almost on a daily basis for the purpose of validation of PRs on sandbox and integration environments. Bugs are also tested manually when they are not covered in an automated test. In QIMAone we think the manual testing is mandatory and we are not targeting a full automation strategy.

However , the execution of our automated tests should be architectured. Jenkins pipelines are scheduled to check everyday if there is a new major version deployed on QA environment. If yes, the full regression tests are run, if not only the sanity tests will be run on the integration environment. It means that everyday a sanity test is triggered on the integration environment and every week a full regression test is triggered on the QA environment.

Today we count almost 1600 automated non regression tests. Among which, almost 600 are tagged as Sanity tests. At Qimaone , automated sanity tests are not executed on production, we execute them in sandbox and integration environment before executing the full regression tests.

Once the regressions are executed by the pipelines, the results of the execution are stored in an xray database then displayed on a grafana dashboard.

After each full regression of a new release, the QA members check the results on grafana and might need to rerun some automated tests on their local machine to analyse failed tests, they either troubleshoot and fix the test if the error comes from the test automation solution or they create bug if the fail is because of a new regression bug detected.

Since the automated scripts are coded in Javascript and the automation tool used is Cypress, when we run automated tests on local machines we use cypress runner. Sometimes we use postman to run apis locally.

d.  Test Adaptation

Some extra tools are used to ease the automation, we mentioned before the cypress runner, we use also the postman runner and newman for api report generation. Mailosaur is used as our testing email service for the accounts we create for and in our tests. We might get help from logrocket to analyse production behaviors on a specific scenario. Some plugins also are used within cypress like mailosaur for emailing service management,  faker for data generation and file uploader to upload data etc.

Conclusion

We mentioned in this article an overview of the activities of the QA team. It helped underlining our TAA and understanding it within the context. We discussed the test generation layer, the test definition layer, the test execution layer and the test adaptation layer.

Put all together we can establish the following TAA :

This architecture made clear for us our process to generate, define and execute automated tests. It allowed us to spot easily the things that we might challenge.

For instance, we noticed for the test generation that we still have potential improvements to do, for example get more benefits from logRocket to generate automatically test scenarios similar to production. We can also work closely to have a test definition layer more solid by defining a test pattern to be followed as a best practice by the automation team.

As a matter of fact, we are working on some next steps to improve our architecture. For example, replacing Jenkins by ArgoCD as it is the CICD tool used for the software under test, so we want to align out test automation solution with it.

Thank you !

Written by Mohamed HAMZA, QA Automation Eng. at QIMA