Developing software is hard and testing software is hard too!

Software testing is the process of evaluating the software against its requirements, and finding the mismatch, commonly known as a defect.

In any software development process, the number of issues are directly proportional to the substandard software quality. Therefore the role of Quality Assurance Engineer becomes vital in the process as they need to find the defects before the actual user of the application finds it. Once a defect is found it is shared with the developer and watched until it gets fixed.

Software testing is quite sophisticated and a repeated process that humans are not very good at, but computers are! Hence automation testing has got traction and many projects use that to find out the defects at an early stage.

In software engineering, the diagram below is famously known as Testing Pyramid. It's an industry-standard approach for testing. The emphasis is to capture bugs at the early stage to reduce the cost and time. Here we will understand the significance of the different stages of the testing.

There are basically 3 layers in the pyramid, each has an important role to play. The below table contains the summary of each layer:

We will describe each individual layer with details!

1) Unit:

It is a technique using which every independent module/unit of software is tested and determined so there aren’t any issues from the developer.

As you can refer to the above table, the pyramid emphasises unit tests are written by developers and there is a convention to cover minimum 70% of the code base as it’s quite modular, independent and fast to write and execute. It is white box testing approach as we are testing the code of the actual application.

Ideally, whenever any new feature is being developed, the unit tests should be written for that in parallel, as it will involve less effort. Writing the unit tests for the developed feature takes longer as developers need to go through the application code again to understand it properly.

Objective:

• Independent unit tests are written for each component/unit
• Every function/procedure/class is tested
• It is integrated with CI/CD pipeline to capture issues in the development phase itself.

Tools:

• JS (Karma, Mocha, Jasmine)
• Python (Unittest)
• Java (JTest, JUnit, TestNG)
• C# (NUnit)

Tips:

• Setup husky which will execute all unit tests while committing the code
• Use “test as your code" approach, so there should be unit test for each developed feature
• Try to use test data resemble to production

2) Integration:

It’s a type of testing where all individual components are combined and then tested.In this approach we need to ensure that software is working after integration as it was expected.

Integration of the different individual and independent modules is important, because no matter how efficiently individual modules are working but if the integration of those modules isn’t working well it will affect the overall quality of software.

Normally integration testing is carried out after unit testing, once all individual modules are tested we can combine them one by one and perform integration testing, the main goal here is to test the interface between modules.

So as we climb the pyramid it suggests that integrated tests should contribute around 20-30% of total tests and can be written by developers(ideally) but if QA is familiar with the integration of modules, they can also write the tests. Here we can mock services to fill in coverage where unit tests may be missed.

Depending on the project along with the integration testing sometimes other testing types can be also used such as Component testing and API testing.

Objective:

• Performed after unit testing
• Focus on checking all modules are working as expected after integrating
• API Testing is a popular choice as part of Integration testing
• It could be a Black box or White box depends on the testing

Integration Testing Tools:

• JS (Enzyme, Jest, React Testing Library)
• PHP (Codeception)
• Java (JUnit 5)

API Testing Tools:

• Postman
• SoapUI
• Katalon Studio
• JS (Axios, Sync-request)
• Java (Rest-assured)

Tips:

• CI/CD pipeline should ensure Unit tests and Integration tests are successfully evaluated before deployment
• Make sure to put enough logger statements in an automated integration test to debug the failures
• Create positive and negative tests

3) UI (E2E):

UI testing is also popularly known as E2E testing and it is a Black box mostly, this is the last stage of the pyramid and the most important from a user perspective.

It involves testing the software workflow from beginning to end and here we will try to replicate the real user scenario to ensure the software is working correctly.

In simpler words it is the process of testing the visual elements/components of the software. Here the tester will check the end-user software and try to find the defects.

This is the top level of pyramid and it is the most expensive form of testing as tests are most brittle and take a long time both in test case development and test execution due to which it’s an ideal to create tests for only important end-to-end scenarios.

Mostly, QA is responsible to carry out this testing.

An example of automation tests for e-commerce application would be:

• Register a new user
• Login to the application, add a product to cart, checkout and purchase, and confirm the order

Objective:

• Performed after unit & integration testing
• Create an end-to-end UI test which tests all major workflow
• Each UI test should be independent and can be executed autonomously
• UI framework/tool should support screenshot of failing test and provide nice reporting
• Must be integrated with CI/CD pipeline

Tools:

• Selenium Webdriver (Java, Python, JavaScript/NodeJS, C#, Ruby)
• Selenium Webdriver based frameworks/tools such as Webdriver.io, Nightwatch.JS, Protractor, Katalon Studio, Ranorex
• Tools/Frameworks not based on Selenium Webdriver like Cypress, Puppeteer, and SikuliX

Tips:

• Do not use static waits
• There should be unique id assigned to web elements, Ideally, we should assign custom data-test-ids
• More tests don’t always mean more coverage so write E2E test cases in a way that it checks/covers the most functionality, so that repetition and execution time can be reduced and coverage can be increased
• Avoid repetitive steps on UI automation, leverage API automation whenever needed. For example, most UI test contains login step and login via UI is time consuming, instead make an API call in backend to do pretty fast login

4) Manual Testing:

On the pyramid at the very top level, there is a cloud of Manual Tests, this is a Black box testing type.

In general automation testing can’t replace manual testing, so practically for complex applications, we can not achieve 100% automation test coverage. There could be several use cases that need to be tested by humans.

Humans are not good with repetitive tasks while computers are, so it is advisable to automate those repetitive tasks/tests and create a regression test suite that should be executable autonomously and automatically on a regular interval.

When to use Manual testing?

• Can be used at the beginning of the project
• Whenever there are a lot of new changes to the application and the application needs to be deployed asap
• If there is need to cut the cost and team is more focused on ad-hoc testing

Why to avoid Manual testing?

• Less reliable as it is executed by humans
• It’s a very time-consuming process especially while executing the entire regression test suites multiple times, so for big and complex projects where re-execution of regression test suites happen, manual testing should be avoided
• Can’t be used for Load and Performance testing

Closing remarks:

• This the ideal approach to testing, its usability could be varied according to the application under test
• Some articles on internet suggest the particular ratio of testing pyramid(i.e: 70% Unit tests, 20% Integration tests & 10% UI tests) but there is no such ideal mixing, it depends on the application under test
• The integration testing can be carried out via multiple ways but the API testing is one of the popular forms of testing which can be achieved in a shorter period time
• Ideally, all the automated tests should be integrated with CI/CD pipeline
• Selenium is a popular open-source test automation tool and it has support in many programming languages due to which it is extensively used, but Cypress is an emerging UI automation framework written in JS, but have some limitations like tab switching and strictly adhere to the cross-origin policy