STLC - Test Execution

What is Test Execution in Software Testing?

After test planning, a framework is created to run automated test cases. Execution of both manual test cases and automated test cases, generation of reports, and identification of bugs is called test execution. This is the main step, as here the test cases run. If any bug is there, then after the bugs get fixed, test cases are executed again for quality assurance. An important aspect of test execution is documentation also. The document has a definite structure that has been followed in the industry for many years.

Test Execution Guidelines

The following are important aspects of test execution that need to be understood before running test cases:

1. Build: Build is standalone software. A source code is converted into a build that is deployed on the test environment, and the build is what is tested for bugs and quality assurance.
2. Test Environment: There is a dedicated server for testing purposes. After the application is tested, it gets deployed to another server for production usage. Nowadays, Saucelabs and Browserstack provide these test environments as a service model. I could use macOS, chrome version 104 as a service instead of having the hardware available.
3. Test Team Size: Test team size is dynamic. The initial project is on test lead, then QA and senior QA are assigned tasks by the lead. The team size keeps changing based on company projects and deadlines.
4. Test Cycles: The testing process also happens in a cycle. During the initial phase, the product is tested to ensure that functionalities are working and no other bugs are left. In the next cycle, the system is retested for exploratory testing and finding deeper bugs.
5. Test Execution Phase: Test execution consists of test scripts, maintenance, and bug reporting. After the test, the environment is set up, and the build is deployed.
6. Smoke and Sanity test cases: When carried out on the build, the testing process comes under smoke and sanity testing. Sanity testing is done to ensure that functionalities are working in the build, and smoke ensures that all the bugs have been fixed in the build deployed.
7. Defect Reporting: The bug life cycle is an important aspect of test execution. Bugs found are reported and maintained in a separate document. Nowadays, we use Bugzilla and JIRA for bug report management. Defect reporting is talked about in depth below.

Test Execution States

Some important test execution states/results have a definite meaning to them.

• Pass: test case is executed, and the expected result is the same.
• Fail: test case is executed, and the expected result is not the same.
• Inconclusive: test case is executed, and there is no clear result.
• Block: test case cannot be executed because one of the test case preconditions is not met.
• Deferred: test case is not executed yet and will run in the future.
• In progress: test case is currently running.
• Not run: test case has not been executed yet.

Bug/Defect Life Cycle

Once a bug is reported, it goes through various stages of assignment and reporting, and then only ultimately, it is fixed, termed as a bug life cycle. The diagram below explains the various stages, but first, let's discuss these bugs' terminologies of status.

Defect status

A bug in its cycle goes through various states, as below.

• New: Every time a new bug is discovered, its status is new.
• Assigned: When a bug gets assigned to the developer, its status changes to assigned.
• Open: When the developer is working on the bug, and it has not been fixed yet, its status is changed to open.
• Fixed: When the developer completes the development changes and is resolved from the developer's end.
• Pending retest: After the bug is fixed, it is sent to the testing team to close it. During this transition, its status is pending a retest.
• Retest: During the retesting phase, its status is changed to retest, as the tester tests the code again to verify if the issue has been fixed or not.
• Verified: Once the bug has been fixed, the testing team validates that the bug status is changed to verified.
• Reopen: In case of retesting, the testing team is not satisfied, and the status of the bug is changed to reopen.
• Closed: A bug can only be closed by the one who raised it, so in case if client raised the bug and is satisfied, the status is changed to closed.
• Duplicate: Suppose the same bug is affecting two functionality, and the testing team has written two bugs, so one of them would become a duplicate.
• Rejected: If the development team feels that the bug raised by testing is not a defect, then the bug status is rejected.
• Deferred: Deferred means that the development team is aware of the bugs but will fix them in the next product release cycle.
• Not a bug: If some features do not affect the core functionality, then they are termed as not a bug.

Defect Cycle Explained

1. During the test execution phase of the testing cycle, if a bug is discovered, the status of the bug is assigned as New
2. Then, the developer assigns the bug to make changes in the code and fix the defect. The status here is Assigned.
3. Developer would make changes in the code, and when the developer validates it's been fixed from his end, the status will change to fix.
4. The fixed bug would go to the testing team for retesting and would be validated if it is fixed or not. The status of the bug here is retesting.
5. If the testing team is satisfied with the changes made by the developer, the bug will be verified.
6. But, if the testing team still finds some issues left in the code, then that bug would reopen to go through the entire bug cycle again.
7. Now, the bug can only be closed by the person who has opened it. Suppose a client has opened the bug, then after the client validates his satisfaction, the bug would be closed.
8. During the cycle, if the development team denies that the bug mentioned is not a bug but a definite feature, then that bug is rejected by the development team.

Defect Severity and Priority

• Defect Severity signifies the impact of the bug on the application. It's important to categorize the bug's impact so that the development team knows how severe the issue in their code is. Severity is High, Medium, and Low.
• Defect priority is a ranking mechanism to tell a developer that if multiple bugs have been allocated to him, which one should deal with first? Priorities are mentioned in the order: Critical, Major, Moderate, and Minor.