Let’s see the typical process involved in delivering software from ‘development phase’ to the ‘testing phase’ for a successful bug-free software release to production/client.
These processes are either overlooked or skipped by software companies, which results in poor test management and thereby’ a “buggy” software releases to the client, which leads to “unsatisfied customers”.
Even though lot of time and great effort being given by the testing team for each and every release, when the released software is not having the quality as defined or bench-marked or not meeting the expected criteria, it will not only affects the company reputation with the customers but also demotivates and demoralizes the project team most importantly the testing team as a whole.
If you are part of a testing team in this scenario, you might keep thinking yourself “how to improve my testing capabilities and is there any better way of overcoming this situation”.
I want to give some tips and suggestions, based on my experience with various testing teams involved in software applications and enterprise products releases with multiple domains and platforms and with multiple testing frameworks, on how to improve the test release processes, which will simplify your professional life as a test engineer or a test manager for delivering world-class software.
What You Will Learn:
Test Release Process
The table below gives an overview of a test release process with three universal phases like Input, Process & Output.
|Process Starts With|
• Installation of Released Software in the Testing Server
|Next Process Needs
• Software which passed Smoke / Sanity Testing
|Information / Document Reference|
• User Requirements Document
• Software Requirements Specifications
• Unit Testing Plan
• Coding Standards
• Code Review Checklist
• Development Plan
• Quality Assurance Plan
• Task Allocation
• Work Package
• Project Schedule
• Project Plan
• Configuration Management Plan
• Risk Management Plan.
• Preparing Test Cases for all the Units
• Development & Unit Testing
• Handling of Non Conformance Procedures
• Implementing Configuration Management Plan.
• Implementing Risk Management Plan
• Project Progress Monitoring
• Error Correction & Reviews
|Internal Customer Needs
• Software Build with Version Number
• Release Report
• Test Cases / Test Suite Document
• Test Execution Planning
• Traceability Matrix
• Test Data
|Incoming Input Verification|
• Project Documents are Reviewed and Approved?
• Coding Standards, Code Review Checklist are available for Reference?
• Task Allocated and Work Package Updated?
• Functional Specification, Development Plan & Quality Plan are Reviewed and Approved?
• Risk Management Plan has Mitigation and Contingency to handle Risk?
• Effectiveness of Project Schedule to Deliver the Product On Time?
• Unit Test Cases should consist of all the Entry and Exit Criteria
• Adherence of Code with Coding Standards
• NCP should be handled as per the Guideline
• Configuration Management steps should adhere to the Configuration Management Plan
• Risk handling should adhere to the Risk Management Plan
• Smoke Testing pass all the major Features and Functionalities
|External Customer Needs
• Bug Free Software
• Human/Hardware/Software/ Resource Allocation
• Hardware Breakdown Maintenance
• Training to Team Members
|Process Ends With|
• Execution of Smoke / Sanity Testing on the Released Build
• Each Unit should pass the first round of Testing
• Tasks to be completed as per the Project Schedule
• Smoke Test should be passed before the Release
• Testing Team Passion to Test the Software
Every testing team should create a unique checklist for software release, as per the domain and platform of the software and the project management methodology (like Agile Scrum etc.,) and as per manual/automated testing framework, to accept the released build, before starting the test execution to save the time and effort.
This is one of the most important efficiency parameters in the test release phase.
Test Release Process Improvement:
1) Review the Release Report for the new functionality, customization/modification of existing functionality, bug fixes from the previous build, which will decide to start executing the Smoke Testing or Sanity Testing or a combination of both.
2) Review the Update Testing Documents as per the new functionality and the bug fixes, if not updated already. Normally, during the software development life cycle, these documents get updated by the testing team based on the regular weekly project review meetings.
3) Review the Software Build in Configuration Repository is updated for the build number, version number, labelled or commented with the release name as per the standards defined in the project plan. Also, ensure the build is successfully compiled and installed on the testing server.
4) Schedule a Quick Project Review Meeting after Release to discuss the pros and cons of the released build, known bugs and critical functionality etc., to avoid any miscommunication and to review any important client requirements. Strictly avoid any oral communication between the development and testing teams as it highly impacts the quality of the software release.
5) Ensure the Bug Tracking Tool is Configured Properly, for the allocated testing team and development team of the project, software build and release numbers as well as the modules/functionality of the software, which will help to log the bugs efficiently. If not, should be escalated to the project manager or test manager on a high priority basis.
6) Return the Build to the Development Team without any compromises, if the build fails in Smoke or Sanity Testing. Strictly, testing should not be continued when the system fails in Smoke Testing. This will save a lot of time and effort and improve the quality of the software released in the subsequent releases.
7) Schedule the Project Release on the 1st Day of the Week which will help the test manager to plan the upcoming test cycle based on the build stability and also to send a quick test report to the project manager which will escalate the quality of the software well in advance. If the development team schedules the project release on Friday, the weekend can be utilized for any slippages as well as any build issues in a manual or automated build framework.
8) Ensure the Testers are Trained Properly on the Domain which will help the testing team to adhere the testing schedule and gather time for the next round of testing. Also, the testing team should be trained and have the exposure to the required technology like Scripting and SQL if the project demands white boxing.
9) Avoid the Allocation of Testers in Multiple Projects as it greatly affects the quality of test execution in real time. In practice, even the experienced testers overlook the features and functionality as well as skip the test cases assuming that some test cases never fails, when overloaded with work or allocated on multiple projects with deadlines.
10) Appreciate the Testing Team to have Passion as testers should not work for the “Day” or comment “Call it a Day”. When software is having multiple modules and the functionalist is completely or partly integrated or interrelated, testers should have the passion for writing/executing the test cases with great coverage and traceability matrix, targeting the quality of the final software/product. Because even a cosmetic issue is a “bug” and counted as “1 bug”.
11) Ensure the Software Installation is Easy and Straightforward as it helps the testing team to re-install the software when required instead of waiting for the development manager or an installation manager to do the same job, which will kill the available testing time unnecessarily. For example, even though windows based installation is easy, but when it involves multiple web servers and wide area networks in a multi-tier testing environment, testers may take hours to install the software. If the software testing covers and installation, uninstallation, patches or updates of software, it is more likely to be discussed in detail the process of executing the test cases with the testing team.
12) Ensure the Automated Tools Are Available with License for an automation testing framework. The execution of test cases in an automated framework is easy compared to a manual testing scenario, provided if the automated tools are properly configured and licensed for multiple users. Especially, when the testing plan involves performance and load testing apart from the regular test case execution and regression testing, testers should cover executing test cases on multiple environments like multiple servers, multiple browsers, multiple users etc.
13) Ensure the Ghosted Machines are Setup for Testing before started test execution. Ghosted machines are machines with the different testing environment. For example, a web application software may be planned to test in multiple environments like Windows 7 & Access DB or Windows 2008 & SQL Server or Windows 8 & Oracle or Mainframe & DB2 etc., with all the Browsers like Chrome, Firefox, Internet Explorer, Safari etc., Some “System Testing” even requires to completely format the hard disk and install a fresh software or update the existing software with patches and updates etc.
14) Avoid Implementing the New Features/Change Request by stopping the test execution and re-release the software for stating the testing phase again. This is a very bad practice in many software organizations as per the business requirements to satisfy the external customers or at least to meet the demands of the management steering committee or sometimes the sales/marketing teams. Even though the change requests from the customers are always encouraged in an ‘Agile’ project environment, it should be properly planned and implemented before the release of the software to the testing team.
Managing and controlling the test release content
Managing and controlling the test release content is most important for any IT software or even for any non-IT software environment which will be depicted in the figure below.
- The project managers and/or the project steering committee depends on the authority of organization matrix, is responsible for selecting the content for each and every release.
- Once the bugs and/or the new features and the change request from the customers are identified and approved, it will be implemented by the development team which should be intimated to the project stakeholders before the development/implementation is started.
- Based on the implemented final release, the testing team will update the related documents and prepare for the testing accordingly.
- Testing team will start Smoke / Sanity testing as per the defined requirements in the release report.
- Once the Sanity passed, testing team will start the test execution as per the schedule and allocated tasks viz., Functional Testing, Non-Functional Testing, Security Testing, System Testing, Performance Testing, Load Testing, User Acceptance Testing etc.
- Once the first round of testing cycle is completed, test reports will be sent to all the stakeholders and the development team manager to plan for the next iteration of test execution.
- Depends on the status of the test reports and bug severity and complexity, a complete cycle of a second round of test execution or regression testing will be planned together with the user acceptance testing.
- After completing the planned test execution cycles, the test reports will be sent to all the stakeholders of the project for the passed/failed/missed of the features, functionality and bug fixes.
Sample Release Report Template:
Note: Sample MS Word template for Release report is also available for download below.
Find below a “Sample Release Report” which covers the main aspects of the release process which makes the whole project team’s professional life much happier than ever before.
GPS Navigation for XYZ Company Limited is being released for internal testing. The released version is 1.0.7, the release number is 14.0 and build number 105.25.03. This software release includes the new features and the major bug fixes from the previous release. Smoke testing is passed from the development phase, but a Smoke & Sanity is required before going for Regression Testing.
GPSNavigation_URD_1.0.12, GPSNavigation_FFD_2.17, GPSNavigation_BusinessUseCases_1.23.10, GPSNavigation_TestPlan_1.44, GPSNavigation_TestSuites_2.10, GPSNavigation_UnitTesting_23.3
#3) Release Description
This release is a controlled release of GPS Navigation and contains following features and functionality.
The features marked in * are new in this release.
The following features are not been implemented in this release.
1. Module 1
1.1 Feature 1
1.1.1 Functionality 1
#4) Configuration Management
We are using Visual Source Safe as configuration management tool. The build is available in the following path.
Internal Link: http://184.108.40.206/internalbuild/gpsnavigation/release1.0.13
External Link: https:// 220.127.116.11/externalbuild/gpsnavigation/release1.0.13
#5) Installation Instructions and Steps
Give the detailed information for the installation of the build to QA/Testing team.
#6) Issues / Bugs Fixed
The status of the bugs is updated in defect tracking system.
#7) Issues / Bugs To Be Fixed
#9) Known Bugs / Issues
#10) Release Check List
|Sl.No/||Description||Y / N|
|1||Have all the files been checked in Visual Source Safe?|
|2||Has the label been put on the proper folder in VSS as per internal standards?|
|3||Is the release identifiable as 'external' / 'internal' release in VSS?|
|4||In the comments, has the version been mentioned in VSS?|
|5||In the comments, has a short description been mentioned in VSS?|
|6||Code has been reviewed and code review issues are logged in Clear Quest?|
|7||Code review checklist has been updated and available in VSS?|
|8||Unit test document has been prepared and reviewed?|
|9||Unit test cases executed and the results updated for the status?|
|10||Updated unit test case document is available in VSS?|
|11||All the Clear Quest issues for this released has been resolved/closed?|
|12||All the work package tasks completed and updated in VSS?|
|13||Is Smoke testing done and passed?|
=> Download: Click here to Download Sample Release Report Template in MS Word format.
How to Improve Test Release Process Continuously
Tip #1) Setup a release engineering team which will take care of the critical factors of maintaining the software releases and builds and responsible for the centralized software configuration management systems.
Tip #2) Motivate and appreciate the project teams for following the process involved in Software Development Life Cycle, Product Development Life Cycle and Software Testing Life Cycle. We can define the process but until and unless it is been followed by people involved, there is no use of defining the process.
Tip #3) Estimate the testing effort based on the experiences and previous history. Writing test cases is totally different from executing the same. The testers should understand what to test, how to test and when to test, otherwise, the efforts given to the testing cycle gets wasted, even though multiple rounds of test cycle happened.
Tip #4) Finally, if possible and feasible, automate the testing phase using some universally accepted testing tools. Use of automated build tools and automated testing tools reduce the testing efforts by more than 50% improving the quality of software and ensures 100% quality if the automation framework is properly designed.
Tip #5) Last but not the least, test release is not just a job, it’s an art of making all the stakeholders life in a project easy and more comfortable.
About the author: Balu A. is an experienced techno-functional IT professional with over two decades of IT software experience and a decade of project and test management experience delivering enterprise applications and mobility solutions across domains using Microsoft, Oracle, Java and Mobile technologies. He is a basically a leader with a passion to promote people to become the leaders with the right attitude and loves to work in a process-oriented environment and believes process improves employee efficiency, quality and productivity.
In next tutorial, we will learn – How to Improve Test Case Efficiency.
Let us know your thoughts/queries in comments below.
Have a Software Release as per the Process!
Complete Testing On Schedule with Great Productivity and Efforts!!
Try to Achieve Bug-Free, Quality Assured Software Delivery!!!
If you like this article consider sharing it with your friends!