Entries Tagged 'Testing Life cycle' ↓

This is a guest article by Author “N. Sandhya Rani”.

For success of any project test estimation and proper execution is equally important as the development cycle. Sticking to the estimation is very important to build good reputation with the client.

Experience play major role in estimating “software testing efforts”. Working on varied projects helps to prepare an accurate estimation for the testing cycle. Obviously one cannot just blindly put some number of days for any testing task. Test estimation should be realistic and accurate.

Continue reading →

Do you know “Most of the bugs in software are due to incomplete or inaccurate functional requirements?”  The software code, doesn’t matter how well it’s written, can’t do anything if there are ambiguities in requirements.

It’s better to catch the requirement ambiguities and fix them in early development life cycle. Cost of fixing the bug after completion of development or product release is too high.  So it’s important to have requirement analysis and catch these incorrect requirements before design specifications and project implementation phases of SDLC.

How to measure functional software requirement specification (SRS) documents?
Well, we need to define some standard tests to measure the requirements. Once each requirement is passed through these tests you can evaluate and freeze the functional requirements.

Let’s take an example. You are working on a web based application. Requirement is as follows:
“Web application should be able to serve the user queries as early as possible”

How will you freeze the requirement in this case?
What will be your requirement satisfaction criteria? To get the answer, ask this question to stakeholders: How much response time is ok for you?
If they say, we will accept the response if it’s within 2 seconds, then this is your requirement measure. Freeze this requirement and carry the same procedure for next requirement.

We just learned how to measure the requirements and freeze those in design, implementation and testing phases.

Now let’s take other example. I was working on a web based project. Client (stakeholders) specified the project requirements for initial phase of the project development. My manager circulated all the requirements in the team for review. When we started discussion on these requirements, we were just shocked! Everyone was having his or her own conception about the requirements. We found lot of ambiguities in the ‘terms’ specified in requirement documents, which later on sent to client for review/clarification.

Client used many ambiguous terms, which were having many different meanings, making it difficult to analyze the exact meaning. The next version of the requirement doc from client was clear enough to freeze for design phase.

From this example we learned “Requirements should be clear and consistent”

Next criteria for testing the requirements specification is “Discover missing requirements”

Many times project designers don’t get clear idea about specific modules and they simply assume some requirements while design phase. Any requirement should not be based on assumptions. Requirements should be complete, covering each and every aspect of the system under development.

Specifications should state both type of requirements i.e. what system should do and what should not.

Generally I use my own method to uncover the unspecified requirements. When I read the software requirements specification document (SRS), I note down my own understanding of the requirements that are specified, plus other requirements SRS document should supposed to cover. This helps me to ask the questions about unspecified requirements making it clearer.

For checking the requirements completeness, divide requirements in three sections, ‘Must implement’ requirements, requirements those are not specified but are ‘assumed’ and third type is ‘imagination’ type of requirements. Check if all type of requirements are addressed before software design phase.

Check if the requirements are related to the project goal.
Some times stakeholders have their own expertise, which they expect to come in system under development. They don’t think if that requirement is relevant to project in hand. Make sure to identify such requirements. Try to avoid the irrelevant requirements in first phase of the project development cycle. If not possible ask the questions to stakeholders: why you want to implement this specific requirement? This will describe the particular requirement in detail making it easier for designing the system considering the future scope.

But how to decide the requirements are relevant or not?
Simple answer: Set the project goal and ask this question: If not implementing this requirement will cause any problem achieving our specified goal? If not, then this is irrelevant requirement. Ask the stakeholders if they really want to implement these types of requirements.

In short requirements specification (SRS) doc should address following:
Project functionality (What should be done and what should not)
Software, Hardware interfaces and user interface
System Correctness, Security and performance criteria
Implementation issues (risks) if any

Conclusion:
I have covered all aspects of requirement measurement. To be specific about requirements, I will summarize requirement testing in one sentence:
“Requirements should be clear and specific with no uncertainty, requirements should be measurable in terms of specific values, requirements should be testable having some evaluation criteria for each requirement, and requirements should be complete, without any contradictions”

Testing should start at requirement phase to avoid further requirement related bugs. Communicate more and more with your stakeholder to clarify all the requirements before starting project design and implementation.

Do you have any experience testing software requirements?  

What is BVT?

Build Verification test is a set of tests run on every new build to verify that build is testable before it is released to test team for further testing. These test cases are core functionality test cases that ensure application is stable and can be tested thoroughly. Typically BVT process is automated. If BVT fails that build is again get assigned to developer for fix.

BVT is also called smoke testing or build acceptance testing (BAT)

New Build is checked mainly for two things:

• Build validation
• Build acceptance

Some BVT basics:

• It is a subset of tests that verify main functionalities.
• The BVT’s are typically run on daily builds and if the BVT fails the build is rejected and a new build is released after the fixes are done.
• The advantage of BVT is it saves the efforts of a test team to setup and test a build when major functionality is broken.
• Design BVTs carefully enough to cover basic functionality.
• Typically BVT should not run more than 30 minutes.
• BVT is a type of regression testing, done on each and every new build.

BVT primarily checks for the project integrity and checks whether all the modules are integrated properly or not. Module integration testing is very important when different teams develop project modules. I heard many cases of application failure due to improper module integration. Even in worst cases complete project gets scraped due to failure in module integration.

What is the main task in build release? Obviously file ‘check in’ i.e. to include all the new and modified project files associated with respective builds. BVT was primarily introduced to check initial build health i.e. to check whether – all the new and modified files are included in release, all file formats are correct, every file version and language, flags associated with each file.
These basic checks are worth before build release to test team for testing. You will save time and money by discovering the build flaws at the very beginning using BVT.

Which test cases should be included in BVT?

This is very tricky decision to take before automating the BVT task. Keep in mind that success of BVT depends on which test cases you include in BVT.

Here are some simple tips to include test cases in your BVT automation suite:

• Include only critical test cases in BVT.
• All test cases included in BVT should be stable.
• All the test cases should have known expected result.
• Make sure all included critical functionality test cases are sufficient for application test coverage.

Also do not includes modules in BVT, which are not yet stable. For some under-development features you can’t predict expected behavior as these modules are unstable and you might know some known failures before testing for these incomplete modules. There is no point using such modules or test cases in BVT.

You can make this critical functionality test cases inclusion task simple by communicating with all those involved in project development and testing life cycle. Such process should negotiate BVT test cases, which ultimately ensure BVT success. Set some BVT quality standards and these standards can be met only by analyzing major project features and scenarios.

Example: Test cases to be included in BVT for Text editor application (Some sample tests only):
1) Test case for creating text file.
2) Test cases for writing something into text editor
3) Test case for copy, cut, paste functionality of text editor
4) Test case for opening, saving, deleting text file.

These are some sample test cases, which can be marked as ‘critical’ and for every minor or major changes in application these basic critical test cases should be executed. This task can be easily accomplished by BVT.

BVT automation suits needs to be maintained and modified time-to-time. E.g. include test cases in BVT when there are new stable project modules available.

What happens when BVT suite run:
Say Build verification automation test suite executed after any new build.
1) The result of BVT execution is sent to all the email ID’s associated with that project.
2) The BVT owner (person executing and maintaining the BVT suite) inspects the result of BVT.
3) If BVT fails then BVT owner diagnose the cause of failure.
4) If the failure cause is defect in build, all the relevant information with failure logs is sent to respective developers.
5) Developer on his initial diagnostic replies to team about the failure cause. Whether this is really a bug? And if it’s a bug then what will be his bug-fixing scenario.
6) On bug fix once again BVT test suite is executed and if build passes BVT, the build is passed to test team for further detail functionality, performance and other testes.

This process gets repeated for every new build.

Why BVT or build fails?
BVT breaks sometimes. This doesn’t mean that there is always bug in the build. There are some other reasons to build fail like test case coding error, automation suite error, infrastructure error, hardware failures etc.
You need to troubleshoot the cause for the BVT break and need to take proper action after diagnosis.

Tips for BVT success:
1) Spend considerable time writing BVT test cases scripts.
2) Log as much detailed info as possible to diagnose the BVT pass or fail result. This will help developer team to debug and quickly know the failure cause.
3) Select stable test cases to include in BVT. For new features if new critical test case passes consistently on different configuration then promote this test case in your BVT suite. This will reduce the probability of frequent build failure due to new unstable modules and test cases.
4) Automate BVT process as much as possible. Right from build release process to BVT result – automate everything.
5) Have some penalties for breaking the build Some chocolates or team coffee party from developer who breaks the build will do.

Conclusion:
BVT is nothing but a set of regression test cases that are executed each time for new build. This is also called as smoke test. Build is not assigned to test team unless and until the BVT passes. BVT can be run by developer or tester and BVT result is communicated throughout the team and immediate action is taken to fix the bug if BVT fails. BVT process is typically automated by writing scripts for test cases. Only critical test cases are included in BVT. These test cases should ensure application test coverage. BVT is very effective for daily as well as long term builds. This saves significant time, cost, resources and after all no frustration of test team for incomplete build.

If you have some experience in BVT process then please share it with our readers in comments below.

If you like this post do consider subscribing to our email newsletter to ensure you will not miss any article on software testing! Click here to get article updates by email. 

What is Bug/Defect?

Simple Wikipedia definition of Bug is: “A computer bug is an error, flaw, mistake, failure, or fault in a computer program that prevents it from working correctly or produces an incorrect result. Bugs arise from mistakes and errors, made by people, in either a program’s source code or its design.”

Other definitions can be:
An unwanted and unintended property of a program or piece of hardware, especially one that causes it to malfunction.

or
A fault in a program, which causes the program to perform in an unintended or unanticipated manner.

Lastly the general definition of bug is: “failure to conform to specifications”.

If you want to detect and resolve the defect in early development stage, defect tracking and software development phases should start simultaneously.

We will discuss more on Writing effective bug report in another article. Let’s concentrate here on bug/defect life cycle.

Life cycle of Bug:

1) Log new defect
When tester logs any new bug the mandatory fields are:
Build version, Submit On, Product, Module, Severity, Synopsis and Description to Reproduce

In above list you can add some optional fields if you are using manual Bug submission template:
These Optional Fields are: Customer name, Browser, Operating system, File Attachments or screenshots.

The following fields remain either specified or blank:
If you have authority to add bug Status, Priority and ‘Assigned to’ fields them you can specify these fields. Otherwise Test manager will set status, Bug priority and assign the bug to respective module owner.

Look at the following Bug life cycle:

[Click on the image to view full size] Ref: Bugzilla bug life cycle

The figure is quite complicated but when you consider the significant steps in bug life cycle you will get quick idea of bug life.

On successful logging the bug is reviewed by Development or Test manager. Test manager can set the bug status as Open, can Assign the bug to developer or bug may be deferred until next release.

When bug gets assigned to developer and can start working on it. Developer can set bug status as won’t fix, Couldn’t reproduce, Need more information or ‘Fixed’.

If the bug status set by developer is either ‘Need more info’ or Fixed then QA responds with specific action. If bug is fixed then QA verifies the bug and can set the bug status as verified closed or Reopen.

Bug status description:
These are various stages of bug life cycle. The status caption may vary depending on the bug tracking system you are using.

1) New: When QA files new bug.

2) Deferred: If the bug is not related to current build or can not be fixed in this release or bug is not important to fix immediately then the project manager can set the bug status as deferred.

3) Assigned: ‘Assigned to’ field is set by project lead or manager and assigns bug to developer.

4) Resolved/Fixed: When developer makes necessary code changes and verifies the changes then he/she can make bug status as ‘Fixed’ and the bug is passed to testing team.

5) Could not reproduce: If developer is not able to reproduce the bug by the steps given in bug report by QA then developer can mark the bug as ‘CNR’. QA needs action to check if bug is reproduced and can assign to developer with detailed reproducing steps.

6) Need more information: If developer is not clear about the bug reproduce steps provided by QA to reproduce the bug, then he/she can mark it as “Need more information’. In this case QA needs to add detailed reproducing steps and assign bug back to dev for fix.

7) Reopen: If QA is not satisfy with the fix and if bug is still reproducible even after fix then QA can mark it as ‘Reopen’ so that developer can take appropriate action.

8 ) Closed: If bug is verified by the QA team and if the fix is ok and problem is solved then QA can mark bug as ‘Closed’.

9) Rejected/Invalid: Some times developer or team lead can mark the bug as Rejected or invalid if the system is working according to specifications and bug is just due to some misinterpretation.

Are you going to start on a new project for testing? Don’t forget to check this Testing Checklist in each and every step of your Project life cycle. List is mostly equivalent to Test plan, it will cover all quality assurance and testing standards.

Testing Checklist:
1 Create System and Acceptance Tests [ ]
2 Start Acceptance test Creation [ ]
3 Identify test team [ ]
Continue reading →

Today I got  interesting question from reader, How testing is carried out in company i.e in practical environment? Those who get just out of college and start for searching the jobs have this curiosity,  How would be the actual working environment in the companies?

Here I focus on software Testing actual working process in the companies. As of now I got good experience of software testing career and day to day testing activities.  So I will try to share more practically rather than theoretically.

Whenever we get any new project there is initial project familiarity meeting. In this meeting we basically discuss on who is client? what is project duration and when is delivery? Who is involved in project i.e manager, Tech leads, QA leads, developers, testers etc etc..?

From the SRS (software requirement specification) project plan is developed. The responsibility of testers is to create software test plan from this SRS and project plan. Developers start coding from the design. The project work is devided into different modules and these project modules are distributed among the developers. In meantime testers responsibility is to create test scenario and write test cases according to assigned modules. We try to cover almost all the functional test cases from SRS.  The data can be maintained manually in some excel test case templates or bug tracking tools.

When developers finish individual modules, those modules are assigned to testers.  Smoke testing is performed on these modules and if they fail this test, modules are reassigned to respective developers for fix. For passed modules manual testing is carried out from the written test cases. If any bug is found that get assigned to module developer and  get logged in bug tracking tool. On bug fix tester do bug verification and regression testing of all related modules. If bug passes the verification it is marked as verified and marked as closed. Otherwise above mentioned bug cycle gets repeated.(I will cover bug life cycle in other post)

Different tests are performed on individual modules and integration testing on module integration. These tests includes Compatibility testing i.e testing application on different hardware, OS versions,  software platform, different browsers etc. Load and stress testing is also carried out according to SRS. Finally system testing is performed by creating virtual client environment. On passing all the test cases test report is prepared and decision is taken to release the product!

So this was a brief outline of process of project life cycle.

Here is detail of each step what testing exactly carried out in each software quality and testing life cycle specified by IEEE and ISO standards:

Review of the software requirement specifications

Objectives is set for the Major releases

Target Date planned for the Releases

Detailed Project Plan is build. This includes the decision on Design Specifications

Develop Test Plan based on Design Specifications

Test Plan : This includes Objectives, Methodology adopted while testing, Features to
be tested and not to be tested, risk criteria, testing schedule, multi-
platform support and the resource allocation for testing.

Test Specifications
This document includes technical details ( Software requirements )
required prior to the testing.

Writing of Test Cases
Smoke(BVT) test cases
Sanity Test cases
Regression Test Cases
Negative Test Cases
Extended Test Cases

Development – Modules developed one by one

Installers Binding: Installers are build around the individual product.

Build procedure :
A build includes Installers of the available products – multiple platforms.

Testing
Smoke Test (BVT)  Basic application test to take decision on further testing

Testing of new features
Cross-platform testing
Stress testing and memory leakage testing.

Bug Reporting
Bug report is created

Development – Code freezing
No more new features are added at this point.

Testing
Builds and regression testing.

Decision to release the product
Post-release Scenario for further objectives.