Entries from October 2007 ↓

I was getting too many questions on when and how to automate testing process. Instead of answering them individually I thought it would be better to have some discussion here. I will put my thoughts about when to automate, how to automate or should we automate our testing work? I know there some of our readers are smarter than me. So it would be always a good idea to start a meaningful discussion on such vast topic to get in-depth idea and thoughts from experts from different areas and their experience in automation testing.

Why Automation testing?
1) You have some new releases and bug fixes in working module. So how will you ensure that the new bug fixes have not introduced any new bug in previous working functionality? You need to test the previous functionality also. So will you test manually all the module functionality every time you have some bug fixes or new functionality addition? Well you might do it manually but then you are not doing testing effectively. Effective in terms of company cost, resources, Time etc. Here comes need of Automation.
- So automate your testing procedure when you have lot of regression work.

2) You are testing a web application where there might be thousands of users interacting with your application simultaneously. How will you test such a web application? How will you create those many users manually and simultaneously? Well very difficult task if done manually.
- Automate your load testing work for creating virtual users to check load capacity of your application.

3) You are testing application where code is changing frequently. You have almost same GUI but functional changes are more so testing rework is more.
- Automate your testing work when your GUI is almost frozen but you have lot of frequently functional changes.

What are the Risks associated in Automation Testing?
There are some distinct situations where you can think of automating your testing work. I have covered some risks of automation testing here. If you have taken decision of automation or are going to take sooner then think of following scenarios first.

1) Do you have skilled resources?
For automation you need to have persons having some programming knowledge. Think of your resources. Do they have sufficient programming knowledge for automation testing? If not do they have technical capabilities or programming background that they can easily adapt to the new technologies? Are you going to invest money to build a good automation team? If your answer is yes then only think to automate your work.

2) Initial cost for Automation is very high:
I agree that manual testing has too much cost associated to hire skilled manual testers. And if you are thinking automation will be the solution for you, Think twice. Automation cost is too high for initial setup i.e. cost associated to automation tool purchase, training and maintenance of test scripts is very high.
There are many unsatisfied customers regretting on their decision to automate their work. If you are spending too much and getting merely some good looking testing tools and some basic automation scripts then what is the use of automation?

3) Do not think to automate your UI if it is not fixed:
Beware before automating user interface. If user interface is changing extensively, cost associated with script maintenance will be very high. Basic UI automation is sufficient in such cases.

4) Is your application is stable enough to automate further testing work?
It would be bad idea to automate testing work in early development cycle (Unless it is agile environment). Script maintenance cost will be very high in such cases.

5) Are you thinking of 100% automation?
Please stop dreaming. You cannot 100% automate your testing work. Certainly you have areas like performance testing, regression testing, load/stress testing where you can have chance of reaching near to 100% automation. Areas like User interface, documentation, installation, compatibility and recovery where testing must be done manually.

6) Do not automate tests that run once:
Identify application areas and test cases that might be running once and not included in regression. Avoid automating such modules or test cases.

7) Will your automation suite be having long lifetime?
Every automation script suite should have enough life time that its building cost should be definitely less than that of manual execution cost. This is bit difficult to analyze the effective cost of each automation script suite. Approximately your automation suite should be used or run at least 15 to 20 times for separate builds (General assumption. depends on specific application complexity) to have good ROI.

Here is the conclusion:

Automation testing is the best way to accomplish most of the testing goals and effective use of resources and time. But you should be cautious before choosing the automation tool. Be sure to have skilled staff before deciding to automate your testing work. Otherwise your tool will remain on the shelf giving you no ROI. Handing over the expensive automation tools to unskilled staff will lead to frustration. Before purchasing the automation tools make sure that tool is a best fit to your requirements. You cannot have the tool that will 100% match with your requirements. So find out the limitations of the tool that is best match with your requirements and then use manual testing techniques to overcome those testing tool limitations. Open source tool is also a good option to start with automation. To know more on choosing automation tools read my previous posts here and here.

Instead of relying 100% on either manual or automation use the best combination of manual and automation testing. This is the best solution (I think) for every project. Automation suite will not find all the bugs and cannot be a replacement for real testers. Ad-hoc testing is also necessary in many cases.

Over to you. I would like to here your experience about automation testing. Any practical experience will always be helpful for our readers.

To keep our readers updated on software testing events and activities, I will compile a weekly list of the online resources found. No doubt these resources are quality resources on software testing and you will get insight of software testing industry.

Microsoft last week launched software testing website – Microsoft tester center. James Whittaker Security Architect, Microsoft said “The goal of launching tester center is to provide community of software testers to share knowledge and learn from each other about software testing, day to day job work, testing tools and processes”.

Software testers and developers can view video’s, technical articles. You can interact technical experts from Microsoft via blog’s and discussion forums.

I am always interested reading the expert views on software testing from Microsoft employees. Michael Hunter’s interview series with software testing experts is also a good resource for testers. Interview of Anu Arora – a test manager from Microsoft’s engineering excellence group is the best of this series.

One interesting question I like from this interview is:

Q. What do you think is the most important thing for a tester to know? To do? For developers to know and do about testing?
Anu Arora: Testers need to understand that testing is not just a lot of automation. Testers also need to understand that it is not the other extreme banging the keyboard.

Testing is a highly skilled and sophisticated discipline and testers are the last line of defense for the customer. For the same reasons, the testers need to continually learn and grow; learn new ways to test, learn about the customer and market, learn to be always on the cutting edge of testing.

Developers need to understand that testers are not babysitters. Developers need to be responsible for their own code. Developers need to ensure what they write, works. In the testing language it is called unit testing. As a Test manager, I have been asked once in a while by the few ignorant ones; that if developers did all that, what will testers do? Testers can them move beyond unit testing to focus on integration, systems, scenario and user testing. These are absolutely critical from the customer point of view the one whom we are building this product for.

Jayant Deo asks:
“Looking at the current scenario from the industry it is seen that the testers are expected to have both technical testing skills as well either need to be from the domain background or have gathered domain knowledge mainly for BFSI is commonly seen.
I would like to know why and when is this domain knowledge imparted to the tester during the testing cycle?”

First of all I would like to introduce three dimensional testing career mentioned by Danny R. Faught. There are three categories of skill that need to be judged before hiring any software tester. What are those three skill categories?
1) Testing skill
2) Domain knowledge
3) Technical expertise.

No doubt that any tester should have the basic testing skills like Manual testing and Automation testing. Tester having the common sense can even find most of the obvious bugs in the software. Then would you say that this much testing is sufficient? Would you release the product on the basis of this much testing done? Certainly not. You will certainly have a product look by the domain expert before the product goes into the market.

While testing any application you should think like a end-user. But every human being has the limitations and one can’t be the expert in all of the three dimensions mentioned above. (If you are the experts in all of the above skills then please let me know ) So you can’t assure that you can think 100% like how the end-user going to use your application. User who is going to use your application may be having a good understanding of the domain he is working on. You need to balance all these skill activities so that all product aspects will get addressed.

Nowadays you can see the professional being hired in different companies are more domain experts than having technical skills. Current software industry is also seeing a good trend that many professional developers and domain experts are moving into software testing.

We can observe one more reason why domain experts are most wanted! When you hire fresh engineers who are just out of college you cannot expect them to compete with the experienced professionals. Why? Because experienced professional certainly have the advantage of domain and testing experience and they have better understandings of different issues and can deliver the application better and faster.

Here are some of the examples where you can see the distinct edge of domain knowledge:
1) Mobile application testing.
2) Wireless application testing
3) VoIP applications
4) Protocol testing
5) Banking applications
6) Network testing

How will you test such applications without knowledge of specific domain? Are you going to test the BFSI applications (Banking, Financial Services and Insurance) just for UI or functionality or security or load or stress? You should know what are the user requirements in banking, working procedures, commerce background, exposure to brokerage etc and should test application accordingly, then only you can say that your testing is enough – Here comes the need of subject-matter experts.

Let’s take example of my current project: I am currently working on search engine application. Where I need to know the basic of search engine terminologies and concepts. Many times I see some other team tester’s asking me questions like what is ‘publishers’ and ‘advertisers’, what is the difference and what they do? Do you think they can test the application based on current online advertising and SEO? Certainly not. Unless and until they get well familiar with these terminologies and functionalities.

When I know the functional domain better I can better write and execute more test cases and can effectively simulate the end user actions which is distinctly a big advantage.

Here is the big list of the required testing knowledge:

• Testing skill
• Bug hunting skill
• Technical skill
• Domain knowledge
• Communication skill
• Automation skill
• Some programming skill
• Quick grasping
• Ability to Work under pressure …

That is going to be a huge list. So you will certainly say, do I need to have these many skills? Its’ depends on you. You can stick to one skill or can be expert in one skill and have good understanding of other skills or balanced approach of all the skills. This is the competitive market and you should definitely take advantage of it. Make sure to be expert in at least one domain before making any move.

What if you don’t have enough domain knowledge?
You will be posted on any project and company can assign any work to you. Then what if you don’t have enough domain knowledge of that project? You need to quickly grasp as many concepts as you can. Try to understand the product as if you are the customer and what customer will do with application. Visit the customer site if possible know how they work with the product, Read online resources about the domain you want to test the application, participate in events addressing on such domain, meet the domain experts. Or either company will provide all this in-house training before assigning any domain specific task to testers.

There is no specific stage where you need this domain knowledge. You need to apply your domain knowledge in each and every software testing life cycle.

If you are reading this article till this point then I would like to hear on which domain you are working on? So that our readers can get better idea of different domains and projects. Comment your domain below.

Update: As per the request from many readers I have updated our software testing resource page for the BFSI domain online documents and articles for downloads.

Here is another post of WinRunner Interview preparation series. You can find here frequently asked WinRunner interview questions with answers. Most of the questions are based on Test Script Language (TSL), recording scripts in WinRunner and introduction to some important basic WinRunner most used commands.

1) What is the purpose of set_window command?
Set_Window command sets the focus to the specified window. We use this command to set the focus to the required window before executing tests on a particular window.

Syntax: set_window(<logical name>, time);
The logical name is the logical name of the window and time is the time the execution has to wait till it gets the given window into focus.

Example:
set_window (“Print”, 12);
button_press (“OK”);

The set_window statement indicates that the Print window is the active window.
The OK button is learned within the context of this window. If you program a test manually, you need to enter the set_window statement then the active window changes. When editing a script, take care not to delete necessary set_window statements.

2) Why don’t we normally load the GUI maps through start up scripts?
a) If we are using a single GUI Map file for the entire AUT then the memory used by the GUI Map may be much high.
b) If there is any change in the object being learned then WinRunner will not be able to recognize the object, as it is not in the GUI Map file loaded in the memory. So we will have to learn the object again and update the GUI File and reload it.

3) How do you unload the GUI map?
We can use GUI_close to unload a specific GUI Map file or else we call use GUI_close_all command to unload all the GUI Map files loaded in the memory.

Syntax: GUI_close(<file_name>); or GUI_close_all;

You can also use GUI_unload and GUI_unload_all functions to unload loaded GUI map files.

4) What is the use of GUI map and what happens when GUI map file get loaded?
When we load a GUI Map file, the information about the windows and the objects with their logical names and physical description are loaded into memory. So when the WinRunner executes a script on a particular window, it can identify the objects using this information loaded in the memory.

5) How do you copy and move objects between different GUI map files?
We can copy and move objects between different GUI Map files using the GUI Map Editor. The steps to be followed are:

1. Choose Tools > GUI Map Editor to open the GUI Map Editor.
2. Choose View > GUI Files.
3. Click Expand in the GUI Map Editor. The dialog box expands to display two GUI map files simultaneously.
4. View a different GUI map file on each side of the dialog box by clicking the file names in the GUI File lists.
5. In one file, select the objects you want to copy or move. Use the Shift key and/or Control key to select multiple objects. To select all objects in a GUI map file, choose Edit > Select All.
6. Click Copy or Move.
7. To restore the GUI Map Editor to its original size, click Collapse.

6) How do you configure GUI map?
a) When WinRunner learns the description of a GUI object, it does not learn all its properties. Instead, it learns the minimum number of properties to provide a unique identification of the object.
b) Many applications also contain custom GUI objects. A custom object is any object not belonging to one of the standard classes used by WinRunner. These objects are therefore assigned to the generic “object” class. When WinRunner records an operation on a custom object, it generates obj_mouse_ statements in the test script.
c) If a custom object is similar to a standard object, you can map it to one of the standard classes. You can also configure the properties WinRunner uses to identify a custom object during Context Sensitive testing. The mapping and the configuration you set are valid only for the current WinRunner session. To make the mapping and the configuration permanent, you must add configuration statements to your startup test script.

7) What is the purpose of GUI spy?
Using the GUI Spy, you can view the properties of any GUI object on your desktop. You use the Spy pointer to point to an object, and the GUI Spy displays the properties and their values in the GUI Spy dialog box. You can choose to view all the properties of an object, or only the selected set of properties that WinRunner learns.

8 ) What is the purpose of different record methods 1) Record 2) Pass up 3) As Object 4) Ignore.
a) Record instructs WinRunner to record all operations performed on a GUI object. This is the default record method for all classes. (The only exception is the static class (static text), for which the default is Pass Up.)
b) Pass Up instructs WinRunner to record an operation performed on this class as an operation performed on the element containing the object. Usually this element is a window, and the operation is recorded as win_mouse_click.
c) As Object instructs WinRunner to record all operations performed on a GUI object as though its class were “object” class.
d) Ignore instructs WinRunner to disregard all operations performed on the class.

9) What are the virtual objects and how do you learn them?
You can teach WinRunner to recognize any bitmap in a window as a GUI object by defining the bitmap as a virtual object.
You can create virtual push buttons, radio buttons, check buttons, lists, or tables, according to the bitmap’s behavior in your application. If none of these is suitable, you can map a virtual object to the general object class.

a) Applications may contain bitmaps that look and behave like GUI objects. WinRunner records operations on these bitmaps using win_mouse_click statements. By defining a bitmap as a virtual object, you can instruct WinRunner to treat it like a GUI object such as a push button, when you record and run tests.
b) Using the Virtual Object wizard, you can assign a bitmap to a standard object class, define the coordinates of that object, and assign it a logical name.

10) What is the use of Virtual Object Wizard and how it is used?
To define a virtual object using the Virtual Object wizard:

1. Choose Tools > Virtual Object Wizard. The Virtual Object wizard opens. Click Next.
2. In the Class list, select a class for the new virtual object. If rows that are displayed in the window. For a table class, select the number of visible rows and columns. Click Next.
3. Click Mark Object. Use the crosshairs pointer to select the area of the virtual object. You can use the arrow keys to make precise adjustments to the area you define with the crosshairs. Press Enter or click the right mouse button to display the virtual object’s coordinates in the wizard. If the object marked is visible on the screen, you can click the Highlight button to view it. Click Next.
4. Assign a logical name to the virtual object. This is the name that appears in the test script when you record on the virtual object. If the object contains text that WinRunner can read, the wizard suggests using this text for the logical name. Otherwise, WinRunner suggests virtual_object, virtual_push_button, virtual_list, etc.
5. You can accept the wizard’s suggestion or type in a different name. WinRunner checks that there are no other objects in the GUI map with the same name before confirming your choice. Click Next.

11) What are the modes of script recording in WinRunner ?
There are 2 modes of recording in WinRunner

• Context Sensitive recording records the operations you perform on your application by identifying Graphical User Interface (GUI) objects.
• Analog recording records keyboard input, mouse clicks, and the precise x- and y-coordinates traveled by the mouse pointer across the screen.

12) What is a checkpoint and what are different types of checkpoints?
Checkpoints allow you to compare the current behavior of the application being tested to its behavior in an earlier version.

You can add four types of checkpoints to your test scripts:

1. GUI checkpoints verify information about GUI objects. For example, you can check that a button is enabled or see which item is selected in a list.
2. Bitmap checkpoints take a snapshot of a window or area of your application and compare this to an image captured in an earlier version.
3. Text checkpoints read text in GUI objects and in bitmaps and enable you to verify their contents.
4. Database checkpoints check the contents and the number of rows and columns of a result set, which is based on a query you create on your database.

If you have missed previous posts on WinRunner interview question series, here is the list of posts:

WinRunner Interview Questions Part 1

WinRunner Interview Questions Part 2

WinRunner Questions from Interviewer point of view

Basic WinRunner Interview questions

You can ask your doubts in the comment section below if you are not clear with some answers.

If you are preparing for CSTE testing certification exam or thinking to give the exam in coming days then this question series will help you for preparation. Here I have included some questions from CSTE objective type question papers.

The software testing/Quality assurance professionals can also take this exam to test their testing knowledge.

You can either take printout and mark the answers or note your answers somewhere on the paper serially for all 25 questions. Verify your answers at the answer page provided at the bottom of this test page.

1. Verification is:
a. Checking that we are building the right system
b. Checking that we are building the system right
c. Performed by an independent test team
d. Making sure that it is what the user really wants

2. A regression test:
a. Will always be automated
b. Will help ensure unchanged areas of the software have not been affected
c. Will help ensure changed areas of the software have not been affected
d. Can only be run during user acceptance testing

3. If an expected result is not specified then:
a. We cannot run the test
b. It may be difficult to repeat the test
c. It may be difficult to determine if the test has passed or failed
d. We cannot automate the user inputs

4. Which of the following could be a reason for a failure
1) Testing fault
2) Software fault
3) Design fault
4) Environment Fault
5) Documentation Fault
a. 2 is a valid reason; 1,3,4 & 5 are not
b. 1,2,3,4 are valid reasons; 5 is not
c. 1,2,3 are valid reasons; 4 & 5 are not
d. All of them are valid reasons for failure

5. Test are prioritized so that:
a. You shorten the time required for testing
b. You do the best testing in the time available
c. You do more effective testing
d. You find more faults

6. Which of the following is not a static testing technique
a. Error guessing
b. Walkthrough
c. Data flow analysis
d. Inspections

7. Which of the following statements about component testing is not true?
a. Component testing should be performed by development
b. Component testing is also know as isolation or module testing
c. Component testing should have completion criteria planned
d. Component testing does not involve regression testing

8. During which test activity could faults be found most cost effectively?
a. Execution
b. Design
c. Planning
d. Check Exit criteria completion

9. Which, in general, is the least required skill of a good tester?
a. Being diplomatic
b. Able to write software
c. Having good attention to detail
d. Able to be relied on

10. The purpose of requirement phase is
a. To freeze requirements
b. To understand user needs
c. To define the scope of testing
d. All of the above

11. The process starting with the terminal modules is called -
a. Top-down integration
b. Bottom-up integration
c. None of the above
d. Module integration

12. The inputs for developing a test plan are taken from
a. Project plan
b. Business plan
c. Support plan
d. None of the above

13. Function/Test matrix is a type of
a. Interim Test report
b. Final test report
c. Project status report
d. Management report

14. Defect Management process does not include
a. Defect prevention
b. Deliverable base-lining
c. Management reporting
d. None of the above

15. What is the difference between testing software developed by contractor outside your country, versus testing software developed by a contractor within your country?
a. Does not meet people needs
b. Cultural difference
c. Loss of control over reallocation of resources
d. Relinquishments of control

16. Software testing accounts to what percent of software development costs?
a. 10-20
b. 40-50
c. 70-80
d. 5-10

17. A reliable system will be one that:
a. Is unlikely to be completed on schedule
b. Is unlikely to cause a failure
c. Is likely to be fault-free
d. Is likely to be liked by the users

18. How much testing is enough
a. This question is impossible to answer
b. The answer depends on the risks for your industry, contract and special requirements
c. The answer depends on the maturity of your developers
d. The answer should be standardized for the software development industry

19. Which of the following is not a characteristic for Testability?
a. Operability
b. Observability
c. Simplicity
d. Robustness

20. Cyclomatic Complexity method comes under which testing method.
a. White box
b. Black box
c. Green box
d. Yellow box

21. Which of these can be successfully tested using Loop Testing methodology?
a. Simple Loops
b. Nested Loops
c. Concatenated Loops
d. All of the above

22. To test a function, the programmer has to write a ______, which calls the function and passes it test data.
a. Stub
b. Driver
c. Proxy
d. None of the above

23. Equivalence partitioning is:
a. A black box testing technique used only by developers
b. A black box testing technique than can only be used during system testing
c. A black box testing technique appropriate to all levels of testing
d. A white box testing technique appropriate for component testing

24. When a new testing tool is purchased, it should be used first by:
a. A small team to establish the best way to use the tool
b. Everyone who may eventually have some use for the tool
c. The independent testing team
d. The vendor contractor to write the initial scripts

25. Inspections can find all the following except
a. Variables not defined in the code
b. Spelling and grammar faults in the documents
c. Requirements that have been omitted from the design documents
d. How much of the code has been covered

Done?

Click here to Verify your answers.