Tools learning for testing

Many times folks used to ask, what tools would be good for testing career. Used to advice to learn popular tools (mostly commercial tools) based on their coding skills and interest. But this scenario is changed in last few years. Testers should be ready to handle the automation based on business needs, rather than just toolsmiths. Also mindset change should be there, instead of sticking with just one tool and one scripting language

Read an article recently in this line and few excerpts..


It’s important to ask yourself, why do you want to learn about a specific tool or set of tools? They are an integral part of software testing, supporting what we do on a daily basis. However, having knowledge of a set of tools is less important than knowing when and where to use the right tool. An effective tester will also have the skill to know when and where to use the right tool because the problem requires it, not because the job market determines it desirable. Using the wrong tool at the wrong time can have adverse effects on your day-to-day work, potentially slowing you down, or worse, giving you biased or flat out false information. That’s not to say you shouldn’t explore new tools, but always remember:

Problem first, tools second
With this concept in mind, let’s explore different categories of tools. Some examples will be offered but they are by no means an exhaustive list. I encourage you to build up a toolbox or reference list of tools you have used in the past.

Original Article - What Tools Should I Learn?

Virtualized Servers for Performance Testing

We have to do benchmark testing as well as stress testing. Already high performance system was purchased and virtual machines are created for performance testing setup. I can see few limitations for this approach, since production boxes all are in physical servers. Earlier I've used virtual machines for automation regression execution.

Issues/Limitations of Virtualized Servers

1. if (production environment != load test environment) then result could be wrong
2. Increasing the risk of deployment failure

Did a google search and found following links, which give all the details, including Vmware claims.

Virtualization: Performance testing

Measuring the Performance Impact of Virtualizing a Web Application Server

Response to 'Load Testing a Virtual Web Application'

Virtualization performance testing tips


Anybody have experienced load testing on Virtualized servers? Please share your experience...

Generic Test Automation Framework

So far, I have developed various automation frameworks and recently using Robot Framework. Listed many features for Ideal Automation Framework and given below. Also would like to get more features from other users.

Reference to earlier posts and Links about automation framework.
Wikipedia - Test Automation
Wikipedia - KeyWord Driven Testing
Is KeyWord Driven Framework enough?
My Thoughts about Automation
What is an Automation Framework - from Dhanasekar's blog

Features for Generic and common Test Automation Framework

Headers	Features
Supported Frameworks	Functional Decomposition
	Data Driven
	Keyword Driven
	Tabel Driven
Independent to	Operating System
	Application
	Technology
Test Scripting	Recording
	Parameterization
	Suite Setup Module
	Suite TearDown Module
	Test Setup Module
	Test TearDown Module
	Database Integration
	Scripting
	Environment Library
Test Execution	Multiple OS
	Multiple platforms
	Parallel Execution
	Test Timeout
	Based on Priority
	Grouping Tests
	Running Failed tests
Result Analysis	Detail reporting for each step execution
	Snapshots for failed tests
	Test level reporting
	Compiled report
	Results by category
Expected Benefits	Less effort to automate
	Less dependence on highly skilled automation professionals
	Exception handling
	Easy to Maintain
	Easy to Integrate with Other tools
	Support to CI (Continuous Integration) tools
Additional Expectations	Dynamic conditions
	Dynamic Loop support
	UI to Edit data
	UI to execute tests
	Update test metrics automatically
	Update captured defects automatically

Selenium RC configuration for Java

Few months back, I was trying to run selenium script with Selenium RC Java. Initially I faced few issues to configure the classpath settings. We need to pass the proxy details, if the net connection is based on proxy. I created three batch scripts to run the server and client. You should have latest java version. For more info about Selenium RC, check out Selenium RC documentation .

Selenium Server to start with proxy
Here user can set the username and password for proxy connection. Checkout java options to do that.

REM ********************************************
REM To Run Selenium server
REM Author: Palani Selvam
REM ********************************************

REM Selenium server path
set SeleniumServer=D:\Selenium_trial\tool\selenium-remote-control-0.9.2\selenium-server-0.9.2

REM to run the Selenium server with proxy
d:
cd %SeleniumServer%
java -Dhttp.proxyHost=proxy.mycompany.com -Dhttp.proxyPort=8080 -jar selenium-server.jar 

Selenium Server to start without proxy
It will be useful for non-proxy connection.

REM ********************************************
REM To Run Selenium server
REM Author: Palani Selvam
REM ********************************************

REM Selenium server path
set SeleniumServer=D:\Selenium_trial\tool\selenium-remote-control-0.9.2\selenium-server-0.9.2

REM to run the Selenium server without proxy

d:
cd %SeleniumServer%
java -jar selenium-server.jar

To run the Selenium client
Below I have given the batch script to run the GetAllStockQuotes.java. Source code is available at Indian Stock Quotes through Selenium

REM ********************************************
REM To Run Selenium client (Java). Also sets path and classpath
REM Author: Palani Selvam
REM ********************************************

REM to set class path and path env variables to run selenium
set javaClientPath=D:\Selenium_trial\tool\selenium-remote-control-1.0.1\selenium-java-client-driver-1.0.1
Set javaClientJars=%javaClientPath%\selenium-java-client-driver.jar

REM JUnit jar
Set jUnitJars=D:\java\JUnit\junit46.jar

REM Selenium Script (Java) dir path
set Stockspath=D:\Selenium_trial

REM Setting classpath
set classpath=%classpath%;%javaClientJars%;%jUnitJars%;%Stockspath%\classes;

d:
cd %Stockspath%

REM Compiling test scripts Selenium-java code
javac -d classes GetAllStockQuotes.java

REM Running the test scripts
java junit.textui.TestRunner com.palani.tests.GetAllStockQuotes

For more details of Selenium tools configuration, you can refer 5 Minute Guide To Selenium IDE and Selenium Remote Control (Java) Test Tools

Is KeyWord Driven Framework enough?

Recently I heard the term Key-Word Driven Framework from many testing people. Even people are showing interest to know about it and to implement it. Sometimes people are saying unknowingly as Keyboard Driven Framework.

Few months back, I have implemented Keyword Driven framework in one of our projects. We are using Silktest and data files are kept in XML format. The other automation team members also impressed with this kind of implementation. I found this article Automated Testing Institute - Building A Keyword Driver Script In 4 Steps useful to the professionals, who wants to implement KeyWord Driven framework.

After that the expectation goes to implement KeyWord Driven Framework to all automation projects. I am not against to Keyword Driven Framework. It can be used for many projects. But it is not the only one solution for all our automation issues or dependencies. For example, chart automation. For charts, verification will be different and It is difficult to generalize. May be Flex charts can be used if tools are able to identify all parts of charts. One more is unlimited Keywords. Assume that one project contains more than 500 keywords and the automation guys should know the functionality for all those words. It is definitely issue, if number of keywords increased beyond the limit. Currently I'm thinking the scenarios, where KeyWord or Table Driven Automation frameworks can not be used or should be avoided. Below I have listed the cases.

1. More Context based Menus
   The menu items are shown based on the screen and object.


2. Multiple Application interactions in single project


3. Total Keywords beyond 200
   Testers should memorize all keywords. It is similar to remember Differential Calculus and Integral Calculus functions. One should remember keywords for action and verification for all set of objects.


4. Many hierarchies for Object identification.
   It can be solved if testing tool supports X-path for object identification.


5. Complex data.
   One of my projects required minimum 40 string data to create a report and it is just a first step for any test case. Also the data may change in the future. More than 1000 cases required like that. I can generalize the inputs data and it is difficult to maintain and modify the data.

To know the automation framework concepts, you can go through following links.

Test Automation White Papers
My Thoughts about Automation
What is an Automation Framework - from Dhanasekar's blog
Wikipedia - Test Automation
Wikipedia - KeyWord Driven Testing

Winrunner -Scope of Variables

Winrunner has four different types of variables. They are used inside the funciton, outside the function and calling from different tests.

auto : An auto variable can only be declared within a function and is local to that function.
It exists only while the function is running. A new copy of the variable is created each time the function is called.

static : A static variable is local to the function, test, or compiled module in which it is declared. The variable retains its value until the test is terminated by a Stop command.

public : A public variable can only be declared within a test or module, and is available for all functions, tests, and compiled modules.

extern : An extern declaration indicates a reference to a public variable declared outside of the current test or module.

For more info, See the Winrunner Help.

Silktest - Difference between tag and identifier

Sometimes testers are confused with tag and identifier. Thats why, I prepared this post.

Identifier
Identifier is equal to a variable and it holds the object's reference. The Constraints, which is applicable for variable, is applicable for identifier also. You can not give special characters such as space, dollar etc into the identifier.
Testcases use the identifier to refer to an object. In Winrunner, it is referred as Logical Name.

tag
Tag is a string information, which contains the object details. Tag is the physical description of the object. SilkTest uses the tag to identify objects in the application under test when recording and when executing testcases. Testcases never use the tag to refer to an object. Tags are classified as tag and multitag.

Types of tags:

1. Caption


2. Prior text


3. Index


4. Window ID


5. Location


6. Attributes - only for HTML objects (Added from Silktest 2006).

More details: See the Silktest help for tag: definition

Test Development

Today Software Testing is not a low end task. Testers are doing variety of testing. Sometimes developers are moved to Test Automation and Performance Testing. I am sure that Code development skills improves Tester's ability. Many test expertises have said about it. I found one interesting article at Testy's blog. He has analyzed in depth. Link ==> Do testers need programming skills?

The below quote content is from I.M.Testy's blog

The debate over whether testers need to at least understand programming concepts is still raging within the discipline. To me this debate is puzzling because it seems to suggest that as a professional, I don't have to really understand or be completely proficient in critical aspects of my trade. Even Cem Kaner noted, "I think that the next generation of testers will have to have programming skills." Actually, there was a time not so long ago when testers had to have programming skills, so it is nice that Cem now acknowledges that skill as useful in testing.

Unfortunately, occasionally even within Microsoft a few people still want to differentiate between STE and SDET by blindly assuming that STE meant non-programming testers. The fact is, that the old STE ladder level guidelines clearly stated skills such as debugging production code, and design and develop effective automation as required skills for Microsoft testers. Unfortunately, some managers chose to selectively ignore these skill requirements and some groups chose to differentiate between GUI testers and any tester who could write code by labeling them as STE and SDET respectively. (This was a horrible abomination of job titles in my opinion.) The new SDET competencies at Microsoft are designed, and supposed to be implemented in a manner, to reinforce the essential skills we expect from our testers so a tester at a certain level in their career stage in one business unit essentially has equitable skills of any other tester at the same level in their career stage in any group in the company.

Advanced Peer-to-Peer Networking (APPN)

In my first job, I used to store few tech terms as texts. One such thing is given below.

Advanced Peer-to-Peer Networking (APPN), part of IBM's Systems Network Architecture (SNA), is a group of protocols for setting up or configuring program-to-program communication within an IBM SNA network. Using APPN, a group of computers can be automatically configured by one of the computers acting as a network controller so that peer programs in various computers will be able to communicate with other using specified network routing.
APPN features include:

Better distributed network control; because the organization is peer-to-peer rather than solely hierarchical, terminal failures can be isolated Dynamic peer-to-peer exchange of information about network topology, which enables easier connections, reconfigurations, and routing Dynamic definition of available network resources, Automation of resouce registration and directory lookup

Handle Same titled windows

Many projects are having same title for the windows or dialog boxes. Assume that two window objects are unique and children of them are different. In this situation, look for unique child objects and re-define your frame like below. It will be useful mainly for installers and web applications. Often the browser title will be same for many screens.

Below I have given for Silktest. Assume that X & Y are two different windows and having same tag. Both windows are differentiated by their unique child objects.

Window X
tag "TestWindow/[TextField]Name/.."

Window Y
tag "TestWindow/[Pushbutton]Search/.."

Data Driven Approach

Data driven scripts are those application-specific scripts manually coded in the automation tool’s proprietary language and then modified to accommodate variable data. Variables will be used for application under test input and execution of program selections. This allows the script to drive the application with external data supplied by the test engine. Most of the automation tools are having the feature to convert a recorded script into data driven script. The tools such as Silktest, WinRunner and QuickTest Professional are using Excel file as data file format.

Though these methodologies recommended by conspicuous experts are powerful but the utilization of the advantages propounded by these methods in the Test Automation architecture is under question.

Limitations of Data Driven Approach:

• Multiple data files are essential for this approach and hence the maintenance of these files becomes a nightmare. Each and every Test Case requires inputs and verifications from different files.


• Testers should carefully enter the data if anything at all changes in all the required files.


• Script processing errors are common if a proper attention is not given to the file formats.


• The process of coding and maintaining the tests was tedious and time-consuming. It took a lot of time to create all these tests because there was no modularity and therefore no code reuse.

Due to the ineffective usage of these methods, data driven scripts still contain the hard coded and sometimes very fragile recognition strings for the window components they navigate. When this is the case, the scripts are easily broken when an application change or revision occurs. And when these scripts start breaking (not necessarily just a few). We are sometimes talking about a great many, if not all the scripts, for the entire application.

Fantasy about Record and Playback

Many testers do not have strong programming skills. With the scripting problems, it leads testers to use record and playback techniques. There indeed are many tools that allow scripts to be recorded and then played back, using screen captures for verification.

Every automated test tool vendor will tell you that their tool is "easy to use" and that your non-technical user-type testers can easily automate all of their tests by simply recording their actions, and then playing back the recorded scripts. This one statement alone is probably the most responsible for the majority of automated test tool software that is gathering dust on shelves in companies around the world. We have seen the vendor’s sample applications. We have seen the tools play nice with those applications. And we try to get the tools to play nice with our applications just as fluently. Inherently, project after project, we do not achieve the same level of success.

The Record / Playback Myth:
The problem that an always crop up is that the layouts are changed, invalidating the screen captures and then the interface controls change making playback fail. Now the scripts must be re-recorded from scratch. Record and playback tools provide an easy way to create throwaway test suites.

You can easily notice that identification of controls and windows is through the co- ordinates of the screen. The scripts will be broken easily when there is a change in resolution of the screen.

The scripts resulting from this method contain hard-coded values, which must change if anything at all changes in the application.

If the tester makes an error entering data, etc., the test must be re-recorded. If the application changes, the test must be re-recorded.

Sometimes this manifesto is hard to explain to people who have not yet performed significant test automation with these capture \ replay tools. But it usually takes less than a week, often less than a day, to hear the most repeated phrase: "It worked when I recorded it, but now it fails when I play it back". But this is true in case of record and play back tool.

Common Problems in Test Automation

Few years back, I have prepared few documents about automation scripting and concepts. This post will list the common problems faced by software industries in their test automation attempts.

Automation is not cheap
Test Automation is not cheap. Usually it takes between 3 to 10 times as long to create, verify and document the automated test as it takes to create and run once by hand. Test Automation takes lesser time to create than to test each test scripts.

Easy to run test alone is automated
Many test groups automate only the easy to run tests because early in testing, these are easy to design and the program might not be capable of running more complex test cases. Hence harsh test needs to be done by the skilled manual tester.

Duplicate information was kept in multiple repositories
Most of the project teams purchases a test management tool in addition to the already existing automated testing tools. Duplicate information was kept in multiple repositories and was very difficult to maintain. In several instances, the implementation of more tools actually results in less productivity.

Test Scripts duplicates the development effort
Test script development results in an almost complete duplication of the development effort, through overuse of the testing tool’s programming language. In normal case the application itself used a complex algorithm and again the tester recreates these algorithms using the testing tool. Too much time was spent on automating scripts, without much additional value gained. The test team must be careful not to duplicate the development effort; this is a risk when developing elaborate test scripts.

A lack of test development guidelines was noted
One program had several test automation engineers, each using a different style for creating test scripts. Maintaining the scripts is a nightmare. This is due to the non-availability of development guidelines and hence script readability and maintainability becomes complicated.

Reports generated by the tool is futile
More time is spent by the testers in setting up elaborate customized reports, which is part of the automated testing tool. The reports were never used, since the data required for the report is never accumulated in the tool.

SEO - Search Engine Optimization

After creating my blog, I searched my blog on Google. My blog is not displayed in search results. Then I started to read more about Publicizing the blog. Also I read about Search Engine Optimization (SEO). SEO helps to create high visibility on the leading search engines. It also helps to bring more visitors to your page and indirectly marketing your website.

Brad Callen is one of the search engine optimization experts. He has written a book about SEO. Brad has mentioned two types of optimization. One is OnPage optimization and another one is OffPage optimization. You can download his E-book from docstoc.com.

To know more about SEO, see this Wiki page . Also now You can do Search on my blog.

Screen Saver Off

All GUI test suite are not executed if screen saver screen is displayed or machine gets locked. Before running the scripts, automation engineer should confirm not to get the machine lock or screen saver. In few environments, user would not able to control.

Earlier I also faced this problem. Then I used to login as a local user instead of network user or company login id. Later we used VmWare Workstations to run the regression.

You can avoid this situation by few ways.

1. Using Registry


2. Using scripting while regression starts


3. Restart the Machine if system gets locked

Using Registry
It is a simple method. But it should not overriden by your company policy. System Admin may run your company screen savers.
1. Go to HKEY_CURRENT_USER
Path => [HKEY_CURRENT_USER\Software\Policies\Microsoft\Windows\Control Panel\Desktop]
2. Change the following values for following keys.
"ScreenSaveActive"="1"
"ScreenSaveTimeOut"="30000"

Using scripting while regression starts
1. Disable the screen saver. Below code is given for Silktest. Put into ScriptEnter block.
// Declaration
dll "user32.dll"
LONG SystemParametersInfo (LONG uAction, LONG uParam, LONG lpvParam, LONG fuWinIni) alias "SystemParametersInfoA"

// Calling statement
-> SystemParametersInfo (17, 0, 0, 0)

2. Enable the screen saver. Keep below code into ScriptExit function.
// Calling statement
-> SystemParametersInfo (17, 1, 0, 0)

Restart the Machine if system gets locked
User can call any batch script or system restart command. But you need to re-execute the scripts, from where it was stopped. It is a bit complex trick.

Special characters recognition

Any automation tool used to have few special characters to be masked to identify them. Silktest also not supports few characters in tag. You can't give following chars directly into tag.

"
[
]
{
&
*
/
?
|

Silktest is using those chars for different purpose. What can you do is, you can create a function to create correct tag for any string.

Writing a Problem Report

Many testers are not concentrating the details, while entering a defect. I have seen few guys entering just one line as bug description. Also they are not ready to put the details of the environment. For few scenarios, they can improve it by attaching proper snapshots. As a tester, we should give the right steps to reproduce the issue. Managers should not encourage informal ways like oral communication, email notes, etc.

Cem Kaner has written few articles about Bug Advocacy.
Bug Advocacy: PDF
Bug Advocacy: Video

Also I would like to give two quotes from his famous book.
Book: Testing Computer Software By Cem Kaner, Jack Falk, Hung Quoc Nguyen
ISBN: 0471358460

Quote1:

If your reports are not clear and understandable, bugs won't get fixed. You should spend minimum time needed to describe a problem in a way that maximizes the probability that it will be fixed. The content and tone of your reports affect that probability. The point of writing problem reports is to get bugs fixed.

Quote2:

Nobody likes being told that what did wrong, wrong, wrong. As a tester, that's what you tell people everyday. You can ensure your unpopular by describing problems in a way that tells the programmer you think (s)he is sloppy, stupid or unprofessional.

Documenting automated Scripts

Recently a colleague (new to our team) asked me the documentation for our suite. I do not have any document or HELP files. Earlier My team thought about it and one member came with a silktest script to do that. We have to add more features like search, organizing by attributes. That utility did not give the most needed feature. To ease the documentation job, we need to use set of tags and detailed info for them. We are unable to follow this more than a week.

Most of the automated testing tools are not having documenting facility. I am expecting similar to PerlDoc (for Perl scripts) or JavaDoc (for Java). It is the duty of script developer to ensure the adequate comments for particular function or code block. I used to put two or three line comments, While writing the functions.

Generally these kind of tools or utilities, have a parser. It can parse the source code and get the details by the pre-defined tags or keywords. Then It creates the HELP files for each code block or script. The usage will be meaningful if documentation is regularly updated. Now most utilities are creating HTML files and it is easy to navigate and share the information. I was going through J2Se doc comments. JavaDoc is having the flexibility of including HTML tags.

Capturing failures

In the previous post, we have seen the importance of Test Results Reporting. Capturing failures is helpful to debug the failure. Script will try to capture the error message in case of failure. If it is unable to capture the expected message or details, then it will capture the screen shot of particular object or screen. Usually these snapshots are bitmap files. Later I used an utility to convert a bitmap image to GIF image. It helps me to reduce the storage space.

Most of the load testing tools are capturing the objects as images or HTML files. Now Functional tools also providing this feature. Silktest provides the same feature through TrueLog Explorer. It is available from Silktest8.0 I hope that QTP also has same feature in different name.

Test Results Reporting

Results Reporting is very much important as automating functionalities. It will be better, if results are logged with screen shots for failed cases. Most of us, not concentrating much on Results Reporting. It will improve your automation level higher.

Automation Engineers are developing lot of test scripts. How will you check all scripts results? We can't check each testcase by testcase. Summary of test results is so important. Also the compiled results for all results is good to have.

Long back, I studied one article related with Automatic Results processing. That document discussed about extracting some keywords from particular log files and store the results in different format. I have implemented using Excel sheet and it will give result for each testcase(function) as well as compiled result for each scripts. If summary and results are stored into HTML files, we can easily get to know the results.

I agree, however one person need to look the results. Some of the tools are giving summary details after execution. I want to go one step ahead. I need to see all automation results of particular project from one file (HTML or Flat file or XL sheet). It is like maintaining history.

But Results processing is very common for test automation as well as general applications. You can follow like any standard log files format. You can maintain 2-3 log files for single script. One for all the activities of your suite and another one for Just giving Pass or Fail result for each testcase. It will be better, if all the log files are HTML files. At last prepare a single file to show the compiled results for the whole execution. You can publish the results easily.