Open2test.org Free Automation Frameworks

Open2test.org team has developed many frameworks for different tools like QTP, Selenium, TestPartner, e-Tester, Silktest and OpenScript. Couple of years back, frameworks are available only for QTP & Selenium

Available frameworks

• QTP for Web Framework
• QTP for Windows Framework
• QTP for .NET Framework
• QTP for .NET Syncfusion Framework
• QTP for JAVA Framework
• QTP for SAP Framework
• QTP for Oracle Apps Framework
• QTP for Flex Framework
• QTP for PowerBuilder Framework
• QTP for Mainframe Framework
• Selenium for Web Framework
• Testpartner for Web Framework
• Testpartner for .NET Framework
• e-Tester for Web Framework
• SilkTest for JAVA Framework
• SilkTest for Web Framework
• OpenScript for Web Framework

RobotFramework - Overview

I have executed many automation projects and also developed various automation frameworks using different tools like Silktest, QTP, Selenium, CodedUI, MS UIA, VisualTest (dead) and Winrunner (dead). Tried few open source frameworks and found RobotFramework as the best one. Successfully completed five projects using RobotFramework. Main advantage is, I'm able to integrate with different test tools and libraries. It is neither related to IBM Rational tools or any other commercial tools.

Summary in RobotFramework Homepage
Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has easy-to-use tabular test data syntax and utilizes the keyword-driven testing approach. Its testing capabilities can be extended by test libraries implemented either with Python or Java, and users can create new keywords from existing ones using the same syntax that is used for creating test cases.

Robot Framework is open source software released under Apache License 2.0. Its copyrights are owned and development supported by Nokia Siemens Networks.

Robot Framework Features

1. Advanced Keyword Driven framework
2. Enable functional team contribution for automation
3. Reusable/custom keywords
4. Dynamic variables
5. Easily extendable by Python, C# & Java
6. Supports few test libraries including Selenium
7. Provides simple and powerful Results Reporting
8. Multiple Operating Systems support
9. $ Free

Links for Robot Framework Resources
Robot Framework Home Page
Robot Framework User Guide
Robot Framework Quick Start Guide
Robot IDE
Robot Selenium Library

Robot Framework - Introduction
Robot Framework - Articles
Simple tutorial for Robot Framework

In last 6 months, many updates are happened on Robot framework and its related libraries. It is supporting many of my generic automation framework features.

Headers	Features	Robot Framework
Supported Frameworks	Functional Decomposition	Yes
	Data Driven	Yes
	Keyword Driven	Yes
	Table Driven	Yes
Independent to	Operating System	Yes
	Application	Yes
	Technology	Yes
Test Scripting	Recording	No
	Parametrization	Yes
	Suite Setup Module	Yes
	Suite TearDown Module	Yes
	Test Setup Module	Yes
	Test TearDown Module	Yes
	Database Integration	Yes
	Scripting	Yes
	Environment Library	Yes
Test Execution	Multiple OS	Yes
	Multiple platforms	Yes
	Parallel Execution	No
	Test Timeout	Yes
	Based on Priority	Yes
	Grouping Tests	Yes
	Running Failed tests	Yes
Result Analysis	Detail reporting for each step execution	Yes
	Snapshots for failed tests	Yes
	Test level reporting	Yes
	Compiled report	Yes
	Results by category	Yes
Expected Benefits	Less effort to automate	Yes
	Less dependence on highly skilled automation professionals	Yes
	Exception handling	Yes
	Easy to Maintain	Yes
	Easy to Integrate with Other tools	Yes
	Support to CI (Continuous Integration) tools	Yes
Additional Expectations	Dynamic conditions	Yes
	Dynamic Loop support	Yes
	UI to Edit data	Yes
	UI to execute tests	Yes
	Update test metrics automatically	Yes
	Update captured defects automatically	Yes

Generic Test Automation Framework

So far, I have developed various automation frameworks and recently using Robot Framework. Listed many features for Ideal Automation Framework and given below. Also would like to get more features from other users.

Reference to earlier posts and Links about automation framework.
Wikipedia - Test Automation
Wikipedia - KeyWord Driven Testing
Is KeyWord Driven Framework enough?
My Thoughts about Automation
What is an Automation Framework - from Dhanasekar's blog

Features for Generic and common Test Automation Framework

Headers	Features
Supported Frameworks	Functional Decomposition
	Data Driven
	Keyword Driven
	Tabel Driven
Independent to	Operating System
	Application
	Technology
Test Scripting	Recording
	Parameterization
	Suite Setup Module
	Suite TearDown Module
	Test Setup Module
	Test TearDown Module
	Database Integration
	Scripting
	Environment Library
Test Execution	Multiple OS
	Multiple platforms
	Parallel Execution
	Test Timeout
	Based on Priority
	Grouping Tests
	Running Failed tests
Result Analysis	Detail reporting for each step execution
	Snapshots for failed tests
	Test level reporting
	Compiled report
	Results by category
Expected Benefits	Less effort to automate
	Less dependence on highly skilled automation professionals
	Exception handling
	Easy to Maintain
	Easy to Integrate with Other tools
	Support to CI (Continuous Integration) tools
Additional Expectations	Dynamic conditions
	Dynamic Loop support
	UI to Edit data
	UI to execute tests
	Update test metrics automatically
	Update captured defects automatically

Is KeyWord Driven Framework enough?

Recently I heard the term Key-Word Driven Framework from many testing people. Even people are showing interest to know about it and to implement it. Sometimes people are saying unknowingly as Keyboard Driven Framework.

Few months back, I have implemented Keyword Driven framework in one of our projects. We are using Silktest and data files are kept in XML format. The other automation team members also impressed with this kind of implementation. I found this article Automated Testing Institute - Building A Keyword Driver Script In 4 Steps useful to the professionals, who wants to implement KeyWord Driven framework.

After that the expectation goes to implement KeyWord Driven Framework to all automation projects. I am not against to Keyword Driven Framework. It can be used for many projects. But it is not the only one solution for all our automation issues or dependencies. For example, chart automation. For charts, verification will be different and It is difficult to generalize. May be Flex charts can be used if tools are able to identify all parts of charts. One more is unlimited Keywords. Assume that one project contains more than 500 keywords and the automation guys should know the functionality for all those words. It is definitely issue, if number of keywords increased beyond the limit. Currently I'm thinking the scenarios, where KeyWord or Table Driven Automation frameworks can not be used or should be avoided. Below I have listed the cases.

1. More Context based Menus
   The menu items are shown based on the screen and object.


2. Multiple Application interactions in single project


3. Total Keywords beyond 200
   Testers should memorize all keywords. It is similar to remember Differential Calculus and Integral Calculus functions. One should remember keywords for action and verification for all set of objects.


4. Many hierarchies for Object identification.
   It can be solved if testing tool supports X-path for object identification.


5. Complex data.
   One of my projects required minimum 40 string data to create a report and it is just a first step for any test case. Also the data may change in the future. More than 1000 cases required like that. I can generalize the inputs data and it is difficult to maintain and modify the data.

To know the automation framework concepts, you can go through following links.

Test Automation White Papers
My Thoughts about Automation
What is an Automation Framework - from Dhanasekar's blog
Wikipedia - Test Automation
Wikipedia - KeyWord Driven Testing

Automation White Papers

I used to search framework papers, While starting automation in my first company. I have collected few links and docs. Some of them are not valid today. Listed favorite links in SQA Forums.

My Favorite papers
1. Totally Data-Driven Automated Testing by Keith Zambelich

2. Test Automation Frameworks by Carl J. Nagle

3. Architectures of Test Automation by Cem Kaner

4. Test Automation Snake Oil by James Bach

5. Success with Test Automation by Bret Pettichord

6. Seven Steps to Test Automation Success by Bret Pettichord

7. "When Should a Test Be Automated?" (Brian Marick)

8. Choosing a test automation framework from IBM

My thoughts about Test Automation Framework

Nowadays most of the people are talking about the framework. But most of them are not clear about the framework concepts. Here I'm writing my view about the framework.

Automation Framework is a concept, similar to OOPs. Automation Framework is an advanced thought from Record&Playback or writing some functions and executing them.

Automation Framework is a discipline and should have proper design/architecture. You can understand and implement. Automation Framework should have some of following components.

Components of a Framework
1. AUT Specific libraries
2. Tool wrapper function libraries.
3. Execution Engine (Script Execution should be drived
based on given test data) - Data Driven 4. Results
reporting mechanisms (PASS & FAIL for each
testcases/procedures and compiled results for all. It
would be better, if it captured AUT's snapshots) 5.
Planning for long term automation.
6. Keeping GUI objects info/declarations dynamically
or static.
7. Planned approach of unattended execution.

Also framework design differs by tool, scripting languages and type of AUT etc,. Now a days, people are using PERL, PYTHON, TCL/TK for CLI (Command Line Interface) automation. This design will vary much from GUI specific tool automation's design (QTP, Winrunner, Silktest,Robot and QARun etc).

You can go through following links. Each link is giving different set of components for Automation framework.
http://www.sqa-test.com/w_paper1.html
http://safsdev.sourceforge.net/FRAMESDataDrivenTestAutomationFrameworks.htm
http://www.qacity.com/NR/QACity/Documents/Automation_Framework.PDF
http://www.softwaredioxide.com/testing2004/papers/pankaj_cognizant.pdf
http://www.cbueche.de/FRM_DOC/webhelp_eng/Basic_Concepts.htm

I have developed different frameworks for Rational Visual Test, Winrunner and SilkTest. Some free frameworks are available freely on the net. EMOS is available for Winrunner and QTP. SAFS(http://safsdev.sourceforge.net) gives support for Winrunner and Robot. STAF(http://staf.sourceforge.net) will differ from SAFS.