Sunday, February 12, 2012

RobotFramework Standards

RobotFramework team has given guidelines --> HowToWriteGoodTestCasesWithExamples. Similar to coding standards, Automation team should use certain guidelines for Keyword-Driven framework also. It would be more useful to functional team as well as other teams.

Test Suite / Script

  • Test Script names should be less than 20 characters and file type should be HTML.
  • It should be easily readable and self-explanatory.
  • Remember that suite names are created automatically from file/directory names. Extensions are stripped, underscores are converted to spaces and, if name is all lower case, words are capitalized. For example login_tests.html -> Login Tests and DHCP_and_DNS -> DHCP and DNS.
  • Documentation should be updated with purpose of the script and pre-conditions.
  • Proper keywords should be given for Suite Setup, Suite Teardown, Test Setup and Test Teardown.
  • Should not have too many tests (max 50) in one suite unless they are data-driven.

TestCases / Tests
  • Test case names should be less than 40 characters and file type should be HTML.
  • Test case Name should be in camel case (In a word, First letter is capital and remaining letters are in small).
  • It should be easily readable and self explanatory.
  • Documentation should be updated with manual test case steps, notes and pre-conditions.
  • Proper keywords should be given for Suite Setup, Suite Teardown, Test Setup and Test Teardown.
  • Appropriate tags should be given for each case.
  • Tests should be independent.
  • In dependant tests, detailed notes should be given. Consider verifying the status of the previous test using ${PREV TEST STATUS} variable.
  • Hard coding of object name should be avoided.
  • Should contain many high level keywords instead of repeating steps often.
  • High level keywords should be used for navigation.
  • Local variables should have prefix char ‘t’ as temporary variables

Resources
  • Keep all resource files in single folder.
  • Resource file names should be less than 20 characters and file type should be HTML.
  • All characters should be small characters.
  • Documentation should be updated with purpose.
  • All constants should be maintained in separate resource file.
  • Keep separate resource files for application’s data.
  • Keep separate resource files for all GUI objects page wise or module-wise.
  • Group the high level keywords by categories like business logic, module and general.

High Level Keyword / User Keyword / Function
  • Function names should be less than 35 characters.
  • It should be easily readable and self explanatory.
  • For easy readability, it should be in camel case (Capital & Small letters).
  • Prefixes are sometimes useful. For example, Is - to ask a question about something, Get - get a value, Set - set a value
  • Can have space for better readability.
  • Documentation should have clear details for purpose, variables and returning values.
  • Hard coding of object name should be avoided.
  • Arguments should have char ‘p’ as prefix and returned variable should have char ‘r’ as prefix.
  • Local variables should have prefix char ‘t’ as temporary variables
  • Duplicate Functions should not be added.
  • Can contain some programming logic (for loops, if/else)
  • Complex logic in libraries rather than in user keywords
  • Important variables can have comments on the RHS

Variables
  • Variable names should be less than 20 characters.
  • Variables should have meaningful words
  • For easy readability, it should be in camel case (Capital & Small letters).
  • Local variables should have prefix char ‘t’ as temporary variables
  • Arguments should have char ‘p’ as prefix and returned variable should have char ‘r’ as prefix.
  • GUI object variables should have char ‘o’ as prefix.
  • Constant variables should be in Capital letters. For example, APP_URL, DB_SERVER. All other type of variables should have mixed style (small and capital) characters.
  • Script/Global variables should be defined at the top of the script.
  • Function / test case level variables should be defined at top of the function.
  • Can have space. But try to restrict for minimum

Monday, January 30, 2012

Performance Testing Guidance from Microsoft

I always admire Microsoft for its documentation. Right from OS to Office applications, you can find good documentation. Microsoft has shared very good articles for performance testing and tuning.

You can find more articles related to Performance testing available from
Patterns & Practices: Performance Testing Guidance
. Just listed only How-TOs part. Thanks to Microsoft!!!

Performance Testing


Capacity

Load Testing

Stress Testing

Test Cases/Scripting

Troubleshooting

Tuning

Workload Modeling

Hope all these links would be useful..

Sunday, January 29, 2012

JVM Monitoring

Couple of our applications are using Tomcat server. JMeter was used as load testing tool. To monitor Java memory, I used two options. One is JMX(Java Management Extensions) console and Psi-Probe.

To implement those options, you should add following entries into catalina.sh

set CATALINA_OPTS=-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9005 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false

To access JMX console,Enter command ==> [java_installation]\bin\jconsole hostname:port
To use Probe, you need to deploy in tomcat server instance and then access the probe application.

Sunday, December 4, 2011

Skills for Performance Testing

Earlier I have written about Skills for Automation. Here I attempted to write about the skills for Performance Testing. Because performance testing is very dynamic and has vast areas in different technologies and concepts. It requires different level of of depth knowledge depends on project requirements and environment. It required different technical and non-technical stuff on the job.

Skills for Performance Testing

  • Analytical skills
  • Scripting
  • Operating System Concepts
  • Networking concepts
  • Memory Management concepts
  • Database Concepts
  • Hardware concepts
  • Protocols concepts
  • Debugging
  • Log Analysis
  • Documentation

Suggesting following links...
SQAForums's post - Need opinion from exp. load testers- skillset
Technical Skills For Performance Testers
What Skills Performance Testers Need and How to Get Them?

Saturday, November 26, 2011

JMeter - JDBC Driver issue

I was trying JDBC query execution through JMeter. I was getting the error like java.sql.SQLException: No suitable driver found . I have given ojdbc6.jar on classpath and installed Oracle client. Still JMeter has thrown error. The solution is, ojdbc6.jar should be copied under JMeter Lib folder (<JMeter installation directory>\lib).

Later, the same script was copied into a server and executed the script. Got the same error. Server does not have Oracle client installation and ojdbc6.jar in JMeter lib folder. Just copied the ojdbc6.jar and able to run the script successfully.
Note:
ojdbc6.jar should be used for Oracle 11 version. Also it is supported only Java1.5 and above.

Sunday, October 2, 2011

FOR loops & IF condition in RobotFramework

Using Robot Framework, test engineer can create FOR loops and IF conditions. I don't think so, any other Keyword driven framework is giving this kind of flexibility.

FOR loop can be set in two ways. First one is based on number of items in a List. Another one is based on range like from 1 to 50. Similarly keywords can be executed if condition matches or not. See below example.

FOR Loop & IF condition explantation using Robot Framework

TestCase Action Arguments
VerifyListItems [Arguments] ${Locator} @{ListItems}
[Documentation] Verifies the list of items present in a List Object.
: FOR ${Element} IN @{ListItems}
Select From List ${Locator} ${Element}
List Selection Should Be ${Locator} ${Element}
: FOR ${index} IN RANGE 50
${obj1}= Evaluate ${index}*4+3
Run Keyword if '@{tDriverData}[4]'=='2' Go To MyProfile
Run Keyword Unless '@{tDriverData}[4]'=='2' Go To UserCreation

Monday, August 15, 2011

Robotframework Installation

RobotFramework Installation is a tricky task. We need to install 6-7 utilities. All should be installed in the same order. Here I have given the instructions for Windows OS.

Tools installation
All RobotFramework related software can be the latest. The other software should be as the same version.

  1. Install Python (python-2.6.4.msi). Set installation directory as C:\Python26
  2. Install Robot Framework (robotframework-2.6win32.exe)
  3. Install Robot Selenium Library (robotframework-seleniumlibrary-2.7.win32.exe)
  4. Install Python Widgets (wxPython2.8-win32-unicode-2.8.11.0-py26.exe)
  5. Install Robot IDE (robotide-0.31.win32.exe)
  6. Install Python Win32 module (pywin32-214.win32-py2.6.exe)
  7. OPTIONAL (For SeleniumLibrary)- Install Java (jdk-6u14-windows-i586.exe), if your machine does not have Java.
  8. OPTIONAL (For SeleniumLibrary)- Extract Selenium RC Server (selenium-remote-control-1.0.3.zip)

Environment variables Configuration
To execute RIDE and Pybot, environment variables should set properly.
  1. Create a system level environment variable & value as --> PYTHONHOME=C:\Python26
  2. Create a system level environment variable & value as --> PYTHONPATH=C:\Python26\Lib;C:\Python26\Scripts;C:\Python26\Lib\site-packages;C:\Python26\Lib\site-packages\SeleniumLibrary;C:\Python26\Lib\site-packages\robotide\;C:\Python26\Lib\site-packages\robot\;C:\Python26\Lib\site-packages\wx-2.8-msw-unicode;
  3. Also add to PATH variable --> % PYTHONHOME%;% PYTHONHOME%\Scripts;

Steps for 64 Bit OS
To enable Robot Framework for 64 bit OS, you should install 64 bit python setup in same directory. Also the command prompt should be invoked through setting ‘Run as a Administrator’ if OS is Vista or Windows 7 version.

Verify the Installation
Open new command prompt and type ‘pybot --version’ and output like --> Robot Framework 2.6.0 (Python 2.6.4 on win32).

Tuesday, July 19, 2011

Virtualized Servers for Performance Testing

We have to do benchmark testing as well as stress testing. Already high performance system was purchased and virtual machines are created for performance testing setup. I can see few limitations for this approach, since production boxes all are in physical servers. Earlier I've used virtual machines for automation regression execution.

Issues/Limitations of Virtualized Servers

  1. if (production environment != load test environment) then result could be wrong
  2. Increasing the risk of deployment failure

Did a google search and found following links, which give all the details, including Vmware claims.

Virtualization: Performance testing

Measuring the Performance Impact of Virtualizing a Web Application Server

Response to 'Load Testing a Virtual Web Application'

Virtualization performance testing tips


Anybody have experienced load testing on Virtualized servers? Please share your experience...

Tuesday, July 12, 2011

RobotFramework - Overview

I have executed many automation projects and also developed various automation frameworks using different tools like Silktest, QTP, Selenium, CodedUI, MS UIA, VisualTest (dead) and Winrunner (dead). Tried few open source frameworks and found RobotFramework as the best one. Successfully completed five projects using RobotFramework. Main advantage is, I'm able to integrate with different test tools and libraries. It is neither related to IBM Rational tools or any other commercial tools.

Summary in RobotFramework Homepage
Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has easy-to-use tabular test data syntax and utilizes the keyword-driven testing approach. Its testing capabilities can be extended by test libraries implemented either with Python or Java, and users can create new keywords from existing ones using the same syntax that is used for creating test cases.

Robot Framework is open source software released under Apache License 2.0. Its copyrights are owned and development supported by Nokia Siemens Networks.

Robot Framework Features

  1. Advanced Keyword Driven framework
  2. Enable functional team contribution for automation
  3. Reusable/custom keywords
  4. Dynamic variables
  5. Easily extendable by Python, C# & Java
  6. Supports few test libraries including Selenium
  7. Provides simple and powerful Results Reporting
  8. Multiple Operating Systems support
  9. $ Free


Links for Robot Framework Resources
Robot Framework Home Page
Robot Framework User Guide
Robot Framework Quick Start Guide
Robot IDE
Robot Selenium Library

Robot Framework - Introduction
Robot Framework - Articles
Simple tutorial for Robot Framework

In last 6 months, many updates are happened on Robot framework and its related libraries. It is supporting many of my generic automation framework features.

Headers

Features

Robot Framework

Supported FrameworksFunctional DecompositionYes
Data DrivenYes
Keyword DrivenYes
Table DrivenYes
Independent toOperating SystemYes
ApplicationYes
TechnologyYes
Test ScriptingRecordingNo
ParametrizationYes
Suite Setup ModuleYes
Suite TearDown ModuleYes
Test Setup ModuleYes
Test TearDown ModuleYes
Database IntegrationYes
ScriptingYes
Environment LibraryYes
Test ExecutionMultiple OSYes
Multiple platformsYes
Parallel ExecutionNo
Test TimeoutYes
Based on PriorityYes
Grouping TestsYes
Running Failed testsYes
Result AnalysisDetail reporting for each step executionYes
Snapshots for failed testsYes
Test level reportingYes
Compiled reportYes
Results by categoryYes
Expected BenefitsLess effort to automateYes
Less dependence on highly skilled automation professionalsYes
Exception handlingYes
Easy to MaintainYes
Easy to Integrate with Other toolsYes
Support to CI (Continuous Integration) toolsYes
Additional ExpectationsDynamic conditionsYes
Dynamic Loop supportYes
UI to Edit dataYes
UI to execute testsYes
Update test metrics automaticallyYes
Update captured defects automaticallyYes

Sunday, July 10, 2011

JMeter Tip - javascript eval

For my jmeter script, I was trying to set a javascript statement for this formula --> pitia=${cpi}+${cti}+(${eshortage}/60)
I used the code like this --> ${__javaScript( eval ('${cpi}'.concat('+'\,'${cti}'\,'+'\,'${eshortage}'\,' /60' ) ) )}
The above one was working in a simple script and it is not worked in my business script.

In jmeter.log, I got the exception like below:
2011/07/07 17:30:30 ERROR - jmeter.functions.JavaScript: Error processing Javascript org.mozilla.javascript.EcmaError: SyntaxError: missing ; before statement (<cmd>#1(eval)#1)
at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3229)
at org.mozilla.javascript.DefaultErrorReporter.error(DefaultErrorReporter.java:78)
at org.mozilla.javascript.Parser.addError(Parser.java:126)
at org.mozilla.javascript.Parser.reportError(Parser.java:132)
at org.mozilla.javascript.Parser.statementHelper(Parser.java:1175)
at org.mozilla.javascript.Parser.statement(Parser.java:623)
at org.mozilla.javascript.Parser.parse(Parser.java:355)
at org.mozilla.javascript.Parser.parse(Parser.java:293)

The reason for failure is a simple space, which was given for variable 'cpi'. The jmeter error did not indicate the right reason. You should debug each variable.