Tuesday, July 19, 2011

Virtualized Servers for Performance Testing

We have to do benchmark testing as well as stress testing. Already high performance system was purchased and virtual machines are created for performance testing setup. I can see few limitations for this approach, since production boxes all are in physical servers. Earlier I've used virtual machines for automation regression execution.

Issues/Limitations of Virtualized Servers

  1. if (production environment != load test environment) then result could be wrong
  2. Increasing the risk of deployment failure

Did a google search and found following links, which give all the details, including Vmware claims.

Virtualization: Performance testing

Measuring the Performance Impact of Virtualizing a Web Application Server

Response to 'Load Testing a Virtual Web Application'

Virtualization performance testing tips

Anybody have experienced load testing on Virtualized servers? Please share your experience...

Tuesday, July 12, 2011

RobotFramework - Overview

I have executed many automation projects and also developed various automation frameworks using different tools like Silktest, QTP, Selenium, CodedUI, MS UIA, VisualTest (dead) and Winrunner (dead). Tried few open source frameworks and found RobotFramework as the best one. Successfully completed five projects using RobotFramework. Main advantage is, I'm able to integrate with different test tools and libraries. It is neither related to IBM Rational tools or any other commercial tools.

Summary in RobotFramework Homepage
Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has easy-to-use tabular test data syntax and utilizes the keyword-driven testing approach. Its testing capabilities can be extended by test libraries implemented either with Python or Java, and users can create new keywords from existing ones using the same syntax that is used for creating test cases.

Robot Framework is open source software released under Apache License 2.0. Its copyrights are owned and development supported by Nokia Siemens Networks.

Robot Framework Features

  1. Advanced Keyword Driven framework
  2. Enable functional team contribution for automation
  3. Reusable/custom keywords
  4. Dynamic variables
  5. Easily extendable by Python, C# & Java
  6. Supports few test libraries including Selenium
  7. Provides simple and powerful Results Reporting
  8. Multiple Operating Systems support
  9. $ Free

Links for Robot Framework Resources
Robot Framework Home Page
Robot Framework User Guide
Robot Framework Quick Start Guide
Robot IDE
Robot Selenium Library

Robot Framework - Introduction
Robot Framework - Articles
Simple tutorial for Robot Framework

In last 6 months, many updates are happened on Robot framework and its related libraries. It is supporting many of my generic automation framework features.



Robot Framework

Supported FrameworksFunctional DecompositionYes
Data DrivenYes
Keyword DrivenYes
Table DrivenYes
Independent toOperating SystemYes
Test ScriptingRecordingNo
Suite Setup ModuleYes
Suite TearDown ModuleYes
Test Setup ModuleYes
Test TearDown ModuleYes
Database IntegrationYes
Environment LibraryYes
Test ExecutionMultiple OSYes
Multiple platformsYes
Parallel ExecutionNo
Test TimeoutYes
Based on PriorityYes
Grouping TestsYes
Running Failed testsYes
Result AnalysisDetail reporting for each step executionYes
Snapshots for failed testsYes
Test level reportingYes
Compiled reportYes
Results by categoryYes
Expected BenefitsLess effort to automateYes
Less dependence on highly skilled automation professionalsYes
Exception handlingYes
Easy to MaintainYes
Easy to Integrate with Other toolsYes
Support to CI (Continuous Integration) toolsYes
Additional ExpectationsDynamic conditionsYes
Dynamic Loop supportYes
UI to Edit dataYes
UI to execute testsYes
Update test metrics automaticallyYes
Update captured defects automaticallyYes

Sunday, July 10, 2011

JMeter Tip - javascript eval

For my jmeter script, I was trying to set a javascript statement for this formula --> pitia=${cpi}+${cti}+(${eshortage}/60)
I used the code like this --> ${__javaScript( eval ('${cpi}'.concat('+'\,'${cti}'\,'+'\,'${eshortage}'\,' /60' ) ) )}
The above one was working in a simple script and it is not worked in my business script.

In jmeter.log, I got the exception like below:
2011/07/07 17:30:30 ERROR - jmeter.functions.JavaScript: Error processing Javascript org.mozilla.javascript.EcmaError: SyntaxError: missing ; before statement (<cmd>#1(eval)#1)
at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3229)
at org.mozilla.javascript.DefaultErrorReporter.error(DefaultErrorReporter.java:78)
at org.mozilla.javascript.Parser.addError(Parser.java:126)
at org.mozilla.javascript.Parser.reportError(Parser.java:132)
at org.mozilla.javascript.Parser.statementHelper(Parser.java:1175)
at org.mozilla.javascript.Parser.statement(Parser.java:623)
at org.mozilla.javascript.Parser.parse(Parser.java:355)
at org.mozilla.javascript.Parser.parse(Parser.java:293)

The reason for failure is a simple space, which was given for variable 'cpi'. The jmeter error did not indicate the right reason. You should debug each variable.