Virtualized Servers for Performance Testing

We have to do benchmark testing as well as stress testing. Already high performance system was purchased and virtual machines are created for performance testing setup. I can see few limitations for this approach, since production boxes all are in physical servers. Earlier I've used virtual machines for automation regression execution.

Issues/Limitations of Virtualized Servers

1. if (production environment != load test environment) then result could be wrong
2. Increasing the risk of deployment failure

Did a google search and found following links, which give all the details, including Vmware claims.

Virtualization: Performance testing

Measuring the Performance Impact of Virtualizing a Web Application Server

Response to 'Load Testing a Virtual Web Application'

Virtualization performance testing tips


Anybody have experienced load testing on Virtualized servers? Please share your experience...

RobotFramework - Overview

I have executed many automation projects and also developed various automation frameworks using different tools like Silktest, QTP, Selenium, CodedUI, MS UIA, VisualTest (dead) and Winrunner (dead). Tried few open source frameworks and found RobotFramework as the best one. Successfully completed five projects using RobotFramework. Main advantage is, I'm able to integrate with different test tools and libraries. It is neither related to IBM Rational tools or any other commercial tools.

Summary in RobotFramework Homepage
Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has easy-to-use tabular test data syntax and utilizes the keyword-driven testing approach. Its testing capabilities can be extended by test libraries implemented either with Python or Java, and users can create new keywords from existing ones using the same syntax that is used for creating test cases.

Robot Framework is open source software released under Apache License 2.0. Its copyrights are owned and development supported by Nokia Siemens Networks.

Robot Framework Features

1. Advanced Keyword Driven framework
2. Enable functional team contribution for automation
3. Reusable/custom keywords
4. Dynamic variables
5. Easily extendable by Python, C# & Java
6. Supports few test libraries including Selenium
7. Provides simple and powerful Results Reporting
8. Multiple Operating Systems support
9. $ Free

Links for Robot Framework Resources
Robot Framework Home Page
Robot Framework User Guide
Robot Framework Quick Start Guide
Robot IDE
Robot Selenium Library

Robot Framework - Introduction
Robot Framework - Articles
Simple tutorial for Robot Framework

In last 6 months, many updates are happened on Robot framework and its related libraries. It is supporting many of my generic automation framework features.

Headers	Features	Robot Framework
Supported Frameworks	Functional Decomposition	Yes
	Data Driven	Yes
	Keyword Driven	Yes
	Table Driven	Yes
Independent to	Operating System	Yes
	Application	Yes
	Technology	Yes
Test Scripting	Recording	No
	Parametrization	Yes
	Suite Setup Module	Yes
	Suite TearDown Module	Yes
	Test Setup Module	Yes
	Test TearDown Module	Yes
	Database Integration	Yes
	Scripting	Yes
	Environment Library	Yes
Test Execution	Multiple OS	Yes
	Multiple platforms	Yes
	Parallel Execution	No
	Test Timeout	Yes
	Based on Priority	Yes
	Grouping Tests	Yes
	Running Failed tests	Yes
Result Analysis	Detail reporting for each step execution	Yes
	Snapshots for failed tests	Yes
	Test level reporting	Yes
	Compiled report	Yes
	Results by category	Yes
Expected Benefits	Less effort to automate	Yes
	Less dependence on highly skilled automation professionals	Yes
	Exception handling	Yes
	Easy to Maintain	Yes
	Easy to Integrate with Other tools	Yes
	Support to CI (Continuous Integration) tools	Yes
Additional Expectations	Dynamic conditions	Yes
	Dynamic Loop support	Yes
	UI to Edit data	Yes
	UI to execute tests	Yes
	Update test metrics automatically	Yes
	Update captured defects automatically	Yes

JMeter Tip - javascript eval

For my jmeter script, I was trying to set a javascript statement for this formula --> pitia=${cpi}+${cti}+(${eshortage}/60)
I used the code like this --> ${__javaScript( eval ('${cpi}'.concat('+'\,'${cti}'\,'+'\,'${eshortage}'\,' /60' ) ) )}
The above one was working in a simple script and it is not worked in my business script.

In jmeter.log, I got the exception like below:
2011/07/07 17:30:30 ERROR - jmeter.functions.JavaScript: Error processing Javascript org.mozilla.javascript.EcmaError: SyntaxError: missing ; before statement (<cmd>#1(eval)#1)
at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3229)
at org.mozilla.javascript.DefaultErrorReporter.error(DefaultErrorReporter.java:78)
at org.mozilla.javascript.Parser.addError(Parser.java:126)
at org.mozilla.javascript.Parser.reportError(Parser.java:132)
at org.mozilla.javascript.Parser.statementHelper(Parser.java:1175)
at org.mozilla.javascript.Parser.statement(Parser.java:623)
at org.mozilla.javascript.Parser.parse(Parser.java:355)
at org.mozilla.javascript.Parser.parse(Parser.java:293)

The reason for failure is a simple space, which was given for variable 'cpi'. The jmeter error did not indicate the right reason. You should debug each variable.