Sunday, December 4, 2011

Skills for Performance Testing

Earlier I have written about Skills for Automation. Here I attempted to write about the skills for Performance Testing. Because performance testing is very dynamic and has vast areas in different technologies and concepts. It requires different level of of depth knowledge depends on project requirements and environment. It required different technical and non-technical stuff on the job.

Skills for Performance Testing

  • Analytical skills
  • Scripting
  • Operating System Concepts
  • Networking concepts
  • Memory Management concepts
  • Database Concepts
  • Hardware concepts
  • Protocols concepts
  • Debugging
  • Log Analysis
  • Documentation

Suggesting following links...
SQAForums's post - Need opinion from exp. load testers- skillset
Technical Skills For Performance Testers
What Skills Performance Testers Need and How to Get Them?

Saturday, November 26, 2011

JMeter - JDBC Driver issue

I was trying JDBC query execution through JMeter. I was getting the error like java.sql.SQLException: No suitable driver found . I have given ojdbc6.jar on classpath and installed Oracle client. Still JMeter has thrown error. The solution is, ojdbc6.jar should be copied under JMeter Lib folder (<JMeter installation directory>\lib).

Later, the same script was copied into a server and executed the script. Got the same error. Server does not have Oracle client installation and ojdbc6.jar in JMeter lib folder. Just copied the ojdbc6.jar and able to run the script successfully.
Note:
ojdbc6.jar should be used for Oracle 11 version. Also it is supported only Java1.5 and above.

Sunday, October 2, 2011

FOR loops & IF condition in RobotFramework

Using Robot Framework, test engineer can create FOR loops and IF conditions. I don't think so, any other Keyword driven framework is giving this kind of flexibility.

FOR loop can be set in two ways. First one is based on number of items in a List. Another one is based on range like from 1 to 50. Similarly keywords can be executed if condition matches or not. See below example.

FOR Loop & IF condition explantation using Robot Framework

TestCase Action Arguments
VerifyListItems [Arguments] ${Locator} @{ListItems}
[Documentation] Verifies the list of items present in a List Object.
: FOR ${Element} IN @{ListItems}
Select From List ${Locator} ${Element}
List Selection Should Be ${Locator} ${Element}
: FOR ${index} IN RANGE 50
${obj1}= Evaluate ${index}*4+3
Run Keyword if '@{tDriverData}[4]'=='2' Go To MyProfile
Run Keyword Unless '@{tDriverData}[4]'=='2' Go To UserCreation

Monday, August 15, 2011

Robotframework Installation

RobotFramework Installation is a tricky task. We need to install 6-7 utilities. All should be installed in the same order. Here I have given the instructions for Windows OS.

Tools installation
All RobotFramework related software can be the latest. The other software should be as the same version.

  1. Install Python (python-2.6.4.msi). Set installation directory as C:\Python26
  2. Install Robot Framework (robotframework-2.6win32.exe)
  3. Install Robot Selenium Library (robotframework-seleniumlibrary-2.7.win32.exe)
  4. Install Python Widgets (wxPython2.8-win32-unicode-2.8.11.0-py26.exe)
  5. Install Robot IDE (robotide-0.31.win32.exe)
  6. Install Python Win32 module (pywin32-214.win32-py2.6.exe)
  7. OPTIONAL (For SeleniumLibrary)- Install Java (jdk-6u14-windows-i586.exe), if your machine does not have Java.
  8. OPTIONAL (For SeleniumLibrary)- Extract Selenium RC Server (selenium-remote-control-1.0.3.zip)

Environment variables Configuration
To execute RIDE and Pybot, environment variables should set properly.
  1. Create a system level environment variable & value as --> PYTHONHOME=C:\Python26
  2. Create a system level environment variable & value as --> PYTHONPATH=C:\Python26\Lib;C:\Python26\Scripts;C:\Python26\Lib\site-packages;C:\Python26\Lib\site-packages\SeleniumLibrary;C:\Python26\Lib\site-packages\robotide\;C:\Python26\Lib\site-packages\robot\;C:\Python26\Lib\site-packages\wx-2.8-msw-unicode;
  3. Also add to PATH variable --> % PYTHONHOME%;% PYTHONHOME%\Scripts;

Steps for 64 Bit OS
To enable Robot Framework for 64 bit OS, you should install 64 bit python setup in same directory. Also the command prompt should be invoked through setting ‘Run as a Administrator’ if OS is Vista or Windows 7 version.

Verify the Installation
Open new command prompt and type ‘pybot --version’ and output like --> Robot Framework 2.6.0 (Python 2.6.4 on win32).

Tuesday, July 19, 2011

Virtualized Servers for Performance Testing

We have to do benchmark testing as well as stress testing. Already high performance system was purchased and virtual machines are created for performance testing setup. I can see few limitations for this approach, since production boxes all are in physical servers. Earlier I've used virtual machines for automation regression execution.

Issues/Limitations of Virtualized Servers

  1. if (production environment != load test environment) then result could be wrong
  2. Increasing the risk of deployment failure

Did a google search and found following links, which give all the details, including Vmware claims.

Virtualization: Performance testing

Measuring the Performance Impact of Virtualizing a Web Application Server

Response to 'Load Testing a Virtual Web Application'

Virtualization performance testing tips


Anybody have experienced load testing on Virtualized servers? Please share your experience...

Tuesday, July 12, 2011

RobotFramework - Overview

I have executed many automation projects and also developed various automation frameworks using different tools like Silktest, QTP, Selenium, CodedUI, MS UIA, VisualTest (dead) and Winrunner (dead). Tried few open source frameworks and found RobotFramework as the best one. Successfully completed five projects using RobotFramework. Main advantage is, I'm able to integrate with different test tools and libraries. It is neither related to IBM Rational tools or any other commercial tools.

Summary in RobotFramework Homepage
Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has easy-to-use tabular test data syntax and utilizes the keyword-driven testing approach. Its testing capabilities can be extended by test libraries implemented either with Python or Java, and users can create new keywords from existing ones using the same syntax that is used for creating test cases.

Robot Framework is open source software released under Apache License 2.0. Its copyrights are owned and development supported by Nokia Siemens Networks.

Robot Framework Features

  1. Advanced Keyword Driven framework
  2. Enable functional team contribution for automation
  3. Reusable/custom keywords
  4. Dynamic variables
  5. Easily extendable by Python, C# & Java
  6. Supports few test libraries including Selenium
  7. Provides simple and powerful Results Reporting
  8. Multiple Operating Systems support
  9. $ Free


Links for Robot Framework Resources
Robot Framework Home Page
Robot Framework User Guide
Robot Framework Quick Start Guide
Robot IDE
Robot Selenium Library

Robot Framework - Introduction
Robot Framework - Articles
Simple tutorial for Robot Framework

In last 6 months, many updates are happened on Robot framework and its related libraries. It is supporting many of my generic automation framework features.

Headers

Features

Robot Framework

Supported FrameworksFunctional DecompositionYes
Data DrivenYes
Keyword DrivenYes
Table DrivenYes
Independent toOperating SystemYes
ApplicationYes
TechnologyYes
Test ScriptingRecordingNo
ParametrizationYes
Suite Setup ModuleYes
Suite TearDown ModuleYes
Test Setup ModuleYes
Test TearDown ModuleYes
Database IntegrationYes
ScriptingYes
Environment LibraryYes
Test ExecutionMultiple OSYes
Multiple platformsYes
Parallel ExecutionNo
Test TimeoutYes
Based on PriorityYes
Grouping TestsYes
Running Failed testsYes
Result AnalysisDetail reporting for each step executionYes
Snapshots for failed testsYes
Test level reportingYes
Compiled reportYes
Results by categoryYes
Expected BenefitsLess effort to automateYes
Less dependence on highly skilled automation professionalsYes
Exception handlingYes
Easy to MaintainYes
Easy to Integrate with Other toolsYes
Support to CI (Continuous Integration) toolsYes
Additional ExpectationsDynamic conditionsYes
Dynamic Loop supportYes
UI to Edit dataYes
UI to execute testsYes
Update test metrics automaticallyYes
Update captured defects automaticallyYes

Sunday, July 10, 2011

JMeter Tip - javascript eval

For my jmeter script, I was trying to set a javascript statement for this formula --> pitia=${cpi}+${cti}+(${eshortage}/60)
I used the code like this --> ${__javaScript( eval ('${cpi}'.concat('+'\,'${cti}'\,'+'\,'${eshortage}'\,' /60' ) ) )}
The above one was working in a simple script and it is not worked in my business script.

In jmeter.log, I got the exception like below:
2011/07/07 17:30:30 ERROR - jmeter.functions.JavaScript: Error processing Javascript org.mozilla.javascript.EcmaError: SyntaxError: missing ; before statement (<cmd>#1(eval)#1)
at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3229)
at org.mozilla.javascript.DefaultErrorReporter.error(DefaultErrorReporter.java:78)
at org.mozilla.javascript.Parser.addError(Parser.java:126)
at org.mozilla.javascript.Parser.reportError(Parser.java:132)
at org.mozilla.javascript.Parser.statementHelper(Parser.java:1175)
at org.mozilla.javascript.Parser.statement(Parser.java:623)
at org.mozilla.javascript.Parser.parse(Parser.java:355)
at org.mozilla.javascript.Parser.parse(Parser.java:293)

The reason for failure is a simple space, which was given for variable 'cpi'. The jmeter error did not indicate the right reason. You should debug each variable.

Tuesday, May 31, 2011

Tuesday, April 26, 2011

What is OWASP

Four years back, I was looking help for security testing and I found OWASP with many presentations, books and testing guides. It helps me lot for my deliverables. OWASP's top 10 is a popular one and reflects the current trend.

OWASP by the numbers (Last year report)

  • 420,000 page views per month
  • 6,381 Articles
  • 15,000 downloads per month
  • 21,000 members on mailing lists
  • 2,600 wiki users
  • 1,500 wiki updates per month
  • 160 chapters worldwide
  • 75 individual memberships
  • 118 tool and documentation projects
  • 17 Books
  • 43 corporate/educational memberships
  • 7 Board members (Jeff, Tom, Dave, Seb. Dinis, Matt, Eoin)
  • 39 Committee Volunteers
  • 3 Employees (Paulo, Kate, Alison)
  • 25 projects funded

OWASP - Open Web Application Security Project
OWASP is a community of people passionate about application security. It is a non-profit(501c3 not-for-profit worldwide charitable organization), volunteer driven organization. All members are volunteers and all work is donated by sponsors. They all share a vision of a world where you can confidently trust the software you use. Unfortunately, the current software market doesn’t encourage security – that’s something they are trying to change. One of primary missions is to make application security visible so that people can make informed decisions about risk.

You can find lots of free and open source tools, documents, basic information, guidelines, presentations, video, and blogs at OWASP to help you get started.

  • Worldwide free and open community
  • Focused on improving the security of Web applications
  • Promotes secure software development
  • An open forum for discussion
  • Publications, Articles, Standards
  • Testing and Training Software
  • Local Chapters and Mailing Lists
  • Software libraries and tools

OWASP Software - WebGoat - Training application
WebGoat Project
  • Cross Site Scripting
  • SQL Injection Attacks
  • Thread Safety
  • Field & Parameter Manipulation
  • Session Hijacking and Management
  • Weak Authentication Mechanisms
  • Many more attacks added

OWASP Software - WebScarab - framework for analyzing HTTP/HTTPS traffic
WebScarab Project
  • Fragment Analysis – extract scripts and html as presented to the browser, instead of source code presented by the browser post render
  • Proxy – observe traffic between the browser and server, includes the ability to modify data in transit, expose hidden fields, and perform bandwidth manipulation
  • BeanShell – the ability to execute Java code on requests and responses before being transmitted between the browser and server; allows runtime extension of WebScarab
  • Spider – identifies new URLs within each page viewed
  • SessionID Analysis – Collection and analysis of cookies to determine predictability of session tokens

My Previous posts on Security
Security Attacks - OWASP Top 10
Security Testing - Webscarab tool
Security Testing - CSS or XSS

Thursday, March 31, 2011

Security Attacks - OWASP Top 10

I was looking into cross site scripting for security testing. Look at OWASP top 10 attacks for year 2010, 2007, 2004. Few items have gone up or down, added and removed.

Year 2010 - Top 10

  1. Injection
  2. Cross-Site Scripting (XSS)
  3. Broken Authentication and Session Management
  4. Insecure Direct Object References
  5. Cross-Site Request Forgery (CSRF)
  6. Security Misconfiguration
  7. Insecure Cryptographic Storage
  8. Failure to Restrict URL Access
  9. Insufficient Transport Layer Protection
  10. Unvalidated Redirects and Forwards

Year 2007 - Top 10
  1. Cross Site Scripting (XSS)
  2. Injection Flaws
  3. Malicious File Execution
  4. Insecure Direct Object Reference
  5. Cross Site Request Forgery (CSRF)
  6. Information Leakage and Improper Error Handling
  7. Broken Authentication and Session Management
  8. Insecure Cryptographic Storage
  9. Insecure Communications
  10. Failure to Restrict URL Access

Year 2004 - Top 10
  1. Unvalidated Input
  2. Broken Access Control
  3. Broken Account and Session Management
  4. Cross Site Scripting (XSS) Flaws
  5. Buffer Overflows
  6. Injection Flaws
  7. Improper Error Handling
  8. Insecure Storage
  9. Denial of Service
  10. Insecure configuration management

Monday, February 28, 2011

TaskManager - Tiny Footprint mode

Sometimes back, My laptop did not show menu bar and tabs in Task Manager. I restarted the laptop twice and no success. I searched Microsoft knowledge base and got to know about an interesting thing.

Microsoft KB - Task Manager Menu Bar and Tabs Are Not Visible in Windows XP

Problem
This behavior may occur if Task Manager is running in Tiny Footprint mode.

Resolution
To switch Task Manager to its typical display mode, double-click the top border of the window or the empty space in the border.

Friday, January 28, 2011

Generic Test Automation Framework

So far, I have developed various automation frameworks and recently using Robot Framework. Listed many features for Ideal Automation Framework and given below. Also would like to get more features from other users.

Reference to earlier posts and Links about automation framework.
Wikipedia - Test Automation
Wikipedia - KeyWord Driven Testing
Is KeyWord Driven Framework enough?
My Thoughts about Automation
What is an Automation Framework - from Dhanasekar's blog

Features for Generic and common Test Automation Framework

Headers

Features

Supported FrameworksFunctional Decomposition
Data Driven
Keyword Driven
Tabel Driven
Independent toOperating System
Application
Technology
Test ScriptingRecording
Parameterization
Suite Setup Module
Suite TearDown Module
Test Setup Module
Test TearDown Module
Database Integration
Scripting
Environment Library
Test ExecutionMultiple OS
Multiple platforms
Parallel Execution
Test Timeout
Based on Priority
Grouping Tests
Running Failed tests
Result AnalysisDetail reporting for each step execution
Snapshots for failed tests
Test level reporting
Compiled report
Results by category
Expected BenefitsLess effort to automate
Less dependence on highly skilled automation professionals
Exception handling
Easy to Maintain
Easy to Integrate with Other tools
Support to CI (Continuous Integration) tools
Additional ExpectationsDynamic conditions
Dynamic Loop support
UI to Edit data
UI to execute tests
Update test metrics automatically
Update captured defects automatically

Tuesday, January 25, 2011

Stop Worrying about Weaknesses

Got a interesting email last week. I hope, it would be useful for all...



Little Johnny comes home one day, looks down at his feet, and gives you his report card. You smile at him as you open it up and look inside. Then your smile disappears when you see the F in math. You also see an A (English) and two Bs (history and science). You look down at little Johnny and ask, "What happened in math, Johnny? Why did you get this F?"

We want our kids to be successful at everything they do. And if they're not good at something, we ask why they failed. We tell them to work harder at it. Understand what went wrong, focus, and fix it.

But that's a mistake. The wrong focus. If you dwell on Johnny's failure, on his weakness, you'll be setting him up for a life of struggle and low self-esteem while reducing his chances of reaching his full potential.

Also, as a side note, you won't fix his weakness. You'll just reinforce it.

The problem with a report card is that it measures all students against the same criteria, which ignores that each student is different with unique talents, distinct likes and dislikes, and particular aspirations. And when we see the F on little Johnny's report card, it's easy for us to get distracted from our primary job: to help him deeply enjoy his life and fulfill his potential by developing and deriving pleasure from his unique talents.

Fast forward 20 years. Little Johnny is now big John. As he sits down for a performance review with his manager, she spends a few quiet minutes looking over his review and then raises her eyes to meet his.

"You've worked hard this year John. Your client orientation is superb. You've met your sales goals and you're a solid team player. But you have an area that needs development, specifically, your detail orientation. The spreadsheets we get from you are a mess. Let's talk about how you can get better in that."
An A, two Bs and an F. And his manager handles it the same way his parent did. By focusing the conversation, and John's effort, on his least favorite and weakest area.

We have a report card problem in our companies and it's costing us a tremendous amount of time, money, potential, and happiness. It's costing us talent.
Traditional management systems encourage mediocrity in everything and excellence in nothing. Most performance review systems set an ideal picture of how we want everyone to act (standards, competencies, etc.) and then assesses how closely people match that ideal, nudging them to improve their weaknesses so they "meet or exceed expectations" in every area.

But how will John add the most value to his organization? He's amazing with people, not spreadsheets. He'll work hardest, derive the most pleasure, and contribute his maximum potential with the greatest result if he is able to focus as much time as possible in his area of strength. Which means taking his focus off developing the things in which he's weak. They're just a distraction.

Here's what his manager should say: "You've worked hard this year John. Your client orientation is superb. You've met your sales goals and you're a solid team player. But working on those spreadsheets isn't a good use of your time and it's not your strength. I'm going to ask David to do those for you from now on. He loves spreadsheets and is great at them. I want to spend the rest of our time talking about how you can get even better at working with your clients. That's where you shine — where you add the most value to the company — and you seem to really enjoy it."

An organization should be a platform for unique talent. A performance review system should be flexible enough to reflect and reward the successful contributions of diverse employees. Let's encourage people to be weak in areas in which they are average — because no one can possibly be great at everything — and place all our effort on developing their strengths further.

If it's impossible to take away the part of their job in which they're weak, then help them improve just enough so that it doesn't get in the way of their strength. If you can't take the spreadsheets away from John, help him get a C and move on. That would be preferable to spending the time and effort it would take to get an A or even a B.

Next time little Johnny hands you his report card with an F in math and an A in English, keep smiling and resist the temptation to ask about the F. Instead, ask about the A first. "What happened in English?" you should say to Johnny, "Why did you get this A?" Then let him tell you about how and why he succeeds. What is it about the work that excites him? What about the teacher? How did he study?

Then, if you want him to get a little better in math, you can help him recreate the conditions that led to his success in English.

And when you're done with the report card conversation, it might still be a good idea to get him a math tutor. Because school is about exposure to everything while business is about success in something.

And then, if you want to teach him to harness his particular path to success, make sure to get him an English tutor too.

Friday, January 21, 2011

Silktest2010R2

Silktest 2010R2 has been released in this month and Release notes - SilkTest_ReleaseNotes_en.pdf

Enhancements
Microfocus has improved the support for 64 bit Operating Systems and improved the features for Silk4NET and Silk4J Plugins.

  • Java AWT/Swing Application and Applet Record and Replay Support
  • Adobe Flex Version 4.x Support
  • 64-bit Support for .NET and Windows API-based Applications
  • Default Recording Mode for xBrowser Applications is Low-Level Native User Input
  • Dynamically Invoke Methods
  • SilkTest Workbench - Importing and Exporting Assets, ObjectMaps, Embedded Scripting Language & Integration with Additional Micro Focus Products
  • SilkTest Classic - Dynamic Object Recognition Supports Recording Locator Keywords

End-Of-Life (EOL) Components
I am wondering after seeing MSUIA technology in EOL list. Without that I'm sure that no body can support silverlight automation. Are they serious about this?
  • Adobe Air 1.x
  • Classic OCR
  • Firefox 3.0
  • IBM JRE 1.5
  • Java version 1.5
  • MSUIA technology domain (the WPF technology domain is still supported)

This time, Microfocus has developed few more documents and I like the effort for Migrating from the SilkTest Classic Agent to the Open Agent. More details from below links:
Silktest 2010R2 Release materials
Silktest 2010R2 Classic Documentation