Friday, May 5, 2017

Big Data Testing - Part 1

Today many enterprise companies have flooded with data. Current technological advancements have led to overwhelming amount of data from distinctive domains like internet of things (sensors), health care, ecommerce and financial companies. The term big data was coined to capture the meaning of this emerging trend.

Big Data is helpful to take better business decisions on different kind of business. The data should not contain any bad data and incomplete data. It has different analytics for variety of business needs. It is used in customer segmentation, Product performance, fraud detection, social media analytics, sentimental analysis, predictive analytics, Financial risk analysis etc.

Big Data Ecosystem

A generic big data implementation has four stages:
  1. Capture Data from multiple sources (Data Ingestion)
  2. Storage for huge data
  3. Data Processing (Loading and transformation)
  4. Analyze layer (Analytics & Reports)

Like components ecosystem, the big data system can be decomposed into a layered structure and divisible into three layers, i.e., the infrastructure layer, the computing layer, and the application layer, from bottom to top. This layered view only provides a conceptual hierarchy to underscore the complexity of a big data system.

Big Data Testing Approach
Bad data can cause great difficulties for analysis and make decisions. Also, Poor implementation can lead to poor quality, delays in testing and increased cost. The right test strategy should ensure both functional testing and non-functional testing.

Test approach can be varied based on type of data processing like Stream processing and batch processing and type of analytics solution used. If big data implementation has less stages or few components and Testing techniques also will be reduces based on components availability.

Functional Testing
  • Data Flow Validation
  • Data Integrity
  • Data Ingestion Layer validation
  • Data Storage Layer
  • Data Processing Layer
  • Analytics & Reports Layer

Non-Functional Testing
  • Data Quality Monitoring
  • Infrastructure
  • Data Security
  • Performance Benchmarking
  • Fault Tolerance
  • Fail Over Mechanism

Will explain each of testing in next post.

Saturday, March 25, 2017

Tools learning for testing

Many times folks used to ask, what tools would be good for testing career. Used to advice to learn popular tools (mostly commercial tools) based on their coding skills and interest. But this scenario is changed in last few years. Testers should be ready to handle the automation based on business needs, rather than just toolsmiths. Also mindset change should be there, instead of sticking with just one tool and one scripting language

Read an article recently in this line and few excerpts..

It’s important to ask yourself, why do you want to learn about a specific tool or set of tools? They are an integral part of software testing, supporting what we do on a daily basis. However, having knowledge of a set of tools is less important than knowing when and where to use the right tool. An effective tester will also have the skill to know when and where to use the right tool because the problem requires it, not because the job market determines it desirable. Using the wrong tool at the wrong time can have adverse effects on your day-to-day work, potentially slowing you down, or worse, giving you biased or flat out false information. That’s not to say you shouldn’t explore new tools, but always remember:

Problem first, tools second
With this concept in mind, let’s explore different categories of tools. Some examples will be offered but they are by no means an exhaustive list. I encourage you to build up a toolbox or reference list of tools you have used in the past.

Original Article - What Tools Should I Learn?

Sunday, March 19, 2017

Handling multi-dimensional array in vbscript

Was trying to build excel data provider, which can be used for Test Complete and QTP scripts. Created a VBScript to keep all excel contents into a multi-dimensional array.

Declared array with lower limits and then used ReDim to fix with right count. But I was able to retrieve the data only upto the initial array size.

Tried different ways. Then declared array without any size as given in below code.

'------------------------------------------------------------------------- ' File : ExcelReadByArrays.vbs ' Author : PalaniSelvam Thillaiappan ' Purpose : To read the given sheet contents . '------------------------------------------------------------------------- '' Usage '' cscript D:\MyCode\qtp_uft\ExcelReadByArrays.vbs '' test data: D:\MyCode\qtp_uft\testMyApp_1.xlsx '******** Variables Declaration Dim gsExcelFile, giStartRow, giStartCol, giEndRow, giEndCol, giSheetIndex Dim gsLogFile Dim arrExcelData() 'as all Rows data '''Dim arrExcelData(5, 5) 'as all Rows data giSheetIndex = 1 gsExcelFile = "D:\MyCode\qtp_uft\testMyApp_1.xlsx" '' Working fine, but array size should be given properly ExcelDataRead gsExcelFile, giSheetIndex, arrExcelData PrintArrays arrExcelData '-------------------------------------- ' Method : ExcelDataRead ' Author : PalaniSelvam Thillaiappan ' Purpose : Read all the contents from Excel sheet and write into log file ' Parameters: sExcelFile - String, contains the Excel file ' : iSheetIndex - Integer, Value for Sheet Index '-------------------------------------- Sub ExcelDataRead(sExcelFile, iSheetIndex, ByRef arrExcel) Dim sExcelPath 'As Variant 'Excel file '********** Excel object declaration **********' ' Excel Application object Dim objExcel 'As Excel.Application Dim objXLWorkbooks 'As Excel.Workbooks Dim objXLWorkbook 'As Excel.Workbook Dim WorkSheetCount 'As Variant 'Work sheets count in a excel Dim CurrentWorkSheet 'As Excel.Worksheet ' Current worksheet Dim objCells 'As Excel.Range Dim objCurrentCell 'As Variant Dim objFont 'As Variant ' Result contents Dim sCellText 'As Variant Dim sFontName 'As Variant Dim sFontStyle 'As Variant Dim iFontSize 'As Variant Dim iCellTextColorIndex 'As Variant Dim iCellInteriorColorIndex 'As Variant Dim sResult 'As Variant Dim sChartFile 'As String ' Row and Col integer variables Dim iUsedRowsCount 'As Integer Dim iUsedColsCount 'As Integer Dim iTop, iLeft 'As Integer Dim iRow 'As Integer 'Row item Dim iCol 'As Integer 'Col item Dim iCurRow 'As Integer Dim iCurCol 'As Integer Dim arrColList 'As Array - Column Name If (sExcelFile = "") Then sExcelPath = "D:\MyCode\qtp_uft\testMyApp_1.xlsx" Else sExcelPath = sExcelFile End If If (iSheetIndex = "") Then iSheetIndex = 2 End If Set objExcel = CreateObject("Excel.Application") objExcel.Workbooks.Open sExcelPath, False, True On Error Resume Next WorkSheetCount = objExcel.Worksheets.Count Set objXLWorkbook = objExcel.ActiveWorkbook 'objXLWorkbook.RunAutoMacros Set CurrentWorkSheet = objExcel.ActiveWorkbook.Worksheets(iSheetIndex) 'iSheetIndex worksheet iUsedRowsCount = CurrentWorkSheet.UsedRange.Rows.Count iUsedColsCount = CurrentWorkSheet.UsedRange.Columns.Count iTop = CurrentWorkSheet.UsedRange.Row iLeft = CurrentWorkSheet.UsedRange.Column CurrentWorkSheet.UsedRange.Columns.AutoFit() ' Cells object CurrentWorkSheet.Cells.Activate ReDim Preserve arrExcel(iUsedRowsCount, iUsedColsCount) ''ReDim arrColList(iUsedColsCount) For iRow = iTop To iUsedRowsCount '(iUsedRowsCount - 1) 'Read All Columns For iCol = iLeft To iUsedColsCount '(iUsedColsCount - 1) sResult = "" Set objCurrentCell = CurrentWorkSheet.Cells(iRow, iCol) sCellText = objCurrentCell.Text sResult = "Reading Cell {" & CStr(iRow) & ", " & CStr(iCol) & "}^" & sCellText & "^" WScript.echo sResult Set objCurrentCell = Nothing arrExcel (iRow-1, iCol-1) = sCellText Next Next ' This will prevent Excel from prompting us to save the workbook. objExcel.ActiveWorkbook.Saved = True Set CurrentWorkSheet = Nothing 'objExcel.Worksbooks.Close objExcel.Quit ''Set CurrentWorkSheet = Nothing Set objExcel = Nothing 'MsgBox "Read Completed.", vbOKOnly, "Exec Over" Exit Sub ErrorHandler1: 'MsgBox "Error # " & CStr(Err.Number) & " " & Err.Description 'Err.Clear ' Clear the error. End Sub '' To display given array Sub PrintArrays( arrList) '' To Print all list items Dim aValue 'array item ==> dictionary Dim aItem 'dictionaly item ==> key-pair value Dim iRow Dim iCol WScript.echo "Array Rows Count: ",UBound (arrList, 1) '' First Array WScript.echo "Array Columns Count: ",UBound (arrList, 2) ''Second Array For iRow =LBound(arrList) to UBound (arrList)-1 For iCol=LBound(arrList,2) to UBound (arrList,2)-1 aValue = arrList(iRow, iCol) WScript.echo iRow, iCol, " Value:", aValue Next Next End Sub