Articles / Open Source Software Testin…

Open Source Software Testing Beyond Java

As mentioned in my previous article on Open Source software testing tools, I am going to provide an overview of some more test tools hosted by SourceForge and continue to demonstrate how they can improve the quality and productivity of a development effort. The previous article focused on testing tools that can be used for applications written in Java and Web-based applications. This article will provide an overview of testing tools for SOAP, databases, Testing suites/frameworks, .Net, and Linux, among others. Some interesting new Java testing tools have also been developed since the time I wrote the previous article, and I would like to include them. Like the previous article, this one will focus primarily on functional testing tools.

Java Application Testing Tools

First up is jDiffChaser. Relatively brand new, jDiffChaser automates visual GUI comparisons between two versions of a Swing application. It plays previously recorded scenarios and takes a screenshot of the last production version GUI rendering and another screenshot of the current development version. It then compares both screens, finds the differences between them, and lists all in an HTML report. You can define zones to ignore during comparisons. Scenarios can be played locally on a single host (sequentially), or they can be played at the same time on two hosts when real time data is needed by the tested application. It is a good tool for applications that emphasize color and font size roles, applications for which every visual effect (background, layout, and font) is important because it can have a business meaning, as in air traffic or rail control systems. But jDiffChaser also does a nice job for less critical applications that are subject to side effects because of huge component reusing. It can also be used as a feedback tool to list every visual modification between two versions.

Testing Tools for SOAP Applications

Beyond Java, Web Services-based applications are the most important development in recent years. The following tools can be used for testing applications which use the SOAP framework.

If you develop SOAP-based Web Services, there is a tool that you will not want to miss: soapUI, an Open Source GUI for easily and very quickly creating and firing off SOAP requests. Basically, you feed it WSDL, it creates sample SOAP requests for you, you edit those in a simple but functional XML editor, and then you send it to see the response. This is one of the advantages of an SOA using XML as the platform-neutral lingua franca of message exchange for integration purposes: A machine-readable representation (XML schema and WSDL) to describe services and make them discoverable and easily invokable by tools such as soapUI. And there are no JAR files as for RMI services, no IDL compilers... It's a data format, there is a commonly-agreed-on description of it, and tools can construct and consume messages easily, even "simple" tools. Some of the key benefits include:

  • Use of a UI for creating/configuring tests. This makes it possible for non-programmers to develop tests, which means that testing of Web services can be performed by a QA/system test group.
  • Auto-generation of test messages based on WSDL.
  • The ability to externalize test data using property files.
  • Built-in assertions, such as checking for SOAP faults and the validity of the schema.
  • Built-in stress testing capabilities.
  • Management of endpoint URLs.

However, one needs to understand shortcomings of this approach. soapUI uses its own Web services client. The way it creates test messages is different from how regular (JAX-WS-based and other) Web services clients work. soapUI completely skips the step of generating Java classes from WSDL. It does not have to deal with XML marshaling/unmarshaling from/to Java objects. As a result, you are not invoking your service the way it will be invoked by real Web service consumers. Technically, it may not be an issue in a majority of cases. The fact that soapUI is able to parse your WSDL/schema and generate test message means that you have a valid WSDL that should also be consumable by other technologies. However, it is possible that your WSDL/ schema may lead to sub-optimal (albeit perfectly valid) generated classes.

For example, the schema that defines <addresses><address>...</address>...</addresses> XML will result in a JAXB compiler generating an Addresses class which returns the list of type Address. This may or may not be how you want Java classes to look for the clients. You may want to consider getting rid of <addresses> so that no extra class is generated. In other words, using soapUI (or similar UI-based Web services testing tools, such as SOAPScope) does not allow you to understand all the details of how your Web service will be consumed in the real world. That said, even technical business analysts can perform functional testing of services apart from performance and stress testing features, which are also incorporated.

LISA, iTKO's wizard-based testing software, models how an SOA should perform (regardless of how it's built) and evaluates the effects of the different services within the application. LISA WS-Testing is a fully functional, no-code Web services test authoring and execution solution. It allows local workstation use without a Web-based license, as well as UDDI registry browsing/testing, WS-Security validation and SSL support, SAML tokens, and configurable Web Services simulation within test cases. It also has tools to analyze and stress-test an SOA app, no matter what mix of middleware is involved in its construction. The tool simulates users accessing the Web services, in order to gauge response times. Note: To perpetually license LISA, you will need to register for WS-Testing at the developer's Web site, and iTKO Support will issue you a license and access to the iTKO LISA Forums.

MActor is an extensible integration test tool. It can facilitate tests of any XML-based integration. Adapters for TIBCO RV, HTTP, SOAP, filesystem and IBM MQSeries are included. Tests have to be specified in XML using the Test Specification Language, and input data to the tests are specified in HTML (in a similar manner to that utilized in the FitNesse testing Framework). Messages that are published from the tests are typically constructed based on template messages, where the values of some elements/attributes are replaced using XPath, but a number of other approaches are possible.

Tools for testing Ajax applications

Ajax is a Web development technique used commonly for creating interactive Web applications and to make Web pages feel more responsive by exchanging small amounts of data with the server behind the scenes so that the entire Web page does not have to be reloaded each time the user requires a change. Tools for testing Ajax-based applications will be discussed now.

TIBCO's GITAK (General Interface Test Automation Kit) has now been Open Sourced by the company. This gives developers a free tool for building automatic quality assurance testing routines for enterprise application components like asynchronous communications, JavaScript, AJAX, and RIA (Rich Internet Applications) components. Based on the Selenium Web applications test tool, GITAK runs in a browser alongside target JavaScript and AJAX applications (both of which have a reputation for being difficult to debug) and provides the tools that allow companies to create automated test cases and run scenarios. These can be used to validate that an application is performing as it should. The idea is that users build a library of test cases to suit their particular needs. This library is then run against the applications, which provides an environment in which both new and changed applications can be tested automatically, effectively with the push of a button. TIBCO also contends that it has the additional advantage of testing the entire application rather than just the application components as independent units. This allows GITAK to detect errors that may not be exposed until runtime.

Selenium itself can be used to test Ajax apps. Most of the normal Selenium actions don't work well for the Ajax calls, as they expect synchronous communication, so some people use the "waitForValue" and "waitForCondition" extensions, which can just sit and watch for the DOM or input values to change after the async call was made.

Eclipse plugins for Testing

Eclipse is a very widely used Open Source Integrated Development Environment (IDE), particularly for Java- and Web-based applications. It's very widely extensible with the use of scores of plugins available to help in developing these applications, so here I would like to discuss some of the plugins available for software testing.

Eclipse TPTP (an evolution of Eclipse's prior "Hyades" project) is an Open Source collaborative project that seeks to provide a common platform upon which specialized, differentiated, and interoperable offerings for software test and performance tools are created. TPTP, which stands for Test and Performance Tools Platform, supplies extensible frameworks and services for test and performance tools that are used throughout the application lifecycle, from development through production. It also delivers extensible exemplary tools that verify the utility of, illustrate the appropriate use of, and support the development and maintenance of the platform itself.

TPTP tries to provide a reference implementation of the recent UML2 Testing Profile based upon the Eclipse Modeling Framework (EMF). For logging and tracing, TPTP employs the Common-Base-Event (CBE) Format, which defines the structure of events in a unifying format. Data pools are a concept which appears to be innovative for testing in Eclipse. A datapool contains data usable during a test run. Data pools are stored in a comma-separated file and can be displayed as an Excel-alike data sheet. Test cases can load datapools in the setUp() method and use the dedicated iterations to retrieve cell values for evaluation purposes. Separating test data from the tests themselves appears to be new functionality. Test Deployment requires the allocation of test artifacts to test locations. These locations can also be workbench locations. Through the concept of service abstraction, the Automatable services framework introduces a Service-Oriented Architecture (SOA) concept to TPTP. The framework introduces a layered architecture in which a client at a lower layer employs a model adapter to employ a service provided at a higher layer. Automation Client Adapters allow normal Java programs to consume Eclipse services. These adapters can launch a headless Eclipse from a specified Eclipse home, and are available through the tptp-automation-client jar. Various properties are made available through this API (e.g., the project property specifying a specific project in the workspace of the Eclipse instance).

Solex is a Web application testing tool built as a plugin for the Eclipse IDE. It provides functions to record a client session, adjust it according to various parameters, and replay it later, typically in order to ensure non-regression of the application's behavior (with stress testing capabilities added at a later stage). Solex does nothing you couldn't achieve using raw network dumps, but it wraps it in a decent GUI and allows you to debug HTTP sessions quickly and effectively. Solex acts as an HTTP proxy and records all HTTP requests and responses going through the wire between a Web client (e.g., a Web browser) and a Web server. The task of replaying a scenario consists of sending the previously-recorded (and eventually customized) HTTP requests to the server and asserting each response.

Database Testing Tools

There are probably very few Java- and Web-based applications that do not use a database, so it is worth noting some of the tools available to test databases in general.

LReport is a toolkit for comparing data stored in databases and in csv files. It is written in Perl and can be used as commandline utility or a Perl module for your code. It also provides tools which support nice formatting of data retrieved from the database or csv files. It automatically creates nicely-formatted reports from selected tables, which helps with automatically finding differences between results of the group of selects.

The database testing tool DBFire runs from the commandline and produces the results of the test run in a simple text file. The tool runs pre-determined SQL queries in a prepared-statement-like format. You give it the statement configuration in an XML file, and it does the rest. It contains a config.xml file that starts with the testfire element and contains one or more test elements and one or more dictionary elements.

Test data generator is a utility that generates data, tables, views, procedures, etc. for database testing purposes: performance testing, QA testing, loading tests, or usability testing. It helps in automatically creating database data, objects, and sets of SQL statements, if necessary. A project may contain multiple tables, each with its own data generation rules, value range, and parameters, which makes using the tool very convenient. The user interface is a bit hard to understand at first, and the online help is not particularly helpful, but a bit of spelunking reveals some really useful things. The basic steps are to create a set of rules for generating your sample data (and maybe tables and other objects like views), save this as a project, and execute the project steps. You can reorder the project steps, and you can create steps that execute before the entire project and after the entire project runs. Each step can have a before and after part to it as well. There are lots of options for generating the data that will be placed in each table. These include lookup lists, flat files, lists of values that are part of DTM, and several others, including using foreign keys from master tables. Each project has a number of settings as well for controlling things like whether the generated SQL is kept in a file, whether the database schema has to be verified before running the project, and whether the project actually makes live changes to the database or not. The UI is sufficiently hard to grasp that it's not obvious how you assign different data generation schemes to each column in a table.

Tools for Testing .Net Applications

The current trend is to make use of Web services in most Web-based applications, which probably use not only Java-based applications, but also share resources with .Net-based applications, too. Here, I would like to introduce some tools that are useful for testing the .Net-based applications.

SharpRobo is a Functional Testing and Recording tool for .NET WinForms applications. It can run coded or FIT tests. Recorded FIT tests can be used with OpenWiki to generate HTML files. It supports all the standard WinForm controls. SharpRobo records the tests in FIT format, which can be played back using Fit (File or Directory Runner). You need to derive your forms and controls from SharpRobo base classes.

WatiN stands for "Web Application Testing In .Net", and it's a framework that enables Web application testing through Internet Explorer. WatiN is inspired by WatiR, a Web application testing framework for Ruby. The way that WatiN works is very "easy", knowing that there's a lot of work behind a framework like this. WatiN lets you open Internet Explorer instances and, through interop, interact with the elements in the form. With WatiN, you can get and set values from the elements in the form and you can fire events of any of the elements in the document, too. It effectively allows you to script interactions with Internet Explorer and do things like fill out forms, click through, and test the results that come back. This makes testing things such as a process flow really easy to do in a repeatable manner. WatiN also supports handling AJAX changes quite nicely. The library is very useful, but sometimes, writing a test is a tedious process. For this reason, there are some freeware recorders that are able to record navigation in IE and automatically create code for a WatiN test. Together with the WatiN Test Recorder, it's an ideal tool for functional testing of C#, VB.NET, and PHP application testing. The WatiN Test Recorder is capable of visualizing the DOM of the page, visualizing all the WatiN element that can be referenced on the page, and more. Another alternative recorder is the Watin Recorder, which seems to work very well with IE6, but not with IE7 (unless the site being recorded is added to the "trusted sites" list).

A derived project from WatiN is ItiN, which uses a hybrid approach, allowing manipulation of an InfoPath document directly using XPath and the XML dom (or the view using traditional WatiN automation) and serves as a framework for testing InfoPath forms. The full set of HTML controls supported by WatiN is not required for ItiN, and only buttons, checkboxes, radio buttons, textboxes, and select lists are supported. ItiN needs Visual Studio 2005 and InfoPath 2003 with the .Net programmability support installed to work. This is currently under development, and is a project to watch.

Testing Tools and Linux

No Open Source discussion is complete without Linux, so it is appropriate to mention testing aids that fall under the Linux radar.

BugnuX is a Linux distribution made for software testing based on the concept of distributions that are currently available for security testing. It is a LiveCD-based Linux distribution that is also installable, packaged with useful Testing Tools to help testers and quality assurance personnel. It consists of Open Source Tools for testing Web applications, Java, load, and performance. Software testers can quickly test applications without the need to install a complete OS, using any PC. This is especially useful if they need to validate features on Linux. BugnuX is also handy for consultants who have limited resources and want to test with the tools they know best. Companies can quickly provide a standard Linux system with testing tools to consultants or temporary testers. Thus, BugnuX meets the goal of providing many useful and usable Open Source testing tools that will be essential for testers and quality assurance personnel to do a good job with minimum time wasted in downloading and installing necessary operating system and essential software in a downloadable ISO format which would fit on a CD or USB drive.

Conclusions

When trying to choose Open Source test tools, several factors should be considered, such as how active the development is, the support for the tool, the quality of the documentation, etc. As in the previous article, the table below will give an idea of the development activity, such as the current version as of September 2007, support via mailing lists and public forums, documentation quality, etc. of the tools listed above.

Java Application Tools

Tool

Current Version

Date Released

Public Forums

Mailing Lists

CV Tracker

Docs

jDiffChaser

0.5

31/05/2007

Active

1

Active

Good

SOAP Application Tools

Tool

Current Version

Date Released

Public Forums

Mailing Lists

CV Tracker

Docs

soapUI

1.7.5

06/08/2007

Very Active

1

Very Active

Excellent

iTKO LISA

3.6b

22/06/2007

Very Active

8

Very Active

Excellent

MActor

1.0.14

13/07/2007

Active

0

Very Active

Very Good

AJAX Application Tools

Tool

Current Version

Date Released

Public Forums

Mailing Lists

CV Tracker

Docs

GITAK

0.7

27/04/2007

Very Active

0

Very Active

Good

Selenium

0.8.7

21/03/2007

Very Active

1

Very Active

Excellent

Eclipse plugins

Tool

Current Version

Date Released

Public Forums

Mailing Lists

CV Tracker

Docs

TPTP

4.4.0.2

17/08/2007

Very Active

7

Very Active

Excellent

Solex

0.5.3

05/01/2006

Not Active

1

Not Active

Limited

Database Tools

Tool

Current Version

Date Released

Public Forums

Mailing Lists

CV Tracker

Docs

LReport

0.20

15/03/2007

Very Active

0

Very Active

Very Good

DBFire

2.3

12/05/2005

Not Active

0

Not Active

Limited

DTM Data Generator

1.17

23/08/2007

Active

0

Very Active

Very Good

.Net Tools

Tool

Current Version

Date Released

Public Forums

Mailing Lists

CV Tracker

Docs

SharpRobo

0.51

03/01/2007

Active

0

Active

Limited

WatiN

1.1.2

16/05/2007

Very Active

3

Very Active

Excellent

ItiN

Dev

N/A

Active

0

Very Active

N/A

Linux Tools

Tool

Current Version

Date Released

Public Forums

Mailing Lists

CV Tracker

Docs

BugnuX

2007

31/08/2007

Very Active

1

Very Active

Very Good

The tools mentioned are not the definitive list of tools available, but the most popular ones hosted on SourceForge. I am planning soon to write a continuation to this article which will have details on Open Source bug-tracking tools, tools for testing other non-Java software applications, and tools available for stress and performance testing.

Recent comments

09 May 2008 04:47 Avatar gruendung

Testing Java Applications
As a beginner in Java I found great &amp; fast help by reading this article. I'd just wish to find good summaries like this for every problem to solve. So, thank you.

25 Feb 2008 09:52 Avatar StudentProgrammer

Re: A lot of these don't RERO as much as I'd like
Great Bit of information, never posted here before so interested about the "Feel free to share your opinions, but remember to keep the discussions calm; don't flame and don't insult others. You'll be given a preview of what your comment looks like in step 2.

I guess people get into it here :-)

Student Programmer
My Blog

18 Feb 2008 06:13 Avatar ronfox

Other application domains.
A very nice article... not all of us are web programmers. For me the test tools I use:

cppunit for my C++ work

tcltest for my Tcl scrpiting work.

the Ruby test frameworks for when I do go into the web.. as I approach the web on Rails.

16 Feb 2008 14:51 Avatar imipak

A lot of these don't RERO as much as I'd like
Selenium's official release has quite a lot of dust, these days, but the repository copy fixes a lot of problems in the current release and provides some nice extensions. The same applies to a lot of testing tools. Great code, great ideas, way too long between releases. I understand the need for stability and the need for quality control, but if the previous version has enough problems or limitations that the mailing lists generally advocate using the test release, then the test release has become an official release in all but name.

Screenshot

Project Spotlight

Kigo Video Converter Ultimate for Mac

A tool for converting and editing videos.

Screenshot

Project Spotlight

Kid3

An efficient tagger for MP3, Ogg/Vorbis, and FLAC files.