List of Testing Tools

INCOMPLETE LIST OF TESTING TOOLS

The following is a list of tools we've looked into so far. It's not in any particular order and is far from exhaustive. (It doesn't come near to covering all the tools in the lists cited in our main page.) We are attempting to use the following set of categories to describe the tools (with many tools falling into more than one category):

automatic test generation
source code analysis
configuration/version control
capture & playback
property verification
coverage tools
test suite management

Technical papers regarding the tools are divided into two categories:

Independent Reviews: Papers about a tool written by an impartial third party, evaluating the tool in an objective manner
Manufacturer's Literature: Papers about a tool written by the manufacturer. These contain useful technical information but also a lot of marketing stuff

Name: Visual Test
Company: Rational (aquired from Microsoft 1996)
Web Site: http://www.rational.com/products/visualtest/
Category: capture & playback, test suite management
Languages: any, but special integration with Visual C++
Platforms: Windows, integrated with Microsoft Developer Studio
Independent Review: Visual Test 4.0 White Paper by Thomas Arnold, ST Labs, Inc. Discusses automated testing in general as well as Visual Test. Author feels that VisualTest is useful for users with programming experience, but warns that it doesn't do much "hand-holding" for new users. Paper contains detailed description of Visual Test funcation as well as examples.
Description: Test cases and suites described as program (in special TestBasic language) rather than scripts. Develop test programs, manage suites of tests. May write tests by hand or Scenario Recorder can record session and make into a test (capture & playback). Tests created by the Scenario Recorder can be edited. Can write failure tests to catch exceptions. System testing only, not unit testing.

Name: SQA Suite
Company: Rational
Web Site: http://www.rational.com/products/sqasuite/
Category: capture & playback, test suite management, simulation of network conditions
Languages: "focused on IS and Internet developers using Visual Basic®, PowerBuilder®, and
Developer/2000"
Platforms: Windows
Evaluation Software: an evaluation CD is available by mail (not electronically) from PtS (in the U.K., so this would take some time!)
Independent Review:
           Product Review: SQA Suite[tm] 5.1 Favorable review by Gerry Ocampo from ST Labs.   He found SQA Suite to be fairly easy to learn and user-friendly.
Manufacturer's Literature:
        Internet Testing with SQA Suite
        SQA LoadTest Network Configuration Guides
        SQA LoadTest Solves Five Major Challenges of Client/Server Testing (short online discussion plus a downloadable paper)
        What's New in SQA Suite 6.0
Description: Integrated collection of tools:

testing of Windows client-server applications (Web servers)
can port test cases across Windows platforms
SQA Repository: database of information about tests & results, accessible over the Internet for use of testers on multiple machines
SQA WebEntry: lets users query & modify the repository over the Internet (other applications update the repository also)
SQA Manager: helps manage a QA team, including notifying team members of defects via e-mail
SQA Robot: creates & runs test cases. Uses capture & playback to create tests scripts, which may be edited. Collects performance data.
SQA Process: a formal methodology for testing client/server applications (you get software, a written guide and a few days with a consultant to help set up a QA strategy for your organization)
SQA LoadTest: helps test performance with heavy loads by simulating multiple users (running the same test case with different data each time)
Tests created by capture & playback will work even after changes to Web pages
Integrated with Rational development and requirements tools, which can log requirements to be tested.

Name: SQA TestFoundation for PeopleSoft applications
Company: Rational
Web Site: http://www.rational.com/products/testfoundation/
Category: capture & playback, test suite management, simulation of network conditions
Languages: PeopleSoft
Manufacturer's Literature: Assuring Success in PeopleSoft Implementations Written by Rational, aimed at propsective customers, with lots about the reasons for automated testing, but also some information about how TestFoundation works.
Description: PeopleSoft offers customizable packages for business use, including accounting, materials management, distribution, manufacturing, and human resources. Rational has taken much of SQA Suite and integrated it with PeopleSoft packages, including some new customized features.

Name: PureCoverage
Company: Rational
Web Site: http://www.rational.com/products/pure_coverage
Category: coverage tool
Languages: C, C++, Fortran
Platforms: Unix
Evaluation Software: may download for 15-day evaluation
Independent Review: Marick's comments
Manufacturer's Literature:
Product Encapsulation: Using Purify, PureCoverage, and Quantify with SoftBench
Using Rational Developer Products, Purify, PureCoverage, PureLink, and Quantify, with SunSoft's Visual WorkShop
Description: Measures program coverage -- by function, block, or line. Customizable graphical user-interface, updated dynamically as tests are run. Supports C and C++ programming environments; also says it "detects untested code with or without source code". Integrated with Purify.

Name: Purify
Company: Rational
Web Site: http://www.rational.com/products/purify
Category: property verification (run-time errors)
Languages: can identify errors in any code, even without source, but handles source code for C, C++, Fortran
Platforms: Windows & Unix
Evaluation Software: may download for 15-day evaluation
Manufacturer's Literature:
         Product Encapsulation: Using Purify, PureCoverage, and Quantify with SoftBench
         Purify: Fast Detection of Memory Leaks and Access Errors
         Using Custom Memory Allocators with Purify on HP-UX, Sun/OS, and Solaris
         Using Rational Developer Products, Purify, PureCoverage, PureLink, and Quantify, with SunSoft's Visual WorkShop
         Getting Ahead with Purify
Independent Review:
         NSTL Benchmark Report: Performance Testing of Purify (PDF format)
Description: Run-time error detection for Windows and Unix platforms. Checks for memory leaks and other run-time errors (using uninitialized memory, array bounds violations, etc.) by inserting object code before every load and store instruction. Can check code even if source is not available.

Name: PreVue
Company: Rational
Web Site: http://www.rational.com/products/prevue/
Category: capture & playback, simulation of network conditions
Languages: (I found no mention of specific language support)
Platforms: Windows, Unix
Independent Review: High Confidence Load Testing: A Practical Approach to Low Risk Implementation, by Oracle Corporation (html or pdf)
Manufacturer's Literature: 10,001 Users - A Test Odyssey, by Oracle Corporation (html or pdf)
Description: Emulates network traffic. Generates "client emulation scripts" automatically from recorded data from actual runs; scripts can be edited by user. Versions: PreVue-CS for database servers, PreVue-Web for Web servers, PreVue-X for X Windows, and PreVue-ASCII for character-based servers.

Name: Performix
Company: Rational
Web Site: http://www.rational.com/products/performix/
Category: capture & playback, simulation of network conditions
Languages: (I found no mention of specific language support)
Platforms: Unix & Windows (??)
Manufacturer's Literature: Performix/Web for Web Applications
Description: Similar to PreVue -- I'm not sure of the difference from reading the web pages.

Name: Testmate
Company: Rational
Web Site: http://www.rational.com/products/testmate/ (Ada),
http://www.rational.com/products/testmate/cpp/ (C, C++)
Category: test suite management, coverage tool, automatic test generation (?)
Languages: C, C++, Ada
Platforms: Sun Solaris, HP-UX, Digital UNIX, SGI IRIX, Sun SunOS, IBM RS 6000 AIX
Evaluation Software: RMC has it.
Independent Reviews:
Manufacturer's Literature:
Description: Tools for organizing and maintaining test suites for both module and system testing. Automatic execution and reporting. Integrated with Rational's Apex development environment. Generates test cases from templates.

Name: C-Cover
Company: Bullseye Testing Technology
Web Site: http://www.bullseye.com/webCcover.html
Category: coverage tool
Languages: C, C++
Platforms: Windows 95 or NT, Unix, OS/2, some embedded systems
Evaluation Software: may download for one month evaluation
Independent Review:
        Marick's comments
        C-Cover 3.1, by Bullseye Testing Technology    Review by Tom Arnold at ST Labs. Reviewer thinks it's a good tool but a bit over-priced.
Description: Test coverage analyzer; branch or condition coverage.   Marick says it's easy to use.

Name: BoundsChecker
Company: Nu-Mega
Web Site: http://www.numega.com/products/vc/vc.htm
Category: property verification (run-time errors)
Languages: C++
Platforms: Windows
Evaluation Software: may download for 14-day evaluation from PtS
Independent Review:
        BoundsChecker: Useful for Developers and Testers   Favorable review by Dan Hodge at ST Labs.
        links to numerous Nu-Mega papers in BoundsChecker web page
Description: Identifies run-time errors such as arguments out of range, memory leaks, dangling or nil pointers used.   When an error occurs, report includes call stack and other trace information.

Name: ObjectCoverage
Company: ObjectSoftware Inc.
Category: coverage tool
Languages: C++
Platforms: SPARC - SunOs 4.1.X and Solaris 2.X
Evaluation Software: may be available by filling out an evaluation form
Independent Reviews: Marick's comments
Manufacturer's Literature: technical paper available to download
Description: Produces branch coverage reports for C++ programs.

Name: ObjectTester
Company: ObjectSoftware Inc.
Category: automatic test generation
Languages: C++
Platforms:
Evaluation Software: may be available by filling out an evaluation form
Independent Reviews:
Manufacturer's Literature: ObjectTester(TM) - A Tool To Generate Unit Test Scripts For C++ Classes Rather badly written -- details without an overview.
Description: Generates unit testing scripts for C++ classes. It's hard to tell from how the tests are generated.

Name: TestCenter
Company: CenterLine Software
Web Site: http://www.centerline.com/productline/test_center/test_center.html
Category: coverage tool & property verification (run-time errors)
Languages: C, C++
Platforms: Sun, HP, AIX
Evaluation Software:
Independent Reviews: Marick's comments
Manufacturer's Literature:
Description: measures line coverage of test programs, helps check for run-time errors and memory leaks, can force errors such as "disk full" or "out of memory". Integrated with other development tools.

Name: AcquaProva
Company: CenterLine Software
Web Site: http://www.centerline.com/productline/AcquaProva/AcquaProva.html
Category: source code analysis, property verification (runtime errors)
Languages: C, C++
Platforms: Sun, NCR
Evaluation Software: you can send programs to a demo server and look at the results.
Independent Reviews:
Manufacturer's Literature:
Description: Checks source code for errors as well as providing programming advice based on books by Scott Meyers. User can add rules to enforce company coding practices. Also detects runtime errors.

Name: QC/Replay
Company: CenterLine Software
Web Site: http://www.centerline.com/productline/qcreplay/qcreplay.html
Category: capture and playback
Languages:
Platforms: XWindows on SunOS, Solaris 2.x, HP/UX, AIX
Evaluation Software:
Independent Reviews:
Manufacturer's Literature:
Description: Capture and playback for XWindows programs.

Name: Insure++
Company: ParaSoft
Web Site: http://www.parasoft.com/insure/home.htm
Category: property verification (runtime errors), coverage tool
Languages: C, C++
Platforms: SunOS, HP, IBM RS6000, SGI 5.X, Digital UNIX, Lynx, Linux, SCO
Evaluation Software: trial copies available
Independent Reviews:
        Marick's comments
        Comparison: Debugging Tools (From Advanced Systems, October 1994, pp 56-62)
            compares Purify, Sentinal and Insight (old name of Insure++) and likes Insight best
        Insure++ Product Review (from The X Journal Magazine, Summer 1996)
Manufacturer's Literature:
        Insure++: A Tool To Support Total Quality Software
        Parasoft Insure++ Slides
        Insure++ Technical Papers Archive
Description: Detects many kinds of compile-time and runtime errors, such as memory leaks. Coverage analysis is available as an optional add-on.

Name: CTC++
Company: Testwell
Web Site: http://www.testwell.sci.fi/ctcdesc.html
Category: coverage tool
Languages: C, C++
Platforms: MS-DOS, Windows 3.x/95/NT, DJGPP, HPUX, Sun/Solaris&SunOS, Linux, DEC Alpha OSF1.
Evaluation Software: 30-day evaluation copies available
Independent Reviews: Marick's comments
Manufacturer's Literature:
Description: coverage analysis for C++, by function, statement, branch or condition. Also provides timing information. May be integrated with Microsoft Visual C++ Developer Studio.

Name: TCMON
Company: Testwell
Web Site: http://www.testwell.sci.fi/tcmdesc.html
Category: coverage tool
Languages: Ada 83
Platforms: MS-DOS, VAX/VMS, Sun/Solaris&SunOS. Ada source code available and may be ported to other platforms.
Evaluation Software: 30-day evaluation copies available
Independent Reviews: Marick's comments
Manufacturer's Literature:
Description: coverage analysis for Ada, by statement, condition, or sub-condition. Also provides timing information. Works by producing a modified copy of the source code.

Name: TBGEN
Company: Testwell
Web Site: http://www.testwell.sci.fi/tbgdesc.html
Category: test suite management, unit testing
Languages: Ada
Platforms: MS-DOS, VAX/VMS, Sun/Solaris&SunOS. Ada source code available and may be ported to other platforms.
Evaluation Software: 30-day evaluation copies available
Independent Reviews:
Manufacturer's Literature:
Description: Ada test bed: automates building of test drivers and stubs for unit/module testing. Can test interactively or with a script, including tests, loops and assertions.

Name: CTB
Company: Testwell
Web Site: http://www.testwell.sci.fi/ctbdesc.html
Category: test suite management, unit testing
Languages: C
Platforms: MS-DOS, Windows and others. May be ported to other platforms.
Evaluation Software: 30-day evaluation copies available
Independent Reviews:
Manufacturer's Literature:
Description: C test bed; automates building of test drivers and stubs for unit/module testing.

Name: CMT++
Company: Testwell
Web Site: http://www.testwell.sci.fi/cmtdesc.html
Category: source code analysis, property verification (complexity)
Languages: C, C++
Platforms: MS-DOS, Windows, several Unix platforms
Evaluation Software: 30-day evaluation copies available
Independent Reviews:
Manufacturer's Literature:
Description: analyzies C/C++ code using McCabe's cyclomatic number and Halstead's estimated number of bugs.

Name: GCT
Company: Testing Foundations (Brian Marick)
Web Site: ftp://cs.uiuc.edu/pub/testing/GCT.README
Category: coverage tool
Languages: C
Platforms: Unix
Evaluation Software: it's freeware, available by ftp. See above web site for details.
Independent Reviews:
Manufacturer's Literature:
Description: analyzes test coverage for C. More different coverage measures than most of the other programs, including weak mutation coverage and "race coverage" for multiprocessor systems. Works by instrumenting C source code.

Name: Testworks
Company: Software Research, Inc.
Web Site: http://www.soft.com/Products/index.html
Category: test suite management, coverage tool, capture & playback, automatic test generation, source code analysis
Languages: Java, C, C++ Ada and F77.
Platforms: Windows, Unix
Evaluation Software: limited-time evaluation copies of some components are available; use links from company's home page
Independent Reviews:
Manufacturer's Literature:
Description: An integrated suite of testing tools, providing management of test suites, coverage analysis, and capture and playback for both text and GUI-based applications. The Unix version also includes a tool to generate test cases from templates provided by the user, and source code analysis to measure complexity and identify bugs or possible trouble spots. The compoments may be purchased separately or bundled together.

Name: LDRA Testbed
Company: LDRA Ltd.
Web Site: http://www.ldra.com/index.htm
Category: source code analysis, coverage tool
Languages: Ada, Algol, C/C++, Cobol, Coral 66, Fortran, Pascal, PL/M, PL/1, Assemblers (Intel + Motorola).
Platforms: Solaris, SunOS, Windows 95/NT/3.1x, Vax/VMS, Open VMS, HP-UX, Digital Unix, WinOS/2, AIX, IRIX, SCO, MVS.
Evaluation Software:
Independent Reviews: Marick's comments
Manufacturer's Literature:
Description: Source code analysis includes metrics on quality and complexity, data flow analysis, and variable cross references. Coverage analysis includes statement, branch and subcondition coverage. Also includes data flow analysis.

Name: Hindsight
Company: IntegriSoft, Inc.
Web Site: http://www.integrisoft.com/hindsight.htm
Category: coverage tool, test suite management, test suite analysis,
Languages: C, C++, Fortran
Platforms: SunOS, Solaris, HP/UX, and IBM AIX
Evaluation Software:
Independent Reviews:
Manufacturer's Literature:
Description: Several functions:

segment, branch and condition coverage analysis
generates test drivers and stubs for unit testing
cross-references test cases with functions or classes
recommends minimal set of test cases
complexity metrics

Name: Visual Testing Toolset
Company: McCabe and Associates
Web Site: http://www.mccabe.com/vtt/
Category: coverage tool
Languages & Platforms: says it supports many languages and platforms, but no details
Evaluation Software:
Independent Reviews:
Manufacturer's Literature: The company has a web page listing papers, including journal articles by McCabe. You have to fill out a form to access them, and it's not shown which are on-line.
Description: Reports and graphical displays with several kinds of coverage measures. Computes code complexity and basis paths and shows which paths have been exercised by tests.

Name: XRunner
Company: Mercury Interactive
Web Site: http://www.merc-int.com/products/xrunguide.html
Category: capture & replay, automatic test generation
Languages:
Platforms: all Unix platforms, may be ported to Windows
Evaluation Software: you have to fill out a form to get information about evaluation copies
Independent Reviews:
"Testing Can't Wait" -- from ComputerWorld, 1997, contains a brief but favorable mention
SCO World Magazine, July 1997 - "GUI Testing Made Simple" (not available online, but they quote favorable comments)
Manufacturer's Literature:
Description: Generates test suites for XWindows applications. Test results can be compared as bitmaps, or at a higher level. You can also create your own test suites by capture & replay or with the help of a graphical interface to your program and a test script language

Name: WinRunner
Company: Mercury Interactive
Web Site: http://www.merc-int.com/products/winrunguide.html
Category: capture & replay, automatic test generation
Languages:
Platforms: Windows
Evaluation Software: you have to fill out a form to get information about evaluation copies
Independent Reviews:
"Testing SAP R/3: Mercury Interactive's Insurance Policy for Users"
"Correction Checker -- Mercury suite tests converted code" -- Information Week, July 21, 1997
Manufacturer's Literature: web page contains links to numerous "white papers" and testimonials
Description: Sounds just like XRunner, but for Windows GUI applications.

Name: LoadRunner
Company: Mercury Interactive
Web Site: http://www.merc-int.com/products/loadrunguide.html
Category: load testing
Languages:
Platforms: Windows, Unix
Evaluation Software: you have to fill out a form to get information about evaluation copies
Independent Reviews: product web page contains many links to reviews and inside papers
Manufacturer's Literature:
Description: Emulates a multi-user network load, with each virtual user running tests (scripts may be developed with XRunner or WinRunner).

Name: TestDirector
Company: Mercury Interactive
Web Site:
Category: test suite management
Languages:
Platforms: Windows
Evaluation Software: you have to fill out a form to get information about evaluation copies
Independent Reviews:
Manufacturer's Literature:
Description: Tool for managing test suites, maintaining records about test plans for multi-user projects, tracking progress in finding and fixing bugs. Integrated with WinRunner. (TestSuite = WinRunner + TestDirector.)

Name: Astra SiteTest
Company: Mercury Interactive
Web Site: http://www.merc-int.com/products/astrastguide.html
Category: load testing
Languages:
Platforms: Windows
Evaluation Software: evaluation copies available via a toll-free number listed on the web page
Independent Reviews: web page contains links to lots of reviews and internal papers
Manufacturer's Literature:
Description: Stress testing for web sites. Includes LoadRunner, plus other stuff specific to web sites. May be integrated with Astra SiteManger, a tool for managing web sites -- can gather statistics about usage which can be used to generate testing scenarios.

Name: TestBytes
Company: Mercury Interactive
Web Site: http://www.merc-int.com/products/testbytesguide.html
Category:
Languages:
Platforms:
Evaluation Software: you have to fill out a form to get information about evaluation copies
Independent Reviews:
Manufacturer's Literature:
Description: Data test generation; couldn't access full description due to Network problem.

Name: ITEX - Interactive TTCN Editor and Executor
Company: Telelogic
Web Site: http://www.telelogic.com
Category: automatic test generation, test suite management
Languages:
Platforms: Win 95/NT, Sun Solaris/SunOS, HP-UX
Evaluation Software:
Independent Reviews:
Manufacturer's Literature:
Description: If you have an SDL specification of the system being tested, provides "semi-automatic" test case generation. If you write your own test cases, they are checked against the specification.