Free beta release of combinatorial testing tool 
The tool, named Advanced Combinatorial Testing Suite (ACTS), can compute tests for 2-way through 6-way interactions.  
An easy-to-use GUI is included.    A comparison of ACTS with similar tools shows that ACTS produces
smaller test sets (with the same degree of coverage) and is faster than others. 

To request a copy, send email to Rick Kuhn - kuhn@nist.gov.  

This software is provided free of charge and will remain free in the future.  NIST is an agency of the US Government.
Please Note:  Prior to Feb 09, the name "FireEye" was used for the ACTS tool. Some papers and articles on the site
refer to FireEye; this is the same tool now called ACTS.

Who uses our combinatorial testing tool?  We have approximately 200 users as of July 2009, in IT, defense, finance,
telecom, and many other industries.  Here is a breakdown of our user base.  

User Feedback
Users have been very positive, and are applying ACTS to a wide variety of software.

  •  "At present I am working with a development team to enable automated testing of a complex 
    rating algorithm under development within a property/casualty insurance system. The system
    component under test accepts several hundred independent input variables which are used
    to produce rating factors as output.  I have had some opportunity to use [NIST-ACTS] as a
    means to calculate test coverage and identify missing test cases in our current population of
    test data. I'm pleased to report that the combinatorial coverage data has proven useful to us
    in identifying gaps. Eventually, I would like to use this data to extend our test coverage as
    well -- at coverage strength 3 we're producing a volume of test cases that should be
    manageable when paired with automated test data generation. I can also foresee other uses
    which would pair automated testing tools with [NIST-ACTS] output to improve our test
    coverage for other systems."
  •       "The tool is pretty easy to use, and has already reduced our planned number of tests by 20% or so."

  • "Nice... the XML configuration can be used for both CLI and application."
  • "Our feedback on [NIST-ACTS] is extremely positive. The tool implement in a very efficient 
    way the algorithm of N-Wise reduction we were looking at. In fact, we would be interested
    in including your technology in our tool [ ]."
  • "[NIST-ACTS] is a good application. It was easy to figure out and use. ... I would use this 
    application again."
  • ""I've fund it Preintuitive and greatly helped to reduce the test size. I'll continue to use this tool at 
    next opportunity."
  •  "I think that [NIST-ACTS] has been very helpful in determining test coverage for simple test 
    cases with lots of variables."
  • " Not only does [NIST-ACTS] handle phenomenal combinatorial complexity without breaking 
    a sweat, it is also very easy and straightforward to use.   The user interface is very well thought
    out, intuitive, and delightfully uncluttered. "

ACTS definition
Defining variables in system under test

ACTS constraint screen
Option 1:  Constraints among variables can be added.

ACTS variable strength interactions
Option2:  Different interaction strengths can be used for sets of variables.

ACTS test array
Finished covering array can be output as Excel spreadsheet, text or XML.

IPO (FireEye) Tconfig ITCH Jenny Test Vector Generator AllPairs CTE-XL
n -way Size Time Size Time Size Time Size Time Size Time Size Time Size Time
2 100 0.8s 108 >1 hour 120 0.73s 108 0.001s 101 2.75s 103 3.82s 114 2s
3 400 0.35s 472 >12 hours 2388 17 min 413 0.71s 9158 3.07s 469 43.88s
4 1361 3.05s 1478 >21 hours 1484 15 hours 1536 3.54s 64696 2 min 7s
5 4219 18.41s n/a >1 day n/a >1 day 4580 43.54s 313056 25 min 49s
6 10919 65.03s n/a >1 day n/a >1 day 11625 7 m 50 s 1070048 3 hr 30 min
n-way type Multiway Multiway Multiway Multiway Multiway 2 - way 2-way, 3-way
Developer NIST/UT Arlington  Univ of Ottawa IBM Bob Jenkins SourceForge Satisfice Daimler Chrystler
Language used Java Java Java C Java Perl (Not Known)
Free/Commercial Free Free Free Free/Open Source Free/Open Source Commercial Commercial

 Comparison of Extended IPO Algorithm (ACTS/FireEye) with Other Covering Array Generators

Input (12 variables)
Cur_Vertical_Sep : { 299, 300, 601 };
High_Confidence : boolean;
Two_of_Three_Reports_Valid : boolean;
Own_Tracked_Alt : { 1, 2 };
Other_Tracked_Alt : { 1, 2 };
Own_Tracked_Alt_Rate : { 600, 601 };
Alt_Layer_Value : 0..3;
Up_Separation : { 0, 399, 400, 499, 500, 639, 640, 739, 740, 840 };
Down_Separation : { 0, 399, 400, 499, 500, 639, 640, 739, 740, 840 };
Other_RAC : { NO_INTENT, DO_NOT_CLIMB, DO_NOT_DESCEND };
Other_Capability : { TCAS_TA, OTHER };
Climb_Inhibit : boolean;