Defining A Random Seed

To define a random combination seed, use the seed parameter. For example, the following command generates test cases with a default TupleGenerator that uses the specified seed value.

mvn tcases:tcases -Dseed=299293214

If you already have a generator definition file for a project, this parameter will update the file by adding or changing the default seed value, as shown below. If no genDef file exists, it will create one.

<Generators> 
  <TupleGenerator seed="299293214"/> 
</Generators> 

If you'd like to randomize combinations but you're not particular about the seed value, set the newSeed parameter to true, and Tcases will choose a random seed value for you. This option can be handy when you want to see if a different seed value might produce more interesting test case combinations.

Defining The Default Coverage Level

To define the default coverage level for all functions, use the defaultTupleSize parameter. For example, the following command generates test cases with a default TupleGenerator that uses the specified coverage level.

mvn tcases:tcases -DdefaultTupleSize=2

If you already have a generator definition file for a project, this parameter will update the file by adding or changing the default tuple size value, as shown below. If no genDef file exists, it will create one.

<Generators> 
  <TupleGenerator tuples="2"/> 
</Generators> 

Mix It Up: Random Combinations

By default, Tcases creates combinations of input variables by marching through the system input definition top-to-bottom, picking things up in the order in which it finds them. You might try to exploit that natural order, although satisfying constraints can take things off a predictable sequence. That's why you really shouldn't care too much about which combinations Tcases comes up with. Even better? Ask Tcases to randomize its combination procedure.

You can define random combinations in your generator definition by using the seed attribute --- see the example below. This integer value acts as the seed for a random number generator that controls the combination process. Alternatively, you can (re)define the seed value using a configuration parameter. By specifying the seed explicitly, you ensure that exactly the same random combinations will be used every time you run Tcases with this generator definition.

<Generators> 
  <TupleGenerator function="find" seed="200712190644"> 
    ... 
  </TupleGenerator> 
</Generators> 

The results of random combination can be very interesting. First, you can end up with test cases that you might not have considered, even though they are perfectly valid and produce the same coverage. Sometimes that's just enough to expose a defect that might otherwise have been overlooked, simply because no one thought to try that case. This is an application of the principle of "gratuitous variety" to improve your tests. This also produces another benefit --- sometimes an unusual combination can demonstrate a flaw in your test design. If a combination just doesn't make sense, then it's likely that a constraint is missing or incorrect.

Finally, random combinations can occasionally reduce the number of test cases needed to meet your coverage requirements. That's because some combinations may "consume" variable tuples more efficiently than other equally-valid combinations. Tcases does not attempt to spend the enormous effort needed to guarantee an optimally minimal set of test cases. It simply starts at the beginning and does its best to get quickly to the end. But a random walk through the combinations may lead Tcases to a more efficient path. If you're concerned about the size of your test suite, try using the tcases:reduce goal.

Avoiding Unneeded Combinations

Even when Tcases is generating test cases for the default 1-tuple coverage, it's typical to see some input values used many times. This is most likely for those Var elements that contain only a few Value definitions. Even after these values have been used, Tcases will continue to reuse them to fill out the remaining test cases needed to complete the test suite. In some situations, this can be a bit of a pain. Sometimes there is a Value that you need to test at least once, but for various reasons, including it multiple times adds complexity without really increasing the likelihood of finding new failures. In this case, you can use the once attribute as a hint to avoiding reusing a value more than once.

For example, consider the find command example. The find command requires that the pattern must not exceed the maximum length of a line in the file. Even one line longer than the pattern would be enough to avoid this error condition. In fact, the principles of boundary value testing suggest that it's a good idea to have a test case that has exactly one line longer. Therefore:

<VarSet name="file" when="fileName"> 
  ... 
  <VarSet name="contents" when="fileExists"> 
    <Var name="linesLongerThanPattern"> 
      <Value name="one" property="matchable"/> 
      ...
    </Var> 
    ... 
  </VarSet> 
</VarSet> 

But this is a corner case that doesn't bear repeating. It's a chore to create a test file that meets this special condition, and it's complicated to stretch such a file to meet additional conditions. Moreover, it's unlikely that this special condition will have higher-order interactions with other variable combinations. So let's add once="true" to request Tcases to include this value in only one test case.

<VarSet name="file" when="fileName"> 
  ... 
  <VarSet name="contents" when="fileExists"> 
    <Var name="linesLongerThanPattern"> 
      <Value name="one" property="matchable" once="true"/> 
      ...
    </Var> 
    ... 
  </VarSet> 
</VarSet> 

Nice! But keep in mind that this hint may not always be respected. Even when once="true", a Value may be used more than once if it is needed to satisfy a constraint in remaining test cases.

Although the once shortcut applies only to a default 1-tuple for a single variable, there is also a more general way to define once-only exceptions to higher-order combinations, by adding Once elements to your generator definition.

Transforming Test Cases

The test case definitions that Tcases produces are not directly executable. Their purpose is to specify and guide the construction of actual tests. But because test case definitions appear in a well-defined XML document, it's not hard to transform them into a more concrete form, using XSLT stylesheets and output annotations.

For example, the following configuration sends Tcases output to a file named ${projectName}-Output.doc. But first it transforms the standard test case definition output document using the XSLT stylesheet in the myStylesheet.xsl file.

<plugins>
  <plugin>
    <groupId>org.cornutum.tcases</groupId>
    <artifactId>tcases-maven-plugin</artifactId>
    ...
    <configuration>
      <outFile>*-Output.doc</outFile>
      <transformDef>${basedir}/src/test/tcases/myStylesheet.xsl</transformDef>
    </configuration>
  </plugin>
  ...
</plugins>

If your XSLT stylesheet uses parameters, you can assign them values using the transformParams configuration. For example:

<plugins>
  <plugin>
    <groupId>org.cornutum.tcases</groupId>
    <artifactId>tcases-maven-plugin</artifactId>
    ...
    <configuration>
      <outFile>*-Output.doc</outFile>
      <transformDef>${basedir}/src/test/tcases/myStylesheet.xsl</transformDef>
      <transformParams>
        <param1>value1</param1>
        <param2>value2</param2>
        ...
      </transformParams>
    </configuration>
  </plugin>
  ...
</plugins>

You can even transform test case definitions into executable source code! But to do so, you may need more information about the functions, variables, and values in your input model. Try extending your input model with output annotations that are propagated to your XSLT transform.

Generating API Test Cases

Let's assume you are using the default configuration for tcases:api. Suppose you have an OpenAPI definition named ${basedir}/src/test/tcases/openapi/org/myapp/My-API.yaml

What happens when you run the following command?

mvn tcases:api

API test cases happen! You'll find them in the ${basedir}/target/tcases/openapi/org/myapp directory, in two new files named My-API-Requests-Test.json and My-API-Responses-Test.json. These JSON files describe the inputs needed for tests that fully cover both the requests and the responses defined by My-API.yaml. Each file is an example of a Tcases "test definition document" -- for details, see Tcases: The Complete Guide.

These tests "fully cover" the API? And exactly sort of "coverage" is this? Good questions. For answers, see Why Tcases for OpenAPI?.

Why separate test cases for requests and responses? Because there are two sides to the input space for your API.

This simple command will actually run Tcases for OpenAPI on all OpenAPI definitions (i.e. any *.json or *.yaml file) in the default inputDir. You can use the apiDefs, apiDef, or project parameters to process only specific API definitions. For example the following command will run Tcases for OpenAPI only on the spec(s) named My-API.json or My-API.yaml.

mvn tcases:api -Dproject=My-API

Generating Executable API Tests

Let's assume you are using the default configuration for tcases:api. Suppose you have an OpenAPI definition named ${basedir}/src/test/tcases/openapi/org/myapp/My-API.yaml

What happens when you run the following command, this time using the api-test goal

mvn tcases:api-test -Dproject=My-API 

Congratulations! You just created a new JUnit test for your API! You'll find the source for the new test class in ${basedir}/target/generated-test-sources/java/org/myapp/MyApiTest.java. Build and execute this test to run all test cases for every request defined in My-API.yaml.

How was this test constructed? And what does it really do? For answers, see Generating executable tests.

Generating Request Inputs

Let's assume you are using the default configuration for tcases:api. Suppose you have an OpenAPI definition named ${basedir}/src/test/tcases/openapi/org/myapp/My-API.yaml

What happens when you run the following command?

mvn tcases:api -Dproject=My-API -DrequestCases=true

Instead of the usual descriptions of API test cases, you'll find a different file in the ${basedir}/target/tcases/openapi/org/myapp directory named My-API-Request-Cases.json. This is a request test definition that lists actual input values for all API requests defined in My-API.yaml.

Why is this useful? A request test definition automates one step in the creation of a test program that executes all API request test cases against an actual API server. For details, see Running API Test Cases.

Handling API Input Modelling Conditions

Tcases for OpenAPI reports conditions in your OpenAPI definition that will affect how test cases are generated. Warning conditions are reported with an explanation of the situation. Error conditions report elements in your definition that may need to be changed to generate tests correctly. By default, conditions are reported by writing log messages. But you can use the onModellingCondition parameter to request different handling of such conditions.

For example, when you run the following command, any input modelling condition in your definitions will cause tcases:api to report a failure.

mvn tcases:api -DonModellingCondition=fail

What is the API Input Space?

Tcases for OpenAPI generates tests cases from the API "input space" models that it automatically derives from your OpenAPI definition. But what are they? To find out, run the following command.

mvn tcases:api -DinputModels=true

Again, assuming the default configuration for tcases:api and an OpenAPI definition named ${basedir}/src/test/tcases/openapi/org/myapp/My-API.yaml, the result will be two new files named My-API-Requests-Input.json and My-API-Responses-Input.json. Each file is an example of a Tcases "input definition document" -- for details, see Tcases: The Complete Guide.