Tcases: The JSON Guide
Version 3.4.0 (May 22, 2020)
© 2012-2020 Cornutum Project

Contents

Introduction

Traditionally, Tcases reads and produces data in the form of XML documents. But now you can do everything using JSON instead. This guide explains how.

Why JSON? Many people just find this format easier to read and write. Also, the JSON format is especially useful if you are accessing Tcases using a Web service. In fact, using JSON, there is a way to define all Tcases inputs in a single request. For details, see Running Tcases As A Web Service.

This guide assumes that you already understand what Tcases does and how it works. If not, you should start by reading Tcases: The Complete Guide. Although the examples there are all in XML, the concepts are exactly the same in JSON. Then read this guide to learn how to use JSON to create system input definitions and coverage generator definitions and how to run Tcases to generate a JSON version of a system test definition.

Each type of JSON document used by Tcases is defined by a schema that is compliant with the JSON Schema Specification, Draft 07. For complete details, see JSON Document Schemas.

Running With JSON

Getting Started

There are many ways to use Tcases. For example, you can integrate Tcases APIs directly into your Java application. Or you can run Tcases using the Tcases Maven Plugin. Or you can run Tcases as a shell command from the command line. Most examples in this guide use the Tcases shell command to illustrate how Tcases works.

To get the command line version of Tcases, download the Tcases binary distribution file from the Maven Central Repository. You can find the details here. After installing Tcases, as a quick check, you can run one of the JSON examples that comes with Tcases, using the following commands.

> cd <tcases-release-dir>
> cd docs/examples/json
> tcases -T json < find-Input.json

For details about the interface to the tcases command, see the Javadoc for the TcasesCommand.Options class. To get help at the command line, run tcases -help.

Running a JSON Project

A Tcases project must deal with several closely-related files: a system input definition, zero or more generator definitions, and the system test definition document that is generated from them (possibly in multiple forms). The tcases command implements some conventions that make it easier to keep these files organized. The same conventions used for XML files also apply to JSON files — just substitute the "json" extension in place of "xml".

For example, for a Tcases project named ${myProjectName}, if you are following the conventions, your system input definition file will be named ${myProjectName}-Input.json, and you can generate the test cases for this project by running the following command.

> tcases ${myProjectName}

Specifying JSON Content

Tcases takes its cue from the system input definition file. If it is a *.json file, then all of the files for your project are assumed to be JSON, based on standard conventions. However, if you choose not to following these conventions, you can use the -T option to explicitly specify "json" as the default file content type. In particular, this option is necessary whenever Tcases reads the system input definition from standard input. For example, the following command tells Tcases to read a JSON system input definition from standard input and write a JSON system test definition document to standard output.

> tcases -T json < ${myProjectName}.json

Defining The Input Space

Tcases creates test definitions based on a system input definition that models the "input space" of the system-under-test (SUT). What is a "system input space model"? Learn all about it here.

An Example: The find Command

The examples shown here are based on the same find command example used in the Complete Guide. You can see the complete JSON version of this system input definition here.

Defining System Functions

A JSON system input definition is an object that defines the name of the SUT and that maps the name of each system function to its input definitions. The following example shows a SUT named "Examples" that consists of a single function named "find".

{
  "system": "Examples",
  "find" : {
    ...
  }
}

Defining Input Variables

For each function to be tested, you need to define all of the dimensions of variation in its input space. Tcases refers to each such dimension as a variable .

Within the JSON definition of function inputs, variables are organized into sets by input type. Input type names can be anything you like, but it is conventional to use the name "arg" to identify direct function inputs (such as the file name argument of the find command) and the name "env" to identify indirect "environmental" inputs that affect function behavior (such as the existence of a file with the given file name). The system input definition for the find command contains variables of both of these types

{
  "system": "Examples",
  "find": {
    "arg" : {
      ...
    },
    "env" : {
      ...
    }
  }
}

Defining Input Values

For Tcases to create a test case, it must choose values for all of the input variables. How can it do that? Because we describe all of the possible values for each basic input variable using one or more value definitions.

By default, a value definition defines a valid value, one that the function-under-test is expected to accept. But we can use the optional failure property to identify an value that is invalid and expected to cause the function to produce some kind of failure response. Tcases uses these input values to generate two types of test cases — "success" cases, which use only valid values for all variables, and "failure" cases, which use a failure value for exactly one variable.

For the find command, the "fileName" variable defines two possible values: "defined" (meaning that a value for this required input is specified) and "missing" (meaning that no file name value is given — an error).

{
  "system": "Examples",
  "find": {
    "arg": {
      ...
      "fileName": {
        "values" : {
          "defined" : {
            ...
          },
          "missing" : {
            "failure": true
          }
        }
      }
    },
    "env": {
      ...
    }
  }
}

Note that the way value definitions are organized and named is entirely up to you. Your choices should reflect how you want to model the inputs for your SUT. You can find more information about modeling input values here.

Defining Variable Sets

It's common to find that a single logical input actually has lots of different characteristics, each of which creates a different "dimension of variation" in the input space. You can model this complex sort of input as a variable set.

In its JSON form, a variable set is just like a basic variable, except that instead of a "values" property containing value definitions, there is a "members" property containing definitions of the variables that are members of this variable set. In this way, you can describe a single logical input as a set of multiple variable definitions. A variable set can even contain another variable set, creating a hierarchy of logical inputs that can be extended to any number of levels.

For example, for the find command, the state of the file being searched can be represented as a variable set named "file", which consists of the "exists" variable (i.e. does the file exist or not?) and another variable set named "contents" (i.e. what kind of pattern matches does the file contain?). In turn, the "contents" variable set consists of variables "linesLongerThanPattern" (i.e. how many lines are longer than the match pattern?), "patterns" (i.e. how many instances of the pattern exist?), and "patternsInLine" (i.e. what's the maximum number of pattern instance that can be found in a single line?).

{
  "system": "Examples",
  "find": {
    "arg": {
      ...
    },
    "env": {
      "file": {
        ...
        "members" : {
          "exists": {
            ...
          }
          ...
          "contents": {
            "members" : {
              "linesLongerThanPattern": {
                ...
              },
              "patterns": {
                ...
              },
              "patternsInLine": {
                ...
              }
            }
          }
        }
      }
    }
  }
}

Defining Constraints: Properties and Conditions

Commonly, some of the "dimensions of variation" described by variable definitions are not entirely independent of each other. Instead, there are relationships among these variables that constrain which combinations of values are feasible. We need a way to define those relationships so that infeasible combinations can be excluded from our test cases. With Tcases, you can do that using properties and conditions.

Value properties

A value definition can define a property named "properties" that contains a list of properties for this value. For example:

{
  "system": "Examples",
  "find": {
    "arg": {
      ...
      "pattern": {
        "members": {
          "size": {
            "values": {
              "empty": { "properties": ["empty"]},
              "singleChar": { "properties": ["singleChar"]},
              "manyChars": {}
            }
          },
          "quoted": {
            "values": {
              "yes": { "properties": ["quoted"]},
              "no": {},
              "unterminated": { "failure": true }
            }
          },
          "blanks": {
            "values": {
              "none": {},
              "one": {},
              "many": {}
            }
          },
          "embeddedQuotes": {
            "values": {
              "none": {},
              "one": {},
              "many": {}
            }
          }
        }
      }
      ...
    }
}

Each string in the properties list is just a name that you invent for yourself to identify an important characteristic of this value. The concept is that when this value is included in a test case, it contributes all of its properties — these now become properties of the test case itself. That makes it possible for us to later define "conditions" on the properties that a test case must (or must not!) have for certain values to be included.

For example, the definition above for the pattern.size variable says that when we choose the value empty for a test case, the test case acquires a property named empty. But if we choose the value singleChar, the test case acquires a different property named singleChar. And if we choose the value manyChars, no new properties are added to the test case. Note that the correspondence between these particular names of the values and properties is not exactly accidental — it helps us understand what these elements mean — but it has no special significance. We could have named any of them differently if we wanted to.

But note that all of this applies only to valid value definitions, not to failure value definitions for which "failure" is true. Why? Because failure values are different!.

Value conditions

We can define the conditions required for a value to be included in a test case using the "when" property of a value definition. Adding a "when" condition means "for this value to be included in a test case, the properties of the test case must satisfy this boolean condition".

For example, consider the conditions we can define for the various characteristics of the "pattern" variable.

{
  "system": "Examples",
  "find": {
    "arg": {
      "pattern": {
        ...
        "members": {
          "size": {
            "values": {
              "empty": { "properties": ["empty"] },
              "singleChar": { "properties": ["singleChar"] },
              "manyChars": {}
            }
          },
          "quoted": {
            "values": {
              "yes": { "properties": ["quoted"] },
              "no": {
                "when": {
                  "hasNone": ["empty"]
                }

              },
              "unterminated": { "failure": true }
            }
          },
          "blanks": {
            ...
            "values": {
              "none": {},
              "one": {
                "when": {
                  "hasAll": ["quoted", "singleChar"]
                }

              },
              "many": {
                "when": {
                  "allOf": [
                    { "hasAll": ["quoted"] },
                    { "hasNone": ["singleChar"] }
                  ]
                }

              }
            }
          },
          "embeddedQuotes": {
            ...
            "values": {
              "none": {},
              "one": {},
              "many": {}
            }
          }
        }
      }
      ...
    },
    "env": {
      ...
    }
  }
}

This defines a constraint on the "pattern.quoted" variable. We want to have a test case in which the value for this variable is "no", i.e. the pattern string is not quoted. But in this case, the "pattern.size" cannot be "empty". Because that combination doesn't make sense, we want to exclude it from the test cases generated by Tcases.

Similarly, we define a constraint on the "pattern.blanks" variable, which specifies how many blanks should be in the pattern string. We want a test case in which the value is "many". But in such a test case, the pattern must be quoted (otherwise, a blank is not possible) and it must not be a single character (which would contradict the requirement for multiple blanks).

Condition expressions

The value of a "when" property is a boolean expression object. A boolean expression is a JSON object with a single property, which must be one of the following.

Variable conditions

You may find that, under certain conditions, an input variable becomes irrelevant. It doesn't matter which value you choose — none of them make a difference in function behavior. It's easy to model this situation — just define a "when" condition on the variable definition itself.

For example, when we're testing the find command, we want to try all of the values defined for every dimension of the "pattern" variable set. But, in the case when the pattern string is empty, the question of how many blanks it contains is pointless. In this case, the "pattern.blanks" variable is irrelevant. Similarly, when we test a pattern string that is only one character, the "pattern.embeddedQuotes" variable is meaningless. We can capture these facts about the input space by adding variable constraints, as shown below.

{
  "system": "Examples",
  "find": {
    "arg": {
      "pattern": {
        "when": {
          "hasAll": ["fileExists"]
        }
,
        "members": {
          "size": {
            "values": {
              "empty": { "properties": ["empty"] },
              "singleChar": { "properties": ["singleChar"] },
              "manyChars": {}
            }
          },
          "quoted": {
            "values": {
              "yes": { "properties": ["quoted"] },
              "no": {
                "when": {
                  "hasNone": ["empty"]
                }
              },
              "unterminated": { "failure": true }
            }
          },
          "blanks": {
            "when": {
              "hasNone": ["empty"]
            }
,
            "values": {
              "none": {},
              "one": {
                "when": {
                  "hasAll": ["quoted", "singleChar"]
                }
              },
              "many": {
                "when": {
                  "allOf": [
                    { "hasAll": ["quoted"] },
                    { "hasNone": ["singleChar"] }
                  ]
                }
              }
            }
          },
          "embeddedQuotes": {
            "when": {
              "hasNone": ["singleChar", "empty"]
            }
,
            "values": {
              "none": {},
              "one": {},
              "many": {}
            }
          }
        }
      }
      ...
    },
    "env": {
      ...
    }
  }
}

You can define variable constraints at any level of a variable set hierarchy. For example, you can see in the example above that a constraint is defined for the entire pattern variable set. This constraint models the fact that the pattern is irrelevant when the file specified to search doesn't even exist.

What effect does a variable "when" condition have on the test cases generated by Tcases? Trying running Tcases on the sample system input definition for the find command example and see for yourself how variables are marked with "NA" (not applicable) when irrelevant.

Trouble with conditions?

Careful! Faulty condition definitions can make it impossible for Tcases to work. The Complete Guide has some tips on how to avoid such trouble.

Defining Coverage Generators

For test cases generated by Tcases, the definition of "coverage" is based on combinatorial testing concepts, and test coverage is controlled by creating generator definitions. This sections shows how to create generator defintions using JSON.

Generators definitions are optional — if omitted, the default coverage generator is used for each function. By convention, generator definitions appear in a file named ${myProjectName}-Generators.json. But, if you prefer, you can specify a different file name using the -g option of the tcases command.

A Simple Generator

The simplest possible generator definition is equivalent to the default: for all functions, 1-tuple coverage for all variables. It looks like this:

{
  "*": {}
}

A Detailed Generator

A more detailed generator definition might look like the one below. This defines two generators: a simple 1-tuple generator for most functions and a more complex generator for the "find" function.

The "find" generator specifies a default of 2-tuple ("pairwise") coverage for most variables and randomizes combinations using a specific random seed. Moreover, this uses a combiner to specify a higher level of 3-tuple coverage for a specific subset of variable, including all variables in the "pattern" and "file.contents" variable sets (except for the "linesLongerThanPattern" variable).

{
  
  "*": {
    "tuples": 1
  },

  "find": {
    "tuples": 2,
    "seed": 7502311452031152128,
    "combiners": [
      {
        "tuples": 3,
        "include": ["pattern.**", "file.contents.**"],
        "exclude": ["file.contents.linesLongerThanPattern"]
      }
    ]
  }
}

Understanding Test Case Definitions

When you run Tcases with a system input definition (perhaps together with a coverage generator definition), the result is a system test definition document. For each function in the specified system, this lists a set of test case objects, each identified by a unique "id" number.

For example, try running the following commands.

> cd <tcases-release-dir>
> cd docs/examples/json
> tcases -T json < find-Input.json

You'll then see something like the following test case definitions printed to standard output.

{
  "system": "Examples",
  "find": {
    "testCases": [
      {
        "id": 0,
        ...
      },
      ...
      {
        "id": 9,
        ...
      }
    ]
  }
}

Each "success" case lists a valid value for each basic variable for the function. As in the system input definition, variables are organized by input type, such as "arg", "env", etc. Each basic variable is identified by its "path". For a top-level variable, this is simply the variable name. For a member of a variable set, the path includes the path to its parent variable set. Note that when a variable is irrelevant to a test case, it is not given a "value" but instead shows that the "NA" (not applicable) property is true.

{
  "system": "Examples",
  "find": {
    "testCases": [
      {
        "id": 0,
        "has": {
          "properties": "empty,fileExists,fileName,quoted"
        },
        "arg": {
          "fileName": {
            "value": "defined"
          },
          "pattern.blanks": {
            "NA": true
          },
          "pattern.embeddedQuotes": {
            "NA": true
          },
          "pattern.quoted": {
            "value": "yes"
          },
          "pattern.size": {
            "value": "empty"
          }
        },
        "env": {
          "file.contents.linesLongerThanPattern": {
            "NA": true
          },
          "file.contents.patterns": {
            "NA": true
          },
          "file.contents.patternsInLine": {
            "NA": true
          },
          "file.exists": {
            "value": "yes"
          }
        }
      },
      ...
    ]
  }
}

Each failure case is just the same, except that exactly one variable is assigned an invalid value, which is designated with a "failure" property that is true.

{
  "system": "Examples",
  "find": {
    "testCases": [
      ...
      {
        "id": 7,
        "arg": {
          "fileName": {
            "failure": true ,
            "value": "missing"
          },
          "pattern.blanks": {
            "NA": true
          },
          "pattern.embeddedQuotes": {
            "NA": true
          },
          "pattern.quoted": {
            "NA": true
          },
          "pattern.size": {
            "NA": true
          }
        },
        "env": {
          "file.contents.linesLongerThanPattern": {
            "NA": true
          },
          "file.contents.patterns": {
            "NA": true
          },
          "file.contents.patternsInLine": {
            "NA": true
          },
          "file.exists": {
            "NA": true
          }
        }
      },
      ...
    ]
  }
}

Note also that each test case also lists a "has" property which shows all of the value properties that are associated with the test case. The "has" property also contains the value of every output annotation associated with the test case.

{
  "system": "Examples",
  "find": {
    "testCases": [
      {
        "id": 0,
        "has": {
          "properties": "empty,fileExists,fileName,quoted"
        }
,
        "arg": {
          "fileName": {
            "value": "defined"
          },
          "pattern.blanks": {
            "NA": true
          },
          "pattern.embeddedQuotes": {
            "NA": true
          },
          "pattern.quoted": {
            "value": "yes"
          },
          "pattern.size": {
            "value": "empty"
          }
        },
        "env": {
          "file.contents.linesLongerThanPattern": {
            "NA": true
          },
          "file.contents.patterns": {
            "NA": true
          },
          "file.contents.patternsInLine": {
            "NA": true
          },
          "file.exists": {
            "value": "yes"
          }
        }
      },
      ...
    ]
  }
}

Running Tcases As A Web Service

This section assumes you want to integrate Tcases into a Java-based Web service, submitting requests for Tcases services and getting generated test cases in response.

Defining a Tcases Project

You can describe all of the elements of Tcases project in a JSON document called a project definition.. A project definition contains a system input definition (or a reference to its location). Optionally, a project definition may also contain a generator definition and a system test definition that supplies the base tests for generating new test cases.

Defining Elements Directly

The simplest possible project definition consists of only a JSON system input definition.

{
  "inputDef": {
    ...
  }
}

A more complete project definition could also include a JSON generator definition and a set of JSON base test case definitions.

{
  "inputDef": {
    ...
  },
  "generators": {
    ...
  },
  "baseTests": {
    ...
  }
}

Defining Element Locations

Alternatively, any of the elements of a Tcases project can be defined by a URL for the corresponding JSON document.

{
  "inputDef": "http://www.cornutum.org/tcases/docs/examples/json/find-Input.json",
  "generators": "http://www.cornutum.org/tcases/docs/examples/json/find-Generators.json",
  "baseTests": "http://www.cornutum.org/tcases/docs/examples/json/find-Tests.json"
}

Document locations can either be absolute URLs or URIs relative to a specified "refBase" property. A project definition can also contain a mix of actual JSON documents and URL references.

{
  "refBase": "http://www.cornutum.org/tcases/docs/examples/json",
  "inputDef": "find-Input.json",
  "generators": "find-Generators.json",
  "baseTests": {
    ...
  }
}

Running a Tcases Project

Given an input stream that reads a JSON project definition, you easily generate project test cases and send the resulting JSON system test definition document to an output stream, using methods of the TcasesJson class.

TcasesJson.writeTests
  ( TcasesJson.getTests( projectInputStream),
    testCaseOutputStream);

More Fun With JSON

There's a lot more to learn about Tcases that is beyond the scope of this guide. As always, look to the Complete Guide for all the details.

But here are a few more things that all JSON users ought to know.

Reducing Test Cases

A random walk through the combinations may lead Tcases to a smaller set of test cases. You can automate that process by running the Tcases Reducer. Naturally, the tcases-reducer command also supports JSON projects, using the same conventions and command line options as tcases command.

For example, the following commands will produce a new find-Generators.json file that reduces test cases for the find-Input.json input model.

> cd <tcases-release-dir>
> cd docs/examples/json
> tcases-reducer find

Avoiding Unneeded Combinations

Sometimes there is a variable value that you need to test at least once, but for various reasons, including it multiple times adds complexity without really increasing the likelihood of finding new failures. In this case, you can use the "once" property as a hint to avoiding reusing a value more than once.

For example, when definining the "file.contents.linesLongerThanPattern" variable, setting "once" to true for the value "one" tells Tcases to generate only one test case using that value.

{
  "system": "Examples",
  "find": {
    "arg": {
      ...
    },
    "env": {
      "file": {
        ...
        "members": {
          ...
          "contents": {
            ...,
            "members": {
              "linesLongerThanPattern": {
                "values": {
                  "one": {
                    "once": true ,
                    "properties": ["matchable"]
                  },
                  ...
                }
              },
              ...
            }
          }
        }
      }
    }
  }
}

Actually, the "once" attribute is just a special shortcut that applies only when using the default 1-tuple coverage. But you can make the same kind of exception for higher-order tuples by adding "once" tuple lists to your generator definition.

For example, the following generator definition for the "find" function specifies 2-tuple coverage for all variables but tells Tcases to create only one test case that uses a certain 2-tuple combination: a quoted multi-character pattern.

{
  "find": {
    "combiners": [
      {
        "tuples": 2,
        "once": [
          {
            "pattern.size": "manyChars",
            "pattern.quoted": "yes"
          }
        ]

      }
    ]
  }
}

Using Output Annotations

An output annotation is a special property setting — a name-value pair — that you can add to various elements of a system input definition. It has no effect on test cases that Tcases generates. But Tcases will accumulate output annotations and attach them to the resulting system test definition document.

You can see illustrations of the following kinds of output annotations in the annotations-Input.json example.

Transforming Test Cases

The test case definitions generated by Tcases for Open API are not directly executable. Their purpose is to specify and guide the construction of actual tests. But because test case definitions can appear in a well-defined format, it's possible transform them into a more concrete form. Here are some of the options possible.

JSON Document Schemas

The JSON documents used by Tcases are defined by the following schemas, which are compliant with the JSON Schema Specification, Draft 07.