# UTE Manual

## Version: 0.0.1

List of Tables

1.1. `xrun` command options
1.2. Common harness settings and options
1.3. Harness settings for `tonga`
1.4. Harness settings for `jtreg`
1.5. Harness settings for `javatest`
1.6. Harness settings for `jck`
1.7. Harness settings for `refworkload`
1.8. Harness settings for `rtpresto2`
1.9. Harness settings for `rtpresto3`
1.10. Harness settings for `bigapps`
1.11. Harness module functions
1.12. Harness internal variables
1.13. Utility functions
1.14. Phare settings

## Chapter 1. Unified Test Execution (UTE)

## 1.1. Introduction

UTE is a tool for test execution automation. UTE unifies execution for different test harnesses. UTE has modular architecture, where test harness is pluggable module.

This manual is intended for all people who run tests and for quality engineers. It describes both how to use UTE and how to extend it.

Note that UTE deals only with test harness configuration preparation and execution. Full testrun preparation (including things like JDK, browser, testbase installation, exclude and known failure list preparation, results postprocessing) and distributed test execution are outsid e of scope of UTE and may be provided by GTEE or done manually.

This manual consists of several parts.

- to begin with UTE, read Section 1.2, “Concepts”,
- for examples of standalone usage of UTE, refer to Section 1.3.4, “Examples of UTE standalone usage”,
- for examples of usage of UTE with GTEE, refer to Section 1.4.2, “Examples of UTE usage with GTEE”,
- to get information about how to use UTE standalone, refer to Section 1.3, “Standalone usage”,
- to get information about how to use UTE with GTEE, refer to Section 1.4, “Usage with GTEE”,
- to get details on harness modules, refer to Section 1.5, “Harness modules”,
- to learn how to add or modify harness module, refer to Section 1.6, “Harness interface”.

## 1.2. Concepts

Test harness is a tool to run tests. Different test harnesses have different capabilities, setup and actual way of execution. Examples of test harness include Tonga, JavaTest, jtreg and others.

Testbase (sometimes referred to as workspace) is a collection of tests. Testbase is usually located inside particular directory on file system (referred as testbase location). Testbase is executed with particular test harness. Examples of testbase include VM Testbase, other SQE Testbases, JCK Runtime 6 (or other version), JDK or HotSpot regression tests.

Test component identifies class of testing that is performed. Among the examples are SQE functional components (`vm, jmx, swing,` etc.), `jck, javase` (tests from JavaSE workspace), `bigapps.` For UTE, normally test component has only informational value and no impact on functionality (`tonga` and `dtonga` are notable exceptions).

Product is the program being tested (for example JDK 6u10 b26).

Testrun is a particular execution of tests.

Configuration of testrun may include several parameters, for example, particular product, product command-line options, harness module, testbase, test options or configuration, test lists or suites, other required entities (such as browsers and plugins).

Harness module is a part of UTE which allows to run testing in a particular way with particular harness. In this manual, harness module is sometimes simply referred as harness. During testrun, one particular harness module is used.

Result directory is the directory where data related to testrun is stored. It usually contains test results, configuration files, intermidiate data files and scripts, testrun and test log files.

## 1.3. Standalone usage

### 1.3.1. Overview

In the simplest case, UTE can be used standalone for quick execution of few tests with little configuration. If installed and configured with GTEE, it is enough to specify JDK location and test name (refer to Section 1.3.2, “Usage of `xrun` command”): UTE will correctly auto-detect settings, configure test harness (including different lists) and do the testing (provided that `xrun` has been set up properly, see Section 1.3.6, “UTE setup for `xrun` command”). Most of the settings can be overriden by additional options.

`xrun` command uses `tlists` component configuration from GTEE to determine the component name for the test specified in the command line. This component name also determines how which lists are generated for `tlists` and how UTE configures the testrun, including the information about test harness, testbase and other locations (there is one configuration for each component).

`run` command may be used in cases when automatic configuration is not necessary and all settings are specified in command line. Note that `xrun` command itself is implemented on top of `run` command.

UTE is run directly from command line:

    ute -jdk [jdk] -test [test] ...       (recommended, equivalent to xrun command)
    ute xrun -jdk [jdk] -test [test] ...
    ute run -harness [harness] ...

Refer to

- Section 1.3.2, “Usage of `xrun` command” for description of `xrun` command,
- Section 1.3.3, “Usage of `run` command” for description of `run` command,
- Section 1.3.4, “Examples of UTE standalone usage” for list of examples of UTE usage,
- Section 1.5.2, “Common settings and options” for description of common command line options and settings,
- Sections under Section 1.5, “Harness modules” for options and settings specific to harness module.

### 1.3.2. Usage of `xrun` command

`xrun` command supports all common and harness-specific options (refer to Section 1.5.2, “Common settings and options” and subsequent sections). Note that the only required options are `-jdk` and `-test.` The values of other options are normally derived from `xrun` setup (refer to Section 1.3.6, “UTE setup for `xrun` command”).

The current directory is used for results directory by default.

In addition, the following options are supported:

Table 1.1. `xrun` command options

Name | Command line option | Set by | Default value | Description
--- | --- | --- | --- | ---
TLISTS | -tlists | user |  | `tlists` processing mode

Refer to Section 1.3.4, “Examples of UTE standalone usage” for specific examples.

### 1.3.3. Usage of `run` command

`run` command also supports all common and harness-specific options (refer to Section 1.5.2, “Common settings and options” and subsequent sections). However, unlike `xrun,` the values of these options have to be specified: nothing is derived from setup.

### 1.3.4. Examples of UTE standalone usage

- Run `vm` test `jit/Arrays/ArrayBounds` with JDK 6u10:

      ute -jdk /net/vmsqe.russia/export/jdk/6/build/u10/solaris-i586 -test jit/Arrays/ArrayBounds

- Run `vm` test `nsk/logging/ConsoleHandler/close/close001` from VM testbase 1.5 with JDK 6u10 in `comp` mode with `-XX:CompileThreshold=2` option:

      ute -jdk /net/vmsqe.russia/export/jdk/6/build/u10/solaris-i586 -test nsk/logging/ConsoleHandler/close/close001 \
      -testbase /net/vmsqe.russia/export/testbase/sqe/vm/1.5/build/execution/vm -vmmode comp -vmopts -XX:CompileThreshold=2

- Run JavaSE (regression) test with JDK 6u10, 64-bit mode and `-server -Xint` options:

      ute -jdk /net/vmsqe.russia/export/jdk/6/build/u10/solaris-amd64 -test closed/compiler/6724218/Test.java -bits d64 -vmopts '-server -Xint'

- Run `runThese` `quick` suite with JDK 6u10:

      ute run -jdk /net/sqenfs-1.sfbay/export1/comp/vm/jdk/re/6u10/latest/binaries/linux-i586 -test runThese/quick

- Run tests with `rtj/dtrace/` prefix (from `realtime` testbase):

      ute -jdk /net/vmsqe.russia/export/jdk/rtj/2.1/build/b11/solaris-i586 -test rtj/dtrace/

- Run `dtrace/DTrace_rtsjProvider.java` from Java RTS testbase:

      ute -jdk /net/mackdrive.france/export/jrts_sqe/jdk/rtj/2.1/build/b11/solaris-i586/ -test dtrace/DTrace_rtsjProvider.java

- Run 2 iterations of `specjvm98 (refworkload)`

      ute run -harness refworkload -testbase /net/vmsqe.russia/export/testbase/refworkload \
      -jdk /net/vmsqe.russia/export/jdk/rtj/2.1/build/b11/solaris-i586 -results . -benchmark specjvm98 -iterations 2

- Run 10 minutes of `Tomcat` bigapp:

      ute run -harness bigapps -testbase /net/vmsqe.russia/export/testbase/bigapps/Tomcat/apache-tomcat-5.5.26 \
      -jdk /net/vmsqe.russia/export/jdk/rtj/2.1/build/b11/solaris-i586 -results . \
      -runbigapp /net/vmsqe.russia/export/tool/runbigapp.sh -bigapp Tomcat -duration 10

### 1.3.5. UTE setup for `run` command

The following steps are required for UTE installation. These steps are only needed for standalone UTE. If UTE is installed as part of GTEE `(UTE_HOME = $GTEE_HOME/harness/ute),` it may be used as is without additional setup.

- Put UTE to a directory `(UTE_HOME)` either by unpacking it, or by checking out from repository.
- Ensure that `UTE_HOME/stable-jdk` points to directory with binaries for stable JDK.
- `etc/?`

### 1.3.6. UTE setup for `xrun` command

Setup for UTE command `xrun` requires effort and knowledge of test configuration for particular component and locations for particular site.

For each component, there is a component script under `$UTE_HOME/etc/component` subdirectory which is sourced in context of `xrun` command once the component is determined (and has access to UTE internal variables). The script is responsible for configuring any additional settings for the component. At least `HARNESS` and `TESTBASE` and mandatory settings for the harness should be set.

It is recommended for the component script to set most of the values to make it easier for users: any settings will not set have to be set by the user in the command line for each run.

Note that the settings may depend on JDK release being tested: the value is accessible as `jdk_release` variable.

Note that GTEE includes common configurations for most components (under `site/common/etc/ute`) which assumes certain structure of directories for particular site. and allow to define only base directory.

## 1.4. Usage with GTEE

### 1.4.1. Overview

UTE provides `Run.sh` which may be used as GTEE suite. In GTEE installation, UTE suite is usually available as `ute` and is specified in job definition file as

    <workspacename>ute</workspacename>

In this case, UTE is configured naturally: user-defs and other elements in job definition file are transformed to UTE settings. There are two mandatory settings in job definition file for UTE: `HARNESS` and `TESTBASE.` Some harness modules may also require other settings. Refer to Section 1.5.2, “Common settings and options” for description of common settings and to corresponding section under Section 1.5, “Harness modules” for settings specific to harness module.

### 1.4.2. Examples of UTE usage with GTEE

Refer to Section 1.5, “Harness modules” for description of the settings.

- Job definition file fragment for `vm` component (refer to VM Testbase 6 documentation):

      	<user-defs key="HARNESS" value="tonga"/>
      	<user-defs key="TESTBASE" value="/net/vmsqe.russia/export/testbase/sqe/vm/6/build/nightly/vm"/>
      	<user-defs key="TONGA_JAR" value="/net/vmsqe.russia/export/tool/tonga/tonga.jar"/>
      	<user-defs key="TONGA_OPTS" value="-suiteDir src -localTZ"/>
      	<user-defs key="JCK12a" value="/net/vmsqe.russia/export/testbase/jck/1.2"/>
      	<user-defs key="JCK122" value="/net/vmsqe.russia/export/testbase/jck/1.2.2"/>
      	<user-defs key="JCK60" value="/net/vmsqe.russia/export/testbase/jck/6a/dist"/>
      	<user-defs key="JCK60_LIB_PATH" value="/net/vmsqe.russia/export/testbase/jck/6a/jckjni"/>
      	<user-defs key="DISPLAY" value="vmsqe-p4-08:1"/>
      ...
      	<testlistset includes="spb.nightly.testlist">

- Job definition file fragments for `HS_REGRESION` and `JDI_REGRESSION` subsets of Java SE regression tests:

      	<user-defs key="HARNESS" value="jtreg"/>
      	<user-defs key="JTREG_HOME" value="/net/sqenfs-1.sfbay/export1/comp/vm/tool/jtreg_4.0"/>
      	<user-defs key="TESTBASE" value="/net/sqenfs-1.sfbay/export1/comp/vm/testbase/jtreg/7"/>
      ...
      	<testlistset includes="HS_REGRESSION,JDI_REGRESSION"/>

- Job definition file fragment for `quick` suite of `runThese:`

      	<user-defs key="HARNESS" value="runThese"/>
      	<user-defs key="TESTBASE" value="/net/sqenfs-1.sfbay/export1/comp/vm/testbase/runThese"/>
      ...
      	<testlistset includes="quick">

- Job definition file fragment for `realtime` component:

      	<user-defs key="HARNESS" value="tonga"/>
      	<user-defs key="TESTBASE" value="/net/vmsqe.russia/export/testbase/sqe/realtime/build/execution"/>
      	<user-defs key="TONGA_JAR" value="/net/vmsqe.russia/export/tool/tonga/tonga.jar"/>
      	<user-defs key="TONGA_OPTS" value="-suiteDir src"/>

- Job definition file fragment for `javarts` tests:

      	<user-defs key="HARNESS" value="javatest_simple"/>
      	<user-defs key="TESTBASE" value="/net/mackdrive.france/export/jrts_sqe/testbase/jrts"/>
      ...
      	<testlistset includes="testlist">

- Job definition file fragment for 10 iterations of `specjbb2005 (refworkload):`

      	<user-defs key="HARNESS" value="refworkload"/>
      	<user-defs key="TESTBASE" value="/net/vmsqe.russia/export/testbase/refworkload"/>
      	<user-defs key="BENCHMARK" value="specjbb2005"/>
      	<user-defs key="ITERATIONS" value="10"/>

- Job definition file fragment for 120 hours of `Tomcat` bigapp:

      	<user-defs key="HARNESS" value="bigapps"/>
      	<user-defs key="HARNESS_LOCATION" value="/net/vmsqe.russia/export/tool/runbigapp.sh"/>
      	<user-defs key="TESTBASE" value="/net/vmsqe.russia/export/testbase/bigapps/Tomcat/apache-tomcat-5.5.26"/>
      	<user-defs key="BIGAPP" value="Tomcat"/>
      	<user-defs key="DURATION" value="7200"/>

## 1.5. Harness modules

### 1.5.1. Overview

This section is intended for UTE users and describes common module parameters and all currently included harness modules and their parameters. Refer to

- Section 1.3, “Standalone usage” and Section 1.4, “Usage with GTEE” for ways to configure and run UTE.
- Section 1.6, “Harness interface” for detailed description of interface between UTE and harness module and information about how to add or modify harness module.

In UTE, each harness module has an identifier (also the argument to `-harness`).

### 1.5.2. Common settings and options

The following table provides user-visible common settings and options.

Table 1.2. Common harness settings and options

Name | Command line option | Set by | Default value | Description
--- | --- | --- | --- | ---
- | -env | user |  | Arbitrary setting in `name=value` format
- | -envfile | user |  | Arbitrary setting file containing assignments in `name=value` format
HARNESS | -harness | user |  | Test harness being executed
TESTBASE | -testbase | user |  | Location of testbase
COMPONENT | -component | user |  | Test component
TESTLIST | -testlist | user (optional) |  | Testlist to execute
TESTS | -test | user (optional) |  | Tests to execute (interpretation depends on harness module)
RES_DIR | -results | user | - | Result directory
TEST_JAVA_HOME | -jdk | user | - | Product location
VM_OPTS | -vmopts | user (optional) |  | Additional VM options to product
JAVA_32_64_FLAG | -bits | user (optional) |  | Bits flag (d32 or d64)
VM_FLAVOR | -vmflavor | user (optional) |  | VM flavor
VM_MODE | -vmmode | user (optional) |  | VM mode
PLATFORM | -platform | user (optional) |  |
JDK_ARCH | -jdkarch | user (optional) |  | Test JDK architecture
JAVA_EXE | -javaexe | user (optional) | java | Name of java executable
STABLE_JAVA_HOME | -stablejavahome | user (optional) |  | Stable JDK location
STABLE_VMOPTS | -stablevmopts | user (optional) |  | Stable JDK options to use
RUN_SHELL | -shell | user (optional) | ksh | Shell used for scripts
WORK_SUBDIR | -worksubdir | user (optional) | workDir | Subdirectory of result directory to store test harness work results
REPORT_SUBDIR | -reportsubdir | user (optional) | report | Subdirectory of result directory to store test harness reports
DISPLAY | -display | user (optional) |  | X11 DISPLAY variable
SETTINGS | -settings | user (optional) |  | Additional settings file in `name=value` format

### 1.5.3. `tonga` - UTB-compatible Tonga

`tonga` module is UTB-compatible module for Java SE SQE functional testbase. UTB (Unified Test Base) is a collection of scripts to execute SQE functional tests. UTB provides uniform way to run different SQE Testbases (vm, libs, awt, ...).

Note that support for some UTB features is limited, and some do not work. Only features related to harness configuration and test execution itself (section 1.1 in UTB documentation) are supported. Features that are provided by test execution environment (e.g. JDK installation, local mapping) are omitted intentionaly.

#### 1.5.3.1. Configuration

The following table provides user-visible settings for `tonga` module. For other settings refer to Table 1.2, “Common harness settings and options”. Note that component may define other settings, some of which may be mandatory (refer to component documentation).

Table 1.3. Harness settings for `tonga`

Name | Command line option | Set by | Default value | Description
--- | --- | --- | --- | ---
COMPONENT, J2SE_COMPONENT_DIR | -component | user |  | SQE functional component. Required.
SUBCOMPONENT | -subcomponent | user |  | Subcomponent (optional)
TESTBASE | -testbase | user |  | Location of testbase (note: includes component subdirectory)
TESTLIST | -testlist | user | testlist | Testlist to use
KEYWORDS | -keywords | user | !manual | Keyword expression to filter tests by
TONGA_JAR | -tonga | user |  | Path to Tonga JAR
TONGA_OPTS | -tongaopts | user |  | Additional options to Tonga
TONGA_MAIN_CLASS | - | user | Tonga | Main class to run from Tonga JAR file.
GENERATE_RERUN_SCRIPT |  | user | true | true to generate rerun script for test
Options processed by component script |  |  |  |
DISPLAY | -display | user |  | X11 DISPLAY setting
KNOWN_FAIL_LIST |  | user |  | Known failure list for Tonga
EXCLUDE_LIST |  | user |  | Exclude list for Tonga
TONGA_REMOVE |  | user | PASS | Tests to remove results for (PASS, NONE, ALL)

#### 1.5.3.2. Implementation details

Part of `tonga` module is also used for `dtonga` module, so its code must be changed carefully. `tonga_setup` is split in two parts: `tonga_setup_common` which is also used by `dtonga` and `tonga_setup_component` which is only used by `tonga` module itself.

#### 1.5.3.3. Component interface

This component interface is completely identical to component interface between UTB and SQE component. Each SQE component (also a subdirectory in SQE testbase) has it's own `Run_component.sh` which contains component-specific definitions. This script is provided by component owners.

Because in UTE `TESTBASE` already points to component subdirectory, this is also where `Run_component.sh` is located. This script is expected to contain shell functions that define the interface between `tonga` module and component. Note that component script only expected to prepare Tonga INI file and the execution of Tonga itself is done by UTE.

The following functions are used:

- `create_tonga_ini inifile` (mandatory) - is called by `tonga` module to create Tonga ini file for execution with one parameter - path to Tonga ini file. The settings for testrun can be read and used as environment variables.
- `component_preparation` (optional) - may be used for component-specific initialization.
- `component_initialize` (optional) - may be used for component-specific early initialization. This should only be used to override default values of some UTE variables. When this function is called, UTE is not yet fully initialized, so it should be used with caution.
- `component_post_cleanup` (optional) - may be used for component-specific cleanup.

### 1.5.4. `dtonga` - DTonga (distributed Tonga)

#### 1.5.4.1. Configuration

#### 1.5.4.2. Implementation details

### 1.5.5. `jtreg` - Jtreg, OpenJDK test harness based on JavaTest

#### 1.5.5.1. Configuration

The following table provides user-visible settings for `jtreg` module (for other settings refer to Table 1.2, “Common harness settings and options”):

Table 1.4. Harness settings for `jtreg`

Name | Command line option | Set by | Default value | Description
--- | --- | --- | --- | ---
JTREG_HOME, JCT_TOOLS | -jtreg | user |  | Location of jtreg harness. Use JTREG_HOME (JCT_TOOLS exists only for backward compatibility and may be removed in the future)
TIMEOUT_FACTOR | -tfactor | user | 1 | Timeout factor (all test timeouts are multiplied by this positive number)
JTREG_OPTS | -jtregopts | user |  | Additional options to jtreg
TESTLIST | -testlist | user |  | Test list (a subdirectory of TESTBASE). It may be used to select particular testbase (e.g. JDI_REGRESSION) or subdirectory of tests.
WORK_SUBDIR |  | user | workDir | Subdirectory of result directory to be used as work directory for JavaTest
REPORT_SUBDIR |  | user | report | Subdirectory to store JavaTest report
JTR_IF_PASSED |  | user | false | If true, retain .jtr log files for passed tests, do not otherwise

#### 1.5.5.2. Implementation details

TBD

### 1.5.6. `javatest` - JavaTest basic support

#### 1.5.6.1. Configuration

`javatest` harness module provides generate template-based support for JavaTest harness. It creates javatest config file from provided template and UTE settings. Javatest property `"a.b.c"` corresponds to setting `a_b_c.`

The following table provides user-visible settings for `javatest` module (for other settings refer to Table 1.2, “Common harness settings and options”):

Table 1.5. Harness settings for `javatest`

Name | Command line option | Set by | Default value | Description
--- | --- | --- | --- | ---
JAVATEST_JAR | -javatest | user |  | Location of JavaTest JAR.
TESTLIST | -testlist | user |  | Test list (a subdirectory of TESTBASE). It may be used to select particular testbase (e.g. JDI_REGRESSION) or subdirectory of tests.
WORK_SUBDIR |  | user | workDir | Subdirectory of result directory to be used as work directory for jtreg
REPORT_SUBDIR |  | user | report | Subdirectory to store jtreg report
JAVATEST_CONFIG_TEMPLATE |  | user |  | Template for JavaTest config
JTR_IF_PASSED |  | user | false | If true, retain .jtr log files for passed tests, do not otherwise

#### 1.5.6.2. Implementation details

### 1.5.7. `javatest_simple` - JavaTest SimpleTest harness

#### 1.5.7.1. Configuration

`javatest_simple` harness module is an extension of `javatest` module which also automatically sets `SimpleHarness` variables in the template.

`javatest_simple` harness module does not provide additional user-visible settings. Refer to Table 1.5, “Harness settings for `javatest`” for information about `javatest` harness module settings.

#### 1.5.7.2. Implementation details

### 1.5.8. `jck` - JCK

#### 1.5.8.1. Configuration

`jck` harness module is an extension of `javatest` module which offers some additional features:

- JCK configuration template is included
- Some JCK configuration settings are set automatically
- JCK agent and tnameserv are started for the run
- Exclude list is converted to javatest format

The following table provides user-visible settings for `jck` module (for other settings refer to Table 1.2, “Common harness settings and options” and Table 1.5, “Harness settings for `javatest`”):

Table 1.6. Harness settings for `jck`

Name | Command line option | Set by | Default value | Description
--- | --- | --- | --- | ---
JCK_START_AGENT | -agent | user | true | true to start JCK agent
JCK_START_TNAMESERV | -tnameserv | user | true | true to start tnameserv

#### 1.5.8.2. Implementation details

### 1.5.9. `runThese` - runThese, reliability harness

#### 1.5.9.1. Configuration

`runThese` harness module does not have any additional settings. Refer to Table 1.2, “Common harness settings and options” for information about common settings.

#### 1.5.9.2. Implementation details

### 1.5.10. `refworkload` - refworkload, collection of performance benchmarks

#### 1.5.10.1. Configuration

Table 1.7. Harness settings for `refworkload`

Name | Command line option | Set by | Default value | Description
--- | --- | --- | --- | ---
BENCHMARK | -benchmark | user |  | Refworkload benchmark
ITERATIONS | -iterations | user |  | Number of iterations

Note: in addition to these settings, `refworkload` harness module supports Phare submission. Refer to Section 1.7.1, “Phare support” for more information.

#### 1.5.10.2. Implementation details

### 1.5.11. `rtpresto2` - rtpresto2, Java Realtime benchmark

#### 1.5.11.1. Configuration

Table 1.8. Harness settings for `rtpresto2`

Name | Command line option | Set by | Default value | Description
--- | --- | --- | --- | ---
PERIOD | -period | user |  | Period, in microseconds
ITERATIONS | -iterations | user |  | Number of iterations
TOOLS | -tools | user |  | Base location of GNU tools (should include gnuplot)

#### 1.5.11.2. Implementation details

rtpresto2 is called through make. Under Linux, `prep-linux.sh` is called first.

### 1.5.12. `rtpresto3` - rtpresto3, Java Realtime benchmark

#### 1.5.12.1. Configuration

Table 1.9. Harness settings for `rtpresto3`

Name | Command line option | Set by | Default value | Description
--- | --- | --- | --- | ---
PERIOD | -period | user |  | Period, in microseconds
ITERATIONS | -iterations | user |  | Number of iterations
TOOLS | -tools | user |  | Base location of GNU tools (should include gnuplot)
GNUPLOT | -gnuplot | user |  | Location of gnuplot
CFGS | -cfgs | user |  | Configurations to run (whitespace separated)
PROPS | -props | user |  | Properties file to use
DSCRIPT | -dscript | user |  | dscript to use
PRE | -pre | user |  | Enable creation of precompilation and preinitialization lists (true/false)

Note: in addition to these settings, `rtpresto3` harness module supports Phare submission. Refer to Section 1.7.1, “Phare support” for more information.

#### 1.5.12.2. Implementation details

### 1.5.13. `bigapps` - runbigapp, common BigApps harness

`bigapps` module is the integration module for runbigapp.sh, a script for BigApps automation.

#### 1.5.13.1. Configuration

The following table provides user-visible settings for `bigapps` module (for other settings refer to Table 1.2, “Common harness settings and options”):

Table 1.10. Harness settings for `bigapps`

Name | Command line option | Set by | Default value | Description
--- | --- | --- | --- | ---
HARNESS_LOCATION | -runbigapp | user |  | Location of `runbigapp` harness (directory)
BIGAPP | -bigapp | user |  | BigApp to run
TESTBASE | -testbase | user |  | Location of BigApp (subdirectory in BigApps workspace)
EXEC_DURATION | -duration | user |  | Duration of run in minutes

Note: in addition to these settings, `bigapps` harness module supports Phare submission. Refer to Section 1.7.1, “Phare support” for more information.

#### 1.5.13.2. Implementation details

`bigapps` module is very simple. After doing validation of settings, runbigapp.sh is called with a number of parameters.

## 1.6. Harness interface

### 1.6.1. Overview

Harness interface is a convention between UTE and a harness module.

Harness module is parsable shell script `$UTE_HOME/harness/$harness` (where `$harness` is the harness module identifier). This shell script must define a number of shell functions, some of which are mandatory and others are optional. These functions are called by UTE during harness module lifecycle to perform particular task (refer to Section 1.6.2, “Harness module functions”). Harness module may use additional tools which are placed under `$UTE_HOME/tool/$tool/` (where `tool` is the tool identifier) and data files which are placed under `$UTE_HOME/data/.`

UTE immediately fails if any command exits with non-zero status (except test harness, the exit code of which is processed). It means that harness module must be written to process expected non-zero exit codes. Additionally, it is recommended to validate any settings before usage to avoid starting testrun with wrong settings.

Harness module functions are expected to use a number of internal variables set by UTE. Refer to Table 1.12, “Harness internal variables” for list of these variables. Additionally, harness module may expect user to define a number of additional settings, in which case, it should validate them before execution.

Additionally, UTE defines a number of utility functions which are recommended to be used by harness modules (refer to Section 1.6.4, “Utility and validation functions”).

### 1.6.2. Harness module functions

By convention, all harness module functions are prefixed with the identifier of harness module.

The following table provides the list of functions. For particular harness module, `harness` is expected to be replaced with harness module identifier.

Table 1.11. Harness module functions

Name | Mandatory | Description
--- | --- | ---
harness_setup | yes | Testrun preparation
harness_run | yes | Testrun execution
harness_initialize | no | Harness module initialization (called early before and before `harness_setup)`)
harness_postprocess |  | Harness module cleanup (called late after `harness_run`)

### 1.6.3. Variables

Several kinds of variables are accessible to harness module:

- Settings provided by user
- Auxiliary variables set by UTE
- Auxiliary variables set by harness module itself

Some of user settings are mandatory and validated by UTE, thus there is no need for harness module to validate them. In addition, UTE defines or calculates a number of internal variables which are then used by harness module. Note that these variables allow for flexible test configuration and their usage is mandatory for harness module.

There may be harness module-specific settings (mandatory or optional). There are no constrains or limitations on these settings, although there are several recommendations:

- The number of mandatory settings should be kept to minimum to simplify user experience.
- At the same time, it is recommended to allow for flexible reconfiguration of testrun (ideally as flexible as test harness capabilities). Last two items together mean that testrun configuration is defined by a number of parameters, most of which have sensible default value.
- All settings should always be validated during `harness_setup` or `harness_initialize` steps. In most cases, one of validation functions should be used (refer to Section 1.6.4, “Utility and validation functions”).

Refer to Section 1.5.2, “Common settings and options” for list of settings provided by user.

The following table provides the list of internal UTE variables.

Table 1.12. Harness internal variables

Name | Command line option | Set by | Default value | Description
--- | --- | --- | --- | ---
UTE_HOME | - | UTE | - | UTE location
PLATFORM | - | UTE | - | Machine platform (e.g. solaris-sparc, linux-amd64, windows-i586)
JDK_ARCH | - | UTE |  | Product architecture (e.g. solaris-sparc, linux-amd64, windows-i586)
PS | - | UTE |  | Path component separator (':'/';')
TEST_JAVA | - | UTE |  | Full path to java binary (from TEST_JAVA_HOME and JAVA_EXE)
VM_OPTS_ALL | - | UTE |  | All product options
STABLE_JAVA | - | UTE |  | Full path to stable java binary

### 1.6.4. Utility and validation functions

Note: all validation functions immediately fail (and cause UTE to fail with non-zero exit code) if validation fails.

The following table provides the list of supported utility and validation functions:

Table 1.13. Utility functions

Usage | Description |
`info "message", debug "message", warn "message", error "message` | Log message at corresponding log level |
`fail "message"` | Fail current function with given message |
`ivar_set "name" "value" ivar_get "name"` | Set or get variable with given name (indirectly) |
`harness_run command arg ...` | Run harness with given arguments |
`process_start id command arg ... process_stop id` | Start or stop background process with given command line |
`set_default_var name value` | Set value of variable `name` to `value` if it has not been set yet (by user) |
`check_not_empty name` | Check that variable with name `name` is set (has non-empty value) |
`check_dir dir` | Check that `dir` is existing directory |
`check_dir_var name` | Check that variable `name` contains an existing directory |
`check_file filename` | Check that `filename` is an existing file |
`check_file_var name` | Check that variable `name` contains an existing file |

## 1.7. Other features

### 1.7.1. Phare support

Phare is Performance Hierarchical Analysis and Reporting Environment. It is used for performance, predictability and other statistical results. UTE supports submission of results to Phare (for some harnesses).

Table 1.14. Phare settings

Name | Command line option | Set by | Default value | Description
--- | --- | --- | --- | ---
PHARE | -phare | user | false | Set `true` for enabling Phare submission
PHARE_HOME |  | user |  | Phare home directory
PRODUCT_NAME |  | user |  | Product name
RUN_TYPE |  | user |  | Run type
BENCHMARK |  | benchmark |  | Benchmark
BENCHMARK_CONFIG |  | user |  | Benchmark config
PRODUCT_CONFIG |  | user |  | Product config