A Testcase is created through the "Create Testcase" menu option. A Testcase must exist within a TestSuite, so it is necessary to first create a TestSuite before creating Testcases.
When a Testcase is first created, it will be given a default name. The name can be amended by entering a new name in the edit box and pressing the Update button on the Testcase editor. However, the name of the Testcase must be unique among all other Testcases at the same level of the hierarchy.
There is a wealth of literature describing the concept of a Testcase, and techniques for designing effective Testcases in order to most efficiently exercise the features of the system under test. The following title is a seminal work in this field:
The Art of Software Testing
Glenford J. Myers
John Wiley and Sons Inc
ISBN: 0471043281
The following is another significant work, which also focuses on the area of test automation which a tool such as Test Manager is designed to facilitate:
Software Test Automation
Mark Fewster, Dorothy Graham
Addison Wesley
ISBN: 0201331403
There are many, many other significant contributions to the field of software testing. The reader is advised to consult the literature in order to understand how to design good Testcases for his or her system.
There are three elements that define a Testcase within Test Manager:
Input Data
Testcase-specific input data is supplied to the Test Manager application
in the form of files and directories stored on disk. This will be a sub-directory
of the TestSuite that the Testcase is contained within. The Testcase editor
displays the input location, and provides a View button that opens a file
manager application on the directory on disk.
The user should place any test input files and directories into this directory. When the test is executed, these input files will be copied to a "sandbox" area on disk while the execution takes place. This ensures that the original input files are not destroyed or modified when the test is executed.
Execution Procedures
The Execution Procedures for a Testcase define what exactly happens
when the Testcase is executed. Each Testcase may have one or more execution
procedures, which are executed in sequence.
Execution procedures are Java components that implement a particular Java interface. The user can select from a list of pre-built components, in order to define the execution procedures for a given Testcase. The Test Manager application is supplied with a set of components that can be used to perform standard tasks such as running a program, or executing an SQL query on a database. In addition, it is possible to create custom components that perform tasks that are more specific to the particular system under test.
The Testcase editor displays a table of all execution procedures that have been instantiated in this Testcase. When the Add button is pressed, a list of all execution procedure components that the application is aware of, will be displayed for the user to select from. Each Testcase may instantiate one or more of these components, and change the sequence of execution by using the Up and Down buttons. It is possible to use multiple instances of the same component within a single Testcase, so that, for example, a number of SQL queries may be executed. It is this feature that allows complex testcase procedures to be constructed from a set of simple building-blocks.
Each execution procedure component is configurable. The components are configured with a set of properties (name/value pairs). The Testcase editor allows these properties to be created and edited. The properties can be attached to the procedure by selecting the procedure in the table, then adding the properties to the table on the right-hand side. Alternatively, the properties may be attached to the Testcase by entering them in the table above. Properties that are attached to the Testcase will be applied to all the procedures in the Testcase, so this may be used to ensure that properties are given the same value across multiple execution procedures. It is also possible to set property values common to a set of Testcases, or an entire TestSuite.
The list of properties that a component supports is shown in the execution procedures table. (The tables support tool-tips, so if the value is too large to display, the entire value can be seen by moving the mouse over that area of the screen). This table only shows the property names. In order to find out the set of valid values for a particular property, it is necessary to consult the documentation of that execution procedure.
Built-In Execution Procedures
The following procedures are supplied with the application. The properties
that each procedure uses is listed below the procedure description.
RunExecutable This procedure will create a new process, and run a batch program until completion. The output and error streams from the program are stored in the execution log file.
ExeName - the pathname of the executable
ExeArguments - arguments to the executable (optional)RunInteractiveProgram This procedure will create a new process, and run a command-line program until completion. User input required by the program can be supplied in an input file. The output and error streams from the program are stored in the execution log file.
ExeName - the pathname of the executable
ExeArguments - arguments to the executable (optional)
InputStream - the name of a file containing user input to the programRunSQLQuery This procedure will execute an SQL query on a database using JDBC. It will then display the results of the query in an ASCII file.
URL - the JDBC connection string for the database
Username - database username (optional)
Password - database password (optional)
QueryString - the SQL query to execute
ResultFile - the name of the output ASCII file
JDBCDriver - the classname of the JDBC driver to load (optional)RunSQLUpdate his procedure will execute an SQL update statement (in other words, a statement that does not return any results) on a database database using JDBC.
URL - the JDBC connection string for the database
Username - database username (optional)
Password - database password (optional)
QueryString - the SQL statement to execute
JDBCDriver - the classname of the JDBC driver to load (optional)RunHTTPQuery This procedure will execute a request on a HTTP server, and store the response stream in a file on disk.
QueryString - the HTTP query to run (without the http:// part)
ResultFile - The file for storing the returned stream from the HTTP server
InvokeBrowser - a boolean flag to indicate that the test should also display the results in a browser, as well as storing in a file (optional)
Username - The username for pages that require a login (optional)
Password - The password for pages that require a login (optional)
FollowRedirects - a boolean flag to indicate whether or not the procedure should attempt to follow up a redirect code from the server (HTTP status codes 3xx). The default value for this property is false.
Verification Procedures
When a Testcase is executed, each execution procedure will be run
in sequence. At this point, the application will attempt to verify whether
or not
the Testcase has passed or failed. When a Testcase is constructed, the
user must define the pass/fail criteria for a Testcase. This criteria
is determined by the verification procedures. These procedures define
the actions that the application will take in order to determine that
the Testcase has passed or failed.
Verification procedures are very similar to execution procedures. They are Java components that implement a particular Java interface. The Test Manager application is supplied with a number of pre-built components for performing common verification tasks such as file comparison and ensuring that the execution time is within some limit. Again, it is possible to create custom components that perform verification tasks that are more specific to the particular system under test.
The instantiation, sequencing, and configuration of verification procedures is identical to that of execution procedures.
It is not absolutely necessary to instantiate any verification procedures. If a Testcase is defined with execution procedures only, the Testcase will execute but willl not be verified. This is a valid approach for automatically executing, but manually verifying a Testcase. It may be necessary to do this if the Testcase verification is difficult to automate.
Built-In Verification Procedures
The following procedures are supplied with the application. The properties
that each procedure uses is listed below the procedure description.
VerifyFileEquality Compare a file against the benchmark version of the file for this Testcase. Report any differences found. This will work
recursively on directories too.
Filename - the name of the file or directory to compareVerifyExecutionLog Compare the log file for the Testcase against the benchmark version. Report any differences found.
VerifyExecutionTime Test whether the execution time of the Testcase (the total time taken to complete all of the execution procedures) is within a specified maximum.
MaxTime - the maximum execution time in seconds