Triaged Tester

January 12, 2009

Test Terms

Filed under: Black Box Testing — Triaged Tester @ 5:40 am

These are few of the testing terms that normally do rounds in office

Term AKA Definition
Agent daemon Automation System component – A process running on any machine which is a part of the automation system.  Talks to the distributor to give “ready” and “busy” status. Also usually gathers machine software and hardware configuration data to upload to a configuration database.
Automation   Executable test code that requires no human intervention during the execution in order to complete successfully
Automation System   An end to end solution for automated test case execution. Provides the infrastructure for test case automation to be executed and reported on. 
Automation Target Machine/Machine Under Test   Automation System Component – Machine where Harness is launched and test cases are executed.   A single machine.
Code Coverage   The degree to which a component or product has had automation touch some metric of code.  Metric can be function coverage, branch coverage, block coverage, or arc coverage.  Block coverage is most often used.  Product code needs to be instrumented for code coverage in order to track it.
Code Generators   Agents send machine software and hardware information to this database.  Used by the scheduler to find valid automation target machines for automation.
Configuration database   Automation System Component – Agents send machine software and hardware information to this database.  Used by the scheduler to find valid automation target machines for automation.
Code Writer Code Generators A tool that can query some structure of a target of test (an assembly, a database) and generate test code that can be executed against that target of test.
Configuration Matrix   The hardware and software configuration dimensions supported for a product release.  The dimensions are multiplied into a full matrix
Data Driven Automation   Automation that accepts variable input at run time.  Variables are stored in some persisted state (database, file) and read into the code as it is executing.
Delta diff The resultant difference between 2 things.
Distributor/scheduler/controller Scheduler, controller Automation System Component – This is the part of the automation system where test automation is queued up to be executed on machines within a machine pool (lab).  It often has access to configuration data from the automation target machines in order to send tests to the correct automation targets (e.g. If a test requires IA64). The distributor is usually in communication with agents on the Automation Target Machines.   In some cases, Distributors are overloaded with coordination instructions for multi-machine automation (better to do this in a harness, for portability).  Distributors are basically fancy ShellExec engines.
Genetic Algorithms   Data driven automation that “evolves” towards the favored result.  User “selects” the favored result prior to execution and the genetic algorithm can mutate and become more and more efficient at finding the least number of steps required to get from start to favored result.
Harness   Automation System Component – The end point executable run on an Automation Target Machine which knows how to interpret structure in a binary or some other blob which contains test cases and  execution instructions.  Easy to use harnesses have both a GUI and a command line mode (for use with a distributor).
Instrumented Build   A build of the product which has been modified in some way to trace code behaviors, such as code coverage, or memory tracing.
ITE   Integrated Test Environment.  A tool used to create and execute Model Based Tests.
Log File   Verbose data about the Run, parameters used, system configuration used, trace data, and the result of test cases.   Log files can be are parsed and can have result data uploaded to a result database.   Log files are often saved to a file share and linked to results if needed for later analysis.
Logger Logging API API used by test automation authors to write comments and pass/fail results to the log file.
Machine Paver   Automation System Component – A component that can provision machines depending on test requirements.  A paver can blow away and restore machines in the Lab to allow for the install of custom, or standard test baseline images. May be reactive (test cases submitted to the system and request a configuration that is not available so the paver blows away machines and restores or creates the correct image for the TC), or proactive (paver sets up machines to base configurations prior to test case execution, test cases are designed to run on those configs available in the lab only)
Machine Pool   Automation System Component – A logical grouping of random machines, usually in a Lab.  Might be secured.
MAGEN   A tool that allows users to create dimensions and expand them into matrices.
Model Based Testing   A testing method that usually uses a finite state machine from which test cases and paths for testing are derived.
OASYS   Office Automation System.  One of two end to end automation systems at Microsoft.
Offline Run   Automation System Component – a persisted representation of a Run, can be used for offline automation.
Offline automation portable automation Automation that can run outside of DMZ.  The only part of the system used is the Harness. Offline Run is passed to the harness and tests are executed.  Assumption is that the automation target machine(s) have been created already and exist in the correct state for the test cases to be executed.
Parameter Matrix   A set of dimensions and values for a method signature which takes more than 1 parameter.
PICT PairWise Independent Cominatorics A tool that can be used to create equivalency classes based on pair wise combination of a set of dimensions.  Results in far fewer outputs than a full matrix expansion.  Guarantees that all values from any 2 dimensions are combined at least once in the result set.
Report   Visualization of automation results.
Result   The actual Pass/Fail/Other from a test case.   Also a logical object in the result database.   Linked to test case and contains metadata describing the Automation Target machine on which the test was executed.   Data for the result is retrieved from the log file.
Run Job A logical object that represents the execution instructions, parameters, binaries, command line, and any other information required by the harness to execute a test case or set of test cases.
Scenario End user case An end to end test case that covers any user scenario.
TCM Test Case Manager A database where test cases can be created, modified, stored, and deleted.  Stores metadata about automated test cases, stores descriptions and test steps for manual test cases.
Test Case (automated)   • A test case *must* be atomic – 100% self contained, can be executed independently, or in any order with any other set of test cases.
• Test cases *must* be implemented so that they can be run in any order (API and UI)
• API (not UI) Test cases *must* be implemented so that they can be run in parallel (or multi-threaded).
• Test cases *must* be globalized, that is to say that any test can run on any international platform/OS.
• Test cases *must not* have dependencies on each other, execution sequence *must* be irrelevant
• Test cases *should* clean up any data that it wrote, or restore any data it deleted or modified during execution.  Test Cases *should not* have side/after-effects that could affect other tests.
• Test cases *must* verify a valid starting state exists on the machine under test.
• Test cases *must* be able configure the machine under test with data required for the test.
• Test cases *must* be able to execute successfully on a valid automation target machine that is not connected to a network.
• Test cases *must* specify an execution timeout which if exceeded results in a failure.
• A test case *should* map to a method, or function in an Assembly DLL. (assuming we will use a Harness like NUnit, or Junit)
– For example, a single assembly could contain multiple test cases, but each of those cases should be independently executable. 
– One programmatic entry point per test case is a good rule of thumb.
• A test case *must* specify any special execution constraints that are beyond the baseline, and be able to verify those execution constraints prior to execution.
– For example, a test case might be specifically built to run *only* on a specific SKU of Green.  The test case needs to verify that it is being executed on that SKU prior to test step execution to avoid a false failure due to invalid execution environment.
• A test case *must* log one and only one Pass/Fail result per execution
– Internal test steps can log pass fail, but the Test case pass/fail is the result that we record for reporting purposes. 
– Test step results are logically “anded” together to create the test case result.
• If you have a “test case” that requires another “test case” to execute before it, you actually have 2 test steps within 1 test case.  Log one result.  This is critical for results to mean something for reporting later on. Test Case: TCM level definition (metadata):
• A test case is represented in a Test Case manager, with the following required data:
– A test case *must* have a description of each execution step in the test. 
– A test case *must* have a description of required environment/hardware that is beyond the baseline configuration which is needed for successful execution.
– A test case *must* have a reference to the automation scripts/binaries as well as source code.
– A test case *must* have a description and a reference to any data required to set up the test.  

Test Suite   a collection of test cases.
Test Variation   Same test case, using different variables for each execution.

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at

%d bloggers like this: