API - JSON QueryTest

Overview

The QueryTest method allows a JSON RPC client to monitor the status of a Test Run that has been submitted to the N2SVCD.

The JSON RPC client is expected to periodically poll for Test Run status. The N2SVCD service will maintain the Test Run status in memory until one of the following occurs.

  1. The N2SVCD process is restarted, or
  2. The CancelTest method is used to cancel the Test Run, or
  3. The QueryTest method is called on a completed Test Run.

Request Parameters

The QueryTest method has a single parameter:

Parameter Type Description
run_id String The Test Run ID of a previously started Test Run.

Example Request

Here is an example QueryTest request.

{
    "jsonrpc" : "2.0",
    "id" : 1,
    "method" : "QueryTest",
    "params" : {
        "run_id" : "tr_9538"
    }
}

Successful Response

The QueryTest result is an object with the following attributes.

Attribute Type Description
run_id String The unique run identifier assigned to this run.
completed Integer 0 (completed) or 1 (in progress).
time_now Array The "now" system time on the N2SVCD clock.
[0] Integer The "now" epoch seconds.
[1] Integer The "now" epoch microseconds.
time_start Array Test run start time on the N2SVCD clock.
[0] Integer The "time_start" epoch seconds.
[1] Integer The "time_start" epoch microseconds.
time_finish Array Test run start finish on the N2SVCD clock (if the run is completed).
[0] Integer The "time_finish" epoch seconds.
[1] Integer The "time_finish" epoch microseconds.
duration Float Test run duration in seconds (if the run is completed).
counts Object Object with test instance counts.
.started Integer Number of instances started.
.finished Integer Number of instances finished (passed, failed, and aborted).
.failed Integer Number of instances completed with a failed non-mandatory check.
.aborted Integer Number of instances aborted by a failed mandatory check.
.cancelled Integer Number of instances cancelled by user-requested run cancel.
globals Object Any global variables defined during the Test Run.
halted Array Array of halted test instances (if any). See description below.
first_fails Array Array of "first fail" reasons. See description below.
first_aborts Array Array of "first abort" reasons. See description below.
buckets Array Time-based call distribution buckets. See description below.
first_call_summary Object Information about the first Test Instance in the Run.

Note: The first_call_summary is strictly speaking a summary of the first Test Instance, which may in practice implement none, one or more Telephony Calls.

Response - “globals”

The globals attribute is an Object whose values have been set during the Test Run execution. This is usually performed via the eval operation. For example the following operation:


    ...
    {
        "type": "eval",
        "expression": "$g->{foo} = 'bar'"
    }
    ...

…would result in the following globals attribute value at the conclusion of the Test Run:


    ...
    "globals": {
        "foo": "bar"
    },
    ...

Global variables can be used in this way to gather additional information about the Test Run or Test Instances for return back to the client.

Response - “halted”

The halted attribute is an array of Objects giving information about any currently halted instances within the run. Each Object in the halted Array has the following attributes.

Attribute Type Description
instance_id Integer The id of the halted instance within the test run.
operation Object The operation which caused the instance to halt.

E.g. the following entry indicates that Test Instance with ID “41” is halted at a halt operation which is operation index 13 in the test run operations array.


    ...
    "halted": [
        {
            "instance_id": "41",
            "operation" : {
                "arguments" : {
                    "seconds" : "5"
                },
                "type" : "halt",
                "static" : 1,
                "idx" : "13"
            }
        }
    ]
    ...

The operation Object has the following attributes at least:

Attribute Type Description
idx Integer The index of the halted operation within the array of test operations.
type String The operation type name, which is typically the halt operation.
arguments Object The arguments of this operation as provided in the ExecuteTest request.
... [Any] Any other operation attributes which may have been provided in the ExecuteTest request or may be used by the TestRun for its internal test management purposes.

The instance_id from the halted Object must be specified as the instance_id parameter to the ResumeInstance RPC method when resuming a halted Test Instance. If a Test Instance is not resumed before the halt timer expires, the instance will be aborted.

Response - “first_fails/aborts”

During a load test (i.e. when more than one Test Instance is executed in a Test Run) detailed information is only available about the first Test Instance in the Run. The first_fails and first_aborts response attributes provide some limited information about fail and abort reasons in subsequent Test Instances.

The first_fails attribute is an Array which tracks the first failed check which occurs during each Test Instance. The number of Test Instances which fail with the same unique text message and operation is counted and reported.

Each Object in the first_fails and first_aborts attribute has the following attributes:

Attribute Type Description
op_idx Integer The index in the operations at which this fail/abort occured.
text String The text fail/abort message generated by this failed/aborted check.
count Integer The number of Test Instances whose first fail/abort occured as described.

E.g. the following entry indicates that 300 Test Instances all encountered their first failed check at operation[4] and with the text “‘initialCallSegment_cause’ should = 30, but = 31.”


    ...
    "first_fails" : [
        {
            "count" : 300,
            "op_idx" : 4,
            "text" : "'initialCallSegment_cause' should = 30, but = 31."
        }
    ]
    ...

Response - “buckets”

The buckets attribute is used to indicate the distribution of calls over time. Although it is always returned, the information is only statistically relevant in a sustained load test scenario (e.g. at least 100 Test Instances).

Between 1 and 20 buckets will be provided. Each bucket has the following attributes:

Attribute Type Description
num_calls_started Integer Number of Test Instances started during this time period.
num_calls_finished Integer Number of Test Instances finished during this time period.
time_start Array N2SVCD timestamp for start of interval covered by this bucket.
[0] Integer The "time_start" epoch seconds.
[1] Integer The "time_start" epoch microseconds.

The time_start attribute is not present for the first bucket, as the the “time_start” of the Test Run should be used instead. There is no “time_finish” attribute on the bucket, as the “time_start” of the next bucket should be used, or the “time_finish” of the Test Run.

The following example shows a complete bucket count. The actual number of buckets provided is determined by the N2SVCD.


    ...
    "buckets" : [
        {
            "num_calls_started" : 15,
            "num_calls_finished" : 15
        },
        {
            "time_start" : [
                1389748329,
                67575
            ],
            "num_calls_started" : 15,
            "num_calls_finished" : 15
        },

        ...17 more buckets...

        {
            "time_start" : [
                1389748356,
                87250
            ],
            "num_calls_started" : 15,
            "num_calls_finished" : 15
        }
    ]
    ...

Response - “first_call_summary”

The first_call_summary attribute gives information about the processing of the first Test Instance executed in the Test Run. It contains the following attributes:

Attribute Type Description
idx Integer The Instance ID within the Test Run. Always = 0.
completed Integer 0 (completed) or 1 (in progress).
time_start Array N2SVCD timestamp at which this Test Instance started.
[0] Integer The "time_start" epoch seconds.
[1] Integer The "time_start" epoch microseconds.
time_finish Array N2SVCD timestamp at which this Test Instance finished (if completed).
[0] Integer The "time_start" epoch seconds.
[1] Integer The "time_start" epoch microseconds.
duration Float Test Instance duration in seconds (if completed).
check_log Array List of checks performed against this Test Instance.
trace_log Array List of tracing events logged against this Test Instance.
See Application - Trace Log Structure.

Each check_log entry has the following structure:

Attribute Type Description
op_idx Integer The index in the operations at which this check occured.
step Integer The incrementing operation counter at which this check occured.
status String pass, fail, or abort.
text String A text message describing the result of the check.

Note: The step counter is initialised to 0 at the start of the Test Instance and is incremented every time a new operation is performed. If there are no branches, loops or jumps in the operation sequence, then the step will always be identical to the op_idx.

Each trace_log entry has structure dependent on the type of event being logged. Refer to Application - Trace Log Structure for the detailed description of the check_log event format.

Note: Other interfaces may implement new path identifiers and associated attributes.

Example Successful Response

Here is a possible successful QueryTest response.

{
    "jsonrpc" : "2.0",
    "id" : "2",
    "result" : {
        "run_id" : "tr_7309",
        "completed" : 1,
        "time_now" : [
            1389747502,
            87993
        ],
        "time_start" : [
            1389747502,
            85922
        ],
        "time_finish" : [
            1389747502,
            87871
        ],
        "duration" : 0.001949,
        "counts" : {
            "started" : 1,
            "finished" : 1,
            "aborted" : 0,
            "failed" : 1,
            "cancelled" : 0
        },
        "globals" : {},
        "first_fails" : [
            {
                "count" : 1,
                "op_idx" : 4,
                "text" : "'initialCallSegment_cause' should = 30, but = 31."
            }
        ],
        "first_aborts" : [],
        "buckets" : [
            {
                "num_calls_started" : 1,
                "num_calls_finished" : 1
            }
        ],
        "first_call_summary" : {
            "idx" : "0",
            "completed": 1,
            "time_start" : [
                1389747502,
                85986
            ],
            "time_finish" : [
                1389747502,
                87776
            ],
            "duration" : 0.00179,
            "check_log" : [
                {
                    "op_idx" : 4,
                    "step" : 8,
                    "status" : "pass",
                    "text" : "Got inap.ssp_from_scp.ReleaseCall as expected."
                },
                {
                    "op_idx" : 4,
                    "step" : 8,
                    "status" : "fail",
                    "text" : "'initialCallSegment_cause' should = 30, but = 31."
                },
                {
                    "op_idx" : 5,
                    "step" : 9,
                    "status" : "pass",
                    "text" : "OK Cause 31, is > 22."
                }
            ],
            "trace_log" : [
                {
                    "op_idx" : 3,
                    "step" : 7,
                    "time" : [
                        1389747502,
                        86887
                    ],
                    "path" : "tcap.ssp_to_scp",
                    "protocol" : "inap",
                    "hex" : "6281894804008b05dc6b1a2818060700118605010101a00d600ba1090607040000011503046c65a163020101020100305b80020bb88207031081009909108307831120674523018501f78a088497161405512102af0e300c02020190a1060404424f535397029181980d300ba0098101028201018801009a022001bb0580038090a39c01039f36044d3b8fa0"
                },
                {
                    "op_idx" : 4,
                    "step" : 8,
                    "time" : [
                        1389747502,
                        87524
                    ],
                    "path" : "tcap.ssp_from_scp",
                    "protocol" : "inap",
                    "hex" : "64144904008b05dc6c0ca10a0201010201160402809f"
                }
            ]
        }
    }
}