Run Test

Overview

The run_test binary program allows users to execute one or more JSON-defined test scripts against a running N2SVCD TesterApp application. This program uses the ExecuteTest REST API method to begin executing each test run, and then uses the QueryTest REST API method to periodically check each test run until it completes.

This program may be used to either:

  1. A single test run, which executes a single instance.
  2. A single test run, which executes multiple instances (specify --total greater than one).
  3. Multiple test runs, each of which run in sequence (specify more than one .json file).
  4. Multiple test runs, running in parallel (specify more than one .json file, and specify --parallel greater than one).

The default installation location for this binary is /usr/share/n2svcd/bin/run_test.

Running run_test --help will show the program usage.

usage: run_test <json-file> [... <json-file>]
        --host <json-rpc-host>          Default = 'localhost'.
        --port <json-rpc-port>          Default = 9009.
        --quiet                         (suppress progress noise)
        --space                         (output a newline between tests)
        --echo                          (echo JSON RPC request content)
        --failure-only                  (when running multiple test scripts only print failed tests in summary)
        --parallel <number_of_tests>    The number of tests to run in parallel. Up to this number will be actioned at once.

        --dpc <point-code>              Default = 4114.
        --no-dpc                        (don't send a DPC)
        --dssn <subsystem-number>       Default = 106.
        --no-dssn                       (don't send DSSN)
        --dgt_digits <digits>           Default = ''.
        --dgt_noa <nature-of-address>   Default = ''.
        --dgt_np <numbering-plan>       Default = ''.
        --dgt_tt <translation-type>     Default = ''.
        --ssp_ac <ati3|camel1|camel2|camel3|camel4|cs1|etsi|ussd2|a.b...>
                                        Default = 'camel3'.

        --smpp <smpp-app-name>          Default = 'default'.
        --pi <pi-app-name>              Default = 'default'.
        --osd <osd-app-name>            Default = 'default'.
        --diameter <diameter-app-name>  Default = 'default'.
        --sigtran <sigtran-app-name>    Default = 'default'.
        --rest <rest-app-name>          Default = 'default'.

        --google-test-output <output_file_location> Output a Google test formatted XML file that can be used by invoking applications to display run statistics.

        --cps <cps>
        --backlog <backlog>
        --ramp <ramp-seconds>           (requires --cps and/or --backlog)
        --total <total>                 (requires --cps and/or --backlog)

        --trace_level <0|1|2|3>         (request n2svcd to return debug traces)
        --trace <0|1|2|3>               (request n2svcd to return debug traces)
        --debug                         (--trace_level = 1)
        --dump                          (--trace_level = 2)
        --spam                          (--trace_level = 3)

        --load-details                  (track test details under load)

        --globals                       (Print $g at the end of the run)

Usage

run_test can be used to run a single JSON test definition, in which case a single test summary will be output.

./run_test ../test/regression/set2-INAP/tester_app/gb001_a_party_hangup.json

[START ../test/regression/set2-INAP/tester_app/gb001_a_party_hangup.json]
[STARTED...]
[COMPLETED 1(1) of 1 instances in 0.208 seconds.  Aborted = 0, Failed = 0, Cancelled = 0, Rate = 4.815/sec]
>>> FIRST CALL CHECKS
... [S=01/O=01] [PASS] Received operation inap.ssp_from_scp.FurnishChargingInformation as expected.
... [S=01/O=01] [PASS] String 'FCI.[0].tag_hex' is present and has expected value '001c'.
... [S=01/O=01] [PASS] String 'FCI.[0].value_hex' is present and has expected value '0100ffffffffffffffffffffffffffffffffffff'.
... [S=01/O=01] [PASS] String 'FCI.[1].tag_hex' is present and has expected value '002c'.
... [S=01/O=01] [PASS] String 'FCI.[1].value_hex' is present and has expected value '0200000000000000001000000000000000000000'.
... [S=01/O=01] [PASS] String 'FCI.[2].tag_hex' is present and has expected value '004c'.
... [S=01/O=01] [PASS] String 'FCI.[2].value_hex' is present and has expected value '0000000000000017228700000049300000005042'.
... [S=01/O=01] [PASS] String 'FCI.[3].tag_hex' is present and has expected value '999c'.
... [S=01/O=01] [PASS] String 'FCI.[3].value_hex' is present and has expected value '0000000000000000000000000000000000000000'.
... [S=02/O=02] [PASS] Received operation inap.ssp_from_scp.ApplyCharging as expected.
... [S=02/O=02] [PASS] Integer 'R1.maxConversationTime' is present and has expected value 120.
... [S=02/O=02] [PASS] Integer 'R1.treatmentForFirstWarning.warningTimeBeforeExpiry' is present and has expected value 110.
... [S=02/O=02] [PASS] Integer 'R1.treatmentForFirstWarning.informationToSend.inbandInfo.messageID.elementaryMessageID' is present and has expected value 500.
... [S=02/O=02] [PASS] Integer 'R1.treatmentForFirstWarning.informationToSend.inbandInfo.numberOfRepetitions' is present and has expected value 1.
... [S=02/O=02] [PASS] Integer 'R1.treatmentForFirstWarning.informationToSend.inbandInfo.duration' is present and has expected value 0.
... [S=02/O=02] [PASS] Integer 'R1.treatmentForFirstWarning.informationToSend.inbandInfo.interval' is present and has expected value 0.
... [S=03/O=03] [PASS] Received operation inap.ssp_from_scp.RequestReportBCSMEvent as expected.
... [S=03/O=03] [PASS] Integer 'BCSM(5,2)monitorMode' is present and has expected value 0.
... [S=03/O=03] [PASS] Integer 'BCSM(6,2)monitorMode' is present and has expected value 0.
... [S=03/O=03] [PASS] Integer 'BCSM(6,2)dPSpecificCriteria.applicationTimer' is present and has expected value 300.
... [S=03/O=03] [PASS] Integer 'BCSM(10,1)monitorMode' is present and has expected value 1.
... [S=03/O=03] [PASS] Integer 'BCSM(4)monitorMode' is present and has expected value 0.
... [S=03/O=03] [PASS] Integer 'BCSM(9,2)monitorMode' is present and has expected value 0.
... [S=03/O=03] [PASS] Integer 'BCSM(9,1)monitorMode' is present and has expected value 1.
... [S=04/O=04] [PASS] Received operation inap.ssp_from_scp.Connect as expected.
... [S=04/O=04] [PASS] String 'destinationRoutingAddress_digits' is present and has expected value '049340221'.
... [S=04/O=04] [PASS] Integer 'destinationRoutingAddress_noa' is present and has expected value 3.
... [S=05/O=05] [PASS] This is a static pass string.
... [S=06/O=06] [PASS] Correctly received DRA digits 049340221.
>>> First Instance all tests returned PASS.

run_test may also be run using multiple test definitions. This will output a summary of each individual test (see above), as well as a final summary of all the tests when complete.

./run_test ../test/regression/set2-INAP/tester_app/*.json
...

SUMMARY OF 16 RUNS.
  IDX 1st-PASS/FAIL : #START #FINISH #ABORT #FAILED #CANCEL : JSON Filename
    0       29    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/gb001_a_party_hangup.json
    1       27    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/gb002_b_party_hangup.json
    2       32    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/gb003_timer_expires.json
    3       39    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/gb004_post_call_interaction.json
    4       27    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/gb100_simple_charged.json
    5        5    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap001_idp_ext_rc_logic.json
    6       15    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap002_ctr_pa_pacui.json
    7       11    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap003_sinap_fci.json
    8       22    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap004_etc_pa_pacui_double.json
    9       11    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap004_etc_pa_pacui.json
   10       11    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap005_etc_pa_pacui_error.json
   11       10    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap006_etc_pa_pacui_abort.json
   12       13    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap007_connect_bcsm_cir.json
   13        3    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap100_simple_connect.json
   14        3    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap101_vars_connect.json
   15       11    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap200_intester_camel.json

Common Options

The following common options may be specified when executing run_test.

Option Type Description
--host String Specify a non standard JSON-RPC server listen host for TesterApp.
(Default: localhost)
--port String Specify a non standard JSON-RPC server listen port for TesterApp.
(Default: 9009)
--quiet Switch Specifying --quiet will suppress the more verbose output logging of each assertion statement from the generated output.
--echo Switch Outputs the JSON result message from the JSON REST RPC after running each test.
--trace_level Number Sets the global trace level that N2SVCD will pass to all executing applications during the test.
--trace Number Increments the global trace level to match that of the provided--trace level. This is effectively an alias for --trace_level.
--debug Switch Sets the global trace level to DEBUG. This is the same as setting trace_level = 1.
--dump Switch Sets the global trace level to DUMP. This is the same as setting trace_level = 2.
--spam Switch Sets the global trace level to SPAM. This is the same as setting trace_level = 3.
--globals Switch Outputs any stored global variables that were generated during the test run.
--space Switch Inserts whitespace between test results when executing multiple tests.
--failure-only Switch When executing multiple tests only failed tests will be reported when generating the final execution summary.
--parallel Number When executing multiple tests parallel indicates the number of tests to have running at one time. Default = 1
--google-test-output String If specified a Google test output formatted XML file will be generated to the provided path. Google test XML may be used by downstream services such as Teamcity in order to generate reporting for test executions.

Load Testing Options

The following load testing options may be specified when executing run_test. Using these options will automatically execute run_test in performance testing mode, which will disable some standard output.

Option Type Description
--cps Number Executes a concurrent number of calls equal to the --cps value. If no --total is provided the total number of calls will be ten times the value of --cps.
--backlog Number Instructs run_test to process tests as fast as possible, having a minimum of the --backlog specified value in progress at one time. If no --total flag is provided, the total number of calls will be ten times the value of --backlog.
--ramp Number Increments the concurrent number of calls by the specified value of --ramp over the duration of the performance test. This is intended to be used with --total and a combination of either --cps or --backlog, otherwise only one test will be performed.
--total Number Determines the number of total test iterations that run_test will perform. If specified, at least one of --cps or --backlog must also be specified.
--load-details Switch Allows the matching of inner JSON test operation content under load test conditions.

SCCP Protocol Options

The following SCCP protocol options may be specified when executing run_test.

Option Type Description
--dpc Number Specifies the Destination Point Code when sending outbound SCCP messages.
(Default: 4114)
--no-dpc Switch Disables the inclusion of the Destination Point Code when sending outbound SCCP messages. If specified, the value of --dpc will be ignored.
--dssn Number Specifies the Destination Sub-System Number when sending outbound SCCP messages.
(Default: 106)
--no-dssn Switch Disables the inclusion of the Destination Sub-System Number when sending outbound SCCP messages. If specified, the value of --dssn will be ignored.
--dgt_digits String Specifies the Destination Global Title Digits when sending outbound SCCP messages. If not specified, this message parameter will not be sent.
--dgt_noa Number Specifies the Destination Global Title Nature of Address Indicator when sending outbound SCCP messages. If not specified, this message parameter will not be sent.
--dgt_np Number Specifies the Destination Global Title Numbering Plan when sending outbound SCCP messages. If not specified, this message parameter will not be sent.
--dgt_tt Number Specifies the Destination Global Title Translation Type when sending outbound SCCP messages. If not specified, this message parameter will not be sent.
--ssp_ac String Specifies the Application Context to use when sending INAP encoded messages.
(Default: camel3)

Application Options

The following application options may be specified when executing run_test.

Option Type Description
--smpp String Specifies the SMPP application to utilize when executing a test that has SMPP operations. The value must be either a real or virtual named application defined in the N2SVCD configuration file.
(Default: default)
--pi String Specifies the PI application to utilize when executing a test that has PI operations. The value must be either a real or virtual named application defined in the N2SVCD configuration file.
(Default: default)
--osd String Specifies the OSD application to utilize when executing a test that has OSD operations. The value must be either a real or virtual named application defined in the N2SVCD configuration file.
(Default: default)
--diameter String Specifies the Diameter application to utilize when executing a test that has Diameter operations. The value must be either a real or virtual named application defined in the N2SVCD configuration file.
(Default: default)
--sigtran String Specifies the SIGTRAN application to utilize when executing a test that has SIGTRAN operations. The value must be either a real or virtual named application defined in the N2SVCD configuration file.
(Default: default)
--rest String Specifies the REST application to utilize when executing a test that has REST operations. The value must be either a real or virtual named application defined in the N2SVCD configuration file.
(Default: default)