Protocol Conformance Testing ¶

Contents

Protocol Conformance Testing

General ¶

The purpose of conformance testing is to determine to what extent a single implementation of a particular standard conforms to the individual requirements of that standard. Please find additional and more detailed information about conformance testing at ETSI’s Center for Testing & Interoperability

The ISO standard for the methodology of conformance testing (ISO/IEC 9646-1 and ISO/IEC 9646-2) as well as the ETSI rules for conformance testing (ETSI ETS 300 406) are used as a basis for the test methodology.

To implement this methodology we require several intermediary artefacts. Those single artefacts break down the whole complexity of conformance testing into smaller pieces, each with a specific perspective on the problem.

Test Suite Structure ¶

In the first step we define a TSS for a specific IUT.

The TSS reflects the coverage of the reference specification by the TS: it is a synopsis of “which tests are performed on which aspects of the reference specification”. The conformance requirements and the ICS proforma of the base specification are an essential source of cross-reference to check that the coverage of the test suite specified by the TSS&TP is acceptable.

Test Configurations ¶

TODO: Why do we need Test configurations?

Test Purpose Catalogues ¶

A TP (Test Purpose) is a formal description of a test case. A formal description in the form of a TP offers a possibility of describing the purpose of a test without having the later technical implementation in mind. Following the TSS the tester is supported in systematically covering the complete IUT specification.

The listing below shows a simple MQTT TP specified in TDL-TO.

Test Purpose {
  TP Id TP_MQTT_Broker_CONNECT_001

  Test objective
  "The IUT MUST close the network connection if fixed header flags in CONNECT Control Packet are invalid"
  Reference
    "[MQTT-2.2.2-1], [MQTT-2.2.2-2], [MQTT-3.1.4-1], [MQTT-3.2.2-6]"
  PICS Selection PICS_BROKER_BASIC

  Expected behaviour
  ensure that {
    when {
        the IUT entity receives a CONNECT message containing
        header_flags indicating value '1111'B;
    }
    then {
        the IUT entity closes the TCP_CONNECTION
    }
  }
}

The exemple below shows a simplified tabular representation for the TP.

TP-ID	TP_MQTT_BROKER_CONNECT_01
Selection	PICS_Broker
Summary	The IUT MUST close the network connection if…
Reference	[MQTT-2.2.2-1], [MQTT-2.2.2-2]
Expected bahaviour
initial condition statement
ensure that statement

IoT-Testware Test Suites ¶

This steps focuses on a technical implementation of the TPs. We use TTCN-3 and Eclipse Titan to implement each TP into a TC and orchestrate to executable test suites.

Protocol Conformance Testing¶

General¶

Test Suite Structure¶

Test Configurations¶

Test Purpose Catalogues¶

IoT-Testware Test Suites¶