Protocol Conformance Testing¶

General¶

The purpose of conformance testing is to determine to what extent a single implementation of a particular standard conforms to the individual requirements of that standard. Please find additional and more detailed information about conformance testing at ETSI’s Center for Testing & Interoperability

The ISO standard for the methodology of conformance testing (ISO/IEC 9646-1 and ISO/IEC 9646-2) as well as the ETSI rules for conformance testing (ETSI ETS 300 406) are used as a basis for the test methodology.

To implement this methodology we require several intermediary artefacts. Those single artefacts break down the whole complexity of conformance testing into smaller pieces, each with a specific perspective on the problem.

Test Suite Structure¶

In the first step we define a TSS for a specific IUT.

The TSS reflects the coverage of the reference specification by the TS: it is a synopsis of “which tests are performed on which aspects of the reference specification”. The conformance requirements and the ICS proforma of the base specification are an essential source of cross-reference to check that the coverage of the test suite specified by the TSS&TP is acceptable.

Test Configurations¶

TODO: Why do we need Test configurations?

Test Purpose Catalogues¶

A TP (Test Purpose) is a formal description of a test case. A formal description in the form of a TP offers a possibility of describing the purpose of a test without having the later technical implementation in mind. Following the TSS the tester is supported in systematically covering the complete IUT specification.

The listing below shows a simple MQTT TP specified in TDL-TO.

Test Purpose {
TP Id TP_MQTT_Broker_CONNECT_001

Test objective
"The IUT MUST close the network connection if fixed header flags in CONNECT Control Packet are invalid"
Reference
"[MQTT-2.2.2-1], [MQTT-2.2.2-2], [MQTT-3.1.4-1], [MQTT-3.2.2-6]"
PICS Selection PICS_BROKER_BASIC

Expected behaviour
ensure that {
when {
the IUT entity receives a CONNECT message containing
}
then {
the IUT entity closes the TCP_CONNECTION
}
}
}


The exemple below shows a simplified tabular representation for the TP.

 TP-ID TP_MQTT_BROKER_CONNECT_01 Selection PICS_Broker Summary The IUT MUST close the network connection if… Reference [MQTT-2.2.2-1], [MQTT-2.2.2-2] Expected bahaviour initial condition statement ensure that statement

IoT-Testware Test Suites¶

This steps focuses on a technical implementation of the TPs. We use TTCN-3 and Eclipse Titan to implement each TP into a TC and orchestrate to executable test suites.