Block-based Testing Guidelines

Guidelines for testing block-based components in this SDR component library.

Introduction

Components that process samples in groups should use the block-based generators in ocpi_testing, which produce sample messages with consideration for the processing length. Examples of block-based components include:

  • Framer components, where samples are processed in an internal buffer and are only output once the buffer is full.

  • Decimating components, where the sampling rate is reduced.

  • Any other component with an internal buffer or run-length processing where the grouping of samples is significant.

Unit Test XML Description File

Test cases are ordered as follows in the unit test description XML file:

  • typical

  • property

  • sample

  • input_stressing

  • message_size

  • time

  • sample_interval

  • flush

  • discontinuity

  • metadata

  • time_calculation

  • opcode_interaction

  • custom (only if custom input data is required)

  • soak

Typical

This is to test a component under normal operating circumstances, with property values set to that which is considered “typical”. The generated input data contains only sample messages.

<tests usehdlfileio="true">
  <property test="true" name="subcase" type="string"/>
  <case>
    <input port="input" script="generate.py --case typical" messagesinfile="true" messagesize="16384"/>
    <output port="output" script="verify.py" messagesinfile="true"/>
    <property name="A" value="1"/>
    <property name="my_component_s.hdl.B" value="false"/>
  </case>

Property

A separate test case exists for each of the properties of a component, in this example there are two properties. The properties are individually set to different values around zero, minimum and maximum to thoroughly exercise them.

In this worked example we will take A to be a short, where the minimum is -32768 and the maximum is 32767. B is a boolean and therefore setting it to the inverse of its value in the typical test case is sufficient.

Note that for all other test cases properties should generally use the same values as the typical test case, apart from the property test case where they are being exercised, or the final soak test where they are set to random values.

The generated input data contains only sample messages.

<case>
  <input port="input" script="generate.py --case property" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" values="0,1,-1,2,-2,100,-100,1024,-1024,32766,32767,-32767,-32768"/>
  <property name="my_component_s.hdl.B" value="false"/>
</case>
<case>
  <input port="input" script="generate.py --case property" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="true"/>
</case>

Sample

This test case generates only sample messages whose samples are around an order of magnitude defined by subcase. Since the component input port is of type short (“_s” suffix implies both input and output ports are short), the subcases result in the following input data being generated:

  • all_zero is only 0s

  • all_maximum is only 32767s

  • all_minimum is only -32768s

  • large_positive is values near 32767

  • large_negative is values near -32768

  • near_zero is values near 0

  • random is random values that are a subset of the supported range of short

  • sinusoidal is the superposition of several sinusoidal waves

<case>
  <input port="input" script="generate.py --case sample" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="all_zero,all_maximum,all_minimum,large_positive,large_negative,near_zero,random,sinusoidal">
</case>

Depending on the input port type, options for different subcase values exist.

Input Type

Subcase Values

Boolean

all_zero,all_maximum

Character, Short, Long, Long-long
all_zero,all_maximum,all_minimum,large_positive,
large_negative,near_zero,random,sinusoidal
Unsigned Character, Unsigned Short,
Unsigned Long, Unsigned Long-long
all_zero,all_maximum,large_positive,near_zero,
random,sinusoidal
Complex Character, Complex Short,
Complex Long, Complex Long-long
all_zero,all_maximum,all_minimum,real_zero,imaginary_zero,
large_positive,large_negative,near_zero,random,sinusoidal

Float, Double
all_zero,all_maximum,all_minimum,large_positive,
large_negative,near_zero,positive_infinity,
negative_infinity,not_a_number,random,sinusoidal

Complex Float, Complex Double
all_zero,all_maximum,all_minimum,real_zero,imaginary_zero,
large_positive,large_negative,near_zero,positive_infinity,
negative_infinity,not_a_number,random,sinusoidal

Input Stressing

Setting stressormode to full varies the range of input behaviour to provide coverage of what HDL workers might see on their input ports.

The generated input data contains only sample messages.

<case>
  <input port="input" script="generate.py --case input_stressing" messagesinfile="true" messagesize="16384" stressormode="full"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
</case>

Message Size

This tests a component’s ability to process shortest, longest and mixed message sizes. For the short timed sample protocol, the “shortest” message size contains 1 sample, which is 2 bytes long, and the “longest” message size contains 8192 samples, which is 16384 bytes / 2 bytes. The “different_sizes” subcase value results in a mixture of different message sizes being generated.

The generated input data contains only sample messages.

<case>
  <input port="input" script="generate.py --case message_size" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="shortest,longest,different_sizes">
</case>

Time

The generated input data for this test is a sample message, followed by a time message, and a final sample message. The subcase value determines the value of the time message.

Other subcases are:

  • single : Generates a time message first, and again between blocks.

  • intra_single : Generates a time message in the middle of a block.

  • consecutive : Generates two consecutive time messages between blocks.

  • intra_consecutive : Generates two consecutive time messages in the middle of a block.

  • intra_multiple : Generates time messages with various buffer fill states.

<case>
  <input port="input" script="generate.py --case time" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="zero,positive,maximum,single,intra_single,consecutive,intra_consecutive,intra_multiple">
</case>

Sample Interval

The generated input data for this test is a sample message, followed by a sample_interval message, and a final sample message. The subcase value determines the value of the sample_interval message.

Other subcases are:

  • single : Generates a sample_interval message first, and again between blocks.

  • intra_single : Generates a sample_interval message in the middle of a block.

  • consecutive : Generates two consecutive sample_interval messages between blocks.

  • intra_consecutive : Generates two consecutive sample_interval messages in the middle of a block.

  • intra_multiple : Generates sample_interval messages with various buffer fill states.

<case>
  <input port="input" script="generate.py --case sample_interval" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="zero,positive,maximum,single,intra_single,consecutive,intra_consecutive,intra_multiple">
</case>

Flush

The following subcases are supported:

  • single : Generates a flush message first, and again between blocks.

  • intra_single : Generates a flush message in the middle of a block.

  • consecutive : Generates two consecutive flush messages between blocks.

  • intra_consecutive : Generates two consecutive flush messages in the middle of a block.

  • intra_multiple : Generates flush messages with various buffer fill states.

<case>
  <input port="input" script="generate.py --case flush" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="single,intra_single,consecutive,intra_consecutive,intra_multiple">
</case>

Discontinuity

The following subcases are supported:

  • single : Generates a discontinuity message first, and again between blocks.

  • intra_single : Generates a discontinuity message in the middle of a block.

  • consecutive : Generates two consecutive discontinuity messages between blocks.

  • intra_consecutive : Generates two consecutive discontinuity messages in the middle of a block.

  • intra_multiple : Generates discontinuity messages with various buffer fill states.

<case>
  <input port="input" script="generate.py --case discontinuity" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="single,intra_single,consecutive,intra_consecutive,intra_multiple">
</case>

Metadata

The generated input data for this test is a sample message, followed by a metadata message, and a final sample message. The subcase value determines the value of the metadata message.

Other subcases are:

  • none : Generates only sample messages.

  • single : Generates a metadata message first, and again between blocks.

  • intra_single : Generates a metadata message in the middle of a block.

  • consecutive : Generates two consecutive metadata messages between blocks.

  • intra_consecutive : Generates two consecutive metadata messages in the middle of a block.

  • intra_multiple : Generates metadata messages with various buffer fill states.

<case>
  <input port="input" script="generate.py --case metadata" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="zero,positive,maximum,none,single,intra_single,consecutive,intra_consecutive,intra_multiple">
</case>

Time Calculation

The generated input data contains sample_interval, time, and sample messages to test the ability of a component to calculate and output correct time and sample interval values. All subcases include a non-zero sample_interval as the first message and have a flush message at the end.

The following subcases are supported:

  • interval_change: Generates a change in sample interval during a partially filled buffer.

  • time_change: Generates a change in time during a partially filled buffer.

  • minimum_increment: Generates the smallest valid sample_interval to test minimum time increments.

  • single : Generates a time message, and again between blocks.

  • intra_single : Generates a time message in the middle of a block.

  • consecutive : Generates two consecutive time messages between blocks.

  • intra_consecutive : Generates two consecutive time messages in the middle of a block.

  • intra_multiple : Generates time messages with various buffer fill states.

<case>
  <input port="input" script="generate.py --case time_calculation" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="interval_change,time_change,minimum_increment,single,intra_single,consecutive,intra_consecutive,intra_multiple">
</case>

Opcode Interaction

The generated input data contains messages to test the interaction between and re-ordering of different messages and buffer fill states. All subcases include a non-zero sample_interval as the first message.

The following subcases are supported:

  • time_flush: Generates time, flush and sample messages for interaction testing.

  • time_discontinuity: Generates time, discontinuity and sample messages for interaction testing.

  • time_metadata: Generates time, metadata and sample messages for interaction testing. Four consecutive metadata opcodes are generated, which can be used to test metadata FIFO overflow handling (by setting the FIFO size to less than four).

  • all_opcodes: Generates all permutations of messages. This is suited to components with complex message interactions where the previous tests might not pick up all edge cases.

  • single : Generates a time message, and again between blocks.

  • intra_single : Generates a time message in the middle of a block.

  • consecutive : Generates two consecutive time messages between blocks.

  • intra_consecutive : Generates two consecutive time messages in the middle of a block.

  • intra_multiple : Generates time messages with various buffer fill states.

<case>
  <input port="input" script="generate.py --case opcode_interaction" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="fifo_size" values="4,3"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="time_flush,time_discontinuity,time_metadata,all_opcodes,single,intra_single,consecutive,intra_consecutive,intra_multiple">
</case>

Custom

If a custom test case is required, the Generator() class custom method in generate.py can be overridden. The subcase property allows for multiple tests with different generated input data.

<case>
  <input port="input" script="generate.py --case custom" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="custom_test_1,custom_test_2">
</case>

Soak

The final soak test exercises random combinations of property values and generated input data. Multiple random property values are selected for this test. When the subcase value is sample_only only sample message are generated on the input, and when it is all_opcodes a mix of message types is generated.

  <case>
    <input port="input" script="generate.py --case soak" messagesinfile="true" messagesize="16384"/>
    <output port="output" script="verify.py" messagesinfile="true"/>
    <property name="A" values="22756,-12173,-23562,2742,10479,31312,6912,-1850"/>
    <property name="my_component_s.hdl.B" values="false,true"/>
    <property name="subcase" values="sample_only,all_opcodes">
  </case>
</tests>