Component Testing Guidelines

Guidelines for testing components in this SDR component library.

Introduction

Most unit tests for components in this library follow a standard format, making use of ocpi_testing and ocpi_protocols Python modules, which are delivered as part of the OpenCPI framework. The tests are specifically for components that use the timed sample protocols.

A worked example is provided below that demonstrates this, although there are some components for which the worked example is not entirely suitable, such as framers and components with multiple or no input ports. In these instances it is recommended that the component designer refers to the source code of similar components present in this library.

For components that perform FFTs, decimation or framing, Block-based Testing Guidelines are available.

Unit Test XML Description File

Test cases are ordered as follows in the unit test description XML file:

  • typical

  • property

  • sample

  • input_stressing

  • message_size

  • time

  • sample_interval

  • flush

  • discontinuity

  • metadata

  • custom (only if custom input data is required)

  • soak

The following settings should be declared globally within the test XML file:

  • Top level attribute usehdlfileio is set to true for faster simulation times.

  • Declaring a test property subcase allows for subcases within test cases.

  • The ordering in each test case is:

    1. ports (inputs first, followed by outputs),

    2. properties (same ordering as in OCS XML file),

    3. worker specific properties (e.g. my_component_s.hdl.property_name),

    4. test specific properties (if defined),

    5. subcase (if needed).

Typical

This is to test a component under normal operating circumstances, with property values set to that which is considered “typical”. The generated input data contains only sample messages.

<tests usehdlfileio="true">
  <property test="true" name="subcase" type="string"/>
  <case>
    <input port="input" script="generate.py --case typical" messagesinfile="true" messagesize="16384"/>
    <output port="output" script="verify.py" messagesinfile="true"/>
    <property name="A" value="1"/>
    <property name="my_component_s.hdl.B" value="false"/>
  </case>

Property

A separate test case exists for each of the properties of a component, in this example there are two properties. The properties are individually set to different values around zero, minimum and maximum to thoroughly exercise them.

In this worked example we will take A to be a short, where the minimum is -32768 and the maximum is 32767. B is a boolean and therefore setting it to the inverse of its value in the typical test case is sufficient.

Note that for all other test cases properties should generally use the same values as the typical test case, apart from the property test case where they are being exercised, or the final soak test where they are set to random values.

The generated input data contains only sample messages.

<case>
  <input port="input" script="generate.py --case property" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" values="0,1,-1,2,-2,100,-100,1024,-1024,32766,32767,-32767,-32768"/>
  <property name="my_component_s.hdl.B" value="false"/>
</case>
<case>
  <input port="input" script="generate.py --case property" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="true"/>
</case>

Sample

This test case generates only sample messages whose samples are around an order of magnitude defined by subcase. Since the component input port is of type short (“_s” suffix implies both input and output ports are short), the subcases result in the following input data being generated:

  • all_zero is only 0s

  • all_maximum is only 32767s

  • all_minimum is only -32768s

  • large_positive is values near 32767

  • large_negative is values near -32768

  • near_zero is values near 0

<case>
  <input port="input" script="generate.py --case sample" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="all_zero,all_maximum,all_minimum,large_positive,large_negative,near_zero">
</case>

Depending on the input port type, options for different subcase values exist. All of these options should be specified to ensure comprehensive testing.

Input Type

Subcase Values

Boolean

all_zero,all_maximum

Character, Short, Long, Long-long
all_zero,all_maximum,all_minimum,large_positive,
large_negative,near_zero
Unsigned Character, Unsigned Short,
Unsigned Long, Unsigned Long-long

all_zero,all_maximum,large_positive,near_zero
Complex Character, Complex Short,
Complex Long, Complex Long-long
all_zero,all_maximum,all_minimum,real_zero,imaginary_zero,
large_positive,large_negative,near_zero

Float, Double
all_zero,all_maximum,all_minimum,large_positive,
large_negative,near_zero,positive_infinity,
negative_infinity,not_a_number

Complex Float, Complex Double
all_zero,all_maximum,all_minimum,real_zero,imaginary_zero,
large_positive,large_negative,near_zero,positive_infinity,
negative_infinity,not_a_number

Input Stressing

Setting stressormode to full varies the range of input behaviour to provide coverage of what HDL workers might see on their input ports.

The generated input data contains only sample messages.

<case>
  <input port="input" script="generate.py --case input_stressing" messagesinfile="true" messagesize="16384" stressormode="full"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
</case>

Message Size

This tests a component’s ability to process shortest, longest and mixed message sizes. For the short timed sample protocol, the “shortest” message size contains 1 sample, which is 2 bytes long, and the “longest” message size contains 8192 samples, which is 16384 bytes / 2 bytes. The “different_sizes” subcase value results in a mixture of different message sizes being generated.

The generated input data contains only sample messages.

<case>
  <input port="input" script="generate.py --case message_size" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="shortest,longest,different_sizes">
</case>

Time

The generated input data for this test is a sample message, followed by a time message, and a final sample message. The subcase value determines the value of the time message. The “consecutive” subcase value generates two consecutive time messages between two sample messages.

<case>
  <input port="input" script="generate.py --case time" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="zero,positive,maximum,consecutive">
</case>

Sample Interval

The generated input data for this test is a sample message, followed by a sample_interval message, and a final sample message. The subcase value determines the value of the sample_interval message. The “consecutive” subcase value generates two consecutive sample_interval messages between two sample messages.

<case>
  <input port="input" script="generate.py --case sample_interval" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="zero,positive,maximum,consecutive">
</case>

Flush

The generated input data for this test is a sample message, followed by a flush message, and a final sample message. The “consecutive” subcase value generates two consecutive flush messages between two sample messages.

<case>
  <input port="input" script="generate.py --case flush" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="single,consecutive">
</case>

Discontinuity

The generated input data for this test is a sample message, followed by a discontinuity message, and a final sample message. The “consecutive” subcase value generates two consecutive discontinuity messages between two sample messages.

<case>
  <input port="input" script="generate.py --case discontinuity" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="single,consecutive">
</case>

Metadata

The generated input data for this test is a sample message, followed by a metadata message, and a final sample message. The subcase value determines the value of the metadata message. The “consecutive” subcase value generates two consecutive metadata messages between two sample messages.

<case>
  <input port="input" script="generate.py --case metadata" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="zero,positive,maximum,consecutive">
</case>

Custom

If a custom test case is required, the Generator() class custom method in generate.py can be overridden. The subcase property allows for multiple tests with different generated input data.

<case>
  <input port="input" script="generate.py --case custom" messagesinfile="true" messagesize="16384"/>
  <output port="output" script="verify.py" messagesinfile="true"/>
  <property name="A" value="1"/>
  <property name="my_component_s.hdl.B" value="false"/>
  <property name="subcase" values="custom_test_1,custom_test_2">
</case>

Soak

The final soak test exercises random combinations of property values and generated input data. Multiple random property values are selected for this test. When the subcase value is “sample_only” only sample message are generated on the input, and when it is “all_opcodes” a mix of message types is generated.

  <case>
    <input port="input" script="generate.py --case soak" messagesinfile="true" messagesize="16384"/>
    <output port="output" script="verify.py" messagesinfile="true"/>
    <property name="A" values="22756,-12173,-23562,2742,10479,31312,6912,-1850"/>
    <property name="my_component_s.hdl.B" values="false,true"/>
    <property name="subcase" values="sample_only,all_opcodes">
  </case>
</tests>

Additional Details

  • An HDL worker’s build configuration XML file should contain only configurations intended to be tested by the component unit tests. There should exist enough configurations to achieve sufficient test coverage. The minimum supported platform for unit tests is the PlutoSDR.

  • As previously mentioned in Introduction, the above worked example is not suitable for all components. It is recommended component designers refer to the source code of existing similar components and ideally write code that is consistent with the code base.

  • generate.py is called for input ports, and uses ocpi_testing and ocpi_protocols Python libraries to generate input test data and write it to files, respectively.

    • A seed ensures the randomly generated input data remains the same every time tests are run.

    • It does not always make sense to generate random input data, such as for framers. In these instances the Generator() methods can be overridden or custom methods added in generate.py.

      • Generator() methods should not be unnecessarily overridden; it is important to understand the purpose of tests and the impact on test coverage before doing so. Whether custom tests would be more appropriate is also a deciding factor, since they typically only add to test coverage.

  • verify.py is called for output ports and uses the ocpi_testing Python library for verification. It is possible to write a custom comparison method if required, although current comparison methods include:

    • Equal - Checks output samples are exactly equal

    • Bounded - Checks output samples are within a fixed bound, +/- BOUND

    • Bounded with Exception - Same as Bounded, but allows a settable fraction of samples to exceed BOUND up to within +/- EXCEPTION_BOUND

    • Relative - Checks output samples are within a tolerance that is relative to their value

    • Statistical - Checks output samples are statistically identical, using standard deviation

  • Google-Style Python docstrings are typically used for documenting Python code.

def my_function(value):
"""One line summary of the function

Any further details explaining the function, it's expected usage

Args:
   value (``type``): Description of this value, including intended type.

Returns:
   Description on the returned value. Omit this section if nothing is explicitly returned.
"""

  • There are directories that contain common code for components to reuse: VHDL in hdl/primitives/, C++ in components/<library>/common/, and Python in scripts/. These directories are meant only for reusable code, e.g. the core functionality of a FIR or a Polyphase Interpolator implementation.