The Regression Test Suite

Icarus Verilog development includes a regression test suite that is included along with the source. The “ivtest” directory contains the regression test suite, and this suite is used by the github actions as continuous integration to make sure the code is always going forward.

NOTE: There are scripts written in perl to run the regression tests, but they are being gradually replaced with a newer set of scripts. It is the newer method that is described here.

Test Descriptions

Regression tests are listed in the regress-vvp.list file. Each line lists the name of the test and the path to the dest description. The list file is therefore pretty simple, and all the description of the test is in the description file:

macro_str_esc  vvp_tests/macro_str_esc.json

The “name” is a simple name, and the test-description-file is the path (relative the ivtest directory) to the description file. A simple test description file is a JSON file, like this:

{
  "type"   : "normal",
  "source" : "macro_str_esc.v",
  "gold"   : "macro_str_esc"
}

This description file contains all the information that the vvp_reg.py script needs to run the regression test. The sections below describe the keys and values in the description file dictionary.

source (required)

This specifies the name of the source file. The file is actually to be found in the ivltests/ directory.

type (required)

This describes the kind of test to run. The valid values are:

  • normal - Compile the source using the iverilog compiler vvp target, and if that succeeds execute it using the vvp command. If there is no gold file specified, then look for an output line with the “PASSED” string.

  • normal-vlog95 - This is similar to the normal case, but uses the -tvlog95 target in a first pass to generate simplified verilog, then a regular iverilog command with the -tvvp target to generate the actual executable. This tests the -tvlog95 target.

  • NI - Mark the test as not implemented. The test will be skipped without running or reporting an error.

  • CE - Compile, but expect the compiler to fail. This means the compiler command process must return an error exit.

  • EF - Compile and run, but expect the run time to fail. This means the run time program must return an error exit.

gold (optional)

If this is specified, it replaces the “Passed” condition with a comparison of the output with a gold file. The argument is the name of the gold file set, which will be found in the “gold/” directory. The name here is actually the basename of the gold files, with separate actual gold files for the iverilog and vvp stderr and stdout. For example, if a “normal” test includes a gold file, then the program is compiled and run, and the outputs are compared with the gold file to make sure it ran properly.

The way the regression suite works, there are 4 log files created for each test:

  • foo-iverilog-stdout.log

  • foo-iverilog-stderr.log

  • foo-vvp-stdout.log

  • foo-vvp-stderr.log

The “gold” value is the name of the gold file set. If the gold value is “foo”, Then the actual gold files are called:

  • gold/foo-iverilog-stdout.gold

  • gold/foo-iverilog-stderr.gold

  • gold/foo-vvp-stdout.gold

  • gold/foo/vvp-stderr.gold

If any of those files is empty, then the gold file doesn’t need to be present at all. The log files and the gold files are compared byte for byte, so if the output you are getting is correct, then copy the log to the corresponding gold, and you’re done.

If the run type is “CE” or “RE”, then the gold files still work, and can be used to check that the error message is correct. If the gold file setting is present, the error return is required, and also the gold files must match.

iverilog-args (optional)

If this is specified, it is a list of strings that are passed as arguments to the iverilog command line.

vvp-args (optional)

If this is specified, it is a list of strings that are passed as arguments to the vvp command. These arguments go before the vvp input file that is to be run.

vvp-args-extended (optional)

If this is specified, it is a lost of strings that are passed as arguments to the vvp command. These are extended arguments, and are placed after the vvp input file that is being run. This is where you place things like plusargs.