7.1. `harness` — Test harness¶

In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. It has two main parts: the test execution engine and the test script repository.

This module implements the test execution engine.

The test scripts are part of the build system.

7.1.1. Stubs¶

Symbols can be stubbed per C-file using the STUB() macro and STUB make variable. The STUB make variable is a list of source files and the symbols to stub within given file.

For example, stub functions foo_bar() and foo_fie() in fum.c by defining stub functions STUB(foo_bar)() and STUB(foo_fie)(), and set the make variable STUB to fum.c:foo_bar,foo_fie.

Prototypes for foo_bar() and foo_fie() in foo.h:

int foo_bar();
int foo_fie();

foo_bar() and foo_fie() called in fum.c. Both function calls will call the stubbed version on the respective function.

int fum_init()
{
   foo_bar();
   foo_fie();
}

The stubbed implementations, often defined in the test suite file main.c:

int STUB(foo_bar)()
{
    return (0);
}

int STUB(foo_fie)()
{
    return (0);
}

And last, add the stubbed symbol to the test suite makefile Makefile:

STUB = fum.c:foo_bar,foo_fie

7.1.2. Example test suite¶

Below is an example of a test suite using the harness. It has three test cases; test_passed, test_failed and test_skipped.

The test macro BTASSERT(condition) should be used to validate conditions.

#include "simba.h"

static int test_passed(struct harness_t *harness_p)
{
    /* Return zero(0) when a test case passes. */
    return (0);
}

static int test_failed(struct harness_t *harness_p)
{
    /* Return a negative integer when a test case fails. BTASSERT
       will return -1 when the condition is false. */
    BTASSERT(0);

    return (0);
}

static int test_skipped(struct harness_t *harness_p)
{
    /* Return a positive integer when a test case is skipped. */
    return (1);
}

int main()
{
    /* Test harness and NULL terminated list of test cases.*/
    struct harness_t harness;
    struct harness_testcase_t harness_testcases[] = {
        { test_passed, "test_passed" },
        { test_failed, "test_failed" },
        { test_skipped, "test_skipped" },
        { NULL, NULL }
    };

    sys_start();

    harness_init(&harness);
    harness_run(&harness, harness_testcases);

    return (0);
}

The output from the test suite is:

app:    test_suite-7.0.0 built 2016-07-25 17:38 CEST by erik.
board:  Linux
mcu:    Linux

enter: test_passed
exit: test_passed: PASSED

enter: test_failed
exit: test_failed: FAILED

enter: test_skipped
exit: test_skipped: SKIPPED

            NAME        STATE  PRIO   CPU  LOGMASK
            main      current     0    0%     0x0f
                        ready   127    0%     0x0f
harness report: total(3), passed(1), failed(1), skipped(1)

There are plenty of test suites in the tst folder on Github.

Source code: src/debug/harness.h, src/debug/harness.c

Defines

_ASSERTFMT(fmt, ...) std_printf(FSTR(fmt "\n"), ##__VA_ARGS__);¶

_ASSERTHEX(actual_str, actual, expected_str, expected, size) std_printf(FSTR(":: \r\n" \ "Memory buffer '" actual_str "'\r\n")); \ std_hexdump(sys_get_stdout(), actual, size); \ std_printf(FSTR("is not equal to memory buffer '" expected_str "'\r\n")); \ std_hexdump(sys_get_stdout(), expected, size);¶

BTASSERTRM(cond, cond_str, res, msg) if (!(cond)) { \ std_printf(FSTR(__FILE__ ":" STRINGIFY(__LINE__) ": BTASSERT: " \ cond_str " ")); \ msg; \ return (res); \ }¶: Assert given condition. Print an error message and return given value res on error.

BTASSERTR(cond, cond_str, res, ...) BTASSERTRM(cond, cond_str, res, _ASSERTFMT(__VA_ARGS__));¶: Assert given condition. Print an error message and return given value res on error.

BTASSERTN(cond, ...) BTASSERTR(cond, #cond, NULL, __VA_ARGS__)¶: Assert given condition. Print an error message and return given value on error.

BTASSERT(cond, ...) BTASSERTR(cond, #cond, -1, __VA_ARGS__)¶: Assert given condition. Print an error message and return.

BTASSERTI(actual, operator, expected) do { \ int UNIQUE(_actual); \ int UNIQUE(_expected); \ UNIQUE(_actual) = (actual); \ UNIQUE(_expected) = (expected); \ BTASSERTR(UNIQUE(_actual) operator UNIQUE(_expected), \ #actual " " #operator " " #expected, \ -1, \ ":: Condition '%d " #operator " %d' ('0x%x " \ #operator " 0x%x') is not true.", \ UNIQUE(_actual), \ UNIQUE(_expected), \ UNIQUE(_actual), \ UNIQUE(_expected)); \ } while (0)¶: Compare two integers actual and expected with given operator operator. Print an error message if the condition is not true and return.

BTASSERTM(actual, expected, size) do { \ const void *UNIQUE(_actual); \ const void *UNIQUE(_expected); \ UNIQUE(_actual) = (actual); \ UNIQUE(_expected) = (expected); \ BTASSERTRM(memcmp(UNIQUE(_actual), UNIQUE(_expected), size) == 0, \ "memcmp(" #actual ", " #expected ", " #size ") == 0", \ -1, \ _ASSERTHEX(#actual, UNIQUE(_actual), \ #expected, UNIQUE(_expected), \ size)); \ } while (0)¶: Comapre two memory positions actual and expected. Print an error message if they are not equal and return.

BTASSERTV(cond, ...) if (!(cond)) { \ std_printf(FSTR(__FILE__ ":" STRINGIFY(__LINE__) ": BTASSERT: " #cond " ")); \ _ASSERTFMT(__VA_ARGS__); \ return; \ }¶: Assert given condition in a testcase. Print an error message and return -1 on error.

STUB(function) __stub_ ## function¶: Stub given function. Used with the make variable STUB to preprocess object file(s).

Typedefs

typedef int (*harness_testcase_cb_t)(struct harness_t *harness_p)¶

The testcase function callback.

Return

zero(0) if the testcase passed, a negative error code if the testcase failed, and a positive value if the testcase was skipped.

Parameters

harness_t: The harness object.

Functions

int harness_init(struct harness_t *self_p)¶

Initialize given test harness.

Return

zero(0) or negative error code.

Parameters

self_p: Test harness to initialize.

int harness_run(struct harness_t *self_p, struct harness_testcase_t *testcases_p)¶

Run given testcases in given test harness.

Return

zero(0) or negative error code.

Parameters

self_p: Test harness.
testcases_p: An array of testcases to run. The last element in the array must have callback and name_p set to NULL.

int harness_expect(void *chan_p, const char *pattern_p, const struct time_t *timeout_p)¶

Continiously read from the channel and return when given pattern has been read, or when a timeout occurs.

Return

Number of bytes read from the channel when match occured, or negative error code.

Parameters

chan_p: Channel to read from.
pattern_p: Pattern to wait for.
timeout_p: Timeout, or NULL to wait forever.

ssize_t harness_mock_write(const char *id_p, const void *buf_p, size_t size)¶

Write given data buffer to a mock entry with given id.

Return

Number of written words or negative error code.

Parameters

id_p: Mock id string to write.
buf_p: Data for given mock id.
size: Buffer size in words.

ssize_t harness_mock_read(const char *id_p, void *buf_p, size_t size)¶

Read data from mock entry with given id.

Return

Number of read words or negative error code.

Parameters

id_p: Mock id string to read.
buf_p: Buffer to read into.
size: Buffer size in words.

struct harness_testcase_t¶

Public Members

harness_testcase_cb_t callback¶

const char *name_p¶

struct harness_t¶

Public Members

struct uart_driver_t uart¶

7.1. harness — Test harness¶

7.1.1. Stubs¶

7.1.2. Example test suite¶

7.1. `harness` — Test harness¶