7.1. `harness` — Test harness¶

In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. It has two main parts: the test execution engine and the test script repository.

This module implements the test execution engine.

The test scripts are part of the build system.

7.1.1. Stubs¶

Symbols can be stubbed per C-file using the STUB() macro and STUB make variable. The STUB make variable is a list of source files and the symbols to stub within given file.

For example, stub functions foo_bar() and foo_fie() in fum.c by defining stub functions STUB(foo_bar)() and STUB(foo_fie)(), and set the make variable STUB to fum.c:foo_bar,foo_fie.

Prototypes for foo_bar() and foo_fie() in foo.h:

/**
 * The bar function.
 */
int foo_bar();

/**
 * The fie function.
 */
int foo_fie();

foo_bar() and foo_fie() called in fum.c. Both function calls will call the stubbed version on the respective function.

int fum_init()
{
   foo_bar();
   foo_fie();

   return (0);
}

The stubbed implementations, often defined in the test suite file main.c:

int STUB(foo_bar)()
{
    return (0);
}

int STUB(foo_fie)()
{
    return (0);
}

And last, add the stubbed symbol to the test suite makefile Makefile:

STUB = fum.c:foo_bar,foo_fie

7.1.2. Mocking¶

The test harness also supports function mocking with mock_write_*() and STUB(*) functions. These functions are generated by the script stub.py.

For example, generate a mock stub of foo.h with the command stub.py generate foo.h .. Two files are created; foo_mock.h and foo_mock.c. Include foo_mock.h in the test suite and call mock_write_*() to prepare the stub for function calls. The stub will verify that all input arguments matches their expected values, and write data to output arguments.

Note

The user may have to manually modify parts of the generated stubs to match her use case, as the stub script does not handle all situations properly.

#include "simba.h"
#include "foo_mock.h"

static int test_init(void)
{
    /* Make foo_bar() return 1 and foo_fie() 5 once called. */
    mock_write_foo_bar(1);
    mock_write_foo_fie(5);

    BTASSERT(fum_init() == 0);

    return (0);
}

int main()
{
    struct harness_testcase_t testcases[] = {
        { test_init, "test_init" },
        { NULL, NULL }
    };

    sys_start();

    harness_run(testcases);

    return (0);
}

Add the stub source file to the list of files to build.

STUB = fum.c:foo_bar,foo_fie
SRC += foo_mock.c

7.1.3. Example test suite¶

Below is an example of a test suite using the harness. It has three test cases; test_passed, test_failed and test_skipped.

The test macro BTASSERT(condition) should be used to validate conditions.

#include "simba.h"

static int test_passed(void)
{
    /* Return zero(0) when a test case passes. */
    return (0);
}

static int test_failed(void)
{
    /* Return a negative integer when a test case fails. BTASSERT
       will return -1 when the condition is false. */
    BTASSERT(0);

    return (0);
}

static int test_skipped(void)
{
    /* Return a positive integer when a test case is skipped. */
    return (1);
}

int main()
{
    /* Test harness and NULL terminated list of test cases.*/
    struct harness_testcase_t testcases[] = {
        { test_passed, "test_passed" },
        { test_failed, "test_failed" },
        { test_skipped, "test_skipped" },
        { NULL, NULL }
    };

    sys_start();

    harness_run(testcases);

    return (0);
}

The output from the test suite is:

app:    test_suite-7.0.0 built 2016-07-25 17:38 CEST by erik.
board:  Linux
mcu:    Linux

enter: test_passed
exit: test_passed: PASSED

enter: test_failed
exit: test_failed: FAILED

enter: test_skipped
exit: test_skipped: SKIPPED

            NAME        STATE  PRIO   CPU  LOGMASK
            main      current     0    0%     0x0f
                        ready   127    0%     0x0f
harness report: total(3), passed(1), failed(1), skipped(1)

There are plenty of test suites in the tst folder on Github.

Source code: src/debug/harness.h, src/debug/harness.c

Defines

_ASSERTFMT(fmt, ...)¶

_ASSERTHEX(actual_str, actual, expected_str, expected, actual_size, expected_size)¶

BTASSERTRM(cond, cond_str, res, msg)¶: Assert given condition. Mark testcase as failed, print an error message and return given value res on error.

BTASSERTR(cond, cond_str, res, ...)¶: Assert given condition. Mark testcase as failed, print an error message and return given value res on error.

BTASSERTN(cond, ...)¶: Assert given condition. Mark testcase as failed, print an error message and return given value on error.

BTASSERT(cond, ...)¶: Assert given condition. Mark testcase as failed, print an error message and return -1 on error.

BTASSERTI(actual, operator, expected)¶: Compare two integers actual and expected with given operator operator. Mark testcase as failed, print an error message if the condition is not true and return -1 on error.

BTASSERTM(actual, expected, size)¶: Comapre two memory positions actual and expected. Mark testcase as failed, print an error message if they are not equal and return -1 on error.

BTASSERTV(cond, ...)¶: Assert given condition in a testcase. Mark testcase as failed, print an error message and return on error.

BTASSERT_IN_RANGE(value, low, high)¶: Assert that given value is in given range. Mark testcase as failed, print an error message and return.

STUB(function)¶: Stub given function. Used with the make variable STUB to preprocess object file(s).

Typedefs

typedef int (*harness_testcase_cb_t)(void)¶

The testcase function callback.

Return: zero(0) if the testcase passed, a negative error code if the testcase failed, and a positive value if the testcase was skipped.

typedef int (*harness_mock_cb_t)(void *arg_p, void *buf_p, size_t *size_p)¶

The read/assert callback function. Modifies a copy of the original mock entry. The modified mock entry is read by read/assert functions.

Return

true(1) if the mock entry shall be removed, otherwise false(0).

Parameters

arg_p: arg_size bytes copied from arg_p given to harness_mock_cwrite().
buf_p: Mock entry data buffer, equivalent to buf_p given to harness_mock_cwrite().
size_p: Mock entry size, equivalent to size given to harness_mock_cwrite().

Functions

int harness_run(struct harness_testcase_t *testcases_p)¶

Run given testcases in the test harness.

Return

Never returns.

Parameters

testcases_p: An array of testcases to run. The last element in the array must have callback and name_p set to NULL.

int harness_expect(void *chan_p, const char *pattern_p, const struct time_t *timeout_p)¶

Continiously read from given channel and return when given pattern has been read, or when given timeout occurs.

Return

Number of bytes read from the channel when match occured, or negative error code.

Parameters

chan_p: Channel to read from.
pattern_p: Pattern to wait for.
timeout_p: Timeout, or NULL to wait the default timeout of one second.

ssize_t harness_mock_write(const char *id_p, const void *buf_p, size_t size)¶

Write given data buffer to a mock entry with given id. The mock entry can later be read with harness_mock_read() or harness_mock_try_read().

Return

Number of written words or negative error code.

Parameters

id_p: Mock id string to write.

            NOTE: Only a reference to this string is stored in
                  the mock entry.

buf_p: Data for given mock id, or NULL if no data shall be written.
size: Buffer size in words, or zero(0) if buf_p is NULL.

ssize_t harness_mock_mwrite(const char *id_p, const void *buf_p, size_t size, int length)¶

Write given data buffer to a mock entry with given id. The mock entry can later be read length times with harness_mock_read(), harness_mock_try_read() or harness_mock_assert().

Return

Number of written words or negative error code.

Parameters

id_p: Mock id string to write.

            NOTE: Only a reference to this string is stored in
                  the mock entry.

buf_p: Data for given mock id, or NULL if no data shall be written.
size: Buffer size in words, or zero(0) if buf_p is NULL.
length: Number of times this mock entry will be read/asserted.

ssize_t harness_mock_cwrite(const char *id_p, const void *buf_p, size_t size, harness_mock_cb_t cb, void *arg_p, size_t arg_size)¶

Write given data buffer to a mock entry with given id. The mock entry can later be read with harness_mock_read(), harness_mock_try_read() or harness_mock_assert() until the callback cb returns true(1).

Return

Number of written words or negative error code.

Parameters

id_p: Mock id string to write.

            NOTE: Only a reference to this string is stored in
                  the mock entry.

buf_p: Data for given mock id, or NULL if no data shall be written.
size: Buffer size in words, or zero(0) if buf_p is NULL.
cb: Callback function called on each read/assert of this mock entry. The mock entry will be removed once the callback returns true(1).
arg_p: Callback argument pointer.
arg_size: Callback argument size.

ssize_t harness_mock_read(const char *id_p, void *buf_p, size_t size)¶

Read data from mock entry with given id, and make the testcase fail if the mock id is not found or if given size does not match the size in the mock entry.

Return

Number of read words or negative error code.

Parameters

id_p: Mock id string to read.
buf_p: Buffer to read into, or NULL if no data shall be read.
size: Buffer size in words, or zero(0) if buf_p is NULL.

ssize_t harness_mock_try_read(const char *id_p, void *buf_p, size_t size)¶

Try to read data from mock entry with given id. The testcase does not fail if the mock entry is missing. However, the test case fails if the mock id is found and the data size does not match.

Return

Number of read words, -ENOENT if no mock entry was found for given id, or negative error code.

Parameters

id_p: Mock id string to read.
buf_p: Buffer to read into, or NULL if no data shall be loaded.
size: Buffer size in words, or zero(0) if buf_p is NULL.

int harness_mock_assert(const char *id_p, const void *buf_p, size_t size)¶

Find mock entry with given id and compare its data to given buffer. The testcase fails if the mock id is not found or on data mismatch.

Return

zero(0) or negative error code.

Parameters

id_p: Mock id string to assert.
buf_p: Buffer with expected data, or NULL if no data shall be compared.
size: Buffer size in words, or zero(0) if buf_p is NULL.

ssize_t harness_mock_write_notify(const char *id_p, const void *buf_p, size_t size)¶

Write given data buffer to a mock entry with given id and notify all raeders that data is available. The harness_mock_write_notify() and harness_mock_read_wait() functions are useful to mock communication interfaces between threads.

Return

Number of written words or negative error code.

Parameters

id_p: Mock id string to write.

            NOTE: Only a reference to this string is stored in
                  the mock entry.

buf_p: Data for given mock id, or NULL if no data shall be written.
size: Buffer size in words, or zero(0) if buf_p is NULL.

ssize_t harness_mock_read_wait(const char *id_p, void *buf_p, size_t size, struct time_t *timeout_p)¶

Read data from mock entry with given id. Suspends the current thread if the mock id is not found.

Return

Number of read words or negative error code.

Parameters

id_p: Mock id string to read.
buf_p: Buffer to read into, or NULL if no data shall be read.
size: Buffer size in words, or zero(0) if buf_p is NULL.
timeout_p: Read timeout.

int harness_set_testcase_result(int result)¶

Set currently executing testcase result to passed(0), skipped(1) or failed(-1).

Return

zero(0) or negative error code.

Parameters

result: Testcase result to set.

int harness_get_testcase_result(void)¶

Get currently executing testcase result.

Return: passed(0), skipped(1) or failed(-1).

struct harness_testcase_t¶

Public Members

harness_testcase_cb_t callback¶

const char *name_p¶

7.1. harness — Test harness¶

7.1.1. Stubs¶

7.1.2. Mocking¶

7.1.3. Example test suite¶

7.1. `harness` — Test harness¶