7.1. harness
— Test harness¶
In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. It has two main parts: the test execution engine and the test script repository.
This module implements the test execution engine.
The test scripts are part of the build system.
7.1.1. Stubs¶
Symbols can be stubbed per C-file using the STUB()
macro and
STUB
make variable. The STUB
make variable is a list of source
files and the symbols to stub within given file.
For example, stub functions foo_bar()
and foo_fie()
in
fum.c
by defining stub functions STUB(foo_bar)()
and
STUB(foo_fie)()
, and set the make variable STUB
to
fum.c:foo_bar,foo_fie
.
Prototypes for foo_bar()
and foo_fie()
in foo.h
:
int foo_bar();
int foo_fie();
foo_bar()
and foo_fie()
called in fum.c
. Both function
calls will call the stubbed version on the respective function.
int fum_init()
{
foo_bar();
foo_fie();
}
The stubbed implementations, often defined in the test suite file
main.c
:
int STUB(foo_bar)()
{
return (0);
}
int STUB(foo_fie)()
{
return (0);
}
And last, add the stubbed symbol to the test suite makefile
Makefile
:
STUB = fum.c:foo_bar,foo_fie
7.1.2. Example test suite¶
Below is an example of a test suite using the harness. It has three
test cases; test_passed
, test_failed
and test_skipped
.
The test macro BTASSERT(condition)
should be used to validate
conditions.
#include "simba.h"
static int test_passed(struct harness_t *harness_p)
{
/* Return zero(0) when a test case passes. */
return (0);
}
static int test_failed(struct harness_t *harness_p)
{
/* Return a negative integer when a test case fails. BTASSERT
will return -1 when the condition is false. */
BTASSERT(0);
return (0);
}
static int test_skipped(struct harness_t *harness_p)
{
/* Return a positive integer when a test case is skipped. */
return (1);
}
int main()
{
/* Test harness and NULL terminated list of test cases.*/
struct harness_t harness;
struct harness_testcase_t harness_testcases[] = {
{ test_passed, "test_passed" },
{ test_failed, "test_failed" },
{ test_skipped, "test_skipped" },
{ NULL, NULL }
};
sys_start();
harness_init(&harness);
harness_run(&harness, harness_testcases);
return (0);
}
The output from the test suite is:
app: test_suite-7.0.0 built 2016-07-25 17:38 CEST by erik.
board: Linux
mcu: Linux
enter: test_passed
exit: test_passed: PASSED
enter: test_failed
exit: test_failed: FAILED
enter: test_skipped
exit: test_skipped: SKIPPED
NAME STATE PRIO CPU LOGMASK
main current 0 0% 0x0f
ready 127 0% 0x0f
harness report: total(3), passed(1), failed(1), skipped(1)
There are plenty of test suites in the tst folder on Github.
Source code: src/debug/harness.h, src/debug/harness.c
Defines
-
_ASSERTFMT
(fmt, ...) std_printf(FSTR(fmt "\n"), ##__VA_ARGS__);¶
-
_ASSERTHEX
(actual_str, actual, expected_str, expected, size) std_printf(FSTR(":: \r\n" \ "Memory buffer '" actual_str "'\r\n")); \ std_hexdump(sys_get_stdout(), actual, size); \ std_printf(FSTR("is not equal to memory buffer '" expected_str "'\r\n")); \ std_hexdump(sys_get_stdout(), expected, size);¶
-
BTASSERTRM
(cond, cond_str, res, msg) if (!(cond)) { \ std_printf(FSTR(__FILE__ ":" STRINGIFY(__LINE__) ": BTASSERT: " \ cond_str " ")); \ msg; \ return (res); \ }¶ Assert given condition. Print an error message and return given value
res
on error.
-
BTASSERTR
(cond, cond_str, res, ...) BTASSERTRM(cond, cond_str, res, _ASSERTFMT(__VA_ARGS__));¶ Assert given condition. Print an error message and return given value
res
on error.
-
BTASSERTN
(cond, ...) BTASSERTR(cond, #cond, NULL, __VA_ARGS__)¶ Assert given condition. Print an error message and return given value on error.
-
BTASSERT
(cond, ...) BTASSERTR(cond, #cond, -1, __VA_ARGS__)¶ Assert given condition. Print an error message and return.
-
BTASSERTI
(actual, operator, expected) do { \ int UNIQUE(_actual); \ int UNIQUE(_expected); \ UNIQUE(_actual) = (actual); \ UNIQUE(_expected) = (expected); \ BTASSERTR(UNIQUE(_actual) operator UNIQUE(_expected), \ #actual " " #operator " " #expected, \ -1, \ ":: Condition '%d " #operator " %d' ('0x%x " \ #operator " 0x%x') is not true.", \ UNIQUE(_actual), \ UNIQUE(_expected), \ UNIQUE(_actual), \ UNIQUE(_expected)); \ } while (0)¶ Compare two integers
actual
andexpected
with given operatoroperator
. Print an error message if the condition is not true and return.
-
BTASSERTM
(actual, expected, size) do { \ const void *UNIQUE(_actual); \ const void *UNIQUE(_expected); \ UNIQUE(_actual) = (actual); \ UNIQUE(_expected) = (expected); \ BTASSERTRM(memcmp(UNIQUE(_actual), UNIQUE(_expected), size) == 0, \ "memcmp(" #actual ", " #expected ", " #size ") == 0", \ -1, \ _ASSERTHEX(#actual, UNIQUE(_actual), \ #expected, UNIQUE(_expected), \ size)); \ } while (0)¶ Comapre two memory positions
actual
andexpected
. Print an error message if they are not equal and return.
-
BTASSERTV
(cond, ...) if (!(cond)) { \ std_printf(FSTR(__FILE__ ":" STRINGIFY(__LINE__) ": BTASSERT: " #cond " ")); \ _ASSERTFMT(__VA_ARGS__); \ return; \ }¶ Assert given condition in a testcase. Print an error message and return -1 on error.
-
STUB
(function) __stub_ ## function¶ Stub given function. Used with the make variable STUB to preprocess object file(s).
Typedefs
Functions
-
int
harness_init
(struct harness_t *self_p)¶ Initialize given test harness.
- Return
- zero(0) or negative error code.
- Parameters
self_p
: Test harness to initialize.
-
int
harness_run
(struct harness_t *self_p, struct harness_testcase_t *testcases_p)¶ Run given testcases in given test harness.
- Return
- zero(0) or negative error code.
- Parameters
self_p
: Test harness.testcases_p
: An array of testcases to run. The last element in the array must havecallback
andname_p
set to NULL.
-
int
harness_expect
(void *chan_p, const char *pattern_p, const struct time_t *timeout_p)¶ Continiously read from the channel and return when given pattern has been read, or when a timeout occurs.
- Return
- Number of bytes read from the channel when match occured, or negative error code.
- Parameters
chan_p
: Channel to read from.pattern_p
: Pattern to wait for.timeout_p
: Timeout, or NULL to wait forever.
-
ssize_t
harness_mock_write
(const char *id_p, const void *buf_p, size_t size)¶ Write given data buffer to a mock entry with given id.
- Return
- Number of written words or negative error code.
- Parameters
id_p
: Mock id string to write.buf_p
: Data for given mock id.size
: Buffer size in words.
-
ssize_t
harness_mock_read
(const char *id_p, void *buf_p, size_t size)¶ Read data from mock entry with given id.
- Return
- Number of read words or negative error code.
- Parameters
id_p
: Mock id string to read.buf_p
: Buffer to read into.size
: Buffer size in words.
-
struct
harness_testcase_t
¶