7.1. harness
— Test harness¶
In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. It has two main parts: the test execution engine and the test script repository.
This module implements the test execution engine.
The test scripts are part of the build system.
7.1.1. Stubs¶
Symbols can be stubbed per C-file using the STUB()
macro and
STUB
make variable. The STUB
make variable is a list of source
files and the symbols to stub within given file.
For example, stub functions foo_bar()
and foo_fie()
in
fum.c
by defining stub functions STUB(foo_bar)()
and
STUB(foo_fie)()
, and set the make variable STUB
to
fum.c:foo_bar,foo_fie
.
Prototypes for foo_bar()
and foo_fie()
in foo.h
:
/**
* The bar function.
*/
int foo_bar();
/**
* The fie function.
*/
int foo_fie();
foo_bar()
and foo_fie()
called in fum.c
. Both function
calls will call the stubbed version on the respective function.
int fum_init()
{
foo_bar();
foo_fie();
return (0);
}
The stubbed implementations, often defined in the test suite file
main.c
:
int STUB(foo_bar)()
{
return (0);
}
int STUB(foo_fie)()
{
return (0);
}
And last, add the stubbed symbol to the test suite makefile
Makefile
:
STUB = fum.c:foo_bar,foo_fie
7.1.2. Mocking¶
The test harness also supports function mocking with
mock_write_*()
and STUB(*)
functions. These functions are
generated by the script stub.py.
For example, generate a mock stub of foo.h
with the command
stub.py generate foo.h .
. Two files are created; foo_mock.h
and foo_mock.c
. Include foo_mock.h
in the test suite and call
mock_write_*()
to prepare the stub for function calls. The stub
will verify that all input arguments matches their expected values,
and write data to output arguments.
Note
The user may have to manually modify parts of the generated stubs to match her use case, as the stub script does not handle all situations properly.
#include "simba.h"
#include "foo_mock.h"
static int test_init(void)
{
/* Make foo_bar() return 1 and foo_fie() 5 once called. */
mock_write_foo_bar(1);
mock_write_foo_fie(5);
BTASSERT(fum_init() == 0);
return (0);
}
int main()
{
struct harness_testcase_t testcases[] = {
{ test_init, "test_init" },
{ NULL, NULL }
};
sys_start();
harness_run(testcases);
return (0);
}
Add the stub source file to the list of files to build.
STUB = fum.c:foo_bar,foo_fie
SRC += foo_mock.c
7.1.3. Example test suite¶
Below is an example of a test suite using the harness. It has three
test cases; test_passed
, test_failed
and test_skipped
.
The test macro BTASSERT(condition)
should be used to validate
conditions.
#include "simba.h"
static int test_passed(void)
{
/* Return zero(0) when a test case passes. */
return (0);
}
static int test_failed(void)
{
/* Return a negative integer when a test case fails. BTASSERT
will return -1 when the condition is false. */
BTASSERT(0);
return (0);
}
static int test_skipped(void)
{
/* Return a positive integer when a test case is skipped. */
return (1);
}
int main()
{
/* Test harness and NULL terminated list of test cases.*/
struct harness_testcase_t testcases[] = {
{ test_passed, "test_passed" },
{ test_failed, "test_failed" },
{ test_skipped, "test_skipped" },
{ NULL, NULL }
};
sys_start();
harness_run(testcases);
return (0);
}
The output from the test suite is:
app: test_suite-7.0.0 built 2016-07-25 17:38 CEST by erik.
board: Linux
mcu: Linux
enter: test_passed
exit: test_passed: PASSED
enter: test_failed
exit: test_failed: FAILED
enter: test_skipped
exit: test_skipped: SKIPPED
NAME STATE PRIO CPU LOGMASK
main current 0 0% 0x0f
ready 127 0% 0x0f
harness report: total(3), passed(1), failed(1), skipped(1)
There are plenty of test suites in the tst folder on Github.
Source code: src/debug/harness.h, src/debug/harness.c
Defines
-
_ASSERTFMT
(fmt, ...)¶
-
_ASSERTHEX
(actual_str, actual, expected_str, expected, actual_size, expected_size)¶
-
BTASSERTRM
(cond, cond_str, res, msg)¶ Assert given condition. Mark testcase as failed, print an error message and return given value
res
on error.
-
BTASSERTR
(cond, cond_str, res, ...)¶ Assert given condition. Mark testcase as failed, print an error message and return given value
res
on error.
-
BTASSERTN
(cond, ...)¶ Assert given condition. Mark testcase as failed, print an error message and return given value on error.
-
BTASSERT
(cond, ...)¶ Assert given condition. Mark testcase as failed, print an error message and return -1 on error.
-
BTASSERTI
(actual, operator, expected)¶ Compare two integers
actual
andexpected
with given operatoroperator
. Mark testcase as failed, print an error message if the condition is not true and return -1 on error.
-
BTASSERTM
(actual, expected, size)¶ Comapre two memory positions
actual
andexpected
. Mark testcase as failed, print an error message if they are not equal and return -1 on error.
-
BTASSERTV
(cond, ...)¶ Assert given condition in a testcase. Mark testcase as failed, print an error message and return on error.
-
BTASSERT_IN_RANGE
(value, low, high)¶ Assert that given value is in given range. Mark testcase as failed, print an error message and return.
-
STUB
(function)¶ Stub given function. Used with the make variable STUB to preprocess object file(s).
Typedefs
-
typedef int (*
harness_testcase_cb_t
)(void)¶ The testcase function callback.
- Return
- zero(0) if the testcase passed, a negative error code if the testcase failed, and a positive value if the testcase was skipped.
-
typedef int (*
harness_mock_cb_t
)(void *arg_p, void *buf_p, size_t *size_p)¶ The read/assert callback function. Modifies a copy of the original mock entry. The modified mock entry is read by read/assert functions.
- Return
- true(1) if the mock entry shall be removed, otherwise false(0).
- Parameters
arg_p
:arg_size
bytes copied fromarg_p
given toharness_mock_cwrite()
.buf_p
: Mock entry data buffer, equivalent tobuf_p
given toharness_mock_cwrite()
.size_p
: Mock entry size, equivalent tosize
given toharness_mock_cwrite()
.
Functions
-
int
harness_run
(struct harness_testcase_t *testcases_p)¶ Run given testcases in the test harness.
- Return
- Never returns.
- Parameters
testcases_p
: An array of testcases to run. The last element in the array must havecallback
andname_p
set to NULL.
-
int
harness_expect
(void *chan_p, const char *pattern_p, const struct time_t *timeout_p)¶ Continiously read from given channel and return when given pattern has been read, or when given timeout occurs.
- Return
- Number of bytes read from the channel when match occured, or negative error code.
- Parameters
chan_p
: Channel to read from.pattern_p
: Pattern to wait for.timeout_p
: Timeout, or NULL to wait the default timeout of one second.
-
ssize_t
harness_mock_write
(const char *id_p, const void *buf_p, size_t size)¶ Write given data buffer to a mock entry with given id. The mock entry can later be read with
harness_mock_read()
orharness_mock_try_read()
.- Return
- Number of written words or negative error code.
- Parameters
id_p
: Mock id string to write.NOTE: Only a reference to this string is stored in the mock entry.
buf_p
: Data for given mock id, or NULL if no data shall be written.size
: Buffer size in words, or zero(0) if buf_p is NULL.
-
ssize_t
harness_mock_mwrite
(const char *id_p, const void *buf_p, size_t size, int length)¶ Write given data buffer to a mock entry with given id. The mock entry can later be read
length
times withharness_mock_read()
,harness_mock_try_read()
orharness_mock_assert()
.- Return
- Number of written words or negative error code.
- Parameters
id_p
: Mock id string to write.NOTE: Only a reference to this string is stored in the mock entry.
buf_p
: Data for given mock id, or NULL if no data shall be written.size
: Buffer size in words, or zero(0) if buf_p is NULL.length
: Number of times this mock entry will be read/asserted.
-
ssize_t
harness_mock_cwrite
(const char *id_p, const void *buf_p, size_t size, harness_mock_cb_t cb, void *arg_p, size_t arg_size)¶ Write given data buffer to a mock entry with given id. The mock entry can later be read with
harness_mock_read()
,harness_mock_try_read()
orharness_mock_assert()
until the callbackcb
returns true(1).- Return
- Number of written words or negative error code.
- Parameters
id_p
: Mock id string to write.NOTE: Only a reference to this string is stored in the mock entry.
buf_p
: Data for given mock id, or NULL if no data shall be written.size
: Buffer size in words, or zero(0) if buf_p is NULL.cb
: Callback function called on each read/assert of this mock entry. The mock entry will be removed once the callback returns true(1).arg_p
: Callback argument pointer.arg_size
: Callback argument size.
-
ssize_t
harness_mock_read
(const char *id_p, void *buf_p, size_t size)¶ Read data from mock entry with given id, and make the testcase fail if the mock id is not found or if given size does not match the size in the mock entry.
- Return
- Number of read words or negative error code.
- Parameters
id_p
: Mock id string to read.buf_p
: Buffer to read into, or NULL if no data shall be read.size
: Buffer size in words, or zero(0) if buf_p is NULL.
-
ssize_t
harness_mock_try_read
(const char *id_p, void *buf_p, size_t size)¶ Try to read data from mock entry with given id. The testcase does not fail if the mock entry is missing. However, the test case fails if the mock id is found and the data size does not match.
- Return
- Number of read words, -ENOENT if no mock entry was found for given id, or negative error code.
- Parameters
id_p
: Mock id string to read.buf_p
: Buffer to read into, or NULL if no data shall be loaded.size
: Buffer size in words, or zero(0) if buf_p is NULL.
-
int
harness_mock_assert
(const char *id_p, const void *buf_p, size_t size)¶ Find mock entry with given id and compare its data to given buffer. The testcase fails if the mock id is not found or on data mismatch.
- Return
- zero(0) or negative error code.
- Parameters
id_p
: Mock id string to assert.buf_p
: Buffer with expected data, or NULL if no data shall be compared.size
: Buffer size in words, or zero(0) if buf_p is NULL.
-
ssize_t
harness_mock_write_notify
(const char *id_p, const void *buf_p, size_t size)¶ Write given data buffer to a mock entry with given id and notify all raeders that data is available. The
harness_mock_write_notify()
andharness_mock_read_wait()
functions are useful to mock communication interfaces between threads.- Return
- Number of written words or negative error code.
- Parameters
id_p
: Mock id string to write.NOTE: Only a reference to this string is stored in the mock entry.
buf_p
: Data for given mock id, or NULL if no data shall be written.size
: Buffer size in words, or zero(0) if buf_p is NULL.
-
ssize_t
harness_mock_read_wait
(const char *id_p, void *buf_p, size_t size, struct time_t *timeout_p)¶ Read data from mock entry with given id. Suspends the current thread if the mock id is not found.
- Return
- Number of read words or negative error code.
- Parameters
id_p
: Mock id string to read.buf_p
: Buffer to read into, or NULL if no data shall be read.size
: Buffer size in words, or zero(0) if buf_p is NULL.timeout_p
: Read timeout.
-
int
harness_set_testcase_result
(int result)¶ Set currently executing testcase result to passed(0), skipped(1) or failed(-1).
- Return
- zero(0) or negative error code.
- Parameters
result
: Testcase result to set.
-
int
harness_get_testcase_result
(void)¶ Get currently executing testcase result.
- Return
- passed(0), skipped(1) or failed(-1).
-
struct
harness_testcase_t
¶