Changing the reporting mechanism of Cgreen.

Replacing the reporter

In every test suite so far, we have run the tests with this line...

return run_test_suite(our_tests(), create_text_reporter());

We can change the reporting mechanism just by changing this method. Here is the code for create_text_reporter()...

TestReporter *create_text_reporter() {
    TestReporter *reporter = create_reporter();
    reporter->start = &text_reporter_start;
    reporter->finish = &text_reporter_finish;
    reporter->show_fail = &show_fail;
    reporter->show_incomplete = &show_incomplete;
    return reporter;
}

The TestReporter structure contains function pointers that control the reporting. When called from create_reporter() constructor, these pointers are set up with functions that display nothing. The text reporter code replaces these with something more dramatic, and then returns a pointer to this new object. Thus the create_text_reporter() function effectively extends the object from create_reporter().

The text reporter only outputs content at the start of the first test, at the end of the test run to display the results, when a failure occours, and when a test fails to complete. A quick look at the text_reporter.c file in Cgreen reveals that the overrides just output a message and chain to the versions in reporter.h.

To change the reporting mechanism ourselves, we just have to know a little about the methods in the TestReporter structure.

The TestReporter structure

The Cgreen TestReporter is a pseudo class that looks something like...

typedef struct _TestReporter TestReporter;
struct _TestReporter {
    void (*destroy)(TestReporter *);
    void (*start)(TestReporter *, const char *);
    void (*finish)(TestReporter *, const char *);
    void (*show_pass)(TestReporter *, const char *, int, const char *, va_list);
    void (*show_fail)(TestReporter *, const char *, int, const char *, va_list);
    void (*show_incomplete)(TestReporter *, const char *);
    void (*assert_true)(TestReporter *, const char *, int, int, const char *, ...);
    int passes;
    int failures;
    int exceptions;
    void *breadcrumb;
    int ipc;
    void *memo;
};

The first block are the methods that can be overridden.

void (*destroy)(TestReporter *): This is the destructor for the default structure. If this is overridden, then the overriding function must call destroy_reporter(TestReporter *reporter) to finish the clean up.
void (*start)(TestReporter *, const char *): The first of the callbacks. At the start of each test suite Cgreen will call this method on the reporter with the name of the suite being entered. The default version keeps track of the stack of tests in the breadcrumb pointer of TestReporter. If you make use of the breadcrumb functions, as the defaults do, then you will need to call reporter_start() to keep the book keeping in sync.
void (*finish)(TestReporter *, const char *): The counterpart to the (*start)() call called on leaving the test suite. It needs to be chained to the reporter_finish() to keep track of the breadcrumb book keeping. The text reporter uses the state of the breadcrumb to decide if it is ending teh top level test. If so, it prints the familiar summary of passes and fails.
void (*show_pass)(TestReporter *, const char *, int, const char *, va_list): This method is initially empty, so there is no need to chain the call to any other function. Besides the pointer to the reporter structure, Cgreen also passes the file name of the test, the line number of failed assertion, the message to show and any additional parameters to substitute into the message. The message comes in as printf() style format string, and so the variable argument list should match the substitutions.
void (*show_fail)(TestReporter *, const char *, int, const char *, va_list): The partner of show_pass(), and the one you'll likely overload first.
void (*show_incomplete)(TestReporter *, const char *): When a test fails to complete, this is the handler that is called. As it's an unexpected outcome, no message is received, but we do get the name of the test. The text reporter combines this with the breadcrumb to produce the exception report.
void (*assert_true)(TestReporter *, const char *, int, int, const char *, ...): This is not normally overridden and is really internal. It is the raw entry point for the test messages from the test suite. By default it dispatches teh call to either show_pass() or show_fail().

The second block is simply resources and book keeping that the reporter can use to liven up the messages...

passes: The number of passes so far.
failures: The number of failures generated so far.
exceptions: The number of test functions that have failed to complete so far.
breadcrumb: This is a pointer to the list of test names in the stack.

The breadcrumb pointer is different and needs a little explanation. Basically it is a stack, analagous to the breadcrumb trail you see on websites. Everytime the start() handler is invoked, the name is placed in this stack. When the finish() message handler is invoked, a name is popped off.

There are a bunch of utility functions in cgreen/breadcrumb.h that can read the state of this stack. Most useful are get_current_from_breadcrumb() which takes the breadcrumb pointer and returns the curent test name, and get_breadcrumb_depth() which gives the current depth of the stack. A depth of zero means that the test run has finished.

If you need to traverse all the names in the breadcrumb, then you can call walk_breadcrumb(). Here is the full signature...

void walk_breadcrumb(Breadcrumb *breadcrumb, void (*walker)(const char *, void *), void *memo);

The void (*walker)(const char *, void *) is a callback that will be passed the name of the test suite for each level of nesting. It is also poassed the memo pointer that was passed to the walk_breadcrumb() call. You can use this pointer for anything you want, as all Cgreen does is pass it from call to call. This is so aggregate information can be kept track of whilst still being reentrant.

The last parts of the TestReporter structure are...

ipc: This is an internal structure for handling the messaging between reporter and test suite. You shouldn't touch this.
memo: By contrast, this is a spare pointer for your own expansion.

An example XML reporter

Let's make things real with an example. Suppose we want to send the output from Cgreen in XML format, say for storing in a repository or for sending across the network.

Suppose also that we have come up with the following format...

<?xml?>
<test name="Top Level">
    <test name="A Group">
        <test name="a_test_that_passes">
        </test>
        <test name="a_test_that_fails">
            <fail>
                <message><![CDATA[A failure]]></message>
                <location file="test_as_xml.c" line="8"/>
            </fail>
        </test>
    </test>
</test>

In other words a simple nesting of tests with only failures encoded. The absence of failure is a pass.

Here is a test script, test_in_xml.c that we can use to construct the above output...

#include "cgreen/cgreen.h"

void a_test_that_passes() {
    assert_true(1);
}

void a_test_that_fails() {
    assert_true_with_message(0, "A failure");
}

TestSuite *create_test_group() {
    TestSuite *suite = create_named_test_suite("A Group");
    add_test(suite, a_test_that_passes);
    add_test(suite, a_test_that_fails);
    return suite;
}

int main(int argc, char **argv) {
    TestSuite *suite = create_named_test_suite("Top Level");
    add_suite(suite, create_test_group());
    return run_test_suite(suite, create_text_reporter());
}

The text reporter is used just to confirm that everything is working. So far it is.

Running "Top Level"...
Failure!: A Group -> a_test_that_fails -> A failure at [test_as_xml.c] line [8]
Completed "Top Level": 1 pass, 1 failure, 0 exceptions.

Our first move is to switch the reporter from text, to our not yet written XML version...

#include "cgreen/cgreen.h
#include "xml_reporter.h"

...

int main(int argc, char **argv) {
    TestSuite *suite = create_named_test_suite("Top Level");
    add_suite(suite, create_test_group());
    return run_test_suite(suite, create_xml_reporter());
}

We'll start the ball rolling with the xml_reporter.h header file...

#ifndef _XML_REPORTER_HEADER_
#define _XML_REPORTER_HEADER_

#include "cgreen/reporter.h"

TestReporter *create_xml_reporter();

#endif

...and the simplest possible reporter in reporter.c.

#include "xml_reporter.h"
#include "cgreen/reporter.h"

TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    return reporter;
}

One that outputs nothing.

gcc -c test_as_xml.c
gcc -c xml_reporter.c
gcc xml_reporter.o test_as_xml.o cgreen/cgreen.a -o xml
./xml

Yep, nothing.

Let's add the outer test tags first, so that we can see Cgreen navigating the test suite...

#include "xml_reporter.h"
#include "cgreen/reporter.h"
#include <stdio.h>

static void xml_reporter_start(TestReporter *reporter, const char *name);
static void xml_reporter_finish(TestReporter *reporter, const char *name);

TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    reporter->start = &xml_reporter_start;
    reporter->finish = &xml_reporter_finish;
    return reporter;
}

static void xml_reporter_start(TestReporter *reporter, const char *name) {
    printf("<test name=\"%s\">\n", name);
    reporter_start(reporter, name);
}

static void xml_reporter_finish(TestReporter *reporter, const char *name) {
    reporter_finish(reporter, name);
    printf("</test>\n");
}

Although chaining to the underlying reporter_start() and reporter_finish() functions is optional, I want to make use of some of the facilities later.

Our output meanwhile, is making it's first tentative steps...

<test name="Top Level">
<test name="A Group">
<test name="a_test_that_passes">
</test>
<test name="a_test_that_fails">
</test>
</test>
</test>

We don't want a passing message, so the show_fail() function is all we need...

...
static void xml_show_fail(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments);

TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    reporter->start = &xml_reporter_start;
    reporter->finish = &xml_reporter_finish;
    reporter->show_fail = &xml_show_fail;
    return reporter;
}

...

static void xml_show_fail(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments) {
    printf("<fail>\n");
    printf("\t<message><![CDATA[");
    vprintf(message, arguments);
    printf("]]></message>\n");
    printf("\t<location file=\"%s\" line=\"%d\"/>\n", file, line);
    printf("</fail>\n");
}

We have to use vprintf() to handle the variable argument list passed to us. This will probably mean including the stdarg.h header as well as stdio.h.

This gets us pretty close to what we want...

<test name="Top Level">
<test name="A Group">
<test name="a_test_that_passes">
</test>
<test name="a_test_that_fails">
<fail>
    <message><![CDATA[A failure]]></message>
    <location file="test_as_xml.c" line="9"/></fail>
</test>
</test>
</test>

For completeness we should add a tag for an incomplete test. We'll output this as a failure, athough we don't get a location this time...

static void xml_show_incomplete(TestReporter *reporter, const char *name) {
    printf("<fail>\n");
    printf("\t<message><![CDATA[Failed to complete]]></message>\n");
    printf("</fail>\n");
}

All that's left then is the XML declaration and the thorny issue of indenting. Although the indenting is not strictly necessary, it would make the output a lot more readable.

The test depth is kept track of for us with the breadcrumb object in the TestReporter structure. We'll add an indent() function that outputs the correct number of tabs...

static indent(TestReporter *reporter) {
    int depth = get_breadcrumb_depth((Breadcrumb *)reporter->breadcrumb);
    while (depth-- > 0) {
        printf("\t");
    }
}

The get_breadcrumb_depth() function just gives the current test depth. As that is just the number of tabs to output, the implementation is trivial.

We can then use this function in the rest of the code. Here is the complete listing...

#include "xml_reporter.h"
#include "cgreen/reporter.h"
#include "cgreen/breadcrumb.h"
#include <stdio.h>
#include <stdarg.h>

static indent(TestReporter *reporter);
static void xml_reporter_start(TestReporter *reporter, const char *name);
static void xml_reporter_finish(TestReporter *reporter, const char *name);
static void xml_show_fail(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments);
static void xml_show_incomplete(TestReporter *reporter, const char *name);

TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    reporter->start = &xml_reporter_start;
    reporter->finish = &xml_reporter_finish;
    reporter->show_fail = &xml_show_fail;
    reporter->show_incomplete = &xml_show_incomplete;
    return reporter;
}

static indent(TestReporter *reporter) {
    int depth = get_breadcrumb_depth((Breadcrumb *)reporter->breadcrumb);
    while (depth-- > 0) {
        printf("\t");
    }
}

static void xml_reporter_start(TestReporter *reporter, const char *name) {
    if (get_breadcrumb_depth((Breadcrumb *)reporter->breadcrumb) == 0) {
        printf("<?xml?>\n");
    }
    indent(reporter);
    printf("<test name=\"%s\">\n", name);
    reporter_start(reporter, name);
}

static void xml_reporter_finish(TestReporter *reporter, const char *name) {
    reporter_finish(reporter, name);
    indent(reporter);
    printf("</test>\n");
}

static void xml_show_fail(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments) {
    indent(reporter);
    printf("<fail>\n");
    indent(reporter);
    printf("\t<message><![CDATA[");
    vprintf(message, arguments);
    printf("]]></message>\n");
    indent(reporter);
    printf("\t<location file=\"%s\" line=\"%d\"/>\n", file, line);
    indent(reporter);
    printf("</fail>\n");
}

static void xml_show_incomplete(TestReporter *reporter, const char *name) {
    indent(reporter);
    printf("<fail>\n");
    indent(reporter);
    printf("\t<message><![CDATA[Failed to complete]]></message>\n");
    indent(reporter);
    printf("</fail>\n");
}

And finally the desired output...

<?xml?>
<test name="Top Level">
    <test name="A Group">
        <test name="a_test_that_passes">
        </test>
        <test name="a_test_that_fails">
            <fail>
                <message><![CDATA[A failure]]></message>
                <location file="test_as_xml.c" line="9"/>
            </fail>
        </test>
    </test>
</test>

Job done.

Possible other extensions include reporters that write to syslog, talk to IDE plug-ins, paint pretty printed documents or just return a boolean for monitoring purposes.

Changing Cgreen Reporting

Replacing the reporter

The TestReporter structure

An example XML reporter