Nowadays comprehensive testing is a must for any software product. However, it isn’t such a general rule when it comes to graphics applications. Many developers face difficulties when they have to test their rendering codes. Manual tests and visual feedback is sometimes satisfactory but if one would like to have automated regression tests usual approaches seem to fail. Even if at first sight unit testing of rendering code doesn’t look really straightforward, in fact it is. OpenGL is not an exception from this rule as well. In this article I would like to briefly present a few methods how to unit test OpenGL rendering code and also present my choice and the reasons behind the decision.

There are several ways how to create automated test cases for rendering code. To present the different approaches we first have to select a small portion of our rendering code to demonstrate the differences of each technique, mentioning the strengths and weaknesses of them.

Before going any further, we have to lay down our requirements against a good OpenGL unit testing environment:

• Verifies results – This is the most basic requirement for any testing framework. We have to have the ability to check whether the rendering code executed by the module is valid and works as expected.
• Productive – The usage and maintenance of the framework shall require minimal effort. Many times unit testing is attacked because it requires additional code writing. While this is generally true, a nice unit testing environment can be kept very simple yet flexible. An OpenGL testing environment shouldn’t be different.
• Fast – This is a general requirement for any unit testing environment especially when combined with a continuous integration framework. We want our test results as fast as possible as long feedback cycles severely slow down the development process.
• Standalone – Does not require complex setup or environmental support in order to be executed. This is a general requirement when we deal with unit testing as if the code is tightly coupled by any of the surroundings then both development and maintenance costs increase.
• Compatible – Does not require any special hardware so it can be tested on a machine that wouldn’t necessarily be suitable for manually testing the actual product. This is especially important when the target hardware is some type of embedded platform. It is also important to ensure that it will work on hardware provided by different vendors. In one word, it should comply to the standard, not to driver implementations.
• Cross-platform – Does not rely on the services of a particular operating system or platform, instead it can be executed on any machine as usually all unit tests. Of course, this restriction can be relaxed depending on actual use case scenarios.

Now that we know what we would like to achieve, we can continue with a sample use case. Lets say we would like to create an OpenGL 3.2 based rendering engine. One of the first things that we would write is a class (or set of classes) that will help us handling OpenGL buffer objects as it seems to be one of the main building blocks of such a system. As a very basic example, our first version of the buffer handling class will act simply as a wrapper for buffer objects having the following interface:

class Buffer {
public:
    Buffer();
    virtual ~Buffer();
};

As it can be seen for now we just require that our class to handle the creation and deletion of a buffer object. Obviously, our test has to check that the constructor successfully creates a buffer by calling glGenBuffers and the destructor deletes that by calling glDeleteBuffers with proper arguments. Now lets see what possibilities we have to test OpenGL rendering code and whether it conforms to our requirements and is able to test our simple module.

Checking rendered image

The most naive solution for creating automated tests for rendering code is to actually execute the OpenGL commands and check whether the rendering happened as expected. This can be done by comparing reference rendering results to the actual ones. This approach has the benefit that we actually verify the concrete behavior but lets see how it looks like when we check against our previously laid down requirements:

• Verifies results – Partially fulfilled. We check against the correct behavior, however, the ability to reproduce the actual same image is often difficult if not impossible due to different relaxations regarding to precision in both the standard and driver implementations. In order to have reproducible results the testing environment shall also provide some mechanisms to allow slight differences.
• Productive – Not met. It can be quite expensive to create an assertion system. Also, the production of reference data can be quite time consuming.
• Fast – Not met. Even if the checkers are highly optimized components of the framework, it wouldn’t fit into the time-frame of unit test cycles to execute possibly thousands of test cases that require complete verification of the produced image.
• Standalone – Not met. We have to setup a complete rendering environment in order to test even the simplest rendering code. Also, it relies on the assumption that the rendering code actually produces some image. As we can see in our buffer handling example, this is not always the case.
• Compatible – Not met. We need a testing machine that has the hardware capabilities to execute the rendering code and produce the required image.
• Cross-platform – Partially fulfilled. If our rendering code is cross-platform then it is possible to test it on any of the supported platforms. However, this makes the assertion system even more complicated as it also has to support the target platforms. Also, driver implementations may vary even further when dealing with different operating systems.

As we can see, even if this version is quite natural way of thinking for anybody it’s simply impractical and not feasible for actual use. To be able to find a good solution we must look deeper into what unit testing exactly is as the presented solution has nothing to do with it. In order to be able to do real unit testing we have to eliminate the dependency on OpenGL driver implementations and strictly concentrating on the module under test.

Fake OpenGL driver

The second presented solution is to create a layer between the code under testing and the actual OpenGL driver implementation. This can be easily achieved by creating a fake driver, as an example a dynamic library called opengl32.dll in case of Windows. This additional layer would do nothing else than just recording and checking whether the required API calls happened as expected. Providing an interface towards the testing environment that can be used to request the informations needed to make a verdict about the successfulness of the test case.

Beside that this version accommodates much more to the idea behind unit testing it also has the benefit that it is acting as a totally independent layer and does not directly disturb the development of the actual code. Still, if we go back to our checklist we have some issues that raise some concerns regarding to the applicability of this approach:

• Verifies results – Partially fulfilled. It is up to the implementation of the new layer whether it provides the required facilities to properly check the behavior of our tested code. Nevertheless, it also highly depends on the implementation on how we define correct behavior and the responsibility of the library.
• Productive – Partially fulfilled. Now we have a separate module that helps us in the testing. This may introduce some additional maintenance work but, of course, this depends on how intelligently is the library actually implemented.
• Fast – Mostly resolved. We do not have expensive assertions, however, as we have a quite restricted interface between our testing environment and the new layer we most probably met situations when we have to make trade-offs between speed and flexibility.
• Standalone – Resolved. We have a totally independent module that is responsible to simulate the surrounding environment of the code under testing as it should be when doing unit test. However, the question arises whether we would like this layer to be that separated from the testing code.
• Compatible – Resolved. There is no dependency on dedicated graphics hardware or any other piece of metal. In case of a robust driver simulation layer we can test our code on whatever platform we prefer.
• Cross-platform – Resolved. As previously mentioned, if the additional layer is well designed, there should be no problems regarding to this issue.

Now we have a resolution that can be seriously taken into consideration as a good way to test rendering code. It can also be simply applied to test our buffer handling code as well. Also, as it is a totally standalone software element it is also very portable so it is easy to reuse between projects written in different programming languages and for different platforms.

Still, there is one thing that may need further investigation. Most probably for the other portions of our production code we already use some kind of mocking mechanisms for our unit testing. Having an additional interface type to handle the OpenGL related mocking (as the presented fake driver approach is nothing more than a mock library for OpenGL) may reduce the productivity of our developers. Also, it can make the testing code less uniform so introducing a slight maintenance penalty. At least for comparison, we should try to integrate the OpenGL mocking into our existing mocking facilities.

API mocks

All the people who seriously do unit testing use some mocking techniques to eliminate dependency on any external software element like databases, network or another code element. Why should the OpenGL API be different?

As I already written about that I use GoogleMock to test my C++ code. Lets see how this mocking framework is capable for removing OpenGL related dependencies. By default, GoogleMock does support only class mocks, however it is fairly straightforward to mock out OpenGL API functions as well. As an example, our buffer handling class needs at least a mock for the glGenBuffers and glDeleteBuffers API functions. These mocks can be very easily created using GoogleMock as part of a class in the following way:

class CGLMock {
public:
    MOCK_METHOD2( GenBuffers, void(GLsizei n, GLuint* buffers) );
    MOCK_METHOD2( DeleteBuffers, void(GLsizei n, GLuint* buffers) );
};
CGLMock GLMock;

This, however is not enough to replace the already existing real API function pointers with the fake ones. I did this with a nasty little trick by taking advantage of the C preprocessor:

#undef glGenBuffers
#define glGenBuffers                  GLMock.GenBuffers
#undef glDeleteBuffers
#define glDeleteBuffers               GLMock.DeleteBuffers

The #undef is needed because I use GLEW for accessing OpenGL API functions and it uses macros for the API function names as well.

All these are put into a file that can be called like glmock.h. In order to force the production code to use these definitions when trying to access the API inside a test case we have to create a wrapper header called something like opengl.h that will include original headers in case of normal build and include the mock library in case of unit test build. This is kind of a workaround but it works quite well in practice.

In theory, this trick can be applied in case of any mocking framework. As a result, from now we can write a very simple test case to check the creation and deletion of our buffer object as easily as the following few lines of code:

TEST(BufferTest, CreationAndDestruction) {
    EXPECT_CALL(GLMock, GenBuffers(1,_))
        .WillOnce(SetArgumentPointee<1>(13));
    Buffer* buffer = new Buffer;
    EXPECT_CALL(GLMock, DeleteBuffers(1,Pointee(13)));
    delete buffer;
}

I would not like to go into the details related to the interface of GoogleMock. In one word, the test case above checks whether the constructor calls glGenBuffers with a number of 1 for the requested number of buffer objects and returns a buffer ID in the pointer argument, and at the end it checks if glDeleteBuffers was called with the buffer ID value got at creation.

It is maybe a matter of taste whether the second or this third solution is more attractive for you. My choice was this last solution because I didn’t want to develop an separate library and also was afraid of messing up my test code with different syntactical representations of mocks. Finally, lets sum up the achievements of this last version:

• Verifies results – Fulfilled. An existing mocking framework is used for emulating the OpenGL API thus we have all the facilities required for the proper checking of the API calls.
• Productive – Fulfilled. Again, we don’t have to deal with writing an own mocking mechanisms as we have everything out of the box. We can also incrementally extend our mock library on-the-fly while editing the test cases and the production code.
• Fast – Resolved. Our rendering related unit test cases should be as fast as any other test codes as they are indifferent, just the purposes are dissimilar.
• Standalone – Mostly resolved. The mocking library is independent, however, as we’ve seen, the introduction may require some nasty tricks in order to inject foreign code into the production code.
• Compatible – Resolved. From this point of view, this approach behaves the same as the previous version.
• Cross-platform – Resolved. Again, the same like in the previous case, maybe even a bit easier to make it portable.

Conclusion

We’ve seen a few ways how we can extend our testing environment in order to support the verification of rendering code. We’ve also seen that the range varies from techniques that provide high level methods suitable especially for functional testing, until very low level methods that tightly integrate in the mocking methodology of unit testing. These, of course, do not replace traditional testing methods rather they extend it in order to find problems in the early phases of software development.

I also tried to present a very basic example of production code that needs such a facility in order to be tested, as well as a sample test case written using GoogleMocks applying the last presented technique.

While writing this article I got the idea that it would be nice to have a complete and general framework for OpenGL testing. If there is interest for it, maybe I’ll allocate some time to write one. I’m also interested which approach is the most attractive for you, especially if you have some concrete experience with any of these or with some other technique.

Many people are looking for information about which particular C++ unit testing framework they should use for their project and there are also many articles discuss the topic but few articles talk about mock frameworks which are even more important factor when applying unit testing in practice and they have much greater effect on the productivity when doing test-driven development.

Test-driven development in a nutshell

My first experience with unit testing happened to be during my university studies. As we learned Java development, we also worked a little with JUnit as it is the most popular unit testing framework for Java and also the grandfather of almost all current frameworks. I felt about it like pain in the a** because it seemed ridiculous to use it for our very dummy programs and because I’m really not a big fan of Java and it’s monolithic development environment called NetBeans, but I will talk about it in another article.

Next I’ve met unit testing at my current job where we used TNSDLUnit which is a proprietary framework developed by the employees as an internal open-source project for our TNSDL language. Basically this framework is just a TNSDL specific version of the well known CPPUTest framework. This time I’ve seen real life examples and I soon discovered the potentials in unit testing and in particular its application in test-driven development.

Test-driven development, or TDD in short, is a software development process that became very popular in the recent past and not accidentally as it is a very powerful tool in good hands. It introduces a very simple development cycle that results in simple, clean and test covered code. The process consists of five straightforward steps:

1. Add a test – this is made based on a particular requirement
2. Run all tests and see if the new one fails – this ensures that the test really needs modification to the code
3. Write some code – modify code to make the new test pass
4. Run all tests and see them succeed – if the implementation is good then the new test now must pass
5. Refactor code – tidy up the code as you can be sure now that if you do something wrong then tests will fail

Not just this development method provides 100% test coverage for your code but it forces you to make the implementation based on requirements not vice versa.

TDD in a more strict manner also states that one must not write more code than it is necessary to make the test pass. This results sometimes in very awkward implementation stubs at the beginning but it prevents the developer from creating untested code (note that 100% coverage usually does not enough for proper testing).

Why use unit testing?

Many people have concerns about the necessity of testing and ask why test and why use unit testing. Well, as it maybe seems ridiculous to unit test a code which consists of a few hundreds or thousands of source line and is perspicuous even for the less competent people, the importance of testing comes into force when developing huge sized codes which are continuously modified by a group of people. One can not just test whether a newly implemented functionality is working as expected but can execute automated regression sets in order to ensure that no old functionality is crashed due to a modification.

Others have concerns about that unit testing doubles the code as so increases maintenance because the testing code is as big as the implementation itself. By the way, in a real life example the testing code can even be double the size of the production code. However, the added benefit of a well tested product eliminates most of the cost of bug fixes as they are discovered in the very beginning of the development process.

C++ unit test frameworks

When I’ve started to work with C++ one of my first thing was to look around for a good unit test framework. There are plenty of them so I will mention only some of the most popular ones: CPPUnit, CPPUTest, CxxTest, UnitTest++, Boost Test Library, GoogleTest, etc.

Most people spend too much time on deciding which unit test framework to use while they all have very similar syntax structure and all support the most needed assertions and facilities that are needed to get started with TDD. I think this is just a matter of taste.

Resolving dependencies

As one starts to actively use unit testing will face difficulties that rise from the dependencies between different elements of the software system, whether it be an external database, a foreign module or just an own class referenced by the code unit under test. These problems force people to write stubs or mocks to remove external dependencies.

Stubs are just not flexible enough and writing mocks sometimes seems to be too expensive. However, both have a great penalty on development time and maintenance cost. So, at a point, dependencies become the main problem and as such the relevance of which unit test framework to use gets hidden and people start to look for a mock framework. These libraries not just enable easy creation of mock objects but also integrates them tightly into the unit test framework and sometimes they even provide automated mock generation. Fortunately there are also plenty of mock frameworks for C++. Here are some of them: mockpp, Mock Objects, mockcpp, GoogleMock, etc.

Here the differences are much more visible from both from usability, portability and maturity point of view. Some of them automatically generate mocks, others need code for that as well. Some of them depend on specific application binary interface (ABI) formats, others not. Some of them need exotic language features while others work on all the significant compilers. Also, due to the complexity of the C++ language it is very hard to correctly parse code that is heavily obfuscated by macros, so in many cases the automatic generation of mocks simply doesn’t work.

I’ve put my vote on GoogleMock and as such I started to use GoogleTest as a unit test framework because of the following reasons:

• It does not depend on any special language feature or ABI format so it’s portable
• It is natively integrates with GoogleTest so no need to worry about compatibility issues
• Mocks have to be hand written, however it needs just a few lines of code (it also has a mock generator, however I don’t use it)

Summary

As a final conclusion, if you take my advice then you don’t take anybody’s advice. Try it out yourself and see it whether a particular tool set fits your taste or not. It strongly depends on what you plan to unit test. Just keep it simple and have fun with TDD. Yes, it’s really fun!