Nowadays comprehensive testing is a must for any software product. However, it isn’t such a general rule when it comes to graphics applications. Many developers face difficulties when they have to test their rendering codes. Manual tests and visual feedback is sometimes satisfactory but if one would like to have automated regression tests usual approaches seem to fail. Even if at first sight unit testing of rendering code doesn’t look really straightforward, in fact it is. OpenGL is not an exception from this rule as well. In this article I would like to briefly present a few methods how to unit test OpenGL rendering code and also present my choice and the reasons behind the decision.

There are several ways how to create automated test cases for rendering code. To present the different approaches we first have to select a small portion of our rendering code to demonstrate the differences of each technique, mentioning the strengths and weaknesses of them.

Before going any further, we have to lay down our requirements against a good OpenGL unit testing environment:

• Verifies results – This is the most basic requirement for any testing framework. We have to have the ability to check whether the rendering code executed by the module is valid and works as expected.
• Productive – The usage and maintenance of the framework shall require minimal effort. Many times unit testing is attacked because it requires additional code writing. While this is generally true, a nice unit testing environment can be kept very simple yet flexible. An OpenGL testing environment shouldn’t be different.
• Fast – This is a general requirement for any unit testing environment especially when combined with a continuous integration framework. We want our test results as fast as possible as long feedback cycles severely slow down the development process.
• Standalone – Does not require complex setup or environmental support in order to be executed. This is a general requirement when we deal with unit testing as if the code is tightly coupled by any of the surroundings then both development and maintenance costs increase.
• Compatible – Does not require any special hardware so it can be tested on a machine that wouldn’t necessarily be suitable for manually testing the actual product. This is especially important when the target hardware is some type of embedded platform. It is also important to ensure that it will work on hardware provided by different vendors. In one word, it should comply to the standard, not to driver implementations.
• Cross-platform – Does not rely on the services of a particular operating system or platform, instead it can be executed on any machine as usually all unit tests. Of course, this restriction can be relaxed depending on actual use case scenarios.

Now that we know what we would like to achieve, we can continue with a sample use case. Lets say we would like to create an OpenGL 3.2 based rendering engine. One of the first things that we would write is a class (or set of classes) that will help us handling OpenGL buffer objects as it seems to be one of the main building blocks of such a system. As a very basic example, our first version of the buffer handling class will act simply as a wrapper for buffer objects having the following interface:

class Buffer {
public:
    Buffer();
    virtual ~Buffer();
};

As it can be seen for now we just require that our class to handle the creation and deletion of a buffer object. Obviously, our test has to check that the constructor successfully creates a buffer by calling glGenBuffers and the destructor deletes that by calling glDeleteBuffers with proper arguments. Now lets see what possibilities we have to test OpenGL rendering code and whether it conforms to our requirements and is able to test our simple module.

Checking rendered image

The most naive solution for creating automated tests for rendering code is to actually execute the OpenGL commands and check whether the rendering happened as expected. This can be done by comparing reference rendering results to the actual ones. This approach has the benefit that we actually verify the concrete behavior but lets see how it looks like when we check against our previously laid down requirements:

• Verifies results – Partially fulfilled. We check against the correct behavior, however, the ability to reproduce the actual same image is often difficult if not impossible due to different relaxations regarding to precision in both the standard and driver implementations. In order to have reproducible results the testing environment shall also provide some mechanisms to allow slight differences.
• Productive – Not met. It can be quite expensive to create an assertion system. Also, the production of reference data can be quite time consuming.
• Fast – Not met. Even if the checkers are highly optimized components of the framework, it wouldn’t fit into the time-frame of unit test cycles to execute possibly thousands of test cases that require complete verification of the produced image.
• Standalone – Not met. We have to setup a complete rendering environment in order to test even the simplest rendering code. Also, it relies on the assumption that the rendering code actually produces some image. As we can see in our buffer handling example, this is not always the case.
• Compatible – Not met. We need a testing machine that has the hardware capabilities to execute the rendering code and produce the required image.
• Cross-platform – Partially fulfilled. If our rendering code is cross-platform then it is possible to test it on any of the supported platforms. However, this makes the assertion system even more complicated as it also has to support the target platforms. Also, driver implementations may vary even further when dealing with different operating systems.

As we can see, even if this version is quite natural way of thinking for anybody it’s simply impractical and not feasible for actual use. To be able to find a good solution we must look deeper into what unit testing exactly is as the presented solution has nothing to do with it. In order to be able to do real unit testing we have to eliminate the dependency on OpenGL driver implementations and strictly concentrating on the module under test.

Fake OpenGL driver

The second presented solution is to create a layer between the code under testing and the actual OpenGL driver implementation. This can be easily achieved by creating a fake driver, as an example a dynamic library called opengl32.dll in case of Windows. This additional layer would do nothing else than just recording and checking whether the required API calls happened as expected. Providing an interface towards the testing environment that can be used to request the informations needed to make a verdict about the successfulness of the test case.

Beside that this version accommodates much more to the idea behind unit testing it also has the benefit that it is acting as a totally independent layer and does not directly disturb the development of the actual code. Still, if we go back to our checklist we have some issues that raise some concerns regarding to the applicability of this approach:

• Verifies results – Partially fulfilled. It is up to the implementation of the new layer whether it provides the required facilities to properly check the behavior of our tested code. Nevertheless, it also highly depends on the implementation on how we define correct behavior and the responsibility of the library.
• Productive – Partially fulfilled. Now we have a separate module that helps us in the testing. This may introduce some additional maintenance work but, of course, this depends on how intelligently is the library actually implemented.
• Fast – Mostly resolved. We do not have expensive assertions, however, as we have a quite restricted interface between our testing environment and the new layer we most probably met situations when we have to make trade-offs between speed and flexibility.
• Standalone – Resolved. We have a totally independent module that is responsible to simulate the surrounding environment of the code under testing as it should be when doing unit test. However, the question arises whether we would like this layer to be that separated from the testing code.
• Compatible – Resolved. There is no dependency on dedicated graphics hardware or any other piece of metal. In case of a robust driver simulation layer we can test our code on whatever platform we prefer.
• Cross-platform – Resolved. As previously mentioned, if the additional layer is well designed, there should be no problems regarding to this issue.

Now we have a resolution that can be seriously taken into consideration as a good way to test rendering code. It can also be simply applied to test our buffer handling code as well. Also, as it is a totally standalone software element it is also very portable so it is easy to reuse between projects written in different programming languages and for different platforms.

Still, there is one thing that may need further investigation. Most probably for the other portions of our production code we already use some kind of mocking mechanisms for our unit testing. Having an additional interface type to handle the OpenGL related mocking (as the presented fake driver approach is nothing more than a mock library for OpenGL) may reduce the productivity of our developers. Also, it can make the testing code less uniform so introducing a slight maintenance penalty. At least for comparison, we should try to integrate the OpenGL mocking into our existing mocking facilities.

API mocks

All the people who seriously do unit testing use some mocking techniques to eliminate dependency on any external software element like databases, network or another code element. Why should the OpenGL API be different?

As I already written about that I use GoogleMock to test my C++ code. Lets see how this mocking framework is capable for removing OpenGL related dependencies. By default, GoogleMock does support only class mocks, however it is fairly straightforward to mock out OpenGL API functions as well. As an example, our buffer handling class needs at least a mock for the glGenBuffers and glDeleteBuffers API functions. These mocks can be very easily created using GoogleMock as part of a class in the following way:

class CGLMock {
public:
    MOCK_METHOD2( GenBuffers, void(GLsizei n, GLuint* buffers) );
    MOCK_METHOD2( DeleteBuffers, void(GLsizei n, GLuint* buffers) );
};
CGLMock GLMock;

This, however is not enough to replace the already existing real API function pointers with the fake ones. I did this with a nasty little trick by taking advantage of the C preprocessor:

#undef glGenBuffers
#define glGenBuffers                  GLMock.GenBuffers
#undef glDeleteBuffers
#define glDeleteBuffers               GLMock.DeleteBuffers

The #undef is needed because I use GLEW for accessing OpenGL API functions and it uses macros for the API function names as well.

All these are put into a file that can be called like glmock.h. In order to force the production code to use these definitions when trying to access the API inside a test case we have to create a wrapper header called something like opengl.h that will include original headers in case of normal build and include the mock library in case of unit test build. This is kind of a workaround but it works quite well in practice.

In theory, this trick can be applied in case of any mocking framework. As a result, from now we can write a very simple test case to check the creation and deletion of our buffer object as easily as the following few lines of code:

TEST(BufferTest, CreationAndDestruction) {
    EXPECT_CALL(GLMock, GenBuffers(1,_))
        .WillOnce(SetArgumentPointee<1>(13));
    Buffer* buffer = new Buffer;
    EXPECT_CALL(GLMock, DeleteBuffers(1,Pointee(13)));
    delete buffer;
}

I would not like to go into the details related to the interface of GoogleMock. In one word, the test case above checks whether the constructor calls glGenBuffers with a number of 1 for the requested number of buffer objects and returns a buffer ID in the pointer argument, and at the end it checks if glDeleteBuffers was called with the buffer ID value got at creation.

It is maybe a matter of taste whether the second or this third solution is more attractive for you. My choice was this last solution because I didn’t want to develop an separate library and also was afraid of messing up my test code with different syntactical representations of mocks. Finally, lets sum up the achievements of this last version:

• Verifies results – Fulfilled. An existing mocking framework is used for emulating the OpenGL API thus we have all the facilities required for the proper checking of the API calls.
• Productive – Fulfilled. Again, we don’t have to deal with writing an own mocking mechanisms as we have everything out of the box. We can also incrementally extend our mock library on-the-fly while editing the test cases and the production code.
• Fast – Resolved. Our rendering related unit test cases should be as fast as any other test codes as they are indifferent, just the purposes are dissimilar.
• Standalone – Mostly resolved. The mocking library is independent, however, as we’ve seen, the introduction may require some nasty tricks in order to inject foreign code into the production code.
• Compatible – Resolved. From this point of view, this approach behaves the same as the previous version.
• Cross-platform – Resolved. Again, the same like in the previous case, maybe even a bit easier to make it portable.

Conclusion

We’ve seen a few ways how we can extend our testing environment in order to support the verification of rendering code. We’ve also seen that the range varies from techniques that provide high level methods suitable especially for functional testing, until very low level methods that tightly integrate in the mocking methodology of unit testing. These, of course, do not replace traditional testing methods rather they extend it in order to find problems in the early phases of software development.

I also tried to present a very basic example of production code that needs such a facility in order to be tested, as well as a sample test case written using GoogleMocks applying the last presented technique.

While writing this article I got the idea that it would be nice to have a complete and general framework for OpenGL testing. If there is interest for it, maybe I’ll allocate some time to write one. I’m also interested which approach is the most attractive for you, especially if you have some concrete experience with any of these or with some other technique.

Previously I talked about how one can easily take advantage of multiprocessing using OpenMP. Even if the C pragmas introduced by the parallel programming API standard is very straightforward for simple programs, it simply doesn’t fit nicely in a complex C++ application that is built from the ground with the OOP in mind. To smoothly introduce OpenMP into such projects one need higher level constructs that hide the actual implementation details. This is the first article of a series that will try to provide reference implementations of such an abstraction. First, we will start with synchronizable primitives that try to reflect the functionality provided by the “synchronized” statement of Java.

This article is highly inspired by an article written by Achilleas Margaritis and is mostly equivalent with his thoughts. My article tries to provide a portable reference implementation of a slightly modified version of the trick presented by Margaritis that uses OpenMP as the multiprocessing API back-end.

Motivation

According to the OO paradigm, classes and consequently objects provide an abstract interface to the underlying internal data or services of the modeled entity or entity class. When it comes to parallel programing we should provide facilities to enable concurrent access to shared resources that are in this case objects. Using plain OpenMP can be satisfactory, however when used extensively the OpenMP pragmas and API function calls introduced can greatly affect the readability and the maintainability of the code. Nevertheless, there can be platforms that use other APIs for handling race conditions. It is obvious that we need to encapsulate these facilities and provide an abstract tool-set instead.

Implementation

The very first building block of such a framework can be a mutex class that provides mutually exclusive access to certain resources. In the world of OpenMP this should look like something similar to the following:

class Mutex {
public:
    Mutex() { omp_init_lock(&_mutex); }
    ~Mutex() { omp_destroy_lock(&_mutex); }
    void lock() { omp_set_lock(&_mutex); }
    void unlock() { omp_unset_lock(&_mutex); }
private:
    omp_lock_t _mutex;
};

This seems already enough for us to make our Java-like “synchronized” statement, however we would like to create a framework that makes usage as easy and safe as possible. In order to get closer to this goal we apply the RAII (Resource Acquisition Is Initialization) design pattern to create our lock class:

class Lock {
public:
    Lock(Mutex& mutex) : _mutex(mutex), _release(false) { _mutex.lock(); }
    ~Lock() { _mutex.unlock(); }
    bool operator() const { return !_release; }
    void release() { _release = true; }
private:
    Mutex& _mutex;
    bool _release;
};

Our goal is to provide an inheritable interface for such objects that needs synchronization. However, this step has to involve severe considerations regarding to the provided interface as we explicitly need to conform to the following requirements:

• The interface shall not expose the interface of the underlying synchronization primitive, in our case the mutex class methods.
• The interface shall be available only to the synchronizable objects but not for the external world as we would like to not just hide the implementation details of our abstract entity but also prevent the users to synchronize our objects as it should be the responsibility of the object itself.
• The interface shall expose methods which are less prone to name collision, for convenience.

If we take care of the presented conventions we end up with an interface similar to the following:

class Synchronizable: protected Mutex {
protected:
	void enterSyncBlock() { this->lock(); }
	void exitSyncBlock() { this->unlock(); }
};

Now we are almost at the finish line. We just need to inherit this class in order to have the needed facilities for an object that needs synchronization. However, using this interface directly is not the most comfortable and safe. If we would like to have a Java-like “synchronized” statement we have to call for additional help. Fortunately, we have our not so well respected C macro language coming to rescue us as we can use it to make some pseudo-language extensions. The simplest way to define our new statement is using the following line:

#define synchronized(obj)  for(Lock obj##_lock = *obj; obj##_lock; obj##_lock.release())

From now, we can really use object synchronization in C++ as easy as in Java, we just need the following syntax in the method of our shared objects:

synchronized(this) {
    // some code that needs synchronization
}

Now it is clearly visible how handy the RAII pattern became in our case. Beside that it is now very straightforward to use this statement it provides additional benefits:

• It makes the code more readable and as a result it is easier to maintain.
• No need to call inconveniently named methods and use lock variables.
• The synchronized code has it’s own scope inside the code.
• It is exception-safe as the mutex is unlocked upon destruction.

Additionally, we can also take advantage of the inherent problem in C++ regarding to multiple inheritance. If we inherit our object from other two synchronized objects then using a simple type casting we can explicitly specify which ancestor we would like to synchronize in a particular block. Also, to ease this we can define our synchronization statement instead of the Java-like one using the following line:

#define synchronized(cls)  for(Lock obj##_lock = *static_cast<cls*>(this); obj##_lock; obj##_lock.release())

In this case we pass the class name instead of the object pointer this. Using this later construct we can easily specify the correct ancestor that we would like to synchronize in case when we deal with multiple inheritance situations. Personally I prefer the later syntax as it is much more customized for C++ use cases.

As from now we don’t need a direct interface for entering and exiting our synchronization block we can simplify our synchronizable interface to the following chunk:

class Synchronizable: protected Mutex {
};

This is enough from now to provide the facilities needed for a synchronization block but still complies to the requirement that we would like to hide the synchronization primitive related details.

Beside this, Jörg came up with the idea today to replace the for loop in our macro with a single if statement. This seems reasonable as we don’t have to sacrifice any scoping and safety related benefits of our framework. This simplifies our lock class to the following:

class Lock {
public:
    Lock(Mutex& mutex) : _mutex(mutex) { _mutex.lock(); }
    ~Lock() { _mutex.unlock(); }
    bool operator() const { return true; }
private:
    Mutex& _mutex;
};

This definition of the lock class is satisfactory if we redefine our synchronized macro to use an if statement instead:

/* Java-like synchronized statement */
#define synchronized(obj)  if (Lock obj##_lock = *obj)
/* alternative synchronized statement to support multiple inheritance */
#define synchronized(cls)  if (Lock obj##_lock = *static_cast<cls*>(this))

Thanks to the useful comments we even managed to further optimize and minimize the support code needed for our new pseudo-language extension.

Conclusion

We have seen an example how one can implement an easy to use synchronizable interface for C++. Also, we’ve provided a concrete implementation that is based on OpenMP. This library is still far from an API that provides all the necessary constructs that one needs for using parallel programming in their C++ projects, however we made our first step and I will recap on the subject in subsequent articles to further extend this framework.

Credits go to Achilleas Margaritis whose article inspired me to write mine and to Jörg for the useful improvement ideas.

Full source code

Language: C++
Platform: cross-platform
Dependency: OpenMP
Download link: omp_sync.h
Comments: In order to use it as it is, you will need a C++ compiler supporting OpenMP like GCC 4.2 or Visual C++ 2008.