Currently there are several ways to feed data to the GPU no matter of what API we use and what type of application we develop. In case of OpenGL we have uniform buffers, texture buffers, texture images, etc. The same is true for OpenCL and other compute APIs that even provide more fine-grained memory management taking advantage of the local data store (LDS) available on today’s hardware. In this article I’ll present the memory access performance characteristics of AMD’s Evergreen-class GPUs focusing on what this all means from OpenGL point of view. While most of the data is about the HD5870, the general principles and relative performance characteristics are valid for other GPUs, including ones from other vendors.

Introduction

Traditional CPU based applications don’t have to worry too much about where they put their data as they have a simple set of possibilities: registers and global memory (accessed through a series of linear caches called L1, L2 and on newer architectures also L3). While this and its details can be already quite cumbersome to utilize efficiently, GPU based algorithms need even more investigation as their architecture is based on a more complex multi-level memory design.

Typical questions an OpenGL graphics developer could ask nowadays are:

• Where should I put my per-object data?
• From where should I source animation data?
• Should I use uniform buffers, texture buffers or vertex buffers for my per-instance data?
• What does it mean from performance point of view if I use read-write buffers or textures?

Of course, the list could continue and answering the individual questions is not easy and often requires performance measurements to prove our suspicions. Instead of trying to answer all these questions it is easier to take a look at the actual hardware performance characteristics and solve the individual issues based on that.

I’ve already touched the topic in the past with the article Uniform Buffers VS Texture Buffers where I’ve presented the key differences between the two data access method and a few examples when to use one or the other. In this article I’ll go further and try to provide more accurate data about how various memory access methods perform in practice.

Earlier there were little to no detailed information about the actual performance of API level memory access methods but fortunately the increasing popularity of OpenCL made vendors to provide more technical details about the architecture and performance of their products to enable software developers to fully leverage the power of today’s GPUs. While these documents focus on OpenCL or other compute APIs, most of the data applies indirectly to OpenGL as well.

The Evergreen architecture

In order to be able to provide some actual performance data, I’ve selected as reference AMD’s Evergreen architecture and the Radeon HD5870 as the target hardware. Note that most of the presented details roughly apply to all other modern GPUs, including NVIDIA’s Fermi architecture. Each time there is a clear difference between the two, I’ll try to point it out. However, I cannot be 100% sure what are these differences as ATI’s OpenCL programming guide is somewhat more talkative about actual performance details than that of NVIDIA’s OpenCL programming guide.

OpenCL Platform Model

From OpenCL platform model’s point of view the Radeon HD5870 is structured in the following way:

• Total of 20 compute units.
• Each compute unit consists of 16 stream cores.
• Each stream core consists of 5 processing elements (4 traditional, 1 transcendental).

This sums up to a total of 1600 processing elements on the Radeon HD5870.

The basic OpenCL architecture applies in the same way to NVIDIA GPUs, however, there is are differences between AMD’s and NVIDIA’s GPU architecture. AMD uses a special super-scalar architecture since their HD2000 series that allows them to execute 5 separate instructions in each core.

ATI super-scalar architecture consisting of one transcendental unit (left), four traditional units and a dedicated branch execution unit (right).

What this already reveals us from OpenGL point of view is that AMD’s architecture groups together 16 stream cores so fragment shaders are most probably running on 4×4 tiles of fragments in sync. As an example, it is important to note this in case we use heavy dynamic branching in shaders as we should be aware of that in case the branch selection is not coherent for the specified fragment neighborhood, performance can drop due to the fact that hardware masks out those processing elements that did not select the appropriate branch.

Also, it is important to note that usually one out of four or five processing elements (depending on hardware generation and vendor) are capable of executing transcendental instructions such as logarithm, exponential or trigonometric functions.

Memory capacity and performance

AMD is very clear about the memory capacity and performance details in their OpenCL programming guide. The figure below showcases these hardware characteristics of the Radeon HD5870:

GPRs – General Purpose Registers
LDS – Local Data Store
Direct-addressed constant – a constant accessed using a constant address.
Same-indexed constant – a varying-indexed constant where each processing element accesses the same index.
Varying-indexed constant – a varying-indexed constant where the processing elements access different indices.

Of course, consider this data for fetches that are properly aligned. In case of unaligned data access the actual throughput can be much lower. In order to be able to reach the peak bandwidth we have to align our data usually to multiples of 4, 8 or 16 bytes (depending on actual hardware).

As it can be seen, constant storage can also fall into three different access performance categories so do buffers and images. While actual numbers differ on various platforms, the guidelines apply to most of modern GPUs: use a particular addressing method wisely and take in consideration access locality in order to get optimum performance.

These numbers are no different in case of OpenGL terminology either, just replace the word “constant” with uniform buffers and think about images and global data as texture images or buffer objects. The only exception is that there is no direct alternative for local memory in OpenGL.

An additional thing to consider since Shader Model 5.0 hardware is read-write images and buffers. AMD refers to the two memory access method as FastPath and CompletePath. This means that in case of read-only textures or buffers the GPU uses the FastPath that is able to take full advantage of the L2 cache while read-write textures and buffers usually use the so called CompletePath that sacrifices the advantages of the L2 cache to enable the use of atomic operations on global memory objects. This, of course, has a quite huge performance effect reducing the throughput of the GPU about five times on the Radeon HD5870:

Summary

Well, now we’ve seen that how various OpenCL memory types perform in reality, let’s see how all these information translate to the OpenGL world. Here are my top-10 recommendations about when and how to use the various data acquiring possibilities present in modern OpenGL:

1. Align your data to multiples of 16 bytes and fetch them accordingly.
2. Use direct-addressing of data in uniform buffers and try to avoid indexing into uniform buffers.
3. If you must use indexing into uniform buffers, make sure that the indices are coherent across processing elements working in sync.
4. If you heavily use indexed data consider using texture buffers instead of uniform buffers to take advantage of the L1 and L2 cache.
5. Texture and buffer caches are linear so consider this when planning you access patterns.
6. Bind textures and buffers for read-write mode only when it is really necessary, use regular texture binding otherwise to ensure optimum performance.
7. A single atomic buffer operation forces the shader to use the slow path so use atomic operations wisely.
8. Do not use atomic buffer operations to implement atomic counters, use built-in hardware atomic counters instead as they are much faster.
9. Consider using dynamic branching to avoid costly memory operations as often as possible.
10. Try to make your branch selection coherent across processing elements working in sync (e.g. 4×4 fragment tile in case of a fragment shader).

Relative performance characteristics of memory access methods (higher is better).

Note: This article may contain inaccurate data and some advices may not apply to other hardware platforms. I’ve made this article with the hope that it may prove useful for some developers out there. For accurate details or more information, please contact your hardware vendor.

Previously I talked about how one can easily take advantage of multiprocessing using OpenMP. Even if the C pragmas introduced by the parallel programming API standard is very straightforward for simple programs, it simply doesn’t fit nicely in a complex C++ application that is built from the ground with the OOP in mind. To smoothly introduce OpenMP into such projects one need higher level constructs that hide the actual implementation details. This is the first article of a series that will try to provide reference implementations of such an abstraction. First, we will start with synchronizable primitives that try to reflect the functionality provided by the “synchronized” statement of Java.

This article is highly inspired by an article written by Achilleas Margaritis and is mostly equivalent with his thoughts. My article tries to provide a portable reference implementation of a slightly modified version of the trick presented by Margaritis that uses OpenMP as the multiprocessing API back-end.

Motivation

According to the OO paradigm, classes and consequently objects provide an abstract interface to the underlying internal data or services of the modeled entity or entity class. When it comes to parallel programing we should provide facilities to enable concurrent access to shared resources that are in this case objects. Using plain OpenMP can be satisfactory, however when used extensively the OpenMP pragmas and API function calls introduced can greatly affect the readability and the maintainability of the code. Nevertheless, there can be platforms that use other APIs for handling race conditions. It is obvious that we need to encapsulate these facilities and provide an abstract tool-set instead.

Implementation

The very first building block of such a framework can be a mutex class that provides mutually exclusive access to certain resources. In the world of OpenMP this should look like something similar to the following:

class Mutex {
public:
    Mutex() { omp_init_lock(&_mutex); }
    ~Mutex() { omp_destroy_lock(&_mutex); }
    void lock() { omp_set_lock(&_mutex); }
    void unlock() { omp_unset_lock(&_mutex); }
private:
    omp_lock_t _mutex;
};

This seems already enough for us to make our Java-like “synchronized” statement, however we would like to create a framework that makes usage as easy and safe as possible. In order to get closer to this goal we apply the RAII (Resource Acquisition Is Initialization) design pattern to create our lock class:

class Lock {
public:
    Lock(Mutex& mutex) : _mutex(mutex), _release(false) { _mutex.lock(); }
    ~Lock() { _mutex.unlock(); }
    bool operator() const { return !_release; }
    void release() { _release = true; }
private:
    Mutex& _mutex;
    bool _release;
};

Our goal is to provide an inheritable interface for such objects that needs synchronization. However, this step has to involve severe considerations regarding to the provided interface as we explicitly need to conform to the following requirements:

• The interface shall not expose the interface of the underlying synchronization primitive, in our case the mutex class methods.
• The interface shall be available only to the synchronizable objects but not for the external world as we would like to not just hide the implementation details of our abstract entity but also prevent the users to synchronize our objects as it should be the responsibility of the object itself.
• The interface shall expose methods which are less prone to name collision, for convenience.

If we take care of the presented conventions we end up with an interface similar to the following:

class Synchronizable: protected Mutex {
protected:
	void enterSyncBlock() { this->lock(); }
	void exitSyncBlock() { this->unlock(); }
};

Now we are almost at the finish line. We just need to inherit this class in order to have the needed facilities for an object that needs synchronization. However, using this interface directly is not the most comfortable and safe. If we would like to have a Java-like “synchronized” statement we have to call for additional help. Fortunately, we have our not so well respected C macro language coming to rescue us as we can use it to make some pseudo-language extensions. The simplest way to define our new statement is using the following line:

#define synchronized(obj)  for(Lock obj##_lock = *obj; obj##_lock; obj##_lock.release())

From now, we can really use object synchronization in C++ as easy as in Java, we just need the following syntax in the method of our shared objects:

synchronized(this) {
    // some code that needs synchronization
}

Now it is clearly visible how handy the RAII pattern became in our case. Beside that it is now very straightforward to use this statement it provides additional benefits:

• It makes the code more readable and as a result it is easier to maintain.
• No need to call inconveniently named methods and use lock variables.
• The synchronized code has it’s own scope inside the code.
• It is exception-safe as the mutex is unlocked upon destruction.

Additionally, we can also take advantage of the inherent problem in C++ regarding to multiple inheritance. If we inherit our object from other two synchronized objects then using a simple type casting we can explicitly specify which ancestor we would like to synchronize in a particular block. Also, to ease this we can define our synchronization statement instead of the Java-like one using the following line:

#define synchronized(cls)  for(Lock obj##_lock = *static_cast<cls*>(this); obj##_lock; obj##_lock.release())

In this case we pass the class name instead of the object pointer this. Using this later construct we can easily specify the correct ancestor that we would like to synchronize in case when we deal with multiple inheritance situations. Personally I prefer the later syntax as it is much more customized for C++ use cases.

As from now we don’t need a direct interface for entering and exiting our synchronization block we can simplify our synchronizable interface to the following chunk:

class Synchronizable: protected Mutex {
};

This is enough from now to provide the facilities needed for a synchronization block but still complies to the requirement that we would like to hide the synchronization primitive related details.

Beside this, Jörg came up with the idea today to replace the for loop in our macro with a single if statement. This seems reasonable as we don’t have to sacrifice any scoping and safety related benefits of our framework. This simplifies our lock class to the following:

class Lock {
public:
    Lock(Mutex& mutex) : _mutex(mutex) { _mutex.lock(); }
    ~Lock() { _mutex.unlock(); }
    bool operator() const { return true; }
private:
    Mutex& _mutex;
};

This definition of the lock class is satisfactory if we redefine our synchronized macro to use an if statement instead:

/* Java-like synchronized statement */
#define synchronized(obj)  if (Lock obj##_lock = *obj)
/* alternative synchronized statement to support multiple inheritance */
#define synchronized(cls)  if (Lock obj##_lock = *static_cast<cls*>(this))

Thanks to the useful comments we even managed to further optimize and minimize the support code needed for our new pseudo-language extension.

Conclusion

We have seen an example how one can implement an easy to use synchronizable interface for C++. Also, we’ve provided a concrete implementation that is based on OpenMP. This library is still far from an API that provides all the necessary constructs that one needs for using parallel programming in their C++ projects, however we made our first step and I will recap on the subject in subsequent articles to further extend this framework.

Credits go to Achilleas Margaritis whose article inspired me to write mine and to Jörg for the useful improvement ideas.

Full source code

Language: C++
Platform: cross-platform
Dependency: OpenMP
Download link: omp_sync.h
Comments: In order to use it as it is, you will need a C++ compiler supporting OpenMP like GCC 4.2 or Visual C++ 2008.

Multiprocessing has been there for decades as a premium feature for enterprise applications but adopting this technology still brings huge burden to software companies that still maintain and develop legacy code. Nowadays, as most commodity hardware already have highly parallelized architectures, a modern application is almost unimaginable without proper multi-threading capabilities even if we talk about text editor or a multimedia application. The transition from traditional software development to multiprocessing is not an easy and painless task. Fortunately we have such tools in our hand like OpenMP.

Currently the biggest hit is OpenCL as it seems to be the ultimate solution to harness the power of highly parallel architectures like multi-core CPUs, DSPs and probably most important is that it can leverage the huge raw computational capabilities of GPUs. However it is one of the most important standard that came out lately, it is not the answer for all questions. For those who would like to converge their legacy code with multiprocessing technology maybe it’s a better advice to look around for other solutions.

My intension was not related to this when I started to search around for a multiprocessing framework. I just wanted to find something that provides an easy to use interface to introduce multi-threading and the needed shared memory semantics into my hobby projects. This is how I found OpenMP.

What is OpenMP?

Basically, OpenMP is an API specification for parallel programming that is intended to extend the most preferred programming languages used for computationally heavy and scientific calculations with a tool set that enables cross-platform multi-threading support tightly integrated into the language itself. Namely, OpenMP adds shared memory parallel programming capabilities to the C, C++ and Fortran languages.

While OpenMP is limited to these particular programming languages, it is truly an open and multi-platform API that is very well supported by different compilers (at least as far as I can tell). The standard itself is developed and maintained in a similar fashion like OpenGL as it has it’s own Architecture Review Board with representatives from all major hardware and software vendors like AMD, HP, IBM, Intel, Sun Microsystems, Microsoft and others.

The specification itself is maintained in two different versions: one for C/C++ and another for Fortran. As I was never involved in development with Fortran, I dug deeper only in the C/C++ specific details, however the facilities provided by the API are basically the same for Fortran as well.

The language extensions are introduced using OpenMP specific pragmas and a run-time library. At first sight this does not seem to be the most elegant solution but this fits very well into all versions of the programming language specifications so there are no further interworking issues and the OpenMP standard can be maintained totally separated from the underlying language itself. Looking at the evolution process of the C and the C++ programming languages this makes sense by the way.

Say Hello World to parallel programming

I think the best way to show the power and simplicity of OpenMP is to show a basic example on how easy is to add parallel computing capabilities even to the most straightforward algorithms:

void quicksort(int *a, int lo, int hi) {
    int i=lo, j=hi, h;
    int x=a[(lo+hi)/2];

    do {
        while (a[i] < x) i++;
        while (a[j] > x) j--;
        if (i <= j) {
            h=a[i]; a[i]=a[j]; a[j]=h;
            i++; j--;
        }
    } while (i <= j);

    #pragma omp parallel sections
    {
        #pragma omp section
        if (lo < j) quicksort(a, lo, j);
        #pragma omp section
        if (i < hi) quicksort(a, i, hi);
    }
}

This is the quick sort algorithm in OpenMP fashion. As you may already observed this function is not really different from the original sequential version of the famous sorting technique. The only added content is the presence of the three OpenMP specific pragmas and an additional block.

I will now explain how we exploited parallel programming with just these few added lines but I don’t want to go into details as it is always better to read the specification itself before starting to heavily use OpenMP. First, we’ve created “parallel sections” which means that we expressed our intension that we would like to separate the tasks in the next code block between multiple threads. Next we’ve specified the actual “sections” that one thread should execute.

This way each time we’ve split up the array in two pieces we sort the separate regions using separate threads. Of course, for a very huge this would not mean that the number of threads will exponentially grow as it will be saturated at some point. However, this is just one parameter that is fully controlled by the programmer.

Parallelize loops with minimal effort

Many times happens that the performance bottleneck is inside a for loop that moves or does calculations on huge data arrays. One example is an algorithm that interpolates two float arrays to another one. This can be of course parallelized using the “sections” semantics presented earlier, however it would need modification to the original algorithm and after this it would not clearly reflect the purpose of that anymore. OpenMP supports also such cases very elegantly:

#pragma omp parallel for
for(int i = 1; i < size; ++i)
    C[i] = A[i] * alpha + B[i] * (1 - alpha);

Notice that there are no loop-carried dependencies. This means that one iteration of the loop does not depend upon the result of another iteration of the loop. This makes it appropriate for parallelization. Only by adding a single pragma the time needed to execute this loop may scale down perfectly on multi-core systems.

For more control over how many threads will likely to carry out the results of this for loop one can specify the exact number of threads that should be used for the operation by adding another option to the pragma:

#pragma omp parallel for num_threads(4)

Of course there are plenty of other configuration possibilities that control how the parallelized code will actually execute but, again, this article is not meant to be a through guide on the usage of OpenMP instead it’s just a foretaste to raise interest for getting more details about this prominent tool.

More than just threads

We’ve seen so far that OpenMP enables the introduction of basic work sharing support for an already existing project with minimal effort. However, OpenMP is more than just another way to execute separate threads, it also provide very easy to use facilities for synchronization and shared data handling that can be the building blocks of any multiprocessing application including, but not limited to the following features:

• Explicitly scoped variables to indicate shared and thread private storage
• Atomic operations and critical sections
• Execution barriers for fine grained synchronization

The best thing in these is that you just specify the appropriate pragmas for the affected statements or variables and the rest is carried out by OpenMP. For more information on the usage of these please refer to the OpenMP specification.

Compiler support

One of the best things in OpenMP is that it is well supported by most of the major C/C++ compiler vendors:

• GCC version 4.3.2 and later (enabled with the -fopenmp compiler switch)
• Visual C++ 2008 and later (enabled with the /openmp compiler switch)
• Intel C/C++ compiler version 10.1 and later (using -Qopenmp on Windows or -openmp on Linux or MacOSX)

For a complete list of supported compiler please refer to the official site of OpenMP.

Another advantage that raises from the fact how the actual language integration of OpenMP has been designed is that it usually gracefully degrades on compilers without support for OpenMP as the pragmas can be silently ignored. I intentionally used the word “usually” as in case that the business logic of the application is consciously using the multi-threaded semantics then it wouldn’t execute in the exact same way with or without OpenMP. However, the responsibility to monitor such situations is up to the developer.

Conclusion

My personal opinion about OpenMP that it best suites those situations when a gradual transition is needed for legacy code towards a parallelized system or when one searches for the easiest possible way to take advantage multiprocessing capable environments. Still, OpenMP is suitable to fulfill almost all the tasks that are needed to implement completely new applications with parallel programming in mind and so I recommend it to everybody even for general use.