Frei-Chen edge detector

In this article, I would like to present you an edge detection algorithm that shares similar performance characteristics like the well-known Sobel operator but provides slightly better edge detection and can be seamlessly extended with little to no performance overhead to also detect corners alongside with edges. The algorithm works on a 3×3 texel footprint similarly like the Sobel filter but applies a total of nine convolution masks over the image that can be used for either edge or corner detection. The article presents the mathematical background that is needed to implement the edge detector and provides a reference implementation written in C/C++ using OpenGL that showcases both the Frei-Chen and the Sobel edge detection filter applied to the same image.

I met with the algorithm during my computer graphics studies when one of my homeworks was to implement the Frei-Chen edge detector. As I already mentioned it in an earlier post, I am willing to provide source code for more basic graphics algorithms after seeing the success of my former post about the Gaussian blur filter. This one is a very similarly basic article, taking in consideration it shows only how to apply a particular convolution filter based algorithm on a still image, while the possibilities this edge detection algorithm brings is a more complex topic that is out of the scope of this article.

As the provided reference implementation also showcases applying the Sobel operator on an image, I would like to present that first and then continue with the presentation of the Frei-Chen masking set. Those who are already well familiar with edge detection and the Sobel operator can skip the following two sections.

Edge detection

Before getting deep into how to implement edge detectors, let’s first talk about what is an edge detector and why we need it.

In general, edge detection is one of the most fundamental image processing tools, particularly used in the areas of feature detection and feature extraction. The aim of the technique is to identify points of a digital image at which the intensity changes sharply. The reason of these intensity changes can be either discontinuities in depth, surface orientation, lighting condition changes and many other factors. In the ideal case, the result of applying an edge detector to an image leads us to a set of connected lines or curves that indicate the boundaries of objects.

Not going that far, what an edge detector gives us from the very beginning is a gray-scale image where each pixel intensity tries to approximate the likelihood of whether that pixel belongs to an object boundary. How well a particular algorithm can detect such pixels depends on many factors and usually it is better to try multiple edge detectors in order to choose one that fits most for the particular use case.

After we got this gray-scale image we usually have to define a threshold value that will be used as an acceptance criteria for edge pixels. If the intensity value previously calculated is above this threshold then we accept the pixel as an edge otherwise we don’t. This part is the so called binarization stage. Additionally, subsequent image processing algorithms can be used to further interpret the edge image.

In computer graphics, edge detection is usually used to implement various image decoration algorithms. Maybe the most popular applications of edge detectors nowadays are non-photorealistic rendering (NPR) and screen-space anti-aliasing techniques.

Sobel filter

The Sobel edge detection filter works on a 3×3 texel footprint and applies two convolution masks to the image that are intended to detect horizontal and vertical gradients of the image. The filter weights can be seen in on the figure below:

These masks are applied to the intensities gathered from the 3×3 footprint of the image and then are accumulated to produce the final gradient value in the following way:

The actual algorithm can be seen in the accompanying demo that provides a GLSL based implementation. The algorithm is defined to work on one channel image, however it can be easily extended to be applied either separately on a usual three-channel RGB image or by first calculating a gray-scale value based on the color component values. The former is more computationally intensive but usually provides better results by defining the threshold criteria in a way that a pixel is accepted as boundary point if the gradient value is larger than the threshold for either of the color channels. The reference implementation, however is based on the later approach for the sake of simplicity so for each pixel first an intensity value is calculated simply by taking the length of the vector comprised of the RGB components.

Frei-Chen filter

The Frei-Chen edge detector also works on a 3×3 texel footprint but applies a total of nine convolution masks to the image. Frei-Chen masks are unique masks, which contain all of the basis vectors. This implies that a 3×3 image area is represented with the weighted sum of nine Frei-Chen masks that can be seen below:

The first four Frei-Chen masks above are used for edges, the next four are used for lines and the last mask is used to compute averages. For edge detection, appropriate masks are chosen and the image is projected onto it. The projection equation is given below:

When we are using the Frei-Chen masks for edge detection we are searching for the cosine defined above and we use the first four masks as the elements of importance so the first sum above goes from one to four.

The application of a threshold and applying the filter to multi-channel images works exactly the same way like in case of the Sobel filter. Similarly, the reference implementation applies the filter on the image as it would be a single-channel image by first calculating the intensity value for each texel in the same fashion like with the previously presented filter.

Comparison

Based on my experience, the Frei-Chen edge detector looks better than the Sobel filter as it is less sensitive to noise and is able to detect edges that have small gradients and thus are not found by the basic Sobel filter. For a comparison, you can check the figure below:

Comparison of edge detectors: original image (left), Sobel filter (middle), Frei-Chen filter (right).

The reason why the Frei-Chen edge detector seems to work better is because its construction includes a normalization factor as well as other factors that are meant to exclude all other features except edges. A normalization factor can be also added to the Sobel filter by having a third mask that is equivalent with the ninth Frei-Chen mask and is used to normalize the gradients. This could help in reducing the number of undetected edges and the amount of noise that arises from the fact that the Sobel filter calculates absolute gradients rather than relative ones.

From performance point of view, the Frei-Chen edge detector is much more heavyweight as it uses nine masks instead of two, however, in practice, the performance difference between the two is much less taking in consideration that both use the same sized texel footprint and the computational performance of today’s GPUs is usually much higher than their texture fetching performance.

Conclusion

We managed to present an alternative algorithm for the Sobel filter in the form of the Frei-Chen edge detector that, even though having little impact on the performance compared to the Sobel operator, provides better edge detection quality. Having little to no difference in the way how the input data has to be organized and how the result is output, the Frei-Chen edge detector can be easily used as a drop-in replacement for implementations that used the Sobel filter before.

Source code and Win32 binary can be acquired in the downloads section.

I would like to encourage those who read this article to add the Frei-Chen edge detector into their software for making a comparison about whether it yields to better results than the Sobel filter for applications that rely on the output of the edge detection filter. I would be interested how the filter works in real-life computer graphics scenarios.

Thanks in advance and hope you enjoyed the article!

OpenGL 4.0 - Mountains demo

Dynamic geometry level-of-detail (LOD) algorithms are very popular and powerful algorithms that provide a great level of rendering performance optimization while preserving detail by using less detailed geometry for objects that are far away, too small or otherwise less significant in the quality of the final rendering. Many of these are used since the very beginning of computer graphics technologies and are present in some form in current CAD softwares, video games and other graphics applications. While determining the appropriate geometry LOD was previously the task of the CPU, with todays hardware it is possible to also offload this to the GPU which excels at handling large amount of objects in parallel.

Introduction

With the advent of Shader Model 5.0 GPUs and the appearance of programmable tessellation hardware it may seem like the geometry LOD problem is solved once and for all. However, in many cases it is simply not enough as for far away objects even a patch pass-through tessellation shader already produces too much geometry than the added detail worths. As a result, classic geometry LOD algorithms are still a good-to-have feature in the tool-box of the developer. Not to mention that all vendors recommend disabling tessellation shaders at all if we don’t need any geometry amplification as even a pass-through tessellation shader does have its payload.

This means that there has to be still a conventional rendering path for geometries that should not be tessellated. Then why not to try offloading the geometry LOD determination to the GPU if possible?

This article presents a technique that was already presented by AMD’s March of the Froblins demo and by NVIDIA’s Skinned Instancing demo and allows GPU based dynamic geometry LOD determination using a geometry shader that selects the most appropriate LOD from a group of geometry LODs based on the object’s distance from camera. While this article and the reference implementation (OpenGL 4.0 – Mountains demo) presents the application of the technique only for instanced geometry, the same method can be easily extended to support heterogeneous objects by taking advantage of the latest functionalities introduced in OpenGL 4.

The algorithm

The technique is based on the geometry shader’s ability to emit or deny the emission of primitives into a transform feedback buffer as done in the mentioned DX based implementations. One major improvement compared to earlier approaches is that the LOD determination is done in a single pass rather than requiring a separate pass for each geometry LOD. Additionally, this LOD determination pass can be also merged together with other visibility determination passes like Instance Cloud Reduction or Hierarchical-Z map based occlusion culling as it is done in the reference implementation. This was made possible thanks to the latest transform feedback capabilities introduced in OpenGL 4.0 (see the extension ARB_transform_feedback3) that enables the geometry shader to output data to separate primitive streams.

Flow-chart presenting the culling and dynamic LOD algorithms used in AMD's March of the Froblins demo. The implementation needs five passes for culling and separating three detail levels and performs two asynchronous queries meanwhile. Requires OpenGL 3 compliant hardware.

Flow-chart presenting the culling and dynamic LOD algorithm used in our Mountains demo. The implementation requires only one pass for culling and separating three detail levels without the need to use asynchronous queries. Requires OpenGL 4 compliant hardware.

The algorithm itself is very simple and straightforward. For each object instance determine the appropriate geometry LOD based on it’s distance from the camera and the LOD distances passed as uniform to the shader. After this, output the instance’s data to the output stream ID that corresponds to the determined LOD’s index. Here you can see a GLSL implementation of the algorithm:

#version 400 core

uniform mat4 ModelViewMatrix;
uniform vec2 LodDistance;

layout(points) in;
layout(points, max_vertices = 1) out;

in vec3 InstancePosition[1];

layout(stream=0) out vec3 InstPosLOD0;
layout(stream=1) out vec3 InstPosLOD1;
layout(stream=2) out vec3 InstPosLOD2;

void main() {
  float distance = length(ModelViewMatrix * vec4(InstancePosition[0], 1.0));
  if ( distance < LodDistance.x ) {
    InstPosLOD0 = InstancePosition[0];
    EmitStreamVertex(0);
  } else
  if ( distance < LodDistance.y ) {
    InstPosLOD1 = InstancePosition[0];
    EmitStreamVertex(1);
  } else {
    InstPosLOD2 = InstancePosition[0];
    EmitStreamVertex(2);
  }
}

Additionally, the geometry LOD determination pass has to be executed with primitive queries enabled for all the relevant output streams to acquire the number of instances for each geometry LOD index:

for (int i=0; i<NUM_LOD; i++)
  glBeginQueryIndexed(GL_PRIMITIVES_GENERATED, i, lodQuery[i]);

glBeginTransformFeedback(GL_POINTS);
  glDrawArrays(GL_POINTS, 0, instanceCount);
glEndTransformFeedback();

for (int i=0; i<NUM_LOD; i++)
  glEndQueryIndexed(GL_PRIMITIVES_GENERATED, i);

Finally, the only thing what is left is to issue an instanced draw call for each geometry LOD index to draw all the instances:

for (int i=0; i<NUM_LOD; i++) {
  glGetQueryObjectiv(lodQuery[i], GL_QUERY_RESULT, instanceCountLOD[i]);
  if ( instanceCountLOD[i] > 0 )
    glDrawElementsInstanced(..., instanceCountLOD[i]);
}

That’s all, and what you get as a result is a fully GPU based geometry LOD selection algorithm.

The Mountains demo

The reference implementation provided as part of the OpenGL 4.0 – Mountains demo that is available with full source code and Windows executable in the downloads section. The demo application implements the same visibility determination algorithms that were presented in the SIGGRAPH 2008 Course Notes besides the dynamic geometry LOD algorithm presented here in a single pass.

Dynamic LOD can be enabled in the demo by using the F3 key. After enabled, the demo separates the various geometry detail levels according to the LOD distances configured. As it can be seen, there is almost no visible difference between the scene rendered with dynamic geometry LOD enabled and disabled. Also, by setting the LOD distances appropriately, the algorithm provides seamless transition between subsequent geometry detail levels as the camera is moved.

Close-up view of distant objects to compare the image quality without (left) and with (right) dynamic LOD.

Geometry LOD visualization: LOD 0 (red), LOD 1 (green), LOD 2 (blue).

When dyamic LOD is enabled, the demo also makes it possible to visualize the various geometry detail levels by pressing the F4 key. The highest detail LOD is marked with red, mid-level with green and the lowest detail geometries are marked as blue. It can be seen that as the camera moves the renderer automatically adjusts the detail of each individual instance.

Besides maintaining a constant quality without the viewer to observe any transitions between the various detail levels, the algorithm provides a huge performance gain in case of complex geometries as it can be seen on the figure below:

Performance comparison of the demo in frames per second on a Radeon HD5770 (higher is better): no culling (bottom), instance cloud reduction (middle), ICR + Hi-Z map based occlusion culling (top), no geometry LOD (blue), dynamic geometry LOD (red).

Conclusion

We’ve seen how straightforward is to implement GPU based dynamic geometry LOD determination using geometry shaders on OpenGL 4.0 compliant hardware providing also a reference implementation that uses the algorithm to efficiently determine detail levels for large number of instanced geometry. We also briefly mentioned that the algorithm can be extended to handle arbitrary object sets. We discussed about a possible OpenGL 3 based implementation but we did not provide one as it requires several rendering passes to perform all the operations that can be implemented in a single pass on Shader Model 5.0 hardware.

Even though the algorithm is already extremely efficient, it still involves the use of asynchronous primitive queries that may induce some latency. Of course, this latency can be easily hidden by performing other operations on the CPU/GPU until the results are available.

Furthermore, taking full advantage of Shader Model 5.0 GPUs it would be possible to eliminate the need of asynchronous queries by using atomic counters and indirect rendering, however the core OpenGL specification does not expose yet such functionality so this improvement is left for a future release of the demo.

Classic dynamic geometry LOD algorithms are still first class citizens of every rendering system and even though the introduction of hardware tessellation somewhat subsumes the need for these classic techniques, practice shows that the best way to implement a full-fledged dynamic LOD system is by using geometry LOD selection and tessellation together rather that one instead of the other.

OpenGL 4.0 - Mountains demo

Hierarchical-Z is a well known and standard feature of modern GPUs that allows them to speed up depth testing by rejecting large group of incoming fragments using a reduced and compressed version of the depth buffer that resides in on-chip memory. The technique presented in this article uses the same basic idea to allow batched occlusion culling for large amount of individual objects using a geometry shader without the need of any CPU intervention that is unavoidable using traditional occlusion queries. The article also provides a reference implementation in the form of the OpenGL 4.0 Mountains demo that uses the technique for culling thousands of object instances.

Introduction

Occlusion culling is a visibility determination algorithm that is used to identify those objects that did reside in the view volume but still aren’t visible on the screen due to occlusion. That means they are hidden by such objects that reside closer to the camera.

For several generations now GPUs allow hardware accelerated methods to perform occlusion culling in the form of occlusion queries. OpenGL provides the functionality via the extension ARB_occlusion_query. Occlusion queries are very simple: when you draw an object with occlusion query enabled the query returns the number of samples that passed the depth test (or simply return true or false based on whether any samples of the objects passed the depth test or not as it is provided by the OpenGL extension ARB_occlusion_query2).

So actually performing occlusion culling using occlusion queries means simply the following:

1. Draw the object while occlusion query is enabled.
2. If the query result is that the object is visible then draw the object.

At first, this may sound stupid as you have to draw the object in order to tell whether it is visible or not. While in this form it really sounds silly, in practice occlusion query can save a lot of work for the GPU. Think about you have a complex object with several thousands of triangles. If you would like to determine the visibility of it using occlusion query you would simply render e.g. the bounding box of the object and if the bounding box is visible (occlusion query returns that some samples have passed) then it means the object itself is most probably visible. This way you can save the GPU from the unnecessary processing of large amount of geometry.

I have to mention here that I intentionally used the expression “most probably visible” as occlusion queries provide just a conservative estimate on whether the object is visible or not rather than an exact result. This is because the bounding box occupies a different (larger) portion of the screen than the original geometry. So what we expect from an occlusion culling algorithm is to give one of the following results: the object is not visible or the object is most probably visible. The bigger this probability is the better the occlusion culling effectiveness is.

While we would always want an occlusion culling algorithm to be as effective as possible usually we have to make a trade-off between effectiveness and efficiency. In the above example if we would like to have 100% effectiveness then we would have to draw the whole object and that would defeat most of the goals of occlusion culling. The algorithm presented in this article is somewhat even more conservative but enables the use of occlusion culling for much larger datasets.

Motivation

While hardware accelerated occlusion query is a powerful tool to use in visibility determination it puts a quite reasonable burden on the application to manage the occlusion queries and to draw the objects based on the results when they are available (taking in consideration the asynchronous nature of occlusion queries). The most naive use of occlusion queries would be to execute the query right before we have to draw the object. While this seems like a feasible idea, it does not perform well in practice as the CPU has to be stalled until the result of the query is available and that involves also empty cycles on the GPU as well thus results in unacceptable performance. In order to resolve this, the application has to fill the time between the query execution and the drawing of the object based on the query result. While there are techniques to accomplish this, it definitely comes at a cost as the implementation becomes more complex.

The aforementioned problem is somewhat resolved by using conditional rendering introduced in OpenGL 3 (NV_conditional_render extension). However, this extension does nothing just in case the results of the query are not available yet then we simply draw the object no matter if it is visible or not. This can avoid the stalling of the rendering pipeline and can be done in software if the extension is not available, however, it somewhat defeats the purpose of occlusion culling.

Another deficit when using occlusion queries is that there is still need for CPU intervention in order to make a decision about the visibility of the object. For today’s hardware where proper batching is one of the most crucial aspects of the renderer such an approach is rather ineffective.

The occlusion culling technique presented in this article solves both these issues by providing an implementation that is very simple to integrate into any renderer, does put little to no burden on the renderer and makes decision about the visibility of objects entirely on the GPU.

The algorithm

As in case of many other GPU based culling algorithm presented by me and others, the hierarchical-Z map based occlusion culling uses the geometry shader’s ability to deny the emission of primitives that are determined to be invisible on the final rendering. The shader will only emit data for those objects that are visible and this data is streamed out into a buffer object using transform feedback.

The algorithm itself is similar in spirit to the hierarchical Z testing that is implemented in modern GPUs. After rendering all the occluders in the scene, we construct a hierarchical depth image from the depth buffer which we will refer to as the Hi-Z map. This texture map is a mip-mapped, screen resolution image where each texel in mip level i contains the maximum depth of all corresponding texels in mip level i-1. This depth information can be collected during the main rendering pass for the occluding objects as we need a texture of the same resolution so we don’t need a separate depth pass. This can be simply accomplished using OpenGL framebuffer objects.

After the construction of the Hi-Z map, occlusion culling can be performed by comparing depth value of the object’s bounding volume and the depth information stored in the Hi-Z map. This is when the hierarchical mip-mapped structure of the Hi-Z map comes handy as we can do conservative depth comparisons with less texture fetches by sampling directly from a particular mip level.

This is why we constructed the Hi-Z map using a “store maximum depth” policy. This will work with a usual depth buffer setup where the depth comparison function is either GREATER or GEQUAL. For a reverse directed depth buffer the “store minimum depth” policy has to be used.

Hi-Z map construction

In case of single-sample rendering, one can use the Hi-Z map as the main depth buffer for rendering the scene. The technique extends also to multi-sampled rendering but in this case a separate full-screen quad pass is needed to calculate the maximum depth of each individual sample in the multi-sampled depth buffer and store it in the single-sampled Hi-Z map. This is possible since OpenGL 3.2 or using the extension ARB_texture_multisample. Besides this additional step, the algorithm remains the same.

The Hi-Z map can be constructed using OpenGL framebuffer objects by rendering a full-screen quad pass for each mip level where the previous mip level is bound as the input texture and the current mip level is bound as render target. As OpenGL does allow rendering from and to the same texture object as far as we don’t access the same mip level for both reading and writing, the algorithm simply looks like the following:

// bind depth texture
glBindTexture(GL_TEXTURE_2D, depthTexture);
// calculate the number of mipmap levels for NPOT texture
int numLevels = 1 + (int)floorf(log2f(fmaxf(SCREEN_WIDTH, SCREEN_HEIGHT)));
int currentWidth = SCREEN_WIDTH;
int currentHeight = SCREEN_HEIGHT;
for (int i=1; i<numLevels; i++) {
  // calculate next viewport size
  currentWidth /= 2;
  currentHeight /= 2;
  // ensure that the viewport size is always at least 1x1
  currentWidth = currentWidth > 0 ? currentWidth : 1;
  currentHeight = currentHeight > 0 ? currentHeight : 1;
  glViewport(0, 0, currentWidth, currentHeight);
  // bind next level for rendering but first restrict fetches only to previous level
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_BASE_LEVEL, i-1);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAX_LEVEL, i-1);
  glFramebufferTexture2D(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT,
                         GL_TEXTURE_2D, depthTexture, i);
  // draw full-screen quad
  ............
}
// reset mipmap level range for the depth image
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_BASE_LEVEL, 0);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAX_LEVEL, numLevels-1);

It is very important not to forget about the step when we ensure that the viewport size is always at least 1×1 as in case of non-power-of-two (NPOT) textures due to rounding problems. I forgot this first and I was wondering an hour why my last mip level didn’t get filled.

While one may wonder how this technique can be efficient after so many full-screen quad passes, it is in fact very efficient and it constructs the Hi-Z map on my Radeon HD5770 in less than 0.2 milliseconds. The measurement should be quite accurate as I’ve done it using OpenGL timer queries (see the extension ARB_timer_query).

The fragment shader used for the construction of the Hi-Z map is very straightforward except one thing. We use an NPOT depth texture due to the aspect ratio of the window and as NPOT textures use a “floor” convention to determine the size of subsequent mip levels (see the extension ARB_texture_non_power_of_two) we need predicated fetches as in case of reduction from odd-sized mip levels we should not forgot about the edge texels:

#version 400 core

uniform sampler2D LastMip;
uniform ivec2 LastMipSize;

in vec2 TexCoord;

void main(void)
{
  vec4 texels;
  texels.x = texture( LastMip, TexCoord ).x;
  texels.y = textureOffset( LastMip, TexCoord, ivec2(-1, 0) ).x;
  texels.z = textureOffset( LastMip, TexCoord, ivec2(-1,-1) ).x;
  texels.w = textureOffset( LastMip, TexCoord, ivec2( 0,-1) ).x;

  float maxZ = max( max( texels.x, texels.y ), max( texels.z, texels.w ) );

  vec3 extra;
  // if we are reducing an odd-width texture then fetch the edge texels
  if ( ( (LastMipSize.x & 1) != 0 ) && ( int(gl_FragCoord.x) == LastMipSize.x-3 ) ) {
    // if both edges are odd, fetch the top-left corner texel
    if ( ( (LastMipSize.y & 1) != 0 ) && ( int(gl_FragCoord.y) == LastMipSize.y-3 ) ) {
      extra.z = textureOffset( LastMip, TexCoord, ivec2( 1, 1) ).x;
      maxZ = max( maxZ, extra.z );
    }
    extra.x = textureOffset( LastMip, TexCoord, ivec2( 1, 0) ).x;
    extra.y = textureOffset( LastMip, TexCoord, ivec2( 1,-1) ).x;
    maxZ = max( maxZ, max( extra.x, extra.y ) );
  } else
  // if we are reducing an odd-height texture then fetch the edge texels
  if ( ( (LastMipSize.y & 1) != 0 ) && ( int(gl_FragCoord.y) == LastMipSize.y-3 ) ) {
    extra.x = textureOffset( LastMip, TexCoord, ivec2( 0, 1) ).x;
    extra.y = textureOffset( LastMip, TexCoord, ivec2(-1, 1) ).x;
    maxZ = max( maxZ, max( extra.x, extra.y ) );
  }

  gl_FragDepth = maxZ;
}

I was experimenting with using texture gather lookups to reduce the number of texture fetches from 4-to-7 fetches per fragment down to 1-to-3 fetches per fragment (see the extension ARB_texture_gather) it seems that texture gather works only if the image is linearly sampled and to avoid the additional burden involved by switching filtering state during rendering I stuck to simple texture lookups as using texture gather lookups did not show any visible effect on the construction time of the Hi-Z map.

Various mip levels of the Hi-Z map. The Hi-Z map size is 1024x768 and the displayed mip levels are: level 4 (left), level 5 (middle) and level 6 (right).

For debugging and demonstration purposes the Mountains demo has built-in function to display the content of the various mip levels of the Hi-Z map. This is available by pressing the F4 key while Hi-Z map based occlusion culling is enabled. The + and – keys can be used to switch between the mip levels.

In order to better visualize the depth information in the depth buffer I converted the non-linear depth values stored in the depth texture into linear depth values as presented in [GeeXLab] How to Visualize the Depth Buffer in GLSL.

Culling with the Hi-Z map

Once we have constructed the Hi-Z map, we can perform the actual occlusion culling by fetching the 2×2 texel neighborhood corresponding to the screen area occupied by the bounding volume of the object whose visibility has to be determined. In the demo I used bounding boxes but any other bounding volume can be used (e.g. a bounding sphere is usually accurate enough for this technique).

First, we have to calculate the clip space bounding rectangle of the bounding volume. In the bounding box case this is done by transforming the bounding box vertices into clip space and then calculate the minimum and maximum X and Y coordinates. This bounding rectangle will be used for two things: it defines the texture coordinates that we’ll have to use for the Hi-Z map lookup and it helps determining the appropriate LOD for the texture lookup.

In order to determine the texture LOD that we’ll have to fetch we have to calculate the screen space size of the bounding square corresponding to the clip space bounding rectangle determined previously. This can be simply done by calculating the width and height of the bounding rectangle in clip space and then transforming this into screen space:

float ViewSizeX = (BoundingRect[1].x-BoundingRect[0].x) * Transform.Viewport.y;
float ViewSizeY = (BoundingRect[1].y-BoundingRect[0].y) * Transform.Viewport.z;

After this, the texture LOD can be simply calculated using the following formula:

float LOD = ceil( log2( max( ViewSizeX, ViewSizeY ) / 2.0 ) );

Finally, as we have the texture coordinates (the vertices of the clip space bounding rectangle) and the texture LOD, we simply have to make four texture lookups into the Hi-Z map using these parameters, calculate the maximum of the four depth values returned and compare it to the depth value corresponding to the object (this is the object’s front-most point’s depth value that comes also from the clip space coordinates of the bounding box). If the object depth is greater than the reference depth the object is occluded and so it is culled by the geometry shader as usual.

One may ask why we use a 2×2 texel footprint for calculating the reference depth value why not just fetch the next mip level only once (as there we also get the maximum values of a 2×2 texel footprint due to the Hi-Z map construction method). That’s what I’ve also asked myself at first sight but quickly figured out the reason (see the figure below).

Comparison of number of fetches used for occlusion culling. Both figures show the magnified screen coverage of a single Hi-Z map texel at mip level N, texel coverage for mip level N-1 is in cyan and texel coverage for mip level N-2 is in blue. Object is show as red and yellow indicates the fetched texels.

In case of four texels not just the determination of the texture LOD is much easier but also it better encompasses the actual object bounding rectangle. In case of one texture fetch the computation of texture LOD is more complicated and expensive but the main problem is that a larger LOD has to be fetched and it is not always the LOD determined in the case of four fetches plus one. In the most extreme situation (if the bounding rectangle is right at the middle of the screen) it is possible that we have to fetch the largest LOD. This does not result in any false culling but it severely degrades the effectiveness of the culling.

Of course, it is possible to use more complex screen space bounding polygon as well as more fetches but those would increase the effectiveness of the culling much less than the additional burden and expensive operations worth.

Conclusion

We’ve seen how traditional hardware occlusion culling works by using occlusion queries. We also discussed that we sometimes need a better algorithm that does the occlusion culling for large amount of objects without CPU intervention.

The article also described a way to implement such an occlusion culling algorithm by using a hierarchical-Z map and geometry shaders. We’ve also managed to provide a reference implementation in the form of the demo called Mountains that can be downloaded with full source code in the downloads section.

The algorithm performs very well in practice on current hardware. The Hi-Z map construction takes less than 0.2 milliseconds and the actual culling comes at almost no cost for even thousands of objects. For more detail about performance comparison between rendering with and without hierarchical-Z map based occlusion culling read the article about the OpenGL 4.0 Mountains Demo.

While the demo uses the technique only for culling instances of the same object, the technique can be easily extended to work for heterogeneous set of objects as the actual culling algorithm works on a per-object basis and is completely indifferent regarding to the method used for rendering the actual geometry.

This technique can be thought of as the next step towards a completely GPU based visibility determination and scene management system.

Acknowledgements go to Jeremy Shopf, Joshua Barczak, Christopher Oat and Natalya Tatarchuk and their SIGGRAPH 2008 Course Notes that inspired this work.

OpenGL 4.0 - Mountains demo

OpenGL 3.0 capable GPUs introduced a level of processing power and programming flexibility that isn’t comparable with any earlier generations. After that, OpenGL 4.0 and the hardware supporting it even further pushed the limits of what previously seemed to be impossible. Thanks to these features nowadays more and more possibilities are available for the graphics developers to implement GPU based scene management and culling algorithms. The Mountains demo showcases some of these rendering techniques that, as far as I know, were never implemented so far using OpenGL. In this article I will present the key features of the demo that will be discussed in more detail in subsequent articles. Demo binaries with full source code are also published.

The demo itself is mainly inspired by the March of the Froblins demo released by AMD and the SIGGRAPH 2008 Course Notes by Jeremy Shopf, Joshua Barczak, Christopher Oat and Natalya Tatarchuk presenting the actual implementation in detail. That demo targeted the Radeon HD4800 series and presented several practical GPU based culling algorithms implemented using DirectX10. The Mountains demo implements these techniques in OpenGL and further improves the technique used in AMD’s demo by unleashing the new features introduced by Shader Model 5.0 hardware and OpenGL 4.0.

While this article briefly presents the demo and the used rendering techniques, the details of each individual technique will be presented in subsequent articles as the thorough examination of them needs a longer discussion that would render this article simply too long and overwhelming.

Introduction

The Mountains demo renders a tiled terrain block with thousands of high detail tree models (the full detail tree model is over five thousand triangles). Due to the view distance used in the demo is quite large, several tiles of the terrain block are potentially visible on the screen and this results in a huge explosion in the number of triangles the GPU has to render. Also, with traditional methods the rendering of the terrain blocks and the several thousand tree models would need loads of draw calls. In order to solve this problem, the demo renders the trees using geometry instancing to minimize the number of draw calls.

In a traditional rendering engine CPU based culling methods would be used. While that would even work in practice, it is more convenient to perform the culling on the GPU as every information needed to do it is available there. Nevertheless, culling is a typical algorithm that can easily take advantage of the highly parallel architecture of the GPU. Also, performing the culling on the CPU would make geometry instancing barely beneficial.

Another problem with a scene like this is that a simple per-object view frustum culling would not solve the problem completely as most of trees in the view frustum are not visible due that they are hidden by the terrain. In traditional OpenGL the way how to solve this problem would be the use of per-object occlusion queries and rendering of bounding volumes. While this may work in practice, it involves too much CPU intervention even if we take advantage of conditional rendering and nevertheless, this also breaks instancing.

These are the issues that motivated me in creating this demo and I established the following goals for the project:

View from above

• All the object-level information must stay on the GPU and the CPU should not make decisions on a per-object basis.
• The renderer should use as few draw calls as possible in order to solve the problem of visibility determination.
• Don’t draw anything that is not inside the view frustum or is occluded by terrain.

The result is a renderer that does little to no scene management on the CPU, instead uses the GPU for visibility determination that is, in most cases, able to reduce the scene’s geometric complexity from over 400 million triangles under one million triangles providing an interactive experience on a Radeon HD5770 with around 200 frames per second.

Implementation

The scene consists of a tiled terrain with over 130 thousands of triangles and more than 1400 tree instances each with almost 6 thousands of triangles. This sums up to 8 million triangles for a single tile block of terrain. As the view range is needed to be quite large we actually deal with a 7×7 tile of terrain that is dynamically placed in a way that the camera always resides in the middle block of the tile. What all this means that even though we dynamically generate the scenery around the camera, we still have to deal with a scene consisting of over 400 million triangles. This is simply too much for the GPU to deal with.

The first step done in order to reduce the geometric complexity of the scene is done on the CPU by performing a view frustum culling on a per-terrain-block basis. This will limit our 7×7 tile to a smaller subset that contains only those blocks that are lying within the view frustum. The result is a scene usually around 50 million triangles.

While this is already a reasonable amount of simplification, in order to further reduce the amount of geometry we have to render we have to do per-object culling. But as mentioned before, we would not like to do such fine grained scene management on the CPU so we need some sophisticated methods to do it on the GPU.

In order to accomplish this, we will take advantage of the geometry shader’s capability of discarding geometry. We will use it to do the per-object decisions in order to cull the tree instances that are not visible. The three techniques implemented in the culling geometry shader and the accompanying vertex shader are the following:

• Instance Cloud Reduction (ICR) – This method does view frustum culling on a per-instance basis based on the bounding box of the instanced geometry, in this case the tree. The technique was first presented in my previous article titled Instance culling using geometry shaders and then further improved according to the instructions presented in Instance Cloud Reduction reloaded. In this case, the technique allows us to do a more fine grained yet still high level view frustum culling of the tree instances than that allowed by the simple per-tile culling performed on the CPU.
• Hierarchical-Z Map based Occlusion Culling – This technique allows for conservative per-instance occlusion culling completely done and evaluated on the GPU using a similar algorithm that the hardware depth buffer uses to hierarchically reject fragments based on their depth values. Using this technique, a coarse occlusion culling can be performed on the instances without the need of occlusion queries and CPU intervention. Update! The technique is discussed in detail in the article Hierarchical-Z map based occlusion culling.
• Dynamic Level-of-Detail Determination – This method allows us to dynamically select a suitable geometry level-of-detail on a per-instance basis completely on the GPU based on the application provided LOD parameters and the distance of the instance from the camera. The Mountains demo uses three LOD levels for the tree object: one with 5811 triangles, another with 2893 triangles and the lowest detailed version contains 1492 triangles. Update! The technical details of the algorithm are presented in the article GPU based dynamic geometry LOD.

While in the Mountains demo all these techniques are used to determine the visibility and the LOD of static scenery (as trees are unlikely to move) the truth is that these methods apply with no modification also to dynamic scenery. This is a very important thing to note as usually dynamic objects are those that makes many of the CPU based scene management and visibility determination algorithms difficult to use or simply inefficient.

The key improvement compared to how these techniques are used in AMD’s demo is that my implementation applies all the algorithms to the instance set in a single rendering pass compared to the several passes needed by the original implementation. This is because the Mountains demo takes advantage of the latest technologies introduced by OpenGL 4.0 and the supporting hardware (in this case the functionality provided by the extension GL_ARB_transform_feedback3).

By using these techniques the GPU is able to reduce the geometric complexity of the scene from 50 million triangles down to around a few millions, sometimes even under a million. Of course, the actually reduction efficiency is heavily influenced by the view position and direction.

Besides the scene management and visibility determination techniques, the demo also showcases a few simple visual effects:

View horizon and sky

• A simple infinitely far skybox generated using a geometry shader.
• Simple diffuse lighting applied to the tree instances.
• Global illumination-like effect that simulates the terrain to cast shadows over the trees even though no shadow rendering technique is applied.
• Fog effect to smooth out the disappearance of the terrain at the far clip plane.
• Simplistic fake depth-of-field effect that makes far away objects look blurry.

Maybe I will present also some of these techniques in detail in another article if there is interest for it.

As I mentioned, I used a geometry shader to render the skybox and so I did when rendering full screen quads to apply image space algorithms. I’ve done this because I always feel kind of stupid when I have to put such a simple geometry like a skybox or a full screen quad into a vertex buffer. In these situations I feel like I would simply use immediate mode to draw that damn little piece of geometry but I want to stick to core OpenGL so I quickly change my mind. As a simple alternative, I rather used geometry shaders to emit these simple geometric objects that are used so often that I even wonder how OpenGL does not have e.g. a glDrawScreenQuad-like command. Of course, the geometry shaders don’t start by themselves so I used dummy draw commands to make the geometry shader do its job.

Performance

Now let’s see how our GPU based optimizations perform in practice. I’ve collected results from typical view positions from where a moderate number of trees are visible. The tests were done on a Radeon HD 5770. Other configuration parameters are not really relevant as the demo is clearly GPU bound as only a few state changes and render commands are executed on the CPU. Of course, this is kind of a synthetic demo as you would usually want to balance the workload between the CPU and the GPU but usually you have AI, physics and other things for the CPU so transferring as much work to the GPU as possible usually gives a great benefit.

Performance comparison of the demo in frames per second on a Radeon HD5770 (higher is better): no culling (bottom), instance cloud reduction (middle), ICR + Hi-Z map based occlusion culling (top), no geometry LOD (blue), dynamic geometry LOD (red).

As you can see on the figure above, using all the optimizations clearly shows its benefits on the frame rate of the demo, even though the Hi-Z map based occlusion query requires several additional draw passes due to the construction of the Hi-Z map. It is also clearly visible that in a scene like this where there are a lot of occluders, ICR is simply not sufficient on its own. One final note that the application of dynamic LOD has a more significant effect without Hi-Z as occlusion culling removes the largest ratio of the instances.

Amount of visible geometry after culling in millions of triangles: no culling (bottom), instance cloud reduction (middle), ICR + Hi-Z map based occlusion culling (top), no geometry LOD (blue), dynamic geometry LOD (red).

Our next chart shows the amount of geometry that is finally drawn after culling in millions of triangles. On this figure we see exactly the inverse of the previous chart and it is not surprising as obviously we have a geometry throughput bottleneck. It also clearly shows how important dynamic LOD is even if we don’t perform more sophisticated visibility determination algorithms.

Finally, in the table above we’ve listed the number of draw calls needed by each technique from the reference point of view. The techniques applied do not have a significant effect on the amount of draw calls: we have a fixed number of draw calls and additionally two draw calls if we use LOD. The only exception is when we use Hi-Z map based occlusion culling as the Hi-Z map is a full mipmap chain and we need ten additional draw calls to generate all the mip-levels.

Conclusion

The techniques presented are rather simple to implement and can provide huge performance increases. Nevertheless, they allow the renderer to offload even some of the object-level algorithms from the CPU to the GPU and obviously this is the direction to go in the future.

We’ve also met mostly our goals set at the beginning. Of course not fully as the occlusion culling performed is rather a coarse culling method and does not eliminate completely all the instances that will not contribute to the final image.

Future work

While the implementation almost completely eliminates all need of CPU intervention during the rendering phase, I still had to use a few asynchronous queries to get the amount of visible instances for each geometry LOD, although the latency incurred by the use of query objects is hidden in the demo by rendering the skybox between the initiation of the queries and the retrieving of the results.

Deep in the forest

As soon as we get atomic counters into core OpenGL and consequently when we’ll have drivers supporting it, I will further improve the technique using indirect rendering and atomic counters so even the need for these queries will be eliminated.

Additionally, as mentioned several times, I plan to write detailed articles about the individual techniques I used in the demo. I decided to go in this direction as a thorough description of all the details of the demo would be simply too long in one piece.

Running the demo

The demo uses OpenGL 4.0 so a Shader Model 5.0 capable graphics card is a must. Even though most of the used techniques makes it possible to create an implementation running on OpenGL 3.x, this time I wanted to stick to GL 4.0 as I took advantage of the new features of it to even further improve the implementation.

First, don’t be afraid if after startup the demo will run on very low frame rates. This is because by default all GPU based optimizations are disabled.

You can use the SPACE button to switch between the various culling methods:

• No culling at all
• Instance cloud reduction
• ICR with Hi-Z map based occlusion culling

Finally, you can turn dynamic LOD on and off using the F3 key.

There are a few other controls present in the demo that you may figure out if you read the code, but I don’t want to go into the details of them as they will be presented in the upcoming articles where I will present Hi-Z map based occlusion culling and dynamic LOD in detail. So stay tuned: follow me on twitter or subscribe to the RSS feed.

The demo can be downloaded with full source code in the downloads section.

Gaussian blur

Gaussian blur is an image space effect that is used to create a softly blurred version of the original image. This image then can be used by more sophisticated algorithms to produce effects like bloom, depth-of-field, heat haze or fuzzy glass. In this article I will present how to take advantage of the various properties of the Gaussian filter to create an efficient implementation as well as a technique that can greatly improve the performance of a naive Gaussian blur filter implementation by taking advantage of bilinear texture filtering to reduce the number of necessary texture lookups. While the article focuses on the Gaussian blur filter, most of the principles presented are valid for most convolution filters used in real-time graphics.

Gaussian blur is a widely used technique in the domain of computer graphics and many rendering techniques rely on it in order to produce convincing photorealistic effects, no matter if we talk about an offline renderer or a game engine. Since the advent of configurable fragment processing through texture combiners and then using fragment shaders the use of Gaussian blur or some other blur filter is almost a must for every rendering engine. While the basic convolution filter algorithm is a rather expensive one, there are a lot of neat techniques that can drastically reduce the computational cost of it, making it available for real-time rendering even on pretty outdated hardware. This article will be most like a tutorial article that tries to present most of the available optimization techniques. Some of them may be familiar to all of you but maybe the linear sampling will bring you some surprise, but let’s not go that far but start with the basics.

Terminology

In order to precede any possibility of confusion, I’ll start the article with the introduction of some terms and concepts that I will use in the post.

Convolution filter – An algorithm that combines the color value of a group of pixels.

NxN-tap filter – A filter that uses a square shaped footprint of pixels with the square’s side length being N pixels.

N-tap filter – A filter that uses an N-pixel footprint. Note that an N-tap filter does *not* necessarily mean that the filter has to sample N texels as we will see that an N-tap filter can be implemented using less than N texel fetches.

Filter kernel – A collection of relative coordinates and weights that are used to combine the pixel footprint of the filter.

Discrete sampling – Texture sampling method when we fetch the data of exactly one texel (aka GL_NEAREST filtering).

Linear sampling – Texture sampling method when we fetch a footprint of 2×2 texels and we apply a bilinear filter to aquire the final color information (aka GL_LINEAR filtering).

Gaussian filter

The image space Gaussian filter is an NxN-tap convolution filter that weights the pixels inside of its footprint based on the Gaussian function:

The pixels of the filter footprint are weighted using the values got from the Gaussian function thus providing a blur effect. The spacial representation of the Gaussian filter, sometimes referred to as the “bell surface”, demonstrates how much the individual pixels of the footprint contribute to the final pixel color.

The graphical representation of the 2-dimensional Gaussian function

Based on this some of you may already say “aha, so we simply need to do NxN texture fetches and weight them together and voilà”. While this is true, it is not that efficient as it looks like. In case of a 1024×1024 image, using a fragment shader that implements a 33×33-tap Gaussian filter based on this approach would need an enormous number of 1024*1024*33*33 ≈ 1.14 billion texture fetches in order to apply the blur filter for the whole image.

In order to get to a more efficient algorithm we have to analyze a bit some of the nice properties of the Gaussian function:

• The 2-dimensional Gaussian function can be calculated by multiplying two 1-dimensional Gaussian function:

• A Gaussian function with a distribution of 2σ is equivalent with the product of two Gaussian functions with a distribution of σ.

Both of these properties of the Gaussian function give us room for heavy optimization.

Based on the first property, we can separate our 2-dimensional Gaussian function into two 1-dimensional one. In case of the fragment shader implementation this means that we can separate our Gaussian filter into a horizontal blur filter and the vertical blur filter, still getting the accurate results after the rendering. This results in two N-tap filters and an additional rendering pass needed for the second filter. Getting back to our example, applying the two filters to a 1024×1024 image using two 33-tap Gaussian filters will get us to 1024*1024*33*2 ≈ 69 million texture fetches. That is already more than an order of magnitude less than the original approach made possible.

Using the second property of the Gaussian function, we can separate our 33×33-tap filter into three 9×9-tap filter (9+8=17, 17+16=33). Back to our example, for the 1024×1024 sized image this results in 1024*1024*9*9*3 ≈ 255 million texture fetches. As we can see, we also spared a large amount of the necessary texture fetches using this approach as well.

Of course, the combination of the two techniques is also possible. That means we both separate our filter to a vertical and horizontal filter as well as decompose our 33-tap filter into three 9-tap filter. This will get us to the almost optimal number of 1024*1024*9*3*2 ≈ 56 million texture fetches.

Gaussian kernel weights

We’ve seen how to implement an efficient Gaussian blur filter for our application, at least in theory, but we haven’t talked about how we should calculate the weights for each pixel we combine using the filter in order to get the proper results. The most straightforward way to determine the kernel weights is by simply calculating the value of the Gaussian function for various distribution and coordinate values. While this is the most generic solution, there is a simpler way to get some weights by using the binomial coefficients. Why we can do that? Because the Gaussian function is actually the distribution function of the normal distribution and the normal distribution’s discrete equivalent is the binomial distribution which uses the binomial coefficients for weighting its samples.

The Pascal triangle showcasing the binomial coefficients that can be used to calculate the kernel weights (each element in the succeeding rows is the sum of its "parents").

For implementing our 9-tap horizontal and vertical Gaussian filter we will use the last row of the Pascal triangle illustrated above in order to calculate our weights. One may ask why we don’t use the row with index 8 as it has 9 coefficients. This is a justifiable question, but it is rather easy to answer it. This is because with a typical 32 bit color buffer the outermost coefficients don’t have any effect on the final image while the second outermost ones have little to no effect. We would like to minimize the number of texture fetches but provide the highest quality blur as possible with our 9-tap filter. Obviously, in case very high precision results are a must and a higher precision color buffer is available, preferably a floating point one, using the row with index 8 is better. But let’s stick to our original idea and use the last row…

By having the necessary coefficients, it is very easy to calculate the weights that will be used to linearly interpolate our pixels. We just have to divide the coefficient by the sum of the coefficients that is 4096 in this case. Of course, for correcting the elimination of the four outermost coefficients, we shall reduce the sum to 4070, otherwise if we apply the filter several times the image may get darker.

Now, as we have our weights it is very straightforward to implement our fragment shaders. Let’s see how the vertical file shader will look like in GLSL:

uniform sampler2D image;

out vec4 FragmentColor;

uniform float offset[5] = float[]( 0.0, 1.0, 2.0, 3.0, 4.0 );
uniform float weight[5] = float[]( 0.2270270270, 0.1945945946, 0.1216216216,
                                   0.0540540541, 0.0162162162 );

void main(void)
{
    FragmentColor = texture2D( image, vec2(gl_FragCoord)/1024.0 ) * weight[0];
    for (int i=1; i<5; i++) {
        FragmentColor +=
            texture2D( image, ( vec2(gl_FragCoord)+vec2(0.0, offset[i]) )/1024.0 )
                * weight[i];
        FragmentColor +=
            texture2D( image, ( vec2(gl_FragCoord)-vec2(0.0, offset[i]) )/1024.0 )
                * weight[i];
    }
}

Obviously the horizontal filter is no different just the offset value is applied to the X component rather than to the Y component of the fragment coordinate. Note that we hardcoded here the size of the image as we divide the resulting window space coordinate by 1024. In a real life scenario one may replace that with a uniform or simply use texture rectangles that don’t use normalized texture coordinates.

If you have to apply the filter several times in order to get a more strong blur effect, the only thing you have to do is ping-pong between two framebuffers and apply the shaders to the result of the previous step.

9-tap Gaussian blur filter applied to an image of size 1024x1024: no filter applied (left), applied once (middle), applied nine times (right). Click to view the full-sized image in order to better see the difference.

Linear sampling

So far, we were able to see how to implement a separable Gaussian filter using two rendering pass in order to get a 9-tap Gaussian blur. We’ve also seen that we can run this filter three times over a 1024×1024 sized image in order to get a 33-tap Gaussian blur by using only 56 million texture fetches. While this is already quite efficient it does not really expose any possibilities of the GPUs as this form of the algorithm would work perfectly almost unmodified on a CPU as well.

Now, we will see that we can take advantage of the fixed function hardware available on the GPU that can even further reduce the number of required texture fetches. In order to get to this optimization let’s discuss one of the assumptions that we made from the beginning of the article:

So far, we assumed that in order to get information about a single pixel we have to make a texture fetch, that means for 9 pixels we need 9 texture fetches. While this is true in case of a CPU implementation, it is not necessarily true in case of a GPU implementation. This is because in the GPU case we have bilinear texture filtering at our disposal that comes with practically no cost. That means if we don’t fetch at texel center positions our texture then we can get information about multiple pixels. As we already use the separability property of the Gaussian function we actually working in 1D so for us bilinear filter will provide information about two pixels. The amount of how much each texel contribute to the final color value is based on the coordinate that we use.

By properly adjusting the texture coordinate offsets we can get the accurate information of two texels or pixels using a single texture fetch. That means for implementing a 9-tap horizontal/vertical Gaussian filter we need only 5 texture fetches. In general, for an N-tap filter we need [N/2] texture fetches.

What this will mean for our weight values previously used for the discrete sampled Gaussian filter? It means that each case we use a single texture fetch to get information about two texels we have to weight the color value retrieved by the sum of the weights corresponding to the two texels. Now that we know what are our weights, we just have to calculate the texture coordinate offsets properly.

For texture coordinates, we can simply use the middle coordinate between the two texel centers. While this is a good approximation, we won’t accept it as we can calculate much better coordinates that will result us exactly the same values as when we used discrete sampling.

In case of such a merge of two texels we have to adjust the coordinates that the distance of the determined coordinate from the texel #1 center should be equal to the weight of texel #2 divided by the sum of the two weights. In the same style, the distance of the determined coordinate from the texel #2 center should be equal to the weight of texel #1 divided by the sum of the two weights.

As a result, we get the following formulas to determine the weights and offsets for our linear sampled Gaussian blur filter:

By using this information we just have to replace our uniform constants and decrease the number of iterations in our vertical filter shader and we get the following:

uniform sampler2D image;

out vec4 FragmentColor;

uniform float offset[3] = float[]( 0.0, 1.3846153846, 3.2307692308 );
uniform float weight[3] = float[]( 0.2270270270, 0.3162162162, 0.0702702703 );

void main(void)
{
    FragmentColor = texture2D( image, vec2(gl_FragCoord)/1024.0 ) * weight[0];
    for (int i=1; i<3; i++) {
        FragmentColor +=
            texture2D( image, ( vec2(gl_FragCoord)+vec2(0.0, offset[i]) )/1024.0 )
                * weight[i];
        FragmentColor +=
            texture2D( image, ( vec2(gl_FragCoord)-vec2(0.0, offset[i]) )/1024.0 )
                * weight[i];
    }
}

This simplification of the algorithm is mathematically correct and if we don’t consider possible rounding errors resulting from the hardware implementation of the bilinear filter we should get the exact same result with our linear sampling shader like in case of the discrete sampling one.

9-tap Gaussian blur applied nine times with discrete sampling (left) and linear sampling (right). Click for the full resolution of the image. Note that there is no visible difference between the two techniques even after several passes.

While the implementation of the linear sampling is pretty straightforward, it has a quite visible effect on the performance of the Gaussian blur filter. Taking into consideration that we managed to implement a 9-tap filter using just five texture fetches instead of nine, back to our example, blurring a 1024×1024 image with a 33-tap filter takes only 1024*1024*5*3*2 ≈ 31 million texture fetches instead of the 56 million required by discrete sampling. This is a quite reasonable difference and in order to better present how much that matters I’ve done some experiment to measure the difference between the two techniques. The result speaks for itself:

Performance comparison of the 9-tap Gaussian blur filter with discrete and linear sampling on a Radeon HD5770. The vertical axis is the frames per second (higher is better) and the horizontal axis represents results with various number of blur steps (higher is blurrier).

As we can see, the performance of the Gaussian filter implemented with linear sampling is about 60% faster than the one implemented with discrete sampling indifferent from the number of blur steps applied to the image. This roughly proportional to the number of texture fetches spared by using linear filtering.

Conclusion

We’ve seen that implementing an efficient Gaussian blur filter is quite straightforward and the result is a very fast real-time algorithm, especially using the linear sampling, that can be used as the basis of more advanced rendering techniques.

Even though we concentrated on Gaussian blur in this article, many of the discussed principles apply to most convolution filter types. Also, most of the theory applies in case we need a blurred image of reduced size like it is usually needed by the bloom effect, even the linear sampling. The only thing that is really different in case of a reduced size blurred image is that our center pixel is also a “double-pixel”. This means that we have to use a row from our Pascal triangle that has even number of coefficients as we would like to linear sample the middle texels as well.

We’ve also had a brief insight into the computational complexity of the various techniques and how the filter can be efficiently implemented on the GPU.

The demo application used for the measurements performed to compare the discrete and linear sampling method can be downloaded here:

Binary release

Platform: Windows
Dependency: OpenGL 3.3 capable graphics driver
Download link: gaussian_win32.zip (2.96MB)

Source code

Language: C++
Platform: cross-platform
Dependency: GLEW, SFML, GLM
Download link: gaussian_src.zip (5.37KB)

P.S.: Sorry for the high minimum requirements of the application just I would really like to stick to strict OpenGL 3+ demos.

OpenGL 3.3 - Nature

A few months ago I’ve presented an object culling mechanism that I’ve named Instance Cloud Reduction (ICR) in the article Instance culling using geometry shaders. The technique targets the first generation of OpenGL 3 capable cards and takes advantage of geometry shaders’ capability to reduce the emitted geometry amount in order to get to a fully GPU accelerated algorithm that performs view frustum culling on instanced geometry without the need of OpenCL or any other GPU compute API. After the culling step the reduced set of instance data is fed to the drawing pass in the form of a texture buffers. In this article I will present an improved version of the algorithm that exploits the use of instanced arrays introduced lately in OpenGL 3.3 to further optimize it.

Lets recap the basics of the algorithm before I present the improved technique. The geometry shaders have a very nice feature that they cannot just emit a modified version of the input geometry but can also alter the number of emitted primitives compared to the number of received ones. This is a both-way ability what means that we cannot just increase but also decrease the number of primitives. That is what the technique takes advantage.

In the first pass we feed a simple vertex shader – geometry shader pair with the instance data of the geometries as they’ve been the data of point primitives. The vertex shader then checks whether the actual instance is inside the view frustum or not and sends the result to the geometry shader. If the result is yes then the geometry shader outputs the instance data otherwise discards it. The primitives emitted by the geometry shaders are captured then using transform feedback into a buffer object. Also a query object is needed in order to be able to get the amount of instances that passed the view frustum culling. In the drawing pass we use the result of the query to decide how many instances we have to draw and the captured feedback buffer is used as instance data.

Instance Cloud Reduction - Combined view of Pass 1 + Pass 2

This is a very brief description of the culling mechanism so for a complete specification please read the original article.

Motivation

While Instance Cloud Reduction is a quite robust technique that can severely simplify and speed up the rendering of high amount of instanced geometry its performance is also limited due to some hardware and API restrictions. The most important ones are the following:

• Needs an extra rendering pass to perform the culling.
• Requires the usage of asynchronous queries to determine the number of visible instances.
• Uses texture fetching in the vertex shader of the actual drawing pass.

The first mentioned drawback means that more draw commands are required that use the output of the first pass as input. This and the second disadvantage may cause stalls due to the fact that the CPU has to wait for the data to be ready before issuing the second pass thus the GPU is not used effectively.

What this improvement tries to solve is the third problem. Texture fetching itself is quite fast in the latest generation of hardware, however it causes some slowdowns anyway due to the latency introduced by texture fetches even though GPUs use some latency hiding techniques.

Instanced arrays provide us a way to replace texture fetching with vertex fetching that is usually done by different hardware element that works synchronously with the execution of vertex shaders. I’ve expected quite a reasonable speedup by taking advantage of instanced arrays, however we will see that actual results were far from my initial expectations.

Implementation

Traditional vertex fetching happens in a way that one element is fetched from each enabled input attribute buffer and the vertex shader is issued with these values. One element in a vertex attribute buffer can mean up to four floating point or integer values and for each execution of the vertex shader one set of these elements is used. There is an internal counter that is increased after each fetch and the next vertex attribute fetch will use this counter as an index into the buffer object.

While this mechanism is satisfactory for the most attributes of a vertex, it is not practical for instance data as such data belongs to an instance rather than a vertex. In order to source instance data from vertex attributes in case of traditional vertex fetching, high amount of redundant storage is required in order to get the same information for all the vertices belonging to a particular instance. This is not just waste of memory but also waste of bandwidth and it also defeats the goal of Instance Cloud Reduction.

Compared to traditional vertex fetching, instanced arrays provide a way to increase the internal counter used as the index into the vertex attribute buffer in a different way, in particular one can set the frequency of increase using a vertex attribute divisor that specifies after how many instances the counter shall be increased. This is a per-attribute property and by setting it to one we end up with exactly what we need: one vertex fetch per instance.

This means that actually we need just a very minor change compared to the original technique, more precisely we replace our texture buffer with a vertex attribute buffer that has a divisor of one and use it as the source of instance data in the vertex shader of the drawing pass.

Execution results

As we are not talking about a new technique but just an optimized implementation of the same method, the best way to evaluate it is by comparing the performance of the new version with the original one.

As I’ve mentioned earlier, I expected a reasonable performance increase by replacing texture fetches with vertex fetches, in practice the difference was not so significant. However, the performance difference between the two implementation can heavily depend on the underlying hardware implementation so various cards from various vendors and GPU generations can show more diverging behavior. In fact even driver versions may have an effect on the results.

Performance comparison of the old implementation and the presented one on an AMD Radeon HD5770. Scale is in frames per second (higher value is better).

Due to lack of hardware to use for testing, I’ve checked only with one card, namely a Radeon HD5770 with Catalyst 10.6 drivers. I noticed roughly a 10% speedup as the the new version of the Nature demo showed 100 FPS compared to the 90 FPS observed with the old implementation.

Even though this was not exactly the outcome I’ve expected from the new implementation, maybe the assumption is still valid for older generation of GPUs or for NVIDIA cards. I suspect so because for Shader Model 4.0 cards the hardware implementation of the texture fetching unit and the vertex fetching unit was most probably more differentiated than that of the latest GPUs. Also my guess is that on NVIDIA cards the difference is maybe higher as the vertex fetching hardware in SM 4.0 GeForce cards is less flexible than that of AMD’s taking in consideration that the first HD series Radeons already had some form of tessellation functionality that requires more freedom from the vertex pushing hardware.

In order to get a better picture about how effective the presented optimization is, I would like to ask all the visitors of this post to try the two releases and send me feedback about it.

Conclusion

We’ve seen that how easy it was to take advantage of instanced arrays in an existing implementation of the ICR technique and how does it perform on the latest generation of GPUs compared to the previous version. While this small addition provides some benefits, it also comes at a cost and we have to talk about that as well.

Advantages:

• Eliminates the need for texture fetching in the vertex shader thus improving performance.
• Does not compromise the goal and the implementation architecture of the original method.
• Frees up one texture unit that was previously reserved for the texture buffer containing the instance data.

Disadvantages:

• Requires OpenGL 3.3 or the GL_ARB_instanced_arrays extension in addition to the OpenGL 3.2 features.
• We have to possibly sacrifice multiple vertex input attributes to feed the instance data to the shaders.

Most of the mentioned benefits and drawbacks are self-explanatory, however I would like to say a few words about the last mentioned one…

For the purpose of showcase I used a simple translation factor as instance data that means a single vector of floats. In real life situation one may need more complex transformation data that can only be stored in the matrix. While in the demo the feeding of instance data consumed only one vertex attribute slot, in case of a full transformation matrix it would require four of them (not to mention other possible instance attributes). As the maximum number of input attributes is severely limited, usually to 16, the application of the optimization is restricted to situations when all the vertex and instance attributes fit into this limit.

In case of the original implementation, where a texture buffer was used as input, this did not cause any problem as the vertex shader is free to fetch any number of texels from that (still, performance can be a concern in this case). In order to help situations when input attribute slots are at a premium, in real life scenarios it is recommended to use quaternions instead of transformation matrices as they consume two times less attribute resources. Actually this can be a general recommendation as using quaternions decreases the bandwidth requirements of the instance data fetch thus increasing performance even in situations when there are enough input attribute slots available.

In order to ease the performance comparison for you, you can find download links for both versions of the Nature demo.

Old version binary release

Platform: Windows
Dependency: OpenGL 3.2 capable graphics driver
Download link: nature12_win32.zip (3.58MB)
Comments: This version does NOT include the optimization presented in this article.

Old version source code

Language: C++
Platform: cross-platform
Dependency: GLEW, SFML, GLM
Download link: nature12_src.zip (12.6KB)
Comments: This version does NOT include the optimization presented in this article.

New version binary release

Platform: Windows
Dependency: OpenGL 3.3 capable graphics driver
Download link: nature20_win32.zip (3.58MB)
Comments: This version includes the optimization presented in this article.

New version source code

Language: C++
Platform: cross-platform
Dependency: GLEW, SFML, GLM
Download link: nature20_src.zip (12.8KB)
Comments: This version includes the optimization presented in this article.

Many things have changed since the first time the public put their hands on the first mobile phone device as these days the end user rarely makes their choices when buying a mobile equipment based on their telephony capabilities. In fact, nowadays these devices are one of the most popular entertainment platforms out there. The main problem for application developers is that these platforms tended to be very heterogeneous from point of view of hardware architecture as well as that of API support. Meanwhile things have changed. While the underlying hardware still varies a lot from device to device the work of application developers has been eased by having cross platform mobile operating systems and open standards. In particular OpenGL ES that is an embedded version of the popular graphics API. In this article I would like to talk about some of the big players of the mobile OS industry and about using OpenGL ES for creating impressive mobile applications.

The first version of the OpenGL ES specification has been released in order to provide a lightweight API for embedded graphics using a well-defined subset of the functionalities provided by the desktop version of OpenGL. While the specification is already out quite for a while, the wide adoption in the industry and the interest of application developers for it became strong only in the recent past. Currently, we have several mobile platforms that are bundled with 3D accelerators and provide a set of features via OpenGL ES that makes developers capable of creating games that weren’t possible even on desktop platforms about ten years ago.

Going 3D on mobiles

Those who know me, know that well that I was always interested in graphics, especially when using it for entertainment purposes. In particular, I was about to develop video games since the first time I’ve put my hands on a computer. This is no different now as well as now I’m writing about OpenGL ES and mobile platforms because I got interested in creating games for mobile phones.

As I’ve already mentioned before, the problem with developing for mobile equipments is the variety of hardware and software platforms that they are built on. As being somebody who is already familiar with desktop OpenGL, having OpenGL ES in the tool-set already eliminates some of the burden that I must face with.

Also when talking about application platform things have also changed a lot. Nowadays, we have just a few big players in the mobile OS industry thus easing the work of the developers. More precisely, if an application developer plans to go mobile and would like to grab the biggest market audience, can limit their efforts on the following platforms:

• iPhone OS – This is the one that drives Apple’s iPhone mobile devices as well as the iPod Touch. It provides an application platform similar to that Mac developers got used to. It can be said that this platform is the most popular in the industry, especially when dealing with gaming applications.
• Android – This is the newest player in the field, brought by Google. While it’s a newbie in the industry it already captured the attention of tons of developers. We can say that currently Android and iPhone are dictating the direction of mobile entertainment.
• Symbian OS – Symbian has the largest share in most markets worldwide, still not that popular in the mobile gaming industry. It is the operating system running most of today’s Nokia phones.
• Windows Mobile – Microsoft’s product built on Windows CE, the company’s embedded operating system.
• RIM Blackberry OS – Operating system primarily designed for the business industry.

While most of these mobile operating systems are built on the same design conceptions it is very difficult for the developer to create cross-platform applications for all these platforms as they vary on the language and tool-set support that minimizes the possibilities for code reuse. Unfortunately this is against the one of the most important rule of mobile development as to maximize portability.

It is not 100% true that there is no way to provide optimum portability for all these platforms, but if we choose this direction we are limited to two possibilities: cross-platform Java applications and web-based applications. While these seem to be excellent alternatives to native programming of the platforms, they severely limit the developer in creating applications that fully take advantage of the underlying hardware. This is when OpenGL ES comes into picture as all these platforms have API support thus providing at least some form of code reuse possibility when dealing with entertainment applications.

Now, I would like to continue with talking about the two platforms that I’m most interested in.

iPhone OS

I started to get involved in iPhone game development because one of my friends pushed me to after seeing the great success of his brother-in-law, zhooley who had some great titles. Currently I don’t have a Mac yet to develop on, but already read some stuff about iPhone development. This is where the following information come from.

iPhone is currently is the most important platform for mobile application developers. It became such an important factor in the industry thanks to Apple’s AppStore. Previously there was little to no way for the end users to extend their mobile software base so easily. While this is good for the end user, it is maybe even better for application developers as AppStore provides them quite a large market audience.

The secret why iPhone is an excellent gaming platform lies in the palette of features that the phone hardware and the software frameworks provide. Just to mention the most important ones:

• Touch screen control with support for multi-touch events capturing the movement of up to five fingers.
• Three accelerometers for tracking the spacial movement and direction of the device in all axes.
• MVC inspired GUI framework for enhanced productivity.
• Support for several industry standard APIs like OpenGL ES, OpenAL and much more.

But that’s enough from the general speaking, let’s see what’s about OpenGL ES support on the iPhones…

As far as I can tell, not being an iPhone owner, the graphics hardware bundled with the mobile comes in form of PowerVR accelerators: MBX and SGX.

The PowerVR MBX has OpenGL ES 1.1 support, that is roughly equivalent to OpenGL 1.5, running a tile-based deferred renderer that is suitable for most 3D applications. That means it has only fixed function capabilities, however that is usually enough for most mobile applications. Also note that it has very limited amount of texture memory of 24MB.

The PowerVR SGX is a more powerful processor that also supports OpenGL ES 2.0, roughly equivalent to OpenGL 2.0, but has optimized fixed function shaders that provide flawless backward compatibility for OpenGL ES 1.1 applications.

The most important thing is still that all iPhones are able to do floating point maths natively and efficiently that is an important factor when dealing with OpenGL applications as the usage of the fixed point types can be quite a burden for developers, especially for those migrating from desktop development.

Additionally, the OpenGL ES implementation on iPhone provides some nice extensions like GL_OES_framebuffer_object, GL_OES_compressed_paletted_texture and GL_OES_point_sprite. Also, thanks to the iPhone simulator that comes with the SDK it is easy to test the application during development without an actual device. Still, one important hint to mention is that the iPhone simulator has different OpenGL ES capabilities than the actual hardwares and also the performance characteristics measured on the simulator should not be taken as valid measurements because the simulator does not really simulate the graphics hardware but only the software platform.

iPhone development is done using the Cocoa API and preferably Objective-C, however C, C++ and Objective-C++ can be also used for development. One just has to interface somehow the Cocoa API and the rest can be done almost in any native programming language. That is one of the key advantages of the iPhone platform that one can develop native applications and no need for Java or web-based solutions.

While iPhone may seem to be a perfect choice for mobile game platform, we should not forget about one big disadvantage of it, in particular that one cannot develop legal iPhone applications without owning a Mac.

Android

The Android platform was suggested by one of my workmates who just brought a Droid. That phone is actually a device capable to compete with the iPhone from both features and performance point of view.

Android is the big hit of the last year and my forecast is that it will be one of the most relevant platforms of the upcoming years. Google adopted the idea of Apple and they also created an open market for the softwares that the end user can easily download and install on their devices. This is the AndroidMarket that can easily become a powerful competitor of the AppStore.

While, as I said earlier, the Motorola Droid, as an example, does support about the same feature set that makes the iPhone an excellent gaming platform, this cannot be said about most of the phones running Android on them. This is maybe one of the biggest disadvantages of the Android platform. However, we can take this also as an advantage as it makes it possible for more phones to adopt this operating system.

As the Android operating system is running on various phones from different vendors with different hardware capabilities, there isn’t too much to talk about the graphics hardware capabilities except that some devices not just don’t have a graphics accelerator but they also lack of floating point support. This is another disadvantage as it forces developers to stick to fixed point math in their OpenGL ES applications to maximize portability or they have to maintain two different rendering paths.

Originally, Android supported only OpenGL ES 1.0 that is roughly equivalent to OpenGL 1.3. However, since NDK r3 there is also OpenGL ES 2.0 support for Android as well. The feature set here varies much more from both hardware point of view and extension support.

Development for Android is done in Java using a proprietary SDK for accessing the Android API. The SDK comes with a simulator that works fine, except the long initial boot time that I was really surprised about when first trying it out.

One advantage of the SDK that it can be used in virtually any operating system so application developers can work on either Windows, Linux, MacOSX or other platform. There is also a nice Eclipse plugin that makes application development for Android even easier. That’s why I started with this one.

Just to illustrate how easy to put together some working demo with a good SDK, I’ve created a simple box rotating app to demonstrate OpenGL ES usage on Android. From installation till having a working application it took no more than two hours. You can find the download links for both the source code and the binary release at the end of the article.

Why mobile games?

I am a person who was, is and will be interested in developing computer games. Previously, I was working with desktop platforms and at the time when I was 10 years old it was satisfactory to put together some simple 2D game but not now.

I had always planned to create a state-of-the-art game engine and use it for some game, like most people like me do, but the efforts of one is simply unsatisfactory to compete with the players in the industry out there. Even if I feel the capability to be able to write such an engine but it would take that much time that I simply don’t have since I am working. Even if I would manage to accomplish it in a year or two then the problem with content creation comes into picture. For an AAA PC game content creation takes several times more than the actual programming and here I even lack the knowledge to achieve it. On the other hand mobile game creation is a much shorter process when you can get to actual results in a matter of weeks that is far better compared to PC game creation.

Also, I would never use third party game engines, except some basic libraries like OpenGL, a physics library and things like that because otherwise I wouldn’t feel the results being my own creation.

Having game development as a hobby works well during high school and university but it gets quite difficult after you are out there in the world having a job and responsibilities. Maybe I should have been already taking my time before to develop something concrete for PC but, as most fellow hobbyist know, you usually end up having hundreds of unfinished projects.

While I would never forget about desktop platforms and I will actively keep myself up with the evolution of the industry, mobile application development opened another world for me where I can unfold myself.

HelloAndroid Demo

Source code: helloandroid_src.zip
Binary release: HelloAndroid.apk

OpenGL 3.2 - Nature

Since the appearance of Shader Model 4.0 people wonder how to take advantage of the newly introduced programmable pipeline stage. The most important feature enabled by geometry shaders is that one can change the amount of emitted primitives inside the pipeline. The first thing that a naive developer would try to do with it is geometry tesselation. However, the new shader performs very bad when used for tesselation in a real life scenario even though there are demos show casting this possibility. If we take a closer look at the new feature we observe that the most revolutionary in it is not that it can raise the number of emitted primitives but that it can discard them. This article would like to present a rendering technique that takes advantage of this aspect of geometry shaders to enable the GPU accelerated culling of higher order primitives.

Geometry shaders can be used for many different advanced rendering techniques that were impossible before the introduction of this flexible programmable shader stage. In this article I would like to present one use case that for me seemed to be one of the most practical application of primitive manipulation possibilities introduced by geometry shaders. As I haven’t seen any whitepaper talking specifically about this particular technique, even if some of them inherently used it, I would dare name the technique myself as Instance Cloud Reduction. I will also present a demo program that shows how to take advantage of the technique in a heavy workload situation.

The idea itself was inspired by AMD’s  tech demo for the Radeon 4800 series cards called March of the Froblins. An almost identical technique presented in this article is used in the mentioned demo for the culling of large amount of animated creatures against the view frustum. Also a somewhat similar technique is used in NVIDIA’s Skinned Instancing demo for determining LOD instance sets. Unfortunately, both demos are for DirectX only and, as far as I can tell, there is no OpenGL demo showing any of the aforementioned rendering techniques.

Motivation

Nowadays, as the computational capabilities of GPUs is growing in a much faster pace than that of CPUs, graphics developers meet more and more optimization problems related to CPU bound applications. More and more focus is on minimizing the number of driver invocations, actually that’s what motivated the restructuring of the two most commonly used graphics APIs. As a result we have now DirectX 10+ and OpenGL 3+. However, even if the introduction of geometry instancing, texture arrays and local memory buffer storage for the most important inputs of the rendering, there is still need for wise decisions from graphics programmers to take full advantage of the horsepower coming with the latest GPUs.

Earlier graphics applications strongly relied on CPU based culling techniques, whether it be the usage of the quite outdated BSPs or the more generic and still heavily applied hierarchical culling techniques. We’ve already reached the point that sometimes even the most efficient CPU based culling techniques seem to be too expensive and usually introduce the small batch problem. Instanced rendering is not an exception.

The applicability of geometry instancing is strongly limited by several factors. One of the most important ones is the culling of instanced geometries. One may choose to cull these objects in the same fashion as others, using the CPU, but that usually breaks the batch and maybe we loose the benefits of geometry instancing. It is more and more imminent to have a GPU based alternative. Without CPU based culling, by sending the whole bunch of instances down the graphics pipeline may choke our vertex processor in case we have high poly geometries and quite large amount of instances of it.

The rendering technique presented in this article will try to achieve this goal. We will use a multi-pass technique that in the first pass culls the object instances against the view frustum using the GPU and in the second pass renders only those instances that are likely to be visible in the final scene. This way we can severely reduce the amount of vertex data sent through the graphics pipeline.

Implementation

For some people it might seem that the promise for such a technique is simply too naive and is most probably relying on very exotic OpenGL features, heavy misuse of some basic features or need of data conversions during the frame rendering. Wondrously, this is not the case as we have all we need in OpenGL 3.2 to implement the object culling method sketched above. All we need are the followings:

• instanced rendering (core since OpenGL 3.1)
• geometry shaders (core since OpenGL 3.2)
• transform feedback (core since OpenGL 3.0)
• uniform or texture buffers (core since OpenGL 3.1)

The method itself is a multi-pass rendering technique, however, unlike other multi-pass rendering techniques it does not produce any fragments in the first pass, instead the first pass does the view frustum culling and processes data entirely only inside buffer objects.

Culling pass

In the first pass we will feed the graphics pipeline with information about the instances that are needed to perform the view frustum culling. For this we need two inputs for the executed shaders in order to be able to perform the required calculations:

1. Instance transformation data (whether it be a simple transformation matrix or quaternions or whatever) -- This preferably comes from one or more buffer objects that are bound as vertex buffers to the context.
2. Object extents information -- Beside the instance positions we have to know the extents of an instance in order to perform correct culling. This can be either a single float representing the object radius if we choose to use bounding spheres for the culling or a three-dimensional extent vector if we would like to use bounding boxes.

Using these as input we can feed in the instance transformation data as attributes of point primitives to our culling shader. The culling shader is composed of a vertex and a geometry shader. In a typical setup the role of each is the following: the vertex shader determines whether the actual object instance’s bounding volume is inside the view frustum and sends a flag about the culling to the geometry shader, that will emit the instance data to the destination buffer if the flag says that the instance is likely to be visible or does not emit anything if it is determined that the object instance is out of view.

Next, transform feedback is used to capture the primitives emitted by the geometry shader into another buffer object that will be used in the actual rendering pass to source instance transformation data. Beside this, we also need to have an asynchronous query to determine the number of primitives generated to know how many instances of the object do we actually need to render. The following figure shows the workflow of the first pass:

Instance Cloud Reduction - Pass 1: Culling

The actual geometry shader implementation needed to perform the actual culling based on the view frustum check performed by the vertex shader should look like the following chunk:

#version 150 core

layout(points) in;
layout(points, max_vertices = 1) out;

in vec4 OrigPosition[1];
flat in int objectVisible[1];

out vec4 CulledPosition;

void main() {

	/* only emit primitive if the object is visible */
	if ( objectVisible[0] == 1 )
	{
		CulledPosition = OrigPosition[0];
		EmitVertex();
		EndPrimitive();
	}
}

In this example we used only simply a four-component position vector for the instance transformation data but the technique works well for transformation matrices and quaternions as well.

One more thing is that beside that we set up transform feedback in a way that we feed our buffer object dedicated for the culled instance data and we also started an asynchronous query to be able to determine the number of primitives written into the buffer object, it is also useful to turn of rasterization as we wouldn’t like to produce any fragments as a result of the first pass.

Rendering pass

In the second pass there is nothing special to do. Simply use whatever rendering setup you would like to use. The only things that need to be changed in this step compared to your already existing rendering path is that the instance data for the rendering must be sourced from the generated culled instance data buffer and, as a result, the number of instances passed for the instanced drawing functions shall be changed in order to render only the visible instances. This number can be read from the asynchronous query’s result that we started in the first pass.

The instance data in the rendering pass can be, of course, sourced from either a uniform or a texture buffer object. This depends on the actual use case and is more clearly explained in the article Uniform Buffers VS Texture Buffers.

Important note is that when one has to deal with several instanced geometries it is recommended to do the culling phase prior to rendering any instanced primitives because of the following reasons:

• The result of the first instance cloud’s culling is more likely to be finished on the GPU so no sync issues arise from reading the asynchronous query result to determine the number of visible instances.
• Probably less state changes are needed as very different setup is required by the two passes.
• Results in tidier renderer design as culling is clearly separated from actual rendering.

Putting everything together, the application of the presented technique would result in the following workflow on the GPU:

Instance Cloud Reduction - Combined view of Pass 1 + Pass 2

Conclusion

We’ve seen that the presented advanced rendering technique is able to help in situations when we have to deal with large number of instanced geometries and how to take advantage of the latest features of graphics cards and OpenGL to perform view frustum culling calculations on the GPU. This prevents us from having to deal with complicated and expensive CPU based object culling methods that break the drawing batches, especially when dealing with dynamic objects. For ease the decision whether to incorporate this technique in your rendering engine I would like to present the advantages and disadvantages of it.

Advantages:

• Heavily reduces the amount of processed data in a naive implementation.
• No need for any space partitioning methods in the host application to handle the culling of dynamic objects.
• Can handle huge amount of instanced objects due to the enormous horsepower of today’s GPUs.
• Scales well with increased number of instances as the per-instance calculation is relatively low.
• Relies strictly on OpenGL 3.2 core features.
• No need for OpenCL capable hardware.

Disadvantages:

• Needs an extra rendering pass to perform the culling.
• Requires the usage of asynchronous queries to determine the number of visible instances.

I hope you agree with me and think about this technique as one more step towards fully GPU based scene management. If you have any remarks or improvement ideas regarding to the rendering technique itself feel free to tell me.

The Demo

As I promised, the technique presented above comes with a live demo that actually took most of my time dedicated to writing this blog in the last two weeks. The demo itself is more like a technical show cast rather than a presentation of a real-life use case scenario.

First of all, I used high polygon count models for the rendering to emphasize the amount of time the culling phase spares from the very valuable time of our GPU. In a real world application one would never do something like this. As a result, the demo is more like a benchmark than an interactive application. However, maybe on high-end graphics cards it can perform pretty well.

The demo scene consists of two object types: trees and grass blocks. The tree model is further divided into two parts as they need different textures: the tree trunk and the tree foliage. Obviously, this additional burden can be prevented by using texture arrays to avoid the need of separate draw calls to render the trunk and the foliage.

The tree trunk consists of 33138 triangles, the tree foliage has 16069 triangles and the faking-free grass block consists of 8961 triangles which I had to model myself as didn’t found any suitable model. Actually this modeling step consumed quite a reasonable amount of my time spent with the demo as I’m not an expert in this domain.As you can see, these models are not the ones that one might use in an interactive real-time application like games. However, they seemed to be very suitable for the purpose of the demonstration.

What really kicks off the boundaries of GPUs is that the demo renders 10,000 trees and 250,000 grass blocks using instancing. This ends up in more than 2.7 billion triangles in the scene. This is far more that a GPU can handle without the aid of some scene management and culling. However, we will use no scene management at all and the only culling method that we will use is the one presented in this article.

The actual results are quite promising. The view frustum culling step usually spares more than 99.9% of the GPU horsepower as the amount of actually rendered triangles after the culling step is far below 2 million triangles. This is still quite much but as we use high polygon count models and we don’t use any LOD techniques this seems reasonable.

Even if the demo scene statistics doesn’t seem like a typical use case scenario, the ease of the implementation and the compelling visual results made me pleased anyway:

www.youtube.com/watch?v=srbOFTLTe8k

On my Radeon HD2600XT I have achieved 6-7 frames per second which is acceptable taking in consideration the huge amount of geometry data still passed to the graphics card. On more recent cards I suppose it should run with good frame rates, however, due to the lack of hardware to test on, these are my only results. If anybody manages to take a better screen capture than mine above then please let me know.

Implementation details

Just to tell a few words about what techniques and tricks I’ve used during the creation of the demo here is a listing of the most important ones:

• Three models are used as mentioned previously with high instance counts with over 2.7 billion of total triangles in the scene as mentioned already.
• Three 512x512 RGBA textures are used for the models that are partially handmade, and again, I’m not a texture artist so sorry if they don’t look flawless.
• The wavefront model and TGA image loader that accompany the demo are very roughly implemented only for the demo so I would strongly encourage you not to use it to any purpose as it handles only a subset of the possibilities of the file formats.
• The vertex data from the wavefront model files is transferred in a very naive way so vertex reuse isn’t taken into account.
• The instance data consists of simple four-component vectors representing the world-space position of the instance. This seemed to be the most simple for the demonstration purposes.
• In the second pass, the instance data is sourced from a texture buffer but not really because the visible instance count exceeded the amount that would fit in a uniform buffer. I used texture buffers because for this simple demonstration they seemed to be a little bit more easy to be integrated.
• The morphing effect that simulated wind blow is done using hard-coded geometry deformation in the vertex shader. It is not physically correct but visually compelling.
• The lighting is a simple directional light using Phong’s shading and reflection model.
• Simple fog is simulated with some awkward formula that I’ve chosen after a few test runs.
• Alpha testing is achieved by using the discard operation in the fragment shader.

Driver issues

During the development of the demonstration program I’ve met several driver related problems as I’ve never used so heavily the latest OpenGL features previously. I’ve worked with Catalyst 9.12 and 10.1 but both seemed to lack of a proper GLSL compiler. Here are some of the issues I’ve met:

• When I’ve forgot to declare the varyings in the geometry shader as arrays like the standard requires then still the driver hasn’t complained about any syntax error but when tried to execute the code the program crashed.
• Except the texture sampler uniform, all other uniforms failed to work when used in the fragment shader only so I’ve put them all in the vertex shader.
• For loops seemed not to work when used inside the geometry shader, that’s why the culling itself is done in the vertex shader in the demo.

All these problems resulted in nasty tricks to make things working and ended up in awful shader code. Sorry for that. At least now it works on my configuration but pretty unsure whether it will work on other graphics card and driver combos. Please report me any success or failure when trying out the demo. Anyway, be sure to have the latest graphics drivers installed as, at least in case of AMD, OpenGL 3.2 drivers came out only at the fall of 2009.

Edit:

Thanks to the information got from Pierre Boudier from AMD I’ve updated both the source and binary releases to support the latest drivers properly. The problem was that I didn’t use attribute location binding as specified in the standard.

Also have to mention that with my new Radeon HD5770 I managed to achieve over 90 frames per second that actually show that this technique can be in fact used for games and other interactive applications.

One more thing in the end. As you know this version of the Nature demo uses a texture buffer to source instance positions. I plan to create another version that will take advantage of the instanced arrays introduced in core with OpenGL 3.4. I expect quite a reasonable speedup as that would eliminate the need for texture fetches in the vertex array by rather dedicating a vertex fetcher for the purpose thus increasing the overall performance of the technique.

Binary release

Platform: Windows
Dependency: OpenGL 3.2 capable graphics driver
Download link: nature12_win32.zip (3.58MB)
Comments: Includes the update that makes it work even with the latest drivers.

Full source code

Language: C++
Platform: cross-platform
Dependency: GLEW, SFML, GLM
Download link: nature12_src.zip (12.6KB)
Comments: Sorry for the many dependencies, however, I would recommend the mentioned libraries for everybody who is doing OpenGL development.

Previously I talked about how one can easily take advantage of multiprocessing using OpenMP. Even if the C pragmas introduced by the parallel programming API standard is very straightforward for simple programs, it simply doesn’t fit nicely in a complex C++ application that is built from the ground with the OOP in mind. To smoothly introduce OpenMP into such projects one need higher level constructs that hide the actual implementation details. This is the first article of a series that will try to provide reference implementations of such an abstraction. First, we will start with synchronizable primitives that try to reflect the functionality provided by the “synchronized” statement of Java.

This article is highly inspired by an article written by Achilleas Margaritis and is mostly equivalent with his thoughts. My article tries to provide a portable reference implementation of a slightly modified version of the trick presented by Margaritis that uses OpenMP as the multiprocessing API back-end.

Motivation

According to the OO paradigm, classes and consequently objects provide an abstract interface to the underlying internal data or services of the modeled entity or entity class. When it comes to parallel programing we should provide facilities to enable concurrent access to shared resources that are in this case objects. Using plain OpenMP can be satisfactory, however when used extensively the OpenMP pragmas and API function calls introduced can greatly affect the readability and the maintainability of the code. Nevertheless, there can be platforms that use other APIs for handling race conditions. It is obvious that we need to encapsulate these facilities and provide an abstract tool-set instead.

Implementation

The very first building block of such a framework can be a mutex class that provides mutually exclusive access to certain resources. In the world of OpenMP this should look like something similar to the following:

class Mutex {
public:
    Mutex() { omp_init_lock(&_mutex); }
    ~Mutex() { omp_destroy_lock(&_mutex); }
    void lock() { omp_set_lock(&_mutex); }
    void unlock() { omp_unset_lock(&_mutex); }
private:
    omp_lock_t _mutex;
};

This seems already enough for us to make our Java-like “synchronized” statement, however we would like to create a framework that makes usage as easy and safe as possible. In order to get closer to this goal we apply the RAII (Resource Acquisition Is Initialization) design pattern to create our lock class:

class Lock {
public:
    Lock(Mutex& mutex) : _mutex(mutex), _release(false) { _mutex.lock(); }
    ~Lock() { _mutex.unlock(); }
    bool operator() const { return !_release; }
    void release() { _release = true; }
private:
    Mutex& _mutex;
    bool _release;
};

Our goal is to provide an inheritable interface for such objects that needs synchronization. However, this step has to involve severe considerations regarding to the provided interface as we explicitly need to conform to the following requirements:

• The interface shall not expose the interface of the underlying synchronization primitive, in our case the mutex class methods.
• The interface shall be available only to the synchronizable objects but not for the external world as we would like to not just hide the implementation details of our abstract entity but also prevent the users to synchronize our objects as it should be the responsibility of the object itself.
• The interface shall expose methods which are less prone to name collision, for convenience.

If we take care of the presented conventions we end up with an interface similar to the following:

class Synchronizable: protected Mutex {
protected:
	void enterSyncBlock() { this->lock(); }
	void exitSyncBlock() { this->unlock(); }
};

Now we are almost at the finish line. We just need to inherit this class in order to have the needed facilities for an object that needs synchronization. However, using this interface directly is not the most comfortable and safe. If we would like to have a Java-like “synchronized” statement we have to call for additional help. Fortunately, we have our not so well respected C macro language coming to rescue us as we can use it to make some pseudo-language extensions. The simplest way to define our new statement is using the following line:

#define synchronized(obj)  for(Lock obj##_lock = *obj; obj##_lock; obj##_lock.release())

From now, we can really use object synchronization in C++ as easy as in Java, we just need the following syntax in the method of our shared objects:

synchronized(this) {
    // some code that needs synchronization
}

Now it is clearly visible how handy the RAII pattern became in our case. Beside that it is now very straightforward to use this statement it provides additional benefits:

• It makes the code more readable and as a result it is easier to maintain.
• No need to call inconveniently named methods and use lock variables.
• The synchronized code has it’s own scope inside the code.
• It is exception-safe as the mutex is unlocked upon destruction.

Additionally, we can also take advantage of the inherent problem in C++ regarding to multiple inheritance. If we inherit our object from other two synchronized objects then using a simple type casting we can explicitly specify which ancestor we would like to synchronize in a particular block. Also, to ease this we can define our synchronization statement instead of the Java-like one using the following line:

#define synchronized(cls)  for(Lock obj##_lock = *static_cast<cls*>(this); obj##_lock; obj##_lock.release())

In this case we pass the class name instead of the object pointer this. Using this later construct we can easily specify the correct ancestor that we would like to synchronize in case when we deal with multiple inheritance situations. Personally I prefer the later syntax as it is much more customized for C++ use cases.

As from now we don’t need a direct interface for entering and exiting our synchronization block we can simplify our synchronizable interface to the following chunk:

class Synchronizable: protected Mutex {
};

This is enough from now to provide the facilities needed for a synchronization block but still complies to the requirement that we would like to hide the synchronization primitive related details.

Beside this, Jörg came up with the idea today to replace the for loop in our macro with a single if statement. This seems reasonable as we don’t have to sacrifice any scoping and safety related benefits of our framework. This simplifies our lock class to the following:

class Lock {
public:
    Lock(Mutex& mutex) : _mutex(mutex) { _mutex.lock(); }
    ~Lock() { _mutex.unlock(); }
    bool operator() const { return true; }
private:
    Mutex& _mutex;
};

This definition of the lock class is satisfactory if we redefine our synchronized macro to use an if statement instead:

/* Java-like synchronized statement */
#define synchronized(obj)  if (Lock obj##_lock = *obj)
/* alternative synchronized statement to support multiple inheritance */
#define synchronized(cls)  if (Lock obj##_lock = *static_cast<cls*>(this))

Thanks to the useful comments we even managed to further optimize and minimize the support code needed for our new pseudo-language extension.

Conclusion

We have seen an example how one can implement an easy to use synchronizable interface for C++. Also, we’ve provided a concrete implementation that is based on OpenMP. This library is still far from an API that provides all the necessary constructs that one needs for using parallel programming in their C++ projects, however we made our first step and I will recap on the subject in subsequent articles to further extend this framework.

Credits go to Achilleas Margaritis whose article inspired me to write mine and to Jörg for the useful improvement ideas.

Full source code

Language: C++
Platform: cross-platform
Dependency: OpenMP
Download link: omp_sync.h
Comments: In order to use it as it is, you will need a C++ compiler supporting OpenMP like GCC 4.2 or Visual C++ 2008.