The War Is Far From Over

I’ve chosen the title based on the popular article that tries to prove that OpenGL lost the war against Direct3D. To be honest, I didn’t really like the article at all. First, because it compared OpenGL 3 which targeted Shader Model 4.0 hardware and DirectX 11 which targeted Shader Model 5.0 hardware. Besides that, as we will see, the war is really far from over… This article aims to list the most important features introduced by OpenGL 3.x, OpenGL 4.x, Direct3D 10, Direct3D 11 and we will also talk about the promised features of the upcoming Direct3D 11.1 to be fair with DirectX

After I wrote my article about the latest features introduced in OpenGL someone asked me whether I can write an article about the comparison of the hardware features exposed by OpenGL and Direct3D. Instead of a long explanation, I decided to simply create a table of the features introduced by the APIs. Please note that the list focuses on hardware features and does not discuss API feature differences between the two APIs. The list may be far from complete and I’m happy to get feedback about what is missing from the table so that I can extend it. Also there are features for which I did not find whether an equivalent exists in D3D and are marked with a question mark. If anybody can point me to the answer, I would be happy, but I did not find a specification of the HLSL versions.

[1] There is no support for these counters in OpenGL, however they can be implemented with the help of shader atomic counters.
[2] There is no support in Direct3D to use the dedicated atomic counter hardware (supported currently only by AMD GPUs) only by using an append/consume buffer. Though, as atomic counters are the part of UAVs and arbitrary number of UAVs can be attached to a single resource, the same functionality is supported indirectly.
[3] There is read/write buffer and texture support in Direct3D 11, however it is available only in the fragment (pixel) shader. Direct3D 11.1 plans to remove this restriction.
[4] There is no support for texture format casting in OpenGL, conversion, however, can be done by doing a copy preferably using pixel buffer objects.
[5] There is no support for automatic storage of atomic counter values in buffers in Direct3D, however, their value can be manually copied to arbitrary resources.

As a conclusion, I would like to say just one thing: even though there are some features that are not supported by either OpenGL or Direct3D, we really can say that the two APIs are on par with the number of hardware features they expose.

(Sorry in advance for any mistakes, it took quite some time to create this table and I may became too tired at the end)

After the release of the OpenGL 4.1 specification the Khronos Group slowed down the pace a little bit but they didn’t left OpenGL developers without a new specification version for too long as a few weeks ago they’ve released OpenGL 4.2. The new version of the specification brings several API improvements as well as exposes some important pieces of hardware functionality that makes OpenGL 4.x class hardware a great step forward in GPU history. This article aims to present the newly introduced features in the latest version of the OpenGL specification and, as a few months ago I wrote an article about Suggestions for OpenGL 4.2 and beyond, I will write a few words about how does the new specification reflect my forecast.

New features in OpenGL 4.2

OpenGL 4.2 finally filled the holes in the capability matrix of Shader Model 5.0 hardware with some long waited extensions from which some of the functionalities were actually already accessible through cross-vendor and vendor specific extensions. Also, the new version of the specification brings some important API improvement extensions and GLSL constructs that continue the transition to a more easy to use state and shader management.

ARB_texture_compression_bptc

This extension adds the new block compression texture formats called BC7 and BC6H in Direct3D terminology. The extension is actually available for quite some time, since the release of OpenGL 4.0 but now it became core. The formats provide high quality block compression for fixed point RGBA and sRGB textures as well as two floating point texture compression formats for signed and unsigned data.

Traditional block compression methods (as S3TC or RGTC) use the gradients in a block of pixels which works fine for smooth images but does provide poor results in case of sharp edges. BPTC solves the issue by dividing blocks into multiple partitions which are compressed using independent gradients thus providing better overall quality.

When comparing compression efficiency, BPTC has a compression ratio of 3:1 compared to 6:1, 4:1 and 2:1 that are the compression ratios of the S3TC DXT1, S3TC DXT5 and RGTC formats respectively.

ARB_compressed_texture_pixel_storage

This is an interesting extension that solves a problem that I didn’t even know is such a big issue. The extension is designed primarily to support compressed image formats with fixed-size blocks as that of BPTC as an example. The application can use this extension to configure pixel store parameters so that subtexture operations can provide consistent results in all cases.

ARB_texture_storage

This is again an interesting extension that provides API improvement over how texture storage is allocated in classic OpenGL. As we all know, OpenGL was always too ad hoc on resource management, from the point of view of when actual resources are allocated for a particular API primitive. This is especially a problem in case of textures where we potentially talk about large amount of data. In classic OpenGL the driver could not know from the beginning for example whether the application will need mipmaps for the texture or how many levels are required. This could easily result in bad allocation patterns and/or large reallocations. This extension introduces the concept of immutable texture images where all the levels are allocated up-front for a texture object.

ARB_transform_feedback_instanced

This extension extends the so called “AutoDraw” feature by providing instanced “AutoDraw”. This means that geometry captured using transform feedback can be rendered multiple time using geometry instancing. This is actually a feature that even D3D11 does not provide and being such, I didn’t even think that hardware supports it, even though I think the list usage patterns of the extensions is most probably pretty narrow.

ARB_base_instance

This extension is actually the feature I called ARB_instanced_arrays2 in my suggestion list. The extension provides three new draw commands, one is kind of illy named as DrawElementsInstancedBaseVertexBaseInstance, even though this command can be called the “basic” indexed draw commands that specifies all parameters. Also, the parameter list of the indirect indexed draw command is extended with the base instance parameter. Fortunately, however, the ARB chosen to add new commands rather than a SetBaseInstance-style state specifier command to introduce the new concept. Funnily this feature was missing for a long time as, as far as I know, it is supported by all GPUs capable of doing instanced drawing, and is available in D3D as well.

ARB_shader_image_load_store

This is where things get start really interesting. This new extension is the ARBified version of the extension EXT_shader_image_load_store which fortunately didn’t make it into core in its current form.

The extension provides GLSL built-in functions allowing shaders to load from, store to, and perform atomic read-modify-write operations to a single level of a texture called an image from any shader stage. Also, the extension indirectly enables the same set of operations for buffer objects by using buffer textures. This enables developers to implement more sophisticated algorithms using shaders that require more complex data structures than just plain arrays.

This, together with atomic counters that we will talk about later, enables the possibility to implement append/consume buffers and rendering techniques like AMD’s Order-Independent Transparency (OIT) algorithm as presented at GDC10.

As the introduction of the new write operations to fragment shaders besides the traditional framebuffer writes makes the execution of the shader have side effects and thus sensitive to whether early-Z is used or not by the hardware, so the extension also provides a mechanism to force or disable early-Z in the fragment shader.

A similar issue is in case of vertex shaders as the post-transform cache may be no longer valid in case of certain usage patterns of load/store images so, based on how smart the shader compiler is, the post-transform cache could be easily disabled in case a vertex shader uses load/store images resulting in downgraded performance, so care must be taken when using read/write images in vertex shaders as OpenGL does not have any mechanism to help these issues (but I actually have a proposal that I’ll talk about in a future article).

The API of this extension is greatly improved compared to the EXT version, especially when dealing with various texture image formats. The extension also provides a future-proof DSA-style API. Further, the ARB version of the extension supports loads from any texture format and corrected some specification bugs of the EXT version.

From hardware implementation point of view, it must be noted that in case a shader contains atomic operations applied to a particular read/write image the driver uses a different hardware path, as required by atomic read-modify-writes so that care must be taken to use atomic operations only when necessary. Also note that this decision is made statically at compile time by the driver so even a single atomic operation in an unlikely taken branch will result it degraded performance. This is another reason why to use atomic counters to implement append/consume buffers instead of using read/write image atomics.

ARB_shader_atomic_counters

This the other long waited feature that I also suggested and was still missing from OpenGL but was available in D3D11. The specification was actually ongoing for a long time now (about a year) and it even appeared for a while in AMD’s OpenGL drivers sometimes as EXT, sometimes as ARB extension. The extension provides API to access a number of hardware atomic counters that provide efficient counter operations on a GPU global scale. Atomic counters come handy in many cases like append/consume buffers or indirect draw buffer construction.

The extension provides access to these atomic counters from GLSL and also makes it possible to back them up with buffer objects so after OpenGL draw calls the value of the counters is preserved in these buffers for later use.

The OpenGL implementation is superior compared to D3D’s as it provides access to atomic counters from all shader stages, with caveats of course as, it was mentioned in the previous section, the side effects made possible with read/write images and atomic counters require special care in case of fragment and vertex shaders as they may result in invalid rendering and/or lower performance.

On hardware vendor implementations, it must be noted that atomic counters are much, much more faster than read/write image atomics, at least on AMD hardware which has dedicated hardware for atomic counters. On NVIDIA hardware, though, it seems that there is no different hardware path for atomic counters as their performance is roughly the same as in case of read/write image atomics.

The dedicated hardware implementation of atomic counters, however, comes with a trade-off as the number of atomic counters is severely limited on AMD hardware, but one can still use read/write image atomics if ran out of atomic counters.

ARB_conservative_depth

This is another extension I’ve suggested and that fills another functionality hole compared to D3D11. The extension is actually an ARBified version of AMD_conservative_depth that extends the application developer’s control over eary depth and stencil tests. ARB_shader_image_load_store  already provides a way to force or disable eary-Z and this extension provides further modes that provide a hint to the driver about how depth is modified in a fragment shader that outputs depth. This passes enough information to the GL implementation to activate some early depth test optimizations safely while still preserving the ability to account the final depth value in the depth test.

The extension exposes the new capability in the form of fragment shader input layout qualifiers called “depth_any”, “depth_greater”, “depth_less” and “depth_unchanged”. The interesting ones are the one that assume a greater or less depth value as output and provide the ability to early reject groups of fragments using Hi-Z and early-Z even when depth is modified. This technique can greatly improve the rendering performance of volumetric particles, decals and billboards.

As far as I can tell, though, the extension provides performance benefits only the AMD hardware currently as NVIDIA hardware does not have such functionality thus using the extension would still force NVIDIA GPUs to disable early-Z in case the fragment shader outputs a depth value, but future hardware may change this.

ARB_shading_language_420pack

This is a strangely named extension that provides a lot of improvements to GLSL. These are mostly API improvements only, but have a great value when looking at source code maintainability and resource management.

I think the most useful addition of the extension is the “binding” layout qualifier that I referred to as ARB_explicit_sampler_location and ARB_explicit_uniform_block_index in my suggestion list. This enables shader writers to explicitly bind a uniform block binding index to a uniform block as well as explicitly bind sampler, texture and image binding points to a sampler or image variable.

Besides that, the extension adds other minor improvements, like implicit conversion of return values of functions, UTF-8 character set support, C-style initializer list support and scalar swizzle operators.

ARB_internalformat_query

This is another kind of strangely named extension that was meant to provide the possibility to query information about the internal format of textures, however, it actually failed it as it provides only the ability to query the maximum number of samples available for different texture formats.

The extension was ambitious as it planned to provide internal format information like the ability to query the actual internal format used, whether the format is renderable, accessible in a particular shader stage, whether it can be used as read/write image, and even to provide performance hint about using a particular texture internal format. Unfortunately all these were left for a future extension.

ARB_map_buffer_alignment

This is the last new extension introduced in OpenGL 4.2 that trivially adds the requirement to the pointer returned by buffer mapping commands that they provide a minimum of 64 byte alignment to support processing of the data directly with special CPU instructions like SSE or AVX. This can provide further performance increase when client is modifying buffer data.

Conclusion

OpenGL 4.2 again proven that OpenGL is not dead, but in fact plans to be again the ultimate choice of 3D API by pushing the exposed hardware capabilities over the line set by D3D11. When thinking about the list of expected extensions I presented in my earlier article, Suggestions for OpenGL 4.2 and beyond we can see that OpenGL 4.2 fulfilled all my expectations and even my wish list was partly fulfilled, but here’s the list for a better overview:

My expectations for OpenGL 4.2:

GL_EXT_shader_image_load_store
- added in the form of GL_ARB_shader_image_load_store

GL_ARB_shader_atomic_counters
- added as is

GL_ARB_instanced_arrays2
- added in the form of GL_ARB_base_instance

GL_ARB_explicit_sampler_location
- added in the form of GL_ARB_shading_language_420pack

GL_ARB_explicit_uniform_block_index
- added in the form of GL_ARB_shading_language_420pack

My personal wish-list for OpenGL 4.2:

GL_ARB_draw_indirect2
- still missing, though partly available though GL_AMD_multi_draw_indirect

GL_ARB_direct_state_access
- still missing, however, there is hope that it will be included in the next release where the ARB plans to rewrite the whole structure of the core specification

GL_NV_texture_barrier
- not in core but it is implicitly subsumed by GL_ARB_shader_image_load_store, they say

GL_AMD_conservative_depth
- added in the form of GL_ARB_conservative_depth, despite lack of NVIDIA support

GL_ARB_texture_gather_lod
- still missing, because of lack of supporting hardware

GL_NV_copy_image
- still missing, even though it could be a good API improvement

GL_EXT_texture_filter_anisotropic
- still missing, as I was informed, because of patent issues

GL_ARB_shader_stencil_export
- still missing, most probably because of lack of NVIDIA hardware support

GL_AMD_depth_clamp_separate
- still missing, most probably because of lack of NVIDIA hardware support

GL_AMD_transform_feedback3_lines_triangles
- still missing, most probably because of lack of NVIDIA hardware support

Frei-Chen edge detector

In this article, I would like to present you an edge detection algorithm that shares similar performance characteristics like the well-known Sobel operator but provides slightly better edge detection and can be seamlessly extended with little to no performance overhead to also detect corners alongside with edges. The algorithm works on a 3×3 texel footprint similarly like the Sobel filter but applies a total of nine convolution masks over the image that can be used for either edge or corner detection. The article presents the mathematical background that is needed to implement the edge detector and provides a reference implementation written in C/C++ using OpenGL that showcases both the Frei-Chen and the Sobel edge detection filter applied to the same image.

I met with the algorithm during my computer graphics studies when one of my homeworks was to implement the Frei-Chen edge detector. As I already mentioned it in an earlier post, I am willing to provide source code for more basic graphics algorithms after seeing the success of my former post about the Gaussian blur filter. This one is a very similarly basic article, taking in consideration it shows only how to apply a particular convolution filter based algorithm on a still image, while the possibilities this edge detection algorithm brings is a more complex topic that is out of the scope of this article.

As the provided reference implementation also showcases applying the Sobel operator on an image, I would like to present that first and then continue with the presentation of the Frei-Chen masking set. Those who are already well familiar with edge detection and the Sobel operator can skip the following two sections.

Edge detection

Before getting deep into how to implement edge detectors, let’s first talk about what is an edge detector and why we need it.

In general, edge detection is one of the most fundamental image processing tools, particularly used in the areas of feature detection and feature extraction. The aim of the technique is to identify points of a digital image at which the intensity changes sharply. The reason of these intensity changes can be either discontinuities in depth, surface orientation, lighting condition changes and many other factors. In the ideal case, the result of applying an edge detector to an image leads us to a set of connected lines or curves that indicate the boundaries of objects.

Not going that far, what an edge detector gives us from the very beginning is a gray-scale image where each pixel intensity tries to approximate the likelihood of whether that pixel belongs to an object boundary. How well a particular algorithm can detect such pixels depends on many factors and usually it is better to try multiple edge detectors in order to choose one that fits most for the particular use case.

After we got this gray-scale image we usually have to define a threshold value that will be used as an acceptance criteria for edge pixels. If the intensity value previously calculated is above this threshold then we accept the pixel as an edge otherwise we don’t. This part is the so called binarization stage. Additionally, subsequent image processing algorithms can be used to further interpret the edge image.

In computer graphics, edge detection is usually used to implement various image decoration algorithms. Maybe the most popular applications of edge detectors nowadays are non-photorealistic rendering (NPR) and screen-space anti-aliasing techniques.

Sobel filter

The Sobel edge detection filter works on a 3×3 texel footprint and applies two convolution masks to the image that are intended to detect horizontal and vertical gradients of the image. The filter weights can be seen in on the figure below:

These masks are applied to the intensities gathered from the 3×3 footprint of the image and then are accumulated to produce the final gradient value in the following way:

The actual algorithm can be seen in the accompanying demo that provides a GLSL based implementation. The algorithm is defined to work on one channel image, however it can be easily extended to be applied either separately on a usual three-channel RGB image or by first calculating a gray-scale value based on the color component values. The former is more computationally intensive but usually provides better results by defining the threshold criteria in a way that a pixel is accepted as boundary point if the gradient value is larger than the threshold for either of the color channels. The reference implementation, however is based on the later approach for the sake of simplicity so for each pixel first an intensity value is calculated simply by taking the length of the vector comprised of the RGB components.

Frei-Chen filter

The Frei-Chen edge detector also works on a 3×3 texel footprint but applies a total of nine convolution masks to the image. Frei-Chen masks are unique masks, which contain all of the basis vectors. This implies that a 3×3 image area is represented with the weighted sum of nine Frei-Chen masks that can be seen below:

The first four Frei-Chen masks above are used for edges, the next four are used for lines and the last mask is used to compute averages. For edge detection, appropriate masks are chosen and the image is projected onto it. The projection equation is given below:

When we are using the Frei-Chen masks for edge detection we are searching for the cosine defined above and we use the first four masks as the elements of importance so the first sum above goes from one to four.

The application of a threshold and applying the filter to multi-channel images works exactly the same way like in case of the Sobel filter. Similarly, the reference implementation applies the filter on the image as it would be a single-channel image by first calculating the intensity value for each texel in the same fashion like with the previously presented filter.

Comparison

Based on my experience, the Frei-Chen edge detector looks better than the Sobel filter as it is less sensitive to noise and is able to detect edges that have small gradients and thus are not found by the basic Sobel filter. For a comparison, you can check the figure below:

Comparison of edge detectors: original image (left), Sobel filter (middle), Frei-Chen filter (right).

The reason why the Frei-Chen edge detector seems to work better is because its construction includes a normalization factor as well as other factors that are meant to exclude all other features except edges. A normalization factor can be also added to the Sobel filter by having a third mask that is equivalent with the ninth Frei-Chen mask and is used to normalize the gradients. This could help in reducing the number of undetected edges and the amount of noise that arises from the fact that the Sobel filter calculates absolute gradients rather than relative ones.

From performance point of view, the Frei-Chen edge detector is much more heavyweight as it uses nine masks instead of two, however, in practice, the performance difference between the two is much less taking in consideration that both use the same sized texel footprint and the computational performance of today’s GPUs is usually much higher than their texture fetching performance.

Conclusion

We managed to present an alternative algorithm for the Sobel filter in the form of the Frei-Chen edge detector that, even though having little impact on the performance compared to the Sobel operator, provides better edge detection quality. Having little to no difference in the way how the input data has to be organized and how the result is output, the Frei-Chen edge detector can be easily used as a drop-in replacement for implementations that used the Sobel filter before.

Source code and Win32 binary can be acquired in the downloads section.

I would like to encourage those who read this article to add the Frei-Chen edge detector into their software for making a comparison about whether it yields to better results than the Sobel filter for applications that rely on the output of the edge detection filter. I would be interested how the filter works in real-life computer graphics scenarios.

Thanks in advance and hope you enjoyed the article!

The Khronos Group did a great job in the last few years to once again prove that OpenGL is still in game and that it can become the ultimate graphics API of choice, if it is not that already. However, we must note that it is not quite yet true that OpenGL 4.1 is a superset of its competitor, DirectX 11. We still have some holes that still have to be filled and I think the ARB should not stop just there as there is much more potential in the current hardware architectures than that is currently exposed by any graphics API so establishing the future of OpenGL should start by going one step further than DX11. In this article I would like to present my vision of items of importance that should be included in the next revision of the specification and how I see the future of OpenGL.

Since the original OpenGL Longs Peak announcement, graphics developers were really excited to get their hands on the completely revised OpenGL 3 specification. Still, due to severe backward compatibility and portability issues the original plan seemed to be failed and developers expressed their great sense of disappointment about the ARB’s decision to choose rather a more evolutionary move away from the legacy API instead of the radical rewrite, the Khronos Group has proved that the decision was not necessarily bad for OpenGL and in fact we got now a pretty powerful API, even though the coexistence of the legacy and the new design greatly increased the complexity of the specification.

What we have now is an API that can really compete with DirectX 11 but I strongly believe that this is not the end of the story yet as we still have a lot of things to do in ahead of us. I mean this both from point of view of exposing more hardware capabilities as well as streamlining the API language itself to increase the productivity of the developers who use it. My plan is to target both of these issues in this article, also trying to focus on hardware functionalities that are not even exposed by other graphics APIs yet.

Exposing more hardware capabilities

In this chapter of the article I will talk about some familiar and some not so familiar hardware features and corresponding OpenGL extensions that should be included in the next revision of the specification in order to be able to confidently say that OpenGL is a strict superset of the competing graphics APIs. The extensions listed here are not in any particular priority order, they are just listed in a way that ease the discussion about their functionality.

GL_EXT_shader_image_load_store

This extension provides GLSL built-in functions allowing shaders to load from, store to, and perform atomic read-modify-write operations to a single level of a texture from any shader stage. Also, the extension also indirectly enables the same operations for buffer objects by using texture buffers. This enables developers to implement more sophisticated algorithms using shaders that require more complex data structures than just plain arrays.

An example use case can be the implementation of Order-Independent Transparency (OIT) using fragment linked lists as presented by AMD at GDC10. Of course, there are a lot of other techniques that could benefit from hardware accelerated random access images (called UAV textures/buffers in DX11 terminology) including algorithms related to global illumination, ray tracing, and my personal favorite: scene management.

As the introduction of new write operations to fragment shaders besides the traditional framebuffer writes makes the execution of the shaders sensitive to whether early-Z is used or not by the hardware, the extension also introduces a new fragment shader input layout qualifier called “early_fragment_tests” to force OpenGL to use early depth and stencil test. Otherwise the specification language is valid stating that the depth and stencil tests are performed after fragment shader execution.

Finally, the extension enables some form of control over the order of image loads, stores, and atomics relative to other pipeline operations accessing the same memory region both using the OpenGL API and from within shaders.

The API itself provides a DSA-style binding mechanism that enables binding to so called “image units” that are separate from that of texture image units. In the same style, the specification language and GLSL refers to the introduced read-write textures with the term “image”.

In my opinion this is one of the most important extensions that should be made core with OpenGL 4.2 and I’m pretty sure this will actually happen.

GL_NV_texture_barrier

This extension relaxes the restrictions of OpenGL on rendering to a currently bound texture and provides a mechanism to avoid read-after-write problems. More precisely, the extension allows rendering to a currently bound texture in the following cases:

• If the reads and writes are from/to disjoint sets of texels (after accounting for texture filtering rules) so it should work unless the drawn areas overlap, or
• If there is only a single read and write of each texel, and the read is in the fragment shader invocation that writes the same texel (e.g. using texelFetch2D).

Some of these situations were already supported implicitly like rendering to a texture level and fetching from another texture level. But the extension goes further and provides an API function to put an explicit barrier between draw calls to ensure proper rendering.

The extension can be used to accomplish a limited form of programmable blending and can eliminate the need of any image or buffer data copy in case we can live with the restrictions mentioned above.

One may ask why we need this extension if we have the GL_EXT_shader_image_load_store extension as this one is just a subset of the functionality provided by that. The answer is simple: performance. While read-write textures can mimic the same functionality they usually use different hardware paths that are slower than regular read-only texture accesses. So it would be a definite benefit to having also this extension in core OpenGL.

GL_ARB_shader_atomic_counters

This extension does not have public specifications yet, however it can be found in the extension lists of the latest Catalyst driver releases sometimes with EXT, sometimes with ARB prefix. The extension itself provides API to access a number of hardware atomic counters that provide efficient counter operations on a GPU global scale.

Atomic counters come handy when one has to read or write individual elements of a buffer or texture. As an example, this extension is needed to be able to efficiently implement the OIT algorithm mentioned earlier as, when constructing the fragment linked list, we need to have unique offsets to the linked list buffer. This unique offset can be, of course, acquired by using atomic read-modify-write operations but those perform much slower than hardware atomic counters.

Besides the mentioned example, atomic counters are useful in many algorithms from many domains, one important use case is to perform feedback operations similar to that provided by transform feedback. Such feedback operations can be used to perform various scene management or culling mechanisms.

The extension provides access to these atomic counters from GLSL and also makes it possible to back them up with buffer objects so after OpenGL draw calls the value of the counters is conserved in these buffers for subsequent use.

GL_AMD_conservative_depth

Early depth test is a common optimization for hardware accelerated graphics that can skip the evaluation of fragment shaders for fragments that end up being discarded because they don’t pass the depth test. The problem is that in case the fragment shader modifies the depth value of the fragment then the early depth test is disabled. One can force early depth test with the functionality introduced by the extension GL_EXT_shader_image_load_store but that can lead to some rendering artifacts as the modified depth value output by the fragment shader is not taken into account.

This extension allows the application to pass enough information to the GL implementation to activate some early depth test optimizations safely while still preserving the ability to account the final depth value in the depth test. In order to solve this, the extension introduces four new fragment shader input layout qualifiers called “depth_unchanged, “depth_any”, “depth_greater” and “depth_less”. The most interesting ones are the latest two that provide the ability to do early-Z and hierarchical-Z tests from one direction to discard some groups of fragments and still allow the fragment shader to safely modify the depth value.

This technique comes very handy in case of rendering volumetric particles, decals or billboards. Without this extension one have to sacrifice the possibility to do early rejection of fragments in order to be able to create the volumetric primitives mentioned.

As far as I know this feature is also present in DirectX 11 so it should be a must for OpenGL 4.x also. As the extension is an AMD one, I don’t know whether NVIDIA GPUs do support anything like this in hardware but even if not, they can simply ignore the new layout qualifiers and do late depth test instead. Of course, it would result in lower performance but if only functionality is concerned it should be just okay.

GL_ARB_instanced_arrays2

OpenGL provides two means to perform geometry instancing via the extensions GL_ARB_draw_instanced and GL_ARB_instanced_arrays. While this (yet non-existent) extension would extend both, it is more relevant in case of the extension mentioned later so I named it accordingly.

The extension should trivially add the possibility to specify a “first instance” parameter for the instanced draw commands. Whether this is accomplished by introducing new variants of the glDrawElement* and glDrawArrays* draw commands or having a separate command for specifying the new parameter is up to the ARB. The extension should also interact with GL_ARB_draw_indirect which already mentions the lack of the parameter in GL and reserved already a field in the indirect draw command structure for specifying the “first instance” parameter.

This extension itself would be much more a bug fix rather than a completely new feature as this functionality should have been already exposed at the first time instancing was introduced to OpenGL.

GL_ARB_draw_indirect2

This is one of the extensions I would be the most happy to see in the next release of the OpenGL specification. It would be a functional addition to the GL_ARB_draw_indirect extension that currently only allows the execution of a single instanced draw command that sources its parameter from a buffer object.

The new extension would add a new buffer binding point called e.g. GL_DRAW_INDIRECT_PRIMITIVE_COUNT that would specify the source of the “primcount” parameter to the following newly introduced draw commands:

    void MultiDrawArraysIndirect( enum mode, sizei stride,
                                  const void *indirect,
                                  const void *primcount );
    void MultiDrawElementsIndirect( enum mode, enum type, sizei stride,
                                    const void *indirect,
                                    const void *primcount );

This would not just allow for executing multiple indirect draw commands at once, without further CPU action, but also would source the “primcount” parameter from a buffer object thus if the draw commands are generated using transform feedback, read-write buffers or OpenCL (e.g. based on some GPU based scene management algorithm) then the application does not have to use asynchronous queries or other means that may introduce sync points in the rendering to be able to feed the “primcount” parameter.

Some people said that this is quite a futuristic feature to expect and most probably such functionality will be available only on newer generation of GPUs and maybe with OpenGL 5. I was not that pessimistic so I decided to raise my question to the relevant ARB members of NVIDIA and AMD. While I did not receive any answer from NVIDIA, I did received some good news from AMD as they said that this functionality can be implemented for Shader Model 5.0 level hardware.

What this extension would give developers is a way to efficiently implement GPU based scene management where the GPU bakes together all the rendering commands for the current frame using atomic counters and buffer writes, and the CPU just have to issue a few or maybe just a single MultiDraw*Indirect command to render the whole scene. But of course, the feature can increase draw command throughput also in case of CPU based scene management.

So my message to the Khronos Group is please, start working on such an extension as this would not just make developers happy, but you can also strengthen OpenGL’s position in the industry by putting something into the specification that even DirectX 11 cannot do.

GL_AMD_transform_feedback3_lines_triangles

OpenGL 4.0 introduced the extension GL_ARB_transform_feedback3 that further extended the transform feedback capabilities provided by earlier extensions to allow ouput to separate vertex streams. However there is one caveat: separate vertex streams are only supported for point primitives.

This new AMD extension does nothing more than just simply removes that restrictions for separate output streams allowing the same set of primitive types to be used with multiple transform feedback streams as with a single stream as long as the primitive types are the same for all output streams.

Limiting the possible output primitive types for transform feedback into multiple streams should not be a problem unless you want also to rasterize some triangles at the same time you output. Without relaxing this restriction can do this only by issuing two separate draw commands that incurs a performance hit.

I don’t know if the restriction is present in the ARB extension because NVIDIA does not support this in hardware but if this is not the case then I think this extension should be included in the next release of the specification. Otherwise, please NVIDIA include this feature in your next GPU generation.

GL_NV_copy_image

OpenGL 3.1 already introduced a method to provide GPU accelerated copy of buffer data. This NVIDIA extension provides a similar functionality that can be used to execute efficient image data transfer between image objects (i.e. textures and renderbuffers).

While there are already methods to perform image data copies between textures e.g. using the GL_EXT_framebuffer_blit extension promoted to core with OpenGL 3.0 these require expensive framebuffer object operations and they also lack direct support for transferring 3D image data.

This extension simply introduces a single command that allows such image data copies for every type of textures (including cube maps, 3D textures and array textures) without the need to bind the image objects or otherwise configure the rendering.

GL_AMD_depth_clamp_separate

The extension GL_ARB_depth_clamp promoted to core with OpenGL 3.2 introduced the ability to control the clamping of the depth value for both the near and far clip planes. This eliminates artifacts like seeing inside an object happening when the object’s geometry is clipped by the near clip plane.

This new extension provides a mean for the application to enable depth clamp separately for the near and the far clip plane. This increases the flexibility of depth clamping and can save some fill-rate in certain situations.

GL_EXT_texture_filter_anisotropic

I don’t think that I have to talk too much about this extension as it should be familiar to all of you. It simply enables the possibility to use anisotropic filtering on a per-texture basis. I really wonder how this extension didn’t make its way into core as it is supported by hardware since more than a decade.

I know that the extension itself is supported by all relevant graphics driver vendors but really, why we can’t just simply include it in the core specification?

GL_ARB_texture_gather_lod

This is another yet non-existent extension that would extend GL_ARB_texture_gather by adding GLSL built-in functions called textureGatherLod that would allow gathered fetches with explicit LOD. I’m not sure if these functions are missing from the specification because of lack of hardware support or just because the ARB thought they might not be of any use. Anyway, if the hardware supports it then OpenGL should expose it to developers as there are certain situations when one has to use explicit LOD and could benefit from the increased fetching performance enabled by gathered fetches.

GL_ARB_shader_stencil_export

This extension was published at the time the OpenGL 4.1 specification came out and provides the ability for the fragment shader to output the stencil reference value that was otherwise configurable only using API calls. This enables a great level of flexibility to existing and future stencil buffer based algorithms making it possible also to directly write independent values to the stencil buffer on a per-fragment basis.

The predecessor of the extension is GL_AMD_shader_stencil_export and as such it indicates that maybe it is only supported in hardware on AMD GPUs. However, if this is not the case and NVIDIA could support this also then I think it worths to promote this feature also to core OpenGL.

Streamlining the API

After discussing the long list of functional features that would be nice to be included into the next release of OpenGL let’s focus on the API improvement extensions and ideas that are necessary to improve the usability of the API itself. Actually this part could go way longer than I’ll discuss because as we get more and more features to OpenGL, developers struggle with the increased complexity of the API. I’ll try to focus on the most crucial issues.

GL_EXT_direct_state_access

This is the extension what all OpenGL developers are waiting for a long time now. Direct state access eliminates the OpenGL API’s stupid “bind-to-modify” nature.

For a very long time the only vendor supporting the extension was NVIDIA. Fortunately, since Catalyst 10.7 AMD also exposes the extension to developers. Still, I have one problem: this extension is very poorly designed.

The main problem with the extension is that the functions were designed in a way that a naive implementation could be done by simply using “bind-to-modify” under the hood. That’s what resulted in crazy API functions like MultiTexParameter* and friends. Also, enabling DSA for all of the deprecated functionalities would result in an explosion of the API specification and as a consequence it would result in bloated specification language. Finally, I would also like to object somewhat the lack of creativity of the contributors regarding to the awkward naming conventions present in the current DSA extension.

In my opinion the Khronos Group has to address the issue by creating a new ARB version of the DSA extension that focuses strictly on core functionalities, throwing away DSA support for deprecated features (if somebody needs to use deprecated features they can still use the EXT version) and provide a naming convention that fits much better into the current API language.

Anyway, I completely agree with the other developers out there and scream for DSA. I think the Khronos Group has to eliminate the problem of the “bind-to-modify” semantics as soon as possible otherwise, even though the core specification exposes more and more hardware features, developers will not be attracted to use OpenGL.

GL_ARB_explicit_sampler_location

The ARB moved in the right direction when they introduced the GL_ARB_explicit_attrib_location extension by eliminating the need to use dummy API calls to bind vertex attributes and output buffers to shader variables but they should not stop here. One of the most important addition could be adding a similar language syntax to GLSL that would allow us to bind sampler uniforms to texture image units. Obviously, the same goes for read-write images if GL_EXT_shader_image_load_store is included.

GL_ARB_explicit_uniform_block_index

Similar to the previous request, uniform block indices should be as well explicitly specifiable in the shaders themselves. This extension would add exactly such functionality. The implementation is also straightforward: just a simple uniform block layout qualifier has to be added.

Other API clarifications

Besides the major issues the current specification language also has some bugs and unclear parts that should be addressed as well:

• Program pipeline objects are created by binding the object name which is not in align with the rest of the API language.
• No language is about whether program pipeline objects are shared among contexts or not which suggests that they aren’t which is not in align with the fact that program and shader objects are shared.

Most probably there are a lot more issues with the specification language but for now just these came into my mind. Maybe some of you can extend the list with tons of other specification mistakes.

OpenGL 4.2 and beyond

While my feature requests cover most of the needed functionality that should be included in the next revision of the OpenGL specification, there are a lot of other things that could be very useful for developers but are very unlikely to get their way into the specification any soon. I will talk about these features in this section of the article as these raise much more questions than just to be able to simply include it in OpenGL 4.2.

Affinity contexts

We have multi-GPU designs like SLI and CrossFire for a long time now. Fortunately, we have also vendor specific extensions to create affinity contexts that are associated with a single GPU of a multi-GPU configuration. We have WGL_AMD_gpu_association and WGL_NV_gpu_affinity for Windows and GLX_AMD_gpu_association on GLX based platforms. I have just two problems with this:

• First, these are vendor specific extensions.
• Second, NVIDIA exposes its affinity context support only on Windows and just for their professional cards, leaving consumer hardware owners without affinity context support.

I would be pleased to see in the future extensions like WGL_ARB_gpu_affinity_context and GLX_ARB_gpu_affinity_context that will be supported by both NVIDIA and AMD, and that are supported on both professional and consumer hardware.

Command buffers

I would like to see something similar in OpenGL that what we have in OpenCL. Having several separate command buffers for a single OpenGL context can have its performance benefits as some of the implicit sync points that are otherwise present in OpenGL draw commands could be eliminated. Another solution would be to use simply multiple GL contexts but it is much more complicated and context switches are quite heavy-weight operations. This would be something like how framebuffer objects replaced pbuffers.

Also this could go that far as we can encapsulate state manipulation data into command buffers in a similar way how display lists allowed this in many cases just in a more efficient and hardware centric manner.

Immutable state objects

Another thing strongly related to the previous idea would be immutable state objects. If state management data could not be efficiently stored in such a command buffer we could use instead immutable state objects that would be very similar in nature to display lists that are hiding the underlying representation of the commands.

Display lists are deprecated and I don’t think it was a wrong decision. It made the API language complex and you’ve never knew which command compiles into display lists and how. I remember the time I was making an OpenGL app on my GeForce2 and used DrawElements calls inside display lists that referenced buffer object data. Funnily it was working on NVIDIA hardware, even though the specification says otherwise, and I was wondering why I my app crashes on ATI cards.

Anyway, display lists are gone, but we need some complex state objects that could fill those holes that were left after them.

More callbacks

I was very happy to see the appearance of an extension that introduced the callback concept into OpenGL (GL_AMD_debug_output). Since that, the functionality was promoted to an ARB extension meaning that the ARB has accepted the fact that we need callbacks.

What I would like to see in the future is more OpenGL callbacks. One of the most trivial things I can think of are asynchronous queries. It would so much easier if we would be able to receive a callback from OpenGL when the results of our asynchronous queries are available, rather than having to manually poll it for result in various phases of the rendering.

Actually, I could imagine callbacks for every rendering command issued that will be called by the driver as soon as the actual rendering is complete on the GPU side.

Programmable blending

This is one another thing that developers are screaming for. Fortunately now we have indirect methods to solve most of the issues of programmable blending via the extensions GL_EXT_shader_image_load_store and GL_NV_texture_barrier, however a more general solution would be welcomed.

I don’t know whether this would be actually possible on current hardware but if not, then this is a message to hardware vendors to solve the issue in the near future.

Summary

We’ve seen that even though OpenGL is on track and the Khronos Group is keeping up the pace with its competitors, still there are lots of room for improvement regarding to the OpenGL specification from both functional point of view as well as from API design point of view.

I would like to end the article with a summary of what I expect to be part of the OpenGL 4.2 specification and my personal wish-list beyond those in some kind of priority order.

My expectations for OpenGL 4.2:

• GL_EXT_shader_image_load_store
• GL_ARB_shader_atomic_counters
• GL_ARB_instanced_arrays2
• GL_ARB_explicit_sampler_location
• GL_ARB_explicit_uniform_block_index

My personal wish-list for OpenGL 4.2:

• GL_ARB_draw_indirect2
• GL_ARB_direct_state_access
• GL_NV_texture_barrier
• GL_AMD_conservative_depth
• GL_ARB_texture_gather_lod
• GL_NV_copy_image
• GL_EXT_texture_filter_anisotropic
• GL_ARB_shader_stencil_export
• GL_AMD_depth_clamp_separate
• GL_AMD_transform_feedback3_lines_triangles

OpenGL 4.0 - Mountains demo

Hierarchical-Z is a well known and standard feature of modern GPUs that allows them to speed up depth testing by rejecting large group of incoming fragments using a reduced and compressed version of the depth buffer that resides in on-chip memory. The technique presented in this article uses the same basic idea to allow batched occlusion culling for large amount of individual objects using a geometry shader without the need of any CPU intervention that is unavoidable using traditional occlusion queries. The article also provides a reference implementation in the form of the OpenGL 4.0 Mountains demo that uses the technique for culling thousands of object instances.

Introduction

Occlusion culling is a visibility determination algorithm that is used to identify those objects that did reside in the view volume but still aren’t visible on the screen due to occlusion. That means they are hidden by such objects that reside closer to the camera.

For several generations now GPUs allow hardware accelerated methods to perform occlusion culling in the form of occlusion queries. OpenGL provides the functionality via the extension ARB_occlusion_query. Occlusion queries are very simple: when you draw an object with occlusion query enabled the query returns the number of samples that passed the depth test (or simply return true or false based on whether any samples of the objects passed the depth test or not as it is provided by the OpenGL extension ARB_occlusion_query2).

So actually performing occlusion culling using occlusion queries means simply the following:

1. Draw the object while occlusion query is enabled.
2. If the query result is that the object is visible then draw the object.

At first, this may sound stupid as you have to draw the object in order to tell whether it is visible or not. While in this form it really sounds silly, in practice occlusion query can save a lot of work for the GPU. Think about you have a complex object with several thousands of triangles. If you would like to determine the visibility of it using occlusion query you would simply render e.g. the bounding box of the object and if the bounding box is visible (occlusion query returns that some samples have passed) then it means the object itself is most probably visible. This way you can save the GPU from the unnecessary processing of large amount of geometry.

I have to mention here that I intentionally used the expression “most probably visible” as occlusion queries provide just a conservative estimate on whether the object is visible or not rather than an exact result. This is because the bounding box occupies a different (larger) portion of the screen than the original geometry. So what we expect from an occlusion culling algorithm is to give one of the following results: the object is not visible or the object is most probably visible. The bigger this probability is the better the occlusion culling effectiveness is.

While we would always want an occlusion culling algorithm to be as effective as possible usually we have to make a trade-off between effectiveness and efficiency. In the above example if we would like to have 100% effectiveness then we would have to draw the whole object and that would defeat most of the goals of occlusion culling. The algorithm presented in this article is somewhat even more conservative but enables the use of occlusion culling for much larger datasets.

Motivation

While hardware accelerated occlusion query is a powerful tool to use in visibility determination it puts a quite reasonable burden on the application to manage the occlusion queries and to draw the objects based on the results when they are available (taking in consideration the asynchronous nature of occlusion queries). The most naive use of occlusion queries would be to execute the query right before we have to draw the object. While this seems like a feasible idea, it does not perform well in practice as the CPU has to be stalled until the result of the query is available and that involves also empty cycles on the GPU as well thus results in unacceptable performance. In order to resolve this, the application has to fill the time between the query execution and the drawing of the object based on the query result. While there are techniques to accomplish this, it definitely comes at a cost as the implementation becomes more complex.

The aforementioned problem is somewhat resolved by using conditional rendering introduced in OpenGL 3 (NV_conditional_render extension). However, this extension does nothing just in case the results of the query are not available yet then we simply draw the object no matter if it is visible or not. This can avoid the stalling of the rendering pipeline and can be done in software if the extension is not available, however, it somewhat defeats the purpose of occlusion culling.

Another deficit when using occlusion queries is that there is still need for CPU intervention in order to make a decision about the visibility of the object. For today’s hardware where proper batching is one of the most crucial aspects of the renderer such an approach is rather ineffective.

The occlusion culling technique presented in this article solves both these issues by providing an implementation that is very simple to integrate into any renderer, does put little to no burden on the renderer and makes decision about the visibility of objects entirely on the GPU.

The algorithm

As in case of many other GPU based culling algorithm presented by me and others, the hierarchical-Z map based occlusion culling uses the geometry shader’s ability to deny the emission of primitives that are determined to be invisible on the final rendering. The shader will only emit data for those objects that are visible and this data is streamed out into a buffer object using transform feedback.

The algorithm itself is similar in spirit to the hierarchical Z testing that is implemented in modern GPUs. After rendering all the occluders in the scene, we construct a hierarchical depth image from the depth buffer which we will refer to as the Hi-Z map. This texture map is a mip-mapped, screen resolution image where each texel in mip level i contains the maximum depth of all corresponding texels in mip level i-1. This depth information can be collected during the main rendering pass for the occluding objects as we need a texture of the same resolution so we don’t need a separate depth pass. This can be simply accomplished using OpenGL framebuffer objects.

After the construction of the Hi-Z map, occlusion culling can be performed by comparing depth value of the object’s bounding volume and the depth information stored in the Hi-Z map. This is when the hierarchical mip-mapped structure of the Hi-Z map comes handy as we can do conservative depth comparisons with less texture fetches by sampling directly from a particular mip level.

This is why we constructed the Hi-Z map using a “store maximum depth” policy. This will work with a usual depth buffer setup where the depth comparison function is either GREATER or GEQUAL. For a reverse directed depth buffer the “store minimum depth” policy has to be used.

Hi-Z map construction

In case of single-sample rendering, one can use the Hi-Z map as the main depth buffer for rendering the scene. The technique extends also to multi-sampled rendering but in this case a separate full-screen quad pass is needed to calculate the maximum depth of each individual sample in the multi-sampled depth buffer and store it in the single-sampled Hi-Z map. This is possible since OpenGL 3.2 or using the extension ARB_texture_multisample. Besides this additional step, the algorithm remains the same.

The Hi-Z map can be constructed using OpenGL framebuffer objects by rendering a full-screen quad pass for each mip level where the previous mip level is bound as the input texture and the current mip level is bound as render target. As OpenGL does allow rendering from and to the same texture object as far as we don’t access the same mip level for both reading and writing, the algorithm simply looks like the following:

// bind depth texture
glBindTexture(GL_TEXTURE_2D, depthTexture);
// calculate the number of mipmap levels for NPOT texture
int numLevels = 1 + (int)floorf(log2f(fmaxf(SCREEN_WIDTH, SCREEN_HEIGHT)));
int currentWidth = SCREEN_WIDTH;
int currentHeight = SCREEN_HEIGHT;
for (int i=1; i<numLevels; i++) {
  // calculate next viewport size
  currentWidth /= 2;
  currentHeight /= 2;
  // ensure that the viewport size is always at least 1x1
  currentWidth = currentWidth > 0 ? currentWidth : 1;
  currentHeight = currentHeight > 0 ? currentHeight : 1;
  glViewport(0, 0, currentWidth, currentHeight);
  // bind next level for rendering but first restrict fetches only to previous level
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_BASE_LEVEL, i-1);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAX_LEVEL, i-1);
  glFramebufferTexture2D(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT,
                         GL_TEXTURE_2D, depthTexture, i);
  // draw full-screen quad
  ............
}
// reset mipmap level range for the depth image
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_BASE_LEVEL, 0);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAX_LEVEL, numLevels-1);

It is very important not to forget about the step when we ensure that the viewport size is always at least 1×1 as in case of non-power-of-two (NPOT) textures due to rounding problems. I forgot this first and I was wondering an hour why my last mip level didn’t get filled.

While one may wonder how this technique can be efficient after so many full-screen quad passes, it is in fact very efficient and it constructs the Hi-Z map on my Radeon HD5770 in less than 0.2 milliseconds. The measurement should be quite accurate as I’ve done it using OpenGL timer queries (see the extension ARB_timer_query).

The fragment shader used for the construction of the Hi-Z map is very straightforward except one thing. We use an NPOT depth texture due to the aspect ratio of the window and as NPOT textures use a “floor” convention to determine the size of subsequent mip levels (see the extension ARB_texture_non_power_of_two) we need predicated fetches as in case of reduction from odd-sized mip levels we should not forgot about the edge texels:

#version 400 core

uniform sampler2D LastMip;
uniform ivec2 LastMipSize;

in vec2 TexCoord;

void main(void)
{
  vec4 texels;
  texels.x = texture( LastMip, TexCoord ).x;
  texels.y = textureOffset( LastMip, TexCoord, ivec2(-1, 0) ).x;
  texels.z = textureOffset( LastMip, TexCoord, ivec2(-1,-1) ).x;
  texels.w = textureOffset( LastMip, TexCoord, ivec2( 0,-1) ).x;

  float maxZ = max( max( texels.x, texels.y ), max( texels.z, texels.w ) );

  vec3 extra;
  // if we are reducing an odd-width texture then fetch the edge texels
  if ( ( (LastMipSize.x & 1) != 0 ) && ( int(gl_FragCoord.x) == LastMipSize.x-3 ) ) {
    // if both edges are odd, fetch the top-left corner texel
    if ( ( (LastMipSize.y & 1) != 0 ) && ( int(gl_FragCoord.y) == LastMipSize.y-3 ) ) {
      extra.z = textureOffset( LastMip, TexCoord, ivec2( 1, 1) ).x;
      maxZ = max( maxZ, extra.z );
    }
    extra.x = textureOffset( LastMip, TexCoord, ivec2( 1, 0) ).x;
    extra.y = textureOffset( LastMip, TexCoord, ivec2( 1,-1) ).x;
    maxZ = max( maxZ, max( extra.x, extra.y ) );
  } else
  // if we are reducing an odd-height texture then fetch the edge texels
  if ( ( (LastMipSize.y & 1) != 0 ) && ( int(gl_FragCoord.y) == LastMipSize.y-3 ) ) {
    extra.x = textureOffset( LastMip, TexCoord, ivec2( 0, 1) ).x;
    extra.y = textureOffset( LastMip, TexCoord, ivec2(-1, 1) ).x;
    maxZ = max( maxZ, max( extra.x, extra.y ) );
  }

  gl_FragDepth = maxZ;
}

I was experimenting with using texture gather lookups to reduce the number of texture fetches from 4-to-7 fetches per fragment down to 1-to-3 fetches per fragment (see the extension ARB_texture_gather) it seems that texture gather works only if the image is linearly sampled and to avoid the additional burden involved by switching filtering state during rendering I stuck to simple texture lookups as using texture gather lookups did not show any visible effect on the construction time of the Hi-Z map.

Various mip levels of the Hi-Z map. The Hi-Z map size is 1024x768 and the displayed mip levels are: level 4 (left), level 5 (middle) and level 6 (right).

For debugging and demonstration purposes the Mountains demo has built-in function to display the content of the various mip levels of the Hi-Z map. This is available by pressing the F4 key while Hi-Z map based occlusion culling is enabled. The + and – keys can be used to switch between the mip levels.

In order to better visualize the depth information in the depth buffer I converted the non-linear depth values stored in the depth texture into linear depth values as presented in [GeeXLab] How to Visualize the Depth Buffer in GLSL.

Culling with the Hi-Z map

Once we have constructed the Hi-Z map, we can perform the actual occlusion culling by fetching the 2×2 texel neighborhood corresponding to the screen area occupied by the bounding volume of the object whose visibility has to be determined. In the demo I used bounding boxes but any other bounding volume can be used (e.g. a bounding sphere is usually accurate enough for this technique).

First, we have to calculate the clip space bounding rectangle of the bounding volume. In the bounding box case this is done by transforming the bounding box vertices into clip space and then calculate the minimum and maximum X and Y coordinates. This bounding rectangle will be used for two things: it defines the texture coordinates that we’ll have to use for the Hi-Z map lookup and it helps determining the appropriate LOD for the texture lookup.

In order to determine the texture LOD that we’ll have to fetch we have to calculate the screen space size of the bounding square corresponding to the clip space bounding rectangle determined previously. This can be simply done by calculating the width and height of the bounding rectangle in clip space and then transforming this into screen space:

float ViewSizeX = (BoundingRect[1].x-BoundingRect[0].x) * Transform.Viewport.y;
float ViewSizeY = (BoundingRect[1].y-BoundingRect[0].y) * Transform.Viewport.z;

After this, the texture LOD can be simply calculated using the following formula:

float LOD = ceil( log2( max( ViewSizeX, ViewSizeY ) / 2.0 ) );

Finally, as we have the texture coordinates (the vertices of the clip space bounding rectangle) and the texture LOD, we simply have to make four texture lookups into the Hi-Z map using these parameters, calculate the maximum of the four depth values returned and compare it to the depth value corresponding to the object (this is the object’s front-most point’s depth value that comes also from the clip space coordinates of the bounding box). If the object depth is greater than the reference depth the object is occluded and so it is culled by the geometry shader as usual.

One may ask why we use a 2×2 texel footprint for calculating the reference depth value why not just fetch the next mip level only once (as there we also get the maximum values of a 2×2 texel footprint due to the Hi-Z map construction method). That’s what I’ve also asked myself at first sight but quickly figured out the reason (see the figure below).

Comparison of number of fetches used for occlusion culling. Both figures show the magnified screen coverage of a single Hi-Z map texel at mip level N, texel coverage for mip level N-1 is in cyan and texel coverage for mip level N-2 is in blue. Object is show as red and yellow indicates the fetched texels.

In case of four texels not just the determination of the texture LOD is much easier but also it better encompasses the actual object bounding rectangle. In case of one texture fetch the computation of texture LOD is more complicated and expensive but the main problem is that a larger LOD has to be fetched and it is not always the LOD determined in the case of four fetches plus one. In the most extreme situation (if the bounding rectangle is right at the middle of the screen) it is possible that we have to fetch the largest LOD. This does not result in any false culling but it severely degrades the effectiveness of the culling.

Of course, it is possible to use more complex screen space bounding polygon as well as more fetches but those would increase the effectiveness of the culling much less than the additional burden and expensive operations worth.

Conclusion

We’ve seen how traditional hardware occlusion culling works by using occlusion queries. We also discussed that we sometimes need a better algorithm that does the occlusion culling for large amount of objects without CPU intervention.

The article also described a way to implement such an occlusion culling algorithm by using a hierarchical-Z map and geometry shaders. We’ve also managed to provide a reference implementation in the form of the demo called Mountains that can be downloaded with full source code in the downloads section.

The algorithm performs very well in practice on current hardware. The Hi-Z map construction takes less than 0.2 milliseconds and the actual culling comes at almost no cost for even thousands of objects. For more detail about performance comparison between rendering with and without hierarchical-Z map based occlusion culling read the article about the OpenGL 4.0 Mountains Demo.

While the demo uses the technique only for culling instances of the same object, the technique can be easily extended to work for heterogeneous set of objects as the actual culling algorithm works on a per-object basis and is completely indifferent regarding to the method used for rendering the actual geometry.

This technique can be thought of as the next step towards a completely GPU based visibility determination and scene management system.

Acknowledgements go to Jeremy Shopf, Joshua Barczak, Christopher Oat and Natalya Tatarchuk and their SIGGRAPH 2008 Course Notes that inspired this work.

Gaussian blur

Gaussian blur is an image space effect that is used to create a softly blurred version of the original image. This image then can be used by more sophisticated algorithms to produce effects like bloom, depth-of-field, heat haze or fuzzy glass. In this article I will present how to take advantage of the various properties of the Gaussian filter to create an efficient implementation as well as a technique that can greatly improve the performance of a naive Gaussian blur filter implementation by taking advantage of bilinear texture filtering to reduce the number of necessary texture lookups. While the article focuses on the Gaussian blur filter, most of the principles presented are valid for most convolution filters used in real-time graphics.

Gaussian blur is a widely used technique in the domain of computer graphics and many rendering techniques rely on it in order to produce convincing photorealistic effects, no matter if we talk about an offline renderer or a game engine. Since the advent of configurable fragment processing through texture combiners and then using fragment shaders the use of Gaussian blur or some other blur filter is almost a must for every rendering engine. While the basic convolution filter algorithm is a rather expensive one, there are a lot of neat techniques that can drastically reduce the computational cost of it, making it available for real-time rendering even on pretty outdated hardware. This article will be most like a tutorial article that tries to present most of the available optimization techniques. Some of them may be familiar to all of you but maybe the linear sampling will bring you some surprise, but let’s not go that far but start with the basics.

Terminology

In order to precede any possibility of confusion, I’ll start the article with the introduction of some terms and concepts that I will use in the post.

Convolution filter – An algorithm that combines the color value of a group of pixels.

NxN-tap filter – A filter that uses a square shaped footprint of pixels with the square’s side length being N pixels.

N-tap filter – A filter that uses an N-pixel footprint. Note that an N-tap filter does *not* necessarily mean that the filter has to sample N texels as we will see that an N-tap filter can be implemented using less than N texel fetches.

Filter kernel – A collection of relative coordinates and weights that are used to combine the pixel footprint of the filter.

Discrete sampling – Texture sampling method when we fetch the data of exactly one texel (aka GL_NEAREST filtering).

Linear sampling – Texture sampling method when we fetch a footprint of 2×2 texels and we apply a bilinear filter to aquire the final color information (aka GL_LINEAR filtering).

Gaussian filter

The image space Gaussian filter is an NxN-tap convolution filter that weights the pixels inside of its footprint based on the Gaussian function:

The pixels of the filter footprint are weighted using the values got from the Gaussian function thus providing a blur effect. The spacial representation of the Gaussian filter, sometimes referred to as the “bell surface”, demonstrates how much the individual pixels of the footprint contribute to the final pixel color.

The graphical representation of the 2-dimensional Gaussian function

Based on this some of you may already say “aha, so we simply need to do NxN texture fetches and weight them together and voilà”. While this is true, it is not that efficient as it looks like. In case of a 1024×1024 image, using a fragment shader that implements a 33×33-tap Gaussian filter based on this approach would need an enormous number of 1024*1024*33*33 ≈ 1.14 billion texture fetches in order to apply the blur filter for the whole image.

In order to get to a more efficient algorithm we have to analyze a bit some of the nice properties of the Gaussian function:

• The 2-dimensional Gaussian function can be calculated by multiplying two 1-dimensional Gaussian function:

• A Gaussian function with a distribution of 2σ is equivalent with the product of two Gaussian functions with a distribution of σ.

Both of these properties of the Gaussian function give us room for heavy optimization.

Based on the first property, we can separate our 2-dimensional Gaussian function into two 1-dimensional one. In case of the fragment shader implementation this means that we can separate our Gaussian filter into a horizontal blur filter and the vertical blur filter, still getting the accurate results after the rendering. This results in two N-tap filters and an additional rendering pass needed for the second filter. Getting back to our example, applying the two filters to a 1024×1024 image using two 33-tap Gaussian filters will get us to 1024*1024*33*2 ≈ 69 million texture fetches. That is already more than an order of magnitude less than the original approach made possible.

Using the second property of the Gaussian function, we can separate our 33×33-tap filter into three 9×9-tap filter (9+8=17, 17+16=33). Back to our example, for the 1024×1024 sized image this results in 1024*1024*9*9*3 ≈ 255 million texture fetches. As we can see, we also spared a large amount of the necessary texture fetches using this approach as well.

Of course, the combination of the two techniques is also possible. That means we both separate our filter to a vertical and horizontal filter as well as decompose our 33-tap filter into three 9-tap filter. This will get us to the almost optimal number of 1024*1024*9*3*2 ≈ 56 million texture fetches.

Gaussian kernel weights

We’ve seen how to implement an efficient Gaussian blur filter for our application, at least in theory, but we haven’t talked about how we should calculate the weights for each pixel we combine using the filter in order to get the proper results. The most straightforward way to determine the kernel weights is by simply calculating the value of the Gaussian function for various distribution and coordinate values. While this is the most generic solution, there is a simpler way to get some weights by using the binomial coefficients. Why we can do that? Because the Gaussian function is actually the distribution function of the normal distribution and the normal distribution’s discrete equivalent is the binomial distribution which uses the binomial coefficients for weighting its samples.

The Pascal triangle showcasing the binomial coefficients that can be used to calculate the kernel weights (each element in the succeeding rows is the sum of its "parents").

For implementing our 9-tap horizontal and vertical Gaussian filter we will use the last row of the Pascal triangle illustrated above in order to calculate our weights. One may ask why we don’t use the row with index 8 as it has 9 coefficients. This is a justifiable question, but it is rather easy to answer it. This is because with a typical 32 bit color buffer the outermost coefficients don’t have any effect on the final image while the second outermost ones have little to no effect. We would like to minimize the number of texture fetches but provide the highest quality blur as possible with our 9-tap filter. Obviously, in case very high precision results are a must and a higher precision color buffer is available, preferably a floating point one, using the row with index 8 is better. But let’s stick to our original idea and use the last row…

By having the necessary coefficients, it is very easy to calculate the weights that will be used to linearly interpolate our pixels. We just have to divide the coefficient by the sum of the coefficients that is 4096 in this case. Of course, for correcting the elimination of the four outermost coefficients, we shall reduce the sum to 4070, otherwise if we apply the filter several times the image may get darker.

Now, as we have our weights it is very straightforward to implement our fragment shaders. Let’s see how the vertical file shader will look like in GLSL:

uniform sampler2D image;

out vec4 FragmentColor;

uniform float offset[5] = float[]( 0.0, 1.0, 2.0, 3.0, 4.0 );
uniform float weight[5] = float[]( 0.2270270270, 0.1945945946, 0.1216216216,
                                   0.0540540541, 0.0162162162 );

void main(void)
{
    FragmentColor = texture2D( image, vec2(gl_FragCoord)/1024.0 ) * weight[0];
    for (int i=1; i<5; i++) {
        FragmentColor +=
            texture2D( image, ( vec2(gl_FragCoord)+vec2(0.0, offset[i]) )/1024.0 )
                * weight[i];
        FragmentColor +=
            texture2D( image, ( vec2(gl_FragCoord)-vec2(0.0, offset[i]) )/1024.0 )
                * weight[i];
    }
}

Obviously the horizontal filter is no different just the offset value is applied to the X component rather than to the Y component of the fragment coordinate. Note that we hardcoded here the size of the image as we divide the resulting window space coordinate by 1024. In a real life scenario one may replace that with a uniform or simply use texture rectangles that don’t use normalized texture coordinates.

If you have to apply the filter several times in order to get a more strong blur effect, the only thing you have to do is ping-pong between two framebuffers and apply the shaders to the result of the previous step.

9-tap Gaussian blur filter applied to an image of size 1024x1024: no filter applied (left), applied once (middle), applied nine times (right). Click to view the full-sized image in order to better see the difference.

Linear sampling

So far, we were able to see how to implement a separable Gaussian filter using two rendering pass in order to get a 9-tap Gaussian blur. We’ve also seen that we can run this filter three times over a 1024×1024 sized image in order to get a 33-tap Gaussian blur by using only 56 million texture fetches. While this is already quite efficient it does not really expose any possibilities of the GPUs as this form of the algorithm would work perfectly almost unmodified on a CPU as well.

Now, we will see that we can take advantage of the fixed function hardware available on the GPU that can even further reduce the number of required texture fetches. In order to get to this optimization let’s discuss one of the assumptions that we made from the beginning of the article:

So far, we assumed that in order to get information about a single pixel we have to make a texture fetch, that means for 9 pixels we need 9 texture fetches. While this is true in case of a CPU implementation, it is not necessarily true in case of a GPU implementation. This is because in the GPU case we have bilinear texture filtering at our disposal that comes with practically no cost. That means if we don’t fetch at texel center positions our texture then we can get information about multiple pixels. As we already use the separability property of the Gaussian function we actually working in 1D so for us bilinear filter will provide information about two pixels. The amount of how much each texel contribute to the final color value is based on the coordinate that we use.

By properly adjusting the texture coordinate offsets we can get the accurate information of two texels or pixels using a single texture fetch. That means for implementing a 9-tap horizontal/vertical Gaussian filter we need only 5 texture fetches. In general, for an N-tap filter we need [N/2] texture fetches.

What this will mean for our weight values previously used for the discrete sampled Gaussian filter? It means that each case we use a single texture fetch to get information about two texels we have to weight the color value retrieved by the sum of the weights corresponding to the two texels. Now that we know what are our weights, we just have to calculate the texture coordinate offsets properly.

For texture coordinates, we can simply use the middle coordinate between the two texel centers. While this is a good approximation, we won’t accept it as we can calculate much better coordinates that will result us exactly the same values as when we used discrete sampling.

In case of such a merge of two texels we have to adjust the coordinates that the distance of the determined coordinate from the texel #1 center should be equal to the weight of texel #2 divided by the sum of the two weights. In the same style, the distance of the determined coordinate from the texel #2 center should be equal to the weight of texel #1 divided by the sum of the two weights.

As a result, we get the following formulas to determine the weights and offsets for our linear sampled Gaussian blur filter:

By using this information we just have to replace our uniform constants and decrease the number of iterations in our vertical filter shader and we get the following:

uniform sampler2D image;

out vec4 FragmentColor;

uniform float offset[3] = float[]( 0.0, 1.3846153846, 3.2307692308 );
uniform float weight[3] = float[]( 0.2270270270, 0.3162162162, 0.0702702703 );

void main(void)
{
    FragmentColor = texture2D( image, vec2(gl_FragCoord)/1024.0 ) * weight[0];
    for (int i=1; i<3; i++) {
        FragmentColor +=
            texture2D( image, ( vec2(gl_FragCoord)+vec2(0.0, offset[i]) )/1024.0 )
                * weight[i];
        FragmentColor +=
            texture2D( image, ( vec2(gl_FragCoord)-vec2(0.0, offset[i]) )/1024.0 )
                * weight[i];
    }
}

This simplification of the algorithm is mathematically correct and if we don’t consider possible rounding errors resulting from the hardware implementation of the bilinear filter we should get the exact same result with our linear sampling shader like in case of the discrete sampling one.

9-tap Gaussian blur applied nine times with discrete sampling (left) and linear sampling (right). Click for the full resolution of the image. Note that there is no visible difference between the two techniques even after several passes.

While the implementation of the linear sampling is pretty straightforward, it has a quite visible effect on the performance of the Gaussian blur filter. Taking into consideration that we managed to implement a 9-tap filter using just five texture fetches instead of nine, back to our example, blurring a 1024×1024 image with a 33-tap filter takes only 1024*1024*5*3*2 ≈ 31 million texture fetches instead of the 56 million required by discrete sampling. This is a quite reasonable difference and in order to better present how much that matters I’ve done some experiment to measure the difference between the two techniques. The result speaks for itself:

Performance comparison of the 9-tap Gaussian blur filter with discrete and linear sampling on a Radeon HD5770. The vertical axis is the frames per second (higher is better) and the horizontal axis represents results with various number of blur steps (higher is blurrier).

As we can see, the performance of the Gaussian filter implemented with linear sampling is about 60% faster than the one implemented with discrete sampling indifferent from the number of blur steps applied to the image. This roughly proportional to the number of texture fetches spared by using linear filtering.

Conclusion

We’ve seen that implementing an efficient Gaussian blur filter is quite straightforward and the result is a very fast real-time algorithm, especially using the linear sampling, that can be used as the basis of more advanced rendering techniques.

Even though we concentrated on Gaussian blur in this article, many of the discussed principles apply to most convolution filter types. Also, most of the theory applies in case we need a blurred image of reduced size like it is usually needed by the bloom effect, even the linear sampling. The only thing that is really different in case of a reduced size blurred image is that our center pixel is also a “double-pixel”. This means that we have to use a row from our Pascal triangle that has even number of coefficients as we would like to linear sample the middle texels as well.

We’ve also had a brief insight into the computational complexity of the various techniques and how the filter can be efficiently implemented on the GPU.

The demo application used for the measurements performed to compare the discrete and linear sampling method can be downloaded here:

Binary release

Platform: Windows
Dependency: OpenGL 3.3 capable graphics driver
Download link: gaussian_win32.zip (2.96MB)

Source code

Language: C++
Platform: cross-platform
Dependency: GLEW, SFML, GLM
Download link: gaussian_src.zip (5.37KB)

P.S.: Sorry for the high minimum requirements of the application just I would really like to stick to strict OpenGL 3+ demos.

The Khronos Group keeps the pace that they set themselves being able to deliver the latest specification of OpenGL less than half year after the revolutionary appearance of OpenGL 4. Abandoning the OpenGL 3.x line of the specification (at least for a while) the new update concentrates on Shader Model 5.0 class GPUs and extensions heavily promoted by the community. Beside all this, the Khronos Group now confessedly opens towards convergence to OpenGL ES making the desktop version of the specification downward compatible with its embedded brother. In this article I would like to present the features introduced with the latest revision of the specification.

At the time of the release of the OpenGL 4 specification I was able to quickly deliver you a thorough presentation of all the new features introduced by that revision of the specification. This time I am already quite late, however I hope that this article will still prove as value for lots of you, especially for those who haven’t had time in the recent past to dig into the details of the new API version.

OpenGL 4.1 is not as revolutionary and feature-rich as its predecessor, however the latest revision was well received by the community as it brought such core extensions to the API that the community was waiting for a long time now. The new revision of the specification was accompanied with the appearance of a couple of other ARB extensions that have not yet been included into core, however I will still talk about some of them as they indicate a slight shift in the force of influence of various vendors and representatives inside the Architecture Review Board (ARB).

New features of OpenGL 4.1

Let’s start with the presentation of the new features arriving with the OpenGL 4.1 specification primarily targeting Shader Model 5.0 hardware. Here you will see a lot of harmonization features as well as community’s choice features that squarely intended to increase OpenGL development efficiency and feedom.

ARB_ES2_compatibility

There have been for a long time rumors about the Khronos Group preparing a convergence between desktop OpenGL and OpenGL ES. This extension of the core specification clearly makes the first step towards this goal by providing an all-in-one specification pack that makes the desktop version of the specification downward compatible with ES. The extension adds support for features of OpenGL ES 2.0 that are missing from OpenGL 3+. According to the extension specification, enabling these features will ease the process of porting applications from OpenGL ES 2.0 to OpenGL.

More precisely, GL_ARB_ES2_compatibility exposes not just all the functions and tokens that weren’t present in the desktop version of the specification but also completes it with all the semantics that were exclusively specified only in the embedded version. Just to mention few of these issues:

• Vertex data format is now extended with the possibility to use 16-bit fixed point values by exposing the GL_FIXED type identifier token.
• Providing possibility to query the precision format used internally by shaders.
• Enable the use of GLSL ES for writing shaders for desktop GL.

While having this extension under the hood does not mean that we can simply pick our last game made for e.g. Symbian and just drop it on our PC, this extension may prove to be great value for GL ES developers migrating their software to desktop platforms.

ARB_get_program_binary

This is one of the most waited additions to the core specification by the developer community. This extension introduces the possibility to acquire some sort of binary format of the compiled and linked shaders that can be later used to specify the program object directly with its binary code thus providing caching possibility to eliminate the need of compilation and linking next time the shader has to be used. This also makes it possible to create an offline GLSL compiler just using the OpenGL API itself.

Still, it has to be mentioned that having this feature in our hand does not necessarily mean that we can simply create our shader binaries offline and then distribute our software without the shader source itself as the binary formats supported by a particular implementation heavily depend on the hardware vendor as well as driver version. This is due to the fact that the shader binary most probably consists of instructions specially generated for the particular GPU-driver combo. The only way to relax this limitation would be to have some sort of cross-platform byte-code for shaders but that would in fact defeat most of the benefits of the extension on its own. Additionally, this extension does not provide any binary formats but leaves this to vendor specific extensions. It only exposes a common infrastructure for acquiring and loading program binaries.

While the usage of this extension does not completely eliminates the need for shader source compilation, it can limit the need for recompilation and relink to an installation time or first-run time compilation instead and use the stored binaries later. It also opens up room for SDK tools providing shader compilers with more aggressive optimization at their disposal being used offline. Such tools can truly be introduced as the specification explicitly mentions that run-time generated binaries by the GL should be interchangeable with those generated by offline SDK tools.

ARB_separate_shader_objects

This is one another extension requested over several forums by the community. This feature has a longer history as it is actually based on the already existing and widely supported extension GL_EXT_separate_shader_objects by NVIDIA. For those who are already familiar with the predecessor of this extension won’t really find too much new stuff reading the specification of the ARB version of the extension, however it is still a must to read for them as well as even though there aren’t too much semantic differences between the functionality of the two, the usage of them still differs quite a lot as the ARB version solved the design issues of its predecessor by introducing a new type of GL object that I will talk about just in a moment.

In a nutshell, this extension provides a way to create program objects using any variation of shaders and bind them together to the current rendering context. Previously there was no way to bind multiple program objects to the context as the program object was designed to be a container for all the shaders forming the rendering pipeline of the context. This was a design decision during the development of GLSL that, before this extension, made the connection between the varyings of subsequent shader stages using a name based binding. As name information is available for shaders latest in the link stage, shaders were tightly coupled meaning that a change in any shader stage code required the relinking of the complete program object.

This proved to be very unpleasant for OpenGL developers as usually every rendering engine has its own set of vertex and fragment shaders (maybe accompanied with other shader types) that are used in various combinations. As an example, let’s take two vertex shaders: a simple MVP matrix based transformation shader and a more complex one that also supports skeletal animation. Also let’s take two fragment shaders: one for diffuse material and one for reflective material. We can have several types of objects: static with diffuse material, static with reflective material, animated with diffuse material and animated with reflective material.

In traditional GLSL the vertex and fragment shaders are bound together at link time rather than at the time they are bound to the context, like it was in case of legacy shaders (GL_ARB_vertex_program, GL_ARB_fragment_program and others). This means that in order to be able to use any of the combinations of vertex and fragment shaders (and maybe some geometry and tesselation shaders as well) we end up with two possible solutions, both having their severe drawbacks:

Link every combination of the shader objects

While this sounds as a viable solution and is still used by most of the developers, it has several problems. First of all, it wastes resources as we now have several copies of the same piece of code and the number of combinations can be pretty high, especially if not just vertex and fragment shaders are in use. While this is already quite a reasonable issue with the solution, the biggest problem arises for the application developer when he or she has to maintain an individual set of uniform locations as well as binding points for vertex attributes, draw buffers and possibly transform feedback buffers. While the GL_ARB_explicit_attrib_location extension already eliminates the need for maintaining binding points for vertex attributes, this solution is still simply unacceptable.

Link the program objects on an on-demand basis

In case of this alternative we are said to link the shader objects only when they are actually needed. While this solution eliminates the need for a possibly huge number of program objects, it introduces a reasonable run-time performance hit due to the additional relink process needed. Additionally, this solution proves to be more inferior even compared to the previous one as the uniform locations are determined at link time so it makes no less headache to the application developer.

This is the rationale behind this extension and why it is included into the core specification. The extension relaxes the strict tightly coupled behavior of the GLSL and adopts a mix-and-match shader stage model allowing multiple different program objects to be bound at once each to an individual set of rendering pipeline stage independently of other stage bindings.

Due to the fact that from now program objects are not the top most containers for the code used currently by the rendering pipeline, the ARB decided to introduce a new container object called a “program pipeline object” that can contain a set of program objects bound to their very own set of shader stages. This is the main difference between the EXT and the ARB version of the extension. I think it was a good decision to introduce this new type of object and the associated semantics as I always thought that the EXT version of the extension doesn’t have a really good design as I’ve seen it kind of a hack to relax the limitations of GLSL. The program pipeline object idea is definitely superior and I hope that the GLSL does not have too much of such annoying design issues hidden within.

ARB_shader_precision

This extension is much more a clarification to the existing specification rather than a new feature. It restricts more clearly the precision requirements of implementations of GLSL. According to the specification, the extension is meant to more precisely define the precision of arithmetic operations (addition, multiplication, etc.), transcendentals (log, exp, pow, etc.), when NaNs (not-a-number) and INFs (infinites) will be accepted and generated and denorm flushing behavior. The precision of the rest of the operations, including trigonometric operations are not addressed by the extension. For further details, please refer to the extension specification.

ARB_vertex_attrib_64bit

This extension trivially introduces 64-bit floating-point types into the list of supported vertex attribute component types. Nominally OpenGL did support this already from the very early stages of its history, however in practice only the latest generation of hardware does really accept vertex attributes in double precision floating-point type. While OpenGL 4 already introduced support for 64-bit floating-point values in GLSL and most of the shaders’ environment, vertex attributes gained the 64-bit precision only with this new extension.

This new feature makes it possible to use high precision for positioning data and other attributes of our geometries. While this sounds pretty awesome and it is actually, still for game developers and other real-time graphics users this shouldn’t mean that they should quickly switch to the new precision only in such cases when the precision requirements of the application really need it as using 64-bit floating-point values for vertex attributes does not just double the memory consumption but also involves a serious hit on performance due to bandwidth limitations and vertex attributes of this type may count double against the implementation-dependent limit on the number of vertex shader attribute vectors.

ARB_viewport_array

Previously, the configuration of the viewport, aka the transformation that generates the screen space coordinates based on the incoming view space coordinates of the vertices, was a global configuration that had effect on all draw commands meaning that in order to draw a primitive into multiple viewports the OpenGL viewport had to be changed between several draw calls. While previously this limitation wasn’t really an issue, due to the introduction of geometry shaders the possibility to amplify geometry and produce multiple output primitives for each primitive input justifies the need of several separately configurable viewports. Why? Because even though one was able to render the output primitives into separate render targets, they still shared the same global viewport.

This extension enhances OpenGL by providing a mechanism to specify multiple viewports and a new ability for the geometry shader being able to select the used viewport on a per-primitive basis. This does not just mean that separate viewports can be used for separate render targets but also enables to use multiple viewports to render to the same render target.

Additionally, the introduction of a viewport array means that we’re gonna have separate scissor rectangle for each viewport in the array as well. This can come handy for deferred shading based renderers that often use the scissor rectangle to limit the number of pixels to be accessed in case of rendering the effect of a light source. Having multiple scissors means that we have to change state less often, thus batching is much less an issue even in case of heavy scissor rectangle usage.

Finally, the new viewport specification commands accept floating point values thus providing additional flexibility to the application developer to define their very own pixel center conventions.

I’m pretty unsure whether this feature depends on any Shader Model 5.0 hardware, maybe others are more aware of this. Anyway, I wouldn’t be surprised if this extension will be supported by a much larger range of graphics cards than just pure SM5 GPUs. Actually this is true for many other extensions introduced by OpenGL 4.1 but let’s not guess but wait for the upcoming drivers to see whether I’m right or wrong.

Some other interesting extensions

So far I presented the new features of the latest revision of the OpenGL specification. While this was the main topic of this article, at about the same time the specification was published, a lot of other ARB extensions just appeared in the registry. While these extensions are not yet included into core and I cannot know whether they will be ever included, I would like to talk about some of them as it made me get to an interesting conclusion.

ARB_shader_stencil_export

The stencil test is a powerful mechanism of OpenGL to selectively discard fragments based on the content of the stencil buffer that is used in a wide variety of rendering techniques including shadow volumes and deferred shading. However, the whole configuration of the stencil test and stencil operations is completely fixed function that is limited to operations such as incrementing, decrementing the existing value, or replacing the existing value in the stencil buffer with a fixed reference value.

This extension provides some programmability to the fixed function stencil operations by enabling the fragment shader to output a stencil reference value on a per-fragment basis. When stencil testing is enabled, this allows the test to be performed against the value generated in the shader. Also, when the stencil operation is set to GL_REPLACE, this allows a value generated in the shader to be written to the stencil buffer directly.

This opens up a lot of possibilities, however, I need to think much more about it as the best use cases of this feature are pretty much not basic ones. Obviously, by using the stencil reference value export inside a fragment shader disables early stencil test in the same style as exporting an new depth value from within a fragment shader disables early depth test.

ARB_debug_output

This extension allows OpenGL to notify the application when various events occur that can come handy during application development and debugging. These events include errors, usage of deprecated functionalities, using configuration that results in undefined behavior, portability or performance issues. The application is notified about these events using a callback function that is defined by passing a function pointer to the appropriate OpenGL command.

While this extension provides a callback mechanism only for debugging purposes, the most revolutionary thing by having such an ARB extension is that this is the first official appearance of a feature that supports callbacks to the application code. Most probably not I’m the only person who would like to see a lot of other callbacks in the future included in the OpenGL API as we can benefit from it by getting notification about e.g. the completion of various asynchronous commands issued previously. This does not just provide a lot of flexibility but may also help in optimizing the rendering code based on the additional information previously available only if we use polling.

Why these extensions are so interesting?

The two extensions presented above already great value on their own but this isn’t why I mentioned them. The reason why I found these extensions so interesting as they are both obviously based on some vendor specific extensions released in the recent past by AMD, namely GL_AMD_shader_stencil_export and GL_AMD_debug_output. This conspicuously reveals that AMD has serious plans with their OpenGL support and this is something that a lot of those crazy folks waited for, who develop OpenGL stuff using ATI cards like me.

I think this also means that the NVIDIA monopoly in the ARB is over and this results in concurency and competition from what OpenGL and its community will definitely benefit in the long run.

Conclusion

The article ran out of control again, like the one I wrote about the previous release of the specification. Again, hope there are at least a few of you who kept up reading and finally got to this last chapter of the article. We can again quote the always recurring question of the community:

Where is direct state access?

Well, it is still not here, however, finally AMD has finished implementing it as well and published it finally. They have been working on it for quite some time but it became officially public only with Catalyst 10.7. Haven’t used it so far so maybe plenty of hidden bugs are still in it but at least they have it. This is one another thing that strengthens my prognostication that AMD committed itself for support OpenGL as previously they barely added support for any other extensions beside core features.

Back to the topic of the OpenGL 4.1 specification, while it is not as revolutionary as we got used to after reading the previous update, OpenGL is still on track and this is thanks to the Khronos Group and obviously to the great community. If OpenGL will get its iterative evolution in this pace like we’ve seen in the last two years, Microsoft will have a difficult time to keep up.

Thanks for reading this not-so-short article!

The Khronos Group continues the progress of streamlining the OpenGL API. One very important step in this battle has been made just a few days ago by releasing two concurrent core releases of the OpenGL specification, namely version 3.3 and 4.0. This is a major update of the standard containing many revolutionary additions to the tool-set of OpenGL that need careful examination. In this article I would like to talk about these new features trying to point out their importance and touching also some practical use case scenarios.

This is the fourth revision of the OpenGL API standard in the last two years. This fast pace revolution started about one and half years ago with the release of the version 3.0 of the specification. At that time, a great feel of disappointment has overcame the developers due to the lack of the promised rewrite of the whole API. Others, who had to deal with legacy code were also disappointed but they felt so because the new revision of the API threatened them with removing old features. These two opposing forces have put the Khronos Group into a situation where there was very difficult to make a decision that would make everybody happy. After two releases, this issue has been mostly resolved with OpenGL 3.2 and also lots of missing features have been integrated into the core API meanwhile.

Even though great steps has been made in order to fulfill everybody’s needs, the gap between the core functionality of OpenGL and the DirectX API still increased, especially due to the introduction of Shader Model 5.0 hardware. OpenGL was in a position when it had to adopt the features of the new hardware generation and also try to make up leeway in case of Shader Model 4.0 hardware. My personal wish was that there should be two new versions of the API: one that complements the OpenGL 3.x API with the missing features and another that catches up to DirectX 11. Actually my wish became true as the first time in the history of OpenGL we got two new releases of the standard at once, and finally, we got an API that is a really competitive alternative for Microsoft’s DirectX API. I think I can say this in the name of every OpenGL developer: Thank you Khronos!

Okay, but that’s enough about history and acknowledgements. Lets see what’s under the hood of the new API revisions! When I read the good news at OpenGL.org I felt myself like a child at Christmas just taking the first look at the presents under the tree: I was in great ecstasy and started to “open the presents” as fast as I could…

New features of OpenGL 3.3

Let’s start with the new version of the API targeting Shader Model 4.x hardware. It seems that the concentration on the major release 4.0 didn’t capture the attention of the ARB explicitly as we have many interesting features already in the first box…

ARB_blend_func_extended

This is a feature for what I’ve seen many requests on the OpenGL discussion forums. It enables fragment shaders to output an additional color per render target that can be used as a blending factor for either source or destination colors providing an additional degree of freedom to affect the way how fragments are blended into the destination buffers. This is one functionality that is supported by the underlying hardware for a while but without API support it was impossible to take advantage of it. As it is very straightforward how this feature works I would not even talk about it too much. Just one additional comment: surprisingly AMD already supports this extension in its latest graphics drivers which is a remarkable thing taking in consideration that AMD drivers were always a step behind the NVIDIA ones in the race of adopting latest OpenGL features. It seems that now AMD takes seriously the OpenGL support and this is good news for all the developers out there, especially for me, being an ATI fan.

ARB_explicit_attrib_location

Most probably not just for me, the way how the binding of vertex attributes to shader attributes and the binding of shader outputs to render targets happened earlier caused a big headache from both the point of view of modular software design and efficiency. Previously, the application developer had little to no control over how to automatically connect these elements together in a shader independent way. This tight coupling between the host application code and the shaders just make the work of the developers cumbersome. This feature leverages the way how this binding process is done by allowing to globally assign a particular semantic meaning to an attribute location without knowing how that attribute will be named in any particular shader, decoupling the host application from the shaders. This extension is a typical example how design abstractions can ease the life of the developer without any dependency on hardware support.

ARB_occlusion_query2

Well, there isn’t too much to say about this extension as it just adds a new occlusion query type that reports just a boolean value about the visibility of the object rather than the actual samples. It is somewhat equivalent to the occlusion query extensions prior to ARB_occlusion_query. Don’t ask me why this feature is important but they felt that it might be useful. One thing I can think about that with such a query we might get our results about the occlusion query of the proxy object sooner as we have to wait only till the first passed sample but I’m not confident whether such thing is supported by either the hardware or the drivers.

ARB_sampler_objects

This is one another feature that people have been waiting for years. This extension decouples texture image data from sampler state. Previously, if a texture image had to be used with different sampler modes, no matter if we talk about various filtering modes or texture coordinate wrapping, one had to do expensive state changes to modify the sampler state of the texture object, accomplish the needed filtering or wrapping from within shaders or, in worst case, duplicating texture image data in order to have access to the same texture with different sampler parameters. The primary intend of this feature is to solve these problems.

One thing to remark regarding to this extension is that even though it is a long waited addition to the API, several people already expressed their discontent regarding to the fact that the texture unit semantics have been kept. Nevertheless, I also expected that the introduction of this feature should be the point when the texture unit semantics has to go but after seeing the example of ARB_explicit_attrib_location as a way to decouple the shader code from the host application code I tend to agree with Khronos in this decision as we can think about the texture units from now as an adapter layer between GPU and CPU code and as such the decision seems reasonable.

ARB_shader_bit_encoding

This extension adds built-in functions for getting and setting the bit encoding for floating-point values in the OpenGL Shading Language. As it is more like an indicator extension regarding to added functionality in the Shading Language I would rather not go into details as I will talk about the new Shading Language later.

ARB_texture_rgb10_a2ui

Again, an extension that is quite self-explanatory: new texture image format called RGB10_A2 with non-normalized unsigned integers in them. This is nothing more than another hole filled in the gap between hardware and API support.

ARB_texture_swizzle

Especially when using one or two component texture formats, like in the case of shadow maps, the specification was somewhat unclear how these components are finally mapped to RGBA quadruples and provided little to no facilities to control this process. If the developers weren’t already fed up with this, the possibility of a problem increased even further because often the driver implementations behaved differently as well. This issue has been finally clarified with this extension by providing an explicit tool for the application developer to control the swizzling of the components that is done implicitly afterwards in case of every single texture fetch. The new state is introduced as part of texture object state that provides fine grained control over when and how to use the swizzling. According to the extension specification, this feature has a notable role in helping porting issues of legacy OpenGL applications as well as those of the games written for PlayStation 3 as the console provides such functionality already.

ARB_timer_query

Prior to this extension, runtime performance measurements were limited to the use of client side timing information or relying on the use of offline profiling mechanisms like that of AMD’s GPUPerfStudio. During development, this timing information can help identify application, driver or GPU bottlenecks. At runtime, this data can be used to dynamically optimize the scene to achieve reasonable frame rates. While today’s hardware provides a great repertoire of performance measurement metrics there was no API support to access these previously. This feature provides an additional asynchronous query type that enables application developers to measure the driver and GPU time that is required to complete a set of rendering commands, thus providing additional flexibility for both offline and runtime optimizations. While this extension does not guarantee 100% consistency and repeatability, the information gathered with timer queries will definitely make it possible to identify server side bottlenecks and the reasons behind them.

ARB_instanced_arrays

Many people argued with me at the OpenGL discussion forums when I stated that instanced arrays should be included in core OpenGL. Their reasoning was built on the fact that we already have the ARB_draw_instanced extension that provides a shader based thus much more flexible way to handle instanced geometry. While from this point of view I tend to agree with them, there are many non-trivial use cases which prove that my reasoning is not pointless. It seems that Khronos agrees with me regarding to this topic.

In a nutshell, the instanced arrays feature enables the use of vertex attributes as a source of instance data. This is done by introducing a so called “array divisor” that specifies how the corresponding vertex attributes are mapped to instances. Usually a vertex attribute advances on a per-vertex basis. In case of instanced arrays this advance happens only after every Nth conceptual draw calls that is equivalent to  a traditional draw command, excluding instanced draw commands.

One use case can be when one deals with huge number of instances where the per-instance data simply not fits into uniform buffers. While in such cases one can use a texture buffer instead to source the instance data like it was mentioned in my article Uniform Buffers VS Texture Buffers, accepting the additional overhead of using texture fetches may prove to be a not-so-performance-wise decision. Beside standard instancing use cases, there are plenty of nasty tricks that can be efficiently achieved using this feature but that goes far beyond the scope of this article and requires a separate discussion on what I will most probably recap in the near future.

ARB_vertex_type_2_10_10_10_rev

We’ve arrived to the final new extension included in core OpenGL 3.3. This is another gap filling extension to provide two new vertex attribute data formats: a signed and an unsigned format with 10 bits for each significant coordinate. The most typical use of this format is to store vertex normals in the signed-normalized version of the format in order to have a compact (4 bytes per normal) yet high precision (due to 10 bits per component) format that can reduce memory needs and bandwidth requirements while retaining sufficient precision. Previously, there was no way to have such high precision for the vertex attributes in case of a 4-byte footprint.

The OpenGL Shading Language 3.30

The first remarkable thing is the shift in the versioning of the Shading Language. It seems that from now it will be in align with the core specification version. This decision was most probably made because of the introduction of two release branches of the standard specification in order to avoid confusion regarding to the correspondence between API and Shading Language versioning.

As in case of talking about the OpenGL Shading Language it is much more difficult to easily summarize the new features with corresponding use cases I will simply limit my comments to an excerpt from its specification regarding to the features added in this new version:

• Layout qualifiers can be used to declare the location of vertex shader inputs and fragment shader outputs in align with the API functionality provided by ARB_explicit_attrib_location as mentioned before.
• Built-in functions provided to converting floating-point values to integer ones representing their encoding.
• Some clarification of already existing facilities of the language.

New features of OpenGL 4.0

It is very obvious that the major version number change indicates that this revision of the specification is targeting Shader Model 5.0 hardware. To be honest, as I was never really interested in DirectX, I barely know all the features introduced by DX11 but seems that there are some great facilities in OpenGL 4.0 that I’ve never heard that hardware supports it. This can be due to DX11 does not even support such functionalities but it is maybe because I don’t know enough details about DX11. Anyway, let’s see the revolutionary things that we face we checking out the latest version of the OpenGL specification…

ARB_draw_buffers_blend

Using this feature one is able to select individual blend equations and blend functions for each render target. This extension was already exposed for a few months now so most probably everybody heard about it or even if not the functionality is very straightforward. It simply removes some of the restrictions when dealing with multiple render targets (MRT). One interesting thing is still that the Khronos Group decided to include this extension in the 4.0 version of the API but not in 3.3. This is odd as Shader Model 4.0 capable hardware already supports this feature or at least I have the extension on my Radeon HD2600 which raises the question: why only in 4.0? Unfortunately, I don’t know the answer but I hope the ARB has a good reason behind this, as we will see later, there are other features that for some reason were only exposed in the latest version of the API but not in core for Shader Model 4.0 hardware.

ARB_sample_shading

In case of traditional multisample rendering the hardware optimizes the multisampling in a way that the fragment shader is executed only once for each fragment. This can be done as the standard specification relaxes the way how the implementation behaves regarding to feeding color and texture coordinate values for each sample. While this optimization usually does not provide any rendering artifacts and it heavily reduces the amount of pressure on the GPU, there are some situations when this optimization results in aliasing artifacts. One sample use case is when alpha-tested primitives are rendered.

This extension provides a global state for enabling and disabling sample shading and a way to control how fine-grained per-sample shading should be by supplying a minimum number of samples that need to be shaded. Beside this, it also introduces the required language elements to the OpenGL Shading Language to support sample shading.

ARB_shader_subroutine

In my humble opinion, this is one of the most important features introduced in this new version of the API specification. So far, many engine and shader developers faced the problems that where inherently there in the Shading Language that heavily reduced the ability to create a modular shader design in order to separate the independent tasks done in shaders nowadays. One initiative was the idea behind the EXT_separate_shader_objects extension. While that extension removed the dependency between shader stages, it does not address the problem with tight coupling inside one shader stage, also the aforementioned extension defeats some of the design goals of the Shading Language introducing complicated language semantics in order to solve the problem of inter-stage dependency.

Just to emphasize the importance of this new functionality with a very basic example, let’s take a simple rendering engine that supports skeletal animated geometry, materials and lights. In such a use case both the vertex and fragment shaders have multiple roles: the vertex shader has to perform the skeletal animation (property of the geometry) and the view transformation (property of the camera or of the light in case of shadow map rendering), and the fragment shader has to calculate the incident light to the surface point (property of the light) and then calculate the illuminance factor (property of the material). With the traditional tool-set these components of the shaders were tightly coupled and in order to support the combination of any geometry type (animated or not, skeletal or morph animation, etc.), any light type (directional, point, etc.) and material type (diffuse, phong, environment mapped, etc.), one had to compile all possible combinations of the shaders or create uber-shaders that do run-time decisions in order to solve the problem of heterogeneous inputs. Both of these solutions provide additional hardware resource usage and possible runtime overhead.

This extension adds some kind of polymorphism support to shaders. This way a single shader can include many alternative subroutines for a particular task and dynamically select through the API which subroutine is called from each call site. This opens the doors for modular shader designs while retaining most of the performance of specialized shaders.

ARB_tessellation_shader

Yes, this is about the new geometry tessellation mechanism introduced by Shader Model 5.0 hardware. The extension itself introduces three new stages that are roughly situated between the vertex shader and the geometry shader:

• Tessellation Control Shader – This new shader type operates on a patch that is actually nothing more than a fixed-size collection of vertices, each with per-vertex attributes and a number of associated per-patch attributes. Also note that while it operates on a patch, it is invoked on a per-vertex basis. The most important rule of this shader is to perturb the tessellation level for the patch that controls how finely the patch will be tesselated. Usually think about a patch as a triangle or quad. This shader is equivalent to DX11′s hull shader.
• Fixed-function tessellation primitive generator – The role of this new stage is to subdivide the incoming patch based on the tessellation level and related configuration that the unit gets as input.
• Tessellation Evaluation Shader – This new shader type is responsible of calculating the position and other attributes of the vertices produced by the tesselator. This shader is equivalent to DX11′s domain shader.

One important thing to notice is that a new primitive type is introduced, namely a patch. A patch on its own it is not directly or indirectly related to any traditional OpenGL primitive as it cannot be directly rendered. It is used only as the input type for the tesselator, however, a patch supplies the control grid of the geometry to be generated via tessellation so in practice it is most likely to be equivalent with triangles or quads but it is important to remark the difference.

As this is maybe the most well known feature of Shader Model 5.0 hardware I wouldn’t like to talk about it more as everybody knows what is it for and it would be rather long to explain how to use it. Also, it is not the intension of this article to fully cover the usage of all the new features, it is just a quick summarization of the new possibilities.

ARB_texture_buffer_object_rgb32

Yet another extension that introduces an additional format, now for texture buffers. Previously, texture buffers supported only four-component formats, this is extended with three-component formats. As currently there is no any practical use case in my mind when this can be useful, I would rather not come up with one. However, my opinion is that these formats most probably work with reduced performance compared to the four-component ones even though the memory footprint and bandwidth usage is maybe somewhat lower, I have concerns regarding to alignment related performance issues.

ARB_texture_cube_map_array

Those who already use texture arrays to help batching issues and remove unnecessary state changes most probably adore this extension as it enables texture array capabilities also for cube map textures. This comes handy especially in case when many materials use environment cube maps or when shadow cube maps are used for point lights. To come up with even a more concrete example, you can render the shadow cube maps for many hundreds of point lights with a single draw call by taking advantage of the layered rendering capability of geometry shaders and the possibility to bind texture arrays as render targets.

One more thing to notice here is that cube map arrays are already supported by Shader Model 4.1 hardware so the question to the ARB is again there, however, as OpenGL 3.3 still targets Shader Model 4.0 hardware maybe we will see a 3.x version of the specification that will also include this extension. The judgement is up to you whether you agree with me or not.

ARB_texture_gather

Another feature from the repertoire of Shader Model 4.1. This extension introduces new texture fetching functions to the Shading Language that determine a 2×2 footprint of the texture that would be used for linear filtering in a texture lookup and returns a vector consisting of the first component from each of the four texels in the footprint. This is the so called Gather4 texture fetching mode and can be useful to accelerate percentage closer filtering of shadow maps as it can fetch four samples at once. Still, there are some limitations on the use of this fetching mode, one important thing is that a shader cannot use normal and gather fetches on the same sampler. This makes me think about whether this feature is not part of the sampler object state instead of being a Shading Language construct. Anyway, as in typical use cases these limitations does not defeat the goal of the feature, I would not consider this problem a design issue.

ARB_transform_feedback2

The transform feedback mechanism already proved to me that is a great addition to the tool-set of graphics application developers. This feature extends transform feedback with an object type that encapsulates transform feedback related state to enable configuration reuse. Also it provides a way to pause and resume transform feedback mode if, for some reason, some rendering commands should be excluded from the feedback process.

The last and maybe most important benefit of this extension is the ability to draw primitives captured in transform feedback mode without querying the captured primitive count. It is roughly equivalent to DX10′s AutoDraw feature and the purpose of it is to eliminate the need to query the number of previously generated primitives in order to supply it to an OpenGL draw command. This solves the synchronization issues that previously happened between the CPU and the GPU.

One example is when a skeletal animated geometry has to be used in a multipass rendering technique. We can think about traditional forward rendering or when dealing with multiple shadow maps that have to be generated. Anyway, as the calculations needed to perform skeletal animation are rather expensive, it is wastage to perform these calculations in each pass.  A common way to solve this problem is to use transform feedback to capture the geometry emitted by a vertex shader that simply executes the skeletal animation on the input geometry. In subsequent rendering passes this feedback buffer can be used to source the geometry data to eliminate the need to recompute the animation. Without this extension, in such cases the application is most probably stalled until the feedback process ends as it needs to query the number of generated primitives. With this extension, this is solved as we don’t have to know the results of the previous transform feedback in order to issue a draw command that sources the data from the feedback buffer. By the way, this seems to be logical as the information is already on the GPU so why it should ping-pong between the CPU and the GPU?

As I mentioned before, the functionality provided by this extension is equivalent to DX10′s AutoDraw feature. This time my question is really serious: why this feature haven’t been included in OpenGL 3.3? It would provide a great benefit for those who use transform feedback and I don’t see any reason behind not supporting it because, as far as I can tell, it is supported on the corresponding hardware.

ARB_transform_feedback3

Surprisingly, OpenGL 4.0 comes with another transform feedback extension as well but this time a true Shader Model 5.0 feature. The new hardware generation has the ability to emit vertices from the geometry shader to multiple vertex streams. In order to provide clever API support, the ARB decided to relax the previous limitation of transform feedback mode that output can be in either interleaved format or to separate buffers. This new extension enables the use of both together also providing a way to group geometry shader outputs to groups in order to target the individual vertex streams.

The most important benefit of this feature is still that we have separate streams, each with its own primitive emission counter so the outputs should not necessarily have the same granularity. This provide room for very clever rendering techniques. As an example, remember NVIDIA’s Skinned Instancing demo that used one draw call per geometry LOD to sort instance data on a per-LOD basis. Using this extension, this preprocessing step can be done with a single draw call, but the abilities of this feature goes far beyond such a simple use case, I will also talk a bit about another in the next section.

One of my less technical notes is that it seems that the Khronos Group members have good sense of humor. I realized this when I met the “manbearbig” when reading one of the examples in the extension specification.

ARB_draw_indirect

We’ve arrived to the most culminating point in the list of features introduced. It is hard to say such a thing, but in my humble opinion this extension can be the Holy Grail of next generation rendering engines. I will explain why I think so…

The extension provides a way to source the parameters of instanced draw commands from within buffer objects. One naive use case would be to put all the rendering command parameters to a buffer object using the host application and then draw everything with a single command. While this simple method already has its benefits, this feature provides much more flexibility than this. The most revolutionary is that, using this extension, one is able to generate instanced draw commands with the GPU on-the-fly. Together with ARB_transform_feedback3 it is possible to write a completely GPU based scene management system.

Those who remember my Instance Cloud Reduction (ICR) algorithm, presented in the article Instance culling using geometry shaders, know that the required synchronization points between the CPU and the GPU heavily limited the practical utility of the culling technique. By taking advantage of the aforementioned features in case of ICR does not just eliminate the synchronization issues that I’ve spoken of but makes the technique practical also in case of heavily heterogeneous scenes with virtually any number of geometries even if there are multiple number of LOD level for them, and this whole stuff can be done with even less number of draw calls than that of the demo that accompanied my article. As soon as we will see OpenGL 4.0 capable drivers I will write an article about this technique, supplying also a reference implementation.

ARB_texture_query_lod

This extension provides new fragment shader texture functions, namely textureLOD*, that return the results of automatic LOD computations that would be performed if a texture lookup would be performed. These functions return a two-component vector. The X component of the result vector contains information about the mipmap level that would be used if a normal texture lookup would have been made with the same coordinates. This value can be a concrete mipmap level or a value between two levels if trilinear filtering is in use. The Y component of the result holds the computed LOD lambda-prime, see the OpenGL specification in order to check out where it is actually coming from and how it is calculated.

One interesting thing that this extension can be used for is when one implements some shader based filtering and addressing method for textures. As an example, lets take a mega-texture implemented that uses a 3D texture for storage, without actual mipmaps, and the addressing, filtering and mipmapping is done with shader code. As right now this is the only example that came into my mind and this is already awkward enough, I would rather leave the further discussion of the importance of this feature to more competent people.

ARB_gpu_shader5

Basically, this extension is nothing more than a big umbrella feature under what all the additional general or minor API changes go. Just to sum up the miscellaneous features provided by this extension, here is an excerpt from the extension specification:

• Support for indexing into arrays of samplers using non-constant indices.
• Support for indexing into an array of uniform blocks.
• Extending Gather4 with the ability to select any single component of a multi-component texture, to perform per-sample depth comparison, and to specify arbitrary offsets computed at runtime when gathering the 2×2 footprint.
• Support for instanced geometry shaders, where a geometry shader may be run multiple times for each primitive.

For a full list of new facilities introduced by the extension refer to the extension specification.

ARB_gpu_shader_fp64

This extension enables the use of double-precision floating-point data types and arithmetic from within shaders, also providing API entry points for double-precision data where it was missing. While one may think that the added precision is somewhat wastage in case of real-time graphics, it is important to note that GPUs are more and more often used for scientific calculations, not even necessarily in case of graphics related tasks. Taking in consideration this fact, the importance of double-precision floating-point support should not be underestimated. Beside that, maybe standard graphics application developers can also take advantage of the higher precision in some extreme use case scenarios.

The OpenGL Shading Language 4.00

Beside what I’ve already mentioned, there is no important thing to mention regarding to the Shading Language. There where many changes but most of them are simply provide Shading Language support to the API extensions. What I haven’t mentioned so far is the synchronization possibility for tessellation shaders, more implicit conversions, more integer functions, packing and unpacking facilities for floating-point formats and a new qualifier to force precision and disallow optimizations that re-order operations or treat different instances of the same operator with different precision.

Conclusion

I hope that some of you didn’t give up the reading so far. Sorry, but it seems that this article gone wild and still didn’t manage to cover all the topics I intended to talk about. But still, maybe I’ll recap on those subjects later.

Where is direct state access?

The original promise of eliminating the bind-to-modify semantics from the OpenGL API is still not done. The first reaction of many people is still to ask this question. While the bind-to-modify semantics is a rather annoying “feature” of OpenGL, I tend to state that if we are not talking about legacy OpenGL, the importance of direct state access is less and less relevant as we can already heavily reduce the number of state changes and API calls in our applications, thanks to the fast pace evolution of OpenGL. I sincerely think that with a modern rendering engine design built upon the idioms behind the new versions of the OpenGL API one should not face any significant scalability issues due to the outdated bind-to-modify semantics but maybe I’m wrong.

Personally, I have only one problem with the newly released specification versions that I’ve already tried to emphasize several times: the fact that so far many Shader Model 4.x features are missing from the 3.x line of the API specification. Hopefully that will be solved sooner or later, however addressing these issues should happen before the hardware to support will become outdated.

Anyway, we should not have any harsh complains as the Khronos Group did a great job again. They managed to keep again the half-year schedule and they even published two parallel releases at once! If someone still says that the DirectX API is superior compared to OpenGL should think it twice, as it seems that the tendency is that OpenGL just starts to evolve more and more fast. Beside that as now also AMD is being active in the OpenGL world, we can expect good support from both industry and developer community point of view.

My respect for the Khronos Group and thanks for reading the article!

OpenGL 3.2 - Nature

Since the appearance of Shader Model 4.0 people wonder how to take advantage of the newly introduced programmable pipeline stage. The most important feature enabled by geometry shaders is that one can change the amount of emitted primitives inside the pipeline. The first thing that a naive developer would try to do with it is geometry tesselation. However, the new shader performs very bad when used for tesselation in a real life scenario even though there are demos show casting this possibility. If we take a closer look at the new feature we observe that the most revolutionary in it is not that it can raise the number of emitted primitives but that it can discard them. This article would like to present a rendering technique that takes advantage of this aspect of geometry shaders to enable the GPU accelerated culling of higher order primitives.

Geometry shaders can be used for many different advanced rendering techniques that were impossible before the introduction of this flexible programmable shader stage. In this article I would like to present one use case that for me seemed to be one of the most practical application of primitive manipulation possibilities introduced by geometry shaders. As I haven’t seen any whitepaper talking specifically about this particular technique, even if some of them inherently used it, I would dare name the technique myself as Instance Cloud Reduction. I will also present a demo program that shows how to take advantage of the technique in a heavy workload situation.

The idea itself was inspired by AMD’s  tech demo for the Radeon 4800 series cards called March of the Froblins. An almost identical technique presented in this article is used in the mentioned demo for the culling of large amount of animated creatures against the view frustum. Also a somewhat similar technique is used in NVIDIA’s Skinned Instancing demo for determining LOD instance sets. Unfortunately, both demos are for DirectX only and, as far as I can tell, there is no OpenGL demo showing any of the aforementioned rendering techniques.

Motivation

Nowadays, as the computational capabilities of GPUs is growing in a much faster pace than that of CPUs, graphics developers meet more and more optimization problems related to CPU bound applications. More and more focus is on minimizing the number of driver invocations, actually that’s what motivated the restructuring of the two most commonly used graphics APIs. As a result we have now DirectX 10+ and OpenGL 3+. However, even if the introduction of geometry instancing, texture arrays and local memory buffer storage for the most important inputs of the rendering, there is still need for wise decisions from graphics programmers to take full advantage of the horsepower coming with the latest GPUs.

Earlier graphics applications strongly relied on CPU based culling techniques, whether it be the usage of the quite outdated BSPs or the more generic and still heavily applied hierarchical culling techniques. We’ve already reached the point that sometimes even the most efficient CPU based culling techniques seem to be too expensive and usually introduce the small batch problem. Instanced rendering is not an exception.

The applicability of geometry instancing is strongly limited by several factors. One of the most important ones is the culling of instanced geometries. One may choose to cull these objects in the same fashion as others, using the CPU, but that usually breaks the batch and maybe we loose the benefits of geometry instancing. It is more and more imminent to have a GPU based alternative. Without CPU based culling, by sending the whole bunch of instances down the graphics pipeline may choke our vertex processor in case we have high poly geometries and quite large amount of instances of it.

The rendering technique presented in this article will try to achieve this goal. We will use a multi-pass technique that in the first pass culls the object instances against the view frustum using the GPU and in the second pass renders only those instances that are likely to be visible in the final scene. This way we can severely reduce the amount of vertex data sent through the graphics pipeline.

Implementation

For some people it might seem that the promise for such a technique is simply too naive and is most probably relying on very exotic OpenGL features, heavy misuse of some basic features or need of data conversions during the frame rendering. Wondrously, this is not the case as we have all we need in OpenGL 3.2 to implement the object culling method sketched above. All we need are the followings:

• instanced rendering (core since OpenGL 3.1)
• geometry shaders (core since OpenGL 3.2)
• transform feedback (core since OpenGL 3.0)
• uniform or texture buffers (core since OpenGL 3.1)

The method itself is a multi-pass rendering technique, however, unlike other multi-pass rendering techniques it does not produce any fragments in the first pass, instead the first pass does the view frustum culling and processes data entirely only inside buffer objects.

Culling pass

In the first pass we will feed the graphics pipeline with information about the instances that are needed to perform the view frustum culling. For this we need two inputs for the executed shaders in order to be able to perform the required calculations:

1. Instance transformation data (whether it be a simple transformation matrix or quaternions or whatever) -- This preferably comes from one or more buffer objects that are bound as vertex buffers to the context.
2. Object extents information -- Beside the instance positions we have to know the extents of an instance in order to perform correct culling. This can be either a single float representing the object radius if we choose to use bounding spheres for the culling or a three-dimensional extent vector if we would like to use bounding boxes.

Using these as input we can feed in the instance transformation data as attributes of point primitives to our culling shader. The culling shader is composed of a vertex and a geometry shader. In a typical setup the role of each is the following: the vertex shader determines whether the actual object instance’s bounding volume is inside the view frustum and sends a flag about the culling to the geometry shader, that will emit the instance data to the destination buffer if the flag says that the instance is likely to be visible or does not emit anything if it is determined that the object instance is out of view.

Next, transform feedback is used to capture the primitives emitted by the geometry shader into another buffer object that will be used in the actual rendering pass to source instance transformation data. Beside this, we also need to have an asynchronous query to determine the number of primitives generated to know how many instances of the object do we actually need to render. The following figure shows the workflow of the first pass:

Instance Cloud Reduction - Pass 1: Culling

The actual geometry shader implementation needed to perform the actual culling based on the view frustum check performed by the vertex shader should look like the following chunk:

#version 150 core

layout(points) in;
layout(points, max_vertices = 1) out;

in vec4 OrigPosition[1];
flat in int objectVisible[1];

out vec4 CulledPosition;

void main() {

	/* only emit primitive if the object is visible */
	if ( objectVisible[0] == 1 )
	{
		CulledPosition = OrigPosition[0];
		EmitVertex();
		EndPrimitive();
	}
}

In this example we used only simply a four-component position vector for the instance transformation data but the technique works well for transformation matrices and quaternions as well.

One more thing is that beside that we set up transform feedback in a way that we feed our buffer object dedicated for the culled instance data and we also started an asynchronous query to be able to determine the number of primitives written into the buffer object, it is also useful to turn of rasterization as we wouldn’t like to produce any fragments as a result of the first pass.

Rendering pass

In the second pass there is nothing special to do. Simply use whatever rendering setup you would like to use. The only things that need to be changed in this step compared to your already existing rendering path is that the instance data for the rendering must be sourced from the generated culled instance data buffer and, as a result, the number of instances passed for the instanced drawing functions shall be changed in order to render only the visible instances. This number can be read from the asynchronous query’s result that we started in the first pass.

The instance data in the rendering pass can be, of course, sourced from either a uniform or a texture buffer object. This depends on the actual use case and is more clearly explained in the article Uniform Buffers VS Texture Buffers.

Important note is that when one has to deal with several instanced geometries it is recommended to do the culling phase prior to rendering any instanced primitives because of the following reasons:

• The result of the first instance cloud’s culling is more likely to be finished on the GPU so no sync issues arise from reading the asynchronous query result to determine the number of visible instances.
• Probably less state changes are needed as very different setup is required by the two passes.
• Results in tidier renderer design as culling is clearly separated from actual rendering.

Putting everything together, the application of the presented technique would result in the following workflow on the GPU:

Instance Cloud Reduction - Combined view of Pass 1 + Pass 2

Conclusion

We’ve seen that the presented advanced rendering technique is able to help in situations when we have to deal with large number of instanced geometries and how to take advantage of the latest features of graphics cards and OpenGL to perform view frustum culling calculations on the GPU. This prevents us from having to deal with complicated and expensive CPU based object culling methods that break the drawing batches, especially when dealing with dynamic objects. For ease the decision whether to incorporate this technique in your rendering engine I would like to present the advantages and disadvantages of it.

Advantages:

• Heavily reduces the amount of processed data in a naive implementation.
• No need for any space partitioning methods in the host application to handle the culling of dynamic objects.
• Can handle huge amount of instanced objects due to the enormous horsepower of today’s GPUs.
• Scales well with increased number of instances as the per-instance calculation is relatively low.
• Relies strictly on OpenGL 3.2 core features.
• No need for OpenCL capable hardware.

Disadvantages:

• Needs an extra rendering pass to perform the culling.
• Requires the usage of asynchronous queries to determine the number of visible instances.

I hope you agree with me and think about this technique as one more step towards fully GPU based scene management. If you have any remarks or improvement ideas regarding to the rendering technique itself feel free to tell me.

The Demo

As I promised, the technique presented above comes with a live demo that actually took most of my time dedicated to writing this blog in the last two weeks. The demo itself is more like a technical show cast rather than a presentation of a real-life use case scenario.

First of all, I used high polygon count models for the rendering to emphasize the amount of time the culling phase spares from the very valuable time of our GPU. In a real world application one would never do something like this. As a result, the demo is more like a benchmark than an interactive application. However, maybe on high-end graphics cards it can perform pretty well.

The demo scene consists of two object types: trees and grass blocks. The tree model is further divided into two parts as they need different textures: the tree trunk and the tree foliage. Obviously, this additional burden can be prevented by using texture arrays to avoid the need of separate draw calls to render the trunk and the foliage.

The tree trunk consists of 33138 triangles, the tree foliage has 16069 triangles and the faking-free grass block consists of 8961 triangles which I had to model myself as didn’t found any suitable model. Actually this modeling step consumed quite a reasonable amount of my time spent with the demo as I’m not an expert in this domain.As you can see, these models are not the ones that one might use in an interactive real-time application like games. However, they seemed to be very suitable for the purpose of the demonstration.

What really kicks off the boundaries of GPUs is that the demo renders 10,000 trees and 250,000 grass blocks using instancing. This ends up in more than 2.7 billion triangles in the scene. This is far more that a GPU can handle without the aid of some scene management and culling. However, we will use no scene management at all and the only culling method that we will use is the one presented in this article.

The actual results are quite promising. The view frustum culling step usually spares more than 99.9% of the GPU horsepower as the amount of actually rendered triangles after the culling step is far below 2 million triangles. This is still quite much but as we use high polygon count models and we don’t use any LOD techniques this seems reasonable.

Even if the demo scene statistics doesn’t seem like a typical use case scenario, the ease of the implementation and the compelling visual results made me pleased anyway:

www.youtube.com/watch?v=srbOFTLTe8k

On my Radeon HD2600XT I have achieved 6-7 frames per second which is acceptable taking in consideration the huge amount of geometry data still passed to the graphics card. On more recent cards I suppose it should run with good frame rates, however, due to the lack of hardware to test on, these are my only results. If anybody manages to take a better screen capture than mine above then please let me know.

Implementation details

Just to tell a few words about what techniques and tricks I’ve used during the creation of the demo here is a listing of the most important ones:

• Three models are used as mentioned previously with high instance counts with over 2.7 billion of total triangles in the scene as mentioned already.
• Three 512x512 RGBA textures are used for the models that are partially handmade, and again, I’m not a texture artist so sorry if they don’t look flawless.
• The wavefront model and TGA image loader that accompany the demo are very roughly implemented only for the demo so I would strongly encourage you not to use it to any purpose as it handles only a subset of the possibilities of the file formats.
• The vertex data from the wavefront model files is transferred in a very naive way so vertex reuse isn’t taken into account.
• The instance data consists of simple four-component vectors representing the world-space position of the instance. This seemed to be the most simple for the demonstration purposes.
• In the second pass, the instance data is sourced from a texture buffer but not really because the visible instance count exceeded the amount that would fit in a uniform buffer. I used texture buffers because for this simple demonstration they seemed to be a little bit more easy to be integrated.
• The morphing effect that simulated wind blow is done using hard-coded geometry deformation in the vertex shader. It is not physically correct but visually compelling.
• The lighting is a simple directional light using Phong’s shading and reflection model.
• Simple fog is simulated with some awkward formula that I’ve chosen after a few test runs.
• Alpha testing is achieved by using the discard operation in the fragment shader.

Driver issues

During the development of the demonstration program I’ve met several driver related problems as I’ve never used so heavily the latest OpenGL features previously. I’ve worked with Catalyst 9.12 and 10.1 but both seemed to lack of a proper GLSL compiler. Here are some of the issues I’ve met:

• When I’ve forgot to declare the varyings in the geometry shader as arrays like the standard requires then still the driver hasn’t complained about any syntax error but when tried to execute the code the program crashed.
• Except the texture sampler uniform, all other uniforms failed to work when used in the fragment shader only so I’ve put them all in the vertex shader.
• For loops seemed not to work when used inside the geometry shader, that’s why the culling itself is done in the vertex shader in the demo.

All these problems resulted in nasty tricks to make things working and ended up in awful shader code. Sorry for that. At least now it works on my configuration but pretty unsure whether it will work on other graphics card and driver combos. Please report me any success or failure when trying out the demo. Anyway, be sure to have the latest graphics drivers installed as, at least in case of AMD, OpenGL 3.2 drivers came out only at the fall of 2009.

Edit:

Thanks to the information got from Pierre Boudier from AMD I’ve updated both the source and binary releases to support the latest drivers properly. The problem was that I didn’t use attribute location binding as specified in the standard.

Also have to mention that with my new Radeon HD5770 I managed to achieve over 90 frames per second that actually show that this technique can be in fact used for games and other interactive applications.

One more thing in the end. As you know this version of the Nature demo uses a texture buffer to source instance positions. I plan to create another version that will take advantage of the instanced arrays introduced in core with OpenGL 3.4. I expect quite a reasonable speedup as that would eliminate the need for texture fetches in the vertex array by rather dedicating a vertex fetcher for the purpose thus increasing the overall performance of the technique.

Binary release

Platform: Windows
Dependency: OpenGL 3.2 capable graphics driver
Download link: nature12_win32.zip (3.58MB)
Comments: Includes the update that makes it work even with the latest drivers.

Full source code

Language: C++
Platform: cross-platform
Dependency: GLEW, SFML, GLM
Download link: nature12_src.zip (12.6KB)
Comments: Sorry for the many dependencies, however, I would recommend the mentioned libraries for everybody who is doing OpenGL development.