piglit (III): How to write GLSL shader tests

Posted by Samuel Iglesias on May 29, 2015

In earlier posts I talked about how to install piglit on your system and run the first time and how to tailor your piglit run. I have given a talk in FOSDEM 2015 about how to test your OpenGL drivers using Free Software where I explained how to run piglit and dEQP. This post and the next one are going to introduce the topic of creating new tests for piglit, as this was not covered before in this post series.

Brief introduction

Before start talking about how to create a GLSL test, let me explain you a couple of things about how piglit organizes its tests.

First is that all the tests are defined inside tests/ directory.

  • The ones related to a specific spec (like OpenGL extension, GL version, GLSL version, etc) are placed inside tests/spec subdirectory.
  • Shader tests are usually defined inside  tests/shaders/ directory, but not always. If they are specific to a OpenGL extension, they will be in the its subdirectory (see first bullet).
  • Tests specific to other APIs: tests/egl, tests/cl, etc.
  • Etc.

Take your time browsing all these directories and figure out the best place for your tests or, if you are looking for a specific test, where it is likely placed.

Second thing is that binary tests should be defined inside tests/all.py to be executed in a full piglit run. This is not the case for GLSL tests or shader runner tests as we are going to see in this post.

The most common language to write these tests is C but there are a couple of alternatives if you are interested on writing GLSL shader tests. Let's start with a couple of examples of GLSL shader tests as it is the easiest way to contribute new tests to piglit.

GLSL compiler test

When creating a GLSL compiler test, most of the time you can avoid to write a C code example with all those bells and whistles. Actually, it is pretty straightforward if you just want to check that a GLSL shader compiles successfully or not according to your expectations.

Usually, the rule of thumb is to keep GLSL tests simple and focused to detect failures (or success) in shaders' compilation. For example, you want to check that defining a shader storage buffer block is only possible if ARB_shader_storage_buffer_object extension is enabled in the driver.

#version 120
#extension GL_ARB_shader_storage_buffer_object: disable

buffer ssbo {
    vec4 a;
};

void foo(void) {
}

Once you write the GLSL shader, save it to a file named like extension-disabled-shader-storage-block.frag. There are several suffixes that you can use depending of the GLSL shader type: .vert for vertex shader, .geom for geometry shader, .frag for fragment shader. Notice that the name itself is a summary of what the test does, so other people can find what it does without opening the file.

There is a piglit binary called glslparser that it is going to pick this shader and compile it checking for errors. This binary needs some parameters to know how to run this test and the expected result, so we provide them. Add at the top of the shader the following content:

// [config]
// expect_result: fail
// glsl_version: 1.20
// require_extensions: GL_ARB_shader_storage_buffer_object
// [end config]

There we write down that we expect the test to fail, which is the minimum supported GLSL version to run this test and the required extensions. At this case we need GLSL 1.20 version and GL_ARB_shader_storage_buffer_object extension.

// [config]
// expect_result: fail
// glsl_version: 1.20
// require_extensions: GL_ARB_shader_storage_buffer_object
// [end config]

#version 120
#extension GL_ARB_shader_storage_buffer_object: disable

buffer ssbo {
    vec4 a;
};

void foo(void) {
}

Once you have it, you should save it in the proper place. The directory will be the same than the extension name it is going to test, usually it is saved in a subdirectory called compiler because we are testing the GLSL shader compilation, for this example: tests/spec/arb_shader_storage_buffer_object/compiler.

And that's it. Next time you run piglit with tests/all.py profile, one script will find this test, parse it and execute glslparser with the provided configuration checking if the result of your shader is the expected one or not.

You can execute it manually by running the following command:

$ bin/glslparser tests/spec/arb_shader_storage_buffer_object/compiler/extension-disabled-shader-storage-block.frag fail 1.20 GL_ARB_shader_storage_buffer_object

As you see, the last arguments come from the config we defined in the test file but we added them by hand.

It is possible to link a GLSL shader and look for errors by using check_link with true or false, depending if you expect the test to succeed or to fail at link time:

// check_link: true

Nevertheless, it only links one shader. If you want to link more than one shader, I recommend you to use another type of piglit tests (see next section).

As you see, you can easily write new GLSL shader tests to verify some bits of a specification: valid/invalid keywords, initialization, special cases explained in the spec, etc.

GLSL shader linker test

Sometimes compiling/linking only one shader is not enough, there are cases that you want to compile and link different but related shaders: link a vertex and a fragment shader or a geometry shader between them, etc. In the previous example, we saw that the GLSL shader test does not link different shaders (actually there is only one). When there are several GLSL shaders, you need to use another type of piglit tests: shader_runner tests.

As usual, you first start writing the GLSL shaders. In some cases, you can substitute one shader by its pass-through counterpart to avoid writing "useless" shaders when there is not any user provided interface dependency.

Let's start with an example of a pass-through vertex shader:

# ARB_gpu_shader5 spec says:
#   "If an implementation supports  vertex streams, the individual
#   streams are numbered 0 through -1"
#
# This test verifies that a link error occurs if EmitStreamVertex()
# is called with a stream value which is negative.

[require]
GLSL >= 1.50
GL_ARB_gpu_shader5

[vertex shader passthrough]

[geometry shader]

#extension GL_ARB_gpu_shader5 : enable

layout(points) in;
layout(points, max_vertices=3) out;

void main()
{
    gl_Position = vec4(1.0, 1.0, 1.0, 1.0);
    EmitStreamVertex(-1);
    EndStreamPrimitive(-1);
}

[fragment shader]

out vec4 color;

void main()
{
  color = vec4(0.0, 1.0, 0.0, 1.0);
}

[test]
link error

The file starts with a comment describing what the test does and which spec (or part of it) is checking.

Next step is very similar to the previous example: specify the minimum GL and/or GLSL version required and the extensions needed for the test execution. This information is written in the [require] section.

Then the different GLSL shader definitions start: vertex shader is pass-through as it has no user-defined interface dependency to next stage (geometry shader at this case). After it, it defines the  geometry and fragment shaders and finally the test commands to run. In this case, we expect the test to fail at link time.

However you are not limited to link checking, there are other available commands to run in the test commands part:

    • Load uniform values
      • uniform <type> <name> <value>
uniform vec4 color 0.0 0.5 0.0 0.0
    • Draw rectangles
      • draw rect <coordinates>
draw rect -1 -1 2 2
    • Probe a pixel color value and compare it to an expected value
      • probe {rgb, rgba} 
probe rgb 1 1 0.0 1.0 0.0
    • Probe all pixel values.
      • probe all {rgb,rgba} 
probe all rgba 0.0 1.0 0.0 0.0

Or even more complex commands:

    • Load data into a vertex array object and render primitives using that vertex array data.
[vertex data]
vertex/float/3
 1.0  1.0  1.0
-1.0  1.0  1.0
-1.0 -1.0  1.0
 1.0 -1.0  1.0

[test]
draw arrays GL_TRIANGLE_FAN 0 4
    •  Relative probe pixel color
relative probe rgb (.5, .5) (0.0, 1.0, 0.0)
    • And much more:
[fragment program]
!!ARBfp1.0
OPTION ARB_fragment_program_shadow;
TXP result.color, fragment.texcoord[0], texture[0], SHADOWRECT;
END

[test]
texture shadowRect 0 (32, 32)
texparameter Rect depth_mode luminance
texparameter Rect compare_func greater

Just check other tests to figure out if they are doing something like what you want to do in your test and copy the interesting bits!

Save the file with a good name describing briefly what it does (example's filename is stream-negative-value.shader_test) and place it in the corresponding subdirectory. For this example, the proper place is under GL_ARB_gpu_shader5 test directory and linker subdirectory because this is a linker test: tests/spec/arb_gpu_shader5/linker/

In order to execute this test with tests/all.py profile, you don't need to do anything else. A script will find this test file and call shader_runner binary with the filename as a parameter.

In case you want to run it by yourself, the command is easy:

$ bin/shader_runner tests/spec/arb_gpu_shader5/linker/stream-negative-value.shader_test

Next post will cover the creation of GL binary tests within the piglit framework. I hope this post and next one will help you to contribute to piglit!