Last post I gave an introduction to piglit, an open-source test suite for OpenGL implementations. On that post, I explained how to compile the source code, how to run full piglit test suite and how to analyze the results.
However, you can tailor a piglit run to execute the specific tests that match your needs. For example I might want to check if a specific subset of tests pass because I am testing a new feature in Mesa as part of my job at Igalia.
Configure the piglit run
There are several parameters that configure the piglit run to match our needs:
- --dry-run: do not execute the tests. Very useful to check if the parameters you give are doing what you expect.
- --valgrind: It runs Valgrind memcheck on each test program that it's going to execute. If it finds any error, the test fails and it saves the valgrind output into results file.
- --all-concurrent: run all tests concurrently.
- --no-concurrency: disable concurrent test runs.
- --sync: sync results to disk after every test so you don't lose information if something bad happens to your system.
- --platform {glx,x11_egl,wayland,gbm,mixed_glx_egl}: if you compiled waffle with different windows system, this is the name of the one passed to waffle.
There is a help parameter if you want further information:
$ ./piglit run -h
Skip tests
Sometimes you prefer to skip some tests because they are causing a GPU hang or taking a lot of time and their output is not interesting for you. The way to skip a test is using the -x parameter:
$ ./piglit run tests/all -x texture-rg results/all-reference-except-texture-rg
Run specific tests
You can run some specific subset of tests using a filter by test name. Remember that it's filtering by test name but not by functionality, so you might miss some test programs that check the same functionality.
$ ./piglit run tests/all -t color -t format results/color-format
Also, you can concatenate more parameters to add/remove more tests that are going to be executed.
$ ./piglit run tests/all -t color -t format -t tex -x texture-rg results/color-format-tex
Run standalone tests
There is another way to run standalone tests apart from using the -t argument to piglit run. You might be interested in this way if you want to run GDB on a failing tests to debug what is going on.
If we remember how was the HTML output for a given test:
The command field specifies the program name executed for this test together with all its arguments. Let me explain what they mean in this example.
- Some binaries receives arguments that specify the data type to test (like at this case GL_RGB16_SNORM) or any other data: number of samples, msaa, etc.
- -fbo: it draws in an off-screen framebuffer.
- -auto: it automatically run the test and when it finishes, it closes the window and prints if the test failed or passed.
Occasionally, you might run the test program without fbo and auto parameters because you want to see what it draws on the window to understand better the bug you are debugging.
Create your own test profile
Besides explicitly adding/removing tests from tests/all.py (or other profiles) in the piglit run command, there is other way of running a specific subset of tests: profiles.
A test profile in piglit is a script written in Python that selects the tests to execute.
There are several profiles already defined in piglit, two of them we already saw them in this post series: tests/sanity.tests and tests/all. First one is useful to check if piglit was correctly compiled together with its dependencies, while the second runs all piglit tests in one shot. Of course there are more profiles inside tests/ directory: cl.py (OpenCL tests), es3conform.py (OpenGL ES 3.0 conformance tests), gpu.py, etc.
Eventually you will write your own profile because adding/removing tests in the console command is tiresome and prone to errors.
This is an example of a profile based on tests/gpu.py that I was recently using for testing the gallium llvmpipe driver.
# -*- coding: utf-8 -*-
# quick.tests minus compiler tests.
from tests.quick import profile
from framework.glsl_parser_test import GLSLParserTest
__all__ = ['profile']
# Remove all glsl_parser_tests, as they are compiler test
profile.filter_tests(lambda p, t: not isinstance(t, GLSLParserTest))
# Drop ARB_vertex_program/ARB_fragment_program compiler tests.
del profile.tests['spec']['ARB_vertex_program']
del profile.tests['spec']['ARB_fragment_program']
del profile.tests['asmparsertest']
# Too much time on gallium llvmpipe
profile.tests['spec']['!OpenGL 1.0'].pop('gl-1.0-front-invalidate-back', None)
profile.tests['spec']['!OpenGL 1.0'].pop('gl-1.0-swapbuffers-behavior', None)
profile.tests['spec']['!OpenGL 1.1'].pop('read-front', None)
profile.tests['spec']['!OpenGL 1.1'].pop('read-front clear-front-first', None)
profile.tests['spec']['!OpenGL 1.1'].pop('drawbuffer-modes', None)
profile.tests['spec']['EXT_framebuffer_blit'].pop('fbo-sys-blit', None)
profile.tests['spec']['EXT_framebuffer_blit'].pop('fbo-sys-sub-blit', None)
As you see, it picks the test lists from quick.py but with some changes: it drops all the tests related to a couple of OpenGL extensions (ARB_vertex_program and ARB_fragment_program) and it drops other seven tests because they took too much time when I was testing them with gallium llvmpipe driver on my laptop.
I recommend you to spend some time playing with profiles as it is a very powerful tool when tailoring a piglit run.
What else?
In some situations, you want to know which test produces kernel error messages in dmesg, or which test is being ran now. For both cases, piglit provides parameters for run command:
$ ./piglit run tests/all -v --dmesg results/all-reference
- Verbose (-v) prints a line of output for each test before and after it runs, so you can find which takes longer or outputs errors without need to wait until piglit finishes.
- Dmesg (--dmesg): it saves the difference of dmesg outputs before and after each test program is executed. Thanks to that, you can easily find which test produces kernel errors in the graphics device driver.
Wrapping up
After giving an introduction to piglit in my blog, this post explains how to configure a piglit run, change the list of tests to execute and how to run a standalone test. As you see, piglit is very powerful tool that requires some time to learn how to use it appropriately.
In next post I will talk about a more advanced topic: how to create your own tests.