<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:base="en">
	<title>Tvrtko&#39;s blog</title>
	<subtitle>Graphics, kernel stuff, etc</subtitle>
	<link href="https://blogs.igalia.com/tursulin/feed/feed.xml" rel="self"/>
	<link href="https://blogs.igalia.com/tursulin/"/>
	<updated>2025-06-23T00:00:00Z</updated>
	<id>https://blogs.igalia.com/</id>
	<author>
		<name>Tvrtko Ursulin</name>
		<email>tursulin@igalia.com</email>
	</author>
	
	<entry>
		<title>Fair(er) DRM GPU scheduler</title>
		<link href="https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/"/>
		<updated>2025-06-23T00:00:00Z</updated>
		<id>https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/</id>
		<content type="html">&lt;h2 id=&quot;introduction&quot; tabindex=&quot;-1&quot;&gt;Introduction &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The DRM GPU scheduler is a shared Direct Rendering Manager (DRM) Linux Kernel level component used by a number of GPU drivers for managing job submissions from multiple rendering contexts to the hardware. Some of the basic functions it can provide are dependency resolving, timeout detection, and most importantly for this article, scheduling algorithms whose essential purpose is picking the next queued unit of work to execute once there is capacity on the GPU.&lt;/p&gt;
&lt;p&gt;Different kernel drivers use the scheduler in slightly different ways - some simply need the dependency resolving and timeout detection part, while the actual scheduling happens in the proprietary firmware, while others rely on the scheduler’s algorithms for choosing what to run next. The latter ones is what the work described here is suggesting to improve.&lt;/p&gt;
&lt;p&gt;More details about the other functionality provided by the scheduler, including some low level implementation details, are available in the generated kernel documentation repository&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref1&quot;&gt;[1]&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;h2 id=&quot;basic-concepts-and-terminology&quot; tabindex=&quot;-1&quot;&gt;Basic concepts and terminology &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Three DRM scheduler data structures (or objects) are relevant for this topic: the scheduler, scheduling entities and jobs.&lt;/p&gt;
&lt;p&gt;First we have a &lt;em&gt;scheduler&lt;/em&gt; itself, which usually corresponds with some hardware unit which can execute certain types of work. For example, the render engine can often be single hardware instance in a GPU and needs arbitration for multiple clients to be able to use it simultaneously.&lt;/p&gt;
&lt;p&gt;Then there are &lt;em&gt;scheduling entities&lt;/em&gt;, or in short entities, which broadly speaking correspond with userspace rendering contexts. Typically when an userspace client opens a render node, one such rendering context is created. Some drivers also allow userspace to create multiple contexts per open file.&lt;/p&gt;
&lt;p&gt;Finally there are &lt;em&gt;jobs&lt;/em&gt; which represent units of work submitted from userspace into the kernel. These are typically created as a result of userspace doing an &lt;em&gt;ioctl(2)&lt;/em&gt; operation, which are specific to the driver in question.&lt;/p&gt;
&lt;p&gt;Jobs are usually associated with entities and entities are then executed by schedulers. Each scheduler instance will have a list of runnable entities (entities with least one queued job) and when the GPU is available to execute something it will need to pick one of them.&lt;/p&gt;
&lt;p&gt;Typically every userspace client will submit at least one such job per rendered frame and the desktop compositor may issue one or more to render the final screen image. Hence, on a busy graphical desktop, we can find dozens of active entities submitting multiple GPU jobs, sixty or more times per second.&lt;/p&gt;
&lt;h2 id=&quot;the-current-scheduling-algorithm&quot; tabindex=&quot;-1&quot;&gt;The current scheduling algorithm &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;In order to select the next entity to run, the scheduler defaults to the First In First Out (FIFO) mode of operation where selection criteria is the job submit time.&lt;/p&gt;
&lt;p&gt;The FIFO algorithm in general has some well known disadvantages around the areas of fairness and latency, and also because selection criteria is based on job submit time, it couples the selection with the CPU scheduler, which is also not desirable because it creates an artifical coupling between different schedulers, different sets of tasks (CPU processes and GPU tasks), and different hardware blocks.&lt;/p&gt;
&lt;p&gt;This is further amplified by the lack of guarantee that clients are submitting jobs with equal pacing (not all clients may be synchronised to the display refresh rate, or not all may be able to maintain it), the fact their per frame submissions may consist of unequal number of jobs, and last but not least the lack of preemption support. The latter is true both for the DRM scheduler itself, but also for many GPUs in their hardware capabilities.&lt;/p&gt;
&lt;p&gt;Apart from uneven GPU time distribution, the end result of the FIFO algorithm picking the sub-optimal entity can be dropped frames and choppy rendering.&lt;/p&gt;
&lt;h2 id=&quot;round-robin-backup-algorithm&quot; tabindex=&quot;-1&quot;&gt;Round-robin backup algorithm &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Apart from the default FIFO scheduling algorithm, the scheduler also implements the round-robin (RR) strategy, which can be selected as an alternative at kernel boot time via a kernel argument. Round-robin, however, suffers from its own set of problems.&lt;/p&gt;
&lt;p&gt;Whereas round-robin is typically considered a fair algorithm when used in systems with preemption support and ability to assign fixed execution quanta, in the context of GPU scheduling this fairness property does not hold. Here quanta are defined by userspace job submissions and, as mentioned before, the number of submitted jobs per rendered frame can also differ between different clients.&lt;/p&gt;
&lt;p&gt;The final result can again be unfair distribution of GPU time and missed deadlines.&lt;/p&gt;
&lt;p&gt;In fact, round-robin was the initial and only algorithm until FIFO was added to resolve some of these issue. More can be read in the relevant kernel commit. &lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref2&quot;&gt;[2]&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;
&lt;h2 id=&quot;priority-starvation-issues&quot; tabindex=&quot;-1&quot;&gt;Priority starvation issues &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Another issue in the current scheduler design are the priority queues and the strict priority order execution.&lt;/p&gt;
&lt;p&gt;Priority queues serve the purpose of implementing support for entity priority, which usually maps to userspace constructs such as VK_EXT_global_priority and similar. If we look at the wording for this specific Vulkan extension, it is described like this&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref3&quot;&gt;[3]&lt;/a&gt;&lt;/sup&gt;:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;The driver implementation *will attempt* to skew hardware resource allocation in favour of the higher-priority task. Therefore, higher-priority work *may retain similar* latency and throughput characteristics even if the system is congested with lower priority work.
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;As emphasised, the wording is giving implementations leeway to not be entirely strict, while the current scheduler implementation only executes lower priorities when the higher priority queues are all empty. This over strictness can lead to complete starvation of the lower priorities.&lt;/p&gt;
&lt;h2 id=&quot;fair-er-algorithm&quot; tabindex=&quot;-1&quot;&gt;Fair(er) algorithm &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;To solve both the issue of the weak scheduling algorithm and the issue of priority starvation we tried an algorithm inspired by the Linux kernel’s original Completely Fair Scheduler (CFS)&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref4&quot;&gt;[4]&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;With this algorithm the next entity to run will be the one with least virtual GPU time spent so far, where &lt;em&gt;virtual GPU time&lt;/em&gt; is calculated from the the real GPU time scaled by a factor based on the entity priority.&lt;/p&gt;
&lt;p&gt;Since the scheduler already manages a &lt;em&gt;rbtree&lt;/em&gt;&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref5&quot;&gt;[5]&lt;/a&gt;&lt;/sup&gt; of entities, sorted by the job submit timestamp, we were able to simply replace that timestamp with the calculated virtual GPU time.&lt;/p&gt;
&lt;p&gt;When an entity has nothing more to run it gets removed from the tree and we store the delta between its virtual GPU time and the top of the queue. And when the entity re-enters the tree with a fresh submission, this delta is used to give it a new relative position considering the current head of the queue.&lt;/p&gt;
&lt;p&gt;Because the scheduler does not currently track GPU time spent per entity this is something that we needed to add to make this possible. It however did not pose a significant challenge, apart having a slight weakness with the up to date utilisation potentially lagging slightly behind the actual numbers due some DRM scheduler internal design choices. But that is a different and wider topic which is out of the intended scope for this write-up.&lt;/p&gt;
&lt;p&gt;The virtual GPU time selection criteria largely decouples the scheduling decisions from job submission times, to an extent from submission patterns too, and allows for more fair GPU time distribution. With a caveat that it is still not entirely fair because, as mentioned before, neither the DRM scheduler nor many GPUs support preemption, which would be required for more fairness.&lt;/p&gt;
&lt;h2 id=&quot;solving-the-priority-starvation&quot; tabindex=&quot;-1&quot;&gt;Solving the priority starvation &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Because priority is now consolidated into a single entity selection criteria we were also able to remove the per priority queues and eliminate priority based starvation. All entities are now in a single run queue, sorted by the virtual GPU time, and the relative distribution of GPU time between entities of different priorities is controlled by the scaling factor which converts the real GPU time into virtual GPU time.&lt;/p&gt;
&lt;h2 id=&quot;code-base-simplification&quot; tabindex=&quot;-1&quot;&gt;Code base simplification &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Another benefit of being able to remove per priority run queues is a code base simplification. Going further than that, if we are able to establish that the fair scheduling algorithm has no regressions compared to FIFO and RR, we can also remove those two which further consolidates the scheduler. So far no regressions have indeed been identified.&lt;/p&gt;
&lt;h2 id=&quot;real-world-examples&quot; tabindex=&quot;-1&quot;&gt;Real world examples &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;As an first example we set up three demanding graphical clients, one of which was set to run with low priority (&lt;em&gt;VK_QUEUE_GLOBAL_PRIORITY_LOW_EXT&lt;/em&gt;).&lt;/p&gt;
&lt;p&gt;One client is the Unigine Heaven benchmark&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref6&quot;&gt;[6]&lt;/a&gt;&lt;/sup&gt; which is simulating a game, while the other two are two instances of the &lt;em&gt;deferredmultisampling&lt;/em&gt; Vulkan demo from Sascha Willems&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref7&quot;&gt;[7]&lt;/a&gt;&lt;/sup&gt;, modified to support running with the user specified global priority. Those two are simulating very heavy GPU load running simultaneouosly with the game.&lt;/p&gt;
&lt;p&gt;All tests are run on a Valve Steam Deck OLED with an AMD integrated GPU.&lt;/p&gt;
&lt;p&gt;First we try the current FIFO based scheduler and we monitor the GPU utilisation using the &lt;em&gt;gputop&lt;/em&gt;&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref8&quot;&gt;[8]&lt;/a&gt;&lt;/sup&gt; tool. We can observe two things:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;That the distribution of GPU time between the normal priority clients is not equal.&lt;/li&gt;
&lt;li&gt;That the low priority client is not getting any GPU time.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/avif&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/h4fuWuxOck-2224.avif 2224w&quot;&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/h4fuWuxOck-2224.webp 2224w&quot;&gt;&lt;img alt=&quot;FIFO scheduling uneven GPU distribution and low priority starvation&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;https://blogs.igalia.com/tursulin/img/h4fuWuxOck-2224.png&quot; width=&quot;2224&quot; height=&quot;612&quot;&gt;&lt;/picture&gt;&lt;/p&gt;
&lt;p&gt;Switching to the CFS inspired (fair) scheduler the situation changes drastically:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;GPU time distribution between normal priority clients is much closer together.&lt;/li&gt;
&lt;li&gt;Low priority client is not starved, but receiving a small share of the GPU.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/avif&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/NtooPsstET-2231.avif 2231w&quot;&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/NtooPsstET-2231.webp 2231w&quot;&gt;&lt;img alt=&quot;New scheduler even GPU distribution and no low priority starvation&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;https://blogs.igalia.com/tursulin/img/NtooPsstET-2231.png&quot; width=&quot;2231&quot; height=&quot;604&quot;&gt;&lt;/picture&gt;&lt;/p&gt;
&lt;p&gt;Note that the absolute numbers are not static but represent a trend.&lt;/p&gt;
&lt;p&gt;This proves that the new algorithm can make the low priority useful for running heavy GPU tasks in the background, similar to what can be done on the CPU side of things using the &lt;em&gt;nice(1)&lt;/em&gt; process priorities.&lt;/p&gt;
&lt;h2 id=&quot;synthetic-tests&quot; tabindex=&quot;-1&quot;&gt;Synthetic tests &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Apart from experimenting with real world workloads, another functionality we implemented in the scope of this work is a collection of simulated workloads implemented as kernel unit tests based on the recently merged DRM scheduler mock scheduler unit test framework&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref9&quot;&gt;[9]&lt;/a&gt;&lt;/sup&gt;&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref10&quot;&gt;[10]&lt;/a&gt;&lt;/sup&gt;. The idea behind those is to make it easy for developers to check for scheduling regressions when modifying the code, without the need to set up sometimes complicated testing environments.&lt;/p&gt;
&lt;p&gt;Let us look at a few examples on how the new scheduler compares with FIFO when using those simulated workloads.&lt;/p&gt;
&lt;p&gt;First an easy, albeit exaggerated, illustration of priority starvation improvements.&lt;/p&gt;
&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/avif&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/3Y38OrD4Ow-986.avif 986w&quot;&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/3Y38OrD4Ow-986.webp 986w&quot;&gt;&lt;img alt=&quot;Solved low priority starvation&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;https://blogs.igalia.com/tursulin/img/3Y38OrD4Ow-986.png&quot; width=&quot;986&quot; height=&quot;555&quot;&gt;&lt;/picture&gt;&lt;/p&gt;
&lt;p&gt;Here we have a normal priority client and a low priority client submitting many jobs asynchronously (only waiting for the submission to finish after having submitted the last job). We look at the number of outstanding jobs (queue depth - qd) on the Y axis and the passage of time on the X axis. With the FIFO scheduler (blue) we see that the low priority client is not making any progress whatsoever, all until the all submission of the normal client have been completed. Switching to the CFS inspired scheduler (red) this improves dramatically and we can see the low priority client making slow but steady progress from the start.&lt;/p&gt;
&lt;p&gt;Second example is about fairness where two clients are of equal priority:&lt;/p&gt;
&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/avif&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/3xbzj8QWTo-989.avif 989w&quot;&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/3xbzj8QWTo-989.webp 989w&quot;&gt;&lt;img alt=&quot;Fair GPU time distribution&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;https://blogs.igalia.com/tursulin/img/3xbzj8QWTo-989.png&quot; width=&quot;989&quot; height=&quot;556&quot;&gt;&lt;/picture&gt;&lt;/p&gt;
&lt;p&gt;Here the interesting observation is that the new scheduler graphed lines are much more straight. This means that the GPU time distribution is more equal, or fair, because the selection criteria is decoupled from the job submission time but based on each client’s GPU time utilisation.&lt;/p&gt;
&lt;p&gt;For the final set of test workloads we will look at the rate of progress (aka frames per second, or fps) between different clients.&lt;/p&gt;
&lt;p&gt;In both cases we have one client representing a heavy graphical load, and one representing an interactive, lightweight client. They are running in parallel but we will only look at the interactive client in the graphs. Because the goal is to look at what frame rate the interactive client can achieve when competing for the GPU. In other words we use that as a proxy for assessing user experience of using the desktop while there is simultaneous heavy GPU usage from another client.&lt;/p&gt;
&lt;p&gt;The interactive client is set up to spend 1ms of GPU time in every 10ms period, resulting in an effective GPU load of 10%.&lt;/p&gt;
&lt;p&gt;First test is with a heavy client wanting to utilise 75% of the GPU by submitting three 2.5ms jobs back to back, repeating that cycle every 10ms.&lt;/p&gt;
&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/avif&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/_DeVevyLn6-986.avif 986w&quot;&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/_DeVevyLn6-986.webp 986w&quot;&gt;&lt;img alt=&quot;Interactive client vs heavy load&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;https://blogs.igalia.com/tursulin/img/_DeVevyLn6-986.png&quot; width=&quot;986&quot; height=&quot;555&quot;&gt;&lt;/picture&gt;&lt;/p&gt;
&lt;p&gt;We can see that the average frame rate the interactive client achieves with the new scheduler is much higher than under the current FIFO algorithm.&lt;/p&gt;
&lt;p&gt;For the second test we made the heavy GPU load client even more demanding by making it want to completely monopolise the GPU. It is now submitting four 50ms jobs back to back, and only backing off for 1us before repeating the loop.&lt;/p&gt;
&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/avif&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/-T42diLZod-986.avif 986w&quot;&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/-T42diLZod-986.webp 986w&quot;&gt;&lt;img alt=&quot;Interactive client vs very heavy load&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;https://blogs.igalia.com/tursulin/img/-T42diLZod-986.png&quot; width=&quot;986&quot; height=&quot;555&quot;&gt;&lt;/picture&gt;&lt;/p&gt;
&lt;p&gt;Again the new scheduler is able to give significantly more GPU time to the interactive client compared to what FIFO is able to do.&lt;/p&gt;
&lt;h2 id=&quot;conclusions&quot; tabindex=&quot;-1&quot;&gt;Conclusions &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;From all the above it appears that the experiment was successful. We were able to simplify the code base, solve the priority starvation and improve scheduling fairness and GPU time allocation for interactive clients. No scheduling regressions have been identified to date.&lt;/p&gt;
&lt;p&gt;The complete patch series implementing these changes is available at&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref11&quot;&gt;[11]&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;h2 id=&quot;potential-for-further-refinements&quot; tabindex=&quot;-1&quot;&gt;Potential for further refinements &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Because this work has simplified the scheduler code base and introduced entity GPU time tracking, it also opens up the possibilities for future experimenting with other modern algorithms. One example could be an EEVDF&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref12&quot;&gt;[12]&lt;/a&gt;&lt;/sup&gt; inspired scheduler, given that algorithm has recently improved upon the kernel’s CPU scheduler and is looking potentially promising for it is combining fairness and latency in one algorithm.&lt;/p&gt;
&lt;h2 id=&quot;connection-with-the-drm-scheduling-cgroup-controller-proposal&quot; tabindex=&quot;-1&quot;&gt;Connection with the DRM scheduling cgroup controller proposal &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Another interesting angle is that, as this work implements scheduling based on virtual GPU time, which as a reminder is calculated by scaling the real time by a factor based on entity priority, it can be tied really elegantly to the previously proposed DRM scheduling cgroup controller.&lt;/p&gt;
&lt;p&gt;There we had group weights already which can now be used when scaling the virtual time and lead to a simple but effective cgroup controller. This has already been prototyped&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; id=&quot;fnref13&quot;&gt;[13]&lt;/a&gt;&lt;/sup&gt;, but more on that in a following blog post.&lt;/p&gt;
&lt;h2 id=&quot;references&quot; tabindex=&quot;-1&quot;&gt;References &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;hr class=&quot;footnotes-sep&quot;&gt;
&lt;section class=&quot;footnotes&quot;&gt;
&lt;ol class=&quot;footnotes-list&quot;&gt;
&lt;li id=&quot;fn1&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://docs.kernel.org/gpu/drm-mm.html#gpu-scheduler&quot;&gt;https://docs.kernel.org/gpu/drm-mm.html#gpu-scheduler&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn2&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.16-rc2&amp;amp;id=08fb97de03aa2205c6791301bd83a095abc1949c&quot;&gt;https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.16-rc2&amp;amp;id=08fb97de03aa2205c6791301bd83a095abc1949c&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn3&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://registry.khronos.org/vulkan/specs/latest/man/html/VK_EXT_global_priority.html&quot;&gt;https://registry.khronos.org/vulkan/specs/latest/man/html/VK_EXT_global_priority.html&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn4&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Completely_Fair_Scheduler&quot;&gt;https://en.wikipedia.org/wiki/Completely_Fair_Scheduler&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn5&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Red%E2%80%93black_tree&quot;&gt;https://en.wikipedia.org/wiki/Red–black_tree&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn6&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://benchmark.unigine.com/heaven&quot;&gt;https://benchmark.unigine.com/heaven&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn7&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/SaschaWillems/Vulkan&quot;&gt;https://github.com/SaschaWillems/Vulkan&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn8&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/tools/gputop.c?ref_type=heads&quot;&gt;https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/tools/gputop.c?ref_type=heads&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn9&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://gitlab.freedesktop.org/tursulin/kernel/-/commit/486bdcac6121cfc5356ab75641fc702e41324e27&quot;&gt;https://gitlab.freedesktop.org/tursulin/kernel/-/commit/486bdcac6121cfc5356ab75641fc702e41324e27&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn10&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://gitlab.freedesktop.org/tursulin/kernel/-/commit/50898d37f652b1f26e9dac225ecd86b3215a4558&quot;&gt;https://gitlab.freedesktop.org/tursulin/kernel/-/commit/50898d37f652b1f26e9dac225ecd86b3215a4558&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn11&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://gitlab.freedesktop.org/tursulin/kernel/-/tree/drm-sched-cfs?ref_type=heads&quot;&gt;https://gitlab.freedesktop.org/tursulin/kernel/-/tree/drm-sched-cfs?ref_type=heads&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn12&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://lwn.net/Articles/925371/&quot;&gt;https://lwn.net/Articles/925371/&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn13&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://lore.kernel.org/dri-devel/20250502123256.50540-1-tvrtko.ursulin@igalia.com/&quot;&gt;https://lore.kernel.org/dri-devel/20250502123256.50540-1-tvrtko.ursulin@igalia.com/&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/fair-er-drm-gpu-scheduler/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
	</entry>
	
	<entry>
		<title>DRM scheduling cgroup controller</title>
		<link href="https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/"/>
		<updated>2024-09-04T00:00:00Z</updated>
		<id>https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/</id>
		<content type="html">&lt;h2 id=&quot;introduction&quot; tabindex=&quot;-1&quot;&gt;Introduction &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The topic of a Direct Rendering Manager (DRM) cgroup controller is something which has been proposed a few times in the past, but so far is still missing from the Linux graphics stack. Some of those attempts were focusing on controlling the GPU memory usage aspect, while some were concerned with scheduling. As I am continuing to explore this area as part of my work at Igalia, in this post we will discuss one possible way of implementing the latter.&lt;/p&gt;
&lt;p&gt;General problem statement which we are trying to address is the fact many GPUs (and their respective kernel drivers) can simultaneously schedule workloads from different clients and that there are use-cases where having external control over scheduling decisions would be beneficial.&lt;/p&gt;
&lt;p&gt;But first to clarify what we mean by &lt;em&gt;“external control”&lt;/em&gt;. By that term we refer to the scheduling decisions being influenced from the outside of the actual process doing the rendering. If we were to draw a parallel to CPU scheduling, that would be the difference between a process (or a thread) issuing a system call such as &lt;em&gt;setpriority(2)&lt;/em&gt; or &lt;em&gt;nice(2)&lt;/em&gt; itself (&lt;em&gt;“internal control”&lt;/em&gt;), versus its scheduling priority being modified by an external entity such as the user issuing the &lt;em&gt;renice(1)&lt;/em&gt; shell command, launching the executable via the &lt;em&gt;nice(1)&lt;/em&gt; shell command, or even using the CPU scheduling cgroup controller &lt;em&gt;(“external control”)&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;This has two benefits. Firstly, it is the user who typically knows which tasks are higher priority and which should run in the background and therefore be as much as it is possible isolated from starving the foreground tasks from resources. Secondly, external control can be applied on any process in an unified manner, without the need for applications to individually expose the means to control their scheduling priority.&lt;/p&gt;
&lt;p&gt;If we now return back to the world of GPU scheduling we find ourselves in a landscape where internal scheduling control is possible with many GPU drivers, but the external control is not. To improve on that there are some technical and conceptual challenges, because GPUs are not as nice and uniform in their scheduling needs and capabilities as CPUs are, but if we would be able to come up with something reasonable even if not perfect, it could bring improvements to the user experience in a variety of scenarios.&lt;/p&gt;
&lt;h2 id=&quot;past-attempts-priority-based-controllers&quot; tabindex=&quot;-1&quot;&gt;Past attempts - Priority based controllers &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The earliest attempt I can remember was from 2018, by Matt Roper&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot; id=&quot;fnref1&quot;&gt;[1]&lt;/a&gt;&lt;/sup&gt;, who proposed to implement a driver-specific priority based controller. The RFC limited itself to i915 (kernel driver for Intel GPUs) and, although the priority-based setup is well established in the world of CPU scheduling, and it is easy to understand its effects, the proposal did not gain much traction.&lt;/p&gt;
&lt;p&gt;Because of the aforementioned advantages, when I proposed my version of the controller in 2022&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot; id=&quot;fnref2&quot;&gt;[2]&lt;/a&gt;&lt;/sup&gt;, it also included a slightly different version of a priority-based controller. In contrast to the earlier one, this proposal was in principle driver-agnostic and the priority levels were also abstracted.&lt;/p&gt;
&lt;p&gt;The proposal was also accompanied by benchmark results showing that the approach was effective in allowing users on Linux to launch GPU tasks in the background, while leaving more GPU bandwidth to the foreground task than when not using the controller. Similarly on ChromeOS, when wired into the focused versus un-focused window cgroup management, it was able to demonstrate relatively more GPU time given to the foreground window.&lt;/p&gt;
&lt;h2 id=&quot;current-proposal-weight-based-controller&quot; tabindex=&quot;-1&quot;&gt;Current proposal - Weight based controller &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Anticipating the potential lack of sufficient support for this approach the same RFC also included a second controller which takes a different route. It abstracts things one step further and implements a weight based controller based on GPU utilisation&lt;sup class=&quot;footnote-ref&quot;&gt;&lt;a href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot; id=&quot;fnref3&quot;&gt;[3]&lt;/a&gt;&lt;/sup&gt;.&lt;/p&gt;
&lt;p&gt;The basic idea is that the GPU time budget is split based on relative group weights across the cgroup hierarchy, and that the controller notifies the individual DRM drivers when their clients are over budget. From there it is left for the individual drivers to know how to best manage this situation, depending on the specific scheduling capabilities of the driver and the GPU hardware.&lt;/p&gt;
&lt;p&gt;The user interface completely mimics the exiting CPU and IO cgroup controllers with the single &lt;em&gt;drm.weight&lt;/em&gt; control file. The weights carry no absolute meaning and are only relative within a single group of siblings. Their only purpose is to split out the time budget between them.&lt;/p&gt;
&lt;p&gt;Visually one potential cgroup configuration could look like this:&lt;/p&gt;
&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/avif&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/KCnzzeDYF8-778.avif 778w&quot;&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/KCnzzeDYF8-778.webp 778w&quot;&gt;&lt;img alt=&quot;A visual representation of cgroup hierarchy&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;https://blogs.igalia.com/tursulin/img/KCnzzeDYF8-778.png&quot; width=&quot;778&quot; height=&quot;606&quot;&gt;&lt;/picture&gt;&lt;/p&gt;
&lt;p&gt;The DRM cgroup controller then executes a periodic scanning task which queries each DRM client for its GPU usage and notifies drivers when clients are over their allocated budget.&lt;/p&gt;
&lt;p&gt;If we expand the concept with runtime adjustment of group weights based on window focus status, with two graphically active clients such as a game and a web browser, we can end up with the following two scenarios:&lt;/p&gt;
&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/avif&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/YNndVChgoh-1650.avif 1650w&quot;&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/YNndVChgoh-1650.webp 1650w&quot;&gt;&lt;img alt=&quot;Example cgroup hierararchy re-configured based on window focus&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;https://blogs.igalia.com/tursulin/img/YNndVChgoh-1650.png&quot; width=&quot;1650&quot; height=&quot;904&quot;&gt;&lt;/picture&gt;&lt;/p&gt;
&lt;p&gt;Here we show the actual GPU utilisation of each group together with their &lt;em&gt;drm.weight&lt;/em&gt;. On the left hand side the web browser is the focused window, with the weights 100-to-10 in its favour.&lt;/p&gt;
&lt;p&gt;The compositor is not using its full &lt;em&gt;200 / (200 + 100)&lt;/em&gt; so a portion is passed on to the desktop group to the extent of the full 80% required. Inside the desktop group the game is currently using 70%, while its actual allocation is &lt;em&gt;80% * (10 / (100 + 10)) = 7.27%&lt;/em&gt;. Therefore it is currently consuming is more than the budget and the corresponding DRM driver will be notified by the controller and will be able to do something about it.&lt;/p&gt;
&lt;p&gt;After the user has given focus to the game window, relative weights will be adjusted and so will the budgets. Now the web browser will be over budget and therefore it can be throttled down, limiting the effect of its background activity on the foreground game window.&lt;/p&gt;
&lt;h3 id=&quot;first-driver-implementation-i915&quot; tabindex=&quot;-1&quot;&gt;First driver implementation - i915 &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Back when I started developing this idea Intel GPU’s were my main focus, which is why i915 was the first driver I wired up with the controller.&lt;/p&gt;
&lt;p&gt;There I implemented a rather simple approach of dynamically adjusting the scheduling priority of the throttled contexts, to the amount proportional to how much client is over budget in relative terms.&lt;/p&gt;
&lt;p&gt;Implementation would also cross-check against the physical engine utilisation, since in i915 we have easy access to that metric, and only throttle if the latter is close to being fully utilised. (Why this makes sense could be an interesting digression relating to the fact that a single cgroup can in theory contain multiple GPUs and multiple clients using a mix of those GPUs. But lets leave that for later.)&lt;/p&gt;
&lt;p&gt;One of the scenarios I used to test how well this works is to run two demanding GPU clients, each in its own cgroup, tweak their relative weights, and see what happens. The results were encouraging and are shown in the following table.&lt;/p&gt;
&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/avif&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/VokU69nDsn-720.avif 720w&quot;&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/VokU69nDsn-720.webp 720w&quot;&gt;&lt;img alt=&quot;Benchmark results under Linux&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;https://blogs.igalia.com/tursulin/img/VokU69nDsn-720.png&quot; width=&quot;720&quot; height=&quot;302&quot;&gt;&lt;/picture&gt;&lt;/p&gt;
&lt;p&gt;We can see that, when a clients group weight was decreased, the GPU bandwidth it was receiving also went down, as a consequence of the lowered context priority after receiving the over-budget notification.&lt;/p&gt;
&lt;p&gt;This is a suitable moment to mention how the DRM cgroup controller does not promise perfect control, that is, achieving the actual GPU sharing ratios as expressed by group-relative weights. As we have mentioned before, GPU scheduling is not nearly at the same level of quality and granularity as in the CPU world, so the goal it sets is simply to improve things - do something which has a positive impact on user experience. At the same time, the mechanism and control interface proposed does not preclude individual drivers doing as good job as they can. Or even a future possibility of replacing the inner workings with a controller with something smarter, with no need to change the user space control interface.&lt;/p&gt;
&lt;p&gt;Going back to the initial i915 implementation, the second test I have done was attempting to wire up with the background/foreground window focus handling in ChromeOS. There I experimented with a game (Android VM) running in parallel with a WebGL demo in a browser. At a certain point after both clients were running I lowered the weight of the background game and on the below screenshot we can see how the FPS metric in a browser jumped up.&lt;/p&gt;
&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/avif&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/uqaMw7GZxb-2374.avif 2374w&quot;&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/uqaMw7GZxb-2374.webp 2374w&quot;&gt;&lt;img alt=&quot;Screenshot of foreground versus background change under ChromeOS&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;https://blogs.igalia.com/tursulin/img/uqaMw7GZxb-2374.png&quot; width=&quot;2374&quot; height=&quot;2072&quot;&gt;&lt;/picture&gt;&lt;/p&gt;
&lt;p&gt;This illustrates how having the controller can indeed improve the user experience. The user’s focus will be at the foreground window and therefore it does make sense to prioritise GPU access to that client for better interactiveness and smoother rendering there. In fact, in this example the actual FPS jumped from around 48-49 to 60fps. Meaning that throttling the background client has allowed the foreground one to match its rendering to display’s refresh rate.&lt;/p&gt;
&lt;h3 id=&quot;second-implementation-amdgpu&quot; tabindex=&quot;-1&quot;&gt;Second implementation - amdgpu &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;AMD’s kernel module was the next interesting driver which I wired up with the controller.&lt;/p&gt;
&lt;p&gt;The fact that its scheduling is built on top of the DRM scheduler with only three distinct priority levels mandated a different approach to throttling. We keep a sorted list of “most offending” clients (most out of budget, or most borrowed unused budget from the sibling group), with the idea that the top client on that list gets throttled by lowering its scheduling priority. That was relatively straightforward to implement and sounded like it could potentially satisfy the most basic use case of background task isolation.&lt;/p&gt;
&lt;p&gt;To test the runtime behaviour we set up two sibling cgroups and vary their relative scheduling weights. In one cgroup we run glxgears with vsync turned off and log its frame rate over time, while in the second group we run glmark2.&lt;/p&gt;
&lt;p&gt;Let us first have a look on how glxgears frame rate varies during this test, depending on three different scheduling weight ratios between the cgroups. Scheduling weight ratio is expressed as glxgears:glmark2 ie. 10:1 means glxgears scheduling weight was ten times as much as configured for glmark2.&lt;/p&gt;
&lt;p&gt;We can observe that, as the glmark2 is progressing through its various sub-benchmarks, glxgears frame rate is changing too. But it was overall higher in the runs where the scheduling weight ratio was in its favour. That is a positive result showing that even a simple implementation seems to be having the desired effect, at least to some extent.&lt;/p&gt;
&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/avif&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/uwX9_zLGqc-1754.avif 1754w&quot;&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/uwX9_zLGqc-1754.webp 1754w&quot;&gt;&lt;img alt=&quot;Scheduling weight effect on glxgears running in parallel with glmark2&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;https://blogs.igalia.com/tursulin/img/uwX9_zLGqc-1754.png&quot; width=&quot;1754&quot; height=&quot;874&quot;&gt;&lt;/picture&gt;&lt;/p&gt;
&lt;p&gt;For the second test we can look from the perspective of glmark2, checking how the benchmark score change depending on the ratio of scheduling weights.&lt;/p&gt;
&lt;p&gt;&lt;picture&gt;&lt;source type=&quot;image/avif&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/e4Yc0RoIPg-2368.avif 2368w&quot;&gt;&lt;source type=&quot;image/webp&quot; srcset=&quot;https://blogs.igalia.com/tursulin/img/e4Yc0RoIPg-2368.webp 2368w&quot;&gt;&lt;img alt=&quot;Scheduling weight effect on glmark2 running in parallel with glxgears&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; src=&quot;https://blogs.igalia.com/tursulin/img/e4Yc0RoIPg-2368.png&quot; width=&quot;2368&quot; height=&quot;1898&quot;&gt;&lt;/picture&gt;&lt;/p&gt;
&lt;p&gt;Again we see that the scores are generally improving when the scheduling weight ratio is increased in favour of the benchmark.&lt;/p&gt;
&lt;p&gt;However, in neither case the change of the result is proportional to actual ratios. This is because the primitive implementation is not able to precisely limit the “background” client, but is only able to achieve some throttling. Also, there is an inherent delay in how fast the controller can react given the control loop is based on periodic scanning. This period is configurable and was set to two seconds for the above tests.&lt;/p&gt;
&lt;h2 id=&quot;conclusion&quot; tabindex=&quot;-1&quot;&gt;Conclusion &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Hopefully this write-up has managed to demonstrate two main points:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;First, that a generic and driver agnostic approach to DRM scheduling cgroup controller can improve user experience and enable new use cases. While at the same time following the established control interface as it exists for CPU and IO control, which makes it future-proof and extendable;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Secondly, that even relatively basic driver implementations can be somewhat effective in providing positive control effects.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;It also probably needs to be re-iterated that neither the driver implementations or the cgroup controller implementation itself are limited by the user interface proposed. Both could be independently improved under the hood in the future.&lt;/p&gt;
&lt;p&gt;What is next? There is more work to be done such as conducting more detailed testing, polishing the implementation and potentially attempting to wire up more drivers to the controller. Further advocacy work in the DRM community too.&lt;/p&gt;
&lt;h2 id=&quot;references&quot; tabindex=&quot;-1&quot;&gt;References &lt;a class=&quot;header-anchor&quot; href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;hr class=&quot;footnotes-sep&quot;&gt;
&lt;section class=&quot;footnotes&quot;&gt;
&lt;ol class=&quot;footnotes-list&quot;&gt;
&lt;li id=&quot;fn1&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://lore.kernel.org/dri-devel/20180120015141.10118-1-matthew.d.roper@intel.com/&quot;&gt;https://lore.kernel.org/dri-devel/20180120015141.10118-1-matthew.d.roper@intel.com/&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn2&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://lore.kernel.org/lkml/20221019173254.3361334-1-tvrtko.ursulin@linux.intel.com/&quot;&gt;https://lore.kernel.org/lkml/20221019173254.3361334-1-tvrtko.ursulin@linux.intel.com/&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li id=&quot;fn3&quot; class=&quot;footnote-item&quot;&gt;&lt;p&gt;&lt;a href=&quot;https://lore.kernel.org/lkml/ZVE3shwiRbUQyAqs@mtj.duckdns.org/T/&quot;&gt;https://lore.kernel.org/lkml/ZVE3shwiRbUQyAqs@mtj.duckdns.org/T/&lt;/a&gt; &lt;a href=&quot;https://blogs.igalia.com/tursulin/drm-scheduling-cgroup-controller/&quot; class=&quot;footnote-backref&quot;&gt;↩︎&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;/section&gt;
</content>
	</entry>
</feed>
