Changwoo Writes Code

sched_ext: scheduler architecture and interfaces (Part 2)

2024-06-24T00:00:00Z

This is the second blog post about the sched_ext, a BPF-based extensible scheduler class. Just in case you didn’t read the first one, you can find it here. In this blog post, I briefly update what has been happening in sched_ext, then introduce the scheduler architecture and the sched_ext API. After reading this, you should have a good understanding of the sched_ext architecture and be ready to read the source code of any sched_ext schedulers. Let’s get started!

Now, it is happening! #

After a long debate about the sched_ext patch, Linus finally approved its inclusion in the 6.11 kernel. As of the time of this writing, the V7 patch set has been posted to the mailing list. In addition, the developer and user communities around the sched_ext are expanding. More developers have been attending the weekly office hours. If you are interested in scheduler development, please join our Slack channel. Also, many enthusiasts (especially from the CachyOS community) have been actively testing, benchmarking, and bug-reporting the sched_ext schedulers.

Where to start developing a sched_ext scheduler? #

As discussed in the first blog post, sched_ext is an extensible scheduler class, so more than one sched_ext-based scheduler exists (e.g., scx_lavd, scx_rusty, and scx_rustland). To play with sched_ext schedulers, you should use a sched_ext-enabled kernel. There are three ways to get a sched_ext-enabled kernel:

You can install a Linux distribution that natively supports the sched_ext kernel by default. As of this writing, CachyOS—an Arch-based Linux distribution—supports the sched_ext enabled kernel and the BPF schedulers. This might be the most straightforward if you plan to set up a new machine. See this tutorial, which explains how to use sched_ext schedulers in CachyOS.
If you plan to explore the sched_ext kernel as well, you can start with virtual machine. virtme_ng is an excellent environment, which makes your VM-based kernel development seamless. See Andrea’s blog post on virtme_ng for more details.
If you finally decide to run your own sched_ext kernel on your machine, you should build the kernel with proper config options. See the details in my first blog post.

Now, you are ready to hack. Congrats!

Understanding the big picture of Linux schedulers #

Before getting your hands dirty, let’s step back and see the big picture of the Linux scheduler first.

The following diagram shows the architecture of Linux scheduler. As the figure illustrates, the Linux scheduler conceptually consists of two layers. The core kernel scheduler (core.c) defines the common behavior of all scheduling policies. For example, the core kernel scheduler defines what to do upon a timer interrupt (or scheduler tick). At the high level, it is analogous to the abstract base class for all schedulers in a C++ term.

Concrete scheduling policies are defined on top of the core kernel scheduler. For example, Linux provides real-time scheduling policies (rt.c) such as FIFO (First-In, First-Out, SCHED_FIFO) and RR (Round Robin, SCHED_RR). The fair scheduling policy (SCHED_NORMAL, EEVDF: Earliest Eligible Virtual Deadline First) governs non-realtime regular tasks.

This two-levels architecture allows one to write a new scheduling policy while reusing the logic of the core kernel scheduler. In Linux, such an architecture is called a scheduler class.

                                                  +========================+
                                                  || User-space part of   ||
                                                  || your scheduler       ||
                                                  || (e.g., main.rs)      ||
                                                  +========================+
~~~~~~~~~~~~~~~~ <kernel space vs. user space boundary > ~~~~~~~~~~~~~~~~~~~
                                                  +========================+
                                                  || Your BPF scheduler   ||
                                                  || (e.g., main.bpf.c)   ||
                                                  +========================+
+-----------------------+ +---------------------+ +========================+
| Default scheduler     | | Real-time scheduler | || Sched_ext framework  ||
| (EEVDF)               | | (FIFO, RR)          | ||                      ||
| (kernel/sched/fair.c) | | (kernel/sched/rt.c) | || (kernel/sched/ext.c) ||
+-----------------------+ +---------------------+ +========================+
+--------------------------------------------------------------------------+
|   Core kernel scheduler                                                  |
|   (kernel/sched/core.c)                                                  |
+--------------------------------------------------------------------------+

The sched_ext framework inside the kernel (ext.c) is defined on top of the core kernel scheduler like any other scheduler classes (e.g., fair.c and rt.c). That implies all the scheduler classes should communicate to the core kernel scheduler via the same interface; we will discuss that interface shortly.

The sched_ext framework is a vessel that ships a BPF scheduler with a user-defined policy. The sched_ext framework adds two more layers on top of it: a BPF scheduler layer and a user-space process, which interacts with the BPF scheduler. Writing a new sched_ext scheduler means writing a new BPF scheduler and its user-space counterpart.

You may notice that the idea of a scheduler class is somewhat similar to the idea of a VFS (Virtual File System) layer for file systems. Both define the common behavior of concrete policies (e.g., a new scheduling policy or file system layout) and the common interfaces for them.

Yet, there is a key distinction. VFS, with its focus on spatially partitioned disk space (or drive), allows for the simultaneous existence of multiple file systems, as long as each manages disjoint disk (or drive) space.

 SCHED_FIFO => SCHED_RR => SCHED_NORMAL (either EEVDF or sched_ext)

In contrast, scheduling is about how to slice and use CPU time, so at a certain point, only one scheduling policy can make scheduling decisions. In that sense, the scheduler classes are hierarchical in using CPU time. Real-time schedulers will always take the CPU time first (SCHED_FIFO => SCHED_RR). If no real-time task wants more CPU time, the schedulers for “normal” tasks (SCHED_NORMAL) will take turns with the real-time schedulers. Since EEVDF and sched_ext are both defined as SCHED_NORMAL, they cannot coexist at a certain point. When the sched_ext-based scheduler becomes active, it will take over all normal class tasks, replacing EEVDF.*

(*Note that sched_ext provides a special mode for scheduling only a subset of normal class tasks, mainly for testing purposes. However, I won’t discuss it in this post for brevity of discussion.)

Interface matters #

A sched_ext scheduler consists of four layers, as discussed from bottom to top: 1) core kernel scheduler, 2) sched_ext framework, 3) BPF scheduler, and 4) BPF’s user-space counterpart. These layers interact with each other through (relatively) well-defined interfaces. We now discuss those interfaces to understand how a scheduler works in action.

+========================+
|| User-space part of   ||
|| your scheduler       ||
|| (e.g., main.rs)      ||
+========================+
   \\//   ^^^^
   \\//   ^^^^ <== Interface 4: BPF scheduler <=> user-space counter part
   \\//   ^^^^
+========================+
|| Your BPF scheduler   ||
|| (e.g., main.bpf.c)   ||
+========================+
   ^^^^   \\// <== Interface 3: BPF scheduler => sched_ext framework
   ^^^^   \\//
   ^^^^  <======== Interface 2: sched_ext framework => BPF scheduler
+========================+
|| Sched_ext framework  ||
||                      ||
|| (kernel/sched/ext.c) ||
+========================+
          ^^^^
          ^^^^ <== Interface 1: core kernel scheduler => scheduler class
          ^^^^
+------------------------+
| Core kernel scheduler  |
|  (kernel/sched/core.c) |
+------------------------+

Interface 1: core kernel scheduler ⇒ scheduler class #

The core kernel scheduler defines the common underlying behavior for scheduling and defines an interface for concrete scheduler classes (e.g., SCHED_FIFO, SCHED_NORMAL). It defines an ops structure (struct sched_class) – a struct of function pointers – that needs to be filled by a concrete scheduler class to implement a concrete scheduling policy.

In the Linux kernel, a task (thread or process) is represented by struct task_struct, and a runnable (i.e., non-sleeping, ready-to-run) task should be in one of the run queues (struct rq) associated with each CPU. Scheduling is the problem of treating runnable tasks through run queues.

In a particular situation, when each scheduling policy needs its specific action, the core kernel scheduler calls an operation defined in struct sched_class. For example, when the core kernel scheduler needs to select a task to be scheduled, it calls the sched_class.pick_next_task(rq) callback of a concrete scheduling policy. When a task becomes runnable, the core kernel scheduler calls sched_calss.enqueue(rq, p, flags) so the concrete scheduling policy enqueues task p to run queue rq. When a task’s runtime state needs to be updated, the core kernel scheduler calls sched_calss.update_curr(rq).

/* kernel/sched/sched.h */
struct sched_class {
	void (*enqueue_task) (struct rq *rq, struct task_struct *p, int flags);
	void (*dequeue_task) (struct rq *rq, struct task_struct *p, int flags);
	void (*yield_task)   (struct rq *rq);
	bool (*yield_to_task)(struct rq *rq, struct task_struct *p);

	void (*wakeup_preempt)(struct rq *rq, struct task_struct *p, int flags);

	struct task_struct *(*pick_next_task)(struct rq *rq);

	void (*put_prev_task)(struct rq *rq, struct task_struct *p);
	void (*set_next_task)(struct rq *rq, struct task_struct *p, bool first);

	int (*balance)(struct rq *rq, struct task_struct *prev,
                       struct rq_flags *rf);
	int  (*select_task_rq)(struct task_struct *p, int task_cpu, int flags);

	struct task_struct * (*pick_task)(struct rq *rq);

	void (*migrate_task_rq)(struct task_struct *p, int new_cpu);

	void (*task_woken)(struct rq *this_rq, struct task_struct *task);

	void (*set_cpus_allowed)(struct task_struct *p,
                                    struct affinity_context *ctx);

	void (*rq_online)(struct rq *rq);
	void (*rq_offline)(struct rq *rq);

	struct rq *(*find_lock_rq)(struct task_struct *p, struct rq *rq);

	void (*task_tick)(struct rq *rq, struct task_struct *p, int queued);
	void (*task_fork)(struct task_struct *p);
	void (*task_dead)(struct task_struct *p);

	void (*switching_to) (struct rq *this_rq, struct task_struct *task);
	void (*switched_from)(struct rq *this_rq, struct task_struct *task);
	void (*switched_to)  (struct rq *this_rq, struct task_struct *task);
	void (*reweight_task)(struct rq *this_rq, struct task_struct *task,
			      int newprio);
	void (*prio_changed) (struct rq *this_rq, struct task_struct *task,
			      int oldprio);

	unsigned int (*get_rr_interval)(struct rq *rq,
					struct task_struct *task);

	void (*update_curr)(struct rq *rq);
        /* ... */
};

The sched_ext kernel framework implements functions required in struct sched_class. For example, when the core kernel scheduler enqueues a task or selects a next task to be scheduled or updates task’s runtime state, it eventually calls enqueue_task_scx(), pick_next_task_scx(), and update_curr_scx(), respectively, through struct sched_class.

/* kernel/sched/ext.c */
DEFINE_SCHED_CLASS(ext) = {
	.enqueue_task		= enqueue_task_scx,
	.dequeue_task		= dequeue_task_scx,
	.yield_task		= yield_task_scx,
	.yield_to_task		= yield_to_task_scx,

	.wakeup_preempt	= wakeup_preempt_scx,

	.pick_next_task	= pick_next_task_scx,

	.put_prev_task		= put_prev_task_scx,
	.set_next_task		= set_next_task_scx,

#ifdef CONFIG_SMP
	.balance		= balance_scx,
	.select_task_rq	= select_task_rq_scx,
	.set_cpus_allowed	= set_cpus_allowed_scx,

	.rq_online		= rq_online_scx,
	.rq_offline		= rq_offline_scx,
#endif

#ifdef CONFIG_SCHED_CORE
	.pick_task		= pick_task_scx,
#endif

	.task_tick		= task_tick_scx,

	.switching_to		= switching_to_scx,
	.switched_from		= switched_from_scx,
	.switched_to		= switched_to_scx,
	.reweight_task		= reweight_task_scx,
	.prio_changed		= prio_changed_scx,

	.update_curr		= update_curr_scx,

#ifdef CONFIG_UCLAMP_TASK
	.uclamp_enabled	= 1,
#endif
};

The sched_ext framework provides the common implementation for BPF schedulers. For example, when a task’s state needs to be updated, update_curr_scx() in the sched_ext decrements the time slice of the currently running task.

static void update_curr_scx(struct rq *rq)
{
	struct task_struct *curr = rq->curr;
	u64 now = rq_clock_task(rq);
	u64 delta_exec;

	delta_exec = now - curr->se.exec_start;
        /* ... */
	curr->scx.slice -= min(curr->scx.slice, delta_exec);
        /* ... */
}

Interface 2: sched_ext framework ⇒ BPF scheduler #

Let’s think about another case of enqueuing a runnable task to a run queue. The core kernel scheduler will call sched_class.enqueue_task(), actually enqueue_task_scx() in the schd_ext framework. How should the enqueue_task_scx() function be designed? The proper data structure and how tasks are organized in the queue depend on the BPF scheduler’s scheduling policy. For example, a FIFO queue would be the best data structure in the FIFO policy. In the deadline-based scheduling policy, it would be nice to maintain runnable tasks ordered by their deadlines. Any ordered map (e.g., red-black tree) would be appropriate.

Now let’s briefly examine how sched_ext implements enqueue_task_scx(). As the following code snippet shows, enqueue_task_scx() eventually calls scx_ops.enqueue(args), which is an enqueue implementation of a BPF scheduler.

/* kernel/sched/ext.c */

static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags)
{
	int sticky_cpu = p->scx.sticky_cpu;
	enq_flags |= rq->scx.extra_enq_flags;
        /* ... */
	do_enqueue_task(rq, p, enq_flags, sticky_cpu);
}

static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
			    int sticky_cpu)
{
        /* ... */
	SCX_CALL_OP_TASK(SCX_KF_ENQUEUE, enqueue, p, enq_flags);
        /* ... */
}

#define SCX_CALL_OP_TASK(mask, op, task, args...)				\
do {										\
	/* ... */                                                            \
	SCX_CALL_OP(mask, op, task, ##args);					\
	/* ... */                                                            \
} while (0)

#define SCX_CALL_OP(mask, op, args...)                                       \
do {										\
	/* ... */                                                            \
	scx_ops.op(args);                                                    \
	/* ... */                                                            \
} while (0)

The sched_ext framework defines an operation structure – struct sched_ext_ops, where scx_ops.enqueue() is defined. The following code snippet is a simplified version of struct sched_ext_ops. The full version with API description can be found in the source code.

/* kernel/sched/ext.c */
struct sched_ext_ops {
	s32 (*select_cpu)(struct task_struct *p, s32 prev_cpu, u64 wake_flags);

	void (*enqueue)(struct task_struct *p, u64 enq_flags);
	void (*dispatch)(s32 cpu, struct task_struct *prev);

	void (*tick)(struct task_struct *p);

	void (*runnable)(struct task_struct *p, u64 enq_flags);
	void (*running)(struct task_struct *p);
	void (*stopping)(struct task_struct *p, bool runnable);
	void (*quiescent)(struct task_struct *p, u64 deq_flags);

        /* ... */

	void (*update_idle)(s32 cpu, bool idle);

        /* ... */

	s32 (*init_task)(struct task_struct *p, struct scx_init_task_args *args);
	void (*exit_task)(struct task_struct *p, struct scx_exit_task_args *args);

        /* ... */

	s32 (*init)(void);
	void (*exit)(struct scx_exit_info *info);

        /* ... */
};

static struct sched_ext_ops scx_ops;

There seem to be a lot of callbacks in a BPF scheduler, but most of them are pretty straightforward.

When a BPF scheduler is loaded and unloaded, you have control over it. The sched_ext_ops.init() and sched_ext_ops.exit() functions are your tools to initialize and de-initialize BPF scheduler-wide data structures. This is where you can create and destroy per-CPU structures and global run queues managed by the BPF scheduler.

When a new task is created, sched_ext_ops.init_task() is called. As you can expect, sched_ext_ops.exit_task() is called when a task is terminated. You can manage per-task data structures there.

During the lifetime, a task will become first runnable (sched_ext_ops.runnable()), then it will be picked by the scheduler and will be running (sched_ext_ops.running()). Then, when its time slice is exhausted, the task will be stopped and scheduled out (sched_ext_ops.stopping()). The task becomes non-runnable (sched_ext_ops.quiescent()), for example, when waiting for an IO event.

When a task wakes up, for example, when an IO event is delivered, the BPF scheduler should first decide which CPU the task should run on (sched_ext_ops.select_cpu()). Then, the task will be enqueued to a BPF-managed run queue (sched_ext_ops.enqueue()).

Each CPU will continue to execute the assigned tasks to it. When a CPU has nothing to run, it will get a task from the BPF-managed run queue (sched_ext_ops.dispatch()). Even if there is no task to run on the BPF-managed run queue, the CPU will be idle to save power until there is something to run. The state transition from/to idle state can be tracked by sched_ext_ops.update_idle().

Upon every scheduler tick (i.e., every 1/HZ seconds), sched_ext_ops.tick() is called, so the BPF-scheduler can periodically do chores (e.g., power management, preemption, etc).

That’s (almost) all! Writing a new BPF scheduler is nothing but implementing those struct sched_ext_ops functions. Note that the sched_ext framework provides a default implementation of each callback, so when a callback implementation is not provided by the BPF scheduler, the default implementation will be executed. That’s straightforward, huh?

Interface 3: BPF scheduler ⇒ sched_ext framework #

A BPF scheduler needs to talk to the sched_ext kernel framework. For instance, after choosing a task to be run, it should be somehow notified to the sched_ext framework. There are two ways from a BPF scheduler to the sched_ext framework: 1) BPF helper function and 2) dispatch queue (DSQ). DSQ is also created and manipulated by the BPF helper functions. Here is a list of some essential BPF helper functions. A full list of BPF helper functions are here, and their implementations with full API descriptions are here.

/* tools/shed_ext/include/scx/common.bpf.h */
s32 scx_bpf_create_dsq(u64 dsq_id, s32 node);
s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym;
void scx_bpf_destroy_dsq(u64 dsq_id) __ksym;

void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id,
                      u64 slice, u64 enq_flags);
void scx_bpf_dispatch_vtime(struct task_struct *p, u64 dsq_id, u64 slice,
                            u64 vtime, u64 enq_flags);

bool scx_bpf_consume(u64 dsq_id);
bool scx_bpf_consume_task(unsigned long it, struct task_struct *p);

s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu,
                           u64 wake_flags, bool *is_idle);

void scx_bpf_kick_cpu(s32 cpu, u64 flags);

Dispatch queue (DSQ) is a core construct between a BPF scheduler and the sched_ext kernel framework, so many BPF helper functions are around DSQ. DSQ is a queue holding runnable tasks. Tasks in a DSQ can be ordered by an arrival order (FIFO) or a vtime (virtual time) order, where vtime is an integer associated with a task. The sched_ext framework also creates and maintains its internal DSQs: 1) one system-wide (named global DSQ) and 2) one for each CPU (named local DSQ). Both internal DSQs are FIFO-ordered. Each CPU in the sched_ext framework runs a task on its local DSQ in a FIFO order.

A BPF scheduler can also create DSQs (either FIFO or vtime DSQ) to manage runnable tasks in its own way (scx_bpf_create_dsq()). sched_ext_ops.init() is a typical place to create custom DSQs.

A task can be enqueued to the DSQ in a FIFO order (scx_bpf_dispatch()) or vtime order (scx_bpf_dispatch_vtime()), typically at sched_ext_ops.enqueue().

A task in a DSQ can be consumed from tip (scx_bpf_consume()) or a specified task can be consumed (scx_bpf_consume_task()), typically at sched_ext_ops.dispatch(). Here consuming a task means that a task is removed from a custom DSQ and moved to the internal local DSQ, so a CPU can run a task in its locak DSQ.

In addition, the number of tasks in a DSQ can be queried (scx_bpf_dsq_nr_queued()). At the end, a DSQ should be destroyed (scx_bpf_destroy_dsq()).

Besides DSQ-related BPF helpers, two other helper functions are notable: scx_bpf_select_cpu_dfl() and scx_bpf_kick_cpu(). scx_bpf_select_cpu_dfl() finds an appropriate CPU to run a task using default core selection policy. It can be used in sched_ext_ops.select_cpu(). scx_bpf_kick_cpu() sends an IPI (Inter-Processor Interrupt) signal to another CPU for waking up an idle CPU (SCX_KICK_IDLE flag) and asking preempt out the currently running task (SCX_KICK_PREEMPT flag).

Interface 4: BPF scheduler ⟺ user-space counterpart #

Since the BPF scheduler is a regular BPF program, you can use any BPF tricks you like. A user-space program (typically written in C or Rust) can use any libbpf API. Most importantly, a BPF map is accessed and manipulated by a user-space program (e.g., bpf_map_lookup_elem()).

What’s next? #

The best way to digest the sched_ext concept is to read the existing BPF scheduler code. I strongly encourage you to read the source code of one of scx_lavd, scx_rusty, or scx_rustland. Then, you will have a solid foundation to write a new BPF scheduler or improve an existing one. In the next blog post, I will discuss how to find out what to improve as an author of the scx_lavd scheduler. I hope you enjoyed this blog post. Happy scheduling until the next blog post!

sched_ext: a BPF-extensible scheduler class (Part 1)

2023-12-18T00:00:00Z

This blog post introduces sched_ext, a recently proposed extensible scheduler class. sched_ext allows you to write and run your custom process scheduler optimized for your target workloads and hardware architectures using BPF programs. In the rest of the post, I recap the concept of a scheduler at the high level and go over the default schedulers in Linux kernel. Then, I introduce why an extensible scheduler framework, sched_ext, is necessary, how to use it, and provide a BPF scheduler example.

What is a process scheduler? #

A process scheduler is essential in any OS, including the Linux kernel. A scheduler multiplexes CPU time into multiple tasks – processes and threads. It essentially makes two decisions:

Task selection: Which task should be run next?
Time slice: How long should the chosen next task be allowed to run?

With a multi-core CPU, which is a norm today, a scheduler also makes one more decision:

Core selection: Which CPU core should be used for executing the chosen next task?

The above three decisions often conflict each other. For example, a scheduler may assign a long time slice to reduce the context-switching overhead and take advantage of the warmed-up processor cache. However, a time slice that is too long may degrade the responsiveness of a system, hurting the interactive performance.

Also, a scheduler may want to fully utilize multiple CPU cores, so it can assign a task to any idle CPU core. Such work-conserving scheduling may increase overall CPU utilization, but higher CPU utilization does not always result in higher performance. When a task is too frequently migrated from one CPU to another CPU, the cache of the newly assigned CPU should be warmed up again so there is little chance to reuse the cache when the task is scheduled next, resulting in lower performance. The cache re-warming cost will be higher when a system has multiple cache domains, such as NUMA (Non-Uniform Memory Architecture) and AMD CCX (Core Complex) architectures.

How does the Linux scheduler make scheduling decisions? #

Completely Fair Scheduler (CFS) scheduler #

You may wonder how the Linux scheduler makes scheduling decisions to meet such conflicting goals. The Completely Fair Scheduler (CFS) has been the default scheduler for the last 15+ years since merged into the 2.6.23 kernel (October 2007). The core idea of CFS is simple (of course, the devil is in the details), but it works surprisingly well in many workloads. CFS manages the virtual total execution time of a task – vruntime. The actual execution time of a task is adjusted based on the task’s static priority (i.e., nice value). CFS pursues the complete fairness of tasks’ vruntime to equally distribute the CPU time to each task based on its priority. It always chooses a task with the smallest vruntime value for complete fair use of CPU time. CFS internally manages a run queue – a queue of runnable tasks – using a red-black tree indexed by each task’s vruntime. The red-black tree offers O(logN) time complexity in picking the next task where N is the number of runnable tasks. CFS handles IO-intensive tasks pretty well. The most IO-intensive tasks sleep most of the time, waiting for IO events, so their vruntime values tend to be small. Hence, whenever an IO event occurs and the IO-intensive task is wakened up and becomes runnable, CFS quickly chooses that task since it has a small vruntime value.

However, CFS scheduler does not enforce any scheduling deadlines and has many corner cases where a latency-critical task is not scheduled on time – such scheduling delay of latency-critical tasks results in high tail latency. For instance, Wine – a Windows-API emulation layer on Linux – has a wineserver process. The wineserver process is a single-threaded event router which relays Window events to other processes. It is super latency-critical, so scheduling delay of wineserver triggers cascading delays of dependent tasks. However, CFS does not always schedule wineserver immediately. Since wineserver consumes a fair amount of CPU time as a central event rounter, its vruntime value would be pretty high so CFS won’t immediately pick the wineserver task all the time.

Earliest Eligible Virtual Deadline First (EEVDF) scheduler #

Due to such limitations, the CFS scheduler was recently retired after serving 15+ years. The Earliest Eligible Virtual Deadline First (EEVDF) scheduler was introduced as a new default scheduler in the Linux kernel 6.6 release (November 2023).

Like CFS, EEVDF pursues the fair use of CPU time among tasks. For this, EEVDF introduces two notions: eligibility and lag of a task. EEVDF manages how much CPU time should be allocated to a task adjusted by the task’s static (nice) priority. Task’s lag is the gap between the ideal CPU time and the actual CPU time allocated to the task. The positive lag means less CPU time has been allocated. On the other hand, the negative lag means too much CPU time is allocated to the task, so scheduling such a task with a negative lag value violates the fair use of CPU time. With the EEVDF terms, tasks with only positive lag values are eligible for the scheduling. EEVDF scheduler does not schedule an ineligible task, which has a negative lag value, for fairness.

Another essential concept in EEVDF is virtual deadline. A virtual deadline is the time to finish executing the task’s requested time slice. It is calculated by adding the task’s allocated time slice, and its heuristically estimated eligible time, which is when a task may start to run. Conceptually, a task may start at the eligible time and finish its time slice at the virtual deadline. EEVDF assumes the task’s time slice is given by setting a latency nice value. The EEVDF scheduler always chooses an eligible task with the earliest virtual deadline.

EEVDF scheduler naturally handles latency-critical tasks unlike CFS. A latency-critical task has a small latency nice value, so it has a shorter time slice. Therefore, its virtual deadline is earlier than non-latency critical tasks with longer slices. Hence, suppose two tasks have the same nice value with different latency nice values. The latency-critical task will be scheduled more frequently with a shorter slice.

One size does not fit all, so does a scheduler #

The upsteam schedulers, like CFS and EEVDF, aim to generically work well for most workloads (e.g., databases, games, micro-services) on most hardware platforms (e.g., x86, ARM, NUMA, non-NUMA, CCX, etc.). It is not uncommon that one scheduler optimization works well in some workloads and shows pathological degradations in other workloads and hardware combinations. To avoid such performance regression bugs, the kernel community sets a high bar in changing the scheduler code.

However, if you always run a particular set of workloads on a specific hardware, would it be best to use such generic schedulers? In the real world, there are many such cases. For example, Linux-based gaming devices, like the Steam Deck, usually run Windows-based games. Hyperscalars, like Google and Meta, standardize their servers to a few types (e.g., OCP) for maintenance and run a certain set of workloads in a cluster. Similarly, many other companies (e.g., Airbnb, eBay) run their own workloads on their own mini-datacenters.

Adopting a custom scheduler for specific workload/hardware combinations would be more beneficial than using generic schedulers. However, implementing and maintaining custom schedulers is very painful. That is not only because replacing or customizing the in-kernel scheduler is challenging. But also, it is unlikely that a custom scheduling policy is upstreamed so that the custom scheduler will impose a heavy maintenance burden in the long term.

Here comes the `sched_ext` framework #

sched_ext was proposed to address the problems mentioned above. It allows users to write a custom scheduling policy using BPF without modifying the kernel code. You don’t need to struggle to maintain the out-of-tree custom scheduler. In addition, BPF provides a safe kernel programming environment. In particular, the BPF verifier ensures that your custom scheduler has neither a memory bug nor an infinite loop. Also, if your custom scheduler misbehaves – like failing to schedule a task for too long (say 30 seconds), the kernel portion of sched_ext kills your custom scheduler and falls back to the default kernel scheduler (CFS or EEVDF). Last but not least, you can update the BPF scheduler without reinstalling the kernel and rebooting a server. It is a charm in a large data center with hundreds of thousands of servers. Rebooting that many servers in a rolling manner easily takes several weeks.

Obviously, sched_ext is a safe and nice framework that allows for the rapid experimentation and deployment of various scheduling policies. In the long term, I believe providing such an environment will also positively affect the default scheduler’s improvement. Who knows that some scheduling policies battle-tested with sched_ext have turned out beneficial to generic workloads (with some tweaks) so they can be re-implemented on the default scheduler?

How to build and run `sched_ext` #

Enough talking. Now, let’s get our hands dirty. As of this writing, the fifth version of the sched_ext patch set has been proposed. It still needs to be merged into the mainline kernel. Hence, you should apply the patch set to the kernel source or use the sched_ext kernel code on GitHub. Either way, the following kernel configuration should be enabled to turn on sched_ext and enable BPF. Make sure the following bits are included in your .config.

# essential configs to enable sched_ext
CONFIG_BPF=y
CONFIG_SCHED_CLASS_EXT=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_PAHOLE_HAS_BTF_TAG=y
CONFIG_DEBUG_INFO_BTF=y

# useful debug features for sched_ext
CONFIG_DEBUG_INFO=y
CONFIG_SCHED_DEBUG=y
CONFIG_DEBUG_INFO_DWARF5=y
CONFIG_DEBUG_INFO_BTF_MODULES=y

Once the kernel configuration is done, you just follow your usual kernel compilation and installation procedure – one example is here.

After the kernel is compiled, it’s time to compile the BPF schedulers. The sched_ext scheduler sources provide several BPF schedulers under tools/sched_ext. BPF code should be compiled with clang/LLVM, so the following options should be added to the make command. Note the clang version should be 16 or higher to compile BPF code correctly.

$> cd tools/sched_ext
$> clang --version    # check if clang version is 16 or higher
$> make CC=clang LLVM=1 -j

After rebooting with the sched_ext kernel, let’s run any of the sched_ext BPF schedulers.

$> cd tools/sched_ext/build/bin
$> sudo ./scx_simple  # a simple vruntime-based scheduler, like CFS
local=2 global=0
local=885 global=5
local=895 global=12
local=906 global=20
...

Yeah! Now the simple vruntime-based scheduler, scx_simple, is up and running. The log message shows how many tasks are scheduled from the local or global dispatch queue (DSQ). For now, please consider DSQ as a run queue in sched_ext holding runnable tasks. I will explain the concept of the dispatch queue in more detail in my next blog post. Stay tuned!

While simple, scx_simple performs better than the CFS scheduler under some workloads. You can check the status of sched_ext from sysfs:

$> sudo cat /sys/kernel/debug/sched/ext
ops                           : simple
enabled                       : 1
switching_all                 : 1
switched_all                  : 1
enable_state                  : enabled
nr_rejected                   : 0

If the BPF scheduler prints out some messages (using, for example, bpf_printk()), you can check the message at /sys/kernel/debug/tracing/trace_pipe as in any other BPF programs.

By the way, if you are lucky enough to use Arch Linux or Ubuntu Linux (24.04 NobleNumbat), you can install a sched_ext-enabled kernel and the necessary tools from package repositories. Please find the detailed instructions to install the sched_ext packages in Arch or Ubuntu Linux here.

What’s inside `scx_simple`? #

Let’s take a look the source code of scx_simple. The scx_simple scheduler has two files: scx_simple.c and scx_simple.bpf.c.

scx_simple.c opens and load the BPF scheduler (scx_simple__open() and scx_simple__load()). Then it enables the operations provided by the BPF scheduler (bpf_map__attach_struct_ops(skel->maps.simple_ops)). Once the BPF scheduler operations are enabled – the BPF scheduling policy is enabled, it periodically reads and prints the status.

int main(int argc, char **argv)
{
	/* ... */
	skel = scx_simple__open();
	SCX_BUG_ON(!skel, "Failed to open skel");

	/* ... */
	SCX_BUG_ON(scx_simple__load(skel), "Failed to load skel");

	link = bpf_map__attach_struct_ops(skel->maps.simple_ops);
	SCX_BUG_ON(!link, "Failed to attach struct_ops");

	while (!exit_req && !uei_exited(&skel->bss->uei)) {
		__u64 stats[2];
		read_stats(skel, stats);
		printf("local=%llu global=%llu\n", stats[0], stats[1]);
		fflush(stdout);
		sleep(1);
	}

	bpf_link__destroy(link);
	uei_print(&skel->bss->uei);
	scx_simple__destroy(skel);
	return 0;
}

sched_ext defines a struct of callback functions, struct sched_ext_ops defined at include/linux/sched/ext.h. A BPF scheduler, scx_simple.bpf.c, implements those callback functions, which will be called in various situations by the in-kernel portion of sched_ext.

SEC(".struct_ops.link")
struct sched_ext_ops simple_ops = {
	.enqueue		= (void *)simple_enqueue,
	.dispatch		= (void *)simple_dispatch,
	.running		= (void *)simple_running,
	.stopping		= (void *)simple_stopping,
	.enable			= (void *)simple_enable,
	.init			= (void *)simple_init,
	.exit			= (void *)simple_exit,
	.name			= "simple",
};

When the BPF scheduler is initialized, an init() callback will be called. simple_init () changes all tasks (governed by the default CFS/EEVDF scheduler) to be under the control of scx_simple (scx_bpf_switch_all()) if a command line option is specified (!switch_partial). Then it creates a centralized task queue (scx_bpf_create_dsq(SHARED_DSQ, -1)).

s32 BPF_STRUCT_OPS_SLEEPABLE(simple_init)
{
	if (!switch_partial)
		scx_bpf_switch_all();

	return scx_bpf_create_dsq(SHARED_DSQ, -1);
}

When a task starts to be controlled by sched_ext (for instance, after fork(), clone(), scx_bpf_switch_all()), an enable() callback will be called. It is the best time to initialize per-task information. struct sched_ext_entity is defined to store sched_ext-specific scheduling information, such as time slice (slice) and task’s virtual time (dsq_vtime), and it is added to struct task_struct (p->scx). simple_enable() simply initializes the task’s virtual time to the current time (p->scx.dsq_vtime = vtime_now).

void BPF_STRUCT_OPS(simple_enable, struct task_struct *p,
		    struct scx_enable_args *args)
{
	p->scx.dsq_vtime = vtime_now;
}

When a task p becomes ready to run, the enqueue() callback is called. simple_enqueue() dispatches a task to either the local or global dispatch queue (DSQ) (SCX_DSQ_LOCAL vs. SHARED_DSQ inscx_bpf_dispatch()) for execution depending on its flag (enq_flag) set by the underlying sched_ext framework. The local dispatch queue (SCX_DSQ_LOCAL) is a kind of staging area waiting for execution, so the underlying sched_ext framework fetches a task from the per-CPU local DSQ and schedules that task to run on a CPU. simple_enqueue() manages tasks on the local DSQ in a FIFO order (scx_bpf_dispatch()) but tasks on the global DSQ in a vtime order (scx_bpf_dispatch_vtime()). When dispatching a task to a DSQ, the time slice of a task is set as an argument of scx_bpf_dispatch(). scx_simple sets the time slice to the default value, SCX_SLICE_DFL (20 msec).

void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p, u64 enq_flags)
{
	/*
	 * If scx_select_cpu_dfl() is setting %SCX_ENQ_LOCAL, it indicates that
	 * running @p on its CPU directly shouldn't affect fairness. Just queue
	 * it on the local FIFO.
	 */
	if (enq_flags & SCX_ENQ_LOCAL) {
		stat_inc(0);	/* count local queueing */
		scx_bpf_dispatch(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, enq_flags);
		return;
	}

	stat_inc(1);	/* count global queueing */

	if (fifo_sched) {
		scx_bpf_dispatch(p, SHARED_DSQ, SCX_SLICE_DFL, enq_flags);
	} else {
		u64 vtime = p->scx.dsq_vtime;

		/*
		 * Limit the amount of budget that an idling task can accumulate
		 * to one slice.
		 */
		if (vtime_before(vtime, vtime_now - SCX_SLICE_DFL))
			vtime = vtime_now - SCX_SLICE_DFL;

		scx_bpf_dispatch_vtime(p, SHARED_DSQ, SCX_SLICE_DFL, vtime,
				       enq_flags);
	}
}

When a task p gets scheduled out, the stopping() callback is called. simple_stopping() accumulates the consumed time slice so far (SCX_SLICE_DFL - p->scx.slice) to the task’s vtime (p->scx.dsq_vtime) based on its nice priority (p->scx.weight).

void BPF_STRUCT_OPS(simple_stopping, struct task_struct *p, bool runnable)
{
	/*
	 * Scale the execution time by the inverse of the weight and charge.
	 *
	 * Note that the default yield implementation yields by setting
	 * @p->scx.slice to zero and the following would treat the yielding task
	 * as if it has consumed all its slice. If this penalizes yielding tasks
	 * too much, determine the execution time by taking explicit timestamps
	 * instead of depending on @p->scx.slice.
	 */
	p->scx.dsq_vtime += (SCX_SLICE_DFL - p->scx.slice) * 100 / p->scx.weight;
}

When a per-CPU local DSQ becomes empty, there is no task to run on a CPU. In this case, the dispatch() callback is called. simple_dispatch() moves a task from the global DSQ to a local DSQ (scx_bpf_consume(SHARED_DSQ)) for execution. Since the global DSQ is ordered by vtime (scx_bpf_dispatch_vtime(p, SHARED_DSQ, ...)), scx_simple picks a task with the smallest vtime, which consumed the least amount of CPU time so far, and it moves the task to the local DSQ for the execution.

void BPF_STRUCT_OPS(simple_dispatch, s32 cpu, struct task_struct *prev)
{
	scx_bpf_consume(SHARED_DSQ);
}

What’s next? #

So far, I have introduced the concept of process scheduler, and the key idea of the default scheduler in Linux (CFS and EEVDF). Then, I shared the motivation of sched_ext framework and how to run on your machine. Finally, I skimmed through how a BPF scheduler is organized. In the next blog post, I will dive into what’s going on under the hood. Please stay tuned!

This is a post with no description

2023-10-06T00:00:00Z

When the post has not description: field, this first paragraph is the excerpt but we want it to be rendered using markdown. The rest of the post content is not relevant for this particular case.

Changwoo Writes Code

sched_ext: scheduler architecture and interfaces (Part 2)

Now, it is happening! #

Where to start developing a sched_ext scheduler? #

Understanding the big picture of Linux schedulers #

Sharing time is different from sharing space #

Interface matters #

Interface 1: core kernel scheduler ⇒ scheduler class #

Interface 2: sched_ext framework ⇒ BPF scheduler #

Interface 3: BPF scheduler ⇒ sched_ext framework #

Interface 4: BPF scheduler ⟺ user-space counterpart #

What’s next? #

sched_ext: a BPF-extensible scheduler class (Part 1)

What is a process scheduler? #

How does the Linux scheduler make scheduling decisions? #

Completely Fair Scheduler (CFS) scheduler #

Earliest Eligible Virtual Deadline First (EEVDF) scheduler #

One size does not fit all, so does a scheduler #

Here comes the sched_ext framework #

How to build and run sched_ext #

What’s inside scx_simple? #

What’s next? #

This is a post with no description

Here comes the `sched_ext` framework #

How to build and run `sched_ext` #

What’s inside `scx_simple`? #