sched: Implement BPF extensible scheduler class
Subsystems (14)
Tejun Heo
maintainerPATCH v6 1/1 sched: Implement BPF extensible scheduler class
Updates and Changes
-------------------
This is v6 of sched_ext (SCX) patchset.
During the past five months, both the development and adoption of sched_ext have been progressing briskly. Here are some highlights around adoption:
- Valve has been working with Igalia to implement a sched_ext scheduler for Steam Deck. The development is still in its early stages but they are already happy with the results (more consistent FPS) and are planning to enable the scheduler on Steam Deck.
https://github.com/sched-ext/scx/tree/main/scheds/rust/scx_lavd
https://ossna2024.sched.com/event/1aBOT/optimizing-scheduler-for-linux-gaming-changwoo-min-igalia
- Ubuntu is considering to include sched_ext in the upcoming 24.10 release. Andrea Righi of Canonical has been actively working on a userspace scheduling framework since the end of the last year.
https://github.com/sched-ext/scx/tree/main/scheds/rust/scx_rustland https://discourse.ubuntu.com/t/introducing-kernel-6-8-for-the-24-04-noble-numbat-release/41958
- We (Meta) are now deploying a sched_ext scheduler on one large production workload (web), conducting wide-scale verification benchmark on another (ads), and preparing production deployment on yet another workload (ML training). These are all using scx_layered, which is useful for quick prototyping and experiments. In the process, we've identified several common strategies which are useful across multiple workloads (e.g. soft-affinitizing related threads) and are in the process of implementing something more generic and polished.
https://github.com/sched-ext/scx/tree/main/scheds/rust/scx_layered
- Because Google's ghOSt framework (userspace scheduling framework with BPF hooks for optimization) is already available in the Google fleet, that's what Google is currently experimenting with. They are seeing promising results in a couple important workloads (search and cloud hosting) and trying to move on to deployment. The gap between ghOSt and sched_ext is not wide at this point and Google is working to port ghOSt schedulers on top of sched_ext.
- ChromeOS is looking into scx_layered with focus on reducing latency (as a replacement to RT) with a prototype port of sched_ext on ChromeOS.
- Oculus is facing a number of scheduling related challenges and looking into sched_ext. They have an android port that they're experimenting with although any actual deployment would have to wait until a newer platform kernel can be rolled out which will take quite a while.
Although sched_ext is still out of tree, we're seeing wide interest and adoption across multiple organizations and different use cases. Plus, our first-hand experiences at Meta and the reports from other users definitively confirm the hypothesized merits of sched_ext - among others, lowered barrier of entry coupled with rapid and safe experiments leading to insights and performance gains in an easily deployable form.
For example, scx_rusty and scx_layered are proving substantial benefits of better work conservation within L3 domains, soft affinity (flexibly grouping related threads together) and application specific prioritization for request-driven server workloads. scx_lavd is demonstrating the benefits of a new set of heuristics based on the runtime and the frequencies of waking up others and being woken up for gaming, and possibly other interactive, workloads.
We're seeing constantly increasing interests both within Meta and from the wider community. The benefits are inherent and clear enough that I don't see a reason for the trend to change. However, being out of tree does add a lot of overhead in terms of decision making and logistics for everyone involved.
Given that there already is substantial adoption which continues to grow and sched_ext doesn't affect the built-in schedulers or the rest of kernel in an invasive manner, I believe it's reasonable to consider sched_ext for inclusion. I and David Vernet would be happy to run the tree, respond to bug reports, coordinate with scheduler core or any other kernel subsystem that sched_ext may interact with.
If you're interested in high-level arguments for and against. Please refer to the following discussion in the v4 posting:
http://lkml.kernel.org/r/[email protected]
If you're interested in getting your hands dirty, the following repository contains example and practical schedulers along with documentation on how to get started:
https://github.com/sched-ext/scx
The kernel and scheduler packages are available for Ubuntu, CachyOS and Arch (through CachyOS repo). Fedora packaging is in the works.
There are also a slack and weekly office hour:
https://schedextworkspace.slack.com
Office Hour: Mondays at 16:00 UTC (8:00 AM PST, 16:00 UTC, 17:00 CEST,
1:00 AM KST). Please see the #office-hours channel for the zoom invite.
The followings are significant changes from v5
(http://lkml.kernel.org/r/[email protected]). For more detailed list of changes, please refer to the patch descriptions.
- scx_pair, scx_userland, scx_next and the rust schedulers are removed from kernel tree and now hosted in https://github.com/sched-ext/scx along with all other schedulers.
- SCX_OPS_DISABLING state is replaced with the new bypass mechanism which allows temporarily putting the system into simple FIFO scheduling mode. In addition to the shut down path, this is used to isolate the BPF scheduler across power management events.
- ops.prep_enable() is replaced with ops.init_task() and ops.enable/disable() are now called whenever the task enters and leaves sched_ext instead of when the task becomes schedulable on sched_ext and stops being so. A new operation - ops.exit_task() - is called when the task stops being schedulable on sched_ext.
- scx_bpf_dispatch() can now be called from ops.select_cpu() too. This removes the need for communicating local dispatch decision made by ops.select_cpu() to ops.enqueue() via per-task storage.
- SCX_TASK_ENQ_LOCAL which told the BPF scheduler that scx_select_cpu_dfl() wants the task to be dispatched to the local DSQ was removed. Instead, scx_bpf_select_cpu_dfl() now dispatches directly if it finds a suitable idle CPU. If such behavior is not desired, users can use scx_bpf_select_cpu_dfl() which returns the verdict in a bool out param.
- Dispatch decisions made in ops.dispatch() may now be cancelled with a new scx_bpf_dispatch_cancel() kfunc.
- A new SCX_KICK_IDLE flag is available for use with scx_bpf_kick_cpu() to only send a resched IPI if the target CPU is idle.
- exit_code added to scx_exit_info. This is used to indicate different exit conditions on non-error exits and enables e.g. handling CPU hotplugs by restarting the scheduler.
- Debug dump added. When the BPF scheduler gets aborted, the states of all runqueues and runnable tasks are captured and sent to the scheduler binary to aid debugging. See https://github.com/sched-ext/scx/issues/234 for an example of the debug dump being used to root cause a bug in scx_lavd.
- The BPF scheduler can now iterate DSQs and consume specific tasks.
- CPU frequency scaling support added through cpuperf kfunc interface.
- The current state of sched_ext can now be monitored through files under /sys/sched_ext instead of /sys/kernel/debug/sched/ext. This is to enable monitoring on kernels which don't enable debugfs. A drgn script tools/sched_ext/scx_show_state.py is added for additional visibility.
- tools/sched_ext/include/scx/compat[.bpf].h and other facilities to allow schedulers to be loaded on older kernels are added. The current tentative target is maintaining backward compatibility for at least one major kernel release where reasonable.
- Code reorganized so that only the parts necessary to integrate with the rest of the kernel are in the header files.
Overview
--------
This patch set proposes a new scheduler class called ‘ext_sched_class’, or sched_ext, which allows scheduling policies to be implemented as BPF programs.
More details will be provided on the overall architecture of sched_ext throughout the various patches in this set, as well as in the “How” section below. We realize that this patch set is a significant proposal, so we will be going into depth in the following “Motivation” section to explain why we think it’s justified. That section is laid out as follows, touching on three main axes where we believe that sched_ext provides significant value:
1. Ease of experimentation and exploration: Enabling rapid iteration of new scheduling policies.
2. Customization: Building application-specific schedulers which implement policies that are not applicable to general-purpose schedulers.
3. Rapid scheduler deployments: Non-disruptive swap outs of scheduling policies in production environments.
After the motivation section, we’ll provide a more detailed (but still high-level) overview of how sched_ext works.
Motivation
----------
1. Ease of experimentation and exploration
*Why is exploration important?*
Scheduling is a challenging problem space. Small changes in scheduling behavior can have a significant impact on various components of a system, with the corresponding effects varying widely across different platforms, architectures, and workloads.
While complexities have always existed in scheduling, they have increased dramatically over the past 10-15 years. In the mid-late 2000s, cores were typically homogeneous and further apart from each other, with the criteria for scheduling being roughly the same across the entire die.
Systems in the modern age are by comparison much more complex. Modern CPU designs, where the total power budget of all CPU cores often far exceeds the power budget of the socket, with dynamic frequency scaling, and with or without chiplets, have significantly expanded the scheduling problem space. Cache hierarchies have become less uniform, with Core Complex (CCX) designs such as recent AMD processors having multiple shared L3 caches within a single socket. Such topologies resemble NUMA sans persistent NUMA node stickiness.
Use-cases have become increasingly complex and diverse as well. Applications such as mobile and VR have strict latency requirements to avoid missing deadlines that impact user experience. Stacking workloads in servers is constantly pushing the demands on the scheduler in terms of workload isolation and resource distribution.
Experimentation and exploration are important for any non-trivial problem space. However, given the recent hardware and software developments, we believe that experimentation and exploration are not just important, but _critical_ in the scheduling problem space.
Indeed, other approaches in industry are already being explored. AMD has proposed an experimental patch set [0] which enables userspace to provide hints to the scheduler via “Userspace Hinting”. The approach adds a prctl() API which allows callers to set a numerical “hint” value on a struct task_struct. This hint is then optionally read by the scheduler to adjust the cost calculus for various scheduling decisions.
[0]: https://lore.kernel.org/lkml/[email protected]/
Huawei have also expressed interest [1] in enabling some form of programmable scheduling. While we’re unaware of any patch sets which have been sent to the upstream list for this proposal, it similarly illustrates the need for more flexibility in the scheduler.
[1]: https://lore.kernel.org/bpf/[email protected]/
Additionally, Google has developed ghOSt [2] with the goal of enabling custom, userspace driven scheduling policies. Prior presentations at LPC [3] have discussed ghOSt and how BPF can be used to accelerate scheduling.
[2]: https://dl.acm.org/doi/pdf/10.1145/3477132.3483542
[3]: https://lpc.events/event/16/contributions/1365/
*Why can’t we just explore directly with CFS?*
Experimenting with CFS directly or implementing a new sched_class from scratch is of course possible, but is often difficult and time consuming. Newcomers to the scheduler often require years to understand the codebase and become productive contributors. Even for seasoned kernel engineers, experimenting with and upstreaming features can take a very long time. The iteration process itself is also time consuming, as testing scheduler changes on real hardware requires reinstalling the kernel and rebooting the host.
Core scheduling is an example of a feature that took a significant amount of time and effort to integrate into the kernel. Part of the difficulty with core scheduling was the inherent mismatch in abstraction between the desire to perform core-wide scheduling, and the per-cpu design of the kernel scheduler. This caused issues, for example ensuring proper fairness between the independent runqueues of SMT siblings.
The high barrier to entry for working on the scheduler is an impediment to academia as well. Master’s/PhD candidates who are interested in improving the scheduler will spend years ramping-up, only to complete their degrees just as they’re finally ready to make significant changes. A lower entrance barrier would allow researchers to more quickly ramp up, test out hypotheses, and iterate on novel ideas. Research methodology is also severely hampered by the high barrier of entry to make modifications; for example, the Shenango [4] and Shinjuku scheduling policies used sched affinity to replicate the desired policy semantics, due to the difficulty of incorporating these policies into the kernel directly.
[4]: https://www.usenix.org/system/files/nsdi19-ousterhout.pdf
The iterative process itself also imposes a significant cost to working on the scheduler. Testing changes requires developers to recompile and reinstall the kernel, reboot their machines, rewarm their workloads, and then finally rerun their benchmarks. Though some of this overhead could potentially be mitigated by enabling schedulers to be implemented as kernel modules, a machine crash or subtle system state corruption is always only one innocuous mistake away. These problems are exacerbated when testing production workloads in a datacenter environment as well, where multiple hosts may be involved in an experiment; requiring a significantly longer ramp up time. Warming up memcache instances in the Meta production environment takes hours, for example.
*How does sched_ext help with exploration?*
sched_ext attempts to address all of the problems described above. In this section, we’ll describe the benefits to experimentation and exploration that are afforded by sched_ext, provide real-world examples of those benefits, and discuss some of the trade-offs and considerations in our design choices.
One of our main goals was to lower the barrier to entry for experimenting with the scheduler. sched_ext provides ergonomic callbacks and helpers to ease common operations such as managing idle CPUs, scheduling tasks on arbitrary CPUs, handling preemptions from other scheduling classes, and more. While sched_ext does require some ramp-up, the complexity is self-contained, and the learning curve gradual. Developers can ramp up by first implementing simple policies such as global weighted vtime scheduling in only tens of lines of code, and then continue to learn the APIs and building blocks available with sched_ext as they build more featureful and complex schedulers.
Another critical advantage provided by sched_ext is the use of BPF. BPF provides strong safety guarantees by statically analyzing programs at load time to ensure that they cannot corrupt or crash the system. sched_ext guarantees system integrity no matter what BPF scheduler is loaded, and provides mechanisms to safely disable the current BPF scheduler and migrate tasks back to a trusted scheduler. For example, we also implement in-kernel safety mechanisms to guarantee that a misbehaving scheduler cannot indefinitely starve tasks. BPF also enables sched_ext to significantly improve iteration speed for running experiments. Loading and unloading a BPF scheduler is simply a matter of running and terminating a sched_ext binary.
BPF also provides programs with a rich set of APIs, such as maps, kfuncs, and BPF helpers. In addition to providing useful building blocks to programs that run entirely in kernel space (such as many of our example schedulers), these APIs also allow programs to leverage user space in making scheduling decisions. Specifically, the Atropos sample scheduler has a relatively simple weighted vtime or FIFO scheduling layer in BPF, paired with a load balancing component in userspace written in Rust. As described in more detail below, we also built a more general user-space scheduling framework called “rhone” by leveraging various BPF features.
On the other hand, BPF does have shortcomings, as can be plainly seen from the complexity in some of the example schedulers. scx_pair.bpf.c illustrates this point well. To start, it requires a good amount of code to emulate cgroup-local-storage. In the kernel proper, this would simply be a matter of adding another pointer to the struct cgroup, but in BPF, it requires a complex juggling of data amongst multiple different maps, a good amount of boilerplate code, and some unwieldy bpf_loop()‘s and atomics. The code is also littered with explicit and often unnecessary sanity checks to appease the verifier.
That being said, BPF is being rapidly improved. For example, Yonghong Song recently upstreamed a patch set [5] to add a cgroup local storage map type, allowing scx_pair.bpf.c to be simplified. There are plans to address other issues as well, such as providing statically-verified locking, and avoiding the need for unnecessary sanity checks. Addressing these shortcomings is a high priority for BPF, and as progress continues to be made, we expect most deficiencies to be addressed in the not-too-distant future.
[5]: https://lore.kernel.org/bpf/[email protected]/
Yet another exploration advantage of sched_ext is helping widening the scope of experiments. For example, sched_ext makes it easy to defer CPU assignment until a task starts executing, allowing schedulers to share scheduling queues at any granularity (hyper-twin, CCX and so on). Additionally, higher level frameworks can be built on top to further widen the scope. For example, the aforementioned “rhone” [6] library allows implementing scheduling policies in user-space by encapsulating the complexity around communicating scheduling decisions with the kernel. This allows taking advantage of a richer programming environment in user-space, enabling experimenting with, for instance, more complex mathematical models.
[6]: https://github.com/Decave/rhone
sched_ext also allows developers to leverage machine learning. At Meta, we experimented with using machine learning to predict whether a running task would soon yield its CPU. These predictions can be used to aid the scheduler in deciding whether to keep a runnable task on its current CPU rather than migrating it to an idle CPU, with the hope of avoiding unnecessary cache misses. Using a tiny neural net model with only one hidden layer of size 16, and a decaying count of 64 syscalls as a feature, we were able to achieve a 15% throughput improvement on an Nginx benchmark, with an 87% inference accuracy.
2. Customization
This section discusses how sched_ext can enable users to run workloads on application-specific schedulers.
*Why deploy custom schedulers rather than improving CFS?*
Implementing application-specific schedulers and improving CFS are not conflicting goals. Scheduling features explored with sched_ext which yield beneficial results, and which are sufficiently generalizable, can and should be integrated into CFS. However, CFS is fundamentally designed to be a general purpose scheduler, and thus is not conducive to being extended with some highly targeted application or hardware specific changes.
Targeted, bespoke scheduling has many potential use cases. For example, VM scheduling can make certain optimizations that are infeasible in CFS due to the constrained problem space (scheduling a static number of long-running VCPUs versus an arbitrary number of threads). Additionally, certain applications might want to make targeted policy decisions based on hints directly from the application (for example, a service that knows the different deadlines of incoming RPCs).
Google has also experimented with some promising, novel scheduling policies. One example is “central” scheduling, wherein a single CPU makes all scheduling decisions for the entire system. This allows most cores on the system to be fully dedicated to running workloads, and can have significant performance improvements for certain use cases. For example, central scheduling with VCPUs can avoid expensive vmexits and cache flushes, by instead delegating the responsibility of preemption checks from the tick to a single CPU. See scx_central.bpf.c for a simple example of a central scheduling policy built in sched_ext.
Some workloads also have non-generalizable constraints which enable optimizations in a scheduling policy which would otherwise not be feasible. For example,VM workloads at Google typically have a low overcommit ratio compared to the number of physical CPUs. This allows the scheduler to support bounded tail latencies, as well as longer blocks of uninterrupted time.
Yet another interesting use case is the scx_flatcg scheduler, which is in 0024-sched_ext-Add-cgroup-support.patch and provides a flattened hierarchical vtree for cgroups. This scheduler does not account for thundering herd problems among cgroups, and therefore may not be suitable for inclusion in CFS. However, in a simple benchmark using wrk[8] on apache serving a CGI script calculating sha1sum of a small file, it outperformed CFS by ~3% with CPU controller disabled and by ~10% with two apache instances competing with 2:1 weight ratio nested four level deep.
Certain industries require specific scheduling behaviors that do not apply broadly. For example, ARINC 653 defines scheduling behavior that is widely used by avionic software, and some out-of-tree implementations
(https://ieeexplore.ieee.org/document/7005306) have been built. While the upstream community may decide to merge one such implementation in the future, it would also be entirely reasonable to not do so given the narrowness of use-case, and non-generalizable, strict requirements. Such cases can be well served by sched_ext in all stages of the software development lifecycle -- development, testing, deployment and maintenance.
There are also classes of policy exploration, such as machine learning, or responding in real-time to application hints, that are significantly harder (and not necessarily appropriate) to integrate within the kernel itself.
*Won’t this increase fragmentation?*
We acknowledge that to some degree, sched_ext does run the risk of increasing the fragmentation of scheduler implementations. As a result of exploration, however, we believe that enabling the larger ecosystem to innovate will ultimately accelerate the overall development and performance of Linux.
BPF programs are required to be GPLv2, which is enforced by the verifier on program loads. With regards to API stability, just as with other semi-internal interfaces such as BPF kfuncs, we won’t be providing any API stability guarantees to BPF schedulers. While we intend to make an effort to provide compatibility when possible, we will not provide any explicit, strong guarantees as the kernel typically does with e.g. UAPI headers. For users who decide to keep their schedulers out-of-tree,the licensing and maintenance overheads will be fundamentally the same as for carrying out-of-tree patches.
With regards to the schedulers included in this patch set, and any other schedulers we implement in the future, both Meta and Google will open-source all of the schedulers we implement which have any relevance to the broader upstream community. We expect that some of these, such as the simple example schedulers and scx_rusty scheduler, will be upstreamed as part of the kernel tree. Distros will be able to package and release these schedulers with the kernel, allowing users to utilize these schedulers out-of-the-box without requiring any additional work or dependencies such as clang or building the scheduler programs themselves. Other schedulers and scheduling frameworks such as rhone may be open-sourced through separate per-project repos.
3. Rapid scheduler deployments
Rolling out kernel upgrades is a slow and iterative process. At a large scale it can take months to roll a new kernel out to a fleet of servers. While this latency is expected and inevitable for normal kernel upgrades, it can become highly problematic when kernel changes are required to fix bugs. Livepatch [8] is available to quickly roll out critical security fixes to large fleets, but the scope of changes that can be applied with livepatching is fairly limited, and would likely not be usable for patching scheduling policies. With sched_ext, new scheduling policies can be rapidly rolled out to production environments.
[8]: https://www.kernel.org/doc/html/latest/livepatch/livepatch.html
As an example, one of the variants of the L1 Terminal Fault (L1TF) [9] vulnerability allows a VCPU running a VM to read arbitrary host kernel memory for pages in L1 data cache. The solution was to implement core scheduling, which ensures that tasks running as hypertwins have the same “cookie”.
[9]: https://www.intel.com/content/www/us/en/architecture-and-technology/l1tf.html
While core scheduling works well, it took a long time to finalize and land upstream. This long rollout period was painful, and required organizations to make difficult choices amongst a bad set of options. Some companies such as Google chose to implement and use their own custom L1TF-safe scheduler, others chose to run without hyper-threading enabled, and yet others left hyper-threading enabled and crossed their fingers.
Once core scheduling was upstream, organizations had to upgrade the kernels on their entire fleets. As downtime is not an option for many, these upgrades had to be gradually rolled out, which can take a very long time for large fleets.
An example of an sched_ext scheduler that illustrates core scheduling semantics is scx_pair.bpf.c, which co-schedules pairs of tasks from the same cgroup, and is resilient to L1TF vulnerabilities. While this example scheduler is certainly not suitable for production in its current form, a similar scheduler that is more performant and featureful could be written and deployed if necessary.
Rapid scheduling deployments can similarly be useful to quickly roll-out new scheduling features without requiring kernel upgrades. At Google, for example, it was observed that some low-priority workloads were causing degraded performance for higher-priority workloads due to consuming a disproportionate share of memory bandwidth. While a temporary mitigation was to use sched affinity to limit the footprint of this low-priority workload to a small subset of CPUs, a preferable solution would be to implement a more featureful task-priority mechanism which automatically throttles lower-priority tasks which are causing memory contention for the rest of the system. Implementing this in CFS and rolling it out to the fleet could take a very long time.
sched_ext would directly address these gaps. If another hardware bug or resource contention issue comes in that requires scheduler support to mitigate, sched_ext can be used to experiment with and test different policies. Once a scheduler is available, it can quickly be rolled out to as many hosts as necessary, and function as a stop-gap solution until a longer-term mitigation is upstreamed.
How
---
sched_ext is a new sched_class which allows scheduling policies to be implemented in BPF programs.
sched_ext leverages BPF’s struct_ops feature to define a structure which exports function callbacks and flags to BPF programs that wish to implement scheduling policies. The struct_ops structure exported by sched_ext is struct sched_ext_ops, and is conceptually similar to struct sched_class. The role of sched_ext is to map the complex sched_class callbacks to the more simple and ergonomic struct sched_ext_ops callbacks.
Unlike some other BPF program types which have ABI requirements due to exporting UAPIs, struct_ops has no ABI requirements whatsoever. This provides us with the flexibility to change the APIs provided to schedulers as necessary. BPF struct_ops is also already being used successfully in other subsystems, such as in support of TCP congestion control.
The only struct_ops field that is required to be specified by a scheduler is the ‘name’ field. Otherwise, sched_ext will provide sane default behavior, such as automatically choosing an idle CPU on the task wakeup path if .select_cpu() is missing.
*Dispatch queues*
To bridge the workflow imbalance between the scheduler core and sched_ext_ops callbacks, sched_ext uses simple FIFOs called dispatch queues (dsq's). By default, there is one global dsq (SCX_DSQ_GLOBAL), and one local per-CPU dsq (SCX_DSQ_LOCAL). SCX_DSQ_GLOBAL is provided for convenience and need not be used by a scheduler that doesn't require it. As described in more detail below, SCX_DSQ_LOCAL is the per-CPU FIFO that sched_ext pulls from when putting the next task on the CPU. The BPF scheduler can manage an arbitrary number of dsq's using scx_bpf_create_dsq() and scx_bpf_destroy_dsq().
*Scheduling cycle*
The following briefly shows a typical workflow for how a waking task is scheduled and executed.
1. When a task is waking up, .select_cpu() is the first operation invoked. This serves two purposes. It both allows a scheduler to optimize task placement by specifying a CPU where it expects the task to eventually be scheduled, and the latter is that the selected CPU will be woken if it’s idle.
2. Once the target CPU is selected, .enqueue() is invoked. It can make one of the following decisions:
- Immediately dispatch the task to either the global dsq (SCX_DSQ_GLOBAL) or the current CPU’s local dsq (SCX_DSQ_LOCAL).
- Immediately dispatch the task to a user-created dispatch queue.
- Queue the task on the BPF side, e.g. in an rbtree map for a vruntime scheduler, with the intention of dispatching it at a later time from .dispatch().
3. When a CPU is ready to schedule, it first looks at its local dsq. If empty, it invokes .consume() which should make one or more scx_bpf_consume() calls to consume tasks from dsq's. If a scx_bpf_consume() call succeeds, the CPU has the next task to run and .consume() can return. If .consume() is not defined, sched_ext will by-default consume from only the built-in SCX_DSQ_GLOBAL dsq.
4. If there's still no task to run, .dispatch() is invoked which should make one or more scx_bpf_dispatch() calls to dispatch tasks from the BPF scheduler to one of the dsq's. If more than one task has been dispatched, go back to the previous consumption step.
*Verifying callback behavior*
sched_ext always verifies that any value returned from a callback is valid, and will issue an error and unload the scheduler if it is not. For example, if .select_cpu() returns an invalid CPU, or if an attempt is made to invoke the scx_bpf_dispatch() with invalid enqueue flags. Furthermore, if a task remains runnable for too long without being scheduled, sched_ext will detect it and error-out the scheduler.
Closing Thoughts
----------------
Both Meta and Google have experimented quite a lot with schedulers in the last several years. Google has benchmarked various workloads using user space scheduling, and have achieved performance wins by trading off generality for application specific needs. At Meta, we are actively experimenting with multiple production workloads and seeing significant performance gains, and are in the process of deploying sched_ext schedulers on production workloads at scale. We expect to leverage it extensively to run various experiments and develop customized schedulers for a number of critical workloads.
In closing, both Meta and Google believe that sched_ext will significantly evolve how the broader community explores the scheduling problem space, while also enabling targeted policies for custom applications. We’ll be able to experiment easier and faster, explore uncharted areas, and deploy emergency scheduler changes when necessary. The same applies to anyone who wants to work on the scheduler, including academia and specialized industries. sched_ext will push forward the state of the art when it comes to scheduling and performance in Linux.
Written By
----------
David Vernet <[email protected]>
Josh Don <[email protected]>
Tejun Heo <[email protected]>
Barret Rhoden <[email protected]>
Supported By
------------
Paul Turner <[email protected]>
Neel Natu <[email protected]>
Patrick Bellasi <[email protected]>
Hao Luo <[email protected]>
Dimitrios Skarlatos <[email protected]>
Patchset
--------
This patchset is on top of bpf/for-next as of 2024-04-29:
07801a24e2f1 ("bpf, docs: Clarify PC use in instruction-set.rst")
and contains the following patches:
NOTE: The doc added by 0038 contains a high-level overview and might be good
place to start.
0001-cgroup-Implement-cgroup_show_cftypes.patch 0002-sched-Restructure-sched_class-order-sanity-checks-in.patch 0003-sched-Allow-sched_cgroup_fork-to-fail-and-introduce-.patch 0004-sched-Add-sched_class-reweight_task.patch 0005-sched-Add-sched_class-switching_to-and-expose-check_.patch 0006-sched-Factor-out-cgroup-weight-conversion-functions.patch 0007-sched-Expose-css_tg-and-__setscheduler_prio.patch 0008-sched-Enumerate-CPU-cgroup-file-types.patch 0009-sched-Add-reason-to-sched_class-rq_-on-off-line.patch 0010-sched-Factor-out-update_other_load_avgs-from-__updat.patch 0011-cpufreq_schedutil-Refactor-sugov_cpu_is_busy.patch 0012-sched-Add-normal_policy.patch 0013-sched_ext-Add-boilerplate-for-extensible-scheduler-c.patch 0014-sched_ext-Implement-BPF-extensible-scheduler-class.patch 0015-sched_ext-Add-scx_simple-and-scx_example_qmap-exampl.patch 0016-sched_ext-Add-sysrq-S-which-disables-the-BPF-schedul.patch 0017-sched_ext-Implement-runnable-task-stall-watchdog.patch 0018-sched_ext-Allow-BPF-schedulers-to-disallow-specific-.patch 0019-sched_ext-Print-sched_ext-info-when-dumping-stack.patch 0020-sched_ext-Print-debug-dump-after-an-error-exit.patch 0021-tools-sched_ext-Add-scx_show_state.py.patch 0022-sched_ext-Implement-scx_bpf_kick_cpu-and-task-preemp.patch 0023-sched_ext-Add-a-central-scheduler-which-makes-all-sc.patch 0024-sched_ext-Make-watchdog-handle-ops.dispatch-looping-.patch 0025-sched_ext-Add-task-state-tracking-operations.patch 0026-sched_ext-Implement-tickless-support.patch 0027-sched_ext-Track-tasks-that-are-subjects-of-the-in-fl.patch 0028-sched_ext-Add-cgroup-support.patch 0029-sched_ext-Add-a-cgroup-scheduler-which-uses-flattene.patch 0030-sched_ext-Implement-SCX_KICK_WAIT.patch 0031-sched_ext-Implement-sched_ext_ops.cpu_acquire-releas.patch 0032-sched_ext-Implement-sched_ext_ops.cpu_online-offline.patch 0033-sched_ext-Bypass-BPF-scheduler-while-PM-events-are-i.patch 0034-sched_ext-Implement-core-sched-support.patch 0035-sched_ext-Add-vtime-ordered-priority-queue-to-dispat.patch 0036-sched_ext-Implement-DSQ-iterator.patch 0037-sched_ext-Add-cpuperf-support.patch 0038-sched_ext-Documentation-scheduler-Document-extensibl.patch 0039-sched_ext-Add-selftests.patch
0001 : Cgroup prep.
0002-0012: Scheduler prep.
0013-0015: sched_ext core implementation and a couple example BPF scheduler.
0016-0021: Utility features including safety mechanisms, switch-all and
printing sched_ext state when dumping backtraces.
0022-0027: Kicking and preempting other CPUs, task state transition tracking
and tickless support. Demonstrated with an example central scheduler which makes all scheduling decisions on one CPU.
0029-0030: cgroup support and the ability to wait for other CPUs after
kicking them.
0031-0033: Add CPU preemption and hotplug and power-management support.
0034 : Add core-sched support.
0035-0036: Add DSQ rbtree and iterator support.
0037 : Add cpuperf (frequency scaling) support.
0038 : Add documentation.
0039 : Add selftests.
The patchset is also available in the following git branch:
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git sched_ext-v6
diffstat follows.
Documentation/scheduler/index.rst | 1 Documentation/scheduler/sched-ext.rst | 307 + MAINTAINERS | 13 Makefile | 8 drivers/tty/sysrq.c | 1 include/asm-generic/vmlinux.lds.h | 1 include/linux/cgroup-defs.h | 8 include/linux/cgroup.h | 5 include/linux/sched.h | 5 include/linux/sched/ext.h | 210 include/linux/sched/task.h | 3 include/uapi/linux/sched.h | 1 init/Kconfig | 5 init/init_task.c | 12 kernel/Kconfig.preempt | 24 kernel/cgroup/cgroup.c | 97 kernel/fork.c | 17 kernel/sched/build_policy.c | 8 kernel/sched/core.c | 324 + kernel/sched/cpufreq_schedutil.c | 50 kernel/sched/deadline.c | 4 kernel/sched/debug.c | 3 kernel/sched/ext.c | 6641 +++++++++++++++++++++++++++++++ kernel/sched/ext.h | 139 kernel/sched/fair.c | 25 kernel/sched/idle.c | 2 kernel/sched/rt.c | 4 kernel/sched/sched.h | 123 kernel/sched/topology.c | 4 lib/dump_stack.c | 1 tools/Makefile | 10 tools/sched_ext/.gitignore | 2 tools/sched_ext/Makefile | 246 + tools/sched_ext/README.md | 270 + tools/sched_ext/include/bpf-compat/gnu/stubs.h | 11 tools/sched_ext/include/scx/common.bpf.h | 301 + tools/sched_ext/include/scx/common.h | 71 tools/sched_ext/include/scx/compat.bpf.h | 110 tools/sched_ext/include/scx/compat.h | 197 tools/sched_ext/include/scx/user_exit_info.h | 111 tools/sched_ext/scx_central.bpf.c | 361 + tools/sched_ext/scx_central.c | 135 tools/sched_ext/scx_flatcg.bpf.c | 939 ++++ tools/sched_ext/scx_flatcg.c | 233 + tools/sched_ext/scx_flatcg.h | 51 tools/sched_ext/scx_qmap.bpf.c | 673 +++ tools/sched_ext/scx_qmap.c | 150 tools/sched_ext/scx_show_state.py | 39 tools/sched_ext/scx_simple.bpf.c | 149 tools/sched_ext/scx_simple.c | 107 tools/testing/selftests/sched_ext/.gitignore | 6 tools/testing/selftests/sched_ext/Makefile | 216 + tools/testing/selftests/sched_ext/config | 9 tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c | 42 tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.c | 57 tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c | 39 tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.c | 56 tools/testing/selftests/sched_ext/enq_last_no_enq_fails.bpf.c | 21 tools/testing/selftests/sched_ext/enq_last_no_enq_fails.c | 60 tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c | 43 tools/testing/selftests/sched_ext/enq_select_cpu_fails.c | 61 tools/testing/selftests/sched_ext/exit.bpf.c | 84 tools/testing/selftests/sched_ext/exit.c | 55 tools/testing/selftests/sched_ext/exit_test.h | 20 tools/testing/selftests/sched_ext/hotplug.bpf.c | 55 tools/testing/selftests/sched_ext/hotplug.c | 168 tools/testing/selftests/sched_ext/hotplug_test.h | 15 tools/testing/selftests/sched_ext/init_enable_count.bpf.c | 53 tools/testing/selftests/sched_ext/init_enable_count.c | 166 tools/testing/selftests/sched_ext/maximal.bpf.c | 164 tools/testing/selftests/sched_ext/maximal.c | 51 tools/testing/selftests/sched_ext/maybe_null.bpf.c | 26 tools/testing/selftests/sched_ext/maybe_null.c | 40 tools/testing/selftests/sched_ext/maybe_null_fail.bpf.c | 25 tools/testing/selftests/sched_ext/minimal.bpf.c | 21 tools/testing/selftests/sched_ext/minimal.c | 58 tools/testing/selftests/sched_ext/prog_run.bpf.c | 32 tools/testing/selftests/sched_ext/prog_run.c | 78 tools/testing/selftests/sched_ext/reload_loop.c | 75 tools/testing/selftests/sched_ext/runner.c | 201 tools/testing/selftests/sched_ext/scx_test.h | 131 tools/testing/selftests/sched_ext/select_cpu_dfl.bpf.c | 40 tools/testing/selftests/sched_ext/select_cpu_dfl.c | 72 tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c | 89 tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c | 72 tools/testing/selftests/sched_ext/select_cpu_dispatch.bpf.c | 41 tools/testing/selftests/sched_ext/select_cpu_dispatch.c | 70 tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.bpf.c | 37 tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.c | 56 tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.bpf.c | 38 tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.c | 56 tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c | 92 tools/testing/selftests/sched_ext/select_cpu_vtime.c | 59 tools/testing/selftests/sched_ext/test_example.c | 49 tools/testing/selftests/sched_ext/util.c | 71 tools/testing/selftests/sched_ext/util.h | 13 96 files changed, 15056 insertions(+), 139 deletions(-)
Patchset History
----------------
v4 (http://lkml.kernel.org/r/[email protected]) -> v5:
- Updated to rebase on top of the current bpf/for-next (2023-11-06). '0002-0010: Scheduler prep' were simply rebased on top of new EEVDF scheduler which demonstrate clean cut API boundary between sched-ext and sched core.
- To accommodate 32bit configs, fields which use atomic ops and store_release/load_acquire are switched from 64bits to longs.
- To help triaging, if sched_ext is enabled, backtrace dumps now show the currently active scheduler along with some debug information.
- Fixes for bugs including p->scx.flags corruption due to unsynchronized SCX_TASK_DSQ_ON_PRIQ changes, and overly permissive BTF struct and scx_bpf kfunc access checks.
- Other misc changes including renaming "type" to "kind" in scx_exit_info to ease usage from rust and other languages in which "type" is a reserved keyword.
- scx_atropos is renamed to scx_rusty and received signficant updates to improve scalability. Load metrics are now tracked in BPF and accessed only as necessary from userspace.
- Misc example scheduler improvements including the usage of resizable BPF .bss array, the introduction of SCX_BUG[_ON](), and timer CPU pinning in scx_central.
- Improve Makefile and documentation for example schedulers.
v3 (https://lkml.kernel.org/r/[email protected]) -> v4:
- There aren't any significant changes to the sched_ext API even though we kept experimenting heavily with a couple BPF scheduler implementations indicating that the core API reached a level of maturity.
- 0002-sched-Encapsulate-task-attribute-change-sequence-int.patch which implemented custom guard scope for scheduler attribute changes dropped as upstream is moving towards a more generic implementation.
- Build fixes with different CONFIG combinations.
- Core code cleanups and improvements including how idle CPU is selected and disabling ttwu_queue for tasks on SCX to avoid confusing BPF schedulers expecting ->select_cpu() call. See 0012-sched_ext-Implement-BPF-extensible-scheduler-class.patch for more details.
- "_example" dropped from the example schedulers as the distinction between the example-only and practically-useful isn't black-and-white. Instead, each scheduler has detailed comments and there's also a README file.
- scx_central, scx_pair and scx_flatcg are moved into their own patches as suggested by Josh Don.
- scx_atropos received sustantial updates including fixes for bugs that could cause temporary stalls and improvements in load balancing and wakeup target CPU selection. For details, See 0034-sched_ext-Add-a-rust-userspace-hybrid-example-schedu.patch.
v2 (http://lkml.kernel.org/r/[email protected]) -> v3:
- ops.set_weight() added to allow BPF schedulers to track weight changes without polling p->scx.weight.
- scx_bpf_task_cgroup() kfunc added to allow BPF scheduler to reliably determine the current cpu cgroup under rq lock protection. This required improving the kf_mask SCX operation verification mechanism and adding 0023-sched_ext-Track-tasks-that-are-subjects-of-the-in-fl.patch.
- Updated to use the latest BPF improvements including KF_RCU and the inline iterator.
- scx_example_flatcg added to 0024-sched_ext-Add-cgroup-support.patch. It uses the new BPF RB tree support to implement flattened cgroup hierarchy.
- A DSQ now also contains an rbtree so that it can be used to implement vtime based scheduling among tasks sharing a DSQ conveniently and efficiently. For more details, see 0029-sched_ext-Add-vtime-ordered-priority-queue-to-dispat.patch. All eligible example schedulers are updated to default to weighted vtime scheduilng.
- atropos scheduler's userspace code is substantially restructred and rewritten. The binary is renamed to scx_atropos and can auto-config the domains according to the cache topology.
- Various other example scheduler updates including scx_example_dummy being renamed to scx_example_simple, the example schedulers defaulting to enabling switch_all and clarifying performance expectation of each example scheduler.
- A bunch of fixes and improvements. Please refer to each patch for details.
v1 (http://lkml.kernel.org/r/[email protected]) -> v2:
- Rebased on top of bpf/for-next - a5f6b9d577eb ("Merge branch 'Enable struct_ops programs to be sleepable'"). There were several missing features including generic cpumask helpers and sleepable struct_ops operation support that v1 was working around. The rebase gets rid of all SCX specific temporary helpers.
- Some kfunc helpers are context-sensitive and can only be called from specific operations. v1 didn't restrict kfunc accesses allowing them to be misused which can lead to crashes and other malfunctions. v2 makes more kfuncs safe to be called from anywhere and implements per-task mask based runtime access control for the rest. The longer-term plan is to make the BPF verifier enforce these restrictions. Combined with the above, sans mistakes and bugs, it shouldn't be possible to crash the machine through SCX and its helpers.
- Core-sched support. While v1 implemented the pick_task operation, there were multiple missing pieces for working core-sched support. v2 adds 0027-sched_ext-Implement-core-sched-support.patch. SCX by default implements global FIFO ordering and allows the BPF schedulers to implement custom ordering via scx_ops.core_sched_before(). scx_example_qmap is updated so that the five queues' relative priorities are correctly reflected when core-sched is enabled.
- Dropped balance_scx_on_up() which was called from put_prev_task_balance(). UP support is now contained in SCX proper.
- 0002-sched-Encapsulate-task-attribute-change-sequence-int.patch adds SCHED_CHANGE_BLOCK() which encapsulates the preparation and restoration sequences used for task attribute changes. For SCX, this replaces sched_deq_and_put_task() and sched_enq_and_set_task() from v1.
- 0011-sched-Add-reason-to-sched_move_task.patch dropped from v1. SCX now distinguishes cgroup and autogroup tg's using task_group_is_autogroup().
- Other misc changes including fixes for bugs that Julia Lawall noticed and patch descriptions updates with more details on how the introduced changes are going to be used.
- MAINTAINERS entries added.
The followings are discussion points which were raised but didn't result in code changes in this iteration.
- There were discussions around exposing __setscheduler_prio() and, in v2, SCHED_CHANGE_BLOCK() in kernel/sched/sched.h. Switching scheduler implementations is innate for SCX. At the very least, it needs to be able to turn on and off the BPF scheduler which requires something equivalent to SCHED_CHANGE_BLOCK(). The use of __setscheduler_prio() depends on the behavior we want to present to userspace. The current one of using CFS as the fallback when BPF scheduler is not available seems more friendly and less error-prone to other options.
- Another discussion point was around for_each_active_class() and friends which skip over CFS or SCX when it's known that the sched_class must be empty. I left it as-is for now as it seems to be cleaner and more robust than trying to plug each operation which may added unnecessary overheads.
Thanks.
--
tejun
Tejun Heo
maintainerPATCH 1/39 cgroup: Implement cgroup_show_cftypes()
Implement cgroup_show_cftypes() which shows and hides all cgroup files associated with the specified set of cgroup file types. CFTYPE_HIDDEN flag is added so that files can be created hidden from the get-go.
cgroup_show_cftypes() can be used whether the cftypes are added or not. It also combines with cgroup_show_file() so that a given file is visible iff both its cftype and cfile are visible.
This will be used by a new sched_class to selectively show and hide CPU controller interface files depending on whether they're supported.
--- include/linux/cgroup-defs.h | 8 +++ include/linux/cgroup.h | 1 + kernel/cgroup/cgroup.c | 97 ++++++++++++++++++++++++++++++++++--- 3 files changed, 99 insertions(+), 7 deletions(-)
include/linux/cgroup-defs.h +8 -0
@@ -132,12 +132,18 @@ enum {CFTYPE_WORLD_WRITABLE = (1 <<4 ),/* (DON'T USE FOR NEW FILES) S_IWUGO */ CFTYPE_DEBUG = (1 <<5 ),/* create when cgroup_debug */ +CFTYPE_HIDDEN = (1 <<6 ),/* file type hidden, see cgroup_show_cftypes() */ +/* internal flags, do not use outside cgroup core proper */ __CFTYPE_ONLY_ON_DFL = (1 <<16 ),/* only on default hierarchy */ __CFTYPE_NOT_ON_DFL = (1 <<17 ),/* not on default hierarchy */ __CFTYPE_ADDED = (1 << 18), }; +enum cfile_flags { +CFILE_HIDDEN = (1 <<0 ),/* file instance hidden */ +}; +/* * cgroup_file is the handle for a file instance created in a cgroup which * is used, for example, to generate file changed notifications. This can
@@ -145,7 +151,9 @@ enum {*/ struct cgroup_file {/* do not access any fields from outside cgroup core */ +struct cftype *cft;struct kernfs_node *kn; +unsigned int flags;unsigned long notified_at;struct timer_list notify_timer; };
include/linux/cgroup.h +1 -0
@@ -114,6 +114,7 @@ int cgroup_transfer_tasks(struct cgroup *to, struct cgroup *from);int cgroup_add_dfl_cftypes (struct cgroup_subsys *ss,struct cftype *cfts);int cgroup_add_legacy_cftypes (struct cgroup_subsys *ss,struct cftype *cfts);int cgroup_rm_cftypes (struct cftype *cfts); +void cgroup_show_cftype (struct cftype *cft,bool show);void cgroup_file_notify (struct cgroup_file *cfile);void cgroup_file_show (struct cgroup_file *cfile,bool show);
kernel/cgroup/cgroup.c +90 -7
127 lines not on this page · raw message
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 2/39 sched: Restructure sched_class order sanity checks in sched_init()
In reply to Tejun HeoCurrently, sched_init() checks that the sched_class'es are in the expected order by testing each adjacency which is a bit brittle and makes it cumbersome to add optional sched_class'es. Instead, let's verify whether they're in the expected order using sched_class_above() which is what matters.
--- kernel/sched/core.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/sched/core.c +4 -4
@@ -9924,12 +9924,12 @@ void __init sched_init(void)int i;/* Make sure the linker didn't screw up */ -BUG_ON (&idle_sched_class != &fair_sched_class +1 || - &fair_sched_class != &rt_sched_class +1 || - &rt_sched_class != &dl_sched_class +1 );#ifdef CONFIG_SMP -BUG_ON (&dl_sched_class != &stop_sched_class +1 ); +BUG_ON (!sched_class_above (&stop_sched_class, &dl_sched_class));#endif +BUG_ON (!sched_class_above (&dl_sched_class, &rt_sched_class)); +BUG_ON (!sched_class_above (&rt_sched_class, &fair_sched_class)); +BUG_ON (!sched_class_above (&fair_sched_class, &idle_sched_class));wait_bit_init ();
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 3/39 sched: Allow sched_cgroup_fork() to fail and introduce sched_cancel_fork()
In reply to Tejun HeoA new BPF extensible sched_class will need more control over the forking process. It wants to be able to fail from sched_cgroup_fork() after the new task's sched_task_group is initialized so that the loaded BPF program can prepare the task with its cgroup association is established and reject fork if e.g. allocation fails.
Allow sched_cgroup_fork() to fail by making it return int instead of void and adding sched_cancel_fork() to undo sched_fork() in the error path.
sched_cgroup_fork() doesn't fail yet and this patch shouldn't cause any behavior changes.
v2: Patch description updated to detail the expected use.
--- include/linux/sched/task.h | 3 ++- kernel/fork.c | 15 ++++++++++----- kernel/sched/core.c | 8 +++++++- 3 files changed, 19 insertions(+), 7 deletions(-)
include/linux/sched/task.h +2 -1
@@ -63,7 +63,8 @@ extern asmlinkage void schedule_tail(struct task_struct *prev);extern void init_idle (struct task_struct *idle,int cpu);extern int sched_fork (unsigned long clone_flags,struct task_struct *p); -extern void sched_cgroup_fork (struct task_struct *p,struct kernel_clone_args *kargs); +extern int sched_cgroup_fork (struct task_struct *p,struct kernel_clone_args *kargs); +extern void sched_cancel_fork (struct task_struct *p);extern void sched_post_fork (struct task_struct *p);extern void sched_dead (struct task_struct *p);
kernel/fork.c +10 -5
@@ -2362,7 +2362,7 @@ __latent_entropy struct task_struct *copy_process( retval =perf_event_init_task (p, clone_flags);if (retval) - goto bad_fork_cleanup_policy; + goto bad_fork_sched_cancel_fork; retval =audit_alloc (p);if (retval) goto bad_fork_cleanup_perf;
@@ -2495,7 +2495,9 @@ __latent_entropy struct task_struct *copy_process(* cgroup specific, it unconditionally needs to place the task on a * runqueue. */ -sched_cgroup_fork (p, args); + retval =sched_cgroup_fork (p, args); +if (retval) + goto bad_fork_cancel_cgroup;/* * From this point on we must avoid any synchronous user-space
@@ -2541,13 +2543,13 @@ __latent_entropy struct task_struct *copy_process(/* Don't start children in a dying pid namespace */ if (unlikely (!(ns_of_pid (pid)->pid_allocated &PIDNS_ADDING ))) { retval = -ENOMEM ; - goto bad_fork_cancel_cgroup; + goto bad_fork_core_free; }/* Let kill terminate clone/fork in the middle */ if (fatal_signal_pending (current)) { retval = -EINTR ; - goto bad_fork_cancel_cgroup; + goto bad_fork_core_free; }/* No more failure paths after this point. */
@@ -2621,10 +2623,11 @@ __latent_entropy struct task_struct *copy_process(return p; -bad_fork_cancel_cgroup: +bad_fork_core_free:sched_core_free (p);spin_unlock (¤t->sighand->siglock);write_unlock_irq (&tasklist_lock); +bad_fork_cancel_cgroup:cgroup_cancel_fork (p, args); bad_fork_put_pidfd:if (clone_flags &CLONE_PIDFD ) {
@@ -2663,6 +2666,8 @@ __latent_entropy struct task_struct *copy_process(audit_free (p); bad_fork_cleanup_perf:perf_event_free_task (p); +bad_fork_sched_cancel_fork: +sched_cancel_fork (p); bad_fork_cleanup_policy:lockdep_free_task (p);#ifdef CONFIG_NUMA
kernel/sched/core.c +7 -1
@@ -4817,7 +4817,7 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)return 0 ; } -void sched_cgroup_fork (struct task_struct *p,struct kernel_clone_args *kargs) +int sched_cgroup_fork (struct task_struct *p,struct kernel_clone_args *kargs) {unsigned long flags;
@@ -4844,6 +4844,12 @@ void sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs)if (p->sched_class->task_fork) p->sched_class->task_fork (p);raw_spin_unlock_irqrestore (&p->pi_lock, flags); + +return 0 ; +} + +void sched_cancel_fork (struct task_struct *p) +{ }void sched_post_fork (struct task_struct *p)
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 4/39 sched: Add sched_class->reweight_task()
In reply to Tejun HeoCurrently, during a task weight change, sched core directly calls reweight_task() defined in fair.c if @p is on CFS. Let's make it a proper sched_class operation instead. CFS's reweight_task() is renamed to reweight_task_fair() and now called through sched_class.
While it turns a direct call into an indirect one, set_load_weight() isn't called from a hot path and this change shouldn't cause any noticeable difference. This will be used to implement reweight_task for a new BPF extensible sched_class so that it can keep its cached task weight up-to-date.
--- kernel/sched/core.c | 4 ++-- kernel/sched/fair.c | 3 ++- kernel/sched/sched.h | 4 ++-- 3 files changed, 6 insertions(+), 5 deletions(-)
kernel/sched/core.c +2 -2
@@ -1342,8 +1342,8 @@ static void set_load_weight(struct task_struct *p, bool update_load)* SCHED_OTHER tasks have to update their load when changing their * weight */ -if (update_load && p->sched_class == &fair_sched_class) { -reweight_task (p, prio); +if (update_load && p->sched_class->reweight_task) { + p->sched_class->reweight_task (task_rq (p), p, prio); }else { load->weight =scale_load (sched_prio_to_weight[prio]); load->inv_weight = sched_prio_to_wmult[prio];
kernel/sched/fair.c +2 -1
@@ -3835,7 +3835,7 @@ static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, } } -void reweight_task (struct task_struct *p,int prio) +static void reweight_task_fair (struct rq *rq,struct task_struct *p,int prio) {struct sched_entity *se = &p->se;struct cfs_rq *cfs_rq =cfs_rq_of (se);
@@ -13135,6 +13135,7 @@ DEFINE_SCHED_CLASS(fair) = { .task_tick = task_tick_fair, .task_fork = task_fork_fair, + .reweight_task = reweight_task_fair, .prio_changed = prio_changed_fair, .switched_from = switched_from_fair, .switched_to = switched_to_fair,
kernel/sched/sched.h +2 -2
@@ -2303,6 +2303,8 @@ struct sched_class {*/ void (*switched_from)(struct rq *this_rq,struct task_struct *task);void (*switched_to) (struct rq *this_rq,struct task_struct *task); +void (*reweight_task)(struct rq *this_rq,struct task_struct *task, +int newprio);void (*prio_changed) (struct rq *this_rq,struct task_struct *task,int oldprio);
@@ -2460,8 +2462,6 @@ extern void init_sched_dl_class(void);extern void init_sched_rt_class (void );extern void init_sched_fair_class (void ); -extern void reweight_task (struct task_struct *p,int prio); -extern void resched_curr (struct rq *rq);extern void resched_cpu (int cpu);
Signature · 1 line
2.44.0
Peter Zijlstra
maintainerRe: [PATCH 04/39] sched: Add sched_class->reweight_task()
Quoted 27 lines · Tejun Heo
Currently, during a task weight change, sched core directly calls
reweight_task()defined infair.cif @p is on CFS. Let's make it a propersched_classoperation instead. CFS'sreweight_task()is renamed toreweight_task_fair()and now called throughsched_class.While it turns a direct call into an indirect one,
set_load_weight()isn't called from a hot path and this change shouldn't cause any noticeable difference. This will be used to implementreweight_taskfor a new BPF extensiblesched_classso that it can keep its cached task weight up-to-date.Signed-off-by Tejun Heo <[email protected]>Reviewed-by David Vernet <[email protected]>Acked-by Josh Don <[email protected]>Acked-by Hao Luo <[email protected]>Acked-by Barret Rhoden <[email protected]>--- kernel/sched/core.c | 4 ++-- kernel/sched/fair.c | 3 ++- kernel/sched/sched.h | 4 ++-- 3 files changed, 6 insertions(+), 5 deletions(-)kernel/sched/core.c +0
-0
kernel/sched/core.c static void set_load_weight(struct task_struct *p, bool update_load) · line 1349
Quoted 10 lines
kernel/sched/core.c +2
-2@@ -1342,8 +1342,8 @@ static void set_load_weight(struct task_struct *p, bool update_load)* SCHED_OTHER tasks have to update their load when changing their * weight */ -if (update_load && p->sched_class == &fair_sched_class) { -reweight_task (p, prio); +if (update_load && p->sched_class->reweight_task) { + p->sched_class->reweight_task (task_rq (p), p, prio); }else { load->weight =scale_load (sched_prio_to_weight[prio]); load->inv_weight = sched_prio_to_wmult[prio];
This reminds me, I think we have a bug here...
https://lkml.kernel.org/r/[email protected]
I *think* we want something like the below, hmm?
kernel/sched/core.c +11 -10
@@ -1328,15 +1328,15 @@ int tg_nop(struct task_group *tg, void *data)void set_load_weight (struct task_struct *p,bool update_load) {int prio = p->static_prio -MAX_RT_PRIO ; -struct load_weight *load = &p->se.load; +unsigned long weight; +u32 inv_weight; -/* -* SCHED_IDLE tasks get minimal weight: -*/ if (task_has_idle_policy (p)) { - load->weight =scale_load (WEIGHT_IDLEPRIO ); - load->inv_weight =WMULT_IDLEPRIO ; -return ; + weight =scale_load (WEIGHT_IDLEPRIO ); + inv_weight =WMULT_IDLEPRIO ; + }else { + weight =scale_load (sched_prio_to_weight[prio]); + inv_weight = sched_prio_to_wmult[prio]; }/*
@@ -1344,10 +1344,11 @@ void set_load_weight(struct task_struct *p, bool update_load)* weight */ if (update_load && p->sched_class == &fair_sched_class) { -reweight_task (p, prio); +reweight_task (p, weight, inv_weight); }else { - load->weight =scale_load (sched_prio_to_weight[prio]); - load->inv_weight = sched_prio_to_wmult[prio]; +struct load_weight *lw = &p->se.load; + lw->weight = weight; + lw->inv_weight = inv_weight; } }
kernel/sched/fair.c +1 -1
@@ -3835,7 +3835,7 @@ static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, } } -void reweight_task (struct task_struct *p,int prio) +void reweight_task (struct task_struct *p,unsigned long weight,u32 inv_weight) {struct sched_entity *se = &p->se;struct cfs_rq *cfs_rq =cfs_rq_of (se);
kernel/sched/sched.h +1 -1
@@ -2509,7 +2509,7 @@ extern void init_sched_dl_class(void);extern void init_sched_rt_class (void );extern void init_sched_fair_class (void ); -extern void reweight_task (struct task_struct *p,int prio); +extern void reweight_task (struct task_struct *p,unsigned long weight,u32 inv_weight);extern void resched_curr (struct rq *rq);extern void resched_cpu (int cpu);
Peter Zijlstra
maintainerRe: [PATCH 04/39] sched: Add sched_class->reweight_task()
kernel/sched/fair.c static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, · line 3841
Quoted 13 lines · Peter Zijlstra
kernel/sched/fair.c +1
-1@@ -3835,7 +3835,7 @@ static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, } } -void reweight_task (struct task_struct *p,int prio) +void reweight_task (struct task_struct *p,unsigned long weight,u32 inv_weight) {struct sched_entity *se = &p->se;struct cfs_rq *cfs_rq =cfs_rq_of (se);
Lost something in transition...
kernel/sched/fair.c +2 -3
@@ -3835,15 +3835,14 @@ static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, } } -void reweight_task (struct task_struct *p,int prio) +void reweight_task (struct task_struct *p,unsigned long weight,u32 inv_weight) {struct sched_entity *se = &p->se;struct cfs_rq *cfs_rq =cfs_rq_of (se);struct load_weight *load = &se->load; -unsigned long weight =scale_load (sched_prio_to_weight[prio]);reweight_entity (cfs_rq, se, weight); - load->inv_weight = sched_prio_to_wmult[prio]; + load->inv_weight = inv_weight; }static inline int throttled_hierarchy (struct cfs_rq *cfs_rq);
Tejun Heo
maintainerRe: [PATCH 04/39] sched: Add sched_class->reweight_task()
In reply to Peter ZijlstraHello, Peter.
Quoted 21 lines · Peter Zijlstra
This reminds me, I think we have a bug here...
https://lkml.kernel.org/r/[email protected]
I *think* we want something like the below, hmm?
kernel/sched/core.c +2
-3@@ -1328,15 +1328,15 @@ int tg_nop(struct task_group *tg, void *data)void set_load_weight (struct task_struct *p,bool update_load) {int prio = p->static_prio -MAX_RT_PRIO ; -struct load_weight *load = &p->se.load; +unsigned long weight; +u32 inv_weight; -/* -* SCHED_IDLE tasks get minimal weight:
kernel/sched/core.c int tg_nop(struct task_group *tg, void *data) · line 1339
Quoted 10 lines
kernel/sched/core.c +5
-4-*/ if (task_has_idle_policy (p)) { - load->weight =scale_load (WEIGHT_IDLEPRIO ); - load->inv_weight =WMULT_IDLEPRIO ; -return ; + weight =scale_load (WEIGHT_IDLEPRIO ); + inv_weight =WMULT_IDLEPRIO ; + }else { + weight =scale_load (sched_prio_to_weight[prio]); + inv_weight = sched_prio_to_wmult[prio];
Hmmm... sorry but I'm a bit confused again. Isn't the code doing the same thing before and after?
Before, if @p is SCHED_IDLE, @p->se.load is set to idle values and the function returns. If @update_load && fair, calls reweight_task(). Otherwise, update @p->se.load is updated according to @prio. After the patch, @weight and @inv_weight calcuations are moved to set_load_weight() but it ends up with the same result for all three cases.
Were you trying to say that if the idle policy were to implement ->reweight_task(), it wouldn't be called?
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH 04/39] sched: Add sched_class->reweight_task()
Quoted 2 lines · Tejun Heo
Were you trying to say that if the idle policy were to implement ->reweight_task(), it wouldn't be called?
This. IDLE really is FAIR but with a really small weight.
Tejun Heo
maintainerRe: [PATCH 04/39] sched: Add sched_class->reweight_task()
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Tejun Heo
Were you trying to say that if the idle policy were to implement ->reweight_task(), it wouldn't be called?
This. IDLE really is FAIR but with a really small weight.
Oh, right, I got confused (again) by the idle class and SCHED_IDLE. Will fix.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 1/1 sched/fair: set_load_weight() must also call reweight_task() for SCHED_IDLE tasks
When a task's weight is being changed, set_load_weight() is called with @update_load set. As weight changes aren't trivial for the fair class, set_load_weight() calls fair.c::reweight_task() for fair class tasks.
However, set_load_weight() first tests task_has_idle_policy() on entry and skips calling reweight_task() for SCHED_IDLE tasks. This is buggy as SCHED_IDLE tasks are just fair tasks with a very low weight and they would incorrectly skip load, vlag and position updates.
Fix it by updating reweight_task() to take struct load_weight as idle weight can't be expressed with prio and making set_load_weight() call reweight_task() for SCHED_IDLE tasks too when @update_load is set.
---
Hello,
Peter, I took the liberty to use struct load_weight instead of open coding weight and inv_weight. As it's an obviously user visible bug, I tagged it with the stable tag but given how long it's been broken, I'm not sure how urgent it is, so please use your discretion. I verified that the reweight_task() is called with the right weights on SCHED_IDLE switches.
Thanks.
kernel/sched/core.c | 23 ++++++++++------------- kernel/sched/fair.c | 7 +++---- kernel/sched/sched.h | 2 +- 3 files changed, 14 insertions(+), 18 deletions(-)
kernel/sched/core.c +10 -13
@@ -1328,27 +1328,24 @@ int tg_nop(struct task_group *tg, void *void set_load_weight (struct task_struct *p,bool update_load) {int prio = p->static_prio -MAX_RT_PRIO ; -struct load_weight *load = &p->se.load; +struct load_weight lw; -/* -* SCHED_IDLE tasks get minimal weight: -*/ if (task_has_idle_policy (p)) { - load->weight =scale_load (WEIGHT_IDLEPRIO ); - load->inv_weight =WMULT_IDLEPRIO ; -return ; + lw.weight =scale_load (WEIGHT_IDLEPRIO ); + lw.inv_weight =WMULT_IDLEPRIO ; + }else { + lw.weight =scale_load (sched_prio_to_weight[prio]); + lw.inv_weight = sched_prio_to_wmult[prio]; }/* * SCHED_OTHER tasks have to update their load when changing their * weight */ -if (update_load && p->sched_class == &fair_sched_class) { -reweight_task (p, prio); - }else { - load->weight =scale_load (sched_prio_to_weight[prio]); - load->inv_weight = sched_prio_to_wmult[prio]; - } +if (update_load && p->sched_class == &fair_sched_class) +reweight_task (p, &lw); +else + p->se.load = lw; }#ifdef CONFIG_UCLAMP_TASK
kernel/sched/fair.c +3 -4
@@ -3835,15 +3835,14 @@ static void reweight_entity(struct cfs_r } } -void reweight_task (struct task_struct *p,int prio) +void reweight_task (struct task_struct *p,const struct load_weight *lw) {struct sched_entity *se = &p->se;struct cfs_rq *cfs_rq =cfs_rq_of (se);struct load_weight *load = &se->load; -unsigned long weight =scale_load (sched_prio_to_weight[prio]); -reweight_entity (cfs_rq, se, weight); - load->inv_weight = sched_prio_to_wmult[prio]; +reweight_entity (cfs_rq, se, lw->weight); + load->inv_weight = lw->inv_weight; }static inline int throttled_hierarchy (struct cfs_rq *cfs_rq);
kernel/sched/sched.h +1 -1
@@ -2509,7 +2509,7 @@ extern void init_sched_dl_class(void);extern void init_sched_rt_class (void );extern void init_sched_fair_class (void ); -extern void reweight_task (struct task_struct *p,int prio); +extern void reweight_task (struct task_struct *p,const struct load_weight *lw);extern void resched_curr (struct rq *rq);extern void resched_cpu (int cpu);
Tejun Heo
maintainerPATCH 1/1 sched_ext: Account for idle policy when setting p->scx.weight in scx_ops_enable_task()
In reply to Tejun HeoWhen initializing p->scx.weight, scx_ops_enable_task() wasn't considering whether the task is SCHED_IDLE. Update it to use WEIGHT_IDLEPRIO as the source weight for SCHED_IDLE tasks. This leaves reweight_task_scx() the sole user of set_task_scx_weight(). Open code it. @weight is going to be provided by sched core in the future anyway.
---
This is a related fix for sched_ext.
Thanks.
kernel/sched/ext.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-)
kernel/sched/ext.c +12 -9
@@ -3237,22 +3237,23 @@ static int scx_ops_init_task(struct taskreturn 0 ; } -static void set_task_scx_weight (struct task_struct *p) -{ -u32 weight = sched_prio_to_weight[p->static_prio -MAX_RT_PRIO ]; - - p->scx.weight =sched_weight_to_cgroup (weight); -} -static void scx_ops_enable_task (struct task_struct *p) { +u32 weight; +lockdep_assert_rq_held (task_rq (p));/* * Set the weight before calling ops.enable() so that the scheduler * doesn't see a stale value if they inspect the task struct. */ -set_task_scx_weight (p); +if (task_has_idle_policy (p)) + weight =WEIGHT_IDLEPRIO ; +else + weight = sched_prio_to_weight[p->static_prio -MAX_RT_PRIO ]; + + p->scx.weight =sched_weight_to_cgroup (weight); +if (SCX_HAS_OP (enable))SCX_CALL_OP_TASK (SCX_KF_REST , enable, p);scx_set_task_state (p,SCX_TASK_ENABLED );
@@ -3405,9 +3406,11 @@ void sched_ext_free(struct task_struct *static void reweight_task_scx (struct rq *rq,struct task_struct *p,int newprio) { +unsigned long weight = sched_prio_to_weight[newprio]; +lockdep_assert_rq_held (task_rq (p)); -set_task_scx_weight (p); + p->scx.weight =sched_weight_to_cgroup (weight);if (SCX_HAS_OP (set_weight))SCX_CALL_OP_TASK (SCX_KF_REST , set_weight, p, p->scx.weight); }
Tejun Heo
maintainerPATCH v2 1/1 sched_ext: Account for idle policy when setting p->scx.weight in scx_ops_enable_task()
From e98abd22fbcada509f776229af688b7f74d6cdba Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Mon, 8 Jul 2024 09:19:14 -1000
Subject: [PATCH] sched_ext: Account for idle policy when setting p->scx.weight
in scx_ops_enable_task()
When initializing p->scx.weight, scx_ops_enable_task() wasn't considering whether the task is SCHED_IDLE. Update it to use WEIGHT_IDLEPRIO as the source weight for SCHED_IDLE tasks. This leaves reweight_task_scx() the sole user of set_task_scx_weight(). Open code it. @weight is going to be provided by sched core in the future anyway.
v2: Use the newly available @lw->weight to set @p->scx.weight in
reweight_task_scx().
---
A slight adjustment in reweight_task_scx() to match the new load_weight argument. Applied to sched_ext/for-6.11.
Thanks.
kernel/sched/ext.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)
kernel/sched/ext.c +10 -9
@@ -3237,22 +3237,23 @@ static int scx_ops_init_task(struct task_struct *p, struct task_group *tg, boolreturn 0 ; } -static void set_task_scx_weight (struct task_struct *p) -{ -u32 weight = sched_prio_to_weight[p->static_prio -MAX_RT_PRIO ]; - - p->scx.weight =sched_weight_to_cgroup (weight); -} -static void scx_ops_enable_task (struct task_struct *p) { +u32 weight; +lockdep_assert_rq_held (task_rq (p));/* * Set the weight before calling ops.enable() so that the scheduler * doesn't see a stale value if they inspect the task struct. */ -set_task_scx_weight (p); +if (task_has_idle_policy (p)) + weight =WEIGHT_IDLEPRIO ; +else + weight = sched_prio_to_weight[p->static_prio -MAX_RT_PRIO ]; + + p->scx.weight =sched_weight_to_cgroup (weight); +if (SCX_HAS_OP (enable))SCX_CALL_OP_TASK (SCX_KF_REST , enable, p);scx_set_task_state (p,SCX_TASK_ENABLED );
@@ -3408,7 +3409,7 @@ static void reweight_task_scx(struct rq *rq, struct task_struct *p, {lockdep_assert_rq_held (task_rq (p)); -set_task_scx_weight (p); + p->scx.weight =sched_weight_to_cgroup (scale_load_down (lw->weight));if (SCX_HAS_OP (set_weight))SCX_CALL_OP_TASK (SCX_KF_REST , set_weight, p, p->scx.weight); }
Signature · 1 line
2.45.2
[tip: sched/core] sched/fair: set_load_weight() must also call reweight_task() for SCHED_IDLE tasks
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: d329605287020c3d1c3b0dadc63d8208e7251382 Gitweb: https://git.kernel.org/tip/d329605287020c3d1c3b0dadc63d8208e7251382 Author: Tejun Heo <[email protected]> AuthorDate: Tue, 25 Jun 2024 15:29:58 -10:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 04 Jul 2024 15:59:52 +02:00
sched/fair: set_load_weight() must also call reweight_task() for SCHED_IDLE tasks
When a task's weight is being changed, set_load_weight() is called with @update_load set. As weight changes aren't trivial for the fair class, set_load_weight() calls fair.c::reweight_task() for fair class tasks.
However, set_load_weight() first tests task_has_idle_policy() on entry and skips calling reweight_task() for SCHED_IDLE tasks. This is buggy as SCHED_IDLE tasks are just fair tasks with a very low weight and they would incorrectly skip load, vlag and position updates.
Fix it by updating reweight_task() to take struct load_weight as idle weight can't be expressed with prio and making set_load_weight() call reweight_task() for SCHED_IDLE tasks too when @update_load is set.
--- kernel/sched/core.c | 23 ++++++++++------------- kernel/sched/fair.c | 7 +++---- kernel/sched/sched.h | 2 +- 3 files changed, 14 insertions(+), 18 deletions(-)
kernel/sched/core.c +10 -13
@@ -1328,27 +1328,24 @@ int tg_nop(struct task_group *tg, void *data)void set_load_weight (struct task_struct *p,bool update_load) {int prio = p->static_prio -MAX_RT_PRIO ; -struct load_weight *load = &p->se.load; +struct load_weight lw; -/* -* SCHED_IDLE tasks get minimal weight: -*/ if (task_has_idle_policy (p)) { - load->weight =scale_load (WEIGHT_IDLEPRIO ); - load->inv_weight =WMULT_IDLEPRIO ; -return ; + lw.weight =scale_load (WEIGHT_IDLEPRIO ); + lw.inv_weight =WMULT_IDLEPRIO ; + }else { + lw.weight =scale_load (sched_prio_to_weight[prio]); + lw.inv_weight = sched_prio_to_wmult[prio]; }/* * SCHED_OTHER tasks have to update their load when changing their * weight */ -if (update_load && p->sched_class == &fair_sched_class) { -reweight_task (p, prio); - }else { - load->weight =scale_load (sched_prio_to_weight[prio]); - load->inv_weight = sched_prio_to_wmult[prio]; - } +if (update_load && p->sched_class == &fair_sched_class) +reweight_task (p, &lw); +else + p->se.load = lw; }#ifdef CONFIG_UCLAMP_TASK
kernel/sched/fair.c +3 -4
@@ -3835,15 +3835,14 @@ static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, } } -void reweight_task (struct task_struct *p,int prio) +void reweight_task (struct task_struct *p,const struct load_weight *lw) {struct sched_entity *se = &p->se;struct cfs_rq *cfs_rq =cfs_rq_of (se);struct load_weight *load = &se->load; -unsigned long weight =scale_load (sched_prio_to_weight[prio]); -reweight_entity (cfs_rq, se, weight); - load->inv_weight = sched_prio_to_wmult[prio]; +reweight_entity (cfs_rq, se, lw->weight); + load->inv_weight = lw->inv_weight; }static inline int throttled_hierarchy (struct cfs_rq *cfs_rq);
kernel/sched/sched.h +1 -1
@@ -2509,7 +2509,7 @@ extern void init_sched_dl_class(void);extern void init_sched_rt_class (void );extern void init_sched_fair_class (void ); -extern void reweight_task (struct task_struct *p,int prio); +extern void reweight_task (struct task_struct *p,const struct load_weight *lw);extern void resched_curr (struct rq *rq);extern void resched_cpu (int cpu);
Tejun Heo
maintainerPATCH 5/39 sched: Add sched_class->switching_to() and expose check_class_changing/changed()
In reply to Tejun HeoWhen a task switches to a new sched_class, the prev and new classes are notified through ->switched_from() and ->switched_to(), respectively, after the switching is done.
A new BPF extensible sched_class will have callbacks that allow the BPF scheduler to keep track of relevant task states (like priority and cpumask). Those callbacks aren't called while a task is on a different sched_class. When a task comes back, we wanna tell the BPF progs the up-to-date state before the task gets enqueued, so we need a hook which is called before the switching is committed.
This patch adds ->switching_to() which is called during sched_class switch through check_class_changing() before the task is restored. Also, this patch exposes check_class_changing/changed() in kernel/sched/sched.h. They will be used by the new BPF extensible sched_class to implement implicit sched_class switching which is used e.g. when falling back to CFS when the BPF scheduler fails or unloads.
This is a prep patch and doesn't cause any behavior changes. The new operation and exposed functions aren't used yet.
v2: Improve patch description w/ details on planned use.
--- kernel/sched/core.c | 19 ++++++++++++++++--- kernel/sched/sched.h | 7 +++++++ 2 files changed, 23 insertions(+), 3 deletions(-)
kernel/sched/core.c +16 -3
@@ -2214,6 +2214,17 @@ inline int task_curr(const struct task_struct *p)return cpu_curr (task_cpu (p)) == p; } +/* +* ->switching_to() is called with the pi_lock and rq_lock held and must not +* mess with locking. +*/ +void check_class_changing (struct rq *rq,struct task_struct *p, +const struct sched_class *prev_class) +{ +if (prev_class != p->sched_class && p->sched_class->switching_to) + p->sched_class->switching_to (rq, p); +} +/* * switched_from, switched_to and prio_changed must _NOT_ drop rq->lock, * use the balance_callback list if you want balancing.
@@ -2221,9 +2232,9 @@ inline int task_curr(const struct task_struct *p)* this means any call to check_class_changed() must be followed by a call to * balance_callback(). */ -static inline void check_class_changed (struct rq *rq,struct task_struct *p, -const struct sched_class *prev_class, -int oldprio) +void check_class_changed (struct rq *rq,struct task_struct *p, +const struct sched_class *prev_class, +int oldprio) {if (prev_class != p->sched_class) {if (prev_class->switched_from)
@@ -7253,6 +7264,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task) }__setscheduler_prio (p, prio); +check_class_changing (rq, p, prev_class);if (queued)enqueue_task (rq, p, queue_flag);
@@ -7898,6 +7910,7 @@ static int __sched_setscheduler(struct task_struct *p,__setscheduler_prio (p, newprio); }__setscheduler_uclamp (p, attr); +check_class_changing (rq, p, prev_class);if (queued) {/*
kernel/sched/sched.h +7 -0
@@ -2301,6 +2301,7 @@ struct sched_class {* cannot assume the switched_from/switched_to pair is serialized by * rq->lock. They are however serialized by p->pi_lock. */ +void (*switching_to) (struct rq *this_rq,struct task_struct *task);void (*switched_from)(struct rq *this_rq,struct task_struct *task);void (*switched_to) (struct rq *this_rq,struct task_struct *task); void (*reweight_task)(struct rq *this_rq, struct task_struct *task,
@@ -2540,6 +2541,12 @@ static inline void sub_nr_running(struct rq *rq, unsigned count)extern void activate_task (struct rq *rq,struct task_struct *p,int flags);extern void deactivate_task (struct rq *rq,struct task_struct *p,int flags); +extern void check_class_changing (struct rq *rq,struct task_struct *p, +const struct sched_class *prev_class); +extern void check_class_changed (struct rq *rq,struct task_struct *p, +const struct sched_class *prev_class, +int oldprio); +extern void wakeup_preempt (struct rq *rq,struct task_struct *p,int flags);#ifdef CONFIG_PREEMPT_RT
Signature · 1 line
2.44.0
Peter Zijlstra
maintainerRe: [PATCH 05/39] sched: Add sched_class->switching_to() and expose check_class_changing/changed()
Quoted 34 lines · Tejun Heo
When a task switches to a new
sched_class, the prev and new classes are notified through ->switched_from() and ->switched_to(), respectively, after the switching is done.A new BPF extensible
sched_classwill have callbacks that allow the BPF scheduler to keep track of relevant task states (like priority and cpumask). Those callbacks aren't called while a task is on a differentsched_class. When a task comes back, we wanna tell the BPF progs the up-to-date state before the task gets enqueued, so we need a hook which is called before the switching is committed.This patch adds ->switching_to() which is called during
sched_classswitch throughcheck_class_changing()before the task is restored. Also, this patch exposes check_class_changing/changed() inkernel/sched/sched.h. They will be used by the new BPF extensiblesched_classto implement implicitsched_classswitching which is used e.g. when falling back to CFS when the BPF scheduler fails or unloads.This is a prep patch and doesn't cause any behavior changes. The new operation and exposed functions aren't used yet.
v2: Improve patch description w/ details on planned use.
Signed-off-by Tejun Heo <[email protected]>Reviewed-by David Vernet <[email protected]>Acked-by Josh Don <[email protected]>Acked-by Hao Luo <[email protected]>Acked-by Barret Rhoden <[email protected]>kernel/sched/sched.h +0
-0
kernel/sched/sched.h struct sched_class { · line 2307
Quoted 7 lines
kernel/sched/sched.h +1
-0@@ -2301,6 +2301,7 @@ struct sched_class {* cannot assume the switched_from/switched_to pair is serialized by * rq->lock. They are however serialized by p->pi_lock. */ +void (*switching_to) (struct rq *this_rq,struct task_struct *task);void (*switched_from)(struct rq *this_rq,struct task_struct *task);void (*switched_to) (struct rq *this_rq,struct task_struct *task); void (*reweight_task)(struct rq *this_rq, struct task_struct *task,
So I *think* that I can handle all the current cases in sched_class::{en,de}queue_task() if we add {EN,DE}QUEUE_CLASS flags.
Would that work for the BPF thing as well?
Something like the very much incomplete below... It would allow removing all these switch{ed,ing}_{to,from}() things entirely, instead of adding yet more.
---
kernel/sched/core.c +17 -5
@@ -6864,15 +6864,22 @@ int default_wake_function(wait_queue_entry_t *curr, unsigned mode, int wake_flag }EXPORT_SYMBOL (default_wake_function); -void __setscheduler_prio (struct task_struct *p,int prio) +struct sched_class *__setscheduler_class (int prio) { +struct sched_class *class; +if (dl_prio (prio)) - p->sched_class = &dl_sched_class; + class = &dl_sched_class;else if (rt_prio (prio)) - p->sched_class = &rt_sched_class; + class = &rt_sched_class;else - p->sched_class = &fair_sched_class; + class = &fair_sched_class; +return class; +} + +void __setscheduler_prio (struct task_struct *p,int prio) +{ p->prio = prio; }
@@ -6919,7 +6926,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task) {int prio, oldprio, queued, running, queue_flag =DEQUEUE_SAVE |DEQUEUE_MOVE |DEQUEUE_NOCLOCK ; -const struct sched_class *prev_class; +const struct sched_class *prev_class, *class;struct rq_flags rf;struct rq *rq;
@@ -6977,6 +6984,10 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task) queue_flag &= ~DEQUEUE_MOVE ; prev_class = p->sched_class; + class =__setscheduler_class (prio); +if (prev_class != class) + queue_flags |=DEQUEUE_CLASS ; + queued =task_on_rq_queued (p); running =task_current (rq, p);if (queued)
@@ -7014,6 +7025,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task) p->rt.timeout =0 ; } + p->class = class;__setscheduler_prio (p, prio);if (queued)
kernel/sched/sched.h +3 -0
@@ -2251,6 +2251,7 @@ extern const u32 sched_prio_to_wmult[40];#define DEQUEUE_MOVE 0x04/* Matches ENQUEUE_MOVE */ #define DEQUEUE_NOCLOCK 0x08/* Matches ENQUEUE_NOCLOCK */ #define DEQUEUE_MIGRATING 0x100/* Matches ENQUEUE_MIGRATING */ +#define DEQUEUE_CLASS 0x200/* Matches ENQUEUE_CLASS */ #define ENQUEUE_WAKEUP 0x01#define ENQUEUE_RESTORE 0x02
@@ -2266,6 +2267,7 @@ extern const u32 sched_prio_to_wmult[40];#endif #define ENQUEUE_INITIAL 0x80#define ENQUEUE_MIGRATING 0x100 +#define ENQUEUE_CLASS 0x200#define RETRY_TASK ((void *)-1UL)
@@ -3603,6 +3605,7 @@ static inline int rt_effective_prio(struct task_struct *p, int prio)extern int __sched_setscheduler (struct task_struct *p,const struct sched_attr *attr,bool user,bool pi);extern int __sched_setaffinity (struct task_struct *p,struct affinity_context *ctx); +extern struct sched_class *__setscheduler_class (int prio);extern void __setscheduler_prio (struct task_struct *p,int prio);extern void set_load_weight (struct task_struct *p,bool update_load);extern void enqueue_task (struct rq *rq,struct task_struct *p,int flags);
kernel/sched/syscalls.c +8 -3
@@ -612,7 +612,7 @@ int __sched_setscheduler(struct task_struct *p, {int oldpolicy =-1 , policy = attr->sched_policy;int retval, oldprio, newprio, queued, running; -const struct sched_class *prev_class; +const struct sched_class *prev_class, *class;struct balance_callback *head;struct rq_flags rf;int reset_on_fork;
@@ -783,6 +783,12 @@ int __sched_setscheduler(struct task_struct *p, queue_flags &= ~DEQUEUE_MOVE ; } + class = prev_class = p->sched_class; +if (!(attr->sched_flags &SCHED_FLAG_KEEP_PARAMS )) + class =__setscheduler_class (newprio); +if (prev_class != class) + queue_flags |=DEQUEUE_CLASS ; + queued =task_on_rq_queued (p); running =task_current (rq, p);if (queued)
@@ -790,10 +796,9 @@ int __sched_setscheduler(struct task_struct *p,if (running)put_prev_task (rq, p); - prev_class = p->sched_class; -if (!(attr->sched_flags &SCHED_FLAG_KEEP_PARAMS )) {__setscheduler_params (p, attr); + p->class = class;__setscheduler_prio (p, newprio); }__setscheduler_uclamp (p, attr);
Tejun Heo
maintainerRe: [PATCH 05/39] sched: Add sched_class->switching_to() and expose check_class_changing/changed()
Hello, Peter.
On Mon, Jun 24, 2024 at 01:06:24PM +0200, Peter Zijlstra wrote:
...
Quoted 7 lines
Review
Quoted 3 lines
(quoted diff) +1
-0+ void (*switching_to) (struct rq *this_rq, struct task_struct *task); void (*switched_from)(struct rq *this_rq, struct task_struct *task); void (*switched_to) (struct rq *this_rq, struct task_struct *task);So I *think* that I can handle all the current cases in sched_class::{en,de}queue_task() if we add {EN,DE}QUEUE_CLASS flags.
Quoted 5 lines
Would that work for the BPF thing as well?
Something like the very much incomplete below... It would allow removing all these switch{ed,ing}_{to,from}() things entirely, instead of adding yet more.
Hmm... so, I tried to make it work for SCX but enqueue() and dequeue() are only called if the task was queued at the time of sched_class change, right? However, these callbacks expect to be called even when the task is not currently queued. Maybe I'm misreading code but it looks like that'd break other classes too. What am I missing?
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH 05/39] sched: Add sched_class->switching_to() and expose check_class_changing/changed()
Quoted 17 lines · Tejun Heo
Hello, Peter.
On Mon, Jun 24, 2024 at 01:06:24PM +0200, Peter Zijlstra wrote:
...Quoted 7 lines
Review
Quoted 3 lines
(quoted diff) +1
-0+ void (*switching_to) (struct rq *this_rq, struct task_struct *task); void (*switched_from)(struct rq *this_rq, struct task_struct *task); void (*switched_to) (struct rq *this_rq, struct task_struct *task);So I *think* that I can handle all the current cases in sched_class::{en,de}queue_task() if we add {EN,DE}QUEUE_CLASS flags.
Quoted 5 lines
Would that work for the BPF thing as well?
Something like the very much incomplete below... It would allow removing all these switch{ed,ing}_{to,from}() things entirely, instead of adding yet more.
Quoted 5 lines
Hmm... so, I tried to make it work for SCX but
enqueue()anddequeue()are only called if the task was queued at the time ofsched_classchange, right? However, these callbacks expect to be called even when the task is not currently queued. Maybe I'm misreading code but it looks like that'd break other classes too. What am I missing?
Ah,.. so I think the RT/DL ones can work (which is what I looked at), they're only concerned with balancing tasks that are on the queue. But yeah, I missed that the fair thing needs it regardless.
Bummer. I was hoping to reduce calls.
Tejun Heo
maintainerPATCH 6/39 sched: Factor out cgroup weight conversion functions
In reply to Tejun HeoFactor out sched_weight_from/to_cgroup() which convert between scheduler shares and cgroup weight. No functional change. The factored out functions will be used by a new BPF extensible sched_class so that the weights can be exposed to the BPF programs in a way which is consistent cgroup weights and easier to interpret.
The weight conversions will be used regardless of cgroup usage. It's just borrowing the cgroup weight range as it's more intuitive. CGROUP_WEIGHT_MIN/DFL/MAX constants are moved outside CONFIG_CGROUPS so that the conversion helpers can always be defined.
v2: The helpers are now defined regardless of COFNIG_CGROUPS.
--- include/linux/cgroup.h | 4 ++-- kernel/sched/core.c | 28 +++++++++++++--------------- kernel/sched/sched.h | 18 ++++++++++++++++++ 3 files changed, 33 insertions(+), 17 deletions(-)
include/linux/cgroup.h +2 -2
@@ -29,8 +29,6 @@struct kernel_clone_args ; -#ifdef CONFIG_CGROUPS -/* * All weight knobs on the default hierarchy should use the following min, * default and max values. The default value is the logarithmic center of
@@ -40,6 +38,8 @@ struct kernel_clone_args;#define CGROUP_WEIGHT_DFL 100#define CGROUP_WEIGHT_MAX 10000 +#ifdef CONFIG_CGROUPS +enum {CSS_TASK_ITER_PROCS = (1U <<0 ),/* walk only threadgroup leaders */ CSS_TASK_ITER_THREADED = (1U << 1), /* walk all threaded css_sets in the domain */
kernel/sched/core.c +13 -15
@@ -11313,29 +11313,27 @@ static int cpu_local_stat_show(struct seq_file *sf, }#ifdef CONFIG_FAIR_GROUP_SCHED + +static unsigned long tg_weight (struct task_group *tg) +{ +return scale_load_down (tg->shares); +} +static u64 cpu_weight_read_u64 (struct cgroup_subsys_state *css,struct cftype *cft) { -struct task_group *tg =css_tg (css); -u64 weight =scale_load_down (tg->shares); - -return DIV_ROUND_CLOSEST_ULL (weight *CGROUP_WEIGHT_DFL ,1024 ); +return sched_weight_to_cgroup (tg_weight (css_tg (css))); }static int cpu_weight_write_u64 (struct cgroup_subsys_state *css, -struct cftype *cft,u64 weight) +struct cftype *cft,u64 cgrp_weight) { -/* -* cgroup weight knobs should use the common MIN, DFL and MAX -* values which are 1, 100 and 10000 respectively. While it loses -* a bit of range on both ends, it maps pretty well onto the shares -* value used by scheduler and the round-trip conversions preserve -* the original value over the entire range. -*/ -if (weight <CGROUP_WEIGHT_MIN || weight >CGROUP_WEIGHT_MAX ) +unsigned long weight; + +if (cgrp_weight <CGROUP_WEIGHT_MIN || cgrp_weight >CGROUP_WEIGHT_MAX )return -ERANGE ; - weight =DIV_ROUND_CLOSEST_ULL (weight *1024 ,CGROUP_WEIGHT_DFL ); + weight =sched_weight_from_cgroup (cgrp_weight);return sched_group_set_shares (css_tg (css),scale_load (weight)); }
@@ -11343,7 +11341,7 @@ static int cpu_weight_write_u64(struct cgroup_subsys_state *css,static s64 cpu_weight_nice_read_s64 (struct cgroup_subsys_state *css,struct cftype *cft) { -unsigned long weight =scale_load_down (css_tg (css)->shares); +unsigned long weight =tg_weight (css_tg (css));int last_delta =INT_MAX ;int prio, delta;
kernel/sched/sched.h +18 -0
@@ -221,6 +221,24 @@ static inline void update_avg(u64 *avg, u64 sample)#define shr_bound (val, shift) \ (val >> min_t(typeof(shift), shift, BITS_PER_TYPE(typeof(val)) - 1)) +/* +* cgroup weight knobs should use the common MIN, DFL and MAX values which are +* 1, 100 and 10000 respectively. While it loses a bit of range on both ends, it +* maps pretty well onto the shares value used by scheduler and the round-trip +* conversions preserve the original value over the entire range. +*/ +static inline unsigned long sched_weight_from_cgroup (unsigned long cgrp_weight) +{ +return DIV_ROUND_CLOSEST_ULL (cgrp_weight *1024 ,CGROUP_WEIGHT_DFL ); +} + +static inline unsigned long sched_weight_to_cgroup (unsigned long weight) +{ + return clamp_t(unsigned long, +DIV_ROUND_CLOSEST_ULL (weight *CGROUP_WEIGHT_DFL ,1024 ), +CGROUP_WEIGHT_MIN ,CGROUP_WEIGHT_MAX ); +} +/* * !! For sched_setattr_nocheck() (kernel) only !! *
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 7/39 sched: Expose css_tg() and __setscheduler_prio()
In reply to Tejun HeoThese will be used by a new BPF extensible sched_class.
css_tg() will be used in the init and exit paths to visit all task_groups by walking cgroups.
__setscheduler_prio() is used to pick the sched_class matching the current prio of the task. For the new BPF extensible sched_class, the mapping from the task configuration to sched_class isn't static and depends on a few factors - e.g. whether the BPF progs implementing the scheduler are loaded and in a serviceable state. That mapping logic will be added to __setscheduler_prio().
When the BPF scheduler progs get loaded and unloaded, the mapping changes and the new sched_class will walk the tasks applying the new mapping using __setscheduler_prio().
v3: Dropped SCHED_CHANGE_BLOCK() as upstream is adding more generic cleanup
mechanism.
v2: Expose SCHED_CHANGE_BLOCK() too and update the description.
--- kernel/sched/core.c | 7 +------ kernel/sched/sched.h | 7 +++++++ 2 files changed, 8 insertions(+), 6 deletions(-)
kernel/sched/core.c +1 -6
@@ -7098,7 +7098,7 @@ int default_wake_function(wait_queue_entry_t *curr, unsigned mode, int wake_flag }EXPORT_SYMBOL (default_wake_function); -static void __setscheduler_prio (struct task_struct *p,int prio) +void __setscheduler_prio (struct task_struct *p,int prio) {if (dl_prio (prio)) p->sched_class = &dl_sched_class;
@@ -10542,11 +10542,6 @@ void sched_move_task(struct task_struct *tsk) } } -static inline struct task_group *css_tg (struct cgroup_subsys_state *css) -{ - return css ? container_of(css, struct task_group, css) : NULL; -} -static struct cgroup_subsys_state *cpu_cgroup_css_alloc (struct cgroup_subsys_state *parent_css) {
kernel/sched/sched.h +7 -0
@@ -477,6 +477,11 @@ static inline int walk_tg_tree(tg_visitor down, tg_visitor up, void *data)return walk_tg_tree_from (&root_task_group, down, up, data); } +static inline struct task_group *css_tg (struct cgroup_subsys_state *css) +{ + return css ? container_of(css, struct task_group, css) : NULL; +} +extern int tg_nop (struct task_group *tg,void *data);#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -2481,6 +2486,8 @@ extern void init_sched_dl_class(void);extern void init_sched_rt_class (void );extern void init_sched_fair_class (void ); +extern void __setscheduler_prio (struct task_struct *p,int prio); +extern void resched_curr (struct rq *rq);extern void resched_cpu (int cpu);
Signature · 1 line
2.44.0
Peter Zijlstra
maintainerRe: [PATCH 07/39] sched: Expose css_tg() and __setscheduler_prio()
Quoted 37 lines · Tejun Heo
These will be used by a new BPF extensible
sched_class.
css_tg()will be used in the init and exit paths to visit alltask_groupsby walking cgroups.
__setscheduler_prio()is used to pick thesched_classmatching the current prio of the task. For the new BPF extensiblesched_class, the mapping from the task configuration tosched_classisn't static and depends on a few factors - e.g. whether the BPF progs implementing the scheduler are loaded and in a serviceable state. That mapping logic will be added to__setscheduler_prio().When the BPF scheduler progs get loaded and unloaded, the mapping changes and the new
sched_classwill walk the tasks applying the new mapping using__setscheduler_prio().v3: Dropped
SCHED_CHANGE_BLOCK()as upstream is adding more generic cleanup
mechanism.v2: Expose
SCHED_CHANGE_BLOCK()too and update the description.Signed-off-by Tejun Heo <[email protected]>Reviewed-by David Vernet <[email protected]>Acked-by Josh Don <[email protected]>Acked-by Hao Luo <[email protected]>Acked-by Barret Rhoden <[email protected]>Reported-by kernel test robot <[email protected]>--- kernel/sched/core.c | 7 +------ kernel/sched/sched.h | 7 +++++++ 2 files changed, 8 insertions(+), 6 deletions(-)kernel/sched/core.c +0
-0
kernel/sched/core.c int default_wake_function(wait_queue_entry_t *curr, unsigned mode, int wake_flag · line 7104
Quoted 8 lines
kernel/sched/core.c +1
-1@@ -7098,7 +7098,7 @@ int default_wake_function(wait_queue_entry_t *curr, unsigned mode, int wake_flag }EXPORT_SYMBOL (default_wake_function); -static void __setscheduler_prio (struct task_struct *p,int prio) +void __setscheduler_prio (struct task_struct *p,int prio) {if (dl_prio (prio)) p->sched_class = &dl_sched_class;
FWIW this conflicts with patches in tip/sched/core, and did so at the time of posting.
Tejun Heo
maintainerRe: [PATCH 07/39] sched: Expose css_tg() and __setscheduler_prio()
Hello,
Quoted 8 lines · Peter Zijlstra
kernel/sched/core.c int default_wake_function(wait_queue_entry_t *curr, unsigned mode, int wake_flag · line 7104
Quoted 5 lines
kernel/sched/core.c +1
-1-static void __setscheduler_prio (struct task_struct *p,int prio) +void __setscheduler_prio (struct task_struct *p,int prio) {if (dl_prio (prio)) p->sched_class = &dl_sched_class;FWIW this conflicts with patches in tip/sched/core, and did so at the time of posting.
Oh yeah, as I noted in another reply, v7 patchset is rebased on top of the current tip/sched/core and already in linux-next. These are all fixed up already.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 8/39 sched: Enumerate CPU cgroup file types
In reply to Tejun HeoRename cpu[_legacy]_files to cpu[_legacy]_cftypes for clarity and add cpu_cftype_id which enumerates every cgroup2 interface file type. This doesn't make any functional difference now. The enums will be used to access specific cftypes by a new BPF extensible sched_class to selectively show and hide CPU controller interface files depending on the capability of the currently loaded BPF scheduler progs.
--- kernel/sched/core.c | 22 +++++++++++----------- kernel/sched/sched.h | 21 +++++++++++++++++++++ 2 files changed, 32 insertions(+), 11 deletions(-)
kernel/sched/core.c +11 -11
@@ -11196,7 +11196,7 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css, }#endif -static struct cftype cpu_legacy_files[] = { +static struct cftype cpu_legacy_cftypes[] = {#ifdef CONFIG_FAIR_GROUP_SCHED { .name = "shares",
@@ -11425,21 +11425,21 @@ static ssize_t cpu_max_write(struct kernfs_open_file *of, }#endif -static struct cftype cpu_files[] = { +struct cftype cpu_cftypes[CPU_CFTYPE_CNT +1 ] = { #ifdef CONFIG_FAIR_GROUP_SCHED - { + [CPU_CFTYPE_WEIGHT] = { .name ="weight" , .flags =CFTYPE_NOT_ON_ROOT , .read_u64 = cpu_weight_read_u64, .write_u64 = cpu_weight_write_u64, }, - { + [CPU_CFTYPE_WEIGHT_NICE ] = { .name ="weight.nice" , .flags =CFTYPE_NOT_ON_ROOT , .read_s64 = cpu_weight_nice_read_s64, .write_s64 = cpu_weight_nice_write_s64, }, - { + [CPU_CFTYPE_IDLE ] = { .name ="idle" , .flags =CFTYPE_NOT_ON_ROOT , .read_s64 = cpu_idle_read_s64,
@@ -11447,13 +11447,13 @@ static struct cftype cpu_files[] = { },#endif #ifdef CONFIG_CFS_BANDWIDTH - { + [CPU_CFTYPE_MAX] = { .name = "max", .flags = CFTYPE_NOT_ON_ROOT, .seq_show = cpu_max_show, .write = cpu_max_write, }, - { + [CPU_CFTYPE_MAX_BURST] = { .name = "max.burst", .flags = CFTYPE_NOT_ON_ROOT, .read_u64 = cpu_cfs_burst_read_u64,
@@ -11461,13 +11461,13 @@ static struct cftype cpu_files[] = { },#endif #ifdef CONFIG_UCLAMP_TASK_GROUP - { + [CPU_CFTYPE_UCLAMP_MIN] = { .name = "uclamp.min", .flags = CFTYPE_NOT_ON_ROOT, .seq_show = cpu_uclamp_min_show, .write = cpu_uclamp_min_write, }, - { + [CPU_CFTYPE_UCLAMP_MAX] = { .name = "uclamp.max", .flags = CFTYPE_NOT_ON_ROOT, .seq_show = cpu_uclamp_max_show,
@@ -11488,8 +11488,8 @@ struct cgroup_subsys cpu_cgrp_subsys = { .can_attach = cpu_cgroup_can_attach, #endif .attach = cpu_cgroup_attach, - .legacy_cftypes = cpu_legacy_files, - .dfl_cftypes = cpu_files, + .legacy_cftypes = cpu_legacy_cftypes, + .dfl_cftypes = cpu_cftypes, .early_init = true, .threaded = true, };
kernel/sched/sched.h +21 -0
@@ -3505,4 +3505,25 @@ static inline void init_sched_mm_cid(struct task_struct *t) { }extern u64 avg_vruntime (struct cfs_rq *cfs_rq);extern int entity_eligible (struct cfs_rq *cfs_rq,struct sched_entity *se); +#ifdef CONFIG_CGROUP_SCHED +enum cpu_cftype_id { +#ifdef CONFIG_FAIR_GROUP_SCHED +CPU_CFTYPE_WEIGHT , +CPU_CFTYPE_WEIGHT_NICE , +CPU_CFTYPE_IDLE , +#endif +#ifdef CONFIG_CFS_BANDWIDTH +CPU_CFTYPE_MAX , +CPU_CFTYPE_MAX_BURST , +#endif +#ifdef CONFIG_UCLAMP_TASK_GROUP +CPU_CFTYPE_UCLAMP_MIN , +CPU_CFTYPE_UCLAMP_MAX , +#endif +CPU_CFTYPE_CNT , +}; + +extern struct cftype cpu_cftypes[CPU_CFTYPE_CNT +1 ]; +#endif /* CONFIG_CGROUP_SCHED */ +#endif /* _KERNEL_SCHED_SCHED_H */
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 9/39 sched: Add @reason to sched_class->rq_{on|off}line()
In reply to Tejun Heo->rq_{on|off}line are called either during CPU hotplug or cpuset partition updates. A planned BPF extensible sched_class wants to tell the BPF scheduler progs about CPU hotplug events in a way that's synchronized with rq state changes.
As the BPF scheduler progs aren't necessarily affected by cpuset partition updates, we need a way to distinguish the two types of events. Let's add an argument to tell them apart.
v2: Patch description updated to detail the expected use.
--- kernel/sched/core.c | 12 ++++++------ kernel/sched/deadline.c | 4 ++-- kernel/sched/fair.c | 4 ++-- kernel/sched/rt.c | 4 ++-- kernel/sched/sched.h | 13 +++++++++---- kernel/sched/topology.c | 4 ++-- 6 files changed, 23 insertions(+), 18 deletions(-)
kernel/sched/core.c +6 -6
@@ -9593,7 +9593,7 @@ static inline void balance_hotplug_wait(void)#endif /* CONFIG_HOTPLUG_CPU */ -void set_rq_online (struct rq *rq) +void set_rq_online (struct rq *rq,enum rq_onoff_reason reason) {if (!rq->online) {const struct sched_class *class;
@@ -9603,12 +9603,12 @@ void set_rq_online(struct rq *rq)for_each_class (class) {if (class->rq_online) - class->rq_online (rq); + class->rq_online (rq, reason); } } } -void set_rq_offline (struct rq *rq) +void set_rq_offline (struct rq *rq,enum rq_onoff_reason reason) {if (rq->online) {const struct sched_class *class;
@@ -9616,7 +9616,7 @@ void set_rq_offline(struct rq *rq)update_rq_clock (rq);for_each_class (class) {if (class->rq_offline) - class->rq_offline (rq); + class->rq_offline (rq, reason); }cpumask_clear_cpu (rq->cpu, rq->rd->online);
@@ -9712,7 +9712,7 @@ int sched_cpu_activate(unsigned int cpu)rq_lock_irqsave (rq, &rf);if (rq->rd) {BUG_ON (!cpumask_test_cpu (cpu, rq->rd->span)); -set_rq_online (rq); +set_rq_online (rq,RQ_ONOFF_HOTPLUG ); }rq_unlock_irqrestore (rq, &rf);
@@ -9756,7 +9756,7 @@ int sched_cpu_deactivate(unsigned int cpu)rq_lock_irqsave (rq, &rf);if (rq->rd) {BUG_ON (!cpumask_test_cpu (cpu, rq->rd->span)); -set_rq_offline (rq); +set_rq_offline (rq,RQ_ONOFF_HOTPLUG ); }rq_unlock_irqrestore (rq, &rf);
kernel/sched/deadline.c +2 -2
@@ -2607,7 +2607,7 @@ static void set_cpus_allowed_dl(struct task_struct *p, }/* Assumes rq->lock is held */ -static void rq_online_dl (struct rq *rq) +static void rq_online_dl (struct rq *rq,enum rq_onoff_reason reason) {if (rq->dl.overloaded)dl_set_overload (rq);
@@ -2618,7 +2618,7 @@ static void rq_online_dl(struct rq *rq) }/* Assumes rq->lock is held */ -static void rq_offline_dl (struct rq *rq) +static void rq_offline_dl (struct rq *rq,enum rq_onoff_reason reason) {if (rq->dl.overloaded)dl_clear_overload (rq);
kernel/sched/fair.c +2 -2
@@ -12446,14 +12446,14 @@ void trigger_load_balance(struct rq *rq)nohz_balancer_kick (rq); } -static void rq_online_fair (struct rq *rq) +static void rq_online_fair (struct rq *rq,enum rq_onoff_reason reason) {update_sysctl ();update_runtime_enabled (rq); } -static void rq_offline_fair (struct rq *rq) +static void rq_offline_fair (struct rq *rq,enum rq_onoff_reason reason) {update_sysctl ();
kernel/sched/rt.c +2 -2
@@ -2426,7 +2426,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) }/* Assumes rq->lock is held */ -static void rq_online_rt (struct rq *rq) +static void rq_online_rt (struct rq *rq,enum rq_onoff_reason reason) {if (rq->rt.overloaded)rt_set_overload (rq);
@@ -2437,7 +2437,7 @@ static void rq_online_rt(struct rq *rq) }/* Assumes rq->lock is held */ -static void rq_offline_rt (struct rq *rq) +static void rq_offline_rt (struct rq *rq,enum rq_onoff_reason reason) {if (rq->rt.overloaded)rt_clear_overload (rq);
kernel/sched/sched.h +9 -4
@@ -2271,6 +2271,11 @@ extern const u32 sched_prio_to_wmult[40];#define RETRY_TASK ((void *)-1UL) +enum rq_onoff_reason { +RQ_ONOFF_HOTPLUG ,/* CPU is going on/offline */ +RQ_ONOFF_TOPOLOGY ,/* sched domain topology update */ +}; +struct affinity_context {const struct cpumask *new_mask;struct cpumask *user_mask;
@@ -2309,8 +2314,8 @@ struct sched_class {void (*set_cpus_allowed)(struct task_struct *p,struct affinity_context *ctx); -void (*rq_online)(struct rq *rq); -void (*rq_offline)(struct rq *rq); +void (*rq_online)(struct rq *rq,enum rq_onoff_reason reason); +void (*rq_offline)(struct rq *rq,enum rq_onoff_reason reason);struct rq *(*find_lock_rq)(struct task_struct *p,struct rq *rq);#endif
@@ -2853,8 +2858,8 @@ static inline void double_rq_unlock(struct rq *rq1, struct rq *rq2) raw_spin_rq_unlock(rq1); } -extern void set_rq_online (struct rq *rq); -extern void set_rq_offline (struct rq *rq); +extern void set_rq_online (struct rq *rq,enum rq_onoff_reason reason); +extern void set_rq_offline (struct rq *rq,enum rq_onoff_reason reason);extern bool sched_smp_initialized;#else /* CONFIG_SMP */
kernel/sched/topology.c +2 -2
@@ -497,7 +497,7 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd) old_rd = rq->rd;if (cpumask_test_cpu (rq->cpu, old_rd->online)) -set_rq_offline (rq); +set_rq_offline (rq,RQ_ONOFF_TOPOLOGY );cpumask_clear_cpu (rq->cpu, old_rd->span);
@@ -515,7 +515,7 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)cpumask_set_cpu (rq->cpu, rd->span);if (cpumask_test_cpu (rq->cpu, cpu_active_mask)) -set_rq_online (rq); +set_rq_online (rq,RQ_ONOFF_TOPOLOGY );rq_unlock_irqrestore (rq, &rf);
Signature · 1 line
2.44.0
Peter Zijlstra
maintainerRe: [PATCH 09/39] sched: Add @reason to sched_class->rq_{on|off}line()
Quoted 8 lines · Tejun Heo
->rq_{on|off}line are called either during CPU hotplug or cpuset partition updates. A planned BPF extensible
sched_classwants to tell the BPF scheduler progs about CPU hotplug events in a way that's synchronized with rq state changes.As the BPF scheduler progs aren't necessarily affected by cpuset partition updates, we need a way to distinguish the two types of events. Let's add an argument to tell them apart.
That would be a bug. Must not be able to ignore partitions.
Tejun Heo
maintainerRe: [PATCH 09/39] sched: Add @reason to sched_class->rq_{on|off}line()
Hello, Peter.
Quoted 11 lines · Peter Zijlstra
Quoted 8 lines · Tejun Heo
->rq_{on|off}line are called either during CPU hotplug or cpuset partition updates. A planned BPF extensible
sched_classwants to tell the BPF scheduler progs about CPU hotplug events in a way that's synchronized with rq state changes.As the BPF scheduler progs aren't necessarily affected by cpuset partition updates, we need a way to distinguish the two types of events. Let's add an argument to tell them apart.
That would be a bug. Must not be able to ignore partitions.
So, first of all, this implementation was brittle in assuming CPU hotplug events would be called in first and broke after recent cpuset changes. In v7, it's replaced by hooks in sched_cpu_[de]activate(), which has the extra benefit of allowing the BPF hotplug methods to be sleepable.
Taking a step back to the sched domains. They don't translate well to sched_ext schedulers where task to CPU associations are often more dynamic (e.g. multiple CPUs sharing a task queue) and load balancing operations can be implemented pretty differently from CFS. The benefits of exposing sched domains directly to the BPF schedulers is unclear as most of relevant information can be obtained from userspace already.
The cgroup support side isn't fully developed yet (e.g. cpu.weight is available but I haven't added cpu.max yet) and plans can always change but I was thinking taking a similar approach as cpu.weight for cpuset's isolation features - ie. give the BPF scheduler a way to access the user's configuration and let it implement whatever it wants to implement.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH 09/39] sched: Add @reason to sched_class->rq_{on|off}line()
Quoted 13 lines · Tejun Heo
Hello, Peter.
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Tejun Heo
->rq_{on|off}line are called either during CPU hotplug or cpuset partition updates. A planned BPF extensible
sched_classwants to tell the BPF scheduler progs about CPU hotplug events in a way that's synchronized with rq state changes.As the BPF scheduler progs aren't necessarily affected by cpuset partition updates, we need a way to distinguish the two types of events. Let's add an argument to tell them apart.
Quoted 6 lines
Quoted 1 line
That would be a bug. Must not be able to ignore partitions.
So, first of all, this implementation was brittle in assuming CPU hotplug events would be called in first and broke after recent cpuset changes. In v7, it's replaced by hooks in sched_cpu_[de]activate(), which has the extra benefit of allowing the BPF hotplug methods to be sleepable.
Urgh, I suppose I should go stare at v7 then.
Quoted 6 lines
Taking a step back to the sched domains. They don't translate well to
sched_extschedulers where task to CPU associations are often more dynamic (e.g. multiple CPUs sharing a task queue) and load balancing operations can be implemented pretty differently from CFS. The benefits of exposing sched domains directly to the BPF schedulers is unclear as most of relevant information can be obtained from userspace already.
Either which way around you want to turn it, you must not violate partitions. If a bpf thing isn't capable of handling partitions, you must refuse loading it when a partition exists and equally disallow creation of partitions when it does load.
For partitions specifically, you only need the root_domain, not the full sched_domain trees.
I'm aware you have these shared runqueues, but you don't *have* to do that. Esp. so if the user explicitly requested partitions.
Tejun Heo
maintainerRe: [PATCH 09/39] sched: Add @reason to sched_class->rq_{on|off}line()
Hello,
On Tue, Jun 25, 2024 at 10:29:26AM +0200, Peter Zijlstra wrote:
...
Quoted 12 lines
Quoted 6 lines
Taking a step back to the sched domains. They don't translate well to
sched_extschedulers where task to CPU associations are often more dynamic (e.g. multiple CPUs sharing a task queue) and load balancing operations can be implemented pretty differently from CFS. The benefits of exposing sched domains directly to the BPF schedulers is unclear as most of relevant information can be obtained from userspace already.Either which way around you want to turn it, you must not violate partitions. If a bpf thing isn't capable of handling partitions, you must refuse loading it when a partition exists and equally disallow creation of partitions when it does load.
Quoted 5 lines
For partitions specifically, you only need the
root_domain, not the fullsched_domaintrees.I'm aware you have these shared runqueues, but you don't *have* to do that. Esp. so if the user explicitly requested partitions.
As a quick work around, I can just disallow / eject the BPF scheduler when partitioning is configured. However, I think I'm still missing something and would appreciate if you can fill me in.
Abiding by core scheduling configuration is critical because it has direct user visible and security implications and this can be tested from userspace
- are two threads which shouldn't be on the same core on the same core or not? So, the violation condition is pretty clear.
However, I'm not sure how partioning is similar. My understanding is that it works as a barrier for the load balancer. LB on this side can't look there and LB on that side can't look here. However, isn't the impact purely performance / isolation difference? IOW, let's say you laod a BPF scheduler which consumes the partition information but doesn't do anything differently based on it. cpumasks are still enforced the same and I can't think of anything which userspace would be able to test to tell whether partitioning is working or not.
If the only difference partitions make is on performance. While it would make sense to communicate partitions to the BPF scheduler, would it make sense to reject BPF scheduler based on it? ie. Assuming that the feature is implemented, what would distinguish between one BPF scheduler which handles partitions specially and the other which doesn't care?
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH 09/39] sched: Add @reason to sched_class->rq_{on|off}line()
Quoted 26 lines · Tejun Heo
Hello,
On Tue, Jun 25, 2024 at 10:29:26AM +0200, Peter Zijlstra wrote:
...Quoted 12 lines
Quoted 6 lines
Taking a step back to the sched domains. They don't translate well to
sched_extschedulers where task to CPU associations are often more dynamic (e.g. multiple CPUs sharing a task queue) and load balancing operations can be implemented pretty differently from CFS. The benefits of exposing sched domains directly to the BPF schedulers is unclear as most of relevant information can be obtained from userspace already.Either which way around you want to turn it, you must not violate partitions. If a bpf thing isn't capable of handling partitions, you must refuse loading it when a partition exists and equally disallow creation of partitions when it does load.
Quoted 5 lines
For partitions specifically, you only need the
root_domain, not the fullsched_domaintrees.I'm aware you have these shared runqueues, but you don't *have* to do that. Esp. so if the user explicitly requested partitions.
As a quick work around, I can just disallow / eject the BPF scheduler when partitioning is configured. However, I think I'm still missing something and would appreciate if you can fill me in.
Quoted 6 lines
Abiding by core scheduling configuration is critical because it has direct user visible and security implications and this can be tested from userspace
- are two threads which shouldn't be on the same core on the same core or not? So, the violation condition is pretty clear.However, I'm not sure how partioning is similar.
I'm not sure what you mean. It's like violating the cpumask, probably not a big deal, but against the express wishes of the user.
Quoted 4 lines
My understanding is that it
works as a barrier for the load balancer. LB on this side can't look there and LB on that side can't look here. However, isn't the impact purely performance / isolation difference?
Yes. But this isolation is very important to some people.
Quoted 5 lines
IOW, let's say you laod a BPF scheduler
which consumes the partition information but doesn't do anything differently based on it. cpumasks are still enforced the same and I can't think of anything which userspace would be able to test to tell whether partitioning is working or not.
So barring a few caveats it really boils down to a task staying in the partition it's part of. If you ever see it leave, you know you got a problem.
Now, there's a bunch of ways to actually create partitions:
- cpuset
- cpuset-v2
- isolcpus boot crap
And they're all subtly different iirc, but IIRC the cpuset ones are simplest since the task is part of a cgroup and the cgroup cpumask is imposed on them and things should be fairly straight forward.
The isolcpus thing creates a pile of single CPU partitions and people have to manually set cpu-affinity, and here we have some hysterical behaviour that I would love to change but have not yet dared do -- because I know there's people doing dodgy things because they've been sending 'bug' reports.
Specifically it is possible to set a cpumask that spans multiple partitions :-( Traditionally the behaviour was that it would place the task on the lowest cpu number, the current behaviour is the task it placed randomly on any CPU in the given mask.
It is my opinion that both behaviours are correct, since after all, we don't violate the given constraint, the user provided mask. If that's not what you wanted, you should be setting something else etc..
I've proposed rejecting a cpumask that spans partitions -- I've not yet done this, because clearly people are doing this, however misguided. But perhaps we should just bite the bullet and cause pain -- dunno.
Anyway, tl;dr, you can have a cpumask wider than a parition and people still not wanting migrations to happen.
Quoted 1 line
If the only difference partitions make is on performance.
People explicitly did not want migrations there -- otherwise they would not have gone to the trouble of setting up the partitions in the first place.
Quoted 5 lines
While it would
make sense to communicate partitions to the BPF scheduler, would it make sense to reject BPF scheduler based on it? ie. Assuming that the feature is implemented, what would distinguish between one BPF scheduler which handles partitions specially and the other which doesn't care?
Correctness? Anyway, can't you handle this in the kernel part, simply never allow a shared runqueue to cross a root_domain's mask and put some WARNs on to ensure constraints are respected etc.? Should be fairly simple to check prev_cpu and new_cpu are having the same root_domain for instance.
Tejun Heo
maintainerRe: [PATCH 09/39] sched: Add @reason to sched_class->rq_{on|off}line()
Hello,
On Wed, Jun 26, 2024 at 10:23:42AM +0200, Peter Zijlstra wrote:
...
Quoted 14 lines
- cpuset
- cpuset-v2
- isolcpus boot crapAnd they're all subtly different iirc, but IIRC the cpuset ones are simplest since the task is part of a cgroup and the cgroup cpumask is imposed on them and things should be fairly straight forward.
The isolcpus thing creates a pile of single CPU partitions and people have to manually set cpu-affinity, and here we have some hysterical behaviour that I would love to change but have not yet dared do -- because I know there's people doing dodgy things because they've been sending 'bug' reports.
Quoted 4 lines
Specifically it is possible to set a cpumask that spans multiple partitions :-( Traditionally the behaviour was that it would place the task on the lowest cpu number, the current behaviour is the task it placed randomly on any CPU in the given mask.
This is what I was missing. I was just thinking cpuset case and as cpuset partitions are always reflected in the task cpumasks, there isn't whole lot to do.
...
Quoted 11 lines
Quoted 5 lines
While it would
make sense to communicate partitions to the BPF scheduler, would it make sense to reject BPF scheduler based on it? ie. Assuming that the feature is implemented, what would distinguish between one BPF scheduler which handles partitions specially and the other which doesn't care?Correctness? Anyway, can't you handle this in the kernel part, simply never allow a shared runqueue to cross a root_domain's mask and put some WARNs on to ensure constraints are respected etc.? Should be fairly simple to check
prev_cpuandnew_cpuare having the sameroot_domainfor instance.
Yeah, I'll plug it. It might as well be just reject and ejecting BPF schedulers when conditions are detected. The BPF scheduler doesn't have to use the built-in DSQs and can decide to dispatch to any CPU from its BPF queues (however that may be implemented, it can also be in userspace), so it's a bit tricky to enforce correctness dynamically after the fact. I'll think more on it.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 1/1 sched_ext: Disallow loading BPF scheduler if isolcpus= domain isolation is in effect
sched_domains regulate the load balancing for sched_classes. A machine can be partitioned into multiple sections that are not load-balanced across using either isolcpus= boot param or cpuset partitions. In such cases, tasks that are in one partition are expected to stay within that partition.
cpuset configured partitions are always reflected in each member task's cpumask. As SCX always honors the task cpumasks, the BPF scheduler is automatically in compliance with the configured partitions.
However, for isolcpus= domain isolation, the isolated CPUs are simply omitted from the top-level sched_domain[s] without further restrictions on tasks' cpumasks, so, for example, a task currently running in an isolated CPU may have more CPUs in its allowed cpumask while expected to remain on the same CPU.
There is no straightforward way to enforce this partitioning preemptively on BPF schedulers and erroring out after a violation can be surprising. isolcpus= domain isolation is being replaced with cpuset partitions anyway, so keep it simple and simply disallow loading a BPF scheduler if isolcpus= domain isolation is in effect.
--- kernel/sched/build_policy.c | 1 + kernel/sched/ext.c | 6 ++++++ 2 files changed, 7 insertions(+)
kernel/sched/build_policy.c +1 -0
@@ -16,6 +16,7 @@#include <linux/sched/clock.h> #include <linux/sched/cputime.h> #include <linux/sched/hotplug.h> +#include <linux/sched/isolation.h> #include <linux/sched/posix-timers.h> #include <linux/sched/rt.h>
kernel/sched/ext.c +6 -0
@@ -4400,6 +4400,12 @@ static int scx_ops_enable(struct sched_eunsigned long timeout;int i, cpu, ret; +if (!cpumask_equal (housekeeping_cpumask (HK_TYPE_DOMAIN ), + cpu_possible_mask)) { +pr_err ("sched_ext: Not compatible with \"isolcpus=\" domain isolation" ); +return -EINVAL ; + } +mutex_lock (&scx_ops_enable_mutex);if (!scx_ops_helper) {
Re: [PATCH sched_ext/for-6.11] sched_ext: Disallow loading BPF scheduler if isolcpus= domain isolation is in effect
Quoted 21 lines · Tejun Heo
sched_domainsregulate the load balancing forsched_classes. A machine can be partitioned into multiple sections that are not load-balanced across using either isolcpus= boot param or cpuset partitions. In such cases, tasks that are in one partition are expected to stay within that partition.cpuset configured partitions are always reflected in each member task's cpumask. As SCX always honors the task cpumasks, the BPF scheduler is automatically in compliance with the configured partitions.
However, for isolcpus= domain isolation, the isolated CPUs are simply omitted from the top-level sched_domain[s] without further restrictions on tasks' cpumasks, so, for example, a task currently running in an isolated CPU may have more CPUs in its allowed cpumask while expected to remain on the same CPU.
There is no straightforward way to enforce this partitioning preemptively on BPF schedulers and erroring out after a violation can be surprising. isolcpus= domain isolation is being replaced with cpuset partitions anyway, so keep it simple and simply disallow loading a BPF scheduler if isolcpus= domain isolation is in effect.
Quoted 5 lines
Signed-off-by Tejun Heo <[email protected]>Cc David Vernet <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Frederic Weisbecker <[email protected]>
Applied to cgroup/for-6.11.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 10/39 sched: Factor out update_other_load_avgs() from __update_blocked_others()
In reply to Tejun HeoRT, DL, thermal and irq load and utilization metrics need to be decayed and updated periodically and before consumption to keep the numbers reasonable. This is currently done from __update_blocked_others() as a part of the fair class load balance path. Let's factor it out to update_other_load_avgs(). Pure refactor. No functional changes.
This will be used by the new BPF extensible scheduling class to ensure that the above metrics are properly maintained.
--- kernel/sched/core.c | 19 +++++++++++++++++++ kernel/sched/fair.c | 16 +++------------- kernel/sched/sched.h | 3 +++ 3 files changed, 25 insertions(+), 13 deletions(-)
kernel/sched/core.c +19 -0
@@ -7486,6 +7486,25 @@ int sched_core_idle_cpu(int cpu)#endif #ifdef CONFIG_SMP +/* +* Load avg and utiliztion metrics need to be updated periodically and before +* consumption. This function updates the metrics for all subsystems except for +* the fair class. @rq must be locked and have its clock updated. +*/ +bool update_other_load_avgs (struct rq *rq) +{ +u64 now =rq_clock_pelt (rq); +const struct sched_class *curr_class = rq->curr->sched_class; +unsigned long thermal_pressure =arch_scale_thermal_pressure (cpu_of (rq)); + +lockdep_assert_rq_held (rq); + +return update_rt_rq_load_avg (now, rq, curr_class == &rt_sched_class) | +update_dl_rq_load_avg (now, rq, curr_class == &dl_sched_class) | +update_thermal_load_avg (rq_clock_thermal (rq), rq, thermal_pressure) | +update_irq_load_avg (rq,0 ); +} +/* * This function computes an effective utilization for the given CPU, to be * used for frequency selection given the linear relation: f = u * f_max.
kernel/sched/fair.c +3 -13
@@ -9283,28 +9283,18 @@ static inline void update_blocked_load_status(struct rq *rq, bool has_blocked) {static bool __update_blocked_others (struct rq *rq,bool *done) { -const struct sched_class *curr_class; -u64 now =rq_clock_pelt (rq); -unsigned long thermal_pressure; -bool decayed; +bool updated;/* * update_load_avg() can call cpufreq_update_util(). Make sure that RT, * DL and IRQ signals have been updated before updating CFS. */ - curr_class = rq->curr->sched_class; - - thermal_pressure =arch_scale_thermal_pressure (cpu_of (rq)); - - decayed =update_rt_rq_load_avg (now, rq, curr_class == &rt_sched_class) | -update_dl_rq_load_avg (now, rq, curr_class == &dl_sched_class) | -update_thermal_load_avg (rq_clock_thermal (rq), rq, thermal_pressure) | -update_irq_load_avg (rq,0 ); + updated =update_other_load_avgs (rq);if (others_have_blocked (rq)) *done = false; -return decayed; +return updated; }#ifdef CONFIG_FAIR_GROUP_SCHED
kernel/sched/sched.h +3 -0
@@ -3042,6 +3042,7 @@ static inline void cpufreq_update_util(struct rq *rq, unsigned int flags) {}#endif #ifdef CONFIG_SMP +bool update_other_load_avgs (struct rq *rq);unsigned long effective_cpu_util (int cpu,unsigned long util_cfs,unsigned long *min,unsigned long *max);
@@ -3084,6 +3085,8 @@ static inline unsigned long cpu_util_rt(struct rq *rq) {return READ_ONCE (rq->avg_rt.util_avg); } +#else +static inline bool update_other_load_avgs (struct rq *rq) {return false; }#endif #ifdef CONFIG_UCLAMP_TASK
Signature · 1 line
2.44.0
Peter Zijlstra
maintainerRe: [PATCH 10/39] sched: Factor out update_other_load_avgs() from __update_blocked_others()
Quoted 33 lines · Tejun Heo
RT, DL, thermal and irq load and utilization metrics need to be decayed and updated periodically and before consumption to keep the numbers reasonable. This is currently done from
__update_blocked_others()as a part of the fair class load balance path. Let's factor it out toupdate_other_load_avgs(). Pure refactor. No functional changes.This will be used by the new BPF extensible scheduling class to ensure that the above metrics are properly maintained.
Signed-off-by Tejun Heo <[email protected]>Reviewed-by David Vernet <[email protected]>--- kernel/sched/core.c | 19 +++++++++++++++++++ kernel/sched/fair.c | 16 +++------------- kernel/sched/sched.h | 3 +++ 3 files changed, 25 insertions(+), 13 deletions(-)kernel/sched/core.c +8
-0@@ -7486,6 +7486,25 @@ int sched_core_idle_cpu(int cpu)#endif #ifdef CONFIG_SMP +/* +* Load avg and utiliztion metrics need to be updated periodically and before +* consumption. This function updates the metrics for all subsystems except for +* the fair class. @rq must be locked and have its clock updated. +*/ +bool update_other_load_avgs (struct rq *rq) +{ +u64 now =rq_clock_pelt (rq);
kernel/sched/core.c int sched_core_idle_cpu(int cpu) · line 7506
Quoted 10 lines
kernel/sched/core.c +10
-0+const struct sched_class *curr_class = rq->curr->sched_class; +unsigned long thermal_pressure =arch_scale_thermal_pressure (cpu_of (rq)); + +lockdep_assert_rq_held (rq); + +return update_rt_rq_load_avg (now, rq, curr_class == &rt_sched_class) | +update_dl_rq_load_avg (now, rq, curr_class == &dl_sched_class) | +update_thermal_load_avg (rq_clock_thermal (rq), rq, thermal_pressure) | +update_irq_load_avg (rq,0 ); +}
Yeah, but you then ignore the return value and don't call into cpufreq.
Vincent, what would be the right thing to do here?
Vincent Guittot
maintainerRe: [PATCH 10/39] sched: Factor out update_other_load_avgs() from __update_blocked_others()
Quoted 48 lines · Peter Zijlstra
Quoted 33 lines · Tejun Heo
RT, DL, thermal and irq load and utilization metrics need to be decayed and updated periodically and before consumption to keep the numbers reasonable. This is currently done from
__update_blocked_others()as a part of the fair class load balance path. Let's factor it out toupdate_other_load_avgs(). Pure refactor. No functional changes.This will be used by the new BPF extensible scheduling class to ensure that the above metrics are properly maintained.
Signed-off-by Tejun Heo <[email protected]>Reviewed-by David Vernet <[email protected]>--- kernel/sched/core.c | 19 +++++++++++++++++++ kernel/sched/fair.c | 16 +++------------- kernel/sched/sched.h | 3 +++ 3 files changed, 25 insertions(+), 13 deletions(-)kernel/sched/core.c +8
-0@@ -7486,6 +7486,25 @@ int sched_core_idle_cpu(int cpu)#endif #ifdef CONFIG_SMP +/* +* Load avg and utiliztion metrics need to be updated periodically and before +* consumption. This function updates the metrics for all subsystems except for +* the fair class. @rq must be locked and have its clock updated. +*/ +bool update_other_load_avgs (struct rq *rq) +{ +u64 now =rq_clock_pelt (rq);kernel/sched/core.c int sched_core_idle_cpu(int cpu) · line 7506
Quoted 10 lines
kernel/sched/core.c +10
-0+const struct sched_class *curr_class = rq->curr->sched_class; +unsigned long thermal_pressure =arch_scale_thermal_pressure (cpu_of (rq)); + +lockdep_assert_rq_held (rq); + +return update_rt_rq_load_avg (now, rq, curr_class == &rt_sched_class) | +update_dl_rq_load_avg (now, rq, curr_class == &dl_sched_class) | +update_thermal_load_avg (rq_clock_thermal (rq), rq, thermal_pressure) | +update_irq_load_avg (rq,0 ); +}Yeah, but you then ignore the return value and don't call into cpufreq.
Quoted 1 line
Vincent, what would be the right thing to do here?
These metrics are only consumed by fair class so my first question would be:
- Do we plan to have a fair and sched_ext to coexist ? Or is it exclusive ? I haven't been able to get a clear answer on this while reading the cover letter
- If sched_ext is exclusive to fair then I'm not sure that you need to update them at all because they will not be used. RT uses a fix freq and DL use the Sum of running DL bandwidth. But if both an coexist we use be sure to update them periodically
Tejun Heo
maintainerRe: [PATCH 10/39] sched: Factor out update_other_load_avgs() from __update_blocked_others()
Hello, Peter, Vincent.
Quoted 16 lines · Vincent Guittot
Quoted 16 lines
kernel/sched/core.c int sched_core_idle_cpu(int cpu) · line 7506
Quoted 13 lines
kernel/sched/core.c +13
-0+bool update_other_load_avgs (struct rq *rq) +{ +u64 now =rq_clock_pelt (rq); +const struct sched_class *curr_class = rq->curr->sched_class; +unsigned long thermal_pressure =arch_scale_thermal_pressure (cpu_of (rq)); + +lockdep_assert_rq_held (rq); + +return update_rt_rq_load_avg (now, rq, curr_class == &rt_sched_class) | +update_dl_rq_load_avg (now, rq, curr_class == &dl_sched_class) | +update_thermal_load_avg (rq_clock_thermal (rq), rq, thermal_pressure) | +update_irq_load_avg (rq,0 ); +}Yeah, but you then ignore the return value and don't call into cpufreq.
Quoted 7 lines
Quoted 1 line
Vincent, what would be the right thing to do here?
These metrics are only consumed by fair class so my first question would be:
- Do we plan to have a fair and
sched_extto coexist ? Or is it exclusive ? I haven't been able to get a clear answer on this while reading the cover letter
Oh, there are two modes. In partial mode (SCX_OPS_SWITCH_PARTIAL), there can be both CFS and SCX tasks running on the same system. If the flag is not set, the whole system is on SCX.
Quoted 4 lines
- If
sched_extis exclusive to fair then I'm not sure that you need to update them at all because they will not be used. RT uses a fix freq and DL use the Sum of running DL bandwidth. But if both an coexist we use be sure to update them periodically
Hmm.... I think I saw RT's schedutil signal stuck high constantly pushing up the frequency. I might be mistaken tho. I'll check again.
Thanks.
Signature · 1 line
tejun
Vincent Guittot
maintainerRe: [PATCH 10/39] sched: Factor out update_other_load_avgs() from __update_blocked_others()
Quoted 24 lines · Tejun Heo
Hello, Peter, Vincent.
Quoted 20 lines · Vincent Guittot
Quoted 16 lines
kernel/sched/core.c int sched_core_idle_cpu(int cpu) · line 7506
Quoted 13 lines
kernel/sched/core.c +13
-0+bool update_other_load_avgs (struct rq *rq) +{ +u64 now =rq_clock_pelt (rq); +const struct sched_class *curr_class = rq->curr->sched_class; +unsigned long thermal_pressure =arch_scale_thermal_pressure (cpu_of (rq)); + +lockdep_assert_rq_held (rq); + +return update_rt_rq_load_avg (now, rq, curr_class == &rt_sched_class) | +update_dl_rq_load_avg (now, rq, curr_class == &dl_sched_class) | +update_thermal_load_avg (rq_clock_thermal (rq), rq, thermal_pressure) | +update_irq_load_avg (rq,0 ); +}Yeah, but you then ignore the return value and don't call into cpufreq.
Quoted 1 line
Vincent, what would be the right thing to do here?
These metrics are only consumed by fair class so my first question would be:
Quoted 7 lines
Quoted 3 lines
- Do we plan to have a fair and
sched_extto coexist ? Or is it exclusive ? I haven't been able to get a clear answer on this while reading the cover letterOh, there are two modes. In partial mode (
SCX_OPS_SWITCH_PARTIAL), there can be both CFS and SCX tasks running on the same system. If the flag is not set, the whole system is on SCX.
ok thanks for the clarification because this will have an impact on how fair tasks are scheduled
Quoted 8 lines
Quoted 5 lines
- If
sched_extis exclusive to fair then I'm not sure that you need to update them at all because they will not be used. RT uses a fix freq and DL use the Sum of running DL bandwidth. But if both an coexist we use be sure to update them periodicallyHmm.... I think I saw RT's schedutil signal stuck high constantly pushing up the frequency. I might be mistaken tho. I'll check again.
This is used when selecting a frequency for fair tasks
Quoted 6 lines
Thanks.
--
tejun
Tejun Heo
maintainerRe: [PATCH 10/39] sched: Factor out update_other_load_avgs() from __update_blocked_others()
Hello,
Quoted 4 lines · Vincent Guittot
Quoted 2 lines
Hmm.... I think I saw RT's schedutil signal stuck high constantly pushing up the frequency. I might be mistaken tho. I'll check again.
This is used when selecting a frequency for fair tasks
When schedutil is used as the governor, sugov_get_util() provides the source utilization information to determine the target frequency. sugov_get_util() gets the CFS util metric from cpu_util_cfs_boost() and then runs it through effective_cpu_util().
effective_cpu_util() does a bunch of things including adding cpu_util_irq(), cpu_util_rt() and cpu_util_dl(), so if SCX doesn't decay these utilization metrics, they never decay and schedutil ends up making decisions with stale stuck-high numbers. I can easily confirm the behavior by sprinkling some trace_printks and commenting out update_other_load_avgs() on the SCX side.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 11/39 cpufreq_schedutil: Refactor sugov_cpu_is_busy()
In reply to Tejun Heosugov_cpu_is_busy() is used to avoid decreasing performance level while the CPU is busy and called by sugov_update_single_freq() and sugov_update_single_perf(). Both callers repeat the same pattern to first test for uclamp and then the business. Let's refactor so that the tests aren't repeated.
The new helper is named sugov_hold_freq() and tests both the uclamp exception and CPU business. No functional changes. This will make adding more exception conditions easier.
--- kernel/sched/cpufreq_schedutil.c | 38 +++++++++++++++----------------- 1 file changed, 18 insertions(+), 20 deletions(-)
kernel/sched/cpufreq_schedutil.c +18 -20
@@ -325,16 +325,27 @@ static unsigned long sugov_iowait_apply(struct sugov_cpu *sg_cpu, u64 time, }#ifdef CONFIG_NO_HZ_COMMON -static bool sugov_cpu_is_busy (struct sugov_cpu *sg_cpu) +static bool sugov_hold_freq (struct sugov_cpu *sg_cpu) { -unsigned long idle_calls =tick_nohz_get_idle_calls_cpu (sg_cpu->cpu); -bool ret = idle_calls == sg_cpu->saved_idle_calls; +unsigned long idle_calls; +bool ret; + +/* if capped by uclamp_max, always update to be in compliance */ +if (uclamp_rq_is_capped (cpu_rq (sg_cpu->cpu))) +return false; + +/* +* Maintain the frequency if the CPU has not been idle recently, as +* reduction is likely to be premature. +*/ + idle_calls =tick_nohz_get_idle_calls_cpu (sg_cpu->cpu); + ret = idle_calls == sg_cpu->saved_idle_calls; sg_cpu->saved_idle_calls = idle_calls;return ret; }#else -static inline bool sugov_cpu_is_busy (struct sugov_cpu *sg_cpu) {return false; } +static inline bool sugov_hold_freq (struct sugov_cpu *sg_cpu) {return false; }#endif /* CONFIG_NO_HZ_COMMON */ /*
@@ -382,14 +393,8 @@ static void sugov_update_single_freq(struct update_util_data *hook, u64 time,return ; next_f =get_next_freq (sg_policy, sg_cpu->util, max_cap); -/* -* Do not reduce the frequency if the CPU has not been idle -* recently, as the reduction is likely to be premature then. -* -* Except when the rq is capped by uclamp_max. -*/ -if (!uclamp_rq_is_capped (cpu_rq (sg_cpu->cpu)) && -sugov_cpu_is_busy (sg_cpu) && next_f < sg_policy->next_freq && + +if (sugov_hold_freq (sg_cpu) && next_f < sg_policy->next_freq && !sg_policy->need_freq_update) { next_f = sg_policy->next_freq;
@@ -436,14 +441,7 @@ static void sugov_update_single_perf(struct update_util_data *hook, u64 time,if (!sugov_update_single_common (sg_cpu, time, max_cap, flags))return ; -/* -* Do not reduce the target performance level if the CPU has not been -* idle recently, as the reduction is likely to be premature then. -* -* Except when the rq is capped by uclamp_max. -*/ -if (!uclamp_rq_is_capped (cpu_rq (sg_cpu->cpu)) && -sugov_cpu_is_busy (sg_cpu) && sg_cpu->util < prev_util) +if (sugov_hold_freq (sg_cpu) && sg_cpu->util < prev_util) sg_cpu->util = prev_util; cpufreq_driver_adjust_perf(sg_cpu->cpu, sg_cpu->bw_min,
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 12/39 sched: Add normal_policy()
In reply to Tejun HeoA new BPF extensible sched_class will need to dynamically change how a task picks its sched_class. For example, if the loaded BPF scheduler progs fail, the tasks will be forced back on CFS even if the task's policy is set to the new sched_class. To support such mapping, add normal_policy() which wraps testing for %SCHED_NORMAL. This doesn't cause any behavior changes.
v2: Update the description with more details on the expected use.
--- kernel/sched/fair.c | 2 +- kernel/sched/sched.h | 8 +++++++- 2 files changed, 8 insertions(+), 2 deletions(-)
kernel/sched/fair.c +1 -1
@@ -8322,7 +8322,7 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int* Batch and idle tasks do not preempt non-idle tasks (their preemption * is driven by the tick): */ -if (unlikely (p->policy !=SCHED_NORMAL ) || !sched_feat (WAKEUP_PREEMPTION )) +if (unlikely (!normal_policy (p->policy)) || !sched_feat (WAKEUP_PREEMPTION ))return ;find_matching_se (&se, &pse);
kernel/sched/sched.h +7 -1
@@ -171,9 +171,15 @@ static inline int idle_policy(int policy) {return policy ==SCHED_IDLE ; } + +static inline int normal_policy (int policy) +{ +return policy ==SCHED_NORMAL ; +} +static inline int fair_policy (int policy) { -return policy ==SCHED_NORMAL || policy ==SCHED_BATCH ; +return normal_policy (policy) || policy ==SCHED_BATCH ; }static inline int rt_policy (int policy)
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 13/39 sched_ext: Add boilerplate for extensible scheduler class
In reply to Tejun HeoThis adds dummy implementations of sched_ext interfaces which interact with the scheduler core and hook them in the correct places. As they're all dummies, this doesn't cause any behavior changes. This is split out to help reviewing.
v2: balance_scx_on_up() dropped. This will be handled in sched_ext proper.
--- include/linux/sched/ext.h | 12 ++++++++++++ kernel/fork.c | 2 ++ kernel/sched/core.c | 32 ++++++++++++++++++++++++-------- kernel/sched/cpufreq_schedutil.c | 4 +++- kernel/sched/ext.h | 25 +++++++++++++++++++++++++ kernel/sched/idle.c | 2 ++ kernel/sched/sched.h | 2 ++ 7 files changed, 70 insertions(+), 9 deletions(-) create mode 100644 include/linux/sched/ext.h create mode 100644 kernel/sched/ext.h
include/linux/sched/ext.h new file +12 -0
@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_SCHED_EXT_H +#define _LINUX_SCHED_EXT_H + +#ifdef CONFIG_SCHED_CLASS_EXT +#error "NOT IMPLEMENTED YET" +#else /* !CONFIG_SCHED_CLASS_EXT */ + +static inline void sched_ext_free (struct task_struct *p) {} + +#endif /* CONFIG_SCHED_CLASS_EXT */ +#endif /* _LINUX_SCHED_EXT_H */
kernel/fork.c +2 -0
@@ -23,6 +23,7 @@#include <linux/sched/task.h> #include <linux/sched/task_stack.h> #include <linux/sched/cputime.h> +#include <linux/sched/ext.h> #include <linux/seq_file.h> #include <linux/rtmutex.h> #include <linux/init.h>
@@ -970,6 +971,7 @@ void __put_task_struct(struct task_struct *tsk)WARN_ON (refcount_read (&tsk->usage));WARN_ON (tsk == current); +sched_ext_free (tsk);io_uring_free (tsk);cgroup_free (tsk);task_numa_free (tsk, true);
kernel/sched/core.c +24 -8
@@ -4767,6 +4767,8 @@ late_initcall(sched_core_sysctl_init);*/ int sched_fork (unsigned long clone_flags,struct task_struct *p) { +int ret; +__sched_fork (clone_flags, p);/* * We mark the process as NEW here. This guarantees that
@@ -4803,12 +4805,16 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p) p->sched_reset_on_fork =0 ; } -if (dl_prio (p->prio)) -return -EAGAIN ; -else if (rt_prio (p->prio)) +scx_pre_fork (p); + +if (dl_prio (p->prio)) { + ret = -EAGAIN ; + goto out_cancel; + }else if (rt_prio (p->prio)) { p->sched_class = &rt_sched_class; -else + }else { p->sched_class = &fair_sched_class; + }init_entity_runnable_average (&p->se);
@@ -4826,6 +4832,10 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)RB_CLEAR_NODE (&p->pushable_dl_tasks);#endif return 0 ; + +out_cancel: +scx_cancel_fork (p); +return ret; }int sched_cgroup_fork (struct task_struct *p,struct kernel_clone_args *kargs)
@@ -4856,16 +4866,18 @@ int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs) p->sched_class->task_fork (p);raw_spin_unlock_irqrestore (&p->pi_lock, flags); -return 0 ; +return scx_fork (p); }void sched_cancel_fork (struct task_struct *p) { +scx_cancel_fork (p); }void sched_post_fork (struct task_struct *p) {uclamp_post_fork (p); +scx_post_fork (p); }unsigned long to_ratio (u64 period,u64 runtime)
@@ -6017,7 +6029,7 @@ static void put_prev_task_balance(struct rq *rq, struct task_struct *prev,* We can terminate the balance pass as soon as we know there is * a runnable task of @class priority or higher. */ - for_class_range(class, prev->sched_class, &idle_sched_class) { + for_balance_class_range(class, prev->sched_class, &idle_sched_class) {if (class->balance (rq, prev, rf))break ; }
@@ -6035,6 +6047,9 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)const struct sched_class *class;struct task_struct *p; +if (scx_enabled ()) + goto restart; +/* * Optimization: we know that if all tasks are in the fair class we can * call that function directly, but only if the @prev task wasn't of a
@@ -6075,7 +6090,7 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)if (prev->dl_server) prev->dl_server =NULL ; -for_each_class (class) { +for_each_active_class (class) { p = class->pick_next_task (rq);if (p)return p;
@@ -6108,7 +6123,7 @@ static inline struct task_struct *pick_task(struct rq *rq)const struct sched_class *class;struct task_struct *p; -for_each_class (class) { +for_each_active_class (class) { p = class->pick_task (rq);if (p)return p;
@@ -10135,6 +10150,7 @@ void __init sched_init(void)balance_push_set (smp_processor_id (), false);#endif init_sched_fair_class (); +init_sched_ext_class ();psi_init ();
kernel/sched/cpufreq_schedutil.c +3 -1
@@ -197,7 +197,9 @@ unsigned long sugov_effective_cpu_perf(int cpu, unsigned long actual,static void sugov_get_util (struct sugov_cpu *sg_cpu,unsigned long boost) { -unsigned long min, max, util =cpu_util_cfs_boost (sg_cpu->cpu); +unsigned long min, max; +unsigned long util =cpu_util_cfs_boost (sg_cpu->cpu) + +scx_cpuperf_target (sg_cpu->cpu); util =effective_cpu_util (sg_cpu->cpu, util, &min, &max); util =max (util, boost);
kernel/sched/ext.h new file +25 -0
@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifdef CONFIG_SCHED_CLASS_EXT +#error "NOT IMPLEMENTED YET" +#else /* CONFIG_SCHED_CLASS_EXT */ + +#define scx_enabled () false + +static inline void scx_pre_fork (struct task_struct *p) {} +static inline int scx_fork (struct task_struct *p) {return 0 ; } +static inline void scx_post_fork (struct task_struct *p) {} +static inline void scx_cancel_fork (struct task_struct *p) {} +static inline void init_sched_ext_class (void ) {} +static inline u32 scx_cpuperf_target (s32 cpu) {return 0 ; } + +#define for_each_active_class for_each_class +#define for_balance_class_range for_class_range + +#endif /* CONFIG_SCHED_CLASS_EXT */ + +#if defined(CONFIG_SCHED_CLASS_EXT ) && defined(CONFIG_SMP ) +#error "NOT IMPLEMENTED YET" +#else +static inline void scx_update_idle (struct rq *rq,bool idle) {} +#endif
kernel/sched/idle.c +2 -0
@@ -458,11 +458,13 @@ static void wakeup_preempt_idle(struct rq *rq, struct task_struct *p, int flags)static void put_prev_task_idle (struct rq *rq,struct task_struct *prev) { +scx_update_idle (rq, false); }static void set_next_task_idle (struct rq *rq,struct task_struct *next,bool first) {update_idle_core (rq); +scx_update_idle (rq, true);schedstat_inc (rq->sched_goidle); }
kernel/sched/sched.h +2 -0
@@ -3540,4 +3540,6 @@ enum cpu_cftype_id {extern struct cftype cpu_cftypes[CPU_CFTYPE_CNT +1 ];#endif /* CONFIG_CGROUP_SCHED */ +#include "ext.h" +#endif /* _KERNEL_SCHED_SCHED_H */
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 14/39 sched_ext: Implement BPF extensible scheduler class
In reply to Tejun HeoImplement a new scheduler class sched_ext (SCX), which allows scheduling policies to be implemented as BPF programs to achieve the following:
1. Ease of experimentation and exploration: Enabling rapid iteration of new scheduling policies.
2. Customization: Building application-specific schedulers which implement policies that are not applicable to general-purpose schedulers.
3. Rapid scheduler deployments: Non-disruptive swap outs of scheduling policies in production environments.
sched_ext leverages BPF’s struct_ops feature to define a structure which exports function callbacks and flags to BPF programs that wish to implement scheduling policies. The struct_ops structure exported by sched_ext is struct sched_ext_ops, and is conceptually similar to struct sched_class. The role of sched_ext is to map the complex sched_class callbacks to the more simple and ergonomic struct sched_ext_ops callbacks.
For more detailed discussion on the motivations and overview, please refer to the cover letter.
Later patches will also add several example schedulers and documentation.
This patch implements the minimum core framework to enable implementation of BPF schedulers. Subsequent patches will gradually add functionalities including safety guarantee mechanisms, nohz and cgroup support.
include/linux/sched/ext.h defines struct sched_ext_ops. With the comment on top, each operation should be self-explanatory. The followings are worth noting:
- Both "sched_ext" and its shorthand "scx" are used. If the identifier already has "sched" in it, "ext" is used; otherwise, "scx".
- In sched_ext_ops, only .name is mandatory. Every operation is optional and if omitted a simple but functional default behavior is provided.
- A new policy constant SCHED_EXT is added and a task can select sched_ext by invoking sched_setscheduler(2) with the new policy constant. However, if the BPF scheduler is not loaded, SCHED_EXT is the same as SCHED_NORMAL and the task is scheduled by CFS. When the BPF scheduler is loaded, all tasks which have the SCHED_EXT policy are switched to sched_ext.
- To bridge the workflow imbalance between the scheduler core and sched_ext_ops callbacks, sched_ext uses simple FIFOs called dispatch queues (dsq's). By default, there is one global dsq (SCX_DSQ_GLOBAL), and one local per-CPU dsq (SCX_DSQ_LOCAL). SCX_DSQ_GLOBAL is provided for convenience and need not be used by a scheduler that doesn't require it. SCX_DSQ_LOCAL is the per-CPU FIFO that sched_ext pulls from when putting the next task on the CPU. The BPF scheduler can manage an arbitrary number of dsq's using scx_bpf_create_dsq() and scx_bpf_destroy_dsq().
- sched_ext guarantees system integrity no matter what the BPF scheduler does. To enable this, each task's ownership is tracked through p->scx.ops_state and all tasks are put on scx_tasks list. The disable path can always recover and revert all tasks back to CFS. See p->scx.ops_state and scx_tasks.
- A task is not tied to its rq while enqueued. This decouples CPU selection from queueing and allows sharing a scheduling queue across an arbitrary subset of CPUs. This adds some complexities as a task may need to be bounced between rq's right before it starts executing. See dispatch_to_local_dsq() and move_task_to_local_dsq().
- One complication that arises from the above weak association between task and rq is that synchronizing with dequeue() gets complicated as dequeue() may happen anytime while the task is enqueued and the dispatch path might need to release the rq lock to transfer the task. Solving this requires a bit of complexity. See the logic around p->scx.sticky_cpu and p->scx.ops_qseq.
- Both enable and disable paths are a bit complicated. The enable path switches all tasks without blocking to avoid issues which can arise from partially switched states (e.g. the switching task itself being starved). The disable path can't trust the BPF scheduler at all, so it also has to guarantee forward progress without blocking. See scx_ops_enable() and scx_ops_disable_workfn().
- When sched_ext is disabled, static_branches are used to shut down the entry points from hot paths.
v6: - SCX_NR_ONLINE_OPS replaced with SCX_OPI_*_BEGIN/END so that multiple
groups can be expressed. Later CPU hotplug operations are put into their own group.
- SCX_OPS_DISABLING state is replaced with the new bypass mechanism which allows temporarily putting the system into simple FIFO scheduling mode bypassing the BPF scheduler. In addition to the shut down path, this will also be used to isolate the BPF scheduler across PM events. Enabling and disabling the bypass mode requires iterating all runnable tasks. rq->scx.runnable_list addition is moved from the later watchdog patch.
- ops.prep_enable() is replaced with ops.init_task() and ops.enable/disable() are now called whenever the task enters and leaves sched_ext instead of when the task becomes schedulable on sched_ext and stops being so. A new operation - ops.exit_task() - is called when the task stops being schedulable on sched_ext.
- scx_bpf_dispatch() can now be called from ops.select_cpu() too. This removes the need for communicating local dispatch decision made by ops.select_cpu() to ops.enqueue() via per-task storage. SCX_KF_SELECT_CPU is added to support the change.
- SCX_TASK_ENQ_LOCAL which told the BPF scheudler that scx_select_cpu_dfl() wants the task to be dispatched to the local DSQ was removed. Instead, scx_bpf_select_cpu_dfl() now dispatches directly if it finds a suitable idle CPU. If such behavior is not desired, users can use scx_bpf_select_cpu_dfl() which returns the verdict in a bool out param.
- scx_select_cpu_dfl() was mishandling WAKE_SYNC and could end up queueing many tasks on a local DSQ which makes tasks to execute in order while other CPUs stay idle which made some hackbench numbers really bad. Fixed.
- The current state of sched_ext can now be monitored through files under /sys/sched_ext instead of /sys/kernel/debug/sched/ext. This is to enable monitoring on kernels which don't enable debugfs.
- sched_ext wasn't telling BPF that ops.dispatch()'s @prev argument may be NULL and a BPF scheduler which derefs the pointer without checking could crash the kernel. Tell BPF. This is currently a bit ugly. A better way to annotate this is expected in the future.
- scx_exit_info updated to carry pointers to message buffers instead of embedding them directly. This decouples buffer sizes from API so that they can be changed without breaking compatibility.
- exit_code added to scx_exit_info. This is used to indicate different exit conditions on non-error exits and will be used to handle e.g. CPU hotplugs.
- The patch "sched_ext: Allow BPF schedulers to switch all eligible tasks into sched_ext" is folded in and the interface is changed so that partial switching is indicated with a new ops flag %SCX_OPS_SWITCH_PARTIAL. This makes scx_bpf_switch_all() unnecessasry and in turn SCX_KF_INIT. ops.init() is now called with SCX_KF_SLEEPABLE.
- Code reorganized so that only the parts necessary to integrate with the rest of the kernel are in the header files.
- Changes to reflect the BPF and other kernel changes including the addition of bpf_sched_ext_ops.cfi_stubs.
v5: - To accommodate 32bit configs, p->scx.ops_state is now atomic_long_t
instead of atomic64_t and scx_dsp_buf_ent.qseq which uses load_acquire/store_release is now unsigned long instead of u64.
- Fix the bug where bpf_scx_btf_struct_access() was allowing write access to arbitrary fields.
- Distinguish kfuncs which can be called from any sched_ext ops and from anywhere. e.g. scx_bpf_pick_idle_cpu() can now be called only from sched_ext ops.
- Rename "type" to "kind" in scx_exit_info to make it easier to use on languages in which "type" is a reserved keyword.
- Since cff9b2332ab7 ("kernel/sched: Modify initial boot task idle setup"), PF_IDLE is not set on idle tasks which haven't been online yet which made scx_task_iter_next_filtered() include those idle tasks in iterations leading to oopses. Update scx_task_iter_next_filtered() to directly test p->sched_class against idle_sched_class instead of using is_idle_task() which tests PF_IDLE.
- Other updates to match upstream changes such as adding const to set_cpumask() param and renaming check_preempt_curr() to wakeup_preempt().
v4: - SCHED_CHANGE_BLOCK replaced with the previous
sched_deq_and_put_task()/sched_enq_and_set_tsak() pair. This is because upstream is adaopting a different generic cleanup mechanism. Once that lands, the code will be adapted accordingly.
- task_on_scx() used to test whether a task should be switched into SCX, which is confusing. Renamed to task_should_scx(). task_on_scx() now tests whether a task is currently on SCX.
- scx_has_idle_cpus is barely used anymore and replaced with direct check on the idle cpumask.
- SCX_PICK_IDLE_CORE added and scx_pick_idle_cpu() improved to prefer fully idle cores.
- ops.enable() now sees up-to-date p->scx.weight value.
- ttwu_queue path is disabled for tasks on SCX to avoid confusing BPF schedulers expecting ->select_cpu() call.
- Use cpu_smt_mask() instead of topology_sibling_cpumask() like the rest of the scheduler.
v3: - ops.set_weight() added to allow BPF schedulers to track weight changes
without polling p->scx.weight.
- move_task_to_local_dsq() was losing SCX-specific enq_flags when enqueueing the task on the target dsq because it goes through activate_task() which loses the upper 32bit of the flags. Carry the flags through rq->scx.extra_enq_flags.
- scx_bpf_dispatch(), scx_bpf_pick_idle_cpu(), scx_bpf_task_running() and scx_bpf_task_cpu() now use the new KF_RCU instead of KF_TRUSTED_ARGS to make it easier for BPF schedulers to call them.
- The kfunc helper access control mechanism implemented through sched_ext_entity.kf_mask is improved. Now SCX_CALL_OP*() is always used when invoking scx_ops operations.
v2: - balance_scx_on_up() is dropped. Instead, on UP, balance_scx() is
called from put_prev_taks_scx() and pick_next_task_scx() as necessary. To determine whether balance_scx() should be called from put_prev_task_scx(), SCX_TASK_DEQD_FOR_SLEEP flag is added. See the comment in put_prev_task_scx() for details.
- sched_deq_and_put_task() / sched_enq_and_set_task() sequences replaced with SCHED_CHANGE_BLOCK().
- Unused all_dsqs list removed. This was a left-over from previous iterations.
- p->scx.kf_mask is added to track and enforce which kfunc helpers are allowed. Also, init/exit sequences are updated to make some kfuncs always safe to call regardless of the current BPF scheduler state. Combined, this should make all the kfuncs safe.
- BPF now supports sleepable struct_ops operations. Hacky workaround removed and operations and kfunc helpers are tagged appropriately.
- BPF now supports bitmask / cpumask helpers. scx_bpf_get_idle_cpumask() and friends are added so that BPF schedulers can use the idle masks with the generic helpers. This replaces the hacky kfunc helpers added by a separate patch in V1.
- CONFIG_SCHED_CLASS_EXT can no longer be enabled if SCHED_CORE is enabled. This restriction will be removed by a later patch which adds core-sched support.
- Add MAINTAINERS entries and other misc changes.
--- MAINTAINERS | 13 + include/asm-generic/vmlinux.lds.h | 1 + include/linux/sched.h | 5 + include/linux/sched/ext.h | 141 +- include/uapi/linux/sched.h | 1 + init/init_task.c | 11 + kernel/Kconfig.preempt | 22 +- kernel/sched/build_policy.c | 7 + kernel/sched/core.c | 70 +- kernel/sched/debug.c | 3 + kernel/sched/ext.c | 4238 +++++++++++++++++++++++++++++ kernel/sched/ext.h | 82 +- kernel/sched/sched.h | 17 + 13 files changed, 4603 insertions(+), 8 deletions(-) create mode 100644 kernel/sched/ext.c
MAINTAINERS +13 -0
@@ -19635,6 +19635,19 @@ F: include/linux/wait.h F: include/uapi/linux/sched.h F: kernel/sched/ +SCHEDULER - SCHED_EXT +R: Tejun Heo <[email protected]> +R: David Vernet <[email protected]> +L: [email protected] +S: Maintained +W: https://github.com/sched-ext/scx +T: git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git +F: include/linux/sched/ext.h +F: kernel/sched/ext.h +F: kernel/sched/ext.c +F: tools/sched_ext/ +F: tools/testing/selftests/sched_ext + SCSI LIBSAS SUBSYSTEM R: John Garry <[email protected]> R: Jason Yan <[email protected]>
include/asm-generic/vmlinux.lds.h +1 -0
@@ -131,6 +131,7 @@ *(__dl_sched_class) \ *(__rt_sched_class) \ *(__fair_sched_class) \ + *(__ext_sched_class) \ *(__idle_sched_class ) \ __sched_class_lowest = .;
include/linux/sched.h +5 -0
@@ -80,6 +80,8 @@ struct task_group;struct task_struct ;struct user_event_mm ; +#include <linux/sched/ext.h> +/* * Task state bitmask. NOTE! These bits are also * encoded in fs/proc/array.c: get_task_state().
@@ -798,6 +800,9 @@ struct task_struct {struct sched_rt_entity rt;struct sched_dl_entity dl;struct sched_dl_entity *dl_server; +#ifdef CONFIG_SCHED_CLASS_EXT +struct sched_ext_entity scx; +#endif const struct sched_class *sched_class;#ifdef CONFIG_SCHED_CORE
include/linux/sched/ext.h +140 -1
149 lines not on this page · raw message
include/uapi/linux/sched.h +1 -0
@@ -118,6 +118,7 @@ struct clone_args {/* SCHED_ISO: reserved but not implemented yet */ #define SCHED_IDLE 5#define SCHED_DEADLINE 6 +#define SCHED_EXT 7/* Can be ORed in to make sure the process is reverted back to SCHED_NORMAL on fork */ #define SCHED_RESET_ON_FORK 0x40000000
init/init_task.c +11 -0
@@ -6,6 +6,7 @@#include <linux/sched/sysctl.h> #include <linux/sched/rt.h> #include <linux/sched/task.h> +#include <linux/sched/ext.h> #include <linux/init.h> #include <linux/fs.h> #include <linux/mm.h>
@@ -97,6 +98,16 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {#endif #ifdef CONFIG_CGROUP_SCHED .sched_task_group = &root_task_group, +#endif +#ifdef CONFIG_SCHED_CLASS_EXT + .scx = { + .dsq_node = LIST_HEAD_INIT(init_task.scx.dsq_node), + .sticky_cpu = -1, + .holding_cpu = -1, + .runnable_node = LIST_HEAD_INIT(init_task.scx.runnable_node), + .ddsp_dsq_id = SCX_DSQ_INVALID, + .slice = SCX_SLICE_DFL, + }, #endif .ptraced = LIST_HEAD_INIT(init_task.ptraced), .ptrace_entry = LIST_HEAD_INIT(init_task.ptrace_entry),
kernel/Kconfig.preempt +21 -1
@@ -133,4 +133,24 @@ config SCHED_CORE which is the likely usage by Linux distributions, there should be no measurable impact on performance. - +config SCHED_CLASS_EXT + bool "Extensible Scheduling Class" + depends on BPF_SYSCALL && BPF_JIT && !SCHED_CORE + help + This option enables a new scheduler class sched_ext (SCX), which + allows scheduling policies to be implemented as BPF programs to + achieve the following: + + - Ease of experimentation and exploration: Enabling rapid + iteration of new scheduling policies. + - Customization: Building application-specific schedulers which + implement policies that are not applicable to general-purpose + schedulers. + - Rapid scheduler deployments: Non-disruptive swap outs of + scheduling policies in production environments. + + sched_ext leverages BPF’s struct_ops feature to define a structure + which exports function callbacks and flags to BPF programs that + wish to implement scheduling policies. The struct_ops structure + exported by sched_ext is struct sched_ext_ops, and is conceptually + similar to struct sched_class.
kernel/sched/build_policy.c +7 -0
@@ -21,13 +21,17 @@#include <linux/cpuidle.h> #include <linux/jiffies.h> +#include <linux/kobject.h> #include <linux/livepatch.h> +#include <linux/pm.h> #include <linux/psi.h> +#include <linux/seq_buf.h> #include <linux/seqlock_api.h> #include <linux/slab.h> #include <linux/suspend.h> #include <linux/tsacct_kern.h> #include <linux/vtime.h> +#include <linux/percpu-rwsem.h> #include <uapi/linux/sched/types.h>
@@ -52,3 +56,6 @@#include "cputime.c" #include "deadline.c" +#ifdef CONFIG_SCHED_CLASS_EXT +# include "ext.c" +#endif
kernel/sched/core.c +67 -3
127 lines not on this page · raw message
kernel/sched/debug.c +3 -0
@@ -1089,6 +1089,9 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,P (dl.runtime);P (dl.deadline); } +#ifdef CONFIG_SCHED_CLASS_EXT +__PS ("ext.enabled" ,task_on_scx (p)); +#endif #undef PN_SCHEDSTAT#undef P_SCHEDSTAT
kernel/sched/ext.c new file +4238 -0
2,594 lines not on this page · raw message
kernel/sched/ext.h +79 -3
82 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.44.0
Part of the diff is not shown here. View the raw message.
Tejun Heo
maintainerPATCH 15/39 sched_ext: Add scx_simple and scx_example_qmap example schedulers
In reply to Tejun HeoAdd two simple example BPF schedulers - simple and qmap.
* simple: In terms of scheduling, it behaves identical to not having any operation implemented at all. The two operations it implements are only to improve visibility and exit handling. On certain homogeneous configurations, this actually can perform pretty well.
* qmap: A fixed five level priority scheduler to demonstrate queueing PIDs on BPF maps for scheduling. While not very practical, this is useful as a simple example and will be used to demonstrate different features.
v6: - Common header files reorganized and cleaned up. Compat helpers are
added to demonstrate how schedulers can maintain backward compatibility with older kernels while making use of newly added features.
- simple_select_cpu() added to keep track of the number of local dispatches. This is needed because the default ops.select_cpu() implementation is updated to dispatch directly and won't call ops.enqueue().
- Updated to reflect the sched_ext API changes. Switching all tasks is the default behavior now and scx_qmap supports partial switching when -p is specified.
- tools/sched_ext/Kconfig dropped. This will be included in the doc instead.
v5: - Improve Makefile. Build artifects are now collected into a separate
dir which change be changed. Install and help targets are added and clean actually cleans everything.
- MEMBER_VPTR() improved to improve access to structs. ARRAY_ELEM_PTR() and RESIZEABLE_ARRAY() are added to support resizable arrays in .bss.
- Add scx_common.h which provides common utilities to user code such as SCX_BUG[_ON]() and RESIZE_ARRAY().
- Use SCX_BUG[_ON]() to simplify error handling.
v4: - Dropped _example prefix from scheduler names.
v3: - Rename scx_example_dummy to scx_example_simple and restructure a bit
to ease later additions. Comment updates.
- Added declarations for BPF inline iterators. In the future, hopefully, these will be consolidated into a generic BPF header so that they don't need to be replicated here.
v2: - Updated with the generic BPF cpumask helpers.
--- Makefile | 8 +- tools/Makefile | 10 +- tools/sched_ext/.gitignore | 2 + tools/sched_ext/Makefile | 246 ++++++++++++++++ .../sched_ext/include/bpf-compat/gnu/stubs.h | 11 + tools/sched_ext/include/scx/common.bpf.h | 275 ++++++++++++++++++ tools/sched_ext/include/scx/common.h | 71 +++++ tools/sched_ext/include/scx/compat.bpf.h | 53 ++++ tools/sched_ext/include/scx/compat.h | 155 ++++++++++ tools/sched_ext/include/scx/user_exit_info.h | 64 ++++ tools/sched_ext/scx_qmap.bpf.c | 268 +++++++++++++++++ tools/sched_ext/scx_qmap.c | 100 +++++++ tools/sched_ext/scx_simple.bpf.c | 63 ++++ tools/sched_ext/scx_simple.c | 99 +++++++ 14 files changed, 1423 insertions(+), 2 deletions(-) create mode 100644 tools/sched_ext/.gitignore create mode 100644 tools/sched_ext/Makefile create mode 100644 tools/sched_ext/include/bpf-compat/gnu/stubs.h create mode 100644 tools/sched_ext/include/scx/common.bpf.h create mode 100644 tools/sched_ext/include/scx/common.h create mode 100644 tools/sched_ext/include/scx/compat.bpf.h create mode 100644 tools/sched_ext/include/scx/compat.h create mode 100644 tools/sched_ext/include/scx/user_exit_info.h create mode 100644 tools/sched_ext/scx_qmap.bpf.c create mode 100644 tools/sched_ext/scx_qmap.c create mode 100644 tools/sched_ext/scx_simple.bpf.c create mode 100644 tools/sched_ext/scx_simple.c
Makefile +7 -1
@@ -1344,6 +1344,12 @@ ifneq ($(wildcard $(resolve_btfids_O)),) $(Q)$(MAKE) -sC $(srctree)/tools/bpf/resolve_btfids O=$(resolve_btfids_O) clean endif +tools-clean-targets := sched_ext +PHONY += $(tools-clean-targets) +$(tools-clean-targets): + $(Q)$(MAKE) -sC tools $@_clean +tools_clean: $(tools-clean-targets) + # Clear a bunch of variables before executing the submake ifeq ($(quiet),silent_) tools_silent=s
@@ -1513,7 +1519,7 @@ PHONY += $(mrproper-dirs) mrproper $(mrproper-dirs): $(Q)$(MAKE) $(clean)=$(patsubst _mrproper_%,%,$@) -mrproper: clean $(mrproper-dirs) +mrproper: clean $(mrproper-dirs) tools_clean $(call cmd,rmfiles) @find . $(RCS_FIND_IGNORE) \ \( -name '*.rmeta' \) \
tools/Makefile +9 -1
@@ -28,6 +28,7 @@ include scripts/Makefile.include @echo ' pci - PCI tools' @echo ' perf - Linux performance measurement and analysis tool' @echo ' selftests - various kernel selftests' + @echo ' sched_ext - sched_ext example schedulers' @echo ' bootconfig - boot config tool' @echo ' spi - spi tools' @echo ' tmon - thermal monitoring and tuning tool'
@@ -91,6 +92,9 @@ perf: FORCE $(Q)mkdir -p $(PERF_O) . $(Q)$(MAKE) --no-print-directory -C perf O=$(PERF_O) subdir= +sched_ext: FORCE + $(call descend,sched_ext) + selftests: FORCE $(call descend,testing/$@)
@@ -184,6 +188,9 @@ install: acpi_install counter_install cpupower_install gpio_install \ $(Q)mkdir -p $(PERF_O) . $(Q)$(MAKE) --no-print-directory -C perf O=$(PERF_O) subdir= clean +sched_ext_clean: + $(call descend,sched_ext,clean) + selftests_clean: $(call descend,testing/$(@:_clean=),clean)
@@ -213,6 +220,7 @@ clean: acpi_clean counter_clean cpupower_clean hv_clean firewire_clean \ mm_clean bpf_clean iio_clean x86_energy_perf_policy_clean tmon_clean \ freefall_clean build_clean libbpf_clean libsubcmd_clean \ gpio_clean objtool_clean leds_clean wmi_clean pci_clean firmware_clean debugging_clean \ - intel-speed-select_clean tracing_clean thermal_clean thermometer_clean thermal-engine_clean + intel-speed-select_clean tracing_clean thermal_clean thermometer_clean thermal-engine_clean \ + sched_ext_clean .PHONY: FORCE
tools/sched_ext/.gitignore new file +2 -0
@@ -0,0 +1,2 @@ +tools/ +build/
tools/sched_ext/Makefile new file +246 -0
246 lines not on this page · raw message
tools/sched_ext/include/bpf-compat/gnu/stubs.h new file +11 -0
@@ -0,0 +1,11 @@ +/* +* Dummy gnu/stubs.h. clang can end up including /usr/include/gnu/stubs.h when +* compiling BPF files although its content doesn't play any role. The file in +* turn includes stubs-64.h or stubs-32.h depending on whether __x86_64__ is +* defined. When compiling a BPF source, __x86_64__ isn't set and thus +* stubs-32.h is selected. However, the file is not there if the system doesn't +* have 32bit glibc devel package installed leading to a build failure. +* +* The problem is worked around by making this file available in the include +* search paths before the system one when building BPF. +*/
tools/sched_ext/include/scx/common.bpf.h new file +275 -0
275 lines not on this page · raw message
tools/sched_ext/include/scx/common.h new file +71 -0
71 lines not on this page · raw message
tools/sched_ext/include/scx/compat.bpf.h new file +53 -0
53 lines not on this page · raw message
tools/sched_ext/include/scx/compat.h new file +155 -0
155 lines not on this page · raw message
tools/sched_ext/include/scx/user_exit_info.h new file +64 -0
64 lines not on this page · raw message
tools/sched_ext/scx_qmap.bpf.c new file +268 -0
268 lines not on this page · raw message
tools/sched_ext/scx_qmap.c new file +100 -0
100 lines not on this page · raw message
tools/sched_ext/scx_simple.bpf.c new file +63 -0
63 lines not on this page · raw message
tools/sched_ext/scx_simple.c new file +99 -0
99 lines not on this page · raw message
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 16/39 sched_ext: Add sysrq-S which disables the BPF scheduler
In reply to Tejun HeoThis enables the admin to abort the BPF scheduler and revert to CFS anytime.
--- drivers/tty/sysrq.c | 1 + kernel/sched/build_policy.c | 1 + kernel/sched/ext.c | 20 ++++++++++++++++++++ 3 files changed, 22 insertions(+)
drivers/tty/sysrq.c +1 -0
@@ -520,6 +520,7 @@ static const struct sysrq_key_op *sysrq_key_table[62] = {NULL ,/* P */ NULL ,/* Q */ NULL ,/* R */ +/* S: May be registered by sched_ext for resetting */ NULL ,/* S */ NULL ,/* T */ NULL, /* U */
kernel/sched/build_policy.c +1 -0
@@ -31,6 +31,7 @@#include <linux/suspend.h> #include <linux/tsacct_kern.h> #include <linux/vtime.h> +#include <linux/sysrq.h> #include <linux/percpu-rwsem.h> #include <uapi/linux/sched/types.h>
kernel/sched/ext.c +20 -0
@@ -20,6 +20,7 @@ enum scx_exit_kind {SCX_EXIT_UNREG =64 ,/* user-space initiated unregistration */ SCX_EXIT_UNREG_BPF ,/* BPF-initiated unregistration */ SCX_EXIT_UNREG_KERN ,/* kernel-initiated unregistration */ +SCX_EXIT_SYSRQ ,/* requested by 'S' sysrq */ SCX_EXIT_ERROR =1024 ,/* runtime error, error msg contains details */ SCX_EXIT_ERROR_BPF, /* ERROR but triggered through scx_bpf_error() */
@@ -2767,6 +2768,8 @@ static const char *scx_exit_reason(enum scx_exit_kind kind)return "Scheduler unregistered from BPF" ;case SCX_EXIT_UNREG_KERN :return "Scheduler unregistered from the main kernel" ; +case SCX_EXIT_SYSRQ : +return "disabled by sysrq-S" ;case SCX_EXIT_ERROR :return "runtime error" ;case SCX_EXIT_ERROR_BPF :
@@ -3506,6 +3509,21 @@ static struct bpf_struct_ops bpf_sched_ext_ops = {* System integration and init. */ +static void sysrq_handle_sched_ext_reset (u8 key) +{ +if (scx_ops_helper) +scx_ops_disable (SCX_EXIT_SYSRQ ); +else +pr_info ("sched_ext: BPF scheduler not yet used\n" ); +} + +static const struct sysrq_key_op sysrq_sched_ext_reset_op = { + .handler = sysrq_handle_sched_ext_reset, + .help_msg ="reset-sched-ext(S)" , + .action_msg ="Disable sched_ext and revert all tasks to CFS" , + .enable_mask =SYSRQ_ENABLE_RTNICE , +}; +void __init init_sched_ext_class (void ) {s32 cpu, v;
@@ -3529,6 +3547,8 @@ void __init init_sched_ext_class(void)init_dsq (&rq->scx.local_dsq,SCX_DSQ_LOCAL );INIT_LIST_HEAD (&rq->scx.runnable_list); } + +register_sysrq_key ('S' , &sysrq_sched_ext_reset_op); }
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 17/39 sched_ext: Implement runnable task stall watchdog
In reply to Tejun HeoFrom: David Vernet <[email protected]>
The most common and critical way that a BPF scheduler can misbehave is by failing to run runnable tasks for too long. This patch implements a watchdog.
* All tasks record when they become runnable.
* A watchdog work periodically scans all runnable tasks. If any task has stayed runnable for too long, the BPF scheduler is aborted.
* scheduler_tick() monitors whether the watchdog itself is stuck. If so, the BPF scheduler is aborted.
Because the watchdog only scans the tasks which are currently runnable and usually very infrequently, the overhead should be negligible. scx_qmap is updated so that it can be told to stall user and/or kernel tasks.
A detected task stall looks like the following:
sched_ext: BPF scheduler "qmap" errored, disabling
sched_ext: runnable task stall (dbus-daemon[953] failed to run for 6.478s)
scx_check_timeout_workfn+0x10e/0x1b0
process_one_work+0x287/0x560
worker_thread+0x234/0x420
kthread+0xe9/0x100
ret_from_fork+0x1f/0x30A detected watchdog stall:
sched_ext: BPF scheduler "qmap" errored, disabling
sched_ext: runnable task stall (watchdog failed to check in for 5.001s)
scheduler_tick+0x2eb/0x340
update_process_times+0x7a/0x90
tick_sched_timer+0xd8/0x130
__hrtimer_run_queues+0x178/0x3b0
hrtimer_interrupt+0xfc/0x390
__sysvec_apic_timer_interrupt+0xb7/0x2b0
sysvec_apic_timer_interrupt+0x90/0xb0
asm_sysvec_apic_timer_interrupt+0x1b/0x20
default_idle+0x14/0x20
arch_cpu_idle+0xf/0x20
default_idle_call+0x50/0x90
do_idle+0xe8/0x240
cpu_startup_entry+0x1d/0x20
kernel_init+0x0/0x190
start_kernel+0x0/0x392
start_kernel+0x324/0x392
x86_64_start_reservations+0x2a/0x2c
x86_64_start_kernel+0x104/0x109
secondary_startup_64_no_verify+0xce/0xdbNote that this patch exposes scx_ops_error[_type]() in kernel/sched/ext.h to inline scx_notify_sched_tick().
v4: - While disabling, cancel_delayed_work_sync(&scx_watchdog_work) was
being called before forward progress was guaranteed and thus could lead to system lockup. Relocated.
- While enabling, it was comparing msecs against jiffies without conversion leading to spurious load failures on lower HZ kernels. Fixed.
- runnable list management is now used by core bypass logic and moved to the patch implementing sched_ext core.
v3: - bpf_scx_init_member() was incorrectly comparing ops->timeout_ms
against SCX_WATCHDOG_MAX_TIMEOUT which is in jiffies without conversion leading to spurious load failures in lower HZ kernels. Fixed.
v2: - Julia Lawall noticed that the watchdog code was mixing msecs and
jiffies. Fix by using jiffies for everything.
--- include/linux/sched/ext.h | 1 + init/init_task.c | 1 + kernel/sched/core.c | 1 + kernel/sched/ext.c | 130 ++++++++++++++++++++++++++++++++- kernel/sched/ext.h | 2 + tools/sched_ext/scx_qmap.bpf.c | 12 +++ tools/sched_ext/scx_qmap.c | 12 ++- 7 files changed, 153 insertions(+), 6 deletions(-)
include/linux/sched/ext.h +1 -0
@@ -122,6 +122,7 @@ struct sched_ext_entity {atomic_long_t ops_state;struct list_head runnable_node;/* rq->scx.runnable_list */ +unsigned long runnable_at;u64 ddsp_dsq_id;u64 ddsp_enq_flags;
init/init_task.c +1 -0
@@ -105,6 +105,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { .sticky_cpu = -1, .holding_cpu = -1, .runnable_node = LIST_HEAD_INIT(init_task.scx.runnable_node), + .runnable_at = INITIAL_JIFFIES, .ddsp_dsq_id = SCX_DSQ_INVALID, .slice = SCX_SLICE_DFL, },
kernel/sched/core.c +1 -0
@@ -5733,6 +5733,7 @@ void scheduler_tick(void)calc_global_load_tick (rq);sched_core_tick (rq);task_tick_mm_cid (rq, curr); +scx_tick (rq);rq_unlock (rq, &rf);
kernel/sched/ext.c +126 -4
236 lines not on this page · raw message
kernel/sched/ext.h +2 -0
@@ -29,6 +29,7 @@ static inline bool task_on_scx(const struct task_struct *p)return scx_enabled () && p->sched_class == &ext_sched_class; } +void scx_tick (struct rq *rq);void init_scx_entity (struct sched_ext_entity *scx);void scx_pre_fork (struct task_struct *p);int scx_fork (struct task_struct *p);
@@ -75,6 +76,7 @@ static inline const struct sched_class *next_active_class(const struct sched_cla#define scx_enabled () false#define scx_switched_all () false +static inline void scx_tick (struct rq *rq) {}static inline void scx_pre_fork (struct task_struct *p) {}static inline int scx_fork (struct task_struct *p) {return 0 ; }static inline void scx_post_fork (struct task_struct *p) {}
tools/sched_ext/scx_qmap.bpf.c +12 -0
@@ -29,6 +29,8 @@ enum consts { char _license[] SEC("license") = "GPL";const volatile u64 slice_ns =SCX_SLICE_DFL ; +const volatile u32 stall_user_nth; +const volatile u32 stall_kernel_nth;const volatile u32 dsp_batch;const volatile bool switch_partial;
@@ -130,11 +132,20 @@ static int weight_to_idx(u32 weight)void BPF_STRUCT_OPS (qmap_enqueue ,struct task_struct *p,u64 enq_flags) { +static u32 user_cnt, kernel_cnt;struct task_ctx *tctx;u32 pid = p->pid;int idx =weight_to_idx (p->scx.weight);void *ring; +if (p->flags &PF_KTHREAD ) { +if (stall_kernel_nth && !(++kernel_cnt % stall_kernel_nth)) +return ; + }else { +if (stall_user_nth && !(++user_cnt % stall_user_nth)) +return ; + } +if (test_error_cnt && !--test_error_cnt)scx_bpf_error ("test triggering error" );
@@ -265,4 +276,5 @@ SCX_OPS_DEFINE(qmap_ops, .init_task = (void *)qmap_init_task, .init = (void *)qmap_init, .exit = (void *)qmap_exit, + .timeout_ms = 5000U, .name = "qmap");
tools/sched_ext/scx_qmap.c +10 -2
@@ -19,10 +19,12 @@ const char help_fmt[] ="\n" "See the top-level comment in .bpf.c for more details.\n" "\n" -"Usage: %s [-s SLICE_US] [-e COUNT] [-b COUNT] [-p] [-v]\n" +"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT] [-p] [-v]\n" "\n" " -s SLICE_US Override slice duration\n" " -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n" +" -t COUNT Stall every COUNT'th user thread\n" +" -T COUNT Stall every COUNT'th kernel thread\n" " -b COUNT Dispatch upto COUNT tasks together\n" " -p Switch only tasks on SCHED_EXT policy intead of all\n" " -v Print libbpf debug messages\n"
@@ -55,7 +57,7 @@ int main(int argc, char **argv) skel =SCX_OPS_OPEN (qmap_ops, scx_qmap); -while ((opt =getopt (argc, argv,"s:e:b:pvh" )) !=-1 ) { +while ((opt =getopt (argc, argv,"s:e:t:T:b:pvh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -63,6 +65,12 @@ int main(int argc, char **argv)case 'e' : skel->bss->test_error_cnt =strtoul (optarg,NULL ,0 );break ; +case 't' : + skel->rodata->stall_user_nth =strtoul (optarg,NULL ,0 ); +break ; +case 'T' : + skel->rodata->stall_kernel_nth =strtoul (optarg,NULL ,0 ); +break ;case 'b' : skel->rodata->dsp_batch =strtoul (optarg,NULL ,0 );break ;
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 18/39 sched_ext: Allow BPF schedulers to disallow specific tasks from joining SCHED_EXT
In reply to Tejun HeoBPF schedulers might not want to schedule certain tasks - e.g. kernel threads. This patch adds p->scx.disallow which can be set by BPF schedulers in such cases. The field can be changed anytime and setting it in ops.prep_enable() guarantees that the task can never be scheduled by sched_ext.
scx_qmap is updated with the -d option to disallow a specific PID:
# echo $$ 1092 # grep -E '(policy)|(ext\.enabled)' /proc/self/sched policy : 0 ext.enabled : 0 # ./set-scx 1092 # grep -E '(policy)|(ext\.enabled)' /proc/self/sched policy : 7 ext.enabled : 0
Run "scx_qmap -d 1092" in another terminal.
# cat /sys/kernel/sched_ext/nr_rejected 1 # grep -E '(policy)|(ext\.enabled)' /proc/self/sched policy : 0 ext.enabled : 1 # ./set-scx 1092 setparam failed for 1092 (Permission denied)
- v3: Update description to reflect /sys/kernel/sched_ext interface change.
- v2: Use atomic_long_t instead of atomic64_t for scx_kick_cpus_pnt_seqs to accommodate 32bit archs.
--- include/linux/sched/ext.h | 12 ++++++++ kernel/sched/core.c | 4 +++ kernel/sched/ext.c | 50 ++++++++++++++++++++++++++++++++++ kernel/sched/ext.h | 2 ++ tools/sched_ext/scx_qmap.bpf.c | 4 +++ tools/sched_ext/scx_qmap.c | 11 ++++++-- 6 files changed, 81 insertions(+), 2 deletions(-)
include/linux/sched/ext.h +12 -0
@@ -137,6 +137,18 @@ struct sched_ext_entity {*/ u64 slice; +/* +* If set, reject future sched_setscheduler(2) calls updating the policy +* to %SCHED_EXT with -%EACCES. +* +* If set from ops.init_task() and the task's policy is already +* %SCHED_EXT, which can happen while the BPF scheduler is being loaded +* or by inhering the parent's policy during fork, the task's policy is +* rejected and forcefully reverted to %SCHED_NORMAL. The number of +* such events are reported through /sys/kernel/debug/sched_ext::nr_rejected. +*/ +bool disallow;/* reject switching into SCX */ +/* cold fields */ /* must be the last field, see init_scx_entity() */ struct list_head tasks_node;
kernel/sched/core.c +4 -0
@@ -7866,6 +7866,10 @@ static int __sched_setscheduler(struct task_struct *p, goto unlock; } + retval =scx_check_setscheduler (p, policy); +if (retval) + goto unlock; +/* * If not changing anything there's no need to proceed further, * but store a possible modification of reset_on_fork.
kernel/sched/ext.c +50 -0
95 lines not on this page · raw message
kernel/sched/ext.h +2 -0
@@ -35,6 +35,7 @@ void scx_pre_fork(struct task_struct *p);int scx_fork (struct task_struct *p);void scx_post_fork (struct task_struct *p);void scx_cancel_fork (struct task_struct *p); +int scx_check_setscheduler (struct task_struct *p,int policy);bool task_should_scx (struct task_struct *p);void init_sched_ext_class (void );
@@ -81,6 +82,7 @@ static inline void scx_pre_fork(struct task_struct *p) {}static inline int scx_fork (struct task_struct *p) {return 0 ; }static inline void scx_post_fork (struct task_struct *p) {}static inline void scx_cancel_fork (struct task_struct *p) {} +static inline int scx_check_setscheduler (struct task_struct *p,int policy) {return 0 ; }static inline bool task_on_scx (const struct task_struct *p) {return false; }static inline void init_sched_ext_class (void ) {}static inline u32 scx_cpuperf_target (s32 cpu) {return 0 ; }
tools/sched_ext/scx_qmap.bpf.c +4 -0
@@ -32,6 +32,7 @@ const volatile u64 slice_ns = SCX_SLICE_DFL;const volatile u32 stall_user_nth;const volatile u32 stall_kernel_nth;const volatile u32 dsp_batch; +const volatile s32 disallow_tgid;const volatile bool switch_partial;u32 test_error_cnt;
@@ -244,6 +245,9 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)s32 BPF_STRUCT_OPS (qmap_init_task ,struct task_struct *p,struct scx_init_task_args *args) { +if (p->tgid == disallow_tgid) + p->scx.disallow = true; +/* * @p is new. Let's ensure that its task_ctx is available. We can sleep * in this function and the following will automatically use GFP_KERNEL.
tools/sched_ext/scx_qmap.c +9 -2
@@ -19,13 +19,15 @@ const char help_fmt[] ="\n" "See the top-level comment in .bpf.c for more details.\n" "\n" -"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT] [-p] [-v]\n" +"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT]\n" +" [-d PID] [-p] [-v]\n" "\n" " -s SLICE_US Override slice duration\n" " -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n" " -t COUNT Stall every COUNT'th user thread\n" " -T COUNT Stall every COUNT'th kernel thread\n" " -b COUNT Dispatch upto COUNT tasks together\n" +" -d PID Disallow a process from switching into SCHED_EXT (-1 for self)\n" " -p Switch only tasks on SCHED_EXT policy intead of all\n" " -v Print libbpf debug messages\n" " -h Display this help and exit\n" ;
@@ -57,7 +59,7 @@ int main(int argc, char **argv) skel =SCX_OPS_OPEN (qmap_ops, scx_qmap); -while ((opt =getopt (argc, argv,"s:e:t:T:b:pvh" )) !=-1 ) { +while ((opt =getopt (argc, argv,"s:e:t:T:b:d:pvh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -74,6 +76,11 @@ int main(int argc, char **argv)case 'b' : skel->rodata->dsp_batch =strtoul (optarg,NULL ,0 );break ; +case 'd' : + skel->rodata->disallow_tgid =strtol (optarg,NULL ,0 ); +if (skel->rodata->disallow_tgid <0 ) + skel->rodata->disallow_tgid =getpid (); +break ;case 'p' : skel->rodata->switch_partial = true; skel->struct_ops.qmap_ops->flags |= __COMPAT_SCX_OPS_SWITCH_PARTIAL;
Signature · 1 line
2.44.0
Peter Zijlstra
maintainerRe: [PATCH 18/39] sched_ext: Allow BPF schedulers to disallow specific tasks from joining SCHED_EXT
Quoted 5 lines · Tejun Heo
BPF schedulers might not want to schedule certain tasks - e.g. kernel threads. This patch adds
p->scx.disallowwhich can be set by BPF schedulers in such cases. The field can be changed anytime and setting it inops.prep_enable()guarantees that the task can never be scheduled bysched_ext.
Why ?!?!
By leaving kernel threads fair, and fair sitting above the BPF thing, it is not dissimilar to promoting them to FIFO. They will instantly preempt the BPF thing and keep running for as long as they need. The only real difference between this and actual FIFO is the behaviour on contention.
This seems like a very bad thing to have, and your 'changelog' has no justification what so ever.
Tejun Heo
maintainerRe: [PATCH 18/39] sched_ext: Allow BPF schedulers to disallow specific tasks from joining SCHED_EXT
Hello, Peter.
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Tejun Heo
BPF schedulers might not want to schedule certain tasks - e.g. kernel threads. This patch adds
p->scx.disallowwhich can be set by BPF schedulers in such cases. The field can be changed anytime and setting it inops.prep_enable()guarantees that the task can never be scheduled bysched_ext.
Quoted 7 lines
Why ?!?!
By leaving kernel threads fair, and fair sitting above the BPF thing, it is not dissimilar to promoting them to FIFO. They will instantly preempt the BPF thing and keep running for as long as they need. The only real difference between this and actual FIFO is the behaviour on contention.
Yes, from sched_ext's POV, in partial mode, CFS isn't all that different from FIFO. Whenever there are tasks to run in CFS, CPUs are taken away. Right now, partial mode can be useful for leaving a part of system on CFS (e.g. in a cpuset partitioned system), when the scheduler is narrowly focused and doesn't cover everything necessary (e.g. EAS).
Quoted 2 lines
This seems like a very bad thing to have, and your 'changelog' has no justification what so ever.
This is a bit of duplicate interface in that in partial mode sched_ext can already be opted in by setting per-thread sched class. However, some use cases wanted this so that the BPF scheduler has the final say over who can be on it rather than the userspace. It's a convenience feature for some use cases.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 19/39 sched_ext: Print sched_ext info when dumping stack
In reply to Tejun HeoFrom: David Vernet <[email protected]>
It would be useful to see what the sched_ext scheduler state is, and what scheduler is running, when we're dumping a task's stack. This patch therefore adds a new print_scx_info() function that's called in the same context as print_worker_info() and print_stop_info(). An example dump follows.
BUG: kernel NULL pointer dereference, address: 0000000000000999 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 0 P4D 0 Oops: 0002 [#1] PREEMPT SMP CPU: 13 PID: 2047 Comm: insmod Tainted: G O 6.6.0-work-10323-gb58d4cae8e99-dirty #34 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS unknown 2/2/2022 Sched_ext: qmap (enabled+all), task: runnable_at=-17ms RIP: 0010:init_module+0x9/0x1000 [test_module] ...
v3: - scx_ops_enable_state_str[] definition moved to an earlier patch as
it's now used by core implementation.
- Convert jiffy delta to msecs using jiffies_to_msecs() instead of multiplying by (HZ / MSEC_PER_SEC). The conversion is implemented in jiffies_delta_msecs().
v2: - We are now using scx_ops_enable_state_str[] outside
CONFIG_SCHED_DEBUG. Move it outside of CONFIG_SCHED_DEBUG and to the top. This was reported by Changwoo and Andrea.
--- include/linux/sched/ext.h | 2 ++ kernel/sched/core.c | 1 + kernel/sched/ext.c | 53 +++++++++++++++++++++++++++++++++++++++ lib/dump_stack.c | 1 + 4 files changed, 57 insertions(+)
include/linux/sched/ext.h +2 -0
@@ -155,10 +155,12 @@ struct sched_ext_entity { };void sched_ext_free (struct task_struct *p); +void print_scx_info (const char *log_lvl,struct task_struct *p);#else /* !CONFIG_SCHED_CLASS_EXT */ static inline void sched_ext_free (struct task_struct *p) {} +static inline void print_scx_info (const char *log_lvl,struct task_struct *p) {}#endif /* CONFIG_SCHED_CLASS_EXT */ #endif /* _LINUX_SCHED_EXT_H */
kernel/sched/core.c +1 -0
@@ -9272,6 +9272,7 @@ void sched_show_task(struct task_struct *p)print_worker_info (KERN_INFO , p);print_stop_info (KERN_INFO , p); +print_scx_info (KERN_INFO , p);show_stack (p,NULL ,KERN_INFO );put_task_stack (p); }
kernel/sched/ext.c +53 -0
65 lines not on this page · raw message
lib/dump_stack.c +1 -0
@@ -68,6 +68,7 @@ void dump_stack_print_info(const char *log_lvl)print_worker_info (log_lvl, current);print_stop_info (log_lvl, current); +print_scx_info (log_lvl, current); }/**
Signature · 1 line
2.44.0
Peter Zijlstra
maintainerRe: [PATCH 19/39] sched_ext: Print sched_ext info when dumping stack
kernel/sched/ext.c static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind, · line 586
Quoted 7 lines · Tejun Heo
kernel/sched/ext.c +7
-0+static long jiffies_delta_msecs (unsigned long at,unsigned long now) +{ +if (time_after (at, now)) +return jiffies_to_msecs (at - now); +else +return -(long )jiffies_to_msecs (now - at); +}
You have this weird superfluous else:
if () return foo; else return bar;
pattern in multiple patches, please change that to:
if () return foo; return bar;
Throughout the series.
Also, if we consider 2s complement, does the above actually make sense?
Re: [PATCH 19/39] sched_ext: Print sched_ext info when dumping stack
Quoted 2 lines · Peter Zijlstra
Also, if we consider 2s complement, does the above actually make sense?
jiffies_to_msecs() does not work on 2s complement. It just takes 'unsigned long', and just considers large positive numbers to be exactly that - not a negative jiffy.
We have a separate jiffies_delta_to_msecs(), but that clamps to zero. So again, it doesn't actually work for negative jiffies.
Side note: jiffies_to_msecs() seems to return an "unsigned int". That seems a bit odd.
Linus
Tejun Heo
maintainerRe: [PATCH 19/39] sched_ext: Print sched_ext info when dumping stack
In reply to Peter ZijlstraHello, Peter.
Quoted 17 lines · Peter Zijlstra
kernel/sched/ext.c static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind, · line 586
Quoted 7 lines · Tejun Heo
kernel/sched/ext.c +7
-0+static long jiffies_delta_msecs (unsigned long at,unsigned long now) +{ +if (time_after (at, now)) +return jiffies_to_msecs (at - now); +else +return -(long )jiffies_to_msecs (now - at); +}You have this weird superfluous else:
if () return foo; else return bar;
Quoted 7 lines
pattern in multiple patches, please change that to:
if () return foo; return bar;Throughout the series.
It's just my personal preference. When the if and else blocks are relatively short and completely symmetric like the above case, using exlicit if/else structure looks more immediately intuitive to me. If it isn't too bothersome to everybody, I'd like to keep it that way.
Quoted 1 line
Also, if we consider 2s complement, does the above actually make sense?
Maybe we can update jiffifes_to_msecs() to be 2s complement transparent but probably not a good idea given how many are calling it.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 20/39 sched_ext: Print debug dump after an error exit
In reply to Tejun HeoIf a BPF scheduler triggers an error, the scheduler is aborted and the system is reverted to the built-in scheduler. In the process, a lot of information which may be useful for figuring out what happened can be lost.
This patch adds debug dump which captures information which may be useful for debugging including runqueue and runnable thread states at the time of failure. The following shows a debug dump after triggering the watchdog:
# os/work/tools/sched_ext/build/bin/scx_qmap -t 100 enq=0, dsp=0, delta=0, deq=0 enq=21, dsp=21, delta=0, deq=0 enq=96, dsp=96, delta=0, deq=1
DEBUG DUMP
================================================================================
kworker/u16:0[11] triggered exit kind 1026:
runnable task stall (scx_qmap[1524] failed to run for 5.659s)
Backtrace:
scx_watchdog_workfn+0x138/0x1c0
process_scheduled_works+0x245/0x4e0
worker_thread+0x270/0x360
kthread+0xeb/0x110
ret_from_fork+0x36/0x40
ret_from_fork_asm+0x11/0x20 Runqueue states
---------------
CPU 2 : nr_run=1 ops_qseq=34
curr=kworker/u16:0[11] class=ext_sched_class
*R kworker/u16:0[11] +0ms
scx_state/flags=3/0xd ops_state/qseq=0/0
sticky/holding_cpu=-1/-1 dsq_id=(n/a)
cpus=ff
scx_ops_error_irq_workfn+0x25d/0x340
irq_work_run_list+0x7d/0xc0
irq_work_run+0x18/0x30
__sysvec_irq_work+0x38/0x100
sysvec_irq_work+0x69/0x80
asm_sysvec_irq_work+0x1b/0x20
scx_watchdog_workfn+0x15d/0x1c0
process_scheduled_works+0x245/0x4e0
worker_thread+0x270/0x360
kthread+0xeb/0x110
ret_from_fork+0x36/0x40
ret_from_fork_asm+0x11/0x20CPU 7 : nr_run=1 ops_qseq=11
curr=swapper/7[0] class=idle_sched_class
R scx_qmap[1524] -5659ms
scx_state/flags=3/0x9 ops_state/qseq=2/3
sticky/holding_cpu=-1/-1 dsq_id=(n/a)
cpus=ff
common_nsleep+0x34/0x50
__x64_sys_clock_nanosleep+0xd9/0x120
do_syscall_64+0x7e/0x150
entry_SYSCALL_64_after_hwframe+0x46/0x4e================================================================================
EXIT: runnable task stall (scx_qmap[1524] failed to run for 5.659s)
It shows that CPU 2 was running the watchdog when it triggered the error condition and the scx_qmap thread has been queued on CPU 7 for over 5 seconds but failed to run. This dump has proved pretty useful for developing and debugging BPF schedulers.
Currently, it uses fixed 32k buffer and doesn't provide any way for the BPF scheduler to add additional information. These will be improved in the future.
--- kernel/sched/ext.c | 122 ++++++++++++++++++- tools/sched_ext/include/scx/compat.h | 11 +- tools/sched_ext/include/scx/user_exit_info.h | 19 +++ tools/sched_ext/scx_qmap.c | 10 +- tools/sched_ext/scx_simple.c | 2 +- 5 files changed, 155 insertions(+), 9 deletions(-)
kernel/sched/ext.c +119 -3
@@ -12,6 +12,7 @@ enum scx_consts {SCX_EXIT_BT_LEN =64 ,SCX_EXIT_MSG_LEN =1024 , + SCX_EXIT_DUMP_DFL_LEN = 32768, }; enum scx_exit_kind {
@@ -48,6 +49,9 @@ struct scx_exit_info {/* informational message */ char *msg; + +/* debug dump */ +char *dump; };/* sched_ext_ops.flags */
@@ -330,6 +334,12 @@ struct sched_ext_ops {*/ u32 timeout_ms; +/** +* exit_dump_len - scx_exit_info.dump buffer length. If 0, the default +* value of 32768 is used. +*/ +u32 exit_dump_len; +/** * name - BPF scheduler's name *
@@ -2888,12 +2898,13 @@ static void scx_ops_bypass(bool bypass)static void free_exit_info (struct scx_exit_info *ei) { +kfree (ei->dump);kfree (ei->msg);kfree (ei->bt);kfree (ei); } -static struct scx_exit_info *alloc_exit_info (void ) +static struct scx_exit_info *alloc_exit_info (size_t exit_dump_len) {struct scx_exit_info *ei;
@@ -2903,8 +2914,9 @@ static struct scx_exit_info *alloc_exit_info(void) ei->bt =kcalloc (sizeof (ei->bt[0 ]),SCX_EXIT_BT_LEN ,GFP_KERNEL ); ei->msg =kzalloc (SCX_EXIT_MSG_LEN ,GFP_KERNEL ); + ei->dump =kzalloc (exit_dump_len,GFP_KERNEL ); -if (!ei->bt || !ei->msg) { +if (!ei->bt || !ei->msg || !ei->dump) {free_exit_info (ei);return NULL ; }
@@ -3104,8 +3116,101 @@ static void scx_ops_disable(enum scx_exit_kind kind)schedule_scx_ops_disable_work (); } +static void scx_dump_task (struct seq_buf *s,struct task_struct *p,char marker, +unsigned long now) +{ +static unsigned long bt[SCX_EXIT_BT_LEN ]; +char dsq_id_buf[19 ] ="(n/a)" ; +unsigned long ops_state =atomic_long_read (&p->scx.ops_state); +unsigned int bt_len; +size_t avail, used; +char *buf; + +if (p->scx.dsq) +scnprintf (dsq_id_buf,sizeof (dsq_id_buf),"0x%llx" , + (unsigned long long )p->scx.dsq->id); + +seq_buf_printf (s,"\n %c%c %s[%d] %+ldms\n" , + marker,task_state_to_char (p), p->comm, p->pid, +jiffies_delta_msecs (p->scx.runnable_at, now)); +seq_buf_printf (s," scx_state/flags=%u/0x%x ops_state/qseq=%lu/%lu\n" , +scx_get_task_state (p), + p->scx.flags & ~SCX_TASK_STATE_MASK , + ops_state &SCX_OPSS_STATE_MASK , + ops_state >>SCX_OPSS_QSEQ_SHIFT ); +seq_buf_printf (s," sticky/holding_cpu=%d/%d dsq_id=%s\n" , + p->scx.sticky_cpu, p->scx.holding_cpu, dsq_id_buf); +seq_buf_printf (s," cpus=%*pb\n\n" ,cpumask_pr_args (p->cpus_ptr)); + + bt_len =stack_trace_save_tsk (p, bt,SCX_EXIT_BT_LEN ,1 ); + + avail =seq_buf_get_buf (s, &buf); + used =stack_trace_snprint (buf, avail, bt, bt_len,3 ); +seq_buf_commit (s, used < avail ? used :-1 ); +} + +static void scx_dump_state (struct scx_exit_info *ei,size_t dump_len) +{ +const char trunc_marker[] ="\n\n~~~~ TRUNCATED ~~~~\n" ; +unsigned long now = jiffies; +struct seq_buf s; +size_t avail, used; +char *buf; +int cpu; + +if (dump_len <=sizeof (trunc_marker)) +return ; + +seq_buf_init (&s, ei->dump, dump_len -sizeof (trunc_marker)); + +seq_buf_printf (&s,"%s[%d] triggered exit kind %d:\n %s (%s)\n\n" , + current->comm, current->pid, ei->kind, ei->reason, ei->msg); +seq_buf_printf (&s,"Backtrace:\n" ); + avail =seq_buf_get_buf (&s, &buf); + used =stack_trace_snprint (buf, avail, ei->bt, ei->bt_len,1 ); +seq_buf_commit (&s, used < avail ? used :-1 ); + +seq_buf_printf (&s,"\nRunqueue states\n" ); +seq_buf_printf (&s,"---------------\n" ); + +for_each_possible_cpu (cpu) { +struct rq *rq =cpu_rq (cpu); +struct rq_flags rf; +struct task_struct *p; + +rq_lock (rq, &rf); + +if (list_empty (&rq->scx.runnable_list) && + rq->curr->sched_class == &idle_sched_class) + goto next; + +seq_buf_printf (&s,"\nCPU %-4d: nr_run=%u ops_qseq=%lu\n" , + cpu, rq->scx.nr_running, rq->scx.ops_qseq); +seq_buf_printf (&s," curr=%s[%d] class=%ps\n" , + rq->curr->comm, rq->curr->pid, + rq->curr->sched_class); + +if (rq->curr->sched_class == &ext_sched_class) +scx_dump_task (&s, rq->curr,'*' , now); + +list_for_each_entry (p, &rq->scx.runnable_list, scx.runnable_node) +scx_dump_task (&s, p,' ' , now); + next: +rq_unlock (rq, &rf); + } + +if (seq_buf_has_overflowed (&s)) +memcpy (ei->dump +seq_buf_used (&s) -1 , trunc_marker, +sizeof (trunc_marker)); +} +static void scx_ops_error_irq_workfn (struct irq_work *irq_work) { +struct scx_exit_info *ei = scx_exit_info; + +if (ei->kind >=SCX_EXIT_ERROR ) +scx_dump_state (ei, scx_ops.exit_dump_len); +schedule_scx_ops_disable_work (); }
@@ -3131,6 +3236,13 @@ static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind,vscnprintf (ei->msg,SCX_EXIT_MSG_LEN , fmt, args);va_end (args); +/* +* Set ei->kind and ->reason for scx_dump_state(). They'll be set again +* in scx_ops_disable_workfn(). +*/ + ei->kind = kind; + ei->reason =scx_exit_reason (ei->kind); +irq_work_queue (&scx_ops_error_irq_work); }
@@ -3192,7 +3304,7 @@ static int scx_ops_enable(struct sched_ext_ops *ops)if (ret <0 ) goto err; - scx_exit_info =alloc_exit_info (); + scx_exit_info =alloc_exit_info (ops->exit_dump_len);if (!scx_exit_info) { ret = -ENOMEM ; goto err_del;
@@ -3572,6 +3684,10 @@ static int bpf_scx_init_member(const struct btf_type *t,return -E2BIG ; ops->timeout_ms = *(u32 *)(udata + moff);return 1 ; +case offsetof(struct sched_ext_ops , exit_dump_len): + ops->exit_dump_len = + *(u32 *)(udata + moff) ?:SCX_EXIT_DUMP_DFL_LEN ; +return 1 ; }return 0 ;
tools/sched_ext/include/scx/compat.h +9 -2
@@ -126,7 +126,8 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field* and attach it, backward compatibility is automatically maintained where * reasonable. * -* - sched_ext_ops.tick(): Ignored on older kernels with a warning. +* - ops.tick(): Ignored on older kernels with a warning. +* - ops.exit_dump_len: Cleared to zero on older kernels with a warning. */ #define SCX_OPS_OPEN (__ops_name, __scx_name) ({ \ struct __scx_name *__skel; \
@@ -136,7 +137,13 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field __skel; \ }) -#define SCX_OPS_LOAD (__skel, __ops_name, __scx_name) ({ \ +#define SCX_OPS_LOAD (__skel, __ops_name, __scx_name, __uei_name) ({ \ + UEI_SET_SIZE(__skel, __ops_name, __uei_name); \ + if (!__COMPAT_struct_has_field("sched_ext_ops", "exit_dump_len") && \ + (__skel)->struct_ops.__ops_name->exit_dump_len) { \ + fprintf(stderr, "WARNING: kernel doesn't support setting exit dump len\n"); \ + (__skel)->struct_ops.__ops_name->exit_dump_len = 0; \ + } \ if (!__COMPAT_struct_has_field("sched_ext_ops", "tick") && \ (__skel)->struct_ops.__ops_name->tick) { \ fprintf(stderr, "WARNING: kernel doesn't support ops.tick()\n"); \
tools/sched_ext/include/scx/user_exit_info.h +19 -0
@@ -13,6 +13,7 @@enum uei_sizes {UEI_REASON_LEN =128 ,UEI_MSG_LEN =1024 , +UEI_DUMP_DFL_LEN =32768 , };struct user_exit_info {
@@ -28,6 +29,8 @@ struct user_exit_info {#include <bpf/bpf_core_read.h> #define UEI_DEFINE (__name) \ + char RESIZABLE_ARRAY(data, __name##_dump); \ + const volatile u32 __name##_dump_len; \ struct user_exit_info __name SEC(".data")#define UEI_RECORD (__uei_name, __ei) ({ \
@@ -35,6 +38,8 @@ struct user_exit_info { sizeof(__uei_name.reason), (__ei)->reason); \bpf_probe_read_kernel_str (__uei_name.msg, \sizeof (__uei_name.msg), (__ei)->msg); \ + bpf_probe_read_kernel_str(__uei_name##_dump, \ + __uei_name##_dump_len, (__ei)->dump); \if (bpf_core_field_exists ((__ei)->exit_code)) \ __uei_name.exit_code = (__ei)->exit_code; \/* use __sync to force memory barrier */ \
@@ -47,6 +52,13 @@ struct user_exit_info {#include <stdio.h> #include <stdbool.h> +/* no need to call the following explicitly if SCX_OPS_LOAD() is used */ +#define UEI_SET_SIZE (__skel, __ops_name, __uei_name) ({ \ + u32 __len = (__skel)->struct_ops.__ops_name->exit_dump_len ?: UEI_DUMP_DFL_LEN; \ + (__skel)->rodata->__uei_name##_dump_len = __len; \ + RESIZE_ARRAY(data, __uei_name##_dump, __len); \ +}) +#define UEI_EXITED (__skel, __uei_name) ({ \/* use __sync to force memory barrier */ \__sync_val_compare_and_swap (&(__skel)->data->__uei_name.kind,-1 ,-1 ); \
@@ -54,6 +66,13 @@ struct user_exit_info {#define UEI_REPORT (__skel, __uei_name) ({ \ struct user_exit_info *__uei = &(__skel)->data->__uei_name; \ + char *__uei_dump = (__skel)->data_##__uei_name##_dump->__uei_name##_dump; \ + if (__uei_dump[0] != '\0') { \ + fputs("\nDEBUG DUMP\n", stderr); \ + fputs("================================================================================\n\n", stderr); \ + fputs(__uei_dump, stderr); \ + fputs("\n================================================================================\n\n", stderr); \ + } \ fprintf(stderr, "EXIT: %s", __uei->reason); \ if (__uei->msg[0] != '\0') \ fprintf(stderr, " (%s)", __uei->msg); \
tools/sched_ext/scx_qmap.c +7 -3
@@ -20,7 +20,7 @@ const char help_fmt[] ="See the top-level comment in .bpf.c for more details.\n" "\n" "Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT]\n" -" [-d PID] [-p] [-v]\n" +" [-d PID] [-D LEN] [-p] [-v]\n" "\n" " -s SLICE_US Override slice duration\n" " -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
@@ -28,6 +28,7 @@ const char help_fmt[] =" -T COUNT Stall every COUNT'th kernel thread\n" " -b COUNT Dispatch upto COUNT tasks together\n" " -d PID Disallow a process from switching into SCHED_EXT (-1 for self)\n" +" -D LEN Set scx_exit_info.dump buffer length\n" " -p Switch only tasks on SCHED_EXT policy intead of all\n" " -v Print libbpf debug messages\n" " -h Display this help and exit\n" ;
@@ -59,7 +60,7 @@ int main(int argc, char **argv) skel =SCX_OPS_OPEN (qmap_ops, scx_qmap); -while ((opt =getopt (argc, argv,"s:e:t:T:b:d:pvh" )) !=-1 ) { +while ((opt =getopt (argc, argv,"s:e:t:T:b:d:D:pvh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -81,6 +82,9 @@ int main(int argc, char **argv)if (skel->rodata->disallow_tgid <0 ) skel->rodata->disallow_tgid =getpid ();break ; +case 'D' : + skel->struct_ops.qmap_ops->exit_dump_len =strtoul (optarg,NULL ,0 ); +break ;case 'p' : skel->rodata->switch_partial = true; skel->struct_ops.qmap_ops->flags |= __COMPAT_SCX_OPS_SWITCH_PARTIAL;
@@ -94,7 +98,7 @@ int main(int argc, char **argv) } } -SCX_OPS_LOAD (skel, qmap_ops, scx_qmap); +SCX_OPS_LOAD (skel, qmap_ops, scx_qmap, uei); link =SCX_OPS_ATTACH (skel, qmap_ops);while (!exit_req && !UEI_EXITED (skel, uei)) {
tools/sched_ext/scx_simple.c +1 -1
@@ -80,7 +80,7 @@ int main(int argc, char **argv) } } -SCX_OPS_LOAD (skel, simple_ops, scx_simple); +SCX_OPS_LOAD (skel, simple_ops, scx_simple, uei); link =SCX_OPS_ATTACH (skel, simple_ops);while (!exit_req && !UEI_EXITED (skel, uei)) {
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 21/39 tools/sched_ext: Add scx_show_state.py
In reply to Tejun HeoThere are states which are interesting but don't quite fit the interface exposed under /sys/kernel/sched_ext. Add tools/scx_show_state.py to show them.
--- tools/sched_ext/scx_show_state.py | 39 +++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 tools/sched_ext/scx_show_state.py
tools/sched_ext/scx_show_state.py new file +39 -0
@@ -0,0 +1,39 @@ +#!/usr/bin/env drgn +# +# Copyright (C) 2024 Tejun Heo <[email protected]> +# Copyright (C) 2024 Meta Platforms, Inc. and affiliates. + +desc = """ +This is a drgn script to show the current sched_ext state. +For more info on drgn, visit https://github.com/osandov/drgn. +""" + +import drgn +import sys + +def err(s): + print(s, file=sys.stderr, flush=True) + sys.exit(1) + +def read_int(name): + return int(prog[name].value_()) + +def read_atomic(name): + return prog[name].counter.value_() + +def read_static_key(name): + return prog[name].key.enabled.counter.value_() + +def ops_state_str(state): + return prog['scx_ops_enable_state_str'][state].string_().decode() + +ops = prog['scx_ops'] +enable_state = read_atomic("scx_ops_enable_state_var") + +print(f'ops : {ops.name.string_().decode()}') +print(f'enabled : {read_static_key("__scx_ops_enabled")}') +print(f'switching_all : {read_int("scx_switching_all")}') +print(f'switched_all : {read_static_key("__scx_switched_all")}') +print(f'enable_state : {ops_state_str(enable_state)} ({enable_state})') +print(f'bypass_depth : {read_atomic("scx_ops_bypass_depth")}') +print(f'nr_rejected : {read_atomic("scx_nr_rejected")}')
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 22/39 sched_ext: Implement scx_bpf_kick_cpu() and task preemption support
In reply to Tejun HeoIt's often useful to wake up and/or trigger reschedule on other CPUs. This patch adds scx_bpf_kick_cpu() kfunc helper that BPF scheduler can call to kick the target CPU into the scheduling path.
As a sched_ext task relinquishes its CPU only after its slice is depleted, this patch also adds SCX_KICK_PREEMPT and SCX_ENQ_PREEMPT which clears the slice of the target CPU's current task to guarantee that sched_ext's scheduling path runs on the CPU.
If SCX_KICK_IDLE is specified, the target CPU is kicked iff the CPU is idle to guarantee that the target CPU will go through at least one full sched_ext scheduling cycle after the kicking. This can be used to wake up idle CPUs without incurring unnecessary overhead if it isn't currently idle.
As a demonstration of how backward compatibility can be supported using BPF CO-RE, tools/sched_ext/include/scx/compat.bpf.h is added. It provides __COMPAT_scx_bpf_kick_cpu_IDLE() which uses SCX_KICK_IDLE if available or becomes a regular kicking otherwise. This allows schedulers to use the new SCX_KICK_IDLE while maintaining support for older kernels. The plan is to temporarily use compat helpers to ease API updates and drop them after a few kernel releases.
v5: - SCX_KICK_IDLE added. Note that this also adds a compat mechanism for
schedulers so that they can support kernels without SCX_KICK_IDLE. This is useful as a demonstration of how new feature flags can be added in a backward compatible way.
- kick_cpus_irq_workfn() reimplemented so that it touches the pending cpumasks only as necessary to reduce kicking overhead on machines with a lot of CPUs.
- tools/sched_ext/include/scx/compat.bpf.h added.
v4: - Move example scheduler to its own patch.
v3: - Make scx_example_central switch all tasks by default.
- Convert to BPF inline iterators.
v2: - Julia Lawall reported that scx_example_central can overflow the
dispatch buffer and malfunction. As scheduling for other CPUs can't be handled by the automatic retry mechanism, fix by implementing an explicit overflow and retry handling.
- Updated to use generic BPF cpumask helpers.
--- include/linux/sched/ext.h | 4 + kernel/sched/ext.c | 225 +++++++++++++++++++++-- kernel/sched/sched.h | 10 + tools/sched_ext/include/scx/common.bpf.h | 1 + tools/sched_ext/include/scx/compat.bpf.h | 16 ++ 5 files changed, 243 insertions(+), 13 deletions(-)
include/linux/sched/ext.h +4 -0
@@ -134,6 +134,10 @@ struct sched_ext_entity {* scx_bpf_dispatch() but can also be modified directly by the BPF * scheduler. Automatically decreased by SCX as the task executes. On * depletion, a scheduling event is triggered. +* +* This value is cleared to zero if the task is preempted by +* %SCX_KICK_PREEMPT and shouldn't be used to determine how long the +* task ran. Use p->se.sum_exec_runtime instead. */ u64 slice;
kernel/sched/ext.c +212 -13
333 lines not on this page · raw message
kernel/sched/sched.h +10 -0
@@ -709,12 +709,22 @@ struct cfs_rq { };#ifdef CONFIG_SCHED_CLASS_EXT +/* scx_rq->flags, protected by the rq lock */ +enum scx_rq_flags { +SCX_RQ_BALANCING =1 <<0 , +}; +struct scx_rq {struct scx_dispatch_q local_dsq;struct list_head runnable_list;/* runnable tasks on this rq */ unsigned long ops_qseq;u64 extra_enq_flags;/* see move_task_to_local_dsq() */ u32 nr_running; +u32 flags; +cpumask_var_t cpus_to_kick; +cpumask_var_t cpus_to_kick_if_idle; +cpumask_var_t cpus_to_preempt; +struct irq_work kick_cpus_irq_work; };#endif /* CONFIG_SCHED_CLASS_EXT */
tools/sched_ext/include/scx/common.bpf.h +1 -0
@@ -34,6 +34,7 @@ void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flag u32 scx_bpf_dispatch_nr_slots(void) __ksym; void scx_bpf_dispatch_cancel(void) __ksym; bool scx_bpf_consume(u64 dsq_id) __ksym; +void scx_bpf_kick_cpu(s32 cpu, u64 flags) __ksym; s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym; void scx_bpf_destroy_dsq(u64 dsq_id) __ksym; void scx_bpf_exit_bstr(s64 exit_code, char *fmt, unsigned long long *data, u32 data__sz) __ksym __weak;
tools/sched_ext/include/scx/compat.bpf.h +16 -0
@@ -7,6 +7,22 @@#ifndef __SCX_COMPAT_BPF_H#define __SCX_COMPAT_BPF_H +#define __COMPAT_ENUM_OR_ZERO (__type, __ent) \ +({ \ + __type __ret = 0; \ + if (bpf_core_enum_value_exists(__type, __ent)) \ + __ret = __ent; \ + __ret; \ +}) + +/* +* %SCX_KICK_IDLE is a later addition. To support both before and after, use +* %__COMPAT_SCX_KICK_IDLE which becomes 0 on kernels which don't support it. +* Users can use %SCX_KICK_IDLE directly in the future. +*/ +#define __COMPAT_SCX_KICK_IDLE \ + __COMPAT_ENUM_OR_ZERO(enum scx_kick_flags, SCX_KICK_IDLE) +/* * scx_switch_all() was replaced by %SCX_OPS_SWITCH_PARTIAL. See * %__COMPAT_SCX_OPS_SWITCH_PARTIAL in compat.h. This can be dropped in the
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 23/39 sched_ext: Add a central scheduler which makes all scheduling decisions on one CPU
In reply to Tejun HeoThis patch adds a new example scheduler, scx_central, which demonstrates central scheduling where one CPU is responsible for making all scheduling decisions in the system using scx_bpf_kick_cpu(). The central CPU makes scheduling decisions for all CPUs in the system, queues tasks on the appropriate local dsq's and preempts the worker CPUs. The worker CPUs in turn preempt the central CPU when it needs tasks to run.
Currently, every CPU depends on its own tick to expire the current task. A follow-up patch implementing tickless support for sched_ext will allow the worker CPUs to go full tickless so that they can run completely undisturbed.
v3: - Kumar fixed a bug where the dispatch path could overflow the dispatch
buffer if too many are dispatched to the fallback DSQ.
- Use the new SCX_KICK_IDLE to wake up non-central CPUs.
- Dropped '-p' option.
v2: - Use RESIZABLE_ARRAY() instead of fixed MAX_CPUS and use SCX_BUG[_ON]()
to simplify error handling.
--- tools/sched_ext/Makefile | 2 +- tools/sched_ext/scx_central.bpf.c | 214 ++++++++++++++++++++++++++++++ tools/sched_ext/scx_central.c | 105 +++++++++++++++ 3 files changed, 320 insertions(+), 1 deletion(-) create mode 100644 tools/sched_ext/scx_central.bpf.c create mode 100644 tools/sched_ext/scx_central.c
tools/sched_ext/Makefile +1 -1
@@ -176,7 +176,7 @@ $(INCLUDE_DIR)/%.bpf.skel.h: $(SCXOBJ_DIR)/%.bpf.o $(INCLUDE_DIR)/vmlinux.h $(BP SCX_COMMON_DEPS := include/scx/common.h include/scx/user_exit_info.h | $(BINDIR) -c-sched-targets = scx_simple scx_qmap +c-sched-targets = scx_simple scx_qmap scx_central $(addprefix $(BINDIR)/,$(c-sched-targets)): \ $(BINDIR)/%: \
tools/sched_ext/scx_central.bpf.c new file +214 -0
214 lines not on this page · raw message
tools/sched_ext/scx_central.c new file +105 -0
105 lines not on this page · raw message
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 24/39 sched_ext: Make watchdog handle ops.dispatch() looping stall
In reply to Tejun HeoThe dispatch path retries if the local DSQ is still empty after ops.dispatch() either dispatched or consumed a task. This is both out of necessity and for convenience. It has to retry because the dispatch path might lose the tasks to dequeue while the rq lock is released while trying to migrate tasks across CPUs, and the retry mechanism makes ops.dispatch() implementation easier as it only needs to make some forward progress each iteration.
However, this makes it possible for ops.dispatch() to stall CPUs by repeatedly dispatching ineligible tasks. If all CPUs are stalled that way, the watchdog or sysrq handler can't run and the system can't be saved. Let's address the issue by breaking out of the dispatch loop after 32 iterations.
It is unlikely but not impossible for ops.dispatch() to legitimately go over the iteration limit. We want to come back to the dispatch path in such cases as not doing so risks stalling the CPU by idling with runnable tasks pending. As the previous task is still current in balance_scx(), resched_curr() doesn't do anything - it will just get cleared. Let's instead use scx_kick_bpf() which will trigger reschedule after switching to the next task which will likely be the idle task.
--- kernel/sched/ext.c | 17 +++++++++++++++++ tools/sched_ext/scx_qmap.bpf.c | 15 +++++++++++++++ tools/sched_ext/scx_qmap.c | 8 ++++++-- 3 files changed, 38 insertions(+), 2 deletions(-)
kernel/sched/ext.c +17 -0
@@ -8,6 +8,7 @@enum scx_consts {SCX_DSP_DFL_MAX_BATCH =32 , +SCX_DSP_MAX_LOOPS =32 ,SCX_WATCHDOG_MAX_TIMEOUT =30 *HZ ,SCX_EXIT_BT_LEN =64 ,
@@ -598,6 +599,7 @@ static DEFINE_PER_CPU(struct scx_dsp_ctx, scx_dsp_ctx);static struct kset *scx_kset;static struct kobject *scx_root_kobj; +static void scx_bpf_kick_cpu (s32 cpu,u64 flags); static __printf(3, 4) void scx_ops_exit_kind(enum scx_exit_kind kind, s64 exit_code, const char *fmt, ...);
@@ -1840,6 +1842,7 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,struct scx_rq *scx_rq = &rq->scx;struct scx_dsp_ctx *dspc =this_cpu_ptr (&scx_dsp_ctx);bool prev_on_scx = prev->sched_class == &ext_sched_class; +int nr_loops =SCX_DSP_MAX_LOOPS ;bool has_tasks = false;lockdep_assert_rq_held (rq);
@@ -1896,6 +1899,20 @@ static int balance_scx(struct rq *rq, struct task_struct *prev, goto has_tasks;if (consume_dispatch_q (rq, rf, &scx_dsq_global)) goto has_tasks; + +/* +* ops.dispatch() can trap us in this loop by repeatedly +* dispatching ineligible tasks. Break out once in a while to +* allow the watchdog to run. As IRQ can't be enabled in +* balance(), we want to complete this scheduling cycle and then +* start a new one. IOW, we want to call resched_curr() on the +* next, most likely idle, task, not the current one. Use +* scx_bpf_kick_cpu() for deferred kicking. +*/ +if (unlikely (!--nr_loops)) { +scx_bpf_kick_cpu (cpu_of (rq),0 ); +break ; + } }while (dspc->nr_tasks); goto out;
tools/sched_ext/scx_qmap.bpf.c +15 -0
@@ -31,6 +31,7 @@ char _license[] SEC("license") = "GPL";const volatile u64 slice_ns =SCX_SLICE_DFL ;const volatile u32 stall_user_nth;const volatile u32 stall_kernel_nth; +const volatile u32 dsp_inf_loop_after;const volatile u32 dsp_batch;const volatile s32 disallow_tgid;const volatile bool switch_partial;
@@ -198,6 +199,20 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)if (scx_bpf_consume (SHARED_DSQ ))return ; +if (dsp_inf_loop_after && nr_dispatched > dsp_inf_loop_after) { +/* +* PID 2 should be kthreadd which should mostly be idle and off +* the scheduler. Let's keep dispatching it to force the kernel +* to call this function over and over again. +*/ + p =bpf_task_from_pid (2 ); +if (p) { +scx_bpf_dispatch (p,SCX_DSQ_LOCAL , slice_ns,0 ); +bpf_task_release (p); +return ; + } + } +if (!(cpuc =bpf_map_lookup_elem (&cpu_ctx_stor, &zero))) {scx_bpf_error ("failed to look up cpu_ctx" );return ;
tools/sched_ext/scx_qmap.c +6 -2
@@ -19,13 +19,14 @@ const char help_fmt[] ="\n" "See the top-level comment in .bpf.c for more details.\n" "\n" -"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-b COUNT]\n" +"Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n" " [-d PID] [-D LEN] [-p] [-v]\n" "\n" " -s SLICE_US Override slice duration\n" " -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n" " -t COUNT Stall every COUNT'th user thread\n" " -T COUNT Stall every COUNT'th kernel thread\n" +" -l COUNT Trigger dispatch infinite looping after COUNT dispatches\n" " -b COUNT Dispatch upto COUNT tasks together\n" " -d PID Disallow a process from switching into SCHED_EXT (-1 for self)\n" " -D LEN Set scx_exit_info.dump buffer length\n"
@@ -60,7 +61,7 @@ int main(int argc, char **argv) skel =SCX_OPS_OPEN (qmap_ops, scx_qmap); -while ((opt =getopt (argc, argv,"s:e:t:T:b:d:D:pvh" )) !=-1 ) { +while ((opt =getopt (argc, argv,"s:e:t:T:l:b:d:D:pvh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -74,6 +75,9 @@ int main(int argc, char **argv)case 'T' : skel->rodata->stall_kernel_nth =strtoul (optarg,NULL ,0 );break ; +case 'l' : + skel->rodata->dsp_inf_loop_after =strtoul (optarg,NULL ,0 ); +break ;case 'b' : skel->rodata->dsp_batch =strtoul (optarg,NULL ,0 );break ;
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 25/39 sched_ext: Add task state tracking operations
In reply to Tejun HeoBeing able to track the task runnable and running state transitions are useful for a variety of purposes including latency tracking and load factor calculation.
Currently, BPF schedulers don't have a good way of tracking these transitions. Becoming runnable can be determined from ops.enqueue() but becoming quiescent can only be inferred from the lack of subsequent enqueue. Also, as the local dsq can have multiple tasks and some events are handled in the sched_ext core, it's difficult to determine when a given task starts and stops executing.
This patch adds sched_ext_ops.runnable(), .running(), .stopping() and .quiescent() operations to track the task runnable and running state transitions. They're mostly self explanatory; however, we want to ensure that running <-> stopping transitions are always contained within runnable <-> quiescent transitions which is a bit different from how the scheduler core behaves. This adds a bit of complication. See the comment in dequeue_task_scx().
--- kernel/sched/ext.c | 104 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+)
kernel/sched/ext.c +104 -0
@@ -203,6 +203,71 @@ struct sched_ext_ops {*/ void (*tick)(struct task_struct *p); +/** +* runnable - A task is becoming runnable on its associated CPU +* @p: task becoming runnable +* @enq_flags: %SCX_ENQ_* +* +* This and the following three functions can be used to track a task's +* execution state transitions. A task becomes ->runnable() on a CPU, +* and then goes through one or more ->running() and ->stopping() pairs +* as it runs on the CPU, and eventually becomes ->quiescent() when it's +* done running on the CPU. +* +* @p is becoming runnable on the CPU because it's +* +* - waking up (%SCX_ENQ_WAKEUP) +* - being moved from another CPU +* - being restored after temporarily taken off the queue for an +* attribute change. +* +* This and ->enqueue() are related but not coupled. This operation +* notifies @p's state transition and may not be followed by ->enqueue() +* e.g. when @p is being dispatched to a remote CPU. Likewise, a task +* may be ->enqueue()'d without being preceded by this operation e.g. +* after exhausting its slice. +*/ +void (*runnable)(struct task_struct *p,u64 enq_flags); + +/** +* running - A task is starting to run on its associated CPU +* @p: task starting to run +* +* See ->runnable() for explanation on the task state notifiers. +*/ +void (*running)(struct task_struct *p); + +/** +* stopping - A task is stopping execution +* @p: task stopping to run +* @runnable: is task @p still runnable? +* +* See ->runnable() for explanation on the task state notifiers. If +* !@runnable, ->quiescent() will be invoked after this operation +* returns. +*/ +void (*stopping)(struct task_struct *p,bool runnable); + +/** +* quiescent - A task is becoming not runnable on its associated CPU +* @p: task becoming not runnable +* @deq_flags: %SCX_DEQ_* +* +* See ->runnable() for explanation on the task state notifiers. +* +* @p is becoming quiescent on the CPU because it's +* +* - sleeping (%SCX_DEQ_SLEEP) +* - being moved to another CPU +* - being temporarily taken off the queue for an attribute change +* (%SCX_DEQ_SAVE) +* +* This and ->dequeue() are related but not coupled. This operation +* notifies @p's state transition and may not be preceded by ->dequeue() +* e.g. when @p is being dispatched to a remote CPU. +*/ +void (*quiescent)(struct task_struct *p,u64 deq_flags); +/** * yield - Yield CPU * @from: yielding task
@@ -1289,6 +1354,9 @@ static void enqueue_task_scx(struct rq *rq, struct task_struct *p, int enq_flags rq->scx.nr_running++;add_nr_running (rq,1 ); +if (SCX_HAS_OP (runnable)) +SCX_CALL_OP (SCX_KF_REST , runnable, p, enq_flags); +do_enqueue_task (rq, p, enq_flags, sticky_cpu); }
@@ -1350,6 +1418,26 @@ static void dequeue_task_scx(struct rq *rq, struct task_struct *p, int deq_flagsops_dequeue (p, deq_flags); +/* +* A currently running task which is going off @rq first gets dequeued +* and then stops running. As we want running <-> stopping transitions +* to be contained within runnable <-> quiescent transitions, trigger +* ->stopping() early here instead of in put_prev_task_scx(). +* +* @p may go through multiple stopping <-> running transitions between +* here and put_prev_task_scx() if task attribute changes occur while +* balance_scx() leaves @rq unlocked. However, they don't contain any +* information meaningful to the BPF scheduler and can be suppressed by +* skipping the callbacks if the task is !QUEUED. +*/ +if (SCX_HAS_OP (stopping) &&task_current (rq, p)) { +update_curr_scx (rq); +SCX_CALL_OP (SCX_KF_REST , stopping, p, false); + } + +if (SCX_HAS_OP (quiescent)) +SCX_CALL_OP (SCX_KF_REST , quiescent, p, deq_flags); +if (deq_flags &SCX_DEQ_SLEEP ) p->scx.flags |=SCX_TASK_DEQD_FOR_SLEEP ;else
@@ -1933,6 +2021,10 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first) p->se.exec_start =rq_clock_task (rq); +/* see dequeue_task_scx() on why we skip when !QUEUED */ +if (SCX_HAS_OP (running) && (p->scx.flags &SCX_TASK_QUEUED )) +SCX_CALL_OP (SCX_KF_REST , running, p); +clr_task_runnable (p, true); }
@@ -1971,6 +2063,10 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p)update_curr_scx (rq); +/* see dequeue_task_scx() on why we skip when !QUEUED */ +if (SCX_HAS_OP (stopping) && (p->scx.flags &SCX_TASK_QUEUED )) +SCX_CALL_OP (SCX_KF_REST , stopping, p, true); +/* * If we're being called from put_prev_task_balance(), balance_scx() may * have decided that @p should keep running.
@@ -3830,6 +3926,10 @@ static s32 select_cpu_stub(struct task_struct *p, s32 prev_cpu, u64 wake_flags)static void enqueue_stub (struct task_struct *p,u64 enq_flags) {}static void dequeue_stub (struct task_struct *p,u64 enq_flags) {}static void dispatch_stub (s32 prev_cpu,struct task_struct *p) {} +static void runnable_stub (struct task_struct *p,u64 enq_flags) {} +static void running_stub (struct task_struct *p) {} +static void stopping_stub (struct task_struct *p,bool runnable) {} +static void quiescent_stub (struct task_struct *p,u64 deq_flags) {}static bool yield_stub (struct task_struct *from,struct task_struct *to) {return false; }static void set_weight_stub (struct task_struct *p,u32 weight) {}static void set_cpumask_stub (struct task_struct *p,const struct cpumask *mask) {}
@@ -3846,6 +3946,10 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = { .enqueue = enqueue_stub, .dequeue = dequeue_stub, .dispatch = dispatch_stub, + .runnable = runnable_stub, + .running = running_stub, + .stopping = stopping_stub, + .quiescent = quiescent_stub, .yield = yield_stub, .set_weight = set_weight_stub, .set_cpumask = set_cpumask_stub,
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 26/39 sched_ext: Implement tickless support
In reply to Tejun HeoAllow BPF schedulers to indicate tickless operation by setting p->scx.slice to SCX_SLICE_INF. A CPU whose current task has infinte slice goes into tickless operation.
scx_central is updated to use tickless operations for all tasks and instead use a BPF timer to expire slices. This also uses the SCX_ENQ_PREEMPT and task state tracking added by the previous patches.
Currently, there is no way to pin the timer on the central CPU, so it may end up on one of the worker CPUs; however, outside of that, the worker CPUs can go tickless both while running sched_ext tasks and idling.
With schbench running, scx_central shows:
root@test ~# grep ^LOC /proc/interrupts; sleep 10; grep ^LOC /proc/interrupts
LOC: 142024 656 664 449 Local timer interrupts LOC: 161663 663 665 449 Local timer interrupts
Without it:
root@test ~ [SIGINT]# grep ^LOC /proc/interrupts; sleep 10; grep ^LOC /proc/interrupts
LOC: 188778 3142 3793 3993 Local timer interrupts LOC: 198993 5314 6323 6438 Local timer interrupts
While scx_central itself is too barebone to be useful as a production scheduler, a more featureful central scheduler can be built using the same approach. Google's experience shows that such an approach can have significant benefits for certain applications such as VM hosting.
v4: Allow operation even if BPF_F_TIMER_CPU_PIN is not available.
v3: Pin the central scheduler's timer on the central_cpu using
BPF_F_TIMER_CPU_PIN.
v2: Convert to BPF inline iterators.
--- include/linux/sched/ext.h | 1 + kernel/sched/core.c | 9 +- kernel/sched/ext.c | 52 +++++++++- kernel/sched/ext.h | 2 + kernel/sched/sched.h | 1 + tools/sched_ext/scx_central.bpf.c | 159 ++++++++++++++++++++++++++++-- tools/sched_ext/scx_central.c | 29 +++++- 7 files changed, 241 insertions(+), 12 deletions(-)
include/linux/sched/ext.h +1 -0
@@ -16,6 +16,7 @@ enum scx_public_consts {SCX_OPS_NAME_LEN =128 ,SCX_SLICE_DFL =20 *NSEC_PER_MSEC , + SCX_SLICE_INF = U64_MAX, /* infinite, implies nohz */ };/*
kernel/sched/core.c +6 -3
@@ -1255,13 +1255,16 @@ bool sched_can_stop_tick(struct rq *rq)return true;/* -* If there are no DL,RR/FIFO tasks, there must only be CFS tasks left; -* if there's more than one we need the tick for involuntary -* preemption. +* If there are no DL,RR/FIFO tasks, there must only be CFS or SCX tasks +* left. For CFS, if there's more than one we need the tick for +* involuntary preemption. For SCX, ask. */ if (!scx_switched_all () && rq->nr_running >1 )return false; +if (scx_enabled () && !scx_can_stop_tick (rq)) +return false; +/* * If there is one task and it has CFS runtime bandwidth constraints * and it's on the cpu now we don't want to stop the tick.
kernel/sched/ext.c +50 -2
82 lines not on this page · raw message
kernel/sched/ext.h +2 -0
@@ -36,6 +36,7 @@ int scx_fork(struct task_struct *p);void scx_post_fork (struct task_struct *p);void scx_cancel_fork (struct task_struct *p);int scx_check_setscheduler (struct task_struct *p,int policy); +bool scx_can_stop_tick (struct rq *rq);bool task_should_scx (struct task_struct *p);void init_sched_ext_class (void );
@@ -83,6 +84,7 @@ static inline int scx_fork(struct task_struct *p) { return 0; }static inline void scx_post_fork (struct task_struct *p) {}static inline void scx_cancel_fork (struct task_struct *p) {}static inline int scx_check_setscheduler (struct task_struct *p,int policy) {return 0 ; } +static inline bool scx_can_stop_tick (struct rq *rq) {return true; }static inline bool task_on_scx (const struct task_struct *p) {return false; }static inline void init_sched_ext_class (void ) {}static inline u32 scx_cpuperf_target (s32 cpu) {return 0 ; }
kernel/sched/sched.h +1 -0
@@ -712,6 +712,7 @@ struct cfs_rq {/* scx_rq->flags, protected by the rq lock */ enum scx_rq_flags {SCX_RQ_BALANCING =1 <<0 , +SCX_RQ_CAN_STOP_TICK =1 <<1 , };struct scx_rq {
tools/sched_ext/scx_central.bpf.c +153 -6
218 lines not on this page · raw message
tools/sched_ext/scx_central.c +28 -1
@@ -48,6 +48,7 @@ int main(int argc, char **argv)struct bpf_link *link;__u64 seq =0 ;__s32 opt; +cpu_set_t *cpuset;libbpf_set_print (libbpf_print_fn);signal (SIGINT , sigint_handler);
@@ -77,10 +78,35 @@ int main(int argc, char **argv)/* Resize arrays so their element count is equal to cpu count. */ RESIZE_ARRAY (data, cpu_gimme_task, skel->rodata->nr_cpu_ids); +RESIZE_ARRAY (data, cpu_started_at, skel->rodata->nr_cpu_ids);SCX_OPS_LOAD (skel, central_ops, scx_central, uei); + +/* +* Affinitize the loading thread to the central CPU, as: +* - That's where the BPF timer is first invoked in the BPF program. +* - We probably don't want this user space component to take up a core +* from a task that would benefit from avoiding preemption on one of +* the tickless cores. +* +* Until BPF supports pinning the timer, it's not guaranteed that it +* will always be invoked on the central CPU. In practice, this +* suffices the majority of the time. +*/ + cpuset =CPU_ALLOC (skel->rodata->nr_cpu_ids); +SCX_BUG_ON (!cpuset,"Failed to allocate cpuset" ); +CPU_ZERO (cpuset); +CPU_SET (skel->rodata->central_cpu, cpuset); +SCX_BUG_ON (sched_setaffinity (0 ,sizeof (cpuset), cpuset), +"Failed to affinitize to central CPU %d (max %d)" , + skel->rodata->central_cpu, skel->rodata->nr_cpu_ids -1 ); +CPU_FREE (cpuset); + link =SCX_OPS_ATTACH (skel, central_ops); +if (!skel->data->timer_pinned) +printf ("WARNING : BPF_F_TIMER_CPU_PIN not available, timer not pinned to central\n" ); +while (!exit_req && !UEI_EXITED (skel, uei)) {printf ("[SEQ %llu]\n" , seq++); printf("total :%10" PRIu64 " local:%10" PRIu64 " queued:%10" PRIu64 " lost:%10" PRIu64 "\n",
@@ -88,7 +114,8 @@ int main(int argc, char **argv) skel->bss->nr_locals, skel->bss->nr_queued, skel->bss->nr_lost_pids); -printf (" dispatch:%10" PRIu64" mismatch:%10" PRIu64" retry:%10" PRIu64"\n" , +printf ("timer :%10" PRIu64" dispatch:%10" PRIu64" mismatch:%10" PRIu64" retry:%10" PRIu64"\n" , + skel->bss->nr_timers, skel->bss->nr_dispatches, skel->bss->nr_mismatches, skel->bss->nr_retries);
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 27/39 sched_ext: Track tasks that are subjects of the in-flight SCX operation
In reply to Tejun HeoWhen some SCX operations are in flight, it is known that the subject task's rq lock is held throughout which makes it safe to access certain fields of the task - e.g. its current task_group. We want to add SCX kfunc helpers that can make use of this guarantee - e.g. to help determining the currently associated CPU cgroup from the task's current task_group.
As it'd be dangerous call such a helper on a task which isn't rq lock protected, the helper should be able to verify the input task and reject accordingly. This patch adds sched_ext_entity.kf_tasks[] that track the tasks which are currently being operated on by a terminal SCX operation. The new SCX_CALL_OP_[2]TASK[_RET]() can be used when invoking SCX operations which take tasks as arguments and the scx_kf_allowed_on_arg_tasks() can be used by kfunc helpers to verify the input task status.
Note that as sched_ext_entity.kf_tasks[] can't handle nesting, the tracking is currently only limited to terminal SCX operations. If needed in the future, this restriction can be removed by moving the tracking to the task side with a couple per-task counters.
v2: Updated to reflect the addition of SCX_KF_SELECT_CPU.
--- include/linux/sched/ext.h | 2 + kernel/sched/ext.c | 91 +++++++++++++++++++++++++++++++-------- 2 files changed, 76 insertions(+), 17 deletions(-)
include/linux/sched/ext.h +2 -0
@@ -106,6 +106,7 @@ enum scx_kf_mask { __SCX_KF_RQ_LOCKED =SCX_KF_DISPATCH |SCX_KF_ENQUEUE |SCX_KF_SELECT_CPU |SCX_KF_REST , + __SCX_KF_TERMINAL = SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST, };/*
@@ -120,6 +121,7 @@ struct sched_ext_entity {s32 sticky_cpu;s32 holding_cpu;u32 kf_mask;/* see scx_kf_mask above */ +struct task_struct *kf_tasks[2 ];/* see SCX_CALL_OP_TASK() */ atomic_long_t ops_state;struct list_head runnable_node;/* rq->scx.runnable_list */
kernel/sched/ext.c +74 -17
184 lines not on this page · raw message
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 28/39 sched_ext: Add cgroup support
In reply to Tejun HeoAdd sched_ext_ops operations to init/exit cgroups, and track task migrations and config changes. Because different BPF schedulers may implement different subsets of CPU control features, allow BPF schedulers to pick which cgroup interface files to enable using SCX_OPS_CGROUP_KNOB_* flags. For now, only the weight knobs are supported but adding more should be straightforward.
While a BPF scheduler is being enabled and disabled, relevant cgroup operations are locked out using scx_cgroup_rwsem. This avoids situations like task prep taking place while the task is being moved across cgroups, making things easier for BPF schedulers.
v5: - Flipped the locking order between scx_cgroup_rwsem and
cpus_read_lock() to avoid locking order conflict w/ cpuset. Better documentation around locking.
- sched_move_task() takes an early exit if the source and destination are identical. This triggered the warning in scx_cgroup_can_attach() as it left p->scx.cgrp_moving_from uncleared. Updated the cgroup migration path so that ops.cgroup_prep_move() is skipped for identity migrations so that its invocations always match ops.cgroup_move() one-to-one.
v4: - Example schedulers moved into their own patches.
- Fix build failure when !CONFIG_CGROUP_SCHED, reported by Andrea Righi.
v3: - Make scx_example_pair switch all tasks by default.
- Convert to BPF inline iterators.
- scx_bpf_task_cgroup() is added to determine the current cgroup from CPU controller's POV. This allows BPF schedulers to accurately track CPU cgroup membership.
- scx_example_flatcg added. This demonstrates flattened hierarchy implementation of CPU cgroup control and shows significant performance improvement when cgroups which are nested multiple levels are under competition.
v2: - Build fixes for different CONFIG combinations.
--- include/linux/sched/ext.h | 3 + init/Kconfig | 5 + kernel/sched/core.c | 70 ++- kernel/sched/ext.c | 524 ++++++++++++++++++++++- kernel/sched/ext.h | 20 + kernel/sched/sched.h | 12 +- tools/sched_ext/include/scx/common.bpf.h | 1 + 7 files changed, 614 insertions(+), 21 deletions(-)
include/linux/sched/ext.h +3 -0
@@ -157,6 +157,9 @@ struct sched_ext_entity {bool disallow;/* reject switching into SCX */ /* cold fields */ +#ifdef CONFIG_EXT_GROUP_SCHED +struct cgroup *cgrp_moving_from; +#endif /* must be the last field, see init_scx_entity() */ struct list_head tasks_node; };
init/Kconfig +5 -0
@@ -1033,6 +1033,11 @@ config RT_GROUP_SCHED realtime bandwidth for them. See Documentation/scheduler/sched-rt-group.rst for more information. +config EXT_GROUP_SCHED + bool + depends on SCHED_CLASS_EXT && CGROUP_SCHED + default y + endif #CGROUP_SCHED config SCHED_MM_CID
kernel/sched/core.c +61 -9
189 lines not on this page · raw message
kernel/sched/ext.c +513 -11
693 lines not on this page · raw message
kernel/sched/ext.h +20 -0
@@ -105,3 +105,23 @@ static inline void scx_update_idle(struct rq *rq, bool idle)#else static inline void scx_update_idle (struct rq *rq,bool idle) {}#endif + +#ifdef CONFIG_CGROUP_SCHED +#ifdef CONFIG_EXT_GROUP_SCHED +int scx_tg_online (struct task_group *tg); +void scx_tg_offline (struct task_group *tg); +int scx_cgroup_can_attach (struct cgroup_taskset *tset); +void scx_move_task (struct task_struct *p); +void scx_cgroup_finish_attach (void ); +void scx_cgroup_cancel_attach (struct cgroup_taskset *tset); +void scx_group_set_weight (struct task_group *tg,unsigned long cgrp_weight); +#else /* CONFIG_EXT_GROUP_SCHED */ +static inline int scx_tg_online (struct task_group *tg) {return 0 ; } +static inline void scx_tg_offline (struct task_group *tg) {} +static inline int scx_cgroup_can_attach (struct cgroup_taskset *tset) {return 0 ; } +static inline void scx_move_task (struct task_struct *p) {} +static inline void scx_cgroup_finish_attach (void ) {} +static inline void scx_cgroup_cancel_attach (struct cgroup_taskset *tset) {} +static inline void scx_group_set_weight (struct task_group *tg,unsigned long cgrp_weight) {} +#endif /* CONFIG_EXT_GROUP_SCHED */ +#endif /* CONFIG_CGROUP_SCHED */
kernel/sched/sched.h +11 -1
@@ -432,6 +432,11 @@ struct task_group {struct rt_bandwidth rt_bandwidth;#endif +#ifdef CONFIG_EXT_GROUP_SCHED +u32 scx_flags;/* SCX_TG_* */ +u32 scx_weight; +#endif +struct rcu_head rcu;struct list_head list;
@@ -548,6 +553,11 @@ extern void set_task_rq_fair(struct sched_entity *se,static inline void set_task_rq_fair (struct sched_entity *se,struct cfs_rq *prev,struct cfs_rq *next) { }#endif /* CONFIG_SMP */ +#else /* CONFIG_FAIR_GROUP_SCHED */ +static inline int sched_group_set_shares (struct task_group *tg,unsigned long shares) +{ +return 0 ; +}#endif /* CONFIG_FAIR_GROUP_SCHED */ #else /* CONFIG_CGROUP_SCHED */
@@ -3549,7 +3559,7 @@ extern int entity_eligible(struct cfs_rq *cfs_rq, struct sched_entity *se); #ifdef CONFIG_CGROUP_SCHED enum cpu_cftype_id { -#ifdef CONFIG_FAIR_GROUP_SCHED +#if defined(CONFIG_FAIR_GROUP_SCHED) || defined(CONFIG_EXT_GROUP_SCHED) CPU_CFTYPE_WEIGHT, CPU_CFTYPE_WEIGHT_NICE, CPU_CFTYPE_IDLE,
tools/sched_ext/include/scx/common.bpf.h +1 -0
@@ -51,6 +51,7 @@ s32 scx_bpf_pick_idle_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; bool scx_bpf_task_running(const struct task_struct *p) __ksym; s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym; +struct cgroup *scx_bpf_task_cgroup (struct task_struct *p) __ksym;static inline __attribute__((format (printf,1 ,2 )))void ___scx_bpf_exit_format_checker (const char *fmt, ...) {}
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 29/39 sched_ext: Add a cgroup scheduler which uses flattened hierarchy
In reply to Tejun HeoThis patch adds scx_flatcg example scheduler which implements hierarchical weight-based cgroup CPU control by flattening the cgroup hierarchy into a single layer by compounding the active weight share at each level.
This flattening of hierarchy can bring a substantial performance gain when the cgroup hierarchy is nested multiple levels. in a simple benchmark using wrk[8] on apache serving a CGI script calculating sha1sum of a small file, it outperforms CFS by ~3% with CPU controller disabled and by ~10% with two apache instances competing with 2:1 weight ratio nested four level deep.
However, the gain comes at the cost of not being able to properly handle thundering herd of cgroups. For example, if many cgroups which are nested behind a low priority parent cgroup wake up around the same time, they may be able to consume more CPU cycles than they are entitled to. In many use cases, this isn't a real concern especially given the performance gain. Also, there are ways to mitigate the problem further by e.g. introducing an extra scheduling layer on cgroup delegation boundaries.
v3: - Updated to reflect the core API changes including ops.init/exit_task()
and direct dispatch from ops.select_cpu(). Fixes and improvements including additional statistics.
- Use reference counted kptr for cgv_node instead of xchg'ing against stash location.
- Dropped '-p' option.
v2: - Use SCX_BUG[_ON]() to simplify error handling.
--- tools/sched_ext/Makefile | 2 +- tools/sched_ext/scx_flatcg.bpf.c | 861 +++++++++++++++++++++++++++++++ tools/sched_ext/scx_flatcg.c | 225 ++++++++ tools/sched_ext/scx_flatcg.h | 51 ++ 4 files changed, 1138 insertions(+), 1 deletion(-) create mode 100644 tools/sched_ext/scx_flatcg.bpf.c create mode 100644 tools/sched_ext/scx_flatcg.c create mode 100644 tools/sched_ext/scx_flatcg.h
tools/sched_ext/Makefile +1 -1
@@ -176,7 +176,7 @@ $(INCLUDE_DIR)/%.bpf.skel.h: $(SCXOBJ_DIR)/%.bpf.o $(INCLUDE_DIR)/vmlinux.h $(BP SCX_COMMON_DEPS := include/scx/common.h include/scx/user_exit_info.h | $(BINDIR) -c-sched-targets = scx_simple scx_qmap scx_central +c-sched-targets = scx_simple scx_qmap scx_central scx_flatcg $(addprefix $(BINDIR)/,$(c-sched-targets)): \ $(BINDIR)/%: \
tools/sched_ext/scx_flatcg.bpf.c new file +861 -0
861 lines not on this page · raw message
tools/sched_ext/scx_flatcg.c new file +225 -0
225 lines not on this page · raw message
tools/sched_ext/scx_flatcg.h new file +51 -0
51 lines not on this page · raw message
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 30/39 sched_ext: Implement SCX_KICK_WAIT
In reply to Tejun HeoFrom: David Vernet <[email protected]>
If set when calling scx_bpf_kick_cpu(), the invoking CPU will busy wait for the kicked cpu to enter the scheduler. See the following for example usage:
https://github.com/sched-ext/scx/blob/main/scheds/c/scx_pair.bpf.c
v2: - Updated to fit the updated kick_cpus_irq_workfn() implementation.
- Include SCX_KICK_WAIT related information in debug dump.
--- kernel/sched/core.c | 4 ++- kernel/sched/ext.c | 82 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/ext.h | 4 +++ kernel/sched/sched.h | 2 ++ 4 files changed, 85 insertions(+), 7 deletions(-)
kernel/sched/core.c +3 -1
@@ -6115,8 +6115,10 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)for_each_active_class (class) { p = class->pick_next_task (rq); -if (p) +if (p) { +scx_next_task_picked (rq, p, class);return p; + } }BUG ();/* The idle class should always have a runnable task. */
kernel/sched/ext.c +76 -6
173 lines not on this page · raw message
kernel/sched/ext.h +4 -0
@@ -29,6 +29,8 @@ static inline bool task_on_scx(const struct task_struct *p)return scx_enabled () && p->sched_class == &ext_sched_class; } +void scx_next_task_picked (struct rq *rq,struct task_struct *p, +const struct sched_class *active);void scx_tick (struct rq *rq);void init_scx_entity (struct sched_ext_entity *scx);void scx_pre_fork (struct task_struct *p);
@@ -78,6 +80,8 @@ static inline const struct sched_class *next_active_class(const struct sched_cla#define scx_enabled () false#define scx_switched_all () false +static inline void scx_next_task_picked (struct rq *rq,struct task_struct *p, +const struct sched_class *active) {}static inline void scx_tick (struct rq *rq) {}static inline void scx_pre_fork (struct task_struct *p) {}static inline int scx_fork (struct task_struct *p) {return 0 ; }
kernel/sched/sched.h +2 -0
@@ -735,6 +735,8 @@ struct scx_rq {cpumask_var_t cpus_to_kick;cpumask_var_t cpus_to_kick_if_idle;cpumask_var_t cpus_to_preempt; +cpumask_var_t cpus_to_wait; +unsigned long pnt_seq; struct irq_work kick_cpus_irq_work; };#endif /* CONFIG_SCHED_CLASS_EXT */
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 31/39 sched_ext: Implement sched_ext_ops.cpu_acquire/release()
In reply to Tejun HeoFrom: David Vernet <[email protected]>
Scheduler classes are strictly ordered and when a higher priority class has tasks to run, the lower priority ones lose access to the CPU. Being able to monitor and act on these events are necessary for use cases includling strict core-scheduling and latency management.
This patch adds two operations ops.cpu_acquire() and .cpu_release(). The former is invoked when a CPU becomes available to the BPF scheduler and the opposite for the latter. This patch also implements scx_bpf_reenqueue_local() which can be called from .cpu_release() to trigger requeueing of all tasks in the local dsq of the CPU so that the tasks can be reassigned to other available CPUs.
scx_pair is updated to use .cpu_acquire/release() along with %SCX_KICK_WAIT to make the pair scheduling guarantee strict even when a CPU is preempted by a higher priority scheduler class.
scx_qmap is updated to use .cpu_acquire/release() to empty the local dsq of a preempted CPU. A similar approach can be adopted by BPF schedulers that want to have a tight control over latency.
v4: Use the new SCX_KICK_IDLE to wake up a CPU after re-enqueueing.
v3: Drop the const qualifier from scx_cpu_release_args.task. BPF enforces
access control through the verifier, so the qualifier isn't actually operative and only gets in the way when interacting with various helpers.
v2: Add p->scx.kf_mask annotation to allow calling scx_bpf_reenqueue_local()
from ops.cpu_release() nested inside ops.init() and other sleepable operations.
--- include/linux/sched/ext.h | 4 +- kernel/sched/ext.c | 200 ++++++++++++++++++++++- kernel/sched/ext.h | 2 + kernel/sched/sched.h | 1 + tools/sched_ext/include/scx/common.bpf.h | 1 + tools/sched_ext/scx_qmap.bpf.c | 37 ++++- tools/sched_ext/scx_qmap.c | 4 +- 7 files changed, 242 insertions(+), 7 deletions(-)
include/linux/sched/ext.h +3 -1
@@ -98,13 +98,15 @@ enum scx_kf_mask {SCX_KF_UNLOCKED =0 ,/* not sleepable, not rq locked */ /* all non-sleepables may be nested inside SLEEPABLE */ SCX_KF_SLEEPABLE =1 <<0 ,/* sleepable init operations */ +/* ENQUEUE and DISPATCH may be nested inside CPU_RELEASE */ +SCX_KF_CPU_RELEASE =1 <<1 ,/* ops.cpu_release() */ /* ops.dequeue (in REST) may be nested inside DISPATCH */ SCX_KF_DISPATCH =1 <<2 ,/* ops.dispatch() */ SCX_KF_ENQUEUE =1 <<3 ,/* ops.enqueue() and ops.select_cpu() */ SCX_KF_SELECT_CPU =1 <<4 ,/* ops.select_cpu() */ SCX_KF_REST =1 <<5 ,/* other rq-locked operations */ - __SCX_KF_RQ_LOCKED =SCX_KF_DISPATCH | + __SCX_KF_RQ_LOCKED =SCX_KF_CPU_RELEASE |SCX_KF_DISPATCH |SCX_KF_ENQUEUE |SCX_KF_SELECT_CPU |SCX_KF_REST , __SCX_KF_TERMINAL = SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST, };
kernel/sched/ext.c +197 -3
297 lines not on this page · raw message
kernel/sched/ext.h +2 -0
@@ -24,6 +24,8 @@ DECLARE_STATIC_KEY_FALSE(__scx_switched_all);#define scx_enabled () static_branch_unlikely(&__scx_ops_enabled)#define scx_switched_all () static_branch_unlikely(&__scx_switched_all) +DECLARE_STATIC_KEY_FALSE (scx_ops_cpu_preempt); +static inline bool task_on_scx (const struct task_struct *p) {return scx_enabled () && p->sched_class == &ext_sched_class;
kernel/sched/sched.h +1 -0
@@ -732,6 +732,7 @@ struct scx_rq {u64 extra_enq_flags;/* see move_task_to_local_dsq() */ u32 nr_running;u32 flags; +bool cpu_released;cpumask_var_t cpus_to_kick;cpumask_var_t cpus_to_kick_if_idle;cpumask_var_t cpus_to_preempt;
tools/sched_ext/include/scx/common.bpf.h +1 -0
@@ -34,6 +34,7 @@ void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flag u32 scx_bpf_dispatch_nr_slots(void) __ksym; void scx_bpf_dispatch_cancel(void) __ksym; bool scx_bpf_consume(u64 dsq_id) __ksym; +u32 scx_bpf_reenqueue_local(void) __ksym; void scx_bpf_kick_cpu(s32 cpu, u64 flags) __ksym; s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym; void scx_bpf_destroy_dsq(u64 dsq_id) __ksym;
tools/sched_ext/scx_qmap.bpf.c +36 -1
@@ -11,6 +11,8 @@* * - BPF-side queueing using PIDs. * - Sleepable per-task storage allocation using ops.prep_enable(). +* - Using ops.cpu_release() to handle a higher priority scheduling class taking +* the CPU away. * * This scheduler is primarily for demonstration and testing of sched_ext * features and unlikely to be useful for actual workloads.
@@ -90,7 +92,7 @@ struct { } cpu_ctx_stor SEC(".maps");/* Statistics */ -u64 nr_enqueued, nr_dispatched, nr_dequeued; +u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_dequeued;s32 BPF_STRUCT_OPS (qmap_select_cpu ,struct task_struct *p,s32 prev_cpu,u64 wake_flags)
@@ -164,6 +166,22 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)return ; } +/* +* If the task was re-enqueued due to the CPU being preempted by a +* higher priority scheduling class, just re-enqueue the task directly +* on the global DSQ. As we want another CPU to pick it up, find and +* kick an idle CPU. +*/ +if (enq_flags &SCX_ENQ_REENQ ) { +s32 cpu; + +scx_bpf_dispatch (p,SHARED_DSQ ,0 , enq_flags); + cpu =scx_bpf_pick_idle_cpu (p->cpus_ptr,0 ); +if (cpu >=0 ) +scx_bpf_kick_cpu (cpu, __COMPAT_SCX_KICK_IDLE); +return ; + } + ring =bpf_map_lookup_elem (&queue_arr, &idx);if (!ring) {scx_bpf_error ("failed to find ring %d" , idx);
@@ -257,6 +275,22 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev) } } +void BPF_STRUCT_OPS (qmap_cpu_release ,s32 cpu,struct scx_cpu_release_args *args) +{ +u32 cnt; + +/* +* Called when @cpu is taken by a higher priority scheduling class. This +* makes @cpu no longer available for executing sched_ext tasks. As we +* don't want the tasks in @cpu's local dsq to sit there until @cpu +* becomes available again, re-enqueue them into the global dsq. See +* %SCX_ENQ_REENQ handling in qmap_enqueue(). +*/ + cnt =scx_bpf_reenqueue_local (); +if (cnt) +__sync_fetch_and_add (&nr_reenqueued, cnt); +} +s32 BPF_STRUCT_OPS (qmap_init_task ,struct task_struct *p,struct scx_init_task_args *args) {
@@ -292,6 +326,7 @@ SCX_OPS_DEFINE(qmap_ops, .enqueue = (void *)qmap_enqueue, .dequeue = (void *)qmap_dequeue, .dispatch = (void *)qmap_dispatch, + .cpu_release = (void *)qmap_cpu_release, .init_task = (void *)qmap_init_task, .init = (void *)qmap_init, .exit = (void *)qmap_exit,
tools/sched_ext/scx_qmap.c +2 -2
@@ -109,9 +109,9 @@ int main(int argc, char **argv)long nr_enqueued = skel->bss->nr_enqueued;long nr_dispatched = skel->bss->nr_dispatched; -printf ("stats : enq=%lu dsp=%lu delta=%ld deq=%" PRIu64"\n" , +printf ("stats : enq=%lu dsp=%lu delta=%ld reenq=%" PRIu64" deq=%" PRIu64"\n" , nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched, - skel->bss->nr_dequeued); + skel->bss->nr_reenqueued, skel->bss->nr_dequeued);fflush (stdout);sleep (1 ); }
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 32/39 sched_ext: Implement sched_ext_ops.cpu_online/offline()
In reply to Tejun HeoAdd ops.cpu_online/offline() which are invoked when CPUs come online and offline respectively. As the enqueue path already automatically bypasses tasks to the local dsq on a deactivated CPU, BPF schedulers are guaranteed to see tasks only on CPUs which are between online() and offline().
If the BPF scheduler doesn't implement ops.cpu_online/offline(), the scheduler is automatically exited with SCX_ECODE_RESTART | SCX_ECODE_RSN_HOTPLUG. Userspace can implement CPU hotpplug support trivially by simply reinitializing and reloading the scheduler.
scx_qmap is updated to print out online CPUs on hotplug events. Other schedulers are updated to restart based on ecode.
v2: - To accommodate lock ordering change between scx_cgroup_rwsem and
cpus_read_lock(), CPU hotplug operations are put into its own SCX_OPI block and enabled eariler during scx_ope_enable() so that cpus_read_lock() can be dropped before acquiring scx_cgroup_rwsem.
- Auto exit with ECODE added.
--- kernel/sched/ext.c | 135 ++++++++++++++++++- tools/sched_ext/include/scx/compat.h | 29 ++++ tools/sched_ext/include/scx/user_exit_info.h | 28 ++++ tools/sched_ext/scx_central.c | 9 +- tools/sched_ext/scx_flatcg.c | 8 +- tools/sched_ext/scx_qmap.bpf.c | 60 +++++++++ tools/sched_ext/scx_qmap.c | 4 + tools/sched_ext/scx_simple.c | 8 +- 8 files changed, 271 insertions(+), 10 deletions(-)
kernel/sched/ext.c +132 -3
@@ -30,6 +30,29 @@ enum scx_exit_kind { SCX_EXIT_ERROR_STALL, /* watchdog detected stalled runnable tasks */ }; +/* +* An exit code can be specified when exiting with scx_bpf_exit() or +* scx_ops_exit(), corresponding to exit_kind UNREG_BPF and UNREG_KERN +* respectively. The codes are 64bit of the format: +* +* Bits: [63 .. 48 47 .. 32 31 .. 0] +* [ SYS ACT ] [ SYS RSN ] [ USR ] +* +* SYS ACT: System-defined exit actions +* SYS RSN: System-defined exit reasons +* USR : User-defined exit codes and reasons +* +* Using the above, users may communicate intention and context by ORing system +* actions and/or system reasons with a user-defined exit code. +*/ +enum scx_exit_code { +/* Reasons */ +SCX_ECODE_RSN_HOTPLUG =1LLU <<32 , + +/* Actions */ +SCX_ECODE_ACT_RESTART =1LLU <<48 , +}; +/* * scx_exit_info is passed to ops.exit() to describe why the BPF scheduler is * being disabled.
@@ -504,7 +527,29 @@ struct sched_ext_ops {#endif /* CONFIG_CGROUPS */ /* -* All online ops must come before ops.init(). +* All online ops must come before ops.cpu_online(). +*/ + +/** +* cpu_online - A CPU became online +* @cpu: CPU which just came up +* +* @cpu just came online. @cpu doesn't call ops.enqueue() or run tasks +* associated with other CPUs beforehand. +*/ +void (*cpu_online)(s32 cpu); + +/** +* cpu_offline - A CPU is going offline +* @cpu: CPU which is going offline +* +* @cpu is going offline. @cpu doesn't call ops.enqueue() or run tasks +* associated with other CPUs afterwards. +*/ +void (*cpu_offline)(s32 cpu); + +/* +* All CPU hotplug ops must come before ops.init(). */ /**
@@ -543,6 +588,15 @@ struct sched_ext_ops {*/ u32 exit_dump_len; +/** +* hotplug_seq - A sequence number that may be set by the scheduler to +* detect when a hotplug event has occurred during the loading process. +* If 0, no detection occurs. Otherwise, the scheduler will fail to +* load if the sequence number does not match @scx_hotplug_seq on the +* enable path. +*/ +u64 hotplug_seq; +/** * name - BPF scheduler's name *
@@ -556,7 +610,9 @@ struct sched_ext_ops {enum scx_opi {SCX_OPI_BEGIN =0 ,SCX_OPI_NORMAL_BEGIN =0 , -SCX_OPI_NORMAL_END =SCX_OP_IDX (init), +SCX_OPI_NORMAL_END =SCX_OP_IDX (cpu_online), +SCX_OPI_CPU_HOTPLUG_BEGIN =SCX_OP_IDX (cpu_online), +SCX_OPI_CPU_HOTPLUG_END =SCX_OP_IDX (init),SCX_OPI_END =SCX_OP_IDX (init), };
@@ -746,6 +802,7 @@ static atomic_t scx_exit_kind = ATOMIC_INIT(SCX_EXIT_DONE);static struct scx_exit_info *scx_exit_info;static atomic_long_t scx_nr_rejected =ATOMIC_LONG_INIT (0 ); +static atomic_long_t scx_hotplug_seq =ATOMIC_LONG_INIT (0 );/* * The maximum amount of time in jiffies that a task may be runnable without
@@ -2172,7 +2229,8 @@ static int balance_scx(struct rq *rq, struct task_struct *prev,* emitted in scx_next_task_picked(). */ if (SCX_HAS_OP (cpu_acquire)) -SCX_CALL_OP (0 , cpu_acquire,cpu_of (rq),NULL ); +SCX_CALL_OP (SCX_KF_UNLOCKED , cpu_acquire,cpu_of (rq), +NULL ); rq->scx.cpu_released = false; }
@@ -2700,6 +2758,34 @@ void __scx_update_idle(struct rq *rq, bool idle)#endif } +static void handle_hotplug (struct rq *rq,bool online) +{ +int cpu =cpu_of (rq); + +atomic_long_inc (&scx_hotplug_seq); + +if (online &&SCX_HAS_OP (cpu_online)) +SCX_CALL_OP (SCX_KF_REST , cpu_online, cpu); +else if (!online &&SCX_HAS_OP (cpu_offline)) +SCX_CALL_OP (SCX_KF_REST , cpu_offline, cpu); +else +scx_ops_exit (SCX_ECODE_ACT_RESTART |SCX_ECODE_RSN_HOTPLUG , +"cpu %d going %s, exiting scheduler" , cpu, + online ?"online" :"offline" ); +} + +static void rq_online_scx (struct rq *rq,enum rq_onoff_reason reason) +{ +if (reason ==RQ_ONOFF_HOTPLUG ) +handle_hotplug (rq, true); +} + +static void rq_offline_scx (struct rq *rq,enum rq_onoff_reason reason) +{ +if (reason ==RQ_ONOFF_HOTPLUG ) +handle_hotplug (rq, false); +} +#else /* CONFIG_SMP */ static bool test_and_clear_cpu_idle (int cpu) {return false; }
@@ -3328,6 +3414,9 @@ DEFINE_SCHED_CLASS(ext) = { .balance = balance_scx, .select_task_rq = select_task_rq_scx, .set_cpus_allowed = set_cpus_allowed_scx, + + .rq_online = rq_online_scx, + .rq_offline = rq_offline_scx, #endif .task_tick = task_tick_scx,
@@ -3584,10 +3673,18 @@ static ssize_t scx_attr_nr_rejected_show(struct kobject *kobj, }SCX_ATTR (nr_rejected); +static ssize_t scx_attr_hotplug_seq_show (struct kobject *kobj, +struct kobj_attribute *ka,char *buf) +{ +return sysfs_emit (buf,"%ld\n" ,atomic_long_read (&scx_hotplug_seq)); +} +SCX_ATTR (hotplug_seq); +static struct attribute *scx_global_attrs[] = { &scx_attr_state.attr, &scx_attr_switch_all.attr, &scx_attr_nr_rejected.attr, + &scx_attr_hotplug_seq.attr,NULL , };
@@ -4110,6 +4207,25 @@ static struct kthread_worker *scx_create_rt_helper(const char *name)return helper; } +static void check_hotplug_seq (const struct sched_ext_ops *ops) +{ +unsigned long long global_hotplug_seq; + +/* +* If a hotplug event has occurred between when a scheduler was +* initialized, and when we were able to attach, exit and notify user +* space about it. +*/ +if (ops->hotplug_seq) { + global_hotplug_seq =atomic_long_read (&scx_hotplug_seq); +if (ops->hotplug_seq != global_hotplug_seq) { +scx_ops_exit (SCX_ECODE_ACT_RESTART |SCX_ECODE_RSN_HOTPLUG , +"expected hotplug seq %llu did not match actual %llu" , + ops->hotplug_seq, global_hotplug_seq); + } + } +} +static int validate_ops (const struct sched_ext_ops *ops) {/*
@@ -4192,6 +4308,10 @@ static int scx_ops_enable(struct sched_ext_ops *ops) } } +for (i =SCX_OPI_CPU_HOTPLUG_BEGIN ; i <SCX_OPI_CPU_HOTPLUG_END ; i++) +if (((void (**)(void ))ops)[i]) +static_branch_enable_cpuslocked (&scx_has_op[i]); +cpus_read_unlock (); ret =validate_ops (ops);
@@ -4239,6 +4359,8 @@ static int scx_ops_enable(struct sched_ext_ops *ops)cpus_read_lock ();scx_cgroup_lock (); +check_hotplug_seq (ops); +for (i =SCX_OPI_NORMAL_BEGIN ; i <SCX_OPI_NORMAL_END ; i++)if (((void (**)(void ))ops)[i])static_branch_enable_cpuslocked (&scx_has_op[i]);
@@ -4563,6 +4685,9 @@ static int bpf_scx_init_member(const struct btf_type *t, ops->exit_dump_len = *(u32 *)(udata + moff) ?:SCX_EXIT_DUMP_DFL_LEN ;return 1 ; +case offsetof(struct sched_ext_ops , hotplug_seq): + ops->hotplug_seq = *(u64 *)(udata + moff); +return 1 ; }return 0 ;
@@ -4659,6 +4784,8 @@ static void cgroup_move_stub(struct task_struct *p, struct cgroup *from, structstatic void cgroup_cancel_move_stub (struct task_struct *p,struct cgroup *from,struct cgroup *to) {}static void cgroup_set_weight_stub (struct cgroup *cgrp,u32 weight) {}#endif +static void cpu_online_stub (s32 cpu) {} +static void cpu_offline_stub (s32 cpu) {}static s32 init_stub (void ) {return -EINVAL ; }static void exit_stub (struct scx_exit_info *info) {}
@@ -4689,6 +4816,8 @@ static struct sched_ext_ops __bpf_ops_sched_ext_ops = { .cgroup_cancel_move = cgroup_cancel_move_stub, .cgroup_set_weight = cgroup_set_weight_stub, #endif + .cpu_online = cpu_online_stub, + .cpu_offline = cpu_offline_stub, .init = init_stub, .exit = exit_stub, };
tools/sched_ext/include/scx/compat.h +29 -0
@@ -8,6 +8,9 @@#define __SCX_COMPAT_H#include <bpf/btf.h> +#include <fcntl.h> +#include <stdlib.h> +#include <unistd.h> struct btf *__COMPAT_vmlinux_btf __attribute__((weak));
@@ -120,6 +123,28 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field#define __COMPAT_HAS_CPUMASKS \ __COMPAT_has_ksym("scx_bpf_nr_cpu_ids") +static inline long scx_hotplug_seq (void ) +{ +int fd; +char buf[32 ]; +ssize_t len; +long val; + + fd =open ("/sys/kernel/sched_ext/hotplug_seq" ,O_RDONLY ); +if (fd <0 ) +return -ENOENT ; + + len =read (fd, buf,sizeof (buf) -1 ); +SCX_BUG_ON (len <=0 ,"read failed (%ld)" , len); + buf[len] =0 ; +close (fd); + + val =strtoul (buf,NULL ,10 ); +SCX_BUG_ON (val <0 ,"invalid num hotplug events: %lu" , val); + +return val; +} +/* * struct sched_ext_ops can change over time. If compat.bpf.h::SCX_OPS_DEFINE() * is used to define ops and compat.h::SCX_OPS_LOAD/ATTACH() are used to load
@@ -128,12 +153,16 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field* * - ops.tick(): Ignored on older kernels with a warning. * - ops.exit_dump_len: Cleared to zero on older kernels with a warning. +* - ops.hotplug_seq: Ignored on older kernels. */ #define SCX_OPS_OPEN (__ops_name, __scx_name) ({ \ struct __scx_name *__skel; \ \ __skel = __scx_name##__open(); \ SCX_BUG_ON(!__skel, "Could not open " #__scx_name); \ + \ + if (__COMPAT_struct_has_field("sched_ext_ops", "hotplug_seq")) \ + __skel->struct_ops.__ops_name->hotplug_seq = scx_hotplug_seq(); \ __skel; \ })
tools/sched_ext/include/scx/user_exit_info.h +28 -0
@@ -77,7 +77,35 @@ struct user_exit_info {if (__uei->msg[0 ] !='\0' ) \fprintf (stderr," (%s)" , __uei->msg); \fputs ("\n" , stderr); \ + __uei->exit_code; \ }) +/* +* We can't import vmlinux.h while compiling user C code. Let's duplicate +* scx_exit_code definition. +*/ +enum scx_exit_code { +/* Reasons */ +SCX_ECODE_RSN_HOTPLUG =1LLU <<32 , + +/* Actions */ +SCX_ECODE_ACT_RESTART =1LLU <<48 , +}; + +enum uei_ecode_mask { +UEI_ECODE_USER_MASK = ((1LLU <<32 ) -1 ), +UEI_ECODE_SYS_RSN_MASK = ((1LLU <<16 ) -1 ) <<32 , +UEI_ECODE_SYS_ACT_MASK = ((1LLU <<16 ) -1 ) <<48 , +}; + +/* +* These macro interpret the ecode returned from UEI_REPORT(). +*/ +#define UEI_ECODE_USER (__ecode) ((__ecode) & UEI_ECODE_USER_MASK) +#define UEI_ECODE_SYS_RSN (__ecode) ((__ecode) & UEI_ECODE_SYS_RSN_MASK) +#define UEI_ECODE_SYS_ACT (__ecode) ((__ecode) & UEI_ECODE_SYS_ACT_MASK) + +#define UEI_ECODE_RESTART (__ecode) (UEI_ECODE_SYS_ACT((__ecode)) == SCX_ECODE_ACT_RESTART) +#endif /* __bpf__ */ #endif /* __USER_EXIT_INFO_H */
tools/sched_ext/scx_central.c +6 -3
@@ -46,14 +46,14 @@ int main(int argc, char **argv) {struct scx_central *skel;struct bpf_link *link; -__u64 seq =0 ; +__u64 seq =0 , ecode;__s32 opt;cpu_set_t *cpuset;libbpf_set_print (libbpf_print_fn);signal (SIGINT , sigint_handler);signal (SIGTERM , sigint_handler); - +restart: skel =SCX_OPS_OPEN (central_ops, scx_central); skel->rodata->central_cpu =0 ;
@@ -126,7 +126,10 @@ int main(int argc, char **argv) }bpf_link__destroy (link); -UEI_REPORT (skel, uei); + ecode =UEI_REPORT (skel, uei);scx_central__destroy (skel); + +if (UEI_ECODE_RESTART (ecode)) + goto restart;return 0 ; }
tools/sched_ext/scx_flatcg.c +6 -2
@@ -127,11 +127,12 @@ int main(int argc, char **argv)__u64 last_stats[FCG_NR_STATS ] = {};unsigned long seq =0 ;__s32 opt; +__u64 ecode;libbpf_set_print (libbpf_print_fn);signal (SIGINT , sigint_handler);signal (SIGTERM , sigint_handler); - +restart: skel =SCX_OPS_OPEN (flatcg_ops, scx_flatcg); skel->rodata->nr_cpus =libbpf_num_possible_cpus ();
@@ -219,7 +220,10 @@ int main(int argc, char **argv) }bpf_link__destroy (link); -UEI_REPORT (skel, uei); + ecode =UEI_REPORT (skel, uei);scx_flatcg__destroy (skel); + +if (UEI_ECODE_RESTART (ecode)) + goto restart;return 0 ; }
tools/sched_ext/scx_qmap.bpf.c +60 -0
77 lines not on this page · raw message
tools/sched_ext/scx_qmap.c +4 -0
@@ -119,5 +119,9 @@ int main(int argc, char **argv)bpf_link__destroy (link);UEI_REPORT (skel, uei);scx_qmap__destroy (skel); +/* +* scx_qmap implements ops.cpu_on/offline() and doesn't need to restart +* on CPU hotplug events. +*/ return 0 ; }
tools/sched_ext/scx_simple.c +6 -2
@@ -62,11 +62,12 @@ int main(int argc, char **argv)struct scx_simple *skel;struct bpf_link *link;__u32 opt; +__u64 ecode;libbpf_set_print (libbpf_print_fn);signal (SIGINT , sigint_handler);signal (SIGTERM , sigint_handler); - +restart: skel =SCX_OPS_OPEN (simple_ops, scx_simple);while ((opt =getopt (argc, argv,"vh" )) !=-1 ) {
@@ -93,7 +94,10 @@ int main(int argc, char **argv) }bpf_link__destroy (link); -UEI_REPORT (skel, uei); + ecode =UEI_REPORT (skel, uei);scx_simple__destroy (skel); + +if (UEI_ECODE_RESTART (ecode)) + goto restart;return 0 ; }
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 33/39 sched_ext: Bypass BPF scheduler while PM events are in progress
In reply to Tejun HeoPM operations freeze userspace. Some BPF schedulers have active userspace component and may misbehave as expected across PM events. While the system is frozen, nothing too interesting is happening in terms of scheduling and we can get by just fine with the fallback FIFO behavior. Let's make things easier by always bypassing the BPF scheduler while PM events are in progress.
--- kernel/sched/ext.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
kernel/sched/ext.c +34 -0
@@ -5010,6 +5010,34 @@ void print_scx_info(const char *log_lvl, struct task_struct *p) runnable_at_buf); } +static int scx_pm_handler (struct notifier_block *nb,unsigned long event,void *ptr) +{ +/* +* SCX schedulers often have userspace components which are sometimes +* involved in critial scheduling paths. PM operations involve freezing +* userspace which can lead to scheduling misbehaviors including stalls. +* Let's bypass while PM operations are in progress. +*/ +switch (event) { +case PM_HIBERNATION_PREPARE : +case PM_SUSPEND_PREPARE : +case PM_RESTORE_PREPARE : +scx_ops_bypass (true); +break ; +case PM_POST_HIBERNATION : +case PM_POST_SUSPEND : +case PM_POST_RESTORE : +scx_ops_bypass (false); +break ; + } + +return NOTIFY_OK ; +} + +static struct notifier_block scx_pm_notifier = { + .notifier_call = scx_pm_handler, +}; +void __init init_sched_ext_class (void ) {s32 cpu, v;
@@ -5902,6 +5930,12 @@ static int __init scx_init(void)return ret; } + ret =register_pm_notifier (&scx_pm_notifier); +if (ret) { +pr_err ("sched_ext: Failed to register PM notifier (%d)\n" , ret); +return ret; + } + scx_kset =kset_create_and_add ("sched_ext" , &scx_uevent_ops, kernel_kobj);if (!scx_kset) {pr_err ("sched_ext: Failed to create /sys/kernel/sched_ext\n" );
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 34/39 sched_ext: Implement core-sched support
In reply to Tejun HeoThe core-sched support is composed of the following parts:
- task_struct->scx.core_sched_at is added. This is a timestamp which can be used to order tasks. Depending on whether the BPF scheduler implements custom ordering, it tracks either global FIFO ordering of all tasks or local-DSQ ordering within the dispatched tasks on a CPU.
- prio_less() is updated to call scx_prio_less() when comparing SCX tasks. scx_prio_less() calls ops.core_sched_before() if available or uses the core_sched_at timestamp. For global FIFO ordering, the BPF scheduler doesn't need to do anything. Otherwise, it should implement ops.core_sched_before() which reflects the ordering.
- When core-sched is enabled, balance_scx() balances all SMT siblings so that they all have tasks dispatched if necessary before pick_task_scx() is called. pick_task_scx() picks between the current task and the first dispatched task on the local DSQ based on availability and the core_sched_at timestamps. Note that FIFO ordering is expected among the already dispatched tasks whether running or on the local DSQ, so this path always compares core_sched_at instead of calling into ops.core_sched_before().
qmap_core_sched_before() is added to scx_qmap. It scales the distances from the heads of the queues to compare the tasks across different priority queues and seems to behave as expected.
v3: Fixed build error when !CONFIG_SCHED_SMT reported by Andrea Righi.
v2: Sched core added the const qualifiers to prio_less task arguments.
Explicitly drop them for ops.core_sched_before() task arguments. BPF enforces access control through the verifier, so the qualifier isn't actually operative and only gets in the way when interacting with various helpers.
--- include/linux/sched/ext.h | 3 + kernel/Kconfig.preempt | 2 +- kernel/sched/core.c | 10 +- kernel/sched/ext.c | 250 +++++++++++++++++++++++++++++++-- kernel/sched/ext.h | 5 + tools/sched_ext/scx_qmap.bpf.c | 87 +++++++++++- tools/sched_ext/scx_qmap.c | 5 +- 7 files changed, 344 insertions(+), 18 deletions(-)
include/linux/sched/ext.h +3 -0
@@ -129,6 +129,9 @@ struct sched_ext_entity {struct list_head runnable_node;/* rq->scx.runnable_list */ unsigned long runnable_at; +#ifdef CONFIG_SCHED_CORE +u64 core_sched_at;/* see scx_prio_less() */ +#endif u64 ddsp_dsq_id;u64 ddsp_enq_flags;
kernel/Kconfig.preempt +1 -1
@@ -135,7 +135,7 @@ config SCHED_CORE config SCHED_CLASS_EXT bool "Extensible Scheduling Class" - depends on BPF_SYSCALL && BPF_JIT && !SCHED_CORE + depends on BPF_SYSCALL && BPF_JIT help This option enables a new scheduler class sched_ext (SCX), which allows scheduling policies to be implemented as BPF programs to
kernel/sched/core.c +9 -1
@@ -168,7 +168,10 @@ static inline int __task_prio(const struct task_struct *p)if (p->sched_class == &idle_sched_class)return MAX_RT_PRIO +NICE_WIDTH ;/* 140 */ -return MAX_RT_PRIO +MAX_NICE ;/* 120, squash fair */ +if (task_on_scx (p)) +return MAX_RT_PRIO +MAX_NICE +1 ;/* 120, squash ext */ + +return MAX_RT_PRIO +MAX_NICE ;/* 119, squash fair */ }/*
@@ -197,6 +200,11 @@ static inline bool prio_less(const struct task_struct *a,if (pa ==MAX_RT_PRIO +MAX_NICE )/* fair */ return cfs_prio_less (a, b, in_fi); +#ifdef CONFIG_SCHED_CLASS_EXT +if (pa ==MAX_RT_PRIO +MAX_NICE +1 )/* ext */ +return scx_prio_less (a, b, in_fi); +#endif +return false; }
kernel/sched/ext.c +238 -12
385 lines not on this page · raw message
kernel/sched/ext.h +5 -0
@@ -77,6 +77,11 @@ static inline const struct sched_class *next_active_class(const struct sched_clafor_active_class_range (class, (prev_class) > &ext_sched_class ? \ &ext_sched_class : (prev_class), (end_class)) +#ifdef CONFIG_SCHED_CORE +bool scx_prio_less (const struct task_struct *a,const struct task_struct *b, +bool in_fi); +#endif +#else /* CONFIG_SCHED_CLASS_EXT */ #define scx_enabled () false
tools/sched_ext/scx_qmap.bpf.c +85 -2
143 lines not on this page · raw message
tools/sched_ext/scx_qmap.c +3 -2
@@ -109,9 +109,10 @@ int main(int argc, char **argv)long nr_enqueued = skel->bss->nr_enqueued;long nr_dispatched = skel->bss->nr_dispatched; -printf ("stats : enq=%lu dsp=%lu delta=%ld reenq=%" PRIu64" deq=%" PRIu64"\n" , +printf ("stats : enq=%lu dsp=%lu delta=%ld reenq=%" PRIu64" deq=%" PRIu64" core=%" PRIu64"\n" , nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched, - skel->bss->nr_reenqueued, skel->bss->nr_dequeued); + skel->bss->nr_reenqueued, skel->bss->nr_dequeued, + skel->bss->nr_core_sched_execed);fflush (stdout);sleep (1 ); }
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 35/39 sched_ext: Add vtime-ordered priority queue to dispatch_q's
In reply to Tejun HeoCurrently, a dsq is always a FIFO. A task which is dispatched earlier gets consumed or executed earlier. While this is sufficient when dsq's are used for simple staging areas for tasks which are ready to execute, it'd make dsq's a lot more useful if they can implement custom ordering.
This patch adds a vtime-ordered priority queue to dsq's. When the BPF scheduler dispatches a task with the new scx_bpf_dispatch_vtime() helper, it can specify the vtime tha the task should be inserted at and the task is inserted into the priority queue in the dsq which is ordered according to time_before64() comparison of the vtime values.
A DSQ can either be a FIFO or priority queue and automatically switches between the two depending on whether scx_bpf_dispatch() or scx_bpf_dispatch_vtime() is used. Using the wrong variant while the DSQ already has the other type queued is not allowed and triggers an ops error. Built-in DSQs must always be FIFOs.
This makes it very easy for the BPF schedulers to implement proper vtime based scheduling within each dsq very easy and efficient at a negligible cost in terms of code complexity and overhead.
scx_simple and scx_example_flatcg are updated to default to weighted vtime scheduling (the latter within each cgroup). FIFO scheduling can be selected with -f option.
v4: - As allowing mixing priority queue and FIFO on the same DSQ sometimes
led to unexpected starvations, DSQs now error out if both modes are used at the same time and the built-in DSQs are no longer allowed to be priority queues.
- Explicit type struct scx_dsq_node added to contain fields needed to be linked on DSQs. This will be used to implement stateful iterator.
- Tasks are now always linked on dsq->list whether the DSQ is in FIFO or PRIQ mode. This confines PRIQ related complexities to the enqueue and dequeue paths. Other paths only need to look at dsq->list. This will also ease implementing BPF iterator.
- Print p->scx.dsq_flags in debug dump.
v3: - SCX_TASK_DSQ_ON_PRIQ flag is moved from p->scx.flags into its own
p->scx.dsq_flags. The flag is protected with the dsq lock unlike other
flags in p->scx.flags. This led to flag corruption in some cases.
- Add comments explaining the interaction between using consumption of
p->scx.slice to determine vtime progress and yielding.v2: - p->scx.dsq_vtime was not initialized on load or across cgroup
migrations leading to some tasks being stalled for extended period of time depending on how saturated the machine is. Fixed.
--- include/linux/sched/ext.h | 29 ++++- init/init_task.c | 2 +- kernel/sched/ext.c | 150 ++++++++++++++++++++--- tools/sched_ext/include/scx/common.bpf.h | 1 + tools/sched_ext/scx_flatcg.bpf.c | 80 +++++++++++- tools/sched_ext/scx_flatcg.c | 8 +- tools/sched_ext/scx_simple.bpf.c | 90 +++++++++++++- tools/sched_ext/scx_simple.c | 8 +- 8 files changed, 342 insertions(+), 26 deletions(-)
include/linux/sched/ext.h +26 -3
@@ -49,12 +49,15 @@ enum scx_dsq_id_flags { };/* -* Dispatch queue (dsq) is a simple FIFO which is used to buffer between the -* scheduler core and the BPF scheduler. See the documentation for more details. +* A dispatch queue (DSQ) can be either a FIFO or p->scx.dsq_vtime ordered +* queue. A built-in DSQ is always a FIFO. The built-in local DSQs are used to +* buffer between the scheduler core and the BPF scheduler. See the +* documentation for more details. */ struct scx_dispatch_q {raw_spinlock_t lock;struct list_head list;/* tasks in dispatch order */ +struct rb_root priq;/* used to order by p->scx.dsq_vtime */ u32 nr;u64 id;struct rhash_head hash_node;
@@ -86,6 +89,11 @@ enum scx_task_state { SCX_TASK_NR_STATES, }; +/* scx_entity.dsq_flags */ +enum scx_ent_dsq_flags { +SCX_TASK_DSQ_ON_PRIQ =1 <<0 ,/* task is queued on the priority queue of a dsq */ +}; +/* * Mask bits for scx_entity.kf_mask. Not all kfuncs can be called from * everywhere and the following bits track which kfunc sets are currently
@@ -111,13 +119,19 @@ enum scx_kf_mask { __SCX_KF_TERMINAL = SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST, }; +struct scx_dsq_node { +struct list_head list;/* dispatch order */ +struct rb_node priq;/* p->scx.dsq_vtime order */ +u32 flags;/* SCX_TASK_DSQ_* flags */ +}; +/* * The following is embedded in task_struct and contains all fields necessary * for a task to be scheduled by SCX. */ struct sched_ext_entity {struct scx_dispatch_q *dsq; -struct list_head dsq_node; +struct scx_dsq_node dsq_node;/* protected by dsq lock */ u32 flags;/* protected by rq lock */ u32 weight;s32 sticky_cpu;
@@ -149,6 +163,15 @@ struct sched_ext_entity {*/ u64 slice; +/* +* Used to order tasks when dispatching to the vtime-ordered priority +* queue of a dsq. This is usually set through scx_bpf_dispatch_vtime() +* but can also be modified directly by the BPF scheduler. Modifying it +* while a task is queued on a dsq may mangle the ordering and is not +* recommended. +*/ +u64 dsq_vtime; +/* * If set, reject future sched_setscheduler(2) calls updating the policy * to %SCHED_EXT with -%EACCES.
init/init_task.c +1 -1
@@ -101,7 +101,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {#endif #ifdef CONFIG_SCHED_CLASS_EXT .scx = { - .dsq_node = LIST_HEAD_INIT(init_task.scx.dsq_node), + .dsq_node.list = LIST_HEAD_INIT(init_task.scx.dsq_node.list), .sticky_cpu = -1, .holding_cpu = -1, .runnable_node = LIST_HEAD_INIT(init_task.scx.runnable_node),
kernel/sched/ext.c +135 -15
252 lines not on this page · raw message
tools/sched_ext/include/scx/common.bpf.h +1 -0
@@ -31,6 +31,7 @@ static inline void ___vmlinux_h_sanity_check___(void) s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) __ksym; s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym; void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym; +void scx_bpf_dispatch_vtime(struct task_struct *p, u64 dsq_id, u64 slice, u64 vtime, u64 enq_flags) __ksym; u32 scx_bpf_dispatch_nr_slots(void) __ksym; void scx_bpf_dispatch_cancel(void) __ksym; bool scx_bpf_consume(u64 dsq_id) __ksym;
tools/sched_ext/scx_flatcg.bpf.c +79 -1
140 lines not on this page · raw message
tools/sched_ext/scx_flatcg.c +6 -2
@@ -26,10 +26,11 @@ const char help_fmt[] ="\n" "See the top-level comment in .bpf.c for more details.\n" "\n" -"Usage: %s [-s SLICE_US] [-i INTERVAL] [-v]\n" +"Usage: %s [-s SLICE_US] [-i INTERVAL] [-f] [-v]\n" "\n" " -s SLICE_US Override slice duration\n" " -i INTERVAL Report interval\n" +" -f Use FIFO scheduling instead of weighted vtime scheduling\n" " -v Print libbpf debug messages\n" " -h Display this help and exit\n" ;
@@ -137,7 +138,7 @@ int main(int argc, char **argv) skel->rodata->nr_cpus =libbpf_num_possible_cpus (); -while ((opt =getopt (argc, argv,"s:i:dvh" )) !=-1 ) { +while ((opt =getopt (argc, argv,"s:i:dfvh" )) !=-1 ) {double v;switch (opt) {
@@ -153,6 +154,9 @@ int main(int argc, char **argv)case 'd' : dump_cgrps = true;break ; +case 'f' : + skel->rodata->fifo_sched = true; +break ;case 'v' : verbose = true;break ;
tools/sched_ext/scx_simple.bpf.c +88 -2
125 lines not on this page · raw message
tools/sched_ext/scx_simple.c +6 -2
@@ -17,8 +17,9 @@ const char help_fmt[] ="\n" "See the top-level comment in .bpf.c for more details.\n" "\n" -"Usage: %s [-v]\n" +"Usage: %s [-f] [-v]\n" "\n" +" -f Use FIFO scheduling instead of weighted vtime scheduling\n" " -v Print libbpf debug messages\n" " -h Display this help and exit\n" ;
@@ -70,8 +71,11 @@ int main(int argc, char **argv) restart: skel =SCX_OPS_OPEN (simple_ops, scx_simple); -while ((opt =getopt (argc, argv,"vh" )) !=-1 ) { +while ((opt =getopt (argc, argv,"fvh" )) !=-1 ) {switch (opt) { +case 'f' : + skel->rodata->fifo_sched = true; +break ;case 'v' : verbose = true;break ;
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 36/39 sched_ext: Implement DSQ iterator
In reply to Tejun HeoDSQs are very opaque in the consumption path. The BPF scheduler has no way of knowing which tasks are being considered and which is picked. This patch adds BPF DSQ iterator.
- Allows iterating tasks queued on a DSQ in the dispatch order or reverse from anywhere using bpf_for_each(scx_dsq) or calling the iterator kfuncs directly.
- Allows consuming arbitrary tasks on the DSQ in any order while iterating in the dispatch path using the new scx_bpf_consume_task().
- Has ordering guarantee where only tasks which were already queued when the iteration started are visible and consumable during the iteration.
Note that scx_bpf_consume_task() does a bit of dance to pass in the pointer to the iterator to __scx_bpf_consume_task(). This is to work around the current limitation in the BPF verifier where it doesn't allow the memory area used for an iterator to be passed into kfuncs. We should be able to remove this workaround in the future.
scx_qmap is updated to implement periodic dumping of the shared DSQ and a rather silly prioritization mechanism to demonstrate the use of DSQ iteration and selective consumption.
--- include/linux/sched/ext.h | 4 + kernel/sched/ext.c | 271 ++++++++++++++++++++++- tools/sched_ext/include/scx/common.bpf.h | 19 ++ tools/sched_ext/include/scx/compat.bpf.h | 15 ++ tools/sched_ext/include/scx/compat.h | 6 + tools/sched_ext/scx_qmap.bpf.c | 98 +++++++- tools/sched_ext/scx_qmap.c | 22 +- 7 files changed, 422 insertions(+), 13 deletions(-)
include/linux/sched/ext.h +4 -0
@@ -59,6 +59,7 @@ struct scx_dispatch_q {struct list_head list;/* tasks in dispatch order */ struct rb_root priq;/* used to order by p->scx.dsq_vtime */ u32 nr; +u64 seq;/* used by BPF iter */ u64 id;struct rhash_head hash_node;struct llist_node free_node;
@@ -92,6 +93,8 @@ enum scx_task_state {/* scx_entity.dsq_flags */ enum scx_ent_dsq_flags {SCX_TASK_DSQ_ON_PRIQ =1 <<0 ,/* task is queued on the priority queue of a dsq */ + +SCX_TASK_DSQ_CURSOR =1 <<31 ,/* iteration cursor, not a task */ };/*
@@ -132,6 +135,7 @@ struct scx_dsq_node {struct sched_ext_entity {struct scx_dispatch_q *dsq;struct scx_dsq_node dsq_node;/* protected by dsq lock */ +u64 dsq_seq;u32 flags;/* protected by rq lock */ u32 weight;s32 sticky_cpu;
kernel/sched/ext.c +266 -5
332 lines not on this page · raw message
tools/sched_ext/include/scx/common.bpf.h +19 -0
@@ -35,10 +35,14 @@ void scx_bpf_dispatch_vtime(struct task_struct *p, u64 dsq_id, u64 slice, u64 vt u32 scx_bpf_dispatch_nr_slots(void) __ksym; void scx_bpf_dispatch_cancel(void) __ksym; bool scx_bpf_consume(u64 dsq_id) __ksym; +bool __scx_bpf_consume_task(unsigned long it, struct task_struct *p) __ksym __weak; u32 scx_bpf_reenqueue_local(void) __ksym; void scx_bpf_kick_cpu(s32 cpu, u64 flags) __ksym; s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym; void scx_bpf_destroy_dsq(u64 dsq_id) __ksym; +int bpf_iter_scx_dsq_new(struct bpf_iter_scx_dsq *it, u64 dsq_id, bool rev) __ksym __weak; +struct task_struct *bpf_iter_scx_dsq_next (struct bpf_iter_scx_dsq *it) __ksym__weak ; +void bpf_iter_scx_dsq_destroy(struct bpf_iter_scx_dsq *it) __ksym __weak; void scx_bpf_exit_bstr(s64 exit_code, char *fmt, unsigned long long *data, u32 data__sz) __ksym __weak; void scx_bpf_error_bstr(char *fmt, unsigned long long *data, u32 data_len) __ksym; u32 scx_bpf_nr_cpu_ids(void) __ksym __weak;
@@ -55,6 +59,21 @@ bool scx_bpf_task_running(const struct task_struct *p) __ksym; s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym;struct cgroup *scx_bpf_task_cgroup (struct task_struct *p) __ksym; +/* +* Use the following as @it when calling scx_bpf_consume_task() from whitin +* bpf_for_each() loops. +*/ +#define BPF_FOR_EACH_ITER (&___it) + +/* hopefully temporary wrapper to work around BPF restriction */ +static inline bool scx_bpf_consume_task (struct bpf_iter_scx_dsq *it, +struct task_struct *p) +{ +unsigned long ptr; +bpf_probe_read_kernel (&ptr,sizeof (ptr), it); +return __scx_bpf_consume_task (ptr, p); +} +static inline __attribute__((format (printf,1 ,2 )))void ___scx_bpf_exit_format_checker (const char *fmt, ...) {}
tools/sched_ext/include/scx/compat.bpf.h +15 -0
@@ -56,6 +56,21 @@ static inline void __COMPAT_scx_bpf_switch_all(void)#define __COMPAT_HAS_CPUMASKS \ bpf_ksym_exists(scx_bpf_nr_cpu_ids) +/* +* Iteration and scx_bpf_consume_task() are new. The following become noop on +* older kernels. The users can switch to bpf_for_each(scx_dsq) and directly +* call scx_bpf_consume_task() in the future. +*/ +#define __COMPAT_DSQ_FOR_EACH (p, dsq_id, flags) \ + if (bpf_ksym_exists(bpf_iter_scx_dsq_new)) \ + bpf_for_each(scx_dsq, (p), (dsq_id), (flags)) + +static inline bool __COMPAT_scx_bpf_consume_task (struct bpf_iter_scx_dsq *it, +struct task_struct *p) +{ +return false; +} +/* * Define sched_ext_ops. This may be expanded to define multiple variants for * backward compatibility. See compat.h::SCX_OPS_LOAD/ATTACH().
tools/sched_ext/include/scx/compat.h +6 -0
@@ -123,6 +123,12 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field#define __COMPAT_HAS_CPUMASKS \ __COMPAT_has_ksym("scx_bpf_nr_cpu_ids") +/* +* DSQ iterator is new. Users will be able to assume existence in the future. +*/ +#define __COMPAT_HAS_DSQ_ITER \ + __COMPAT_has_ksym("bpf_iter_scx_dsq_new") +static inline long scx_hotplug_seq (void ) {int fd;
tools/sched_ext/scx_qmap.bpf.c +94 -4
147 lines not on this page · raw message
tools/sched_ext/scx_qmap.c +18 -4
@@ -20,7 +20,7 @@ const char help_fmt[] ="See the top-level comment in .bpf.c for more details.\n" "\n" "Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n" -" [-d PID] [-D LEN] [-p] [-v]\n" +" [-P] [-E PREFIX] [-d PID] [-D LEN] [-p] [-v]\n" "\n" " -s SLICE_US Override slice duration\n" " -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
@@ -28,6 +28,9 @@ const char help_fmt[] =" -T COUNT Stall every COUNT'th kernel thread\n" " -l COUNT Trigger dispatch infinite looping after COUNT dispatches\n" " -b COUNT Dispatch upto COUNT tasks together\n" +" -P Print out DSQ content to trace_pipe every second, use with -b\n" +" -E PREFIX Expedite consumption of threads w/ matching comm, use with -b\n" +" (e.g. match shell on a loaded system)\n" " -d PID Disallow a process from switching into SCHED_EXT (-1 for self)\n" " -D LEN Set scx_exit_info.dump buffer length\n" " -p Switch only tasks on SCHED_EXT policy intead of all\n"
@@ -61,7 +64,7 @@ int main(int argc, char **argv) skel =SCX_OPS_OPEN (qmap_ops, scx_qmap); -while ((opt =getopt (argc, argv,"s:e:t:T:l:b:d:D:pvh" )) !=-1 ) { +while ((opt =getopt (argc, argv,"s:e:t:T:l:b:PE:d:D:pvh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -81,6 +84,13 @@ int main(int argc, char **argv)case 'b' : skel->rodata->dsp_batch =strtoul (optarg,NULL ,0 );break ; +case 'P' : + skel->rodata->print_shared_dsq = true; +break ; +case 'E' : +strncpy (skel->rodata->exp_prefix, optarg, +sizeof (skel->rodata->exp_prefix) -1 ); +break ;case 'd' : skel->rodata->disallow_tgid =strtol (optarg,NULL ,0 );if (skel->rodata->disallow_tgid <0 )
@@ -102,6 +112,10 @@ int main(int argc, char **argv) } } +if (!__COMPAT_HAS_DSQ_ITER && + (skel->rodata->print_shared_dsq ||strlen (skel->rodata->exp_prefix))) +fprintf (stderr,"kernel doesn't support DSQ iteration\n" ); +SCX_OPS_LOAD (skel, qmap_ops, scx_qmap, uei); link =SCX_OPS_ATTACH (skel, qmap_ops);
@@ -109,10 +123,10 @@ int main(int argc, char **argv)long nr_enqueued = skel->bss->nr_enqueued;long nr_dispatched = skel->bss->nr_dispatched; -printf ("stats : enq=%lu dsp=%lu delta=%ld reenq=%" PRIu64" deq=%" PRIu64" core=%" PRIu64"\n" , +printf ("stats : enq=%lu dsp=%lu delta=%ld reenq=%" PRIu64" deq=%" PRIu64" core=%" PRIu64" exp=%" PRIu64"\n" , nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched, skel->bss->nr_reenqueued, skel->bss->nr_dequeued, - skel->bss->nr_core_sched_execed); + skel->bss->nr_core_sched_execed, skel->bss->nr_expedited);fflush (stdout);sleep (1 ); }
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 37/39 sched_ext: Add cpuperf support
In reply to Tejun HeoThis patch adds CPU performance monitoring and scaling support by integrating into schedutil. The following kfuncs are added:
- scx_bpf_cpuperf_cap(): Query the relative performance capacity of different CPUs in the system.
- scx_bpf_cpuperf_cur(): Query the current performance level of a CPU relative to its max performance.
- scx_bpf_cpuperf_set(): Set the current target performance level of a CPU.
This gives direct control over CPU performance setting to the BPF scheduler. The only change on the schedutil side is disabling frequency holding heuristics as it doesn't apply well to sched_ext schedulers which may have a lot weaker conneciton between tasks and their current / last CPU.
A toy implementation of cpuperf is added to scx_qmap as a demonstration of the feature.
--- kernel/sched/cpufreq_schedutil.c | 8 ++ kernel/sched/ext.c | 83 +++++++++++++++++- kernel/sched/ext.h | 3 +- kernel/sched/sched.h | 1 + tools/sched_ext/include/scx/common.bpf.h | 3 + tools/sched_ext/include/scx/compat.bpf.h | 26 ++++++ tools/sched_ext/scx_qmap.bpf.c | 106 +++++++++++++++++++++++ tools/sched_ext/scx_qmap.c | 8 ++ 8 files changed, 234 insertions(+), 4 deletions(-)
kernel/sched/cpufreq_schedutil.c +8 -0
@@ -332,6 +332,14 @@ static bool sugov_hold_freq(struct sugov_cpu *sg_cpu)unsigned long idle_calls;bool ret; +/* +* The heuristics in this function is for the fair class. For SCX, the +* performance target comes directly from the BPF scheduler. Let's just +* follow it. +*/ +if (scx_switched_all ()) +return false; +/* if capped by uclamp_max, always update to be in compliance */ if (uclamp_rq_is_capped (cpu_rq (sg_cpu->cpu)))return false;
kernel/sched/ext.c +81 -2
119 lines not on this page · raw message
kernel/sched/ext.h +1 -2
@@ -46,9 +46,8 @@ void init_sched_ext_class(void);static inline u32 scx_cpuperf_target (s32 cpu) { -/* for now, peg cpus at max performance while enabled */ if (scx_enabled ()) -return SCHED_CAPACITY_SCALE ; +return cpu_rq (cpu)->scx.cpuperf_target;else return 0 ; }
kernel/sched/sched.h +1 -0
@@ -732,6 +732,7 @@ struct scx_rq {u64 extra_enq_flags;/* see move_task_to_local_dsq() */ u32 nr_running;u32 flags; +u32 cpuperf_target;/* [0, SCHED_CAPACITY_SCALE] */ bool cpu_released;cpumask_var_t cpus_to_kick;cpumask_var_t cpus_to_kick_if_idle;
tools/sched_ext/include/scx/common.bpf.h +3 -0
@@ -45,6 +45,9 @@ struct task_struct *bpf_iter_scx_dsq_next(struct bpf_iter_scx_dsq *it) __ksym __ void bpf_iter_scx_dsq_destroy(struct bpf_iter_scx_dsq *it) __ksym __weak; void scx_bpf_exit_bstr(s64 exit_code, char *fmt, unsigned long long *data, u32 data__sz) __ksym __weak; void scx_bpf_error_bstr(char *fmt, unsigned long long *data, u32 data_len) __ksym; +u32 scx_bpf_cpuperf_cap(s32 cpu) __ksym __weak; +u32 scx_bpf_cpuperf_cur(s32 cpu) __ksym __weak; +void scx_bpf_cpuperf_set(s32 cpu, u32 perf) __ksym __weak; u32 scx_bpf_nr_cpu_ids(void) __ksym __weak;const struct cpumask *scx_bpf_get_possible_cpumask (void ) __ksym__weak ;const struct cpumask *scx_bpf_get_online_cpumask (void ) __ksym__weak ;
tools/sched_ext/include/scx/compat.bpf.h +26 -0
@@ -56,6 +56,32 @@ static inline void __COMPAT_scx_bpf_switch_all(void)#define __COMPAT_HAS_CPUMASKS \ bpf_ksym_exists(scx_bpf_nr_cpu_ids) +/* +* cpuperf is new. The followings become noop on older kernels. Callers can be +* updated to call cpuperf kfuncs directly in the future. +*/ +static inline u32 __COMPAT_scx_bpf_cpuperf_cap (s32 cpu) +{ +if (bpf_ksym_exists (scx_bpf_cpuperf_cap)) +return scx_bpf_cpuperf_cap (cpu); +else +return 1024 ; +} + +static inline u32 __COMPAT_scx_bpf_cpuperf_cur (s32 cpu) +{ +if (bpf_ksym_exists (scx_bpf_cpuperf_cur)) +return scx_bpf_cpuperf_cur (cpu); +else +return 1024 ; +} + +static inline void __COMPAT_scx_bpf_cpuperf_set (s32 cpu,u32 perf) +{ +if (bpf_ksym_exists (scx_bpf_cpuperf_set)) +return scx_bpf_cpuperf_set (cpu, perf); +} +/* * Iteration and scx_bpf_consume_task() are new. The following become noop on * older kernels. The users can switch to bpf_for_each(scx_dsq) and directly
tools/sched_ext/scx_qmap.bpf.c +106 -0
148 lines not on this page · raw message
tools/sched_ext/scx_qmap.c +8 -0
@@ -127,6 +127,14 @@ int main(int argc, char **argv) nr_enqueued, nr_dispatched, nr_enqueued - nr_dispatched, skel->bss->nr_reenqueued, skel->bss->nr_dequeued, skel->bss->nr_core_sched_execed, skel->bss->nr_expedited); +if (__COMPAT_has_ksym ("scx_bpf_cpuperf_cur" )) +printf ("cpuperf: cur min/avg/max=%u/%u/%u target min/avg/max=%u/%u/%u\n" , + skel->bss->cpuperf_min, + skel->bss->cpuperf_avg, + skel->bss->cpuperf_max, + skel->bss->cpuperf_target_min, + skel->bss->cpuperf_target_avg, + skel->bss->cpuperf_target_max);fflush (stdout);sleep (1 ); }
Signature · 1 line
2.44.0
Tejun Heo
maintainerPATCH 38/39 sched_ext: Documentation: scheduler: Document extensible scheduler class
In reply to Tejun HeoAdd Documentation/scheduler/sched-ext.rst which gives a high-level overview and pointers to the examples.
v5: - Updated to reflect /sys/kernel interface change. Kconfig options
added.
v4: - README improved, reformatted in markdown and renamed to README.md.
v3: - Added tools/sched_ext/README.
- Dropped _example prefix from scheduler names.
v2: - Apply minor edits suggested by Bagas. Caveats section dropped as all
of them are addressed.
--- Documentation/scheduler/index.rst | 1 + Documentation/scheduler/sched-ext.rst | 307 ++++++++++++++++++++++++++ include/linux/sched/ext.h | 2 + kernel/Kconfig.preempt | 2 + kernel/sched/ext.c | 2 + kernel/sched/ext.h | 2 + tools/sched_ext/README.md | 270 ++++++++++++++++++++++ 7 files changed, 586 insertions(+) create mode 100644 Documentation/scheduler/sched-ext.rst create mode 100644 tools/sched_ext/README.md
Documentation/scheduler/index.rst +1 -0
@@ -20,6 +20,7 @@ Scheduler sched-nice-design sched-rt-group sched-stats + sched-ext sched-debug text_files
Documentation/scheduler/sched-ext.rst new file +307 -0
307 lines not on this page · raw message
include/linux/sched/ext.h +2 -0
@@ -1,5 +1,7 @@/* SPDX-License-Identifier: GPL-2.0 */ /* +* BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst +* * Copyright (c) 2022 Meta Platforms, Inc. and affiliates. * Copyright (c) 2022 Tejun Heo <[email protected]> * Copyright (c) 2022 David Vernet <[email protected]>
kernel/Kconfig.preempt +2 -0
@@ -154,3 +154,5 @@ config SCHED_CLASS_EXT wish to implement scheduling policies. The struct_ops structure exported by sched_ext is struct sched_ext_ops, and is conceptually similar to struct sched_class. + + See Documentation/scheduler/sched-ext.rst for more details.
kernel/sched/ext.c +2 -0
@@ -1,5 +1,7 @@/* SPDX-License-Identifier: GPL-2.0 */ /* +* BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst +* * Copyright (c) 2022 Meta Platforms, Inc. and affiliates. * Copyright (c) 2022 Tejun Heo <[email protected]> * Copyright (c) 2022 David Vernet <[email protected]>
kernel/sched/ext.h +2 -0
@@ -1,5 +1,7 @@/* SPDX-License-Identifier: GPL-2.0 */ /* +* BPF extensible scheduler class: Documentation/scheduler/sched-ext.rst +* * Copyright (c) 2022 Meta Platforms, Inc. and affiliates. * Copyright (c) 2022 Tejun Heo <[email protected]> * Copyright (c) 2022 David Vernet <[email protected]>
tools/sched_ext/README.md new file +270 -0
270 lines not on this page · raw message
Signature · 1 line
2.44.0
Re: [PATCH 38/39] sched_ext: Documentation: scheduler: Document extensible scheduler class
Quoted 3 lines · Tejun Heo
Add
Documentation/scheduler/sched-ext.rstwhich gives a high-level overview and pointers to the examples.
The doc LGTM, thanks!
Signature · 1 line
An old man doll... just what I always wanted! - Clara
Tejun Heo
maintainerPATCH 39/39 sched_ext: Add selftests
In reply to Tejun HeoFrom: David Vernet <[email protected]>
Add basic selftests.
--- tools/testing/selftests/sched_ext/.gitignore | 6 + tools/testing/selftests/sched_ext/Makefile | 216 ++++++++++++++++++ tools/testing/selftests/sched_ext/config | 9 + .../sched_ext/ddsp_bogus_dsq_fail.bpf.c | 42 ++++ .../selftests/sched_ext/ddsp_bogus_dsq_fail.c | 57 +++++ .../sched_ext/ddsp_vtimelocal_fail.bpf.c | 39 ++++ .../sched_ext/ddsp_vtimelocal_fail.c | 56 +++++ .../sched_ext/enq_last_no_enq_fails.bpf.c | 21 ++ .../sched_ext/enq_last_no_enq_fails.c | 60 +++++ .../sched_ext/enq_select_cpu_fails.bpf.c | 43 ++++ .../sched_ext/enq_select_cpu_fails.c | 61 +++++ tools/testing/selftests/sched_ext/exit.bpf.c | 84 +++++++ tools/testing/selftests/sched_ext/exit.c | 55 +++++ tools/testing/selftests/sched_ext/exit_test.h | 20 ++ .../testing/selftests/sched_ext/hotplug.bpf.c | 55 +++++ tools/testing/selftests/sched_ext/hotplug.c | 168 ++++++++++++++ .../selftests/sched_ext/hotplug_test.h | 15 ++ .../sched_ext/init_enable_count.bpf.c | 53 +++++ .../selftests/sched_ext/init_enable_count.c | 166 ++++++++++++++ .../testing/selftests/sched_ext/maximal.bpf.c | 164 +++++++++++++ tools/testing/selftests/sched_ext/maximal.c | 51 +++++ .../selftests/sched_ext/maybe_null.bpf.c | 26 +++ .../testing/selftests/sched_ext/maybe_null.c | 40 ++++ .../selftests/sched_ext/maybe_null_fail.bpf.c | 25 ++ .../testing/selftests/sched_ext/minimal.bpf.c | 21 ++ tools/testing/selftests/sched_ext/minimal.c | 58 +++++ .../selftests/sched_ext/prog_run.bpf.c | 32 +++ tools/testing/selftests/sched_ext/prog_run.c | 78 +++++++ .../testing/selftests/sched_ext/reload_loop.c | 75 ++++++ tools/testing/selftests/sched_ext/runner.c | 201 ++++++++++++++++ tools/testing/selftests/sched_ext/scx_test.h | 131 +++++++++++ .../selftests/sched_ext/select_cpu_dfl.bpf.c | 40 ++++ .../selftests/sched_ext/select_cpu_dfl.c | 72 ++++++ .../sched_ext/select_cpu_dfl_nodispatch.bpf.c | 89 ++++++++ .../sched_ext/select_cpu_dfl_nodispatch.c | 72 ++++++ .../sched_ext/select_cpu_dispatch.bpf.c | 41 ++++ .../selftests/sched_ext/select_cpu_dispatch.c | 70 ++++++ .../select_cpu_dispatch_bad_dsq.bpf.c | 37 +++ .../sched_ext/select_cpu_dispatch_bad_dsq.c | 56 +++++ .../select_cpu_dispatch_dbl_dsp.bpf.c | 38 +++ .../sched_ext/select_cpu_dispatch_dbl_dsp.c | 56 +++++ .../sched_ext/select_cpu_vtime.bpf.c | 92 ++++++++ .../selftests/sched_ext/select_cpu_vtime.c | 59 +++++ .../selftests/sched_ext/test_example.c | 49 ++++ tools/testing/selftests/sched_ext/util.c | 71 ++++++ tools/testing/selftests/sched_ext/util.h | 13 ++ 46 files changed, 2983 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/.gitignore create mode 100644 tools/testing/selftests/sched_ext/Makefile create mode 100644 tools/testing/selftests/sched_ext/config create mode 100644 tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c create mode 100644 tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.c create mode 100644 tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c create mode 100644 tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.c create mode 100644 tools/testing/selftests/sched_ext/enq_last_no_enq_fails.bpf.c create mode 100644 tools/testing/selftests/sched_ext/enq_last_no_enq_fails.c create mode 100644 tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c create mode 100644 tools/testing/selftests/sched_ext/enq_select_cpu_fails.c create mode 100644 tools/testing/selftests/sched_ext/exit.bpf.c create mode 100644 tools/testing/selftests/sched_ext/exit.c create mode 100644 tools/testing/selftests/sched_ext/exit_test.h create mode 100644 tools/testing/selftests/sched_ext/hotplug.bpf.c create mode 100644 tools/testing/selftests/sched_ext/hotplug.c create mode 100644 tools/testing/selftests/sched_ext/hotplug_test.h create mode 100644 tools/testing/selftests/sched_ext/init_enable_count.bpf.c create mode 100644 tools/testing/selftests/sched_ext/init_enable_count.c create mode 100644 tools/testing/selftests/sched_ext/maximal.bpf.c create mode 100644 tools/testing/selftests/sched_ext/maximal.c create mode 100644 tools/testing/selftests/sched_ext/maybe_null.bpf.c create mode 100644 tools/testing/selftests/sched_ext/maybe_null.c create mode 100644 tools/testing/selftests/sched_ext/maybe_null_fail.bpf.c create mode 100644 tools/testing/selftests/sched_ext/minimal.bpf.c create mode 100644 tools/testing/selftests/sched_ext/minimal.c create mode 100644 tools/testing/selftests/sched_ext/prog_run.bpf.c create mode 100644 tools/testing/selftests/sched_ext/prog_run.c create mode 100644 tools/testing/selftests/sched_ext/reload_loop.c create mode 100644 tools/testing/selftests/sched_ext/runner.c create mode 100644 tools/testing/selftests/sched_ext/scx_test.h create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dfl.bpf.c create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dfl.c create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch.bpf.c create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch.c create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.bpf.c create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.c create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.bpf.c create mode 100644 tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.c create mode 100644 tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c create mode 100644 tools/testing/selftests/sched_ext/select_cpu_vtime.c create mode 100644 tools/testing/selftests/sched_ext/test_example.c create mode 100644 tools/testing/selftests/sched_ext/util.c create mode 100644 tools/testing/selftests/sched_ext/util.h
tools/testing/selftests/sched_ext/.gitignore new file +6 -0
@@ -0,0 +1,6 @@ +* +!*.c +!*.h +!Makefile +!.gitignore +!config
tools/testing/selftests/sched_ext/Makefile new file +216 -0
216 lines not on this page · raw message
tools/testing/selftests/sched_ext/config new file +9 -0
@@ -0,0 +1,9 @@ +CONFIG_SCHED_DEBUG=y +CONFIG_SCHED_CLASS_EXT=y +CONFIG_CGROUPS=y +CONFIG_CGROUP_SCHED=y +CONFIG_EXT_GROUP_SCHED=y +CONFIG_BPF=y +CONFIG_BPF_SYSCALL=y +CONFIG_DEBUG_INFO=y +CONFIG_DEBUG_INFO_BTF=y
tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c new file +42 -0
42 lines not on this page · raw message
tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.c new file +57 -0
57 lines not on this page · raw message
tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c new file +39 -0
@@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. +* Copyright (c) 2024 David Vernet <[email protected]> +* Copyright (c) 2024 Tejun Heo <[email protected]> +*/ +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +UEI_DEFINE (uei); + +s32 BPF_STRUCT_OPS (ddsp_vtimelocal_fail_select_cpu ,struct task_struct *p, +s32 prev_cpu,u64 wake_flags) +{ +s32 cpu =scx_bpf_pick_idle_cpu (p->cpus_ptr,0 ); + +if (cpu >=0 ) { +/* Shouldn't be allowed to vtime dispatch to a builtin DSQ. */ +scx_bpf_dispatch_vtime (p,SCX_DSQ_LOCAL ,SCX_SLICE_DFL , + p->scx.dsq_vtime,0 ); +return cpu; + } + +return prev_cpu; +} + +void BPF_STRUCT_OPS (ddsp_vtimelocal_fail_exit ,struct scx_exit_info *ei) +{ +UEI_RECORD (uei, ei); +} + +SEC(".struct_ops.link") +struct sched_ext_ops ddsp_vtimelocal_fail_ops = { + .select_cpu = ddsp_vtimelocal_fail_select_cpu, + .exit = ddsp_vtimelocal_fail_exit, + .name ="ddsp_vtimelocal_fail" , + .timeout_ms =1000U , +};
tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.c new file +56 -0
56 lines not on this page · raw message
tools/testing/selftests/sched_ext/enq_last_no_enq_fails.bpf.c new file +21 -0
@@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* A scheduler that validates the behavior of direct dispatching with a default +* select_cpu implementation. +* +* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. +* Copyright (c) 2023 David Vernet <[email protected]> +* Copyright (c) 2023 Tejun Heo <[email protected]> +*/ + +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +SEC(".struct_ops.link") +struct sched_ext_ops enq_last_no_enq_fails_ops = { + .name ="enq_last_no_enq_fails" , +/* Need to define ops.enqueue() with SCX_OPS_ENQ_LAST */ + .flags =SCX_OPS_ENQ_LAST , + .timeout_ms =1000U , +};
tools/testing/selftests/sched_ext/enq_last_no_enq_fails.c new file +60 -0
60 lines not on this page · raw message
tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c new file +43 -0
43 lines not on this page · raw message
tools/testing/selftests/sched_ext/enq_select_cpu_fails.c new file +61 -0
61 lines not on this page · raw message
tools/testing/selftests/sched_ext/exit.bpf.c new file +84 -0
84 lines not on this page · raw message
tools/testing/selftests/sched_ext/exit.c new file +55 -0
55 lines not on this page · raw message
tools/testing/selftests/sched_ext/exit_test.h new file +20 -0
@@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. +* Copyright (c) 2024 David Vernet <[email protected]> +*/ + +#ifndef __EXIT_TEST_H__ +#define __EXIT_TEST_H__ + +enum exit_test_case { +EXIT_SELECT_CPU , +EXIT_ENQUEUE , +EXIT_DISPATCH , +EXIT_ENABLE , +EXIT_INIT_TASK , +EXIT_INIT , +NUM_EXITS , +}; + +#endif // # __EXIT_TEST_H__
tools/testing/selftests/sched_ext/hotplug.bpf.c new file +55 -0
55 lines not on this page · raw message
tools/testing/selftests/sched_ext/hotplug.c new file +168 -0
168 lines not on this page · raw message
tools/testing/selftests/sched_ext/hotplug_test.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. +* Copyright (c) 2024 David Vernet <[email protected]> +*/ + +#ifndef __HOTPLUG_TEST_H__ +#define __HOTPLUG_TEST_H__ + +enum hotplug_test_flags { +HOTPLUG_EXIT_RSN =1LLU <<0 , +HOTPLUG_ONLINING =1LLU <<1 , +}; + +#endif // # __HOTPLUG_TEST_H__
tools/testing/selftests/sched_ext/init_enable_count.bpf.c new file +53 -0
53 lines not on this page · raw message
tools/testing/selftests/sched_ext/init_enable_count.c new file +166 -0
166 lines not on this page · raw message
tools/testing/selftests/sched_ext/maximal.bpf.c new file +164 -0
164 lines not on this page · raw message
tools/testing/selftests/sched_ext/maximal.c new file +51 -0
51 lines not on this page · raw message
tools/testing/selftests/sched_ext/maybe_null.bpf.c new file +26 -0
@@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. +*/ + +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +u64 vtime_test; + +void BPF_STRUCT_OPS (maybe_null_running ,struct task_struct *p) +{} + +void BPF_STRUCT_OPS (maybe_null_success_dispatch ,s32 cpu,struct task_struct *p) +{ +if (p !=NULL ) + vtime_test = p->scx.dsq_vtime; +} + +SEC(".struct_ops.link") +struct sched_ext_ops maybe_null_success = { + .dispatch = maybe_null_success_dispatch, + .enable = maybe_null_running, + .name ="minimal" , +};
tools/testing/selftests/sched_ext/maybe_null.c new file +40 -0
@@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. +*/ +#include <bpf/bpf.h> +#include <scx/common.h> +#include <sys/wait.h> +#include <unistd.h> +#include "maybe_null.bpf.skel.h" +#include "maybe_null_fail.bpf.skel.h" +#include "scx_test.h" + +static enum scx_test_status run (void *ctx) +{ +struct maybe_null *skel; +struct maybe_null_fail *fail_skel; + + skel =maybe_null__open_and_load (); +if (!skel) { +SCX_ERR ("Failed to open and load maybe_null skel" ); +return SCX_TEST_FAIL ; + } +maybe_null__destroy (skel); + + fail_skel =maybe_null_fail__open_and_load (); +if (fail_skel) { +maybe_null_fail__destroy (fail_skel); +SCX_ERR ("Should failed to open and load maybe_null_fail skel" ); +return SCX_TEST_FAIL ; + } + +return SCX_TEST_PASS ; +} + +struct scx_test maybe_null = { + .name ="maybe_null" , + .description ="Verify if PTR_MAYBE_NULL work for .dispatch" , + .run = run, +}; +REGISTER_SCX_TEST (&maybe_null)
tools/testing/selftests/sched_ext/maybe_null_fail.bpf.c new file +25 -0
@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. +*/ + +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +u64 vtime_test; + +void BPF_STRUCT_OPS (maybe_null_running ,struct task_struct *p) +{} + +void BPF_STRUCT_OPS (maybe_null_fail_dispatch ,s32 cpu,struct task_struct *p) +{ + vtime_test = p->scx.dsq_vtime; +} + +SEC(".struct_ops.link") +struct sched_ext_ops maybe_null_fail = { + .dispatch = maybe_null_fail_dispatch, + .enable = maybe_null_running, + .name ="minimal" , +};
tools/testing/selftests/sched_ext/minimal.bpf.c new file +21 -0
@@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* A completely minimal scheduler. +* +* This scheduler defines the absolute minimal set of struct sched_ext_ops +* fields: its name. It should _not_ fail to be loaded, and can be used to +* exercise the default scheduling paths in ext.c. +* +* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. +* Copyright (c) 2023 David Vernet <[email protected]> +* Copyright (c) 2023 Tejun Heo <[email protected]> +*/ + +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +SEC(".struct_ops.link") +struct sched_ext_ops minimal_ops = { + .name ="minimal" , +};
tools/testing/selftests/sched_ext/minimal.c new file +58 -0
58 lines not on this page · raw message
tools/testing/selftests/sched_ext/prog_run.bpf.c new file +32 -0
@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* A scheduler that validates that we can invoke sched_ext kfuncs in +* BPF_PROG_TYPE_SYSCALL programs. +* +* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. +* Copyright (c) 2024 David Vernet <[email protected]> +*/ + +#include <scx/common.bpf.h> + +UEI_DEFINE (uei); + +char _license[] SEC("license") = "GPL"; + +SEC("syscall") +int BPF_PROG (prog_run_syscall ) +{ +scx_bpf_exit (0xdeadbeef ,"Exited from PROG_RUN" ); +return 0 ; +} + +void BPF_STRUCT_OPS (prog_run_exit ,struct scx_exit_info *ei) +{ +UEI_RECORD (uei, ei); +} + +SEC(".struct_ops.link") +struct sched_ext_ops prog_run_ops = { + .exit = prog_run_exit, + .name ="prog_run" , +};
tools/testing/selftests/sched_ext/prog_run.c new file +78 -0
78 lines not on this page · raw message
tools/testing/selftests/sched_ext/reload_loop.c new file +75 -0
75 lines not on this page · raw message
tools/testing/selftests/sched_ext/runner.c new file +201 -0
201 lines not on this page · raw message
tools/testing/selftests/sched_ext/scx_test.h new file +131 -0
131 lines not on this page · raw message
tools/testing/selftests/sched_ext/select_cpu_dfl.bpf.c new file +40 -0
@@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* A scheduler that validates the behavior of direct dispatching with a default +* select_cpu implementation. +* +* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. +* Copyright (c) 2023 David Vernet <[email protected]> +* Copyright (c) 2023 Tejun Heo <[email protected]> +*/ + +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +bool saw_local = false; + +static bool task_is_test (const struct task_struct *p) +{ +return !bpf_strncmp (p->comm,9 ,"select_cpu" ); +} + +void BPF_STRUCT_OPS (select_cpu_dfl_enqueue ,struct task_struct *p, +u64 enq_flags) +{ +const struct cpumask *idle_mask =scx_bpf_get_idle_cpumask (); + +if (task_is_test (p) && +bpf_cpumask_test_cpu (scx_bpf_task_cpu (p), idle_mask)) { + saw_local = true; + } +scx_bpf_put_idle_cpumask (idle_mask); + +scx_bpf_dispatch (p,SCX_DSQ_GLOBAL ,SCX_SLICE_DFL , enq_flags); +} + +SEC(".struct_ops.link") +struct sched_ext_ops select_cpu_dfl_ops = { + .enqueue = select_cpu_dfl_enqueue, + .name ="select_cpu_dfl" , +};
tools/testing/selftests/sched_ext/select_cpu_dfl.c new file +72 -0
72 lines not on this page · raw message
tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c new file +89 -0
89 lines not on this page · raw message
tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c new file +72 -0
72 lines not on this page · raw message
tools/testing/selftests/sched_ext/select_cpu_dispatch.bpf.c new file +41 -0
41 lines not on this page · raw message
tools/testing/selftests/sched_ext/select_cpu_dispatch.c new file +70 -0
70 lines not on this page · raw message
tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.bpf.c new file +37 -0
@@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* A scheduler that validates the behavior of direct dispatching with a default +* select_cpu implementation. +* +* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. +* Copyright (c) 2023 David Vernet <[email protected]> +* Copyright (c) 2023 Tejun Heo <[email protected]> +*/ + +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +UEI_DEFINE (uei); + +s32 BPF_STRUCT_OPS (select_cpu_dispatch_bad_dsq_select_cpu ,struct task_struct *p, +s32 prev_cpu,u64 wake_flags) +{ +/* Dispatching to a random DSQ should fail. */ +scx_bpf_dispatch (p,0xcafef00d ,SCX_SLICE_DFL ,0 ); + +return prev_cpu; +} + +void BPF_STRUCT_OPS (select_cpu_dispatch_bad_dsq_exit ,struct scx_exit_info *ei) +{ +UEI_RECORD (uei, ei); +} + +SEC(".struct_ops.link") +struct sched_ext_ops select_cpu_dispatch_bad_dsq_ops = { + .select_cpu = select_cpu_dispatch_bad_dsq_select_cpu, + .exit = select_cpu_dispatch_bad_dsq_exit, + .name ="select_cpu_dispatch_bad_dsq" , + .timeout_ms =1000U , +};
tools/testing/selftests/sched_ext/select_cpu_dispatch_bad_dsq.c new file +56 -0
56 lines not on this page · raw message
tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.bpf.c new file +38 -0
@@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* A scheduler that validates the behavior of direct dispatching with a default +* select_cpu implementation. +* +* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. +* Copyright (c) 2023 David Vernet <[email protected]> +* Copyright (c) 2023 Tejun Heo <[email protected]> +*/ + +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +UEI_DEFINE (uei); + +s32 BPF_STRUCT_OPS (select_cpu_dispatch_dbl_dsp_select_cpu ,struct task_struct *p, +s32 prev_cpu,u64 wake_flags) +{ +/* Dispatching twice in a row is disallowed. */ +scx_bpf_dispatch (p,SCX_DSQ_GLOBAL ,SCX_SLICE_DFL ,0 ); +scx_bpf_dispatch (p,SCX_DSQ_GLOBAL ,SCX_SLICE_DFL ,0 ); + +return prev_cpu; +} + +void BPF_STRUCT_OPS (select_cpu_dispatch_dbl_dsp_exit ,struct scx_exit_info *ei) +{ +UEI_RECORD (uei, ei); +} + +SEC(".struct_ops.link") +struct sched_ext_ops select_cpu_dispatch_dbl_dsp_ops = { + .select_cpu = select_cpu_dispatch_dbl_dsp_select_cpu, + .exit = select_cpu_dispatch_dbl_dsp_exit, + .name ="select_cpu_dispatch_dbl_dsp" , + .timeout_ms =1000U , +};
tools/testing/selftests/sched_ext/select_cpu_dispatch_dbl_dsp.c new file +56 -0
56 lines not on this page · raw message
tools/testing/selftests/sched_ext/select_cpu_vtime.bpf.c new file +92 -0
92 lines not on this page · raw message
tools/testing/selftests/sched_ext/select_cpu_vtime.c new file +59 -0
59 lines not on this page · raw message
tools/testing/selftests/sched_ext/test_example.c new file +49 -0
49 lines not on this page · raw message
tools/testing/selftests/sched_ext/util.c new file +71 -0
71 lines not on this page · raw message
tools/testing/selftests/sched_ext/util.h new file +13 -0
@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. +* Copyright (c) 2024 David Vernet <[email protected]> +*/ + +#ifndef __SCX_TEST_UTIL_H__ +#define __SCX_TEST_UTIL_H__ + +long file_read_long (const char *path); +int file_write_long (const char *path,long val); + +#endif // __SCX_TEST_H__
Signature · 1 line
2.44.0
Peter Zijlstra
maintainerCan you please put your efforts and the touted Google collaboration in fixing the existing cgroup mess?
Tejun Heo
maintainerHello, Peter.
Quoted 2 lines · Peter Zijlstra
Can you please put your efforts and the touted Google collaboration in fixing the existing cgroup mess?
I suppose you're referring to Rik's flattened hierarchy patchset.
https://lore.kernel.org/all/[email protected]
Rik spent a lot of time and energy on it and IIRC one of the reasons why it didn't get pushed further was the lack of any enthusiasm or support from the upstream community.
We can resurrect the discussion on that patchset but how is that connected to sched_ext? One of the example schedulers, scx_flatcg, does employ the same approach with a twist (instead of flattening completely, it builds two-level hierarchy so that a leaf cgroup can be treated as a single entity) to demonstrate the idea but the two projects don't really have much in common otherwise. Are you saying that Meta and Google working on the flattened hierarchy is a prerequisite for landing sched_ext?
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 8 lines · Tejun Heo
Hello, Peter.
Quoted 2 lines · Peter Zijlstra
Can you please put your efforts and the touted Google collaboration in fixing the existing cgroup mess?
I suppose you're referring to Rik's flattened hierarchy patchset.
Quoted 8 lines
https://lore.kernel.org/all/[email protected]
Rik spent a lot of time and energy on it and IIRC one of the reasons why it didn't get pushed further was the lack of any enthusiasm or support from the upstream community.
We can resurrect the discussion on that patchset but how is that connected to
sched_ext?
I'm absolutely not taking any of this until at the very least the cgroup situation that's been created is solved. And even then, I fundamentally believe the approach to be detrimental to the scheduler eco-system. Witness the metric ton of toy schedulers written for it, that's all effort not put into improving the existing code.
You guys Google/Facebook got us the cgroup thing, Google did a lot of the work for cpu-cgroup, and now you Facebook say you can't live with it because it's too expensive. Yes Rik did put a lot of effort into it, but Google shot it down. What am I to do?
You Google/Facebook are touting collaboration, collaborate on fixing it. Instead of re-posting this over and over. After all, your main motivation for starting this was the cpu-cgroup overhead.
From where I'm sitting, you created a problem (cpu-cgroup) and now you're creating an even bigger problem as a work-around. Very much not appreciated.
Tejun Heo
maintainerHello,
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Tejun Heo
We can resurrect the discussion on that patchset but how is that connected to
sched_ext?I'm absolutely not taking any of this until at the very least the cgroup situation that's been created is solved. And even then, I fundamentally believe the approach to be detrimental to the scheduler eco-system.
Please see below for more on the flattened hierarchy patchset. However, no matter how that discussion works out, what you seem to be suggesting - suspending discussion or further push for upstream on sched_ext until a mostly unrelated work is done - doesn't seem reasonable, especially when most of the input that you have provided is not constructive.
Even if we agree that, for some reason, the two projects are linked and Meta and Google owe to push the flattened hierarchy patchset to land sched_ext upstream, it should be obvious how your proposition puts us in an impossible spot - A is a prerequisite for B but B isn't going to happen. That's not a motivating situation for anyone.
If working on the flattened hierarchy patchset is something you want us to commit to as a gesture of good will, we can surely consider that, but that shouldn't block further discussions on sched_ext or its upstreaming.
(I reordered your comment about the number of sched_ext schedulers and developer attention towards the end of the reply to avoid jumping back and forth between subjects.)
Quoted 1 line
You guys Google/Facebook got us the cgroup thing, Google did a lot of
We can't divorce ourselves completely from the organizations that we work for but the above is still a pretty broad stroke. Neither David nor I was involved in the CPU controller design or implementation and I don't think it's the same group of people on the Google side either. We sure can discuss how to proceed on the flattened hierarchy patchset but I don't think the picture you're painting is a fair depiction of the overall situation.
Quoted 3 lines
the work for cpu-cgroup, and now you Facebook say you can't live with it because it's too expensive. Yes Rik did put a lot of effort into it, but Google shot it down. What am I to do?
You could have encouraged and guided the project if it felt important enough. You didn't have to but that was an option.
Quoted 3 lines
You Google/Facebook are touting collaboration, collaborate on fixing it. Instead of re-posting this over and over. After all, your main motivation for starting this was the cpu-cgroup overhead.
The hierarchical scheduling overhead isn't the main motivation for us. We can't use the CPU controller for all workloads and while it'd be nice to improve that, it's pretty easy to work around especially with constantly increasing number of CPUs. Currently, most sched_ext experiments are without cgroups and even when cgroups are considered, they're just used as grouping hints.
In fact, we want to try implementing hierarchical scheduling by dynamically soft-affinitizing cgroups to CPUs, which would be a bridge too far for the in-kernel scheduler, at least for now, as it wouldn't be able to handle custom affinities properly, but it is an idea worth exploring. Enabling experiments like that is definitely one of our main motivations.
Quoted 3 lines
From where I'm sitting, you created a problem (cpu-cgroup) and now you're creating an even bigger problem as a work-around. Very much not appreciated.
I have a hard time agreeing. These projects don't overlap all that much. Their scopes are wildly different. That said, if this is somehow the blocker, we can talk and try to find a solution but such a solution would have to be reasonable from our end too. How else would it work?
Quoted 2 lines
Witness the metric ton of toy schedulers written for it, that's all effort not put into improving the existing code.
This view works only if you assume that the entire world contains only a handful of developers who can work on schedulers. The only way that would be the case is if the barrier of entry is raised unreasonably high. Sometimes a high barrier of entry can't be avoided or is beneficial. However, if it's pushed up high enough to leave only a handful of people to work on an area as large as scheduling, something probably is wrong.
You know better than anyone that there's no such thing as the perfect scheduler for all, or even most, workloads. There are too many interacting factors and second-order effects for a single implementation, no matter how advanced, to be perfect or even great for the multitudes of situations that scheduling encounters. With hardware and workloads becoming more complex, the situation isn't getting any better. This partially explains why we can easily achieve significantly better behaviors for specific workloads even with a toy scheduler which is just there to demonstrate an idea.
The built-in scheduler has to be good enough for everyone, and, thanks to the effort of you and the other sched maintainers, it serves that role admirably. However, that requirement also comes with stringent constraints. Radical ideas are difficult to play with. Each change has to make some sense for every use case. Nothing drastic can be introduced unless the future path can reasonably be forecast. So, development efforts must be highly orchestrated and stay consistent which justifies a higher barrier of entry and strict control.
Yet, the many different ways that even simple schedulers can demonstrates sometimes significant behavior and performance benefits for specific workloads suggest that there are a lot of low hanging fruits in the area. Low hanging fruits that we can't easily reach from our current local optimum. A single implementation which has to satisfy all users all the time is unlikely to be an effective vehicle for mapping out such landscape.
I believe we agree that we want more people contributing to the scheduling area. We need that. However, I have a hard time seeing how that would be achieved in the current structure. Most people can't afford to sink six months, a year, two years into a project only to eventually be nacked without any way to deploy and prove their ideas and efforts. Unfortunately, that is where we end up today in many cases.
There are many smart people with bright ideas just outside the fence who are eager to develop, tune and even just play with schedulers. I believe they will flourish when they can work in an environment where scheduling experimentation is accessible and encouraged. In fact, we are already seeing that. Out of the four non-trivial sched_ext schedulers, three are either primarily driven by or have significant contributions from people who had not and wouldn't have worked on the in-kernel schedulers at all.
So, here's my proposition. Let's please open it up. sched_ext hooks into sched infra but the contact surface is limited and we'll try our best to stay out of your way. I can't promise that it won't ever get in your way, but, if it ever does, just ping me and David. Resolving such situations would be our highest priority. Let us and others try out crazy ideas and find out what works.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 7 lines · Tejun Heo
Quoted 3 lines
You Google/Facebook are touting collaboration, collaborate on fixing it. Instead of re-posting this over and over. After all, your main motivation for starting this was the cpu-cgroup overhead.
The hierarchical scheduling overhead isn't the main motivation for us. We can't use the CPU controller for all workloads and while it'd be nice to improve that,
Hurmph, I had the impression from the earlier threads that this ~5% cgroup overhead was most definitely a problem and a motivator for all this.
The overhead was prohibitive, it was claimed, and you needed a solution. Did not previous versions use this very argument in order to push for all this?
By improving the cgroup mess -- I very much agree that the cgroup thing is not very nice. This whole argument goes away and we all get a better cgroup implementation.
Quoted 6 lines
This view works only if you assume that the entire world contains only a handful of developers who can work on schedulers. The only way that would be the case is if the barrier of entry is raised unreasonably high. Sometimes a high barrier of entry can't be avoided or is beneficial. However, if it's pushed up high enough to leave only a handful of people to work on an area as large as scheduling, something probably is wrong.
I've never really felt there were too few sched patches to stare at on any one day (quite the opposite on many days in fact).
There have also always been plenty out of tree scheduler patches -- although I rarely if ever have time to look at them.
Writing a custom scheduler isn't that hard, simply ripping out fair_sched_class and replacing it with something simple really isn't *that* hard.
The only really hard requirement is respecting affinities, you'll crash and burn real hard if you get that wrong (think of all the per-cpu kthreads that hard rely on the per-cpu-ness of them).
But you can easily ignore cgroups, uclamp and a ton of other stuff and still boot and play around.
Quoted 2 lines
I believe we agree that we want more people contributing to the scheduling area.
I think therein lies the rub -- contribution. If we were to do this thing, random loadable BPF schedulers, then how do we ensure people will contribute back?
That is, from where I am sitting I see $vendor mandate their $enterprise product needs their $BPF scheduler. At which point $vendor will have no incentive to ever contribute back.
And customers of $vendor that want to run additional workloads on their machine are then stuck with that scheduler, irrespective of it being suitable for them or not. This is not a good experience.
So I don't at all mind people playing around with schedulers -- they can do so today, there are a ton of out of tree patches to start or learn from, or like I said, it really isn't all that hard to just rip out fair and write something new.
Open source, you get to do your own thing. Have at.
But part of what made Linux work so well, is in my opinion the GPL. GPL forces people to contribute back -- to work on the shared project. And I see the whole BPF thing as a run-around on that.
Even the large cloud vendors and service providers (Amazon, Google, Facebook etc.) contribute back because of rebase pain -- as you well know. The rebase pain offsets the 'TIVO hole'.
But with the BPF muck; where is the motivation to help improve things?
Keeping a rando github repo with BPF schedulers is not contributing. That's just a repo with multiple out of tree schedulers to be ignored. Who will put in the effort of upsteaming things if they can hack up a BPF and throw it over the wall?
So yeah, I'm very much NOT supportive of this effort. From where I'm sitting there is simply not a single benefit. You're not making my life better, so why would I care?
How does this BPF muck translate into better quality patches for me?
Steven Rostedt
reviewerQuoted 6 lines · On Mon, 13 May 2024 10:03:59 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 2 lines
I believe we agree that we want more people contributing to the scheduling area.
I think therein lies the rub -- contribution. If we were to do this thing, random loadable BPF schedulers, then how do we ensure people will contribute back?
Hi Peter,
I'm somewhat agnostic to sched_ext itself, but I have been an advocate for a plugable scheduler infrastructure. And we are seriously looking at adding it to ChromeOS.
Quoted 4 lines
That is, from where I am sitting I see $vendor mandate their $enterprise product needs their $BPF scheduler. At which point $vendor will have no incentive to ever contribute back.
Believe me they already have their own scheduler, and because its so different, it's very hard to contribute back.
Quoted 4 lines
And customers of $vendor that want to run additional workloads on their machine are then stuck with that scheduler, irrespective of it being suitable for them or not. This is not a good experience.
And $vendor usually has a unique workload that their changes will likely cause regressions in other workloads, making it even harder to contribute back.
Quoted 5 lines
So I don't at all mind people playing around with schedulers -- they can do so today, there are a ton of out of tree patches to start or learn from, or like I said, it really isn't all that hard to just rip out fair and write something new.
For cloud servers, I bet a lot of schedulers are not public. Although, my company tries to publish the schedulers they use.
Quoted 10 lines
Open source, you get to do your own thing. Have at.
But part of what made Linux work so well, is in my opinion the GPL. GPL forces people to contribute back -- to work on the shared project. And I see the whole BPF thing as a run-around on that.
Even the large cloud vendors and service providers (Amazon, Google, Facebook etc.) contribute back because of rebase pain -- as you well know. The rebase pain offsets the 'TIVO hole'.
From what I understand (I don't work on production, but Chromebooks), a lot of changes cannot be contributed back because their updates are far from what is upstream. Having a plugable scheduler would actually allow them to contribute *more*.
Quoted 2 lines
But with the BPF muck; where is the motivation to help improve things?
For the same reasons you mention about GPL and why it works. Collaboration. Sharing ideas helps everyone. If there's some secret sauce scheduler then they would likely just replace the scheduler, as its more performant. I don't believe it would be worth while to use BPF for that purpose.
Quoted 2 lines
Keeping a rando github repo with BPF schedulers is not contributing.
Agreed, and I would guess having them in the Linux kernel tree would be more beneficial.
Quoted 3 lines
That's just a repo with multiple out of tree schedulers to be ignored. Who will put in the effort of upsteaming things if they can hack up a BPF and throw it over the wall?
If there's a place in the Linux kernel tree, I'm sure there would be motivation to place it there. Having it in the kernel proper does give more visibility of code, and therefore enhancements to that code. This was the same rationale for putting perf into the kernel proper.
Quoted 6 lines
So yeah, I'm very much NOT supportive of this effort. From where I'm sitting there is simply not a single benefit. You're not making my life better, so why would I care?
How does this BPF muck translate into better quality patches for me?
Here's how we will be using it (we will likely be porting sched_ext to ChromeOS regardless of its acceptance).
Doing testing of scheduler changes in the field is extremely time consuming and complex. We tested EEVDF vs CFS by backporting EEVDF to 5.15 (as that is the kernel version we are using on the chromebooks we were testing on), and then we need to add a user space "switch" to change the scheduler. Note, this also risks causing a bug in adding these changes. Then we push the kernel out, and then start our experiment that enables our feature to a small percentage, and slowly increases the number of users until we have a enough for a statistical result.
What sched_ext would give us is a easy way to try different scheduling algorithms and get feedback much quicker. Once we determine a solution that improves things, we would then spend the time to implement it in the scheduler, and yes, send it upstream.
To me, sched_ext should never be the final solution, but it can be extremely useful in testing various changes quickly in the field. Which to me would encourage more contributions.
-- Steve
Quoted 102 lines · Steven Rostedt
Quoted 6 lines · On Mon, 13 May 2024 10:03:59 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 2 lines
I believe we agree that we want more people contributing to the scheduling area.
I think therein lies the rub -- contribution. If we were to do this thing, random loadable BPF schedulers, then how do we ensure people will contribute back?
Hi Peter,
I'm somewhat agnostic to
sched_extitself, but I have been an advocate for a plugable scheduler infrastructure. And we are seriously looking at adding it to ChromeOS.Quoted 4 lines
That is, from where I am sitting I see $vendor mandate their $enterprise product needs their $BPF scheduler. At which point $vendor will have no incentive to ever contribute back.
Believe me they already have their own scheduler, and because its so different, it's very hard to contribute back.
Quoted 4 lines
And customers of $vendor that want to run additional workloads on their machine are then stuck with that scheduler, irrespective of it being suitable for them or not. This is not a good experience.
And $vendor usually has a unique workload that their changes will likely cause regressions in other workloads, making it even harder to contribute back.
Quoted 5 lines
So I don't at all mind people playing around with schedulers -- they can do so today, there are a ton of out of tree patches to start or learn from, or like I said, it really isn't all that hard to just rip out fair and write something new.
For cloud servers, I bet a lot of schedulers are not public. Although, my company tries to publish the schedulers they use.
Quoted 10 lines
Open source, you get to do your own thing. Have at.
But part of what made Linux work so well, is in my opinion the GPL. GPL forces people to contribute back -- to work on the shared project. And I see the whole BPF thing as a run-around on that.
Even the large cloud vendors and service providers (Amazon, Google, Facebook etc.) contribute back because of rebase pain -- as you well know. The rebase pain offsets the 'TIVO hole'.
From what I understand (I don't work on production, but Chromebooks), a lot of changes cannot be contributed back because their updates are far from what is upstream. Having a plugable scheduler would actually allow them to contribute *more*.
Quoted 2 lines
But with the BPF muck; where is the motivation to help improve things?
For the same reasons you mention about GPL and why it works. Collaboration. Sharing ideas helps everyone. If there's some secret sauce scheduler then they would likely just replace the scheduler, as its more performant. I don't believe it would be worth while to use BPF for that purpose.
Quoted 2 lines
Keeping a rando github repo with BPF schedulers is not contributing.
Agreed, and I would guess having them in the Linux kernel tree would be more beneficial.
Quoted 3 lines
That's just a repo with multiple out of tree schedulers to be ignored. Who will put in the effort of upsteaming things if they can hack up a BPF and throw it over the wall?
If there's a place in the Linux kernel tree, I'm sure there would be motivation to place it there. Having it in the kernel proper does give more visibility of code, and therefore enhancements to that code. This was the same rationale for putting perf into the kernel proper.
Quoted 6 lines
So yeah, I'm very much NOT supportive of this effort. From where I'm sitting there is simply not a single benefit. You're not making my life better, so why would I care?
How does this BPF muck translate into better quality patches for me?
Here's how we will be using it (we will likely be porting
sched_extto ChromeOS regardless of its acceptance).Doing testing of scheduler changes in the field is extremely time consuming and complex. We tested EEVDF vs CFS by backporting EEVDF to 5.15 (as that is the kernel version we are using on the chromebooks we were testing on), and then we need to add a user space "switch" to change the scheduler. Note, this also risks causing a bug in adding these changes. Then we push the kernel out, and then start our experiment that enables our feature to a small percentage, and slowly increases the number of users until we have a enough for a statistical result.
Quoted 8 lines
What
sched_extwould give us is a easy way to try different scheduling algorithms and get feedback much quicker. Once we determine a solution that improves things, we would then spend the time to implement it in the scheduler, and yes, send it upstream.To me,
sched_extshould never be the final solution, but it can be extremely useful in testing various changes quickly in the field. Which to me would encourage more contributions.
I really don't think the problems we have are because of EEVDF vs CFS vs anything else. Other major OSes have one scheduler, but what they exceed on is providing better QoS interfaces and mechanism to handle specific scenarios that Linux lacks.
The confusion I see again and again over the years is the fragmentation of Linux eco system and app writers don't know how to do things properly on Linux vs other OSes. Note our CONFIG system is part of this fragmentation.
The addition of more flavours which inevitably will lead to custom QoS specific to that scheduler and libraries built on top of it that require that particular extension available is a recipe for more confusion and fragmentation. Not to mention big players are likely to take over, and I wouldn't be surprised if new business models start to spring up on top of that. Add to the lot the potential security issues with the ease to lure people to download sneaky sched extension that gives great promises but full of malware (more dangerous with the greater power of BPF/sudo misused).
I really don't buy the rapid development aspect too. The scheduler was heavily influenced by the early contributors which come from server market that had (few) very specific workloads they needed to optimize for and throughput had a heavier weight vs latency. Fast forward to now, things are different. Even on server market latency/responsiveness has become more important. Power and thermal are important on a larger class of systems now too. I'd dare say even on server market. How do you know when it's okay for an app/task to consume too much power and when it is not? Hint hint, you can't unless someone in userspace tells you. Similarly for latency vs throughput. What is the correct way to write an application to provide this info? Then we can ask what is missing in the scheduler to enable this.
Note the original min/wakeup_granularity_ns, latency_ns etc were tuned by default for throughput by the way (server market bias). You can manipulate those and get better latencies.
And this brings me to the major point, we really need to stop thinking that we must improve everything at system level. Workloads need to evolve to take best out of systems and we need new libraries for performance and power management. And this means they need to get new APIs and libraries do a be able to do a better job and scale well.
I agree with Peter it is not hard to write something to make specific workload better. But what we really need is enable workloads to be written better and be more portable to take best of the hardware they run on, AND coexist with other workloads. For example, how do you write a good multi threaded application that can scale well across systems (including big.LITTLE) and not trip over other workloads stealing resources sometimes? You need something like this
https://developer.apple.com/documentation/DISPATCH
which has a linux port
https://github.com/apple/swift-corelibs-libdispatch
not a new scheduler.
How do you write an app that can manage bad thermal situations?
https://developer.android.com/games/optimize/adpf/thermal
POSIX is dormant, and every OS has to wing new interfaces to deal with the new realities. And I don't see a lot of these discussions. Linux is lagging behind in general in this aspect. The trend I see is how do I make existing stuff better, and believe me I've seen strcmp(task->comm, ...) to hand pick things. Which I am sure we'll end up down this path if we let things loose.
So I am against any custom extension. I think it all has to be part of the kernel tree and adhere to all of its supported interfaces. Which I think what we really ought on focusing to evolve and improve. This is the biggest friction point IMO, not the scheduler algorithm. If the latter need to change, it needs to be as the result of this friction - which what EEVDF came about from to my understanding. To enable implementing a latency interface easier. But Vincent had a working implementation with CFS too which I think would have worked fine by the way.
I do hope we can reconsider some of our default behaviors though (that bias to perf and throughput specifically).
FWIW IMO the biggest issues I see in the scheduler is that its testability and debuggability is hard. I think BPF can be a good fit for that. For the latter I started this project, yet I am still trying to figure out how to add tracer for the difficult paths to help people more easily report when a bad decision has happened to provide more info about the internal state of the scheduler, in hope to accelerate the process of finding solutions. I think people are getting stuck explaining why things are failing, which makes finding a common solution hard if not impossible. We need better way to understand the problems people are seeing
https://github.com/qais-yousef/sched-analyzer
Similar methodology can be used to create a BPF based sched test framework. I don't have cycles to start this, but hope to if no one beats me to it.
I think it would be great to have a clear list of the current limitations people see in the scheduler. It could be a failure on my end, but I haven't seen specifics of problems and what was tried and failed to the point it is impossible to move forward. From what I see, I am hitting bugs here and there all the time. But they are hard to debug to truly understand where things went wrong. Like this one for example where PTHREAD_PRIO_PI is a NOP for fair tasks. Many thought using this flag doesn't help (rather than buggy)..
https://lore.kernel.org/lkml/[email protected]/
David Vernet
reviewerOn Tue, May 14, 2024 at 01:07:15AM +0100, Qais Yousef wrote:
[...]
Quoted 16 lines
Quoted 16 lines
Quoted 2 lines
How does this BPF muck translate into better quality patches for me?
Here's how we will be using it (we will likely be porting
sched_extto ChromeOS regardless of its acceptance).Doing testing of scheduler changes in the field is extremely time consuming and complex. We tested EEVDF vs CFS by backporting EEVDF to 5.15 (as that is the kernel version we are using on the chromebooks we were testing on), and then we need to add a user space "switch" to change the scheduler. Note, this also risks causing a bug in adding these changes. Then we push the kernel out, and then start our experiment that enables our feature to a small percentage, and slowly increases the number of users until we have a enough for a statistical result.
Quoted 8 lines
Quoted 8 lines
What
sched_extwould give us is a easy way to try different scheduling algorithms and get feedback much quicker. Once we determine a solution that improves things, we would then spend the time to implement it in the scheduler, and yes, send it upstream.To me,
sched_extshould never be the final solution, but it can be extremely useful in testing various changes quickly in the field. Which to me would encourage more contributions.
Hello Qais,
[...]
Quoted 1 line
I really don't buy the rapid development aspect too. The scheduler was heavily
There are already several examples from users who have shown that the rapid development and experimentation is extremely useful. Imagine if you're iterating on the scheduler to improve p99 frame rates on the Steam Deck, as Changwoo described. It's much more efficient to be able to just tweak and load a BPF scheduler (that is safe and can't crash the machine) to try some random idea out than it is to:
1. Tweak and recompile the kernel
2. Reinstall the kernel on the Steam Deck
3. Reboot the Steam Deck
4. Reload a game and let caches rewarm
5. Measure FPS
You're talking about a 5 second compile job + 1 second to reload a safe BPF scheduler vs. having to do all of the above steps _and_ potentially making a mistake that brings the machine down. These benefits are also extremely useful for testing workloads on production servers, etc. Let’s also not forget that unlike many other kernel features, you probably can’t get reliable scheduling results from running in a VM. The experimentation overhead is very real.
[...]
Quoted 10 lines
influenced by the early contributors which come from server market that had (few) very specific workloads they needed to optimize for and throughput had a heavier weight vs latency. Fast forward to now, things are different. Even on server market latency/responsiveness has become more important. Power and thermal are important on a larger class of systems now too. I'd dare say even on server market. How do you know when it's okay for an app/task to consume too much power and when it is not? Hint hint, you can't unless someone in userspace tells you. Similarly for latency vs throughput. What is the correct way to write an application to provide this info? Then we can ask what is missing in the scheduler to enable this.
Hmm, you seem to be arguing that the way forward here is to have our one general purpose scheduler be entirely driven by user space hinting. Assuming I’m not misunderstanding you, I strongly disagree with this sentiment. User space hinting can be powerful, but I think we need to have a general purpose scheduler that's completely agnostic to whatever is running in user space. We’ve also been able to get strong results from sched_ext schedulers that don’t use any user space hinting.
Also, even if this ended up being the way forward, I don’t see it being practical to implement. Wouldn’t it require us to update all of user space globally just to update how it interfaces with the scheduler?
[...]
Quoted 3 lines
Note the original min/wakeup_granularity_ns,
latency_nsetc were tuned by default for throughput by the way (server market bias). You can manipulate those and get better latencies.
Those knobs aren't available anymore in EEVDF.
[...]
Quoted 5 lines
point IMO, not the scheduler algorithm. If the latter need to change, it needs to be as the result of this friction - which what EEVDF came about from to my understanding. To enable implementing a latency interface easier. But Vincent had a working implementation with CFS too which I think would have worked fine by the way.
This friction is nothing new. It's why we already find ourselves in the unfortunate position of having a large corpus of out of tree scheduler patches. If there is a lot of performance being left on the table, vendors are going to find a way to get that performance. Corporations don't need our consent to ship kernels with custom schedulers on their devices. They've already been doing it for years, and it's ultimately the users who suffer.
I genuinely believe that the fair.c scheduler will benefit from being able to apply ideas conceived in a sched_ext scheduler which end up working well for general use cases. For example, in scx_rusty, we’re able to get very good interactivity [0] by determining a task’s deadline as a function of its average runtime (along with some other great ideas that Changwoo first added to scx_lavd) rather than from its eligibility + slice as with what EEVDF does. Over the course of a day or two, I tried way more ideas that didn’t work than would have been possible in that time frame than with a recompile-reboot cycle, and ended up finding one that seems to work very well. It would be awesome if these ideas were added to EEVDF so that everyone can benefit.
[0]: https://drive.google.com/file/d/1fyHt9BYGha6apl7HAkibwpy52UTi8-AQ/view?usp=drive_link
Thanks,
David
Quoted 48 lines · David Vernet
On Tue, May 14, 2024 at 01:07:15AM +0100, Qais Yousef wrote:
[...]
Quoted 16 lines
Quoted 16 lines
Quoted 2 lines
How does this BPF muck translate into better quality patches for me?
Here's how we will be using it (we will likely be porting
sched_extto ChromeOS regardless of its acceptance).Doing testing of scheduler changes in the field is extremely time consuming and complex. We tested EEVDF vs CFS by backporting EEVDF to 5.15 (as that is the kernel version we are using on the chromebooks we were testing on), and then we need to add a user space "switch" to change the scheduler. Note, this also risks causing a bug in adding these changes. Then we push the kernel out, and then start our experiment that enables our feature to a small percentage, and slowly increases the number of users until we have a enough for a statistical result.
Quoted 8 lines
Quoted 8 lines
What
sched_extwould give us is a easy way to try different scheduling algorithms and get feedback much quicker. Once we determine a solution that improves things, we would then spend the time to implement it in the scheduler, and yes, send it upstream.To me,
sched_extshould never be the final solution, but it can be extremely useful in testing various changes quickly in the field. Which to me would encourage more contributions.Hello Qais,
[...]
Quoted 1 line
I really don't buy the rapid development aspect too. The scheduler was heavily
There are already several examples from users who have shown that the rapid development and experimentation is extremely useful. Imagine if you're iterating on the scheduler to improve p99 frame rates on the Steam Deck, as Changwoo described. It's much more efficient to be able to just tweak and load a BPF scheduler (that is safe and can't crash the machine) to try some random idea out than it is to:
1. Tweak and recompile the kernel
2. Reinstall the kernel on the Steam Deck
3. Reboot the Steam Deck
4. Reload a game and let caches rewarm
5. Measure FPS
Quoted 6 lines
You're talking about a 5 second compile job + 1 second to reload a safe BPF scheduler vs. having to do all of the above steps _and_ potentially making a mistake that brings the machine down. These benefits are also extremely useful for testing workloads on production servers, etc. Let’s also not forget that unlike many other kernel features, you probably can’t get reliable scheduling results from running in a VM. The experimentation overhead is very real.
What I read here is that I can hack my system quickly. Is the intention to extend the kernel? If yes, I can't see how this experimentation is actually valid if not implemented in the kernel first taking into account the real constraint that you have to deal with sooner or later.
Quoted 14 lines
[...]
Quoted 10 lines
influenced by the early contributors which come from server market that had (few) very specific workloads they needed to optimize for and throughput had a heavier weight vs latency. Fast forward to now, things are different. Even on server market latency/responsiveness has become more important. Power and thermal are important on a larger class of systems now too. I'd dare say even on server market. How do you know when it's okay for an app/task to consume too much power and when it is not? Hint hint, you can't unless someone in userspace tells you. Similarly for latency vs throughput. What is the correct way to write an application to provide this info? Then we can ask what is missing in the scheduler to enable this.
Quoted 7 lines
Hmm, you seem to be arguing that the way forward here is to have our one general purpose scheduler be entirely driven by user space hinting. Assuming I’m not misunderstanding you, I strongly disagree with this sentiment. User space hinting can be powerful, but I think we need to have a general purpose scheduler that's completely agnostic to whatever is running in user space. We’ve also been able to get strong results from
sched_extschedulers that don’t use any user space hinting.
I'm curious. If you believe in general purpose system, what work was done to improve the current one? What debugging and analysis was done to improve the current situation? It seems you reached a conclusion that we need something different - but no reasons behind it why is that.
Is the problem with the default behavior of the system? Or are your problems focused on corner cases where things seem to fail?
Quoted 3 lines
Also, even if this ended up being the way forward, I don’t see it being practical to implement. Wouldn’t it require us to update all of user space
People swear by Apple's GCD by the way. It'd be really great if someone can create something similar that works properly on Linux. I have never tried the libdispatch port to see how well it does.
And have you seen these?
https://developer.android.com/stories/games/mediatek-adpf
Quoted 1 line
globally just to update how it interfaces with the scheduler?
I think you're confusing default scheduler behavior and dealing with corner cases that are impossible for the scheduler to resolve. These corner cases are when help is needed. Note that the thermal API is actually info from the system to the app. If the app decides to listen, then they can help reduce the thermal impact without causing throttling. If they decide not to listen, then the best the system can do is throttle everything hard to protect from damage. And under bad thermal pressure, the scheduler can know which tasks to prioritize to performance if it has explicit knowledge/hints.
If the default behavior is not working for you; could you provide more details on what goes wrong? It's unlikely that a new algorithm is the solution, but likely a bug somewhere or some configuration problem.
And if someone wants to optimize for best perf, power and thermal, they need to do the work. There's only so much you can do on their behalf that is actually scalable.
System designers want apps (all type of apps) to take best advantage of the hardware they built.
App writers want to write portable software that gives the desired experience on all type of systems without special optimization.
Quoted 8 lines
[...]
Quoted 3 lines
Note the original min/wakeup_granularity_ns,
latency_nsetc were tuned by default for throughput by the way (server market bias). You can manipulate those and get better latencies.Those knobs aren't available anymore in EEVDF.
I generalized my statement as I didn't expect many have moved to 6.6 LTS, which is the only one that has EEVDF.
EEVDF has base_slice_ns. What value do you read on your system? What is your TICK value and how m any CPUs do you have?
Quoted 9 lines
[...]
Quoted 5 lines
point IMO, not the scheduler algorithm. If the latter need to change, it needs to be as the result of this friction - which what EEVDF came about from to my understanding. To enable implementing a latency interface easier. But Vincent had a working implementation with CFS too which I think would have worked fine by the way.
Quoted 6 lines
This friction is nothing new. It's why we already find ourselves in the unfortunate position of having a large corpus of out of tree scheduler patches. If there is a lot of performance being left on the table, vendors are going to find a way to get that performance. Corporations don't need our consent to ship kernels with custom schedulers on their devices. They've already been doing it for years, and it's ultimately the users who suffer.
I think everyone agrees on the need to improve. But..
Quoted 5 lines
I genuinely believe that the
fair.cscheduler will benefit from being able to apply ideas conceived in asched_extscheduler which end up working well for general use cases. For example, inscx_rusty, we’re able to get very good interactivity [0] by determining a task’s deadline as a function of its average
.. I am really failing to see why you jumped to the fact we need a new scheduler. And you'll find a lot of skepticism about the validity of your results. We have no clue what kind of unknown constraint you've left out with these test. And how limited your environment is.
Quoted 2 lines
runtime (along with some other great ideas that Changwoo first added to
scx_lavd) rather than from its eligibility + slice as with what EEVDF does.
You'll find soon that the concept of runtime is hard. And generally there's a big soup of tasks running in the system. Most of which have no real deadline. Only few do. And corner cases are the complexity of any situation when you have more tasks that need to run immediately than you have CPUs to distribute them on. I don't think we can figure this out automatically based on runtime.
Quoted 4 lines
Over the course of a day or two, I tried way more ideas that didn’t work than would have been possible in that time frame than with a recompile-reboot cycle, and ended up finding one that seems to work very well. It would be awesome if these ideas were added to EEVDF so that everyone can benefit.
Why do you think they're applicable? And how do you know you're not working around different problems? Or have missed constraints in your testing once applied will make the whole results invalid?
Too many unknowns IMHO. I am not against a different scheduler algorithm if it proves to be a more generic default. But you'll find first you have to explain what has failed in current one and what kind of analysis made you reach this conclusion. And then you'll find you'll need to actually do it in the kernel taking into account all the constraints that you must handle to prove it is still as valid as you initially thought.
And I can only share my experience, I don't think the algorithm itself is the bottleneck here. The devil is in the corner cases. And these are hard to deal with without explicit hints.
The biggest issue I see generally with the default behavior is that traditionally it has been biased towards throughput because those folks are the one that keep reporting regressions when anything changes on the list.
Please add your voice and report problems when you notice things don't work for you. That's the best way to ensure there's visibility of these issues. It seems to me you're hitting problems that people expect to work. But I have no clue what problems you have. I am not sure if this was reported somewhere else, but it seems not.
Thanks!
--
Qais Yousef
Tejun Heo
maintainerHello,
BTW, David is off for the week and might be a bit slow to respond. I just want to comment on one part.
On Mon, May 27, 2024 at 10:25:40PM +0100, Qais Yousef wrote:
...
Quoted 3 lines
And I can only share my experience, I don't think the algorithm itself is the bottleneck here. The devil is in the corner cases. And these are hard to deal with without explicit hints.
Our perceptions of the scope of the problem space seem very different. To me, it seems pretty unexplored. Here's just one area: Constantly increasing number of cores and popularization of more complex cache hierarchies.
Over a hundred CPUs in a system is fairly normal now with a couple layers of cache hierarchy. Once we have so many, things can look a bit different from the days when we had a few. Flipping the approach so that we can dynamically assign close-by CPUs to related groups of threads becomes attractive.
e.g. If you have a bunch of services which aren't latency critical but are needed to maintain system integrity (updates, monitoring, security and so on), soft-affining them to a number of CPUs while allowing some CPU headroom can give you noticeable gain both in performance (partly from cleaner caches) and power consumption while not adding that much to latency. This is something the scheduler can and, I believe, should do transparently.
It's not obvious how to do it though. It doesn't quite fit the current LB model. cgroup hierarchy seems to provide some hints on how threads can be grouped but the boundaries might not match that well. Even if we figure out how to define these groups, figuring out group-vs-group competition isn't trivial (naive load-sums don't work when comparing across groups spanning multiple CPUs).
Also, what about the threads with oddball cpumasks? Should we begin to treat CPUs more like other resources, e.g., memory? We don't generally allow applications to specify which specific physical pages they get because that doesn't buy anything while adding a lot of constraints. If we have dozens and hundreds of CPUs, are there fundamental reason to view them differently from other resources which are treated fungible?
The claim that the current scheduler has the fundamentals all figured out and it's mostly about handling edge cases and educating users seems wildly off mark to me.
Maybe we can develop all that in the current framework in a gradual fashion, but when the problem space is so wide open, that is not a good approach to take. The cost of constricting is likely significantly higher than the benefits of having a single code base. Imagine having to develop all the features of btrfs in the ext2 code base. It's probably doable, at least theoretically, but that would have been massively stifling, maybe to the point of most of it not happening.
To the above particular problem of soft-affinity, scx_layered has something really simple and dumb implemented and we're testing and deploying it in the fleet with noticeable perf gains, and there are early efforts to see whether we can automatically figure out grouping based on the cgroup hierarchy and possibly minimal xattr hints on them.
I don't yet know what generic form soft-affinity should take eventually, but, with sched_ext, we have a way to try out different ideas in production and iterate on them learning each step of the way. Given how generic both the problem and benefits from solving it are, we'll have to reach some generic solution at one point. Maybe it will come from sched_ext or maybe it will come from people working on fair like yourself. Either way, sched_ext is already showing us what can be achieved and prodding people towards solving it.
Thanks.
Signature · 1 line
tejun
Quoted 11 lines · Tejun Heo
Hello,
BTW, David is off for the week and might be a bit slow to respond. I just want to comment on one part.
On Mon, May 27, 2024 at 10:25:40PM +0100, Qais Yousef wrote:
...Quoted 3 lines
And I can only share my experience, I don't think the algorithm itself is the bottleneck here. The devil is in the corner cases. And these are hard to deal with without explicit hints.
Quoted 8 lines
Our perceptions of the scope of the problem space seem very different. To me, it seems pretty unexplored. Here's just one area: Constantly increasing number of cores and popularization of more complex cache hierarchies.
Over a hundred CPUs in a system is fairly normal now with a couple layers of cache hierarchy. Once we have so many, things can look a bit different from the days when we had a few. Flipping the approach so that we can dynamically assign close-by CPUs to related groups of threads becomes attractive.
I had this use case in mind actually for sched-qos [1] idea I am trying to develop. There are workloads that can benefit if 2 or 3 tasks are kept withing the closest cache. And I think we can describe that with a hint.
I was thinking to borrow from core scheduling concept of cookie to tag a group of task via the hint and try to find reasonable higher level behavior that we can translate correctly into different systems.
Quoted 7 lines
e.g. If you have a bunch of services which aren't latency critical but are needed to maintain system integrity (updates, monitoring, security and so on), soft-affining them to a number of CPUs while allowing some CPU headroom can give you noticeable gain both in performance (partly from cleaner caches) and power consumption while not adding that much to latency. This is something the scheduler can and, I believe, should do transparently.
This looks similar to what I am trying to do with uclamp_max and extending load balancer to allow to balance workloads based on power - keeping in mind freeing resources for tasks that need performance too. I don't think we can fix this problem on wake up balance only. The system is in a constant flux and we need load balancer to do corrections when other things wake up and we need better decisions to be made.
Generally if we have EAS type of behavior available for SMP systems where we don't distribute by default but try to pack based on compute demand - and a hint to tell us that some tasks really want to be spread as an exception for those that packing really hurts them, I think we'd be in a much better place to be able to distribute resources like you describe.
Quoted 4 lines
It's not obvious how to do it though. It doesn't quite fit the current LB model. cgroup hierarchy seems to provide some hints on how threads can be grouped but the boundaries might not match that well. Even if we figure out
cgroups is too aggressive IMHO. We really need per-task hints. It's coarse vs fine grained hinting. There's only so much classification you can give to a larger group of tasks. Especially if you can't control the codebase of this group of tasks.
Some people can get invested in tuning specific apps. But this is not scalable and fragile.
Quoted 3 lines
how to define these groups, figuring out group-vs-group competition isn't trivial (naive load-sums don't work when comparing across groups spanning multiple CPUs).
I think the implementation is trickier than the definition. There's lots of demands to keep the fast path as fast as possible. To do smarter decisions this will get expensive. Personally I think today we have abundant of compute power and the challenge is how to smartly distribute resources, which justify slowing things down in favour of making better choices. But I don't know how much we can afford to be honest.
Generally as I was telling David, people who tend to come forward more to support or complain are those who have pure throughput in mind. Maybe I am wrong, but from my perception a lot of decisions were biased this way. We need to be more vocal about our needs to make sure that things move in the right direction. It's hard to help a use case or fix a problem when you don't know about it.
Quoted 7 lines
Also, what about the threads with oddball cpumasks? Should we begin to treat CPUs more like other resources, e.g., memory? We don't generally allow applications to specify which specific physical pages they get because that doesn't buy anything while adding a lot of constraints. If we have dozens and hundreds of CPUs, are there fundamental reason to view them differently from other resources which are treated fungible?
I'd be more than happy to see affinity and cpuset disappear :) But I fear it might be a little too late..
Can't some selinux rule or some syscall filter be used to block userspace from playing with affinity?
I'm assuming you're not referring to in-kernel usage of affinity. Which might be worth scrutinizing. But we have more control over that in general to make it better when a problem arises.
Quoted 4 lines
The claim that the current scheduler has the fundamentals all figured out and it's mostly about handling edge cases and educating users seems wildly off mark to me.
I don't think anyone claimed that. But EEVDF or CFS is about how tasks enqueued on the CPU will be ordered and run. It's not about selecting which CPU to run the task on.
EAS modifies the selection algorithm (which is not what David was talking about IIUC). It seems your problems are more with CPU selection then?
Quoted 10 lines
Maybe we can develop all that in the current framework in a gradual fashion, but when the problem space is so wide open, that is not a good approach to take. The cost of constricting is likely significantly higher than the benefits of having a single code base. Imagine having to develop all the features of btrfs in the ext2 code base. It's probably doable, at least theoretically, but that would have been massively stifling, maybe to the point of most of it not happening.
To the above particular problem of soft-affinity,
scx_layeredhas something
What layered refers to here? Is it akin to different sched classes?
Quoted 5 lines
really simple and dumb implemented and we're testing and deploying it in the fleet with noticeable perf gains, and there are early efforts to see whether we can automatically figure out grouping based on the cgroup hierarchy and possibly minimal xattr hints on them.
Quoted 8 lines
I don't yet know what generic form soft-affinity should take eventually, but, with
sched_ext, we have a way to try out different ideas in production and iterate on them learning each step of the way. Given how generic both the problem and benefits from solving it are, we'll have to reach some generic solution at one point. Maybe it will come fromsched_extor maybe it will come from people working on fair like yourself. Either way,sched_extis already showing us what can be achieved and prodding people towards solving it.
To be honest this doesn't look any different to all the hacks out there that do the same. The path I see this is going into is the same as I mentioned above where some people manually tune for specific usage. I really struggle to see how this is going to be applicable later and all I see a divergence and parallel universes - which ultimately will hurt the user as Linux behavior is just not predictable.
This Linus rant [2] is relevant to the situation. In this case people who write applications will just find that Linux is not reliable because every system doesn't behave the same.
[1] https://lore.kernel.org/lkml/20230916213316.p36nhgnibsidoggt@airbuntu/
[2] https://lore.kernel.org/lkml/CAHk-=wgtb7y-bEh7tPDvDWru7ZKQ8-KMjZ53Tsk37zsPPdwXbA@mail.gmail.com/
Thanks!
--
Qais Yousef
Peter Zijlstra
maintainerQuoted 8 lines · Qais Yousef
Quoted 6 lines · Steven Rostedt
Quoted 3 lines
That is, from where I am sitting I see $vendor mandate their $enterprise product needs their $BPF scheduler. At which point $vendor will have no incentive to ever contribute back.
Believe me they already have their own scheduler, and because its so different, it's very hard to contribute back.
'They' are free to have their own scheduler, but since 'nobody' is using it and 'they' want to have their product work on RHEL / SLES / etc.. therefore are bound to respect the common interfaces, no?
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
So I don't at all mind people playing around with schedulers -- they can do so today, there are a ton of out of tree patches to start or learn from, or like I said, it really isn't all that hard to just rip out fair and write something new.
For cloud servers, I bet a lot of schedulers are not public. Although, my company tries to publish the schedulers they use.
Yeah, it's the TIVO thing. Keeping all that private creates the rebase pain. Outside of that there's nothing we can do.
Anyway, instead of doing magic mushroom schedulers, what does the cloud crud actually want? I know the KVM people were somewhat looking forward to the EEVDF sched_attr::sched_runtime extension because virt likes the longer slices. Less preemption more better for them.
In fact, some of the facebook workloads also wanted longer slices (and no wakeup preemption).
Quoted 4 lines
Quoted 4 lines
From what I understand (I don't work on production, but Chromebooks), a lot of changes cannot be contributed back because their updates are far from what is upstream. Having a plugable scheduler would actually allow them to contribute *more*.
So can we please start by telling what kind of magic hacks ChromeOS has and whatfor?
The term contributing seems to mean different things to us. Building a external scheduler isn't contributing, it's fragmenting.
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Keeping a rando github repo with BPF schedulers is not contributing.
Agreed, and I would guess having them in the Linux kernel tree would be more beneficial.
Yeah, no. Same thing. It's just a pile of junk until someone puts the time in to figure out how to properly integrate it. Very much like Qais argues below.
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
That's just a repo with multiple out of tree schedulers to be ignored. Who will put in the effort of upsteaming things if they can hack up a BPF and throw it over the wall?
If there's a place in the Linux kernel tree, I'm sure there would be motivation to place it there. Having it in the kernel proper does give more visibility of code, and therefore enhancements to that code. This was the same rationale for putting perf into the kernel proper.
These things are very much not the same. A pile of random hacks vs a single unified interface to PMUs. They're like the polar opposite of one another.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
So yeah, I'm very much NOT supportive of this effort. From where I'm sitting there is simply not a single benefit. You're not making my life better, so why would I care?
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
How does this BPF muck translate into better quality patches for me?
Here's how we will be using it (we will likely be porting
sched_extto ChromeOS regardless of its acceptance).Doing testing of scheduler changes in the field is extremely time consuming and complex. We tested EEVDF vs CFS by backporting EEVDF to 5.15 (as that is the kernel version we are using on the chromebooks we
/me mumbles something about necro-kernels...
Quoted 11 lines
Quoted 11 lines
were testing on), and then we need to add a user space "switch" to change the scheduler. Note, this also risks causing a bug in adding these changes. Then we push the kernel out, and then start our experiment that enables our feature to a small percentage, and slowly increases the number of users until we have a enough for a statistical result.
What
sched_extwould give us is a easy way to try different scheduling algorithms and get feedback much quicker. Once we determine a solution that improves things, we would then spend the time to implement it in the scheduler, and yes, send it upstream.
This sounds a little backwards... ok, a lot. How do you do actual problem analysis in this case? Having random statistics is not really useful - beyond determining there might be a problem.
The next step is isolating that problem locally and reproducing it. Then analysing *what* the actual problem is and how it happens, and then try and think of a solution.
(preferably one that then doesn't break another thing :-)
Quoted 3 lines
Quoted 3 lines
To me,
sched_extshould never be the final solution, but it can be extremely useful in testing various changes quickly in the field. Which to me would encourage more contributions.
Well, the thing is, the moment sched_ext itself lands upstream, it will become the final solution for a fair number of people and leave us, the wider Linux scheduler community, up a creek without no paddles on.
There is absolutely no inherent incentive to further contribute. Your immediate problem is solved, you get assigned the next problem. That is reality.
Worse, they can share the BPF hack and get warm fuzzy feeling of 'contribution' while in fact it's useless. At best we know 'random hack changed something for them'. No problem description, no reproducer, no nothing.
Anyway, if you feel you need BPF hackery to do this, by all means, do so. But realize that it is a debug tool and in general we don't merge debug tools.
Also, I would argue that perhaps a scheduler livepatch would be more convenient to actually debug / A-B test things.
Quoted 4 lines
I really don't think the problems we have are because of EEVDF vs CFS vs anything else. Other major OSes have one scheduler, but what they exceed on is providing better QoS interfaces and mechanism to handle specific scenarios that Linux lacks.
Quite possibly. The immediate problem being that adding interfaces is terrifying. Linus has a rather strong opinion about breaking stuff, and getting this wrong will very quickly result in a paint-into-corner type problem.
We can/could add fields to sched_attr under the understanding that they're purely optional and try thing, *however* too many such fields and we're up a creek again.
Quoted 7 lines
The confusion I see again and again over the years is the fragmentation of Linux eco system and app writers don't know how to do things properly on Linux vs other OSes. Note our CONFIG system is part of this fragmentation.
The addition of more flavours which inevitably will lead to custom QoS specific to that scheduler and libraries built on top of it that require that particular extension available is a recipe for more confusion and fragmentation.
Yes, this!
Quoted 7 lines
I really don't buy the rapid development aspect too. The scheduler was heavily influenced by the early contributors which come from server market that had (few) very specific workloads they needed to optimize for and throughput had a heavier weight vs latency. Fast forward to now, things are different. Even on server market latency/responsiveness has become more important. Power and thermal are important on a larger class of systems now too. I'd dare say even on server market.
Absolutely, AFAIU racks are both power and thermal limited. There are some crazy ACPI protocols to manage some of this.
Quoted 3 lines
How do you know when it's okay for an app/task to consume too
much power and when it is not? Hint hint, you can't unless someone in userspace tells you.
Yes, cluster/cloud infrastructure needs to manage that. There is nothing smart the kernel can do here on its own, except respect the ACPI lunacy and hard throttle itself when the panic signal comes.
Quoted 3 lines
Similarly for latency vs throughput. What is the correct way to write an application to provide this info? Then we can ask what is missing in the scheduler to enable this.
Right, so the EEVDF thing is a start here. By providing a per task request size, applications can indicate if they want frequent and short activations or more infrequent longer activations.
An application can know it's (average) activation time, the kernel has no clue when work starts and is completed. Applications can fairly trivially measure this using CLOCK_THREAD_CPUTIME_ID reads before and after and communicate this (very much like SCHED_DEADLINE).
Anyway, yes, userspace needs to change and provide more information. The trick ofcourse is figuring out which bit of information is critical / useful etc.
There is a definite limit on the amount of constraints you want to solve at runtime.
Everybody going off and hacking their own thing does not help, we need collaboration to figure out what it is that is needed.
Quoted 3 lines
Note the original min/wakeup_granularity_ns,
latency_nsetc were tuned by default for throughput by the way (server market bias). You can manipulate those and get better latencies.
The immediate problem with those knobs is that they are system wide. But yes, everybody was randomly poking them knobs, sometimes in obviously insane ways.
Quoted 6 lines
FWIW IMO the biggest issues I see in the scheduler is that its testability and debuggability is hard. I think BPF can be a good fit for that. For the latter I started this project, yet I am still trying to figure out how to add tracer for the difficult paths to help people more easily report when a bad decision has happened to provide more info about the internal state of the scheduler, in hope to accelerate the process of finding solutions.
So the pitfalls here are that exposing that information for debug purposes can/will lead to people consuming this information for non-debug purposes and then when we want to change things we're stuck because suddenly someone relies something we believed was an implementation detail :/
I've been bitten by this before and this is why I'm so very hesitant to put tracepoints in the scheduler.
Quoted 4 lines
I think it would be great to have a clear list of the current limitations people see in the scheduler. It could be a failure on my end, but I haven't seen specifics of problems and what was tried and failed to the point it is impossible to move forward.
Right, list, but also ideally reproducers (yeah, I know, really hard).
The moment we merge sched_ext all motivation to do any of this work goes out the window.
Quoted 4 lines
From what I see, I am hitting bugs here and there
all the time. But they are hard to debug to truly understand where things went wrong. Like this one for example wherePTHREAD_PRIO_PIis a NOP for fair tasks. Many thought using this flag doesn't help (rather than buggy)..
Yay for the terminal backlog :/ I'll try and have a look.
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines
I really don't think the problems we have are because of EEVDF vs CFS vs anything else. Other major OSes have one scheduler, but what they exceed on is providing better QoS interfaces and mechanism to handle specific scenarios that Linux lacks.
Quite possibly. The immediate problem being that adding interfaces is terrifying. Linus has a rather strong opinion about breaking stuff, and getting this wrong will very quickly result in a paint-into-corner type problem.
We need to move forward though. Let us find an approach and agree with Linus on what will constitute regressions when things have to disappear.
My general worry is more about default behavior not just interfaces. As pointed out below, the default behavior favoured throughput for a long time because folks who care about throughput were more vocal, but it seems we have a silent majority problem of people who need latency by default but find no path forward.
And of course it'll never be possible to make them both happy by default. Regression reports in this area need to consider the wider impact on other users when deciding whether it needs to be fixed or not.
We do seem to be held hostages sometimes by older systems/workloads who IMHO if can't move to use new facilities provided, have no good reason to complain about regressions as we need to look forward for what new workloads and system need by default. The world is moving on too fast - but we can't catch up due to these regression reports. We need a better balance IMHO.
Quoted 4 lines
We can/could add fields to
sched_attrunder the understanding that they're purely optional and try thing, *however* too many such fields and we're up a creek again.
My personal vision on this is this
https://lore.kernel.org/lkml/20230916213316.p36nhgnibsidoggt@airbuntu/
We don't need to continue to add new fields as this is a problem actually when it comes to integrating to libc (who yet to have proper wrappers in pthreads for the things we added). A u32 should be virtually infinite number of hints. We should be able to deprecate at ease by making a specific hint type return an error when it longer supported (-ENOSYS). We can even create uclamp alias for this (that is called performance_hint given how widely people interpret uclamp as a bandwidth hint) and make it the soruce of QoS truth.
Quoted 4 lines
Quoted 3 lines
Similarly for latency vs throughput. What is the correct way to write an application to provide this info? Then we can ask what is missing in the scheduler to enable this.
Quoted 8 lines
Right, so the EEVDF thing is a start here. By providing a per task request size, applications can indicate if they want frequent and short activations or more infrequent longer activations.
An application can know it's (average) activation time, the kernel has no clue when work starts and is completed. Applications can fairly trivially measure this using
CLOCK_THREAD_CPUTIME_IDreads before and after and communicate this (very much likeSCHED_DEADLINE).
I fear the concept of time in userspace will be hard to get right without some further help from us due to DVFS/HMP having a Black Hole effect and causing extreme Time Dilution problem. It is actually a problem for schedutil that I am trying to find a reasonable fix for as part of my magic margins series. On one system I ran a test on, it took 30ms to take off from util 0. And the system stayed running at the lowest frequency for 42ms! Our utilization invariance is very good for estimating compute demand, but terrible for bursty tasks - which I are very common on interactive systems. I think this is a cause of many 'latency' woos in general. Things can end up running slower for longer. But this is a different problem for a different series/day.
FWIW even for userspace trying to create dynamic uclamp control are struggling because they can't measure time reliable. A task can seem happy, but only because something else on another CPU with shared policy had a 'heavy' tasks running. As soon as this goes to sleep things look really different from the tasks runtime perspective. It was running super fast by accident.
The average time will be hard to get right in general due to the interactive nature for some workloads and things could have big variations in practice based on my experience. Ie; the frame to frame variations could be larger than expected.
I think it is a helpful interface, but won't address all workload demands. The thing I care about for example they really want to run ASAP and that's it. They could run for a longer period of time, or a shorter period of time. next_buddy type of behavior will help these tasks. But might need to be stronger than current implementation. I am trying to find out..
And oversubscribed scenarios are important. It is common to have a sudden surge of activities that cause delays that requires load balancer's help to better distribute as some CPUs can get less busy sooner but wakeup preemption won't save those already enqueued tasks from getting CPU time ASAP without some additional external trigger. In contrary, I do see problems today (older CFS LTS kernel) where a surge of short running tasks can delay enqueued tasks considerably. I have no clue what's going on yet. I don't have a reproducer but creeps up often enough when I look at traces.
Generally I think latency is more important in majority of systems these days and it might be better to default to more responsive system and let those who want throughput to opt-in, rather than the other way around.
In theory, there should be (very) few tasks in the system that are actually need the next_buddy type of behavior to skip the queue and run ASAP if the average default latency is good (1-2ms).
I also think we need to enable HRTICK by default too. We will have more timely preemption points then. I generally think sched_feat should be a good way to give admins the power to control certain aspects of the scheduler. We can also make uclamp a sched_feat and ensure it can be made available on any system
- unlike today where it's not enabled by default on Debian at least and this hit me and looks like the Asahi folks who managed to get good power improvements and thankfully has higher weight than me asking for this to be enabled by default. Read Energy Aware Scheduling section here
https://asahilinux.org/2024/01/fedora-asahi-new/
As a general topic for discussion not just for scheduler, there are core features that must always be there from programmer's perspective. We are shooting ourselves in the foot here by being too flexible with our usage of CONFIGs, IMHO :)
Too much random babbling from my side maybe, but I think there's a series of seemingly independent issues that are actually interconnected and one is leading to the other but people are trying to find the one root-cause which I don't think exists.
Quoted 7 lines
Anyway, yes, userspace needs to change and provide more information. The trick ofcourse is figuring out which bit of information is critical / useful etc.
There is a definite limit on the amount of constraints you want to solve at runtime.
+1
Quoted 3 lines
Everybody going off and hacking their own thing does not help, we need collaboration to figure out what it is that is needed.
+2 - I've been trying to snoop on many use cases to further understand what truly goes wrong. Some of the issues I've seen were actually due to bugs in the kernel. Other issues could already be fixed with existing facilities, but users didn't know how to use them. So the task is not easy to untangle.
Quoted 8 lines
Quoted 4 lines
Note the original min/wakeup_granularity_ns,
latency_nsetc were tuned by default for throughput by the way (server market bias). You can manipulate those and get better latencies.The immediate problem with those knobs is that they are system wide. But yes, everybody was randomly poking them knobs, sometimes in obviously insane ways.
Yes. I fear though that because they were system wide is that the out-of-the-box experience for many (especially with CFS defaults) were bad latencies. I like the new 3ms base_slice_ns, but I think for many who care about 120Hz refresh for example this is too large. It's almost half of the frame time.
The TICK value plays a big role too. On 4ms TICK, this 3ms will become 4ms if wakeup preemption decided not to preempt immediately.
Is this 3ms a constant by the way? I see it still depends on NR_CPUS, but I read it on different systems and I got 3ms. I think having a constant value across all systems makes more sense. With EAS (which I think someone should put effort to enable it for SMP systems) we tend to pack. And a lot of systems have too few of CPUs and things being packed is common case - I think the rationale in the past was that we distribute tasks to idle CPUs at wake up which is good for latency, but I don't know if this is a good assumption to make still to decide these values.
And looks like we have a bug. I didn't spend a lot of time on studying EEVDF impact on latencies, but I had this simple run with my pi_test [1]. You need sched-analyzer/sched-analyzer-pp somewhere in your path which you can download from [2]. setup Perfetto traced [3]. Running on 6.8.8 M1 Mac Mini
./run.sh 0 0 0
=====================================
:: 2255 | pi_test | ['./pi_test'] ::
====================================================================================================
----------------------------------------------------------------------------------------------------
───────────────────────────── Sum Time in State Exclude Sleeping (ms) ──────────────────────────────
R ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 4533.25
Running ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 4533.75
────────────────────────────── % Time in State Exclude Sleeping (ms) ───────────────────────────────
R ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 50.0
Running ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 50.0
─────────────────────────────────── Sum Time Running on CPU (ms) ─────────────────────────────────── CPU0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 4533.75
──────────────────────────────────── % Time Running on CPU (ms) ──────────────────────────────────── CPU0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 100.0
Time in State (ms):
----------------------------------------------------------------------------------------------------
count mean std min 50% 75% 90% 95% 99% max
state
R 1149.0 3.95 1.12 -0.0 4.0 4.0 6.0 6.01 7.0 7.01
Running 1149.0 3.95 1.11 0.0 4.0 4.0 6.0 6.00 7.0 7.00
Time Running on CPU (ms):
----------------------------------------------------------------------------------------------------
count mean std min 50% 75% 90% 95% 99% max
cpu
0.0 1149.0 3.95 1.11 0.0 4.0 4.0 6.0 6.0 7.0 7.0
=========================================
:: 2257 | pi_test_low | ['./pi_test'] ::
====================================================================================================
----------------------------------------------------------------------------------------------------
───────────────────────────── Sum Time in State Exclude Sleeping (ms) ──────────────────────────────
R ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 4533.89
Running ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 4533.11
────────────────────────────── % Time in State Exclude Sleeping (ms) ───────────────────────────────
R ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 50.0
Running ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 50.0
─────────────────────────────────── Sum Time Running on CPU (ms) ─────────────────────────────────── CPU0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 4533.11
──────────────────────────────────── % Time Running on CPU (ms) ──────────────────────────────────── CPU0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 100.0
Time in State (ms):
----------------------------------------------------------------------------------------------------
count mean std min 50% 75% 90% 95% 99% max
state
R 1154.0 3.93 1.14 0.0 4.0 4.0 6.0 6.0 7.0 7.0
Running 1154.0 3.93 1.13 -0.0 4.0 4.0 6.0 6.0 7.0 7.0
Time Running on CPU (ms):
----------------------------------------------------------------------------------------------------
count mean std min 50% 75% 90% 95% 99% max
cpu
0.0 1154.0 3.93 1.13 -0.0 4.0 4.0 6.0 6.0 7.0 7.0
Note that the average RUNNING (R is for RUNNABLE) time is ~4ms instead of 3ms. Oour P90 and max values are almost double the 3ms slice. I am running with 1ms TICK, so I think there has to be a bug somewhere preventing timely preemption..
If I enable HRTICK it looks much better
=====================================
:: 2517 | pi_test | ['./pi_test'] ::
====================================================================================================
----------------------------------------------------------------------------------------------------
───────────────────────────── Sum Time in State Exclude Sleeping (ms) ──────────────────────────────
R ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 4911.05
Running ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 4911.96
────────────────────────────── % Time in State Exclude Sleeping (ms) ───────────────────────────────
R ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 50.0
Running ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 50.0
─────────────────────────────────── Sum Time Running on CPU (ms) ─────────────────────────────────── CPU0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 4911.96
──────────────────────────────────── % Time Running on CPU (ms) ──────────────────────────────────── CPU0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 100.0
Time in State (ms):
----------------------------------------------------------------------------------------------------
count mean std min 50% 75% 90% 95% 99% max
state
R 1645.0 2.99 0.22 0.0 3.0 3.0 3.0 3.0 3.01 3.97
Running 1646.0 2.98 0.17 -0.0 3.0 3.0 3.0 3.0 3.00 3.01
Time Running on CPU (ms):
----------------------------------------------------------------------------------------------------
count mean std min 50% 75% 90% 95% 99% max
cpu
0.0 1646.0 2.98 0.17 -0.0 3.0 3.0 3.0 3.0 3.0 3.01
=========================================
:: 2519 | pi_test_low | ['./pi_test'] ::
====================================================================================================
----------------------------------------------------------------------------------------------------
───────────────────────────── Sum Time in State Exclude Sleeping (ms) ──────────────────────────────
R ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 4912.11
Running ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 4910.89
────────────────────────────── % Time in State Exclude Sleeping (ms) ───────────────────────────────
R ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 50.01
Running ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 49.99
─────────────────────────────────── Sum Time Running on CPU (ms) ─────────────────────────────────── CPU0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 4910.89
──────────────────────────────────── % Time Running on CPU (ms) ──────────────────────────────────── CPU0.0 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 100.0
Time in State (ms):
----------------------------------------------------------------------------------------------------
count mean std min 50% 75% 90% 95% 99% max
state
R 1644.0 2.99 0.19 -0.00 3.0 3.0 3.0 3.0 3.01 3.95
Running 1643.0 2.99 0.16 0.51 3.0 3.0 3.0 3.0 3.00 3.01
Time Running on CPU (ms):
----------------------------------------------------------------------------------------------------
count mean std min 50% 75% 90% 95% 99% max
cpu
0.0 1643.0 2.99 0.16 0.51 3.0 3.0 3.0 3.0 3.0 3.01
[1] https://github.com/qais-yousef/pi_test
[2] https://github.com/qais-yousef/sched-analyzer/releases
[3] https://github.com/qais-yousef/sched-analyzer?tab=readme-ov-file#perfetto-mode
Quoted 8 lines
Quoted 7 lines
FWIW IMO the biggest issues I see in the scheduler is that its testability and debuggability is hard. I think BPF can be a good fit for that. For the latter I started this project, yet I am still trying to figure out how to add tracer for the difficult paths to help people more easily report when a bad decision has happened to provide more info about the internal state of the scheduler, in hope to accelerate the process of finding solutions.
Quoted 8 lines
So the pitfalls here are that exposing that information for debug purposes can/will lead to people consuming this information for non-debug purposes and then when we want to change things we're stuck because suddenly someone relies something we believed was an implementation detail :/
I've been bitten by this before and this is why I'm so very hesitant to put tracepoints in the scheduler.
I was hoping the 'bare' tracepoint approach I added is okay? I don't need more than that. Function signature and structure internals can never be ABIs. I already had to deal with util_est changes across kernel versions.
If our emperor penguin is reading, it'd be great if he has new thoughts on debug features and userspace dependency. I think we really need to help people to better debug and understand why things aren't behaving as they anticipate. Or at least make it easier to provide info on the list to help us understand what could have gone wrong.
Your concerns are real. These should not prevent code from moving on without worrying about breakages. If anyone latched into those I hope we can tell them sorry, but this one is expected breakage.. I think by design the bare tracepoints can never be ABI though.
Quoted 6 lines
Quoted 4 lines
From what I see, I am hitting bugs here and there
all the time. But they are hard to debug to truly understand where things went wrong. Like this one for example wherePTHREAD_PRIO_PIis a NOP for fair tasks. Many thought using this flag doesn't help (rather than buggy)..Yay for the terminal backlog :/ I'll try and have a look.
It seems hard to fix without Proxy Execution :( If you have ideas for a temporary solution that'd be great. But looks like we just need to get PE merged and available for users - I think John's series doesn't tie this to futex_pi yet.
Quoted 49 lines · Peter Zijlstra
Quoted 7 lines · Tejun Heo
Quoted 3 lines
You Google/Facebook are touting collaboration, collaborate on fixing it. Instead of re-posting this over and over. After all, your main motivation for starting this was the cpu-cgroup overhead.
The hierarchical scheduling overhead isn't the main motivation for us. We can't use the CPU controller for all workloads and while it'd be nice to improve that,
Hurmph, I had the impression from the earlier threads that this ~5% cgroup overhead was most definitely a problem and a motivator for all this.
The overhead was prohibitive, it was claimed, and you needed a solution. Did not previous versions use this very argument in order to push for all this?
By improving the cgroup mess -- I very much agree that the cgroup thing is not very nice. This whole argument goes away and we all get a better cgroup implementation.
Quoted 6 lines
This view works only if you assume that the entire world contains only a handful of developers who can work on schedulers. The only way that would be the case is if the barrier of entry is raised unreasonably high. Sometimes a high barrier of entry can't be avoided or is beneficial. However, if it's pushed up high enough to leave only a handful of people to work on an area as large as scheduling, something probably is wrong.
I've never really felt there were too few sched patches to stare at on any one day (quite the opposite on many days in fact).
There have also always been plenty out of tree scheduler patches -- although I rarely if ever have time to look at them.
Writing a custom scheduler isn't that hard, simply ripping out
fair_sched_classand replacing it with something simple really isn't *that* hard.The only really hard requirement is respecting affinities, you'll crash and burn real hard if you get that wrong (think of all the per-cpu kthreads that hard rely on the per-cpu-ness of them).
But you can easily ignore cgroups, uclamp and a ton of other stuff and still boot and play around.
Quoted 2 lines
I believe we agree that we want more people contributing to the scheduling area.
Quoted 7 lines
I think therein lies the rub -- contribution. If we were to do this thing, random loadable BPF schedulers, then how do we ensure people will contribute back?
That is, from where I am sitting I see $vendor mandate their $enterprise product needs their $BPF scheduler. At which point $vendor will have no incentive to ever contribute back.
Especially in the scheduler space, the incentive to contribute back today is somewhat inverted. As you mention above, it's relatively easy to make custom things, and it's also very difficult to get features and patches included. The cost of maintaining patches out of tree is relatively low in comparison with the cost of working through inclusion, and the scheduler stands out in terms of how hard it is to land changes.
I think the scheduler balances the needs of a wide variety of workloads exceptionally well, but based on the volume of out of tree scheduler infrastructure, it feels like the community is struggling to meet their collaboration needs in the upstream tree.
Just like I can’t imagine one filesystem working for everything, I think we need to open up the field a little on schedulers. As we develop for new variations in workloads, power management, and hardware types, I think sched_ext gives us a way to do more collaboration in the upstream tree, and while I’m not pretending it’s perfect, it’s definitely ready for expansion and broader use.
I do think that sched_ext developers will keep participating upstream, and I agree with a lot of the points that Steve makes in his reply. People are going to keep sending patches in because the kernel community is just the best place to build and maintain this functionality.
-chris
Quoted 15 lines · Peter Zijlstra
Quoted 7 lines · Tejun Heo
Quoted 3 lines
You Google/Facebook are touting collaboration, collaborate on fixing it. Instead of re-posting this over and over. After all, your main motivation for starting this was the cpu-cgroup overhead.
The hierarchical scheduling overhead isn't the main motivation for us. We can't use the CPU controller for all workloads and while it'd be nice to improve that,
Hurmph, I had the impression from the earlier threads that this ~5% cgroup overhead was most definitely a problem and a motivator for all this.
Quoted 7 lines
The overhead was prohibitive, it was claimed, and you needed a solution. Did not previous versions use this very argument in order to push for all this?
By improving the cgroup mess -- I very much agree that the cgroup thing is not very nice. This whole argument goes away and we all get a better cgroup implementation.
I talked with pjt to get some historical context on these patches, it sounds like these were some advocated performance improvements but had fairness issues that Paul pointed out. We're happy to help take a look at this again, but this is all independent from the motivation for sched_ext. Sounds like we're on the same page about this now though :)
Thus, cgroups are not a primary motivator for sched_ext. However, one aspect of cgroups is made quite a bit nicer by the pluggable scheduling. This is the fact that cgroups are a second class citizen in CFS, because they are still a compile time option, so everything must be built to support a thread-only model. That makes it really hard to write group schedulers; fundamentally, task placement, load balancing, etc. is operating on lists of tasks, not lists of cgroups. As you can see, improving hierarchical performance of cgroups is nice to have but not related to this goal.
Quoted 3 lines
Writing a custom scheduler isn't that hard, simply ripping out
fair_sched_classand replacing it with something simple really isn't *that* hard.
Getting that custom scheduler back into upstream is pretty hard though. I like Chris' analogy of filesystems because it gives a really good sense of what a bigger ecosystem might look like with schedulers. It simply is not feasible to implement certain types of behavior in CFS, because the behavior is too specialized for certain classes of workloads, and the model/heuristics could not be made to work in a general purpose scheduling environment, which CFS strives to achieve. Taking these ideas and putting them into a different scheduling class is also a bit of a non-starter. The scheduler is optimized to have most tasks running in a single scheduling class (CFS), and adding additional classes both adds additional static overhead, as well as complexity from dealing with the need to ensure non-starvation of lower priority sched classes (due to the strict priority ordering of sched classes).
As an example, a few years ago Xi posted a simple new scheduling class optimized for high frequency context switching
(https://lkml.org/lkml/2019/9/6/177), which was nack'd offline. There's an argument to be made that some of the functionality there could have been rolled into RT, but I think it serves as a good example of the friction of adding even simple new sched classes.
Pluggable scheduling helps to fill the gap here for policies that have elements making them not a good fit for CFS, given that we don't want a plethora of new sched classes in the tree. That's not to say that just because a policy cannot be fully integrated into CFS it has no benefit for upstream contribution; there are pieces that might make sense to adapt to CFS. For example, we've been experimenting with a policy that schedules based on cgroups rather than simply tasks, and uses that to provide better CCX locality for the cgroup, with better control of how the group spills over to remote CCX. The group based scheduling nature cannot be easily integrated into CFS, as described above, but the CCX scheduling portion could find its way into CFS by means of a more nuanced evaluation of migration_cost, or new placement heuristics.
Quoted 2 lines
But you can easily ignore cgroups, uclamp and a ton of other stuff and still boot and play around.
Please don't underestimate the value that swapping policies around at runtime has. Sure, playing around in a VM isn't bad, but getting performance data from real hardware can take quite a while between boots, not including the time to actually restart the workload and have it warmed up. We're talking several orders of magnitude more latency in the iterative policy development and analysis portion here. I imagine it would have been nice, for example, to swap in and out successive versions of EEVDF while hackbench was actively running, and observe how each swap out changes running averages of latency, etc. Even in a world where we committed to having a single scheduler, I think it would be nice if we had in-tree CFS BPF programs that implemented several of the hooks, just for the purpose of improving development velocity. What BPF has done for networking could be made to improve heuristic heavy areas like select_task_rq, for example.
My point here is that allowing new sched policies is a big benefit, but also placing those policies in BPF is another (separate) big benefit that sched_ext provides natively.
Best,
Josh
Tejun Heo
maintainerHello, Peter.
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Tejun Heo
The hierarchical scheduling overhead isn't the main motivation for us. We can't use the CPU controller for all workloads and while it'd be nice to improve that,
Quoted 7 lines
Hurmph, I had the impression from the earlier threads that this ~5% cgroup overhead was most definitely a problem and a motivator for all this.
The overhead was prohibitive, it was claimed, and you needed a solution. Did not previous versions use this very argument in order to push for all this?
Being able to experiment with potential solutions for problems like hierarchical scheduling overhead is important and something we wanted to demonstrate for sched_ext. It's true that the current hierarchical scheduling is too expensive to deploy on certain workloads but as I wrote before it's also not that difficult to work around and isn't a high priority problem for us.
Quoted 3 lines
By improving the cgroup mess -- I very much agree that the cgroup thing is not very nice. This whole argument goes away and we all get a better cgroup implementation.
Improving the cgroup CPU controller performance would be great. However, I don't see how that'd be an argument against sched_ext. Sure, with sched_ext, we can easily test out potential ideas which can lower the hierarchical scheduling overhead but, if anything, that should make us want it more. Why wouldn't we want to have such ability for other problems too?
Quoted 12 lines
Quoted 6 lines
This view works only if you assume that the entire world contains only a handful of developers who can work on schedulers. The only way that would be the case is if the barrier of entry is raised unreasonably high. Sometimes a high barrier of entry can't be avoided or is beneficial. However, if it's pushed up high enough to leave only a handful of people to work on an area as large as scheduling, something probably is wrong.
I've never really felt there were too few sched patches to stare at on any one day (quite the opposite on many days in fact).
There have also always been plenty out of tree scheduler patches -- although I rarely if ever have time to look at them.
...
Quoted 6 lines
Quoted 2 lines
I believe we agree that we want more people contributing to the scheduling area.
I think therein lies the rub -- contribution. If we were to do this thing, random loadable BPF schedulers, then how do we ensure people will contribute back?
Everything has cost and benefits. Forcing potential contributors into a single narrow funnel has the benefit of concentrating the effort as you're pointing out. However, the cost is that it's a single funnel. In addition to the inherent downsides of having only one of anything, it can handle only so much, and pushes people away from even considering contributing.
There are multiple types of contributions. Getting concrete patches into the main scheduler is one. Trying out wildly different ideas and exploring the problem space is another. Providing a viable competing implementation can be an important contribution too by keeping everyone on their toes. If we concentrate just on direct code contributions, we can lose the sight of the bigger picture costing us in other areas.
During the short period of time that we've been experimenting with sched_ext, we've already found multiple fairly generic approaches that show significant gains. That's not because people who have been playing with sched_ext have special abilities, but rather because there are plenty of sometimes obvious things which have been difficult to try with the in-kernel scheduler. Sure, anyone can modify the kernel, but, without a practical way to publish, deploy and maintain such modifications, it’s really difficult to justify such effort when the chance of landing upstream is really low. If our experience up to this point is any indication, capable engineers who are interested in the area don't seem to be in particularly short supply. What is in short supply is an environment in which they can participate, develop and refine their ideas.
Opportunity cost is often more difficult to appreciate but it is as real as any cost. While there may be more than enough patches for you to review, we are leaving a lot of opportunities unpursued and potential contributors outside the fence because the funnel is too narrow and the barrier of entry too high. Yes, there are benefits to the current setup where we tell everyone to contribute to a single code base but at this point I believe it's costing us more than benefiting.
Quoted 7 lines
That is, from where I am sitting I see $vendor mandate their $enterprise product needs their $BPF scheduler. At which point $vendor will have no incentive to ever contribute back.
And customers of $vendor that want to run additional workloads on their machine are then stuck with that scheduler, irrespective of it being suitable for them or not. This is not a good experience.
The above scenario sounds contrived to me. The situation is already like this with vendor patched kernels. Just like for patched kernels, the vendor has to share the code for sched_ext schedulers due to GPL. After all, the BPF verifier will flat out reject loading any non-GPL programs. In addition, sched_ext has benefits in terms of user experience. Because sched_ext is designed to be supplemental to the default scheduler, its users have an easy out - falling back to CFS/EEVDF by simply unloading the sched_ext scheduler. With patched kernels, they'd have to reboot and a stock kernel might not even be available.
Quoted 7 lines
So I don't at all mind people playing around with schedulers -- they can do so today, there are a ton of out of tree patches to start or learn from, or like I said, it really isn't all that hard to just rip out fair and write something new.
Open source, you get to do your own thing. Have at.
Quoted 7 lines
But part of what made Linux work so well, is in my opinion the GPL. GPL forces people to contribute back -- to work on the shared project. And I see the whole BPF thing as a run-around on that.
Even the large cloud vendors and service providers (Amazon, Google, Facebook etc.) contribute back because of rebase pain -- as you well know. The rebase pain offsets the 'TIVO hole'.
Two things are being conflated here. What GPL gives us is that ideas and code don't get locked up behind a paywall. If someone based their work on a GPL project, others get to take a look at what they did to learn and copy from them. The upstream pressure is a separate mechanism which nudges people towards upstream because the overhead of rebase is painful regardless of the license requirements.
The upstream pressure works well but as I wrote above it also can be pushed too far to the point where it costs rather than benefits long term development. Controlling too tight runs the risk of pushing changes and proposals worth considering under the ground and potential contributors away. It may be difficult to judge and agree on where the current situation exactly is but it is not difficult to see signs of stress. Even just for us, scheduling is one of the common pain points for both server workloads and Oculus. Talking to other organizations, we hear similar concerns.
You said two conflicting things - that people can have at it as it's open source but at the same time that even large organizations are forced to the funnel due to the rebase pain. It's true that even for large organizations, deviating from upstream is expensive. However, big orgs can still do it because the benefit usually scales with the number of machines allowing them to cross the break-even point and thus pay for it.
But the same pain applies to smaller organizations, researchers and individuals. Imagine how big a deterrence the current situation would be for them. It's extremely challenging for them to build a user base and community as it's very awkward to deploy and painful to maintain custom kernels. Some still persevere but most would be discouraged even from starting if the prospect of their work being useful is so slim. This limits potential contributions from a lot of organizations.
CFS / EEVDF is an excellent general purpose scheduler. It obviously is the most used and most important scheduler in the whole world. It's difficult to believe that the only way to get enough people to contribute to it is by suppressing alternatives. The current approach of funneling potential contributors to a single code base with a very high bar creates a lot of pain for those potential contributors, and probably feels unnecessarily punitive to anyone new to the space. If we have to really worry about losing contributors to the main Linux scheduler just because sched_ext creates an additional space that interested engineers can work in, something has gone really wrong and I don't believe that matches the reality.
Quoted 6 lines
But with the BPF muck; where is the motivation to help improve things?
Keeping a rando github repo with BPF schedulers is not contributing. That's just a repo with multiple out of tree schedulers to be ignored. Who will put in the effort of upsteaming things if they can hack up a BPF and throw it over the wall?
I wouldn't be so dismissive about development happening outside the kernel tree. We already see strong community and collaborations in the SCX repo which is serving as an umbrella project for the sched_ext schedulers. Different schedulers are chasing different directions but they actively learn and borrow from each other. It can definitely serve as an incubator to prove and refine new ideas which can be adopted widely and to grow scheduling engineers.
For an example, it's still early but Changwoo's work on interactivity in scx_lavd seems generally useful and has already been adopted by scx_rusty. It's something which can easily be applied to EEVDF too. Changwoo may or may not work on EEVDF directly (he says he wants to) but the code change necessary is neither big nor difficult. Figuring out what actually works was the hard part, not the implementation. Not all ideas would be like this but this serves as a good example of how contribution is not just directly writing patches and how work outside the tree can benefit the kernel.
Quoted 5 lines
So yeah, I'm very much NOT supportive of this effort. From where I'm sitting there is simply not a single benefit. You're not making my life better, so why would I care?
How does this BPF muck translate into better quality patches for me?
I'm not sure whether it would make your life better but I firmly believe that it will benefit overall Linux scheduling in the long term. You don't necessarily have to care. We'll do our best to ensure that it bothers you as little as possible.
Maybe I'm mistaken and we won't find much that'd be useful enough for EEVDF but also maybe there are enough things that we haven't tried that will make things better for everyone. I believe in the latter and the indications till now seem to agree. You don't have to share my optimism but wouldn’t it at least be worthwhile to find out?
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerHello, Peter.
A gentle ping. I don't think this subthread reached its conclusion. Would you mind sharing your thoughts?
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerHello,
It has been a couple weeks, so I take it that you aren't intending to respond. I think it'd be useful to summarize the arguments against sched_ext and list the counter-points.
(1) Merging sched_ext will weaken the incentive to contribute.
While this may partially be true, it isn't looking at the whole picture. This argument looks at the costs of sched_ext while ignoring the benefits, and it ignores the costs of funneling all scheduler work through one codebase.
If you look at the whole picture, I think you’ll see that:
- The problem space of CPU scheduling is too big for a single code base to be effective. Hardware has changed a lot and so have the workloads. There are many areas that we haven't mapped out. It's difficult to try anything radical in a code base which has to satisfy everyone all the time, but holding the bar so high that experimentation is suppressed means we will all be worse off.
- The bar for contribution is too high, driving away potential contributors. Many vendors and users carry internal patches as the upstreaming cost is too high. We are already seeing multiple developers who have not previously contributed to fair.c actively participating in and driving sched_ext schedulers. It’s possible those developers will eventually contribute to fair.c, but if sched_ext didn’t exist this would be less likely.
The constraint of only one scheduler codebase makes it very difficult to contribute. You say that this constraint is necessary to force collaboration, but I think the opposite is happening - many people don't bother trying to contribute because the bar is too high. If sched_ext is merged, the scheduler code base may lose some of the enforcement. However, in the longer term, I believe we will gain more talented and motivated engineers working in the problem space and some of them will surely find it worthwhile to contribute to fair.c. It will be the most widely used scheduler in the world no matter what, and will be attractive for people to work on.
EEVDF worked out because you have worked on the scheduler for a long time and have gained a ton of context on what works and doesn't. It also worked out because you were more confident that it'd get merged. How do we build confidence in other developers who want to explore whatever comes after EEVDF without worrying that it is hopeless to try? sched_ext provides an outlet for people who aren't already established to take a smaller risk first, which is likely to lead to more people contributing.
(2) Efforts and developments out of the kernel tree are worthless.
I believe this is too narrow a view. Direct contribution is one form of contribution but there are many others, including research. EEVDF itself is based on a research paper. Figuring out what works and sharing them seems as important as anything to me.
One reason cited for the uselessness is that out-of-tree efforts are often throw-away and don't build up to anything. There is some truth to this but the main reason is the difficulty of working with out-of-tree kernel modifications. Rebase is painful and there is no convenient way to distribute to users. Some still power through but it's near impossible to build a user base and community for things that are out-of-tree. sched_ext solves these problems and the umbrella repo serves as the central repository for the developers to collaborate and learn from each other. This isn’t a prediction for the future, it is something which is already actively happening.
Given the right environment, they will keep flourishing and finding new ways to improve scheduling. Many of them won't be applicable to the built-in scheduler, but some will. It's also likely that, in the long term, the larger scheduler developer base will be directly beneficial to the built-in scheduler too.
(3) This will lead to vendor-specific fragmentation.
This is already happening with or without sched_ext whether that's in the form of out-of-tree scheduler patches or people trying to circumvent the scheduler with creative uses of the RT class.
sched_ext will introduce a different mode of doing it. There are scenarios where the situation can become a bit worse but I don't believe the difference would be drastic. Because all sched_ext schedulers have to be under the GPL, any vendor shipping a sched_ext scheduler to a customer will have to publish the code. If there are useful ideas we'll be just as free to take them as now. Also, users would have the benefit that it's a lot easier to opt out of the vendor's scheduler.
On balance, yes, sched_ext may lead to more or at least different types of fragmentation, but that seems like a minor downside compared to the overall benefits especially given that we have to live with some level of fragmentation no matter what.
(4) sched_ext is a debug tool and we don't merge debug tools.
I think both parts of the above claim are wrong. sched_ext can be used purely as a debug tool but it's also performant and flexible enough to readily enable non-trivial practical use cases. We are using it in production today, and as stated elsewhere in this thread, there are multiple other companies in various stages of rolling it out to production. It can be a debug tool, a temporary bridge to field early ideas while working on something more permanent, a proper solution to specific problems which don't quite fit the general scheduler (an extreme example would be standard-dictated scheduling for avionics), and so on.
Also, we merge debug tools all the time. Lockdep is a debug tool. The code base is full of debug features and components. Why wouldn't we merge something if it makes the lives of the developers and users better by making it easier to understand and debug problems? We don't merge printks someone sprinkled over the code base to debug one particular problem. We do and should merge tools and frameworks which improve visibility and debugging.
To reiterate our proposition: Let’s please open it up. Scheduling doesn’t have to be this closed. Many open subsystems survive fine and often thrive thanks to their openness. sched_ext hooks into the core scheduling but the contact surface is limited, and, if they ever get in the way, we’ll do our best to resolve them. The balance in the trade-offs seems pretty obvious.
Thanks.
--
tejun
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Tejun Heo
Hello, Peter.
Quoted 3 lines · Peter Zijlstra
Can you please put your efforts and the touted Google collaboration in
fixing the existing cgroup mess?I suppose you're referring to Rik's flattened hierarchy patchset.
Quoted 6 lines
You guys Google/Facebook got us the cgroup thing, Google did a lot of the work for cpu-cgroup, and now you Facebook say you can't live with it
because it's too expensive. Yes Rik did put a lot of effort into it, but
Google shot it down. What am I to do?
I believe the issues that Paul pointed out with my flattened cgroup code are fixable. I ended up not getting back to this code because it took me a few months to think of ways to fix the issues Paul found, and by then I had moved on to other projects.
For reference, Paul found these two (very real) issues with my implementation.
1) Thundering herd problem. If many tasks in a low priority cgroup wake up at the same time, they can end up swamping a CPU.
I believe this can be solved with the same idea I had for reimplementing CONFIG_CFS_BANDWIDTH. Specifically, the code that determines the time slice length for a task already has a way to determine whether a CPU is "overloaded", and time slices need to be shortened. Once we reach that situation, we can place woken up tasks on a secondary heap of per-cgroup runqueues, from which we do not directly run tasks, but pick the lowest vruntime task from the lowest vruntime cgroup and put that on the main runqueue, if the previously running task has a vruntime that is higher than that of a task in the secondary group. If a task is woken up in a cgroup that already has tasks on that secondary queue, we wake up the task onto that secondary queue.
This means on overloaded CPUs, we move back to
a task selection mechanism closer to what we
currently have, while in the non-overloaded
situation we use a flat runqueue.
This same scheme could be used to implement
CFS bandwidth control. A task belonging to a
throttled group would be placed on the group's
queue, not the CPU's flat runqueue.
2) The vruntime for a task can be advanced by way to much at once. If we have tasks A & B running, and task B has a priority that is 1/100th of that of task A, its vruntime would be advanced 100x as much as task A, when running the same length time slice.
This creates a big issue if we get a wakeup of task C, at the same priority as task B, and then task A goes to sleep. Due to the very far advanced runtime of task B, task C could get to monopolize the CPU for a considerable amount of time, and task B could get starved.
A potential fix for this is to never account more than the maximum time slice length at a time, while any excess delta_exec time for the task gets remembered.
At pick_next_entity time, the scheduler can see that task B has a lot of delta_exec time left, and account up to the maximum slice length to the task's vruntime, and place it back in the queue if the next task now has a lower vruntime.
For a steady state of a high priority task A and a low priority task B, this makes pick_next_task more expensive, but when task A disappears and task C appears, CPU time will continue to be fair between them.
Limiting the total weight of tasks on the flat runqueue, using the mechanism for thundering herd and CFS bandwidth outlined above, should keep this overhead bounded to something reasonable.
Does the above sound like it would work?
Does it sound like code that you would be ok with merging?
Is it a large enough improvement over the current hierarchical runqueue that it would be worth doing?
This would be a fairly large project, so we should probably discuss some of the details before investing too much time in it.
Signature · 1 line
All Rights Reversed.
Tejun Heo
maintainerHello, Rik.
Quoted 10 lines · Rik van Riel
I believe the issues that Paul pointed out with my flattened cgroup code are fixable. I ended up not getting back to this code because it took me a few months to think of ways to fix the issues Paul found, and by then I had moved on to other projects.
For reference, Paul found these two (very real) issues with my implementation.
1) Thundering herd problem. If many tasks in a low priority cgroup wake up at the same time, they can end up swamping a CPU.
The way that scx_flatcg (which seems broken right now, will fix it) works around this problem is by always doing two-level scheduling. ie. The top level rbtree hosts the tasks in the root cgroup and all the active cgroups, where each cgroup is scheduled according to their current flattened hierarchical share. This seems to work pretty well as what becomes really expensive is the repeated nesting which can easily go 6+ levels.
This doesn't solve the thundering herd problem completely but shifts it one level. ie. Thundering herds of threads can be handled easily. However, thunderding herds of cgroups can still cause unfairness. I can imagine cases where this can lead to scheduling issues but they all seem pretty convoluted and artificial. Sure, somebody who's intentionally adversarial can cause temporary issues but perfect isolation of adversarial actors isn't what cgroups can or should practically target.
Even in the case this is an actual issue, we can solve it by limited nesting. cgroups already have deligation boundaries. Maybe it needs to be made more explicit but one solution could be adding a nesting layer only on delegation boundaries so that misbehaviors are better contained within each delegation domain.
Quoted 8 lines
I believe this can be solved with the same idea I had for reimplementing
CONFIG_CFS_BANDWIDTH. Specifically, the code that determines the time slice length for a task already has a way to determine whether a CPU is "overloaded", and time slices need to be shortened. Once we reach that situation, we can place woken up tasks on a secondary heap of per-cgroup runqueues, from which we do not directly run tasks, but pick the lowest vruntime task from the lowest vruntime cgroup and put that on the main runqueue, if the previously running
When overloaded, are the cgroups being put on a single rbtree? If so, they'd be using flattened shares, right? I wonder what you're suggesting for the overloaded case is pretty simliar to what flatcg is doing plus avoiding one level of indirection while not overloaded.
Thanks.
Signature · 1 line
tejun
Quoted 17 lines · Tejun Heo
Quoted 5 lines · Rik van Riel
I believe this can be solved with the same idea I had for reimplementing
CONFIG_CFS_BANDWIDTH. Specifically, the code that determines the time slice length for a task already has a way to determine whether a CPU is "overloaded", and time slices need to beQuoted 8 lines
shortened. Once we reach that situation, we can place woken up tasks on
a secondary heap of per-cgroup runqueues, from which we do not directly
run tasks, but pick the lowest vruntime task from the lowest vruntime
cgroup and put that on the main runqueue, if the previously running
Quoted 7 lines
When overloaded, are the cgroups being put on a single rbtree? If so, they'd
be using flattened shares, right? I wonder what you're suggesting for the
overloaded case is pretty simliar to what flatcg is doing plus avoiding one
level of indirection while not overloaded.
It does indeed sound like flatcg is doing almost the same thing.
I'm not entirely sure what to make of the fact that we both came up with the same solution to the problem. I suppose it's a nice improvement over a fully hierarchical solution, especially when it comes to overhead, but there is still a fair amount of complexity left.
I don't know if there is a simpler solution to this problem.
There might not be.
Signature · 1 line
All Rights Reversed.
Changwoo Min
reviewerHello,
I'd like to reaffirm Valve and Igalia's backing for the sched_ext proposal.
Let's delve into the context first. Valve, in collaboration with Igalia and other firms, has been dedicated to enhancing the gaming experience on Linux. Our endeavor involves utilizing a standard Linux distribution (SteamOS) to execute unaltered Windows games on the Linux kernel with the aid of Wine and various other software components. The overarching objective is to refine the Linux desktop environment for gaming and interactive purposes. As part of our commitment, we adhere to an "upstream everything" policy, contributing to the Linux kernel and numerous open-source projects. For those interested, you can explore the details of our contributions through the following link:
From our perspective, sched_ext holds significant promise and utility, particularly in facilitating rapid experimentation with new ideas. Our experimental ideas may or may not align with the existing scheduler designs, be it CFS or EEVDF.
Specifically, our research into the characteristics of gaming workloads for schedulers has unveiled intriguing insights that could inform better scheduling decisions. For instance, tasks within the gaming software stack, such as game engines, Wine, and graphics drivers, often exhibit very short duration when scheduled, necessitating frequent scheduling activities. Moreover, multiple tasks across software layers collaborate to complete a single application-level task, forming task chains. Inadequate scheduling decisions within these chains can lead to high tail latency, commonly known as "stuttering" in the gaming community.
Quoted 2 lines
Witness the metric ton of toy schedulers written for it, that's all effort not put into improving the existing code.
While these properties offer valuable insights for improving scheduling decisions for gaming workloads, their applicability to general-purpose schedulers like EEVDF remains uncertain. The most effective means to evaluate their broader utility is through practical experimentation. In this regard, sched_ext provides an excellent platform for rapid testing of new ideas.
One may question why not just experiment out-of-tree? In reality, we can’t just trivially patch _general-purpose_ EEVDF (and CFS too) to be better for _all_ use cases, especially when upstream has resisted tons of niche complexity in the upstream scheduler. It is a very hard problem, and we believe having the sched_ext upstream for more users/distros will encourage more progress. Our case of Linux gaming demonstrates that working on the existing code is neither always possible nor effective. Further details of our findings can be found through the following link:
https://ossna2024.sched.com/event/1aBOT/optimizing-scheduler-for-linux-gaming-changwoo-min-igalia
Quoted 2 lines
situation that's been created is solved. And even then, I fundamentally believe the approach to be detrimental to the scheduler eco-system.
Contrary to the notion that sched_ext might prove detrimental to the scheduler ecosystem, we hold a different view. The successful implementation of sched_ext enriches the scheduler community with fresh insights, ideas, and code. For instance, our adoption of a virtual deadline-based approach in designing LAVD (Latency-criticality Aware Virtual Deadline), our sched_ext-based scheduler for gaming, represents a deliberate design choice. Aligning our heuristics and findings with EEVDF through a similar virtual deadline-based approach enables us to contribute our discoveries to EEVDF in the future once proven to be more universally applicable. Notably, the concept of "latency criticality" in LAVD holds promise beyond gaming workloads, potentially benefiting various interactive workloads. If you are interested in, you can find the source code of LAVD in the following link:
https://github.com/sched-ext/scx/tree/main/scheds/rust/scx_lavd
In essence, I envision sched_ext and its community as an incubator for new ideas, invigorating the scheduler ecosystem. Some of the "toy" schedulers may evolve into specialized solutions tailored for specific problem domains, such as HPC, AI/ML, or gaming. Lessons learned from these experimental schedulers will invariably contribute, directly or indirectly, to the evolution of the EEVDF scheduler.
Sincerely,
Changwoo Min
Hi everyone,
We (CachyOS [0]) want to support to proposal of sched_ext.
CachyOS is an archlinux based distribution, which improves the performance, throughput and interactivity for the desktop but also on servers. We have a big community, which likes testing new features on the kernel, GPU drivers, frequency drivers, new locking technologies and schedulers.
[0]: https://cachyos.org/
For CachyOS, schedulers have always been a key matter. We have numerous variants of kernels with different schedulers so that everyone can be satisfied. So far, schedulers in the kernel have faced one problem, namely: 1 kernel = 1 scheduler. It was impossible to change the CPU scheduler without restarting the computer and booting another kernel. scx-scheduler simply and extremely successfully changes this approach and allows you to change schedulers at runtime. The approach with a single default scheduler in the kernel also leads to a problematic situation, because in our opinion it is impossible to develop a single scheduler that is the optimal solution for all possible tasks. Again, and in this case, scx-scheds makes this problem obsolete.
The EEVDF Scheduler gives a good overall experience, but it currently lacks good interactivity under load, specific scenarios like gaming and also the latency. Rusty gives CachyOS currently a very good replacement for desktop usage. We are planning to provide the Rusty Scheduler in the future as default. The interactivity of Rusty is dramatically better than EEVDF, especially when a heavy workload is running in the background, e.g. during compilation. Using the desktop under such a workload can be very challenging for the default scheduler and at rusty it is not even noticeable that during compilation is running. Also in gaming scenarios rusty seems to provide excellent performance.
The LAVD Scheduler is integrated as default in our handheld variant and shows very good impressive results in frame times as well as 1% lows. This gives our user on the Handheld Edition a better experience.
It is also worth mentioning that our community takes an active part in sched-ext testing and regularly reports bugs and suggestions for changes. Cooperation with developers is also perfect - they immediately make corrections if there are any regressions.
Regards,
Peter Jung
CachyOS
On Wed, May 01, 2024 at 05:09:35AM -1000, Tejun Heo wrote:
...
Quoted 6 lines
- Ubuntu is considering to include
sched_extin the upcoming 24.10 release. Andrea Righi of Canonical has been actively working on a userspace scheduling framework since the end of the last year.https://github.com/sched-ext/scx/tree/main/scheds/rust/scx_rustland https://discourse.ubuntu.com/t/introducing-kernel-6-8-for-the-24-04-noble-numbat-release/41958
Regarding this topic, I can confirm that Ubuntu intends to provide official support for a sched_ext kernel in the 24.10 release.
We still have to finalize all the specific details of the plan, but essentially, there will be a separate "derivative" kernel featuring sched_ext, alongside a user-space scx package(s) for the schedulers and tools (which, ideally, we would also like to upstream into Debian).
-Andrea
[ Tejun reminded me about this, and discussion hasn't really gone anywhere for much too long, so now I just need to be the person who makes a decision and people can hate on ]
Quoted 5 lines · Tejun Heo
This is v6 of
sched_ext(SCX) patchset.During the past five months, both the development and adoption of
sched_exthave been progressing briskly. Here are some highlights around adoption:
[...]
I honestly see no reason to delay this any more. This whole patchset was the major (private) discussion at last year's kernel maintainer summit, and I don't find any value in having the same discussion (whether off-list or as an actual event) at the upcoming maintainer summit one year later, so to make any kind of sane progress, my current plan is to merge this for 6.11.
At least that way, we're making progress, and the discussion at KS 2024 can be about my mental acuity - or lack thereof - rather than about rehashing the same thing that clearly made no progress last year.
I've never been a huge believer in trying to make everybody happy with code that is out of tree - we're better off working together on it in-tree.
And using the "in order to accept this, some other thing has to be fixed first" argument doesn't really work well either (and _that_ has been discussed for over a decade at various maintainer summits).
Maybe the people who have concerns about this can work on those concerns when it's in-tree.
I'm also not a believer in the argument that has been used (multiple times) that the BPF scheduler would keep people from participating in scheduler development. I personally think the main thing that keeps people from participating is too high barriers to participation.
Anyway, this is the heads-up to Tejun to please just send me a pull request for the next merge window.
And for everybody else as a "It's happening" heads-up.
[ Please just mentally insert the "IT'S HAPPENING" meme gif here - because if I actually were to include it here, lkml would just reject this email. Sometimes the anti-html rules don't work in our favor ].
Linus
Tejun Heo
maintainerHello,
Quoted 2 lines · Linus Torvalds
Anyway, this is the heads-up to Tejun to please just send me a pull request for the next merge window.
As there have been some bug fixes and minor updates since v6, I'll refresh and post the v7 patchset and start the sched_ext korg tree off of it. To be on the safe side, I'll separate out parts which interact with other subsystems (e.g. cgroup and schedutil) into their own patchsets and handle them as normal patches targeting the sched_ext tree.
Thank you so much.
Signature · 1 line
tejun
Linus!
Quoted 6 lines · Linus Torvalds
Quoted 5 lines · Tejun Heo
This is v6 of
sched_ext(SCX) patchset.During the past five months, both the development and adoption of
sched_exthave been progressing briskly. Here are some highlights around adoption:
Quoted 8 lines
[...]
I honestly see no reason to delay this any more. This whole patchset was the major (private) discussion at last year's kernel maintainer summit, and I don't find any value in having the same discussion (whether off-list or as an actual event) at the upcoming maintainer summit one year later, so to make any kind of sane progress, my current plan is to merge this for 6.11.
I was part of that discussion and sat down for quite some time with the sched_ext people to find a constructive way out of this situation. My memory might trick me, but I remember clearly that there was consensus to resolve this in a constructive and collaborative way.
Unfortunately I ran out of cycles after Richmond to follow up and the fact that Peter wrecked his shoulder and was AFK for months did not make it any better.
However, the sched_ext people did not follow up either especially not regarding a clean integration along the scheme I asked them for in November. Contrary to that the series gained more ad hoc warts.
That's water under the bridge, but it clearly shows how non-constructive this has become.
So instead of "solving" this brute force and thereby proliferating the non-constructive situation, can you please hold off with that plan to merge it as is and give us three month to get this onto a collaborative and constructive track?
I can make cycles available to work with both sides to get this resolved for the benefit of everyone.
A clean integration will help both ends and makes both the existing code and the new code better and easier to maintain together. IIRC, that's something you yourself asked people to do in the past.
Thanks,
Thomas
Quoted 5 lines · Thomas Gleixner
So instead of "solving" this brute force and thereby proliferating the non-constructive situation, can you please hold off with that plan to merge it as is and give us three month to get this onto a collaborative and constructive track?
The thing is, I have seen absolutely _nothing_ in the last 9 months or so.
So to me, "three more months" sounds like just delay.
Linus
Linus!
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Thomas Gleixner
So instead of "solving" this brute force and thereby proliferating the non-constructive situation, can you please hold off with that plan to merge it as is and give us three month to get this onto a collaborative and constructive track?
The thing is, I have seen absolutely _nothing_ in the last 9 months or so.
Right, but that applies to both sides, no?
Quoted 1 line
So to me, "three more months" sounds like just delay.
Three months delay are well worth to give it try, but that's your decision.
Thanks,
Thomas
Quoted 7 lines · Thomas Gleixner
Quoted 3 lines · Linus Torvalds
The thing is, I have seen absolutely _nothing_ in the last 9 months or so.
Right, but that applies to both sides, no?
But Thomas, this isn't a "both sides" issue.
This is a "people want to do new code and features, and the scheduler people ARE ACTIVELY HOLDING IT UP" issue.
Yes, part of that "actively holding it up" is trying to make rules for "you need to do this other XYZ thing to make us happy".
But no, then "not doing XYZ" does *NOT* make it some "but but other side" issue.
This, btw, is not some new thing. It's something that has been discussed multiple times over the years at the maintainer summit for different maintainers. When people come in and propose feature X, it's not kosher to then say "you have to do Y first".
And yes, maybe everybody even agrees that Y would be a good thing, and yes, wouldn't it be lovely if somebody did it. But the people who wanted X didn't care about Y, and trying to get Y done by then gating X is simply not ok.
Now, if there was some technical argument against X itself, that would be one thing. But the arguments I've heard have basically fallen into two camps: the political one ("We don't want to do X because we simply don't want an extensible scheduler, because we want people to work on _our_ scheduler") and the tying one ("X is ok but we want Y solved first").
I was hoping the tying argument would get solved. I saw a couple of half-hearted emails to that effect, and Rik at some point saying "maybe the problems are solvable", referring to his work from a couple of years ago, but again, nothing actually happened.
And I don't see the argument that the way to make something happen is to continue to do nothing.
https://www.youtube.com/watch?v=lOTyUfOHgas
Because if you are serious about making forward progress *with* the BPF extensions, why not merge them and actually work with that as the base?
IOW, what is the argument for _not_ merging it?
Linus
Linus!
Quoted 4 lines · Linus Torvalds
Quoted 1 line · Thomas Gleixner
Right, but that applies to both sides, no?
But Thomas, this isn't a "both sides" issue.
It is whether you like it or not.
I clearly gave the sched_ext people deep technical feedback on the way how they integrated this seven months ago in Richmond.
I sat down with them because _you_ asked me explicitely for it as _you_ did not want to deal with it.
That feedback got ignored completely. Thank you very much for wasting _my_ time.
Quoted 6 lines
This is a "people want to do new code and features, and the scheduler people ARE ACTIVELY HOLDING IT UP" issue.
Yes, part of that "actively holding it up" is trying to make rules for "you need to do this other XYZ thing to make us happy".
Quoted 6 lines
But no, then "not doing XYZ" does *NOT* make it some "but but other side" issue.
This, btw, is not some new thing. It's something that has been discussed multiple times over the years at the maintainer summit for different maintainers. When people come in and propose feature X, it's not kosher to then say "you have to do Y first".
Seriously? You yourself requested this from me more than once in the past 20 years. But let's not go there.
Quoted 4 lines
And yes, maybe everybody even agrees that Y would be a good thing, and yes, wouldn't it be lovely if somebody did it. But the people who wanted X didn't care about Y, and trying to get Y done by then gating X is simply not ok.
I very well remember the discussions about fix X first before Y.
But you are completely ignoring the fact that in this case
- problem X got introduced by the very same people who are now pushing for Y
- the resolution to problem X was not rejected by the scheduler people at all
It stayed unresolved because the very same parties which are now so cosily working together on Y (aka sched_ext0 could not agree on anything
So the mess they created in the first place stays as is and both parties "solved" their problem with Y (aka sched_ext) and have their own implementations to work around it how they see fit.
IOW, what you are saying is:
#1 Push for your solution X and get it merged
#2 Ignore the resulting problems related to X for at least a decade
#3 Come up with a half baken workaround Y to be able to deal with #2
and leave everyone else behind to deal with the results of #1 and #2
#4 Make Linus decide that Y is the right thing to do because other
people suck
The past discussions about fix "X" before "Y" were distinctly different.
Quoted 6 lines
Now, if there was some technical argument against X itself, that would be one thing. But the arguments I've heard have basically fallen into two camps: the political one ("We don't want to do X because we simply don't want an extensible scheduler, because we want people to work on _our_ scheduler") and the tying one ("X is ok but we want Y solved first").
There have been voiced a lot of technical arguments, which never got addressed and at some point people gave up due to being ignored.
When I sat there in Richmond with the sched_ext people I gave them very deep technical feedback especially on the way how they integrate it:
Sprinkle hooks and callbacks all over the place until it works by some definition of works.
That's perfectly fine for a PoC, but not for something which gets merged into the core of an OS. I clearly asked them to refactor the existing code so that these warts go away and everything is contained into the scheduler classes and at the very end sched_ext falls into place. That's a basic engineering principle as far as I know.
They nodded, ignored my feedback and just continued to persue their way.
Are you still claiming with a straight face that this is a problem rooted in one party?
It's a problem of one interest group to get this into the tree no matter what.
I sat in the room at OSPM 2023 when the introduction of the sched ext advertisement started with "This saves us (FB) millions and Google is collaborating with us [unspoken - to save millions too by eliminating their unmaintainable scheduler hacks]', which is clearly a technical argument, right?
Quoted 4 lines
I was hoping the tying argument would get solved. I saw a couple of half-hearted emails to that effect, and Rik at some point saying "maybe the problems are solvable", referring to his work from a couple of years ago, but again, nothing actually happened.
See above. It's not a problem of the scheduler people.
It's a problem created by the very same people who refuse to solve it and at the same time push for their new magic cure.
Quoted 2 lines
And I don't see the argument that the way to make something happen is to continue to do nothing.
I clearly offered you to try to resolve this amicably within a reasonable time frame.
How exaclty is that equivalent to "continue to do nothing" ?
Quoted 3 lines
Because if you are serious about making forward progress *with* the BPF extensions, why not merge them and actually work with that as the base?
There is a very simple argument to this:
- The way how it is integrated sucks in a big way on purely technical grounds. If you want details, I'm happy to reiterate them again in a separate mail just in case you can't find them in your inbox or can't remember what I told you seven month ago.
- This got pointed out by myself seven months ago to the sched_ext people
- They sat with me at the table and nodded
- Then they went off and ignored it for seven months and just added more warts.
Now you are asking me seriously why I don't want to see this merged in the technical state which it is in right now?
Are you seriously expecting that this is resolved amicably just by forcing this into the tree on the obvious expense of the people who were working on the existing code for decades?
That aside, your whole argument about the so "very high" participation barrier on the scheduler is just based on hearsay and not on facts, which you could easily have gathered yourself. Let me do your homework.
In the past five years there have been:
~4000 mail (patch) threads related to scheduler issues ~1600 commits (i.e. 1.6 commits per work day on average)
~300 different authors
300 different authors is clearly not a sign of a problematic community neither is the ratio of commits to mail threads is 1:2.5.
The fact tell clearly that this is a healthy community and not a sign of a participation barrier problem.
The fact, that the contributions and contribution attempts from the proponents of sched_ext are close to zero cannot be abused to claim that there is a high bar to get patches into the scheduler subsystem.
If you don't try in the first place then you can't complain about it, no?
Just for the record, the scheduler people and myself spent a lot of time to help to get intrusive features like UMCG into mainline, but the efforts were dropped by the submitters for no reason. Short time after that sched_ext came around.
Can we please agree that the root of this has left the technical grounds long ago?
If you think that the correct non-technical solution is to resolve this brute force without giving those who are willing to work this out in the proper way a completely irrelevant delay of three month, then I really have to ask you what you are trying to achive.
If you pull that stuff as is then you create a patently bad precedent and on top of that you slap everyone who worked and works collaboratively and constructively with maintainers and the wider community to get their features merged straight into their face.
You are obviously free to do so, but then please clearly state that this is the new world order by merging an unreviewed patch against Documentation/process/* which makes this as a general rule applicable for everyone.
We've been there and done that during the 2.5 period and it took us years to recover from it, but maybe you have forgotten about that because you merely had to merge the fixes which were created by people who cared and lost their sleep over the mess.
Whatever the outcome is, we definitely have to have a major discussion about the underlying problem at the maintainer summit whether you like it or not.
That said, my offer stands to work on an amicable and collaborative solution for this nuisance, but that's obviously all I can do.
Up to you.
Thanks,
Thomas
[ I'll try to look more at this tomorrow, but I'll send this part early ]
Quoted 6 lines · Thomas Gleixner
When I sat there in Richmond with the
sched_extpeople I gave them very deep technical feedback especially on the way how they integrate it:Sprinkle hooks and callbacks all over the place until it works by some definition of works.
Are we even talking about the same thing?
There are basically two new hooks, for reweight_task (which is something the fair scheduler wanted and was the only user of) and for switching_to(), which is the class changing (again, mainly because there's now not a hardcoded "normal" class).
And yes, there are a couple of other things where the CFS rules were just encoded in the core scheduler code, and they got an extra check or whatever (eg the SCHED_NORMAL changes and things like stop-tick - things that changed simply because now there isn't a single normal scheduler any more).
The rest are mostly all the existing scheduler call-ins, afaik. Or _exactly_ the same thing that other schedulers already do, like the task_prio() stuff.
Yes, there's scx_rq_activate/deactivate at CPU up/down time. Doesn't look unreasonable to me. Same goes for the idle cpu management.
In other cases, it takes a few code sequences, turns them into helper functions, just to be able to re-use them.
The ugliest parts are from what I can see the whole "ok, stop using user space input over PM events" and that "bypass" stuff is sure not pretty.
But that's pretty much all internal to sched_ext, and seems mostly like a sane approach to "what if we do policy in user space"?
And scx_next_task_picked() isn't pretty - as far as I understand, it's because there's only a "class X picked" callback ("pick_next_task()"), and no way to tell other classes they weren't picked.
But "sprinkle hooks and callbacks all over the place"?
Could things like that next_active_class() perhaps be done more prettily? I'm sure.
But I get the very strong feeling that people wanted to limit the amount of changes they made to the core scheduler code.
Quoted 4 lines
I clearly offered you to try to resolve this amicably within a reasonable time frame.
How exaclty is that equivalent to "continue to do nothing" ?
So if we actually *can* resolve this amicably in three months, then that sounds worth it.
But my reaction is "what changed"? Nothing has become more amicable in the last nine months. What makes the next three months special?
Linus
Quoted 4 lines · Linus Torvalds
And
scx_next_task_picked()isn't pretty - as far as I understand, it's because there's only a "class X picked" callback ("pick_next_task()"), and no way to tell other classes they weren't picked.
I guess that could be a class callback, something like this:
p = class->pick_next_task(rq);
if (p)
if (p) {
- scx_next_task_picked(rq, p, class);
+ struct sched_class *prev = last->sched_class;
+ if (class != prev && prev->switch_class)
+ prev->switch_class(rq);
return p;
}and that would be arguably much prettier. But maybe I've mis-understood the reason for that scx_next_task_picked() thing.
Tejun?
Linus
Tejun Heo
maintainerHello, Linus.
Quoted 19 lines · Linus Torvalds
Quoted 4 lines · Linus Torvalds
And
scx_next_task_picked()isn't pretty - as far as I understand, it's because there's only a "class X picked" callback ("pick_next_task()"), and no way to tell other classes they weren't picked.I guess that could be a class callback, something like this:
p = class->pick_next_task(rq); if (p) if (p) {(quoted diff) +3
-1- scx_next_task_picked(rq, p, class); + struct sched_class *prev = last->sched_class; + if (class != prev && prev->switch_class) + prev->switch_class(rq); return p; }
Quoted 2 lines
and that would be arguably much prettier. But maybe I've mis-understood the reason for that
scx_next_task_picked()thing.
Yes, this would work. The callback is there to notify the BPF scheduler when SCX class is preempted by one of the higher priority classes so that e.g. the BPF scheduler can punt the task[s] that was running on or waiting for the CPU to other CPUs. I'll prep a patch to make it a sched_class callback.
There are other hooks which are trickier. e.g. scx_tick() wants to be called regardless of the class of the current task for the watchdog and scx_rq_[de]activate() are there for two reasons - 1. sched core doesn't distinguish actual CPU hotplugs and sched domain updates but the latter doesn't translate well to BPF schedulers 2. it's nice to give sleeping context to the BPF scheduler. The fork hooks are in a similar boat as SCX just needs more synchronization and sleepable context where other classes don't and likely won't.
I can make all of them callbacks but I'm not sure that'd be all that useful for other classes and the semantics would be different from other callbacks, so it's unclear that'd be an overall win.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 1/1 sched, sched_ext: Replace scx_next_task_picked() with sched_class->switch_class()
scx_next_task_picked() is used by sched_ext to notify the BPF scheduler when a CPU is taken away by a task dispatched from a higher priority sched_class so that the BPF scheduler can, e.g., punt the task[s] which was running or were waiting for the CPU to other CPUs.
Replace the sched_ext specific hook scx_next_task_picked() with a new sched_class operation switch_class().
The changes are straightforward and the code looks better afterwards. However, when !CONFIG_SCHED_CLASS_EXT, this just ends up adding an unused hook which is unlikely to be useful to other sched_classes. We can #ifdef the op with CONFIG_SCHED_CLASS_EXT but then I'm not sure the code necessarily looks better afterwards.
Please let me know the preference. If adding #ifdef's is preferable, that's okay too.
--- kernel/sched/core.c | 5 ++++- kernel/sched/ext.c | 20 ++++++++++---------- kernel/sched/ext.h | 4 ---- kernel/sched/sched.h | 2 ++ 4 files changed, 16 insertions(+), 15 deletions(-)
kernel/sched/core.c +4 -1
@@ -5907,7 +5907,10 @@ restart:for_each_active_class (class) { p = class->pick_next_task (rq);if (p) { -scx_next_task_picked (rq, p, class); +const struct sched_class *prev_class = prev->sched_class; + +if (class != prev_class && prev_class->switch_class) + prev_class->switch_class (rq, p);return p; } }
kernel/sched/ext.c +10 -10
@@ -2749,10 +2749,9 @@ preempt_reason_from_class(const struct sreturn SCX_CPU_PREEMPT_UNKNOWN ; } -void scx_next_task_picked (struct rq *rq,struct task_struct *p, -const struct sched_class *active) +static void switch_class_scx (struct rq *rq,struct task_struct *next) { -lockdep_assert_rq_held (rq); +const struct sched_class *next_class = next->sched_class;if (!scx_enabled ())return ;
@@ -2769,12 +2768,11 @@ void scx_next_task_picked(struct rq *rq,/* * The callback is conceptually meant to convey that the CPU is no -* longer under the control of SCX. Therefore, don't invoke the -* callback if the CPU is is staying on SCX, or going idle (in which -* case the SCX scheduler has actively decided not to schedule any -* tasks on the CPU). +* longer under the control of SCX. Therefore, don't invoke the callback +* if the next class is below SCX (in which case the BPF scheduler has +* actively decided not to schedule any tasks on the CPU). */ -if (likely (active >= &ext_sched_class)) +if (sched_class_above (&ext_sched_class, next_class))return ;/*
@@ -2789,8 +2787,8 @@ void scx_next_task_picked(struct rq *rq,if (!rq->scx.cpu_released) {if (SCX_HAS_OP (cpu_release)) {struct scx_cpu_release_args args = { - .reason =preempt_reason_from_class (active), - .task = p, + .reason =preempt_reason_from_class (next_class), + .task = next, }; SCX_CALL_OP(SCX_KF_CPU_RELEASE,
@@ -3496,6 +3494,8 @@ DEFINE_SCHED_CLASS(ext) = { .put_prev_task = put_prev_task_scx, .set_next_task = set_next_task_scx, + .switch_class = switch_class_scx, + #ifdef CONFIG_SMP .balance = balance_scx, .select_task_rq = select_task_rq_scx,
kernel/sched/ext.h +0 -4
@@ -33,8 +33,6 @@ static inline bool task_on_scx(const strreturn scx_enabled () && p->sched_class == &ext_sched_class; } -void scx_next_task_picked (struct rq *rq,struct task_struct *p, -const struct sched_class *active);void scx_tick (struct rq *rq);void init_scx_entity (struct sched_ext_entity *scx);void scx_pre_fork (struct task_struct *p);
@@ -82,8 +80,6 @@ bool scx_prio_less(const struct task_str#define scx_enabled () false#define scx_switched_all () false -static inline void scx_next_task_picked (struct rq *rq,struct task_struct *p, -const struct sched_class *active) {}static inline void scx_tick (struct rq *rq) {}static inline void scx_pre_fork (struct task_struct *p) {}static inline int scx_fork (struct task_struct *p) {return 0 ; }
kernel/sched/sched.h +2 -0
@@ -2355,6 +2355,8 @@ struct sched_class {void (*put_prev_task)(struct rq *rq,struct task_struct *p);void (*set_next_task)(struct rq *rq,struct task_struct *p,bool first); +void (*switch_class)(struct rq *rq,struct task_struct *next); +#ifdef CONFIG_SMP int (*balance)(struct rq *rq,struct task_struct *prev,struct rq_flags *rf);int (*select_task_rq)(struct task_struct *p,int task_cpu,int flags);
Re: [PATCH sched_ext/for-6.11] sched, sched_ext: Replace scx_next_task_picked() with sched_class->switch_class()
Quoted 6 lines · Tejun Heo
The changes are straightforward and the code looks better afterwards. However, when !CONFIG_SCHED_CLASS_EXT, this just ends up adding an unused hook which is unlikely to be useful to other
sched_classes. We can #ifdef the op withCONFIG_SCHED_CLASS_EXTbut then I'm not sure the code necessarily looks better afterwards.
So honestly, if people _really_ care about performance here, then I think that in the long run the right thing to do is
- expose all the DEFINE_SCHED_CLASS() definitions in a header file
- rename for_each_class() to FOR_EACH_CLASS() and make it unroll the
whole damn loop statically
which would turn the indirect branches into actual direct branches, and would statically just remove any "if (!class->zyz)" conditionals.
Pretty? No. But it probably wouldn't be hugely ugly either, and honestly, looking at the existing for_each_class() uses (and the one single "for_class_range()" one), they are so small and the number of classes is so small that unrolling the loop entirely doesn't sound bad.
It wouldn't help deal with *this* case (since it's a "call variable class"), but considering that the current __pick_next_task()
(a) special-cases one class as-is
(b) does a "for_each_class()" and calls an indirect call for each
when that doesn't trigger
I would claim that people don't care enough about this that one test for a NULL 'switch_class' function would be worth worrying about.
Btw, indirect calls are now expensive enough that when you have only a handful of choices, instead of a variable
class->some_callback(some_arguments);
you might literally be better off with a macro that does
#define call_sched_fn(class, name, arg...) switch (class) { \
case &fair_name_class: fair_name_class.name(arg); break; \
... unroll them all here..which then just generates a (very small) tree of if-statements.
Again, this is entirely too ugly to do unless people *really* care. But for situations where you have a small handful of cases known at compile-time, it's not out of the question, and it probably does generate better code.
NOTE NOTE NOTE! This is a comp[letely independent aside, and has nothing to do with sched_ext except for the very obvious indirect fact that sched_ext would be one of the classes in this kind of code.
And yes, I suspect it is too ugly to actually do this.
Linus
Tejun Heo
maintainerRe: [PATCH sched_ext/for-6.11] sched, sched_ext: Replace scx_next_task_picked() with sched_class->switch_class()
Hello,
On Thu, Jun 20, 2024 at 03:42:48PM -0700, Linus Torvalds wrote:
...
Quoted 11 lines
Btw, indirect calls are now expensive enough that when you have only a handful of choices, instead of a variable
class->some_callback(some_arguments);you might literally be better off with a macro that does
#define call_sched_fn(class, name, arg...) switch (class) { \ case &fair_name_class: fair_name_class.name(arg); break; \ ... unroll them all here..
Quoted 6 lines
which then just generates a (very small) tree of if-statements.
Again, this is entirely too ugly to do unless people *really* care. But for situations where you have a small handful of cases known at compile-time, it's not out of the question, and it probably does generate better code.
I'll update the patch description to point to the previous message just in case and apply it to sched_ext/for-6.11.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH sched_ext/for-6.11] sched, sched_ext: Replace scx_next_task_picked() with sched_class->switch_class()
Quoted 17 lines · Tejun Heo
Hello,
On Thu, Jun 20, 2024 at 03:42:48PM -0700, Linus Torvalds wrote:
...Quoted 13 lines
Btw, indirect calls are now expensive enough that when you have only a handful of choices, instead of a variable
class->some_callback(some_arguments);you might literally be better off with a macro that does
#define call_sched_fn(class, name, arg...) switch (class) { \ case &fair_name_class: fair_name_class.name(arg); break; \ ... unroll them all here..which then just generates a (very small) tree of if-statements.
Quoted 7 lines
Quoted 4 lines
Again, this is entirely too ugly to do unless people *really* care. But for situations where you have a small handful of cases known at compile-time, it's not out of the question, and it probably does generate better code.
I'll update the patch description to point to the previous message just in case and apply it to sched_ext/for-6.11.
Can you please back merge and keep it a sane series? I'm going to have to review it (even though I still very strongly disagree with the whole thing) and there really is nothing worse than a series that introduces things only to remove/change them again later.
Tejun Heo
maintainerRe: [PATCH sched_ext/for-6.11] sched, sched_ext: Replace scx_next_task_picked() with sched_class->switch_class()
Hello, Peter.
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines
I'll update the patch description to point to the previous message just in case and apply it to sched_ext/for-6.11.
Can you please back merge and keep it a sane series? I'm going to have to review it (even though I still very strongly disagree with the whole thing) and there really is nothing worse than a series that introduces things only to remove/change them again later.
I started the sched_ext/for-6.11 branch last week with the v7 patchset:
http://lkml.kernel.org/r/[email protected]
and would much prefer to just run it as a normal branch. No matter what form the patchset gets applied in, we'll have constant flow of changes, fixes, reverts and improvements no matter what, so I'd much prefer to avoid another round of reshuffling and reposting.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerRe: [PATCH sched_ext/for-6.11] sched, sched_ext: Replace scx_next_task_picked() with sched_class->switch_class()
In reply to Linus TorvaldsQuoted 18 lines · Linus Torvalds
Btw, indirect calls are now expensive enough that when you have only a handful of choices, instead of a variable
class->some_callback(some_arguments);you might literally be better off with a macro that does
#define call_sched_fn(class, name, arg...) switch (class) { \ case &fair_name_class: fair_name_class.name(arg); break; \ ... unroll them all here..which then just generates a (very small) tree of if-statements.
Again, this is entirely too ugly to do unless people *really* care. But for situations where you have a small handful of cases known at compile-time, it's not out of the question, and it probably does generate better code.
Quoted 5 lines
NOTE NOTE NOTE! This is a comp[letely independent aside, and has nothing to do with
sched_extexcept for the very obvious indirect fact thatsched_extwould be one of the classes in this kind of code.And yes, I suspect it is too ugly to actually do this.
Very early on in the retpoline mess I briefly considered doing this, but I decided against doing the ugly until someone came with numbers bad enough to warrant them.
We're now many years later and I'm very glad we never really *had* to go down that route.
PATCH 1/1 sched, sched_ext: Replace scx_next_task_picked() with sched_class->switch_class()
In reply to Tejun HeoFrom b999e365c2982dbd50f01fec520215d3c61ea2aa Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Thu, 20 Jun 2024 12:15:02 -1000
scx_next_task_picked() is used by sched_ext to notify the BPF scheduler when a CPU is taken away by a task dispatched from a higher priority sched_class so that the BPF scheduler can, e.g., punt the task[s] which was running or were waiting for the CPU to other CPUs.
Replace the sched_ext specific hook scx_next_task_picked() with a new sched_class operation switch_class().
The changes are straightforward and the code looks better afterwards. However, when !CONFIG_SCHED_CLASS_EXT, this ends up adding an unused hook which is unlikely to be useful to other sched_classes. For further discussion on this subject, please refer to the following:
http://lkml.kernel.org/r/CAHk-=wjFPLqo7AXu8maAGEGnOy6reUg-F4zzFhVB0Kyu22h7pw@mail.gmail.com
---
Hello,
Applied to sched_ext/for-6.11. The only difference from the previous posting is the description.
Thanks.
kernel/sched/core.c | 5 ++++- kernel/sched/ext.c | 20 ++++++++++---------- kernel/sched/ext.h | 4 ---- kernel/sched/sched.h | 2 ++ 4 files changed, 16 insertions(+), 15 deletions(-)
kernel/sched/core.c +4 -1
@@ -5907,7 +5907,10 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)for_each_active_class (class) { p = class->pick_next_task (rq);if (p) { -scx_next_task_picked (rq, p, class); +const struct sched_class *prev_class = prev->sched_class; + +if (class != prev_class && prev_class->switch_class) + prev_class->switch_class (rq, p);return p; } }
kernel/sched/ext.c +10 -10
@@ -2749,10 +2749,9 @@ preempt_reason_from_class(const struct sched_class *class)return SCX_CPU_PREEMPT_UNKNOWN ; } -void scx_next_task_picked (struct rq *rq,struct task_struct *p, -const struct sched_class *active) +static void switch_class_scx (struct rq *rq,struct task_struct *next) { -lockdep_assert_rq_held (rq); +const struct sched_class *next_class = next->sched_class;if (!scx_enabled ())return ;
@@ -2769,12 +2768,11 @@ void scx_next_task_picked(struct rq *rq, struct task_struct *p,/* * The callback is conceptually meant to convey that the CPU is no -* longer under the control of SCX. Therefore, don't invoke the -* callback if the CPU is is staying on SCX, or going idle (in which -* case the SCX scheduler has actively decided not to schedule any -* tasks on the CPU). +* longer under the control of SCX. Therefore, don't invoke the callback +* if the next class is below SCX (in which case the BPF scheduler has +* actively decided not to schedule any tasks on the CPU). */ -if (likely (active >= &ext_sched_class)) +if (sched_class_above (&ext_sched_class, next_class))return ;/*
@@ -2789,8 +2787,8 @@ void scx_next_task_picked(struct rq *rq, struct task_struct *p,if (!rq->scx.cpu_released) {if (SCX_HAS_OP (cpu_release)) {struct scx_cpu_release_args args = { - .reason =preempt_reason_from_class (active), - .task = p, + .reason =preempt_reason_from_class (next_class), + .task = next, }; SCX_CALL_OP(SCX_KF_CPU_RELEASE,
@@ -3496,6 +3494,8 @@ DEFINE_SCHED_CLASS(ext) = { .put_prev_task = put_prev_task_scx, .set_next_task = set_next_task_scx, + .switch_class = switch_class_scx, + #ifdef CONFIG_SMP .balance = balance_scx, .select_task_rq = select_task_rq_scx,
kernel/sched/ext.h +0 -4
@@ -33,8 +33,6 @@ static inline bool task_on_scx(const struct task_struct *p)return scx_enabled () && p->sched_class == &ext_sched_class; } -void scx_next_task_picked (struct rq *rq,struct task_struct *p, -const struct sched_class *active);void scx_tick (struct rq *rq);void init_scx_entity (struct sched_ext_entity *scx);void scx_pre_fork (struct task_struct *p);
@@ -82,8 +80,6 @@ bool scx_prio_less(const struct task_struct *a, const struct task_struct *b,#define scx_enabled () false#define scx_switched_all () false -static inline void scx_next_task_picked (struct rq *rq,struct task_struct *p, -const struct sched_class *active) {}static inline void scx_tick (struct rq *rq) {}static inline void scx_pre_fork (struct task_struct *p) {}static inline int scx_fork (struct task_struct *p) {return 0 ; }
kernel/sched/sched.h +2 -0
@@ -2355,6 +2355,8 @@ struct sched_class {void (*put_prev_task)(struct rq *rq,struct task_struct *p);void (*set_next_task)(struct rq *rq,struct task_struct *p,bool first); +void (*switch_class)(struct rq *rq,struct task_struct *next); +#ifdef CONFIG_SMP int (*balance)(struct rq *rq,struct task_struct *prev,struct rq_flags *rf);int (*select_task_rq)(struct task_struct *p,int task_cpu,int flags);
Signature · 1 line
2.45.2
Peter Zijlstra
maintainerRe: [PATCH sched_ext/for-6.11] sched, sched_ext: Replace scx_next_task_picked() with sched_class->switch_class()
In reply to Tejun HeoQuoted 33 lines · Tejun Heo
scx_next_task_picked()is used bysched_extto notify the BPF scheduler when a CPU is taken away by a task dispatched from a higher prioritysched_classso that the BPF scheduler can, e.g., punt the task[s] which was running or were waiting for the CPU to other CPUs.Replace the
sched_extspecific hookscx_next_task_picked()with a newsched_classoperationswitch_class().The changes are straightforward and the code looks better afterwards. However, when !CONFIG_SCHED_CLASS_EXT, this just ends up adding an unused hook which is unlikely to be useful to other
sched_classes. We can #ifdef the op withCONFIG_SCHED_CLASS_EXTbut then I'm not sure the code necessarily looks better afterwards.Please let me know the preference. If adding #ifdef's is preferable, that's okay too.
Signed-off-by Tejun Heo <[email protected]>Suggested-by Linus Torvalds <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>--- kernel/sched/core.c | 5 ++++- kernel/sched/ext.c | 20 ++++++++++---------- kernel/sched/ext.h | 4 ---- kernel/sched/sched.h | 2 ++ 4 files changed, 16 insertions(+), 15 deletions(-)kernel/sched/core.c +0
-0@@ -5907,7 +5907,10 @@ restart:for_each_active_class (class) {
kernel/sched/core.c restart: · line 5916
Quoted 10 lines
kernel/sched/core.c +4
-1p = class->pick_next_task (rq);if (p) { -scx_next_task_picked (rq, p, class); +const struct sched_class *prev_class = prev->sched_class; + +if (class != prev_class && prev_class->switch_class) + prev_class->switch_class (rq, p);return p; } }
I would much rather see sched_class::pick_next_task() get an extra argument so that the BPF thing can do what it needs in there and we can avoid this extra code here.
Tejun Heo
maintainerRe: [PATCH sched_ext/for-6.11] sched, sched_ext: Replace scx_next_task_picked() with sched_class->switch_class()
Hello, Peter.
Quoted 5 lines · Peter Zijlstra
Quoted 5 lines
kernel/sched/core.c +0
-1@@ -5907,7 +5907,10 @@ restart:for_each_active_class (class) { p = class->pick_next_task (rq);if (p) { -scx_next_task_picked (rq, p, class);
Quoted 8 lines
kernel/sched/core.c restart: · line 5913
Quoted 4 lines
kernel/sched/core.c +4
-0+const struct sched_class *prev_class = prev->sched_class; + +if (class != prev_class && prev_class->switch_class) + prev_class->switch_class (rq, p);I would much rather see
sched_class::pick_next_task()get an extra argument so that the BPF thing can do what it needs in there and we can avoid this extra code here.
Hmm... but here, the previous class's ->pick_next_task() might not be called at all, so I'm not sure how that'd work. For context, sched_ext is using this to tell the BPF scheduler that it lost a CPU to a higher priority class (be that RT or CFS) os that the BPF scheduler can respond if necessary (e.g. punting tasks that were queued on that CPU somewhere else and so on).
Imagine a case where a sched_ext task was running but then a RT task wakes up on the CPU. We'd enter the scheduling path, RT's pick_next_task() would return the new RT task to run. We now need to tell the BPF scheduler that we lost the CPU to the RT task but haven't called its pick_next_task() yet.
Thanks.
Signature · 1 line
tejun