sched_ext: Add cgroup support
Subsystems (4)
Tejun Heo
maintainerPATCH 1/1 sched_ext: Add cgroup support
This patchset is the cgroup integration part of the sched_ext v6 patchset:
http://lkml.kernel.org/r/[email protected]
This patchset is on top of sched_ext/for-6.12 (2c390dda9e03 ("sched_ext:
Make task_can_run_on_remote_rq() use common task_allowed_on_cpu()")) and contains the following patches:
0001-cgroup-Implement-cgroup_show_cftypes.patch 0002-sched-Expose-css_tg.patch 0003-sched-Enumerate-CPU-cgroup-file-types.patch 0004-sched-Make-cpu_shares_read_u64-use-tg_weight.patch 0005-sched-Introduce-CONFIG_GROUP_SCHED_WEIGHT.patch 0006-sched_ext-Add-cgroup-support.patch 0007-sched_ext-Add-a-cgroup-scheduler-which-uses-flattene.patch
0001 implements cgroup_show_cftype() so that SCX schedulers can only show cgroup knobs that it implements.
0002-0005 are sched core preparations. Should be pretty straightforward. No functional changes.
0006 adds cgroup support to sched_ext.
0007 implements an example scheduler which uses the cgroup support.
The patchset is also available in the following git branch:
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-cgroup
diffstat follows. Thanks.
include/linux/cgroup-defs.h | 8 include/linux/cgroup.h | 1 include/linux/sched/ext.h | 3 init/Kconfig | 10 kernel/cgroup/cgroup.c | 97 ++++ kernel/sched/core.c | 115 +++-- kernel/sched/ext.c | 522 ++++++++++++++++++++++++- kernel/sched/ext.h | 20 kernel/sched/sched.h | 35 + tools/sched_ext/Makefile | 2 tools/sched_ext/README.md | 12 tools/sched_ext/include/scx/common.bpf.h | 1 tools/sched_ext/scx_flatcg.bpf.c | 949 ++++++++++++++++++++++++++++++++++++++++++++++ tools/sched_ext/scx_flatcg.c | 233 +++++++++++ tools/sched_ext/scx_flatcg.h | 51 ++ tools/testing/selftests/sched_ext/maximal.bpf.c | 32 + 16 files changed, 2035 insertions(+), 56 deletions(-)
--
tejun
Tejun Heo
maintainerPATCH 1/7 cgroup: Implement cgroup_show_cftypes()
Implement cgroup_show_cftypes() which shows and hides all cgroup files associated with the specified set of cgroup file types. CFTYPE_HIDDEN flag is added so that files can be created hidden from the get-go.
cgroup_show_cftypes() can be used whether the cftypes are added or not. It also combines with cgroup_show_file() so that a given file is visible iff both its cftype and cfile are visible.
This will be used by a new sched_class to selectively show and hide CPU controller interface files depending on whether they're supported.
--- include/linux/cgroup-defs.h | 8 +++ include/linux/cgroup.h | 1 + kernel/cgroup/cgroup.c | 97 ++++++++++++++++++++++++++++++++++--- 3 files changed, 99 insertions(+), 7 deletions(-)
include/linux/cgroup-defs.h +8 -0
@@ -137,12 +137,18 @@ enum {CFTYPE_WORLD_WRITABLE = (1 <<4 ),/* (DON'T USE FOR NEW FILES) S_IWUGO */ CFTYPE_DEBUG = (1 <<5 ),/* create when cgroup_debug */ +CFTYPE_HIDDEN = (1 <<6 ),/* file type hidden, see cgroup_show_cftypes() */ +/* internal flags, do not use outside cgroup core proper */ __CFTYPE_ONLY_ON_DFL = (1 <<16 ),/* only on default hierarchy */ __CFTYPE_NOT_ON_DFL = (1 <<17 ),/* not on default hierarchy */ __CFTYPE_ADDED = (1 << 18), }; +enum cfile_flags { +CFILE_HIDDEN = (1 <<0 ),/* file instance hidden */ +}; +/* * cgroup_file is the handle for a file instance created in a cgroup which * is used, for example, to generate file changed notifications. This can
@@ -150,7 +156,9 @@ enum {*/ struct cgroup_file {/* do not access any fields from outside cgroup core */ +struct cftype *cft;struct kernfs_node *kn; +unsigned int flags;unsigned long notified_at;struct timer_list notify_timer; };
include/linux/cgroup.h +1 -0
@@ -113,6 +113,7 @@ int cgroup_transfer_tasks(struct cgroup *to, struct cgroup *from);int cgroup_add_dfl_cftypes (struct cgroup_subsys *ss,struct cftype *cfts);int cgroup_add_legacy_cftypes (struct cgroup_subsys *ss,struct cftype *cfts);int cgroup_rm_cftypes (struct cftype *cfts); +void cgroup_show_cftype (struct cftype *cft,bool show);void cgroup_file_notify (struct cgroup_file *cfile);void cgroup_file_show (struct cgroup_file *cfile,bool show);
kernel/cgroup/cgroup.c +90 -7
127 lines not on this page · raw message
Signature · 1 line
2.46.0
Tejun Heo
maintainerPATCH 2/7 sched: Expose css_tg()
In reply to Tejun HeoA new BPF extensible sched_class will use css_tg() in the init and exit paths to visit all task_groups by walking cgroups.
v4: __setscheduler_prio() is already exposed. Dropped from this patch.
v3: Dropped SCHED_CHANGE_BLOCK() as upstream is adding more generic cleanup
mechanism.
v2: Expose SCHED_CHANGE_BLOCK() too and update the description.
--- kernel/sched/core.c | 5 ----- kernel/sched/sched.h | 5 +++++ 2 files changed, 5 insertions(+), 5 deletions(-)
kernel/sched/core.c +0 -5
@@ -8918,11 +8918,6 @@ void sched_move_task(struct task_struct *tsk) } } -static inline struct task_group *css_tg (struct cgroup_subsys_state *css) -{ - return css ? container_of(css, struct task_group, css) : NULL; -} -static struct cgroup_subsys_state *cpu_cgroup_css_alloc (struct cgroup_subsys_state *parent_css) {
kernel/sched/sched.h +5 -0
@@ -514,6 +514,11 @@ static inline int walk_tg_tree(tg_visitor down, tg_visitor up, void *data)return walk_tg_tree_from (&root_task_group, down, up, data); } +static inline struct task_group *css_tg (struct cgroup_subsys_state *css) +{ + return css ? container_of(css, struct task_group, css) : NULL; +} +extern int tg_nop (struct task_group *tg,void *data);#ifdef CONFIG_FAIR_GROUP_SCHED
Signature · 1 line
2.46.0
Tejun Heo
maintainerPATCH 3/7 sched: Enumerate CPU cgroup file types
In reply to Tejun HeoRename cpu[_legacy]_files to cpu[_legacy]_cftypes for clarity and add cpu_cftype_id which enumerates every cgroup2 interface file type. This doesn't make any functional difference now. The enums will be used to access specific cftypes by a new BPF extensible sched_class to selectively show and hide CPU controller interface files depending on the capability of the currently loaded BPF scheduler progs.
--- kernel/sched/core.c | 22 +++++++++++----------- kernel/sched/sched.h | 21 +++++++++++++++++++++ 2 files changed, 32 insertions(+), 11 deletions(-)
kernel/sched/core.c +11 -11
@@ -9572,7 +9572,7 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css, }#endif -static struct cftype cpu_legacy_files[] = { +static struct cftype cpu_legacy_cftypes[] = {#ifdef CONFIG_FAIR_GROUP_SCHED { .name = "shares",
@@ -9801,21 +9801,21 @@ static ssize_t cpu_max_write(struct kernfs_open_file *of, }#endif -static struct cftype cpu_files[] = { +struct cftype cpu_cftypes[CPU_CFTYPE_CNT +1 ] = { #ifdef CONFIG_FAIR_GROUP_SCHED - { + [CPU_CFTYPE_WEIGHT] = { .name ="weight" , .flags =CFTYPE_NOT_ON_ROOT , .read_u64 = cpu_weight_read_u64, .write_u64 = cpu_weight_write_u64, }, - { + [CPU_CFTYPE_WEIGHT_NICE ] = { .name ="weight.nice" , .flags =CFTYPE_NOT_ON_ROOT , .read_s64 = cpu_weight_nice_read_s64, .write_s64 = cpu_weight_nice_write_s64, }, - { + [CPU_CFTYPE_IDLE ] = { .name ="idle" , .flags =CFTYPE_NOT_ON_ROOT , .read_s64 = cpu_idle_read_s64,
@@ -9823,13 +9823,13 @@ static struct cftype cpu_files[] = { },#endif #ifdef CONFIG_CFS_BANDWIDTH - { + [CPU_CFTYPE_MAX] = { .name = "max", .flags = CFTYPE_NOT_ON_ROOT, .seq_show = cpu_max_show, .write = cpu_max_write, }, - { + [CPU_CFTYPE_MAX_BURST] = { .name = "max.burst", .flags = CFTYPE_NOT_ON_ROOT, .read_u64 = cpu_cfs_burst_read_u64,
@@ -9837,13 +9837,13 @@ static struct cftype cpu_files[] = { },#endif #ifdef CONFIG_UCLAMP_TASK_GROUP - { + [CPU_CFTYPE_UCLAMP_MIN] = { .name = "uclamp.min", .flags = CFTYPE_NOT_ON_ROOT, .seq_show = cpu_uclamp_min_show, .write = cpu_uclamp_min_write, }, - { + [CPU_CFTYPE_UCLAMP_MAX] = { .name = "uclamp.max", .flags = CFTYPE_NOT_ON_ROOT, .seq_show = cpu_uclamp_max_show,
@@ -9864,8 +9864,8 @@ struct cgroup_subsys cpu_cgrp_subsys = { .can_attach = cpu_cgroup_can_attach, #endif .attach = cpu_cgroup_attach, - .legacy_cftypes = cpu_legacy_files, - .dfl_cftypes = cpu_files, + .legacy_cftypes = cpu_legacy_cftypes, + .dfl_cftypes = cpu_cftypes, .early_init = true, .threaded = true, };
kernel/sched/sched.h +21 -0
@@ -3785,4 +3785,25 @@ void sched_enq_and_set_task(struct sched_enq_and_set_ctx *ctx);#include "ext.h" +#ifdef CONFIG_CGROUP_SCHED +enum cpu_cftype_id { +#ifdef CONFIG_FAIR_GROUP_SCHED +CPU_CFTYPE_WEIGHT , +CPU_CFTYPE_WEIGHT_NICE , +CPU_CFTYPE_IDLE , +#endif +#ifdef CONFIG_CFS_BANDWIDTH +CPU_CFTYPE_MAX , +CPU_CFTYPE_MAX_BURST , +#endif +#ifdef CONFIG_UCLAMP_TASK_GROUP +CPU_CFTYPE_UCLAMP_MIN , +CPU_CFTYPE_UCLAMP_MAX , +#endif +CPU_CFTYPE_CNT , +}; + +extern struct cftype cpu_cftypes[CPU_CFTYPE_CNT +1 ]; +#endif /* CONFIG_CGROUP_SCHED */ +#endif /* _KERNEL_SCHED_SCHED_H */
Signature · 1 line
2.46.0
Tejun Heo
maintainerPATCH 4/7 sched: Make cpu_shares_read_u64() use tg_weight()
In reply to Tejun HeoMove tg_weight() upward and make cpu_shares_read_u64() use it too. This makes the weight retrieval shared between cgroup v1 and v2 paths and will be used to implement cgroup support for sched_ext.
No functional changes.
--- kernel/sched/core.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
kernel/sched/core.c +6 -8
@@ -9170,6 +9170,11 @@ static int cpu_uclamp_max_show(struct seq_file *sf, void *v)#endif /* CONFIG_UCLAMP_TASK_GROUP */ #ifdef CONFIG_FAIR_GROUP_SCHED +static unsigned long tg_weight (struct task_group *tg) +{ +return scale_load_down (tg->shares); +} +static int cpu_shares_write_u64 (struct cgroup_subsys_state *css,struct cftype *cftype,u64 shareval) {
@@ -9181,9 +9186,7 @@ static int cpu_shares_write_u64(struct cgroup_subsys_state *css,static u64 cpu_shares_read_u64 (struct cgroup_subsys_state *css,struct cftype *cft) { -struct task_group *tg =css_tg (css); - -return (u64 )scale_load_down (tg->shares); +return tg_weight (css_tg (css)); }#ifdef CONFIG_CFS_BANDWIDTH
@@ -9685,11 +9688,6 @@ static int cpu_local_stat_show(struct seq_file *sf,#ifdef CONFIG_FAIR_GROUP_SCHED -static unsigned long tg_weight (struct task_group *tg) -{ -return scale_load_down (tg->shares); -} -static u64 cpu_weight_read_u64 (struct cgroup_subsys_state *css,struct cftype *cft) {
Signature · 1 line
2.46.0
Tejun Heo
maintainerPATCH 5/7 sched: Introduce CONFIG_GROUP_SCHED_WEIGHT
In reply to Tejun Heosched_ext will soon add cgroup cpu.weigh support. The cgroup interface code is currently gated behind CONFIG_FAIR_GROUP_SCHED. As the fair class and/or SCX may implement the feature, put the interface code behind the new CONFIG_CGROUP_SCHED_WEIGHT which is selected by CONFIG_FAIR_GROUP_SCHED. This allows either sched class to enable the itnerface code without ading more complex CONFIG tests.
When !CONFIG_FAIR_GROUP_SCHED, a dummy version of sched_group_set_shares() is added to support later CONFIG_CGROUP_SCHED_WEIGHT && !CONFIG_FAIR_GROUP_SCHED builds.
No functional changes.
--- init/Kconfig | 4 ++++ kernel/sched/core.c | 16 ++++++++++------ kernel/sched/sched.h | 6 ++++-- 3 files changed, 18 insertions(+), 8 deletions(-)
init/Kconfig +4 -0
@@ -1024,9 +1024,13 @@ menuconfig CGROUP_SCHED tasks. if CGROUP_SCHED +config GROUP_SCHED_WEIGHT + def_bool n + config FAIR_GROUP_SCHED bool "Group scheduling for SCHED_OTHER" depends on CGROUP_SCHED + select GROUP_SCHED_WEIGHT default CGROUP_SCHED config CFS_BANDWIDTH
kernel/sched/core.c +10 -6
@@ -9169,7 +9169,7 @@ static int cpu_uclamp_max_show(struct seq_file *sf, void *v) }#endif /* CONFIG_UCLAMP_TASK_GROUP */ -#ifdef CONFIG_FAIR_GROUP_SCHED +#ifdef CONFIG_GROUP_SCHED_WEIGHT static unsigned long tg_weight (struct task_group *tg) {return scale_load_down (tg->shares);
@@ -9188,6 +9188,7 @@ static u64 cpu_shares_read_u64(struct cgroup_subsys_state *css, {return tg_weight (css_tg (css)); } +#endif /* CONFIG_GROUP_SCHED_WEIGHT */ #ifdef CONFIG_CFS_BANDWIDTH static DEFINE_MUTEX (cfs_constraints_mutex);
@@ -9533,7 +9534,6 @@ static int cpu_cfs_local_stat_show(struct seq_file *sf, void *v)return 0 ; }#endif /* CONFIG_CFS_BANDWIDTH */ -#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED static int cpu_rt_runtime_write(struct cgroup_subsys_state *css,
@@ -9576,12 +9576,14 @@ static int cpu_idle_write_s64(struct cgroup_subsys_state *css,#endif static struct cftype cpu_legacy_cftypes[] = { -#ifdef CONFIG_FAIR_GROUP_SCHED +#ifdef CONFIG_GROUP_SCHED_WEIGHT { .name = "shares", .read_u64 = cpu_shares_read_u64, .write_u64 = cpu_shares_write_u64, }, +#endif +#ifdef CONFIG_FAIR_GROUP_SCHED { .name = "idle", .read_s64 = cpu_idle_read_s64,
@@ -9686,7 +9688,7 @@ static int cpu_local_stat_show(struct seq_file *sf,return 0 ; } -#ifdef CONFIG_FAIR_GROUP_SCHED +#ifdef CONFIG_GROUP_SCHED_WEIGHT static u64 cpu_weight_read_u64 (struct cgroup_subsys_state *css,struct cftype *cft)
@@ -9740,7 +9742,7 @@ static int cpu_weight_nice_write_s64(struct cgroup_subsys_state *css, return sched_group_set_shares(css_tg(css), scale_load(weight)); } -#endif +#endif /* CONFIG_GROUP_SCHED_WEIGHT */ static void __maybe_unused cpu_period_quota_print (struct seq_file *sf,long period,long quota)
@@ -9800,7 +9802,7 @@ static ssize_t cpu_max_write(struct kernfs_open_file *of,#endif struct cftype cpu_cftypes[CPU_CFTYPE_CNT +1 ] = { -#ifdef CONFIG_FAIR_GROUP_SCHED +#ifdef CONFIG_GROUP_SCHED_WEIGHT [CPU_CFTYPE_WEIGHT] = { .name ="weight" , .flags =CFTYPE_NOT_ON_ROOT ,
@@ -9813,6 +9815,8 @@ struct cftype cpu_cftypes[CPU_CFTYPE_CNT + 1] = { .read_s64 = cpu_weight_nice_read_s64, .write_s64 = cpu_weight_nice_write_s64, }, +#endif +#ifdef CONFIG_FAIR_GROUP_SCHED [CPU_CFTYPE_IDLE] = { .name = "idle", .flags = CFTYPE_NOT_ON_ROOT,
kernel/sched/sched.h +4 -2
@@ -483,7 +483,7 @@ struct task_group { }; -#ifdef CONFIG_FAIR_GROUP_SCHED +#ifdef CONFIG_GROUP_SCHED_WEIGHT #define ROOT_TASK_GROUP_LOAD NICE_0_LOAD/*
@@ -575,6 +575,8 @@ extern void set_task_rq_fair(struct sched_entity *se,static inline void set_task_rq_fair (struct sched_entity *se,struct cfs_rq *prev,struct cfs_rq *next) { }#endif /* CONFIG_SMP */ +#else /* !CONFIG_FAIR_GROUP_SCHED */ +static inline int sched_group_set_shares (struct task_group *tg,unsigned long shares) {return 0 ; }#endif /* CONFIG_FAIR_GROUP_SCHED */ #else /* CONFIG_CGROUP_SCHED */
@@ -3787,7 +3789,7 @@ void sched_enq_and_set_task(struct sched_enq_and_set_ctx *ctx); #ifdef CONFIG_CGROUP_SCHED enum cpu_cftype_id { -#ifdef CONFIG_FAIR_GROUP_SCHED +#ifdef CONFIG_GROUP_SCHED_WEIGHT CPU_CFTYPE_WEIGHT, CPU_CFTYPE_WEIGHT_NICE, CPU_CFTYPE_IDLE,
Signature · 1 line
2.46.0
Tejun Heo
maintainerPATCH 6/7 sched_ext: Add cgroup support
In reply to Tejun HeoAdd sched_ext_ops operations to init/exit cgroups, and track task migrations and config changes. Because different BPF schedulers may implement different subsets of CPU control features, allow BPF schedulers to pick which cgroup interface files to enable using SCX_OPS_CGROUP_KNOB_* flags. For now, only the weight knobs are supported but adding more should be straightforward.
While a BPF scheduler is being enabled and disabled, relevant cgroup operations are locked out using scx_cgroup_rwsem. This avoids situations like task prep taking place while the task is being moved across cgroups, making things easier for BPF schedulers.
v6: - Updated to reflect the removal of SCX_KF_SLEEPABLE.
- Updated to use CONFIG_GROUP_SCHED_WEIGHT and fixes for !CONFIG_FAIR_GROUP_SCHED && CONFIG_EXT_GROUP_SCHED.
v5: - Flipped the locking order between scx_cgroup_rwsem and
cpus_read_lock() to avoid locking order conflict w/ cpuset. Better documentation around locking.
- sched_move_task() takes an early exit if the source and destination are identical. This triggered the warning in scx_cgroup_can_attach() as it left p->scx.cgrp_moving_from uncleared. Updated the cgroup migration path so that ops.cgroup_prep_move() is skipped for identity migrations so that its invocations always match ops.cgroup_move() one-to-one.
v4: - Example schedulers moved into their own patches.
- Fix build failure when !CONFIG_CGROUP_SCHED, reported by Andrea Righi.
v3: - Make scx_example_pair switch all tasks by default.
- Convert to BPF inline iterators.
- scx_bpf_task_cgroup() is added to determine the current cgroup from CPU controller's POV. This allows BPF schedulers to accurately track CPU cgroup membership.
- scx_example_flatcg added. This demonstrates flattened hierarchy implementation of CPU cgroup control and shows significant performance improvement when cgroups which are nested multiple levels are under competition.
v2: - Build fixes for different CONFIG combinations.
--- include/linux/sched/ext.h | 3 + init/Kconfig | 6 + kernel/sched/core.c | 60 +- kernel/sched/ext.c | 522 +++++++++++++++++- kernel/sched/ext.h | 20 + kernel/sched/sched.h | 5 + tools/sched_ext/include/scx/common.bpf.h | 1 + .../testing/selftests/sched_ext/maximal.bpf.c | 32 ++ 8 files changed, 631 insertions(+), 18 deletions(-)
include/linux/sched/ext.h +3 -0
@@ -189,6 +189,9 @@ struct sched_ext_entity {bool disallow;/* reject switching into SCX */ /* cold fields */ +#ifdef CONFIG_EXT_GROUP_SCHED +struct cgroup *cgrp_moving_from; +#endif /* must be the last field, see init_scx_entity() */ struct list_head tasks_node; };
init/Kconfig +6 -0
@@ -1055,6 +1055,12 @@ config RT_GROUP_SCHED realtime bandwidth for them. See Documentation/scheduler/sched-rt-group.rst for more information. +config EXT_GROUP_SCHED + bool + depends on SCHED_CLASS_EXT && CGROUP_SCHED + select GROUP_SCHED_WEIGHT + default y + endif #CGROUP_SCHED config SCHED_MM_CID
kernel/sched/core.c +51 -9
161 lines not on this page · raw message
kernel/sched/ext.c +513 -9
691 lines not on this page · raw message
kernel/sched/ext.h +20 -0
@@ -67,3 +67,23 @@ static inline void scx_update_idle(struct rq *rq, bool idle)#else static inline void scx_update_idle (struct rq *rq,bool idle) {}#endif + +#ifdef CONFIG_CGROUP_SCHED +#ifdef CONFIG_EXT_GROUP_SCHED +int scx_tg_online (struct task_group *tg); +void scx_tg_offline (struct task_group *tg); +int scx_cgroup_can_attach (struct cgroup_taskset *tset); +void scx_move_task (struct task_struct *p); +void scx_cgroup_finish_attach (void ); +void scx_cgroup_cancel_attach (struct cgroup_taskset *tset); +void scx_group_set_weight (struct task_group *tg,unsigned long cgrp_weight); +#else /* CONFIG_EXT_GROUP_SCHED */ +static inline int scx_tg_online (struct task_group *tg) {return 0 ; } +static inline void scx_tg_offline (struct task_group *tg) {} +static inline int scx_cgroup_can_attach (struct cgroup_taskset *tset) {return 0 ; } +static inline void scx_move_task (struct task_struct *p) {} +static inline void scx_cgroup_finish_attach (void ) {} +static inline void scx_cgroup_cancel_attach (struct cgroup_taskset *tset) {} +static inline void scx_group_set_weight (struct task_group *tg,unsigned long cgrp_weight) {} +#endif /* CONFIG_EXT_GROUP_SCHED */ +#endif /* CONFIG_CGROUP_SCHED */
kernel/sched/sched.h +5 -0
@@ -459,6 +459,11 @@ struct task_group {struct rt_bandwidth rt_bandwidth;#endif +#ifdef CONFIG_EXT_GROUP_SCHED +u32 scx_flags;/* SCX_TG_* */ +u32 scx_weight; +#endif +struct rcu_head rcu;struct list_head list;
tools/sched_ext/include/scx/common.bpf.h +1 -0
@@ -61,6 +61,7 @@ s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; bool scx_bpf_task_running(const struct task_struct *p) __ksym; s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym;struct rq *scx_bpf_cpu_rq (s32 cpu) __ksym; +struct cgroup *scx_bpf_task_cgroup (struct task_struct *p) __ksym;static inline __attribute__((format (printf,1 ,2 )))void ___scx_bpf_bstr_format_checker (const char *fmt, ...) {}
tools/testing/selftests/sched_ext/maximal.bpf.c +32 -0
@@ -95,6 +95,32 @@ void BPF_STRUCT_OPS(maximal_exit_task, struct task_struct *p,void BPF_STRUCT_OPS (maximal_disable ,struct task_struct *p) {} +s32 BPF_STRUCT_OPS (maximal_cgroup_init ,struct cgroup *cgrp, +struct scx_cgroup_init_args *args) +{ +return 0 ; +} + +void BPF_STRUCT_OPS (maximal_cgroup_exit ,struct cgroup *cgrp) +{} + +s32 BPF_STRUCT_OPS (maximal_cgroup_prep_move ,struct task_struct *p, +struct cgroup *from,struct cgroup *to) +{ +return 0 ; +} + +void BPF_STRUCT_OPS (maximal_cgroup_move ,struct task_struct *p, +struct cgroup *from,struct cgroup *to) +{} + +void BPF_STRUCT_OPS (maximal_cgroup_cancel_move ,struct task_struct *p, +struct cgroup *from,struct cgroup *to) +{} + +void BPF_STRUCT_OPS (maximal_cgroup_set_weight ,struct cgroup *cgrp,u32 weight) +{} +s32 BPF_STRUCT_OPS_SLEEPABLE (maximal_init ) {return 0 ;
@@ -126,6 +152,12 @@ struct sched_ext_ops maximal_ops = { .enable = maximal_enable, .exit_task = maximal_exit_task, .disable = maximal_disable, + .cgroup_init = maximal_cgroup_init, + .cgroup_exit = maximal_cgroup_exit, + .cgroup_prep_move = maximal_cgroup_prep_move, + .cgroup_move = maximal_cgroup_move, + .cgroup_cancel_move = maximal_cgroup_cancel_move, + .cgroup_set_weight = maximal_cgroup_set_weight, .init = maximal_init, .exit = maximal_exit, .name = "maximal",
Signature · 1 line
2.46.0
Tejun Heo
maintainerPATCH 7/7 sched_ext: Add a cgroup scheduler which uses flattened hierarchy
In reply to Tejun HeoThis patch adds scx_flatcg example scheduler which implements hierarchical weight-based cgroup CPU control by flattening the cgroup hierarchy into a single layer by compounding the active weight share at each level.
This flattening of hierarchy can bring a substantial performance gain when the cgroup hierarchy is nested multiple levels. in a simple benchmark using wrk[8] on apache serving a CGI script calculating sha1sum of a small file, it outperforms CFS by ~3% with CPU controller disabled and by ~10% with two apache instances competing with 2:1 weight ratio nested four level deep.
However, the gain comes at the cost of not being able to properly handle thundering herd of cgroups. For example, if many cgroups which are nested behind a low priority parent cgroup wake up around the same time, they may be able to consume more CPU cycles than they are entitled to. In many use cases, this isn't a real concern especially given the performance gain. Also, there are ways to mitigate the problem further by e.g. introducing an extra scheduling layer on cgroup delegation boundaries.
v4: - Revert reference counted kptr for cgv_node as the change caused easily
reproducible stalls.
v3: - Updated to reflect the core API changes including ops.init/exit_task()
and direct dispatch from ops.select_cpu(). Fixes and improvements including additional statistics.
- Use reference counted kptr for cgv_node instead of xchg'ing against stash location.
- Dropped '-p' option.
v2: - Use SCX_BUG[_ON]() to simplify error handling.
--- tools/sched_ext/Makefile | 2 +- tools/sched_ext/README.md | 12 + tools/sched_ext/scx_flatcg.bpf.c | 949 +++++++++++++++++++++++++++++++ tools/sched_ext/scx_flatcg.c | 233 ++++++++ tools/sched_ext/scx_flatcg.h | 51 ++ 5 files changed, 1246 insertions(+), 1 deletion(-) create mode 100644 tools/sched_ext/scx_flatcg.bpf.c create mode 100644 tools/sched_ext/scx_flatcg.c create mode 100644 tools/sched_ext/scx_flatcg.h
tools/sched_ext/Makefile +1 -1
@@ -176,7 +176,7 @@ $(INCLUDE_DIR)/%.bpf.skel.h: $(SCXOBJ_DIR)/%.bpf.o $(INCLUDE_DIR)/vmlinux.h $(BP SCX_COMMON_DEPS := include/scx/common.h include/scx/user_exit_info.h | $(BINDIR) -c-sched-targets = scx_simple scx_qmap scx_central +c-sched-targets = scx_simple scx_qmap scx_central scx_flatcg $(addprefix $(BINDIR)/,$(c-sched-targets)): \ $(BINDIR)/%: \
tools/sched_ext/README.md +12 -0
@@ -192,6 +192,18 @@ where this could be particularly useful is running VMs, where running with infinite slices and no timer ticks allows the VM to avoid unnecessary expensive vmexits. +## scx_flatcg + +A flattened cgroup hierarchy scheduler. This scheduler implements hierarchical +weight-based cgroup CPU control by flattening the cgroup hierarchy into a single +layer, by compounding the active weight share at each level. The effect of this +is a much more performant CPU controller, which does not need to descend down +cgroup trees in order to properly compute a cgroup's share. + +Similar to scx_simple, in limited scenarios, this scheduler can perform +reasonably well on single socket-socket systems with a unified L3 cache and show +significantly lowered hierarchical scheduling overhead. + # Troubleshooting
tools/sched_ext/scx_flatcg.bpf.c new file +949 -0
949 lines not on this page · raw message
tools/sched_ext/scx_flatcg.c new file +233 -0
233 lines not on this page · raw message
tools/sched_ext/scx_flatcg.h new file +51 -0
51 lines not on this page · raw message
Signature · 1 line
2.46.0
Re: [PATCH 4/7] sched: Make cpu_shares_read_u64() use tg_weight()
In reply to Tejun HeoQuoted 7 lines · Tejun Heo
Move
tg_weight()upward and makecpu_shares_read_u64()use it too. This makes the weight retrieval shared between cgroup v1 and v2 paths and will be used to implement cgroup support forsched_ext.No functional changes.
Signed-off-by Tejun Heo <[email protected]>
Michal Koutný
maintainerRe: [PATCH 1/7] cgroup: Implement cgroup_show_cftypes()
In reply to Tejun Heokernel/cgroup/cgroup.c static void offline_css(struct cgroup_subsys_state *css) · line 5583
Quoted 4 lines · Tejun Heo
kernel/cgroup/cgroup.c +4
-0+void cgroup_show_cftype (struct cftype *cft,bool show) +{ +struct cgroup_subsys *ss = cft->ss; +struct cgroup *root = ss ? &ss->root->cgrp : &cgrp_dfl_root.cgrp;
Strictly speaking, this (v1) cgroup_root dereference should be guarded with cgroup_mutex too (should be the root destroy concurrently).
Quoted 3 lines
kernel/cgroup/cgroup.c +3
-0+struct cgroup_subsys_state *css; + +mutex_lock (&cgroup_mutex);
Tejun Heo
maintainerRe: [PATCH 1/7] cgroup: Implement cgroup_show_cftypes()
Hello, Michal.
Quoted 8 lines · Michal Koutný
kernel/cgroup/cgroup.c static void offline_css(struct cgroup_subsys_state *css) · line 5583
Quoted 4 lines · Tejun Heo
kernel/cgroup/cgroup.c +4
-0+void cgroup_show_cftype (struct cftype *cft,bool show) +{ +struct cgroup_subsys *ss = cft->ss; +struct cgroup *root = ss ? &ss->root->cgrp : &cgrp_dfl_root.cgrp;Strictly speaking, this (v1)
cgroup_rootdereference should be guarded withcgroup_mutextoo (should be the root destroy concurrently).
Ah, right. I think I might just drop this patch. Hiding and showing the interface files dynamically doesn't seem to work that well in practive (ie. it causes more production problems than it solves). Something a bit more boring like triggering a warning when a unimplemented feature is used is probably better here.
Thanks.
Signature · 1 line
tejun