Skip to contentSkip to conversation

SCHED_DEADLINE server infrastructure

RFC PATCH Applied Peter Zijlstra 12 participants 63 messages Jul 26, 2019 – Jun 16, 2020

Subsystems (5)
lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

cover SCHED_DEADLINE server infrastructure

Hi,

So recently syzcaller ran into the big deadline/period issue (again), and I figured I should at least propose a patch that puts limits on that -- see Patch 1.

During that discussion; SCHED_OTHER servers got mentioned (also again), and I figured I should have a poke at that too. So I took some inspiration from patches Alessio Balsini send a while back and cobbled something together for that too.

Included are also a bunch of patches I did for core scheduling (2-8), which I'm probably going to just merge as they're generic cleanups. They're included here because they make pick_next_task() simpler and thereby thinking about the nested pick_next_task() logic inherent in servers was less of a head-ache. (I think it can work fine without them, but its easier with them on)

Anyway; after it all compiled it booted a kvm image all the way to userspace on the first run, so clearly this code isn't to be trusted at all.

There's still lots of missing bits and pieces -- like changelogs and the fair server isn't configurable or hooked into the bandwidth accounting, but the foundation is there I think.

Enjoy!

author maintainer
#1 lore raw

RFC PATCH 1/13 sched/deadline: Impose global limits on sched_attr::sched_period

Cc Daniel Bristot de Oliveira <[email protected]>
Cc Luca Abeni <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Dmitry Vyukov <[email protected]>
---
 include/linux/sched/sysctl.h |    3 +++
 kernel/sched/deadline.c      |   23 +++++++++++++++++++++--
 kernel/sysctl.c              |   14 ++++++++++++++
 3 files changed, 38 insertions(+), 2 deletions(-)
include/linux/sched/sysctl.h +3 -0
@@ -56,6 +56,9 @@ int sched_proc_update_handler(struct ctl
 extern unsigned int sysctl_sched_rt_period;
 extern int sysctl_sched_rt_runtime;
 
+extern unsigned int sysctl_sched_dl_period_max;
+extern unsigned int sysctl_sched_dl_period_min;
+
 #ifdef CONFIG_UCLAMP_TASK
 extern unsigned int sysctl_sched_uclamp_util_min;
 extern unsigned int sysctl_sched_uclamp_util_max;
kernel/sched/deadline.c +21 -2
@@ -2597,6 +2597,14 @@ void __getparam_dl(struct task_struct *p
 }
 
 /*
+ * Default limits for DL period; on the top end we guard against small util
+ * tasks still getting rediculous long effective runtimes, on the bottom end we
+ * guard against timer DoS.
+ */
+unsigned int sysctl_sched_dl_period_max = 1 << 22; /* ~4 seconds */
+unsigned int sysctl_sched_dl_period_min = 100;     /* 100 us */
+
+/*
  * This function validates the new parameters of a -deadline task.
  * We ask for the deadline not being zero, and greater or equal
  * than the runtime, as well as the period of being zero or
@@ -2608,6 +2616,8 @@ void __getparam_dl(struct task_struct *p
  */
 bool __checkparam_dl(const struct sched_attr *attr)
 {
+	u64 period, max, min;
+
 	/* special dl tasks don't actually use any parameter */
 	if (attr->sched_flags & SCHED_FLAG_SUGOV)
 		return true;
@@ -2631,12 +2641,21 @@ bool __checkparam_dl(const struct sched_
 	    attr->sched_period & (1ULL << 63))
 		return false;
 
+	period = attr->sched_period;
+	if (!period)
+		period = attr->sched_deadline;
+
 	/* runtime <= deadline <= period (if period != 0) */
-	if ((attr->sched_period != 0 &&
-	     attr->sched_period < attr->sched_deadline) ||
+	if (period < attr->sched_deadline ||
 	    attr->sched_deadline < attr->sched_runtime)
 		return false;
 
+	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
+	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
+
+	if (period < min || period > max)
+		return false;
+
 	return true;
 }
 
kernel/sysctl.c +14 -0
@@ -443,6 +443,20 @@ static struct ctl_table kern_table[] = {
 		.proc_handler	= sched_rt_handler,
 	},
 	{
+		.procname	= "sched_deadline_period_max_us",
+		.data		= &sysctl_sched_dl_period_max,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
+	{
+		.procname	= "sched_deadline_period_min_us",
+		.data		= &sysctl_sched_dl_period_min,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
+	{
 		.procname	= "sched_rr_timeslice_ms",
 		.data		= &sysctl_sched_rr_timeslice,
 		.maxlen		= sizeof(int),
maintainer Acked-by on v1 1/13
#2 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Hi,

Quoted 27 lines · Peter Zijlstra
Cc Daniel Bristot de Oliveira <[email protected]>
Cc Luca Abeni <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Dmitry Vyukov <[email protected]>
---
 include/linux/sched/sysctl.h |    3 +++
 kernel/sched/deadline.c      |   23 +++++++++++++++++++++--
 kernel/sysctl.c              |   14 ++++++++++++++
 3 files changed, 38 insertions(+), 2 deletions(-)
include/linux/sched/sysctl.h +3 -0
@@ -56,6 +56,9 @@ int sched_proc_update_handler(struct ctl
 extern unsigned int sysctl_sched_rt_period;
 extern int sysctl_sched_rt_runtime;
 
+extern unsigned int sysctl_sched_dl_period_max;
+extern unsigned int sysctl_sched_dl_period_min;
+
 #ifdef CONFIG_UCLAMP_TASK
 extern unsigned int sysctl_sched_uclamp_util_min;
 extern unsigned int sysctl_sched_uclamp_util_max;

kernel/sched/deadline.c void __getparam_dl(struct task_struct *p · line 2601

Quoted 5 lines
kernel/sched/deadline.c +2 -0
@@ -2597,6 +2597,14 @@ void __getparam_dl(struct task_struct *p
 }
 
 /*
+ * Default limits for DL period; on the top end we guard against small util
+ * tasks still getting rediculous long effective runtimes, on the bottom end we

s/rediculous/ridiculous/

kernel/sched/deadline.c void __getparam_dl(struct task_struct *p · line 2605

Quoted 4 lines
kernel/sched/deadline.c +4 -0
+ * guard against timer DoS.
+ */
+unsigned int sysctl_sched_dl_period_max = 1 << 22; /* ~4 seconds */
+unsigned int sysctl_sched_dl_period_min = 100;     /* 100 us */

These limits look sane to me. I've actually been experimenting with 10us period tasks and throttling seemed to behave fine, but I guess 100us is a saner default.

So, (with a few lines of changelog :)

Thanks for doing this.

Best,

Juri

#3 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Quoted 44 lines · Juri Lelli

Hi,

Quoted 26 lines · Peter Zijlstra
Cc Daniel Bristot de Oliveira <[email protected]>
Cc Luca Abeni <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Dmitry Vyukov <[email protected]>
---
 include/linux/sched/sysctl.h |    3 +++
 kernel/sched/deadline.c      |   23 +++++++++++++++++++++--
 kernel/sysctl.c              |   14 ++++++++++++++
 3 files changed, 38 insertions(+), 2 deletions(-)
include/linux/sched/sysctl.h +3 -0
@@ -56,6 +56,9 @@ int sched_proc_update_handler(struct ctl
 extern unsigned int sysctl_sched_rt_period;
 extern int sysctl_sched_rt_runtime;
 
+extern unsigned int sysctl_sched_dl_period_max;
+extern unsigned int sysctl_sched_dl_period_min;
+
 #ifdef CONFIG_UCLAMP_TASK
 extern unsigned int sysctl_sched_uclamp_util_min;
 extern unsigned int sysctl_sched_uclamp_util_max;

kernel/sched/deadline.c void __getparam_dl(struct task_struct *p · line 2601

Quoted 5 lines
kernel/sched/deadline.c +2 -0
@@ -2597,6 +2597,14 @@ void __getparam_dl(struct task_struct *p
 }
 
 /*
+ * Default limits for DL period; on the top end we guard against small util
+ * tasks still getting rediculous long effective runtimes, on the bottom end we

s/rediculous/ridiculous/

kernel/sched/deadline.c void __getparam_dl(struct task_struct *p · line 2605

Quoted 4 lines
kernel/sched/deadline.c +4 -0
+ * guard against timer DoS.
+ */
+unsigned int sysctl_sched_dl_period_max = 1 << 22; /* ~4 seconds */
+unsigned int sysctl_sched_dl_period_min = 100;     /* 100 us */

These limits look sane to me. I've actually been experimenting with 10us period tasks and throttling seemed to behave fine, but I guess 100us is a saner default.

Quoted 3 lines

So, (with a few lines of changelog :)

Looks sane to me too!

-- Daniel

#4 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

In reply to Peter Zijlstra

Hi Peter,

This is indeed an important missing piece.

I think it would be worth having some simple checks, e.g.:
- period_min <= period_max;
- period_min >= (1ULL << DL_SCALE).

To be even more picky, I'm wondering if it would be a good practice to fail the write operation if there are already dl-tasks in the system that violate the new constraint. I'm afraid there is no cheap way of achieving such check, so, I think we can skip this last tricky thing for now.

Thanks,
Alessio

Quoted 99 lines · Peter Zijlstra
Cc Daniel Bristot de Oliveira <[email protected]>
Cc Luca Abeni <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Dmitry Vyukov <[email protected]>
---
 include/linux/sched/sysctl.h |    3 +++
 kernel/sched/deadline.c      |   23 +++++++++++++++++++++--
 kernel/sysctl.c              |   14 ++++++++++++++
 3 files changed, 38 insertions(+), 2 deletions(-)
include/linux/sched/sysctl.h +3 -0
@@ -56,6 +56,9 @@ int sched_proc_update_handler(struct ctl
 extern unsigned int sysctl_sched_rt_period;
 extern int sysctl_sched_rt_runtime;
 
+extern unsigned int sysctl_sched_dl_period_max;
+extern unsigned int sysctl_sched_dl_period_min;
+
 #ifdef CONFIG_UCLAMP_TASK
 extern unsigned int sysctl_sched_uclamp_util_min;
 extern unsigned int sysctl_sched_uclamp_util_max;
kernel/sched/deadline.c +21 -2
@@ -2597,6 +2597,14 @@ void __getparam_dl(struct task_struct *p
 }
 
 /*
+ * Default limits for DL period; on the top end we guard against small util
+ * tasks still getting rediculous long effective runtimes, on the bottom end we
+ * guard against timer DoS.
+ */
+unsigned int sysctl_sched_dl_period_max = 1 << 22; /* ~4 seconds */
+unsigned int sysctl_sched_dl_period_min = 100;     /* 100 us */
+
+/*
  * This function validates the new parameters of a -deadline task.
  * We ask for the deadline not being zero, and greater or equal
  * than the runtime, as well as the period of being zero or
@@ -2608,6 +2616,8 @@ void __getparam_dl(struct task_struct *p
  */
 bool __checkparam_dl(const struct sched_attr *attr)
 {
+	u64 period, max, min;
+
 	/* special dl tasks don't actually use any parameter */
 	if (attr->sched_flags & SCHED_FLAG_SUGOV)
 		return true;
@@ -2631,12 +2641,21 @@ bool __checkparam_dl(const struct sched_
 	    attr->sched_period & (1ULL << 63))
 		return false;
 
+	period = attr->sched_period;
+	if (!period)
+		period = attr->sched_deadline;
+
 	/* runtime <= deadline <= period (if period != 0) */
-	if ((attr->sched_period != 0 &&
-	     attr->sched_period < attr->sched_deadline) ||
+	if (period < attr->sched_deadline ||
 	    attr->sched_deadline < attr->sched_runtime)
 		return false;
 
+	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
+	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
+
+	if (period < min || period > max)
+		return false;
+
 	return true;
 }
 
kernel/sysctl.c +14 -0
@@ -443,6 +443,20 @@ static struct ctl_table kern_table[] = {
 		.proc_handler	= sched_rt_handler,
 	},
 	{
+		.procname	= "sched_deadline_period_max_us",
+		.data		= &sysctl_sched_dl_period_max,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
+	{
+		.procname	= "sched_deadline_period_min_us",
+		.data		= &sysctl_sched_dl_period_min,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
+	{
 		.procname	= "sched_rr_timeslice_ms",
 		.data		= &sysctl_sched_rr_timeslice,
 		.maxlen		= sizeof(int),
author maintainer
#5 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Quoted 8 lines · Alessio Balsini

Hi Peter,

This is indeed an important missing piece.

I think it would be worth having some simple checks, e.g.:
- period_min <= period_max;
- period_min >= (1ULL << DL_SCALE).

Quoted 5 lines

To be even more picky, I'm wondering if it would be a good practice to fail the write operation if there are already dl-tasks in the system that violate the new constraint. I'm afraid there is no cheap way of achieving such check, so, I think we can skip this last tricky thing for now.

Like so?

---
Subject: sched/deadline: Impose global limits on sched_attr::sched_period
From: Peter Zijlstra <[email protected]>
Date: Tue Jul 23 16:01:29 CEST 2019

There are two DoS scenarios with SCHED_DEADLINE related to sched_attr::sched_period:

- since access-control only looks at utilization and density, a very large period can allow a very large runtime, which in turn can incur a very large latency to lower priority tasks.

- for very short periods we can end up spending more time programming the hardware timer than actually running the task.

Mitigate these by imposing limits on the period.

Cc Daniel Bristot de Oliveira <[email protected]>
Cc Luca Abeni <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Dmitry Vyukov <[email protected]>
---
 include/linux/sched/sysctl.h |    7 ++++
 kernel/sched/deadline.c      |   61 +++++++++++++++++++++++++++++++++++++++++--
 kernel/sysctl.c              |   14 +++++++++
 3 files changed, 80 insertions(+), 2 deletions(-)
include/linux/sched/sysctl.h +7 -0
@@ -56,6 +56,13 @@ int sched_proc_update_handler(struct ctl
 extern unsigned int sysctl_sched_rt_period;
 extern int sysctl_sched_rt_runtime;
 
+extern unsigned int sysctl_sched_dl_period_max;
+extern unsigned int sysctl_sched_dl_period_min;
+
+extern int sched_dl_period_handler(struct ctl_table *table, int write,
+		void __user *buffer, size_t *lenp,
+		loff_t *ppos);
+
 #ifdef CONFIG_UCLAMP_TASK
 extern unsigned int sysctl_sched_uclamp_util_min;
 extern unsigned int sysctl_sched_uclamp_util_max;
kernel/sched/deadline.c +59 -2

83 lines not on this page · raw message

kernel/sysctl.c +14 -0
@@ -443,6 +443,20 @@ static struct ctl_table kern_table[] = {
 		.proc_handler	= sched_rt_handler,
 	},
 	{
+		.procname	= "sched_deadline_period_max_us",
+		.data		= &sysctl_sched_dl_period_max,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sched_dl_period_handler,
+	},
+	{
+		.procname	= "sched_deadline_period_min_us",
+		.data		= &sysctl_sched_dl_period_min,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sched_dl_period_handler,
+	},
+	{
 		.procname	= "sched_rr_timeslice_ms",
 		.data		= &sysctl_sched_rr_timeslice,
 		.maxlen		= sizeof(int),
#6 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Quoted 3 lines · Peter Zijlstra

Like so?

Yes, that's exactly what I meant!
What about this refactoring?

Thanks,
Alessio

---
From 459d5488acb3fac938b0f35f480a81a6e401ef92 Mon Sep 17 00:00:00 2001
From: Alessio Balsini <[email protected]>
Date: Thu, 22 Aug 2019 12:55:55 +0100
Subject: [PATCH] sched/deadline: Impose global limits on
sched_attr::sched_period

There are two DoS scenarios with SCHED_DEADLINE related to sched_attr::sched_period:

- since access-control only looks at utilization and density, a very large period can allow a very large runtime, which in turn can incur a very large latency to lower priority tasks.

- for very short periods we can end up spending more time programming the hardware timer than actually running the task.

Mitigate these by imposing limits on the period.

---
 include/linux/sched/sysctl.h |  7 +++++
 kernel/sched/deadline.c      | 51 ++++++++++++++++++++++++++++++++++--
 kernel/sysctl.c              | 14 ++++++++++
 3 files changed, 70 insertions(+), 2 deletions(-)
include/linux/sched/sysctl.h +7 -0
index d4f6215ee03f7..7c8ef07e52133 100644
@@ -56,6 +56,13 @@ int sched_proc_update_handler(struct ctl_table *table, int write,
 extern unsigned int sysctl_sched_rt_period;
 extern int sysctl_sched_rt_runtime;
 
+extern unsigned int sysctl_sched_dl_period_max;
+extern unsigned int sysctl_sched_dl_period_min;
+
+extern int sched_dl_period_handler(struct ctl_table *table, int write,
+		void __user *buffer, size_t *lenp,
+		loff_t *ppos);
+
 #ifdef CONFIG_UCLAMP_TASK
 extern unsigned int sysctl_sched_uclamp_util_min;
 extern unsigned int sysctl_sched_uclamp_util_max;
kernel/sched/deadline.c +49 -2

73 lines not on this page · raw message

kernel/sysctl.c +14 -0
index 078950d9605ba..0d07e4707e9d2 100644
@@ -442,6 +442,20 @@ static struct ctl_table kern_table[] = {
 		.mode		= 0644,
 		.proc_handler	= sched_rt_handler,
 	},
+	{
+		.procname	= "sched_deadline_period_max_us",
+		.data		= &sysctl_sched_dl_period_max,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sched_dl_period_handler,
+	},
+	{
+		.procname	= "sched_deadline_period_min_us",
+		.data		= &sysctl_sched_dl_period_min,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sched_dl_period_handler,
+	},
 	{
 		.procname	= "sched_rr_timeslice_ms",
 		.data		= &sysctl_sched_rr_timeslice,
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
author maintainer
#7 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Quoted 2 lines · Alessio Balsini

Yes, that's exactly what I meant!
What about this refactoring?

Makes sense; and now I spot a race (and sched_rt_handler() from which I copied this is susceptible too):

Review

Quoted 13 lines
(quoted diff) +13 -0
+int sched_dl_period_handler(struct ctl_table *table, int write,
+			    void __user *buffer, size_t *lenp,
+			    loff_t *ppos)
+{
+	unsigned int old_max, old_min;
+	static DEFINE_MUTEX(mutex);
+	int ret;
+
+	mutex_lock(&mutex);
+	old_max = sysctl_sched_dl_period_max;
+	old_min = sysctl_sched_dl_period_min;
+
+	ret = proc_douintvec(table, write, buffer, lenp, ppos);

Here the sysctl_* values have been changed, interleave with:

Quoted 22 lines
(quoted diff) +17 -0
+	if (!ret && write) {
+		u64 max = (u64)sysctl_sched_dl_period_max * NSEC_PER_USEC;
+		u64 min = (u64)sysctl_sched_dl_period_min * NSEC_PER_USEC;
+
+		if (min < 1ULL << DL_SCALE || max < min) {
+			sysctl_sched_dl_period_max = old_max;
+			sysctl_sched_dl_period_min = old_min;
+			ret = -EINVAL;
+		}
+	}
+	mutex_unlock(&mutex);
+
+	return ret;
+}
@@ -2675,12 +2713,21 @@ bool __checkparam_dl(const struct sched_attr *attr)
 	    attr->sched_period & (1ULL << 63))
 		return false;
 
+	period = attr->sched_period;
+	if (!period)
+		period = attr->sched_deadline;

Review bool __checkparam_dl(const struct sched_attr *attr) · line 2726

Quoted 10 lines
(quoted diff) +4 -2
+
 	/* runtime <= deadline <= period (if period != 0) */
-	if ((attr->sched_period != 0 &&
-	     attr->sched_period < attr->sched_deadline) ||
+	if (period < attr->sched_deadline ||
 	    attr->sched_deadline < attr->sched_runtime)
 		return false;
 
+	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
+	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;

this, and we're using unvalidated numbers.

Review bool __checkparam_dl(const struct sched_attr *attr) · line 2731

Quoted 5 lines
(quoted diff) +3 -0
+	if (period < min || period > max)
+		return false;
+
 	return true;
 }

Something like the completely untested (again, sorry) below ought to cure that I think; the same needs doing to sched_rt_handler() I'm thinking.

---
Subject: sched/deadline: Impose global limits on sched_attr::sched_period
From: Alessio Balsini <[email protected]>
Date: Thu, 22 Aug 2019 13:29:49 +0100

There are two DoS scenarios with SCHED_DEADLINE related to sched_attr::sched_period:

- since access-control only looks at utilization and density, a very large period can allow a very large runtime, which in turn can incur a very large latency to lower priority tasks.

- for very short periods we can end up spending more time programming the hardware timer than actually running the task.

Mitigate these by imposing limits on the period.

---
 include/linux/sched/sysctl.h |    7 +++++
 kernel/sched/deadline.c      |   58 +++++++++++++++++++++++++++++++++++++++++--
 kernel/sysctl.c              |   14 ++++++++++
 3 files changed, 77 insertions(+), 2 deletions(-)
include/linux/sched/sysctl.h +7 -0
@@ -56,6 +56,13 @@ int sched_proc_update_handler(struct ctl
 extern unsigned int sysctl_sched_rt_period;
 extern int sysctl_sched_rt_runtime;
 
+extern unsigned int sysctl_sched_dl_period_max;
+extern unsigned int sysctl_sched_dl_period_min;
+
+extern int sched_dl_period_handler(struct ctl_table *table, int write,
+		void __user *buffer, size_t *lenp,
+		loff_t *ppos);
+
 #ifdef CONFIG_UCLAMP_TASK
 extern unsigned int sysctl_sched_uclamp_util_min;
 extern unsigned int sysctl_sched_uclamp_util_max;
kernel/sched/deadline.c +56 -2

80 lines not on this page · raw message

kernel/sysctl.c +14 -0
@@ -443,6 +443,20 @@ static struct ctl_table kern_table[] = {
 		.proc_handler	= sched_rt_handler,
 	},
 	{
+		.procname	= "sched_deadline_period_max_us",
+		.data		= &sysctl_sched_dl_period_max,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sched_dl_period_handler,
+	},
+	{
+		.procname	= "sched_deadline_period_min_us",
+		.data		= &sysctl_sched_dl_period_min,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sched_dl_period_handler,
+	},
+	{
 		.procname	= "sched_rr_timeslice_ms",
 		.data		= &sysctl_sched_rr_timeslice,
 		.maxlen		= sizeof(int),
#8 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Right!

Verified that sysctl_sched_dl_period_max and sysctl_sched_dl_period_min values are now always consistent.

I spent some time in trying to figure out if not having any mutex in __checkparam_dl() is safe. There can surely happen that "max < min", e.g.:

          |              |               periods
User1     | User2        | checkparam_dl()  | sysctl_sched_dl_*
----------|--------------|------------------|-------------------
          |              |                  | [x, x]
p_min = 5 |              |                  |
          |              |                  | [5, x]
p_max = 5 |              |                  |
          |              |                  | [5, 5]
          | setattr(p=8) |                  |
          |              | p = 8            |
          |              | [x, 5]           |
p_max = 9 |              |                  |
          |              |                  | [5, 9]
p_min = 6 |              |                  |
          |              |                  | [6, 9]
          |              | [6, 5]           |
----------|--------------|------------------|-------------------

Sharing my thoughts, a "BUG_ON(max < min)" in __checkparam_dl() is then a guaranteed source of explosions, but the good news is that "if (period < min || period > max" in __checkparam_dl() surely fails if "max < min". Also the fact that, when we are writing the new sysctl_sched_dl_* values, only one is actually changed at a time, that surely helps to preserve the consistency.

But is that enough?

Well, I couldn't find any counterexample to make __checkparam_dl() pass with wrong parameters. And the tests I made are happy.

kernel/sched/deadline.c

Quoted 12 lines · Peter Zijlstra
 include/linux/sched/sysctl.h |    7 +++++
 kernel/sched/deadline.c      |   58 +++++++++++++++++++++++++++++++++++++++++--
 kernel/sysctl.c              |   14 ++++++++++
 3 files changed, 77 insertions(+), 2 deletions(-)
kernel/sched/deadline.c +5 -0
+int sched_dl_period_handler(struct ctl_table *table, int write,
+			    void __user *buffer, size_t *lenp,
+			    loff_t *ppos)
+{
+		if (min > 1ULL << DL_SCALE && max > min) {

s/>/>=/

kernel/sched/deadline.c

Quoted 2 lines
kernel/sched/deadline.c +2 -0
+			WRITE_ONCE(sysctl_sched_dl_period_max, new_max);
+			WRITE_ONCE(sysctl_sched_dl_period_min, new_min);

Besides the inline comment above, this is my ack to your patch.

Otherwise, here follows a slightly more convoluted version of your patch with a couple of changes to sched_dl_period_handler(), summarized as:
- handle new_table only if writing;
- directly compare the us min and max (saves one multiplication);
- atomic-writes only the sysctl_sched_dl_period_XXX which changed.

M2c.

Thanks!
-Alessio

---
From cb4481233b57e42ff9dd315811f7919168a28162 Mon Sep 17 00:00:00 2001
From: Peter Zijlstra <[email protected]>
Date: Thu, 22 Aug 2019 18:51:25 +0200
Subject: [PATCH] sched/deadline: Impose global limits on
sched_attr::sched_period

There are two DoS scenarios with SCHED_DEADLINE related to sched_attr::sched_period:

- since access-control only looks at utilization and density, a very large period can allow a very large runtime, which in turn can incur a very large latency to lower priority tasks.

- for very short periods we can end up spending more time programming the hardware timer than actually running the task.

Mitigate these by imposing limits on the period.

---
 include/linux/sched/sysctl.h |  7 ++++
 kernel/sched/deadline.c      | 68 ++++++++++++++++++++++++++++++++++--
 kernel/sysctl.c              | 14 ++++++++
 3 files changed, 87 insertions(+), 2 deletions(-)
include/linux/sched/sysctl.h +7 -0
index d4f6215ee03f7..7c8ef07e52133 100644
@@ -56,6 +56,13 @@ int sched_proc_update_handler(struct ctl_table *table, int write,
 extern unsigned int sysctl_sched_rt_period;
 extern int sysctl_sched_rt_runtime;
 
+extern unsigned int sysctl_sched_dl_period_max;
+extern unsigned int sysctl_sched_dl_period_min;
+
+extern int sched_dl_period_handler(struct ctl_table *table, int write,
+		void __user *buffer, size_t *lenp,
+		loff_t *ppos);
+
 #ifdef CONFIG_UCLAMP_TASK
 extern unsigned int sysctl_sched_uclamp_util_min;
 extern unsigned int sysctl_sched_uclamp_util_max;
kernel/sched/deadline.c +66 -2

90 lines not on this page · raw message

kernel/sysctl.c +14 -0
index 078950d9605ba..0d07e4707e9d2 100644
@@ -442,6 +442,20 @@ static struct ctl_table kern_table[] = {
 		.mode		= 0644,
 		.proc_handler	= sched_rt_handler,
 	},
+	{
+		.procname	= "sched_deadline_period_max_us",
+		.data		= &sysctl_sched_dl_period_max,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sched_dl_period_handler,
+	},
+	{
+		.procname	= "sched_deadline_period_min_us",
+		.data		= &sysctl_sched_dl_period_min,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sched_dl_period_handler,
+	},
 	{
 		.procname	= "sched_rr_timeslice_ms",
 		.data		= &sysctl_sched_rr_timeslice,
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
author maintainer
#9 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Quoted 26 lines · Alessio Balsini

Right!

Verified that sysctl_sched_dl_period_max and sysctl_sched_dl_period_min values are now always consistent.

I spent some time in trying to figure out if not having any mutex in __checkparam_dl() is safe. There can surely happen that "max < min", e.g.:

          |              |               periods
User1     | User2        | checkparam_dl()  | sysctl_sched_dl_*
----------|--------------|------------------|-------------------
          |              |                  | [x, x]
p_min = 5 |              |                  |
          |              |                  | [5, x]
p_max = 5 |              |                  |
          |              |                  | [5, 5]
          | setattr(p=8) |                  |
          |              | p = 8            |
          |              | [x, 5]           |
p_max = 9 |              |                  |
          |              |                  | [5, 9]
p_min = 6 |              |                  |
          |              |                  | [6, 9]
          |              | [6, 5]           |
----------|--------------|------------------|-------------------
Quoted 7 lines

Sharing my thoughts, a "BUG_ON(max < min)" in __checkparam_dl() is then a guaranteed source of explosions, but the good news is that "if (period < min || period > max" in __checkparam_dl() surely fails if "max < min". Also the fact that, when we are writing the new sysctl_sched_dl_* values, only one is actually changed at a time, that surely helps to preserve the consistency.

But is that enough?

Strictly speaking, no, I suppose it is not. We can have two changes in between the two READ_ONCE()s and then we'd be able to observe a violation.

The easy way to fix that is do something like:

+	synchronize_rcu();
	mutex_unlock(&mutex);

in sched_dl_period_handler(). And do:

+	preempt_disable();
	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;

+ preempt_enable();

in __checkparam_dl().

That would prohibit we see two changes, and seeing only the single change is safe.

author maintainer
#10 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Quoted 33 lines · Peter Zijlstra
Quoted 8 lines · Alessio Balsini

Right!

Verified that sysctl_sched_dl_period_max and sysctl_sched_dl_period_min values are now always consistent.

I spent some time in trying to figure out if not having any mutex in __checkparam_dl() is safe. There can surely happen that "max < min", e.g.:

Quoted 7 lines

Sharing my thoughts, a "BUG_ON(max < min)" in __checkparam_dl() is then a guaranteed source of explosions, but the good news is that "if (period < min || period > max" in __checkparam_dl() surely fails if "max < min". Also the fact that, when we are writing the new sysctl_sched_dl_* values, only one is actually changed at a time, that surely helps to preserve the consistency.

But is that enough?

Strictly speaking, no, I suppose it is not. We can have two changes in between the two READ_ONCE()s and then we'd be able to observe a violation.

The easy way to fix that is do something like:

(quoted diff) +1 -0
+	synchronize_rcu();
	mutex_unlock(&mutex);

in sched_dl_period_handler(). And do:

(quoted diff) +1 -0
+	preempt_disable();
	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
(quoted diff) +1 -0
+	preempt_enable();
Quoted 4 lines

in __checkparam_dl().

That would prohibit we see two changes, and seeing only the single change is safe.

I pushed out a new version; and added patch to sched_rt_handler() on top.

Please have a look at:

git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/wip-deadline

I'll move these two patches to sched/core if everything looks good.

#11 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

In reply to Peter Zijlstra

Review

Quoted 6 lines · On Mon, 2 Sep 2019 11:16:23 +0200 Peter Zijlstra <[email protected]> wrote:

in sched_dl_period_handler(). And do:

(quoted diff) +1 -0
+	preempt_disable();
	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
(quoted diff) +1 -0
+	preempt_enable();

Hmm, I'm curious. Doesn't the preempt_disable/enable() also add compiler barriers which would remove the need for the READ_ONCE()s here?

-- Steve

author maintainer
#12 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Quoted 5 lines · Steven Rostedt
Quoted 2 lines · On Mon, 2 Sep 2019 11:16:23 +0200 Peter Zijlstra <[email protected]> wrote:

in sched_dl_period_handler(). And do:

Quoted 7 lines

Review

Quoted 4 lines
(quoted diff) +1 -0
+	preempt_disable();
	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
(quoted diff) +1 -0
+	preempt_enable();

Hmm, I'm curious. Doesn't the preempt_disable/enable() also add compiler barriers which would remove the need for the READ_ONCE()s here?

They do add compiler barriers; but they do not avoid the compiler tearing stuff up.

So while Linus has declared that compilers should not be tearing shit up, I'm hesitant to actually trust compilers much these days.

#13 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Quoted 14 lines · Peter Zijlstra
Quoted 5 lines · Steven Rostedt
Quoted 2 lines · On Mon, 2 Sep 2019 11:16:23 +0200 Peter Zijlstra <[email protected]> wrote:

in sched_dl_period_handler(). And do:

Quoted 7 lines

Review

Quoted 4 lines
(quoted diff) +1 -0
+	preempt_disable();
	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
(quoted diff) +1 -0
+	preempt_enable();

Hmm, I'm curious. Doesn't the preempt_disable/enable() also add compiler barriers which would remove the need for the READ_ONCE()s here?

Quoted 2 lines

They do add compiler barriers; but they do not avoid the compiler tearing stuff up.

Neither does WRITE_ONCE() on some possibly buggy but currently circulating compilers :(

As Will said in:
https://lore.kernel.org/lkml/20190821103200.kpufwtviqhpbuv2n@willie-the-truck/

void bar(u64 *x)

{
	*(volatile u64 *)x = 0xabcdef10abcdef10;
}

gives:

bar:

	mov	w1, 61200
	movk	w1, 0xabcd, lsl 16
	str	w1, [x0]
	str	w1, [x0, 4]
	ret

Speaking of which, Will, is there a plan to have compiler folks address this tearing issue and are bugs filed somewhere? I believe aarch64 gcc is buggy, and clang is better but is still buggy?

thanks,

- Joel

#14 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Hi Joel,

Quoted 15 lines · Joel Fernandes
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Steven Rostedt

Review

Quoted 6 lines · On Mon, 2 Sep 2019 11:16:23 +0200 Peter Zijlstra <[email protected]> wrote:

in sched_dl_period_handler(). And do:

(quoted diff) +1 -0
+	preempt_disable();
	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
(quoted diff) +1 -0
+	preempt_enable();

Hmm, I'm curious. Doesn't the preempt_disable/enable() also add compiler barriers which would remove the need for the READ_ONCE()s here?

Quoted 5 lines
Quoted 2 lines

They do add compiler barriers; but they do not avoid the compiler tearing stuff up.

Neither does WRITE_ONCE() on some possibly buggy but currently circulating compilers :(

Hmm. The example above is using READ_ONCE, which is a different kettle of frogs.

Quoted 17 lines

As Will said in:
https://lore.kernel.org/lkml/20190821103200.kpufwtviqhpbuv2n@willie-the-truck/

void bar(u64 *x)

{
	*(volatile u64 *)x = 0xabcdef10abcdef10;
}

gives:

bar:

	mov	w1, 61200
	movk	w1, 0xabcd, lsl 16
	str	w1, [x0]
	str	w1, [x0, 4]
	ret
Quoted 3 lines

Speaking of which, Will, is there a plan to have compiler folks address this tearing issue and are bugs filed somewhere? I believe aarch64 gcc is buggy, and clang is better but is still buggy?

Well, it depends on your point of view. Personally, I think tearing a volatile access (e.g. WRITE_ONCE) is buggy and it seems as though the GCC developers agree:

https://gcc.gnu.org/ml/gcc-patches/2019-08/msg01500.html

so it's likely this will be fixed for AArch64 GCC. I couldn't persuade clang to break the volatile case, so think we're good there too.

For the non-volatile case, I don't actually consider it to be a bug, although I sympathise with the desire to avoid a retrospective tree-wide sweep adding random WRITE_ONCE invocations to stores that look like they might be concurrent. In other words, I think I'd suggest:

* The use of WRITE_ONCE in new code (probably with a comment justifying it)
* The introduction of WRITE_ONCE to existing code where it can be shown to be fixing a real bug (e.g. by demonstrating that a compiler actually gets it wrong)

For the /vast/ majority of cases, the compiler will do the right thing even without WRITE_ONCE, simply because that's going to be the most performant choice as well.

Will

#15 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Quoted 18 lines · Will Deacon

Hi Joel,

Quoted 15 lines · Joel Fernandes
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Steven Rostedt

Review

Quoted 6 lines · On Mon, 2 Sep 2019 11:16:23 +0200 Peter Zijlstra <[email protected]> wrote:

in sched_dl_period_handler(). And do:

(quoted diff) +1 -0
+	preempt_disable();
	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
(quoted diff) +1 -0
+	preempt_enable();

Hmm, I'm curious. Doesn't the preempt_disable/enable() also add compiler barriers which would remove the need for the READ_ONCE()s here?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

They do add compiler barriers; but they do not avoid the compiler tearing stuff up.

Neither does WRITE_ONCE() on some possibly buggy but currently circulating compilers :(

Hmm. The example above is using READ_ONCE, which is a different kettle of frogs.

True. But, I equally worry about all *-tearing frog kettles ;-)

Quoted 21 lines
Quoted 17 lines

As Will said in:
https://lore.kernel.org/lkml/20190821103200.kpufwtviqhpbuv2n@willie-the-truck/

void bar(u64 *x)

{
	*(volatile u64 *)x = 0xabcdef10abcdef10;
}

gives:

bar:

	mov	w1, 61200
	movk	w1, 0xabcd, lsl 16
	str	w1, [x0]
	str	w1, [x0, 4]
	ret
Quoted 3 lines

Speaking of which, Will, is there a plan to have compiler folks address this tearing issue and are bugs filed somewhere? I believe aarch64 gcc is buggy, and clang is better but is still buggy?

Quoted 8 lines

Well, it depends on your point of view. Personally, I think tearing a volatile access (e.g. WRITE_ONCE) is buggy and it seems as though the GCC developers agree:

https://gcc.gnu.org/ml/gcc-patches/2019-08/msg01500.html

so it's likely this will be fixed for AArch64 GCC. I couldn't persuade clang to break the volatile case, so think we're good there too.

Glad to know that GCC folks are looking into the issue.

Sorry if this is getting a bit off-topic. Also does the aarch64 clang doing the "memset folding" issue, also need to be looked into?
You had mentioned it in the same thread:
https://lore.kernel.org/lkml/20190821103200.kpufwtviqhpbuv2n@willie-the-truck/
Or, does WRITE_ONCE() resolve such memset store-merging?

Quoted 5 lines

For the non-volatile case, I don't actually consider it to be a bug, although I sympathise with the desire to avoid a retrospective tree-wide sweep adding random WRITE_ONCE invocations to stores that look like they might be concurrent. In other words, I think I'd suggest:

Quoted 8 lines

* The use of WRITE_ONCE in new code (probably with a comment justifying it)
* The introduction of WRITE_ONCE to existing code where it can be shown to be fixing a real bug (e.g. by demonstrating that a compiler actually gets it wrong)

For the /vast/ majority of cases, the compiler will do the right thing even without WRITE_ONCE, simply because that's going to be the most performant choice as well.

Thanks for the thoughts. They seem to be reasonable to me.

thanks,

- Joel

author maintainer
#16 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

In reply to Joel Fernandes
Quoted 15 lines · Joel Fernandes
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Steven Rostedt

Review

Quoted 6 lines · On Mon, 2 Sep 2019 11:16:23 +0200 Peter Zijlstra <[email protected]> wrote:

in sched_dl_period_handler(). And do:

(quoted diff) +1 -0
+	preempt_disable();
	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
(quoted diff) +1 -0
+	preempt_enable();

Hmm, I'm curious. Doesn't the preempt_disable/enable() also add compiler barriers which would remove the need for the READ_ONCE()s here?

Quoted 5 lines
Quoted 2 lines

They do add compiler barriers; but they do not avoid the compiler tearing stuff up.

Neither does WRITE_ONCE() on some possibly buggy but currently circulating compilers :(

Yes, I'm aware :/

#17 lore raw

PATCH 1/1 loop: Add LOOP_SET_DIRECT_IO to compat ioctl

In reply to Peter Zijlstra

[ Upstream commit fdbe4eeeb1aac219b14f10c0ed31ae5d1123e9b8 ]

Enabling Direct I/O with loop devices helps reducing memory usage by avoiding double caching. 32 bit applications running on 64 bits systems are currently not able to request direct I/O because is missing from the lo_compat_ioctl.

This patch fixes the compatibility issue mentioned above by exporting LOOP_SET_DIRECT_IO as additional lo_compat_ioctl() entry. The input argument for this ioctl is a single long converted to a 1-bit boolean, so compatibility is preserved.

---
 drivers/block/loop.c | 1 +
 1 file changed, 1 insertion(+)
drivers/block/loop.c +1 -0
index da3902ac16c86..8aadd4d0c3a88 100644
@@ -1557,6 +1557,7 @@ static int lo_compat_ioctl(struct block_device *bdev, fmode_t mode,
 		arg = (unsigned long) compat_ptr(arg);
 	case LOOP_SET_FD:
 	case LOOP_CHANGE_FD:
+	case LOOP_SET_DIRECT_IO:
 		err = lo_ioctl(bdev, mode, cmd, arg);
 		break;
 	default:
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
#18 lore raw

Re: [PATCH 4.4 4.9 4.14] loop: Add LOOP_SET_DIRECT_IO to compat ioctl

Ops, please forgive the wrong in-reply-to messge id :)

Cheers,
Alessio

Quoted 38 lines · Alessio Balsini

[ Upstream commit fdbe4eeeb1aac219b14f10c0ed31ae5d1123e9b8 ]

Enabling Direct I/O with loop devices helps reducing memory usage by avoiding double caching. 32 bit applications running on 64 bits systems are currently not able to request direct I/O because is missing from the lo_compat_ioctl.

This patch fixes the compatibility issue mentioned above by exporting LOOP_SET_DIRECT_IO as additional lo_compat_ioctl() entry. The input argument for this ioctl is a single long converted to a 1-bit boolean, so compatibility is preserved.

---
 drivers/block/loop.c | 1 +
 1 file changed, 1 insertion(+)
drivers/block/loop.c +1 -0
index da3902ac16c86..8aadd4d0c3a88 100644
@@ -1557,6 +1557,7 @@ static int lo_compat_ioctl(struct block_device *bdev, fmode_t mode,
 		arg = (unsigned long) compat_ptr(arg);
 	case LOOP_SET_FD:
 	case LOOP_CHANGE_FD:
+	case LOOP_SET_DIRECT_IO:
 		err = lo_ioctl(bdev, mode, cmd, arg);
 		break;
 	default:
Signature · 4 lines
2.23.0.866.gb869b98d4c-goog

-- 
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
maintainer Applied
#19 lore raw

Re: [PATCH 4.4 4.9 4.14] loop: Add LOOP_SET_DIRECT_IO to compat ioctl

In reply to Alessio Balsini
Quoted 12 lines · Alessio Balsini

[ Upstream commit fdbe4eeeb1aac219b14f10c0ed31ae5d1123e9b8 ]

Enabling Direct I/O with loop devices helps reducing memory usage by avoiding double caching. 32 bit applications running on 64 bits systems are currently not able to request direct I/O because is missing from the lo_compat_ioctl.

This patch fixes the compatibility issue mentioned above by exporting LOOP_SET_DIRECT_IO as additional lo_compat_ioctl() entry. The input argument for this ioctl is a single long converted to a 1-bit boolean, so compatibility is preserved.

Quoted 4 lines

Queued up, thanks!

Signature · 2 lines
Thanks,
Sasha
maintainer
#20 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

In reply to Peter Zijlstra

Hi Peter,

Quoted 87 lines · Peter Zijlstra
Cc Daniel Bristot de Oliveira <[email protected]>
Cc Luca Abeni <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Dmitry Vyukov <[email protected]>
---
 include/linux/sched/sysctl.h |    3 +++
 kernel/sched/deadline.c      |   23 +++++++++++++++++++++--
 kernel/sysctl.c              |   14 ++++++++++++++
 3 files changed, 38 insertions(+), 2 deletions(-)
include/linux/sched/sysctl.h +3 -0
@@ -56,6 +56,9 @@ int sched_proc_update_handler(struct ctl
 extern unsigned int sysctl_sched_rt_period;
 extern int sysctl_sched_rt_runtime;
 
+extern unsigned int sysctl_sched_dl_period_max;
+extern unsigned int sysctl_sched_dl_period_min;
+
 #ifdef CONFIG_UCLAMP_TASK
 extern unsigned int sysctl_sched_uclamp_util_min;
 extern unsigned int sysctl_sched_uclamp_util_max;
kernel/sched/deadline.c +21 -2
@@ -2597,6 +2597,14 @@ void __getparam_dl(struct task_struct *p
 }
 
 /*
+ * Default limits for DL period; on the top end we guard against small util
+ * tasks still getting rediculous long effective runtimes, on the bottom end we
+ * guard against timer DoS.
+ */
+unsigned int sysctl_sched_dl_period_max = 1 << 22; /* ~4 seconds */
+unsigned int sysctl_sched_dl_period_min = 100;     /* 100 us */
+
+/*
  * This function validates the new parameters of a -deadline task.
  * We ask for the deadline not being zero, and greater or equal
  * than the runtime, as well as the period of being zero or
@@ -2608,6 +2616,8 @@ void __getparam_dl(struct task_struct *p
  */
 bool __checkparam_dl(const struct sched_attr *attr)
 {
+	u64 period, max, min;
+
 	/* special dl tasks don't actually use any parameter */
 	if (attr->sched_flags & SCHED_FLAG_SUGOV)
 		return true;
@@ -2631,12 +2641,21 @@ bool __checkparam_dl(const struct sched_
 	    attr->sched_period & (1ULL << 63))
 		return false;
 
+	period = attr->sched_period;
+	if (!period)
+		period = attr->sched_deadline;
+
 	/* runtime <= deadline <= period (if period != 0) */
-	if ((attr->sched_period != 0 &&
-	     attr->sched_period < attr->sched_deadline) ||
+	if (period < attr->sched_deadline ||
 	    attr->sched_deadline < attr->sched_runtime)
 		return false;
 
+	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
+	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
+
+	if (period < min || period > max)
+		return false;
+
 	return true;
 }
 
kernel/sysctl.c +7 -0
@@ -443,6 +443,20 @@ static struct ctl_table kern_table[] = {
 		.proc_handler	= sched_rt_handler,
 	},
 	{
+		.procname	= "sched_deadline_period_max_us",
+		.data		= &sysctl_sched_dl_period_max,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
+	{

kernel/sysctl.c static struct ctl_table kern_table[] = { · line 462

Quoted 10 lines
kernel/sysctl.c +7 -0
+		.procname	= "sched_deadline_period_min_us",
+		.data		= &sysctl_sched_dl_period_min,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
+	{
 		.procname	= "sched_rr_timeslice_ms",
 		.data		= &sysctl_sched_rr_timeslice,
 		.maxlen		= sizeof(int),

I think this never made it upstream. And I believe both me and Daniel were OK with it. Do you recall if any additional change was needed?

Thanks,

Juri

#21 lore raw

Re: [RFC][PATCH 01/13] sched/deadline: Impose global limits on sched_attr::sched_period

Quoted 93 lines · Juri Lelli

Hi Peter,

Quoted 90 lines · Peter Zijlstra
Cc Daniel Bristot de Oliveira <[email protected]>
Cc Luca Abeni <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Dmitry Vyukov <[email protected]>
---
 include/linux/sched/sysctl.h |    3 +++
 kernel/sched/deadline.c      |   23 +++++++++++++++++++++--
 kernel/sysctl.c              |   14 ++++++++++++++
 3 files changed, 38 insertions(+), 2 deletions(-)
include/linux/sched/sysctl.h +3 -0
@@ -56,6 +56,9 @@ int sched_proc_update_handler(struct ctl
 extern unsigned int sysctl_sched_rt_period;
 extern int sysctl_sched_rt_runtime;
 
+extern unsigned int sysctl_sched_dl_period_max;
+extern unsigned int sysctl_sched_dl_period_min;
+
 #ifdef CONFIG_UCLAMP_TASK
 extern unsigned int sysctl_sched_uclamp_util_min;
 extern unsigned int sysctl_sched_uclamp_util_max;
kernel/sched/deadline.c +21 -2
@@ -2597,6 +2597,14 @@ void __getparam_dl(struct task_struct *p
 }
 
 /*
+ * Default limits for DL period; on the top end we guard against small util
+ * tasks still getting rediculous long effective runtimes, on the bottom end we
+ * guard against timer DoS.
+ */
+unsigned int sysctl_sched_dl_period_max = 1 << 22; /* ~4 seconds */
+unsigned int sysctl_sched_dl_period_min = 100;     /* 100 us */
+
+/*
  * This function validates the new parameters of a -deadline task.
  * We ask for the deadline not being zero, and greater or equal
  * than the runtime, as well as the period of being zero or
@@ -2608,6 +2616,8 @@ void __getparam_dl(struct task_struct *p
  */
 bool __checkparam_dl(const struct sched_attr *attr)
 {
+	u64 period, max, min;
+
 	/* special dl tasks don't actually use any parameter */
 	if (attr->sched_flags & SCHED_FLAG_SUGOV)
 		return true;
@@ -2631,12 +2641,21 @@ bool __checkparam_dl(const struct sched_
 	    attr->sched_period & (1ULL << 63))
 		return false;
 
+	period = attr->sched_period;
+	if (!period)
+		period = attr->sched_deadline;
+
 	/* runtime <= deadline <= period (if period != 0) */
-	if ((attr->sched_period != 0 &&
-	     attr->sched_period < attr->sched_deadline) ||
+	if (period < attr->sched_deadline ||
 	    attr->sched_deadline < attr->sched_runtime)
 		return false;
 
+	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
+	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
+
+	if (period < min || period > max)
+		return false;
+
 	return true;
 }
 
kernel/sysctl.c +11 -0
@@ -443,6 +443,20 @@ static struct ctl_table kern_table[] = {
 		.proc_handler	= sched_rt_handler,
 	},
 	{
+		.procname	= "sched_deadline_period_max_us",
+		.data		= &sysctl_sched_dl_period_max,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
+	{
+		.procname	= "sched_deadline_period_min_us",
+		.data		= &sysctl_sched_dl_period_min,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
Quoted 8 lines

kernel/sysctl.c static struct ctl_table kern_table[] = { · line 462

Quoted 6 lines
kernel/sysctl.c +3 -0
+		.proc_handler	= proc_dointvec,
+	},
+	{
 		.procname	= "sched_rr_timeslice_ms",
 		.data		= &sysctl_sched_rr_timeslice,
 		.maxlen		= sizeof(int),

I think this never made it upstream. And I believe both me and Daniel were OK with it. Do you recall if any additional change was needed?

Just to confirm, yes I am OK with it!

-- Daniel

merge bot Applied
#22 lore raw

[tip: sched/core] sched/deadline: Impose global limits on sched_attr::sched_period

In reply to Peter Zijlstra
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     b4098bfc5efb1fd7ecf40165132a1283aeea3500
Gitweb:        https://git.kernel.org/tip/b4098bfc5efb1fd7ecf40165132a1283aeea3500
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Fri, 26 Jul 2019 16:54:10 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 15 Jun 2020 14:10:04 +02:00

sched/deadline: Impose global limits on sched_attr::sched_period

---
 include/linux/sched/sysctl.h |  3 +++
 kernel/sched/deadline.c      | 23 +++++++++++++++++++++--
 kernel/sysctl.c              | 14 ++++++++++++++
 3 files changed, 38 insertions(+), 2 deletions(-)
include/linux/sched/sysctl.h +3 -0
index 660ac49..24be30a 100644
@@ -61,6 +61,9 @@ int sched_proc_update_handler(struct ctl_table *table, int write,
 extern unsigned int sysctl_sched_rt_period;
 extern int sysctl_sched_rt_runtime;
 
+extern unsigned int sysctl_sched_dl_period_max;
+extern unsigned int sysctl_sched_dl_period_min;
+
 #ifdef CONFIG_UCLAMP_TASK
 extern unsigned int sysctl_sched_uclamp_util_min;
 extern unsigned int sysctl_sched_uclamp_util_max;
kernel/sched/deadline.c +21 -2
index 504d2f5..f31964a 100644
@@ -2635,6 +2635,14 @@ void __getparam_dl(struct task_struct *p, struct sched_attr *attr)
 }
 
 /*
+ * Default limits for DL period; on the top end we guard against small util
+ * tasks still getting rediculous long effective runtimes, on the bottom end we
+ * guard against timer DoS.
+ */
+unsigned int sysctl_sched_dl_period_max = 1 << 22; /* ~4 seconds */
+unsigned int sysctl_sched_dl_period_min = 100;     /* 100 us */
+
+/*
  * This function validates the new parameters of a -deadline task.
  * We ask for the deadline not being zero, and greater or equal
  * than the runtime, as well as the period of being zero or
@@ -2646,6 +2654,8 @@ void __getparam_dl(struct task_struct *p, struct sched_attr *attr)
  */
 bool __checkparam_dl(const struct sched_attr *attr)
 {
+	u64 period, max, min;
+
 	/* special dl tasks don't actually use any parameter */
 	if (attr->sched_flags & SCHED_FLAG_SUGOV)
 		return true;
@@ -2669,12 +2679,21 @@ bool __checkparam_dl(const struct sched_attr *attr)
 	    attr->sched_period & (1ULL << 63))
 		return false;
 
+	period = attr->sched_period;
+	if (!period)
+		period = attr->sched_deadline;
+
 	/* runtime <= deadline <= period (if period != 0) */
-	if ((attr->sched_period != 0 &&
-	     attr->sched_period < attr->sched_deadline) ||
+	if (period < attr->sched_deadline ||
 	    attr->sched_deadline < attr->sched_runtime)
 		return false;
 
+	max = (u64)READ_ONCE(sysctl_sched_dl_period_max) * NSEC_PER_USEC;
+	min = (u64)READ_ONCE(sysctl_sched_dl_period_min) * NSEC_PER_USEC;
+
+	if (period < min || period > max)
+		return false;
+
 	return true;
 }
 
kernel/sysctl.c +14 -0
index db1ce7a..4aea67d 100644
@@ -1780,6 +1780,20 @@ static struct ctl_table kern_table[] = {
 		.proc_handler	= sched_rt_handler,
 	},
 	{
+		.procname	= "sched_deadline_period_max_us",
+		.data		= &sysctl_sched_dl_period_max,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
+	{
+		.procname	= "sched_deadline_period_min_us",
+		.data		= &sysctl_sched_dl_period_min,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
+	{
 		.procname	= "sched_rr_timeslice_ms",
 		.data		= &sysctl_sched_rr_timeslice,
 		.maxlen		= sizeof(int),
author maintainer
#23 lore raw

RFC PATCH 2/13 stop_machine: Fix stop_cpus_in_progress ordering

In reply to Peter Zijlstra

Make sure the entire for loop has stop_cpus_in_progress set.

---
 kernel/stop_machine.c |    2 ++
 1 file changed, 2 insertions(+)
kernel/stop_machine.c +2 -0
@@ -383,6 +383,7 @@ static bool queue_stop_cpus_work(const s
 	 */
 	preempt_disable();
 	stop_cpus_in_progress = true;
+	barrier();
 	for_each_cpu(cpu, cpumask) {
 		work = &per_cpu(cpu_stopper.stop_work, cpu);
 		work->fn = fn;
@@ -391,6 +392,7 @@ static bool queue_stop_cpus_work(const s
 		if (cpu_stop_queue_work(cpu, work))
 			queued = true;
 	}
+	barrier();
 	stop_cpus_in_progress = false;
 	preempt_enable();
 
Reviewed-by on v1 2/13
#24 lore raw

Re: [RFC][PATCH 02/13] stop_machine: Fix stop_cpus_in_progress ordering

Quoted 39 lines · On Fri, Jul 26, 2019 at 04:54:11PM +0200 Peter Zijlstra wrote:

Make sure the entire for loop has stop_cpus_in_progress set.

---
 kernel/stop_machine.c |    2 ++
 1 file changed, 2 insertions(+)
kernel/stop_machine.c +1 -0
@@ -383,6 +383,7 @@ static bool queue_stop_cpus_work(const s
 	 */
 	preempt_disable();
 	stop_cpus_in_progress = true;
+	barrier();
 	for_each_cpu(cpu, cpumask) {
 		work = &per_cpu(cpu_stopper.stop_work, cpu);
 		work->fn = fn;

kernel/stop_machine.c static bool queue_stop_cpus_work(const s · line 398

Quoted 7 lines
kernel/stop_machine.c +1 -0
@@ -391,6 +392,7 @@ static bool queue_stop_cpus_work(const s
 		if (cpu_stop_queue_work(cpu, work))
 			queued = true;
 	}
+	barrier();
 	stop_cpus_in_progress = false;
 	preempt_enable();
 

This looks good.

#25 lore raw

Re: [RFC][PATCH 02/13] stop_machine: Fix stop_cpus_in_progress ordering

In reply to Peter Zijlstra

kernel/stop_machine.c static bool queue_stop_cpus_work(const s · line 386

Quoted 13 lines · On Fri, 26 Jul 2019 16:54:11 +0200 Peter Zijlstra <[email protected]> wrote:

Make sure the entire for loop has stop_cpus_in_progress set.

 kernel/stop_machine.c |    2 ++
 1 file changed, 2 insertions(+)
kernel/stop_machine.c +1 -0
@@ -383,6 +383,7 @@ static bool queue_stop_cpus_work(const s
 	 */
 	preempt_disable();
 	stop_cpus_in_progress = true;
+	barrier();

Like smp_mb() shouldn't barrier() have a comment to what is being ordered and why?

-- Steve

Quoted 12 lines
kernel/stop_machine.c +1 -0
 	for_each_cpu(cpu, cpumask) {
 		work = &per_cpu(cpu_stopper.stop_work, cpu);
 		work->fn = fn;
@@ -391,6 +392,7 @@ static bool queue_stop_cpus_work(const s
 		if (cpu_stop_queue_work(cpu, work))
 			queued = true;
 	}
+	barrier();
 	stop_cpus_in_progress = false;
 	preempt_enable();
 
author maintainer
#26 lore raw

RFC PATCH 3/13 sched: Fix kerneldoc comment for ia64_set_curr_task

In reply to Peter Zijlstra
---
 kernel/sched/core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -6702,7 +6702,7 @@ struct task_struct *curr_task(int cpu)
 
 #ifdef CONFIG_IA64
 /**
- * set_curr_task - set the current task for a given CPU.
+ * ia64_set_curr_task - set the current task for a given CPU.
  * @cpu: the processor in question.
  * @p: the task pointer to set.
  *
author maintainer
#27 lore raw

RFC PATCH 4/13 sched/{rt,deadline}: Fix set_next_task vs pick_next_task

In reply to Peter Zijlstra

Because pick_next_task() implies set_curr_task() and some of the details haven't matter too much, some of what _should_ be in set_curr_task() ended up in pick_next_task, correct this.

This prepares the way for a pick_next_task() variant that does not affect the current state; allowing remote picking.

---
 kernel/sched/deadline.c |   23 ++++++++++++-----------
 kernel/sched/rt.c       |   27 ++++++++++++++-------------
 2 files changed, 26 insertions(+), 24 deletions(-)
kernel/sched/deadline.c +12 -11
@@ -1694,12 +1694,21 @@ static void start_hrtick_dl(struct rq *r
 }
 #endif
 
-static inline void set_next_task(struct rq *rq, struct task_struct *p)
+static void set_next_task_dl(struct rq *rq, struct task_struct *p)
 {
 	p->se.exec_start = rq_clock_task(rq);
 
 	/* You can't push away the running task */
 	dequeue_pushable_dl_task(rq, p);
+
+	if (hrtick_enabled(rq))
+		start_hrtick_dl(rq, p);
+
+	if (rq->curr->sched_class != &dl_sched_class)
+		update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 0);
+
+	if (rq->curr != p)
+		deadline_queue_push_tasks(rq);
 }
 
 static struct sched_dl_entity *pick_next_dl_entity(struct rq *rq,
@@ -1758,15 +1767,7 @@ pick_next_task_dl(struct rq *rq, struct
 
 	p = dl_task_of(dl_se);
 
-	set_next_task(rq, p);
-
-	if (hrtick_enabled(rq))
-		start_hrtick_dl(rq, p);
-
-	deadline_queue_push_tasks(rq);
-
-	if (rq->curr->sched_class != &dl_sched_class)
-		update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 0);
+	set_next_task_dl(rq, p);
 
 	return p;
 }
@@ -1813,7 +1814,7 @@ static void task_fork_dl(struct task_str
 
 static void set_curr_task_dl(struct rq *rq)
 {
-	set_next_task(rq, rq->curr);
+	set_next_task_dl(rq, rq->curr);
 }
 
 #ifdef CONFIG_SMP
kernel/sched/rt.c +14 -13
@@ -1498,12 +1498,23 @@ static void check_preempt_curr_rt(struct
 #endif
 }
 
-static inline void set_next_task(struct rq *rq, struct task_struct *p)
+static inline void set_next_task_rt(struct rq *rq, struct task_struct *p)
 {
 	p->se.exec_start = rq_clock_task(rq);
 
 	/* The running task is never eligible for pushing */
 	dequeue_pushable_task(rq, p);
+
+	/*
+	 * If prev task was rt, put_prev_task() has already updated the
+	 * utilization. We only care of the case where we start to schedule a
+	 * rt task
+	 */
+	if (rq->curr->sched_class != &rt_sched_class)
+		update_rt_rq_load_avg(rq_clock_pelt(rq), rq, 0);
+
+	if (rq->curr != p)
+		rt_queue_push_tasks(rq);
 }
 
 static struct sched_rt_entity *pick_next_rt_entity(struct rq *rq,
@@ -1577,17 +1588,7 @@ pick_next_task_rt(struct rq *rq, struct
 
 	p = _pick_next_task_rt(rq);
 
-	set_next_task(rq, p);
-
-	rt_queue_push_tasks(rq);
-
-	/*
-	 * If prev task was rt, put_prev_task() has already updated the
-	 * utilization. We only care of the case where we start to schedule a
-	 * rt task
-	 */
-	if (rq->curr->sched_class != &rt_sched_class)
-		update_rt_rq_load_avg(rq_clock_pelt(rq), rq, 0);
+	set_next_task_rt(rq, p);
 
 	return p;
 }
@@ -2356,7 +2357,7 @@ static void task_tick_rt(struct rq *rq,
 
 static void set_curr_task_rt(struct rq *rq)
 {
-	set_next_task(rq, rq->curr);
+	set_next_task_rt(rq, rq->curr);
 }
 
 static unsigned int get_rr_interval_rt(struct rq *rq, struct task_struct *task)
maintainer
#28 lore raw

Re: [RFC][PATCH 04/13] sched/{rt,deadline}: Fix set_next_task vs pick_next_task

Hi,

Quoted 25 lines · Peter Zijlstra

Because pick_next_task() implies set_curr_task() and some of the details haven't matter too much, some of what _should_ be in set_curr_task() ended up in pick_next_task, correct this.

This prepares the way for a pick_next_task() variant that does not affect the current state; allowing remote picking.

---
 kernel/sched/deadline.c |   23 ++++++++++++-----------
 kernel/sched/rt.c       |   27 ++++++++++++++-------------
 2 files changed, 26 insertions(+), 24 deletions(-)
kernel/sched/deadline.c +1 -1
@@ -1694,12 +1694,21 @@ static void start_hrtick_dl(struct rq *r
 }
 #endif
 
-static inline void set_next_task(struct rq *rq, struct task_struct *p)
+static void set_next_task_dl(struct rq *rq, struct task_struct *p)
 {
 	p->se.exec_start = rq_clock_task(rq);
 
 	/* You can't push away the running task */

kernel/sched/deadline.c static void start_hrtick_dl(struct rq *r · line 1711

Quoted 10 lines
kernel/sched/deadline.c +9 -0
 	dequeue_pushable_dl_task(rq, p);
+
+	if (hrtick_enabled(rq))
+		start_hrtick_dl(rq, p);
+
+	if (rq->curr->sched_class != &dl_sched_class)
+		update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 0);
+
+	if (rq->curr != p)
+		deadline_queue_push_tasks(rq);

It's a minor thing, but I was wondering why you added the check on curr. deadline_queue_push_tasks() already checks if are there pushable tasks, plus curr can still be of a different class at this point?

Thanks,

Juri

author maintainer
#29 lore raw

Re: [RFC][PATCH 04/13] sched/{rt,deadline}: Fix set_next_task vs pick_next_task

Quoted 27 lines · Juri Lelli

Hi,

Quoted 24 lines · Peter Zijlstra

Because pick_next_task() implies set_curr_task() and some of the details haven't matter too much, some of what _should_ be in set_curr_task() ended up in pick_next_task, correct this.

This prepares the way for a pick_next_task() variant that does not affect the current state; allowing remote picking.

---
 kernel/sched/deadline.c |   23 ++++++++++++-----------
 kernel/sched/rt.c       |   27 ++++++++++++++-------------
 2 files changed, 26 insertions(+), 24 deletions(-)
kernel/sched/deadline.c +1 -1
@@ -1694,12 +1694,21 @@ static void start_hrtick_dl(struct rq *r
 }
 #endif
 
-static inline void set_next_task(struct rq *rq, struct task_struct *p)
+static void set_next_task_dl(struct rq *rq, struct task_struct *p)
 {
 	p->se.exec_start = rq_clock_task(rq);
 
Quoted 15 lines

kernel/sched/deadline.c static void start_hrtick_dl(struct rq *r · line 1711

Quoted 11 lines
kernel/sched/deadline.c +9 -0
 	/* You can't push away the running task */
 	dequeue_pushable_dl_task(rq, p);
+
+	if (hrtick_enabled(rq))
+		start_hrtick_dl(rq, p);
+
+	if (rq->curr->sched_class != &dl_sched_class)
+		update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 0);
+
+	if (rq->curr != p)
+		deadline_queue_push_tasks(rq);

It's a minor thing, but I was wondering why you added the check on curr. deadline_queue_push_tasks() already checks if are there pushable tasks, plus curr can still be of a different class at this point?

Hmm, so by moving that code into set_next_task() it is exposed to the:

  if (queued)
    deuque_task();
  if (running)
    put_prev_task();

/* do stuff */

  if (queued)
    enqueue_task();
  if (running)
    set_next_task();

patter from core.c; and in that case nothing changes. That said; I might've gotten it wrong.

maintainer
#30 lore raw

Re: [RFC][PATCH 04/13] sched/{rt,deadline}: Fix set_next_task vs pick_next_task

Quoted 53 lines · Peter Zijlstra
Quoted 27 lines · Juri Lelli

Hi,

Quoted 24 lines · Peter Zijlstra

Because pick_next_task() implies set_curr_task() and some of the details haven't matter too much, some of what _should_ be in set_curr_task() ended up in pick_next_task, correct this.

This prepares the way for a pick_next_task() variant that does not affect the current state; allowing remote picking.

---
 kernel/sched/deadline.c |   23 ++++++++++++-----------
 kernel/sched/rt.c       |   27 ++++++++++++++-------------
 2 files changed, 26 insertions(+), 24 deletions(-)
kernel/sched/deadline.c +1 -1
@@ -1694,12 +1694,21 @@ static void start_hrtick_dl(struct rq *r
 }
 #endif
 
-static inline void set_next_task(struct rq *rq, struct task_struct *p)
+static void set_next_task_dl(struct rq *rq, struct task_struct *p)
 {
 	p->se.exec_start = rq_clock_task(rq);
 
Quoted 15 lines

kernel/sched/deadline.c static void start_hrtick_dl(struct rq *r · line 1711

Quoted 11 lines
kernel/sched/deadline.c +9 -0
 	/* You can't push away the running task */
 	dequeue_pushable_dl_task(rq, p);
+
+	if (hrtick_enabled(rq))
+		start_hrtick_dl(rq, p);
+
+	if (rq->curr->sched_class != &dl_sched_class)
+		update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 0);
+
+	if (rq->curr != p)
+		deadline_queue_push_tasks(rq);

It's a minor thing, but I was wondering why you added the check on curr. deadline_queue_push_tasks() already checks if are there pushable tasks, plus curr can still be of a different class at this point?

Hmm, so by moving that code into set_next_task() it is exposed to the:

  if (queued)
    deuque_task();
  if (running)
    put_prev_task();

/* do stuff */

Quoted 7 lines
  if (queued)
    enqueue_task();
  if (running)
    set_next_task();

patter from core.c; and in that case nothing changes. That said; I might've gotten it wrong.

Right. But, I was wondering about the __schedule()->pick_next_task() case, where, say, prev (rq->curr) is RT/CFS and next (p) is DEADLINE.

author maintainer
#31 lore raw

Re: [RFC][PATCH 04/13] sched/{rt,deadline}: Fix set_next_task vs pick_next_task

Quoted 59 lines · Juri Lelli
Quoted 58 lines · Peter Zijlstra
Quoted 27 lines · Juri Lelli

Hi,

Quoted 24 lines · Peter Zijlstra

Because pick_next_task() implies set_curr_task() and some of the details haven't matter too much, some of what _should_ be in set_curr_task() ended up in pick_next_task, correct this.

This prepares the way for a pick_next_task() variant that does not affect the current state; allowing remote picking.

---
 kernel/sched/deadline.c |   23 ++++++++++++-----------
 kernel/sched/rt.c       |   27 ++++++++++++++-------------
 2 files changed, 26 insertions(+), 24 deletions(-)
kernel/sched/deadline.c +1 -1
@@ -1694,12 +1694,21 @@ static void start_hrtick_dl(struct rq *r
 }
 #endif
 
-static inline void set_next_task(struct rq *rq, struct task_struct *p)
+static void set_next_task_dl(struct rq *rq, struct task_struct *p)
 {
 	p->se.exec_start = rq_clock_task(rq);
 
Quoted 15 lines

kernel/sched/deadline.c static void start_hrtick_dl(struct rq *r · line 1711

Quoted 11 lines
kernel/sched/deadline.c +9 -0
 	/* You can't push away the running task */
 	dequeue_pushable_dl_task(rq, p);
+
+	if (hrtick_enabled(rq))
+		start_hrtick_dl(rq, p);
+
+	if (rq->curr->sched_class != &dl_sched_class)
+		update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 0);
+
+	if (rq->curr != p)
+		deadline_queue_push_tasks(rq);

It's a minor thing, but I was wondering why you added the check on curr. deadline_queue_push_tasks() already checks if are there pushable tasks, plus curr can still be of a different class at this point?

Hmm, so by moving that code into set_next_task() it is exposed to the:

  if (queued)
    deuque_task();
  if (running)
    put_prev_task();

/* do stuff */

  if (queued)
    enqueue_task();
  if (running)
    set_next_task();
Quoted 5 lines
Quoted 2 lines

patter from core.c; and in that case nothing changes. That said; I might've gotten it wrong.

Right. But, I was wondering about the __schedule()->pick_next_task() case, where, say, prev (rq->curr) is RT/CFS and next (p) is DEADLINE.

So we do pick_next_task() first and then set rq->curr (obviously). So the first set_next_task() will see rq->curr != p and we'll do the push balance stuff.

Then the above pattern will always see rq->curr == p and we'll not trigger push balancing.

Now, looking at it, this also doesn't do push balancing when we re-select the same task, even though we really should be doing it. So I suppose not adding the condition, and always doing the push balance, while wasteful, is not wrong.

Hmm?

maintainer
#32 lore raw

Re: [RFC][PATCH 04/13] sched/{rt,deadline}: Fix set_next_task vs pick_next_task

Quoted 70 lines · Peter Zijlstra
Quoted 59 lines · Juri Lelli
Quoted 58 lines · Peter Zijlstra
Quoted 27 lines · Juri Lelli

Hi,

Quoted 24 lines · Peter Zijlstra

Because pick_next_task() implies set_curr_task() and some of the details haven't matter too much, some of what _should_ be in set_curr_task() ended up in pick_next_task, correct this.

This prepares the way for a pick_next_task() variant that does not affect the current state; allowing remote picking.

---
 kernel/sched/deadline.c |   23 ++++++++++++-----------
 kernel/sched/rt.c       |   27 ++++++++++++++-------------
 2 files changed, 26 insertions(+), 24 deletions(-)
kernel/sched/deadline.c +1 -1
@@ -1694,12 +1694,21 @@ static void start_hrtick_dl(struct rq *r
 }
 #endif
 
-static inline void set_next_task(struct rq *rq, struct task_struct *p)
+static void set_next_task_dl(struct rq *rq, struct task_struct *p)
 {
 	p->se.exec_start = rq_clock_task(rq);
 
Quoted 15 lines

kernel/sched/deadline.c static void start_hrtick_dl(struct rq *r · line 1711

Quoted 11 lines
kernel/sched/deadline.c +9 -0
 	/* You can't push away the running task */
 	dequeue_pushable_dl_task(rq, p);
+
+	if (hrtick_enabled(rq))
+		start_hrtick_dl(rq, p);
+
+	if (rq->curr->sched_class != &dl_sched_class)
+		update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 0);
+
+	if (rq->curr != p)
+		deadline_queue_push_tasks(rq);

It's a minor thing, but I was wondering why you added the check on curr. deadline_queue_push_tasks() already checks if are there pushable tasks, plus curr can still be of a different class at this point?

Hmm, so by moving that code into set_next_task() it is exposed to the:

  if (queued)
    deuque_task();
  if (running)
    put_prev_task();

/* do stuff */

  if (queued)
    enqueue_task();
  if (running)
    set_next_task();
Quoted 5 lines
Quoted 2 lines

patter from core.c; and in that case nothing changes. That said; I might've gotten it wrong.

Right. But, I was wondering about the __schedule()->pick_next_task() case, where, say, prev (rq->curr) is RT/CFS and next (p) is DEADLINE.

So we do pick_next_task() first and then set rq->curr (obviously). So the first set_next_task() will see rq->curr != p and we'll do the push balance stuff.

Quoted 7 lines

Then the above pattern will always see rq->curr == p and we'll not trigger push balancing.

Now, looking at it, this also doesn't do push balancing when we re-select the same task, even though we really should be doing it. So I suppose not adding the condition, and always doing the push balance, while wasteful, is not wrong.

Right, also because deadline_queue_push_tasks() already checks if there are tasks to potentially push around before queuing the balance callback.

author maintainer
#33 lore raw

Re: [RFC][PATCH 04/13] sched/{rt,deadline}: Fix set_next_task vs pick_next_task

Quoted 10 lines · Juri Lelli
Quoted 4 lines · Peter Zijlstra

Now, looking at it, this also doesn't do push balancing when we re-select the same task, even though we really should be doing it. So I suppose not adding the condition, and always doing the push balance, while wasteful, is not wrong.

Right, also because deadline_queue_push_tasks() already checks if there are tasks to potentially push around before queuing the balance callback.

Yes, but in the overloaded case, where there is always a task to push, but nowhere to push it to, we can waste a 'lot' of time looking for naught in case of extra pushes.

So in that regard the check you reference is not sufficient.

Anyway, let me change this for now.

author maintainer
#34 lore raw

RFC PATCH 5/13 sched: Add task_struct pointer to sched_class::set_curr_task

In reply to Peter Zijlstra

In preparation of further separating pick_next_task() and set_curr_task() we have to pass the actual task into it, while there, rename the thing to better pair with put_prev_task().

---
 kernel/sched/core.c      |   12 ++++++------
 kernel/sched/deadline.c  |    7 +------
 kernel/sched/fair.c      |   17 ++++++++++++++---
 kernel/sched/idle.c      |   27 +++++++++++++++------------
 kernel/sched/rt.c        |    7 +------
 kernel/sched/sched.h     |    8 +++++---
 kernel/sched/stop_task.c |   17 +++++++----------
 7 files changed, 49 insertions(+), 46 deletions(-)
kernel/sched/core.c +6 -6
@@ -1494,7 +1494,7 @@ void do_set_cpus_allowed(struct task_str
 	if (queued)
 		enqueue_task(rq, p, ENQUEUE_RESTORE | ENQUEUE_NOCLOCK);
 	if (running)
-		set_curr_task(rq, p);
+		set_next_task(rq, p);
 }
 
 /*
@@ -4273,7 +4273,7 @@ void rt_mutex_setprio(struct task_struct
 	if (queued)
 		enqueue_task(rq, p, queue_flag);
 	if (running)
-		set_curr_task(rq, p);
+		set_next_task(rq, p);
 
 	check_class_changed(rq, p, prev_class, oldprio);
 out_unlock:
@@ -4340,7 +4340,7 @@ void set_user_nice(struct task_struct *p
 			resched_curr(rq);
 	}
 	if (running)
-		set_curr_task(rq, p);
+		set_next_task(rq, p);
 out_unlock:
 	task_rq_unlock(rq, p, &rf);
 }
@@ -4783,7 +4783,7 @@ static int __sched_setscheduler(struct t
 		enqueue_task(rq, p, queue_flags);
 	}
 	if (running)
-		set_curr_task(rq, p);
+		set_next_task(rq, p);
 
 	check_class_changed(rq, p, prev_class, oldprio);
 
@@ -5972,7 +5972,7 @@ void sched_setnuma(struct task_struct *p
 	if (queued)
 		enqueue_task(rq, p, ENQUEUE_RESTORE | ENQUEUE_NOCLOCK);
 	if (running)
-		set_curr_task(rq, p);
+		set_next_task(rq, p);
 	task_rq_unlock(rq, p, &rf);
 }
 #endif /* CONFIG_NUMA_BALANCING */
@@ -6853,7 +6853,7 @@ void sched_move_task(struct task_struct
 	if (queued)
 		enqueue_task(rq, tsk, queue_flags);
 	if (running)
-		set_curr_task(rq, tsk);
+		set_next_task(rq, tsk);
 
 	task_rq_unlock(rq, tsk, &rf);
 }
kernel/sched/deadline.c +1 -6
@@ -1812,11 +1812,6 @@ static void task_fork_dl(struct task_str
 	 */
 }
 
-static void set_curr_task_dl(struct rq *rq)
-{
-	set_next_task_dl(rq, rq->curr);
-}
-
 #ifdef CONFIG_SMP
 
 /* Only try algorithms three times */
@@ -2404,6 +2399,7 @@ const struct sched_class dl_sched_class
 
 	.pick_next_task		= pick_next_task_dl,
 	.put_prev_task		= put_prev_task_dl,
+	.set_next_task		= set_next_task_dl,
 
 #ifdef CONFIG_SMP
 	.select_task_rq		= select_task_rq_dl,
@@ -2414,7 +2410,6 @@ const struct sched_class dl_sched_class
 	.task_woken		= task_woken_dl,
 #endif
 
-	.set_curr_task		= set_curr_task_dl,
 	.task_tick		= task_tick_dl,
 	.task_fork              = task_fork_dl,
 
kernel/sched/fair.c +14 -3
@@ -10121,9 +10121,19 @@ static void switched_to_fair(struct rq *
  * This routine is mostly called to set cfs_rq->curr field when a task
  * migrates between groups/classes.
  */
-static void set_curr_task_fair(struct rq *rq)
+static void set_next_task_fair(struct rq *rq, struct task_struct *p)
 {
-	struct sched_entity *se = &rq->curr->se;
+	struct sched_entity *se = &p->se;
+
+#ifdef CONFIG_SMP
+	if (task_on_rq_queued(p)) {
+		/*
+		 * Move the next running task to the front of the list, so our
+		 * cfs_tasks list becomes MRU one.
+		 */
+		list_move(&se->group_node, &rq->cfs_tasks);
+	}
+#endif
 
 	for_each_sched_entity(se) {
 		struct cfs_rq *cfs_rq = cfs_rq_of(se);
@@ -10394,7 +10404,9 @@ const struct sched_class fair_sched_clas
 	.check_preempt_curr	= check_preempt_wakeup,
 
 	.pick_next_task		= pick_next_task_fair,
+
 	.put_prev_task		= put_prev_task_fair,
+	.set_next_task          = set_next_task_fair,
 
 #ifdef CONFIG_SMP
 	.select_task_rq		= select_task_rq_fair,
@@ -10407,7 +10419,6 @@ const struct sched_class fair_sched_clas
 	.set_cpus_allowed	= set_cpus_allowed_common,
 #endif
 
-	.set_curr_task          = set_curr_task_fair,
 	.task_tick		= task_tick_fair,
 	.task_fork		= task_fork_fair,
 
kernel/sched/idle.c +15 -12
@@ -374,14 +374,25 @@ static void check_preempt_curr_idle(stru
 	resched_curr(rq);
 }
 
+static void put_prev_task_idle(struct rq *rq, struct task_struct *prev)
+{
+}
+
+static void set_next_task_idle(struct rq *rq, struct task_struct *next)
+{
+	update_idle_core(rq);
+	schedstat_inc(rq->sched_goidle);
+}
+
 static struct task_struct *
 pick_next_task_idle(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
+	struct task_struct *next = rq->idle;
+
 	put_prev_task(rq, prev);
-	update_idle_core(rq);
-	schedstat_inc(rq->sched_goidle);
+	set_next_task_idle(rq, next);
 
-	return rq->idle;
+	return next;
 }
 
 /*
@@ -397,10 +408,6 @@ dequeue_task_idle(struct rq *rq, struct
 	raw_spin_lock_irq(&rq->lock);
 }
 
-static void put_prev_task_idle(struct rq *rq, struct task_struct *prev)
-{
-}
-
 /*
  * scheduler tick hitting a task of our scheduling class.
  *
@@ -413,10 +420,6 @@ static void task_tick_idle(struct rq *rq
 {
 }
 
-static void set_curr_task_idle(struct rq *rq)
-{
-}
-
 static void switched_to_idle(struct rq *rq, struct task_struct *p)
 {
 	BUG();
@@ -451,13 +454,13 @@ const struct sched_class idle_sched_clas
 
 	.pick_next_task		= pick_next_task_idle,
 	.put_prev_task		= put_prev_task_idle,
+	.set_next_task          = set_next_task_idle,
 
 #ifdef CONFIG_SMP
 	.select_task_rq		= select_task_rq_idle,
 	.set_cpus_allowed	= set_cpus_allowed_common,
 #endif
 
-	.set_curr_task          = set_curr_task_idle,
 	.task_tick		= task_tick_idle,
 
 	.get_rr_interval	= get_rr_interval_idle,
kernel/sched/rt.c +1 -6
@@ -2355,11 +2355,6 @@ static void task_tick_rt(struct rq *rq,
 	}
 }
 
-static void set_curr_task_rt(struct rq *rq)
-{
-	set_next_task_rt(rq, rq->curr);
-}
-
 static unsigned int get_rr_interval_rt(struct rq *rq, struct task_struct *task)
 {
 	/*
@@ -2381,6 +2376,7 @@ const struct sched_class rt_sched_class
 
 	.pick_next_task		= pick_next_task_rt,
 	.put_prev_task		= put_prev_task_rt,
+	.set_next_task          = set_next_task_rt,
 
 #ifdef CONFIG_SMP
 	.select_task_rq		= select_task_rq_rt,
@@ -2392,7 +2388,6 @@ const struct sched_class rt_sched_class
 	.switched_from		= switched_from_rt,
 #endif
 
-	.set_curr_task          = set_curr_task_rt,
 	.task_tick		= task_tick_rt,
 
 	.get_rr_interval	= get_rr_interval_rt,
kernel/sched/sched.h +5 -3
@@ -1711,6 +1711,7 @@ struct sched_class {
 					       struct task_struct *prev,
 					       struct rq_flags *rf);
 	void (*put_prev_task)(struct rq *rq, struct task_struct *p);
+	void (*set_next_task)(struct rq *rq, struct task_struct *p);
 
 #ifdef CONFIG_SMP
 	int  (*select_task_rq)(struct task_struct *p, int task_cpu, int sd_flag, int flags);
@@ -1725,7 +1726,6 @@ struct sched_class {
 	void (*rq_offline)(struct rq *rq);
 #endif
 
-	void (*set_curr_task)(struct rq *rq);
 	void (*task_tick)(struct rq *rq, struct task_struct *p, int queued);
 	void (*task_fork)(struct task_struct *p);
 	void (*task_dead)(struct task_struct *p);
@@ -1755,12 +1755,14 @@ struct sched_class {
 
 static inline void put_prev_task(struct rq *rq, struct task_struct *prev)
 {
+	WARN_ON_ONCE(rq->curr != prev);
 	prev->sched_class->put_prev_task(rq, prev);
 }
 
-static inline void set_curr_task(struct rq *rq, struct task_struct *curr)
+static inline void set_next_task(struct rq *rq, struct task_struct *next)
 {
-	curr->sched_class->set_curr_task(rq);
+	WARN_ON_ONCE(rq->curr != next);
+	next->sched_class->set_next_task(rq, next);
 }
 
 #ifdef CONFIG_SMP
kernel/sched/stop_task.c +7 -10
@@ -23,6 +23,11 @@ check_preempt_curr_stop(struct rq *rq, s
 	/* we're never preempted */
 }
 
+static void set_next_task_stop(struct rq *rq, struct task_struct *stop)
+{
+	stop->se.exec_start = rq_clock_task(rq);
+}
+
 static struct task_struct *
 pick_next_task_stop(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
@@ -32,8 +37,7 @@ pick_next_task_stop(struct rq *rq, struc
 		return NULL;
 
 	put_prev_task(rq, prev);
-
-	stop->se.exec_start = rq_clock_task(rq);
+	set_next_task_stop(rq, stop);
 
 	return stop;
 }
@@ -86,13 +90,6 @@ static void task_tick_stop(struct rq *rq
 {
 }
 
-static void set_curr_task_stop(struct rq *rq)
-{
-	struct task_struct *stop = rq->stop;
-
-	stop->se.exec_start = rq_clock_task(rq);
-}
-
 static void switched_to_stop(struct rq *rq, struct task_struct *p)
 {
 	BUG(); /* its impossible to change to this class */
@@ -128,13 +125,13 @@ const struct sched_class stop_sched_clas
 
 	.pick_next_task		= pick_next_task_stop,
 	.put_prev_task		= put_prev_task_stop,
+	.set_next_task          = set_next_task_stop,
 
 #ifdef CONFIG_SMP
 	.select_task_rq		= select_task_rq_stop,
 	.set_cpus_allowed	= set_cpus_allowed_common,
 #endif
 
-	.set_curr_task          = set_curr_task_stop,
 	.task_tick		= task_tick_stop,
 
 	.get_rr_interval	= get_rr_interval_stop,
author maintainer
#35 lore raw

RFC PATCH 6/13 sched/fair: Export newidle_balance()

In reply to Peter Zijlstra

For pick_next_task_fair() it is the newidle balance that requires dropping the rq->lock; provided we do put_prev_task() early, we can also detect the condition for doing newidle early.

---
 kernel/sched/fair.c  |   18 ++++++++----------
 kernel/sched/sched.h |    4 ++++
 2 files changed, 12 insertions(+), 10 deletions(-)
kernel/sched/fair.c +8 -10
@@ -3635,8 +3635,6 @@ static inline unsigned long cfs_rq_load_
 	return cfs_rq->avg.load_avg;
 }
 
-static int idle_balance(struct rq *this_rq, struct rq_flags *rf);
-
 static inline unsigned long task_util(struct task_struct *p)
 {
 	return READ_ONCE(p->se.avg.util_avg);
@@ -6848,11 +6846,10 @@ done: __maybe_unused;
 	return p;
 
 idle:
-	update_misfit_status(NULL, rq);
-	new_tasks = idle_balance(rq, rf);
+	new_tasks = newidle_balance(rq, rf);
 
 	/*
-	 * Because idle_balance() releases (and re-acquires) rq->lock, it is
+	 * Because newidle_balance() releases (and re-acquires) rq->lock, it is
 	 * possible for any higher priority task to appear. In that case we
 	 * must re-start the pick_next_entity() loop.
 	 */
@@ -9016,10 +9013,10 @@ static int load_balance(int this_cpu, st
 	ld_moved = 0;
 
 	/*
-	 * idle_balance() disregards balance intervals, so we could repeatedly
-	 * reach this code, which would lead to balance_interval skyrocketting
-	 * in a short amount of time. Skip the balance_interval increase logic
-	 * to avoid that.
+	 * newidle_balance() disregards balance intervals, so we could
+	 * repeatedly reach this code, which would lead to balance_interval
+	 * skyrocketting in a short amount of time. Skip the balance_interval
+	 * increase logic to avoid that.
 	 */
 	if (env.idle == CPU_NEWLY_IDLE)
 		goto out;
@@ -9729,7 +9726,7 @@ static inline void nohz_newidle_balance(
  * idle_balance is called by schedule() if this_cpu is about to become
  * idle. Attempts to pull tasks from other CPUs.
  */
-static int idle_balance(struct rq *this_rq, struct rq_flags *rf)
+int newidle_balance(struct rq *this_rq, struct rq_flags *rf)
 {
 	unsigned long next_balance = jiffies + HZ;
 	int this_cpu = this_rq->cpu;
@@ -9737,6 +9734,7 @@ static int idle_balance(struct rq *this_
 	int pulled_task = 0;
 	u64 curr_cost = 0;
 
+	update_misfit_status(NULL, this_rq);
 	/*
 	 * We must set idle_stamp _before_ calling idle_balance(), such that we
 	 * measure the duration of idle_balance() as idle time.
kernel/sched/sched.h +4 -0
@@ -1449,10 +1449,14 @@ static inline void unregister_sched_doma
 }
 #endif
 
+extern int newidle_balance(struct rq *this_rq, struct rq_flags *rf);
+
 #else
 
 static inline void sched_ttwu_pending(void) { }
 
+static inline int newidle_balance(struct rq *this_rq, struct rq_flags *rf) { return 0; }
+
 #endif /* CONFIG_SMP */
 
 #include "stats.h"
author maintainer
#36 lore raw

RFC PATCH 7/13 sched: Allow put_prev_task() to drop rq->lock

In reply to Peter Zijlstra

Currently the pick_next_task() loop is convoluted and ugly because of how it can drop the rq->lock and needs to restart the picking.

For the RT/Deadline classes, it is put_prev_task() where we do balancing, and we could do this before the picking loop. Make this possible.

---
 kernel/sched/core.c      |    2 +-
 kernel/sched/deadline.c  |   14 +++++++++++++-
 kernel/sched/fair.c      |    2 +-
 kernel/sched/idle.c      |    2 +-
 kernel/sched/rt.c        |   14 +++++++++++++-
 kernel/sched/sched.h     |    4 ++--
 kernel/sched/stop_task.c |    2 +-
 7 files changed, 32 insertions(+), 8 deletions(-)
kernel/sched/core.c +1 -1
@@ -6012,7 +6012,7 @@ static void calc_load_migrate(struct rq
 		atomic_long_add(delta, &calc_load_tasks);
 }
 
-static void put_prev_task_fake(struct rq *rq, struct task_struct *prev)
+static void put_prev_task_fake(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
 }
 
kernel/sched/deadline.c +13 -1
@@ -1772,13 +1772,25 @@ pick_next_task_dl(struct rq *rq, struct
 	return p;
 }
 
-static void put_prev_task_dl(struct rq *rq, struct task_struct *p)
+static void put_prev_task_dl(struct rq *rq, struct task_struct *p, struct rq_flags *rf)
 {
 	update_curr_dl(rq);
 
 	update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 1);
 	if (on_dl_rq(&p->dl) && p->nr_cpus_allowed > 1)
 		enqueue_pushable_dl_task(rq, p);
+
+	if (rf && !on_dl_rq(&p->dl) && need_pull_dl_task(rq, p)) {
+		/*
+		 * This is OK, because current is on_cpu, which avoids it being
+		 * picked for load-balance and preemption/IRQs are still
+		 * disabled avoiding further scheduler activity on it and we've
+		 * not yet started the picking loop.
+		 */
+		rq_unpin_lock(rq, rf);
+		pull_dl_task(rq);
+		rq_repin_lock(rq, rf);
+	}
 }
 
 /*
kernel/sched/fair.c +1 -1
@@ -6871,7 +6871,7 @@ done: __maybe_unused;
 /*
  * Account for a descheduled task:
  */
-static void put_prev_task_fair(struct rq *rq, struct task_struct *prev)
+static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
 	struct sched_entity *se = &prev->se;
 	struct cfs_rq *cfs_rq;
kernel/sched/idle.c +1 -1
@@ -374,7 +374,7 @@ static void check_preempt_curr_idle(stru
 	resched_curr(rq);
 }
 
-static void put_prev_task_idle(struct rq *rq, struct task_struct *prev)
+static void put_prev_task_idle(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
 }
 
kernel/sched/rt.c +13 -1
@@ -1593,7 +1593,7 @@ pick_next_task_rt(struct rq *rq, struct
 	return p;
 }
 
-static void put_prev_task_rt(struct rq *rq, struct task_struct *p)
+static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct rq_flags *rf)
 {
 	update_curr_rt(rq);
 
@@ -1605,6 +1605,18 @@ static void put_prev_task_rt(struct rq *
 	 */
 	if (on_rt_rq(&p->rt) && p->nr_cpus_allowed > 1)
 		enqueue_pushable_task(rq, p);
+
+	if (rf && !on_rt_rq(&p->rt) && need_pull_rt_task(rq, p)) {
+		/*
+		 * This is OK, because current is on_cpu, which avoids it being
+		 * picked for load-balance and preemption/IRQs are still
+		 * disabled avoiding further scheduler activity on it and we've
+		 * not yet started the picking loop.
+		 */
+		rq_unpin_lock(rq, rf);
+		pull_rt_task(rq);
+		rq_repin_lock(rq, rf);
+	}
 }
 
 #ifdef CONFIG_SMP
kernel/sched/sched.h +2 -2
@@ -1714,7 +1714,7 @@ struct sched_class {
 	struct task_struct * (*pick_next_task)(struct rq *rq,
 					       struct task_struct *prev,
 					       struct rq_flags *rf);
-	void (*put_prev_task)(struct rq *rq, struct task_struct *p);
+	void (*put_prev_task)(struct rq *rq, struct task_struct *p, struct rq_flags *rf);
 	void (*set_next_task)(struct rq *rq, struct task_struct *p);
 
 #ifdef CONFIG_SMP
@@ -1760,7 +1760,7 @@ struct sched_class {
 static inline void put_prev_task(struct rq *rq, struct task_struct *prev)
 {
 	WARN_ON_ONCE(rq->curr != prev);
-	prev->sched_class->put_prev_task(rq, prev);
+	prev->sched_class->put_prev_task(rq, prev, NULL);
 }
 
 static inline void set_next_task(struct rq *rq, struct task_struct *next)
kernel/sched/stop_task.c +1 -1
@@ -59,7 +59,7 @@ static void yield_task_stop(struct rq *r
 	BUG(); /* the stop task should never yield, its pointless. */
 }
 
-static void put_prev_task_stop(struct rq *rq, struct task_struct *prev)
+static void put_prev_task_stop(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
 	struct task_struct *curr = rq->curr;
 	u64 delta_exec;
author maintainer
#37 lore raw

RFC PATCH 8/13 sched: Rework pick_next_task() slow-path

In reply to Peter Zijlstra

Avoid the RETRY_TASK case in the pick_next_task() slow path.

By doing the put_prev_task() early, we get the rt/deadline pull done, and by testing rq->nr_running we know if we need newidle_balance().

This then gives a stable state to pick a task from.

Since the fast-path is fair only; it means the other classes will always have pick_next_task(.prev=NULL, .rf=NULL) and we can simplify.

---
 kernel/sched/core.c      |   19 ++++++++++++-------
 kernel/sched/deadline.c  |   30 ++----------------------------
 kernel/sched/fair.c      |    9 ++++++---
 kernel/sched/idle.c      |    4 +++-
 kernel/sched/rt.c        |   29 +----------------------------
 kernel/sched/sched.h     |   13 ++++++++-----
 kernel/sched/stop_task.c |    3 ++-
 7 files changed, 34 insertions(+), 73 deletions(-)
kernel/sched/core.c +12 -7
@@ -3739,7 +3739,7 @@ pick_next_task(struct rq *rq, struct tas
 
 		p = fair_sched_class.pick_next_task(rq, prev, rf);
 		if (unlikely(p == RETRY_TASK))
-			goto again;
+			goto restart;
 
 		/* Assumes fair_sched_class->next == idle_sched_class */
 		if (unlikely(!p))
@@ -3748,14 +3748,19 @@ pick_next_task(struct rq *rq, struct tas
 		return p;
 	}
 
-again:
+restart:
+	/*
+	 * Ensure that we put DL/RT tasks before the pick loop, such that they
+	 * can PULL higher prio tasks when we lower the RQ 'priority'.
+	 */
+	prev->sched_class->put_prev_task(rq, prev, rf);
+	if (!rq->nr_running)
+		newidle_balance(rq, rf);
+
 	for_each_class(class) {
-		p = class->pick_next_task(rq, prev, rf);
-		if (p) {
-			if (unlikely(p == RETRY_TASK))
-				goto again;
+		p = class->pick_next_task(rq, NULL, NULL);
+		if (p)
 			return p;
-		}
 	}
 
 	/* The idle class should always have a runnable task: */
kernel/sched/deadline.c +2 -28
@@ -1729,39 +1729,13 @@ pick_next_task_dl(struct rq *rq, struct
 	struct task_struct *p;
 	struct dl_rq *dl_rq;
 
-	dl_rq = &rq->dl;
-
-	if (need_pull_dl_task(rq, prev)) {
-		/*
-		 * This is OK, because current is on_cpu, which avoids it being
-		 * picked for load-balance and preemption/IRQs are still
-		 * disabled avoiding further scheduler activity on it and we're
-		 * being very careful to re-start the picking loop.
-		 */
-		rq_unpin_lock(rq, rf);
-		pull_dl_task(rq);
-		rq_repin_lock(rq, rf);
-		/*
-		 * pull_dl_task() can drop (and re-acquire) rq->lock; this
-		 * means a stop task can slip in, in which case we need to
-		 * re-start task selection.
-		 */
-		if (rq->stop && task_on_rq_queued(rq->stop))
-			return RETRY_TASK;
-	}
+	WARN_ON_ONCE(prev || rf);
 
-	/*
-	 * When prev is DL, we may throttle it in put_prev_task().
-	 * So, we update time before we check for dl_nr_running.
-	 */
-	if (prev->sched_class == &dl_sched_class)
-		update_curr_dl(rq);
+	dl_rq = &rq->dl;
 
 	if (unlikely(!dl_rq->dl_nr_running))
 		return NULL;
 
-	put_prev_task(rq, prev);
-
 	dl_se = pick_next_dl_entity(rq, dl_rq);
 	BUG_ON(!dl_se);
 
kernel/sched/fair.c +6 -3
@@ -6740,7 +6740,7 @@ pick_next_task_fair(struct rq *rq, struc
 		goto idle;
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-	if (prev->sched_class != &fair_sched_class)
+	if (!prev || prev->sched_class != &fair_sched_class)
 		goto simple;
 
 	/*
@@ -6817,8 +6817,8 @@ pick_next_task_fair(struct rq *rq, struc
 	goto done;
 simple:
 #endif
-
-	put_prev_task(rq, prev);
+	if (prev)
+		put_prev_task(rq, prev);
 
 	do {
 		se = pick_next_entity(cfs_rq, NULL);
@@ -6846,6 +6846,9 @@ done: __maybe_unused;
 	return p;
 
 idle:
+	if (!rf)
+		return NULL;
+
 	new_tasks = newidle_balance(rq, rf);
 
 	/*
kernel/sched/idle.c +3 -1
@@ -389,7 +389,9 @@ pick_next_task_idle(struct rq *rq, struc
 {
 	struct task_struct *next = rq->idle;
 
-	put_prev_task(rq, prev);
+	if (prev)
+		put_prev_task(rq, prev);
+
 	set_next_task_idle(rq, next);
 
 	return next;
kernel/sched/rt.c +1 -28
@@ -1554,38 +1554,11 @@ pick_next_task_rt(struct rq *rq, struct
 	struct task_struct *p;
 	struct rt_rq *rt_rq = &rq->rt;
 
-	if (need_pull_rt_task(rq, prev)) {
-		/*
-		 * This is OK, because current is on_cpu, which avoids it being
-		 * picked for load-balance and preemption/IRQs are still
-		 * disabled avoiding further scheduler activity on it and we're
-		 * being very careful to re-start the picking loop.
-		 */
-		rq_unpin_lock(rq, rf);
-		pull_rt_task(rq);
-		rq_repin_lock(rq, rf);
-		/*
-		 * pull_rt_task() can drop (and re-acquire) rq->lock; this
-		 * means a dl or stop task can slip in, in which case we need
-		 * to re-start task selection.
-		 */
-		if (unlikely((rq->stop && task_on_rq_queued(rq->stop)) ||
-			     rq->dl.dl_nr_running))
-			return RETRY_TASK;
-	}
-
-	/*
-	 * We may dequeue prev's rt_rq in put_prev_task().
-	 * So, we update time before rt_queued check.
-	 */
-	if (prev->sched_class == &rt_sched_class)
-		update_curr_rt(rq);
+	WARN_ON_ONCE(prev || rf);
 
 	if (!rt_rq->rt_queued)
 		return NULL;
 
-	put_prev_task(rq, prev);
-
 	p = _pick_next_task_rt(rq);
 
 	set_next_task_rt(rq, p);
kernel/sched/sched.h +8 -5
@@ -1704,12 +1704,15 @@ struct sched_class {
 	void (*check_preempt_curr)(struct rq *rq, struct task_struct *p, int flags);
 
 	/*
-	 * It is the responsibility of the pick_next_task() method that will
-	 * return the next task to call put_prev_task() on the @prev task or
-	 * something equivalent.
+	 * Both @prev and @rf are optional and may be NULL, in which case the
+	 * caller must already have invoked put_prev_task(rq, prev, rf).
 	 *
-	 * May return RETRY_TASK when it finds a higher prio class has runnable
-	 * tasks.
+	 * Otherwise it is the responsibility of the pick_next_task() to call
+	 * put_prev_task() on the @prev task or something equivalent, IFF it
+	 * returns a next task.
+	 *
+	 * In that case (@rf != NULL) it may return RETRY_TASK when it finds a
+	 * higher prio class has runnable tasks.
 	 */
 	struct task_struct * (*pick_next_task)(struct rq *rq,
 					       struct task_struct *prev,
kernel/sched/stop_task.c +2 -1
@@ -33,10 +33,11 @@ pick_next_task_stop(struct rq *rq, struc
 {
 	struct task_struct *stop = rq->stop;
 
+	WARN_ON_ONCE(prev || rf);
+
 	if (!stop || !task_on_rq_queued(stop))
 		return NULL;
 
-	put_prev_task(rq, prev);
 	set_next_task_stop(rq, stop);
 
 	return stop;
author maintainer
#38 lore raw

RFC PATCH 9/13 sched: Unify runtime accounting across classes

In reply to Peter Zijlstra

All classes use sched_entity::exec_start to track runtime and have copies of the exact same code around to compute runtime.

Collapse all that.

---
 include/linux/sched.h    |    2 -
 kernel/sched/deadline.c  |   17 ++-------------
 kernel/sched/fair.c      |   50 ++++++++++++++++++++++++++++++++++++-----------
 kernel/sched/rt.c        |   17 ++-------------
 kernel/sched/sched.h     |    2 +
 kernel/sched/stop_task.c |   16 ---------------
 6 files changed, 49 insertions(+), 55 deletions(-)
include/linux/sched.h +1 -1
@@ -433,7 +433,7 @@ struct sched_statistics {
 
 	u64				block_start;
 	u64				block_max;
-	u64				exec_max;
+	s64				exec_max;
 	u64				slice_max;
 
 	u64				nr_migrations_cold;
kernel/sched/deadline.c +3 -14
@@ -1147,9 +1147,8 @@ static void update_curr_dl(struct rq *rq
 {
 	struct task_struct *curr = rq->curr;
 	struct sched_dl_entity *dl_se = &curr->dl;
-	u64 delta_exec, scaled_delta_exec;
+	s64 delta_exec, scaled_delta_exec;
 	int cpu = cpu_of(rq);
-	u64 now;
 
 	if (!dl_task(curr) || !on_dl_rq(dl_se))
 		return;
@@ -1162,23 +1161,13 @@ static void update_curr_dl(struct rq *rq
 	 * natural solution, but the full ramifications of this
 	 * approach need further study.
 	 */
-	now = rq_clock_task(rq);
-	delta_exec = now - curr->se.exec_start;
-	if (unlikely((s64)delta_exec <= 0)) {
+	delta_exec = update_curr_common(rq);
+	if (unlikely(delta_exec <= 0)) {
 		if (unlikely(dl_se->dl_yielded))
 			goto throttle;
 		return;
 	}
 
-	schedstat_set(curr->se.statistics.exec_max,
-		      max(curr->se.statistics.exec_max, delta_exec));
-
-	curr->se.sum_exec_runtime += delta_exec;
-	account_group_exec_runtime(curr, delta_exec);
-
-	curr->se.exec_start = now;
-	cgroup_account_cputime(curr, delta_exec);
-
 	if (dl_entity_is_special(dl_se))
 		return;
 
kernel/sched/fair.c +39 -11

69 lines not on this page · raw message

kernel/sched/rt.c +3 -14
@@ -955,26 +955,15 @@ static void update_curr_rt(struct rq *rq
 {
 	struct task_struct *curr = rq->curr;
 	struct sched_rt_entity *rt_se = &curr->rt;
-	u64 delta_exec;
-	u64 now;
+	s64 delta_exec;
 
 	if (curr->sched_class != &rt_sched_class)
 		return;
 
-	now = rq_clock_task(rq);
-	delta_exec = now - curr->se.exec_start;
-	if (unlikely((s64)delta_exec <= 0))
+	delta_exec = update_curr_common(rq);
+	if (unlikely(delta_exec < 0))
 		return;
 
-	schedstat_set(curr->se.statistics.exec_max,
-		      max(curr->se.statistics.exec_max, delta_exec));
-
-	curr->se.sum_exec_runtime += delta_exec;
-	account_group_exec_runtime(curr, delta_exec);
-
-	curr->se.exec_start = now;
-	cgroup_account_cputime(curr, delta_exec);
-
 	if (!rt_bandwidth_enabled())
 		return;
 
kernel/sched/sched.h +2 -0
@@ -1689,6 +1689,8 @@ extern const u32		sched_prio_to_wmult[40
 
 #define RETRY_TASK		((void *)-1UL)
 
+extern s64 update_curr_common(struct rq *rq);
+
 struct sched_class {
 	const struct sched_class *next;
 
kernel/sched/stop_task.c +1 -15
@@ -62,21 +62,7 @@ static void yield_task_stop(struct rq *r
 
 static void put_prev_task_stop(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
-	struct task_struct *curr = rq->curr;
-	u64 delta_exec;
-
-	delta_exec = rq_clock_task(rq) - curr->se.exec_start;
-	if (unlikely((s64)delta_exec < 0))
-		delta_exec = 0;
-
-	schedstat_set(curr->se.statistics.exec_max,
-			max(curr->se.statistics.exec_max, delta_exec));
-
-	curr->se.sum_exec_runtime += delta_exec;
-	account_group_exec_runtime(curr, delta_exec);
-
-	curr->se.exec_start = rq_clock_task(rq);
-	cgroup_account_cputime(curr, delta_exec);
+	update_curr_common(rq);
 }
 
 /*
author maintainer
#39 lore raw

RFC PATCH 10/13 sched/deadline: Collect sched_dl_entity initialization

In reply to Peter Zijlstra

Create a single function that initializes a sched_dl_entity.

---
 kernel/sched/core.c     |    5 +----
 kernel/sched/deadline.c |   22 +++++++++++++++-------
 kernel/sched/sched.h    |    6 ++----
 3 files changed, 18 insertions(+), 15 deletions(-)
kernel/sched/core.c +1 -4
@@ -2581,10 +2581,7 @@ static void __sched_fork(unsigned long c
 	memset(&p->se.statistics, 0, sizeof(p->se.statistics));
 #endif
 
-	RB_CLEAR_NODE(&p->dl.rb_node);
-	init_dl_task_timer(&p->dl);
-	init_dl_inactive_task_timer(&p->dl);
-	__dl_clear_params(p);
+	init_dl_entity(&p->dl);
 
 	INIT_LIST_HEAD(&p->rt.run_list);
 	p->rt.timeout		= 0;
kernel/sched/deadline.c +15 -7
@@ -180,6 +180,8 @@ void dl_change_utilization(struct task_s
 	__add_rq_bw(new_bw, &rq->dl);
 }
 
+static void __dl_clear_params(struct sched_dl_entity *dl_se);
+
 /*
  * The utilization of a task cannot be immediately removed from
  * the rq active utilization (running_bw) when the task blocks.
@@ -278,7 +280,7 @@ static void task_non_contending(struct t
 				sub_rq_bw(&p->dl, &rq->dl);
 			raw_spin_lock(&dl_b->lock);
 			__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
-			__dl_clear_params(p);
+			__dl_clear_params(dl_se);
 			raw_spin_unlock(&dl_b->lock);
 		}
 
@@ -1049,7 +1051,7 @@ static enum hrtimer_restart dl_task_time
 	return HRTIMER_NORESTART;
 }
 
-void init_dl_task_timer(struct sched_dl_entity *dl_se)
+static void init_dl_task_timer(struct sched_dl_entity *dl_se)
 {
 	struct hrtimer *timer = &dl_se->dl_timer;
 
@@ -1261,7 +1263,7 @@ static enum hrtimer_restart inactive_tas
 		raw_spin_lock(&dl_b->lock);
 		__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
 		raw_spin_unlock(&dl_b->lock);
-		__dl_clear_params(p);
+		__dl_clear_params(dl_se);
 
 		goto unlock;
 	}
@@ -1277,7 +1279,7 @@ static enum hrtimer_restart inactive_tas
 	return HRTIMER_NORESTART;
 }
 
-void init_dl_inactive_task_timer(struct sched_dl_entity *dl_se)
+static void init_dl_inactive_task_timer(struct sched_dl_entity *dl_se)
 {
 	struct hrtimer *timer = &dl_se->inactive_timer;
 
@@ -2633,10 +2635,8 @@ bool __checkparam_dl(const struct sched_
 /*
  * This function clears the sched_dl_entity static params.
  */
-void __dl_clear_params(struct task_struct *p)
+static void __dl_clear_params(struct sched_dl_entity *dl_se)
 {
-	struct sched_dl_entity *dl_se = &p->dl;
-
 	dl_se->dl_runtime		= 0;
 	dl_se->dl_deadline		= 0;
 	dl_se->dl_period		= 0;
@@ -2650,6 +2650,14 @@ void __dl_clear_params(struct task_struc
 	dl_se->dl_overrun		= 0;
 }
 
+void init_dl_entity(struct sched_dl_entity *dl_se)
+{
+	RB_CLEAR_NODE(&dl_se->rb_node);
+	init_dl_task_timer(dl_se);
+	init_dl_inactive_task_timer(dl_se);
+	__dl_clear_params(dl_se);
+}
+
 bool dl_param_changed(struct task_struct *p, const struct sched_attr *attr)
 {
 	struct sched_dl_entity *dl_se = &p->dl;
kernel/sched/sched.h +2 -4
@@ -239,8 +239,6 @@ struct rt_bandwidth {
 	unsigned int		rt_period_active;
 };
 
-void __dl_clear_params(struct task_struct *p);
-
 /*
  * To keep the bandwidth of -deadline tasks and groups under control
  * we need some place where:
@@ -1844,10 +1842,10 @@ extern void init_rt_bandwidth(struct rt_
 
 extern struct dl_bandwidth def_dl_bandwidth;
 extern void init_dl_bandwidth(struct dl_bandwidth *dl_b, u64 period, u64 runtime);
-extern void init_dl_task_timer(struct sched_dl_entity *dl_se);
-extern void init_dl_inactive_task_timer(struct sched_dl_entity *dl_se);
 extern void init_dl_rq_bw_ratio(struct dl_rq *dl_rq);
 
+extern void init_dl_entity(struct sched_dl_entity *dl_se);
+
 #define BW_SHIFT		20
 #define BW_UNIT			(1 << BW_SHIFT)
 #define RATIO_SHIFT		8
author maintainer
#40 lore raw

RFC PATCH 11/13 sched/deadline: Move bandwidth accounting into {en,de}queue_dl_entity

In reply to Peter Zijlstra

In preparation of introducing !task sched_dl_entity; move the bandwidth accounting into {en.de}queue_dl_entity().

---
 kernel/sched/deadline.c |  125 ++++++++++++++++++++++++++----------------------
 kernel/sched/sched.h    |    6 ++
 2 files changed, 75 insertions(+), 56 deletions(-)
kernel/sched/deadline.c +69 -56
@@ -236,9 +236,8 @@ static void __dl_clear_params(struct sch
  * up, and checks if the task is still in the "ACTIVE non contending"
  * state or not (in the second case, it updates running_bw).
  */
-static void task_non_contending(struct task_struct *p)
+static void task_non_contending(struct sched_dl_entity *dl_se)
 {
-	struct sched_dl_entity *dl_se = &p->dl;
 	struct hrtimer *timer = &dl_se->inactive_timer;
 	struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
 	struct rq *rq = rq_of_dl_rq(dl_rq);
@@ -271,15 +270,18 @@ static void task_non_contending(struct t
 	 * utilization now, instead of starting a timer
 	 */
 	if ((zerolag_time < 0) || hrtimer_active(&dl_se->inactive_timer)) {
+		struct task_struct *p = dl_task_of(dl_se);
+
 		if (dl_task(p))
 			sub_running_bw(dl_se, dl_rq);
+
 		if (!dl_task(p) || p->state == TASK_DEAD) {
 			struct dl_bw *dl_b = dl_bw_of(task_cpu(p));
 
 			if (p->state == TASK_DEAD)
-				sub_rq_bw(&p->dl, &rq->dl);
+				sub_rq_bw(dl_se, &rq->dl);
 			raw_spin_lock(&dl_b->lock);
-			__dl_sub(dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
+			__dl_sub(dl_b, dl_se->dl_bw, dl_bw_cpus(task_cpu(p)));
 			__dl_clear_params(dl_se);
 			raw_spin_unlock(&dl_b->lock);
 		}
@@ -288,7 +290,7 @@ static void task_non_contending(struct t
 	}
 
 	dl_se->dl_non_contending = 1;
-	get_task_struct(p);
+	get_task_struct(dl_task_of(dl_se));
 	hrtimer_start(timer, ns_to_ktime(zerolag_time), HRTIMER_MODE_REL);
 }
 
@@ -1404,6 +1406,41 @@ enqueue_dl_entity(struct sched_dl_entity
 	BUG_ON(on_dl_rq(dl_se));
 
 	/*
+	 * Check if a constrained deadline task was activated
+	 * after the deadline but before the next period.
+	 * If that is the case, the task will be throttled and
+	 * the replenishment timer will be set to the next period.
+	 */
+	if (!dl_se->dl_throttled && !dl_is_implicit(dl_se))
+		dl_check_constrained_dl(dl_se);
+
+	if (flags & (ENQUEUE_RESTORE|ENQUEUE_MIGRATING)) {
+		struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
+
+		add_rq_bw(dl_se, dl_rq);
+		add_running_bw(dl_se, dl_rq);
+	}
+
+	/*
+	 * If p is throttled, we do not enqueue it. In fact, if it exhausted
+	 * its budget it needs a replenishment and, since it now is on
+	 * its rq, the bandwidth timer callback (which clearly has not
+	 * run yet) will take care of this.
+	 * However, the active utilization does not depend on the fact
+	 * that the task is on the runqueue or not (but depends on the
+	 * task's state - in GRUB parlance, "inactive" vs "active contending").
+	 * In other words, even if a task is throttled its utilization must
+	 * be counted in the active utilization; hence, we need to call
+	 * add_running_bw().
+	 */
+	if (dl_se->dl_throttled && !(flags & ENQUEUE_REPLENISH)) {
+		if (flags & ENQUEUE_WAKEUP)
+			task_contending(dl_se, flags);
+
+		return;
+	}
+
+	/*
 	 * If this is a wakeup or a new instance, the scheduling
 	 * parameters of the task might need updating. Otherwise,
 	 * we want a replenishment of its runtime.
@@ -1422,9 +1459,28 @@ enqueue_dl_entity(struct sched_dl_entity
 	__enqueue_dl_entity(dl_se);
 }
 
-static void dequeue_dl_entity(struct sched_dl_entity *dl_se)
+static void dequeue_dl_entity(struct sched_dl_entity *dl_se, int flags)
 {
 	__dequeue_dl_entity(dl_se);
+
+	if (flags & (DEQUEUE_SAVE|DEQUEUE_MIGRATING)) {
+		struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
+
+		sub_running_bw(dl_se, dl_rq);
+		sub_rq_bw(dl_se, dl_rq);
+	}
+
+	/*
+	 * This check allows to start the inactive timer (or to immediately
+	 * decrease the active utilization, if needed) in two cases:
+	 * when the task blocks and when it is terminating
+	 * (p->state == TASK_DEAD). We can handle the two cases in the same
+	 * way, because from GRUB's point of view the same thing is happening
+	 * (the task moves from "active contending" to "active non contending"
+	 * or "inactive")
+	 */
+	if (flags & DEQUEUE_SLEEP)
+		task_non_contending(dl_se);
 }
 
 static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags)
@@ -1454,38 +1510,8 @@ static void enqueue_task_dl(struct rq *r
 		return;
 	}
 
-	/*
-	 * Check if a constrained deadline task was activated
-	 * after the deadline but before the next period.
-	 * If that is the case, the task will be throttled and
-	 * the replenishment timer will be set to the next period.
-	 */
-	if (!p->dl.dl_throttled && !dl_is_implicit(&p->dl))
-		dl_check_constrained_dl(&p->dl);
-
-	if (p->on_rq == TASK_ON_RQ_MIGRATING || flags & ENQUEUE_RESTORE) {
-		add_rq_bw(&p->dl, &rq->dl);
-		add_running_bw(&p->dl, &rq->dl);
-	}
-
-	/*
-	 * If p is throttled, we do not enqueue it. In fact, if it exhausted
-	 * its budget it needs a replenishment and, since it now is on
-	 * its rq, the bandwidth timer callback (which clearly has not
-	 * run yet) will take care of this.
-	 * However, the active utilization does not depend on the fact
-	 * that the task is on the runqueue or not (but depends on the
-	 * task's state - in GRUB parlance, "inactive" vs "active contending").
-	 * In other words, even if a task is throttled its utilization must
-	 * be counted in the active utilization; hence, we need to call
-	 * add_running_bw().
-	 */
-	if (p->dl.dl_throttled && !(flags & ENQUEUE_REPLENISH)) {
-		if (flags & ENQUEUE_WAKEUP)
-			task_contending(&p->dl, flags);
-
-		return;
-	}
+	if (p->on_rq == TASK_ON_RQ_MIGRATING)
+		flags |= ENQUEUE_MIGRATING;
 
 	enqueue_dl_entity(&p->dl, pi_se, flags);
 
@@ -1495,31 +1521,18 @@ static void enqueue_task_dl(struct rq *r
 
 static void __dequeue_task_dl(struct rq *rq, struct task_struct *p, int flags)
 {
-	dequeue_dl_entity(&p->dl);
+	dequeue_dl_entity(&p->dl, flags);
 	dequeue_pushable_dl_task(rq, p);
 }
 
 static void dequeue_task_dl(struct rq *rq, struct task_struct *p, int flags)
 {
 	update_curr_dl(rq);
-	__dequeue_task_dl(rq, p, flags);
 
-	if (p->on_rq == TASK_ON_RQ_MIGRATING || flags & DEQUEUE_SAVE) {
-		sub_running_bw(&p->dl, &rq->dl);
-		sub_rq_bw(&p->dl, &rq->dl);
-	}
+	if (p->on_rq == TASK_ON_RQ_MIGRATING)
+		flags |= DEQUEUE_MIGRATING;
 
-	/*
-	 * This check allows to start the inactive timer (or to immediately
-	 * decrease the active utilization, if needed) in two cases:
-	 * when the task blocks and when it is terminating
-	 * (p->state == TASK_DEAD). We can handle the two cases in the same
-	 * way, because from GRUB's point of view the same thing is happening
-	 * (the task moves from "active contending" to "active non contending"
-	 * or "inactive")
-	 */
-	if (flags & DEQUEUE_SLEEP)
-		task_non_contending(p);
+	__dequeue_task_dl(rq, p, flags);
 }
 
 /*
@@ -2269,7 +2282,7 @@ static void switched_from_dl(struct rq *
 	 * will reset the task parameters.
 	 */
 	if (task_on_rq_queued(p) && p->dl.dl_runtime)
-		task_non_contending(p);
+		task_non_contending(&p->dl);
 
 	if (!task_on_rq_queued(p)) {
 		/*
kernel/sched/sched.h +6 -0
@@ -1661,6 +1661,10 @@ extern const u32		sched_prio_to_wmult[40
  * MOVE - paired with SAVE/RESTORE, explicitly does not preserve the location
  *        in the runqueue.
  *
+ * NOCLOCK - skip the update_rq_clock() (avoids double updates)
+ *
+ * MIGRATION - p->on_rq == TASK_ON_RQ_MIGRATING (used for DEADLINE)
+ *
  * ENQUEUE_HEAD      - place at front of runqueue (tail if not specified)
  * ENQUEUE_REPLENISH - CBS (replenish runtime and postpone deadline)
  * ENQUEUE_MIGRATED  - the task was migrated during wakeup
@@ -1671,6 +1675,7 @@ extern const u32		sched_prio_to_wmult[40
 #define DEQUEUE_SAVE		0x02 /* Matches ENQUEUE_RESTORE */
 #define DEQUEUE_MOVE		0x04 /* Matches ENQUEUE_MOVE */
 #define DEQUEUE_NOCLOCK		0x08 /* Matches ENQUEUE_NOCLOCK */
+#define DEQUEUE_MIGRATING	0x80 /* Matches ENQUEUE_MIGRATING */
 
 #define ENQUEUE_WAKEUP		0x01
 #define ENQUEUE_RESTORE		0x02
@@ -1684,6 +1689,7 @@ extern const u32		sched_prio_to_wmult[40
 #else
 #define ENQUEUE_MIGRATED	0x00
 #endif
+#define ENQUEUE_MIGRATING	0x80
 
 #define RETRY_TASK		((void *)-1UL)
 
author maintainer
#41 lore raw

RFC PATCH 12/13 sched/deadline: Introduce deadline servers

In reply to Peter Zijlstra
---
 include/linux/sched.h   |   26 +++
 kernel/sched/core.c     |    5 
 kernel/sched/deadline.c |  327 +++++++++++++++++++++++++++++++-----------------
 kernel/sched/fair.c     |    4 
 kernel/sched/sched.h    |   29 ++++
 5 files changed, 277 insertions(+), 114 deletions(-)
include/linux/sched.h +24 -2
@@ -52,12 +52,14 @@ struct robust_list_head;
 struct root_domain;
 struct rq;
 struct sched_attr;
+struct sched_dl_entity;
 struct sched_param;
 struct seq_file;
 struct sighand_struct;
 struct signal_struct;
 struct task_delay_info;
 struct task_group;
+struct task_struct;
 
 /*
  * Task state bitmask. NOTE! These bits are also
@@ -509,6 +511,9 @@ struct sched_rt_entity {
 #endif
 } __randomize_layout;
 
+typedef bool (*dl_server_has_tasks_f)(struct sched_dl_entity *);
+typedef struct task_struct *(*dl_server_pick_f)(struct sched_dl_entity *);
+
 struct sched_dl_entity {
 	struct rb_node			rb_node;
 
@@ -561,6 +566,7 @@ struct sched_dl_entity {
 	unsigned int			dl_yielded        : 1;
 	unsigned int			dl_non_contending : 1;
 	unsigned int			dl_overrun	  : 1;
+	unsigned int			dl_server         : 1;
 
 	/*
 	 * Bandwidth enforcement timer. Each -deadline task has its
@@ -575,7 +581,20 @@ struct sched_dl_entity {
 	 * timer is needed to decrease the active utilization at the correct
 	 * time.
 	 */
-	struct hrtimer inactive_timer;
+	struct hrtimer			inactive_timer;
+
+	/*
+	 * Bits for DL-server functionality. Also see the comment near
+	 * dl_server_update().
+	 *
+	 * @rq the runqueue this server is for
+	 *
+	 * @server_has_tasks() returns true if @server_pick return a
+	 * runnable task.
+	 */
+	struct rq			*rq;
+	dl_server_has_tasks_f		server_has_tasks;
+	dl_server_pick_f		server_pick;
 };
 
 #ifdef CONFIG_UCLAMP_TASK
@@ -688,10 +707,13 @@ struct task_struct {
 	const struct sched_class	*sched_class;
 	struct sched_entity		se;
 	struct sched_rt_entity		rt;
+	struct sched_dl_entity		dl;
+
+	struct sched_dl_entity		*server;
+
 #ifdef CONFIG_CGROUP_SCHED
 	struct task_group		*sched_task_group;
 #endif
-	struct sched_dl_entity		dl;
 
 #ifdef CONFIG_UCLAMP_TASK
 	/* Clamp values requested for a scheduling entity */
kernel/sched/core.c +4 -1
@@ -3756,8 +3756,11 @@ pick_next_task(struct rq *rq, struct tas
 
 	for_each_class(class) {
 		p = class->pick_next_task(rq, NULL, NULL);
-		if (p)
+		if (p) {
+			if (p->sched_class == class && p->server)
+				p->server = NULL;
 			return p;
+		}
 	}
 
 	/* The idle class should always have a runnable task: */
kernel/sched/deadline.c +216 -111

574 lines not on this page · raw message

kernel/sched/fair.c +4 -0
@@ -860,6 +860,8 @@ s64 update_curr_common(struct rq *rq)
 
 	account_group_exec_runtime(curr, delta_exec);
 	cgroup_account_cputime(curr, delta_exec);
+	if (curr->server)
+		dl_server_update(curr->server, delta_exec);
 
 	return delta_exec;
 }
@@ -889,6 +891,8 @@ static void update_curr(struct cfs_rq *c
 		trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
 		cgroup_account_cputime(curtask, delta_exec);
 		account_group_exec_runtime(curtask, delta_exec);
+		if (curtask->server)
+			dl_server_update(curtask->server, delta_exec);
 	}
 
 	account_cfs_rq_runtime(cfs_rq, delta_exec);
kernel/sched/sched.h +29 -0
@@ -316,6 +316,35 @@ extern int  dl_task_can_attach(struct ta
 extern int  dl_cpuset_cpumask_can_shrink(const struct cpumask *cur, const struct cpumask *trial);
 extern bool dl_cpu_busy(unsigned int cpu);
 
+/*
+ * SCHED_DEADLINE supports servers (nested scheduling) with the following
+ * interface:
+ *
+ *   dl_se::rq -- runqueue we belong to.
+ *
+ *   dl_se::server_has_tasks() -- used on bandwidth enforcement; we 'stop' the
+ *                                server when it runs out of tasks to run.
+ *
+ *   dl_se::server_pick() -- nested pick_next_task(); we yield the period if this
+ *                           returns NULL.
+ *
+ *   dl_server_update() -- called from update_curr_common(), propagates runtime
+ *                         to the server.
+ *
+ *   dl_server_start()
+ *   dl_server_stop()  -- start/stop the server when it has (no) tasks
+ *
+ *   dl_server_init()
+ *
+ * XXX
+ */
+extern void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec);
+extern void dl_server_start(struct sched_dl_entity *dl_se);
+extern void dl_server_stop(struct sched_dl_entity *dl_se);
+extern void dl_server_init(struct sched_dl_entity *dl_se, struct rq *rq,
+		    dl_server_has_tasks_f has_tasks,
+		    dl_server_pick_f pick);
+
 #ifdef CONFIG_CGROUP_SCHED
 
 #include <linux/cgroup.h>
#42 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

Quoted 3 lines · Peter Zijlstra

[...]

Review static void update_curr(struct cfs_rq *c · line 896

Quoted 7 lines
(quoted diff) +2 -0
@@ -889,6 +891,8 @@ static void update_curr(struct cfs_rq *c
 		trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
 		cgroup_account_cputime(curtask, delta_exec);
 		account_group_exec_runtime(curtask, delta_exec);
+		if (curtask->server)
+			dl_server_update(curtask->server, delta_exec);
 	}

I get a lockdep_assert_held(&rq->lock) related warning in start_dl_timer() when running the full stack.

...

[    0.530216] root domain span: 0-5 (max cpu_capacity = 1024)
[    0.538655] devtmpfs: initialized
[    0.556485] update_curr: rq mismatch rq[0] != rq[4]
[    0.561519] update_curr: rq mismatch rq[0] != rq[4]
[    0.566497] update_curr: rq mismatch rq[0] != rq[4]
[    0.571443] update_curr: rq mismatch rq[0] != rq[4]
[    0.576762] update_curr: rq mismatch rq[2] != rq[4]
[    0.581674] update_curr: rq mismatch rq[2] != rq[4]
[    0.586569] ------------[ cut here ]------------
[    0.591220] WARNING: CPU: 2 PID: 2 at kernel/sched/deadline.c:916 start_dl_timer+0x160/0x178
[    0.599686] Modules linked in:
[    0.602756] CPU: 2 PID: 2 Comm: kthreadd Tainted: G        W         5.3.0-rc3-00013-ga33cf033cc99-dirty #64
[    0.612620] Hardware name: ARM Juno development board (r0) (DT)
[    0.618560] pstate: 60000085 (nZCv daIf -PAN -UAO)
[    0.623369] pc : start_dl_timer+0x160/0x178
[    0.627572] lr : start_dl_timer+0x160/0x178
[    0.631768] sp : ffff000010013cb0
...
[    0.715075] Call trace:
[    0.717531]  start_dl_timer+0x160/0x178
[    0.721382]  update_curr_dl_se+0x108/0x208
[    0.725494]  dl_server_update+0x2c/0x38
[    0.729348]  update_curr+0x1b4/0x3b8
[    0.732934]  task_tick_fair+0x74/0xa88
[    0.736698]  scheduler_tick+0x94/0x110
[    0.740461]  update_process_times+0x48/0x60

...

Seems to be related to the fact that the rq can change:

kernel/sched/fair.c +10 -1
index e4c14851a34c..5e3130a200ec 100644
@@ -891,8 +891,17 @@ static void update_curr(struct cfs_rq *cfs_rq)
                trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
                cgroup_account_cputime(curtask, delta_exec);
                account_group_exec_runtime(curtask, delta_exec);
-               if (curtask->server)
+               if (curtask->server) {
+                       struct rq *rq = rq_of(cfs_rq);
+                       struct rq *rq2 = curtask->server->rq;
+
+                       if (rq != rq2) {
+                               printk("update_curr: rq mismatch rq[%d] != rq[%d]\n",
+                                      cpu_of(rq), cpu_of(rq2));
+                       }
+
                        dl_server_update(curtask->server, delta_exec);
+               }
        }
 

...

maintainer
#43 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

Hi Dietmar,

Quoted 62 lines · Dietmar Eggemann
Quoted 3 lines · Peter Zijlstra

[...]

kernel/sched/fair.c static void update_curr(struct cfs_rq *c · line 896

Quoted 7 lines
kernel/sched/fair.c +2 -0
@@ -889,6 +891,8 @@ static void update_curr(struct cfs_rq *c
 		trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
 		cgroup_account_cputime(curtask, delta_exec);
 		account_group_exec_runtime(curtask, delta_exec);
+		if (curtask->server)
+			dl_server_update(curtask->server, delta_exec);
 	}

I get a lockdep_assert_held(&rq->lock) related warning in start_dl_timer() when running the full stack.

...

[    0.530216] root domain span: 0-5 (max cpu_capacity = 1024)
[    0.538655] devtmpfs: initialized
[    0.556485] update_curr: rq mismatch rq[0] != rq[4]
[    0.561519] update_curr: rq mismatch rq[0] != rq[4]
[    0.566497] update_curr: rq mismatch rq[0] != rq[4]
[    0.571443] update_curr: rq mismatch rq[0] != rq[4]
[    0.576762] update_curr: rq mismatch rq[2] != rq[4]
[    0.581674] update_curr: rq mismatch rq[2] != rq[4]
[    0.586569] ------------[ cut here ]------------
[    0.591220] WARNING: CPU: 2 PID: 2 at kernel/sched/deadline.c:916 start_dl_timer+0x160/0x178
[    0.599686] Modules linked in:
[    0.602756] CPU: 2 PID: 2 Comm: kthreadd Tainted: G        W         5.3.0-rc3-00013-ga33cf033cc99-dirty #64
[    0.612620] Hardware name: ARM Juno development board (r0) (DT)
[    0.618560] pstate: 60000085 (nZCv daIf -PAN -UAO)
[    0.623369] pc : start_dl_timer+0x160/0x178
[    0.627572] lr : start_dl_timer+0x160/0x178
[    0.631768] sp : ffff000010013cb0
...
[    0.715075] Call trace:
[    0.717531]  start_dl_timer+0x160/0x178
[    0.721382]  update_curr_dl_se+0x108/0x208
[    0.725494]  dl_server_update+0x2c/0x38
[    0.729348]  update_curr+0x1b4/0x3b8
[    0.732934]  task_tick_fair+0x74/0xa88
[    0.736698]  scheduler_tick+0x94/0x110
[    0.740461]  update_process_times+0x48/0x60

...

Seems to be related to the fact that the rq can change:

kernel/sched/fair.c +4 -1
index e4c14851a34c..5e3130a200ec 100644
@@ -891,8 +891,17 @@ static void update_curr(struct cfs_rq *cfs_rq)
                trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
                cgroup_account_cputime(curtask, delta_exec);
                account_group_exec_runtime(curtask, delta_exec);
-               if (curtask->server)
+               if (curtask->server) {
+                       struct rq *rq = rq_of(cfs_rq);
+                       struct rq *rq2 = curtask->server->rq;
+

kernel/sched/fair.c static void update_curr(struct cfs_rq *cfs_rq) · line 906

Quoted 10 lines
kernel/sched/fair.c +6 -0
+                       if (rq != rq2) {
+                               printk("update_curr: rq mismatch rq[%d] != rq[%d]\n",
+                                      cpu_of(rq), cpu_of(rq2));
+                       }
+
                        dl_server_update(curtask->server, delta_exec);
+               }
        }
 
...

Yeah, I actually noticed the same. Some debugging seems to point to early boot spawning of kthreads. I can reliably for example attribute this mismatch to ksoftirqd(s). It looks like they can avoid the dl_server assignment in pick_next_task_dl() and this breaks things. Still need to figure out why this happens and how to fix it, though.

Best,

Juri

#44 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

Quoted 74 lines · Juri Lelli

Hi Dietmar,

Quoted 71 lines · Dietmar Eggemann
Quoted 3 lines · Peter Zijlstra

[...]

kernel/sched/fair.c static void update_curr(struct cfs_rq *c · line 896

Quoted 7 lines
kernel/sched/fair.c +2 -0
@@ -889,6 +891,8 @@ static void update_curr(struct cfs_rq *c
 		trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
 		cgroup_account_cputime(curtask, delta_exec);
 		account_group_exec_runtime(curtask, delta_exec);
+		if (curtask->server)
+			dl_server_update(curtask->server, delta_exec);
 	}

I get a lockdep_assert_held(&rq->lock) related warning in start_dl_timer() when running the full stack.

...

[    0.530216] root domain span: 0-5 (max cpu_capacity = 1024)
[    0.538655] devtmpfs: initialized
[    0.556485] update_curr: rq mismatch rq[0] != rq[4]
[    0.561519] update_curr: rq mismatch rq[0] != rq[4]
[    0.566497] update_curr: rq mismatch rq[0] != rq[4]
[    0.571443] update_curr: rq mismatch rq[0] != rq[4]
[    0.576762] update_curr: rq mismatch rq[2] != rq[4]
[    0.581674] update_curr: rq mismatch rq[2] != rq[4]
[    0.586569] ------------[ cut here ]------------
[    0.591220] WARNING: CPU: 2 PID: 2 at kernel/sched/deadline.c:916 start_dl_timer+0x160/0x178
[    0.599686] Modules linked in:
[    0.602756] CPU: 2 PID: 2 Comm: kthreadd Tainted: G        W         5.3.0-rc3-00013-ga33cf033cc99-dirty #64
[    0.612620] Hardware name: ARM Juno development board (r0) (DT)
[    0.618560] pstate: 60000085 (nZCv daIf -PAN -UAO)
[    0.623369] pc : start_dl_timer+0x160/0x178
[    0.627572] lr : start_dl_timer+0x160/0x178
[    0.631768] sp : ffff000010013cb0
...
[    0.715075] Call trace:
[    0.717531]  start_dl_timer+0x160/0x178
[    0.721382]  update_curr_dl_se+0x108/0x208
[    0.725494]  dl_server_update+0x2c/0x38
[    0.729348]  update_curr+0x1b4/0x3b8
[    0.732934]  task_tick_fair+0x74/0xa88
[    0.736698]  scheduler_tick+0x94/0x110
[    0.740461]  update_process_times+0x48/0x60

...

Seems to be related to the fact that the rq can change:

kernel/sched/fair.c +10 -1
index e4c14851a34c..5e3130a200ec 100644
@@ -891,8 +891,17 @@ static void update_curr(struct cfs_rq *cfs_rq)
                trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
                cgroup_account_cputime(curtask, delta_exec);
                account_group_exec_runtime(curtask, delta_exec);
-               if (curtask->server)
+               if (curtask->server) {
+                       struct rq *rq = rq_of(cfs_rq);
+                       struct rq *rq2 = curtask->server->rq;
+
+                       if (rq != rq2) {
+                               printk("update_curr: rq mismatch rq[%d] != rq[%d]\n",
+                                      cpu_of(rq), cpu_of(rq2));
+                       }
+
                        dl_server_update(curtask->server, delta_exec);
+               }
        }
 
Quoted 7 lines
Quoted 1 line

...

Yeah, I actually noticed the same. Some debugging seems to point to early boot spawning of kthreads. I can reliably for example attribute this mismatch to ksoftirqd(s). It looks like they can avoid the dl_server assignment in pick_next_task_dl() and this breaks things. Still need to figure out why this happens and how to fix it, though.

Yeah, can confirm this:

...

[0.556941] update_curr: rq mismatch rq[0] != rq[4] curr=[kthreadd, 39]
[0.563722] update_curr: rq mismatch rq[0] != rq[4] curr=[netns, 39]
[0.570179] update_curr: rq mismatch rq[0] != rq[4] curr=[netns, 39]
[0.576606] update_curr: rq mismatch rq[0] != rq[4] curr=[netns, 39]
[0.583708] update_curr: rq mismatch rq[2] != rq[1] curr=[kworker/2:1, 40]
[0.590793] update_curr: rq mismatch rq[2] != rq[1] curr=[kworker/2:1, 40]
[0.615096] update_curr: rq mismatch rq[2] != rq[1] curr=[kworker/2:1, 40]
...
[0.626644] WARNING: CPU: 2 PID: 40 at kernel/sched/deadline.c:916 start_dl_timer+0x160/0x178

...

author maintainer
#45 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

In reply to Dietmar Eggemann
Quoted 7 lines · Dietmar Eggemann
Quoted 3 lines · Peter Zijlstra

[...]

Quoted 10 lines

Review static void update_curr(struct cfs_rq *c · line 896

Quoted 7 lines
(quoted diff) +2 -0
@@ -889,6 +891,8 @@ static void update_curr(struct cfs_rq *c
 		trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
 		cgroup_account_cputime(curtask, delta_exec);
 		account_group_exec_runtime(curtask, delta_exec);
+		if (curtask->server)
+			dl_server_update(curtask->server, delta_exec);
 	}

I get a lockdep_assert_held(&rq->lock) related warning in start_dl_timer() when running the full stack.

That would seem to imply a stale curtask->server value; the hunk below:

kernel/sched/core.c +4 -1
@@ -3756,8 +3756,11 @@ pick_next_task(struct rq *rq, struct tas
 
        for_each_class(class) {
                p = class->pick_next_task(rq, NULL, NULL);
-               if (p)
+               if (p) {
+                       if (p->sched_class == class && p->server)
+                               p->server = NULL;
                        return p;
+               }
        }
 
 

Was supposed to clear p->server, but clearly something is going 'funny'.

#46 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

Quoted 36 lines · Peter Zijlstra
Quoted 7 lines · Dietmar Eggemann
Quoted 3 lines · Peter Zijlstra

[...]

Quoted 10 lines

Review static void update_curr(struct cfs_rq *c · line 896

Quoted 7 lines
(quoted diff) +2 -0
@@ -889,6 +891,8 @@ static void update_curr(struct cfs_rq *c
 		trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
 		cgroup_account_cputime(curtask, delta_exec);
 		account_group_exec_runtime(curtask, delta_exec);
+		if (curtask->server)
+			dl_server_update(curtask->server, delta_exec);
 	}

I get a lockdep_assert_held(&rq->lock) related warning in start_dl_timer() when running the full stack.

That would seem to imply a stale curtask->server value; the hunk below:

kernel/sched/core.c +4 -1
@@ -3756,8 +3756,11 @@ pick_next_task(struct rq *rq, struct tas
 
        for_each_class(class) {
                p = class->pick_next_task(rq, NULL, NULL);
-               if (p)
+               if (p) {
+                       if (p->sched_class == class && p->server)
+                               p->server = NULL;
                        return p;
+               }
        }
 
 
Quoted 1 line

Was supposed to clear p->server, but clearly something is going 'funny'.

What about the fast path in pick_next_task()?

kernel/sched/core.c +3 -0
index bffe849b5a42..f1ea6ae16052 100644
@@ -3742,6 +3742,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
                if (unlikely(!p))
                        p = idle_sched_class.pick_next_task(rq, prev, rf);
 
+               if (p->sched_class == &fair_sched_class && p->server)
+                       p->server = NULL;
+
                return p;
        }
maintainer
#47 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

Quoted 46 lines · Dietmar Eggemann
Quoted 36 lines · Peter Zijlstra
Quoted 7 lines · Dietmar Eggemann
Quoted 3 lines · Peter Zijlstra

[...]

Quoted 10 lines

Review static void update_curr(struct cfs_rq *c · line 896

Quoted 7 lines
(quoted diff) +2 -0
@@ -889,6 +891,8 @@ static void update_curr(struct cfs_rq *c
 		trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
 		cgroup_account_cputime(curtask, delta_exec);
 		account_group_exec_runtime(curtask, delta_exec);
+		if (curtask->server)
+			dl_server_update(curtask->server, delta_exec);
 	}

I get a lockdep_assert_held(&rq->lock) related warning in start_dl_timer() when running the full stack.

That would seem to imply a stale curtask->server value; the hunk below:

kernel/sched/core.c +4 -1
@@ -3756,8 +3756,11 @@ pick_next_task(struct rq *rq, struct tas
 
        for_each_class(class) {
                p = class->pick_next_task(rq, NULL, NULL);
-               if (p)
+               if (p) {
+                       if (p->sched_class == class && p->server)
+                               p->server = NULL;
                        return p;
+               }
        }
 
 
Quoted 1 line

Was supposed to clear p->server, but clearly something is going 'funny'.

What about the fast path in pick_next_task()?

kernel/sched/core.c +0 -0
index bffe849b5a42..f1ea6ae16052 100644

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 3747

Quoted 6 lines
kernel/sched/core.c +3 -0
@@ -3742,6 +3742,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
                if (unlikely(!p))
                        p = idle_sched_class.pick_next_task(rq, prev, rf);
 
+               if (p->sched_class == &fair_sched_class && p->server)
+                       p->server = NULL;
+

Hummm, but then who sets it back to the correct server. AFAIU update_curr() needs a ->server to do the correct DL accounting?

#48 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

Quoted 50 lines · Juri Lelli
Quoted 49 lines · Dietmar Eggemann
Quoted 36 lines · Peter Zijlstra
Quoted 7 lines · Dietmar Eggemann
Quoted 3 lines · Peter Zijlstra

[...]

Quoted 10 lines

Review static void update_curr(struct cfs_rq *c · line 896

Quoted 7 lines
(quoted diff) +2 -0
@@ -889,6 +891,8 @@ static void update_curr(struct cfs_rq *c
 		trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
 		cgroup_account_cputime(curtask, delta_exec);
 		account_group_exec_runtime(curtask, delta_exec);
+		if (curtask->server)
+			dl_server_update(curtask->server, delta_exec);
 	}

I get a lockdep_assert_held(&rq->lock) related warning in start_dl_timer() when running the full stack.

That would seem to imply a stale curtask->server value; the hunk below:

kernel/sched/core.c +4 -1
@@ -3756,8 +3756,11 @@ pick_next_task(struct rq *rq, struct tas
 
        for_each_class(class) {
                p = class->pick_next_task(rq, NULL, NULL);
-               if (p)
+               if (p) {
+                       if (p->sched_class == class && p->server)
+                               p->server = NULL;
                        return p;
+               }
        }
 
 
Quoted 1 line

Was supposed to clear p->server, but clearly something is going 'funny'.

What about the fast path in pick_next_task()?

kernel/sched/core.c +0 -0
index bffe849b5a42..f1ea6ae16052 100644
@@ -3742,6 +3742,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
                if (unlikely(!p))
                        p = idle_sched_class.pick_next_task(rq, prev, rf);
 
Quoted 6 lines

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 3747

Quoted 3 lines
kernel/sched/core.c +3 -0
+               if (p->sched_class == &fair_sched_class && p->server)
+                       p->server = NULL;
+

Hummm, but then who sets it back to the correct server. AFAIU update_curr() needs a ->server to do the correct DL accounting?

Ah, OK, this would kill the whole functionality ;-)

maintainer
#49 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

Quoted 51 lines · Dietmar Eggemann
Quoted 50 lines · Juri Lelli
Quoted 49 lines · Dietmar Eggemann
Quoted 36 lines · Peter Zijlstra
Quoted 7 lines · Dietmar Eggemann
Quoted 3 lines · Peter Zijlstra

[...]

Quoted 10 lines

Review static void update_curr(struct cfs_rq *c · line 896

Quoted 7 lines
(quoted diff) +2 -0
@@ -889,6 +891,8 @@ static void update_curr(struct cfs_rq *c
 		trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
 		cgroup_account_cputime(curtask, delta_exec);
 		account_group_exec_runtime(curtask, delta_exec);
+		if (curtask->server)
+			dl_server_update(curtask->server, delta_exec);
 	}

I get a lockdep_assert_held(&rq->lock) related warning in start_dl_timer() when running the full stack.

That would seem to imply a stale curtask->server value; the hunk below:

kernel/sched/core.c +4 -1
@@ -3756,8 +3756,11 @@ pick_next_task(struct rq *rq, struct tas
 
        for_each_class(class) {
                p = class->pick_next_task(rq, NULL, NULL);
-               if (p)
+               if (p) {
+                       if (p->sched_class == class && p->server)
+                               p->server = NULL;
                        return p;
+               }
        }
 
 
Quoted 1 line

Was supposed to clear p->server, but clearly something is going 'funny'.

What about the fast path in pick_next_task()?

kernel/sched/core.c +0 -0
index bffe849b5a42..f1ea6ae16052 100644
@@ -3742,6 +3742,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
                if (unlikely(!p))
                        p = idle_sched_class.pick_next_task(rq, prev, rf);
 
Quoted 9 lines
Quoted 6 lines

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 3747

Quoted 3 lines
kernel/sched/core.c +3 -0
+               if (p->sched_class == &fair_sched_class && p->server)
+                       p->server = NULL;
+

Hummm, but then who sets it back to the correct server. AFAIU update_curr() needs a ->server to do the correct DL accounting?

Ah, OK, this would kill the whole functionality ;-)

I'm thinking we could use &rq->fair_server. It seems to pass the point we are discussing about, but then virt box becomes unresponsive (busy loops).

maintainer
#50 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

Quoted 59 lines · Juri Lelli
Quoted 58 lines · Dietmar Eggemann
Quoted 56 lines · Juri Lelli
Quoted 46 lines · Dietmar Eggemann
Quoted 36 lines · Peter Zijlstra
Quoted 7 lines · Dietmar Eggemann
Quoted 3 lines · Peter Zijlstra

[...]

Quoted 10 lines

Review static void update_curr(struct cfs_rq *c · line 896

Quoted 7 lines
(quoted diff) +2 -0
@@ -889,6 +891,8 @@ static void update_curr(struct cfs_rq *c
 		trace_sched_stat_runtime(curtask, delta_exec, curr->vruntime);
 		cgroup_account_cputime(curtask, delta_exec);
 		account_group_exec_runtime(curtask, delta_exec);
+		if (curtask->server)
+			dl_server_update(curtask->server, delta_exec);
 	}

I get a lockdep_assert_held(&rq->lock) related warning in start_dl_timer() when running the full stack.

That would seem to imply a stale curtask->server value; the hunk below:

kernel/sched/core.c +4 -1
@@ -3756,8 +3756,11 @@ pick_next_task(struct rq *rq, struct tas
 
        for_each_class(class) {
                p = class->pick_next_task(rq, NULL, NULL);
-               if (p)
+               if (p) {
+                       if (p->sched_class == class && p->server)
+                               p->server = NULL;
                        return p;
+               }
        }
 
 
Quoted 1 line

Was supposed to clear p->server, but clearly something is going 'funny'.

What about the fast path in pick_next_task()?

kernel/sched/core.c +0 -0
index bffe849b5a42..f1ea6ae16052 100644

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 3747

Quoted 6 lines
kernel/sched/core.c +3 -0
@@ -3742,6 +3742,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
                if (unlikely(!p))
                        p = idle_sched_class.pick_next_task(rq, prev, rf);
 
+               if (p->sched_class == &fair_sched_class && p->server)
+                       p->server = NULL;
+

Hummm, but then who sets it back to the correct server. AFAIU update_curr() needs a ->server to do the correct DL accounting?

Quoted 6 lines
Quoted 2 lines

Ah, OK, this would kill the whole functionality ;-)

I'm thinking we could use &rq->fair_server. It seems to pass the point we are discussing about, but then virt box becomes unresponsive (busy loops).

I'd like to take this last sentence back, I was able to run a few boot + hackbench + shutdown cycles with the following applied (guess too much debug printks around before).

--->8---

 kernel/sched/deadline.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
kernel/sched/deadline.c +5 -3
index 3fe82b8f7825..d4a20072d4c0 100644
@@ -1312,6 +1312,10 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
 
 void dl_server_start(struct sched_dl_entity *dl_se)
 {
+	if (!dl_server(dl_se)) {
+		dl_se->dl_server = 1;
+		setup_new_dl_entity(dl_se);
+	}
 	enqueue_dl_entity(dl_se, dl_se, ENQUEUE_WAKEUP);
 }
 
@@ -1324,12 +1328,9 @@ void dl_server_init(struct sched_dl_entity *dl_se, struct rq *rq,
 		    dl_server_has_tasks_f has_tasks,
 		    dl_server_pick_f pick)
 {
-	dl_se->dl_server = 1;
 	dl_se->rq = rq;
 	dl_se->server_has_tasks = has_tasks;
 	dl_se->server_pick = pick;
-
-	setup_new_dl_entity(dl_se);
 }
 
 /*
@@ -2829,6 +2830,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se)
 	dl_se->dl_yielded		= 0;
 	dl_se->dl_non_contending	= 0;
 	dl_se->dl_overrun		= 0;
+	dl_se->dl_server		= 0;
 }
 
 void init_dl_entity(struct sched_dl_entity *dl_se)

--->8---

 kernel/sched/core.c | 3 +++
 1 file changed, 3 insertions(+)
kernel/sched/core.c +3 -0
index 7a161472decd..355cb1382aef 100644
@@ -3783,6 +3783,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 		if (unlikely(!p))
 			p = idle_sched_class.pick_next_task(rq, prev, rf);
 
+		if (p->sched_class == &fair_sched_class)
+			p->server = &rq->fair_server;
+
 		return p;
 	}
 
author maintainer
#51 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

Quoted 3 lines · Juri Lelli

I'd like to take this last sentence back, I was able to run a few boot + hackbench + shutdown cycles with the following applied (guess too much debug printks around before).

I've changed that slightly; the merged delta looks like:

kernel/sched/core.c +16 -4
@@ -3913,6 +3913,13 @@ pick_next_task(struct rq *rq, struct tas
 		if (unlikely(!p))
 			p = idle_sched_class.pick_next_task(rq, prev, rf);
 
+		/*
+		 * This is the fast path; it cannot be a DL server pick;
+		 * therefore even if @p == @prev, ->server must be NULL.
+		 */
+		if (prev->server)
+			p->server = NULL;
+
 		return p;
 	}
 
@@ -3925,13 +3932,18 @@ pick_next_task(struct rq *rq, struct tas
 	if (!rq->nr_running)
 		newidle_balance(rq, rf);
 
+	/*
+	 * We've updated @prev and no longer need the server link, clear it.
+	 * Must be done before ->pick_next_task() because that can (re)set
+	 * ->server.
+	 */
+	if (prev->server)
+		prev->server = NULL;
+
 	for_each_class(class) {
 		p = class->pick_next_task(rq, NULL, NULL);
-		if (p) {
-			if (p->sched_class == class && p->server)
-				p->server = NULL;
+		if (p)
 			return p;
-		}
 	}
 
 	/* The idle class should always have a runnable task: */
kernel/sched/deadline.c +5 -3
@@ -1312,6 +1312,10 @@ void dl_server_update(struct sched_dl_en
 
 void dl_server_start(struct sched_dl_entity *dl_se)
 {
+	if (!dl_server(dl_se)) {
+		dl_se->dl_server = 1;
+		setup_new_dl_entity(dl_se);
+	}
 	enqueue_dl_entity(dl_se, dl_se, ENQUEUE_WAKEUP);
 }
 
@@ -1324,12 +1328,9 @@ void dl_server_init(struct sched_dl_enti
 		    dl_server_has_tasks_f has_tasks,
 		    dl_server_pick_f pick)
 {
-	dl_se->dl_server = 1;
 	dl_se->rq = rq;
 	dl_se->server_has_tasks = has_tasks;
 	dl_se->server_pick = pick;
-
-	setup_new_dl_entity(dl_se);
 }
 
 /*
@@ -2855,6 +2856,7 @@ static void __dl_clear_params(struct sch
 	dl_se->dl_yielded		= 0;
 	dl_se->dl_non_contending	= 0;
 	dl_se->dl_overrun		= 0;
+	dl_se->dl_server		= 0;
 }
 
 void init_dl_entity(struct sched_dl_entity *dl_se)
maintainer
#52 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

Hi,

Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Juri Lelli

I'd like to take this last sentence back, I was able to run a few boot + hackbench + shutdown cycles with the following applied (guess too much debug printks around before).

I've changed that slightly; the merged delta looks like:

It looks like I had to further apply what follows on top of your latest wip-deadline branch (d3138279c7f3) to make kvm boot.

--->8---

 kernel/fork.c       | 2 ++
 kernel/sched/core.c | 3 +--
 2 files changed, 3 insertions(+), 2 deletions(-)
kernel/fork.c +2 -0
index 2852d0e76ea3..4f404fe1d9ab 100644
@@ -1983,6 +1983,8 @@ static __latent_entropy struct task_struct *copy_process(
 	p->sequential_io_avg	= 0;
 #endif
 
+	p->server = NULL;
+
 	/* Perform scheduler related setup. Assign this task to a CPU. */
 	retval = sched_fork(clone_flags, p);
 	if (retval)
kernel/sched/core.c +1 -2
index 1fd2a8c68daf..d72579b93a9f 100644
@@ -3917,8 +3917,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 		 * This is the fast path; it cannot be a DL server pick;
 		 * therefore even if @p == @prev, ->server must be NULL.
 		 */
-		if (prev->server)
-			p->server = NULL;
+		p->server = NULL;
 
 		return p;
 	}
author maintainer
#53 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

In reply to Juri Lelli
Quoted 12 lines · Juri Lelli
Quoted 10 lines · Dietmar Eggemann

What about the fast path in pick_next_task()?

kernel/sched/core.c +0 -0
index bffe849b5a42..f1ea6ae16052 100644
@@ -3742,6 +3742,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
                if (unlikely(!p))
                        p = idle_sched_class.pick_next_task(rq, prev, rf);
 
Quoted 6 lines

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 3747

Quoted 3 lines
kernel/sched/core.c +3 -0
+               if (p->sched_class == &fair_sched_class && p->server)
+                       p->server = NULL;
+

Hummm, but then who sets it back to the correct server. AFAIU update_curr() needs a ->server to do the correct DL accounting?

The above looks ok.

The pick_next_task_dl() when it selects a server will set ->server.

maintainer
#54 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

Quoted 20 lines · Peter Zijlstra
Quoted 12 lines · Juri Lelli
Quoted 10 lines · Dietmar Eggemann

What about the fast path in pick_next_task()?

kernel/sched/core.c +0 -0
index bffe849b5a42..f1ea6ae16052 100644
@@ -3742,6 +3742,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
                if (unlikely(!p))
                        p = idle_sched_class.pick_next_task(rq, prev, rf);
 
Quoted 6 lines

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 3747

Quoted 3 lines
kernel/sched/core.c +3 -0
+               if (p->sched_class == &fair_sched_class && p->server)
+                       p->server = NULL;
+

Hummm, but then who sets it back to the correct server. AFAIU update_curr() needs a ->server to do the correct DL accounting?

Quoted 3 lines

The above looks ok.

The pick_next_task_dl() when it selects a server will set ->server.

OK. Yesterday's IRC conversation resolved my perplexity about this point. I tried to summarize what I've got in the comment below (which we might want to add on top of Dietmar's fix).

--->8---
/*
* Since we are executing the fast path it means that the DL
* server was throttled and we'll now be scheduled w/o it. So,
* reset p's pointer to it (accounting won't be performed, as
* it wouldn't make sense while server is thottled). */

maintainer
#55 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

In reply to Peter Zijlstra

Hi,

On 26/07/19 16:54, Peter Zijlstra wrote:

[...]

kernel/sched/deadline.c static void update_curr_dl(struct rq *rq · line 1299

Quoted 10 lines
kernel/sched/deadline.c +10 -0
+void dl_server_init(struct sched_dl_entity *dl_se, struct rq *rq,
+		    dl_server_has_tasks_f has_tasks,
+		    dl_server_pick_f pick)
+{
+	dl_se->dl_server = 1;
+	dl_se->rq = rq;
+	dl_se->server_has_tasks = has_tasks;
+	dl_se->server_pick = pick;
+
+	setup_new_dl_entity(dl_se);

I think we need to postpone calling setup_new_dl_entity() to when dl_server_start() is called for the first time. rq_clock() needs both a caller that holds rq lock (which we can add) and an updated clock (which I'm not sure we can do at this point in time, or maybe we can avoid/trick the debug check?).

Thanks,

Juri

#56 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

In reply to Peter Zijlstra

On 7/26/19 4:54 PM, Peter Zijlstra wrote:

[...]

kernel/sched/deadline.c static void update_curr_dl(struct rq *rq · line 1299

Quoted 10 lines
kernel/sched/deadline.c +10 -0
+void dl_server_init(struct sched_dl_entity *dl_se, struct rq *rq,
+		    dl_server_has_tasks_f has_tasks,
+		    dl_server_pick_f pick)
+{
+	dl_se->dl_server = 1;
+	dl_se->rq = rq;
+	dl_se->server_has_tasks = has_tasks;
+	dl_se->server_pick = pick;
+
+	setup_new_dl_entity(dl_se);

IMHO, this needs rq locking because of the rq_clock(rq) in setup_new_dl_entity().

[    0.000000] WARNING: CPU: 0 PID: 0 at kernel/sched/sched.h:1119
dl_server_init+0x118/0x178
...
[    0.000000] CPU: 0 PID: 0 Comm: swapper Tainted: G        W
5.3.0-rc3-00013-ga33cf033cc99-dirty #10
[    0.000000] Hardware name: ARM Juno development board (r0) (DT)
...
[    0.000000] Call trace:
[    0.000000]  dl_server_init+0x118/0x178
[    0.000000]  fair_server_init+0x5c/0x68
[    0.000000]  sched_init+0x2c8/0x474
[    0.000000]  start_kernel+0x290/0x514

[...]

maintainer
#57 lore raw

Re: [RFC][PATCH 12/13] sched/deadline: Introduce deadline servers

Quoted 31 lines · Dietmar Eggemann

On 7/26/19 4:54 PM, Peter Zijlstra wrote:

[...]

kernel/sched/deadline.c static void update_curr_dl(struct rq *rq · line 1299

Quoted 10 lines
kernel/sched/deadline.c +10 -0
+void dl_server_init(struct sched_dl_entity *dl_se, struct rq *rq,
+		    dl_server_has_tasks_f has_tasks,
+		    dl_server_pick_f pick)
+{
+	dl_se->dl_server = 1;
+	dl_se->rq = rq;
+	dl_se->server_has_tasks = has_tasks;
+	dl_se->server_pick = pick;
+
+	setup_new_dl_entity(dl_se);

IMHO, this needs rq locking because of the rq_clock(rq) in setup_new_dl_entity().

[    0.000000] WARNING: CPU: 0 PID: 0 at kernel/sched/sched.h:1119
dl_server_init+0x118/0x178
...
[    0.000000] CPU: 0 PID: 0 Comm: swapper Tainted: G        W
5.3.0-rc3-00013-ga33cf033cc99-dirty #10
[    0.000000] Hardware name: ARM Juno development board (r0) (DT)
...
[    0.000000] Call trace:
[    0.000000]  dl_server_init+0x118/0x178
[    0.000000]  fair_server_init+0x5c/0x68
[    0.000000]  sched_init+0x2c8/0x474
[    0.000000]  start_kernel+0x290/0x514
Quoted 1 line

[...]

Right.. noticed the same and had a similar thinking the other day (1st hunk): [email protected]

author maintainer
#58 lore raw

RFC PATCH 13/13 sched/fair: Add trivial fair server

In reply to Peter Zijlstra
---
 kernel/sched/core.c     |    6 +++++-
 kernel/sched/deadline.c |    2 ++
 kernel/sched/fair.c     |   29 +++++++++++++++++++++++++++++
 kernel/sched/sched.h    |    6 +++++-
 4 files changed, 41 insertions(+), 2 deletions(-)
kernel/sched/core.c +1 -0
@@ -6501,6 +6504,7 @@ void __init sched_init(void)
 #endif /* CONFIG_SMP */
 		hrtick_rq_init(rq);
 		atomic_set(&rq->nr_iowait, 0);
+		fair_server_init(rq);
 	}
 
 	set_load_weight(&init_task, false);
kernel/sched/fair.c +29 -0
@@ -5239,6 +5239,9 @@ enqueue_task_fair(struct rq *rq, struct
 	 */
 	util_est_enqueue(&rq->cfs, p);
 
+	if (!rq->cfs.h_nr_running)
+		dl_server_start(&rq->fair_server);
+
 	/*
 	 * If in_iowait is set, the code below may not trigger any cpufreq
 	 * utilization updates, so do it here explicitly with the IOWAIT flag
@@ -5374,6 +5377,9 @@ static void dequeue_task_fair(struct rq
 	if (!se)
 		sub_nr_running(rq, 1);
 
+	if (!rq->cfs.h_nr_running)
+		dl_server_stop(&rq->fair_server);
+
 	util_est_dequeue(&rq->cfs, p, task_sleep);
 	hrtick_update(rq);
 }
@@ -6903,6 +6909,29 @@ done: __maybe_unused;
 	return NULL;
 }
 
+static bool fair_server_has_tasks(struct sched_dl_entity *dl_se)
+{
+	return !!dl_se->rq->cfs.nr_running;
+}
+
+static struct task_struct *fair_server_pick(struct sched_dl_entity *dl_se)
+{
+	return pick_next_task_fair(dl_se->rq, NULL, NULL);
+}
+
+void fair_server_init(struct rq *rq)
+{
+	struct sched_dl_entity *dl_se = &rq->fair_server;
+
+	init_dl_entity(dl_se);
+
+	dl_se->dl_runtime = TICK_NSEC;
+	dl_se->dl_deadline = 20 * TICK_NSEC;
+	dl_se->dl_period = 20 * TICK_NSEC;
+
+	dl_server_init(dl_se, rq, fair_server_has_tasks, fair_server_pick);
+}
+
 /*
  * Account for a descheduled task:
  */
kernel/sched/sched.h +5 -1
@@ -345,6 +345,8 @@ extern void dl_server_init(struct sched_
 		    dl_server_has_tasks_f has_tasks,
 		    dl_server_pick_f pick);
 
+extern void fair_server_init(struct rq *);
+
 #ifdef CONFIG_CGROUP_SCHED
 
 #include <linux/cgroup.h>
@@ -905,6 +907,8 @@ struct rq {
 	struct rt_rq		rt;
 	struct dl_rq		dl;
 
+	struct sched_dl_entity	fair_server;
+
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	/* list of leaf cfs_rq on this CPU: */
 	struct list_head	leaf_cfs_rq_list;
@@ -968,7 +972,7 @@ struct rq {
 
 	/* This is used to determine avg_idle's max value */
 	u64			max_idle_balance_cost;
-#endif
+#endif /* CONFIG_SMP */
 
 #ifdef CONFIG_IRQ_TIME_ACCOUNTING
 	u64			prev_irq_time;
#59 lore raw
In reply to Peter Zijlstra

Hi Peter,

Quoted 10 lines · On Fri, 26 Jul 2019 16:54:09 +0200 Peter Zijlstra <[email protected]> wrote:

Hi,

So recently syzcaller ran into the big deadline/period issue (again), and I figured I should at least propose a patch that puts limits on that -- see Patch 1.

During that discussion; SCHED_OTHER servers got mentioned (also again), and I figured I should have a poke at that too. So I took some inspiration from patches Alessio Balsini send a while back and cobbled something together for that too.

I think Patch 1 is a very good idea!

The server patches look interesting (and they seem to be much simpler than our patchset :). I need to have a better look at them, but this seems to be very promising.

Thanks,
Luca

Quoted 18 lines

Included are also a bunch of patches I did for core scheduling (2-8), which I'm probably going to just merge as they're generic cleanups. They're included here because they make pick_next_task() simpler and thereby thinking about the nested pick_next_task() logic inherent in servers was less of a head-ache. (I think it can work fine without them, but its easier with them on)

Anyway; after it all compiled it booted a kvm image all the way to userspace on the first run, so clearly this code isn't to be trusted at all.

There's still lots of missing bits and pieces -- like changelogs and the fair server isn't configurable or hooked into the bandwidth accounting, but the foundation is there I think.

Enjoy!

#60 lore raw
In reply to Peter Zijlstra

Hi Peter,

While testing your series (peterz/sched/wip-deadline 7a9e91d3fe951), I ended up in a panic at boot on a x86_64 kvm guest, would you please have a look? Here attached the backtrace.
Happy to test any suggestion that fixes the issue.

Thanks,
Alessio
---
------>8------

[    0.798326] ------------[ cut here ]------------       
[    0.798328] kernel BUG at kernel/sched/deadline.c:1542!            
[    0.798335] invalid opcode: 0000 [#1] SMP PTI                                  
[    0.798339] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.3.0-rc6+ #28
[    0.798340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
[    0.798349] RIP: 0010:enqueue_dl_entity+0x3f8/0x440
[    0.798351] Code: ff 48 8b 85 60 0a 00 00 8b 48 28 85 c9 0f 85 99 fd ff ff c7 40 28 01 00 00 00 e9 8d fd ff ff 85 c0 75 20 f                                                                                                                             
[    0.798353] RSP: 0000:ffffb68e40154f10 EFLAGS: 00010096
[    0.798356] RAX: 0000000000000020 RBX: ffff974bc74d0c00 RCX: ffff974bc751b200
[    0.798358] RDX: 0000000000000001 RSI: ffff974bc7929410 RDI: ffff974bc7929410
[    0.798359] RBP: 0000000000000009 R08: 00000000a73eb274 R09: 00000000000f4887
[    0.798361] R10: 0000000000000000 R11: 0000000000000000 R12: ffff974bc74d0c80
[    0.798362] R13: 0000000000000000 R14: ffff974bc74d0d00 R15: 0000000000000000
[    0.798365] FS:  0000000000000000(0000) GS:ffff974bc7900000(0000) knlGS:0000000000000000
[    0.798371] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 
[    0.798372] CR2: 00000000ffffffff CR3: 000000000480a000 CR4: 00000000000006e0
[    0.798374] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    0.798375] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    0.798376] Call Trace:                       
[    0.798397]  <IRQ>                                                
[    0.798402]  enqueue_task_fair+0xe69/0x11d0   
[    0.798407]  activate_task+0x58/0x90                                
[    0.798412]  ? kvm_sched_clock_read+0xd/0x20              
[    0.798416]  ttwu_do_activate.isra.96+0x3a/0x50                 
[    0.798420]  sched_ttwu_pending+0x5e/0x90                                              
[    0.798424]  scheduler_ipi+0x9f/0x120               
[    0.798430]  reschedule_interrupt+0xf/0x20           
[    0.798432]  </IRQ>                                                                           
[    0.798436] RIP: 0010:default_idle+0x20/0x140   
[    0.798438] Code: 90 90 90 90 90 90 90 90 90 90 41 55 41 54 55 65 8b 2d f4 40 d7 70 53 0f 1f 44 00 00 e9 07 00 00 00 0f 00 5                                                                                                                             
[    0.798440] RSP: 0000:ffffb68e40083ec0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff02
[    0.798442] RAX: ffffffff8f29c250 RBX: 0000000000000004 RCX: ffff974bc7916000               
[    0.798444] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff974bc791ca80
[    0.798445] RBP: 0000000000000004 R08: 000000009d74022b R09: 0000004d7ebb5820
[    0.798447] R10: 0000000000000400 R11: 0000000000000400 R12: 0000000000000000
[    0.798448] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[    0.798451]  ? __cpuidle_text_start+0x8/0x8 
[    0.798455]  do_idle+0x19e/0x210              
[    0.798458]  cpu_startup_entry+0x14/0x20
[    0.798464]  start_secondary+0x144/0x170                       
[    0.798467]  secondary_startup_64+0xa4/0xb0   
[    0.798469] Modules linked in:              
[    0.798478] ---[ end trace c2be7729c78a55ad ]---
[    0.798482] RIP: 0010:enqueue_dl_entity+0x3f8/0x440                                   
[    0.798484] Code: ff 48 8b 85 60 0a 00 00 8b 48 28 85 c9 0f 85 99 fd ff ff c7 40 28 01 00 00 00 e9 8d fd ff ff 85 c0 75 20 f                                                                                                                             
[    0.798485] RSP: 0000:ffffb68e40154f10 EFLAGS: 00010096
[    0.798487] RAX: 0000000000000020 RBX: ffff974bc74d0c00 RCX: ffff974bc751b200
[    0.798489] RDX: 0000000000000001 RSI: ffff974bc7929410 RDI: ffff974bc7929410
[    0.798490] RBP: 0000000000000009 R08: 00000000a73eb274 R09: 00000000000f4887
[    0.798491] R10: 0000000000000000 R11: 0000000000000000 R12: ffff974bc74d0c80
[    0.798493] R13: 0000000000000000 R14: ffff974bc74d0d00 R15: 0000000000000000
[    0.798495] FS:  0000000000000000(0000) GS:ffff974bc7900000(0000) knlGS:0000000000000000
[    0.798500] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.798501] CR2: 00000000ffffffff CR3: 000000000480a000 CR4: 00000000000006e0
[    0.798502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    0.798504] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    0.798505] Kernel panic - not syncing: Fatal exception in interrupt
[    0.799522] Kernel Offset: 0xd800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    0.875144] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

------8<------

maintainer
#61 lore raw

Hi Alessio,

Quoted 6 lines · Alessio Balsini

Hi Peter,

While testing your series (peterz/sched/wip-deadline 7a9e91d3fe951), I ended up in a panic at boot on a x86_64 kvm guest, would you please have a look? Here attached the backtrace.
Happy to test any suggestion that fixes the issue.

Are you running with latest fix by Peter?

https://lore.kernel.org/lkml/[email protected]/

It seems that his wip tree now has d3138279c7f3 on top (and the fix above has been merged).

Not sure it fixes also what you are seeing, though.

Thanks,

Juri

author maintainer
#62 lore raw
Quoted 10 lines · Juri Lelli

Hi Alessio,

Quoted 6 lines · Alessio Balsini

Hi Peter,

While testing your series (peterz/sched/wip-deadline 7a9e91d3fe951), I ended up in a panic at boot on a x86_64 kvm guest, would you please have a look? Here attached the backtrace.
Happy to test any suggestion that fixes the issue.

Quoted 8 lines

Are you running with latest fix by Peter?

https://lore.kernel.org/lkml/[email protected]/

It seems that his wip tree now has d3138279c7f3 on top (and the fix above has been merged).

Not sure it fixes also what you are seeing, though.

He likely is; but it is also very likely I messed it up somehow; I didn't even boot that branch :/ I'll try and have a look, but I'm running out of time before LPC.