Skip to contentSkip to conversation

sched/uclamp: add SCHED_FLAG_UTIL_CLAMP_RESET flag to reset uclamp

PATCH In review Yun Hsiang 5 participants 18 messages Nov 3, 2020 – Nov 13, 2020 Scheduler

lore
Share & tools
lore

Yun Hsiang

author
#0 lore raw

PATCH v5 1/1 sched/uclamp: add SCHED_FLAG_UTIL_CLAMP_RESET flag to reset uclamp

If the user wants to stop controlling uclamp and let the task inherit the value from the group, we need a method to reset.

Add SCHED_FLAG_UTIL_CLAMP_RESET flag to allow the user to reset uclamp via sched_setattr syscall.

The policy is

_CLAMP_RESET                           => reset both min and max
_CLAMP_RESET | _CLAMP_MIN              => reset min value
_CLAMP_RESET | _CLAMP_MAX              => reset max value

_CLAMP_RESET | _CLAMP_MIN | _CLAMP_MAX => reset both min and max

---
 include/uapi/linux/sched.h |  7 +++--
 kernel/sched/core.c        | 59 ++++++++++++++++++++++++++++----------
 2 files changed, 49 insertions(+), 17 deletions(-)
include/uapi/linux/sched.h +5 -2
index 3bac0a8ceab2..6c823ddb1a1e 100644
@@ -132,17 +132,20 @@ struct clone_args {
 #define SCHED_FLAG_KEEP_PARAMS		0x10
 #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
 #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
+#define SCHED_FLAG_UTIL_CLAMP_RESET	0x80
 
 #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
 				 SCHED_FLAG_KEEP_PARAMS)
 
 #define SCHED_FLAG_UTIL_CLAMP	(SCHED_FLAG_UTIL_CLAMP_MIN | \
-				 SCHED_FLAG_UTIL_CLAMP_MAX)
+				 SCHED_FLAG_UTIL_CLAMP_MAX | \
+				 SCHED_FLAG_UTIL_CLAMP_RESET)
 
 #define SCHED_FLAG_ALL	(SCHED_FLAG_RESET_ON_FORK	| \
 			 SCHED_FLAG_RECLAIM		| \
 			 SCHED_FLAG_DL_OVERRUN		| \
 			 SCHED_FLAG_KEEP_ALL		| \
-			 SCHED_FLAG_UTIL_CLAMP)
+			 SCHED_FLAG_UTIL_CLAMP		| \
+			 SCHED_FLAG_UTIL_CLAMP_RESET)
 
 #endif /* _UAPI_LINUX_SCHED_H */
kernel/sched/core.c +44 -15

103 lines not on this page · raw message

Signature · 1 line
2.25.1
#1 lore raw

Hi Yun

+Juri (A question for you below)

Quoted 24 lines · Yun Hsiang

If the user wants to stop controlling uclamp and let the task inherit the value from the group, we need a method to reset.

Add SCHED_FLAG_UTIL_CLAMP_RESET flag to allow the user to reset uclamp via sched_setattr syscall.

The policy is

_CLAMP_RESET                           => reset both min and max
_CLAMP_RESET | _CLAMP_MIN              => reset min value
_CLAMP_RESET | _CLAMP_MAX              => reset max value

_CLAMP_RESET | _CLAMP_MIN | _CLAMP_MAX => reset both min and max

---
 include/uapi/linux/sched.h |  7 +++--
 kernel/sched/core.c        | 59 ++++++++++++++++++++++++++++----------
 2 files changed, 49 insertions(+), 17 deletions(-)
include/uapi/linux/sched.h +0 -0
index 3bac0a8ceab2..6c823ddb1a1e 100644

include/uapi/linux/sched.h struct clone_args { · line 135

Quoted 4 lines
include/uapi/linux/sched.h +1 -0
@@ -132,17 +132,20 @@ struct clone_args {
 #define SCHED_FLAG_KEEP_PARAMS		0x10
 #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
 #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
+#define SCHED_FLAG_UTIL_CLAMP_RESET	0x80

The new flag needs documentation about how it should be used. It has a none obvious policy that we can't expect users to just get it.

include/uapi/linux/sched.h struct clone_args { · line 142

Quoted 8 lines
include/uapi/linux/sched.h +2 -1
 #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
 				 SCHED_FLAG_KEEP_PARAMS)
 
 #define SCHED_FLAG_UTIL_CLAMP	(SCHED_FLAG_UTIL_CLAMP_MIN | \
-				 SCHED_FLAG_UTIL_CLAMP_MAX)
+				 SCHED_FLAG_UTIL_CLAMP_MAX | \
+				 SCHED_FLAG_UTIL_CLAMP_RESET)

Either do this..

include/uapi/linux/sched.h struct clone_args { · line 149

Quoted 8 lines
include/uapi/linux/sched.h +2 -1
 #define SCHED_FLAG_ALL	(SCHED_FLAG_RESET_ON_FORK	| \
 			 SCHED_FLAG_RECLAIM		| \
 			 SCHED_FLAG_DL_OVERRUN		| \
 			 SCHED_FLAG_KEEP_ALL		| \
-			 SCHED_FLAG_UTIL_CLAMP)
+			 SCHED_FLAG_UTIL_CLAMP		| \
+			 SCHED_FLAG_UTIL_CLAMP_RESET)

Or this.

I checked glibc and it seems they don't use the sched.h from linux and more surprisingly they don't seem to have a wrapper for sched_setattr(). bionic libc from Android does take sched.h from linux, but didn't find any user. So we might be okay with modifying these here.

I still would like us to document better what we expect from these defines. Are they for internal kernel use or really for user space? If the former we should take them out of here. If the latter, then adding the RESET is dangerous as it'll cause an observable change in behavior; that is if an application was using SCHED_FLAG_ALL or SCHED_FLAG_UTIL_CLAMP to update the UTIL_MIN/MAX of a task, existing binaries will find out now that instead of modifying the value they're actually resetting them.

Juri, it seems you originally introduced SCHED_FLAG_ALL. I assume this was some sort of shorthand for user space, not the kernel?

If the latter, I think we should move SCHED_FLAG_ALL and SCHED_FLAG_UTIL_CLAMP to core.c and hope no one is already relying on them.

kernel/sched/core.c unsigned int uclamp_rq_max_value(struct rq *rq, enum uclamp_id clamp_id, · line 1007

Quoted 12 lines
include/uapi/linux/sched.h +0 -0
 #endif /* _UAPI_LINUX_SCHED_H */
kernel/sched/core.c +1 -1
index 8160ab5263f8..6ae463b64834 100644
@@ -1004,7 +1004,7 @@ unsigned int uclamp_rq_max_value(struct rq *rq, enum uclamp_id clamp_id,
 	return uclamp_idle_value(rq, clamp_id, clamp_value);
 }
 
-static void __uclamp_update_util_min_rt_default(struct task_struct *p)
+static inline void __uclamp_update_util_min_rt_default(struct task_struct *p)

Seems unrelated change. Worth a mention in the commit message at least.

Quoted 21 lines
kernel/sched/core.c +8 -2
 {
 	unsigned int default_util_min;
 	struct uclamp_se *uc_se;
@@ -1413,8 +1413,14 @@ int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,
 static int uclamp_validate(struct task_struct *p,
 			   const struct sched_attr *attr)
 {
-	unsigned int lower_bound = p->uclamp_req[UCLAMP_MIN].value;
-	unsigned int upper_bound = p->uclamp_req[UCLAMP_MAX].value;
+	unsigned int lower_bound, upper_bound;
+
+	/* Do not check uclamp attributes values in reset case. */
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_RESET)
+		return 0;
+
+	lower_bound = p->uclamp_req[UCLAMP_MIN].value;
+	upper_bound = p->uclamp_req[UCLAMP_MAX].value;
 
 	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
 		lower_bound = attr->sched_util_min;

kernel/sched/core.c static int uclamp_validate(struct task_struct *p, · line 1447

Quoted 4 lines
kernel/sched/core.c +1 -0
@@ -1438,20 +1444,43 @@ static int uclamp_validate(struct task_struct *p,
 	return 0;
 }
 
+static int uclamp_reset(enum uclamp_id clamp_id, unsigned long flags)

Add the policy part of the commit message as a documentation to this function please.

ie:

/*
* The policy is
* _CLAMP_RESET => reset both min and max
* _CLAMP_RESET | _CLAMP_MIN => reset min value
* _CLAMP_RESET | _CLAMP_MAX => reset max value
* _CLAMP_RESET | _CLAMP_MIN | _CLAMP_MAX => reset both min and max */

Quoted 43 lines
kernel/sched/core.c +25 -3
+{
+	/* No _UCLAMP_RESET flag set: do not reset */
+	if (!(flags & SCHED_FLAG_UTIL_CLAMP_RESET))
+		return false;
+
+	/* Only _UCLAMP_RESET flag set: reset both clamps */
+	if (!(flags & (SCHED_FLAG_UTIL_CLAMP_MIN | SCHED_FLAG_UTIL_CLAMP_MAX)))
+		return true;
+
+	/* Both _UCLAMP_RESET and _UCLAMP_MIN flags are set: reset only min */
+	if ((flags & SCHED_FLAG_UTIL_CLAMP_MIN) && clamp_id == UCLAMP_MIN)
+		return true;
+
+	/* Both _UCLAMP_RESET and _UCLAMP_MAX flags are set: reset only max */
+	if ((flags & SCHED_FLAG_UTIL_CLAMP_MAX) && clamp_id == UCLAMP_MAX)
+		return true;
+
+	return false;
+}
+
 static void __setscheduler_uclamp(struct task_struct *p,
 				  const struct sched_attr *attr)
 {
 	enum uclamp_id clamp_id;
 
 	/*
-	 * On scheduling class change, reset to default clamps for tasks
-	 * without a task-specific value.
+	 * Reset to default clamps on forced _UCLAMP_RESET (always) and
+	 * for tasks without a task-specific value (on scheduling class change).
 	 */
 	for_each_clamp_id(clamp_id) {
+		unsigned int clamp_value;
 		struct uclamp_se *uc_se = &p->uclamp_req[clamp_id];
 
 		/* Keep using defined clamps across class changes */
-		if (uc_se->user_defined)
+		if (!uclamp_reset(clamp_id, attr->sched_flags) &&
+				uc_se->user_defined)
 			continue;
 
 		/*

kernel/sched/core.c static void __setscheduler_uclamp(struct task_struct *p, · line 1495

Quoted 10 lines
kernel/sched/core.c +3 -2
@@ -1459,24 +1488,24 @@ static void __setscheduler_uclamp(struct task_struct *p,
 		 * at runtime.
 		 */
 		if (unlikely(rt_task(p) && clamp_id == UCLAMP_MIN))
-			__uclamp_update_util_min_rt_default(p);
+			clamp_value = sysctl_sched_uclamp_util_min_rt_default;
 		else
-			uclamp_se_set(uc_se, uclamp_none(clamp_id), false);
+			clamp_value = uclamp_none(clamp_id);
 
+		uclamp_se_set(uc_se, clamp_value, false);

This is another unrelated change. Add a comment in the commit message at least please.

Quoted 10 lines
kernel/sched/core.c +3 -2
 	}
 
-	if (likely(!(attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)))
+	if (likely(!(attr->sched_flags && SCHED_FLAG_UTIL_CLAMP)) ||
+		attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_RESET)
 		return;
 
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MIN],

kernel/sched/core.c static void __setscheduler_uclamp(struct task_struct *p, · line 1508

Quoted 10 lines
kernel/sched/core.c +3 -5
-			      attr->sched_util_min, true);
-	}
+				attr->sched_util_min, true);
 
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX)
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MAX],
-			      attr->sched_util_max, true);
-	}
+				attr->sched_util_max, true);

These two hunks seem unrelated too. Multi line statement should still have braces AFAIK. Why change it?

Generally personally I am not fond of mixing 'cleanup' and modifications. Especially when they are unrelated. They come across as churn to me but I won't insist on removing/splitting them but at least document them in the commit message with good reasons please.

Thanks

--
Qais Yousef

Quoted 6 lines
kernel/sched/core.c +0 -0
 }
 
 static void uclamp_fork(struct task_struct *p)
Signature · 1 line
2.25.1
#2 lore raw

Oops, +Juri for real this time.

Quoted 229 lines · Qais Yousef

Hi Yun

+Juri (A question for you below)

Quoted 24 lines · Yun Hsiang

If the user wants to stop controlling uclamp and let the task inherit the value from the group, we need a method to reset.

Add SCHED_FLAG_UTIL_CLAMP_RESET flag to allow the user to reset uclamp via sched_setattr syscall.

The policy is

_CLAMP_RESET                           => reset both min and max
_CLAMP_RESET | _CLAMP_MIN              => reset min value
_CLAMP_RESET | _CLAMP_MAX              => reset max value

_CLAMP_RESET | _CLAMP_MIN | _CLAMP_MAX => reset both min and max

---
 include/uapi/linux/sched.h |  7 +++--
 kernel/sched/core.c        | 59 ++++++++++++++++++++++++++++----------
 2 files changed, 49 insertions(+), 17 deletions(-)
include/uapi/linux/sched.h +0 -0
index 3bac0a8ceab2..6c823ddb1a1e 100644

include/uapi/linux/sched.h struct clone_args { · line 135

Quoted 4 lines
include/uapi/linux/sched.h +1 -0
@@ -132,17 +132,20 @@ struct clone_args {
 #define SCHED_FLAG_KEEP_PARAMS		0x10
 #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
 #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
+#define SCHED_FLAG_UTIL_CLAMP_RESET	0x80

The new flag needs documentation about how it should be used. It has a none obvious policy that we can't expect users to just get it.

include/uapi/linux/sched.h struct clone_args { · line 142

Quoted 8 lines
include/uapi/linux/sched.h +2 -1
 #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
 				 SCHED_FLAG_KEEP_PARAMS)
 
 #define SCHED_FLAG_UTIL_CLAMP	(SCHED_FLAG_UTIL_CLAMP_MIN | \
-				 SCHED_FLAG_UTIL_CLAMP_MAX)
+				 SCHED_FLAG_UTIL_CLAMP_MAX | \
+				 SCHED_FLAG_UTIL_CLAMP_RESET)

Either do this..

include/uapi/linux/sched.h struct clone_args { · line 149

Quoted 8 lines
include/uapi/linux/sched.h +2 -1
 #define SCHED_FLAG_ALL	(SCHED_FLAG_RESET_ON_FORK	| \
 			 SCHED_FLAG_RECLAIM		| \
 			 SCHED_FLAG_DL_OVERRUN		| \
 			 SCHED_FLAG_KEEP_ALL		| \
-			 SCHED_FLAG_UTIL_CLAMP)
+			 SCHED_FLAG_UTIL_CLAMP		| \
+			 SCHED_FLAG_UTIL_CLAMP_RESET)

Or this.

I checked glibc and it seems they don't use the sched.h from linux and more surprisingly they don't seem to have a wrapper for sched_setattr(). bionic libc from Android does take sched.h from linux, but didn't find any user. So we might be okay with modifying these here.

I still would like us to document better what we expect from these defines. Are they for internal kernel use or really for user space? If the former we should take them out of here. If the latter, then adding the RESET is dangerous as it'll cause an observable change in behavior; that is if an application was using SCHED_FLAG_ALL or SCHED_FLAG_UTIL_CLAMP to update the UTIL_MIN/MAX of a task, existing binaries will find out now that instead of modifying the value they're actually resetting them.

Juri, it seems you originally introduced SCHED_FLAG_ALL. I assume this was some sort of shorthand for user space, not the kernel?

If the latter, I think we should move SCHED_FLAG_ALL and SCHED_FLAG_UTIL_CLAMP to core.c and hope no one is already relying on them.

kernel/sched/core.c unsigned int uclamp_rq_max_value(struct rq *rq, enum uclamp_id clamp_id, · line 1007

Quoted 12 lines
include/uapi/linux/sched.h +0 -0
 #endif /* _UAPI_LINUX_SCHED_H */
kernel/sched/core.c +1 -1
index 8160ab5263f8..6ae463b64834 100644
@@ -1004,7 +1004,7 @@ unsigned int uclamp_rq_max_value(struct rq *rq, enum uclamp_id clamp_id,
 	return uclamp_idle_value(rq, clamp_id, clamp_value);
 }
 
-static void __uclamp_update_util_min_rt_default(struct task_struct *p)
+static inline void __uclamp_update_util_min_rt_default(struct task_struct *p)

Seems unrelated change. Worth a mention in the commit message at least.

Quoted 21 lines
kernel/sched/core.c +8 -2
 {
 	unsigned int default_util_min;
 	struct uclamp_se *uc_se;
@@ -1413,8 +1413,14 @@ int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,
 static int uclamp_validate(struct task_struct *p,
 			   const struct sched_attr *attr)
 {
-	unsigned int lower_bound = p->uclamp_req[UCLAMP_MIN].value;
-	unsigned int upper_bound = p->uclamp_req[UCLAMP_MAX].value;
+	unsigned int lower_bound, upper_bound;
+
+	/* Do not check uclamp attributes values in reset case. */
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_RESET)
+		return 0;
+
+	lower_bound = p->uclamp_req[UCLAMP_MIN].value;
+	upper_bound = p->uclamp_req[UCLAMP_MAX].value;
 
 	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
 		lower_bound = attr->sched_util_min;

kernel/sched/core.c static int uclamp_validate(struct task_struct *p, · line 1447

Quoted 4 lines
kernel/sched/core.c +1 -0
@@ -1438,20 +1444,43 @@ static int uclamp_validate(struct task_struct *p,
 	return 0;
 }
 
+static int uclamp_reset(enum uclamp_id clamp_id, unsigned long flags)

Add the policy part of the commit message as a documentation to this function please.

ie:

/*
* The policy is
* _CLAMP_RESET => reset both min and max
* _CLAMP_RESET | _CLAMP_MIN => reset min value
* _CLAMP_RESET | _CLAMP_MAX => reset max value
* _CLAMP_RESET | _CLAMP_MIN | _CLAMP_MAX => reset both min and max */

Quoted 43 lines
kernel/sched/core.c +25 -3
+{
+	/* No _UCLAMP_RESET flag set: do not reset */
+	if (!(flags & SCHED_FLAG_UTIL_CLAMP_RESET))
+		return false;
+
+	/* Only _UCLAMP_RESET flag set: reset both clamps */
+	if (!(flags & (SCHED_FLAG_UTIL_CLAMP_MIN | SCHED_FLAG_UTIL_CLAMP_MAX)))
+		return true;
+
+	/* Both _UCLAMP_RESET and _UCLAMP_MIN flags are set: reset only min */
+	if ((flags & SCHED_FLAG_UTIL_CLAMP_MIN) && clamp_id == UCLAMP_MIN)
+		return true;
+
+	/* Both _UCLAMP_RESET and _UCLAMP_MAX flags are set: reset only max */
+	if ((flags & SCHED_FLAG_UTIL_CLAMP_MAX) && clamp_id == UCLAMP_MAX)
+		return true;
+
+	return false;
+}
+
 static void __setscheduler_uclamp(struct task_struct *p,
 				  const struct sched_attr *attr)
 {
 	enum uclamp_id clamp_id;
 
 	/*
-	 * On scheduling class change, reset to default clamps for tasks
-	 * without a task-specific value.
+	 * Reset to default clamps on forced _UCLAMP_RESET (always) and
+	 * for tasks without a task-specific value (on scheduling class change).
 	 */
 	for_each_clamp_id(clamp_id) {
+		unsigned int clamp_value;
 		struct uclamp_se *uc_se = &p->uclamp_req[clamp_id];
 
 		/* Keep using defined clamps across class changes */
-		if (uc_se->user_defined)
+		if (!uclamp_reset(clamp_id, attr->sched_flags) &&
+				uc_se->user_defined)
 			continue;
 
 		/*

kernel/sched/core.c static void __setscheduler_uclamp(struct task_struct *p, · line 1495

Quoted 10 lines
kernel/sched/core.c +3 -2
@@ -1459,24 +1488,24 @@ static void __setscheduler_uclamp(struct task_struct *p,
 		 * at runtime.
 		 */
 		if (unlikely(rt_task(p) && clamp_id == UCLAMP_MIN))
-			__uclamp_update_util_min_rt_default(p);
+			clamp_value = sysctl_sched_uclamp_util_min_rt_default;
 		else
-			uclamp_se_set(uc_se, uclamp_none(clamp_id), false);
+			clamp_value = uclamp_none(clamp_id);
 
+		uclamp_se_set(uc_se, clamp_value, false);

This is another unrelated change. Add a comment in the commit message at least please.

Quoted 10 lines
kernel/sched/core.c +3 -2
 	}
 
-	if (likely(!(attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)))
+	if (likely(!(attr->sched_flags && SCHED_FLAG_UTIL_CLAMP)) ||
+		attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_RESET)
 		return;
 
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MIN],

kernel/sched/core.c static void __setscheduler_uclamp(struct task_struct *p, · line 1508

Quoted 10 lines
kernel/sched/core.c +3 -5
-			      attr->sched_util_min, true);
-	}
+				attr->sched_util_min, true);
 
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX)
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MAX],
-			      attr->sched_util_max, true);
-	}
+				attr->sched_util_max, true);

These two hunks seem unrelated too. Multi line statement should still have braces AFAIK. Why change it?

Generally personally I am not fond of mixing 'cleanup' and modifications. Especially when they are unrelated. They come across as churn to me but I won't insist on removing/splitting them but at least document them in the commit message with good reasons please.

Thanks

--
Qais Yousef

Quoted 6 lines
kernel/sched/core.c +0 -0
 }
 
 static void uclamp_fork(struct task_struct *p)
Signature · 1 line
2.25.1
#3 lore raw
Quoted 8 lines · Qais Yousef

Oops, +Juri for real this time.

Quoted 5 lines · Qais Yousef

Hi Yun

+Juri (A question for you below)

On 11/03/20 10:37, Yun Hsiang wrote:

[...]

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
 include/uapi/linux/sched.h |  7 +++--
 kernel/sched/core.c        | 59 ++++++++++++++++++++++++++++----------
 2 files changed, 49 insertions(+), 17 deletions(-)
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
include/uapi/linux/sched.h +0 -0
index 3bac0a8ceab2..6c823ddb1a1e 100644
Quoted 8 lines

include/uapi/linux/sched.h struct clone_args { · line 135

Quoted 5 lines
include/uapi/linux/sched.h +1 -0
@@ -132,17 +132,20 @@ struct clone_args {
 #define SCHED_FLAG_KEEP_PARAMS		0x10
 #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
 #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
+#define SCHED_FLAG_UTIL_CLAMP_RESET	0x80

The new flag needs documentation about how it should be used. It has a none obvious policy that we can't expect users to just get it.

See (1) further below.

Quoted 21 lines
Quoted 21 lines

include/uapi/linux/sched.h struct clone_args { · line 142

Quoted 7 lines
include/uapi/linux/sched.h +2 -1
 #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
 				 SCHED_FLAG_KEEP_PARAMS)
 
 #define SCHED_FLAG_UTIL_CLAMP	(SCHED_FLAG_UTIL_CLAMP_MIN | \
-				 SCHED_FLAG_UTIL_CLAMP_MAX)
+				 SCHED_FLAG_UTIL_CLAMP_MAX | \
+				 SCHED_FLAG_UTIL_CLAMP_RESET)

Either do this..

include/uapi/linux/sched.h struct clone_args { · line 149

Quoted 8 lines
include/uapi/linux/sched.h +2 -1
 #define SCHED_FLAG_ALL	(SCHED_FLAG_RESET_ON_FORK	| \
 			 SCHED_FLAG_RECLAIM		| \
 			 SCHED_FLAG_DL_OVERRUN		| \
 			 SCHED_FLAG_KEEP_ALL		| \
-			 SCHED_FLAG_UTIL_CLAMP)
+			 SCHED_FLAG_UTIL_CLAMP		| \
+			 SCHED_FLAG_UTIL_CLAMP_RESET)

Or this.

Quoted 4 lines
Quoted 4 lines

I checked glibc and it seems they don't use the sched.h from linux and more surprisingly they don't seem to have a wrapper for sched_setattr(). bionic libc from Android does take sched.h from linux, but didn't find any user. So we might be okay with modifying these here.

Schould be package linux-libc-dev. Debian 10 (buster-backports) 5.8.10-1~bpo10+1 contains the uclamp bits as well.

/usr/include/linux/sched/types.h
/usr/include/linux/sched.h

/usr/include/linux/sched.h contains SCHED_FLAG_UTIL_CLAMP and SCHED_FLAG_ALL.

But there is no glibc wrapper for sched_[sg]etattr so syscall wrapping is still needed.

Quoted 7 lines
Quoted 7 lines

I still would like us to document better what we expect from these defines. Are they for internal kernel use or really for user space? If the former we should take them out of here. If the latter, then adding the RESET is dangerous as it'll cause an observable change in behavior; that is if an application was using SCHED_FLAG_ALL or SCHED_FLAG_UTIL_CLAMP to update the UTIL_MIN/MAX of a task, existing binaries will find out now that instead of modifying the value they're actually resetting them.

I doubt that any application uses SCHED_FLAG_ALL so far since it already mixes e.g. DL and UCLAMP stuff. AFAIK the only tools supporting uclamp so far is rt-app and uclampset, which both use their own files for DL and uclamp definition.

[..]

Quoted 5 lines
Quoted 5 lines

Add the policy part of the commit message as a documentation to this function please.

ie:

Quoted 8 lines
Quoted 8 lines

/*
* The policy is
* _CLAMP_RESET => reset both min and max
* _CLAMP_RESET | _CLAMP_MIN => reset min value
* _CLAMP_RESET | _CLAMP_MAX => reset max value
* _CLAMP_RESET | _CLAMP_MIN | _CLAMP_MAX => reset both min and max */

(1) Related to documentation, wouldn't it be better to document in include/uapi/linux/sched.h, i.e. where the flags are defined, so it gets exported via linux-libc-dev?
Like it's done for struct sched_attr members in include/uapi/linux/sched/types.h.

[...]

Quoted 4 lines
Quoted 4 lines

include/uapi/linux/sched.h struct clone_args { · line 150

Quoted 4 lines
include/uapi/linux/sched.h +2 -1
-	if (likely(!(attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)))
+	if (likely(!(attr->sched_flags && SCHED_FLAG_UTIL_CLAMP)) ||
+		attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_RESET)
 		return;

Another multi line statement so the 'return' could go with braces.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
include/uapi/linux/sched.h +2 -3
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MIN],
-			      attr->sched_util_min, true);
-	}
+				attr->sched_util_min, true);
Quoted 9 lines
Quoted 9 lines

include/uapi/linux/sched.h

Quoted 6 lines
include/uapi/linux/sched.h +2 -3
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX)
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MAX],
-			      attr->sched_util_max, true);
-	}
+				attr->sched_util_max, true);

These two hunks seem unrelated too. Multi line statement should still have braces AFAIK. Why change it?

+1

[...]

author
#4 lore raw
Quoted 77 lines · Dietmar Eggemann
Quoted 8 lines · Qais Yousef

Oops, +Juri for real this time.

Quoted 5 lines · Qais Yousef

Hi Yun

+Juri (A question for you below)

On 11/03/20 10:37, Yun Hsiang wrote:

[...]

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
 include/uapi/linux/sched.h |  7 +++--
 kernel/sched/core.c        | 59 ++++++++++++++++++++++++++++----------
 2 files changed, 49 insertions(+), 17 deletions(-)
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
include/uapi/linux/sched.h +0 -0
index 3bac0a8ceab2..6c823ddb1a1e 100644
Quoted 8 lines

include/uapi/linux/sched.h struct clone_args { · line 135

Quoted 5 lines
include/uapi/linux/sched.h +1 -0
@@ -132,17 +132,20 @@ struct clone_args {
 #define SCHED_FLAG_KEEP_PARAMS		0x10
 #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
 #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
+#define SCHED_FLAG_UTIL_CLAMP_RESET	0x80

The new flag needs documentation about how it should be used. It has a none obvious policy that we can't expect users to just get it.

See (1) further below.

Quoted 21 lines
Quoted 21 lines

include/uapi/linux/sched.h struct clone_args { · line 142

Quoted 7 lines
include/uapi/linux/sched.h +2 -1
 #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
 				 SCHED_FLAG_KEEP_PARAMS)
 
 #define SCHED_FLAG_UTIL_CLAMP	(SCHED_FLAG_UTIL_CLAMP_MIN | \
-				 SCHED_FLAG_UTIL_CLAMP_MAX)
+				 SCHED_FLAG_UTIL_CLAMP_MAX | \
+				 SCHED_FLAG_UTIL_CLAMP_RESET)

Either do this..

include/uapi/linux/sched.h struct clone_args { · line 149

Quoted 8 lines
include/uapi/linux/sched.h +2 -1
 #define SCHED_FLAG_ALL	(SCHED_FLAG_RESET_ON_FORK	| \
 			 SCHED_FLAG_RECLAIM		| \
 			 SCHED_FLAG_DL_OVERRUN		| \
 			 SCHED_FLAG_KEEP_ALL		| \
-			 SCHED_FLAG_UTIL_CLAMP)
+			 SCHED_FLAG_UTIL_CLAMP		| \
+			 SCHED_FLAG_UTIL_CLAMP_RESET)

Or this.

Quoted 4 lines
Quoted 4 lines

I checked glibc and it seems they don't use the sched.h from linux and more surprisingly they don't seem to have a wrapper for sched_setattr(). bionic libc from Android does take sched.h from linux, but didn't find any user. So we might be okay with modifying these here.

Schould be package linux-libc-dev. Debian 10 (buster-backports) 5.8.10-1~bpo10+1 contains the uclamp bits as well.

/usr/include/linux/sched/types.h
/usr/include/linux/sched.h

/usr/include/linux/sched.h contains SCHED_FLAG_UTIL_CLAMP and SCHED_FLAG_ALL.

But there is no glibc wrapper for sched_[sg]etattr so syscall wrapping is still needed.

Quoted 7 lines
Quoted 7 lines

I still would like us to document better what we expect from these defines. Are they for internal kernel use or really for user space? If the former we should take them out of here. If the latter, then adding the RESET is dangerous as it'll cause an observable change in behavior; that is if an application was using SCHED_FLAG_ALL or SCHED_FLAG_UTIL_CLAMP to update the UTIL_MIN/MAX of a task, existing binaries will find out now that instead of modifying the value they're actually resetting them.

Quoted 5 lines

I doubt that any application uses SCHED_FLAG_ALL so far since it already mixes e.g. DL and UCLAMP stuff. AFAIK the only tools supporting uclamp so far is rt-app and uclampset, which both use their own files for DL and uclamp definition.

I think SCHED_FLAG_ALL is for internal kernel use. So is it better to make it within an #ifdef __KERNEL__ #endif block as Patrick said?

Quoted 16 lines

[..]

Quoted 5 lines
Quoted 5 lines

Add the policy part of the commit message as a documentation to this function please.

ie:

Quoted 8 lines
Quoted 8 lines

/*
* The policy is
* _CLAMP_RESET => reset both min and max
* _CLAMP_RESET | _CLAMP_MIN => reset min value
* _CLAMP_RESET | _CLAMP_MAX => reset max value
* _CLAMP_RESET | _CLAMP_MIN | _CLAMP_MAX => reset both min and max */

Quoted 6 lines

(1) Related to documentation, wouldn't it be better to document in include/uapi/linux/sched.h, i.e. where the flags are defined, so it gets exported via linux-libc-dev?
Like it's done for struct sched_attr members in include/uapi/linux/sched/types.h.

Ok, I'll put the comment for_CLAMP_MIN/_CLAMP_MAX/CLAMP_RESET in include/uapi/linux/sched.h.

Quoted 26 lines

[...]

Quoted 4 lines
Quoted 4 lines

include/uapi/linux/sched.h

Quoted 4 lines
include/uapi/linux/sched.h +2 -1
-	if (likely(!(attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)))
+	if (likely(!(attr->sched_flags && SCHED_FLAG_UTIL_CLAMP)) ||
+		attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_RESET)
 		return;

Another multi line statement so the 'return' could go with braces.

Quoted 17 lines
Quoted 17 lines
Quoted 4 lines
include/uapi/linux/sched.h +1 -1
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MIN],

include/uapi/linux/sched.h

Quoted 10 lines
include/uapi/linux/sched.h +3 -5
-			      attr->sched_util_min, true);
-	}
+				attr->sched_util_min, true);
 
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX)
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MAX],
-			      attr->sched_util_max, true);
-	}
+				attr->sched_util_max, true);

These two hunks seem unrelated too. Multi line statement should still have braces AFAIK. Why change it?

Sorry for the wrong coding style, I'll fix it. And I'll split the modifications to different patches if I want to do some cleanup.

Quoted 3 lines

+1

[...]

#5 lore raw
In reply to Yun Hsiang

Hi Yun,
thanks for keep improving this.

I'm replying here but still considering all other reviewers comments.

Best,
Patrick

On Tue, Nov 03, 2020 at 03:37:56 +0100, Yun Hsiang <[email protected]> wrote...

Quoted 11 lines

If the user wants to stop controlling uclamp and let the task inherit the value from the group, we need a method to reset.

Add SCHED_FLAG_UTIL_CLAMP_RESET flag to allow the user to reset uclamp via sched_setattr syscall.

The policy is

_CLAMP_RESET                           => reset both min and max
_CLAMP_RESET | _CLAMP_MIN              => reset min value
_CLAMP_RESET | _CLAMP_MAX              => reset max value

_CLAMP_RESET | _CLAMP_MIN | _CLAMP_MAX => reset both min and max

This documentation should be added to the uapi header and, most importantly, in:
include/uapi/linux/sched/types.h
where the documentation for struct sched_attr is provided.

Quoted 12 lines
---
 include/uapi/linux/sched.h |  7 +++--
 kernel/sched/core.c        | 59 ++++++++++++++++++++++++++++----------
 2 files changed, 49 insertions(+), 17 deletions(-)
include/uapi/linux/sched.h +0 -0
index 3bac0a8ceab2..6c823ddb1a1e 100644

include/uapi/linux/sched.h struct clone_args { · line 139

Quoted 8 lines
include/uapi/linux/sched.h +1 -0
@@ -132,17 +132,20 @@ struct clone_args {
 #define SCHED_FLAG_KEEP_PARAMS		0x10
 #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
 #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
+#define SCHED_FLAG_UTIL_CLAMP_RESET	0x80
 
 #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
 				 SCHED_FLAG_KEEP_PARAMS)
 

(Related to the following discussion point)
What about adding in a comment here to call out that the following definitions are "internal only"?
Moreover, we could probably wrap the following two define within an #ifdef __KERNEL__/#endif block

Something like:

+ /*
+ * The following definitions are internal only, do not use them to set
+ * set_{set,get}attr() params. Use instead a valid combination of the
+ * flags defined above.
+ */
+ #ifdef __KERNEL__

include/uapi/linux/sched.h struct clone_args { · line 142

Quoted 4 lines
include/uapi/linux/sched.h +2 -1
 #define SCHED_FLAG_UTIL_CLAMP	(SCHED_FLAG_UTIL_CLAMP_MIN | \
-				 SCHED_FLAG_UTIL_CLAMP_MAX)
+				 SCHED_FLAG_UTIL_CLAMP_MAX | \
+				 SCHED_FLAG_UTIL_CLAMP_RESET)

We need the _RESET flag only here (not below), since this is a UCLAMP feature and it's worth/useful to have a single "all uclamp flags" definition...

include/uapi/linux/sched.h struct clone_args { · line 148

Quoted 7 lines
include/uapi/linux/sched.h +2 -1
 #define SCHED_FLAG_ALL	(SCHED_FLAG_RESET_ON_FORK	| \
 			 SCHED_FLAG_RECLAIM		| \
 			 SCHED_FLAG_DL_OVERRUN		| \
 			 SCHED_FLAG_KEEP_ALL		| \
-			 SCHED_FLAG_UTIL_CLAMP)
+			 SCHED_FLAG_UTIL_CLAMP		| \
+			 SCHED_FLAG_UTIL_CLAMP_RESET)

... i.e., you can drop the chunk above.

+ #endif /* __KERNEL__ */

Regarding Qais comment, I had the same Dietmar's thought: I doubt there are apps using _FLAGS_ALL from userspace. For DL tasks, since they cannot fork, it makes no sense to specify, for example _RESET_ON_FORK|_RECLAIM. For CFS/RT tasks, where UCLAMP is supported, it makes no sense to specify DL flags.

It's true however that having this def here when it's supposed to be used only internally, can be kind of "confusing", but it's also useful to keep the definition aligned with the flags defined above. The ifdef wrapping proposed above should make this even more explicit.

Perhaps we can also better call this out also with an additional note right after:

https://elixir.bootlin.com/linux/v5.9.6/source/include/uapi/linux/sched/types.h#L43

In that file, I believe the "Task Utilization Attributes" section can also be improved by adding a description of the _UCLAMP flags semantics.

kernel/sched/core.c unsigned int uclamp_rq_max_value(struct rq *rq, enum uclamp_id clamp_id, · line 1008

Quoted 12 lines
include/uapi/linux/sched.h +0 -0
 #endif /* _UAPI_LINUX_SCHED_H */
kernel/sched/core.c +1 -1
index 8160ab5263f8..6ae463b64834 100644
@@ -1004,7 +1004,7 @@ unsigned int uclamp_rq_max_value(struct rq *rq, enum uclamp_id clamp_id,
 	return uclamp_idle_value(rq, clamp_id, clamp_value);
 }
 
-static void __uclamp_update_util_min_rt_default(struct task_struct *p)
+static inline void __uclamp_update_util_min_rt_default(struct task_struct *p)
 {

Again, IMO, this is _not_ an unrelated change at all. Actually, I still would like to do one step more and inline this function in the _only place_ where it's used. Qais arguments for not doing that where [1]:

Updating the default rt value is done from different contexts. Hence it is important to document the rules under which this update must happen and ensure the update happens through a common path.

I don't see why these arguments are not satisfied when we inline, e.g.

---8<---

kernel/sched/core.c +6 -19
index d2003a7d5ab5..369074154e1d 100644
@@ -1004,25 +1004,9 @@ unsigned int uclamp_rq_max_value(struct rq *rq, enum uclamp_id clamp_id,
        return uclamp_idle_value(rq, clamp_id, clamp_value);
 }
 
-static void __uclamp_update_util_min_rt_default(struct task_struct *p)
-{
-       unsigned int default_util_min;
-       struct uclamp_se *uc_se;
-
-       lockdep_assert_held(&p->pi_lock);
-
-       uc_se = &p->uclamp_req[UCLAMP_MIN];
-
-       /* Only sync if user didn't override the default */
-       if (uc_se->user_defined)
-               return;
-
-       default_util_min = sysctl_sched_uclamp_util_min_rt_default;
-       uclamp_se_set(uc_se, default_util_min, false);
-}
-
 static void uclamp_update_util_min_rt_default(struct task_struct *p)
 {
+       struct uclamp_se *uc_se = &p->uclamp_req[UCLAMP_MIN];
        struct rq_flags rf;
        struct rq *rq;
 
@@ -1031,7 +1015,11 @@ static void uclamp_update_util_min_rt_default(struct task_struct *p)
 
        /* Protect updates to p->uclamp_* */
        rq = task_rq_lock(p, &rf);
-       __uclamp_update_util_min_rt_default(p);
+
+       /* Only sync if user didn't override the default */
+       if (!uc_se->user_defined)
+               uclamp_se_set(uc_se, sysctl_sched_uclamp_util_min_rt_default, false);
+
        task_rq_unlock(rq, p, &rf);
 }
---8<---

[1] https://lore.kernel.org/lkml/20201028182946.6qfmt7q35ewrjjua@e107158-lin/

Quoted 20 lines
kernel/sched/core.c +8 -2
 	unsigned int default_util_min;
 	struct uclamp_se *uc_se;
@@ -1413,8 +1413,14 @@ int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,
 static int uclamp_validate(struct task_struct *p,
 			   const struct sched_attr *attr)
 {
-	unsigned int lower_bound = p->uclamp_req[UCLAMP_MIN].value;
-	unsigned int upper_bound = p->uclamp_req[UCLAMP_MAX].value;
+	unsigned int lower_bound, upper_bound;
+
+	/* Do not check uclamp attributes values in reset case. */
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_RESET)
+		return 0;
+
+	lower_bound = p->uclamp_req[UCLAMP_MIN].value;
+	upper_bound = p->uclamp_req[UCLAMP_MAX].value;
 
 	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
 		lower_bound = attr->sched_util_min;

kernel/sched/core.c static int uclamp_validate(struct task_struct *p, · line 1449

Quoted 6 lines
kernel/sched/core.c +3 -0
@@ -1438,20 +1444,43 @@ static int uclamp_validate(struct task_struct *p,
 	return 0;
 }
 
+static int uclamp_reset(enum uclamp_id clamp_id, unsigned long flags)
+{
+	/* No _UCLAMP_RESET flag set: do not reset */
              ^^^^^^^^^^^^^

Maybe better using the full flag name for all these?

Quoted 6 lines
kernel/sched/core.c +6 -0
+	if (!(flags & SCHED_FLAG_UTIL_CLAMP_RESET))
+		return false;
+
+	/* Only _UCLAMP_RESET flag set: reset both clamps */
+	if (!(flags & (SCHED_FLAG_UTIL_CLAMP_MIN | SCHED_FLAG_UTIL_CLAMP_MAX)))
+		return true;

kernel/sched/core.c static int uclamp_validate(struct task_struct *p, · line 1465

Quoted 10 lines
kernel/sched/core.c +10 -0
+
+	/* Both _UCLAMP_RESET and _UCLAMP_MIN flags are set: reset only min */
+	if ((flags & SCHED_FLAG_UTIL_CLAMP_MIN) && clamp_id == UCLAMP_MIN)
+		return true;
+
+	/* Both _UCLAMP_RESET and _UCLAMP_MAX flags are set: reset only max */
+	if ((flags & SCHED_FLAG_UTIL_CLAMP_MAX) && clamp_id == UCLAMP_MAX)
+		return true;
+
+	return false;

I was suggesting maybe we need READ_ONCE() in the if above but did not addressed previous Dietmar's question [2] on why.

The function above has a correct semantics only when the ordering of the if statement is strictly observed.

Now, since we don't have any data dependency on the multiple if cases, are we sure an overzealous compiler will never come up with some "optimized reordering" of the checks required?

IOW, if the compiler could scramble the checks on an optimization, we would get a wrong semantics which is also very difficult do debug. Hence the idea to use READ_ONCE, to both tell the compiler to not even attempt reordering those checks and also to better document the code about the importance of the ordering on those checks.

Is now more clear? Does that makes sense?

[2] https://lore.kernel.org/lkml/[email protected]/

Quoted 31 lines
kernel/sched/core.c +8 -4
+}
+
 static void __setscheduler_uclamp(struct task_struct *p,
 				  const struct sched_attr *attr)
 {
 	enum uclamp_id clamp_id;
 
 	/*
-	 * On scheduling class change, reset to default clamps for tasks
-	 * without a task-specific value.
+	 * Reset to default clamps on forced _UCLAMP_RESET (always) and
+	 * for tasks without a task-specific value (on scheduling class change).
 	 */
 	for_each_clamp_id(clamp_id) {
+		unsigned int clamp_value;
 		struct uclamp_se *uc_se = &p->uclamp_req[clamp_id];
 
 		/* Keep using defined clamps across class changes */
-		if (uc_se->user_defined)
+		if (!uclamp_reset(clamp_id, attr->sched_flags) &&
+				uc_se->user_defined)
 			continue;
 
 		/*
@@ -1459,24 +1488,24 @@ static void __setscheduler_uclamp(struct task_struct *p,
 		 * at runtime.
 		 */
 		if (unlikely(rt_task(p) && clamp_id == UCLAMP_MIN))
-			__uclamp_update_util_min_rt_default(p);
+			clamp_value = sysctl_sched_uclamp_util_min_rt_default;
 		else

kernel/sched/core.c static void __setscheduler_uclamp(struct task_struct *p, · line 1500

Quoted 10 lines
kernel/sched/core.c +4 -2
-			uclamp_se_set(uc_se, uclamp_none(clamp_id), false);
+			clamp_value = uclamp_none(clamp_id);
 
+		uclamp_se_set(uc_se, clamp_value, false);
 	}
 
-	if (likely(!(attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)))
+	if (likely(!(attr->sched_flags && SCHED_FLAG_UTIL_CLAMP)) ||
+		attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_RESET)
 		return;

Parenthesis required for multi-line is statements.

Following chucks not required.

Quoted 16 lines
kernel/sched/core.c +4 -6
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MIN],
-			      attr->sched_util_min, true);
-	}
+				attr->sched_util_min, true);
 
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX)
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MAX],
-			      attr->sched_util_max, true);
-	}
+				attr->sched_util_max, true);
 }
 
 static void uclamp_fork(struct task_struct *p)
author
#6 lore raw

Hi Patrick,

Quoted 75 lines · Patrick Bellasi

Hi Yun,
thanks for keep improving this.

I'm replying here but still considering all other reviewers comments.

Best,
Patrick

On Tue, Nov 03, 2020 at 03:37:56 +0100, Yun Hsiang <[email protected]> wrote...

Quoted 11 lines

If the user wants to stop controlling uclamp and let the task inherit the value from the group, we need a method to reset.

Add SCHED_FLAG_UTIL_CLAMP_RESET flag to allow the user to reset uclamp via sched_setattr syscall.

The policy is

_CLAMP_RESET                           => reset both min and max
_CLAMP_RESET | _CLAMP_MIN              => reset min value
_CLAMP_RESET | _CLAMP_MAX              => reset max value

_CLAMP_RESET | _CLAMP_MIN | _CLAMP_MAX => reset both min and max

This documentation should be added to the uapi header and, most importantly, in:
include/uapi/linux/sched/types.h
where the documentation for struct sched_attr is provided.

Quoted 12 lines
---
 include/uapi/linux/sched.h |  7 +++--
 kernel/sched/core.c        | 59 ++++++++++++++++++++++++++++----------
 2 files changed, 49 insertions(+), 17 deletions(-)
include/uapi/linux/sched.h +0 -0
index 3bac0a8ceab2..6c823ddb1a1e 100644

include/uapi/linux/sched.h struct clone_args { · line 139

Quoted 8 lines
include/uapi/linux/sched.h +1 -0
@@ -132,17 +132,20 @@ struct clone_args {
 #define SCHED_FLAG_KEEP_PARAMS		0x10
 #define SCHED_FLAG_UTIL_CLAMP_MIN	0x20
 #define SCHED_FLAG_UTIL_CLAMP_MAX	0x40
+#define SCHED_FLAG_UTIL_CLAMP_RESET	0x80
 
 #define SCHED_FLAG_KEEP_ALL	(SCHED_FLAG_KEEP_POLICY | \
 				 SCHED_FLAG_KEEP_PARAMS)
 

(Related to the following discussion point)
What about adding in a comment here to call out that the following definitions are "internal only"?
Moreover, we could probably wrap the following two define within an #ifdef __KERNEL__/#endif block

Something like:

+ /*
+ * The following definitions are internal only, do not use them to set
+ * set_{set,get}attr() params. Use instead a valid combination of the
+ * flags defined above.
+ */
+ #ifdef __KERNEL__

include/uapi/linux/sched.h struct clone_args { · line 142

Quoted 4 lines
include/uapi/linux/sched.h +2 -1
 #define SCHED_FLAG_UTIL_CLAMP	(SCHED_FLAG_UTIL_CLAMP_MIN | \
-				 SCHED_FLAG_UTIL_CLAMP_MAX)
+				 SCHED_FLAG_UTIL_CLAMP_MAX | \
+				 SCHED_FLAG_UTIL_CLAMP_RESET)

We need the _RESET flag only here (not below), since this is a UCLAMP feature and it's worth/useful to have a single "all uclamp flags" definition...

Quoted 9 lines

include/uapi/linux/sched.h struct clone_args { · line 148

Quoted 7 lines
include/uapi/linux/sched.h +2 -1
 #define SCHED_FLAG_ALL	(SCHED_FLAG_RESET_ON_FORK	| \
 			 SCHED_FLAG_RECLAIM		| \
 			 SCHED_FLAG_DL_OVERRUN		| \
 			 SCHED_FLAG_KEEP_ALL		| \
-			 SCHED_FLAG_UTIL_CLAMP)
+			 SCHED_FLAG_UTIL_CLAMP		| \
+			 SCHED_FLAG_UTIL_CLAMP_RESET)

... i.e., you can drop the chunk above.

Yes, this chunk is redundant.

Quoted 14 lines

+ #endif /* __KERNEL__ */

Regarding Qais comment, I had the same Dietmar's thought: I doubt there are apps using _FLAGS_ALL from userspace. For DL tasks, since they cannot fork, it makes no sense to specify, for example _RESET_ON_FORK|_RECLAIM. For CFS/RT tasks, where UCLAMP is supported, it makes no sense to specify DL flags.

It's true however that having this def here when it's supposed to be used only internally, can be kind of "confusing", but it's also useful to keep the definition aligned with the flags defined above. The ifdef wrapping proposed above should make this even more explicit.

Quoted 7 lines

Perhaps we can also better call this out also with an additional note right after:

https://elixir.bootlin.com/linux/v5.9.6/source/include/uapi/linux/sched/types.h#L43

In that file, I believe the "Task Utilization Attributes" section can also be improved by adding a description of the _UCLAMP flags semantics.

I think SCHED_FLAG_ALL is for internal kernel use. So I agree with not exporting it to user.

For the "Task Utilization Attributes" section, should I send a patch to add some descriptions for _UCLAMP_{MIN/MAX} first?

Quoted 71 lines
Quoted 8 lines
include/uapi/linux/sched.h +0 -0
 #endif /* _UAPI_LINUX_SCHED_H */
kernel/sched/core.c +0 -0
index 8160ab5263f8..6ae463b64834 100644

kernel/sched/core.c unsigned int uclamp_rq_max_value(struct rq *rq, enum uclamp_id clamp_id, · line 1008

Quoted 6 lines
kernel/sched/core.c +1 -1
@@ -1004,7 +1004,7 @@ unsigned int uclamp_rq_max_value(struct rq *rq, enum uclamp_id clamp_id,
 	return uclamp_idle_value(rq, clamp_id, clamp_value);
 }
 
-static void __uclamp_update_util_min_rt_default(struct task_struct *p)
+static inline void __uclamp_update_util_min_rt_default(struct task_struct *p)
 {

Again, IMO, this is _not_ an unrelated change at all. Actually, I still would like to do one step more and inline this function in the _only place_ where it's used. Qais arguments for not doing that where [1]:

Updating the default rt value is done from different contexts. Hence it is important to document the rules under which this update must happen and ensure the update happens through a common path.

I don't see why these arguments are not satisfied when we inline, e.g.

---8<---

kernel/sched/core.c +6 -19
index d2003a7d5ab5..369074154e1d 100644
@@ -1004,25 +1004,9 @@ unsigned int uclamp_rq_max_value(struct rq *rq, enum uclamp_id clamp_id,
        return uclamp_idle_value(rq, clamp_id, clamp_value);
 }
 
-static void __uclamp_update_util_min_rt_default(struct task_struct *p)
-{
-       unsigned int default_util_min;
-       struct uclamp_se *uc_se;
-
-       lockdep_assert_held(&p->pi_lock);
-
-       uc_se = &p->uclamp_req[UCLAMP_MIN];
-
-       /* Only sync if user didn't override the default */
-       if (uc_se->user_defined)
-               return;
-
-       default_util_min = sysctl_sched_uclamp_util_min_rt_default;
-       uclamp_se_set(uc_se, default_util_min, false);
-}
-
 static void uclamp_update_util_min_rt_default(struct task_struct *p)
 {
+       struct uclamp_se *uc_se = &p->uclamp_req[UCLAMP_MIN];
        struct rq_flags rf;
        struct rq *rq;
 
@@ -1031,7 +1015,11 @@ static void uclamp_update_util_min_rt_default(struct task_struct *p)
 
        /* Protect updates to p->uclamp_* */
        rq = task_rq_lock(p, &rf);
-       __uclamp_update_util_min_rt_default(p);
+
+       /* Only sync if user didn't override the default */
+       if (!uc_se->user_defined)
+               uclamp_se_set(uc_se, sysctl_sched_uclamp_util_min_rt_default, false);
+
        task_rq_unlock(rq, p, &rf);
 }
---8<---
Quoted 1 line

[1] https://lore.kernel.org/lkml/20201028182946.6qfmt7q35ewrjjua@e107158-lin/

I can split this modification to a different patch.

Quoted 24 lines
Quoted 24 lines
kernel/sched/core.c +8 -2
 	unsigned int default_util_min;
 	struct uclamp_se *uc_se;
@@ -1413,8 +1413,14 @@ int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,
 static int uclamp_validate(struct task_struct *p,
 			   const struct sched_attr *attr)
 {
-	unsigned int lower_bound = p->uclamp_req[UCLAMP_MIN].value;
-	unsigned int upper_bound = p->uclamp_req[UCLAMP_MAX].value;
+	unsigned int lower_bound, upper_bound;
+
+	/* Do not check uclamp attributes values in reset case. */
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_RESET)
+		return 0;
+
+	lower_bound = p->uclamp_req[UCLAMP_MIN].value;
+	upper_bound = p->uclamp_req[UCLAMP_MAX].value;
 
 	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
 		lower_bound = attr->sched_util_min;
@@ -1438,20 +1444,43 @@ static int uclamp_validate(struct task_struct *p,
 	return 0;
 }
 
Quoted 7 lines

kernel/sched/core.c static int uclamp_validate(struct task_struct *p, · line 1449

Quoted 3 lines
kernel/sched/core.c +3 -0
+static int uclamp_reset(enum uclamp_id clamp_id, unsigned long flags)
+{
+	/* No _UCLAMP_RESET flag set: do not reset */
              ^^^^^^^^^^^^^

Maybe better using the full flag name for all these?

Got it!

Quoted 27 lines
Quoted 6 lines
kernel/sched/core.c +6 -0
+	if (!(flags & SCHED_FLAG_UTIL_CLAMP_RESET))
+		return false;
+
+	/* Only _UCLAMP_RESET flag set: reset both clamps */
+	if (!(flags & (SCHED_FLAG_UTIL_CLAMP_MIN | SCHED_FLAG_UTIL_CLAMP_MAX)))
+		return true;

kernel/sched/core.c static int uclamp_validate(struct task_struct *p, · line 1465

Quoted 10 lines
kernel/sched/core.c +10 -0
+
+	/* Both _UCLAMP_RESET and _UCLAMP_MIN flags are set: reset only min */
+	if ((flags & SCHED_FLAG_UTIL_CLAMP_MIN) && clamp_id == UCLAMP_MIN)
+		return true;
+
+	/* Both _UCLAMP_RESET and _UCLAMP_MAX flags are set: reset only max */
+	if ((flags & SCHED_FLAG_UTIL_CLAMP_MAX) && clamp_id == UCLAMP_MAX)
+		return true;
+
+	return false;

I was suggesting maybe we need READ_ONCE() in the if above but did not addressed previous Dietmar's question [2] on why.

The function above has a correct semantics only when the ordering of the if statement is strictly observed.

Now, since we don't have any data dependency on the multiple if cases, are we sure an overzealous compiler will never come up with some "optimized reordering" of the checks required?

Quoted 7 lines

IOW, if the compiler could scramble the checks on an optimization, we would get a wrong semantics which is also very difficult do debug. Hence the idea to use READ_ONCE, to both tell the compiler to not even attempt reordering those checks and also to better document the code about the importance of the ordering on those checks.

Is now more clear? Does that makes sense?

I can undertand what your worries. But I'm not sure is it really needed. If there is no other concern I can add it.

Quoted 40 lines

[2] https://lore.kernel.org/lkml/[email protected]/

Quoted 37 lines
kernel/sched/core.c +10 -5
+}
+
 static void __setscheduler_uclamp(struct task_struct *p,
 				  const struct sched_attr *attr)
 {
 	enum uclamp_id clamp_id;
 
 	/*
-	 * On scheduling class change, reset to default clamps for tasks
-	 * without a task-specific value.
+	 * Reset to default clamps on forced _UCLAMP_RESET (always) and
+	 * for tasks without a task-specific value (on scheduling class change).
 	 */
 	for_each_clamp_id(clamp_id) {
+		unsigned int clamp_value;
 		struct uclamp_se *uc_se = &p->uclamp_req[clamp_id];
 
 		/* Keep using defined clamps across class changes */
-		if (uc_se->user_defined)
+		if (!uclamp_reset(clamp_id, attr->sched_flags) &&
+				uc_se->user_defined)
 			continue;
 
 		/*
@@ -1459,24 +1488,24 @@ static void __setscheduler_uclamp(struct task_struct *p,
 		 * at runtime.
 		 */
 		if (unlikely(rt_task(p) && clamp_id == UCLAMP_MIN))
-			__uclamp_update_util_min_rt_default(p);
+			clamp_value = sysctl_sched_uclamp_util_min_rt_default;
 		else
-			uclamp_se_set(uc_se, uclamp_none(clamp_id), false);
+			clamp_value = uclamp_none(clamp_id);
 
+		uclamp_se_set(uc_se, clamp_value, false);
 	}
 
Quoted 6 lines

kernel/sched/core.c static void __setscheduler_uclamp(struct task_struct *p, · line 1500

Quoted 4 lines
kernel/sched/core.c +2 -1
-	if (likely(!(attr->sched_flags & SCHED_FLAG_UTIL_CLAMP)))
+	if (likely(!(attr->sched_flags && SCHED_FLAG_UTIL_CLAMP)) ||
+		attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_RESET)
 		return;

Parenthesis required for multi-line is statements.

Sorry for wrong coding style. I'll fix it.

Quoted 2 lines

Following chucks not required.

I'll drop these chunks.

Quoted 18 lines
Quoted 17 lines
kernel/sched/core.c +4 -6
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MIN],
-			      attr->sched_util_min, true);
-	}
+				attr->sched_util_min, true);
 
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX)
 		uclamp_se_set(&p->uclamp_req[UCLAMP_MAX],
-			      attr->sched_util_max, true);
-	}
+				attr->sched_util_max, true);
 }
 
 static void uclamp_fork(struct task_struct *p)
#7 lore raw
Quoted 2 lines · Yun Hsiang

I think SCHED_FLAG_ALL is for internal kernel use. So I agree with not exporting it to user.

+1 for the #ifdef __kernel__

Quoted 33 lines
Quoted 33 lines
Quoted 6 lines
(quoted diff) +6 -0
+	if (!(flags & SCHED_FLAG_UTIL_CLAMP_RESET))
+		return false;
+
+	/* Only _UCLAMP_RESET flag set: reset both clamps */
+	if (!(flags & (SCHED_FLAG_UTIL_CLAMP_MIN | SCHED_FLAG_UTIL_CLAMP_MAX)))
+		return true;

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+	/* Both _UCLAMP_RESET and _UCLAMP_MIN flags are set: reset only min */
+	if ((flags & SCHED_FLAG_UTIL_CLAMP_MIN) && clamp_id == UCLAMP_MIN)
+		return true;
+
+	/* Both _UCLAMP_RESET and _UCLAMP_MAX flags are set: reset only max */
+	if ((flags & SCHED_FLAG_UTIL_CLAMP_MAX) && clamp_id == UCLAMP_MAX)
+		return true;
+
+	return false;

I was suggesting maybe we need READ_ONCE() in the if above but did not addressed previous Dietmar's question [2] on why.

The function above has a correct semantics only when the ordering of the if statement is strictly observed.

Now, since we don't have any data dependency on the multiple if cases, are we sure an overzealous compiler will never come up with some "optimized reordering" of the checks required?

IOW, if the compiler could scramble the checks on an optimization, we would get a wrong semantics which is also very difficult do debug. Hence the idea to use READ_ONCE, to both tell the compiler to not even attempt reordering those checks and also to better document the code about the importance of the ordering on those checks.

Quoted 4 lines
Quoted 1 line

Is now more clear? Does that makes sense?

I can undertand what your worries. But I'm not sure is it really needed. If there is no other concern I can add it.

I too don't think the compiler has a power to do such an optimization. It must preserve the order of the checks even if it found a more efficient way to perform them.

Thanks

--
Qais Yousef

maintainer
#8 lore raw
In reply to Patrick Bellasi
Quoted 30 lines · Patrick Bellasi
Quoted 9 lines
(quoted diff) +9 -0
+static int uclamp_reset(enum uclamp_id clamp_id, unsigned long flags)
+{
+	/* No _UCLAMP_RESET flag set: do not reset */
+	if (!(flags & SCHED_FLAG_UTIL_CLAMP_RESET))
+		return false;
+
+	/* Only _UCLAMP_RESET flag set: reset both clamps */
+	if (!(flags & (SCHED_FLAG_UTIL_CLAMP_MIN | SCHED_FLAG_UTIL_CLAMP_MAX)))
+		return true;

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+	/* Both _UCLAMP_RESET and _UCLAMP_MIN flags are set: reset only min */
+	if ((flags & SCHED_FLAG_UTIL_CLAMP_MIN) && clamp_id == UCLAMP_MIN)
+		return true;
+
+	/* Both _UCLAMP_RESET and _UCLAMP_MAX flags are set: reset only max */
+	if ((flags & SCHED_FLAG_UTIL_CLAMP_MAX) && clamp_id == UCLAMP_MAX)
+		return true;
+
+	return false;

I was suggesting maybe we need READ_ONCE() in the if above but did not addressed previous Dietmar's question [2] on why.

The function above has a correct semantics only when the ordering of the if statement is strictly observed.

Now, since we don't have any data dependency on the multiple if cases, are we sure an overzealous compiler will never come up with some "optimized reordering" of the checks required?

Quoted 7 lines

IOW, if the compiler could scramble the checks on an optimization, we would get a wrong semantics which is also very difficult do debug. Hence the idea to use READ_ONCE, to both tell the compiler to not even attempt reordering those checks and also to better document the code about the importance of the ordering on those checks.

Is now more clear? Does that makes sense?

I don't think the compiler is allowed to do as you fear. Specifically it cannot move the first branch down because that would change the meaning of this function and affect observable behaviour even in the traditional single-threaded environment.

maintainer
#9 lore raw
In reply to Yun Hsiang
Quoted 12 lines · Yun Hsiang

If the user wants to stop controlling uclamp and let the task inherit the value from the group, we need a method to reset.

Add SCHED_FLAG_UTIL_CLAMP_RESET flag to allow the user to reset uclamp via sched_setattr syscall.

The policy is

_CLAMP_RESET                           => reset both min and max
_CLAMP_RESET | _CLAMP_MIN              => reset min value
_CLAMP_RESET | _CLAMP_MAX              => reset max value

_CLAMP_RESET | _CLAMP_MIN | _CLAMP_MAX => reset both min and max

The obvious alternative would be to use a magic value in sched_util_{min,max} to indicate reset. After all, we strictly enforce the values are inside [0,1024], which leaves us with many unused values.

Specifically -1 comes to mind. It would allow doing this without an extra flag, OTOH the explicit flag is well, more explicit.

I don't have a strong preference either way, but I wanted to make sure it was considered, and perhaps we can record why this isn't as nice a solution, dunno.

#10 lore raw
Quoted 18 lines · Peter Zijlstra
Quoted 12 lines · Yun Hsiang

If the user wants to stop controlling uclamp and let the task inherit the value from the group, we need a method to reset.

Add SCHED_FLAG_UTIL_CLAMP_RESET flag to allow the user to reset uclamp via sched_setattr syscall.

The policy is

_CLAMP_RESET                           => reset both min and max
_CLAMP_RESET | _CLAMP_MIN              => reset min value
_CLAMP_RESET | _CLAMP_MAX              => reset max value

_CLAMP_RESET | _CLAMP_MIN | _CLAMP_MAX => reset both min and max

The obvious alternative would be to use a magic value in sched_util_{min,max} to indicate reset. After all, we strictly enforce the values are inside [0,1024], which leaves us with many unused values.

Quoted 6 lines

Specifically -1 comes to mind. It would allow doing this without an extra flag, OTOH the explicit flag is well, more explicit.

I don't have a strong preference either way, but I wanted to make sure it was considered, and perhaps we can record why this isn't as nice a solution, dunno.

IMHO the '-1' magic value approach is cleaner. Did some light testing on it.

From 2e6a64fac4f2f66a2c6246de33db22c467fa7d33 Mon Sep 17 00:00:00 2001
From: Dietmar Eggemann <[email protected]>
Date: Wed, 11 Nov 2020 01:14:33 +0100
Subject: [PATCH] sched/uclamp: Allow to reset a task uclamp constraint value

In case the user wants to stop controlling a uclamp constraint value for a task, use the magic value -1 in sched_util_{min,max} with the appropriate sched_flags (SCHED_FLAG_UTIL_CLAMP_{MIN,MAX}) to indicate the reset.

The advantage over the 'additional flag' approach (i.e. introducing SCHED_FLAG_UTIL_CLAMP_RESET) is that no additional flag has to be exported via uapi. This avoids the need to document how this new flag has be used in conjunction with the existing uclamp related flags.

The following subtle issue is fixed as well. When a uclamp constraint value is set on a !user_defined uclamp_se it is currently first reset and then set.
Fix this by AND'ing !user_defined with !SCHED_FLAG_UTIL_CLAMP which stands for the 'sched class change' case.
The related condition 'if (uc_se->user_defined)' moved from __setscheduler_uclamp() into uclamp_reset().

---
 include/uapi/linux/sched/types.h |  4 +-
 kernel/sched/core.c              | 70 +++++++++++++++++++++++---------
 2 files changed, 53 insertions(+), 21 deletions(-)
include/uapi/linux/sched/types.h +2 -2
index c852153ddb0d..b9165f17dddc 100644
@@ -115,8 +115,8 @@ struct sched_attr {
 	__u64 sched_period;
 
 	/* Utilization hints */
-	__u32 sched_util_min;
-	__u32 sched_util_max;
+	__s32 sched_util_min;
+	__s32 sched_util_max;
 
 };
 
kernel/sched/core.c +51 -19

109 lines not on this page · raw message

Signature · 1 line
2.17.1
maintainer
#11 lore raw

include/uapi/linux/sched/types.h struct sched_attr { · line 119

Quoted 12 lines · Dietmar Eggemann
include/uapi/linux/sched/types.h +2 -2
index c852153ddb0d..b9165f17dddc 100644
@@ -115,8 +115,8 @@ struct sched_attr {
 	__u64 sched_period;
 
 	/* Utilization hints */
-	__u32 sched_util_min;
-	__u32 sched_util_max;
+	__s32 sched_util_min;
+	__s32 sched_util_max;

So that's UAPI, not sure we can change the type here.

Quoted 25 lines
include/uapi/linux/sched/types.h +0 -0
 };
 
kernel/sched/core.c +4 -9
index 3dc415f58bd7..caaa2a8434b9 100644
@@ -1413,17 +1413,24 @@ int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,
 static int uclamp_validate(struct task_struct *p,
 			   const struct sched_attr *attr)
 {
-	unsigned int lower_bound = p->uclamp_req[UCLAMP_MIN].value;
-	unsigned int upper_bound = p->uclamp_req[UCLAMP_MAX].value;
+	int util_min = p->uclamp_req[UCLAMP_MIN].value;
+	int util_max = p->uclamp_req[UCLAMP_MAX].value;
 
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
-		lower_bound = attr->sched_util_min;
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX)
-		upper_bound = attr->sched_util_max;
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+		util_min = attr->sched_util_min;
 
-	if (lower_bound > upper_bound)
-		return -EINVAL;
-	if (upper_bound > SCHED_CAPACITY_SCALE)

kernel/sched/core.c int sysctl_sched_uclamp_handler(struct ctl_table *table, int write, · line 1431

Quoted 10 lines
kernel/sched/core.c +10 -0
+		if (util_min < -1 || util_min > SCHED_CAPACITY_SCALE)
+			return -EINVAL;
+	}
+
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+		util_max = attr->sched_util_max;
+
+		if (util_max < -1 || util_max > SCHED_CAPACITY_SCALE)
+			return -EINVAL;
+	}

Luckily we can write that range as a single branch like:

	if (util_{min,max} + 1 > SCHED_CAPACITY_SCALE+1)

which assumes u32 :-)

kernel/sched/core.c int sysctl_sched_uclamp_handler(struct ctl_table *table, int write, · line 1434

Quoted 3 lines
kernel/sched/core.c +2 -0
+
+	if (util_min != -1 && util_max != -1 && util_min > util_max)
 		return -EINVAL;

I think that will compile as is, otherwise write it like ~0u, which is the same bit pattern.

author
#12 lore raw
Quoted 8 lines · Peter Zijlstra
Quoted 7 lines · Dietmar Eggemann
include/uapi/linux/sched/types.h +0 -0
index c852153ddb0d..b9165f17dddc 100644
@@ -115,8 +115,8 @@ struct sched_attr {
 	__u64 sched_period;
 
Quoted 7 lines

include/uapi/linux/sched/types.h struct sched_attr { · line 119

Quoted 5 lines
include/uapi/linux/sched/types.h +2 -2
 	/* Utilization hints */
-	__u32 sched_util_min;
-	__u32 sched_util_max;
+	__s32 sched_util_min;
+	__s32 sched_util_max;

So that's UAPI, not sure we can change the type here.

+1
I am also concerned about changing UAPI.
But if we can chage sched_util_{min/max} to __s32, use -1 to reset is better than adding flags.

Quoted 50 lines
Quoted 26 lines
include/uapi/linux/sched/types.h +0 -0
 };
 
kernel/sched/core.c +4 -9
index 3dc415f58bd7..caaa2a8434b9 100644
@@ -1413,17 +1413,24 @@ int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,
 static int uclamp_validate(struct task_struct *p,
 			   const struct sched_attr *attr)
 {
-	unsigned int lower_bound = p->uclamp_req[UCLAMP_MIN].value;
-	unsigned int upper_bound = p->uclamp_req[UCLAMP_MAX].value;
+	int util_min = p->uclamp_req[UCLAMP_MIN].value;
+	int util_max = p->uclamp_req[UCLAMP_MAX].value;
 
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)
-		lower_bound = attr->sched_util_min;
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX)
-		upper_bound = attr->sched_util_max;
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+		util_min = attr->sched_util_min;
 
-	if (lower_bound > upper_bound)
-		return -EINVAL;
-	if (upper_bound > SCHED_CAPACITY_SCALE)

kernel/sched/core.c int sysctl_sched_uclamp_handler(struct ctl_table *table, int write, · line 1431

Quoted 10 lines
kernel/sched/core.c +10 -0
+		if (util_min < -1 || util_min > SCHED_CAPACITY_SCALE)
+			return -EINVAL;
+	}
+
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+		util_max = attr->sched_util_max;
+
+		if (util_max < -1 || util_max > SCHED_CAPACITY_SCALE)
+			return -EINVAL;
+	}

Luckily we can write that range as a single branch like:

	if (util_{min,max} + 1 > SCHED_CAPACITY_SCALE+1)

which assumes u32 :-)

kernel/sched/core.c int sysctl_sched_uclamp_handler(struct ctl_table *table, int write, · line 1434

Quoted 3 lines
kernel/sched/core.c +2 -0
+
+	if (util_min != -1 && util_max != -1 && util_min > util_max)
 		return -EINVAL;

I think that will compile as is, otherwise write it like ~0u, which is the same bit pattern.

#13 lore raw
In reply to Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 7 lines · Dietmar Eggemann
include/uapi/linux/sched/types.h +0 -0
index c852153ddb0d..b9165f17dddc 100644
@@ -115,8 +115,8 @@ struct sched_attr {
 	__u64 sched_period;
 
Quoted 7 lines

include/uapi/linux/sched/types.h struct sched_attr { · line 119

Quoted 5 lines
include/uapi/linux/sched/types.h +2 -2
 	/* Utilization hints */
-	__u32 sched_util_min;
-	__u32 sched_util_max;
+	__s32 sched_util_min;
+	__s32 sched_util_max;

So that's UAPI, not sure we can change the type here.

Yes, will remove this chunk. Not needed.

I probably should add some documentation there:

(quoted diff) +0 -0

b/include/uapi/linux/sched/types.h
index c852153ddb0d..f2c4589d4dbf 100644

include/uapi/linux/sched/types.h +2 -0
@@ -96,6 +96,8 @@ struct sched_param {
  * on a CPU with a capacity big enough to fit the specified value.
  * A task with a max utilization value smaller than 1024 is more likely
  * scheduled on a CPU with no more capacity than the specified value.
+ *
+ * A task utilization boundary can be reset by setting the attribute to -1.
  */
 struct sched_attr {
        __u32 size;

[...]

Quoted 9 lines

include/uapi/linux/sched/types.h

Quoted 6 lines
include/uapi/linux/sched/types.h +6 -0
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+		util_max = attr->sched_util_max;
+
+		if (util_max < -1 || util_max > SCHED_CAPACITY_SCALE)
+			return -EINVAL;
+	}

Luckily we can write that range as a single branch like:

Quoted 3 lines
	if (util_{min,max} + 1 > SCHED_CAPACITY_SCALE+1)

which assumes u32 :-)

Cool, will change it.

Quoted 6 lines

include/uapi/linux/sched/types.h

Quoted 3 lines
include/uapi/linux/sched/types.h +2 -0
+
+	if (util_min != -1 && util_max != -1 && util_min > util_max)
 		return -EINVAL;

I think that will compile as is, otherwise write it like ~0u, which is the same bit pattern.

Yes, it compiles for me (arm64, gcc 9.2 and arm, gcc 8.3). Started a 0-Day build job to make sure.

Will do some more testing before sending out the updated version.

#14 lore raw
In reply to Dietmar Eggemann
Quoted 19 lines · Dietmar Eggemann
Quoted 18 lines · Peter Zijlstra
Quoted 12 lines · Yun Hsiang

If the user wants to stop controlling uclamp and let the task inherit the value from the group, we need a method to reset.

Add SCHED_FLAG_UTIL_CLAMP_RESET flag to allow the user to reset uclamp via sched_setattr syscall.

The policy is

_CLAMP_RESET                           => reset both min and max
_CLAMP_RESET | _CLAMP_MIN              => reset min value
_CLAMP_RESET | _CLAMP_MAX              => reset max value

_CLAMP_RESET | _CLAMP_MIN | _CLAMP_MAX => reset both min and max

The obvious alternative would be to use a magic value in sched_util_{min,max} to indicate reset. After all, we strictly enforce the values are inside [0,1024], which leaves us with many unused values.

Quoted 8 lines
Quoted 6 lines

Specifically -1 comes to mind. It would allow doing this without an extra flag, OTOH the explicit flag is well, more explicit.

I don't have a strong preference either way, but I wanted to make sure it was considered, and perhaps we can record why this isn't as nice a solution, dunno.

IMHO the '-1' magic value approach is cleaner. Did some light testing on it.

I assume we agree then that we don't want to explicitly document this quirky feature and keep it for advanced users?

I am wary of the UAPI change that is both explicit and implicit. It explicitly requests a reset, but implicitly requests a cgroup behavior change.

With this magic value at least we can more easily return an error if we decided to deprecate it, which has been my main ask so far. I don't want us to end up not being able to easily modify this code in the future.

I don't have strong opinion too though.

If you or Yun would still like to send the patch to protect SCHED_FLAG_UTIL_CLAMP and SCHED_FLAG_ALL with __kernel__ that'd be great.

Quoted 58 lines

From 2e6a64fac4f2f66a2c6246de33db22c467fa7d33 Mon Sep 17 00:00:00 2001
From: Dietmar Eggemann <[email protected]>
Date: Wed, 11 Nov 2020 01:14:33 +0100
Subject: [PATCH] sched/uclamp: Allow to reset a task uclamp constraint value

In case the user wants to stop controlling a uclamp constraint value for a task, use the magic value -1 in sched_util_{min,max} with the appropriate sched_flags (SCHED_FLAG_UTIL_CLAMP_{MIN,MAX}) to indicate the reset.

The advantage over the 'additional flag' approach (i.e. introducing SCHED_FLAG_UTIL_CLAMP_RESET) is that no additional flag has to be exported via uapi. This avoids the need to document how this new flag has be used in conjunction with the existing uclamp related flags.

The following subtle issue is fixed as well. When a uclamp constraint value is set on a !user_defined uclamp_se it is currently first reset and then set.
Fix this by AND'ing !user_defined with !SCHED_FLAG_UTIL_CLAMP which stands for the 'sched class change' case.
The related condition 'if (uc_se->user_defined)' moved from __setscheduler_uclamp() into uclamp_reset().

---
 include/uapi/linux/sched/types.h |  4 +-
 kernel/sched/core.c              | 70 +++++++++++++++++++++++---------
 2 files changed, 53 insertions(+), 21 deletions(-)
include/uapi/linux/sched/types.h +2 -2
index c852153ddb0d..b9165f17dddc 100644
@@ -115,8 +115,8 @@ struct sched_attr {
 	__u64 sched_period;
 
 	/* Utilization hints */
-	__u32 sched_util_min;
-	__u32 sched_util_max;
+	__s32 sched_util_min;
+	__s32 sched_util_max;
 
 };
 
kernel/sched/core.c +2 -3
index 3dc415f58bd7..caaa2a8434b9 100644
@@ -1413,17 +1413,24 @@ int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,
 static int uclamp_validate(struct task_struct *p,
 			   const struct sched_attr *attr)
 {
-	unsigned int lower_bound = p->uclamp_req[UCLAMP_MIN].value;
-	unsigned int upper_bound = p->uclamp_req[UCLAMP_MAX].value;
+	int util_min = p->uclamp_req[UCLAMP_MIN].value;
+	int util_max = p->uclamp_req[UCLAMP_MAX].value;
 
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN)

kernel/sched/core.c int sysctl_sched_uclamp_handler(struct ctl_table *table, int write, · line 1427

Quoted 10 lines
kernel/sched/core.c +2 -5
-		lower_bound = attr->sched_util_min;
-	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX)
-		upper_bound = attr->sched_util_max;
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MIN) {
+		util_min = attr->sched_util_min;
 
 
 
-	if (lower_bound > upper_bound)
-		return -EINVAL;

You removed this check and didn't replace it with equivalent one?

Quoted 4 lines
kernel/sched/core.c +3 -1
-	if (upper_bound > SCHED_CAPACITY_SCALE)
+		if (util_min < -1 || util_min > SCHED_CAPACITY_SCALE)
+			return -EINVAL;
+	}

kernel/sched/core.c int sysctl_sched_uclamp_handler(struct ctl_table *table, int write, · line 1434

Quoted 10 lines
kernel/sched/core.c +9 -0
+
+	if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP_MAX) {
+		util_max = attr->sched_util_max;
+
+		if (util_max < -1 || util_max > SCHED_CAPACITY_SCALE)
+			return -EINVAL;
+	}
+
+	if (util_min != -1 && util_max != -1 && util_min > util_max)
 		return -EINVAL;

Ah, it is here. Never mind.

The approach looks good to me in general.

Thanks

--
Qais Yousef

#15 lore raw
Quoted 3 lines · Qais Yousef
Quoted 2 lines · Dietmar Eggemann
Quoted 1 line · Peter Zijlstra

On Tue, Nov 03, 2020 at 10:37:56AM +0800, Yun Hsiang wrote:

[...]

Quoted 9 lines

I assume we agree then that we don't want to explicitly document this quirky feature and keep it for advanced users?

I am wary of the UAPI change that is both explicit and implicit. It explicitly requests a reset, but implicitly requests a cgroup behavior change.

With this magic value at least we can more easily return an error if we decided to deprecate it, which has been my main ask so far. I don't want us to end up not being able to easily modify this code in the future.

Another advantage for this 'magic value' approach.

Quoted 4 lines

I don't have strong opinion too though.

If you or Yun would still like to send the patch to protect SCHED_FLAG_UTIL_CLAMP and SCHED_FLAG_ALL with __kernel__ that'd be great.

Ah yes! Can add an extra patch for this when sending out the next version.

[...]

#16 lore raw
Quoted 4 lines · Dietmar Eggemann
Quoted 3 lines · Qais Yousef
Quoted 2 lines · Dietmar Eggemann
Quoted 1 line · Peter Zijlstra

On Tue, Nov 03, 2020 at 10:37:56AM +0800, Yun Hsiang wrote:

[...]

Quoted 4 lines
Quoted 2 lines

If you or Yun would still like to send the patch to protect SCHED_FLAG_UTIL_CLAMP and SCHED_FLAG_ALL with __kernel__ that'd be great.

Ah yes! Can add an extra patch for this when sending out the next version.

On second thought, why should we risk a change in UAPI? Since we're now not introducing a new flag the meaning of SCHED_FLAG_ALL or SCHED_FLAG_UTIL_CLAMP won't change.

#17 lore raw
Quoted 8 lines · Dietmar Eggemann
Quoted 4 lines · Dietmar Eggemann
Quoted 3 lines · Qais Yousef
Quoted 2 lines · Dietmar Eggemann
Quoted 1 line · Peter Zijlstra

On Tue, Nov 03, 2020 at 10:37:56AM +0800, Yun Hsiang wrote:

[...]

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

If you or Yun would still like to send the patch to protect SCHED_FLAG_UTIL_CLAMP and SCHED_FLAG_ALL with __kernel__ that'd be great.

Ah yes! Can add an extra patch for this when sending out the next version.

On second thought, why should we risk a change in UAPI? Since we're now not introducing a new flag the meaning of SCHED_FLAG_ALL or SCHED_FLAG_UTIL_CLAMP won't change.

It's a judgement call. Hide them now where it's likely there are no users and hope we won't have to revert it. Or just ignore it and treat it as an ABI and make sure no one change them later.

My judgement call it's better to introduce the __kernel__ while we can. But I can't say for sure nothing will break. All I know it'd be easy to revert if it does cause breakage.

You get to choose :-)

Thanks

--
Qais Yousef