Skip to contentSkip to conversation

selftests/cgroup: Adjust cpu.max quota based on HZ

PATCH Superseded Joe Simmons-Talbott 4 participants 5 messages Jun 22 – Jun 23

Subsystems (2)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 selftests/cgroup: Adjust cpu.max quota based on HZ

For lower HZ values a quota of 1000us is much lower than the amount of microseconds per tick which makes the tests test_cpucg_max and test_cpugc_max_nested fail. Use the amount of microseconds per tick as the quota value.

---
changes since v1:
- Try checking /proc/config.gz to get the actual kernel HZ value and fallback to 1000 if the value cannot be determined.

 tools/testing/selftests/cgroup/test_cpu.c | 33 +++++++++++++++++++++--
 1 file changed, 31 insertions(+), 2 deletions(-)
tools/testing/selftests/cgroup/test_cpu.c +31 -2
index 7a40d76b9548..65e09555309f 100644
@@ -639,6 +639,29 @@ test_cpucg_nested_weight_underprovisioned(const char *root)
 	return run_cpucg_nested_weight_test(root, false);
 }
 
+/*
+ * Best effort attempt to get the kernel's HZ value from the config.
+ * Return the HZ value if found otherwise return -1 to indicate failure.
+ */
+static long
+_get_config_hz(void)
+{
+	long hz = -1;
+	FILE *f;
+	char cmd[256] = "zcat /proc/config.gz 2>/dev/null | grep '^CONFIG_HZ='";
+
+	f = popen(cmd, "r");
+
+	if (!f)
+		goto out;
+
+	fscanf(f, "CONFIG_HZ=%ld", &hz);
+
+out:
+	pclose(f);
+	return hz;
+}
+
 /*
  * This test creates a cgroup with some maximum value within a period, and
  * verifies that a process in the cgroup is not overscheduled.
@@ -646,7 +669,8 @@ test_cpucg_nested_weight_underprovisioned(const char *root)
 static int test_cpucg_max(const char *root)
 {
 	int ret = KSFT_FAIL;
-	long quota_usec = 1000;
+	long hz = _get_config_hz();
+	long quota_usec;
 	long default_period_usec = 100000; /* cpu.max's default period */
 	long duration_seconds = 1;
 
@@ -655,6 +679,8 @@ static int test_cpucg_max(const char *root)
 	char *cpucg;
 	char quota_buf[32];
 
+	quota_usec = hz != -1 ? USEC_PER_SEC / hz : 1000;
+
 	snprintf(quota_buf, sizeof(quota_buf), "%ld", quota_usec);
 
 	cpucg = cg_name(root, "cpucg_test");
@@ -710,7 +736,8 @@ static int test_cpucg_max(const char *root)
 static int test_cpucg_max_nested(const char *root)
 {
 	int ret = KSFT_FAIL;
-	long quota_usec = 1000;
+	long quota_usec;
+	long hz = _get_config_hz();
 	long default_period_usec = 100000; /* cpu.max's default period */
 	long duration_seconds = 1;
 
@@ -719,6 +746,8 @@ static int test_cpucg_max_nested(const char *root)
 	char *parent, *child;
 	char quota_buf[32];
 
+	quota_usec = hz != -1 ? USEC_PER_SEC / hz : 1000;
+
 	snprintf(quota_buf, sizeof(quota_buf), "%ld", quota_usec);
 
 	parent = cg_name(root, "cpucg_parent");
Signature · 1 line
2.54.0
#1 lore raw

Hi Joe,

One comment on the fallback:

  quota_usec = hz != -1 ? USEC_PER_SEC / hz : 1000;

When HZ can't be determined (no CONFIG_IKCONFIG_PROC, or zcat missing), the fallback to 1000 is the exact value that fails at low HZ — so this doesn't actually fix such kernels. A larger fallback (e.g. 10000, the HZ=100 equivalent) would make the tests robust regardless of whether the config is exposed.

Quoted 86 lines · Joe Simmons-Talbott

For lower HZ values a quota of 1000us is much lower than the amount of microseconds per tick which makes the tests test_cpucg_max and test_cpugc_max_nested fail. Use the amount of microseconds per tick as the quota value.

---
changes since v1:
- Try checking /proc/config.gz to get the actual kernel HZ value and fallback to 1000 if the value cannot be determined.

 tools/testing/selftests/cgroup/test_cpu.c | 33 +++++++++++++++++++++--
 1 file changed, 31 insertions(+), 2 deletions(-)
tools/testing/selftests/cgroup/test_cpu.c +31 -2
index 7a40d76b9548..65e09555309f 100644
@@ -639,6 +639,29 @@ test_cpucg_nested_weight_underprovisioned(const char *root)
 	return run_cpucg_nested_weight_test(root, false);
 }
 
+/*
+ * Best effort attempt to get the kernel's HZ value from the config.
+ * Return the HZ value if found otherwise return -1 to indicate failure.
+ */
+static long
+_get_config_hz(void)
+{
+	long hz = -1;
+	FILE *f;
+	char cmd[256] = "zcat /proc/config.gz 2>/dev/null | grep '^CONFIG_HZ='";
+
+	f = popen(cmd, "r");
+
+	if (!f)
+		goto out;
+
+	fscanf(f, "CONFIG_HZ=%ld", &hz);
+
+out:
+	pclose(f);
+	return hz;
+}
+
 /*
  * This test creates a cgroup with some maximum value within a period, and
  * verifies that a process in the cgroup is not overscheduled.
@@ -646,7 +669,8 @@ test_cpucg_nested_weight_underprovisioned(const char *root)
 static int test_cpucg_max(const char *root)
 {
 	int ret = KSFT_FAIL;
-	long quota_usec = 1000;
+	long hz = _get_config_hz();
+	long quota_usec;
 	long default_period_usec = 100000; /* cpu.max's default period */
 	long duration_seconds = 1;
 
@@ -655,6 +679,8 @@ static int test_cpucg_max(const char *root)
 	char *cpucg;
 	char quota_buf[32];
 
+	quota_usec = hz != -1 ? USEC_PER_SEC / hz : 1000;
+
 	snprintf(quota_buf, sizeof(quota_buf), "%ld", quota_usec);
 
 	cpucg = cg_name(root, "cpucg_test");
@@ -710,7 +736,8 @@ static int test_cpucg_max(const char *root)
 static int test_cpucg_max_nested(const char *root)
 {
 	int ret = KSFT_FAIL;
-	long quota_usec = 1000;
+	long quota_usec;
+	long hz = _get_config_hz();
 	long default_period_usec = 100000; /* cpu.max's default period */
 	long duration_seconds = 1;
 
@@ -719,6 +746,8 @@ static int test_cpucg_max_nested(const char *root)
 	char *parent, *child;
 	char quota_buf[32];
 
+	quota_usec = hz != -1 ? USEC_PER_SEC / hz : 1000;
+
 	snprintf(quota_buf, sizeof(quota_buf), "%ld", quota_usec);
 
 	parent = cg_name(root, "cpucg_parent");
#2 lore raw
Quoted 5 lines · Tao Cui

Hi Joe,

One comment on the fallback:

Quoted 7 lines
  quota_usec = hz != -1 ? USEC_PER_SEC / hz : 1000;

When HZ can't be determined (no CONFIG_IKCONFIG_PROC, or zcat missing), the fallback to 1000 is the exact value that fails at low HZ — so this doesn't actually fix such kernels. A larger fallback (e.g. 10000, the HZ=100 equivalent) would make the tests robust regardless of whether the config is exposed.

Hi Tao Cui,

Thank you for your review.

I am happy to use 10000 as the fallback value. I will address this as well as the sashiko comments in v3.

Thanks,
Joe

Quoted 90 lines · Joe Simmons-Talbott
Quoted 86 lines

For lower HZ values a quota of 1000us is much lower than the amount of microseconds per tick which makes the tests test_cpucg_max and test_cpugc_max_nested fail. Use the amount of microseconds per tick as the quota value.

---
changes since v1:
- Try checking /proc/config.gz to get the actual kernel HZ value and fallback to 1000 if the value cannot be determined.

 tools/testing/selftests/cgroup/test_cpu.c | 33 +++++++++++++++++++++--
 1 file changed, 31 insertions(+), 2 deletions(-)
tools/testing/selftests/cgroup/test_cpu.c +31 -2
index 7a40d76b9548..65e09555309f 100644
@@ -639,6 +639,29 @@ test_cpucg_nested_weight_underprovisioned(const char *root)
 	return run_cpucg_nested_weight_test(root, false);
 }
 
+/*
+ * Best effort attempt to get the kernel's HZ value from the config.
+ * Return the HZ value if found otherwise return -1 to indicate failure.
+ */
+static long
+_get_config_hz(void)
+{
+	long hz = -1;
+	FILE *f;
+	char cmd[256] = "zcat /proc/config.gz 2>/dev/null | grep '^CONFIG_HZ='";
+
+	f = popen(cmd, "r");
+
+	if (!f)
+		goto out;
+
+	fscanf(f, "CONFIG_HZ=%ld", &hz);
+
+out:
+	pclose(f);
+	return hz;
+}
+
 /*
  * This test creates a cgroup with some maximum value within a period, and
  * verifies that a process in the cgroup is not overscheduled.
@@ -646,7 +669,8 @@ test_cpucg_nested_weight_underprovisioned(const char *root)
 static int test_cpucg_max(const char *root)
 {
 	int ret = KSFT_FAIL;
-	long quota_usec = 1000;
+	long hz = _get_config_hz();
+	long quota_usec;
 	long default_period_usec = 100000; /* cpu.max's default period */
 	long duration_seconds = 1;
 
@@ -655,6 +679,8 @@ static int test_cpucg_max(const char *root)
 	char *cpucg;
 	char quota_buf[32];
 
+	quota_usec = hz != -1 ? USEC_PER_SEC / hz : 1000;
+
 	snprintf(quota_buf, sizeof(quota_buf), "%ld", quota_usec);
 
 	cpucg = cg_name(root, "cpucg_test");
@@ -710,7 +736,8 @@ static int test_cpucg_max(const char *root)
 static int test_cpucg_max_nested(const char *root)
 {
 	int ret = KSFT_FAIL;
-	long quota_usec = 1000;
+	long quota_usec;
+	long hz = _get_config_hz();
 	long default_period_usec = 100000; /* cpu.max's default period */
 	long duration_seconds = 1;
 
@@ -719,6 +746,8 @@ static int test_cpucg_max_nested(const char *root)
 	char *parent, *child;
 	char quota_buf[32];
 
+	quota_usec = hz != -1 ? USEC_PER_SEC / hz : 1000;
+
 	snprintf(quota_buf, sizeof(quota_buf), "%ld", quota_usec);
 
 	parent = cg_name(root, "cpucg_parent");
maintainer
#3 lore raw
In reply to Joe Simmons-Talbott
Quoted 8 lines · Joe Simmons-Talbott
(quoted diff) +8 -0
+static long
+_get_config_hz(void)
+{
+	long hz = -1;
+	FILE *f;
+	char cmd[256] = "zcat /proc/config.gz 2>/dev/null | grep '^CONFIG_HZ='";
+
+	f = popen(cmd, "r");

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+	if (!f)
+		goto out;
+
+	fscanf(f, "CONFIG_HZ=%ld", &hz);
+
+out:
+	pclose(f);
+	return hz;
+}

I like that you voiced this dependency on CONFIG_HZ and also that _SC_CLK_TCK is useless in this regards.
(I see that BPF selftests have similar infra for this.)

Review test_cpucg_nested_weight_underprovisioned(const char *root) · line 675

Quoted 13 lines
(quoted diff) +3 -1
+
 /*
  * This test creates a cgroup with some maximum value within a period, and
  * verifies that a process in the cgroup is not overscheduled.
@@ -646,7 +669,8 @@ test_cpucg_nested_weight_underprovisioned(const char *root)
 static int test_cpucg_max(const char *root)
 {
 	int ret = KSFT_FAIL;
-	long quota_usec = 1000;
+	long hz = _get_config_hz();
+	long quota_usec;
 	long default_period_usec = 100000; /* cpu.max's default period */
 	long duration_seconds = 1;

I would not bend the tested value but it's expectation (so that approximately same quantity is tested acroos configs).

I reckon the problem might be tasks that overrun the quota due to long tick, fortunately, we can assume this is compensated over multiple periods, so _on average_ quota should be honored (more) precisely. But the test duration may be not well aligned with all the compensation periods, to that must be accounted for in the expectation.

When I write it all down, I get this:

tools/testing/selftests/cgroup/test_cpu.c +13 -4
@@ -651,7 +651,9 @@ static int test_cpucg_max(const char *root)
        long duration_seconds = 1;
 
        long duration_usec = duration_seconds * USEC_PER_SEC;
-       long usage_usec, n_periods, remainder_usec, expected_usage_usec;
+       long usage_usec, expected_usage_usec;
+       long n_periods, spread_periods, unaligned;
+       long tick_usec, low_usage, high_usage;
        char *cpucg;
        char quota_buf[32];
 
@@ -687,9 +689,16 @@ static int test_cpucg_max(const char *root)
         * the cpu hog is set to run as per wall-clock time
         */
        n_periods = duration_usec / default_period_usec;
-       remainder_usec = duration_usec - n_periods * default_period_usec;
-       expected_usage_usec
-               = n_periods * quota_usec + MIN(remainder_usec, quota_usec);
+       tick_usec = USEC_PER_SEC / hz;
+       /* Up to tick_usec (over)run is compensated over multiple periods */
+       spread_periods = MAX(1, tick_usec / quota_usec);
+       low_usage = n_periods / spread_periods;
+       high_usage = (n_periods + spread_periods - 1) / spread_periods;
+       unaligned = n_periods % spread_periods;
+
+       expected_usage_usec = quota_usec * (
+               unaligned * high_usage +
+               (spread_periods - unaligned) * low_usage);
 
        if (!values_close_report(usage_usec, expected_usage_usec, 10))
                goto cleanup;

(I neglected (and dropped) remainder_usec because it is zero with default values)

However, not all preemptions are tick-based, so there'd be noise and one has to tune the values_clone_report(,,err) anyway.

Then to reduce noise, the simpler solution is to let the test run longer

duration_usec = duration_seconds * USEC_PER_SEC * 1000 / hz;

(where 1000 is the CONFIG_HZ=1000 where the test runs sufficiently [1] well.)

Joe, how do to the two variants above (unalignment account and prolonged duration) affect test_cpu behavior on your setup?

(I'm personally wondering what is bigger quantity: systemic error due to HZ quantization or random (SMP) error.)

Thanks,
Michal

[1] Even there one runs into noise depending on nr_cpus, thus even that fixed err=10 is not ideal.

#4 lore raw
Quoted 94 lines · Michal Koutný
Quoted 8 lines · Joe Simmons-Talbott
(quoted diff) +8 -0
+static long
+_get_config_hz(void)
+{
+	long hz = -1;
+	FILE *f;
+	char cmd[256] = "zcat /proc/config.gz 2>/dev/null | grep '^CONFIG_HZ='";
+
+	f = popen(cmd, "r");

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+	if (!f)
+		goto out;
+
+	fscanf(f, "CONFIG_HZ=%ld", &hz);
+
+out:
+	pclose(f);
+	return hz;
+}

I like that you voiced this dependency on CONFIG_HZ and also that _SC_CLK_TCK is useless in this regards.
(I see that BPF selftests have similar infra for this.)

Review test_cpucg_nested_weight_underprovisioned(const char *root) · line 675

Quoted 13 lines
(quoted diff) +3 -1
+
 /*
  * This test creates a cgroup with some maximum value within a period, and
  * verifies that a process in the cgroup is not overscheduled.
@@ -646,7 +669,8 @@ test_cpucg_nested_weight_underprovisioned(const char *root)
 static int test_cpucg_max(const char *root)
 {
 	int ret = KSFT_FAIL;
-	long quota_usec = 1000;
+	long hz = _get_config_hz();
+	long quota_usec;
 	long default_period_usec = 100000; /* cpu.max's default period */
 	long duration_seconds = 1;

I would not bend the tested value but it's expectation (so that approximately same quantity is tested acroos configs).

I reckon the problem might be tasks that overrun the quota due to long tick, fortunately, we can assume this is compensated over multiple periods, so _on average_ quota should be honored (more) precisely. But the test duration may be not well aligned with all the compensation periods, to that must be accounted for in the expectation.

When I write it all down, I get this:

tools/testing/selftests/cgroup/test_cpu.c +13 -4
@@ -651,7 +651,9 @@ static int test_cpucg_max(const char *root)
        long duration_seconds = 1;
 
        long duration_usec = duration_seconds * USEC_PER_SEC;
-       long usage_usec, n_periods, remainder_usec, expected_usage_usec;
+       long usage_usec, expected_usage_usec;
+       long n_periods, spread_periods, unaligned;
+       long tick_usec, low_usage, high_usage;
        char *cpucg;
        char quota_buf[32];
 
@@ -687,9 +689,16 @@ static int test_cpucg_max(const char *root)
         * the cpu hog is set to run as per wall-clock time
         */
        n_periods = duration_usec / default_period_usec;
-       remainder_usec = duration_usec - n_periods * default_period_usec;
-       expected_usage_usec
-               = n_periods * quota_usec + MIN(remainder_usec, quota_usec);
+       tick_usec = USEC_PER_SEC / hz;
+       /* Up to tick_usec (over)run is compensated over multiple periods */
+       spread_periods = MAX(1, tick_usec / quota_usec);
+       low_usage = n_periods / spread_periods;
+       high_usage = (n_periods + spread_periods - 1) / spread_periods;
+       unaligned = n_periods % spread_periods;
+
+       expected_usage_usec = quota_usec * (
+               unaligned * high_usage +
+               (spread_periods - unaligned) * low_usage);
 
        if (!values_close_report(usage_usec, expected_usage_usec, 10))
                goto cleanup;

(I neglected (and dropped) remainder_usec because it is zero with default values)

However, not all preemptions are tick-based, so there'd be noise and one has to tune the values_clone_report(,,err) anyway.

Then to reduce noise, the simpler solution is to let the test run longer

Quoted 6 lines

duration_usec = duration_seconds * USEC_PER_SEC * 1000 / hz;

(where 1000 is the CONFIG_HZ=1000 where the test runs sufficiently [1] well.)

Joe, how do to the two variants above (unalignment account and prolonged duration) affect test_cpu behavior on your setup?

Hi Michal,

Thank you for your review.

I tried both approaches, unalignment account and prolonged duration, and both allowed me to run 10 iterations of the test_cpu tests without any failures. I will use the simpler prolonged duration approach in v3 if that is okay.

Thanks,
Joe

Quoted 9 lines

(I'm personally wondering what is bigger quantity: systemic error due to HZ quantization or random (SMP) error.)

Thanks,
Michal

[1] Even there one runs into noise depending on nr_cpus, thus even that fixed err=10 is not ideal.