Skip to contentSkip to conversation

bpf: Inherit system settings for CPU security mitigations

PATCH Applied Yafang Shao 10 participants 20 messages Oct 5, 2023 – Oct 26, 2023

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bpf: Inherit system settings for CPU security mitigations

Currently, there exists a system-wide setting related to CPU security mitigations, denoted as 'mitigations='. When set to 'mitigations=off', it deactivates all optional CPU mitigations. Therefore, if we implement a system-wide 'mitigations=off' setting, it should inherently bypass Spectre v1 and Spectre v4 in the BPF subsystem.

Please note that there is also a 'nospectre_v1' setting on x86 and ppc architectures, though it is not currently exported. For the time being, let's disregard it.

This idea emerged during our discussion about potential Spectre v1 attacks with Luis[1].

[1]. https://lore.kernel.org/bpf/[email protected]/

---
 include/linux/bpf.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/bpf.h +2 -2
index a82efd34b741..61bde4520f5c 100644
@@ -2164,12 +2164,12 @@ static inline bool bpf_allow_uninit_stack(void)
 
 static inline bool bpf_bypass_spec_v1(void)
 {
-	return perfmon_capable();
+	return perfmon_capable() || cpu_mitigations_off();
 }
 
 static inline bool bpf_bypass_spec_v4(void)
 {
-	return perfmon_capable();
+	return perfmon_capable() || cpu_mitigations_off();
 }
 
 int bpf_map_new_fd(struct bpf_map *map, int flags);
Signature · 1 line
2.30.1 (Apple Git-130)
Acked-by on v1 1/1
#1 lore raw
Quoted 6 lines · Yafang Shao

Currently, there exists a system-wide setting related to CPU security mitigations, denoted as 'mitigations='. When set to 'mitigations=off', it deactivates all optional CPU mitigations. Therefore, if we implement a system-wide 'mitigations=off' setting, it should inherently bypass Spectre v1 and Spectre v4 in the BPF subsystem.

Quoted 8 lines

Please note that there is also a 'nospectre_v1' setting on x86 and ppc architectures, though it is not currently exported. For the time being, let's disregard it.

This idea emerged during our discussion about potential Spectre v1 attacks with Luis[1].

[1]. https://lore.kernel.org/bpf/[email protected]/

Based on the discussion from [1]:

Quoted 28 lines
---
 include/linux/bpf.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/bpf.h +2 -2
index a82efd34b741..61bde4520f5c 100644
@@ -2164,12 +2164,12 @@ static inline bool bpf_allow_uninit_stack(void)
 
 static inline bool bpf_bypass_spec_v1(void)
 {
-	return perfmon_capable();
+	return perfmon_capable() || cpu_mitigations_off();
 }
 
 static inline bool bpf_bypass_spec_v4(void)
 {
-	return perfmon_capable();
+	return perfmon_capable() || cpu_mitigations_off();
 }
 
 int bpf_map_new_fd(struct bpf_map *map, int flags);
Signature · 1 line
2.30.1 (Apple Git-130)
reviewer Acked-by on v1 1/1
#2 lore raw
In reply to Yafang Shao
Quoted 11 lines · Yafang Shao

Currently, there exists a system-wide setting related to CPU security mitigations, denoted as 'mitigations='. When set to 'mitigations=off', it deactivates all optional CPU mitigations. Therefore, if we implement a system-wide 'mitigations=off' setting, it should inherently bypass Spectre v1 and Spectre v4 in the BPF subsystem.

Please note that there is also a 'nospectre_v1' setting on x86 and ppc architectures, though it is not currently exported. For the time being, let's disregard it.

Quoted 7 lines

This idea emerged during our discussion about potential Spectre v1 attacks with Luis[1].

[1]. https://lore.kernel.org/bpf/[email protected]/

Acked-by
#3 lore raw
Quoted 16 lines · Song Liu
Quoted 14 lines · Yafang Shao

Currently, there exists a system-wide setting related to CPU security mitigations, denoted as 'mitigations='. When set to 'mitigations=off', it deactivates all optional CPU mitigations. Therefore, if we implement a system-wide 'mitigations=off' setting, it should inherently bypass Spectre v1 and Spectre v4 in the BPF subsystem.

Please note that there is also a 'nospectre_v1' setting on x86 and ppc architectures, though it is not currently exported. For the time being, let's disregard it.

This idea emerged during our discussion about potential Spectre v1 attacks with Luis[1].

Quoted 7 lines
Quoted 4 lines

[1]. https://lore.kernel.org/bpf/[email protected]/

maintainer
#4 lore raw
Quoted 9 lines · KP Singh
Quoted 8 lines · Song Liu
Quoted 7 lines · Yafang Shao

Currently, there exists a system-wide setting related to CPU security mitigations, denoted as 'mitigations='. When set to 'mitigations=off', it deactivates all optional CPU mitigations. Therefore, if we implement a system-wide 'mitigations=off' setting, it should inherently bypass Spectre v1 and Spectre v4 in the BPF subsystem.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Please note that there is also a 'nospectre_v1' setting on x86 and ppc architectures, though it is not currently exported. For the time being, let's disregard it.

From reading, the cpu_mitigations_off() is a more generic toggle to turn these generally off, so going via cpu_mitigations_off() is fine in our case and does not leave some corner cases behind. I presume you mean above that in future the BPF side could respect some more fine-tuned settings, though it probably might need some more coordination wrt archs to abstract sth generic out of it.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

This idea emerged during our discussion about potential Spectre v1 attacks with Luis[1].

[1]. https://lore.kernel.org/bpf/[email protected]/

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Thanks,
Daniel

#5 lore raw
In reply to Yafang Shao
This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:

Hello:

This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:

Quoted 11 lines · On Thu, 5 Oct 2023 08:41:23 +0000 you wrote:

Currently, there exists a system-wide setting related to CPU security mitigations, denoted as 'mitigations='. When set to 'mitigations=off', it deactivates all optional CPU mitigations. Therefore, if we implement a system-wide 'mitigations=off' setting, it should inherently bypass Spectre v1 and Spectre v4 in the BPF subsystem.

Please note that there is also a 'nospectre_v1' setting on x86 and ppc architectures, though it is not currently exported. For the time being, let's disregard it.

[...]

Here is the summary with links:
- [bpf-next] bpf: Inherit system settings for CPU security mitigations

    https://git.kernel.org/bpf/bpf-next/c/bc5bc309db45

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
#6 lore raw
In reply to Yafang Shao
Quoted 27 lines · Yafang Shao

Currently, there exists a system-wide setting related to CPU security mitigations, denoted as 'mitigations='. When set to 'mitigations=off', it deactivates all optional CPU mitigations. Therefore, if we implement a system-wide 'mitigations=off' setting, it should inherently bypass Spectre v1 and Spectre v4 in the BPF subsystem.

Please note that there is also a 'nospectre_v1' setting on x86 and ppc architectures, though it is not currently exported. For the time being, let's disregard it.

This idea emerged during our discussion about potential Spectre v1 attacks with Luis[1].

[1]. https://lore.kernel.org/bpf/[email protected]/

---
 include/linux/bpf.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/bpf.h +0 -0
index a82efd34b741..61bde4520f5c 100644

include/linux/bpf.h static inline bool bpf_allow_uninit_stack(void) · line 2167

Quoted 5 lines
include/linux/bpf.h +1 -1
@@ -2164,12 +2164,12 @@ static inline bool bpf_allow_uninit_stack(void)
 
 static inline bool bpf_bypass_spec_v1(void)
 {
-       return perfmon_capable();
+       return perfmon_capable() || cpu_mitigations_off();

Should we check cpu_mitigations_off() first before perfmon_capable() to avoid unnecessary capability checks, which generate audit messages?

Quoted 12 lines
include/linux/bpf.h +1 -2
 }
 
 static inline bool bpf_bypass_spec_v4(void)
 {
-       return perfmon_capable();
+       return perfmon_capable() || cpu_mitigations_off();
 }
 
 int bpf_map_new_fd(struct bpf_map *map, int flags);
--

2.30.1 (Apple Git-130)

#7 lore raw
Quoted 30 lines · Andrii Nakryiko
Quoted 28 lines · Yafang Shao

Currently, there exists a system-wide setting related to CPU security mitigations, denoted as 'mitigations='. When set to 'mitigations=off', it deactivates all optional CPU mitigations. Therefore, if we implement a system-wide 'mitigations=off' setting, it should inherently bypass Spectre v1 and Spectre v4 in the BPF subsystem.

Please note that there is also a 'nospectre_v1' setting on x86 and ppc architectures, though it is not currently exported. For the time being, let's disregard it.

This idea emerged during our discussion about potential Spectre v1 attacks with Luis[1].

[1]. https://lore.kernel.org/bpf/[email protected]/

---
 include/linux/bpf.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/bpf.h +0 -0
index a82efd34b741..61bde4520f5c 100644
@@ -2164,12 +2164,12 @@ static inline bool bpf_allow_uninit_stack(void)
 
Quoted 7 lines

include/linux/bpf.h static inline bool bpf_allow_uninit_stack(void) · line 2167

Quoted 4 lines
include/linux/bpf.h +1 -1
 static inline bool bpf_bypass_spec_v1(void)
 {
-       return perfmon_capable();
+       return perfmon_capable() || cpu_mitigations_off();

Should we check cpu_mitigations_off() first before perfmon_capable() to avoid unnecessary capability checks, which generate audit messages?

makes sense.
Should I send an additional patch, or you modify the original patch?

Signature · 2 lines
Regards
Yafang
#8 lore raw
Quoted 41 lines · Yafang Shao
Quoted 30 lines · Andrii Nakryiko
Quoted 28 lines · Yafang Shao

Currently, there exists a system-wide setting related to CPU security mitigations, denoted as 'mitigations='. When set to 'mitigations=off', it deactivates all optional CPU mitigations. Therefore, if we implement a system-wide 'mitigations=off' setting, it should inherently bypass Spectre v1 and Spectre v4 in the BPF subsystem.

Please note that there is also a 'nospectre_v1' setting on x86 and ppc architectures, though it is not currently exported. For the time being, let's disregard it.

This idea emerged during our discussion about potential Spectre v1 attacks with Luis[1].

[1]. https://lore.kernel.org/bpf/[email protected]/

---
 include/linux/bpf.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/bpf.h +0 -0
index a82efd34b741..61bde4520f5c 100644
@@ -2164,12 +2164,12 @@ static inline bool bpf_allow_uninit_stack(void)
 
Quoted 7 lines

include/linux/bpf.h static inline bool bpf_allow_uninit_stack(void) · line 2167

Quoted 4 lines
include/linux/bpf.h +1 -1
 static inline bool bpf_bypass_spec_v1(void)
 {
-       return perfmon_capable();
+       return perfmon_capable() || cpu_mitigations_off();

Should we check cpu_mitigations_off() first before perfmon_capable() to avoid unnecessary capability checks, which generate audit messages?

Quoted 2 lines

makes sense.
Should I send an additional patch, or you modify the original patch?

please send a patch

Quoted 4 lines

--
Regards
Yafang

#9 lore raw
In reply to Yafang Shao
Quoted 29 lines · Yafang Shao

Currently, there exists a system-wide setting related to CPU security mitigations, denoted as 'mitigations='. When set to 'mitigations=off', it deactivates all optional CPU mitigations. Therefore, if we implement a system-wide 'mitigations=off' setting, it should inherently bypass Spectre v1 and Spectre v4 in the BPF subsystem.

Please note that there is also a 'nospectre_v1' setting on x86 and ppc architectures, though it is not currently exported. For the time being, let's disregard it.

This idea emerged during our discussion about potential Spectre v1 attacks with Luis[1].

[1]. https://lore.kernel.org/bpf/[email protected]/

---
 include/linux/bpf.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/bpf.h +0 -0
index a82efd34b741..61bde4520f5c 100644
@@ -2164,12 +2164,12 @@ static inline bool bpf_allow_uninit_stack(void)
 
 static inline bool bpf_bypass_spec_v1(void)

include/linux/bpf.h static inline bool bpf_allow_uninit_stack(void) · line 2173

Quoted 10 lines
include/linux/bpf.h +2 -2
 {
-       return perfmon_capable();
+       return perfmon_capable() || cpu_mitigations_off();
 }
 
 static inline bool bpf_bypass_spec_v4(void)
 {
-       return perfmon_capable();
+       return perfmon_capable() || cpu_mitigations_off();
 }

Yafang,

this patch breaks several
test_progs -t verifier

tests when system is booted with mitigations=off command line.

Please follow up with a patch to fix this.

As you noticed cpu_mitigations_off() is not quite right here. The system might have booted without that command line, but spec_v1 and spec_v4 mitigations are turned off. Unfortunately there is no good way to check that atm. Have you seen this patch set ?
https://lore.kernel.org/all/[email protected]/
Please take a look at it and comment if you think it will help.

In the meantime please fix test_progs -t verifier

Thanks

#10 lore raw
Quoted 44 lines · Alexei Starovoitov
Quoted 29 lines · Yafang Shao

Currently, there exists a system-wide setting related to CPU security mitigations, denoted as 'mitigations='. When set to 'mitigations=off', it deactivates all optional CPU mitigations. Therefore, if we implement a system-wide 'mitigations=off' setting, it should inherently bypass Spectre v1 and Spectre v4 in the BPF subsystem.

Please note that there is also a 'nospectre_v1' setting on x86 and ppc architectures, though it is not currently exported. For the time being, let's disregard it.

This idea emerged during our discussion about potential Spectre v1 attacks with Luis[1].

[1]. https://lore.kernel.org/bpf/[email protected]/

---
 include/linux/bpf.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/bpf.h +0 -0
index a82efd34b741..61bde4520f5c 100644
@@ -2164,12 +2164,12 @@ static inline bool bpf_allow_uninit_stack(void)
 
 static inline bool bpf_bypass_spec_v1(void)

include/linux/bpf.h static inline bool bpf_allow_uninit_stack(void) · line 2173

Quoted 10 lines
include/linux/bpf.h +2 -2
 {
-       return perfmon_capable();
+       return perfmon_capable() || cpu_mitigations_off();
 }
 
 static inline bool bpf_bypass_spec_v4(void)
 {
-       return perfmon_capable();
+       return perfmon_capable() || cpu_mitigations_off();
 }

Yafang,

Quoted 2 lines

this patch breaks several
test_progs -t verifier

Sorry, I miss that.

Quoted 4 lines

tests when system is booted with mitigations=off command line.

Please follow up with a patch to fix this.

will do it.

Quoted 8 lines

As you noticed cpu_mitigations_off() is not quite right here. The system might have booted without that command line, but spec_v1 and spec_v4 mitigations are turned off. Unfortunately there is no good way to check that atm. Have you seen this patch set ?
https://lore.kernel.org/all/[email protected]/
Please take a look at it and comment if you think it will help.

Thanks for your information. will take a look.

Quoted 2 lines

In the meantime please fix test_progs -t verifier

sure

Signature · 2 lines
Regards
Yafang
#11 lore raw

PATCH 1/1 selftests/bpf: Fix selftests broken by mitigations=off

In reply to Alexei Starovoitov

When we configure the kernel command line with 'mitigations=off' and set the sysctl knob 'kernel.unprivileged_bpf_disabled' to 0, the commit bc5bc309db45 ("bpf: Inherit system settings for CPU security mitigations") causes issues in the execution of 'test_progs -t verifier.' This is because 'mitigations=off' bypasses Spectre v1 and Spectre v4 protections.

Currently, when a program requests to run in unprivileged mode (kernel.unprivileged_bpf_disabled = 0), the BPF verifier may prevent it from running due to the following conditions not being enabled:

- bypass_spec_v1
- bypass_spec_v4
- allow_ptr_leaks
- allow_uninit_stack

While 'mitigations=off' enables the first two conditions, it does not enable the latter two. As a result, some test cases in 'test_progs -t verifier' that were expected to fail to run may run successfully, while others still fail but with different error messages. This makes it challenging to address them comprehensively.

Moreover, in the future, we may introduce more fine-grained control over CPU mitigations, such as enabling only bypass_spec_v1 or bypass_spec_v4.

Given the complexity of the situation, rather than fixing each broken test case individually, it's preferable to skip them when 'mitigations=off' is in effect and introduce specific test cases for the new 'mitigations=off' scenario. For instance, we can introduce new BTF declaration tags like '__failure__nospec', '__failure_nospecv1' and '__failure_nospecv4'.

In this patch, the approach is to simply skip the broken test cases when 'mitigations=off' is enabled. The result as follows after this commit,

- without 'mitigations=off'
- kernel.unprivileged_bpf_disabled = 2

    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
  - kernel.unprivileged_bpf_disabled = 0
    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
- with 'mitigations=off'
  - kernel.unprivileged_bpf_disabled = 2
    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
  - kernel.unprivileged_bpf_disabled = 0
    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
---
 tools/testing/selftests/bpf/unpriv_helpers.c | 29 +++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/unpriv_helpers.c +28 -1
index 2a6efbd0401e..2e756c89b37c 100644
@@ -4,9 +4,36 @@
 #include <stdlib.h>
 #include <error.h>
 #include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <fcntl.h>
 
 #include "unpriv_helpers.h"
 
+static bool get_mitigations_off(void)
+{
+	char cmdline[4096], *c;
+	int fd;
+
+	fd = open("/proc/cmdline", O_RDONLY);
+	if (fd < 0) {
+		perror("open /proc/cmdline");
+		return false;
+	}
+
+	if (read(fd, cmdline, sizeof(cmdline) - 1) < 0) {
+		perror("read /proc/cmdline");
+		return false;
+	}
+
+	cmdline[sizeof(cmdline) - 1] = '\0';
+	for (c = strtok(cmdline, " \n"); c; c = strtok(NULL, " \n")) {
+		if (!strncmp(c, "mitigtions=off", strlen(c)))
+			return true;
+	}
+	return false;
+}
+
 bool get_unpriv_disabled(void)
 {
 	bool disabled;
@@ -22,5 +49,5 @@ bool get_unpriv_disabled(void)
 		disabled = true;
 	}
 
-	return disabled;
+	return disabled ? true : !get_mitigations_off();
 }
Signature · 1 line
2.39.3
#12 lore raw

PATCH v2 1/1 selftests/bpf: Fix selftests broken by mitigations=off

When we configure the kernel command line with 'mitigations=off' and set the sysctl knob 'kernel.unprivileged_bpf_disabled' to 0, the commit bc5bc309db45 ("bpf: Inherit system settings for CPU security mitigations") causes issues in the execution of 'test_progs -t verifier.' This is because 'mitigations=off' bypasses Spectre v1 and Spectre v4 protections.

Currently, when a program requests to run in unprivileged mode (kernel.unprivileged_bpf_disabled = 0), the BPF verifier may prevent it from running due to the following conditions not being enabled:

- bypass_spec_v1
- bypass_spec_v4
- allow_ptr_leaks
- allow_uninit_stack

While 'mitigations=off' enables the first two conditions, it does not enable the latter two. As a result, some test cases in 'test_progs -t verifier' that were expected to fail to run may run successfully, while others still fail but with different error messages. This makes it challenging to address them comprehensively.

Moreover, in the future, we may introduce more fine-grained control over CPU mitigations, such as enabling only bypass_spec_v1 or bypass_spec_v4.

Given the complexity of the situation, rather than fixing each broken test case individually, it's preferable to skip them when 'mitigations=off' is in effect and introduce specific test cases for the new 'mitigations=off' scenario. For instance, we can introduce new BTF declaration tags like '__failure__nospec', '__failure_nospecv1' and '__failure_nospecv4'.

In this patch, the approach is to simply skip the broken test cases when 'mitigations=off' is enabled. The result as follows after this commit,

- without 'mitigations=off'
- kernel.unprivileged_bpf_disabled = 2

    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
  - kernel.unprivileged_bpf_disabled = 0
    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
- with 'mitigations=off'
  - kernel.unprivileged_bpf_disabled = 2
    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
  - kernel.unprivileged_bpf_disabled = 0
    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
---
 tools/testing/selftests/bpf/unpriv_helpers.c | 34 +++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

---
v1 -> v2: Fix leaked fd

tools/testing/selftests/bpf/unpriv_helpers.c +33 -1
index 2a6efbd0401e..ca4760795f5d 100644
@@ -4,9 +4,41 @@
 #include <stdlib.h>
 #include <error.h>
 #include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <fcntl.h>
 
 #include "unpriv_helpers.h"
 
+static bool get_mitigations_off(void)
+{
+	char cmdline[4096], *c;
+	int fd, ret = false;
+
+	fd = open("/proc/cmdline", O_RDONLY);
+	if (fd < 0) {
+		perror("open /proc/cmdline");
+		return false;
+	}
+
+	if (read(fd, cmdline, sizeof(cmdline) - 1) < 0) {
+		perror("read /proc/cmdline");
+		goto out;
+	}
+
+	cmdline[sizeof(cmdline) - 1] = '\0';
+	for (c = strtok(cmdline, " \n"); c; c = strtok(NULL, " \n")) {
+		if (!strncmp(c, "mitigtions=off", strlen(c))) {
+			ret = true;
+			break;
+		}
+	}
+
+out:
+	close(fd);
+	return ret;
+}
+
 bool get_unpriv_disabled(void)
 {
 	bool disabled;
@@ -22,5 +54,5 @@ bool get_unpriv_disabled(void)
 		disabled = true;
 	}
 
-	return disabled;
+	return disabled ? true : !get_mitigations_off();
 }
Signature · 1 line
2.39.3
#13 lore raw

Re: [PATCH v2 bpf-next] selftests/bpf: Fix selftests broken by mitigations=off

Quoted 110 lines · Yafang Shao

When we configure the kernel command line with 'mitigations=off' and set the sysctl knob 'kernel.unprivileged_bpf_disabled' to 0, the commit bc5bc309db45 ("bpf: Inherit system settings for CPU security mitigations") causes issues in the execution of 'test_progs -t verifier.' This is because 'mitigations=off' bypasses Spectre v1 and Spectre v4 protections.

Currently, when a program requests to run in unprivileged mode (kernel.unprivileged_bpf_disabled = 0), the BPF verifier may prevent it from running due to the following conditions not being enabled:

- bypass_spec_v1
- bypass_spec_v4
- allow_ptr_leaks
- allow_uninit_stack

While 'mitigations=off' enables the first two conditions, it does not enable the latter two. As a result, some test cases in 'test_progs -t verifier' that were expected to fail to run may run successfully, while others still fail but with different error messages. This makes it challenging to address them comprehensively.

Moreover, in the future, we may introduce more fine-grained control over CPU mitigations, such as enabling only bypass_spec_v1 or bypass_spec_v4.

Given the complexity of the situation, rather than fixing each broken test case individually, it's preferable to skip them when 'mitigations=off' is in effect and introduce specific test cases for the new 'mitigations=off' scenario. For instance, we can introduce new BTF declaration tags like '__failure__nospec', '__failure_nospecv1' and '__failure_nospecv4'.

In this patch, the approach is to simply skip the broken test cases when 'mitigations=off' is enabled. The result as follows after this commit,

- without 'mitigations=off'
- kernel.unprivileged_bpf_disabled = 2

    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
  - kernel.unprivileged_bpf_disabled = 0
    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
- with 'mitigations=off'
  - kernel.unprivileged_bpf_disabled = 2
    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
  - kernel.unprivileged_bpf_disabled = 0
    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
---
 tools/testing/selftests/bpf/unpriv_helpers.c | 34 +++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

---
v1 -> v2: Fix leaked fd

tools/testing/selftests/bpf/unpriv_helpers.c +33 -2
index 2a6efbd0401e..ca4760795f5d 100644
@@ -4,9 +4,41 @@
 #include <stdlib.h>
 #include <error.h>
 #include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <fcntl.h>
 
 #include "unpriv_helpers.h"
 
+static bool get_mitigations_off(void)
+{
+       char cmdline[4096], *c;
+       int fd, ret = false;
+
+       fd = open("/proc/cmdline", O_RDONLY);
+       if (fd < 0) {
+               perror("open /proc/cmdline");
+               return false;
+       }
+
+       if (read(fd, cmdline, sizeof(cmdline) - 1) < 0) {
+               perror("read /proc/cmdline");
+               goto out;
+       }
+
+       cmdline[sizeof(cmdline) - 1] = '\0';
+       for (c = strtok(cmdline, " \n"); c; c = strtok(NULL, " \n")) {
+               if (!strncmp(c, "mitigtions=off", strlen(c))) {
+                       ret = true;
+                       break;
+               }
+       }
+
+out:
+       close(fd);
+       return ret;
+}
+
 bool get_unpriv_disabled(void)
 {
        bool disabled;
@@ -22,5 +54,5 @@ bool get_unpriv_disabled(void)
                disabled = true;
        }
 
-       return disabled;
+       return disabled ? true : !get_mitigations_off();
 }
--
Quoted 2 lines

2.39.3

Pls. just igore this wrong patch. Sorry about the noise. I must be in a sleep state currently. I will send a new one after I get awake ...

Signature · 2 lines
Regards
Yafang
#14 lore raw

Re: [PATCH v2 bpf-next] selftests/bpf: Fix selftests broken by mitigations=off

In reply to Yafang Shao
kernel test robot noticed the following build warnings:

Hi Yafang,

kernel test robot noticed the following build warnings:

[auto build test WARNING on bpf-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Yafang-Shao/selftests-bpf-Fix-selftests-broken-by-mitigations-off/20231022-175022
base:   https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link:    https://lore.kernel.org/r/20231022094906.3003-1-laoar.shao%40gmail.com
patch subject: [PATCH v2 bpf-next] selftests/bpf: Fix selftests broken by mitigations=off
reproduce: (https://download.01.org/0day-ci/archive/20231022/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

# many are suggestions rather than must-fix

WARNING:QUOTED_WHITESPACE_BEFORE_NEWLINE: unnecessary whitespace before a quoted newline #89: FILE: tools/testing/selftests/bpf/unpriv_helpers.c:30:
+ for (c = strtok(cmdline, " \n"); c; c = strtok(NULL, " \n")) {

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#15 lore raw

PATCH v3 1/1 selftests/bpf: Fix selftests broken by mitigations=off

In reply to Alexei Starovoitov

When we configure the kernel command line with 'mitigations=off' and set the sysctl knob 'kernel.unprivileged_bpf_disabled' to 0, the commit bc5bc309db45 ("bpf: Inherit system settings for CPU security mitigations") causes issues in the execution of test_progs -t verifier. This is because 'mitigations=off' bypasses Spectre v1 and Spectre v4 protections.

Currently, when a program requests to run in unprivileged mode (kernel.unprivileged_bpf_disabled = 0), the BPF verifier may prevent it from running due to the following conditions not being enabled:

- bypass_spec_v1
- bypass_spec_v4
- allow_ptr_leaks
- allow_uninit_stack

While 'mitigations=off' enables the first two conditions, it does not enable the latter two. As a result, some test cases in 'test_progs -t verifier' that were expected to fail to run may run successfully, while others still fail but with different error messages. This makes it challenging to address them comprehensively.

Moreover, in the future, we may introduce more fine-grained control over CPU mitigations, such as enabling only bypass_spec_v1 or bypass_spec_v4.

Given the complexity of the situation, rather than fixing each broken test case individually, it's preferable to skip them when 'mitigations=off' is in effect and introduce specific test cases for the new 'mitigations=off' scenario. For instance, we can introduce new BTF declaration tags like '__failure__nospec', '__failure_nospecv1' and '__failure_nospecv4'.

In this patch, the approach is to simply skip the broken test cases when 'mitigations=off' is enabled. The result of test_progs -t verifier as follows after this commit,

Before this commit
==================
- without 'mitigations=off'
- kernel.unprivileged_bpf_disabled = 2

    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
  - kernel.unprivileged_bpf_disabled = 0
    Summary: 74/1336 PASSED, 0 SKIPPED, 0 FAILED   <<<<
- with 'mitigations=off'
  - kernel.unprivileged_bpf_disabled = 2
    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
  - kernel.unprivileged_bpf_disabled = 0
    Summary: 63/1276 PASSED, 0 SKIPPED, 11 FAILED   <<<< 11 FAILED

After this commit
=================
- without 'mitigations=off'
- kernel.unprivileged_bpf_disabled = 2

    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
  - kernel.unprivileged_bpf_disabled = 0
    Summary: 74/1336 PASSED, 0 SKIPPED, 0 FAILED    <<<<
- with this patch, with 'mitigations=off'
  - kernel.unprivileged_bpf_disabled = 2
    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
  - kernel.unprivileged_bpf_disabled = 0
    Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED   <<<< SKIPPED
---
 tools/testing/selftests/bpf/unpriv_helpers.c | 35 +++++++++++++++++++-
 1 file changed, 34 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/unpriv_helpers.c +34 -1
index 2a6efbd0401e..7101e72ef4a3 100644
@@ -4,9 +4,42 @@
 #include <stdlib.h>
 #include <error.h>
 #include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <fcntl.h>
 
 #include "unpriv_helpers.h"
 
+static bool get_mitigations_off(void)
+{
+	char cmdline[4096], *c;
+	int fd, ret = false;
+
+	fd = open("/proc/cmdline", O_RDONLY);
+	if (fd < 0) {
+		perror("open /proc/cmdline");
+		return false;
+	}
+
+	if (read(fd, cmdline, sizeof(cmdline) - 1) < 0) {
+		perror("read /proc/cmdline");
+		goto out;
+	}
+
+	cmdline[sizeof(cmdline) - 1] = '\0';
+	for (c = strtok(cmdline, " \n"); c; c = strtok(NULL, " \n")) {
+		if (strncmp(c, "mitigations=off", strlen(c)))
+			continue;
+
+		ret = true;
+		break;
+	}
+
+out:
+	close(fd);
+	return ret;
+}
+
 bool get_unpriv_disabled(void)
 {
 	bool disabled;
@@ -22,5 +55,5 @@ bool get_unpriv_disabled(void)
 		disabled = true;
 	}
 
-	return disabled;
+	return disabled ? true : get_mitigations_off();
 }
Signature · 1 line
2.39.3
reviewer Acked-by on v3 1/1
#16 lore raw

Re: [PATCH v3 bpf-next] selftests/bpf: Fix selftests broken by mitigations=off

Quoted 60 lines · Yafang Shao

When we configure the kernel command line with 'mitigations=off' and set the sysctl knob 'kernel.unprivileged_bpf_disabled' to 0, the commit bc5bc309db45 ("bpf: Inherit system settings for CPU security mitigations") causes issues in the execution of test_progs -t verifier. This is because 'mitigations=off' bypasses Spectre v1 and Spectre v4 protections.

Currently, when a program requests to run in unprivileged mode (kernel.unprivileged_bpf_disabled = 0), the BPF verifier may prevent it from running due to the following conditions not being enabled:

- bypass_spec_v1
- bypass_spec_v4
- allow_ptr_leaks
- allow_uninit_stack

While 'mitigations=off' enables the first two conditions, it does not enable the latter two. As a result, some test cases in 'test_progs -t verifier' that were expected to fail to run may run successfully, while others still fail but with different error messages. This makes it challenging to address them comprehensively.

Moreover, in the future, we may introduce more fine-grained control over CPU mitigations, such as enabling only bypass_spec_v1 or bypass_spec_v4.

Given the complexity of the situation, rather than fixing each broken test case individually, it's preferable to skip them when 'mitigations=off' is in effect and introduce specific test cases for the new 'mitigations=off' scenario. For instance, we can introduce new BTF declaration tags like '__failure__nospec', '__failure_nospecv1' and '__failure_nospecv4'.

In this patch, the approach is to simply skip the broken test cases when 'mitigations=off' is enabled. The result of test_progs -t verifier as follows after this commit,

Before this commit
==================
- without 'mitigations=off'
- kernel.unprivileged_bpf_disabled = 2

     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
   - kernel.unprivileged_bpf_disabled = 0
     Summary: 74/1336 PASSED, 0 SKIPPED, 0 FAILED   <<<<
- with 'mitigations=off'
   - kernel.unprivileged_bpf_disabled = 2
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
   - kernel.unprivileged_bpf_disabled = 0
     Summary: 63/1276 PASSED, 0 SKIPPED, 11 FAILED   <<<< 11 FAILED

After this commit
=================
- without 'mitigations=off'
- kernel.unprivileged_bpf_disabled = 2

     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
   - kernel.unprivileged_bpf_disabled = 0
     Summary: 74/1336 PASSED, 0 SKIPPED, 0 FAILED    <<<<
- with this patch, with 'mitigations=off'
   - kernel.unprivileged_bpf_disabled = 2
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
   - kernel.unprivileged_bpf_disabled = 0
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED   <<<< SKIPPED
Quoted 4 lines

Ack with a nit below.

Quoted 23 lines
---
  tools/testing/selftests/bpf/unpriv_helpers.c | 35 +++++++++++++++++++-
  1 file changed, 34 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/unpriv_helpers.c +3 -0
index 2a6efbd0401e..7101e72ef4a3 100644
@@ -4,9 +4,42 @@
  #include <stdlib.h>
  #include <error.h>
  #include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <fcntl.h>
 
  #include "unpriv_helpers.h"
 
[...]
  bool get_unpriv_disabled(void)
  {
  	bool disabled;

tools/testing/selftests/bpf/unpriv_helpers.c bool get_unpriv_disabled(void) · line 58

Quoted 5 lines
tools/testing/selftests/bpf/unpriv_helpers.c +1 -1
@@ -22,5 +55,5 @@ bool get_unpriv_disabled(void)
  		disabled = true;
  	}
 
-	return disabled;
+	return disabled ? true : get_mitigations_off();

Above code is correct. But you could slightly simplify it with

	return disabled ? : get_mitigations_off();

I guess maintainer can decide whether simplification is needed or not.

Quoted 1 line
tools/testing/selftests/bpf/unpriv_helpers.c +0 -0
  }
maintainer
#17 lore raw

Re: [PATCH v3 bpf-next] selftests/bpf: Fix selftests broken by mitigations=off

Quoted 69 lines · Yonghong Song
Quoted 60 lines · Yafang Shao

When we configure the kernel command line with 'mitigations=off' and set the sysctl knob 'kernel.unprivileged_bpf_disabled' to 0, the commit bc5bc309db45 ("bpf: Inherit system settings for CPU security mitigations") causes issues in the execution of test_progs -t verifier. This is because 'mitigations=off' bypasses Spectre v1 and Spectre v4 protections.

Currently, when a program requests to run in unprivileged mode (kernel.unprivileged_bpf_disabled = 0), the BPF verifier may prevent it from running due to the following conditions not being enabled:

   - bypass_spec_v1
   - bypass_spec_v4
   - allow_ptr_leaks
   - allow_uninit_stack

While 'mitigations=off' enables the first two conditions, it does not enable the latter two. As a result, some test cases in 'test_progs -t verifier' that were expected to fail to run may run successfully, while others still fail but with different error messages. This makes it challenging to address them comprehensively.

Moreover, in the future, we may introduce more fine-grained control over CPU mitigations, such as enabling only bypass_spec_v1 or bypass_spec_v4.

Given the complexity of the situation, rather than fixing each broken test case individually, it's preferable to skip them when 'mitigations=off' is in effect and introduce specific test cases for the new 'mitigations=off' scenario. For instance, we can introduce new BTF declaration tags like '__failure__nospec', '__failure_nospecv1' and '__failure_nospecv4'.

In this patch, the approach is to simply skip the broken test cases when 'mitigations=off' is enabled. The result of test_progs -t verifier as follows after this commit,

Before this commit
==================
- without 'mitigations=off'
   - kernel.unprivileged_bpf_disabled = 2
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
   - kernel.unprivileged_bpf_disabled = 0
     Summary: 74/1336 PASSED, 0 SKIPPED, 0 FAILED   <<<<
- with 'mitigations=off'
   - kernel.unprivileged_bpf_disabled = 2
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
   - kernel.unprivileged_bpf_disabled = 0
     Summary: 63/1276 PASSED, 0 SKIPPED, 11 FAILED   <<<< 11 FAILED

After this commit
=================
- without 'mitigations=off'
   - kernel.unprivileged_bpf_disabled = 2
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
   - kernel.unprivileged_bpf_disabled = 0
     Summary: 74/1336 PASSED, 0 SKIPPED, 0 FAILED    <<<<
- with this patch, with 'mitigations=off'
   - kernel.unprivileged_bpf_disabled = 2
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
   - kernel.unprivileged_bpf_disabled = 0
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED   <<<< SKIPPED

Quoted 4 lines

Ack with a nit below.

[...]

Quoted 9 lines
Quoted 3 lines

      }
-    return disabled;
+    return disabled ? true : get_mitigations_off();

Above code is correct. But you could slightly simplify it with
    return disabled ? : get_mitigations_off();

I guess maintainer can decide whether simplification is needed or not.

Turns out if you omit, then compiler will complain with a warning :)

[...]
GEN vmlinux.h
unpriv_helpers.c: In function ‘get_unpriv_disabled’:
unpriv_helpers.c:56:27: error: the omitted middle operand in ‘?:’ will always be ‘true’, suggest explicit middle operand [-Werror=parentheses]
56 | return disabled ? : get_mitigations_off();
| ^
cc1: all warnings being treated as errors
make: *** [Makefile:615: /root/linux/tools/testing/selftests/bpf/unpriv_helpers.o] Error 1

So it's okay as is, applied, thanks!

reviewer
#18 lore raw

Re: [PATCH v3 bpf-next] selftests/bpf: Fix selftests broken by mitigations=off

Quoted 83 lines · Daniel Borkmann
Quoted 69 lines · Yonghong Song
Quoted 60 lines · Yafang Shao

When we configure the kernel command line with 'mitigations=off' and set the sysctl knob 'kernel.unprivileged_bpf_disabled' to 0, the commit bc5bc309db45 ("bpf: Inherit system settings for CPU security mitigations") causes issues in the execution of test_progs -t verifier. This is because 'mitigations=off' bypasses Spectre v1 and Spectre v4 protections.

Currently, when a program requests to run in unprivileged mode (kernel.unprivileged_bpf_disabled = 0), the BPF verifier may prevent it from running due to the following conditions not being enabled:

   - bypass_spec_v1
   - bypass_spec_v4
   - allow_ptr_leaks
   - allow_uninit_stack

While 'mitigations=off' enables the first two conditions, it does not enable the latter two. As a result, some test cases in 'test_progs -t verifier' that were expected to fail to run may run successfully, while others still fail but with different error messages. This makes it challenging to address them comprehensively.

Moreover, in the future, we may introduce more fine-grained control over CPU mitigations, such as enabling only bypass_spec_v1 or bypass_spec_v4.

Given the complexity of the situation, rather than fixing each broken test case individually, it's preferable to skip them when 'mitigations=off' is in effect and introduce specific test cases for the new 'mitigations=off' scenario. For instance, we can introduce new BTF declaration tags like '__failure__nospec', '__failure_nospecv1' and '__failure_nospecv4'.

In this patch, the approach is to simply skip the broken test cases when 'mitigations=off' is enabled. The result of test_progs -t verifier as follows after this commit,

Before this commit
==================
- without 'mitigations=off'
   - kernel.unprivileged_bpf_disabled = 2
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
   - kernel.unprivileged_bpf_disabled = 0
     Summary: 74/1336 PASSED, 0 SKIPPED, 0 FAILED <<<<
- with 'mitigations=off'
   - kernel.unprivileged_bpf_disabled = 2
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
   - kernel.unprivileged_bpf_disabled = 0
     Summary: 63/1276 PASSED, 0 SKIPPED, 11 FAILED <<<< 11 FAILED

After this commit
=================
- without 'mitigations=off'
   - kernel.unprivileged_bpf_disabled = 2
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
   - kernel.unprivileged_bpf_disabled = 0
     Summary: 74/1336 PASSED, 0 SKIPPED, 0 FAILED <<<<
- with this patch, with 'mitigations=off'
   - kernel.unprivileged_bpf_disabled = 2
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED
   - kernel.unprivileged_bpf_disabled = 0
     Summary: 74/948 PASSED, 388 SKIPPED, 0 FAILED <<<< SKIPPED

Quoted 4 lines

Ack with a nit below.

[...]

Quoted 9 lines
Quoted 3 lines

      }
-    return disabled;
+    return disabled ? true : get_mitigations_off();

Above code is correct. But you could slightly simplify it with return disabled ? : get_mitigations_off();

I guess maintainer can decide whether simplification is needed or not.

Turns out if you omit, then compiler will complain with a warning :)

Quoted 8 lines

  [...]
  GEN      vmlinux.h
unpriv_helpers.c: In function ‘get_unpriv_disabled’:
unpriv_helpers.c:56:27: error: the omitted middle operand in ‘?:’ will always be ‘true’, suggest explicit middle operand [-Werror=parentheses] 56 |         return disabled ? : get_mitigations_off();
      |                           ^
cc1: all warnings being treated as errors
make: *** [Makefile:615: /root/linux/tools/testing/selftests/bpf/unpriv_helpers.o] Error 1

clang compiler is okay with '?:' change while gcc compiler issued errors. So yes, existing code is good for both compilers. Thanks!

Quoted 3 lines

So it's okay as is, applied, thanks!

#19 lore raw

Re: [PATCH v3 bpf-next] selftests/bpf: Fix selftests broken by mitigations=off

In reply to Yafang Shao
This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:

Hello:

This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:

Quoted 11 lines · On Wed, 25 Oct 2023 03:11:44 +0000 you wrote:

When we configure the kernel command line with 'mitigations=off' and set the sysctl knob 'kernel.unprivileged_bpf_disabled' to 0, the commit bc5bc309db45 ("bpf: Inherit system settings for CPU security mitigations") causes issues in the execution of test_progs -t verifier. This is because 'mitigations=off' bypasses Spectre v1 and Spectre v4 protections.

Currently, when a program requests to run in unprivileged mode (kernel.unprivileged_bpf_disabled = 0), the BPF verifier may prevent it from running due to the following conditions not being enabled:

[...]

Here is the summary with links:
- [v3,bpf-next] selftests/bpf: Fix selftests broken by mitigations=off

    https://git.kernel.org/bpf/bpf-next/c/399f6185a1c0

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html