BPF/Question: PTR_TRUSTED vs PTR_UNTRUSTED
Subsystems (4)
Quoted 5 lines · Matt Bobrowski
Hey David,
I have a quick question in regards to the type tag
PTR_TRUSTED, which I believe is causing one of my BPF programs to no longer successfully pass the BPF verifier. Given the following BPF program as an example:
Hi Matt,
Thanks for the message. I'll add the bpf list to cc so the community can see and discuss.
Quoted 40 lines
---
SEC("lsm.s/bprm_committed_creds")
intBPF_PROG(bprm_committed_creds, struct linux_binprm *bprm){ int ret; char buf[64] = {0}; ret = bpf_d_path(&bprm->file->f_path, buf, sizeof(buf)); if (ret < 0) { bpf_printk("bpf_d_path: ret=%d", ret); return 0; } return 0; }---
In this case, the
PTR_TO_BTF_IDpointer (&bprm->file->f_path) I imagine is considered to be trusted and can be passed to the BPF helperbpf_d_pathwithout the BPF verifier complaining.On the other hand, given the following relatively similar BPF program as an example:
---
SEC("lsm.s/bprm_committed_creds")
intBPF_PROG(bprm_committed_creds){ int ret; char buf[64] = {0}; struct task_struct *current = bpf_get_current_task_btf(); ret = bpf_d_path(¤t->fs->pwd, buf, sizeof(buf)); if (ret < 0) { bpf_printk("bpf_d_path: ret=%d", ret); return 0; } return 0; }
Quoted 6 lines
---
In this case, the
PTR_TO_BTF_IDpointer (¤t->fs->pwd) is considered to be untrusted and cannot be passed to the BPF helperbpf_d_pathas the BPF verifier fails to load the BPF program with the following message:
The reason this is happening is that the struct fs_struct *fs field of struct task_struct is not marked as inheriting its parent task's trusted status. The following diff would fix the issue:
kernel/bpf/verifier.c +1 -0
@@ -5857,6 +5857,7 @@ BTF_TYPE_SAFE_RCU(struct task_struct) {struct css_set __rcu *cgroups;struct task_struct __rcu *real_parent;struct task_struct *group_leader; +struct fs_struct *fs; }; BTF_TYPE_SAFE_RCU(struct cgroup) {
tools/testing/selftests/bpf/progs/task_kfunc_failure.c +21 -0
@@ -324,3 +324,24 @@ int BPF_PROG(task_kfunc_release_in_map, struct task_struct *task, u64 clone_flagreturn 0 ; } + +SEC("lsm.s/bprm_committed_creds") +__success +int BPF_PROG (bprm_committed_creds ) +{ +int ret; +char buf[64 ] = {0 }; +struct task_struct *current =bpf_get_current_task_btf (); +struct fs_struct *fs; + +bpf_rcu_read_lock (); + fs = current->fs; + ret =bpf_d_path (&fs->pwd, buf,sizeof (buf)); +bpf_rcu_read_unlock (); +if (ret <0 ) { +bpf_printk ("bpf_d_path: ret=%d" , ret); +return 0 ; + } + +return 0 ; +}
With this patch, the above test would pass (meaning the program is successfully verified).
Quoted 21 lines
---
; ret =bpf_d_path(¤t->fs->pwd, buf, sizeof(buf));
15: (b7) r3 = 64 ; R3_w=64
16: (85) call bpf_d_path#147
R1 type=untrusted_ptr_ expected=ptr_,trusted_ptr_,rcu_ptr_
---What's interesting is that based on the following commit (
3f00c52393445bpf: Allow trusted pointers to be passed toKF_TRUSTED_ARGSkfuncs) message:---
PTR_TRUSTEDpointers are passed directly from the kernel as a tracepoint orstruct_opscallback argument. Any nested pointer that is obtained from walking aPTR_TRUSTEDpointer is no longerPTR_TRUSTED. From the example above, the structtask_struct*task argument isPTR_TRUSTED, but the 'nested' pointer obtained from 'task->last_wakee' is notPTR_TRUSTED.
---
Quoted 5 lines
I'm reading this as though the first example program should also fail BPF verification given that a nested pointer to struct path is obtained from walking a
PTR_TRUSTEDpointer, which presumably is structlinux_binprmin this case. What subtle details am I missing here? Why is that the first program loads, but the second does not?
The subtle details are in that these are different types. We can't assume that a child pointer automatically inherits its parent's trusted status for all types, so we have to hard code it for now until gcc supports using btf type tags so this can be expressed with annotations like __trusted or __rcu. Consider that some types may have NULL child pointers in certain scenarios. Others may be valid as long as you access it in an RCU read region, others may be valid as long as you access it in an RCU read region and it wasn't NULL.
The struct linux_binprm type's safety is specified in kernel/bpf/verifier.c:
BTF_TYPE_SAFE_TRUSTED(struct linux_binprm) {
struct file *file;
};struct task_struct is specified above:
/* RCU trusted: these fields are trusted in RCU CS and never NULL */
BTF_TYPE_SAFE_RCU(struct task_struct) {
const cpumask_t *cpus_ptr;
struct css_set __rcu *cgroups;
struct task_struct __rcu *real_parent;
struct task_struct *group_leader;
};Quoted 3 lines
What workaround needs to be implemented in order to have the second example program satisfy the
PTR_TRUSTEDcontract with the BPF helperbpf_d_path?
See above, we need to add struct fs_struct * to the list of trusted fields for struct task_struct, and enclose it in an RCU read region.
Thanks,
David
Hey David!
Thanks for getting back to me, and super sorry about the delay.
Quoted 68 lines · David Vernet
Quoted 5 lines · Matt Bobrowski
Hey David,
I have a quick question in regards to the type tag
PTR_TRUSTED, which I believe is causing one of my BPF programs to no longer successfully pass the BPF verifier. Given the following BPF program as an example:Hi Matt,
Thanks for the message. I'll add the bpf list to cc so the community can see and discuss.
Quoted 40 lines
---
SEC("lsm.s/bprm_committed_creds")
intBPF_PROG(bprm_committed_creds, struct linux_binprm *bprm){ int ret; char buf[64] = {0}; ret = bpf_d_path(&bprm->file->f_path, buf, sizeof(buf)); if (ret < 0) { bpf_printk("bpf_d_path: ret=%d", ret); return 0; } return 0; }---
In this case, the
PTR_TO_BTF_IDpointer (&bprm->file->f_path) I imagine is considered to be trusted and can be passed to the BPF helperbpf_d_pathwithout the BPF verifier complaining.On the other hand, given the following relatively similar BPF program as an example:
---
SEC("lsm.s/bprm_committed_creds")
intBPF_PROG(bprm_committed_creds){ int ret; char buf[64] = {0}; struct task_struct *current = bpf_get_current_task_btf(); ret = bpf_d_path(¤t->fs->pwd, buf, sizeof(buf)); if (ret < 0) { bpf_printk("bpf_d_path: ret=%d", ret); return 0; } return 0; }Quoted 6 lines
---
In this case, the
PTR_TO_BTF_IDpointer (¤t->fs->pwd) is considered to be untrusted and cannot be passed to the BPF helperbpf_d_pathas the BPF verifier fails to load the BPF program with the following message:The reason this is happening is that the struct
fs_struct*fs field of structtask_structis not marked as inheriting its parent task's trusted status. The following diff would fix the issue:kernel/bpf/verifier.c +0
-0
kernel/bpf/verifier.c BTF_TYPE_SAFE_RCU(struct task_struct) { · line 5861
Quoted 5 lines
kernel/bpf/verifier.c +1
-0@@ -5857,6 +5857,7 @@ BTF_TYPE_SAFE_RCU(struct task_struct) {struct css_set __rcu *cgroups;struct task_struct __rcu *real_parent;struct task_struct *group_leader; +struct fs_struct *fs; };
Oh, right. So, if we explicitly dereference the struct fs_struct member of struct task_struct within a RCU read-side critical section, the BPF verifier considers the pointer to struct fs_struct as being safe and trusted. Is that right?
Why is it that we need to explicitly add it to such lists so that they're considered to be trusted and cannot simply perform the bpf_rcu_read_lock/unlock() dance from within the BPF program? Also, should we not add the field to BTF_TYPE_SAFE_RCU_OR_NULL() instead of BTF_TYPE_SAFE_RCU(), as struct fs_struct could perhaps be NULL in some circumstances?
Are you OK with me carrying this recommended patch to the mailing list?
Quoted 61 lines
kernel/bpf/verifier.c +0
-0BTF_TYPE_SAFE_RCU(struct cgroup) {tools/testing/selftests/bpf/progs/task_kfunc_failure.c +21
-0@@ -324,3 +324,24 @@ int BPF_PROG(task_kfunc_release_in_map, struct task_struct *task, u64 clone_flagreturn 0 ; } + +SEC("lsm.s/bprm_committed_creds") +__success +int BPF_PROG (bprm_committed_creds ) +{ +int ret; +char buf[64 ] = {0 }; +struct task_struct *current =bpf_get_current_task_btf (); +struct fs_struct *fs; + +bpf_rcu_read_lock (); + fs = current->fs; + ret =bpf_d_path (&fs->pwd, buf,sizeof (buf)); +bpf_rcu_read_unlock (); +if (ret <0 ) { +bpf_printk ("bpf_d_path: ret=%d" , ret); +return 0 ; + } + +return 0 ; +}With this patch, the above test would pass (meaning the program is successfully verified).
Quoted 21 lines
---
; ret =bpf_d_path(¤t->fs->pwd, buf, sizeof(buf));
15: (b7) r3 = 64 ; R3_w=64
16: (85) call bpf_d_path#147
R1 type=untrusted_ptr_ expected=ptr_,trusted_ptr_,rcu_ptr_
---What's interesting is that based on the following commit (
3f00c52393445bpf: Allow trusted pointers to be passed toKF_TRUSTED_ARGSkfuncs) message:---
PTR_TRUSTEDpointers are passed directly from the kernel as a tracepoint orstruct_opscallback argument. Any nested pointer that is obtained from walking aPTR_TRUSTEDpointer is no longerPTR_TRUSTED. From the example above, the structtask_struct*task argument isPTR_TRUSTED, but the 'nested' pointer obtained from 'task->last_wakee' is notPTR_TRUSTED.
---Quoted 5 lines
I'm reading this as though the first example program should also fail BPF verification given that a nested pointer to struct path is obtained from walking a
PTR_TRUSTEDpointer, which presumably is structlinux_binprmin this case. What subtle details am I missing here? Why is that the first program loads, but the second does not?
Quoted 4 lines
The subtle details are in that these are different types. We can't assume that a child pointer automatically inherits its parent's trusted status for all types, so we have to hard code it for now until gcc supports using btf type tags so this can be expressed with annotations like
__trustedor__rcu.
Oh, interesting. You mention that this is explicitly needed for gcc, so I'm now wondering how the semantics differ when using clang? Where would such annotations (__trusted, __rcu) be applied?
Quoted 3 lines
Consider that some types may have NULL child pointers in certain scenarios. Others may be valid as long as you access it in an RCU read region, others may be valid as long as you access it in an RCU read region and it wasn't NULL.
This makes sense. Thanks for the explanation.
Quoted 17 lines
The struct
linux_binprmtype's safety is specified inkernel/bpf/verifier.c:BTF_TYPE_SAFE_TRUSTED(struct linux_binprm) { struct file *file; };struct
task_structis specified above:/* RCU trusted: these fields are trusted in RCU CS and never NULL */
BTF_TYPE_SAFE_RCU(struct task_struct) { const cpumask_t *cpus_ptr; struct css_set __rcu *cgroups; struct task_struct __rcu *real_parent; struct task_struct *group_leader; };
Quoted 6 lines
Quoted 3 lines
What workaround needs to be implemented in order to have the second example program satisfy the
PTR_TRUSTEDcontract with the BPF helperbpf_d_path?See above, we need to add struct
fs_struct* to the list of trusted fields for structtask_struct, and enclose it in an RCU read region.
/M
Quoted 68 lines · Matt Bobrowski
Hey David!
Thanks for getting back to me, and super sorry about the delay.
Quoted 63 lines · David Vernet
Quoted 5 lines · Matt Bobrowski
Hey David,
I have a quick question in regards to the type tag
PTR_TRUSTED, which I believe is causing one of my BPF programs to no longer successfully pass the BPF verifier. Given the following BPF program as an example:Hi Matt,
Thanks for the message. I'll add the bpf list to cc so the community can see and discuss.
Quoted 40 lines
---
SEC("lsm.s/bprm_committed_creds")
intBPF_PROG(bprm_committed_creds, struct linux_binprm *bprm){ int ret; char buf[64] = {0}; ret = bpf_d_path(&bprm->file->f_path, buf, sizeof(buf)); if (ret < 0) { bpf_printk("bpf_d_path: ret=%d", ret); return 0; } return 0; }---
In this case, the
PTR_TO_BTF_IDpointer (&bprm->file->f_path) I imagine is considered to be trusted and can be passed to the BPF helperbpf_d_pathwithout the BPF verifier complaining.On the other hand, given the following relatively similar BPF program as an example:
---
SEC("lsm.s/bprm_committed_creds")
intBPF_PROG(bprm_committed_creds){ int ret; char buf[64] = {0}; struct task_struct *current = bpf_get_current_task_btf(); ret = bpf_d_path(¤t->fs->pwd, buf, sizeof(buf)); if (ret < 0) { bpf_printk("bpf_d_path: ret=%d", ret); return 0; } return 0; }Quoted 6 lines
---
In this case, the
PTR_TO_BTF_IDpointer (¤t->fs->pwd) is considered to be untrusted and cannot be passed to the BPF helperbpf_d_pathas the BPF verifier fails to load the BPF program with the following message:The reason this is happening is that the struct
fs_struct*fs field of structtask_structis not marked as inheriting its parent task's trusted status. The following diff would fix the issue:
Quoted 15 lines
kernel/bpf/verifier.c BTF_TYPE_SAFE_RCU(struct task_struct) { · line 5861
Quoted 10 lines
kernel/bpf/verifier.c +1
-0@@ -5857,6 +5857,7 @@ BTF_TYPE_SAFE_RCU(struct task_struct) {struct css_set __rcu *cgroups;struct task_struct __rcu *real_parent;struct task_struct *group_leader; +struct fs_struct *fs; };Oh, right. So, if we explicitly dereference the struct
fs_structmember of structtask_structwithin a RCU read-side critical section, the BPF verifier considers the pointer to structfs_structas being safe and trusted. Is that right?
With the above patch, yes.
Quoted 6 lines
Why is it that we need to explicitly add it to such lists so that they're considered to be trusted and cannot simply perform the bpf_rcu_read_lock/unlock() dance from within the BPF program? Also, should we not add the field to
BTF_TYPE_SAFE_RCU_OR_NULL()instead ofBTF_TYPE_SAFE_RCU(), as structfs_structcould perhaps be NULL in some circumstances?
I recommend doing some git log / git blame digging. All of this information was captured in prior discussions. For example, in the patch
[0] which added these structs.
[0]: https://lore.kernel.org/bpf/[email protected]/
Quoted 2 lines
Are you OK with me carrying this recommended patch to the mailing list?
Of course
Quoted 61 lines
Quoted 61 lines
kernel/bpf/verifier.c +0
-0BTF_TYPE_SAFE_RCU(struct cgroup) {tools/testing/selftests/bpf/progs/task_kfunc_failure.c +21
-0@@ -324,3 +324,24 @@ int BPF_PROG(task_kfunc_release_in_map, struct task_struct *task, u64 clone_flagreturn 0 ; } + +SEC("lsm.s/bprm_committed_creds") +__success +int BPF_PROG (bprm_committed_creds ) +{ +int ret; +char buf[64 ] = {0 }; +struct task_struct *current =bpf_get_current_task_btf (); +struct fs_struct *fs; + +bpf_rcu_read_lock (); + fs = current->fs; + ret =bpf_d_path (&fs->pwd, buf,sizeof (buf)); +bpf_rcu_read_unlock (); +if (ret <0 ) { +bpf_printk ("bpf_d_path: ret=%d" , ret); +return 0 ; + } + +return 0 ; +}With this patch, the above test would pass (meaning the program is successfully verified).
Quoted 26 lines
---
; ret =bpf_d_path(¤t->fs->pwd, buf, sizeof(buf));
15: (b7) r3 = 64 ; R3_w=64
16: (85) call bpf_d_path#147
R1 type=untrusted_ptr_ expected=ptr_,trusted_ptr_,rcu_ptr_
---What's interesting is that based on the following commit (
3f00c52393445bpf: Allow trusted pointers to be passed toKF_TRUSTED_ARGSkfuncs) message:---
PTR_TRUSTEDpointers are passed directly from the kernel as a tracepoint orstruct_opscallback argument. Any nested pointer that is obtained from walking aPTR_TRUSTEDpointer is no longerPTR_TRUSTED. From the example above, the structtask_struct*task argument isPTR_TRUSTED, but the 'nested' pointer obtained from 'task->last_wakee' is notPTR_TRUSTED.
---I'm reading this as though the first example program should also fail BPF verification given that a nested pointer to struct path is obtained from walking a
PTR_TRUSTEDpointer, which presumably is structlinux_binprmin this case. What subtle details am I missing here? Why is that the first program loads, but the second does not?
Quoted 8 lines
Quoted 4 lines
The subtle details are in that these are different types. We can't assume that a child pointer automatically inherits its parent's trusted status for all types, so we have to hard code it for now until gcc supports using btf type tags so this can be expressed with annotations like
__trustedor__rcu.Oh, interesting. You mention that this is explicitly needed for gcc, so I'm now wondering how the semantics differ when using clang? Where would such annotations (
__trusted,__rcu) be applied?
Please see the patch I linked above :-) The point of Alexei's patch was to keep the same semantics between gcc and clang until gcc finishes adding support for btf type tags. I recommend reading [1] to learn more about BTF, if you're unfamiliar.
[1]: https://docs.kernel.org/bpf/btf.html
Once gcc supports BTF type tags, we can apply __trusted, __rcu, etc annotations directly where the type is defined to encode the information in BTF, and then use that information in the verifier to inform pointer safety guarantees. This is already done in many places throughout the kernel. See e.g. various fields in the definition of struct task_struct.
Quoted 31 lines
Quoted 3 lines
Consider that some types may have NULL child pointers in certain scenarios. Others may be valid as long as you access it in an RCU read region, others may be valid as long as you access it in an RCU read region and it wasn't NULL.
This makes sense. Thanks for the explanation.
Quoted 17 lines
The struct
linux_binprmtype's safety is specified inkernel/bpf/verifier.c:BTF_TYPE_SAFE_TRUSTED(struct linux_binprm) { struct file *file; };struct
task_structis specified above:/* RCU trusted: these fields are trusted in RCU CS and never NULL */
BTF_TYPE_SAFE_RCU(struct task_struct) { const cpumask_t *cpus_ptr; struct css_set __rcu *cgroups; struct task_struct __rcu *real_parent; struct task_struct *group_leader; };Quoted 6 lines
Quoted 3 lines
What workaround needs to be implemented in order to have the second example program satisfy the
PTR_TRUSTEDcontract with the BPF helperbpf_d_path?See above, we need to add struct
fs_struct* to the list of trusted fields for structtask_struct, and enclose it in an RCU read region./M
Quoted 16 lines · David Vernet
Quoted 7 lines
Quoted 7 lines
kernel/bpf/verifier.c +0
-0@@ -5857,6 +5857,7 @@ BTF_TYPE_SAFE_RCU(struct task_struct) {struct css_set __rcu *cgroups;struct task_struct __rcu *real_parent;Quoted 8 lines
kernel/bpf/verifier.c BTF_TYPE_SAFE_RCU(struct task_struct) { · line 5861
Quoted 3 lines
kernel/bpf/verifier.c +1
-0struct task_struct *group_leader; +struct fs_struct *fs; };Oh, right. So, if we explicitly dereference the struct
fs_structmember of structtask_structwithin a RCU read-side critical section, the BPF verifier considers the pointer to structfs_structas being safe and trusted. Is that right?
Quoted 1 line
With the above patch, yes.
After conducting some further tests today, it turns out that making amendments to the struct task_struct BTF_TYPE_SAFE_RCU definition perhaps isn't actually necessary? As of commit afeebf9f57a49 ("bpf:
Undo strict enforcement for walking untagged fields"), if a trusted pointer (in this case being struct task_struct obtained via bpf_get_current_task_btf()) is dereferenced within a RCU read-side critical section, then the pointer that is yielded as a result of the walk/dereference operation is a PTR_TO_BTF_ID. It is neither trusted or untrusted and therefore carries the same level of semantics as a dereferenced pointer before any trust status for pointers was introduced within the BPF verifier.
Have I misunderstood something here?
Quoted 11 lines
Quoted 6 lines
Why is it that we need to explicitly add it to such lists so that they're considered to be trusted and cannot simply perform the bpf_rcu_read_lock/unlock() dance from within the BPF program? Also, should we not add the field to
BTF_TYPE_SAFE_RCU_OR_NULL()instead ofBTF_TYPE_SAFE_RCU(), as structfs_structcould perhaps be NULL in some circumstances?I recommend doing some git log / git blame digging. All of this information was captured in prior discussions. For example, in the patch
[0] which added these structs.
Quoted 6 lines
[0]: https://lore.kernel.org/bpf/[email protected]/
Quoted 2 lines
Are you OK with me carrying this recommended patch to the mailing list?
Of course
Based on what I've mentioned above, perhaps sending through a patch no longer is necessary?
/M
Quoted 31 lines · Matt Bobrowski
Quoted 16 lines · David Vernet
Quoted 7 lines
Quoted 7 lines
kernel/bpf/verifier.c +0
-0@@ -5857,6 +5857,7 @@ BTF_TYPE_SAFE_RCU(struct task_struct) {struct css_set __rcu *cgroups;struct task_struct __rcu *real_parent;Quoted 8 lines
kernel/bpf/verifier.c BTF_TYPE_SAFE_RCU(struct task_struct) { · line 5861
Quoted 3 lines
kernel/bpf/verifier.c +1
-0struct task_struct *group_leader; +struct fs_struct *fs; };Oh, right. So, if we explicitly dereference the struct
fs_structmember of structtask_structwithin a RCU read-side critical section, the BPF verifier considers the pointer to structfs_structas being safe and trusted. Is that right?Quoted 1 line
With the above patch, yes.
After conducting some further tests today, it turns out that making amendments to the struct
task_structBTF_TYPE_SAFE_RCUdefinition perhaps isn't actually necessary? As of commitafeebf9f57a49("bpf:
Undo strict enforcement for walking untagged fields"), if a trusted pointer (in this case being structtask_structobtained viabpf_get_current_task_btf()) is dereferenced within a RCU read-side critical section, then the pointer that is yielded as a result of the walk/dereference operation is aPTR_TO_BTF_ID. It is neither trusted or untrusted and therefore carries the same level of semantics as a dereferenced pointer before any trust status for pointers was introduced within the BPF verifier.
Quoted 1 line
Have I misunderstood something here?
No, that's correct. You only need the aforementioned patch if you need the pointer to be a trusted or RCU pointer.
Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
Why is it that we need to explicitly add it to such lists so that they're considered to be trusted and cannot simply perform the bpf_rcu_read_lock/unlock() dance from within the BPF program? Also, should we not add the field to
BTF_TYPE_SAFE_RCU_OR_NULL()instead ofBTF_TYPE_SAFE_RCU(), as structfs_structcould perhaps be NULL in some circumstances?I recommend doing some git log / git blame digging. All of this information was captured in prior discussions. For example, in the patch
[0] which added these structs.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Are you OK with me carrying this recommended patch to the mailing list?
Of course
Based on what I've mentioned above, perhaps sending through a patch no longer is necessary?
If you only need to call bpf_d_path() then yes, you shouldn't need the patch.