Skip to contentSkip to conversation

bpf: Allow get bpf object with CAP_BPF

RFC PATCH In review Yafang Shao 3 participants 9 messages Nov 29, 2022 – Dec 2, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 bpf: Allow get bpf object with CAP_BPF

In the containerized envriomentation, if a container is not privileged but with CAP_BPF, it is not easy to debug bpf created in this container, let alone using bpftool. Because these bpf objects are invisible if they are not pinned in bpffs. Currently we have to interact with the process which creates these bpf objects to get the information. It may be better if we can control the access to each object the same way as we control the file in bpffs, but now I think we should allow the accessibility of these objects with CAP_BPF.

---
 kernel/bpf/syscall.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/bpf/syscall.c +5 -5
index 35972afb6850..9cd6b41e2d2b 100644
@@ -3660,7 +3660,7 @@ static int bpf_obj_get_next_id(const union bpf_attr *attr,
 	if (CHECK_ATTR(BPF_OBJ_GET_NEXT_ID) || next_id >= INT_MAX)
 		return -EINVAL;
 
-	if (!capable(CAP_SYS_ADMIN))
+	if (!bpf_capable())
 		return -EPERM;
 
 	next_id++;
@@ -3741,7 +3741,7 @@ static int bpf_prog_get_fd_by_id(const union bpf_attr *attr)
 	if (CHECK_ATTR(BPF_PROG_GET_FD_BY_ID))
 		return -EINVAL;
 
-	if (!capable(CAP_SYS_ADMIN))
+	if (!bpf_capable())
 		return -EPERM;
 
 	prog = bpf_prog_by_id(id);
@@ -3768,7 +3768,7 @@ static int bpf_map_get_fd_by_id(const union bpf_attr *attr)
 	    attr->open_flags & ~BPF_OBJ_FLAG_MASK)
 		return -EINVAL;
 
-	if (!capable(CAP_SYS_ADMIN))
+	if (!bpf_capable())
 		return -EPERM;
 
 	f_flags = bpf_get_file_flag(attr->open_flags);
@@ -4345,7 +4345,7 @@ static int bpf_btf_get_fd_by_id(const union bpf_attr *attr)
 	if (CHECK_ATTR(BPF_BTF_GET_FD_BY_ID))
 		return -EINVAL;
 
-	if (!capable(CAP_SYS_ADMIN))
+	if (!bpf_capable())
 		return -EPERM;
 
 	return btf_get_fd_by_id(attr->btf_id);
@@ -4769,7 +4769,7 @@ static int bpf_link_get_fd_by_id(const union bpf_attr *attr)
 	if (CHECK_ATTR(BPF_LINK_GET_FD_BY_ID))
 		return -EINVAL;
 
-	if (!capable(CAP_SYS_ADMIN))
+	if (!bpf_capable())
 		return -EPERM;
 
 	link = bpf_link_by_id(id);
Signature · 1 line
2.30.1 (Apple Git-130)
#1 lore raw
Quoted 10 lines · Yafang Shao

In the containerized envriomentation, if a container is not privileged but with CAP_BPF, it is not easy to debug bpf created in this container, let alone using bpftool. Because these bpf objects are invisible if they are not pinned in bpffs. Currently we have to interact with the process which creates these bpf objects to get the information. It may be better if we can control the access to each object the same way as we control the file in bpffs, but now I think we should allow the accessibility of these objects with CAP_BPF.

Quoted 5 lines
---
 kernel/bpf/syscall.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

As far as I can tell, requiring CAP_SYS_ADMIN on iterating IDs and converting IDs to FDs is intended and is an important design in BPF's security model [1]. So this change does not look good.

From the commit message, I'm not clear how BPF is debugged in containers in your use case. Maybe the debugging process should be required to have CAP_SYS_ADMIN?

[1] https://lore.kernel.org/bpf/[email protected]/

#2 lore raw
Quoted 18 lines · Hao Luo
Quoted 10 lines · Yafang Shao

In the containerized envriomentation, if a container is not privileged but with CAP_BPF, it is not easy to debug bpf created in this container, let alone using bpftool. Because these bpf objects are invisible if they are not pinned in bpffs. Currently we have to interact with the process which creates these bpf objects to get the information. It may be better if we can control the access to each object the same way as we control the file in bpffs, but now I think we should allow the accessibility of these objects with CAP_BPF.

Quoted 5 lines
---
 kernel/bpf/syscall.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
Quoted 4 lines

As far as I can tell, requiring CAP_SYS_ADMIN on iterating IDs and converting IDs to FDs is intended and is an important design in BPF's security model [1]. So this change does not look good.

I understand that allowing ID->FD transition for CAP_SYS_ADMIN only is for security.
But it also prevents the user from transiting its own bpf object ID, that is a problem.

Quoted 4 lines

From the commit message, I'm not clear how BPF is debugged in containers in your use case. Maybe the debugging process should be required to have CAP_SYS_ADMIN?

Some container users will run bpf programs in their container, sometimes they want to check the bpf objects created by themselves by using bpftool or read/write the bpf maps with their own tools. But if the bpf objects are not pinned, the only way to get these bpf objects is via SCM_RIGHTS.
There should be a general way to get the FD of their own objects when CAP_BPF is enabled.
With CAP_SYS_ADMIN, the container user can do almost anything, which is very dangerous.
While with CAP_BPF, the risk can be kept within BPF.

I think we should improve this situation by allowing the user to transit its own bpf object IDs.
There are some possible solutions,
1. introduce BPF_ID namespace
Let's use namespace to isolate the bpf object ID instead of
preventing them from reading all IDs.
2. introduce a global sysctl knob to allow users to do the ID->FD transition for example, introduce a new value into unprivileged_bpf_disabled. -0 Unprivileged calls to ``bpf()`` are enabled
+0 Unprivileged calls to ``bpf()`` are enabled except the calls
+ which explicitly requires ``CAP_BPF`` or ``CAP_SYS_ADMIN`` 1 Unprivileged calls to ``bpf()`` are disabled without recovery 2 Unprivileged calls to ``bpf()`` are disabled
+3 All unprivileged calls to ``bpf()`` are enabled

WDYT ?

Signature · 2 lines
Regards
Yafang
reviewer
#3 lore raw
Quoted 1 line · Yafang Shao

[...]

Quoted 36 lines

I understand that allowing ID->FD transition for CAP_SYS_ADMIN only is for security.
But it also prevents the user from transiting its own bpf object ID, that is a problem.

Quoted 4 lines

From the commit message, I'm not clear how BPF is debugged in containers in your use case. Maybe the debugging process should be required to have CAP_SYS_ADMIN?

Some container users will run bpf programs in their container, sometimes they want to check the bpf objects created by themselves by using bpftool or read/write the bpf maps with their own tools. But if the bpf objects are not pinned, the only way to get these bpf objects is via SCM_RIGHTS.
There should be a general way to get the FD of their own objects when CAP_BPF is enabled.
With CAP_SYS_ADMIN, the container user can do almost anything, which is very dangerous.
While with CAP_BPF, the risk can be kept within BPF.

I think we should improve this situation by allowing the user to transit its own bpf object IDs.
There are some possible solutions,
1. introduce BPF_ID namespace
Let's use namespace to isolate the bpf object ID instead of
preventing them from reading all IDs.
2. introduce a global sysctl knob to allow users to do the ID->FD transition for example, introduce a new value into unprivileged_bpf_disabled. -0 Unprivileged calls to ``bpf()`` are enabled
+0 Unprivileged calls to ``bpf()`` are enabled except the calls
+ which explicitly requires ``CAP_BPF`` or ``CAP_SYS_ADMIN`` 1 Unprivileged calls to ``bpf()`` are disabled without recovery 2 Unprivileged calls to ``bpf()`` are disabled
+3 All unprivileged calls to ``bpf()`` are enabled

Quoted 1 line

WDYT ?

Personally, I think some namespace might be the solution we need. But adding a namespace is a lot of work, so we need to make sure to do it correctly.

This might be a good topic to discuss in the BPF office hour.

Thanks,
Song

#4 lore raw
Quoted 40 lines · Song Liu
Quoted 1 line · Yafang Shao

[...]

Quoted 36 lines

I understand that allowing ID->FD transition for CAP_SYS_ADMIN only is for security.
But it also prevents the user from transiting its own bpf object ID, that is a problem.

Quoted 4 lines

From the commit message, I'm not clear how BPF is debugged in containers in your use case. Maybe the debugging process should be required to have CAP_SYS_ADMIN?

Some container users will run bpf programs in their container, sometimes they want to check the bpf objects created by themselves by using bpftool or read/write the bpf maps with their own tools. But if the bpf objects are not pinned, the only way to get these bpf objects is via SCM_RIGHTS.
There should be a general way to get the FD of their own objects when CAP_BPF is enabled.
With CAP_SYS_ADMIN, the container user can do almost anything, which is very dangerous.
While with CAP_BPF, the risk can be kept within BPF.

I think we should improve this situation by allowing the user to transit its own bpf object IDs.
There are some possible solutions,
1. introduce BPF_ID namespace
Let's use namespace to isolate the bpf object ID instead of
preventing them from reading all IDs.
2. introduce a global sysctl knob to allow users to do the ID->FD transition for example, introduce a new value into unprivileged_bpf_disabled. -0 Unprivileged calls to ``bpf()`` are enabled
+0 Unprivileged calls to ``bpf()`` are enabled except the calls
+ which explicitly requires ``CAP_BPF`` or ``CAP_SYS_ADMIN`` 1 Unprivileged calls to ``bpf()`` are disabled without recovery 2 Unprivileged calls to ``bpf()`` are disabled
+3 All unprivileged calls to ``bpf()`` are enabled

Quoted 8 lines
Quoted 1 line

WDYT ?

Personally, I think some namespace might be the solution we need. But adding a namespace is a lot of work, so we need to make sure to do it correctly.

This might be a good topic to discuss in the BPF office hour.

I think namespace is more preferable. A discussion in the BPF office hour sounds good.

Following are my thoughts:

1. What does the BPF_ID namespace look like? Will it be like the PID namespace, remapping IDs in each namespace? or just restricting the object IDs visible to the users?

2. What's wrong with passing FD? Is it really necessary to introduce a namespace for this purpose?

3. IIRC, Song proposed introducing a namespace for BPF isolation, not just isolating IDs [1]. How does it relate to the BPF_ID namespace?

[1] https://lore.kernel.org/all/CAPhsuW6c17p3XkzSxxo7YBW9LHjqerOqQvt7C1+S--8C9omeng@mail.gmail.com/

#5 lore raw
Quoted 48 lines · Hao Luo
Quoted 46 lines · Song Liu
Quoted 1 line · Yafang Shao

[...]

Quoted 36 lines

I understand that allowing ID->FD transition for CAP_SYS_ADMIN only is for security.
But it also prevents the user from transiting its own bpf object ID, that is a problem.

Quoted 4 lines

From the commit message, I'm not clear how BPF is debugged in containers in your use case. Maybe the debugging process should be required to have CAP_SYS_ADMIN?

Some container users will run bpf programs in their container, sometimes they want to check the bpf objects created by themselves by using bpftool or read/write the bpf maps with their own tools. But if the bpf objects are not pinned, the only way to get these bpf objects is via SCM_RIGHTS.
There should be a general way to get the FD of their own objects when CAP_BPF is enabled.
With CAP_SYS_ADMIN, the container user can do almost anything, which is very dangerous.
While with CAP_BPF, the risk can be kept within BPF.

I think we should improve this situation by allowing the user to transit its own bpf object IDs.
There are some possible solutions,
1. introduce BPF_ID namespace
Let's use namespace to isolate the bpf object ID instead of
preventing them from reading all IDs.
2. introduce a global sysctl knob to allow users to do the ID->FD transition for example, introduce a new value into unprivileged_bpf_disabled. -0 Unprivileged calls to ``bpf()`` are enabled
+0 Unprivileged calls to ``bpf()`` are enabled except the calls
+ which explicitly requires ``CAP_BPF`` or ``CAP_SYS_ADMIN`` 1 Unprivileged calls to ``bpf()`` are disabled without recovery 2 Unprivileged calls to ``bpf()`` are disabled
+3 All unprivileged calls to ``bpf()`` are enabled

Quoted 1 line

WDYT ?

Personally, I think some namespace might be the solution we need. But adding a namespace is a lot of work, so we need to make sure to do it correctly.

Quoted 8 lines
Quoted 2 lines

This might be a good topic to discuss in the BPF office hour.

I think namespace is more preferable. A discussion in the BPF office hour sounds good.

Following are my thoughts:

Thanks for your thoughts.

Quoted 4 lines

1. What does the BPF_ID namespace look like? Will it be like the PID namespace, remapping IDs in each namespace? or just restricting the object IDs visible to the users?

I prefer the former. It looks like the PID namespace, which also uses the idr_alloc().

Quoted 3 lines

2. What's wrong with passing FD? Is it really necessary to introduce a namespace for this purpose?

Passing FD is not flexible, and generic tools like bpftool can't work. In the long run, I think the restriction of CAP_SYS_ADMIN should be replaced by better isolation mechanisms, so introducing a namespace to replace it won't be a bad idea.

Quoted 4 lines

3. IIRC, Song proposed introducing a namespace for BPF isolation, not just isolating IDs [1]. How does it relate to the BPF_ID namespace?

[1] https://lore.kernel.org/all/CAPhsuW6c17p3XkzSxxo7YBW9LHjqerOqQvt7C1+S--8C9omeng@mail.gmail.com/

I have looked through the slides of this proposal, but failed to figure out how Song will design the BPF namespace. Maybe Song can give us a better explanation.
Per my understanding, the goal of Song's proposal should be combined by many namespaces and other isolation mechanisms. For example, with the help of PID namespace, we can make sure only the tasks in this container can be traced by the bpf programs running in it.

Signature · 2 lines
Regards
Yafang
#6 lore raw
Quoted 2 lines · Yafang Shao

On Thu, Dec 1, 2022 at 8:38 AM Hao Luo <[email protected]> wrote:

<...>

Quoted 5 lines
Quoted 4 lines

3. IIRC, Song proposed introducing a namespace for BPF isolation, not just isolating IDs [1]. How does it relate to the BPF_ID namespace?

[1] https://lore.kernel.org/all/CAPhsuW6c17p3XkzSxxo7YBW9LHjqerOqQvt7C1+S--8C9omeng@mail.gmail.com/

Quoted 8 lines

I have looked through the slides of this proposal, but failed to figure out how Song will design the BPF namespace. Maybe Song can give us a better explanation.
Per my understanding, the goal of Song's proposal should be combined by many namespaces and other isolation mechanisms. For example, with the help of PID namespace, we can make sure only the tasks in this container can be traced by the bpf programs running in it.

Among the 5 items in [1], it looks like the third item "Limit which BPF programs are accessible to non-root users" is what you proposed here. The other items are more about isolation, I think. So, the question is, if we have a BPF_ID namespace, would that be sufficient for debugging in containers? If yes, at least it's something useful. We can start from the BPF_ID namespace, bring it for discussion, and gather other requirements gradually.

reviewer
#7 lore raw
Quoted 10 lines · Hao Luo
Quoted 2 lines · Yafang Shao

On Thu, Dec 1, 2022 at 8:38 AM Hao Luo <[email protected]> wrote:

<...>

Quoted 5 lines
Quoted 4 lines

3. IIRC, Song proposed introducing a namespace for BPF isolation, not just isolating IDs [1]. How does it relate to the BPF_ID namespace?

[1] https://lore.kernel.org/all/CAPhsuW6c17p3XkzSxxo7YBW9LHjqerOqQvt7C1+S--8C9omeng@mail.gmail.com/

Quoted 7 lines
Quoted 7 lines

I have looked through the slides of this proposal, but failed to figure out how Song will design the BPF namespace. Maybe Song can give us a better explanation.
Per my understanding, the goal of Song's proposal should be combined by many namespaces and other isolation mechanisms. For example, with the help of PID namespace, we can make sure only the tasks in this container can be traced by the bpf programs running in it.

The proposal didn't really go anywhere. LOL.

As Yafang said, it requires multiple mechanisms to work together, and thus is very complicated. OTOH, we are not sure whether BPF tracing is still useful when it is really safe. Specifically, probe_read is not safe, but really useful.

A related idea is tracer namespace, presented by Mathieu Desnoyers at LPC 2022. [2]

[2] https://lpc.events/event/16/contributions/1237/

Quoted 9 lines
Quoted 1 line

Among the 5 items in [1], it looks like the third item "Limit which BPF programs are accessible to non-root users" is what you proposed here. The other items are more about isolation, I think. So, the question is, if we have a BPF_ID namespace, would that be sufficient for debugging in containers? If yes, at least it's something useful. We can start from the BPF_ID namespace, bring it for discussion, and gather other requirements gradually.

BPF_ID namespace is a better defined idea. I am not sure whether we want the complexity of a namespace, though.

Thanks,
Song

#8 lore raw
In reply to Song Liu
Quoted 40 lines · Song Liu
Quoted 1 line · Yafang Shao

[...]

Quoted 36 lines

I understand that allowing ID->FD transition for CAP_SYS_ADMIN only is for security.
But it also prevents the user from transiting its own bpf object ID, that is a problem.

Quoted 4 lines

From the commit message, I'm not clear how BPF is debugged in containers in your use case. Maybe the debugging process should be required to have CAP_SYS_ADMIN?

Some container users will run bpf programs in their container, sometimes they want to check the bpf objects created by themselves by using bpftool or read/write the bpf maps with their own tools. But if the bpf objects are not pinned, the only way to get these bpf objects is via SCM_RIGHTS.
There should be a general way to get the FD of their own objects when CAP_BPF is enabled.
With CAP_SYS_ADMIN, the container user can do almost anything, which is very dangerous.
While with CAP_BPF, the risk can be kept within BPF.

I think we should improve this situation by allowing the user to transit its own bpf object IDs.
There are some possible solutions,
1. introduce BPF_ID namespace
Let's use namespace to isolate the bpf object ID instead of
preventing them from reading all IDs.
2. introduce a global sysctl knob to allow users to do the ID->FD transition for example, introduce a new value into unprivileged_bpf_disabled. -0 Unprivileged calls to ``bpf()`` are enabled
+0 Unprivileged calls to ``bpf()`` are enabled except the calls
+ which explicitly requires ``CAP_BPF`` or ``CAP_SYS_ADMIN`` 1 Unprivileged calls to ``bpf()`` are disabled without recovery 2 Unprivileged calls to ``bpf()`` are disabled
+3 All unprivileged calls to ``bpf()`` are enabled

Quoted 6 lines
Quoted 1 line

WDYT ?

Personally, I think some namespace might be the solution we need. But adding a namespace is a lot of work, so we need to make sure to do it correctly.

Right, lots of code to write. I will think about it carefully.

Quoted 2 lines

This might be a good topic to discuss in the BPF office hour.

Once I figure out a workable solution, I will post a proposal.

--
Regards
Yafang