Skip to contentSkip to conversation

Improve prog array uref semantics

RFC PATCH In review Daniel Xu 3 participants 9 messages Aug 23, 2023 – Nov 20, 2024

Subsystems (7)
lore
Share & tools
lore

Daniel Xu

author
#0 lore raw

cover Improve prog array uref semantics

This patchset changes the behavior of TC and XDP hooks during attachment such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra uref taken.

The goal behind this change is to try and prevent confusion for the majority of use cases. The current behavior where when the last uref is dropped the prog array map is emptied is quite confusing. Confusing enough for there to be multiple references to it in ebpf-go [0][1].

Completely solving the problem is difficult. As stated in c9da161c6517 ("bpf: fix clearing on persistent program array maps"), it is difficult-to-impossible to walk the full dependency graph b/c it is too dynamic.

However in practice, I've found that all progs in a tailcall chain share the same prog array map. Knowing that, if we take a uref on any used prog array map when the program is attached, we can simplify the majority use case and make it more ergonomic.

I'll be the first to admit this is not a very clean solution. It does not fully solve the problem. Nor does it make overall logic any simpler. But I do think it makes a pretty big usability hole slightly smaller.

I've done some basic testing using a repro program [3] I wrote to debug the original issue that eventually led me to this patchset. If we wanna move forward with this approach, I'll resend with selftests.

[0]: https://github.com/cilium/ebpf/blob/01ebd4c1e2b9f8b3dd4fd2382aa1092c3c9bfc9d/doc.go#L22-L24
[1]: https://github.com/cilium/ebpf/blob/d1a52333f2c0fed085f8d742a5a3c164795d8492/collection.go#L320-L321
[2]: https://github.com/danobi/tc_tailcall_repro

Daniel Xu (2):
net: bpf: Make xdp and cls_bpf use bpf_prog_put_dev()
bpf: Take a uref on BPF_MAP_TYPE_PROG_ARRAY maps during dev attachment

 include/linux/bpf.h  |  1 +
 kernel/bpf/devmap.c  |  8 ++++----
 kernel/bpf/syscall.c | 46 +++++++++++++++++++++++++++++++++++++++++++-
 net/core/dev.c       | 16 +++++++--------
 net/sched/cls_bpf.c  |  4 ++--
 5 files changed, 60 insertions(+), 15 deletions(-)
Signature · 1 line
2.41.0
author
#1 lore raw

RFC PATCH 1/2 net: bpf: Make xdp and cls_bpf use bpf_prog_put_dev()

This commit adds a stubbed bpf_prog_put_dev() that is symmetric to bpf_prog_get_type_dev() such that all bpf device attachments are using a *_dev() API.

This gives core bpf the ability to do special refcnt handling for device attachments.

---
 include/linux/bpf.h  |  1 +
 kernel/bpf/devmap.c  |  8 ++++----
 kernel/bpf/syscall.c |  6 ++++++
 net/core/dev.c       | 16 ++++++++--------
 net/sched/cls_bpf.c  |  4 ++--
 5 files changed, 21 insertions(+), 14 deletions(-)
include/linux/bpf.h +1 -0
index eced6400f778..08269ad8cc45 100644
@@ -2030,6 +2030,7 @@ void bpf_prog_sub(struct bpf_prog *prog, int i);
 void bpf_prog_inc(struct bpf_prog *prog);
 struct bpf_prog * __must_check bpf_prog_inc_not_zero(struct bpf_prog *prog);
 void bpf_prog_put(struct bpf_prog *prog);
+void bpf_prog_put_dev(struct bpf_prog *prog);
 
 void bpf_prog_free_id(struct bpf_prog *prog);
 void bpf_map_free_id(struct bpf_map *map);
kernel/bpf/devmap.c +4 -4
index 4d42f6ed6c11..b5d33a87a560 100644
@@ -212,7 +212,7 @@ static void dev_map_free(struct bpf_map *map)
 			hlist_for_each_entry_safe(dev, next, head, index_hlist) {
 				hlist_del_rcu(&dev->index_hlist);
 				if (dev->xdp_prog)
-					bpf_prog_put(dev->xdp_prog);
+					bpf_prog_put_dev(dev->xdp_prog);
 				dev_put(dev->dev);
 				kfree(dev);
 			}
@@ -228,7 +228,7 @@ static void dev_map_free(struct bpf_map *map)
 				continue;
 
 			if (dev->xdp_prog)
-				bpf_prog_put(dev->xdp_prog);
+				bpf_prog_put_dev(dev->xdp_prog);
 			dev_put(dev->dev);
 			kfree(dev);
 		}
@@ -800,7 +800,7 @@ static void __dev_map_entry_free(struct rcu_head *rcu)
 
 	dev = container_of(rcu, struct bpf_dtab_netdev, rcu);
 	if (dev->xdp_prog)
-		bpf_prog_put(dev->xdp_prog);
+		bpf_prog_put_dev(dev->xdp_prog);
 	dev_put(dev->dev);
 	kfree(dev);
 }
@@ -884,7 +884,7 @@ static struct bpf_dtab_netdev *__dev_map_alloc_node(struct net *net,
 
 	return dev;
 err_put_prog:
-	bpf_prog_put(prog);
+	bpf_prog_put_dev(prog);
 err_put_dev:
 	dev_put(dev->dev);
 err_out:
kernel/bpf/syscall.c +6 -0
index 10666d17b9e3..d8e5530598f3 100644
@@ -2164,6 +2164,12 @@ void bpf_prog_put(struct bpf_prog *prog)
 }
 EXPORT_SYMBOL_GPL(bpf_prog_put);
 
+void bpf_prog_put_dev(struct bpf_prog *prog)
+{
+	bpf_prog_put(prog);
+}
+EXPORT_SYMBOL_GPL(bpf_prog_put_dev);
+
 static int bpf_prog_release(struct inode *inode, struct file *filp)
 {
 	struct bpf_prog *prog = filp->private_data;
net/core/dev.c +8 -8
index 17e6281e408c..ed0ece344416 100644
@@ -5676,7 +5676,7 @@ static int generic_xdp_install(struct net_device *dev, struct netdev_bpf *xdp)
 	case XDP_SETUP_PROG:
 		rcu_assign_pointer(dev->xdp_prog, new);
 		if (old)
-			bpf_prog_put(old);
+			bpf_prog_put_dev(old);
 
 		if (old && !new) {
 			static_branch_dec(&generic_xdp_needed_key);
@@ -9167,7 +9167,7 @@ static int dev_xdp_install(struct net_device *dev, enum bpf_xdp_mode mode,
 	err = bpf_op(dev, &xdp);
 	if (err) {
 		if (prog)
-			bpf_prog_put(prog);
+			bpf_prog_put_dev(prog);
 		return err;
 	}
 
@@ -9202,7 +9202,7 @@ static void dev_xdp_uninstall(struct net_device *dev)
 		if (link)
 			link->dev = NULL;
 		else
-			bpf_prog_put(prog);
+			bpf_prog_put_dev(prog);
 
 		dev_xdp_set_link(dev, mode, NULL);
 	}
@@ -9326,7 +9326,7 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extack
 	else
 		dev_xdp_set_prog(dev, mode, new_prog);
 	if (cur_prog)
-		bpf_prog_put(cur_prog);
+		bpf_prog_put_dev(cur_prog);
 
 	return 0;
 }
@@ -9445,7 +9445,7 @@ static int bpf_xdp_link_update(struct bpf_link *link, struct bpf_prog *new_prog,
 
 	if (old_prog == new_prog) {
 		/* no-op, don't disturb drivers */
-		bpf_prog_put(new_prog);
+		bpf_prog_put_dev(new_prog);
 		goto out_unlock;
 	}
 
@@ -9457,7 +9457,7 @@ static int bpf_xdp_link_update(struct bpf_link *link, struct bpf_prog *new_prog,
 		goto out_unlock;
 
 	old_prog = xchg(&link->prog, new_prog);
-	bpf_prog_put(old_prog);
+	bpf_prog_put_dev(old_prog);
 
 out_unlock:
 	rtnl_unlock();
@@ -9568,9 +9568,9 @@ int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
 
 err_out:
 	if (err && new_prog)
-		bpf_prog_put(new_prog);
+		bpf_prog_put_dev(new_prog);
 	if (old_prog)
-		bpf_prog_put(old_prog);
+		bpf_prog_put_dev(old_prog);
 	return err;
 }
 
net/sched/cls_bpf.c +2 -2
index 382c7a71f81f..20129d73dab4 100644
@@ -258,7 +258,7 @@ static int cls_bpf_init(struct tcf_proto *tp)
 static void cls_bpf_free_parms(struct cls_bpf_prog *prog)
 {
 	if (cls_bpf_is_ebpf(prog))
-		bpf_prog_put(prog->filter);
+		bpf_prog_put_dev(prog->filter);
 	else
 		bpf_prog_destroy(prog->filter);
 
@@ -391,7 +391,7 @@ static int cls_bpf_prog_from_efd(struct nlattr **tb, struct cls_bpf_prog *prog,
 	if (tb[TCA_BPF_NAME]) {
 		name = nla_memdup(tb[TCA_BPF_NAME], GFP_KERNEL);
 		if (!name) {
-			bpf_prog_put(fp);
+			bpf_prog_put_dev(fp);
 			return -ENOMEM;
 		}
 	}
Signature · 1 line
2.41.0
#2 lore raw

Re: [RFC PATCH bpf-next 1/2] net: bpf: Make xdp and cls_bpf use bpf_prog_put_dev()

kernel test robot noticed the following build errors:

Hi Daniel,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on bpf-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Daniel-Xu/net-bpf-Make-xdp-and-cls_bpf-use-bpf_prog_put_dev/20230823-081039
base:   https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link:    https://lore.kernel.org/r/55ffbb77cd34d3460b16a3d79877d41ad7c43a34.1692748902.git.dxu%40dxuuu.xyz
patch subject: [RFC PATCH bpf-next 1/2] net: bpf: Make xdp and cls_bpf use bpf_prog_put_dev()
config: loongarch-randconfig-r023-20230823 (https://download.01.org/0day-ci/archive/20230823/[email protected]/config)
compiler: loongarch64-linux-gcc (GCC) 13.2.0
reproduce: (https://download.01.org/0day-ci/archive/20230823/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

net/core/dev.c: In function 'generic_xdp_install':

>> net/core/dev.c:5679:25: error: implicit declaration of function 'bpf_prog_put_dev'; did you mean 'bpf_prog_put'? [-Werror=implicit-function-declaration]
    5679 |                         bpf_prog_put_dev(old);
         |                         ^~~~~~~~~~~~~~~~
         |                         bpf_prog_put

cc1: some warnings being treated as errors

vim +5679 net/core/dev.c

5668
5669 static int generic_xdp_install(struct net_device *dev, struct netdev_bpf *xdp)
5670 {
5671 struct bpf_prog *old = rtnl_dereference(dev->xdp_prog);
5672 struct bpf_prog *new = xdp->prog;
5673 int ret = 0;
5674
5675 switch (xdp->command) {
5676 case XDP_SETUP_PROG:
5677 rcu_assign_pointer(dev->xdp_prog, new);
5678 if (old)

> 5679				bpf_prog_put_dev(old);

5680
5681 if (old && !new) {
5682 static_branch_dec(&generic_xdp_needed_key);
5683 } else if (new && !old) {
5684 static_branch_inc(&generic_xdp_needed_key);
5685 dev_disable_lro(dev);
5686 dev_disable_gro_hw(dev);
5687 }
5688 break;
5689
5690 default:
5691 ret = -EINVAL;
5692 break;
5693 }
5694
5695 return ret;
5696 }
5697

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author
#3 lore raw

RFC PATCH 2/2 bpf: Take a uref on BPF_MAP_TYPE_PROG_ARRAY maps during dev attachment

In reply to Daniel Xu

This commit changes the behavior of TC and XDP hooks during attachment such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra uref taken.

The goal behind this change is to try and prevent confusion for the majority of use cases. The current behavior where when the last uref is dropped the prog array map is emptied is quite confusing. Confusing enough for there to be multiple references to it in ebpf-go [0][1].

Completely solving the problem is difficult. As stated in c9da161c6517 ("bpf: fix clearing on persistent program array maps"), it is difficult-to-impossible to walk the full dependency graph b/c it is too dynamic.

However in practice, I've found that all progs in a tailcall chain share the same prog array map. Knowing that, if we take a uref on any used prog array map when the program is attached, we can simplify the majority use case and make it more ergonomic.

I'll be the first to admit this is not a very clean solution. It does not fully solve the problem. Nor does it make overall logic any simpler. But I do think it makes a pretty big usability hole slightly smaller.

[0]: https://github.com/cilium/ebpf/blob/01ebd4c1e2b9f8b3dd4fd2382aa1092c3c9bfc9d/doc.go#L22-L24
[1]: https://github.com/cilium/ebpf/blob/d1a52333f2c0fed085f8d742a5a3c164795d8492/collection.go#L320-L321

---
 kernel/bpf/syscall.c | 40 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 39 insertions(+), 1 deletion(-)
kernel/bpf/syscall.c +39 -1
index d8e5530598f3..6706bb1c8e16 100644
@@ -2164,8 +2164,37 @@ void bpf_prog_put(struct bpf_prog *prog)
 }
 EXPORT_SYMBOL_GPL(bpf_prog_put);
 
+/* Whether this program type, when attached, take a uref on prog array maps */
+static bool bpf_prog_should_pin_uref(enum bpf_prog_type type)
+{
+	switch (type) {
+	case BPF_PROG_TYPE_SCHED_CLS:
+	case BPF_PROG_TYPE_SCHED_ACT:
+	case BPF_PROG_TYPE_XDP:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static void __bpf_prog_add_prog_array_urefs(struct bpf_prog *prog, s64 cnt)
+{
+	struct bpf_map *map;
+	u32 i;
+
+	mutex_lock(&prog->aux->used_maps_mutex);
+	for (i = 0; i < prog->aux->used_map_cnt; i++) {
+		map = prog->aux->used_maps[i];
+		if (IS_FD_PROG_ARRAY(map))
+			atomic64_add(cnt, &map->usercnt);
+	}
+	mutex_unlock(&prog->aux->used_maps_mutex);
+}
+
 void bpf_prog_put_dev(struct bpf_prog *prog)
 {
+	if (bpf_prog_should_pin_uref(prog->type))
+		__bpf_prog_add_prog_array_urefs(prog, -1);
 	bpf_prog_put(prog);
 }
 EXPORT_SYMBOL_GPL(bpf_prog_put_dev);
@@ -2366,7 +2395,16 @@ struct bpf_prog *bpf_prog_get(u32 ufd)
 struct bpf_prog *bpf_prog_get_type_dev(u32 ufd, enum bpf_prog_type type,
 				       bool attach_drv)
 {
-	return __bpf_prog_get(ufd, &type, attach_drv);
+	struct bpf_prog *prog;
+
+	prog = __bpf_prog_get(ufd, &type, attach_drv);
+	if (IS_ERR(prog))
+		goto out;
+
+	if (bpf_prog_should_pin_uref(type))
+		__bpf_prog_add_prog_array_urefs(prog, 1);
+out:
+	return prog;
 }
 EXPORT_SYMBOL_GPL(bpf_prog_get_type_dev);
 
Signature · 1 line
2.41.0
author
#4 lore raw
In reply to Daniel Xu

Hey Daniel,

Quoted 27 lines · Daniel Xu

This patchset changes the behavior of TC and XDP hooks during attachment such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra uref taken.

The goal behind this change is to try and prevent confusion for the majority of use cases. The current behavior where when the last uref is dropped the prog array map is emptied is quite confusing. Confusing enough for there to be multiple references to it in ebpf-go [0][1].

Completely solving the problem is difficult. As stated in c9da161c6517 ("bpf: fix clearing on persistent program array maps"), it is difficult-to-impossible to walk the full dependency graph b/c it is too dynamic.

However in practice, I've found that all progs in a tailcall chain share the same prog array map. Knowing that, if we take a uref on any used prog array map when the program is attached, we can simplify the majority use case and make it more ergonomic.

I'll be the first to admit this is not a very clean solution. It does not fully solve the problem. Nor does it make overall logic any simpler. But I do think it makes a pretty big usability hole slightly smaller.

I've done some basic testing using a repro program [3] I wrote to debug the original issue that eventually led me to this patchset. If we wanna move forward with this approach, I'll resend with selftests.

Quoted 5 lines

[0]:
https://github.com/cilium/ebpf/blob/01ebd4c1e2b9f8b3dd4fd2382aa1092c3c9bfc9d/doc.go#L22-L24
[1]:
https://github.com/cilium/ebpf/blob/d1a52333f2c0fed085f8d742a5a3c164795d8492/collection.go#L320-L321
[2]: https://github.com/danobi/tc_tailcall_repro

I recently remembered about this again. Was suggested I poke you in case you're interested. I looked again and I think this is kinda a neat hack. I probably won't have time to pick this back up either way.

#5 lore raw
Quoted 22 lines · Daniel Xu

Hey Daniel,

Quoted 18 lines · Daniel Xu

This patchset changes the behavior of TC and XDP hooks during attachment such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra uref taken.

The goal behind this change is to try and prevent confusion for the majority of use cases. The current behavior where when the last uref is dropped the prog array map is emptied is quite confusing. Confusing enough for there to be multiple references to it in ebpf-go [0][1].

Completely solving the problem is difficult. As stated in c9da161c6517 ("bpf: fix clearing on persistent program array maps"), it is difficult-to-impossible to walk the full dependency graph b/c it is too dynamic.

However in practice, I've found that all progs in a tailcall chain share the same prog array map. Knowing that, if we take a uref on any used prog array map when the program is attached, we can simplify the majority use case and make it more ergonomic.

Are you proposing to inc map uref when prog is attached?

But that re-adds the circular dependency that uref concept is solving. When prog is inserted into prog array prog refcnt is incremented. So if prog also incremented uref. The user space can exit but prog array and progs will stay there though nothing is using them. I guess I'm missing the idea.

Quoted 19 lines
Quoted 9 lines

I'll be the first to admit this is not a very clean solution. It does not fully solve the problem. Nor does it make overall logic any simpler. But I do think it makes a pretty big usability hole slightly smaller.

I've done some basic testing using a repro program [3] I wrote to debug the original issue that eventually led me to this patchset. If we wanna move forward with this approach, I'll resend with selftests.

Quoted 5 lines

[0]:
https://github.com/cilium/ebpf/blob/01ebd4c1e2b9f8b3dd4fd2382aa1092c3c9bfc9d/doc.go#L22-L24
[1]:
https://github.com/cilium/ebpf/blob/d1a52333f2c0fed085f8d742a5a3c164795d8492/collection.go#L320-L321
[2]: https://github.com/danobi/tc_tailcall_repro

I recently remembered about this again. Was suggested I poke you in case you're interested. I looked again and I think this is kinda a neat hack. I probably won't have time to pick this back up either way.

author
#6 lore raw
Quoted 24 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Xu

Hey Daniel,

Quoted 18 lines · Daniel Xu

This patchset changes the behavior of TC and XDP hooks during attachment such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra uref taken.

The goal behind this change is to try and prevent confusion for the majority of use cases. The current behavior where when the last uref is dropped the prog array map is emptied is quite confusing. Confusing enough for there to be multiple references to it in ebpf-go [0][1].

Completely solving the problem is difficult. As stated in c9da161c6517 ("bpf: fix clearing on persistent program array maps"), it is difficult-to-impossible to walk the full dependency graph b/c it is too dynamic.

However in practice, I've found that all progs in a tailcall chain share the same prog array map. Knowing that, if we take a uref on any used prog array map when the program is attached, we can simplify the majority use case and make it more ergonomic.

Quoted 7 lines

Are you proposing to inc map uref when prog is attached?

But that re-adds the circular dependency that uref concept is solving. When prog is inserted into prog array prog refcnt is incremented. So if prog also incremented uref. The user space can exit but prog array and progs will stay there though nothing is using them. I guess I'm missing the idea.

IIRC the old-style tc/xdp attachment is the one incrementing the uref. Once whatever program there is detached the uref is dropped. So I don't think any circular refs can happen unless a prog can somehow prevent its own detachment.

Could be mis-remembering though.

Thanks,
Daniel

#7 lore raw
Quoted 30 lines · Daniel Xu
Quoted 26 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Xu

Hey Daniel,

Quoted 18 lines · Daniel Xu

This patchset changes the behavior of TC and XDP hooks during attachment such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra uref taken.

The goal behind this change is to try and prevent confusion for the majority of use cases. The current behavior where when the last uref is dropped the prog array map is emptied is quite confusing. Confusing enough for there to be multiple references to it in ebpf-go [0][1].

Completely solving the problem is difficult. As stated in c9da161c6517 ("bpf: fix clearing on persistent program array maps"), it is difficult-to-impossible to walk the full dependency graph b/c it is too dynamic.

However in practice, I've found that all progs in a tailcall chain share the same prog array map. Knowing that, if we take a uref on any used prog array map when the program is attached, we can simplify the majority use case and make it more ergonomic.

Are you proposing to inc map uref when prog is attached?

Quoted 7 lines
Quoted 5 lines

But that re-adds the circular dependency that uref concept is solving. When prog is inserted into prog array prog refcnt is incremented. So if prog also incremented uref. The user space can exit but prog array and progs will stay there though nothing is using them. I guess I'm missing the idea.

IIRC the old-style tc/xdp attachment is the one incrementing the uref.

uref is incremented when FD is given to user space and file->release() callback decrements uref.

I don't think any of the attach operations mess with uref. At least they shouldn't.

Quoted 9 lines

Once
whatever program there is detached the uref is dropped. So I don't think any circular refs can happen unless a prog can somehow prevent its own detachment.

Could be mis-remembering though.

Thanks,
Daniel

author
#8 lore raw
Quoted 37 lines · Alexei Starovoitov
Quoted 36 lines · Daniel Xu
Quoted 31 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Xu

Hey Daniel,

Quoted 18 lines · Daniel Xu

This patchset changes the behavior of TC and XDP hooks during attachment such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra uref taken.

The goal behind this change is to try and prevent confusion for the majority of use cases. The current behavior where when the last uref is dropped the prog array map is emptied is quite confusing. Confusing enough for there to be multiple references to it in ebpf-go [0][1].

Completely solving the problem is difficult. As stated in c9da161c6517 ("bpf: fix clearing on persistent program array maps"), it is difficult-to-impossible to walk the full dependency graph b/c it is too dynamic.

However in practice, I've found that all progs in a tailcall chain share the same prog array map. Knowing that, if we take a uref on any used prog array map when the program is attached, we can simplify the majority use case and make it more ergonomic.

Are you proposing to inc map uref when prog is attached?

But that re-adds the circular dependency that uref concept is solving. When prog is inserted into prog array prog refcnt is incremented. So if prog also incremented uref. The user space can exit but prog array and progs will stay there though nothing is using them. I guess I'm missing the idea.

Quoted 7 lines
Quoted 1 line

IIRC the old-style tc/xdp attachment is the one incrementing the uref.

uref is incremented when FD is given to user space and file->release() callback decrements uref.

I don't think any of the attach operations mess with uref. At least they shouldn't.

None yet. My patch was adding it. It's fine if it's too much of a hack - was just an idea.