net: devlink: fix UAF in devlink_compat_running_version()
Subsystems (4)
PATCH 1/1 net: devlink: fix UAF in devlink_compat_running_version()
I got a UAF report as following when doing fault injection test:
BUG: KASAN: use-after-free in devlink_compat_running_version+0x5b9/0x6a0 Read of size 8 at addr ffff88810ac591f0 by task systemd-udevd/458 CPU: 2 PID: 458 Comm: systemd-udevd Not tainted 6.1.0-rc5-00155-ga9d2b54dd4e7-dirty #1359 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014 Call Trace: <TASK> kasan_report+0x90/0x190 devlink_compat_running_version+0x5b9/0x6a0 dev_ethtool+0x2ca/0x340 dev_ioctl+0x16c/0xff0 sock_do_ioctl+0x1ae/0x220 Allocated by task 456: kasan_save_stack+0x1c/0x40 kasan_set_track+0x21/0x30 __kasan_kmalloc+0x7e/0x90 __kmalloc+0x59/0x1b0 devlink_alloc_ns+0xf7/0xa10 nsim_drv_probe+0xa8/0x150 [netdevsim] really_probe+0x1ff/0x660 Freed by task 456: kasan_save_stack+0x1c/0x40 kasan_set_track+0x21/0x30 kasan_save_free_info+0x2a/0x50 __kasan_slab_free+0x102/0x190 __kmem_cache_free+0xca/0x400 nsim_drv_probe.cold.31+0x2af/0xf62 [netdevsim] really_probe+0x1ff/0x660
It happened like this:
processor A processor Bnsim_drv_probe()
devlink_alloc_ns()
nsim_dev_port_add_all()
__nsim_dev_port_add() // add eth1 successful
dev_ethtool()
ethtool_get_drvinfo(eth1)
netdev_to_devlink_get(eth1)
devlink_try_get() // get devlink here
__nsim_dev_port_add() // add eth2 failed, goto error
devlink_free() // it's called in the error path
devlink_compat_running_version() <- causes UAF
devlink_register() // it's in normal path, not called yet
There is two ports to add in nsim_dev_port_add_all(), if first port is added successful, the devlink is visable and can be get by devlink_try_get() on another cpu, but it is not registered yet. And then the second port is failed to added, in the error path, devlink_free() is called, at last it causes UAF.
Add check in devlink_try_get(), if the 'devlink' is not registered it returns NULL to avoid UAF like this case.
--- net/core/devlink.c | 3 +++ 1 file changed, 3 insertions(+)
net/core/devlink.c +3 -0
@@ -250,6 +250,9 @@ void devlink_put(struct devlink *devlink)struct devlink *__must_check devlink_try_get (struct devlink *devlink) { +if (!xa_get_mark (&devlinks, devlink->index,DEVLINK_REGISTERED )) +return NULL ; +if (refcount_inc_not_zero (&devlink->refcount))return devlink;return NULL ;
Signature · 1 line
2.25.1
Quoted 70 lines · Yang Yingliang
I got a UAF report as following when doing fault injection test:
BUG: KASAN: use-after-free in devlink_compat_running_version+0x5b9/0x6a0 Read of size 8 at addr ffff88810ac591f0 by task systemd-udevd/458 CPU: 2 PID: 458 Comm: systemd-udevd Not tainted 6.1.0-rc5-00155-ga9d2b54dd4e7-dirty #1359 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014 Call Trace: <TASK> kasan_report+0x90/0x190 devlink_compat_running_version+0x5b9/0x6a0 dev_ethtool+0x2ca/0x340 dev_ioctl+0x16c/0xff0 sock_do_ioctl+0x1ae/0x220 Allocated by task 456: kasan_save_stack+0x1c/0x40 kasan_set_track+0x21/0x30 __kasan_kmalloc+0x7e/0x90 __kmalloc+0x59/0x1b0 devlink_alloc_ns+0xf7/0xa10 nsim_drv_probe+0xa8/0x150 [netdevsim] really_probe+0x1ff/0x660 Freed by task 456: kasan_save_stack+0x1c/0x40 kasan_set_track+0x21/0x30 kasan_save_free_info+0x2a/0x50 __kasan_slab_free+0x102/0x190 __kmem_cache_free+0xca/0x400 nsim_drv_probe.cold.31+0x2af/0xf62 [netdevsim] really_probe+0x1ff/0x660It happened like this:
processor A processor B
nsim_drv_probe()
devlink_alloc_ns()
nsim_dev_port_add_all()
__nsim_dev_port_add()// add eth1 successful
dev_ethtool()
ethtool_get_drvinfo(eth1)
netdev_to_devlink_get(eth1)
devlink_try_get()// get devlink here
__nsim_dev_port_add()// add eth2 failed, goto error
devlink_free()// it's called in the error path
devlink_compat_running_version()<- causes UAF
devlink_register()// it's in normal path, not called yetThere is two ports to add in
nsim_dev_port_add_all(), if first port is added successful, the devlink is visable and can be get bydevlink_try_get()on another cpu, but it is not registered yet. And then the second port is failed to added, in the error path,devlink_free()is called, at last it causes UAF.Add check in
devlink_try_get(), if the 'devlink' is not registered it returns NULL to avoid UAF like this case.Fixes a62fdbbe9403 ("netdevsim: implement ndo_get_devlink_port")Reported-by Zhengchao Shao <[email protected]>Signed-off-by Yang Yingliang <[email protected]>--- net/core/devlink.c | 3 +++ 1 file changed, 3 insertions(+)net/core/devlink.c +0
-0
net/core/devlink.c void devlink_put(struct devlink *devlink) · line 255
Quoted 6 lines
net/core/devlink.c +3
-0@@ -250,6 +250,9 @@ void devlink_put(struct devlink *devlink)struct devlink *__must_check devlink_try_get (struct devlink *devlink) { +if (!xa_get_mark (&devlinks, devlink->index,DEVLINK_REGISTERED )) +return NULL ; +
Please fix nsim instead of devlink.
Thanks
Quoted 6 lines
net/core/devlink.c +0
-0if (refcount_inc_not_zero (&devlink->refcount))return devlink;return NULL ;Signature · 1 line
2.25.1
Hi,
Quoted 72 lines · Leon Romanovsky
Quoted 71 lines · Yang Yingliang
I got a UAF report as following when doing fault injection test:
BUG: KASAN: use-after-free in devlink_compat_running_version+0x5b9/0x6a0 Read of size 8 at addr ffff88810ac591f0 by task systemd-udevd/458 CPU: 2 PID: 458 Comm: systemd-udevd Not tainted 6.1.0-rc5-00155-ga9d2b54dd4e7-dirty #1359 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014 Call Trace: <TASK> kasan_report+0x90/0x190 devlink_compat_running_version+0x5b9/0x6a0 dev_ethtool+0x2ca/0x340 dev_ioctl+0x16c/0xff0 sock_do_ioctl+0x1ae/0x220 Allocated by task 456: kasan_save_stack+0x1c/0x40 kasan_set_track+0x21/0x30 __kasan_kmalloc+0x7e/0x90 __kmalloc+0x59/0x1b0 devlink_alloc_ns+0xf7/0xa10 nsim_drv_probe+0xa8/0x150 [netdevsim] really_probe+0x1ff/0x660 Freed by task 456: kasan_save_stack+0x1c/0x40 kasan_set_track+0x21/0x30 kasan_save_free_info+0x2a/0x50 __kasan_slab_free+0x102/0x190 __kmem_cache_free+0xca/0x400 nsim_drv_probe.cold.31+0x2af/0xf62 [netdevsim] really_probe+0x1ff/0x660It happened like this:
processor A processor B
nsim_drv_probe()
devlink_alloc_ns()
nsim_dev_port_add_all()
__nsim_dev_port_add()// add eth1 successful
dev_ethtool()
ethtool_get_drvinfo(eth1)
netdev_to_devlink_get(eth1)
devlink_try_get()// get devlink here
__nsim_dev_port_add()// add eth2 failed, goto error
devlink_free()// it's called in the error path
devlink_compat_running_version()<- causes UAF
devlink_register()// it's in normal path, not called yetThere is two ports to add in
nsim_dev_port_add_all(), if first port is added successful, the devlink is visable and can be get bydevlink_try_get()on another cpu, but it is not registered yet. And then the second port is failed to added, in the error path,devlink_free()is called, at last it causes UAF.Add check in
devlink_try_get(), if the 'devlink' is not registered it returns NULL to avoid UAF like this case.Fixes a62fdbbe9403 ("netdevsim: implement ndo_get_devlink_port")Reported-by Zhengchao Shao <[email protected]>Signed-off-by Yang Yingliang <[email protected]>--- net/core/devlink.c | 3 +++ 1 file changed, 3 insertions(+)net/core/devlink.c +0
-0@@ -250,6 +250,9 @@ void devlink_put(struct devlink *devlink)
Quoted 6 lines
net/core/devlink.c void devlink_put(struct devlink *devlink) · line 255
Quoted 5 lines
net/core/devlink.c +3
-0struct devlink *__must_check devlink_try_get (struct devlink *devlink) { +if (!xa_get_mark (&devlinks, devlink->index,DEVLINK_REGISTERED )) +return NULL ; +Please fix nsim instead of devlink.
I think if devlink is not registered, it can not be get and used, but there is no API for driver to check this, can I introduce a new helper name devlink_is_registered() for driver using.
Thanks,
Yang
Quoted 10 lines
Thanks
Quoted 6 lines
net/core/devlink.c +0
-0if (refcount_inc_not_zero (&devlink->refcount))return devlink;return NULL ;Signature · 1 line
2.25.1.
Quoted 80 lines · Yang Yingliang
Hi,
Quoted 66 lines · Leon Romanovsky
Quoted 65 lines · Yang Yingliang
I got a UAF report as following when doing fault injection test:
BUG: KASAN: use-after-free in devlink_compat_running_version+0x5b9/0x6a0 Read of size 8 at addr ffff88810ac591f0 by task systemd-udevd/458 CPU: 2 PID: 458 Comm: systemd-udevd Not tainted 6.1.0-rc5-00155-ga9d2b54dd4e7-dirty #1359 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014 Call Trace: <TASK> kasan_report+0x90/0x190 devlink_compat_running_version+0x5b9/0x6a0 dev_ethtool+0x2ca/0x340 dev_ioctl+0x16c/0xff0 sock_do_ioctl+0x1ae/0x220 Allocated by task 456: kasan_save_stack+0x1c/0x40 kasan_set_track+0x21/0x30 __kasan_kmalloc+0x7e/0x90 __kmalloc+0x59/0x1b0 devlink_alloc_ns+0xf7/0xa10 nsim_drv_probe+0xa8/0x150 [netdevsim] really_probe+0x1ff/0x660 Freed by task 456: kasan_save_stack+0x1c/0x40 kasan_set_track+0x21/0x30 kasan_save_free_info+0x2a/0x50 __kasan_slab_free+0x102/0x190 __kmem_cache_free+0xca/0x400 nsim_drv_probe.cold.31+0x2af/0xf62 [netdevsim] really_probe+0x1ff/0x660It happened like this:
processor A processor B
nsim_drv_probe()
devlink_alloc_ns()
nsim_dev_port_add_all()
__nsim_dev_port_add()// add eth1 successful
dev_ethtool()
ethtool_get_drvinfo(eth1)
netdev_to_devlink_get(eth1)
devlink_try_get()// get devlink here
__nsim_dev_port_add()// add eth2 failed, goto error
devlink_free()// it's called in the error path
devlink_compat_running_version()<- causes UAF
devlink_register()// it's in normal path, not called yetThere is two ports to add in
nsim_dev_port_add_all(), if first port is added successful, the devlink is visable and can be get bydevlink_try_get()on another cpu, but it is not registered yet. And then the second port is failed to added, in the error path,devlink_free()is called, at last it causes UAF.Add check in
devlink_try_get(), if the 'devlink' is not registered it returns NULL to avoid UAF like this case.Fixes a62fdbbe9403 ("netdevsim: implement ndo_get_devlink_port")Reported-by Zhengchao Shao <[email protected]>Signed-off-by Yang Yingliang <[email protected]>--- net/core/devlink.c | 3 +++ 1 file changed, 3 insertions(+)Quoted 11 lines
net/core/devlink.c void devlink_put(struct devlink *devlink) · line 254
Quoted 10 lines
net/core/devlink.c +3
-0@@ -250,6 +250,9 @@ void devlink_put(struct devlink *devlink)struct devlink *__must_check devlink_try_get (struct devlink *devlink) { +if (!xa_get_mark (&devlinks, devlink->index,DEVLINK_REGISTERED )) +return NULL ; +Please fix nsim instead of devlink.
Quoted 3 lines
I think if devlink is not registered, it can not be get and used, but there is no API for driver to check this, can I introduce a new helper name
devlink_is_registered()for driver using.
There is no need in such API as driver calls to devlink_register() and as such it knows when devlink is registered.
This UAF is nsim specific issue. Real devices have single .probe() routine with serialized registration flow. None of them will use devlink_is_registered() call.
Thanks
Quoted 13 lines
Thanks,
YangQuoted 10 lines
Thanks
Quoted 6 lines
net/core/devlink.c +0
-0if (refcount_inc_not_zero (&devlink->refcount))return devlink;return NULL ;Signature · 1 line
2.25.1.
Jakub Kicinski
maintainerQuoted 11 lines · On Tue, 22 Nov 2022 21:04:29 +0200 Leon Romanovsky wrote:
Quoted 4 lines
Quoted 1 line
Please fix nsim instead of devlink.
I think if devlink is not registered, it can not be get and used, but there is no API for driver to check this, can I introduce a new helper name
devlink_is_registered()for driver using.There is no need in such API as driver calls to
devlink_register()and as such it knows when devlink is registered.This UAF is nsim specific issue. Real devices have single .probe() routine with serialized registration flow. None of them will use
devlink_is_registered()call.
Agreed, the fix is to move the register call back. Something along the lines of the untested patch below? Yang Yingliang would you be able to turn that into a real patch?
drivers/net/netdevsim/dev.c +8 -5
@@ -1566,12 +1566,15 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) err =devlink_params_register (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));if (err) - goto err_dl_unregister; + goto err_resource_unregister;nsim_devlink_set_params_init_values (nsim_dev, devlink); +/* here, because params API still expect devlink to be unregistered */ +devl_register (devlink); + err =nsim_dev_dummy_region_init (nsim_dev, devlink);if (err) - goto err_params_unregister; + goto err_dl_unregister; err =nsim_dev_traps_init (devlink);if (err)
@@ -1610,7 +1613,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) nsim_dev->esw_mode =DEVLINK_ESWITCH_MODE_LEGACY ;devlink_set_features (devlink,DEVLINK_F_RELOAD );devl_unlock (devlink); -devlink_register (devlink);return 0 ; err_hwstats_exit:
@@ -1629,10 +1631,11 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)nsim_dev_traps_exit (devlink); err_dummy_region_exit:nsim_dev_dummy_region_exit (nsim_dev); -err_params_unregister: +err_dl_unregister: +devl_unregister (devlink);devlink_params_unregister (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params)); -err_dl_unregister: +err_resource_unregister:devl_resources_unregister (devlink); err_vfc_free:kfree (nsim_dev->vfconfigs);
include/net/devlink.h +8 -0
@@ -1593,6 +1593,14 @@ void devlink_set_features(struct devlink *devlink, u64 features);void devlink_register (struct devlink *devlink);void devlink_unregister (struct devlink *devlink);void devlink_free (struct devlink *devlink); + +void devl_register (struct devlink *devlink) +{ +devl_assert_locked (devlink); +/* doesn't actually take the instance lock */ +devlink_register (devlink); +} +void devlink_port_init (struct devlink *devlink,struct devlink_port *devlink_port);void devlink_port_fini (struct devlink_port *devlink_port);
Quoted 8 lines · Jakub Kicinski
Quoted 7 lines · On Tue, 22 Nov 2022 21:04:29 +0200 Leon Romanovsky wrote:
Quoted 4 lines
Quoted 1 line
Please fix nsim instead of devlink.
I think if devlink is not registered, it can not be get and used, but there is no API for driver to check this, can I introduce a new helper name
devlink_is_registered()for driver using.There is no need in such API as driver calls to
devlink_register()and as such it knows when devlink is registered.
Quoted 6 lines
Quoted 3 lines
This UAF is nsim specific issue. Real devices have single .probe() routine with serialized registration flow. None of them will use
devlink_is_registered()call.Agreed, the fix is to move the register call back. Something along the lines of the untested patch below? Yang Yingliang would you be able to turn that into a real patch?
OK.
Thanks,
Yang
Quoted 65 lines
drivers/net/netdevsim/dev.c +8
-5@@ -1566,12 +1566,15 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) err =devlink_params_register (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));if (err) - goto err_dl_unregister; + goto err_resource_unregister;nsim_devlink_set_params_init_values (nsim_dev, devlink); +/* here, because params API still expect devlink to be unregistered */ +devl_register (devlink); + err =nsim_dev_dummy_region_init (nsim_dev, devlink);if (err) - goto err_params_unregister; + goto err_dl_unregister; err =nsim_dev_traps_init (devlink);if (err)@@ -1610,7 +1613,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) nsim_dev->esw_mode =DEVLINK_ESWITCH_MODE_LEGACY ;devlink_set_features (devlink,DEVLINK_F_RELOAD );devl_unlock (devlink); -devlink_register (devlink);return 0 ; err_hwstats_exit:@@ -1629,10 +1631,11 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)nsim_dev_traps_exit (devlink); err_dummy_region_exit:nsim_dev_dummy_region_exit (nsim_dev); -err_params_unregister: +err_dl_unregister: +devl_unregister (devlink);devlink_params_unregister (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params)); -err_dl_unregister: +err_resource_unregister:devl_resources_unregister (devlink); err_vfc_free:kfree (nsim_dev->vfconfigs);include/net/devlink.h +8
-0@@ -1593,6 +1593,14 @@ void devlink_set_features(struct devlink *devlink, u64 features);void devlink_register (struct devlink *devlink);void devlink_unregister (struct devlink *devlink);void devlink_free (struct devlink *devlink); + +void devl_register (struct devlink *devlink) +{ +devl_assert_locked (devlink); +/* doesn't actually take the instance lock */ +devlink_register (devlink); +} +void devlink_port_init (struct devlink *devlink,struct devlink_port *devlink_port);void devlink_port_fini (struct devlink_port *devlink_port);.
Quoted 20 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 22 Nov 2022 21:04:29 +0200 Leon Romanovsky wrote:
Quoted 4 lines
Quoted 1 line
Please fix nsim instead of devlink.
I think if devlink is not registered, it can not be get and used, but there is no API for driver to check this, can I introduce a new helper name
devlink_is_registered()for driver using.There is no need in such API as driver calls to
devlink_register()and as such it knows when devlink is registered.This UAF is nsim specific issue. Real devices have single .probe() routine with serialized registration flow. None of them will use
devlink_is_registered()call.Agreed, the fix is to move the register call back. Something along the lines of the untested patch below? Yang Yingliang would you be able to turn that into a real patch?
drivers/net/netdevsim/dev.c +0
-0
drivers/net/netdevsim/dev.c int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) · line 1574
Quoted 10 lines
drivers/net/netdevsim/dev.c +4
-1@@ -1566,12 +1566,15 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) err =devlink_params_register (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));if (err) - goto err_dl_unregister; + goto err_resource_unregister;nsim_devlink_set_params_init_values (nsim_dev, devlink); +/* here, because params API still expect devlink to be unregistered */ +devl_register (devlink); +
devlink_set_features() called at last in probe() also needs devlink is not registered.
Quoted 16 lines
drivers/net/netdevsim/dev.c +1
-2err =nsim_dev_dummy_region_init (nsim_dev, devlink);if (err) - goto err_params_unregister; + goto err_dl_unregister; err =nsim_dev_traps_init (devlink);if (err)@@ -1610,7 +1613,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) nsim_dev->esw_mode =DEVLINK_ESWITCH_MODE_LEGACY ;devlink_set_features (devlink,DEVLINK_F_RELOAD );devl_unlock (devlink); -devlink_register (devlink);return 0 ; err_hwstats_exit:
drivers/net/netdevsim/dev.c int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) · line 1635
Quoted 6 lines
drivers/net/netdevsim/dev.c +2
-1@@ -1629,10 +1631,11 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)nsim_dev_traps_exit (devlink); err_dummy_region_exit:nsim_dev_dummy_region_exit (nsim_dev); -err_params_unregister: +err_dl_unregister: +devl_unregister (devlink);
It races with dev_ethtool():dev_ethtool
devlink_try_get()
nsim_drv_probe
devl_lock()
devl_lock()
devlink_unregister()
devlink_put()
wait_for_completion() <- the refcount is got in dev_ethtool, it causes ABBA deadlock
Quoted 16 lines
drivers/net/netdevsim/dev.c +1
-1devlink_params_unregister (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params)); -err_dl_unregister: +err_resource_unregister:devl_resources_unregister (devlink); err_vfc_free:kfree (nsim_dev->vfconfigs);include/net/devlink.h +1
-0@@ -1593,6 +1593,14 @@ void devlink_set_features(struct devlink *devlink, u64 features);void devlink_register (struct devlink *devlink);void devlink_unregister (struct devlink *devlink);void devlink_free (struct devlink *devlink); +
include/net/devlink.h void devlink_set_features(struct devlink *devlink, u64 features); · line 1606
Quoted 10 lines
include/net/devlink.h +7
-0+void devl_register (struct devlink *devlink) +{ +devl_assert_locked (devlink); +/* doesn't actually take the instance lock */ +devlink_register (devlink); +} +void devlink_port_init (struct devlink *devlink,struct devlink_port *devlink_port);void devlink_port_fini (struct devlink_port *devlink_port);
How about the following changes, it's tested ok in my case:
drivers/net/netdevsim/dev.c +0 -0
@@ -1640,6 +1640,8 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
kfree(nsim_dev->vfconfigs);
err_devlink_unlock:
devl_unlock(devlink);
+ devl_put(devlink);
+ devl_wait_for_comp(devlink);
devlink_free(devlink);
dev_set_drvdata(&nsim_bus_dev->dev, NULL);
return err;
include/net/devlink.h +0 -0
@@ -1563,6 +1563,7 @@ static inline struct devlink *devlink_alloc(const struct devlink_ops *ops,
void devlink_set_features(struct devlink *devlink, u64 features); void devlink_register(struct devlink *devlink);
void devlink_unregister(struct devlink *devlink);
+void devl_wait_for_comp(struct devlink *devlink);
void devlink_free(struct devlink *devlink);
void devlink_port_init(struct devlink *devlink,
struct devlink_port *devlink_port);
@@ -1903,4 +1904,6 @@ devlink_compat_switch_id_get(struct net_device *dev,
#endif
+void devl_put(struct devlink *devlink);
+
#endif /* _NET_DEVLINK_H_ */
net/core/devlink.c +0 -0
@@ -248,6 +248,12 @@ void devlink_put(struct devlink *devlink)
call_rcu(&devlink->rcu, __devlink_put_rcu); }
+void devl_put(struct devlink *devlink)
+{
+ devlink_put(devlink);
+}
+EXPORT_SYMBOL_GPL(devl_put);
+
struct devlink *__must_check devlink_try_get(struct devlink *devlink) {
if (refcount_inc_not_zero(&devlink->refcount))
@@ -9788,6 +9794,12 @@ void devlink_unregister(struct devlink *devlink) }
EXPORT_SYMBOL_GPL(devlink_unregister);
+void devl_wait_for_comp(struct devlink *devlink) +{
+ wait_for_completion(&devlink->comp);
+}
+EXPORT_SYMBOL_GPL(devl_wait_for_comp);
+
/**
* devlink_free - Free devlink instance resources *
Quoted 1 line
.
Quoted 55 lines · Yang Yingliang
Quoted 20 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 22 Nov 2022 21:04:29 +0200 Leon Romanovsky wrote:
Quoted 4 lines
Quoted 1 line
Please fix nsim instead of devlink.
I think if devlink is not registered, it can not be get and used, but there is no API for driver to check this, can I introduce a new helper name
devlink_is_registered()for driver using.There is no need in such API as driver calls to
devlink_register()and as such it knows when devlink is registered.This UAF is nsim specific issue. Real devices have single .probe() routine with serialized registration flow. None of them will use
devlink_is_registered()call.Agreed, the fix is to move the register call back. Something along the lines of the untested patch below? Yang Yingliang would you be able to turn that into a real patch?
drivers/net/netdevsim/dev.c +0
-0drivers/net/netdevsim/dev.c int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) · line 1573
Quoted 9 lines
drivers/net/netdevsim/dev.c +4
-1@@ -1566,12 +1566,15 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) err =devlink_params_register (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));if (err) - goto err_dl_unregister; + goto err_resource_unregister;nsim_devlink_set_params_init_values (nsim_dev, devlink); +/* here, because params API still expect devlink to be unregistered */ +devl_register (devlink); +
devlink_set_features()called at last inprobe()also needs devlink is not registered.Quoted 14 lines
drivers/net/netdevsim/dev.c +1
-2err =nsim_dev_dummy_region_init (nsim_dev, devlink);if (err) - goto err_params_unregister; + goto err_dl_unregister; err =nsim_dev_traps_init (devlink);if (err)@@ -1610,7 +1613,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) nsim_dev->esw_mode =DEVLINK_ESWITCH_MODE_LEGACY ;devlink_set_features (devlink,DEVLINK_F_RELOAD );devl_unlock (devlink); -devlink_register (devlink);return 0 ; err_hwstats_exit:drivers/net/netdevsim/dev.c int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) · line 1635
Quoted 6 lines
drivers/net/netdevsim/dev.c +2
-1@@ -1629,10 +1631,11 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)nsim_dev_traps_exit (devlink); err_dummy_region_exit:nsim_dev_dummy_region_exit (nsim_dev); -err_params_unregister: +err_dl_unregister: +devl_unregister (devlink);It races with
dev_ethtool():dev_ethtool
Quoted 8 lines
devlink_try_get()
nsim_drv_probe
devl_lock()
devl_lock()
devlink_unregister()devlink_put()wait_for_completion()<- the refcount is got indev_ethtool, it causes ABBA deadlock
But all these races are nsim specific ones.
Can you please explain why devlink.[c|h] MUST be changed and nsim can't be fixed?
Thanks
Quoted 63 lines · Leon Romanovsky
Quoted 54 lines · Yang Yingliang
Quoted 20 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 22 Nov 2022 21:04:29 +0200 Leon Romanovsky wrote:
Quoted 4 lines
Quoted 1 line
Please fix nsim instead of devlink.
I think if devlink is not registered, it can not be get and used, but there is no API for driver to check this, can I introduce a new helper name
devlink_is_registered()for driver using.There is no need in such API as driver calls to
devlink_register()and as such it knows when devlink is registered.This UAF is nsim specific issue. Real devices have single .probe() routine with serialized registration flow. None of them will use
devlink_is_registered()call.Agreed, the fix is to move the register call back. Something along the lines of the untested patch below? Yang Yingliang would you be able to turn that into a real patch?
drivers/net/netdevsim/dev.c +0
-0drivers/net/netdevsim/dev.c int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) · line 1573
Quoted 9 lines
drivers/net/netdevsim/dev.c +4
-1@@ -1566,12 +1566,15 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) err =devlink_params_register (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));if (err) - goto err_dl_unregister; + goto err_resource_unregister;nsim_devlink_set_params_init_values (nsim_dev, devlink); +/* here, because params API still expect devlink to be unregistered */ +devl_register (devlink); +
devlink_set_features()called at last inprobe()also needs devlink is not registered.Quoted 14 lines
drivers/net/netdevsim/dev.c +1
-2err =nsim_dev_dummy_region_init (nsim_dev, devlink);if (err) - goto err_params_unregister; + goto err_dl_unregister; err =nsim_dev_traps_init (devlink);if (err)@@ -1610,7 +1613,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) nsim_dev->esw_mode =DEVLINK_ESWITCH_MODE_LEGACY ;devlink_set_features (devlink,DEVLINK_F_RELOAD );devl_unlock (devlink); -devlink_register (devlink);return 0 ; err_hwstats_exit:drivers/net/netdevsim/dev.c int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) · line 1635
Quoted 6 lines
drivers/net/netdevsim/dev.c +2
-1@@ -1629,10 +1631,11 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)nsim_dev_traps_exit (devlink); err_dummy_region_exit:nsim_dev_dummy_region_exit (nsim_dev); -err_params_unregister: +err_dl_unregister: +devl_unregister (devlink);It races with
dev_ethtool():dev_ethtoolQuoted 8 lines
devlink_try_get()
nsim_drv_probe
devl_lock()
devl_lock()
devlink_unregister()devlink_put()wait_for_completion()<- the refcount is
got indev_ethtool, it causes ABBA deadlock
Quoted 3 lines
But all these races are nsim specific ones.
Can you please explain why devlink.[c|h] MUST be changed and nsim can't be fixed?
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct nsim_dev, and it can be checked in nsim_get_devlink_port() like this:
drivers/net/netdevsim/netdev.c +0 -0
@@ -242,6 +242,9 @@ static struct devlink_port *nsim_get_devlink_port(struct net_device *dev)
{
struct netdevsim *ns = netdev_priv(dev);
+ if (!ns->nsim_dev->devlink_registered)
+ return NULL;
+
return &ns->nsim_dev_port->devlink_port;
}
Thanks,
Yang
Quoted 3 lines
Thanks
.
Quoted 77 lines · Yang Yingliang
Quoted 63 lines · Leon Romanovsky
Quoted 54 lines · Yang Yingliang
Quoted 20 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 22 Nov 2022 21:04:29 +0200 Leon Romanovsky wrote:
Quoted 4 lines
Quoted 1 line
Please fix nsim instead of devlink.
I think if devlink is not registered, it can not be get and used, but there is no API for driver to check this, can I introduce a new helper name
devlink_is_registered()for driver using.There is no need in such API as driver calls to
devlink_register()and as such it knows when devlink is registered.This UAF is nsim specific issue. Real devices have single .probe() routine with serialized registration flow. None of them will use
devlink_is_registered()call.Agreed, the fix is to move the register call back. Something along the lines of the untested patch below? Yang Yingliang would you be able to turn that into a real patch?
drivers/net/netdevsim/dev.c +0
-0drivers/net/netdevsim/dev.c int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) · line 1573
Quoted 9 lines
drivers/net/netdevsim/dev.c +4
-1@@ -1566,12 +1566,15 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) err =devlink_params_register (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));if (err) - goto err_dl_unregister; + goto err_resource_unregister;nsim_devlink_set_params_init_values (nsim_dev, devlink); +/* here, because params API still expect devlink to be unregistered */ +devl_register (devlink); +
devlink_set_features()called at last inprobe()also needs devlink is not registered.Quoted 14 lines
drivers/net/netdevsim/dev.c +1
-2err =nsim_dev_dummy_region_init (nsim_dev, devlink);if (err) - goto err_params_unregister; + goto err_dl_unregister; err =nsim_dev_traps_init (devlink);if (err)@@ -1610,7 +1613,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) nsim_dev->esw_mode =DEVLINK_ESWITCH_MODE_LEGACY ;devlink_set_features (devlink,DEVLINK_F_RELOAD );devl_unlock (devlink); -devlink_register (devlink);return 0 ; err_hwstats_exit:drivers/net/netdevsim/dev.c int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) · line 1635
Quoted 6 lines
drivers/net/netdevsim/dev.c +2
-1@@ -1629,10 +1631,11 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)nsim_dev_traps_exit (devlink); err_dummy_region_exit:nsim_dev_dummy_region_exit (nsim_dev); -err_params_unregister: +err_dl_unregister: +devl_unregister (devlink);It races with
dev_ethtool():dev_ethtoolQuoted 8 lines
devlink_try_get()
nsim_drv_probe
devl_lock()
devl_lock()
devlink_unregister()devlink_put()wait_for_completion()<- the refcount is
got indev_ethtool, it causes ABBA deadlockQuoted 3 lines
But all these races are nsim specific ones.
Can you please explain why devlink.[c|h] MUST be changed and nsim can't be fixed?I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:
drivers/net/netdevsim/netdev.c static struct devlink_port *nsim_get_devlink_port(struct · line 247
Quoted 6 lines
drivers/net/netdevsim/netdev.c +2
-0@@ -242,6 +242,9 @@ static struct devlink_port *nsim_get_devlink_port(struct net_device *dev) { struct netdevsim *ns = netdev_priv(dev); + if (!ns->nsim_dev->devlink_registered) + return NULL;
Something like that, but you need to make sure that this check is protected by some sort of the locking.
Quoted 9 lines
drivers/net/netdevsim/netdev.c +1
-0+ return &ns->nsim_dev_port->devlink_port; }Thanks,
YangQuoted 3 lines
Thanks
.
Quoted 68 lines · Yang Yingliang
Quoted 63 lines · Leon Romanovsky
Quoted 54 lines · Yang Yingliang
Quoted 20 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 22 Nov 2022 21:04:29 +0200 Leon Romanovsky wrote:
Quoted 4 lines
Quoted 1 line
Please fix nsim instead of devlink.
I think if devlink is not registered, it can not be get and used, but there is no API for driver to check this, can I introduce a new helper name
devlink_is_registered()for driver using.There is no need in such API as driver calls to
devlink_register()and as such it knows when devlink is registered.This UAF is nsim specific issue. Real devices have single .probe() routine with serialized registration flow. None of them will use
devlink_is_registered()call.Agreed, the fix is to move the register call back. Something along the lines of the untested patch below? Yang Yingliang would you be able to turn that into a real patch?
drivers/net/netdevsim/dev.c +0
-0drivers/net/netdevsim/dev.c int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) · line 1573
Quoted 9 lines
drivers/net/netdevsim/dev.c +4
-1@@ -1566,12 +1566,15 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) err =devlink_params_register (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));if (err) - goto err_dl_unregister; + goto err_resource_unregister;nsim_devlink_set_params_init_values (nsim_dev, devlink); +/* here, because params API still expect devlink to be unregistered */ +devl_register (devlink); +
devlink_set_features()called at last inprobe()also needs devlink is not registered.Quoted 14 lines
drivers/net/netdevsim/dev.c +1
-2err =nsim_dev_dummy_region_init (nsim_dev, devlink);if (err) - goto err_params_unregister; + goto err_dl_unregister; err =nsim_dev_traps_init (devlink);if (err)@@ -1610,7 +1613,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) nsim_dev->esw_mode =DEVLINK_ESWITCH_MODE_LEGACY ;devlink_set_features (devlink,DEVLINK_F_RELOAD );devl_unlock (devlink); -devlink_register (devlink);return 0 ; err_hwstats_exit:drivers/net/netdevsim/dev.c int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) · line 1635
Quoted 6 lines
drivers/net/netdevsim/dev.c +2
-1@@ -1629,10 +1631,11 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)nsim_dev_traps_exit (devlink); err_dummy_region_exit:nsim_dev_dummy_region_exit (nsim_dev); -err_params_unregister: +err_dl_unregister: +devl_unregister (devlink);It races with
dev_ethtool():dev_ethtoolQuoted 8 lines
devlink_try_get()
nsim_drv_probe
devl_lock()
devl_lock()
devlink_unregister()devlink_put()wait_for_completion()<- the refcount is
got indev_ethtool, it causes ABBA deadlockQuoted 3 lines
But all these races are nsim specific ones.
Can you please explain why devlink.[c|h] MUST be changed and nsim can't be fixed?
Quoted 6 lines
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:
I read the discussion and it's not clear to me why this is a netdevsim specific problem. The fundamental problem seems to be that it is possible to hold a reference on a devlink instance before it's registered and that devlink_free() will free the instance regardless of its current reference count because it expects devlink_unregister() to block. In this case, the instance was never registered, so devlink_unregister() is not called.
ethtool was able to get a reference on the devlink instance before it was registered because netdevsim registers its netdevs before registering its devlink instance. However, netdevsim is not the only one doing this: funeth, ice, prestera, mlx4, mlxsw, nfp and potentially others do the same thing.
When you think about it, it's strange that it's even possible for ethtool to reach the driver when the netdev used in the request is long gone, but it's not holding a reference on the netdev (it's holding a reference on the devlink instance instead) and devlink_compat_running_version() is called without RTNL.
Jakub Kicinski
maintainerQuoted 21 lines · On Wed, 23 Nov 2022 21:18:14 +0200 Ido Schimmel wrote:
Quoted 6 lines
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:I read the discussion and it's not clear to me why this is a netdevsim specific problem. The fundamental problem seems to be that it is possible to hold a reference on a devlink instance before it's registered and that
devlink_free()will free the instance regardless of its current reference count because it expectsdevlink_unregister()to block. In this case, the instance was never registered, sodevlink_unregister()is not called.ethtool was able to get a reference on the devlink instance before it was registered because netdevsim registers its netdevs before registering its devlink instance. However, netdevsim is not the only one doing this: funeth, ice, prestera, mlx4, mlxsw, nfp and potentially others do the same thing.
Quoted 5 lines
When you think about it, it's strange that it's even possible for ethtool to reach the driver when the netdev used in the request is long gone, but it's not holding a reference on the netdev (it's holding a reference on the devlink instance instead) and
devlink_compat_running_version()is called without RTNL.
Indeed. We did a bit of a flip-flop with the devlink locking rules and the fact that the instance is reachable before it is registered is a leftover from a previous restructuring :(
Hence my preference to get rid of the ordering at the driver level than to try to patch it up in the code. Dunno if that's convincing.
Quoted 28 lines · Jakub Kicinski
Quoted 21 lines · On Wed, 23 Nov 2022 21:18:14 +0200 Ido Schimmel wrote:
Quoted 6 lines
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:I read the discussion and it's not clear to me why this is a netdevsim specific problem. The fundamental problem seems to be that it is possible to hold a reference on a devlink instance before it's registered and that
devlink_free()will free the instance regardless of its current reference count because it expectsdevlink_unregister()to block. In this case, the instance was never registered, sodevlink_unregister()is not called.ethtool was able to get a reference on the devlink instance before it was registered because netdevsim registers its netdevs before registering its devlink instance. However, netdevsim is not the only one doing this: funeth, ice, prestera, mlx4, mlxsw, nfp and potentially others do the same thing.
Quoted 5 lines
When you think about it, it's strange that it's even possible for ethtool to reach the driver when the netdev used in the request is long gone, but it's not holding a reference on the netdev (it's holding a reference on the devlink instance instead) and
devlink_compat_running_version()is called without RTNL.
Quoted 6 lines
Indeed. We did a bit of a flip-flop with the devlink locking rules and the fact that the instance is reachable before it is registered is a leftover from a previous restructuring :(
Hence my preference to get rid of the ordering at the driver level than to try to patch it up in the code. Dunno if that's convincing.
Before you are doing that, let's remind why we do ordering:
1. It was in happy times when some commands were locked and some not. It caused to very exciting things like controlled uaf, e.t.c.
2. We wanted to provide an access from user space after everything devlink related is initialized.
3. Allow sane debuggability as some drivers called to devlink_register and unregister in random places.
The locking (item 1) was fixed.
Sent from mobile.
Quoted 28 lines · Jakub Kicinski
Quoted 21 lines · On Wed, 23 Nov 2022 21:18:14 +0200 Ido Schimmel wrote:
Quoted 6 lines
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:I read the discussion and it's not clear to me why this is a netdevsim specific problem. The fundamental problem seems to be that it is possible to hold a reference on a devlink instance before it's registered and that
devlink_free()will free the instance regardless of its current reference count because it expectsdevlink_unregister()to block. In this case, the instance was never registered, sodevlink_unregister()is not called.ethtool was able to get a reference on the devlink instance before it was registered because netdevsim registers its netdevs before registering its devlink instance. However, netdevsim is not the only one doing this: funeth, ice, prestera, mlx4, mlxsw, nfp and potentially others do the same thing.
Quoted 5 lines
When you think about it, it's strange that it's even possible for ethtool to reach the driver when the netdev used in the request is long gone, but it's not holding a reference on the netdev (it's holding a reference on the devlink instance instead) and
devlink_compat_running_version()is called without RTNL.
Quoted 6 lines
Indeed. We did a bit of a flip-flop with the devlink locking rules and the fact that the instance is reachable before it is registered is a leftover from a previous restructuring :(
Hence my preference to get rid of the ordering at the driver level than to try to patch it up in the code. Dunno if that's convincing.
I don't have a good solution, but changing all the drivers to register their netdevs after the devlink instance is going to be quite painful and too big for 'net'. I feel like the main motivation for this is the ethtool compat stuff, which is not very convincing IMO. I'm quite happy with the current flow where drivers call devlink_register() at the end of their probe.
Regarding a solution for the current crash, assuming we agree it's not a netdevsim specific problem, I think the current fix [1] is OK. Note that while it fixes the crash, it potentially creates other (less severe) problems. After user space receives RTM_NEWLINK notification it will need to wait for a certain period of time before issuing 'ETHTOOL_GDRVINFO' as otherwise it will not get the firmware version. I guess it's not a big deal for drivers that only register one netdev since they will very quickly follow with devlink_register(), but the race window is larger for drivers that need to register many netdevs, for either physical switch or eswitch ports.
Long term, we either need to find a way to make the ethtool compat stuff work correctly or just get rid of it and have affected drivers implement the relevant ethtool operations instead of relying on devlink.
Jiri Pirko
maintainerQuoted 54 lines · Mon, Nov 28, 2022 at 10:20:53AM CET, [email protected] wrote:
Quoted 28 lines · Jakub Kicinski
Quoted 21 lines · On Wed, 23 Nov 2022 21:18:14 +0200 Ido Schimmel wrote:
Quoted 6 lines
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:I read the discussion and it's not clear to me why this is a netdevsim specific problem. The fundamental problem seems to be that it is possible to hold a reference on a devlink instance before it's registered and that
devlink_free()will free the instance regardless of its current reference count because it expectsdevlink_unregister()to block. In this case, the instance was never registered, sodevlink_unregister()is not called.ethtool was able to get a reference on the devlink instance before it was registered because netdevsim registers its netdevs before registering its devlink instance. However, netdevsim is not the only one doing this: funeth, ice, prestera, mlx4, mlxsw, nfp and potentially others do the same thing.
Quoted 5 lines
When you think about it, it's strange that it's even possible for ethtool to reach the driver when the netdev used in the request is long gone, but it's not holding a reference on the netdev (it's holding a reference on the devlink instance instead) and
devlink_compat_running_version()is called without RTNL.Quoted 6 lines
Indeed. We did a bit of a flip-flop with the devlink locking rules and the fact that the instance is reachable before it is registered is a leftover from a previous restructuring :(
Hence my preference to get rid of the ordering at the driver level than to try to patch it up in the code. Dunno if that's convincing.
I don't have a good solution, but changing all the drivers to register their netdevs after the devlink instance is going to be quite painful and too big for 'net'. I feel like the main motivation for this is the ethtool compat stuff, which is not very convincing IMO. I'm quite happy with the current flow where drivers call
devlink_register()at the end of their probe.Regarding a solution for the current crash, assuming we agree it's not a netdevsim specific problem, I think the current fix [1] is OK. Note that while it fixes the crash, it potentially creates other (less severe) problems. After user space receives
RTM_NEWLINKnotification it will need to wait for a certain period of time before issuing 'ETHTOOL_GDRVINFO' as otherwise it will not get the firmware version. I guess it's not a big deal for drivers that only register one netdev since they will very quickly follow withdevlink_register(), but the race window is larger for drivers that need to register many netdevs, for either physical switch or eswitch ports.
Quoted 5 lines
Long term, we either need to find a way to make the ethtool compat stuff work correctly or just get rid of it and have affected drivers implement the relevant ethtool operations instead of relying on devlink.
I just had a call with Ido. We both think that this might be a good solution for -net to avoid the use after free.
For net-next, we eventually should change driver init flows to register devlink instance first and only after that register devlink_port and related netdevice. The ordering is important for the userspace app. For example the init flow:
<- RTnetlink new netdev event
app sees devlink_port handle in IFLA_DEVLINK_PORT
-> query devlink instance using this handle
<- ENODEV
The instance is not registered yet.
So we need to make sure all devlink_port_register() calls are happening after devlink_register(). This is aligned with the original flow before devlink_register() was moved by Leon. Also it is aligned with devlink reload and devlink port split flows.
Quoted 73 lines · Jiri Pirko
Quoted 54 lines · Mon, Nov 28, 2022 at 10:20:53AM CET, [email protected] wrote:
Quoted 28 lines · Jakub Kicinski
Quoted 21 lines · On Wed, 23 Nov 2022 21:18:14 +0200 Ido Schimmel wrote:
Quoted 6 lines
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:I read the discussion and it's not clear to me why this is a netdevsim specific problem. The fundamental problem seems to be that it is possible to hold a reference on a devlink instance before it's registered and that
devlink_free()will free the instance regardless of its current reference count because it expectsdevlink_unregister()to block. In this case, the instance was never registered, sodevlink_unregister()is not called.ethtool was able to get a reference on the devlink instance before it was registered because netdevsim registers its netdevs before registering its devlink instance. However, netdevsim is not the only one doing this: funeth, ice, prestera, mlx4, mlxsw, nfp and potentially others do the same thing.
Quoted 5 lines
When you think about it, it's strange that it's even possible for ethtool to reach the driver when the netdev used in the request is long gone, but it's not holding a reference on the netdev (it's holding a reference on the devlink instance instead) and
devlink_compat_running_version()is called without RTNL.Quoted 6 lines
Indeed. We did a bit of a flip-flop with the devlink locking rules and the fact that the instance is reachable before it is registered is a leftover from a previous restructuring :(
Hence my preference to get rid of the ordering at the driver level than to try to patch it up in the code. Dunno if that's convincing.
I don't have a good solution, but changing all the drivers to register their netdevs after the devlink instance is going to be quite painful and too big for 'net'. I feel like the main motivation for this is the ethtool compat stuff, which is not very convincing IMO. I'm quite happy with the current flow where drivers call
devlink_register()at the end of their probe.Regarding a solution for the current crash, assuming we agree it's not a netdevsim specific problem, I think the current fix [1] is OK. Note that while it fixes the crash, it potentially creates other (less severe) problems. After user space receives
RTM_NEWLINKnotification it will need to wait for a certain period of time before issuing 'ETHTOOL_GDRVINFO' as otherwise it will not get the firmware version. I guess it's not a big deal for drivers that only register one netdev since they will very quickly follow withdevlink_register(), but the race window is larger for drivers that need to register many netdevs, for either physical switch or eswitch ports.Quoted 5 lines
Long term, we either need to find a way to make the ethtool compat stuff work correctly or just get rid of it and have affected drivers implement the relevant ethtool operations instead of relying on devlink.
I just had a call with Ido. We both think that this might be a good solution for -net to avoid the use after free.
For net-next, we eventually should change driver init flows to register devlink instance first and only after that register
devlink_portand related netdevice. The ordering is important for the userspace app. For example the init flow:
<- RTnetlink new netdev event
app seesdevlink_porthandle inIFLA_DEVLINK_PORT
-> query devlink instance using this handle
<- ENODEV
Quoted 1 line
The instance is not registered yet.
This is supposed to be handled by devlink_notify_register() which sends "delayed" notifications after devlink_register() is called.
Unless something is broken, the scenario above shouldn't happen.
Quoted 6 lines
So we need to make sure all
devlink_port_register()calls are happening afterdevlink_register(). This is aligned with the original flow beforedevlink_register()was moved by Leon. Also it is aligned with devlink reload and devlink port split flows.
I don't know what it means.
Thanks
Jiri Pirko
maintainerQuoted 74 lines · Mon, Nov 28, 2022 at 12:50:15PM CET, [email protected] wrote:
Quoted 73 lines · Jiri Pirko
Quoted 54 lines · Mon, Nov 28, 2022 at 10:20:53AM CET, [email protected] wrote:
Quoted 28 lines · Jakub Kicinski
Quoted 21 lines · On Wed, 23 Nov 2022 21:18:14 +0200 Ido Schimmel wrote:
Quoted 6 lines
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:I read the discussion and it's not clear to me why this is a netdevsim specific problem. The fundamental problem seems to be that it is possible to hold a reference on a devlink instance before it's registered and that
devlink_free()will free the instance regardless of its current reference count because it expectsdevlink_unregister()to block. In this case, the instance was never registered, sodevlink_unregister()is not called.ethtool was able to get a reference on the devlink instance before it was registered because netdevsim registers its netdevs before registering its devlink instance. However, netdevsim is not the only one doing this: funeth, ice, prestera, mlx4, mlxsw, nfp and potentially others do the same thing.
Quoted 5 lines
When you think about it, it's strange that it's even possible for ethtool to reach the driver when the netdev used in the request is long gone, but it's not holding a reference on the netdev (it's holding a reference on the devlink instance instead) and
devlink_compat_running_version()is called without RTNL.Quoted 6 lines
Indeed. We did a bit of a flip-flop with the devlink locking rules and the fact that the instance is reachable before it is registered is a leftover from a previous restructuring :(
Hence my preference to get rid of the ordering at the driver level than to try to patch it up in the code. Dunno if that's convincing.
I don't have a good solution, but changing all the drivers to register their netdevs after the devlink instance is going to be quite painful and too big for 'net'. I feel like the main motivation for this is the ethtool compat stuff, which is not very convincing IMO. I'm quite happy with the current flow where drivers call
devlink_register()at the end of their probe.Regarding a solution for the current crash, assuming we agree it's not a netdevsim specific problem, I think the current fix [1] is OK. Note that while it fixes the crash, it potentially creates other (less severe) problems. After user space receives
RTM_NEWLINKnotification it will need to wait for a certain period of time before issuing 'ETHTOOL_GDRVINFO' as otherwise it will not get the firmware version. I guess it's not a big deal for drivers that only register one netdev since they will very quickly follow withdevlink_register(), but the race window is larger for drivers that need to register many netdevs, for either physical switch or eswitch ports.Quoted 5 lines
Long term, we either need to find a way to make the ethtool compat stuff work correctly or just get rid of it and have affected drivers implement the relevant ethtool operations instead of relying on devlink.
I just had a call with Ido. We both think that this might be a good solution for -net to avoid the use after free.
For net-next, we eventually should change driver init flows to register devlink instance first and only after that register
devlink_portand related netdevice. The ordering is important for the userspace app. For example the init flow:
<- RTnetlink new netdev event
app seesdevlink_porthandle inIFLA_DEVLINK_PORT
-> query devlink instance using this handle
<- ENODEV
Quoted 6 lines
Quoted 1 line
The instance is not registered yet.
This is supposed to be handled by
devlink_notify_register()which sends "delayed" notifications afterdevlink_register()is called.Unless something is broken, the scenario above shouldn't happen.
Nope, RTnetlink message for new netdev is not handled by that. It is sent right away.
Quoted 9 lines
Quoted 7 lines
So we need to make sure all
devlink_port_register()calls are happening afterdevlink_register(). This is aligned with the original flow beforedevlink_register()was moved by Leon. Also it is aligned with devlink reload and devlink port split flows.I don't know what it means.
What I mean is that during port split, devlink instance is registered. During port creation and removal during reload, devlink instance is registered. We should maintain the same ordering during init/fini I believe.
Quoted 2 lines
Thanks
Quoted 77 lines · Jiri Pirko
Quoted 76 lines · Mon, Nov 28, 2022 at 12:50:15PM CET, [email protected] wrote:
Quoted 74 lines · Jiri Pirko
Quoted 54 lines · Mon, Nov 28, 2022 at 10:20:53AM CET, [email protected] wrote:
Quoted 28 lines · Jakub Kicinski
Quoted 21 lines · On Wed, 23 Nov 2022 21:18:14 +0200 Ido Schimmel wrote:
Quoted 6 lines
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:I read the discussion and it's not clear to me why this is a netdevsim specific problem. The fundamental problem seems to be that it is possible to hold a reference on a devlink instance before it's registered and that
devlink_free()will free the instance regardless of its current reference count because it expectsdevlink_unregister()to block. In this case, the instance was never registered, sodevlink_unregister()is not called.ethtool was able to get a reference on the devlink instance before it was registered because netdevsim registers its netdevs before registering its devlink instance. However, netdevsim is not the only one doing this: funeth, ice, prestera, mlx4, mlxsw, nfp and potentially others do the same thing.
Quoted 5 lines
When you think about it, it's strange that it's even possible for ethtool to reach the driver when the netdev used in the request is long gone, but it's not holding a reference on the netdev (it's holding a reference on the devlink instance instead) and
devlink_compat_running_version()is called without RTNL.Quoted 6 lines
Indeed. We did a bit of a flip-flop with the devlink locking rules and the fact that the instance is reachable before it is registered is a leftover from a previous restructuring :(
Hence my preference to get rid of the ordering at the driver level than to try to patch it up in the code. Dunno if that's convincing.
I don't have a good solution, but changing all the drivers to register their netdevs after the devlink instance is going to be quite painful and too big for 'net'. I feel like the main motivation for this is the ethtool compat stuff, which is not very convincing IMO. I'm quite happy with the current flow where drivers call
devlink_register()at the end of their probe.Regarding a solution for the current crash, assuming we agree it's not a netdevsim specific problem, I think the current fix [1] is OK. Note that while it fixes the crash, it potentially creates other (less severe) problems. After user space receives
RTM_NEWLINKnotification it will need to wait for a certain period of time before issuing 'ETHTOOL_GDRVINFO' as otherwise it will not get the firmware version. I guess it's not a big deal for drivers that only register one netdev since they will very quickly follow withdevlink_register(), but the race window is larger for drivers that need to register many netdevs, for either physical switch or eswitch ports.Quoted 5 lines
Long term, we either need to find a way to make the ethtool compat stuff work correctly or just get rid of it and have affected drivers implement the relevant ethtool operations instead of relying on devlink.
I just had a call with Ido. We both think that this might be a good solution for -net to avoid the use after free.
For net-next, we eventually should change driver init flows to register devlink instance first and only after that register
devlink_portand related netdevice. The ordering is important for the userspace app. For example the init flow:
<- RTnetlink new netdev event
app seesdevlink_porthandle inIFLA_DEVLINK_PORT
-> query devlink instance using this handle
<- ENODEVThe instance is not registered yet.
Quoted 7 lines
Quoted 4 lines
This is supposed to be handled by
devlink_notify_register()which sends "delayed" notifications afterdevlink_register()is called.Unless something is broken, the scenario above shouldn't happen.
Nope, RTnetlink message for new netdev is not handled by that. It is sent right away.
And why don't you fix your new commit dca56c3038c3 ("net: expose devlink port over rtnetlink") to do not return devlink instance unless it is registered?
Why is it correct to expose devlink port with not ready to use devlink instance?
Thanks
Jiri Pirko
maintainerQuoted 83 lines · Tue, Nov 29, 2022 at 09:44:48AM CET, [email protected] wrote:
Quoted 82 lines · Jiri Pirko
Quoted 80 lines · Mon, Nov 28, 2022 at 12:50:15PM CET, [email protected] wrote:
Quoted 73 lines · Jiri Pirko
Quoted 54 lines · Mon, Nov 28, 2022 at 10:20:53AM CET, [email protected] wrote:
Quoted 28 lines · Jakub Kicinski
Quoted 21 lines · On Wed, 23 Nov 2022 21:18:14 +0200 Ido Schimmel wrote:
Quoted 6 lines
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:I read the discussion and it's not clear to me why this is a netdevsim specific problem. The fundamental problem seems to be that it is possible to hold a reference on a devlink instance before it's registered and that
devlink_free()will free the instance regardless of its current reference count because it expectsdevlink_unregister()to block. In this case, the instance was never registered, sodevlink_unregister()is not called.ethtool was able to get a reference on the devlink instance before it was registered because netdevsim registers its netdevs before registering its devlink instance. However, netdevsim is not the only one doing this: funeth, ice, prestera, mlx4, mlxsw, nfp and potentially others do the same thing.
Quoted 5 lines
When you think about it, it's strange that it's even possible for ethtool to reach the driver when the netdev used in the request is long gone, but it's not holding a reference on the netdev (it's holding a reference on the devlink instance instead) and
devlink_compat_running_version()is called without RTNL.Quoted 6 lines
Indeed. We did a bit of a flip-flop with the devlink locking rules and the fact that the instance is reachable before it is registered is a leftover from a previous restructuring :(
Hence my preference to get rid of the ordering at the driver level than to try to patch it up in the code. Dunno if that's convincing.
I don't have a good solution, but changing all the drivers to register their netdevs after the devlink instance is going to be quite painful and too big for 'net'. I feel like the main motivation for this is the ethtool compat stuff, which is not very convincing IMO. I'm quite happy with the current flow where drivers call
devlink_register()at the end of their probe.Regarding a solution for the current crash, assuming we agree it's not a netdevsim specific problem, I think the current fix [1] is OK. Note that while it fixes the crash, it potentially creates other (less severe) problems. After user space receives
RTM_NEWLINKnotification it will need to wait for a certain period of time before issuing 'ETHTOOL_GDRVINFO' as otherwise it will not get the firmware version. I guess it's not a big deal for drivers that only register one netdev since they will very quickly follow withdevlink_register(), but the race window is larger for drivers that need to register many netdevs, for either physical switch or eswitch ports.Quoted 5 lines
Long term, we either need to find a way to make the ethtool compat stuff work correctly or just get rid of it and have affected drivers implement the relevant ethtool operations instead of relying on devlink.
I just had a call with Ido. We both think that this might be a good solution for -net to avoid the use after free.
For net-next, we eventually should change driver init flows to register devlink instance first and only after that register
devlink_portand related netdevice. The ordering is important for the userspace app. For example the init flow:
<- RTnetlink new netdev event
app seesdevlink_porthandle inIFLA_DEVLINK_PORT
-> query devlink instance using this handle
<- ENODEVQuoted 1 line
The instance is not registered yet.
This is supposed to be handled by
devlink_notify_register()which sends "delayed" notifications afterdevlink_register()is called.Unless something is broken, the scenario above shouldn't happen.
Quoted 8 lines
Quoted 2 lines
Nope, RTnetlink message for new netdev is not handled by that. It is sent right away.
And why don't you fix your new commit
dca56c3038c3("net: expose devlink port over rtnetlink") to do not return devlink instance unless it is registered?Why is it correct to expose devlink port with not ready to use devlink instance?
It is not, but:
Devlink port which is "parent" of the netdev is registered. The netdev is created with devlink_port registered and that it guaranteed to not change during netdev lifetime. Therefore, it would be weird to have 2 RTnetlink events:
1. event of netdev being created without devlink port
2. event of netdev with devlink port
If that is what you suggest.
I'm working on a patchset that is making sure that the flow is always
1) devlink_register & netlink event
2) devlink_port_register & netlink event
3) netdev_register & netlink event
Always the same. That means during init, during reload, during port split.
Quoted 93 lines · Jiri Pirko
Quoted 83 lines · Tue, Nov 29, 2022 at 09:44:48AM CET, [email protected] wrote:
Quoted 82 lines · Jiri Pirko
Quoted 80 lines · Mon, Nov 28, 2022 at 12:50:15PM CET, [email protected] wrote:
Quoted 73 lines · Jiri Pirko
Quoted 54 lines · Mon, Nov 28, 2022 at 10:20:53AM CET, [email protected] wrote:
Quoted 28 lines · Jakub Kicinski
Quoted 21 lines · On Wed, 23 Nov 2022 21:18:14 +0200 Ido Schimmel wrote:
Quoted 6 lines
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:I read the discussion and it's not clear to me why this is a netdevsim specific problem. The fundamental problem seems to be that it is possible to hold a reference on a devlink instance before it's registered and that
devlink_free()will free the instance regardless of its current reference count because it expectsdevlink_unregister()to block. In this case, the instance was never registered, sodevlink_unregister()is not called.ethtool was able to get a reference on the devlink instance before it was registered because netdevsim registers its netdevs before registering its devlink instance. However, netdevsim is not the only one doing this: funeth, ice, prestera, mlx4, mlxsw, nfp and potentially others do the same thing.
Quoted 5 lines
When you think about it, it's strange that it's even possible for ethtool to reach the driver when the netdev used in the request is long gone, but it's not holding a reference on the netdev (it's holding a reference on the devlink instance instead) and
devlink_compat_running_version()is called without RTNL.Quoted 6 lines
Indeed. We did a bit of a flip-flop with the devlink locking rules and the fact that the instance is reachable before it is registered is a leftover from a previous restructuring :(
Hence my preference to get rid of the ordering at the driver level than to try to patch it up in the code. Dunno if that's convincing.
I don't have a good solution, but changing all the drivers to register their netdevs after the devlink instance is going to be quite painful and too big for 'net'. I feel like the main motivation for this is the ethtool compat stuff, which is not very convincing IMO. I'm quite happy with the current flow where drivers call
devlink_register()at the end of their probe.Regarding a solution for the current crash, assuming we agree it's not a netdevsim specific problem, I think the current fix [1] is OK. Note that while it fixes the crash, it potentially creates other (less severe) problems. After user space receives
RTM_NEWLINKnotification it will need to wait for a certain period of time before issuing 'ETHTOOL_GDRVINFO' as otherwise it will not get the firmware version. I guess it's not a big deal for drivers that only register one netdev since they will very quickly follow withdevlink_register(), but the race window is larger for drivers that need to register many netdevs, for either physical switch or eswitch ports.Quoted 5 lines
Long term, we either need to find a way to make the ethtool compat stuff work correctly or just get rid of it and have affected drivers implement the relevant ethtool operations instead of relying on devlink.
I just had a call with Ido. We both think that this might be a good solution for -net to avoid the use after free.
For net-next, we eventually should change driver init flows to register devlink instance first and only after that register
devlink_portand related netdevice. The ordering is important for the userspace app. For example the init flow:
<- RTnetlink new netdev event
app seesdevlink_porthandle inIFLA_DEVLINK_PORT
-> query devlink instance using this handle
<- ENODEVQuoted 1 line
The instance is not registered yet.
This is supposed to be handled by
devlink_notify_register()which sends "delayed" notifications afterdevlink_register()is called.Unless something is broken, the scenario above shouldn't happen.
Quoted 8 lines
Quoted 2 lines
Nope, RTnetlink message for new netdev is not handled by that. It is sent right away.
And why don't you fix your new commit
dca56c3038c3("net: expose devlink port over rtnetlink") to do not return devlink instance unless it is registered?Why is it correct to expose devlink port with not ready to use devlink instance?
Quoted 8 lines
It is not, but:
Devlink port which is "parent" of the netdev is registered. The netdev is created withdevlink_portregistered and that it guaranteed to not change during netdev lifetime. Therefore, it would be weird to have 2 RTnetlink events:
1. event of netdev being created without devlink port
2. event of netdev with devlink port
If that is what you suggest.
Yes, something like that in similar way to IFLA_EVENT_NOTIFY_PEERS.
Quoted 8 lines
I'm working on a patchset that is making sure that the flow is always
1)devlink_register& netlink event
2)devlink_port_register& netlink event
3)netdev_register& netlink eventAlways the same. That means during init, during reload, during port split.
The thing that worries me is unclear lifetime of these devlink objects. devlink_port is aligned with netdev lifetime, devlink is with device one.
It is very strange to me that we have netdev ready without device underneath ready.
Can you please document the lifetime models and how all these objects interact?
Thanks
Jiri Pirko
maintainerQuoted 111 lines · Tue, Nov 29, 2022 at 12:20:01PM CET, [email protected] wrote:
Quoted 93 lines · Jiri Pirko
Quoted 83 lines · Tue, Nov 29, 2022 at 09:44:48AM CET, [email protected] wrote:
Quoted 82 lines · Jiri Pirko
Quoted 80 lines · Mon, Nov 28, 2022 at 12:50:15PM CET, [email protected] wrote:
Quoted 73 lines · Jiri Pirko
Quoted 54 lines · Mon, Nov 28, 2022 at 10:20:53AM CET, [email protected] wrote:
Quoted 28 lines · Jakub Kicinski
Quoted 21 lines · On Wed, 23 Nov 2022 21:18:14 +0200 Ido Schimmel wrote:
Quoted 6 lines
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:I read the discussion and it's not clear to me why this is a netdevsim specific problem. The fundamental problem seems to be that it is possible to hold a reference on a devlink instance before it's registered and that
devlink_free()will free the instance regardless of its current reference count because it expectsdevlink_unregister()to block. In this case, the instance was never registered, sodevlink_unregister()is not called.ethtool was able to get a reference on the devlink instance before it was registered because netdevsim registers its netdevs before registering its devlink instance. However, netdevsim is not the only one doing this: funeth, ice, prestera, mlx4, mlxsw, nfp and potentially others do the same thing.
Quoted 5 lines
When you think about it, it's strange that it's even possible for ethtool to reach the driver when the netdev used in the request is long gone, but it's not holding a reference on the netdev (it's holding a reference on the devlink instance instead) and
devlink_compat_running_version()is called without RTNL.Quoted 6 lines
Indeed. We did a bit of a flip-flop with the devlink locking rules and the fact that the instance is reachable before it is registered is a leftover from a previous restructuring :(
Hence my preference to get rid of the ordering at the driver level than to try to patch it up in the code. Dunno if that's convincing.
I don't have a good solution, but changing all the drivers to register their netdevs after the devlink instance is going to be quite painful and too big for 'net'. I feel like the main motivation for this is the ethtool compat stuff, which is not very convincing IMO. I'm quite happy with the current flow where drivers call
devlink_register()at the end of their probe.Regarding a solution for the current crash, assuming we agree it's not a netdevsim specific problem, I think the current fix [1] is OK. Note that while it fixes the crash, it potentially creates other (less severe) problems. After user space receives
RTM_NEWLINKnotification it will need to wait for a certain period of time before issuing 'ETHTOOL_GDRVINFO' as otherwise it will not get the firmware version. I guess it's not a big deal for drivers that only register one netdev since they will very quickly follow withdevlink_register(), but the race window is larger for drivers that need to register many netdevs, for either physical switch or eswitch ports.Quoted 5 lines
Long term, we either need to find a way to make the ethtool compat stuff work correctly or just get rid of it and have affected drivers implement the relevant ethtool operations instead of relying on devlink.
I just had a call with Ido. We both think that this might be a good solution for -net to avoid the use after free.
For net-next, we eventually should change driver init flows to register devlink instance first and only after that register
devlink_portand related netdevice. The ordering is important for the userspace app. For example the init flow:
<- RTnetlink new netdev event
app seesdevlink_porthandle inIFLA_DEVLINK_PORT
-> query devlink instance using this handle
<- ENODEVQuoted 1 line
The instance is not registered yet.
This is supposed to be handled by
devlink_notify_register()which sends "delayed" notifications afterdevlink_register()is called.Unless something is broken, the scenario above shouldn't happen.
Quoted 8 lines
Quoted 2 lines
Nope, RTnetlink message for new netdev is not handled by that. It is sent right away.
And why don't you fix your new commit
dca56c3038c3("net: expose devlink port over rtnetlink") to do not return devlink instance unless it is registered?Why is it correct to expose devlink port with not ready to use devlink instance?
Quoted 8 lines
It is not, but:
Devlink port which is "parent" of the netdev is registered. The netdev is created withdevlink_portregistered and that it guaranteed to not change during netdev lifetime. Therefore, it would be weird to have 2 RTnetlink events:
1. event of netdev being created without devlink port
2. event of netdev with devlink port
If that is what you suggest.Yes, something like that in similar way to
IFLA_EVENT_NOTIFY_PEERS.Quoted 6 lines
I'm working on a patchset that is making sure that the flow is always
1)devlink_register& netlink event
2)devlink_port_register& netlink event
3)netdev_register& netlink event
Quoted 5 lines
Quoted 2 lines
Always the same. That means during init, during reload, during port split.
The thing that worries me is unclear lifetime of these devlink objects.
devlink_portis aligned with netdev lifetime, devlink is with device one.
Correct. Therefore it makes sense to create device object and propagate the existence to the user, then create port object with netdevice and propagate those.
Quoted 6 lines
It is very strange to me that we have netdev ready without device underneath ready.
Can you please document the lifetime models and how all these objects interact?
Yep. On top of that, I will put an assert into devlink_port_register to check devlink instance is registered.
Quoted 2 lines
Thanks
Jakub Kicinski
maintainerQuoted 18 lines · On Mon, 28 Nov 2022 10:58:58 +0100 Jiri Pirko wrote:
Quoted 5 lines
Long term, we either need to find a way to make the ethtool compat stuff work correctly or just get rid of it and have affected drivers implement the relevant ethtool operations instead of relying on devlink.
I just had a call with Ido. We both think that this might be a good solution for -net to avoid the use after free.
For net-next, we eventually should change driver init flows to register devlink instance first and only after that register
devlink_portand related netdevice. The ordering is important for the userspace app. For example the init flow:
<- RTnetlink new netdev event
app seesdevlink_porthandle inIFLA_DEVLINK_PORT
-> query devlink instance using this handle
<- ENODEV
Quoted 6 lines
The instance is not registered yet.
So we need to make sure all
devlink_port_register()calls are happening afterdevlink_register(). This is aligned with the original flow beforedevlink_register()was moved by Leon. Also it is aligned with devlink reload and devlink port split flows.
Cool. Do you also agree with doing proper refcounting for the devlink instance struct and the liveness check after locking the instance?
Jiri Pirko
maintainerQuoted 21 lines · Mon, Nov 28, 2022 at 07:20:43PM CET, [email protected] wrote:
Quoted 20 lines · On Mon, 28 Nov 2022 10:58:58 +0100 Jiri Pirko wrote:
Quoted 5 lines
Long term, we either need to find a way to make the ethtool compat stuff work correctly or just get rid of it and have affected drivers implement the relevant ethtool operations instead of relying on devlink.
I just had a call with Ido. We both think that this might be a good solution for -net to avoid the use after free.
For net-next, we eventually should change driver init flows to register devlink instance first and only after that register
devlink_portand related netdevice. The ordering is important for the userspace app. For example the init flow:
<- RTnetlink new netdev event
app seesdevlink_porthandle inIFLA_DEVLINK_PORT
-> query devlink instance using this handle
<- ENODEVThe instance is not registered yet.
Quoted 7 lines
Quoted 4 lines
So we need to make sure all
devlink_port_register()calls are happening afterdevlink_register(). This is aligned with the original flow beforedevlink_register()was moved by Leon. Also it is aligned with devlink reload and devlink port split flows.Cool. Do you also agree with doing proper refcounting for the devlink instance struct and the liveness check after locking the instance?
Could you elaborate a bit more? I missed that in the thread and can't find it. Why do we need it?
Jakub Kicinski
maintainerQuoted 5 lines · On Tue, 29 Nov 2022 09:31:40 +0100 Jiri Pirko wrote:
Quoted 2 lines
Cool. Do you also agree with doing proper refcounting for the devlink instance struct and the liveness check after locking the instance?
Could you elaborate a bit more? I missed that in the thread and can't find it. Why do we need it?
Look at the __devlink_free() and changes
to devlink_compat_flash_update() here:
https://lore.kernel.org/netdev/[email protected]/
The model I had in mind (a year ago when it all started) was that the driver takes the devlink instance lock around its entire init path, including the registration of the instance. This way the devlink instance is never visible "half initialized". I mean - it's "visible" as in you can see a notification over netlink before init is done but you can't access it until the init in the driver is completed and it releases the instance lock.
For that to work and to avoid ordering issues with netdev we need to allow unregistering a devlink instance before all references are gone.
So we atomically look up and take a reference on a devlink instance. Then take its lock. Then under the instance lock we check if it's still registered.
For devlink core that's not a problem it's all hidden in the helpers.
Quoted 12 lines · Jakub Kicinski
Quoted 5 lines · On Tue, 29 Nov 2022 09:31:40 +0100 Jiri Pirko wrote:
Quoted 2 lines
Cool. Do you also agree with doing proper refcounting for the devlink instance struct and the liveness check after locking the instance?
Could you elaborate a bit more? I missed that in the thread and can't find it. Why do we need it?
Look at the
__devlink_free()and changes
todevlink_compat_flash_update()here:
Quoted 7 lines
The model I had in mind (a year ago when it all started) was that the driver takes the devlink instance lock around its entire init path, including the registration of the instance. This way the devlink instance is never visible "half initialized". I mean - it's "visible" as in you can see a notification over netlink before init is done but you can't access it until the init in the driver is completed and it releases the instance lock.
In parallel thread, Jiri wanted to avoid this situation of netlink notifications for not-visible yet object. He gave as an example devlink_port which is advertised without devlink being ready.
Thanks
Jiri Pirko
maintainerQuoted 21 lines · Wed, Nov 30, 2022 at 09:54:06AM CET, [email protected] wrote:
Quoted 12 lines · Jakub Kicinski
Quoted 5 lines · On Tue, 29 Nov 2022 09:31:40 +0100 Jiri Pirko wrote:
Quoted 2 lines
Cool. Do you also agree with doing proper refcounting for the devlink instance struct and the liveness check after locking the instance?
Could you elaborate a bit more? I missed that in the thread and can't find it. Why do we need it?
Look at the
__devlink_free()and changes
todevlink_compat_flash_update()here:Quoted 7 lines
The model I had in mind (a year ago when it all started) was that the driver takes the devlink instance lock around its entire init path, including the registration of the instance. This way the devlink instance is never visible "half initialized". I mean - it's "visible" as in you can see a notification over netlink before init is done but you can't access it until the init in the driver is completed and it releases the instance lock.
Quoted 3 lines
In parallel thread, Jiri wanted to avoid this situation of netlink notifications for not-visible yet object. He gave as an example
devlink_portwhich is advertised without devlink being ready.
To be honest, I'm lost tracking what you point at. I never suggested to send notification of devlink_port before devlink notification is sent previously. Perhaps you misread that.
Quoted 2 lines
Thanks
Jakub Kicinski
maintainerQuoted 3 lines · On Wed, 30 Nov 2022 10:54:06 +0200 Leon Romanovsky wrote:
In parallel thread, Jiri wanted to avoid this situation of netlink notifications for not-visible yet object. He gave as an example
devlink_portwhich is advertised without devlink being ready.
To be clear the exact ordering matters much less in my scheme, because the user space caller will wait for the instance lock before checking if the instance is registered. So an early user request will just sleep on the instance lock until init is done.
Jiri Pirko
maintainerQuoted 11 lines · Wed, Nov 30, 2022 at 03:18:26AM CET, [email protected] wrote:
Quoted 5 lines · On Tue, 29 Nov 2022 09:31:40 +0100 Jiri Pirko wrote:
Quoted 2 lines
Cool. Do you also agree with doing proper refcounting for the devlink instance struct and the liveness check after locking the instance?
Could you elaborate a bit more? I missed that in the thread and can't find it. Why do we need it?
Look at the
__devlink_free()and changes
todevlink_compat_flash_update()here:
**)
I see. With the change I suggest, meaning doing devlink_port_register/unregister() and netdev_register/unregister only for registered devlink instance, you don't need this at all. When you hit this compat callback, the netdevice is there and therefore devlink instance is registered for sure.
Quoted 8 lines
The model I had in mind (a year ago when it all started) was that the driver takes the devlink instance lock around its entire init path, including the registration of the instance. This way the devlink instance is never visible "half initialized". I mean - it's "visible" as in you can see a notification over netlink before init is done but you can't access it until the init in the driver is completed and it releases the instance lock.
What is "half-initialized"? Take devlink reload flow for instance. There are multiple things removed/readded, like devlink_port and related netdevice. No problem there.
I think that we really need to go a step back and put the devlink_register at the point of init flow where all things that are "static" during register lifetime are initialized, the rest would be initialized later on. This would make things aligned with devlink_reload() and would make possible to share init/fini/reload code in drivers.
Quoted 3 lines
For that to work and to avoid ordering issues with netdev we need to allow unregistering a devlink instance before all references are gone.
References of what? devlink instance, put by devlink_put()?
Quoted 4 lines
So we atomically look up and take a reference on a devlink instance. Then take its lock. Then under the instance lock we check if it's still registered.
As mentioned above (**), I don't think this is needed.
Quoted 2 lines
For devlink core that's not a problem it's all hidden in the helpers.
Jakub Kicinski
maintainerQuoted 11 lines · On Wed, 30 Nov 2022 12:42:39 +0100 Jiri Pirko wrote:
Quoted 4 lines
Look at the
__devlink_free()and changes
todevlink_compat_flash_update()here:**)
I see. With the change I suggest, meaning doing devlink_port_register/unregister() and netdev_register/unregister only for registered devlink instance, you don't need this at all. When you hit this compat callback, the netdevice is there and therefore devlink instance is registered for sure.
If you move devlink registration up it has to be under the instance lock, otherwise we're back to reload problems. That implies unregister should be under the lock too. But then we can't wait for refs in unregister. Perhaps I don't understand the suggestion.
Quoted 11 lines
Quoted 7 lines
The model I had in mind (a year ago when it all started) was that the driver takes the devlink instance lock around its entire init path, including the registration of the instance. This way the devlink instance is never visible "half initialized". I mean - it's "visible" as in you can see a notification over netlink before init is done but you can't access it until the init in the driver is completed and it releases the instance lock.
What is "half-initialized"? Take devlink reload flow for instance. There are multiple things removed/readded, like
devlink_portand related netdevice. No problem there.
Yes, but reload is under the instance lock, so nothing can mess with a device in a transitional state.
Quoted 6 lines
I think that we really need to go a step back and put the
devlink_registerat the point of init flow where all things that are "static" during register lifetime are initialized, the rest would be initialized later on. This would make things aligned withdevlink_reload()and would make possible to shareinit/fini/reloadcode in drivers.
Yes, I agree that the move should be done but I don't think its sufficient.
Quoted 4 lines
Quoted 2 lines
For that to work and to avoid ordering issues with netdev we need to allow unregistering a devlink instance before all references are gone.
References of what? devlink instance, put by
devlink_put()?
Yes.
Quoted 5 lines
Quoted 3 lines
So we atomically look up and take a reference on a devlink instance. Then take its lock. Then under the instance lock we check if it's still registered.
As mentioned above (**), I don't think this is needed.
But it is, please just let me do it and make the bugs stop 😭
Jiri Pirko
maintainerQuoted 13 lines · Wed, Nov 30, 2022 at 05:46:59PM CET, [email protected] wrote:
Quoted 11 lines · On Wed, 30 Nov 2022 12:42:39 +0100 Jiri Pirko wrote:
Quoted 4 lines
Look at the
__devlink_free()and changes
todevlink_compat_flash_update()here:**)
I see. With the change I suggest, meaning doing devlink_port_register/unregister() and netdev_register/unregister only for registered devlink instance, you don't need this at all. When you hit this compat callback, the netdevice is there and therefore devlink instance is registered for sure.
Quoted 4 lines
If you move devlink registration up it has to be under the instance lock, otherwise we're back to reload problems. That implies unregister should be under the lock too. But then we can't wait for refs in unregister. Perhaps I don't understand the suggestion.
I unlock for register and for the rest of the init I lock again.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
The model I had in mind (a year ago when it all started) was that the driver takes the devlink instance lock around its entire init path, including the registration of the instance. This way the devlink instance is never visible "half initialized". I mean - it's "visible" as in you can see a notification over netlink before init is done but you can't access it until the init in the driver is completed and it releases the instance lock.
Quoted 6 lines
Quoted 3 lines
What is "half-initialized"? Take devlink reload flow for instance. There are multiple things removed/readded, like
devlink_portand related netdevice. No problem there.Yes, but reload is under the instance lock, so nothing can mess with a device in a transitional state.
Sure, that is what I want to do too. To be under instance lock.
Quoted 18 lines
Quoted 7 lines
I think that we really need to go a step back and put the
devlink_registerat the point of init flow where all things that are "static" during register lifetime are initialized, the rest would be initialized later on. This would make things aligned withdevlink_reload()and would make possible to shareinit/fini/reloadcode in drivers.Yes, I agree that the move should be done but I don't think its sufficient.
Quoted 4 lines
Quoted 2 lines
For that to work and to avoid ordering issues with netdev we need to allow unregistering a devlink instance before all references are gone.
References of what? devlink instance, put by
devlink_put()?Yes.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
So we atomically look up and take a reference on a devlink instance. Then take its lock. Then under the instance lock we check if it's still registered.
As mentioned above (**), I don't think this is needed.
But it is, please just let me do it and make the bugs stop 😭
Why exactly is it needed? I don't see it, pardon my ignorance :)
Let me send the RFC of the change tomorrow, you'll see what I mean.
Jakub Kicinski
maintainerQuoted 9 lines · On Wed, 30 Nov 2022 18:00:05 +0100 Jiri Pirko wrote:
Quoted 8 lines · Wed, Nov 30, 2022 at 05:46:59PM CET, [email protected] wrote:
Quoted 6 lines · On Wed, 30 Nov 2022 12:42:39 +0100 Jiri Pirko wrote:
**)
I see. With the change I suggest, meaning doing devlink_port_register/unregister() and netdev_register/unregister only for registered devlink instance, you don't need this at all. When you hit this compat callback, the netdevice is there and therefore devlink instance is registered for sure.
Quoted 6 lines
Quoted 4 lines
If you move devlink registration up it has to be under the instance lock, otherwise we're back to reload problems. That implies unregister should be under the lock too. But then we can't wait for refs in unregister. Perhaps I don't understand the suggestion.
I unlock for register and for the rest of the init I lock again.
The moment you register that instance callbacks can start coming. Leon move the register call last for a good reason - all drivers we looked at had bugs in handling init.
We can come up with fixes in the drivers, flags, devlink_set_features() and all that sort of garbage until the day we die but let's not. The driver facing API should be simple - hold the lock around entire init.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
What is "half-initialized"? Take devlink reload flow for instance. There are multiple things removed/readded, like
devlink_portand related netdevice. No problem there.Yes, but reload is under the instance lock, so nothing can mess with a device in a transitional state.
Sure, that is what I want to do too. To be under instance lock.
I'm confused, you just said "I unlock for register".
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
As mentioned above (**), I don't think this is needed.
But it is, please just let me do it and make the bugs stop 😭
Why exactly is it needed? I don't see it, pardon my ignorance :)
Let me send the RFC of the change tomorrow, you'll see what I mean.
The way I see it Leon had a stab at it, you did too, now it's my turn..
Quoted 15 lines · Jakub Kicinski
Quoted 14 lines · On Wed, 30 Nov 2022 18:00:05 +0100 Jiri Pirko wrote:
Quoted 12 lines · Wed, Nov 30, 2022 at 05:46:59PM CET, [email protected] wrote:
Quoted 6 lines · On Wed, 30 Nov 2022 12:42:39 +0100 Jiri Pirko wrote:
**)
I see. With the change I suggest, meaning doing devlink_port_register/unregister() and netdev_register/unregister only for registered devlink instance, you don't need this at all. When you hit this compat callback, the netdevice is there and therefore devlink instance is registered for sure.If you move devlink registration up it has to be under the instance lock, otherwise we're back to reload problems. That implies unregister should be under the lock too. But then we can't wait for refs in unregister. Perhaps I don't understand the suggestion.
Quoted 5 lines
Quoted 1 line
I unlock for register and for the rest of the init I lock again.
The moment you register that instance callbacks can start coming. Leon move the register call last for a good reason - all drivers we looked at had bugs in handling init.
Plus we had very cozy lock->unlock->lock sequences during devlink command execution, which caused to races between devlink calls and driver initialization.
So I'm also interested to see what Jiri meant by saying "I unlock for register and for the rest of the init I lock again".
Thanks
Jiri Pirko
maintainerQuoted 18 lines · Wed, Nov 30, 2022 at 08:20:34PM CET, [email protected] wrote:
Quoted 17 lines · Jakub Kicinski
Quoted 15 lines · On Wed, 30 Nov 2022 18:00:05 +0100 Jiri Pirko wrote:
Quoted 8 lines · Wed, Nov 30, 2022 at 05:46:59PM CET, [email protected] wrote:
Quoted 6 lines · On Wed, 30 Nov 2022 12:42:39 +0100 Jiri Pirko wrote:
**)
I see. With the change I suggest, meaning doing devlink_port_register/unregister() and netdev_register/unregister only for registered devlink instance, you don't need this at all. When you hit this compat callback, the netdevice is there and therefore devlink instance is registered for sure.Quoted 4 lines
If you move devlink registration up it has to be under the instance lock, otherwise we're back to reload problems. That implies unregister should be under the lock too. But then we can't wait for refs in unregister. Perhaps I don't understand the suggestion.
I unlock for register and for the rest of the init I lock again.
Quoted 7 lines
Quoted 3 lines
The moment you register that instance callbacks can start coming. Leon move the register call last for a good reason - all drivers we looked at had bugs in handling init.
Plus we had very cozy
lock->unlock->locksequences during devlink command execution, which caused to races between devlink calls and driver initialization.
So? Why do you think it is a problem?
Quoted 5 lines
So I'm also interested to see what Jiri meant by saying "I unlock for register and for the rest of the init I lock again".
Thanks
Quoted 23 lines · Jiri Pirko
Quoted 22 lines · Wed, Nov 30, 2022 at 08:20:34PM CET, [email protected] wrote:
Quoted 20 lines · Jakub Kicinski
Quoted 9 lines · On Wed, 30 Nov 2022 18:00:05 +0100 Jiri Pirko wrote:
Quoted 8 lines · Wed, Nov 30, 2022 at 05:46:59PM CET, [email protected] wrote:
Quoted 6 lines · On Wed, 30 Nov 2022 12:42:39 +0100 Jiri Pirko wrote:
**)
I see. With the change I suggest, meaning doing devlink_port_register/unregister() and netdev_register/unregister only for registered devlink instance, you don't need this at all. When you hit this compat callback, the netdevice is there and therefore devlink instance is registered for sure.Quoted 6 lines
Quoted 4 lines
If you move devlink registration up it has to be under the instance lock, otherwise we're back to reload problems. That implies unregister should be under the lock too. But then we can't wait for refs in unregister. Perhaps I don't understand the suggestion.
I unlock for register and for the rest of the init I lock again.
The moment you register that instance callbacks can start coming. Leon move the register call last for a good reason - all drivers we looked at had bugs in handling init.
Quoted 5 lines
Quoted 3 lines
Plus we had very cozy
lock->unlock->locksequences during devlink command execution, which caused to races between devlink calls and driver initialization.So? Why do you think it is a problem?
We need to see the actual implementation. In general, once you unlock you can get other threads to change the state of your device.
Quoted 6 lines
Quoted 6 lines
So I'm also interested to see what Jiri meant by saying "I unlock for register and for the rest of the init I lock again".
Thanks
Jiri Pirko
maintainerQuoted 24 lines · Thu, Dec 01, 2022 at 11:05:38AM CET, [email protected] wrote:
Quoted 23 lines · Jiri Pirko
Quoted 22 lines · Wed, Nov 30, 2022 at 08:20:34PM CET, [email protected] wrote:
Quoted 20 lines · Jakub Kicinski
Quoted 9 lines · On Wed, 30 Nov 2022 18:00:05 +0100 Jiri Pirko wrote:
Quoted 8 lines · Wed, Nov 30, 2022 at 05:46:59PM CET, [email protected] wrote:
Quoted 6 lines · On Wed, 30 Nov 2022 12:42:39 +0100 Jiri Pirko wrote:
**)
I see. With the change I suggest, meaning doing devlink_port_register/unregister() and netdev_register/unregister only for registered devlink instance, you don't need this at all. When you hit this compat callback, the netdevice is there and therefore devlink instance is registered for sure.Quoted 6 lines
Quoted 4 lines
If you move devlink registration up it has to be under the instance lock, otherwise we're back to reload problems. That implies unregister should be under the lock too. But then we can't wait for refs in unregister. Perhaps I don't understand the suggestion.
I unlock for register and for the rest of the init I lock again.
The moment you register that instance callbacks can start coming. Leon move the register call last for a good reason - all drivers we looked at had bugs in handling init.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Plus we had very cozy
lock->unlock->locksequences during devlink command execution, which caused to races between devlink calls and driver initialization.So? Why do you think it is a problem?
We need to see the actual implementation. In general, once you unlock you can get other threads to change the state of your device.
Sure, I still don't understand, why it would be a problem.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
So I'm also interested to see what Jiri meant by saying "I unlock for register and for the rest of the init I lock again".
Thanks
Jiri Pirko
maintainerQuoted 30 lines · Wed, Nov 30, 2022 at 06:20:42PM CET, [email protected] wrote:
Quoted 9 lines · On Wed, 30 Nov 2022 18:00:05 +0100 Jiri Pirko wrote:
Quoted 8 lines · Wed, Nov 30, 2022 at 05:46:59PM CET, [email protected] wrote:
Quoted 6 lines · On Wed, 30 Nov 2022 12:42:39 +0100 Jiri Pirko wrote:
**)
I see. With the change I suggest, meaning doing devlink_port_register/unregister() and netdev_register/unregister only for registered devlink instance, you don't need this at all. When you hit this compat callback, the netdevice is there and therefore devlink instance is registered for sure.Quoted 6 lines
Quoted 4 lines
If you move devlink registration up it has to be under the instance lock, otherwise we're back to reload problems. That implies unregister should be under the lock too. But then we can't wait for refs in unregister. Perhaps I don't understand the suggestion.
I unlock for register and for the rest of the init I lock again.
The moment you register that instance callbacks can start coming. Leon move the register call last for a good reason - all drivers we looked at had bugs in handling init.
We can come up with fixes in the drivers, flags,
devlink_set_features()and all that sort of garbage until the day we die but let's not. The driver facing API should be simple - hold the lock around entire init.Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
What is "half-initialized"? Take devlink reload flow for instance. There are multiple things removed/readded, like
devlink_portand related netdevice. No problem there.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Yes, but reload is under the instance lock, so nothing can mess with a device in a transitional state.
Sure, that is what I want to do too. To be under instance lock.
I'm confused, you just said "I unlock for register".
Ah, right. I got your point now, you don't want the user to see half-init devlink objects. In reload, the secondhalf-uninit&seconfhalf-init happens atomically under instance lock, so the user sees the whole picture still.
But is it a problem? For ports, I don't think so. For the other objects being removed-readded during reload, why do you think it is a problem?
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
As mentioned above (**), I don't think this is needed.
But it is, please just let me do it and make the bugs stop 😭
Why exactly is it needed? I don't see it, pardon my ignorance :)
Let me send the RFC of the change tomorrow, you'll see what I mean.
The way I see it Leon had a stab at it, you did too, now it's my turn..
Up to you.
Quoted 76 lines · Ido Schimmel
Quoted 68 lines · Yang Yingliang
Quoted 63 lines · Leon Romanovsky
Quoted 54 lines · Yang Yingliang
Quoted 20 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 22 Nov 2022 21:04:29 +0200 Leon Romanovsky wrote:
Quoted 4 lines
Quoted 1 line
Please fix nsim instead of devlink.
I think if devlink is not registered, it can not be get and used, but there is no API for driver to check this, can I introduce a new helper name
devlink_is_registered()for driver using.There is no need in such API as driver calls to
devlink_register()and as such it knows when devlink is registered.This UAF is nsim specific issue. Real devices have single .probe() routine with serialized registration flow. None of them will use
devlink_is_registered()call.Agreed, the fix is to move the register call back. Something along the lines of the untested patch below? Yang Yingliang would you be able to turn that into a real patch?
drivers/net/netdevsim/dev.c +0
-0drivers/net/netdevsim/dev.c int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) · line 1573
Quoted 9 lines
drivers/net/netdevsim/dev.c +4
-1@@ -1566,12 +1566,15 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) err =devlink_params_register (devlink, nsim_devlink_params,ARRAY_SIZE (nsim_devlink_params));if (err) - goto err_dl_unregister; + goto err_resource_unregister;nsim_devlink_set_params_init_values (nsim_dev, devlink); +/* here, because params API still expect devlink to be unregistered */ +devl_register (devlink); +
devlink_set_features()called at last inprobe()also needs devlink is not registered.Quoted 14 lines
drivers/net/netdevsim/dev.c +1
-2err =nsim_dev_dummy_region_init (nsim_dev, devlink);if (err) - goto err_params_unregister; + goto err_dl_unregister; err =nsim_dev_traps_init (devlink);if (err)@@ -1610,7 +1613,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) nsim_dev->esw_mode =DEVLINK_ESWITCH_MODE_LEGACY ;devlink_set_features (devlink,DEVLINK_F_RELOAD );devl_unlock (devlink); -devlink_register (devlink);return 0 ; err_hwstats_exit:drivers/net/netdevsim/dev.c int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev) · line 1635
Quoted 6 lines
drivers/net/netdevsim/dev.c +2
-1@@ -1629,10 +1631,11 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)nsim_dev_traps_exit (devlink); err_dummy_region_exit:nsim_dev_dummy_region_exit (nsim_dev); -err_params_unregister: +err_dl_unregister: +devl_unregister (devlink);It races with
dev_ethtool():dev_ethtoolQuoted 8 lines
devlink_try_get()
nsim_drv_probe
devl_lock()
devl_lock()
devlink_unregister()devlink_put()wait_for_completion()<- the refcount is
got indev_ethtool, it causes ABBA deadlockQuoted 3 lines
But all these races are nsim specific ones.
Can you please explain why devlink.[c|h] MUST be changed and nsim can't be fixed?Quoted 6 lines
I used the fix code proposed by Jakub, but it didn't work correctly, so I tried to correct and improve it, and need some devlink helper.
Anyway, it is a nsim problem, if we want fix this without touch devlink, I think we can add a 'registered' field in struct
nsim_dev, and it can be checked innsim_get_devlink_port()like this:
Quoted 8 lines
I read the discussion and it's not clear to me why this is a netdevsim specific problem. The fundamental problem seems to be that it is possible to hold a reference on a devlink instance before it's registered and that
devlink_free()will free the instance regardless of its current reference count because it expectsdevlink_unregister()to block. In this case, the instance was never registered, sodevlink_unregister()is not called.
Shouldn't devlink_free never free until after all references drop? That's how typical reference count works. I guess its just assuming that unregister will prevent this but....
Quoted 6 lines
ethtool was able to get a reference on the devlink instance before it was registered because netdevsim registers its netdevs before registering its devlink instance. However, netdevsim is not the only one doing this: funeth, ice, prestera, mlx4, mlxsw, nfp and potentially others do the same thing.
We used to register devlink before netdev, but this was changed at some point because we wanted to register it late after things had been setup.
Quoted 5 lines
When you think about it, it's strange that it's even possible for ethtool to reach the driver when the netdev used in the request is long gone, but it's not holding a reference on the netdev (it's holding a reference on the devlink instance instead) and
devlink_compat_running_version()is called without RTNL.
Yea that seems weird. I would expect ethtool to get to devlink via netdev? (especially now with the way we changed netdev to hold the devlink?)
Hmm.
Jakub Kicinski
maintainerQuoted 10 lines · On Wed, 23 Nov 2022 14:40:24 +0800 Yang Yingliang wrote:
Review
Quoted 8 lines
if (err)
(quoted diff) +4
-1- goto err_dl_unregister; + goto err_resource_unregister; nsim_devlink_set_params_init_values(nsim_dev, devlink); + /* here, because params API still expect devlink to be unregistered */ + devl_register(devlink); +
devlink_set_features()called at last inprobe()also needs devlink is not registered.
You can move the devlink_set_features() up. It's also a leftover, it was preventing reload from happening before probe has finished. Now the instance is locked until probe is done so there is no race.
Jakub Kicinski
maintainerQuoted 4 lines · On Wed, 23 Nov 2022 14:40:24 +0800 Yang Yingliang wrote:
Review
Quoted 2 lines
(quoted diff) +2
-0+err_dl_unregister: + devl_unregister(devlink);It races with
dev_ethtool():dev_ethtool
Quoted 8 lines
devlink_try_get()
nsim_drv_probe
devl_lock()
devl_lock()
devlink_unregister()devlink_put()wait_for_completion()<- the refcount
is got indev_ethtool, it causes ABBA deadlock
Yeah.. so my original design for the locking had a "devlink_is_alive()" check for this exact reason:
https://lore.kernel.org/netdev/[email protected]/
and the devlink structure was properly refcounted (devlink_put() calls devlink_free() when the last reference is released).
Pure references then need to check if the instance is still alive after locking it. Which is fine, it should only happen in core code.
I think we should go back to that idea.
The waiting for references is a nightmare in the netdev code.
Quoted 18 lines · Jakub Kicinski
Quoted 4 lines · On Wed, 23 Nov 2022 14:40:24 +0800 Yang Yingliang wrote:
Review
Quoted 2 lines
(quoted diff) +2
-0+err_dl_unregister: + devl_unregister(devlink);It races with
dev_ethtool():dev_ethtoolQuoted 8 lines
devlink_try_get()
nsim_drv_probe
devl_lock()
devl_lock()
devlink_unregister()devlink_put()wait_for_completion()<- the refcount
is got indev_ethtool, it causes ABBA deadlockYeah.. so my original design for the locking had a "
devlink_is_alive()" check for this exact reason:
Quoted 7 lines
and the devlink structure was properly refcounted (
devlink_put()callsdevlink_free()when the last reference is released).Pure references then need to check if the instance is still alive after locking it. Which is fine, it should only happen in core code.
I think we should go back to that idea.
But Leon disagree to change devlink code.
I think this problem occurs in the drivers that have multiple ports(netdev):
In some drivers (e.g. mlx5) , one net device uses one devlink instance(see mlx5e_probe()), the instance can not be get until the device is register, in this case, it won't cause UAF.
But in some other drivers(e.g. netdevsim, funeth) multiple ports(net devices) use one devlink instance. If first one is register successful, the instance is visible and can be get through netdev, meanwhile, the second port register failed and goto free the devlink that used by first port(netdevice). So can we fix this in every single driver.
Thanks,
Yang
Quoted 4 lines
The waiting for references is a nightmare in the netdev code.
.
Jiri Pirko
maintainerTue, Nov 22, 2022 at 01:10:48PM CET, [email protected] wrote:
[...]
Quoted 13 lines
Fixes a62fdbbe9403 ("netdevsim: implement ndo_get_devlink_port")Reported-by Zhengchao Shao <[email protected]>Signed-off-by Yang Yingliang <[email protected]>---
net/core/devlink.c| 3 +++1 file changed, 3 insertions(+)net/core/devlink.c +0
-0
net/core/devlink.c void devlink_put(struct devlink *devlink) · line 253
Quoted 4 lines
net/core/devlink.c +1
-0@@ -250,6 +250,9 @@ void devlink_put(struct devlink *devlink)struct devlink *__must_check devlink_try_get (struct devlink *devlink) { +if (!xa_get_mark (&devlinks, devlink->index,DEVLINK_REGISTERED ))
^
You have an extra space here. Checkpatch would warn you.
Quoted 8 lines
net/core/devlink.c +2
-0+return NULL ; +if (refcount_inc_not_zero (&devlink->refcount))return devlink;return NULL ;Signature · 1 line
2.25.1