User SIW fails matching device

PATCH Potnuri Bharat Teja 2 participants 5 messages Jul 12, 2019 InfiniBand

Share & tools

Potnuri Bharat Teja

author

2019-07-12 14:28 UTC #0 lore raw

Hi all,
I observe the following behavior on one of my machines configured for siw.

Issue:
SIW device gets wrong device ops (HW/real rdma driver device ops) instead of siw device ops due to improper device matching.

Root-cause:
In libibverbs, during user cma initialisation, for each entry from the driver list, sysfs device is checked for matching name or device.
If the siw/rxe driver is at the head of the list, then sysfs device matches properly with the corresponding siw driver and gets the corresponding siw/rxe device ops. Now, If the siw/rxe driver is after the real HW driver cxgb4/mlx5 respectively in the driver list, then siw sysfs device matches pci device and wrongly gets the device ops of HW driver (cxgb4/mlx5).

Below debug prints from verbs_register_driver() and driver_list entries, where siw is after cxgb4. I see verbs alloc context landing in cxgb4_alloc_context instead of siw_alloc_context, thus breaking user siw.

<debug> verbs_register_driver_22: 184: driver 0x176e370 <debug> verbs_register_driver_22: 185: name ipathverbs <debug> verbs_register_driver_22: 184: driver 0x176f6a0 <debug> verbs_register_driver_22: 185: name cxgb4 <debug> verbs_register_driver_22: 184: driver 0x176fd50 <debug> verbs_register_driver_22: 185: name cxgb3 <debug> verbs_register_driver_22: 184: driver 0x1777020 <debug> verbs_register_driver_22: 185: name rxe <debug> verbs_register_driver_22: 184: driver 0x1770a30 <debug> verbs_register_driver_22: 185: name siw <debug> verbs_register_driver_22: 184: driver 0x1771120 <debug> verbs_register_driver_22: 185: name mlx4 <debug> verbs_register_driver_22: 184: driver 0x1771990 <debug> verbs_register_driver_22: 185: name mlx5 <debug> verbs_register_driver_22: 184: driver 0x1771ff0 <debug> verbs_register_driver_22: 185: name efa

<debug> try_drivers: 372: driver 0x176e370, sysfs_dev 0x1776b20, name: ipathverbs <debug> try_drivers: 372: driver 0x176f6a0, sysfs_dev 0x1776b20, name: cxgb4 <debug> try_drivers: 372: driver 0x176fd50, sysfs_dev 0x1776b20, name: cxgb3 <debug> try_drivers: 372: driver 0x1777020, sysfs_dev 0x1776b20, name: rxe <debug> try_drivers: 372: driver 0x1770a30, sysfs_dev 0x1776b20, name: siw <debug> try_drivers: 372: driver 0x1771120, sysfs_dev 0x1776b20, name: mlx4 <debug> try_drivers: 372: driver 0x1771990, sysfs_dev 0x1776b20, name: mlx5 <debug> try_drivers: 372: driver 0x1771ff0, sysfs_dev 0x1776b20, name: efa

Proposed fix:
I have the below fix that works. It adds siw/rxe driver to the HEAD of the driver list and the rest to the tail. I am not sure if this fix is the ideal one, so I am attaching it to this mail.

libibverbs/init.c +0 -0

index 930d91811ca9..e44f0d743063 100644
@@ -182,7 +182,10 @@ void verbs_register_driver(const struct verbs_device_ops *ops)

	driver->ops = ops;

- list_add_tail(&driver_list, &driver->entry);
+ if (!strcmp(ops->name, "siw") || !strcmp(ops->name, "rxe"))
+ list_add(&driver_list, &driver->entry);
+ else
+ list_add_tail(&driver_list, &driver->entry);
}
---

Thanks,
Bharat

Jason Gunthorpe

2019-07-12 14:35 UTC #1 lore raw

Quoted 46 lines · Potnuri Bharat Teja

Hi all,
I observe the following behavior on one of my machines configured for siw.
Issue:
SIW device gets wrong device ops (HW/real rdma driver device ops) instead of siw device ops due to improper device matching.
Root-cause:
In libibverbs, during user cma initialisation, for each entry from the driver list, sysfs device is checked for matching name or device.
If the siw/rxe driver is at the head of the list, then sysfs device matches properly with the corresponding siw driver and gets the corresponding siw/rxe device ops. Now, If the siw/rxe driver is after the real HW driver cxgb4/mlx5 respectively in the driver list, then siw sysfs device matches pci device and wrongly gets the device ops of HW driver (cxgb4/mlx5).
Below debug prints from verbs_register_driver() and driver_list entries, where siw is after cxgb4. I see verbs alloc context landing in cxgb4_alloc_context instead of siw_alloc_context, thus breaking user siw.
<debug> verbs_register_driver_22: 184: driver 0x176e370 <debug> verbs_register_driver_22: 185: name ipathverbs <debug> verbs_register_driver_22: 184: driver 0x176f6a0 <debug> verbs_register_driver_22: 185: name cxgb4 <debug> verbs_register_driver_22: 184: driver 0x176fd50 <debug> verbs_register_driver_22: 185: name cxgb3 <debug> verbs_register_driver_22: 184: driver 0x1777020 <debug> verbs_register_driver_22: 185: name rxe <debug> verbs_register_driver_22: 184: driver 0x1770a30 <debug> verbs_register_driver_22: 185: name siw <debug> verbs_register_driver_22: 184: driver 0x1771120 <debug> verbs_register_driver_22: 185: name mlx4 <debug> verbs_register_driver_22: 184: driver 0x1771990 <debug> verbs_register_driver_22: 185: name mlx5 <debug> verbs_register_driver_22: 184: driver 0x1771ff0 <debug> verbs_register_driver_22: 185: name efa
<debug> try_drivers: 372: driver 0x176e370, sysfs_dev 0x1776b20, name: ipathverbs <debug> try_drivers: 372: driver 0x176f6a0, sysfs_dev 0x1776b20, name: cxgb4 <debug> try_drivers: 372: driver 0x176fd50, sysfs_dev 0x1776b20, name: cxgb3 <debug> try_drivers: 372: driver 0x1777020, sysfs_dev 0x1776b20, name: rxe <debug> try_drivers: 372: driver 0x1770a30, sysfs_dev 0x1776b20, name: siw <debug> try_drivers: 372: driver 0x1771120, sysfs_dev 0x1776b20, name: mlx4 <debug> try_drivers: 372: driver 0x1771990, sysfs_dev 0x1776b20, name: mlx5 <debug> try_drivers: 372: driver 0x1771ff0, sysfs_dev 0x1776b20, name: efa

Quoted 4 lines

Proposed fix:
I have the below fix that works. It adds siw/rxe driver to the HEAD of the driver list and the rest to the tail. I am not sure if this fix is the ideal one, so I am attaching it to this mail.

Update your rdma-core to latest and this will be fixed fully by using netlink to match the siw device..

Jason

Potnuri Bharat Teja

author

2019-07-12 15:24 UTC #2 lore raw

Quoted 47 lines · Jason Gunthorpe

Quoted 46 lines · Potnuri Bharat Teja
Hi all,
I observe the following behavior on one of my machines configured for siw.
Issue:
SIW device gets wrong device ops (HW/real rdma driver device ops) instead of siw device ops due to improper device matching.
Root-cause:
In libibverbs, during user cma initialisation, for each entry from the driver list, sysfs device is checked for matching name or device.
If the siw/rxe driver is at the head of the list, then sysfs device matches properly with the corresponding siw driver and gets the corresponding siw/rxe device ops. Now, If the siw/rxe driver is after the real HW driver cxgb4/mlx5 respectively in the driver list, then siw sysfs device matches pci device and wrongly gets the device ops of HW driver (cxgb4/mlx5).
Below debug prints from verbs_register_driver() and driver_list entries, where siw is after cxgb4. I see verbs alloc context landing in cxgb4_alloc_context instead of siw_alloc_context, thus breaking user siw.
<debug> verbs_register_driver_22: 184: driver 0x176e370 <debug> verbs_register_driver_22: 185: name ipathverbs <debug> verbs_register_driver_22: 184: driver 0x176f6a0 <debug> verbs_register_driver_22: 185: name cxgb4 <debug> verbs_register_driver_22: 184: driver 0x176fd50 <debug> verbs_register_driver_22: 185: name cxgb3 <debug> verbs_register_driver_22: 184: driver 0x1777020 <debug> verbs_register_driver_22: 185: name rxe <debug> verbs_register_driver_22: 184: driver 0x1770a30 <debug> verbs_register_driver_22: 185: name siw <debug> verbs_register_driver_22: 184: driver 0x1771120 <debug> verbs_register_driver_22: 185: name mlx4 <debug> verbs_register_driver_22: 184: driver 0x1771990 <debug> verbs_register_driver_22: 185: name mlx5 <debug> verbs_register_driver_22: 184: driver 0x1771ff0 <debug> verbs_register_driver_22: 185: name efa
<debug> try_drivers: 372: driver 0x176e370, sysfs_dev 0x1776b20, name: ipathverbs <debug> try_drivers: 372: driver 0x176f6a0, sysfs_dev 0x1776b20, name: cxgb4 <debug> try_drivers: 372: driver 0x176fd50, sysfs_dev 0x1776b20, name: cxgb3 <debug> try_drivers: 372: driver 0x1777020, sysfs_dev 0x1776b20, name: rxe <debug> try_drivers: 372: driver 0x1770a30, sysfs_dev 0x1776b20, name: siw <debug> try_drivers: 372: driver 0x1771120, sysfs_dev 0x1776b20, name: mlx4 <debug> try_drivers: 372: driver 0x1771990, sysfs_dev 0x1776b20, name: mlx5 <debug> try_drivers: 372: driver 0x1771ff0, sysfs_dev 0x1776b20, name: efa

Quoted 8 lines

Quoted 4 lines
Proposed fix:
I have the below fix that works. It adds siw/rxe driver to the HEAD of the driver list and the rest to the tail. I am not sure if this fix is the ideal one, so I am attaching it to this mail.
Update your rdma-core to latest and this will be fixed fully by using netlink to match the siw device..

I pulled the latest rdma-core, still see the issue.

commit 7ef6077ec3201f661458297fea776746ba752843 (HEAD, upstream/master)
Merge: 837954ff677c 95934b61a74e
Author: Jason Gunthorpe <[email protected]>
Date: Thu Jul 11 16:18:06 2019 -0300

Merge pull request #539 from jgunthorpe/netlink

Use netlink to learn about ibdevs and their related chardevs

-----------

Is there any corresponding kernel change or package dependency? I am currently on Doug's wip/dl-for-next branch.

Jason Gunthorpe

2019-07-12 15:30 UTC #3 lore raw

Quoted 63 lines · Potnuri Bharat Teja

Quoted 47 lines · Jason Gunthorpe
Quoted 46 lines · Potnuri Bharat Teja
Hi all,
I observe the following behavior on one of my machines configured for siw.
Issue:
SIW device gets wrong device ops (HW/real rdma driver device ops) instead of siw device ops due to improper device matching.
Root-cause:
In libibverbs, during user cma initialisation, for each entry from the driver list, sysfs device is checked for matching name or device.
If the siw/rxe driver is at the head of the list, then sysfs device matches properly with the corresponding siw driver and gets the corresponding siw/rxe device ops. Now, If the siw/rxe driver is after the real HW driver cxgb4/mlx5 respectively in the driver list, then siw sysfs device matches pci device and wrongly gets the device ops of HW driver (cxgb4/mlx5).
Below debug prints from verbs_register_driver() and driver_list entries, where siw is after cxgb4. I see verbs alloc context landing in cxgb4_alloc_context instead of siw_alloc_context, thus breaking user siw.
<debug> verbs_register_driver_22: 184: driver 0x176e370 <debug> verbs_register_driver_22: 185: name ipathverbs <debug> verbs_register_driver_22: 184: driver 0x176f6a0 <debug> verbs_register_driver_22: 185: name cxgb4 <debug> verbs_register_driver_22: 184: driver 0x176fd50 <debug> verbs_register_driver_22: 185: name cxgb3 <debug> verbs_register_driver_22: 184: driver 0x1777020 <debug> verbs_register_driver_22: 185: name rxe <debug> verbs_register_driver_22: 184: driver 0x1770a30 <debug> verbs_register_driver_22: 185: name siw <debug> verbs_register_driver_22: 184: driver 0x1771120 <debug> verbs_register_driver_22: 185: name mlx4 <debug> verbs_register_driver_22: 184: driver 0x1771990 <debug> verbs_register_driver_22: 185: name mlx5 <debug> verbs_register_driver_22: 184: driver 0x1771ff0 <debug> verbs_register_driver_22: 185: name efa
<debug> try_drivers: 372: driver 0x176e370, sysfs_dev 0x1776b20, name: ipathverbs <debug> try_drivers: 372: driver 0x176f6a0, sysfs_dev 0x1776b20, name: cxgb4 <debug> try_drivers: 372: driver 0x176fd50, sysfs_dev 0x1776b20, name: cxgb3 <debug> try_drivers: 372: driver 0x1777020, sysfs_dev 0x1776b20, name: rxe <debug> try_drivers: 372: driver 0x1770a30, sysfs_dev 0x1776b20, name: siw <debug> try_drivers: 372: driver 0x1771120, sysfs_dev 0x1776b20, name: mlx4 <debug> try_drivers: 372: driver 0x1771990, sysfs_dev 0x1776b20, name: mlx5 <debug> try_drivers: 372: driver 0x1771ff0, sysfs_dev 0x1776b20, name: efa
Quoted 8 lines
Quoted 4 lines
Proposed fix:
I have the below fix that works. It adds siw/rxe driver to the HEAD of the driver list and the rest to the tail. I am not sure if this fix is the ideal one, so I am attaching it to this mail.
Update your rdma-core to latest and this will be fixed fully by using netlink to match the siw device..
I pulled the latest rdma-core, still see the issue.
commit 7ef6077ec3201f661458297fea776746ba752843 (HEAD, upstream/master)
Merge: 837954ff677c 95934b61a74e
Author: Jason Gunthorpe <[email protected]>
Date: Thu Jul 11 16:18:06 2019 -0300

Quoted 7 lines

Merge pull request #539 from jgunthorpe/netlink
Use netlink to learn about ibdevs and their related chardevs
Is there any corresponding kernel change or package dependency? I am currently on Doug's wip/dl-for-next branch.

That should be good enough for the kernel.. Hmm.. The siw stuff didn't get updated, you need this rdma-core patch too. Please confirm

providers/siw/siw.c +1 -1

index 23e4dd976caf84..41f33fa16123e9 100644
@@ -907,7 +907,7 @@ static void siw_device_free(struct verbs_device *vdev)
 }
 
 static const struct verbs_match_ent rnic_table[] = {
-	VERBS_NAME_MATCH("siw", NULL),
+	VERBS_DRIVER_ID(RDMA_DRIVER_SIW),
 	{},
 };

Potnuri Bharat Teja

author

2019-07-12 15:53 UTC #4 lore raw

Quoted 80 lines · Jason Gunthorpe

Quoted 63 lines · Potnuri Bharat Teja
Quoted 47 lines · Jason Gunthorpe
Quoted 46 lines · Potnuri Bharat Teja
Hi all,
I observe the following behavior on one of my machines configured for siw.
Issue:
SIW device gets wrong device ops (HW/real rdma driver device ops) instead of siw device ops due to improper device matching.
Root-cause:
In libibverbs, during user cma initialisation, for each entry from the driver list, sysfs device is checked for matching name or device.
If the siw/rxe driver is at the head of the list, then sysfs device matches properly with the corresponding siw driver and gets the corresponding siw/rxe device ops. Now, If the siw/rxe driver is after the real HW driver cxgb4/mlx5 respectively in the driver list, then siw sysfs device matches pci device and wrongly gets the device ops of HW driver (cxgb4/mlx5).
Below debug prints from verbs_register_driver() and driver_list entries, where siw is after cxgb4. I see verbs alloc context landing in cxgb4_alloc_context instead of siw_alloc_context, thus breaking user siw.
<debug> verbs_register_driver_22: 184: driver 0x176e370 <debug> verbs_register_driver_22: 185: name ipathverbs <debug> verbs_register_driver_22: 184: driver 0x176f6a0 <debug> verbs_register_driver_22: 185: name cxgb4 <debug> verbs_register_driver_22: 184: driver 0x176fd50 <debug> verbs_register_driver_22: 185: name cxgb3 <debug> verbs_register_driver_22: 184: driver 0x1777020 <debug> verbs_register_driver_22: 185: name rxe <debug> verbs_register_driver_22: 184: driver 0x1770a30 <debug> verbs_register_driver_22: 185: name siw <debug> verbs_register_driver_22: 184: driver 0x1771120 <debug> verbs_register_driver_22: 185: name mlx4 <debug> verbs_register_driver_22: 184: driver 0x1771990 <debug> verbs_register_driver_22: 185: name mlx5 <debug> verbs_register_driver_22: 184: driver 0x1771ff0 <debug> verbs_register_driver_22: 185: name efa
<debug> try_drivers: 372: driver 0x176e370, sysfs_dev 0x1776b20, name: ipathverbs <debug> try_drivers: 372: driver 0x176f6a0, sysfs_dev 0x1776b20, name: cxgb4 <debug> try_drivers: 372: driver 0x176fd50, sysfs_dev 0x1776b20, name: cxgb3 <debug> try_drivers: 372: driver 0x1777020, sysfs_dev 0x1776b20, name: rxe <debug> try_drivers: 372: driver 0x1770a30, sysfs_dev 0x1776b20, name: siw <debug> try_drivers: 372: driver 0x1771120, sysfs_dev 0x1776b20, name: mlx4 <debug> try_drivers: 372: driver 0x1771990, sysfs_dev 0x1776b20, name: mlx5 <debug> try_drivers: 372: driver 0x1771ff0, sysfs_dev 0x1776b20, name: efa
Quoted 8 lines
Quoted 4 lines
Proposed fix:
I have the below fix that works. It adds siw/rxe driver to the HEAD of the driver list and the rest to the tail. I am not sure if this fix is the ideal one, so I am attaching it to this mail.
Update your rdma-core to latest and this will be fixed fully by using netlink to match the siw device..
I pulled the latest rdma-core, still see the issue.
commit 7ef6077ec3201f661458297fea776746ba752843 (HEAD, upstream/master)
Merge: 837954ff677c 95934b61a74e
Author: Jason Gunthorpe <[email protected]>
Date: Thu Jul 11 16:18:06 2019 -0300
Quoted 7 lines
Merge pull request #539 from jgunthorpe/netlink
Use netlink to learn about ibdevs and their related chardevs
Is there any corresponding kernel change or package dependency? I am currently on Doug's wip/dl-for-next branch.
That should be good enough for the kernel.. Hmm.. The siw stuff didn't get updated, you need this rdma-core patch too. Please confirm
providers/siw/siw.c +0 -0
index 23e4dd976caf84..41f33fa16123e9 100644

providers/siw/siw.c static void siw_device_free(struct verbs_device *vdev) · line 913

Quoted 8 lines

providers/siw/siw.c +1 -1

@@ -907,7 +907,7 @@ static void siw_device_free(struct verbs_device *vdev)
 }
 
 static const struct verbs_match_ent rnic_table[] = {
-	VERBS_NAME_MATCH("siw", NULL),
+	VERBS_DRIVER_ID(RDMA_DRIVER_SIW),
 	{},
 };

It works with above change. Thanks!