RDMA/rxe: fix regression caused by recent patch
Subsystems (2)
PATCH v2 1/1 RDMA/rxe: fix regression caused by recent patch
The commit referenced below performs additional checking on devices used for DMA. Specifically it checks that
device->dma_mask != NULL
Rdma_rxe uses this device when pinning MR memory but did not set the value of dma_mask. In fact rdma_rxe does not perform any DMA operations so the value is never used but is checked.
This patch gives dma_mask a valid value extracted from the device backing the ndev used by rxe.
Without this patch rdma_rxe does not function.
N.B. This patch needs to be applied before the recent fix to add back IB_USER_VERBS_CMD_POST_SEND to uverbs_cmd_mask.
Dennis Dallesandro reported that Parav's similar patch did not apply cleanly to rxe. This one does to for-next head of tree as of yesterday.
--- drivers/infiniband/sw/rxe/rxe_verbs.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)
drivers/infiniband/sw/rxe/rxe_verbs.c +15 -2
@@ -1128,19 +1128,32 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name)int err;struct ib_device *dev = &rxe->ib_dev;struct crypto_shash *tfm; +u64 dma_mask;strlcpy (dev->node_desc,"rxe" ,sizeof (dev->node_desc)); dev->node_type =RDMA_NODE_IB_CA ; dev->phys_port_cnt =1 ; dev->num_comp_vectors =num_possible_cpus (); - dev->dev.parent =rxe_dma_device (rxe); dev->local_dma_lkey =0 ;addrconf_addr_eui48 ((unsigned char *)&dev->node_guid, rxe->ndev->dev_addr); dev->dev.dma_parms = &rxe->dma_parms;dma_set_max_seg_size (&dev->dev,UINT_MAX ); -dma_set_coherent_mask (&dev->dev,dma_get_required_mask (&dev->dev)); + +/* rdma_rxe never does real DMA but does rely on +* pinning user memory in MRs to avoid page faults +* in responder and completer tasklets. This code +* supplies a valid dma_mask from the underlying +* network device. It is never used but is checked. +*/ + dev->dev.parent =rxe_dma_device (rxe); + dma_mask = *(dev->dev.parent->dma_mask); + err =dma_coerce_mask_and_coherent (&dev->dev, dma_mask); +if (err) { +pr_warn ("dma_mask not supported\n" ); +return err; + } dev->uverbs_cmd_mask |=BIT_ULL (IB_USER_VERBS_CMD_REQ_NOTIFY_CQ );
Signature · 1 line
2.27.0
Jason Gunthorpe
maintainerQuoted 44 lines · Bob Pearson
The commit referenced below performs additional checking on devices used for DMA. Specifically it checks that
device->dma_mask!= NULL
Rdma_rxeuses this device when pinning MR memory but did not set the value ofdma_mask. In factrdma_rxedoes not perform any DMA operations so the value is never used but is checked.This patch gives
dma_maska valid value extracted from the device backing the ndev used by rxe.Without this patch
rdma_rxedoes not function.N.B. This patch needs to be applied before the recent fix to add back
IB_USER_VERBS_CMD_POST_SENDtouverbs_cmd_mask.Dennis Dallesandro reported that Parav's similar patch did not apply cleanly to rxe. This one does to for-next head of tree as of yesterday.
Fixes f959dcd6ddfd2 ("dma-direct: Fix potential NULL pointer dereference")Signed-off-by Bob Pearson <[email protected]>drivers/infiniband/sw/rxe/rxe_verbs.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)drivers/infiniband/sw/rxe/rxe_verbs.c +2
-1+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c@@ -1128,19 +1128,32 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name)int err;struct ib_device *dev = &rxe->ib_dev;struct crypto_shash *tfm; +u64 dma_mask;strlcpy (dev->node_desc,"rxe" ,sizeof (dev->node_desc)); dev->node_type =RDMA_NODE_IB_CA ; dev->phys_port_cnt =1 ; dev->num_comp_vectors =num_possible_cpus (); - dev->dev.parent =rxe_dma_device (rxe); dev->local_dma_lkey =0 ;addrconf_addr_eui48 ((unsigned char *)&dev->node_guid, rxe->ndev->dev_addr); dev->dev.dma_parms = &rxe->dma_parms;
drivers/infiniband/sw/rxe/rxe_verbs.c int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name) · line 1150
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +8
-1dma_set_max_seg_size (&dev->dev,UINT_MAX ); -dma_set_coherent_mask (&dev->dev,dma_get_required_mask (&dev->dev)); + +/* rdma_rxe never does real DMA but does rely on +* pinning user memory in MRs to avoid page faults +* in responder and completer tasklets. This code +* supplies a valid dma_mask from the underlying +* network device. It is never used but is checked. +*/ + dev->dev.parent =rxe_dma_device (rxe);
Oh! This is another bug, the parent of an ib_device should never be set to a net_device!! This is probably why we get all those mysterious syzkaller faults :| Just leave it NULL
drivers/infiniband/sw/rxe/rxe_verbs.c int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name) · line 1152
Quoted 2 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +2
-0+ dma_mask = *(dev->dev.parent->dma_mask); + err =dma_coerce_mask_and_coherent (&dev->dev, dma_mask);
Why not use Parav's logic?
Jason
Quoted 60 lines · Jason Gunthorpe
Quoted 44 lines · Bob Pearson
The commit referenced below performs additional checking on devices used for DMA. Specifically it checks that
device->dma_mask!= NULL
Rdma_rxeuses this device when pinning MR memory but did not set the value ofdma_mask. In factrdma_rxedoes not perform any DMA operations so the value is never used but is checked.This patch gives
dma_maska valid value extracted from the device backing the ndev used by rxe.Without this patch
rdma_rxedoes not function.N.B. This patch needs to be applied before the recent fix to add back
IB_USER_VERBS_CMD_POST_SENDtouverbs_cmd_mask.Dennis Dallesandro reported that Parav's similar patch did not apply cleanly to rxe. This one does to for-next head of tree as of yesterday.
Fixes f959dcd6ddfd2 ("dma-direct: Fix potential NULL pointer dereference")Signed-off-by Bob Pearson <[email protected]>drivers/infiniband/sw/rxe/rxe_verbs.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)drivers/infiniband/sw/rxe/rxe_verbs.c +2
-1+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c@@ -1128,19 +1128,32 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name)int err;struct ib_device *dev = &rxe->ib_dev;struct crypto_shash *tfm; +u64 dma_mask;strlcpy (dev->node_desc,"rxe" ,sizeof (dev->node_desc)); dev->node_type =RDMA_NODE_IB_CA ; dev->phys_port_cnt =1 ; dev->num_comp_vectors =num_possible_cpus (); - dev->dev.parent =rxe_dma_device (rxe); dev->local_dma_lkey =0 ;addrconf_addr_eui48 ((unsigned char *)&dev->node_guid, rxe->ndev->dev_addr); dev->dev.dma_parms = &rxe->dma_parms;drivers/infiniband/sw/rxe/rxe_verbs.c int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name) · line 1150
Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +8
-1dma_set_max_seg_size (&dev->dev,UINT_MAX ); -dma_set_coherent_mask (&dev->dev,dma_get_required_mask (&dev->dev)); + +/* rdma_rxe never does real DMA but does rely on +* pinning user memory in MRs to avoid page faults +* in responder and completer tasklets. This code +* supplies a valid dma_mask from the underlying +* network device. It is never used but is checked. +*/ + dev->dev.parent =rxe_dma_device (rxe);Oh! This is another bug, the parent of an
ib_deviceshould never be set to anet_device!! This is probably why we get all those mysterious syzkaller faults :| Just leave it NULL
Quoted 7 lines
drivers/infiniband/sw/rxe/rxe_verbs.c int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name) · line 1152
Quoted 2 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +2
-0+ dma_mask = *(dev->dev.parent->dma_mask); + err =dma_coerce_mask_and_coherent (&dev->dev, dma_mask);Why not use Parav's logic?
Jason
It's not the network device. It is the parent of the network device. On 64 bit machines it gives 0xffffffffffffffff as dma_mask.
struct device *rxe_dma_device(struct rxe_dev *rxe)
{
struct net_device *ndev;
ndev = rxe->ndev;
if (is_vlan_dev(ndev))
ndev = vlan_dev_real_dev(ndev);
return ndev->dev.parent;
}His should work too. They will behave the same at the end of the day. I don't really know what the rxe_dma_device() code was trying to do in the first place so I didn't change it. But it was a handy place to get a dma_mask that should work on any architecture. If there is no reason to set dev.parent I can get rid of rxe_dma_device.
Bob
Jason Gunthorpe
maintainerQuoted 18 lines · Bob Pearson
Quoted 18 lines
drivers/infiniband/sw/rxe/rxe_verbs.c int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name) · line 1150
Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +8
-0+ +/* rdma_rxe never does real DMA but does rely on +* pinning user memory in MRs to avoid page faults +* in responder and completer tasklets. This code +* supplies a valid dma_mask from the underlying +* network device. It is never used but is checked. +*/ + dev->dev.parent =rxe_dma_device (rxe);Oh! This is another bug, the parent of an
ib_deviceshould never be set to anet_device!! This is probably why we get all those mysterious syzkaller faults :| Just leave it NULLdrivers/infiniband/sw/rxe/rxe_verbs.c int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name) · line 1152
Quoted 2 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +2
-0+ dma_mask = *(dev->dev.parent->dma_mask); + err =dma_coerce_mask_and_coherent (&dev->dev, dma_mask);Why not use Parav's logic?
Quoted 4 lines
Quoted 1 line
Jason
It's not the network device. It is the parent of the network device. On 64 bit machines it gives
0xffffffffffffffffasdma_mask.
No, it is some weird thing because network devices don't always have physical device parents.
There is no relation between the netdevice RXE is running on and the DMA mask to use for the dummy dma ops, AFAICT
Quoted 2 lines
that should work on any architecture. If there is no reason to set dev.parent I can get rid of
rxe_dma_device.
Please, that arrangement is causing bugs.
Jason