nvme-rdma: Add IB event handling support
PATCH 1/1 nvme-rdma: Add IB event handling support
From: Nitzan Carmi <[email protected]>
IB devices may invoke IB events that need a special treatment from the ib_client. For example, fatal event notification raised to registered clients due to an invalid port/device state after EEH. IB clients should be aware of this fatal event and not post any WR's to the device. Draining the QP, for example, is forbidden and will stuck forever waiting for the flushed work completions.
--- drivers/nvme/host/rdma.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
drivers/nvme/host/rdma.c +36 -0
@@ -45,6 +45,7 @@struct nvme_rdma_device {struct ib_device *dev;struct ib_pd *pd; +struct ib_event_handler event_handler;struct kref ref;struct list_head entry; };
@@ -329,6 +330,7 @@ static void nvme_rdma_free_dev(struct kref *ref)list_del (&ndev->entry);mutex_unlock (&device_list_mutex); +ib_unregister_event_handler (&ndev->event_handler);ib_dealloc_pd (ndev->pd);kfree (ndev); }
@@ -343,6 +345,36 @@ static int nvme_rdma_dev_get(struct nvme_rdma_device *dev)return kref_get_unless_zero (&dev->ref); } +static void nvme_rdma_ib_event_handler (struct ib_event_handler *handler, +struct ib_event *event) +{ +struct nvme_rdma_ctrl *ctrl; +int i; + +pr_debug ("async event %s (%d) on device %s port %d\n" , +ib_event_msg (event->event), event->event, + event->device->name, event->element.port_num); + +switch (event->event) { +case IB_EVENT_DEVICE_FATAL : +mutex_lock (&nvme_rdma_ctrl_mutex); +list_for_each_entry (ctrl, &nvme_rdma_ctrl_list, list) { +if (ctrl->device->dev != event->device) +continue ; + +for (i =0 ; i < ctrl->ctrl.queue_count; i++) +clear_bit (NVME_RDMA_Q_LIVE , + &ctrl->queues[i].flags); +nvme_delete_ctrl (&ctrl->ctrl); + } +mutex_unlock (&nvme_rdma_ctrl_mutex); +break ; +default : +pr_debug ("Unsupported event (%d)\n" , event->event); +break ; + } +} +static struct nvme_rdma_device *nvme_rdma_find_get_device (struct rdma_cm_id *cm_id) {
@@ -374,6 +406,10 @@ static int nvme_rdma_dev_get(struct nvme_rdma_device *dev) goto out_free_pd; } +INIT_IB_EVENT_HANDLER (&ndev->event_handler, ndev->dev, + nvme_rdma_ib_event_handler); +ib_register_event_handler (&ndev->event_handler); +list_add (&ndev->entry, &device_list); out_unlock:mutex_unlock (&device_list_mutex);
Signature · 1 line
1.8.3.1
Christoph Hellwig
maintainerPATCH 1/1 nvme-rdma: Add IB event handling support
Eww. I guess something like this is required, but we really need one coherent rdma event model instead of this mix between rdms/cm, struct ib_client and ib_event.
PATCH 1/1 nvme-rdma: Add IB event handling support
Quoted 4 lines · Christoph Hellwig
Eww. I guess something like this is required, but we really need one coherent rdma event model instead of this mix between rdms/cm, struct
ib_clientandib_event.
Yes, I agree. Let's ignore this patch for now.
Sagi Grimberg
maintainerPATCH 1/1 nvme-rdma: Add IB event handling support
In reply to Max GurtovoyQuoted 7 lines
IB devices may invoke IB events that need a special treatment from the
ib_client. For example, fatal event notification raised to registered clients due to an invalid port/device state after EEH. IB clients should be aware of this fatal event and not post any WR's to the device. Draining the QP, for example, is forbidden and will stuck forever waiting for the flushed work completions.
Where can we find a documentation to what should and should not work in this event?
PATCH 1/1 nvme-rdma: Add IB event handling support
Quoted 10 lines · Sagi Grimberg
Quoted 7 lines
IB devices may invoke IB events that need a special treatment from the
ib_client. For example, fatal event notification raised to registered clients due to an invalid port/device state after EEH. IB clients should be aware of this fatal event and not post any WR's to the device. Draining the QP, for example, is forbidden and will stuck forever waiting for the flushed work completions.Where can we find a documentation to what should and should not work in this event?
I'll probably abandon this commit since there should be low level mechanism that needs to be aware of attempts to post WR to a device in a fatal error and generate a "fake" completions.
I'll update if this patch is needed.