Provide more error details when a QP moves to
Subsystems (4)
cover Provide more error details when a QP moves to
The following series adds debug prints for fatal QP events that are helpful for finding the root cause of the errors.
The ib_get_qp_err_syndrome is called at a work queue since the QP event callback is running on an interrupt context that can't sleep.
The functions is especially useful for debugging purposes for few reasons:
First of all it provides the information in a human readable way, that would make it easier to identify the bug root cause.
Secondly it also allows providing vendor specfic error codes or information that could be very useful to users who know them.
Lastly and most importantly the function provides information about the reason the QP moved to error state, in cases where CQE isn't generated and without this feature it would have been way harder to know the root cause of the error.
An example of such case would be a remote write with RKEY violation, whereas on the remote side no CQE would be generated but this print allows to know the reason behind the failure.
Thanks.
Israel Rukshin (1):
nvme-rdma: add more error details when a QP moves to an error state
Patrisious Haddad (3):
net/mlx5: Introduce CQE error syndrome RDMA/core: Introduce ib_get_qp_err_syndrome function RDMA/mlx5: Implement ib_get_qp_err_syndrome
drivers/infiniband/core/device.c | 1 + drivers/infiniband/core/verbs.c | 8 +++++ drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 + drivers/infiniband/hw/mlx5/qp.c | 42 ++++++++++++++++++++++++- drivers/infiniband/hw/mlx5/qp.h | 2 +- drivers/infiniband/hw/mlx5/qpc.c | 4 ++- drivers/nvme/host/rdma.c | 24 ++++++++++++++ include/linux/mlx5/mlx5_ifc.h | 47 +++++++++++++++++++++++++--- include/rdma/ib_verbs.h | 13 ++++++++ 10 files changed, 135 insertions(+), 8 deletions(-)
Signature · 1 line
2.18.1
PATCH 1/4 net/mlx5: Introduce CQE error syndrome
Introduces CQE error syndrome bits which are inside qp_context_extension and are used to report the reason the QP was moved to error state. Useful for cases in which a CQE is generated, such as remote write rkey violaton.
--- include/linux/mlx5/mlx5_ifc.h | 47 +++++++++++++++++++++++++++++++---- 1 file changed, 42 insertions(+), 5 deletions(-)
include/linux/mlx5/mlx5_ifc.h +42 -5
@@ -1441,7 +1441,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {u8 null_mkey[0x1 ];u8 log_max_klm_list_size[0x6 ]; -u8 reserved_at_120[0xa ]; +u8 reserved_at_120[0x2 ]; +u8 qpc_extension[0x1 ]; +u8 reserved_at_123[0x7 ];u8 log_max_ra_req_dc[0x6 ];u8 reserved_at_130[0x9 ];u8 vnic_env_cq_overrun[0x1 ];
@@ -1605,7 +1607,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {u8 log_bf_reg_size[0x5 ]; -u8 reserved_at_270[0x6 ]; +u8 reserved_at_270[0x3 ]; +u8 qp_error_syndrome[0x1 ]; +u8 reserved_at_274[0x2 ];u8 lag_dct[0x2 ];u8 lag_tx_port_affinity[0x1 ];u8 lag_native_fdb_selection[0x1 ];
@@ -5257,6 +5261,37 @@ struct mlx5_ifc_query_rmp_in_bits {u8 reserved_at_60[0x20 ]; }; +struct mlx5_ifc_cqe_error_syndrome_bits { +u8 hw_error_syndrome[0x8 ]; +u8 hw_syndrome_type[0x4 ]; +u8 reserved_at_c[0x4 ]; +u8 vendor_error_syndrome[0x8 ]; +u8 syndrome[0x8 ]; +}; + +struct mlx5_ifc_qp_context_extension_bits { +u8 reserved_at_0[0x60 ]; + +struct mlx5_ifc_cqe_error_syndrome_bits error_syndrome; + +u8 reserved_at_80[0x580 ]; +}; + +struct mlx5_ifc_qpc_extension_and_pas_list_in_bits { +struct mlx5_ifc_qp_context_extension_bits qpc_data_extension; + +u8 pas[0 ][0x40 ]; +}; + +struct mlx5_ifc_qp_pas_list_in_bits { +struct mlx5_ifc_cmd_pas_bits pas[0 ]; +}; + +union mlx5_ifc_qp_pas_or_qpc_ext_and_pas_bits { +struct mlx5_ifc_qp_pas_list_in_bits qp_pas_list; +struct mlx5_ifc_qpc_extension_and_pas_list_in_bits qpc_ext_and_pas_list; +}; +struct mlx5_ifc_query_qp_out_bits {u8 status[0x8 ];u8 reserved_at_8[0x18 ];
@@ -5273,7 +5308,7 @@ struct mlx5_ifc_query_qp_out_bits {u8 reserved_at_800[0x80 ]; -u8 pas[][0x40 ]; +union mlx5_ifc_qp_pas_or_qpc_ext_and_pas_bits qp_pas_or_qpc_ext_and_pas; };struct mlx5_ifc_query_qp_in_bits {
@@ -5283,7 +5318,8 @@ struct mlx5_ifc_query_qp_in_bits {u8 reserved_at_20[0x10 ];u8 op_mod[0x10 ]; -u8 reserved_at_40[0x8 ]; +u8 qpc_ext[0x1 ]; +u8 reserved_at_41[0x7 ];u8 qpn[0x18 ];u8 reserved_at_60[0x20 ];
@@ -8417,7 +8453,8 @@ struct mlx5_ifc_create_qp_in_bits {u8 reserved_at_20[0x10 ];u8 op_mod[0x10 ]; -u8 reserved_at_40[0x8 ]; +u8 qpc_ext[0x1 ]; +u8 reserved_at_41[0x7 ];u8 input_qpn[0x18 ];u8 reserved_at_60[0x20 ];
Signature · 1 line
2.18.1
PATCH 2/4 RDMA/core: Introduce ib_get_qp_err_syndrome function
In reply to Patrisious HaddadIntroduce ib_get_qp_err_syndrome function, which enables kernel applications to query the reason the QP moved to error state. Even in cases in which no CQE was generated.
--- drivers/infiniband/core/device.c | 1 + drivers/infiniband/core/verbs.c | 8 ++++++++ include/rdma/ib_verbs.h | 13 +++++++++++++ 3 files changed, 22 insertions(+)
drivers/infiniband/core/device.c +1 -0
@@ -2657,6 +2657,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)SET_DEVICE_OP (dev_ops, get_netdev);SET_DEVICE_OP (dev_ops, get_numa_node);SET_DEVICE_OP (dev_ops, get_port_immutable); +SET_DEVICE_OP (dev_ops, get_qp_err_syndrome);SET_DEVICE_OP (dev_ops, get_vector_affinity);SET_DEVICE_OP (dev_ops, get_vf_config);SET_DEVICE_OP (dev_ops, get_vf_guid);
drivers/infiniband/core/verbs.c +8 -0
@@ -1952,6 +1952,14 @@ int ib_query_qp(struct ib_qp *qp, }EXPORT_SYMBOL (ib_query_qp); +int ib_get_qp_err_syndrome (struct ib_qp *qp,char *str) +{ +return qp->device->ops.get_qp_err_syndrome ? + qp->device->ops.get_qp_err_syndrome (qp->real_qp, + str) : -EOPNOTSUPP ; +} +EXPORT_SYMBOL (ib_get_qp_err_syndrome); +int ib_close_qp (struct ib_qp *qp) {struct ib_qp *real_qp;
include/rdma/ib_verbs.h +13 -0
@@ -2465,6 +2465,7 @@ struct ib_device_ops { int qp_attr_mask, struct ib_udata *udata);int (*query_qp)(struct ib_qp *qp,struct ib_qp_attr *qp_attr,int qp_attr_mask,struct ib_qp_init_attr *qp_init_attr); +int (*get_qp_err_syndrome)(struct ib_qp *qp,char *str);int (*destroy_qp)(struct ib_qp *qp,struct ib_udata *udata);int (*create_cq)(struct ib_cq *cq,const struct ib_cq_init_attr *attr,struct ib_udata *udata);
@@ -3777,6 +3778,18 @@ int ib_query_qp(struct ib_qp *qp,int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr); +#define IB_ERR_SYNDROME_LENGTH 256 + +/** +* ib_get_qp_err_syndrome - Returns a string that describes the reason +* the specified QP moved to error state. +* @qp : The QP to query. +* @str: The reason the qp moved to error state. +* +* NOTE: the user must pass a str with size of at least IB_ERR_SYNDROME_LENGTH +*/ +int ib_get_qp_err_syndrome (struct ib_qp *qp,char *str); +/** * ib_destroy_qp - Destroys the specified QP. * @qp: The QP to destroy.
Signature · 1 line
2.18.1
PATCH 3/4 RDMA/mlx5: Implement ib_get_qp_err_syndrome
In reply to Patrisious HaddadImplement ib_get_qp_err_syndrome using a query_qp FW call and return the result in a human readable string.
--- drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 + drivers/infiniband/hw/mlx5/qp.c | 42 +++++++++++++++++++++++++++- drivers/infiniband/hw/mlx5/qp.h | 2 +- drivers/infiniband/hw/mlx5/qpc.c | 4 ++- 5 files changed, 47 insertions(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/main.c +1 -0
@@ -3716,6 +3716,7 @@ static const struct ib_device_ops mlx5_ib_dev_ops = { .get_dev_fw_str = get_dev_fw_str, .get_dma_mr = mlx5_ib_get_dma_mr, .get_link_layer = mlx5_ib_port_link_layer, + .get_qp_err_syndrome = mlx5_ib_get_qp_err_syndrome, .map_mr_sg = mlx5_ib_map_mr_sg, .map_mr_sg_pi = mlx5_ib_map_mr_sg_pi, .mmap = mlx5_ib_mmap,
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -0
@@ -1232,6 +1232,7 @@ int mlx5_ib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask, struct ib_udata *udata);int mlx5_ib_query_qp (struct ib_qp *ibqp,struct ib_qp_attr *qp_attr,int qp_attr_mask,struct ib_qp_init_attr *qp_init_attr); +int mlx5_ib_get_qp_err_syndrome (struct ib_qp *ibqp,char *str);int mlx5_ib_destroy_qp (struct ib_qp *qp,struct ib_udata *udata);void mlx5_ib_drain_sq (struct ib_qp *qp);void mlx5_ib_drain_rq (struct ib_qp *qp);
drivers/infiniband/hw/mlx5/qp.c +41 -1
54 lines not on this page · raw message
drivers/infiniband/hw/mlx5/qp.h +1 -1
@@ -20,7 +20,7 @@ int mlx5_core_qp_modify(struct mlx5_ib_dev *dev, u16 opcode, u32 opt_param_mask,int mlx5_core_destroy_qp (struct mlx5_ib_dev *dev,struct mlx5_core_qp *qp);int mlx5_core_destroy_dct (struct mlx5_ib_dev *dev,struct mlx5_core_dct *dct);int mlx5_core_qp_query (struct mlx5_ib_dev *dev,struct mlx5_core_qp *qp, -u32 *out,int outlen); +u32 *out,int outlen,bool qpc_ext);int mlx5_core_dct_query (struct mlx5_ib_dev *dev,struct mlx5_core_dct *dct,u32 *out,int outlen);
drivers/infiniband/hw/mlx5/qpc.c +3 -1
@@ -504,12 +504,14 @@ void mlx5_cleanup_qp_table(struct mlx5_ib_dev *dev) }int mlx5_core_qp_query (struct mlx5_ib_dev *dev,struct mlx5_core_qp *qp, -u32 *out,int outlen) +u32 *out,int outlen,bool qpc_ext) {u32 in[MLX5_ST_SZ_DW (query_qp_in)] = {};MLX5_SET (query_qp_in, in, opcode,MLX5_CMD_OP_QUERY_QP );MLX5_SET (query_qp_in, in, qpn, qp->qpn); +MLX5_SET (query_qp_in, in, qpc_ext, qpc_ext); +return mlx5_cmd_exec (dev->mdev, in,sizeof (in), out, outlen); }
Signature · 1 line
2.18.1
PATCH 4/4 nvme-rdma: add more error details when a QP moves to an error state
In reply to Patrisious HaddadFrom: Israel Rukshin <[email protected]>
Add debug prints for fatal QP events that are helpful for finding the root cause of the errors. The ib_get_qp_err_syndrome is called at a work queue since the QP event callback is running on an interrupt context that can't sleep.
--- drivers/nvme/host/rdma.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
drivers/nvme/host/rdma.c +24 -0
@@ -99,6 +99,7 @@ struct nvme_rdma_queue {bool pi_support;int cq_size;struct mutex queue_lock; +struct work_struct qp_err_work; };struct nvme_rdma_ctrl {
@@ -237,11 +238,31 @@ static struct nvme_rdma_qe *nvme_rdma_alloc_ring(struct ib_device *ibdev,return NULL ; } +static void nvme_rdma_qp_error_work (struct work_struct *work) +{ +struct nvme_rdma_queue *queue =container_of (work, + struct nvme_rdma_queue, qp_err_work); +int ret; +char err[IB_ERR_SYNDROME_LENGTH ]; + + ret =ib_get_qp_err_syndrome (queue->qp, err); +if (ret) +return ; + +pr_err ("Queue %d got QP error syndrome %s\n" , +nvme_rdma_queue_idx (queue), err); +} +static void nvme_rdma_qp_event (struct ib_event *event,void *context) { +struct nvme_rdma_queue *queue = context; +pr_debug ("QP event %s (%d)\n" ,ib_event_msg (event->event), event->event); +if (event->event ==IB_EVENT_QP_FATAL || + event->event ==IB_EVENT_QP_ACCESS_ERR ) +queue_work (nvme_wq, &queue->qp_err_work); }static int nvme_rdma_wait_for_cm (struct nvme_rdma_queue *queue)
@@ -261,7 +282,9 @@ static int nvme_rdma_create_qp(struct nvme_rdma_queue *queue, const int factor)struct ib_qp_init_attr init_attr;int ret; +INIT_WORK (&queue->qp_err_work, nvme_rdma_qp_error_work);memset (&init_attr,0 ,sizeof (init_attr)); + init_attr.qp_context = queue; init_attr.event_handler = nvme_rdma_qp_event;/* +1 for drain */ init_attr.cap.max_send_wr = factor * queue->queue_size +1 ;
@@ -434,6 +457,7 @@ static void nvme_rdma_destroy_queue_ib(struct nvme_rdma_queue *queue)ib_mr_pool_destroy (queue->qp, &queue->qp->sig_mrs);ib_mr_pool_destroy (queue->qp, &queue->qp->rdma_mrs); +flush_work (&queue->qp_err_work);/* * The cm_id object might have been destroyed during RDMA connection * establishment error flow to avoid getting other cma events, thus
Signature · 1 line
2.18.1
Christoph Hellwig
maintainerRe: [PATCH rdma-next 4/4] nvme-rdma: add more error details when a QP moves to an error state
Quoted 6 lines · Patrisious Haddad
From: Israel Rukshin <[email protected]>
Add debug prints for fatal QP events that are helpful for finding the root cause of the errors. The
ib_get_qp_err_syndromeis called at a work queue since the QP event callback is running on an interrupt context that can't sleep.
What an awkward interface. What prevents us from allowing ib_get_qp_err_syndrome to be called from arbitrary calling contexts, or even better just delivering the error directly as part of the event?
Leon Romanovsky
maintainerRe: [PATCH rdma-next 4/4] nvme-rdma: add more error details when a QP moves to an error state
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Patrisious Haddad
From: Israel Rukshin <[email protected]>
Add debug prints for fatal QP events that are helpful for finding the root cause of the errors. The
ib_get_qp_err_syndromeis called at a work queue since the QP event callback is running on an interrupt context that can't sleep.
Quoted 4 lines
What an awkward interface. What prevents us from allowing
ib_get_qp_err_syndrometo be called from arbitrary calling contexts, or even better just delivering the error directly as part of the event?
We need to call to our FW through command interface and unfortunately it is not possible to do in atomic context.
Thanks
Sagi Grimberg
maintainerRe: [PATCH rdma-next 4/4] nvme-rdma: add more error details when a QP moves to an error state
In reply to Patrisious HaddadQuoted 11 lines
From: Israel Rukshin <[email protected]>
Add debug prints for fatal QP events that are helpful for finding the root cause of the errors. The
ib_get_qp_err_syndromeis called at a work queue since the QP event callback is running on an interrupt context that can't sleep.Signed-off-by Israel Rukshin <[email protected]>Reviewed-by Max Gurtovoy <[email protected]>Reviewed-by Leon Romanovsky <[email protected]>
What makes nvme-rdma special here? Why do you get this in nvme-rdma and not srp/iser/nfs-rdma/rds/smc/ipoib etc?
This entire code needs to move to the rdma core instead of being leaked to ulps.
Re: [PATCH rdma-next 4/4] nvme-rdma: add more error details when a QP moves to an error state
Quoted 12 lines · Sagi Grimberg
Quoted 11 lines
From: Israel Rukshin <[email protected]>
Add debug prints for fatal QP events that are helpful for finding the root cause of the errors. The
ib_get_qp_err_syndromeis called at a work queue since the QP event callback is running on an interrupt context that can't sleep.Signed-off-by Israel Rukshin <[email protected]>Reviewed-by Max Gurtovoy <[email protected]>Reviewed-by Leon Romanovsky <[email protected]>
Quoted 5 lines
What makes nvme-rdma special here? Why do you get this in nvme-rdma and not srp/iser/nfs-rdma/rds/smc/ipoib etc?
This entire code needs to move to the rdma core instead of being leaked to ulps.
We can move, but you will lose connection between queue number, caller and error itself.
As I answered to Christoph, we will need to execute query QP command in a workqueue outside of event handler.
So you will get a print about queue in error state and later you will see parsed error print somewhere in the dmesg.
Thanks
Sagi Grimberg
maintainerRe: [PATCH rdma-next 4/4] nvme-rdma: add more error details when a QP moves to an error state
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
From: Israel Rukshin <[email protected]>
Add debug prints for fatal QP events that are helpful for finding the root cause of the errors. The
ib_get_qp_err_syndromeis called at a work queue since the QP event callback is running on an interrupt context that can't sleep.Signed-off-by Israel Rukshin <[email protected]>Reviewed-by Max Gurtovoy <[email protected]>Reviewed-by Leon Romanovsky <[email protected]>
Quoted 8 lines
Quoted 5 lines
What makes nvme-rdma special here? Why do you get this in nvme-rdma and not srp/iser/nfs-rdma/rds/smc/ipoib etc?
This entire code needs to move to the rdma core instead of being leaked to ulps.
We can move, but you will lose connection between queue number, caller and error itself.
That still doesn't explain why nvme-rdma is special.
In any event, the ulp can log the qpn so the context can be interrogated if that is important.
Christoph Hellwig
maintainerRe: [PATCH rdma-next 4/4] nvme-rdma: add more error details when a QP moves to an error state
Quoted 11 lines · Sagi Grimberg
Quoted 6 lines
Quoted 3 lines
This entire code needs to move to the rdma core instead of being leaked to ulps.
We can move, but you will lose connection between queue number, caller and error itself.
That still doesn't explain why nvme-rdma is special.
In any event, the ulp can log the qpn so the context can be interrogated if that is important.
I also don't see why the QP event handler can't be called from user context to start with. I see absolutely no reason to add boilerplate code to drivers for reporting slighly more verbose errors on one specific piece of hrdware. I'd say clean up the mess that is the QP event handler first, and then once error reporting becomes trivial we can just do it.
Re: [PATCH rdma-next 4/4] nvme-rdma: add more error details when a QP moves to an error state
Quoted 13 lines · Christoph Hellwig
Quoted 11 lines · Sagi Grimberg
Quoted 6 lines
Quoted 3 lines
This entire code needs to move to the rdma core instead of being leaked to ulps.
We can move, but you will lose connection between queue number, caller and error itself.
That still doesn't explain why nvme-rdma is special.
In any event, the ulp can log the qpn so the context can be interrogated if that is important.
Quoted 6 lines
I also don't see why the QP event handler can't be called from user context to start with. I see absolutely no reason to add boilerplate code to drivers for reporting slighly more verbose errors on one specific piece of hrdware. I'd say clean up the mess that is the QP event handler first, and then once error reporting becomes trivial we can just do it.
I don't know, Chuck documented it in 2018:eb93c82ed8c7 ("RDMA/core: Document QP @event_handler function")
1164 struct ib_qp_init_attr {
1165 /* Consumer's event_handler callback must not block */
1166 void (*event_handler)(struct ib_event *, void *);Thanks
Re: [PATCH rdma-next 4/4] nvme-rdma: add more error details when a QP moves to an error state
Quoted 24 lines · Leon Romanovsky
External email: Use caution opening links or attachments
Quoted 13 lines · Christoph Hellwig
Quoted 11 lines · Sagi Grimberg
Quoted 6 lines
Quoted 3 lines
This entire code needs to move to the rdma core instead of being leaked to ulps.
We can move, but you will lose connection between queue number, caller and error itself.
That still doesn't explain why nvme-rdma is special.
In any event, the ulp can log the qpn so the context can be interrogated if that is important.
Quoted 6 lines
I also don't see why the QP event handler can't be called from user context to start with. I see absolutely no reason to add boilerplate code to drivers for reporting slighly more verbose errors on one specific piece of hrdware. I'd say clean up the mess that is the QP event handler first, and then once error reporting becomes trivial we can just do it.
Quoted 6 lines
I don't know, Chuck documented it in 2018:
eb93c82ed8c7("RDMA/core: Document QP @event_handler function")1164 struct ib_qp_init_attr { 1165 /* Consumer's event_handler callback must not block */ 1166 void (*event_handler)(struct ib_event *, void *);
Looks like driver calls it in an atomic way, e.g.:
mlx5_ib_qp_event() -> ibqp->event_handler(&event, ibqp->qp_context);
Could driver also report it as an IB async event, as WQ_CATAS_ERROR is defined as an async event (IB spec C11-39), and QP_FATAL is also an event of enum ib_event_type? Is it a problem that one event is reported twice?
If it is acceptable then ulp could register this event handler with ib_register_event_handler(), which is non-atomic.
Re: [PATCH rdma-next 4/4] nvme-rdma: add more error details when a QP moves to an error state
Quoted 38 lines · Mark Zhang
External email: Use caution opening links or attachments
Quoted 24 lines · Leon Romanovsky
External email: Use caution opening links or attachments
Quoted 13 lines · Christoph Hellwig
Quoted 11 lines · Sagi Grimberg
Quoted 6 lines
Quoted 3 lines
This entire code needs to move to the rdma core instead of being leaked to ulps.
We can move, but you will lose connection between queue number, caller and error itself.
That still doesn't explain why nvme-rdma is special.
In any event, the ulp can log the qpn so the context can be interrogated if that is important.
Quoted 6 lines
I also don't see why the QP event handler can't be called from user context to start with. I see absolutely no reason to add boilerplate code to drivers for reporting slighly more verbose errors on one specific piece of hrdware. I'd say clean up the mess that is the QP event handler first, and then once error reporting becomes trivial we can just do it.
Quoted 6 lines
I don't know, Chuck documented it in 2018:
eb93c82ed8c7("RDMA/core: Document QP @event_handler function")1164 struct
ib_qp_init_attr{
1165 /* Consumer'sevent_handlercallback must not block */ 1166 void (*event_handler)(structib_event*, void *);Looks like driver calls it in an atomic way, e.g.:
mlx5_ib_qp_event()->ibqp->event_handler(&event, ibqp->qp_context);
Quoted 7 lines
Could driver also report it as an IB async event, as
WQ_CATAS_ERRORis defined as an async event (IB spec C11-39), andQP_FATALis also an event of enumib_event_type? Is it a problem that one event is reported twice?If it is acceptable then ulp could register this event handler with
ib_register_event_handler(), which is non-atomic.
Or move qp event handler to non-atomic as Christoph suggested? This means to fix the mlx4/mlx5 driver, to call it in a work queue.
Re: [PATCH rdma-next 4/4] nvme-rdma: add more error details when a QP moves to an error state
In reply to Christoph HellwigQuoted 16 lines · Christoph Hellwig
External email: Use caution opening links or attachments
Quoted 11 lines · Sagi Grimberg
Quoted 6 lines
Quoted 3 lines
This entire code needs to move to the rdma core instead of being leaked to ulps.
We can move, but you will lose connection between queue number, caller and error itself.
That still doesn't explain why nvme-rdma is special.
In any event, the ulp can log the qpn so the context can be interrogated if that is important.
Quoted 6 lines
I also don't see why the QP event handler can't be called from user context to start with. I see absolutely no reason to add boilerplate code to drivers for reporting slighly more verbose errors on one specific piece of hrdware. I'd say clean up the mess that is the QP event handler first, and then once error reporting becomes trivial we can just do it.
I would like to emphasize that it is not just about slightly more verbose error, but mainly it is about an error that wouldn't have been reported at all without this feature, as I previously mentioned error cases in which the remote side doesn't generate a CQE, the remote side wouldn't even know why the QP was moved to error state without this feature.
Re: [PATCH rdma-next 4/4] nvme-rdma: add more error details when a QP moves to an error state
In reply to Sagi GrimbergQuoted 15 lines · Sagi Grimberg
Quoted 15 lines
Quoted 15 lines
Quoted 11 lines
From: Israel Rukshin <[email protected]>
Add debug prints for fatal QP events that are helpful for finding the root cause of the errors. The
ib_get_qp_err_syndromeis called at a work queue since the QP event callback is running on an interrupt context that can't sleep.Signed-off-by Israel Rukshin <[email protected]>Reviewed-by Max Gurtovoy <[email protected]>Reviewed-by Leon Romanovsky <[email protected]>What makes nvme-rdma special here? Why do you get this in nvme-rdma and not srp/iser/nfs-rdma/rds/smc/ipoib etc?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
This entire code needs to move to the rdma core instead of being leaked to ulps.
We can move, but you will lose connection between queue number, caller and error itself.
That still doesn't explain why nvme-rdma is special.
It was important for us to get proper review from at least one ULP, nvme-rdma is not special at all.
Quoted 3 lines
In any event, the ulp can log the qpn so the context can be interrogated if that is important.
ok