Ratelimited ibdev printk functions
Subsystems (2)
v2 cover Ratelimited ibdev printk functions
Hello,
This implements ratelimited ibdev printk functions, similar to their dev_*_ratelimited counterparts.
I've submitted the first patch before [1] without usage, I'm now adding the corresponding EFA patch as well.
cxgb4, hfi1, mlx4, vmw_pvrdma, rdmavt and rxe could also benefit from this if they move to ibvdev prints:
$ git grep ratelimited -- drivers/infiniband/[hs]w
drivers/infiniband/hw/cxgb4/mem.c: pr_warn_ratelimited("%s: dma map failure (non fatal)\n",
drivers/infiniband/hw/cxgb4/resource.c: pr_warn_ratelimited("%s: Out of RQT memory\n",
drivers/infiniband/hw/hfi1/chip.c: dd_dev_info_ratelimited(dd, "DCC Error: fmconfig error: %s\n",
drivers/infiniband/hw/hfi1/chip.c: dd_dev_info_ratelimited(dd, "DCC Error: PortRcv error: %s\n"
drivers/infiniband/hw/hfi1/chip.c: dd_dev_info_ratelimited(dd, "8051 access to LCB blocked\n");
drivers/infiniband/hw/hfi1/chip.c: dd_dev_info_ratelimited(dd, "host access to LCB blocked\n");
drivers/infiniband/hw/hfi1/chip.c: dd_dev_info_ratelimited(dd, "DCC Error: %s\n",
drivers/infiniband/hw/hfi1/chip.c: dd_dev_info_ratelimited(dd, "%s: PortErrorAction bounce\n",
drivers/infiniband/hw/hfi1/chip.c: dd_dev_info_ratelimited(dd, "SDMA engine %u interrupt, but no status bits set\n",
drivers/infiniband/hw/hfi1/hfi.h:#define dd_dev_err_ratelimited(dd, fmt, ...) \
drivers/infiniband/hw/hfi1/hfi.h: dev_err_ratelimited(&(dd)->pcidev->dev, "%s: " fmt, \
drivers/infiniband/hw/hfi1/hfi.h:#define dd_dev_warn_ratelimited(dd, fmt, ...) \
drivers/infiniband/hw/hfi1/hfi.h: dev_warn_ratelimited(&(dd)->pcidev->dev, "%s: " fmt, \
drivers/infiniband/hw/hfi1/hfi.h:#define dd_dev_info_ratelimited(dd, fmt, ...) \
drivers/infiniband/hw/hfi1/hfi.h: dev_info_ratelimited(&(dd)->pcidev->dev, "%s: " fmt, \
drivers/infiniband/hw/hfi1/mad.c: pr_err_ratelimited("hfi1: Invalid trap 0x%0x dropped. Total dropped: %d\n",
drivers/infiniband/hw/hfi1/mad.c: pr_warn_ratelimited("hfi1: Maximum trap limit reached for 0x%0x traps\n",
drivers/infiniband/hw/mlx4/qp.c: pr_warn_ratelimited("Unexpected event type %d on WQ 0x%06x. Events are not supported for WQs\n",
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c: dev_warn_ratelimited(&dev->pdev->dev,
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c: dev_warn_ratelimited(&dev->pdev->dev,
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c: dev_warn_ratelimited(&dev->pdev->dev,
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c: dev_warn_ratelimited(&dev->pdev->dev,
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c: dev_warn_ratelimited(&dev->pdev->dev,
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c: dev_warn_ratelimited(&dev->pdev->dev,
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c: dev_warn_ratelimited(&dev->pdev->dev,
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c: dev_warn_ratelimited(&dev->pdev->dev,
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c: dev_warn_ratelimited(&dev->pdev->dev,
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c: dev_warn_ratelimited(&dev->pdev->dev,
drivers/infiniband/sw/rdmavt/cq.c: rvt_pr_err_ratelimited(rdi, "CQ is full!\n");
drivers/infiniband/sw/rdmavt/vt.h:#define rvt_pr_err_ratelimited(rdi, fmt, ...) \
drivers/infiniband/sw/rdmavt/vt.h: __rvt_pr_err_ratelimited((rdi)->driver_f.get_pci_dev(rdi), \
drivers/infiniband/sw/rdmavt/vt.h:#define __rvt_pr_err_ratelimited(pdev, name, fmt, ...) \
drivers/infiniband/sw/rdmavt/vt.h: dev_err_ratelimited(&(pdev)->dev, "%s: " fmt, name, ##__VA_ARGS__)
drivers/infiniband/sw/rxe/rxe.h: pr_warn_ratelimited("failed crc calculation, err: %d\n", err);
drivers/infiniband/sw/rxe/rxe_net.c: pr_err_ratelimited("no route to %pI4\n", &daddr->s_addr);
drivers/infiniband/sw/rxe/rxe_net.c: pr_err_ratelimited("no route to %pI6\n", daddr);
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("bad qp type\n");
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("bad qp type\n");
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("bad qp type\n");
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("unsupported qp type\n");
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("bad pkey = 0x%x\n", pkey);
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("bad pkey = 0x%0x\n", pkey);
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("bad qkey, got 0x%x expected 0x%x for qpn 0x%x\n",
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("port %d != qp port %d\n",
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("dst addr %pI4 != qp source addr %pI4\n",
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("source addr %pI4 != qp dst addr %pI4\n",
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("dst addr %pI6 != qp source addr %pI6\n",
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("source addr %pI6 != qp dst addr %pI6\n",
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("bad tver\n");
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("no qp matches qpn 0x%x\n", qpn);
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("no grh for mcast qpn\n");
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("failed matching dgid\n");
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("bad ICRC from %pI6c\n",
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("bad ICRC from %pI4\n",
drivers/infiniband/sw/rxe/rxe_recv.c: pr_warn_ratelimited("bad ICRC from unknown\n");
drivers/infiniband/sw/rxe/rxe_resp.c: pr_err_ratelimited("Failed sending ack\n");
drivers/infiniband/sw/rxe/rxe_resp.c: pr_err_ratelimited("Failed sending ack\n");[1] https://patchwork.kernel.org/patch/10926991/
Changelog:
v2:
* Remove the DEBUG part as requested by Leon
Thanks,
Gal
Gal Pressman (2):
RDMA/core: Introduce ratelimited ibdev printk functions RDMA/efa: Rate limit admin queue error prints
drivers/infiniband/hw/efa/efa_com.c | 70 ++++++------ drivers/infiniband/hw/efa/efa_com_cmd.c | 136 ++++++++++++++---------- include/rdma/ib_verbs.h | 42 ++++++++ 3 files changed, 162 insertions(+), 86 deletions(-)
Signature · 1 line
2.22.0
PATCH v2 1/2 RDMA/core: Introduce ratelimited ibdev printk functions
Add ratelimited helpers to the ibdev_* printk functions. Implementation inspired by counterpart dev_*_ratelimited functions.
--- include/rdma/ib_verbs.h | 42 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
include/rdma/ib_verbs.h +42 -0
@@ -107,6 +107,48 @@ static inlinevoid ibdev_dbg (const struct ib_device *ibdev,const char *format, ...) {}#endif +#define ibdev_level_ratelimited (ibdev_level, ibdev, fmt, ...) \ +do { \ + static DEFINE_RATELIMIT_STATE(_rs, \ + DEFAULT_RATELIMIT_INTERVAL, \ + DEFAULT_RATELIMIT_BURST); \ + if (__ratelimit(&_rs)) \ + ibdev_level(ibdev, fmt, ##__VA_ARGS__); \ +} while (0) + +#define ibdev_emerg_ratelimited (ibdev, fmt, ...) \ + ibdev_level_ratelimited(ibdev_emerg, ibdev, fmt, ##__VA_ARGS__) +#define ibdev_alert_ratelimited (ibdev, fmt, ...) \ + ibdev_level_ratelimited(ibdev_alert, ibdev, fmt, ##__VA_ARGS__) +#define ibdev_crit_ratelimited (ibdev, fmt, ...) \ + ibdev_level_ratelimited(ibdev_crit, ibdev, fmt, ##__VA_ARGS__) +#define ibdev_err_ratelimited (ibdev, fmt, ...) \ + ibdev_level_ratelimited(ibdev_err, ibdev, fmt, ##__VA_ARGS__) +#define ibdev_warn_ratelimited (ibdev, fmt, ...) \ + ibdev_level_ratelimited(ibdev_warn, ibdev, fmt, ##__VA_ARGS__) +#define ibdev_notice_ratelimited (ibdev, fmt, ...) \ + ibdev_level_ratelimited(ibdev_notice, ibdev, fmt, ##__VA_ARGS__) +#define ibdev_info_ratelimited (ibdev, fmt, ...) \ + ibdev_level_ratelimited(ibdev_info, ibdev, fmt, ##__VA_ARGS__) + +#if defined(CONFIG_DYNAMIC_DEBUG ) +/* descriptor check is first to prevent flooding with "callbacks suppressed" */ +#define ibdev_dbg_ratelimited (ibdev, fmt, ...) \ +do { \ + static DEFINE_RATELIMIT_STATE(_rs, \ + DEFAULT_RATELIMIT_INTERVAL, \ + DEFAULT_RATELIMIT_BURST); \ + DEFINE_DYNAMIC_DEBUG_METADATA(descriptor, fmt); \ + if (DYNAMIC_DEBUG_BRANCH(descriptor) && __ratelimit(&_rs)) \ + __dynamic_ibdev_dbg(&descriptor, ibdev, fmt, \ + ##__VA_ARGS__); \ +} while (0) +#else +__printf(2, 3) __cold +static inline +void ibdev_dbg_ratelimited (const struct ib_device *ibdev,const char *format, ...) {} +#endif +union ib_gid {u8 raw[16 ];struct {
Signature · 1 line
2.22.0
Re: [PATCH for-next v2 1/2] RDMA/core: Introduce ratelimited ibdev printk functions
Quoted 8 lines · Gal Pressman
Add ratelimited helpers to the ibdev_* printk functions. Implementation inspired by counterpart dev_*_ratelimited functions.
Signed-off-by Gal Pressman <[email protected]>--- include/rdma/ib_verbs.h | 42 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
Thanks,
PATCH v2 2/2 RDMA/efa: Rate limit admin queue error prints
In reply to Gal PressmanAdmin queue error prints should never happen unless something wrong happened to the device. However, in the unfortunate case that it does, we should take extra care not to flood the log with error messages.
--- drivers/infiniband/hw/efa/efa_com.c | 70 ++++++------ drivers/infiniband/hw/efa/efa_com_cmd.c | 136 ++++++++++++++---------- 2 files changed, 120 insertions(+), 86 deletions(-)
drivers/infiniband/hw/efa/efa_com.c +39 -31
@@ -109,17 +109,19 @@ static u32 efa_com_reg_read32(struct efa_com_dev *edev, u16 offset) }while (time_is_after_jiffies (exp_time));if (read_resp->req_id != mmio_read->seq_num) { -ibdev_err (edev->efa_dev, -"Reading register timed out. expected: req id[%u] offset[%#x] actual: req id[%u] offset[%#x]\n" , - mmio_read->seq_num, offset, read_resp->req_id, - read_resp->reg_off); +ibdev_err_ratelimited ( + edev->efa_dev, +"Reading register timed out. expected: req id[%u] offset[%#x] actual: req id[%u] offset[%#x]\n" , + mmio_read->seq_num, offset, read_resp->req_id, + read_resp->reg_off); err =EFA_MMIO_READ_INVALID ; goto out; }if (read_resp->reg_off != offset) { -ibdev_err (edev->efa_dev, -"Reading register failed: wrong offset provided\n" ); +ibdev_err_ratelimited ( + edev->efa_dev, +"Reading register failed: wrong offset provided\n" ); err =EFA_MMIO_READ_INVALID ; goto out; }
@@ -293,9 +295,10 @@ static struct efa_comp_ctx *efa_com_get_comp_ctx(struct efa_com_admin_queue *aq,u16 ctx_id = cmd_id & (aq->depth -1 );if (aq->comp_ctx[ctx_id].occupied && capture) { -ibdev_err (aq->efa_dev, -"Completion context for command_id %#x is occupied\n" , - cmd_id); +ibdev_err_ratelimited ( + aq->efa_dev, +"Completion context for command_id %#x is occupied\n" , + cmd_id);return NULL ; }
@@ -401,7 +404,7 @@ static struct efa_comp_ctx *efa_com_submit_admin_cmd(struct efa_com_admin_queuespin_lock (&aq->sq.lock);if (!test_bit (EFA_AQ_STATE_RUNNING_BIT , &aq->state)) { -ibdev_err (aq->efa_dev,"Admin queue is closed\n" ); +ibdev_err_ratelimited (aq->efa_dev,"Admin queue is closed\n" );spin_unlock (&aq->sq.lock);return ERR_PTR (-ENODEV ); }
@@ -519,8 +522,9 @@ static int efa_com_wait_and_process_admin_cq_polling(struct efa_comp_ctx *comp_cbreak ;if (time_is_before_jiffies (timeout)) { -ibdev_err (aq->efa_dev, -"Wait for completion (polling) timeout\n" ); +ibdev_err_ratelimited ( + aq->efa_dev, +"Wait for completion (polling) timeout\n" );/* EFA didn't have any completion */ atomic64_inc (&aq->stats.no_completion);
@@ -561,17 +565,19 @@ static int efa_com_wait_and_process_admin_cq_interrupts(struct efa_comp_ctx *comatomic64_inc (&aq->stats.no_completion);if (comp_ctx->status ==EFA_CMD_COMPLETED ) -ibdev_err (aq->efa_dev, -"The device sent a completion but the driver didn't receive any MSI-X interrupt for admin cmd %s(%d) status %d (ctx: 0x%p, sq producer: %d, sq consumer: %d, cq consumer: %d)\n" , -efa_com_cmd_str (comp_ctx->cmd_opcode), - comp_ctx->cmd_opcode, comp_ctx->status, - comp_ctx, aq->sq.pc, aq->sq.cc, aq->cq.cc); +ibdev_err_ratelimited ( + aq->efa_dev, +"The device sent a completion but the driver didn't receive any MSI-X interrupt for admin cmd %s(%d) status %d (ctx: 0x%p, sq producer: %d, sq consumer: %d, cq consumer: %d)\n" , +efa_com_cmd_str (comp_ctx->cmd_opcode), + comp_ctx->cmd_opcode, comp_ctx->status, + comp_ctx, aq->sq.pc, aq->sq.cc, aq->cq.cc);else -ibdev_err (aq->efa_dev, -"The device didn't send any completion for admin cmd %s(%d) status %d (ctx 0x%p, sq producer: %d, sq consumer: %d, cq consumer: %d)\n" , -efa_com_cmd_str (comp_ctx->cmd_opcode), - comp_ctx->cmd_opcode, comp_ctx->status, - comp_ctx, aq->sq.pc, aq->sq.cc, aq->cq.cc); +ibdev_err_ratelimited ( + aq->efa_dev, +"The device didn't send any completion for admin cmd %s(%d) status %d (ctx 0x%p, sq producer: %d, sq consumer: %d, cq consumer: %d)\n" , +efa_com_cmd_str (comp_ctx->cmd_opcode), + comp_ctx->cmd_opcode, comp_ctx->status, + comp_ctx, aq->sq.pc, aq->sq.cc, aq->cq.cc);clear_bit (EFA_AQ_STATE_RUNNING_BIT , &aq->state); err = -ETIME ;
@@ -633,10 +639,11 @@ int efa_com_cmd_exec(struct efa_com_admin_queue *aq, cmd->aq_common_descriptor.opcode); comp_ctx =efa_com_submit_admin_cmd (aq, cmd, cmd_size, comp, comp_size);if (IS_ERR (comp_ctx)) { -ibdev_err (aq->efa_dev, -"Failed to submit command %s (opcode %u) err %ld\n" , -efa_com_cmd_str (cmd->aq_common_descriptor.opcode), - cmd->aq_common_descriptor.opcode,PTR_ERR (comp_ctx)); +ibdev_err_ratelimited ( + aq->efa_dev, +"Failed to submit command %s (opcode %u) err %ld\n" , +efa_com_cmd_str (cmd->aq_common_descriptor.opcode), + cmd->aq_common_descriptor.opcode,PTR_ERR (comp_ctx));up (&aq->avail_cmds);return PTR_ERR (comp_ctx);
@@ -644,11 +651,12 @@ int efa_com_cmd_exec(struct efa_com_admin_queue *aq, err =efa_com_wait_and_process_admin_cq (comp_ctx, aq);if (err) -ibdev_err (aq->efa_dev, -"Failed to process command %s (opcode %u) comp_status %d err %d\n" , -efa_com_cmd_str (cmd->aq_common_descriptor.opcode), - cmd->aq_common_descriptor.opcode, - comp_ctx->comp_status, err); +ibdev_err_ratelimited ( + aq->efa_dev, +"Failed to process command %s (opcode %u) comp_status %d err %d\n" , +efa_com_cmd_str (cmd->aq_common_descriptor.opcode), + cmd->aq_common_descriptor.opcode, comp_ctx->comp_status, + err);up (&aq->avail_cmds);
drivers/infiniband/hw/efa/efa_com_cmd.c +81 -55
298 lines not on this page · raw message
Signature · 1 line
2.22.0
Quoted 11 lines · Gal Pressman
Hello,
This implements ratelimited ibdev printk functions, similar to their dev_*_ratelimited counterparts.
I've submitted the first patch before [1] without usage, I'm now adding
the corresponding EFA patch as well.cxgb4, hfi1, mlx4,
vmw_pvrdma, rdmavt and rxe could also benefit from this if they move to ibvdev prints:
[ snip ]
Thanks, series applied, and now the other drivers can convert to ibvdev prints ;-)
Signature · 3 lines
Doug Ledford <[email protected]> GPG KeyID: B826A3330E572FDD Fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD