ionic: make the device command timeout configurable
Subsystems (7)
PATCH v3 2/3 RDMA/ionic: Use the ionic command timeout for the admin queue
In reply to Nikhil P. RaoFrom: Arun Kumar Prakash <[email protected]>
Commit ec8ee714736e ("ionic: stretch heartbeat detection") notes that firmware can take more than 2 seconds to service a device or admin queue request, and is only expected to complete it within the ionic command timeout. The RDMA admin queue watchdog still uses 2 seconds, and when it fires it resets the RDMA LIF and every later verb fails with -ENODEV. Use the ionic command timeout, 5 seconds, instead. The AH create/destroy busy-wait keeps its fixed 2 second bound.
--- drivers/infiniband/hw/ionic/ionic_admin.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
drivers/infiniband/hw/ionic/ionic_admin.c +6 -3
@@ -18,8 +18,6 @@#define IONIC_ADMIN_BUSY_RETRY_COUNT 2000#define IONIC_ADMIN_BUSY_RETRY_MS 1 -/* admin queue will be considered failed if a command takes longer */ -#define IONIC_ADMIN_TIMEOUT (HZ * 2)#define IONIC_ADMIN_WARN (HZ / 8)/* will poll for admin cq to tolerate and report from missed event */
@@ -28,6 +26,11 @@/* work queue for polling the event queue and admin cq */ struct workqueue_struct *ionic_evt_workq; +static unsigned long ionic_admin_timeout (struct ionic_ibdev *dev) +{ +return HZ *ionic_cmd_timeout (dev->lif_cfg.lif); +} +static void ionic_admin_timedout (struct ionic_aq *aq) {struct ionic_ibdev *dev = aq->dev;
@@ -294,7 +297,7 @@ static void ionic_admin_dwork(struct work_struct *ws) }if (time_is_after_eq_jiffies (aq->stamp + -IONIC_ADMIN_TIMEOUT )) { +ionic_admin_timeout (dev))) {/* Timeout threshold not met */ ibdev_dbg (&dev->ibdev,"no progress after %ums\n" , (u32 )jiffies_to_msecs (jiffies - aq->stamp));
Signature · 1 line
2.43.0
v3 cover ionic: make the device command timeout configurable
ionic waits a fixed 5 seconds for device and admin queue commands, and the RDMA driver gives its admin queue 2 seconds. Some commands can take the firmware longer than that, for example while a PFC configuration change flaps the links. The firmware is alive but slow, yet the timed-out command and everything queued behind it fail, and an RDMA admin queue timeout resets the RDMA LIF so that later verbs fail with -ENODEV.
Patch 1 makes the timeout a per-device value without changing it. Patch 2 has the RDMA admin queue use the ionic command timeout instead of its own 2 seconds.
Patch 3 adds a permanent devlink parameter, cmd_timeout, on each PF. The firmware stores the value per PF in NVM, and the PF and its VFs read it at probe and after a reset.
Tested on an AMD Pensando adapter under KASAN and lockdep: get and set of the parameter, and the stored value read back after an FLR.
Patch 2 conflicts with "RDMA/ionic: Wait the device command timeout for admin queue commands", sent to rdma for-rc:
https://patch.msgid.link/[email protected]
When both are merged, keep this series' version: ionic_admin_timeout() replaces IONIC_ADMIN_TIMEOUT, and already defaults to 5 seconds.
This series was developed with the help of an LLM-based coding assistant (Claude).
---
Changes in v3:
- patch 3:
- register the parameter once at probe, on PFs only, instead of
unregistering it on firmware and PCI resets and registering it
again on restart
- get reports the value read at the last probe or reset, or stored by
a later set, instead of sending a device command
- keep the timeout in use when reading the stored value fails,
instead of falling back to the default
- read the stored value right after identify at probe and PCI reset,
so the remaining commands there use it
- rebase on net-next
- Link to v2: https://patch.msgid.link/[email protected]
Changes in v2:
- patch 3:
- comment in ionic_dl_cmd_timeout_get() on why it has no FW_RESET check
- changelog "after probe or reset" -> "after probe or PCI reset"
- Link to v1: https://patch.msgid.link/[email protected]
---
Arun Kumar Prakash (3):
ionic: make the command timeout per device RDMA/ionic: Use the ionic command timeout for the admin queue
ionic: add devlink param for command timeout
Documentation/networking/devlink/ionic.rst | 27 ++++++++ drivers/infiniband/hw/ionic/ionic_admin.c | 9 ++- drivers/net/ethernet/pensando/ionic/ionic.h | 10 ++- drivers/net/ethernet/pensando/ionic/ionic_api.h | 8 +++ drivers/net/ethernet/pensando/ionic/ionic_aux.c | 8 ++- .../net/ethernet/pensando/ionic/ionic_bus_pci.c | 3 + drivers/net/ethernet/pensando/ionic/ionic_dev.c | 70 ++++++++++++++++++-- drivers/net/ethernet/pensando/ionic/ionic_dev.h | 2 + .../net/ethernet/pensando/ionic/ionic_devlink.c | 77 ++++++++++++++++++++++ .../net/ethernet/pensando/ionic/ionic_ethtool.c | 12 ++-- drivers/net/ethernet/pensando/ionic/ionic_fw.c | 8 +-- drivers/net/ethernet/pensando/ionic/ionic_if.h | 8 +++ drivers/net/ethernet/pensando/ionic/ionic_lif.c | 11 ++-- drivers/net/ethernet/pensando/ionic/ionic_main.c | 17 ++--- 14 files changed, 236 insertions(+), 34 deletions(-)
---
base-commit: 44454974aef9c81949e5858ed3e8cbd08e104cad
change-id: 20261102-b4-ionic-cmd-timeout-c704d5304b4f
Best regards,
Signature · 1 line
Nikhil P. Rao <[email protected]>
PATCH v3 1/3 ionic: make the command timeout per device
From: Arun Kumar Prakash <[email protected]>
Device and admin queue command waits, and the firmware heartbeat check interval, use the compile-time DEVCMD_TIMEOUT. Make the timeout a per-device value, so that it can be made configurable through a devlink parameter, and export it to the RDMA driver through ionic_cmd_timeout().
No functional change.
--- drivers/net/ethernet/pensando/ionic/ionic.h | 3 ++- drivers/net/ethernet/pensando/ionic/ionic_api.h | 8 ++++++++ drivers/net/ethernet/pensando/ionic/ionic_aux.c | 8 +++++++- drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c | 1 + drivers/net/ethernet/pensando/ionic/ionic_dev.c | 11 ++++++----- drivers/net/ethernet/pensando/ionic/ionic_ethtool.c | 12 ++++++------ drivers/net/ethernet/pensando/ionic/ionic_fw.c | 8 ++++---- drivers/net/ethernet/pensando/ionic/ionic_lif.c | 10 +++++----- drivers/net/ethernet/pensando/ionic/ionic_main.c | 16 ++++++++-------- 9 files changed, 47 insertions(+), 30 deletions(-)
drivers/net/ethernet/pensando/ionic/ionic.h +2 -1
@@ -18,7 +18,7 @@ struct ionic_lif;#define PCI_DEVICE_ID_PENSANDO_IONIC_ETH_PF 0x1002#define PCI_DEVICE_ID_PENSANDO_IONIC_ETH_VF 0x1003 -#define DEVCMD_TIMEOUT 5 +#define IONIC_CMD_TIMEOUT_DEF 5#define IONIC_ADMINQ_TIME_SLICE msecs_to_jiffies(100)#define IONIC_PHC_UPDATE_NS 10000000000/* 10s in nanoseconds */
@@ -66,6 +66,7 @@ struct ionic {int watchdog_period;void *hwmon_caps_buf;dma_addr_t hwmon_caps_pa; +u8 cmd_timeout; };int ionic_adminq_post (struct ionic_lif *lif,struct ionic_admin_ctx *ctx);
drivers/net/ethernet/pensando/ionic/ionic_api.h +8 -0
@@ -86,6 +86,14 @@ int ionic_error_to_errno(enum ionic_status_code code);*/ void ionic_request_rdma_reset (struct ionic_lif *lif); +/** +* ionic_cmd_timeout - Get the device and admin queue command timeout +* @lif: Logical interface +* +* Return: the command timeout in seconds +*/ +u8 ionic_cmd_timeout (struct ionic_lif *lif); +/** * ionic_intr_alloc - Reserve a device interrupt * @lif: Logical interface
drivers/net/ethernet/pensando/ionic/ionic_aux.c +7 -1
@@ -239,7 +239,7 @@ void ionic_request_rdma_reset(struct ionic_lif *lif)mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_go (&ionic->idev, &cmd); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);mutex_unlock (&ionic->dev_cmd_lock);
@@ -247,3 +247,9 @@ void ionic_request_rdma_reset(struct ionic_lif *lif)pr_warn ("%s request_reset: error %d\n" , __func__, err); }EXPORT_SYMBOL_NS (ionic_request_rdma_reset,"NET_IONIC" ); + +u8 ionic_cmd_timeout (struct ionic_lif *lif) +{ +return lif->ionic->cmd_timeout; +} +EXPORT_SYMBOL_NS (ionic_cmd_timeout,"NET_IONIC" );
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +1 -0
@@ -239,6 +239,7 @@ static int ionic_setup_one(struct ionic *ionic)struct device *dev = ionic->dev;int err; + ionic->cmd_timeout =IONIC_CMD_TIMEOUT_DEF ;ionic_debugfs_add_dev (ionic);/* Setup PCI device */
drivers/net/ethernet/pensando/ionic/ionic_dev.c +6 -5
@@ -212,7 +212,7 @@ static void ionic_map_disc_cmb(struct ionic *ionic)mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_discover_cmb (idev); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);if (!err) { sz =min (sizeof (ident->cmb_layout),sizeof (idev->dev_cmd_regs->data));
@@ -590,10 +590,11 @@ int ionic_heartbeat_check(struct ionic *ionic)return -ENXIO ;/* Because of some variability in the actual FW heartbeat, we -* wait longer than the DEVCMD_TIMEOUT before checking again. +* wait longer than the command timeout before checking again. */ last_check_time = idev->last_hb_time; -if (time_before (check_time, last_check_time +DEVCMD_TIMEOUT *2 *HZ )) +if (time_before (check_time, + last_check_time + ionic->cmd_timeout *2 *HZ ))return 0 ; fw_hb =ioread32 (&idev->dev_info_regs->fw_heartbeat);
@@ -795,7 +796,7 @@ int ionic_set_vf_config(struct ionic *ionic, int vf,mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_go (&ionic->idev, &cmd); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);mutex_unlock (&ionic->dev_cmd_lock);return err;
@@ -812,7 +813,7 @@ void ionic_vf_start(struct ionic *ionic)return ;ionic_dev_cmd_go (&ionic->idev, &cmd); -ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); +ionic_dev_cmd_wait (ionic, ionic->cmd_timeout); }/* LIF commands */
drivers/net/ethernet/pensando/ionic/ionic_ethtool.c +6 -6
@@ -351,7 +351,7 @@ static int ionic_set_link_ksettings(struct net_device *netdev,if (ks->base.autoneg != idev->port_info->config.an_enable) {mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_port_autoneg (idev, ks->base.autoneg); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);mutex_unlock (&ionic->dev_cmd_lock);if (err)return err;
@@ -361,7 +361,7 @@ static int ionic_set_link_ksettings(struct net_device *netdev,if (ks->base.speed !=le32_to_cpu (idev->port_info->config.speed)) {mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_port_speed (idev, ks->base.speed); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);mutex_unlock (&ionic->dev_cmd_lock);if (err)return err;
@@ -411,7 +411,7 @@ static int ionic_set_pauseparam(struct net_device *netdev,mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_port_pause (&lif->ionic->idev, requested_pause); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);mutex_unlock (&ionic->dev_cmd_lock);if (err)return err;
@@ -479,7 +479,7 @@ static int ionic_set_fecparam(struct net_device *netdev,if (fec_type != lif->ionic->idev.port_info->config.fec_type) {mutex_lock (&lif->ionic->dev_cmd_lock);ionic_dev_cmd_port_fec (&lif->ionic->idev, fec_type); - ret =ionic_dev_cmd_wait (lif->ionic,DEVCMD_TIMEOUT ); + ret =ionic_dev_cmd_wait (lif->ionic, lif->ionic->cmd_timeout);mutex_unlock (&lif->ionic->dev_cmd_lock); }
@@ -1131,11 +1131,11 @@ static int ionic_nway_reset(struct net_device *netdev)mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_port_state (&ionic->idev,IONIC_PORT_ADMIN_STATE_DOWN ); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);if (!err) {ionic_dev_cmd_port_state (&ionic->idev,IONIC_PORT_ADMIN_STATE_UP ); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout); }mutex_unlock (&ionic->dev_cmd_lock);
drivers/net/ethernet/pensando/ionic/ionic_fw.c +4 -4
@@ -75,7 +75,7 @@ static int ionic_fw_status_long_wait(struct ionic *ionic, do { mutex_lock(&ionic->dev_cmd_lock); ionic_dev_cmd_go(&ionic->idev, &cmd); - err = ionic_dev_cmd_wait(ionic, DEVCMD_TIMEOUT); + err = ionic_dev_cmd_wait(ionic, ionic->cmd_timeout); mutex_unlock(&ionic->dev_cmd_lock); msleep(20);
@@ -135,7 +135,7 @@ int ionic_firmware_update(struct ionic_lif *lif, const struct firmware *fw,ionic_dev_cmd_firmware_download (idev, offsetof(union ionic_dev_cmd_regs , data), offset, copy_sz); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);mutex_unlock (&ionic->dev_cmd_lock);if (err) { netdev_err(netdev,
@@ -155,7 +155,7 @@ int ionic_firmware_update(struct ionic_lif *lif, const struct firmware *fw,mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_firmware_install (idev); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);ionic_dev_cmd_comp (idev, (union ionic_dev_cmd_comp *)&comp); fw_slot = comp.fw_control.slot;mutex_unlock (&ionic->dev_cmd_lock);
@@ -176,7 +176,7 @@ int ionic_firmware_update(struct ionic_lif *lif, const struct firmware *fw,mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_firmware_activate (idev, fw_slot); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);mutex_unlock (&ionic->dev_cmd_lock);if (err) {NL_SET_ERR_MSG_MOD (extack,"Failed to start firmware select" );
drivers/net/ethernet/pensando/ionic/ionic_lif.c +5 -5
@@ -3382,7 +3382,7 @@ static void ionic_lif_reset(struct ionic_lif *lif)mutex_lock (&lif->ionic->dev_cmd_lock);ionic_dev_cmd_lif_reset (idev, lif->index); -ionic_dev_cmd_wait (lif->ionic,DEVCMD_TIMEOUT ); +ionic_dev_cmd_wait (lif->ionic, lif->ionic->cmd_timeout);mutex_unlock (&lif->ionic->dev_cmd_lock); }
@@ -3589,7 +3589,7 @@ static int ionic_lif_adminq_init(struct ionic_lif *lif)mutex_lock (&lif->ionic->dev_cmd_lock);ionic_dev_cmd_adminq_init (idev, qcq, lif->index, qcq->intr.index); - err =ionic_dev_cmd_wait (lif->ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (lif->ionic, lif->ionic->cmd_timeout);ionic_dev_cmd_comp (idev, (union ionic_dev_cmd_comp *)&comp);mutex_unlock (&lif->ionic->dev_cmd_lock);if (err) {
@@ -3749,7 +3749,7 @@ int ionic_lif_init(struct ionic_lif *lif)mutex_lock (&lif->ionic->dev_cmd_lock);ionic_dev_cmd_lif_init (idev, lif->index, lif->info_pa); - err =ionic_dev_cmd_wait (lif->ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (lif->ionic, lif->ionic->cmd_timeout);ionic_dev_cmd_comp (idev, (union ionic_dev_cmd_comp *)&comp);mutex_unlock (&lif->ionic->dev_cmd_lock);if (err)
@@ -3937,7 +3937,7 @@ static void ionic_lif_queue_identify(struct ionic_lif *lif)mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_queue_identify (idev, lif->lif_type, qtype, ionic_qtype_versions[qtype]); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);if (!err) { qti->version =readb (&q_ident->version); qti->supported =readb (&q_ident->supported);
@@ -4003,7 +4003,7 @@ int ionic_lif_identify(struct ionic *ionic, u8 lif_type,mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_lif_identify (idev, lif_type,IONIC_IDENTITY_VERSION_1 ); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);memcpy_fromio (lid, &idev->dev_cmd_regs->data, sz);mutex_unlock (&ionic->dev_cmd_lock);if (err)
drivers/net/ethernet/pensando/ionic/ionic_main.c +8 -8
@@ -439,7 +439,7 @@ int ionic_adminq_wait(struct ionic_lif *lif, struct ionic_admin_ctx *ctx, } time_start = jiffies; - time_limit = time_start +HZ * (ulong )DEVCMD_TIMEOUT ; + time_limit = time_start +HZ * (ulong )lif->ionic->cmd_timeout; do { remaining = wait_for_completion_timeout(&ctx->work, IONIC_ADMINQ_TIME_SLICE);
@@ -635,7 +635,7 @@ int ionic_identify(struct ionic *ionic)memcpy_toio (&idev->dev_cmd_regs->data, &ident->drv, sz);ionic_dev_cmd_identify (idev,IONIC_DEV_IDENTITY_VERSION_2 ); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);if (!err) { sz =min (sizeof (ident->dev),sizeof (idev->dev_cmd_regs->data));memcpy_fromio (&ident->dev, &idev->dev_cmd_regs->data, sz);
@@ -679,7 +679,7 @@ int ionic_init(struct ionic *ionic)mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_init (idev); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);mutex_unlock (&ionic->dev_cmd_lock);return err;
@@ -695,7 +695,7 @@ int ionic_reset(struct ionic *ionic)mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_reset (idev); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);mutex_unlock (&ionic->dev_cmd_lock);return err;
@@ -711,7 +711,7 @@ int ionic_port_identify(struct ionic *ionic)mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_port_identify (idev); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);if (!err) { sz =min (sizeof (ident->port),sizeof (idev->dev_cmd_regs->data));memcpy_fromio (&ident->port, &idev->dev_cmd_regs->data, sz);
@@ -756,10 +756,10 @@ int ionic_port_init(struct ionic *ionic)memcpy_toio (&idev->dev_cmd_regs->data, &ident->port.config, sz);ionic_dev_cmd_port_init (idev); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);ionic_dev_cmd_port_state (&ionic->idev,IONIC_PORT_ADMIN_STATE_UP ); -ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); +ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);mutex_unlock (&ionic->dev_cmd_lock);if (err) {
@@ -786,7 +786,7 @@ int ionic_port_reset(struct ionic *ionic)if (ionic_is_fw_running (idev)) {mutex_lock (&ionic->dev_cmd_lock);ionic_dev_cmd_port_reset (idev); - err =ionic_dev_cmd_wait (ionic,DEVCMD_TIMEOUT ); + err =ionic_dev_cmd_wait (ionic, ionic->cmd_timeout);mutex_unlock (&ionic->dev_cmd_lock); }
Signature · 1 line
2.43.0
PATCH v3 3/3 ionic: add devlink param for command timeout
In reply to Nikhil P. RaoFrom: Arun Kumar Prakash <[email protected]>
Some device and admin commands can take firmware longer than the driver allows to complete, for example while a PFC configuration change flaps the links. The firmware is alive but slow, yet the timed-out command and everything queued behind it fail.
Make the timeout configurable so it can be tuned to the firmware behaviour seen in production.
Add a permanent devlink parameter, cmd_timeout (u8, seconds), stored in device NVM via a new device attribute, on firmware that advertises the new IONIC_DEV_CAP_CMD_TIMEOUT capability. A non-zero value replaces the device command and admin queue timeouts, and the firmware heartbeat check interval derived from them; 0 restores the default. A new value takes effect on the next driver probe or device reset.
Back-date the initial heartbeat check time by the largest possible interval rather than the watchdog period, so that a larger value does not delay the first heartbeat sample after probe or PCI reset.
Accept 0, or 5 to 255 seconds (the u8 range). Values below the 5 second default are rejected, as the parameter is only meant to lengthen the timeouts. On unresponsive firmware, a larger value also lets callers block for longer: each device or admin queue command can wait up to this value, other commands wait behind it, and RDMA verbs such as QP and CQ create can block for several times this value. A large setting may need a larger kernel.hung_task_timeout_secs.
--- Documentation/networking/devlink/ionic.rst | 27 ++++++++ drivers/net/ethernet/pensando/ionic/ionic.h | 7 ++ drivers/net/ethernet/pensando/ionic/ionic_aux.c | 2 +- .../net/ethernet/pensando/ionic/ionic_bus_pci.c | 4 +- drivers/net/ethernet/pensando/ionic/ionic_dev.c | 61 ++++++++++++++++- drivers/net/ethernet/pensando/ionic/ionic_dev.h | 2 + .../net/ethernet/pensando/ionic/ionic_devlink.c | 77 ++++++++++++++++++++++ drivers/net/ethernet/pensando/ionic/ionic_if.h | 8 +++ drivers/net/ethernet/pensando/ionic/ionic_lif.c | 1 + drivers/net/ethernet/pensando/ionic/ionic_main.c | 3 +- 10 files changed, 187 insertions(+), 5 deletions(-)
Documentation/networking/devlink/ionic.rst +27 -0
@@ -7,6 +7,33 @@ ionic devlink support This document describes the devlink features implemented by the ``ionic`` device driver. +Parameters +========== + +The ``ionic`` driver implements the following driver-specific parameters. + +.. list-table:: Driver-specific parameters implemented + :widths: 5 5 5 85 + + * - Name + - Type + - Mode + - Description + * - ``cmd_timeout`` + - u8 + - permanent + - Command timeout in seconds for this PCI function and its VFs, 0, or + 5 (the default) to 255; values below the default are rejected, as the + parameter is for cases where the default does not suffice. It bounds + device and admin queue commands and the RDMA admin queue watchdog, and + the firmware heartbeat stall check runs at twice this interval. + Setting 0 makes the driver use the default. The value is stored in + device NVM and takes effect on the next driver probe or device reset. + On unresponsive firmware, a larger value lets callers block for + longer; a large setting may need a larger + kernel.hung_task_timeout_secs. Only available on physical functions + whose firmware supports it. + Info versions =============
drivers/net/ethernet/pensando/ionic/ionic.h +7 -0
@@ -67,8 +67,15 @@ struct ionic {void *hwmon_caps_buf;dma_addr_t hwmon_caps_pa;u8 cmd_timeout; +int cmd_timeout_param; }; +static inline bool ionic_cmd_timeout_supported (struct ionic *ionic) +{ +return ionic->ident.dev.capabilities & +cpu_to_le64 (IONIC_DEV_CAP_CMD_TIMEOUT ); +} +int ionic_adminq_post (struct ionic_lif *lif,struct ionic_admin_ctx *ctx);int ionic_adminq_wait (struct ionic_lif *lif,struct ionic_admin_ctx *ctx,const int err,const bool do_msg);
drivers/net/ethernet/pensando/ionic/ionic_aux.c +1 -1
@@ -250,6 +250,6 @@ EXPORT_SYMBOL_NS(ionic_request_rdma_reset, "NET_IONIC");u8 ionic_cmd_timeout (struct ionic_lif *lif) { -return lif->ionic->cmd_timeout; +return READ_ONCE (lif->ionic->cmd_timeout); }EXPORT_SYMBOL_NS (ionic_cmd_timeout,"NET_IONIC" );
drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +3 -1
@@ -239,7 +239,6 @@ static int ionic_setup_one(struct ionic *ionic)struct device *dev = ionic->dev;int err; - ionic->cmd_timeout =IONIC_CMD_TIMEOUT_DEF ;ionic_debugfs_add_dev (ionic);/* Setup PCI device */
@@ -273,6 +272,7 @@ static int ionic_setup_one(struct ionic *ionic)dev_err (dev,"Cannot identify device: %d, aborting\n" , err); goto err_out_teardown; } +ionic_update_cmd_timeout (ionic);ionic_debugfs_add_ident (ionic);ionic_map_cmb (ionic);
@@ -323,6 +323,8 @@ static int ionic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) ionic->dev = dev;pci_set_drvdata (pdev, ionic);mutex_init (&ionic->dev_cmd_lock); + ionic->cmd_timeout =IONIC_CMD_TIMEOUT_DEF ; + ionic->cmd_timeout_param = -EOPNOTSUPP ; err =ida_alloc (&ionic_ida,GFP_KERNEL );if (err <0 )
drivers/net/ethernet/pensando/ionic/ionic_dev.c +59 -2
79 lines not on this page · raw message
drivers/net/ethernet/pensando/ionic/ionic_dev.h +2 -0
@@ -348,6 +348,8 @@ void ionic_dev_cmd_comp(struct ionic_dev *idev, union ionic_dev_cmd_comp *comp);void ionic_dev_cmd_identify (struct ionic_dev *idev,u8 ver);void ionic_dev_cmd_init (struct ionic_dev *idev);void ionic_dev_cmd_reset (struct ionic_dev *idev); +void ionic_update_cmd_timeout (struct ionic *ionic); +int ionic_set_cmd_timeout (struct ionic *ionic,u8 timeout);void ionic_dev_cmd_port_identify (struct ionic_dev *idev);void ionic_dev_cmd_port_init (struct ionic_dev *idev);
drivers/net/ethernet/pensando/ionic/ionic_devlink.c +77 -0
94 lines not on this page · raw message
drivers/net/ethernet/pensando/ionic/ionic_if.h +8 -0
@@ -280,12 +280,14 @@ union ionic_drv_identity {* @IONIC_DEV_CAP_DISC_CMB: Device supports CMB discovery operations * @IONIC_DEV_CAP_HWMON: Device supports hardware monitoring * @IONIC_DEV_CAP_EXTRA_STATS: Device supports extra stats schema +* @IONIC_DEV_CAP_CMD_TIMEOUT: Device supports the command timeout attribute */ enum ionic_dev_capability {IONIC_DEV_CAP_VF_CTRL =BIT (0 ),IONIC_DEV_CAP_DISC_CMB =BIT (1 ),IONIC_DEV_CAP_HWMON =BIT (3 ),IONIC_DEV_CAP_EXTRA_STATS =BIT (4 ), +IONIC_DEV_CAP_CMD_TIMEOUT =BIT (5 ), };/**
@@ -1784,11 +1786,13 @@ enum ionic_dev_state {* @IONIC_DEV_ATTR_STATE: Device state attribute * @IONIC_DEV_ATTR_NAME: Device name attribute * @IONIC_DEV_ATTR_FEATURES: Device feature attributes +* @IONIC_DEV_ATTR_CMD_TIMEOUT: Command timeout */ enum ionic_dev_attr {IONIC_DEV_ATTR_STATE =0 ,IONIC_DEV_ATTR_NAME =1 ,IONIC_DEV_ATTR_FEATURES =2 , +IONIC_DEV_ATTR_CMD_TIMEOUT =3 , };/**
@@ -1799,6 +1803,7 @@ enum ionic_dev_attr {* @state: Device state (enum ionic_dev_state) * @name: The bus info, e.g. PCI slot-device-function, 0 terminated * @features: Device features +* @cmd_timeout: Command timeout in seconds, 0 for the default * @rsvd2: reserved byte(s) */ struct ionic_dev_setattr_cmd {
@@ -1809,6 +1814,7 @@ struct ionic_dev_setattr_cmd {u8 state;char name[IONIC_IFNAMSIZ ];__le64 features; +u8 cmd_timeout;u8 rsvd2[60 ]; } __packed; };
@@ -1848,6 +1854,7 @@ struct ionic_dev_getattr_cmd {* @status: Status of the command (enum ionic_status_code) * @rsvd: reserved byte(s) * @features: Device features +* @cmd_timeout: Command timeout in seconds, 0 for the default * @rsvd2: reserved byte(s) * @color: Color bit */
@@ -1856,6 +1863,7 @@ struct ionic_dev_getattr_comp {u8 rsvd[3 ];union {__le64 features; +u8 cmd_timeout;u8 rsvd2[11 ]; } __packed;u8 color;
drivers/net/ethernet/pensando/ionic/ionic_lif.c +1 -0
@@ -3495,6 +3495,7 @@ static void ionic_lif_handle_fw_up(struct ionic_lif *lif) err =ionic_identify (ionic);if (err) goto err_out; +ionic_update_cmd_timeout (ionic); err =ionic_port_identify (ionic);if (err) goto err_out;
drivers/net/ethernet/pensando/ionic/ionic_main.c +2 -1
@@ -439,7 +439,8 @@ int ionic_adminq_wait(struct ionic_lif *lif, struct ionic_admin_ctx *ctx, } time_start = jiffies; - time_limit = time_start +HZ * (ulong )lif->ionic->cmd_timeout; + time_limit = time_start + +HZ * (ulong )READ_ONCE (lif->ionic->cmd_timeout); do { remaining = wait_for_completion_timeout(&ctx->work, IONIC_ADMINQ_TIME_SLICE);
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH net-next v3 1/3] ionic: make the command timeout per device
In reply to Nikhil P. RaoSashiko has reviewed this patch and found no issues.
Quoted 8 lines
From: Arun Kumar Prakash <[email protected]>
Device and admin queue command waits, and the firmware heartbeat check interval, use the compile-time
DEVCMD_TIMEOUT. Make the timeout a per-device value, so that it can be made configurable through a devlink parameter, and export it to the RDMA driver throughionic_cmd_timeout().
Quoted 5 lines
No functional change.
Signed-off-by Arun Kumar Prakash <[email protected]>Reviewed-by Brett Creeley <[email protected]>Signed-off-by Nikhil P. Rao <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewRe: [PATCH net-next v3 2/3] RDMA/ionic: Use the ionic command timeout for the admin queue
In reply to Nikhil P. RaoSashiko has reviewed this patch and found no issues.
Quoted 10 lines
From: Arun Kumar Prakash <[email protected]>
Commit
ec8ee714736e("ionic: stretch heartbeat detection") notes that firmware can take more than 2 seconds to service a device or admin queue request, and is only expected to complete it within the ionic command timeout. The RDMA admin queue watchdog still uses 2 seconds, and when it fires it resets the RDMA LIF and every later verb fails with-ENODEV. Use the ionic command timeout, 5 seconds, instead. The AH create/destroy busy-wait keeps its fixed 2 second bound.
Quoted 4 lines
Signed-off-by Arun Kumar Prakash <[email protected]>Reviewed-by Brett Creeley <[email protected]>Acked-by Allen Hubbe <[email protected]>Signed-off-by Nikhil P. Rao <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewRe: [PATCH net-next v3 3/3] ionic: add devlink param for command timeout
In reply to Nikhil P. RaoSashiko has reviewed this patch and found no issues.
Quoted 17 lines
From: Arun Kumar Prakash <[email protected]>
Some device and admin commands can take firmware longer than the driver allows to complete, for example while a PFC configuration change flaps the links. The firmware is alive but slow, yet the timed-out command and everything queued behind it fail.
Make the timeout configurable so it can be tuned to the firmware behaviour seen in production.
Add a permanent devlink parameter,
cmd_timeout(u8, seconds), stored in device NVM via a new device attribute, on firmware that advertises the newIONIC_DEV_CAP_CMD_TIMEOUTcapability. A non-zero value replaces the device command and admin queue timeouts, and the firmware heartbeat check interval derived from them; 0 restores the default. A new value [ ... ]
Quoted 4 lines
Signed-off-by Arun Kumar Prakash <[email protected]>Reviewed-by Brett Creeley <[email protected]>Co-developed-by Nikhil P. Rao <[email protected]>Signed-off-by Nikhil P. Rao <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3