Skip to contentSkip to conversation

nvme: sanitize KATO setting

PATCH Superseded Hannes Reinecke 3 participants 6 messages Mar 11, 2021 – Mar 15, 2021 NVMe

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover nvme: sanitize KATO setting

Hi all,

as reserved tag allocation issues are solved with a different patchset, here are just the bits to sanitize and expose the KATO value to sysfs. Exposing it to sysfs will allow userspace to identify persistent discovery controllers (which will have a non-zero KATO value); a patch for that has been sent to the nvme-cli github repository.

As usual, comments and reviews are welcome.

Hannes Reinecke (2):
nvme: sanitize KATO setting
nvme: add 'kato' sysfs attribute

 drivers/nvme/host/core.c    |  8 +++++---
 drivers/nvme/host/fabrics.c |  4 +---
 drivers/nvme/host/nvme.h    | 16 +++++++++++++++-
 3 files changed, 21 insertions(+), 7 deletions(-)
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#1 lore raw

PATCH v3 1/2 nvme: sanitize KATO setting

According to the NVMe base spec the KATO commands should be sent at half of the KATO interval, to properly account for round-trip times.
As we now will only ever send one KATO command per connection we can easily use the recommended values.
This also fixes a potential issue where the request timeout for the KATO command does not match the value in the connect command, which might be causing spurious connection drops from the target.

---
 drivers/nvme/host/core.c    |  6 +++---
 drivers/nvme/host/fabrics.c |  4 +---
 drivers/nvme/host/nvme.h    | 16 +++++++++++++++-
 3 files changed, 19 insertions(+), 7 deletions(-)
drivers/nvme/host/core.c +3 -3
index 82ad5eef9d0c..b56903982b27 100644
@@ -1223,7 +1223,7 @@ static void nvme_keep_alive_end_io(struct request *rq, blk_status_t status)
 		startka = true;
 	spin_unlock_irqrestore(&ctrl->lock, flags);
 	if (startka)
-		queue_delayed_work(nvme_wq, &ctrl->ka_work, ctrl->kato * HZ);
+		nvme_queue_keep_alive_work(ctrl);
 }
 
 static int nvme_keep_alive(struct nvme_ctrl *ctrl)
@@ -1253,7 +1253,7 @@ static void nvme_keep_alive_work(struct work_struct *work)
 		dev_dbg(ctrl->device,
 			"reschedule traffic based keep-alive timer\n");
 		ctrl->comp_seen = false;
-		queue_delayed_work(nvme_wq, &ctrl->ka_work, ctrl->kato * HZ);
+		nvme_queue_keep_alive_work(ctrl);
 		return;
 	}
 
@@ -1270,7 +1270,7 @@ static void nvme_start_keep_alive(struct nvme_ctrl *ctrl)
 	if (unlikely(ctrl->kato == 0))
 		return;
 
-	queue_delayed_work(nvme_wq, &ctrl->ka_work, ctrl->kato * HZ);
+	nvme_queue_keep_alive_work(ctrl);
 }
 
 void nvme_stop_keep_alive(struct nvme_ctrl *ctrl)
drivers/nvme/host/fabrics.c +1 -3
index 604ab0e5a2ad..13c2747e3d00 100644
@@ -379,10 +379,8 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 
 	/*
 	 * Set keep-alive timeout in seconds granularity (ms * 1000)
-	 * and add a grace period for controller kato enforcement
 	 */
-	cmd.connect.kato = ctrl->kato ?
-		cpu_to_le32((ctrl->kato + NVME_KATO_GRACE) * 1000) : 0;
+	cmd.connect.kato = cpu_to_le32(ctrl->kato * 1000);
 
 	if (ctrl->opts->disable_sqflow)
 		cmd.connect.cattr |= NVME_CONNECT_DISABLE_SQFLOW;
drivers/nvme/host/nvme.h +15 -1
index 07b34175c6ce..cc74b9f8d5ff 100644
@@ -27,7 +27,15 @@ extern unsigned int admin_timeout;
 #define NVME_ADMIN_TIMEOUT	(admin_timeout * HZ)
 
 #define NVME_DEFAULT_KATO	5
-#define NVME_KATO_GRACE		10
+
+/*
+ * Recommended frequency for KATO commands
+ *
+ * NVMe 1.4 section 7.12.1 states:
+ * The host should send Keep Alive commands at half of the
+ * Keep Alive Timeout accounting for transport roundtrip times [..].
+ */
+#define NVME_KATO_DELAY(k)	(((k) > 1)?(k) >> 1:(k))
 
 #ifdef CONFIG_ARCH_NO_SG_CHAIN
 #define  NVME_INLINE_SG_CNT  0
@@ -583,6 +591,12 @@ static inline bool nvme_is_aen_req(u16 qid, __u16 command_id)
 	return !qid && command_id >= NVME_AQ_BLK_MQ_DEPTH;
 }
 
+static inline void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)
+{
+	queue_delayed_work(nvme_wq, &ctrl->ka_work,
+			   NVME_KATO_DELAY(ctrl->kato * HZ));
+}
+
 void nvme_complete_rq(struct request *req);
 blk_status_t nvme_host_path_error(struct request *req);
 bool nvme_cancel_request(struct request *req, void *data, bool reserved);
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#2 lore raw

drivers/nvme/host/nvme.h extern unsigned int admin_timeout; · line 38

Quoted 8 lines
drivers/nvme/host/nvme.h +8 -0
+/*
+ * Recommended frequency for KATO commands
+ *
+ * NVMe 1.4 section 7.12.1 states:
+ * The host should send Keep Alive commands at half of the
+ * Keep Alive Timeout accounting for transport roundtrip times [..].
+ */
+#define NVME_KATO_DELAY(k)	(((k) > 1)?(k) >> 1:(k))

This should have been an inline if added at all, and use proper formatting, as well as avoiding the shift obsfucation.

I think the best is to just move this into the only caller, where we also know that ctrl->kato can't be 0 due to the check in nvme_start_keep_alive:

/*
* Recommended frequency for KATO commands per NVMe 1.4 section 7.12.1:
*
* The host should send Keep Alive commands at half of the
* Keep Alive Timeout accounting for transport roundtrip times [..].
*/
static inline void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)

{
	queue_delayed_work(nvme_wq, &ctrl->ka_work, ctrl->kato * HZ / 2);
}

Also please keep this helper inside of core.c.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#3 lore raw
Quoted 23 lines · Christoph Hellwig

drivers/nvme/host/nvme.h extern unsigned int admin_timeout; · line 38

Quoted 8 lines
drivers/nvme/host/nvme.h +8 -0
+/*
+ * Recommended frequency for KATO commands
+ *
+ * NVMe 1.4 section 7.12.1 states:
+ * The host should send Keep Alive commands at half of the
+ * Keep Alive Timeout accounting for transport roundtrip times [..].
+ */
+#define NVME_KATO_DELAY(k)	(((k) > 1)?(k) >> 1:(k))

This should have been an inline if added at all, and use proper formatting, as well as avoiding the shift obsfucation.

I think the best is to just move this into the only caller, where we also know that ctrl->kato can't be 0 due to the check in nvme_start_keep_alive:

/*
* Recommended frequency for KATO commands per NVMe 1.4 section 7.12.1:
*
* The host should send Keep Alive commands at half of the
* Keep Alive Timeout accounting for transport roundtrip times [..].
*/
static inline void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)

Quoted 5 lines
{
	queue_delayed_work(nvme_wq, &ctrl->ka_work, ctrl->kato * HZ / 2);
}

Also please keep this helper inside of core.c.

Agreed.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#4 lore raw

PATCH v3 2/2 nvme: add 'kato' sysfs attribute

In reply to Hannes Reinecke

Add a 'kato' controller sysfs attribute to display the current keep-alive timeout value (if any). This allows userspace to identify persistent discovery controllers, as these will have a non-zero KATO value.

---
 drivers/nvme/host/core.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/nvme/host/core.c +2 -0
index b56903982b27..206e6f883ca8 100644
@@ -3540,6 +3540,7 @@ nvme_show_int_function(cntlid);
 nvme_show_int_function(numa_node);
 nvme_show_int_function(queue_count);
 nvme_show_int_function(sqsize);
+nvme_show_int_function(kato);
 
 static ssize_t nvme_sysfs_delete(struct device *dev,
 				struct device_attribute *attr, const char *buf,
@@ -3706,6 +3707,7 @@ static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_hostid.attr,
 	&dev_attr_ctrl_loss_tmo.attr,
 	&dev_attr_reconnect_delay.attr,
+	&dev_attr_kato.attr,
 	NULL
 };
 
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme