Skip to contentSkip to conversation

Miscellaneous changes for nvme-tcp v2

PATCH New Mikhail Skorzhinskii 3 participants 7 messages Jul 4, 2019 – Jul 9, 2019 NVMe

lore
Share & tools
lore
#0 lore raw

v2 cover Miscellaneous changes for nvme-tcp v2

A few unrelated changes for NVMoF subsystem, that was found during testing of NVMoF TCP.

patch 2: found during extensive testing on various filesystems patch 3: found during testing on debug kernels with VM_DEBUG enabled

v2
- Moved hint patch as separate patch;
- Fix commit description in stable writes patch;
- Simplify check for SLAB check;
- Remove unnecessary replacments of sendpage calls;

Mike Playle (1):
nvme-tcp: set the STABLE_WRITES flag when data digests are enabled

Mikhail Skorzhinskii (1):
nvme-tcp: don't use sendpage for SLAB pages

 drivers/nvme/host/core.c | 5 +++++
 drivers/nvme/host/tcp.c  | 7 ++++++-
 2 files changed, 11 insertions(+), 1 deletion(-)
Signature · 1 line
2.16.4
#1 lore raw

PATCH v2 1/2 nvme-tcp: set the STABLE_WRITES flag when data digests are enabled

There was a few false alarms sighted on target side about wrong data digest while performing high throughput load to XFS filesystem shared through NVMoF TCP.

This flag tells the rest of the kernel to ensure that the data buffer does not change while the write is in flight. It incurs a performance penalty, so only enable it when it is actually needed, i.e. when we are calculating data digests.

Although even with this change in place, ext2 users can steel experience false positives, as ext2 is not respecting this flag. This may be apply to vfat as well.

Signed-off-by Mikhail Skorzhinskii <mskorzhinskiy at solarflare.com>
Signed-off-by Mike Playle <mplayle at solarflare.com>

---

https://lwn.net/Articles/528031/
https://www.redhat.com/archives/dm-devel/2015-May/msg00158.html

---
 drivers/nvme/host/core.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/nvme/host/core.c +5 -0
index 5417110cbf1b..f4340dc1d399 100644
@@ -11,6 +11,7 @@
 #include <linux/hdreg.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/backing-dev.h>
 #include <linux/list_sort.h>
 #include <linux/slab.h>
 #include <linux/types.h>
@@ -3304,6 +3305,10 @@ static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 		goto out_free_ns;
 	}
 
+	if (ctrl->opts->data_digest)
+		ns->queue->backing_dev_info->capabilities
+			|= BDI_CAP_STABLE_WRITES;
+
 	blk_queue_flag_set(QUEUE_FLAG_NONROT, ns->queue);
 	if (ctrl->ops->flags & NVME_F_PCI_P2PDMA)
 		blk_queue_flag_set(QUEUE_FLAG_PCI_P2PDMA, ns->queue);
Signature · 1 line
2.16.4
maintainer
#2 lore raw

PATCH v2 1/2 nvme-tcp: set the STABLE_WRITES flag when data digests are enabled

Reviewed-by Sagi Grimberg <sagi at grimberg.me>
#4 lore raw

PATCH v2 2/2 nvme-tcp: don't use sendpage for SLAB pages

In reply to Mikhail Skorzhinskii

According to commit a10674bf2406 ("tcp: detecting the misuse of .sendpage for Slab objects") and previous discussion, tcp_sendpage should not be used for pages that is managed by SLAB, as SLAB is not taking page reference counters into consideration.

This change prevents sendpage calls for payload sending too, although this is true only for admin commands, so actual data transfer performance should be untouched.

Signed-off-by Mikhail Skorzhinskii <mskorzhinskiy at solarflare.com>
---
 drivers/nvme/host/tcp.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +6 -1
index 08a2501b9357..c4acdbdca520 100644
@@ -860,7 +860,12 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)
 		else
 			flags |= MSG_MORE;
 
-		ret = kernel_sendpage(queue->sock, page, offset, len, flags);
+		/* can't zcopy slab pages */
+		if (unlikely(PageSlab(page))) {
+			ret = sock_no_sendpage(queue->sock, page, offset, len, flags);
+		} else {
+			ret = kernel_sendpage(queue->sock, page, offset, len, flags);
+		}
 		if (ret <= 0)
 			return ret;
 
Signature · 1 line
2.16.4
maintainer
#5 lore raw

PATCH v2 2/2 nvme-tcp: don't use sendpage for SLAB pages

Reviewed-by Sagi Grimberg <sagi at grimberg.me>
#6 lore raw

PATCH v2 2/2 nvme-tcp: don't use sendpage for SLAB pages

In reply to Mikhail Skorzhinskii
Quoted 3 lines · Mikhail Skorzhinskii

This change prevents sendpage calls for payload sending too, although this is true only for admin commands, so actual data transfer performance should be untouched.

XFS can also send slab pages for data.

I've applied to patch to nvme-5.3, but removed the above sentence from the commit log.