Skip to contentSkip to conversation

nvmet: Use direct IO for writes

PATCH New Sagi Grimberg 3 participants 8 messages Sep 21, 2016 – Sep 23, 2016

Subsystems (2)
NVMe NVMe target
lore
Share & tools
lore

Sagi Grimberg

author maintainer
#0 lore raw

PATCH 1/1 nvmet: Use direct IO for writes

We're designed to work with high-end devices where direct IO makes perfect sense. We noticed that we context switch by scheduling kblockd instead of going directly to the device without REQ_SYNC for writes.

Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/target/io-cmd.c | 1 +
 1 file changed, 1 insertion(+)
drivers/nvme/target/io-cmd.c +1 -0
index 2cd069b691ae..4132b6b98182 100644
@@ -58,6 +58,7 @@ static void nvmet_execute_rw(struct nvmet_req *req)
 
 	if (req->cmd->rw.opcode == nvme_cmd_write) {
 		op = REQ_OP_WRITE;
+		op_flags = WRITE_ODIRECT;
 		if (req->cmd->rw.control & cpu_to_le16(NVME_RW_FUA))
 			op_flags |= REQ_FUA;
 	} else {
Signature · 1 line
1.9.1
maintainer
#1 lore raw

PATCH 1/1 nvmet: Use direct IO for writes

Quoted 4 lines · Sagi Grimberg

We're designed to work with high-end devices where direct IO makes perfect sense. We noticed that we context switch by scheduling kblockd instead of going directly to the device without REQ_SYNC for writes.

This looks reasonable. But I still wonder why we bother to inject delay for any fast blk-mq device (background: Sagi told me he is observing issues without this on a NVMe PCIe card backend)

maintainer
#2 lore raw

PATCH 1/1 nvmet: Use direct IO for writes

Quoted 9 lines · Christoph Hellwig
Quoted 4 lines · Sagi Grimberg

We're designed to work with high-end devices where direct IO makes perfect sense. We noticed that we context switch by scheduling kblockd instead of going directly to the device without REQ_SYNC for writes.

This looks reasonable. But I still wonder why we bother to inject delay for any fast blk-mq device (background: Sagi told me he is observing issues without this on a NVMe PCIe card backend)

Batching/merging. It's not like we're doing a calculated explicit delay, it's just async running the queue.

Signature · 1 line
Jens Axboe
author maintainer
#3 lore raw

PATCH 1/1 nvmet: Use direct IO for writes

Quoted 7 lines
Quoted 4 lines

This looks reasonable. But I still wonder why we bother to inject delay for any fast blk-mq device (background: Sagi told me he is observing issues without this on a NVMe PCIe card backend)

Batching/merging. It's not like we're doing a calculated explicit delay, it's just async running the queue.

So are we OK with bypassing that in nvmet?

maintainer
#4 lore raw

PATCH 1/1 nvmet: Use direct IO for writes

Quoted 9 lines · Sagi Grimberg
Quoted 7 lines
Quoted 4 lines

This looks reasonable. But I still wonder why we bother to inject delay for any fast blk-mq device (background: Sagi told me he is observing issues without this on a NVMe PCIe card backend)

Batching/merging. It's not like we're doing a calculated explicit delay, it's just async running the queue.

So are we OK with bypassing that in nvmet?

I'm OK with the patch as posted.

Signature · 1 line
Jens Axboe
author maintainer
#5 lore raw

PATCH 1/1 nvmet: Use direct IO for writes

Quoted 4 lines
Quoted 2 lines

So are we OK with bypassing that in nvmet?

I'm OK with the patch as posted.

Thanks, I'll queue it up for 4.9

maintainer
#6 lore raw

PATCH 1/1 nvmet: Use direct IO for writes

Quoted 6 lines · Sagi Grimberg
Quoted 4 lines
Quoted 2 lines

So are we OK with bypassing that in nvmet?

I'm OK with the patch as posted.

Thanks, I'll queue it up for 4.9

You can add my Reviewed-by, fwiw.

Signature · 1 line
Jens Axboe
maintainer
#7 lore raw

PATCH 1/1 nvmet: Use direct IO for writes

In reply to Sagi Grimberg
Quoted 6 lines · Sagi Grimberg
Quoted 4 lines
Quoted 2 lines

So are we OK with bypassing that in nvmet?

I'm OK with the patch as posted.

Thanks, I'll queue it up for 4.9

Btw, it might be a good idea to send our for-4.9 branch to Jens so it gets some linux-next exposure.