Skip to contentSkip to conversation

nvmet-passthru: Cleanup nvmet_passthru_map_sg()

PATCH In review Logan Gunthorpe 5 participants 14 messages Oct 9, 2020 – Oct 16, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvmet-passthru: Cleanup nvmet_passthru_map_sg()

Clean up some confusing elements of nvmet_passthru_map_sg() by returning early if the request is greater than the maximum bio size. This allows us to drop the sg_cnt variable.

This should not result in any functional change but makes the code clearer and more understandable. The original code allocated a truncated bio then would return EINVAL when bio_add_pc_page() filled that bio. The new code just returns EINVAL early if this would happen.

Fixes c1fef73f793b ("nvmet: add passthru code to process commands")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
---
 drivers/nvme/target/passthru.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/nvme/target/passthru.c +4 -3
index dacfa7435d0b..076e829490a1 100644
@@ -180,18 +180,20 @@ static void nvmet_passthru_req_done(struct request *rq,
 
 static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq)
 {
-	int sg_cnt = req->sg_cnt;
 	struct scatterlist *sg;
 	int op_flags = 0;
 	struct bio *bio;
 	int i, ret;
 
+	if (req->sg_cnt > BIO_MAX_PAGES)
+		return -EINVAL;
+
 	if (req->cmd->common.opcode == nvme_cmd_flush)
 		op_flags = REQ_FUA;
 	else if (nvme_is_write(req->cmd))
 		op_flags = REQ_SYNC | REQ_IDLE;
 
-	bio = bio_alloc(GFP_KERNEL, min(sg_cnt, BIO_MAX_PAGES));
+	bio = bio_alloc(GFP_KERNEL, req->sg_cnt);
 	bio->bi_end_io = bio_put;
 	bio->bi_opf = req_op(rq) | op_flags;
 
@@ -201,7 +203,6 @@ static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq)
 			bio_put(bio);
 			return -EINVAL;
 		}
-		sg_cnt--;
 	}
 
 	ret = blk_rq_append_bio(rq, &bio);

base-commit: 549738f15da0e5a00275977623be199fbbf7df50

Signature · 7 lines
2.20.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Reviewed-by on v1 1/1
#2 lore raw
In reply to Logan Gunthorpe
Quoted 9 lines · Logan Gunthorpe

Clean up some confusing elements of nvmet_passthru_map_sg() by returning early if the request is greater than the maximum bio size. This allows us to drop the sg_cnt variable.

This should not result in any functional change but makes the code clearer and more understandable. The original code allocated a truncated bio then would return EINVAL when bio_add_pc_page() filled that bio. The new code just returns EINVAL early if this would happen.

Quoted 6 lines
Fixes c1fef73f793b ("nvmet: add passthru code to process commands")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>

I have commented offline to Logan about the return from bio_alloc() not being checked for NULL. He assures me that can't happen but I believe that is true only for small scatter-gather lists. So I guess it depends on the size of the data this is planned to be sent through this pass-through.

Quoted 40 lines
---
  drivers/nvme/target/passthru.c | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)
drivers/nvme/target/passthru.c +4 -3
index dacfa7435d0b..076e829490a1 100644
@@ -180,18 +180,20 @@ static void nvmet_passthru_req_done(struct request *rq,
 
  static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq)
  {
-	int sg_cnt = req->sg_cnt;
  	struct scatterlist *sg;
  	int op_flags = 0;
  	struct bio *bio;
  	int i, ret;
 
+	if (req->sg_cnt > BIO_MAX_PAGES)
+		return -EINVAL;
+
  	if (req->cmd->common.opcode == nvme_cmd_flush)
  		op_flags = REQ_FUA;
  	else if (nvme_is_write(req->cmd))
  		op_flags = REQ_SYNC | REQ_IDLE;
 
-	bio = bio_alloc(GFP_KERNEL, min(sg_cnt, BIO_MAX_PAGES));
+	bio = bio_alloc(GFP_KERNEL, req->sg_cnt);
  	bio->bi_end_io = bio_put;
  	bio->bi_opf = req_op(rq) | op_flags;
 
@@ -201,7 +203,6 @@ static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq)
  			bio_put(bio);
  			return -EINVAL;
  		}
-		sg_cnt--;
  	}
 
  	ret = blk_rq_append_bio(rq, &bio);
Quoted 2 lines

base-commit: 549738f15da0e5a00275977623be199fbbf7df50

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#3 lore raw
In reply to Logan Gunthorpe
Quoted 9 lines · Logan Gunthorpe

Clean up some confusing elements of nvmet_passthru_map_sg() by returning early if the request is greater than the maximum bio size. This allows us to drop the sg_cnt variable.

This should not result in any functional change but makes the code clearer and more understandable. The original code allocated a truncated bio then would return EINVAL when bio_add_pc_page() filled that bio. The new code just returns EINVAL early if this would happen.

Quoted 6 lines
Fixes c1fef73f793b ("nvmet: add passthru code to process commands")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>

why the prefix is nvmet-passthru ?

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#4 lore raw
Quoted 10 lines · Chaitanya Kulkarni
Quoted 9 lines · Logan Gunthorpe

Clean up some confusing elements of nvmet_passthru_map_sg() by returning early if the request is greater than the maximum bio size. This allows us to drop the sg_cnt variable.

This should not result in any functional change but makes the code clearer and more understandable. The original code allocated a truncated bio then would return EINVAL when bio_add_pc_page() filled that bio. The new code just returns EINVAL early if this would happen.

Quoted 8 lines
Quoted 6 lines
Fixes c1fef73f793b ("nvmet: add passthru code to process commands")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>

why the prefix is nvmet-passthru ?

Well that's the prefix I used in the first patch, which follows other similar conventions like nvmet-tcp, nvmet-fc, nvmet-rdma, nvmet-loop, etc.

Logan

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#5 lore raw
Quoted 18 lines · Logan Gunthorpe
Quoted 16 lines · Chaitanya Kulkarni
Quoted 15 lines · Logan Gunthorpe

Clean up some confusing elements of nvmet_passthru_map_sg() by returning early if the request is greater than the maximum bio size. This allows us to drop the sg_cnt variable.

This should not result in any functional change but makes the code clearer and more understandable. The original code allocated a truncated bio then would return EINVAL when bio_add_pc_page() filled that bio. The new code just returns EINVAL early if this would happen.

Fixes c1fef73f793b ("nvmet: add passthru code to process commands")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
Quoted 7 lines
Quoted 1 line

why the prefix is nvmet-passthru ?

Well that's the prefix I used in the first patch, which follows other similar conventions like nvmet-tcp, nvmet-fc, nvmet-rdma, nvmet-loop, etc.

Logan

Those are transport types and passthru is a target backend.

I found this [1] in the git log, only one patch has that prefix.

The convention is to not add backend prefixes for target patches (see bdev/file),

I'm not sure we want to start adding prefixes unless there is a specific

reason.

 # git log --oneline  drivers/nvme/target/passthru.c
dc1890b99602 nvmet: add passthru ZNS support
df8dd70a9db8 nvme: lift the file open code from nvme_ctrl_get_by_path
3a6b076168e2 (tag: nvme-5.9-2020-09-17) nvmet: get transport reference
for passthru ctrl
7ee51cf60a90 nvmet: call blk_mq_free_request() directly
a2138fd49467 nvmet: fix oops in pt cmd execution
4db69a3d7cfe nvmet: add ns tear down label for pt-cmd handling
0ceeab96ba59 nvmet-passthru: Reject commands with non-sgl flags set
ba76af676cd0 nvmet: Add passthru enable/disable helpers
c1fef73f793b nvmet: add passthru code to process commands

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#6 lore raw
Quoted 28 lines · Chaitanya Kulkarni
Quoted 18 lines · Logan Gunthorpe
Quoted 16 lines · Chaitanya Kulkarni
Quoted 15 lines · Logan Gunthorpe

Clean up some confusing elements of nvmet_passthru_map_sg() by returning early if the request is greater than the maximum bio size. This allows us to drop the sg_cnt variable.

This should not result in any functional change but makes the code clearer and more understandable. The original code allocated a truncated bio then would return EINVAL when bio_add_pc_page() filled that bio. The new code just returns EINVAL early if this would happen.

Fixes c1fef73f793b ("nvmet: add passthru code to process commands")
Cc Christoph Hellwig <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
Quoted 7 lines
Quoted 1 line

why the prefix is nvmet-passthru ?

Well that's the prefix I used in the first patch, which follows other similar conventions like nvmet-tcp, nvmet-fc, nvmet-rdma, nvmet-loop, etc.

Logan

Those are transport types and passthru is a target backend.

Quoted 8 lines

I found this [1] in the git log, only one patch has that prefix.

The convention is to not add backend prefixes for target patches (see bdev/file),

I'm not sure we want to start adding prefixes unless there is a specific

reason.

Well, I don't see the harm in it, but if whomever merges it wants to change the prefix, I won't object.

Logan

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#7 lore raw
In reply to Logan Gunthorpe

drivers/nvme/target/passthru.c static void nvmet_passthru_req_done(struct request *rq, · line 189

Quoted 10 lines · Logan Gunthorpe

static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq) {

drivers/nvme/target/passthru.c +2 -1
-	int sg_cnt = req->sg_cnt;
 	struct scatterlist *sg;
 	int op_flags = 0;
 	struct bio *bio;
 	int i, ret;
 
+	if (req->sg_cnt > BIO_MAX_PAGES)
+		return -EINVAL;

Don't you need to handle larger requests as well? Or at least limit MDTS?

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#8 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Logan Gunthorpe

static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq) {

drivers/nvme/target/passthru.c +0 -1
-	int sg_cnt = req->sg_cnt;
 	struct scatterlist *sg;
 	int op_flags = 0;
 	struct bio *bio;
 	int i, ret;
Quoted 5 lines

drivers/nvme/target/passthru.c static void nvmet_passthru_req_done(struct request *rq, · line 189

Quoted 2 lines
drivers/nvme/target/passthru.c +2 -0
+	if (req->sg_cnt > BIO_MAX_PAGES)
+		return -EINVAL;

Don't you need to handle larger requests as well? Or at least limit MDTS?

No and Yes: there is already code in nvmet_passthru_override_id_ctrl() to limit MDTS based on max_segments and max_hw_sectors. So any request that doesn't pass this check should be from a misbehaving client and an error should be appropriate.

Logan

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#9 lore raw
Quoted 18 lines · Logan Gunthorpe
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Logan Gunthorpe
  static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq)
  {
drivers/nvme/target/passthru.c +0 -1
-	int sg_cnt = req->sg_cnt;
  	struct scatterlist *sg;
  	int op_flags = 0;
  	struct bio *bio;
  	int i, ret;
Quoted 5 lines

drivers/nvme/target/passthru.c static void nvmet_passthru_req_done(struct request *rq, · line 189

Quoted 2 lines
drivers/nvme/target/passthru.c +2 -0
+	if (req->sg_cnt > BIO_MAX_PAGES)
+		return -EINVAL;

Don't you need to handle larger requests as well? Or at least limit MDTS?

Quoted 4 lines

No and Yes: there is already code in nvmet_passthru_override_id_ctrl() to limit MDTS based on max_segments and max_hw_sectors. So any request that doesn't pass this check should be from a misbehaving client and an error should be appropriate.

Running the numbers: with PAGE_SIZE of 4096 bytes and BIO_MAX_PAGES at 256 gives 1 MiB. From memory the block layer won't accept single requests bigger than 4 MiB (or 8 MiB). Then it is possible that the sgl was built with sgl_alloc_order(order > 0, chainable=false) in which case the maximum (unchained) bio carrying size goes up to:
PAGE_SIZE * (2^order) * 256

I'm using order=3 by default in my driver. So one (unchained) bio will hold as much (or more) than the block layer will take.

Doug Gilbert

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#10 lore raw
In reply to Logan Gunthorpe
Quoted 12 lines · Logan Gunthorpe
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Logan Gunthorpe

static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq) {

drivers/nvme/target/passthru.c +0 -1
-	int sg_cnt = req->sg_cnt;
 	struct scatterlist *sg;
 	int op_flags = 0;
 	struct bio *bio;
 	int i, ret;
Quoted 8 lines
Quoted 5 lines

drivers/nvme/target/passthru.c static void nvmet_passthru_req_done(struct request *rq, · line 189

Quoted 2 lines
drivers/nvme/target/passthru.c +2 -0
+	if (req->sg_cnt > BIO_MAX_PAGES)
+		return -EINVAL;

Don't you need to handle larger requests as well? Or at least limit MDTS?

No and Yes: there is already code in nvmet_passthru_override_id_ctrl() to limit MDTS based on max_segments and max_hw_sectors.

But those are entirely unrelated to the bio size. BIO_MAX_PAGES is 256, so with 4k pages and assuming none can't be merged that is 1MB, while max_segments/max_hw_sectors could be something much larger.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#11 lore raw
Quoted 19 lines · Christoph Hellwig
Quoted 18 lines · Logan Gunthorpe
Quoted 14 lines · Christoph Hellwig

drivers/nvme/target/passthru.c static void nvmet_passthru_req_done(struct request *rq, · line 189

Quoted 10 lines · Logan Gunthorpe

static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq) {

drivers/nvme/target/passthru.c +2 -1
-	int sg_cnt = req->sg_cnt;
 	struct scatterlist *sg;
 	int op_flags = 0;
 	struct bio *bio;
 	int i, ret;
 
+	if (req->sg_cnt > BIO_MAX_PAGES)
+		return -EINVAL;

Don't you need to handle larger requests as well? Or at least limit MDTS?

Quoted 6 lines
Quoted 2 lines

No and Yes: there is already code in nvmet_passthru_override_id_ctrl() to limit MDTS based on max_segments and max_hw_sectors.

But those are entirely unrelated to the bio size. BIO_MAX_PAGES is 256, so with 4k pages and assuming none can't be merged that is 1MB, while max_segments/max_hw_sectors could be something much larger.

Isn't it constrained by max_segments which is set to NVME_MAX_SEGS=127 (for PCI)... less than BIO_MAX_PAGES...

Would the NVME driver even work if max_segments was greater than BIO_MAX_PAGES? Correct me if I'm wrong, but it looks like blk_rq_map_sg() will only map one bio within a request. So there has to be one bio per request by the time it hits nvme_map_data()...

So I'm not really sure how we could even construct a valid passthrough request in nvmet_passthru_map_sg() that is larger than BIO_MAX_PAGES.

If you want me to send a patch to future proof the MDTS limit with BIO_MAX_PAGES, I can do that, but it doesn't look like it will have any effect right now unless big things change.

Logan

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#12 lore raw
Quoted 6 lines · Logan Gunthorpe
Quoted 3 lines

But those are entirely unrelated to the bio size. BIO_MAX_PAGES is 256, so with 4k pages and assuming none can't be merged that is 1MB, while max_segments/max_hw_sectors could be something much larger.

Isn't it constrained by max_segments which is set to NVME_MAX_SEGS=127 (for PCI)... less than BIO_MAX_PAGES...

nvmet-passthrough is not limited to work on PCIe controllers. And even if it did relying on an implicit limit like that is a time bomb. I have work pending for one of the next merge windows to lift that limit for example, and I would not have thought that nvmet-passthrough relies on it in any way.

Quoted 4 lines

Would the NVME driver even work if max_segments was greater than BIO_MAX_PAGES? Correct me if I'm wrong, but it looks like blk_rq_map_sg() will only map one bio within a request. So there has to be one bio per request by the time it hits nvme_map_data()...

blk_rq_map_sg maps the bio chain. Take a look at the for_each_bio loop in __blk_bios_map_sg.

Quoted 3 lines

If you want me to send a patch to future proof the MDTS limit with BIO_MAX_PAGES, I can do that, but it doesn't look like it will have any effect right now unless big things change.

Yes, please do.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#13 lore raw
Quoted 3 lines · Christoph Hellwig

On Thu, Oct 15, 2020 at 12:40:52PM -0600, Logan Gunthorpe wrote:
blk_rq_map_sg maps the bio chain. Take a look at the for_each_bio loop in __blk_bios_map_sg.

Ah, yes, ok... I originally had a loop that created a chain of bios but used bio_add_page() instead of bio_add_pc_page(). This was combined with a separate check for max_segments and max_hw_sectors. When I switched to bio_add_pc_page() to drop the separate check, it no longer made sense to do the chain because there was no way to tell if bio_add_pc_page() failed because of bio_full() or because it exceeded max_segments/hw_sectors. If the bio was full, we'd want to create the chain, but if it exceeded the maximums we'd want to error out...

Quoted 5 lines
Quoted 3 lines

If you want me to send a patch to future proof the MDTS limit with BIO_MAX_PAGES, I can do that, but it doesn't look like it will have any effect right now unless big things change.

Yes, please do.

Will do.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme