Skip to contentSkip to conversation

nvme-loop: set blocking flag

PATCH In review Keith Busch 7 participants 14 messages Oct 17, 2024 – Oct 23, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme-loop: set blocking flag

From: Keith Busch <[email protected]>

Commit 505363957fad ("nvmet: fix nvme status code when namespace is disabled") introduced a mutex lock in the io path for target. The loop target connects this to the blk-mq submission side, so it requires the blocking flag set so that queue_rq doesn't happen inside an rcu context.

---
 drivers/nvme/target/loop.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/target/loop.c +1 -1
index e32790d8fc260..77dd809fe4507 100644
@@ -479,7 +479,7 @@ static void nvme_loop_reset_ctrl_work(struct work_struct *work)
 static const struct nvme_ctrl_ops nvme_loop_ctrl_ops = {
 	.name			= "loop",
 	.module			= THIS_MODULE,
-	.flags			= NVME_F_FABRICS,
+	.flags			= NVME_F_FABRICS | NVME_F_BLOCKING,
 	.reg_read32		= nvmf_reg_read32,
 	.reg_read64		= nvmf_reg_read64,
 	.reg_write32		= nvmf_reg_write32,
Signature · 1 line
2.43.5
#1 lore raw
Quoted 18 lines · Keith Busch

From: Keith Busch <[email protected]>

Commit 505363957fad ("nvmet: fix nvme status code when namespace is disabled") introduced a mutex lock in the io path for target. The loop target connects this to the blk-mq submission side, so it requires the blocking flag set so that queue_rq doesn't happen inside an rcu context.

---
  drivers/nvme/target/loop.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/target/loop.c +0 -0
index e32790d8fc260..77dd809fe4507 100644

drivers/nvme/target/loop.c static void nvme_loop_reset_ctrl_work(struct work_struct *work) · line 485

Quoted 8 lines
drivers/nvme/target/loop.c +1 -1
@@ -479,7 +479,7 @@ static void nvme_loop_reset_ctrl_work(struct work_struct *work)
  static const struct nvme_ctrl_ops nvme_loop_ctrl_ops = {
  	.name			= "loop",
  	.module			= THIS_MODULE,
-	.flags			= NVME_F_FABRICS,
+	.flags			= NVME_F_FABRICS | NVME_F_BLOCKING,
  	.reg_read32		= nvmf_reg_read32,
  	.reg_read64		= nvmf_reg_read64,
  	.reg_write32		= nvmf_reg_write32,

Shouldn't this patch have Fixes: and Cc: stable tags?

Thanks,

Bart.

maintainer
#2 lore raw
In reply to Keith Busch
Quoted 6 lines · Keith Busch

From: Keith Busch <[email protected]>

Commit 505363957fad ("nvmet: fix nvme status code when namespace is disabled") introduced a mutex lock in the io path for target. The loop target connects this to the blk-mq submission side, so it requires the blocking flag set so that queue_rq doesn't happen inside an rcu context.

Looks fine to me, but might be worth looking at doing that a bit differently so that the target loop driver can remain non-blocking as it should be considerably more efficient.

Signature · 1 line
Jens Axboe
maintainer
#3 lore raw
Quoted 11 lines · Jens Axboe
Quoted 6 lines · Keith Busch

From: Keith Busch <[email protected]>

Commit 505363957fad ("nvmet: fix nvme status code when namespace is disabled") introduced a mutex lock in the io path for target. The loop target connects this to the blk-mq submission side, so it requires the blocking flag set so that queue_rq doesn't happen inside an rcu context.

Looks fine to me, but might be worth looking at doing that a bit differently so that the target loop driver can remain non-blocking as it should be considerably more efficient.

That's what I first looked at, but it's too hard. Maybe if we can make a lookup that doesn't rely on searching the configfs group, but it's definitely not this nice one-liner patch. :)

No rush on this today. I'll think more on it.

#4 lore raw
Quoted 7 lines · Keith Busch
Quoted 3 lines

Looks fine to me, but might be worth looking at doing that a bit differently so that the target loop driver can remain non-blocking as it should be considerably more efficient.

That's what I first looked at, but it's too hard. Maybe if we can make a lookup that doesn't rely on searching the configfs group, but it's definitely not this nice one-liner patch. :)

Searching configfs in the I/O path sounds like a train wreck no matter what. I feel a bit bad for not noticing that earlier.

Let me see what I can do.

maintainer
#5 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 6 lines · Keith Busch
Quoted 3 lines

Looks fine to me, but might be worth looking at doing that a bit differently so that the target loop driver can remain non-blocking as it should be considerably more efficient.

That's what I first looked at, but it's too hard. Maybe if we can make a lookup that doesn't rely on searching the configfs group, but it's definitely not this nice one-liner patch. :)

Searching configfs in the I/O path sounds like a train wreck no matter what. I feel a bit bad for not noticing that earlier.

Let me see what I can do.

Note, the search is ONLY in the error case, i.e. when nsid is not found in subsys->namespaces.

I agree its a shame to make the loop driver blocking for this niche case.

Note that this doesn't do anything, because it is really designed to make the host failover work correctly (i.e. not fail IO) when nvmet disabled the ns in one server, and the namespace is available in the same nvmet subsystem on another server.

We could escape this entirely for loop transport because it there are no multi-server nvmet subsystems for loop? There is a weird case where one can use mpath device where one path is loop and another is tcp/rdma... Which is very weird but maybe possible with nvmet.

#6 lore raw
Quoted 4 lines · Sagi Grimberg
Quoted 1 line

Let me see what I can do.

Note, the search is ONLY in the error case, i.e. when nsid is not found in subsys->namespaces.

Which kinda makes it worse as we use an unusual context just from that rare error path.

#7 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Sagi Grimberg
Quoted 1 line

Let me see what I can do.

Note, the search is ONLY in the error case, i.e. when nsid is not found in subsys->namespaces.

Which kinda makes it worse as we use an unusual context just from that rare error path.

I just took a very quick look, and inserting the namespace into the xarray when created and only removing it when it is deleted, and instead checking an enabled flag during lookup should do the work.

Do we have a good test case for commit 505363957fad ?

Quoted 1 line

---end quoted text---

#8 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · Sagi Grimberg
Quoted 1 line

Let me see what I can do.

Note, the search is ONLY in the error case, i.e. when nsid is not found in subsys->namespaces.

Quoted 8 lines
Quoted 2 lines

Which kinda makes it worse as we use an unusual context just from that rare error path.

I just took a very quick look, and inserting the namespace into the xarray when created and only removing it when it is deleted, and instead checking an enabled flag during lookup should do the work.

Do we have a good test case for commit 505363957fad ?

As for the BUG caused by the commit 505363957fad, the blktests test case nvme/052 can confirm if any patch fixes the BUG or not. It is recreate in stable manner on my test node [1], so I'm happy to help testing.

If you mean a test case to confirm that the commit 505363957fad achieves its purpose, I'm not sure about it. According to its commit log, I guess Jirong might have a test environment.

[1] https://lore.kernel.org/linux-nvme/tqcy3sveity7p56v7ywp7ssyviwcb3w4623cnxj3knoobfcanq@yxgt2mjkbkam/

maintainer
#9 lore raw
Quoted 7 lines · Shinichiro Kawasaki

As for the BUG caused by the commit 505363957fad, the blktests test case nvme/052 can confirm if any patch fixes the BUG or not. It is recreate in stable manner on my test node [1], so I'm happy to help testing.

If you mean a test case to confirm that the commit 505363957fad achieves its purpose, I'm not sure about it. According to its commit log, I guess Jirong might have a test environment.

Yes, that's the main thing I want to be able to test.

maintainer
#10 lore raw
In reply to Christoph Hellwig
Quoted 10 lines · Christoph Hellwig
Quoted 6 lines · Christoph Hellwig
Quoted 3 lines · Sagi Grimberg
Quoted 1 line

Let me see what I can do.

Note, the search is ONLY in the error case, i.e. when nsid is not found in subsys->namespaces.

Which kinda makes it worse as we use an unusual context just from that rare error path.

I just took a very quick look, and inserting the namespace into the xarray when created and only removing it when it is deleted, and instead checking an enabled flag during lookup should do the work.

Agreed.

maintainer
#11 lore raw
In reply to Christoph Hellwig
Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Sagi Grimberg
Quoted 1 line

Let me see what I can do.

Note, the search is ONLY in the error case, i.e. when nsid is not found in subsys->namespaces.

Which kinda makes it worse as we use an unusual context just from that rare error path.

I'd say that introducing this search in the hot path not only for error cases would be worse :)

Yes, nvme-loop was an oversight.

#12 lore raw
In reply to Jens Axboe
Quoted 11 lines · Jens Axboe
Quoted 6 lines · Keith Busch

From: Keith Busch <[email protected]>

Commit 505363957fad ("nvmet: fix nvme status code when namespace is disabled") introduced a mutex lock in the io path for target. The loop target connects this to the blk-mq submission side, so it requires the blocking flag set so that queue_rq doesn't happen inside an rcu context.

Looks fine to me, but might be worth looking at doing that a bit differently so that the target loop driver can remain non-blocking as it should be considerably more efficient.

Yes. When designing the nvmet code back in the day (gosh do I feel old) we tried to avoid blocking in the main handling thread in general.

So if we can get rid of the mutex that would be great.

Tested-by on v1 1/1
#13 lore raw
In reply to Keith Busch
Quoted 9 lines · Keith Busch

From: Keith Busch <[email protected]>

Commit 505363957fad ("nvmet: fix nvme status code when namespace is disabled") introduced a mutex lock in the io path for target. The loop target connects this to the blk-mq submission side, so it requires the blocking flag set so that queue_rq doesn't happen inside an rcu context.

Thank you Keith for this quick action. I confirmed the patch avoids the BUG. I also ran whole blktests and observed no regression. In case this patch gets applied as it is,