nvme-multipath: Early exit if no path is available
PATCH v2 1/1 nvme-multipath: Early exit if no path is available
nvme_round_robin_path() should test if the return ns pointer is valid. nvme_next_ns() will return a NULL pointer if there is no path left.
---
v2:
- moved NULL test into the if conditional statement
- added Fixes tag
drivers/nvme/host/multipath.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/multipath.c +1 -1
@@ -221,7 +221,7 @@ static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, }for (ns =nvme_next_ns (head, old); - ns != old; + ns && ns != old; ns =nvme_next_ns (head, ns)) {if (nvme_path_is_disabled (ns))continue ;
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 21 lines · Daniel Wagner
nvme_round_robin_path()should test if the return ns pointer is valid.nvme_next_ns()will return a NULL pointer if there is no path left.Fixes 75c10e732724 ("nvme-multipath: round-robin I/O policy")Cc Hannes Reinecke <[email protected]>Signed-off-by Daniel Wagner <[email protected]>---
v2:
- moved NULL test into the if conditional statement
- added Fixes tagdrivers/nvme/host/multipath.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/host/multipath.c +0
-0
drivers/nvme/host/multipath.c static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, · line 227
Quoted 8 lines
drivers/nvme/host/multipath.c +1
-1@@ -221,7 +221,7 @@ static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, }for (ns =nvme_next_ns (head, old); - ns != old; + ns && ns != old; ns =nvme_next_ns (head, ns)) {if (nvme_path_is_disabled (ns))continue ;
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Thanks,
applied to nvme-5.11.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 20 lines · Daniel Wagner
nvme_round_robin_path()should test if the return ns pointer is valid.nvme_next_ns()will return a NULL pointer if there is no path left.Fixes 75c10e732724 ("nvme-multipath: round-robin I/O policy")Cc Hannes Reinecke <[email protected]>Signed-off-by Daniel Wagner <[email protected]>---
v2:
- moved NULL test into the if conditional statement
- added Fixes tagdrivers/nvme/host/multipath.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/host/multipath.c +0
-0
drivers/nvme/host/multipath.c static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, · line 224
Quoted 5 lines
drivers/nvme/host/multipath.c +1
-1@@ -221,7 +221,7 @@ static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, } for (ns = nvme_next_ns(head, old); - ns != old; + ns && ns != old;
nvme_round_robin_path just be called when !"old". nvme_next_ns should not return NULL when !"old". It seems unnecessary to add checking "ns". Is there a bug that "old" in not in "head" list? If yes, we should fix it.
drivers/nvme/host/multipath.c static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, · line 227
Quoted 4 lines
drivers/nvme/host/multipath.c +0
-0ns = nvme_next_ns(head, ns)) {if (nvme_path_is_disabled (ns))continue ;
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 10 lines · Chao Leng
drivers/nvme/host/multipath.c static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, · line 223
Quoted 7 lines
drivers/nvme/host/multipath.c +1
-1@@ -221,7 +221,7 @@ static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, } for (ns = nvme_next_ns(head, old); - ns != old; + ns && ns != old;
nvme_round_robin_pathjust be called when !"old".nvme_next_nsshould not return NULL when !"old".
It seems unnecessary to add checking "ns".
The problem is when we enter nvme_round_robin_path() and there is no path available. In this case the initialization ns = nvme_next_ns(head, old) could return a NULL pointer.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 12 lines · Daniel Wagner
Quoted 10 lines · Chao Leng
drivers/nvme/host/multipath.c static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, · line 223
Quoted 7 lines
drivers/nvme/host/multipath.c +1
-1@@ -221,7 +221,7 @@ static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, } for (ns = nvme_next_ns(head, old); - ns != old; + ns && ns != old;
nvme_round_robin_pathjust be called when !"old".nvme_next_nsshould not return NULL when !"old".
It seems unnecessary to add checking "ns".
Quoted 3 lines
The problem is when we enter
nvme_round_robin_path()and there is no
path available. In this case the initialization ns = nvme_next_ns(head,
old) could return a NULL pointer."old" should not be NULL, so there is at least one path that is "old".
It is impossible to return NULL for nvme_next_ns(head, old).
Quoted 2 lines
.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 16 lines · Chao Leng
Quoted 13 lines · Daniel Wagner
Quoted 11 lines · Chao Leng
drivers/nvme/host/multipath.c static struct nvme_ns · line 224
Quoted 8 lines
drivers/nvme/host/multipath.c +1
-1@@ -221,7 +221,7 @@ static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, } for (ns = nvme_next_ns(head, old); - ns != old; + ns && ns != old;
nvme_round_robin_pathjust be called when !"old".nvme_next_nsshould not return NULL when !"old".
It seems unnecessary to add checking "ns".
Quoted 4 lines
Quoted 3 lines
The problem is when we enter
nvme_round_robin_path()and there is no
path available. In this case the initialization ns = nvme_next_ns(head,
old) could return a NULL pointer."old" should not be NULL, so there is at least one path that is "old".It is impossible to return NULL for
nvme_next_ns(head, old).
No. list_next_or_null_rcu()/list_first_or_null_rcu() will return NULL when then end of the list is reached.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 16 lines · Hannes Reinecke
Quoted 15 lines · Chao Leng
Quoted 12 lines · Daniel Wagner
Quoted 10 lines · Chao Leng
drivers/nvme/host/multipath.c static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, · line 223
Quoted 7 lines
drivers/nvme/host/multipath.c +1
-1@@ -221,7 +221,7 @@ static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, } for (ns = nvme_next_ns(head, old); - ns != old; + ns && ns != old;
nvme_round_robin_pathjust be called when !"old".nvme_next_nsshould not return NULL when !"old".
It seems unnecessary to add checking "ns".
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
The problem is when we enter
nvme_round_robin_path()and there is no
path available. In this case the initialization ns = nvme_next_ns(head,
old) could return a NULL pointer."old" should not be NULL, so there is at least one path that is "old".It is impossible to return NULL for
nvme_next_ns(head, old).No. list_next_or_null_rcu()/list_first_or_null_rcu() will return NULL when then end of the list is reached.
Although list_next_or_null_rcu()/list_first_or_null_rcu() may return NULL, but nvme_next_ns(head, old) assume that the "old" is in the "head", so nvme_next_ns(head, old) should not return NULL. If the "old" is not in the "head", nvme_next_ns(head, old) will run abnormal.
So there is other bug which cause nvme_next_ns(head, old).
I review the code about head->list and head->current_path, I find 2 bugs may cause nvme_next_ns(head, old) abnormal:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/
Second, in nvme_ns_remove, list_del_rcu is before nvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.
Quoted 4 lines
Cheers,
Hannes
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 1 line · Chao Leng
It is impossible to return NULL for
nvme_next_ns(head, old).
block nvme0n1: no usable path - requeuing I/O
block nvme0n1: no usable path - requeuing I/O
block nvme0n1: no usable path - requeuing I/O
block nvme0n1: no usable path - requeuing I/O
block nvme0n1: no usable path - requeuing I/O
block nvme0n1: no usable path - requeuing I/O
BUG: kernel NULL pointer dereference, address: 0000000000000068 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 8000000ff67bc067 P4D 8000000ff67bc067 PUD ff9ac9067 PMD 0 Oops: 0000 [#1] SMP PTI CPU: 23 PID: 15759 Comm: dt.21.15 Kdump: loaded Tainted: G E 5.3.18-0.gc9fe679-default #1 SLE15-SP2 (unreleased) Hardware name: FUJITSU PRIMERGY RX2540 M2/D3289-B1, BIOS V5.0.0.11 R1.18.0 for D3289-B1x 02/06/2018 RIP: 0010:nvme_ns_head_make_request+0x1d1/0x430 [nvme_core] Code: 54 24 10 0f 84 c9 01 00 00 48 8b 54 24 10 48 83 ea 30 0f 84 ba 01 00 00 48 39 d0 0f 84 01 02 00 00 31 ff eb 05 48 39 d0 74 67 <48> 8b 72 68 83 e6 04 75 13 48 8b 72 68 83 e6 01 75 0a 48 8b 72 10 RSP: 0018:ffffa69d08017af8 EFLAGS: 00010246 RAX: ffff92f261d87800 RBX: ffff92fa555b0010 RCX: ffff92fa555bc570 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 RBP: 0000000000000001 R08: 0000000000001000 R09: 0000000000001000 R10: ffffa69d080179a8 R11: ffff92f264f0c1c0 R12: ffff92f264f7f000 R13: ffff92fa555b0000 R14: 0000000000000001 R15: 0000000000000000 FS: 00007f3962bae700(0000) GS:ffff92f29ffc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000068 CR3: 0000000fd69a2002 CR4: 00000000003606e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: generic_make_request+0x121/0x300 ? submit_bio+0x42/0x1c0 submit_bio+0x42/0x1c0 ext4_io_submit+0x49/0x60 [ext4] ext4_writepages+0x625/0xe90 [ext4] ? do_writepages+0x4b/0xe0 ? ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4] do_writepages+0x4b/0xe0 ? __generic_file_write_iter+0x192/0x1c0 ? __filemap_fdatawrite_range+0xcb/0x100 __filemap_fdatawrite_range+0xcb/0x100 ? ext4_file_write_iter+0x128/0x3c0 [ext4] file_write_and_wait_range+0x5e/0xb0 __generic_file_fsync+0x22/0xb0 ext4_sync_file+0x1f7/0x3c0 [ext4] do_fsync+0x38/0x60 __x64_sys_fsync+0x10/0x20 do_syscall_64+0x5b/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9
You can't see exactly where it dies but I followed the assembly to nvme_round_robin_path(). Maybe it's not the initial nvme_next_ns(head, old) which returns NULL but nvme_next_ns() is returning NULL eventually (list_next_or_null_rcu()).
And I have positive feedback, this patch fixes the above problem.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 50 lines · Daniel Wagner
Quoted 1 line · Chao Leng
It is impossible to return NULL for
nvme_next_ns(head, old).block nvme0n1: no usable path - requeuing I/O
block nvme0n1: no usable path - requeuing I/O
block nvme0n1: no usable path - requeuing I/O
block nvme0n1: no usable path - requeuing I/O
block nvme0n1: no usable path - requeuing I/O
block nvme0n1: no usable path - requeuing I/OBUG: kernel NULL pointer dereference, address: 0000000000000068 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 8000000ff67bc067 P4D 8000000ff67bc067 PUD ff9ac9067 PMD 0 Oops: 0000 [#1] SMP PTI CPU: 23 PID: 15759 Comm: dt.21.15 Kdump: loaded Tainted: G E 5.3.18-0.gc9fe679-default #1 SLE15-SP2 (unreleased) Hardware name: FUJITSU PRIMERGY RX2540 M2/D3289-B1, BIOS V5.0.0.11 R1.18.0 for D3289-B1x 02/06/2018 RIP: 0010:nvme_ns_head_make_request+0x1d1/0x430 [nvme_core] Code: 54 24 10 0f 84 c9 01 00 00 48 8b 54 24 10 48 83 ea 30 0f 84 ba 01 00 00 48 39 d0 0f 84 01 02 00 00 31 ff eb 05 48 39 d0 74 67 <48> 8b 72 68 83 e6 04 75 13 48 8b 72 68 83 e6 01 75 0a 48 8b 72 10 RSP: 0018:ffffa69d08017af8 EFLAGS: 00010246 RAX: ffff92f261d87800 RBX: ffff92fa555b0010 RCX: ffff92fa555bc570 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 RBP: 0000000000000001 R08: 0000000000001000 R09: 0000000000001000 R10: ffffa69d080179a8 R11: ffff92f264f0c1c0 R12: ffff92f264f7f000 R13: ffff92fa555b0000 R14: 0000000000000001 R15: 0000000000000000 FS: 00007f3962bae700(0000) GS:ffff92f29ffc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000068 CR3: 0000000fd69a2002 CR4: 00000000003606e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: generic_make_request+0x121/0x300 ? submit_bio+0x42/0x1c0 submit_bio+0x42/0x1c0 ext4_io_submit+0x49/0x60 [ext4] ext4_writepages+0x625/0xe90 [ext4] ? do_writepages+0x4b/0xe0 ? ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4] do_writepages+0x4b/0xe0 ? __generic_file_write_iter+0x192/0x1c0 ? __filemap_fdatawrite_range+0xcb/0x100 __filemap_fdatawrite_range+0xcb/0x100 ? ext4_file_write_iter+0x128/0x3c0 [ext4] file_write_and_wait_range+0x5e/0xb0 __generic_file_fsync+0x22/0xb0 ext4_sync_file+0x1f7/0x3c0 [ext4] do_fsync+0x38/0x60 __x64_sys_fsync+0x10/0x20 do_syscall_64+0x5b/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 4 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).
So there is other bug cause nvme_next_ns abormal.
I review the code about head->list and head->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/
Second, in nvme_ns_remove, list_del_rcu is before nvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.
Quoted 2 lines
And I have positive feedback, this patch fixes the above problem.
Although adding check ns can fix the crash. There may still be other problems such as in __nvme_find_path which use list_for_each_entry_rcu.
Quoted 3 lines
.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 11 lines
Quoted 5 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).So there is other bug cause
nvme_next_nsabormal.
I review the code abouthead->listandhead->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/
Quoted 4 lines
Second, in
nvme_ns_remove,list_del_rcuis beforenvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.
The reason why we first remove from head->list and only then clear current_path is because the other way around there is no way to guarantee that that the ns won't be assigned as current_path again (because it is in head->list).
nvme_ns_remove fences continue of deletion of the ns by synchronizing the srcu such that for sure the current_path clearance is visible.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 15 lines · Sagi Grimberg
Quoted 5 lines
Quoted 5 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).Quoted 9 lines
So there is other bug cause
nvme_next_nsabormal.
I review the code abouthead->listandhead->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/
Second, innvme_ns_remove,list_del_rcuis beforenvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.
Quoted 4 lines
The reason why we first remove from
head->listand only then clearcurrent_pathis because the other way around there is no way to guarantee that that the ns won't be assigned ascurrent_pathagain (because it is inhead->list).
ok, I see.
Quoted 3 lines
nvme_ns_removefences continue of deletion of the ns by synchronizing the srcu such that for sure thecurrent_pathclearance is visible.
The list will be like this:head->next = ns1;ns1->next = head;old->next = ns1;
This may cause infinite loop in nvme_round_robin_path.
for (ns = nvme_next_ns(head, old); ns != old; ns = nvme_next_ns(head, ns))
The ns will always be ns1, and then infinite loop.
Quoted 1 line
.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 24 lines
Quoted 16 lines
Quoted 11 lines
Quoted 5 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).So there is other bug cause
nvme_next_nsabormal.
I review the code abouthead->listandhead->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/Quoted 4 lines
Second, in
nvme_ns_remove,list_del_rcuis beforenvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.Quoted 4 lines
The reason why we first remove from
head->listand only then clearcurrent_pathis because the other way around there is no way to guarantee that that the ns won't be assigned ascurrent_pathagain (because it is inhead->list).ok, I see.
Quoted 3 lines
nvme_ns_removefences continue of deletion of the ns by synchronizing the srcu such that for sure thecurrent_pathclearance is visible.
Quoted 9 lines
The list will be like this:
head->next= ns1;ns1->next= head;old->next= ns1;
This may cause infinite loop innvme_round_robin_path. for (ns =nvme_next_ns(head, old);
ns != old;
ns =nvme_next_ns(head, ns))
The ns will always be ns1, and then infinite loop.
Who is being removed? I'm not following
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 33 lines · Sagi Grimberg
Quoted 23 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).Quoted 9 lines
So there is other bug cause
nvme_next_nsabormal.
I review the code abouthead->listandhead->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/
Second, innvme_ns_remove,list_del_rcuis beforenvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.Quoted 4 lines
The reason why we first remove from
head->listand only then clearcurrent_pathis because the other way around there is no way to guarantee that that the ns won't be assigned ascurrent_pathagain (because it is inhead->list).ok, I see.
Quoted 3 lines
nvme_ns_removefences continue of deletion of the ns by synchronizing the srcu such that for sure thecurrent_pathclearance is visible.Quoted 9 lines
The list will be like this:
head->next= ns1;ns1->next= head;old->next= ns1;
This may cause infinite loop innvme_round_robin_path. for (ns =nvme_next_ns(head, old);
ns != old;
ns =nvme_next_ns(head, ns))
The ns will always be ns1, and then infinite loop.
Quoted 1 line
Who is being removed? I'm not following
The "old" is being removed path.
Daniel Wagner report crash like this:head->next = head;old->next = head;
So nvme_next_ns(head, old) will return NULL, and then crash. Although check ns can avoid crash, but can not avoid infinite loop.
Similar reason, The list will be like this:head->next = ns1;ns1->next = head;old->next = ns1;
ns1 is other path.
Quoted 1 line
.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 23 lines · Chao Leng
Quoted 16 lines · Sagi Grimberg
Quoted 11 lines
Quoted 5 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).So there is other bug cause
nvme_next_nsabormal.
I review the code abouthead->listandhead->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/Quoted 4 lines
Second, in
nvme_ns_remove,list_del_rcuis beforenvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.Quoted 4 lines
The reason why we first remove from
head->listand only then clearcurrent_pathis because the other way around there is no way to guarantee that that the ns won't be assigned ascurrent_pathagain (because it is inhead->list).
Quoted 8 lines
ok, I see.
Quoted 3 lines
nvme_ns_removefences continue of deletion of the ns by synchronizing the srcu such that for sure thecurrent_pathclearance is visible.The list will be like this:
head->next= ns1;ns1->next= head;old->next= ns1;
Where does 'old' pointing to?
Quoted 5 lines
This may cause infinite loop in
nvme_round_robin_path. for (ns =nvme_next_ns(head, old);
ns != old;
ns =nvme_next_ns(head, ns))
The ns will always be ns1, and then infinite loop.
No. nvme_next_ns() will return NULL.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 34 lines · Hannes Reinecke
Quoted 22 lines · Chao Leng
Quoted 15 lines · Sagi Grimberg
Quoted 5 lines
Quoted 5 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).Quoted 9 lines
So there is other bug cause
nvme_next_nsabormal.
I review the code abouthead->listandhead->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/
Second, innvme_ns_remove,list_del_rcuis beforenvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.Quoted 4 lines
The reason why we first remove from
head->listand only then clearcurrent_pathis because the other way around there is no way to guarantee that that the ns won't be assigned ascurrent_pathagain (because it is inhead->list).Quoted 8 lines
ok, I see.
Quoted 3 lines
nvme_ns_removefences continue of deletion of the ns by synchronizing the srcu such that for sure thecurrent_pathclearance is visible.The list will be like this:
head->next= ns1;ns1->next= head;old->next= ns1;Where does 'old' pointing to?
Quoted 7 lines
Quoted 5 lines
This may cause infinite loop in
nvme_round_robin_path. for (ns =nvme_next_ns(head, old);
ns != old;
ns =nvme_next_ns(head, ns))
The ns will always be ns1, and then infinite loop.No.
nvme_next_ns()will return NULL.
If there is just one path(the "old") and the "old" is deleted, nvme_next_ns() will return NULL.
The list like this:head->next = head;old->next = head;
If there is two or more path and the "old" is deleted, "for" will be infinite loop. because nvme_next_ns() will return the path which in the list except the "old", check condition will be true for ever.
For example, there is two path: one is the "old", another is "ns1". The list like this:head->next = ns1;ns1->next = head;old->next = ns1;nvme_next_ns(head, old) will return ns1;
Then nvme_next_ns(head, ns1) will return ns1;
And then infinite loop.
Quoted 4 lines
Cheers,
Hannes
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 45 lines · Chao Leng
Quoted 35 lines · Hannes Reinecke
Quoted 23 lines · Chao Leng
Quoted 16 lines · Sagi Grimberg
Quoted 11 lines
Quoted 5 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).So there is other bug cause
nvme_next_nsabormal.
I review the code abouthead->listandhead->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/Quoted 4 lines
Second, in
nvme_ns_remove,list_del_rcuis beforenvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.Quoted 4 lines
The reason why we first remove from
head->listand only then clearcurrent_pathis because the other way around there is no way to guarantee that that the ns won't be assigned ascurrent_pathagain (because it is inhead->list).Quoted 8 lines
ok, I see.
Quoted 3 lines
nvme_ns_removefences continue of deletion of the ns by synchronizing the srcu such that for sure thecurrent_pathclearance is visible.The list will be like this:
head->next= ns1;ns1->next= head;old->next= ns1;Where does 'old' pointing to?
Quoted 7 lines
Quoted 5 lines
This may cause infinite loop in
nvme_round_robin_path. for (ns =nvme_next_ns(head, old);
ns != old;
ns =nvme_next_ns(head, ns))
The ns will always be ns1, and then infinite loop.No.
nvme_next_ns()will return NULL.
Quoted 9 lines
If there is just one
path(the "old")and the "old" is deleted,nvme_next_ns()will return NULL.
The list like this:head->next= head;old->next= head;
If there is two or more path and the "old" is deleted, "for" will be infinite loop. becausenvme_next_ns()will return the path which in the list except the "old", check condition will be true for ever.
But that will be caught by the statement above:
if (list_is_singular(&head->list))
no?
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 55 lines · Hannes Reinecke
Quoted 44 lines · Chao Leng
Quoted 34 lines · Hannes Reinecke
Quoted 22 lines · Chao Leng
Quoted 15 lines · Sagi Grimberg
Quoted 5 lines
Quoted 5 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).Quoted 9 lines
So there is other bug cause
nvme_next_nsabormal.
I review the code abouthead->listandhead->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/
Second, innvme_ns_remove,list_del_rcuis beforenvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.Quoted 4 lines
The reason why we first remove from
head->listand only then clearcurrent_pathis because the other way around there is no way to guarantee that that the ns won't be assigned ascurrent_pathagain (because it is inhead->list).Quoted 8 lines
ok, I see.
Quoted 3 lines
nvme_ns_removefences continue of deletion of the ns by synchronizing the srcu such that for sure thecurrent_pathclearance is visible.The list will be like this:
head->next= ns1;ns1->next= head;old->next= ns1;Where does 'old' pointing to?
Quoted 7 lines
Quoted 5 lines
This may cause infinite loop in
nvme_round_robin_path. for (ns =nvme_next_ns(head, old);
ns != old;
ns =nvme_next_ns(head, ns))
The ns will always be ns1, and then infinite loop.No.
nvme_next_ns()will return NULL.Quoted 9 lines
If there is just one
path(the "old")and the "old" is deleted,nvme_next_ns()will return NULL.
The list like this:head->next= head;old->next= head;
If there is two or more path and the "old" is deleted, "for" will be infinite loop. becausenvme_next_ns()will return the path which in the list except the "old", check condition will be true for ever.
Quoted 5 lines
But that will be caught by the statement above:
if (
list_is_singular(&head->list))no?
Two path just a sample example.
If there is just two path, will enter it, may cause no path but there is actually one path. It is falsely assumed that the "old" must be not deleted. If there is more than two path, will cause infinite loop.
Quoted 4 lines
Cheers,
Hannes
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 61 lines · Chao Leng
Quoted 58 lines · Hannes Reinecke
Quoted 45 lines · Chao Leng
Quoted 35 lines · Hannes Reinecke
Quoted 23 lines · Chao Leng
Quoted 16 lines · Sagi Grimberg
Quoted 11 lines
Quoted 5 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).So there is other bug cause
nvme_next_nsabormal.
I review the code abouthead->listandhead->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/Quoted 4 lines
Second, in
nvme_ns_remove,list_del_rcuis beforenvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.Quoted 4 lines
The reason why we first remove from
head->listand only then clearcurrent_pathis because the other way around there is no way to guarantee that that the ns won't be assigned ascurrent_pathagain (because it is inhead->list).Quoted 8 lines
ok, I see.
Quoted 3 lines
nvme_ns_removefences continue of deletion of the ns by synchronizing the srcu such that for sure thecurrent_pathclearance is visible.The list will be like this:
head->next= ns1;ns1->next= head;old->next= ns1;Where does 'old' pointing to?
Quoted 7 lines
Quoted 5 lines
This may cause infinite loop in
nvme_round_robin_path. for (ns =nvme_next_ns(head, old);
ns != old;
ns =nvme_next_ns(head, ns))
The ns will always be ns1, and then infinite loop.No.
nvme_next_ns()will return NULL.Quoted 9 lines
If there is just one
path(the "old")and the "old" is deleted,nvme_next_ns()will return NULL.
The list like this:head->next= head;old->next= head;
If there is two or more path and the "old" is deleted, "for" will be infinite loop. becausenvme_next_ns()will return the path which in the list except the "old", check condition will be true for ever.But that will be caught by the statement above:
Quoted 7 lines
Quoted 3 lines
if (
list_is_singular(&head->list))no?
Two path just a sample example.
If there is just two path, will enter it, may cause no path but there is actually one path. It is falsely assumed that the "old" must be not deleted. If there is more than two path, will cause infinite loop.
So you mean we'll need something like this?
drivers/nvme/host/multipath.c +6 -4
@@ -202,10 +202,12 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node)static struct nvme_ns *nvme_next_ns (struct nvme_ns_head *head,struct nvme_ns *ns) { - ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns, - siblings); -if (ns) -return ns; +if (ns) { + ns =list_next_or_null_rcu (&head->list, &ns->siblings, + struct nvme_ns, siblings); +if (ns) +return ns; + } return list_first_or_null_rcu(&head->list, struct nvme_ns,
siblings);
}
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 78 lines · Hannes Reinecke
Quoted 60 lines · Chao Leng
Quoted 57 lines · Hannes Reinecke
Quoted 44 lines · Chao Leng
Quoted 34 lines · Hannes Reinecke
Quoted 22 lines · Chao Leng
Quoted 15 lines · Sagi Grimberg
Quoted 5 lines
Quoted 5 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).Quoted 9 lines
So there is other bug cause
nvme_next_nsabormal.
I review the code abouthead->listandhead->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/
Second, innvme_ns_remove,list_del_rcuis beforenvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.Quoted 4 lines
The reason why we first remove from
head->listand only then clearcurrent_pathis because the other way around there is no way to guarantee that that the ns won't be assigned ascurrent_pathagain (because it is inhead->list).Quoted 8 lines
ok, I see.
Quoted 3 lines
nvme_ns_removefences continue of deletion of the ns by synchronizing the srcu such that for sure thecurrent_pathclearance is visible.The list will be like this:
head->next= ns1;ns1->next= head;old->next= ns1;Where does 'old' pointing to?
Quoted 7 lines
Quoted 5 lines
This may cause infinite loop in
nvme_round_robin_path. for (ns =nvme_next_ns(head, old);
ns != old;
ns =nvme_next_ns(head, ns))
The ns will always be ns1, and then infinite loop.No.
nvme_next_ns()will return NULL.Quoted 9 lines
If there is just one
path(the "old")and the "old" is deleted,nvme_next_ns()will return NULL.
The list like this:head->next= head;old->next= head;
If there is two or more path and the "old" is deleted, "for" will be infinite loop. becausenvme_next_ns()will return the path which in the list except the "old", check condition will be true for ever.But that will be caught by the statement above:
Quoted 7 lines
Quoted 3 lines
if (
list_is_singular(&head->list))no?
Two path just a sample example.
If there is just two path, will enter it, may cause no path but there is actually one path. It is falsely assumed that the "old" must be not deleted. If there is more than two path, will cause infinite loop.So you mean we'll need something like this?
drivers/nvme/host/multipath.c +0
-0@@ -202,10 +202,12 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node) static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head, struct nvme_ns *ns) {
drivers/nvme/host/multipath.c static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node) · line 210
Quoted 10 lines
drivers/nvme/host/multipath.c +6
-4- ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns, - siblings); - if (ns) - return ns; + if (ns) { + ns = list_next_or_null_rcu(&head->list, &ns->siblings, + struct nvme_ns, siblings); + if (ns) + return ns; + }
No, in the scenario, ns should not be NULL.
May be we can do like this:
drivers/nvme/host/multipath.c +11 -17
@@ -199,30 +199,24 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node)return found; } -static struct nvme_ns *nvme_next_ns (struct nvme_ns_head *head, -struct nvme_ns *ns) -{ - ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns, - siblings); -if (ns) -return ns; - return list_first_or_null_rcu(&head->list, struct nvme_ns, siblings); -} +#define nvme_next_ns_condition (head, current, condition) \ +({ \ + struct nvme_ns *__ptr = list_next_or_null_rcu(&(head)->list, \ + &(current)->siblings, struct nvme_ns, siblings); \ + __ptr ? __ptr : (condition) ? (condition) = false, \ + list_first_or_null_rcu(&(head)->list, struct nvme_ns, \ + siblings) : NULL; \ +})static struct nvme_ns *nvme_round_robin_path (struct nvme_ns_head *head,int node,struct nvme_ns *old) {struct nvme_ns *ns, *found =NULL ; +bool first_half = true; -if (list_is_singular (&head->list)) { -if (nvme_path_is_disabled (old)) -return NULL ; -return old; - } - -for (ns =nvme_next_ns (head, old); +for (ns =nvme_next_ns_condition (head, old, first_half); ns && ns != old; - ns =nvme_next_ns (head, ns)) { + ns =nvme_next_ns_condition (head, ns, first_half)) {if (nvme_path_is_disabled (ns))continue ;
Quoted 6 lines
drivers/nvme/host/multipath.c +0
-0return list_first_or_null_rcu(&head->list, struct nvme_ns, siblings); }Cheers,
Hannes
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 87 lines · Chao Leng
Quoted 84 lines · Hannes Reinecke
Quoted 61 lines · Chao Leng
Quoted 58 lines · Hannes Reinecke
Quoted 45 lines · Chao Leng
Quoted 35 lines · Hannes Reinecke
Quoted 23 lines · Chao Leng
Quoted 16 lines · Sagi Grimberg
Quoted 11 lines
Quoted 5 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).So there is other bug cause
nvme_next_nsabormal.
I review the code abouthead->listandhead->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/Quoted 4 lines
Second, in
nvme_ns_remove,list_del_rcuis beforenvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.Quoted 4 lines
The reason why we first remove from
head->listand only then clearcurrent_pathis because the other way around there is no way to guarantee that that the ns won't be assigned ascurrent_pathagain (because it is inhead->list).Quoted 8 lines
ok, I see.
Quoted 3 lines
nvme_ns_removefences continue of deletion of the ns by synchronizing the srcu such that for sure thecurrent_pathclearance is visible.The list will be like this:
head->next= ns1;ns1->next= head;old->next= ns1;Where does 'old' pointing to?
Quoted 7 lines
Quoted 5 lines
This may cause infinite loop in
nvme_round_robin_path. for (ns =nvme_next_ns(head, old);
ns != old;
ns =nvme_next_ns(head, ns))
The ns will always be ns1, and then infinite loop.No.
nvme_next_ns()will return NULL.Quoted 9 lines
If there is just one
path(the "old")and the "old" is deleted,nvme_next_ns()will return NULL.
The list like this:head->next= head;old->next= head;
If there is two or more path and the "old" is deleted, "for" will be infinite loop. becausenvme_next_ns()will return the path which in the list except the "old", check condition will be true for ever.But that will be caught by the statement above:
Quoted 7 lines
Quoted 3 lines
if (
list_is_singular(&head->list))no?
Two path just a sample example.
If there is just two path, will enter it, may cause no path but there is actually one path. It is falsely assumed that the "old" must be not deleted. If there is more than two path, will cause infinite loop.So you mean we'll need something like this?
drivers/nvme/host/multipath.c +0
-3@@ -202,10 +202,12 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node) static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head, struct nvme_ns *ns) { - ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns, - siblings); - if (ns)
Quoted 8 lines
drivers/nvme/host/multipath.c static struct nvme_ns *__nvme_find_path(struct · line 212
Quoted 7 lines
drivers/nvme/host/multipath.c +6
-1- return ns; + if (ns) { + ns = list_next_or_null_rcu(&head->list, &ns->siblings, + struct nvme_ns, siblings); + if (ns) + return ns; + }No, in the scenario, ns should not be NULL.
Why not? 'ns == NULL' is precisely the corner-case this is trying to fix...
Quoted 21 lines
May be we can do like this:
drivers/nvme/host/multipath.c +0
-8@@ -199,30 +199,24 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node) return found; } -static struct nvme_ns *nvme_next_ns (struct nvme_ns_head *head, - struct nvme_ns *ns) -{ - ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns, - siblings); - if (ns) - return ns; - return list_first_or_null_rcu(&head->list, struct nvme_ns, siblings);
drivers/nvme/host/multipath.c static struct nvme_ns *__nvme_find_path(struct · line 213
Quoted 10 lines
drivers/nvme/host/multipath.c +8
-1-} +#define nvme_next_ns_condition (head, current, condition) \ +({ \ + struct nvme_ns *__ptr = list_next_or_null_rcu(&(head)->list, \ + &(current)->siblings, struct nvme_ns, siblings); \ + __ptr ? __ptr : (condition) ? (condition) = false, \ + list_first_or_null_rcu(&(head)->list, struct nvme_ns, \ + siblings) : NULL; \ +})
Urgh. Please, no. That is well impossible to debug.
Can you please open-code it to demonstrate where the difference to the current (and my fixed) versions is? I'm still not clear where the problem is once we applied both patches.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 119 lines · Hannes Reinecke
Quoted 84 lines · Chao Leng
Quoted 81 lines · Hannes Reinecke
Quoted 60 lines · Chao Leng
Quoted 57 lines · Hannes Reinecke
Quoted 44 lines · Chao Leng
Quoted 34 lines · Hannes Reinecke
Quoted 22 lines · Chao Leng
Quoted 15 lines · Sagi Grimberg
Quoted 5 lines
Quoted 5 lines
You can't see exactly where it dies but I followed the assembly to
nvme_round_robin_path(). Maybe it's not the initialnvme_next_ns(head, old)which returns NULL butnvme_next_ns()is returning NULL eventually (list_next_or_null_rcu()).Quoted 9 lines
So there is other bug cause
nvme_next_nsabormal.
I review the code abouthead->listandhead->current_path, I find 2 bugs may cause the bug:
First, I already send the patch. see:
https://lore.kernel.org/linux-nvme/[email protected]/
Second, innvme_ns_remove,list_del_rcuis beforenvme_mpath_clear_current_path. This may cause "old" is deleted from the "head", but still use "old". I'm not sure there's any other consideration here, I will check it and try to fix it.Quoted 4 lines
The reason why we first remove from
head->listand only then clearcurrent_pathis because the other way around there is no way to guarantee that that the ns won't be assigned ascurrent_pathagain (because it is inhead->list).Quoted 8 lines
ok, I see.
Quoted 3 lines
nvme_ns_removefences continue of deletion of the ns by synchronizing the srcu such that for sure thecurrent_pathclearance is visible.The list will be like this:
head->next= ns1;ns1->next= head;old->next= ns1;Where does 'old' pointing to?
Quoted 7 lines
Quoted 5 lines
This may cause infinite loop in
nvme_round_robin_path. for (ns =nvme_next_ns(head, old);
ns != old;
ns =nvme_next_ns(head, ns))
The ns will always be ns1, and then infinite loop.No.
nvme_next_ns()will return NULL.Quoted 9 lines
If there is just one
path(the "old")and the "old" is deleted,nvme_next_ns()will return NULL.
The list like this:head->next= head;old->next= head;
If there is two or more path and the "old" is deleted, "for" will be infinite loop. becausenvme_next_ns()will return the path which in the list except the "old", check condition will be true for ever.But that will be caught by the statement above:
Quoted 7 lines
Quoted 3 lines
if (
list_is_singular(&head->list))no?
Two path just a sample example.
If there is just two path, will enter it, may cause no path but there is actually one path. It is falsely assumed that the "old" must be not deleted. If there is more than two path, will cause infinite loop.So you mean we'll need something like this?
drivers/nvme/host/multipath.c +0
-3@@ -202,10 +202,12 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node) static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head, struct nvme_ns *ns) { - ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns, - siblings); - if (ns)Quoted 8 lines
drivers/nvme/host/multipath.c static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node) · line 210
Quoted 7 lines
drivers/nvme/host/multipath.c +6
-1- return ns; + if (ns) { + ns = list_next_or_null_rcu(&head->list, &ns->siblings, + struct nvme_ns, siblings); + if (ns) + return ns; + }No, in the scenario, ns should not be NULL.
Why not? 'ns == NULL' is precisely the corner-case this is trying to fix...
Quoted 23 lines
May be we can do like this:
drivers/nvme/host/multipath.c +4
-9@@ -199,30 +199,24 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node) return found; } -static struct nvme_ns *nvme_next_ns (struct nvme_ns_head *head, - struct nvme_ns *ns) -{ - ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns, - siblings); - if (ns) - return ns; - return list_first_or_null_rcu(&head->list, struct nvme_ns, siblings); -} +#define nvme_next_ns_condition (head, current, condition) \ +({ \ + struct nvme_ns *__ptr = list_next_or_null_rcu(&(head)->list, \ + &(current)->siblings, struct nvme_ns, siblings); \
Quoted 8 lines
drivers/nvme/host/multipath.c static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node) · line 210
Quoted 5 lines
drivers/nvme/host/multipath.c +4
-0+ __ptr ? __ptr : (condition) ? (condition) = false, \ + list_first_or_null_rcu(&(head)->list, struct nvme_ns, \ + siblings) : NULL; \ +})Urgh. Please, no. That is well impossible to debug.
Can you please open-code it to demonstrate where the difference to the current (and my fixed) versions is? I'm still not clear where the problem is once we applied both patches.
For example assume the list has three path, and all path is not NVME_ANA_OPTIMIZED:head->next = ns1;ns1->next = ns2;ns2->next = head;old->next = ns2;
My patch work flow:nvme_next_ns_condition(head, old, true) return ns2; nvme_next_ns_condition(head, ns2, true) change the condition to false
and return ns1;
nvme_next_ns_condition(head, ns1, false) return ns2; nvme_next_ns_condition(head, ns2, false) return NULL; And then the loop end.
Your patch work flow:nvme_next_ns(head, old) return ns2;nvme_next_ns(head, ns2) return ns1;nvme_next_ns(head, ns1) return ns2;nvme_next_ns(head, ns2) return ns1;nvme_next_ns(head, ns1) return ns2;
And then the loop is infinite.
Quoted 4 lines
Cheers,
Hannes
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 12 lines · Chao Leng
On 2021/2/1 15:29, Hannes Reinecke wrote:[ .. ]
Quoted 3 lines
Urgh. Please, no. That is well impossible to debug.
Can you please open-code it to demonstrate where the difference to the current (and my fixed) versions is? I'm still not clear where the problem is once we applied both patches.For example assume the list has three path, and all path is not
NVME_ANA_OPTIMIZED:head->next= ns1;ns1->next= ns2;ns2->next= head;old->next= ns2;
And this is where I have issues with.
Where does 'old' come from?
Clearly it was part of the list at one point; so what happened to it?
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 13 lines · Hannes Reinecke
Quoted 12 lines · Chao Leng
On 2021/2/1 15:29, Hannes Reinecke wrote:[ .. ]
Quoted 3 lines
Urgh. Please, no. That is well impossible to debug.
Can you please open-code it to demonstrate where the difference to the current (and my fixed) versions is? I'm still not clear where the problem is once we applied both patches.For example assume the list has three path, and all path is not
NVME_ANA_OPTIMIZED:head->next= ns1;ns1->next= ns2;ns2->next= head;old->next= ns2;
Quoted 3 lines
And this is where I have issues with.
Where does 'old' come from?
Clearly it was part of the list at one point; so what happened to it?
I explained this earlier.
In nvme_ns_remove, there is a hole between list_del_rcu and nvme_mpath_clear_current_path. If head->current_path is the "old", and the "old" is removing. The "old" is already removed from the list by list_del_rcu, but head->current_path is not clear to NULL by nvme_mpath_clear_current_path.
Find path is race with nvme_ns_remove, use the "old" pass to nvme_round_robin_path to find path.
Quoted 4 lines
Cheers,
Hannes
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 19 lines · Chao Leng
Quoted 13 lines · Hannes Reinecke
Quoted 12 lines · Chao Leng
On 2021/2/1 15:29, Hannes Reinecke wrote:[ .. ]
Quoted 3 lines
Urgh. Please, no. That is well impossible to debug.
Can you please open-code it to demonstrate where the difference to the current (and my fixed) versions is? I'm still not clear where the problem is once we applied both patches.For example assume the list has three path, and all path is not
NVME_ANA_OPTIMIZED:head->next= ns1;ns1->next= ns2;ns2->next= head;old->next= ns2;Quoted 3 lines
And this is where I have issues with.
Where does 'old' come from?
Clearly it was part of the list at one point; so what happened to it?
Quoted 8 lines
I explained this earlier.
Innvme_ns_remove, there is a hole betweenlist_del_rcuandnvme_mpath_clear_current_path. Ifhead->current_pathis the "old", and the "old" is removing. The "old" is already removed from the list bylist_del_rcu, buthead->current_pathis not clear to NULL bynvme_mpath_clear_current_path.
Find path is race withnvme_ns_remove, use the "old" pass tonvme_round_robin_pathto find path.
Ah. So this should be better:
@@ -202,10 +202,12 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node)
static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head,
struct nvme_ns *ns)
{
- ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns,
- siblings);
- if (ns)
- return ns;
+ if (ns && !test_bit(NVME_NS_REMOVING, &ns->flags)) {
+ ns = list_next_or_null_rcu(&head->list, &ns->siblings,
+ struct nvme_ns, siblings);
+ if (ns)
+ return ns;
+ }
return list_first_or_null_rcu(&head->list, struct nvme_ns,
siblings);
}
The 'NVME_NS_REMOVING' bit is set before list_del_rcu() is called, so it should guard against the issue you mentioned.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 48 lines · Hannes Reinecke
Quoted 19 lines · Chao Leng
Quoted 13 lines · Hannes Reinecke
Quoted 12 lines · Chao Leng
On 2021/2/1 15:29, Hannes Reinecke wrote:[ .. ]
Quoted 3 lines
Urgh. Please, no. That is well impossible to debug.
Can you please open-code it to demonstrate where the difference to the current (and my fixed) versions is? I'm still not clear where the problem is once we applied both patches.For example assume the list has three path, and all path is not
NVME_ANA_OPTIMIZED:head->next= ns1;ns1->next= ns2;ns2->next= head;old->next= ns2;Quoted 3 lines
And this is where I have issues with.
Where does 'old' come from?
Clearly it was part of the list at one point; so what happened to it?Quoted 8 lines
I explained this earlier.
Innvme_ns_remove, there is a hole betweenlist_del_rcuandnvme_mpath_clear_current_path. Ifhead->current_pathis the "old", and the "old" is removing. The "old" is already removed from the list bylist_del_rcu, buthead->current_pathis not clear to NULL bynvme_mpath_clear_current_path.
Find path is race withnvme_ns_remove, use the "old" pass tonvme_round_robin_pathto find path.Ah. So this should be better:
(quoted diff) +6
-4@@ -202,10 +202,12 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node) static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head, struct nvme_ns *ns) { - ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns, - siblings); - if (ns) - return ns; + if (ns && !test_bit(NVME_NS_REMOVING, &ns->flags)) { + ns = list_next_or_null_rcu(&head->list, &ns->siblings, + struct nvme_ns, siblings); + if (ns) + return ns; + } return list_first_or_null_rcu(&head->list, struct nvme_ns, siblings); }
Quoted 1 line
The '
NVME_NS_REMOVING' bit is set beforelist_del_rcu()is called, so it should guard against the issue you mentioned.
Looks useless, it is still infinite loop. You can check the workflow for the scenario I mentioned.
Quoted 4 lines
Cheers,
Hannes
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 20 lines · Daniel Wagner
nvme_round_robin_path()should test if the return ns pointer is valid.nvme_next_ns()will return a NULL pointer if there is no path left.Fixes 75c10e732724 ("nvme-multipath: round-robin I/O policy")Cc Hannes Reinecke <[email protected]>Signed-off-by Daniel Wagner <[email protected]>---
v2:
- moved NULL test into the if conditional statement
- added Fixes tagdrivers/nvme/host/multipath.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/host/multipath.c +0
-0
drivers/nvme/host/multipath.c static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, · line 224
Quoted 5 lines
drivers/nvme/host/multipath.c +1
-1@@ -221,7 +221,7 @@ static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, } for (ns = nvme_next_ns(head, old); - ns != old; + ns && ns != old;
nvme_round_robin_path just be called when !"old". nvme_next_ns should not return NULL when !"old". It seems unnecessary to add checking "ns". Is there a bug that "old" is not in "head" list? If yes, we should fix it.
drivers/nvme/host/multipath.c static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head, · line 227
Quoted 4 lines
drivers/nvme/host/multipath.c +0
-0ns = nvme_next_ns(head, ns)) {if (nvme_path_is_disabled (ns))continue ;
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme