nvme-multipath: fix double initialization of ANA state
Christoph Hellwig
maintainerPATCH 1/1 nvme-multipath: fix double initialization of ANA state
nvme_init_identify and thus nvme_mpath_init can be called multiple times and thus must not overwrite potentially initialized or in-use fields. Split out a helper for the basic initialization when the controller is initialized and make sure the init_identify path does not blindly change in-use data structures.
--- drivers/nvme/host/core.c | 3 +- drivers/nvme/host/multipath.c | 53 ++++++++++++++++++----------------- drivers/nvme/host/nvme.h | 8 ++++-- 3 files changed, 36 insertions(+), 28 deletions(-)
drivers/nvme/host/core.c +2 -1
@@ -2911,7 +2911,7 @@ static int nvme_init_identify(struct nvme_ctrl *ctrl) ctrl->hmmaxd =le16_to_cpu (id->hmmaxd); } - ret =nvme_mpath_init (ctrl, id); + ret =nvme_mpath_init_identify (ctrl, id);if (ret <0 ) goto out_free;
@@ -4373,6 +4373,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev, min(default_ps_max_latency_us, (unsigned long)S32_MAX));nvme_fault_inject_init (&ctrl->fault_inject,dev_name (ctrl->device)); +nvme_mpath_init_ctrl (ctrl);return 0 ; out_free_name:
drivers/nvme/host/multipath.c +28 -25
73 lines not on this page · raw message
drivers/nvme/host/nvme.h +6 -2
@@ -712,7 +712,8 @@ void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl);int nvme_mpath_alloc_disk (struct nvme_ctrl *ctrl,struct nvme_ns_head *head);void nvme_mpath_add_disk (struct nvme_ns *ns,struct nvme_id_ns *id);void nvme_mpath_remove_disk (struct nvme_ns_head *head); -int nvme_mpath_init (struct nvme_ctrl *ctrl,struct nvme_id_ctrl *id); +int nvme_mpath_init_identify (struct nvme_ctrl *ctrl,struct nvme_id_ctrl *id); +void nvme_mpath_init_ctrl (struct nvme_ctrl *ctrl);void nvme_mpath_uninit (struct nvme_ctrl *ctrl);void nvme_mpath_stop (struct nvme_ctrl *ctrl);bool nvme_mpath_clear_current_path (struct nvme_ns *ns);
@@ -780,7 +781,10 @@ static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)static inline void nvme_trace_bio_complete (struct request *req) { } -static inline int nvme_mpath_init (struct nvme_ctrl *ctrl, +static inline void nvme_mpath_init_ctrl (struct nvme_ctrl *ctrl) +{ +} +static inline int nvme_mpath_init_identify (struct nvme_ctrl *ctrl,struct nvme_id_ctrl *id) {if (ctrl->subsys->cmic &NVME_CTRL_CMIC_ANA )
Signature · 7 lines
2.30.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 5 lines · Christoph Hellwig
nvme_init_identifyand thusnvme_mpath_initcan be called multiple times and thus must not overwrite potentially initialized or in-use fields. Split out a helper for the basic initialization when the controller is initialized and make sure theinit_identifypath does not blindly change in-use data structures.
Looks good.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 6 lines · Christoph Hellwig
nvme_init_identifyand thusnvme_mpath_initcan be called multiple times and thus must not overwrite potentially initialized or in-use fields. Split out a helper for the basic initialization when the controller is initialized and make sure theinit_identifypath does not blindly change in-use data structures.
Quoted 9 lines
Fixes 0d0b660f214d ("nvme: add ANA support")Reported-by Martin Wilck <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>--- drivers/nvme/host/core.c | 3 +- drivers/nvme/host/multipath.c | 53 ++++++++++++++++++----------------- drivers/nvme/host/nvme.h | 8 ++++-- 3 files changed, 36 insertions(+), 28 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 9 lines · Christoph Hellwig
nvme_init_identifyand thusnvme_mpath_initcan be called multiple times and thus must not overwrite potentially initialized or in-use fields. Split out a helper for the basic initialization when the controller is initialized and make sure theinit_identifypath does not blindly change in-use data structures.Fixes 0d0b660f214d ("nvme: add ANA support")Reported-by Martin Wilck <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>
Thank you. I'll prepare another test kernel for our partner.
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Hello Christoph,
Quoted 7 lines · Martin Wilck
Quoted 6 lines · Christoph Hellwig
nvme_init_identifyand thusnvme_mpath_initcan be called multiple times and thus must not overwrite potentially initialized or in-use fields. Split out a helper for the basic initialization when the controller is initialized and make sure theinit_identifypath does not blindly change in-use data structures.
Quoted 5 lines
Quoted 3 lines
Fixes 0d0b660f214d ("nvme: add ANA support")Reported-by Martin Wilck <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Thank you. I'll prepare another test kernel for our partner.
Our partner reported a crash during NVMe controller initialization with the kernel I built with this patch applied. I'm still looking at the dump, and it's not impossible that I made a mistake backporting your patch. But I thought I should inform you anyway.
[ 1010.869437] nvme-fabrics ctl: Failed to read smart log (error -5) [ 1010.869444] nvme nvme0: queue_size 128 > ctrl sqsize 32, clamping down [ 1010.879383] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.1.14:4420 [ 1010.929700] nvme nvme0: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery" [ 1011.041659] BUG: kernel NULL pointer dereference, address: 0000000000000010 [ 1011.041665] #PF: supervisor write access in kernel mode [ 1011.041666] #PF: error_code(0x0002) - not-present page [ 1011.041668] PGD 0 P4D 0 [ 1011.041672] Oops: 0002 [#1] SMP PTI [ 1011.041675] CPU: 13 PID: 0 Comm: swapper/13 Kdump: loaded Tainted: G X 5.3.18-6.g7ea043c-default #1 SLE15-SP2 (unreleased) [ 1011.041678] Hardware name: FUJITSU PRIMERGY RX2530 M2/D3279-B1, BIOS V5.0.0.11 R1.20.0 for D3279-B1x 06/15/2018 [ 1011.041689] RIP: 0010:bio_copy_kern_endio_read+0xc6/0x130 [ 1011.041691] Code: c0 75 87 8b 4e 0c 44 89 df 89 ca 81 e1 ff 0f 00 00 c1 ea 0c 29 cf 48 c1 e2 06 89 f9 48 03 16 e9 6f ff ff ff 48 8b 3e 4c 89 c5 <49> 89 38 4a 8b 7c 0e f8 4b 89 7c 08 f8 49 8d 78 08 4d 01 c8 48 83 [ 1011.041695] RSP: 0018:ffffab41804c8ee8 EFLAGS: 00010212 [ 1011.041697] RAX: 0000000000000000 RBX: ffff9ff1b73e1500 RCX: 0000000000001000 [ 1011.041699] RDX: fffff2b810ce1240 RSI: ffff9ff1b3849000 RDI: 0000000000000000 [ 1011.041701] RBP: 0000000000000010 R08: 0000000000000010 R09: 0000000000001000 [ 1011.041702] R10: 0000000000000001 R11: 0000000000001000 R12: ffff9ff168ace140 [ 1011.041703] R13: 0000000000004810 R14: 0000000000000000 R15: 0000000000000000 [ 1011.041705] FS: 0000000000000000(0000) GS:ffff9ff1ff2c0000(0000) knlGS:0000000000000000 [ 1011.041707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1011.041708] CR2: 0000000000000010 CR3: 00000001de60a006 CR4: 00000000003606e0 [ 1011.041710] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1011.041711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1011.041713] Call Trace: [ 1011.041716] <IRQ> [ 1011.041722] blk_update_request+0x8a/0x3a0 [ 1011.041726] blk_mq_end_request+0x1a/0x130 [ 1011.041729] blk_done_softirq+0x8f/0xc0 [ 1011.041736] __do_softirq+0xe3/0x2dc [ 1011.041744] irq_exit+0xd5/0xe0 [ 1011.041747] call_function_single_interrupt+0xf/0x20 [ 1011.041749] </IRQ>
bio_copy_kern_endio_read() means that this was a command sent via __nvme_submit_sync_cmd(). I don't know yet which one.
Regards,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 6 lines
Our partner reported a crash during NVMe controller initialization with the kernel I built with this patch applied. I'm still looking at the dump, and it's not impossible that I made a mistake backporting your patch. But I thought I should inform you anyway.
Strange, the log indicates the discovery controller, which should not even have any ANA related activity...
Quoted 35 lines
[ 1010.869437] nvme-fabrics ctl: Failed to read smart log (error -5) [ 1010.869444] nvme nvme0: queue_size 128 > ctrl sqsize 32, clamping down [ 1010.879383] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.1.14:4420 [ 1010.929700] nvme nvme0: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery" [ 1011.041659] BUG: kernel NULL pointer dereference, address: 0000000000000010 [ 1011.041665] #PF: supervisor write access in kernel mode [ 1011.041666] #PF: error_code(0x0002) - not-present page [ 1011.041668] PGD 0 P4D 0 [ 1011.041672] Oops: 0002 [#1] SMP PTI [ 1011.041675] CPU: 13 PID: 0 Comm: swapper/13 Kdump: loaded Tainted: G X 5.3.18-6.g7ea043c-default #1 SLE15-SP2 (unreleased) [ 1011.041678] Hardware name: FUJITSU PRIMERGY RX2530 M2/D3279-B1, BIOS V5.0.0.11 R1.20.0 for D3279-B1x 06/15/2018 [ 1011.041689] RIP: 0010:bio_copy_kern_endio_read+0xc6/0x130 [ 1011.041691] Code: c0 75 87 8b 4e 0c 44 89 df 89 ca 81 e1 ff 0f 00 00 c1 ea 0c 29 cf 48 c1 e2 06 89 f9 48 03 16 e9 6f ff ff ff 48 8b 3e 4c 89 c5 <49> 89 38 4a 8b 7c 0e f8 4b 89 7c 08 f8 49 8d 78 08 4d 01 c8 48 83 [ 1011.041695] RSP: 0018:ffffab41804c8ee8 EFLAGS: 00010212 [ 1011.041697] RAX: 0000000000000000 RBX: ffff9ff1b73e1500 RCX: 0000000000001000 [ 1011.041699] RDX: fffff2b810ce1240 RSI: ffff9ff1b3849000 RDI: 0000000000000000 [ 1011.041701] RBP: 0000000000000010 R08: 0000000000000010 R09: 0000000000001000 [ 1011.041702] R10: 0000000000000001 R11: 0000000000001000 R12: ffff9ff168ace140 [ 1011.041703] R13: 0000000000004810 R14: 0000000000000000 R15: 0000000000000000 [ 1011.041705] FS: 0000000000000000(0000) GS:ffff9ff1ff2c0000(0000) knlGS:0000000000000000 [ 1011.041707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1011.041708] CR2: 0000000000000010 CR3: 00000001de60a006 CR4: 00000000003606e0 [ 1011.041710] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1011.041711] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1011.041713] Call Trace: [ 1011.041716] <IRQ> [ 1011.041722] blk_update_request+0x8a/0x3a0 [ 1011.041726] blk_mq_end_request+0x1a/0x130 [ 1011.041729] blk_done_softirq+0x8f/0xc0 [ 1011.041736] __do_softirq+0xe3/0x2dc [ 1011.041744] irq_exit+0xd5/0xe0 [ 1011.041747] call_function_single_interrupt+0xf/0x20 [ 1011.041749] </IRQ>
Quoted 9 lines
bio_copy_kern_endio_read()means that this was a command sent via__nvme_submit_sync_cmd(). I don't know yet which one.Regards,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
maintainerQuoted 14 lines · Martin Wilck
Hello Christoph,
Quoted 11 lines · Martin Wilck
Quoted 9 lines · Christoph Hellwig
nvme_init_identifyand thusnvme_mpath_initcan be called multiple times and thus must not overwrite potentially initialized or in-use fields. Split out a helper for the basic initialization when the controller is initialized and make sure theinit_identifypath does not blindly change in-use data structures.Fixes 0d0b660f214d ("nvme: add ANA support")Reported-by Martin Wilck <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>
Quoted 6 lines
Quoted 1 line
Thank you. I'll prepare another test kernel for our partner.
Our partner reported a crash during NVMe controller initialization with the kernel I built with this patch applied. I'm still looking at the dump, and it's not impossible that I made a mistake backporting your patch. But I thought I should inform you anyway.
It does look weird for sure. Did you backport
"nvmet: use new ana_log_size instead the old one" as well? If not please pick it up.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 29 lines · Christoph Hellwig
Quoted 20 lines · Martin Wilck
Hello Christoph,
Quoted 11 lines · Martin Wilck
Quoted 10 lines · Christoph Hellwig
nvme_init_identifyand thusnvme_mpath_initcan be called multiple
times and thus must not overwrite potentially initialized or in-use
fields. Split out a helper for the basic initialization when the
controller is initialized and make sure theinit_identifypath does
not blindly change in-use data structures.Quoted 5 lines
Quoted 3 lines
Fixes 0d0b660f214d ("nvme: add ANA support")Reported-by Martin Wilck <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Thank you. I'll prepare another test kernel for our partner.
Quoted 7 lines
Our partner reported a crash during NVMe controller initialization with
the kernel I built with this patch applied. I'm still looking at the
dump, and it's not impossible that I made a mistake backporting your
patch. But I thought I should inform you anyway.
Quoted 4 lines
It does look weird for sure. Did you backport
"nvmet: use newana_log_sizeinstead the old one" as well? If not please
pick it up.
I missed that one indeed, and it's the culprit.
Regards and thanks,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 32 lines · Martin Wilck
Quoted 31 lines · Christoph Hellwig
Quoted 29 lines · Martin Wilck
Hello Christoph,
Quoted 12 lines · Martin Wilck
Quoted 11 lines · Christoph Hellwig
nvme_init_identifyand thusnvme_mpath_initcan be called multiple
times and thus must not overwrite potentially initialized or in-use
fields. Split out a helper for the basic initialization when the
controller is initialized and make sure theinit_identifypath
does
not blindly change in-use data structures.Quoted 5 lines
Quoted 3 lines
Fixes 0d0b660f214d ("nvme: add ANA support")Reported-by Martin Wilck <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Thank you. I'll prepare another test kernel for our partner.
Our partner reported a crash during NVMe controller initialization
with
the kernel I built with this patch applied. I'm still looking at the
dump, and it's not impossible that I made a mistake backporting your
patch. But I thought I should inform you anyway.
Quoted 6 lines
Quoted 4 lines
It does look weird for sure. Did you backport
"nvmet: use newana_log_sizeinstead the old one" as well? If not please
pick it up.I missed that one indeed, and it's the culprit.
I wrote nonsense, sorry.
Applying "nvmet: use new ana_log_size instead the old one" fixed the issue for me.
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme