Saw I/O errors while delete/create/attach a namespace on nvme device.
Hi All,
I am working on new nvme device and found this: nguid is changed while delete/create/attach a namespace, we saw some error messages in linux log.
Should we see these errors messages since recreating namespaces or are there some issues in error path?
# dmesg [10524.330156] nvme nvme0: rescanning namespaces. [10524.338810] nvme nvme0: identifiers changed for nsid 1 [10524.343448] block nvme0n1: no usable path - requeuing I/O [10524.480468] block nvme0n1: no available path - failing I/O [10524.480497] block nvme0n1: no available path - failing I/O [10524.480501] Buffer I/O error on dev nvme0n1, logical block 781402912, async page read [10524.480508] block nvme0n1: no available path - failing I/O [10524.480510] Buffer I/O error on dev nvme0n1, logical block 781402913, async page read [10524.480515] block nvme0n1: no available path - failing I/O [10524.480517] Buffer I/O error on dev nvme0n1, logical block 781402914, async page read [10524.480524] block nvme0n1: no available path - failing I/O [10524.480525] Buffer I/O error on dev nvme0n1, logical block 781402915, async page read [10524.480528] block nvme0n1: no available path - failing I/O [10524.480529] Buffer I/O error on dev nvme0n1, logical block 781402916, async page read [10524.480531] block nvme0n1: no available path - failing I/O [10524.480535] Buffer I/O error on dev nvme0n1, logical block 781402917, async page read [10524.480539] block nvme0n1: no available path - failing I/O [10524.480541] Buffer I/O error on dev nvme0n1, logical block 781402918, async page read [10524.480546] block nvme0n1: no available path - failing I/O [10524.480548] Buffer I/O error on dev nvme0n1, logical block 781402919, async page read
Thanks for your help
Wendy
Quoted 4 lines · Wen Xiong
Hi All,
I am working on new nvme device and found this: nguid is changed while delete/create/attach a namespace, we saw some error messages in linux log.
Please provide the detailed steps from start which has lead to this behavior, also please provide git repo/branch/branch head that you are using.
-ck
Quoted 7 lines · Wen Xiong
Hi All,
I am working on new nvme device and found this: nguid is changed while delete/create/attach a namespace, we saw some error messages in linux log.
Should we see these errors messages since recreating namespaces or are there some issues in error path?
What exactly are you doing? To the Linux host code this looks like the NGUID changed for an existing namespace. Are you deleting and recreating nsid1 rapidly and the controller is assigning a different nguid?
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Wen Xiong
Hi All,
I am working on new nvme device and found this: nguid is changed while delete/create/attach a namespace, we saw some error messages in linux log.
Quoted 8 lines
Quoted 2 lines
Should we see these errors messages since recreating namespaces or are there some issues in error path?
What exactly are you doing? To the Linux host code this looks like the NGUID changed for an existing namespace. Are you deleting and recreating nsid1 rapidly and the controller is assigning a different nguid?
exactly "identifiers changed for nsid XXX" comes from nvme_validate_ns() and the only caller is nvme_scan_ns() where ns is present, so scan ns found the namespace and nvme_ns_ids_eaqual() returned an error hence 2nd print
message, but as said earlier we really need to see the exact steps ...
To check exactly which id is problematic something like in [1] can be used, totally untested ...
-ck
[1]
drivers/nvme/host/core.c +0 -0
@@ -1749,10 +1749,23 @@ static void nvme_config_discard(struct gendisk
*disk, struct nvme_ns *ns)
static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, structnvme_ns_ids *b)
{
- return uuid_equal(&a->uuid, &b->uuid) &&
- memcmp(&a->nguid, &b->nguid, sizeof(a->nguid)) == 0 && - memcmp(&a->eui64, &b->eui64, sizeof(a->eui64)) == 0 && - a->csi == b->csi;
+ if (uuid_equal(&a->uuid, &b->uuid) == false) {
+ pr_info("uuid mismatch\n");
+ return false;
+ }
+ if (memcmp(&a->nguid, &b->nguid, sizeof(a->nguid)) != 0) { + pr_info("nguid mismatch\n");
+ return false;
+ }
+ if (memcmp(&a->eui64, &b->eui64, sizeof(a->eui64)) != 0) { + pr_info("euid64 mismaatch\n");
+ return false;
+ }
+ if (a->csi != b->csi) {
+ pr_info("csi mismatch\n");
+ return false;
+ }
+ return true;
}
static int nvme_init_ms(struct nvme_ns *ns, struct nvme_id_ns *id)
Quoted 4 lines
To check exactly which id is problematic something like in [1] can be used, totally untested ...
Steps:
# nvme detach-ns /dev/nvme0 --namespace-id=1 --controllers=0x81 # nvme delete-ns /dev/nvme0 --namespace-id=1 # nvme create-ns /dev/nvme0 --nsze=562805846 --ncap=562805846 --flbas=0 -dps=0 -nmic=1 # nvme attach-ns /dev/nvme0 -n 1 --controller=0x81
Below is linux log with your patch:
[ 149.570987] nvme nvme0: rescanning namespaces. [ 149.578714] nguid mismatch [ 149.578719] nvme nvme0: identifiers changed for nsid 1 [ 149.582291] block nvme0n1: no usable path - requeuing I/O [ 149.722140] block nvme0n1: no available path - failing I/O [ 149.722157] block nvme0n1: no available path - failing I/O [ 149.722165] Buffer I/O error on dev nvme0n1, logical block 281402912, async page read [ 149.722171] block nvme0n1: no available path - failing I/O [ 149.722175] Buffer I/O error on dev nvme0n1, logical block 281402913, async page read [ 149.722181] block nvme0n1: no available path - failing I/O [ 149.722185] Buffer I/O error on dev nvme0n1, logical block 281402914, async page read [ 149.722191] block nvme0n1: no available path - failing I/O [ 149.722195] Buffer I/O error on dev nvme0n1, logical block 281402915, async page read [ 149.722203] block nvme0n1: no available path - failing I/O [ 149.722208] Buffer I/O error on dev nvme0n1, logical block 281402916, async page read [ 149.722217] block nvme0n1: no available path - failing I/O [ 149.722226] Buffer I/O error on dev nvme0n1, logical block 281402917, async page read [ 149.722231] block nvme0n1: no available path - failing I/O [ 149.722233] Buffer I/O error on dev nvme0n1, logical block 281402918, async page read [ 149.722237] block nvme0n1: no available path - failing I/O [ 149.722239] Buffer I/O error on dev nvme0n1, logical block 281402919, async page read
[root@ltcrain119-lp4 ~]#
Below is nguid changes:
# nvme id-ns /dev/nvme0n1|grep nguid nguid : 37444630577000630025384700000245
nvme id-ns /dev/nvme0n1|grep nguid
nguid : 37444630577000630025384700000246
Thanks a lot!
Wendy
Keith Busch
maintainerQuoted 11 lines · Wen Xiong
Quoted 4 lines
To check exactly which id is problematic something like in [1] can be used, totally untested ...
Steps:
# nvme detach-ns /dev/nvme0 --namespace-id=1 --controllers=0x81 # nvme delete-ns /dev/nvme0 --namespace-id=1 # nvme create-ns /dev/nvme0 --nsze=562805846 --ncap=562805846 --flbas=0 -dps=0 -nmic=1 # nvme attach-ns /dev/nvme0 -n 1 --controller=0x81
Quoted 3 lines
Below is linux log with your patch:
[ 149.570987] nvme nvme0: rescanning namespaces.
Are you running these commands in quick succession? There should be a "rescanning namespaces" message right after the 'detach-ns' command, and before subsequent 'attach-ns' command. It looks here that the rescan didn't run until after the 'attach-ns' occured. Instead of tearing down the original, the driver just sees the namespace it previously knew about has changed unexpectedly; the processing for the namespace removal didn't happen prior to the attach-ns command.
Quoted 5 lines
[ 149.578714] nguid mismatch [ 149.578719] nvme nvme0: identifiers changed for nsid 1 [ 149.582291] block nvme0n1: no usable path - requeuing I/O [ 149.722140] block nvme0n1: no available path - failing I/O [ 149.722157] block nvme0n1: no available path - failing I/O
If you drop all open references to /dev/nvme0n1, then the handle should get deleted, and a manual rescan after that should get your new namespace visible.
On 2023-11-07 09:18, Keith Busch wrote:
Hi Keith,
Quoted 6 lines
"rescanning namespaces" message right after the 'detach-ns' command, and before subsequent 'attach-ns' command. It looks here that the rescan didn't run until after the 'attach-ns' occured. Instead of tearing down the original, the driver just sees the namespace it previously knew about has changed unexpectedly; the processing for the namespace removal didn't happen prior to the attach-ns command.
Re-did:
# nvme detach-ns /dev/nvme0 --namespace-id=1 --controllers=0x81 detach-ns: Success, nsid:1 # dmesg [ 4804.431303] nvme nvme0: rescanning namespaces # nvme delete-ns /dev/nvme0 --namespace-id=1 delete-ns: Success, deleted nsid:1 # dmesg [ 4804.431303] nvme nvme0: rescanning namespaces. # nvme create-ns /dev/nvme0 --nsze=562805846 --ncap=562805846 --flbas=0 -dps=0 -nmic=1 create-ns: Success, created nsid:1 [root@ltcrain119-lp4 ~]# dmesg [ 4804.431303] nvme nvme0: rescanning namespaces.
Quoted 3 lines
If you drop all open references to
/dev/nvme0n1, then the handle should get deleted, and a manual rescan after that should get your new namespace visible.
# nvme attach-ns /dev/nvme0 -n 1 --controller=0x81
attach-ns: Success, nsid:1
# dmesg
[ 4804.431303] nvme nvme0: rescanning namespaces.
[ 5219.493625] nvme nvme0: rescanning namespaces.
[ 5219.502136] nguid mismatch
[ 5219.502146] nvme nvme0: identifiers changed for nsid 1
[ 5219.506668] block nvme0n1: no usable path - requeuing I/O
[ 5219.662788] block nvme0n1: no available path - failing I/O
[ 5219.662824] block nvme0n1: no available path - failing I/O
[ 5219.662841] Buffer I/O error on dev nvme0n1, logical block 281402912, async page read
[ 5219.662859] block nvme0n1: no available path - failing I/O
[ 5219.662875] Buffer I/O error on dev nvme0n1, logical block 281402913, async page read
[ 5219.662887] block nvme0n1: no available path - failing I/O
[ 5219.662894] Buffer I/O error on dev nvme0n1, logical block 281402914, async page read
[ 5219.662913] block nvme0n1: no available path - failing I/O
[ 5219.662926] Buffer I/O error on dev nvme0n1, logical block 281402915, async page read
[ 5219.662956] block nvme0n1: no available path - failing I/O
[ 5219.662970] Buffer I/O error on dev nvme0n1, logical block 281402916, async page read
[ 5219.662985] bio_check_eod: 7 callbacks suppressed
[ 5219.662988] systemd-udevd: attempt to access beyond end of device
nvme0n1: rw=0, sector=4502446672, nr_sectors = 16 limit=0
[ 5219.663022] Buffer I/O error on dev nvme0n1, logical block 281402917, async page read
[ 5219.663035] systemd-udevd: attempt to access beyond end of device
nvme0n1: rw=0, sector=4502446688, nr_sectors = 16 limit=0
[ 5219.663052] Buffer I/O error on dev nvme0n1, logical block 281402918, async page read
[ 5219.663065] systemd-udevd: attempt to access beyond end of device
nvme0n1: rw=0, sector=4502446704, nr_sectors = 16 limit=0
[ 5219.663099] Buffer I/O error on dev nvme0n1, logical block 281402919, async page read
# nvme ns-rescan /dev/nvme0n1
/dev/nvme0n1: No such file or directory
Usage: nvme ns-rescan <device> [OPTIONS]Rescans the NVMe namespaces
# nvme ns-rescan /dev/nvme0n1 /dev/nvme0n1: No such file or directory Usage: nvme ns-rescan <device> [OPTIONS]
Rescans the NVMe namespaces
# ls -l /dev/nvme* crw-------. 1 root root 240, 0 Nov 7 08:26 /dev/nvme0 crw-------. 1 root root 240, 1 Nov 7 08:13 /dev/nvme1 brw-rw----. 1 root disk 259, 1 Nov 7 08:13 /dev/nvme1n1
[root@ltcrain119-lp4 ~]# nvme attach-ns /dev/nvme0 -n 1 --controller=0x81
NVMe status: Namespace Already Attached: The controller is already attached to the namespace specified(0x2118)
[root@ltcrain119-lp4 ~]# ls -l /dev/nvme*
crw-------. 1 root root 240, 0 Nov 7 08:26 /dev/nvme0 brw-rw----. 1 root disk 259, 3 Nov 7 09:48 /dev/nvme0n1 crw-------. 1 root root 240, 1 Nov 7 08:13 /dev/nvme1 brw-rw----. 1 root disk 259, 1 Nov 7 08:13 /dev/nvme1n1
After attach-ns command, /dev/nvme0n1 is not showed up in /dev/*, somehow I have to do the 2nd attach-ns command, nvme ns-rescan works after the 2nd attach-ns.
Is a firmware issue on nvme device?
Thanks,
Wendy
On 2023-11-07 09:18, Keith Busch wrote:
Hi Keith,
Quoted 3 lines
If you drop all open references to
/dev/nvme0n1, then the handle should get deleted, and a manual rescan after that should get your new namespace visible.
Should we call nvme_ns_scan(*ctrl, nsid) again if nguid/uuid/eui64/csi are changed when recreating a namespace? Customers don't need to run ns-attach/ns-rescan manually in user space.
Thanks!
Wendy
Quoted 7 lines · Keith Busch
Are you running these commands in quick succession? There should be a "rescanning namespaces" message right after the 'detach-ns' command, and before subsequent 'attach-ns' command. It looks here that the rescan didn't run until after the 'attach-ns' occured. Instead of tearing down the original, the driver just sees the namespace it previously knew about has changed unexpectedly; the processing for the namespace removal didn't happen prior to the attach-ns command.
Yep.
Quoted 10 lines
Quoted 6 lines
[ 149.578714] nguid mismatch [ 149.578719] nvme nvme0: identifiers changed for nsid 1 [ 149.582291] block nvme0n1: no usable path - requeuing I/O [ 149.722140] block nvme0n1: no available path - failing I/O [ 149.722157] block nvme0n1: no available path - failing I/OIf you drop all open references to
/dev/nvme0n1, then the handle should get deleted, and a manual rescan after that should get your new namespace visible.
I fear we still need to handle this somehow. For actual per-spec namespce management we'll just need to snoop the namespace management commands and update the ns_head membership. For out of band management there's not much we can do as-is. A good addition to the spec would be to add the concept of a namespace generation that is incremented every time the nsid is reused.
Quoted 5 lines
What exactly are you doing? To the Linux host code this looks like the NGUID changed for an existing namespace. Are you deleting and recreating nsid1 rapidly and the controller is assigning a different nguid?
Hi Christoph,
Good morning!
Yes. controller assigned a difference nguid to a namespace.
Steps:
Executed nvme detach-ns /dev/nvme0 --namespace-id=1 --controllers=0x81
Executed nvme delete-ns /dev/nvme0 --namespace-id=1
Executed nvme create-ns /dev/nvme0 --nsze=1562805846 --ncap=1562805846 --flbas=0 -dps=0 -nmic=1 Executed nvme attach-ns /dev/nvme --namespace-id=1 --controllers=0x81
Saw NGUID changed and IO errors when executing attach-ns command.
Thanks a lot!
Wendy