Skip to contentSkip to conversation

Saw I/O errors while delete/create/attach a namespace on nvme device.

DISCUSSION Wen Xiong 4 participants 10 messages Nov 7, 2023 – Nov 8, 2023 NVMe

lore
Share & tools
lore

Wen Xiong

author
#0 lore raw

Hi All,

I am working on new nvme device and found this: nguid is changed while delete/create/attach a namespace, we saw some error messages in linux log.

Should we see these errors messages since recreating namespaces or are there some issues in error path?

# dmesg
[10524.330156] nvme nvme0: rescanning namespaces.
[10524.338810] nvme nvme0: identifiers changed for nsid 1
[10524.343448] block nvme0n1: no usable path - requeuing I/O
[10524.480468] block nvme0n1: no available path - failing I/O
[10524.480497] block nvme0n1: no available path - failing I/O
[10524.480501] Buffer I/O error on dev nvme0n1, logical block 781402912, async page read
[10524.480508] block nvme0n1: no available path - failing I/O
[10524.480510] Buffer I/O error on dev nvme0n1, logical block 781402913, async page read
[10524.480515] block nvme0n1: no available path - failing I/O
[10524.480517] Buffer I/O error on dev nvme0n1, logical block 781402914, async page read
[10524.480524] block nvme0n1: no available path - failing I/O
[10524.480525] Buffer I/O error on dev nvme0n1, logical block 781402915, async page read
[10524.480528] block nvme0n1: no available path - failing I/O
[10524.480529] Buffer I/O error on dev nvme0n1, logical block 781402916, async page read
[10524.480531] block nvme0n1: no available path - failing I/O
[10524.480535] Buffer I/O error on dev nvme0n1, logical block 781402917, async page read
[10524.480539] block nvme0n1: no available path - failing I/O
[10524.480541] Buffer I/O error on dev nvme0n1, logical block 781402918, async page read
[10524.480546] block nvme0n1: no available path - failing I/O
[10524.480548] Buffer I/O error on dev nvme0n1, logical block 781402919, async page read

Thanks for your help
Wendy

#1 lore raw
Quoted 4 lines · Wen Xiong

Hi All,

I am working on new nvme device and found this: nguid is changed while delete/create/attach a namespace, we saw some error messages in linux log.

Please provide the detailed steps from start which has lead to this behavior, also please provide git repo/branch/branch head that you are using.

-ck

#2 lore raw
In reply to Wen Xiong
Quoted 7 lines · Wen Xiong

Hi All,

I am working on new nvme device and found this: nguid is changed while delete/create/attach a namespace, we saw some error messages in linux log.

Should we see these errors messages since recreating namespaces or are there some issues in error path?

What exactly are you doing? To the Linux host code this looks like the NGUID changed for an existing namespace. Are you deleting and recreating nsid1 rapidly and the controller is assigning a different nguid?

#3 lore raw
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Wen Xiong

Hi All,

I am working on new nvme device and found this: nguid is changed while delete/create/attach a namespace, we saw some error messages in linux log.

Quoted 8 lines
Quoted 2 lines

Should we see these errors messages since recreating namespaces or are there some issues in error path?

What exactly are you doing? To the Linux host code this looks like the NGUID changed for an existing namespace. Are you deleting and recreating nsid1 rapidly and the controller is assigning a different nguid?

exactly "identifiers changed for nsid XXX" comes from nvme_validate_ns() and the only caller is nvme_scan_ns() where ns is present, so scan ns found the namespace and nvme_ns_ids_eaqual() returned an error hence 2nd print
message, but as said earlier we really need to see the exact steps ...

To check exactly which id is problematic something like in [1] can be used, totally untested ...

-ck

[1]

drivers/nvme/host/core.c +0 -0
index 75a1b58a7a43..84651c922548 100644
@@ -1749,10 +1749,23 @@ static void nvme_config_discard(struct gendisk 

*disk, struct nvme_ns *ns)

 static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct
nvme_ns_ids *b)
 {
-       return uuid_equal(&a->uuid, &b->uuid) &&
-               memcmp(&a->nguid, &b->nguid, sizeof(a->nguid)) == 0 && -               memcmp(&a->eui64, &b->eui64, sizeof(a->eui64)) == 0 && -               a->csi == b->csi;
+       if (uuid_equal(&a->uuid, &b->uuid) == false) {
+               pr_info("uuid mismatch\n");
+               return false;
+       }
+       if (memcmp(&a->nguid, &b->nguid, sizeof(a->nguid)) != 0) { +               pr_info("nguid mismatch\n");
+               return false;
+       }
+       if (memcmp(&a->eui64, &b->eui64, sizeof(a->eui64)) != 0) { +               pr_info("euid64 mismaatch\n");
+               return false;
+       }
+       if (a->csi != b->csi) {
+               pr_info("csi mismatch\n");
+               return false;
+       }
+       return true;
 }

 static int nvme_init_ms(struct nvme_ns *ns, struct nvme_id_ns *id)

author
#4 lore raw
Quoted 4 lines

To check exactly which id is problematic something like in [1] can be used, totally untested ...

Steps:

# nvme detach-ns /dev/nvme0 --namespace-id=1 --controllers=0x81
# nvme delete-ns /dev/nvme0 --namespace-id=1
# nvme create-ns /dev/nvme0 --nsze=562805846 --ncap=562805846 --flbas=0 -dps=0 -nmic=1
# nvme attach-ns /dev/nvme0 -n 1  --controller=0x81

Below is linux log with your patch:

[  149.570987] nvme nvme0: rescanning namespaces.
[  149.578714] nguid mismatch
[  149.578719] nvme nvme0: identifiers changed for nsid 1
[  149.582291] block nvme0n1: no usable path - requeuing I/O
[  149.722140] block nvme0n1: no available path - failing I/O
[  149.722157] block nvme0n1: no available path - failing I/O
[  149.722165] Buffer I/O error on dev nvme0n1, logical block 281402912, async page read
[  149.722171] block nvme0n1: no available path - failing I/O
[  149.722175] Buffer I/O error on dev nvme0n1, logical block 281402913, async page read
[  149.722181] block nvme0n1: no available path - failing I/O
[  149.722185] Buffer I/O error on dev nvme0n1, logical block 281402914, async page read
[  149.722191] block nvme0n1: no available path - failing I/O
[  149.722195] Buffer I/O error on dev nvme0n1, logical block 281402915, async page read
[  149.722203] block nvme0n1: no available path - failing I/O
[  149.722208] Buffer I/O error on dev nvme0n1, logical block 281402916, async page read
[  149.722217] block nvme0n1: no available path - failing I/O
[  149.722226] Buffer I/O error on dev nvme0n1, logical block 281402917, async page read
[  149.722231] block nvme0n1: no available path - failing I/O
[  149.722233] Buffer I/O error on dev nvme0n1, logical block 281402918, async page read
[  149.722237] block nvme0n1: no available path - failing I/O
[  149.722239] Buffer I/O error on dev nvme0n1, logical block 281402919, async page read

[root@ltcrain119-lp4 ~]#

Below is nguid changes:

# nvme id-ns /dev/nvme0n1|grep nguid
nguid   : 37444630577000630025384700000245

nvme id-ns /dev/nvme0n1|grep nguid
nguid : 37444630577000630025384700000246

Thanks a lot!
Wendy

maintainer
#5 lore raw
Quoted 11 lines · Wen Xiong
Quoted 4 lines

To check exactly which id is problematic something like in [1] can be used, totally untested ...

Steps:

# nvme detach-ns /dev/nvme0 --namespace-id=1 --controllers=0x81
# nvme delete-ns /dev/nvme0 --namespace-id=1
# nvme create-ns /dev/nvme0 --nsze=562805846 --ncap=562805846 --flbas=0
-dps=0 -nmic=1
# nvme attach-ns /dev/nvme0 -n 1  --controller=0x81
Quoted 3 lines

Below is linux log with your patch:

[  149.570987] nvme nvme0: rescanning namespaces.

Are you running these commands in quick succession? There should be a "rescanning namespaces" message right after the 'detach-ns' command, and before subsequent 'attach-ns' command. It looks here that the rescan didn't run until after the 'attach-ns' occured. Instead of tearing down the original, the driver just sees the namespace it previously knew about has changed unexpectedly; the processing for the namespace removal didn't happen prior to the attach-ns command.

Quoted 5 lines
[  149.578714] nguid mismatch
[  149.578719] nvme nvme0: identifiers changed for nsid 1
[  149.582291] block nvme0n1: no usable path - requeuing I/O
[  149.722140] block nvme0n1: no available path - failing I/O
[  149.722157] block nvme0n1: no available path - failing I/O

If you drop all open references to /dev/nvme0n1, then the handle should get deleted, and a manual rescan after that should get your new namespace visible.

author
#6 lore raw

On 2023-11-07 09:18, Keith Busch wrote:

Hi Keith,

Quoted 6 lines

"rescanning namespaces" message right after the 'detach-ns' command, and before subsequent 'attach-ns' command. It looks here that the rescan didn't run until after the 'attach-ns' occured. Instead of tearing down the original, the driver just sees the namespace it previously knew about has changed unexpectedly; the processing for the namespace removal didn't happen prior to the attach-ns command.

Re-did:

# nvme detach-ns /dev/nvme0 --namespace-id=1 --controllers=0x81
detach-ns: Success, nsid:1
# dmesg
[ 4804.431303] nvme nvme0: rescanning namespaces
# nvme delete-ns /dev/nvme0 --namespace-id=1
delete-ns: Success, deleted nsid:1
# dmesg
[ 4804.431303] nvme nvme0: rescanning namespaces.
# nvme create-ns /dev/nvme0 --nsze=562805846 --ncap=562805846 --flbas=0 -dps=0 -nmic=1
create-ns: Success, created nsid:1
[root@ltcrain119-lp4 ~]# dmesg
[ 4804.431303] nvme nvme0: rescanning namespaces.
Quoted 3 lines

If you drop all open references to /dev/nvme0n1, then the handle should get deleted, and a manual rescan after that should get your new namespace visible.

# nvme attach-ns /dev/nvme0 -n 1  --controller=0x81
attach-ns: Success, nsid:1
# dmesg
[ 4804.431303] nvme nvme0: rescanning namespaces.
[ 5219.493625] nvme nvme0: rescanning namespaces.
[ 5219.502136] nguid mismatch
[ 5219.502146] nvme nvme0: identifiers changed for nsid 1
[ 5219.506668] block nvme0n1: no usable path - requeuing I/O
[ 5219.662788] block nvme0n1: no available path - failing I/O
[ 5219.662824] block nvme0n1: no available path - failing I/O
[ 5219.662841] Buffer I/O error on dev nvme0n1, logical block 281402912, async page read
[ 5219.662859] block nvme0n1: no available path - failing I/O
[ 5219.662875] Buffer I/O error on dev nvme0n1, logical block 281402913, async page read
[ 5219.662887] block nvme0n1: no available path - failing I/O
[ 5219.662894] Buffer I/O error on dev nvme0n1, logical block 281402914, async page read
[ 5219.662913] block nvme0n1: no available path - failing I/O
[ 5219.662926] Buffer I/O error on dev nvme0n1, logical block 281402915, async page read
[ 5219.662956] block nvme0n1: no available path - failing I/O
[ 5219.662970] Buffer I/O error on dev nvme0n1, logical block 281402916, async page read
[ 5219.662985] bio_check_eod: 7 callbacks suppressed
[ 5219.662988] systemd-udevd: attempt to access beyond end of device
               nvme0n1: rw=0, sector=4502446672, nr_sectors = 16 limit=0
[ 5219.663022] Buffer I/O error on dev nvme0n1, logical block 281402917, async page read
[ 5219.663035] systemd-udevd: attempt to access beyond end of device
               nvme0n1: rw=0, sector=4502446688, nr_sectors = 16 limit=0
[ 5219.663052] Buffer I/O error on dev nvme0n1, logical block 281402918, async page read
[ 5219.663065] systemd-udevd: attempt to access beyond end of device
               nvme0n1: rw=0, sector=4502446704, nr_sectors = 16 limit=0
[ 5219.663099] Buffer I/O error on dev nvme0n1, logical block 281402919, async page read
# nvme ns-rescan /dev/nvme0n1
/dev/nvme0n1: No such file or directory
Usage: nvme ns-rescan <device> [OPTIONS]

Rescans the NVMe namespaces

# nvme ns-rescan /dev/nvme0n1
/dev/nvme0n1: No such file or directory
Usage: nvme ns-rescan <device> [OPTIONS]

Rescans the NVMe namespaces

# ls -l /dev/nvme*
crw-------. 1 root root 240, 0 Nov  7 08:26 /dev/nvme0
crw-------. 1 root root 240, 1 Nov  7 08:13 /dev/nvme1
brw-rw----. 1 root disk 259, 1 Nov  7 08:13 /dev/nvme1n1

[root@ltcrain119-lp4 ~]# nvme attach-ns /dev/nvme0 -n 1 --controller=0x81
NVMe status: Namespace Already Attached: The controller is already attached to the namespace specified(0x2118)

[root@ltcrain119-lp4 ~]# ls -l /dev/nvme*

crw-------. 1 root root 240, 0 Nov  7 08:26 /dev/nvme0
brw-rw----. 1 root disk 259, 3 Nov  7 09:48 /dev/nvme0n1
crw-------. 1 root root 240, 1 Nov  7 08:13 /dev/nvme1
brw-rw----. 1 root disk 259, 1 Nov  7 08:13 /dev/nvme1n1

After attach-ns command, /dev/nvme0n1 is not showed up in /dev/*, somehow I have to do the 2nd attach-ns command, nvme ns-rescan works after the 2nd attach-ns.

Is a firmware issue on nvme device?

Thanks,
Wendy

author
#7 lore raw
In reply to Keith Busch

On 2023-11-07 09:18, Keith Busch wrote:

Hi Keith,

Quoted 3 lines

If you drop all open references to /dev/nvme0n1, then the handle should get deleted, and a manual rescan after that should get your new namespace visible.

Should we call nvme_ns_scan(*ctrl, nsid) again if nguid/uuid/eui64/csi are changed when recreating a namespace? Customers don't need to run ns-attach/ns-rescan manually in user space.

Thanks!
Wendy

#8 lore raw
In reply to Keith Busch
Quoted 7 lines · Keith Busch

Are you running these commands in quick succession? There should be a "rescanning namespaces" message right after the 'detach-ns' command, and before subsequent 'attach-ns' command. It looks here that the rescan didn't run until after the 'attach-ns' occured. Instead of tearing down the original, the driver just sees the namespace it previously knew about has changed unexpectedly; the processing for the namespace removal didn't happen prior to the attach-ns command.

Yep.

Quoted 10 lines
Quoted 6 lines
[  149.578714] nguid mismatch
[  149.578719] nvme nvme0: identifiers changed for nsid 1
[  149.582291] block nvme0n1: no usable path - requeuing I/O
[  149.722140] block nvme0n1: no available path - failing I/O
[  149.722157] block nvme0n1: no available path - failing I/O

If you drop all open references to /dev/nvme0n1, then the handle should get deleted, and a manual rescan after that should get your new namespace visible.

I fear we still need to handle this somehow. For actual per-spec namespce management we'll just need to snoop the namespace management commands and update the ns_head membership. For out of band management there's not much we can do as-is. A good addition to the spec would be to add the concept of a namespace generation that is incremented every time the nsid is reused.

author
#9 lore raw
In reply to Christoph Hellwig
Quoted 5 lines

What exactly are you doing? To the Linux host code this looks like the NGUID changed for an existing namespace. Are you deleting and recreating nsid1 rapidly and the controller is assigning a different nguid?

Hi Christoph,

Good morning!

Yes. controller assigned a difference nguid to a namespace.

Steps:
Executed nvme detach-ns /dev/nvme0 --namespace-id=1 --controllers=0x81
Executed nvme delete-ns /dev/nvme0 --namespace-id=1
Executed nvme create-ns /dev/nvme0 --nsze=1562805846 --ncap=1562805846 --flbas=0 -dps=0 -nmic=1 Executed nvme attach-ns /dev/nvme --namespace-id=1 --controllers=0x81

Saw NGUID changed and IO errors when executing attach-ns command.

Thanks a lot!
Wendy