Skip to contentSkip to conversation

PCI: vmd: Create domain symlink before pci_bus_add_devices

PATCH Superseded Jiwei Sun 3 participants 4 messages Jun 3, 2024 – Jun 4, 2024

Subsystems (3)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jiwei Sun

author
#0 lore raw

PATCH 1/1 PCI: vmd: Create domain symlink before pci_bus_add_devices

From: Jiwei Sun <[email protected]>

During booting into the kernel, the following error message appears:

(udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: Unable to get real path for '/sys/bus/pci/drivers/vmd/0000:c7:00.5/domain/device'' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: /dev/nvme1n1 is not attached to Intel(R) RAID controller.' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: No OROM/EFI properties for /dev/nvme1n1' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: no RAID superblock on /dev/nvme1n1.' (udev-worker)[2149]: nvme1n1: Process '/sbin/mdadm -I /dev/nvme1n1' failed with exit code 1.

This symptom prevents the OS from booting successfully.

After a NVMe disk is probed/added by the nvme driver, the udevd executes some rule scripts by invoking mdadm command to detect if there is a mdraid associated with this NVMe disk. The mdadm determines if one NVMe devce is connected to a particular VMD domain by checking the domain symlink. Here is the root cause:

Thread A Thread B Thread mdadm
vmd_enable_domain
pci_bus_add_devices
__driver_probe_device
...
work_on_cpu
schedule_work_on
: wakeup Thread B
nvme_probe
: wakeup scan_work
to scan nvme disk
and add nvme disk
then wakeup udevd
: udevd executes
mdadm command
flush_work main
: wait for nvme_probe done ...
__driver_probe_device find_driver_devices
: probe next nvme device : 1) Detect the domain
... symlink; 2) Find the
... domain symlink from
... vmd sysfs; 3) The
... domain symlink is not
... created yet, failed
sysfs_create_link
: create domain symlink

sysfs_create_link is invoked at the end of vmd_enable_domain. However, this implementation introduces a timing issue, where mdadm might fail to retrieve the vmd symlink path because the symlink has not been created yet.

Fix the issue by creating VMD domain symlinks before invoking pci_bus_add_devices.

---
 drivers/pci/controller/vmd.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/pci/controller/vmd.c +3 -3
index 87b7856f375a..3f208c5f9ec9 100644
@@ -961,12 +961,12 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	list_for_each_entry(child, &vmd->bus->children, node)
 		pcie_bus_configure_settings(child);
 
+	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
+			       "domain"), "Can't create symlink to domain\n");
+
 	pci_bus_add_devices(vmd->bus);
 
 	vmd_acpi_end();
-
-	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
-			       "domain"), "Can't create symlink to domain\n");
 	return 0;
 }
 
Signature · 1 line
2.27.0
#1 lore raw
Quoted 4 lines · Jiwei Sun

From: Jiwei Sun <[email protected]>

During booting into the kernel, the following error message appears:

Quoted 8 lines
   (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: Unable to get real path for '/sys/bus/pci/drivers/vmd/0000:c7:00.5/domain/device''
   (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: /dev/nvme1n1 is not attached to Intel(R) RAID controller.'
   (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: No OROM/EFI properties for /dev/nvme1n1'
   (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: no RAID superblock on /dev/nvme1n1.'
   (udev-worker)[2149]: nvme1n1: Process '/sbin/mdadm -I /dev/nvme1n1' failed with exit code 1.

This symptom prevents the OS from booting successfully.

I'm just curious: has this been doing this forever or has this just started recently?

Paul

Quoted 67 lines

After a NVMe disk is probed/added by the nvme driver, the udevd executes some rule scripts by invoking mdadm command to detect if there is a mdraid associated with this NVMe disk. The mdadm determines if one NVMe devce is connected to a particular VMD domain by checking the domain symlink. Here is the root cause:

Thread A Thread B Thread mdadm
vmd_enable_domain
pci_bus_add_devices
__driver_probe_device
...
work_on_cpu
schedule_work_on
: wakeup Thread B
nvme_probe
: wakeup scan_work
to scan nvme disk
and add nvme disk
then wakeup udevd
: udevd executes
mdadm command
flush_work main
: wait for nvme_probe done ...
__driver_probe_device find_driver_devices
: probe next nvme device : 1) Detect the domain
... symlink; 2) Find the
... domain symlink from
... vmd sysfs; 3) The
... domain symlink is not
... created yet, failed
sysfs_create_link
: create domain symlink

sysfs_create_link is invoked at the end of vmd_enable_domain. However, this implementation introduces a timing issue, where mdadm might fail to retrieve the vmd symlink path because the symlink has not been created yet.

Fix the issue by creating VMD domain symlinks before invoking pci_bus_add_devices.

---
  drivers/pci/controller/vmd.c | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
drivers/pci/controller/vmd.c +3 -3
index 87b7856f375a..3f208c5f9ec9 100644
@@ -961,12 +961,12 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
  	list_for_each_entry(child, &vmd->bus->children, node)
  		pcie_bus_configure_settings(child);
 
+	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
+			       "domain"), "Can't create symlink to domain\n");
+
  	pci_bus_add_devices(vmd->bus);
 
  	vmd_acpi_end();
-
-	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
-			       "domain"), "Can't create symlink to domain\n");
  	return 0;
  }
 
author
#2 lore raw
Quoted 11 lines · Paul M Stillwell Jr
Quoted 10 lines · Jiwei Sun

From: Jiwei Sun <[email protected]>

During booting into the kernel, the following error message appears:

   (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: Unable to get real path for '/sys/bus/pci/drivers/vmd/0000:c7:00.5/domain/device'' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: /dev/nvme1n1 is not attached to Intel(R) RAID controller.' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: No OROM/EFI properties for /dev/nvme1n1' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: no RAID superblock on /dev/nvme1n1.' (udev-worker)[2149]: nvme1n1: Process '/sbin/mdadm -I /dev/nvme1n1' failed with exit code 1.

Quoted 4 lines
Quoted 2 lines

This symptom prevents the OS from booting successfully.

I'm just curious: has this been doing this forever or has this just started recently?

Thanks for your reply.

The issue was only reproduced in certain specific servers (VROC configuration with RAID1 in two NVMe drives, 7mm NVME 2-bay rear RAID enablement kits), and the VROC RAID1 disk was installed with SLES15.6 (kernel 6.4). According to our test, the issue has been easily reproduced on this configured server since kernel 6.2.
And according to the journalctl log, we found that the systemd-udevd starts running earlier than NVMe device added, it exposes this timing issue.

Thanks,
Regards,
Jiwei

Quoted 68 lines

Paul

Quoted 65 lines

After a NVMe disk is probed/added by the nvme driver, the udevd executes some rule scripts by invoking mdadm command to detect if there is a mdraid associated with this NVMe disk. The mdadm determines if one NVMe devce is connected to a particular VMD domain by checking the domain symlink. Here is the root cause:

Thread A                   Thread B             Thread mdadm vmd_enable_domain
   pci_bus_add_devices
     __driver_probe_device
      ...
      work_on_cpu
        schedule_work_on
        : wakeup Thread B
                            nvme_probe
                            : wakeup scan_work
                              to scan nvme disk
                              and add nvme disk
                              then wakeup udevd
                                                 : udevd executes mdadm command flush_work                               main
        : wait for nvme_probe done                ... __driver_probe_device                        find_driver_devices : probe next nvme device                     : 1) Detect the domain ...                                            symlink; 2) Find the ...                                            domain symlink from ...                                            vmd sysfs; 3) The ...                                            domain symlink is not ...                                            created yet, failed sysfs_create_link
   : create domain symlink

sysfs_create_link is invoked at the end of vmd_enable_domain. However, this implementation introduces a timing issue, where mdadm might fail to retrieve the vmd symlink path because the symlink has not been created yet.

Fix the issue by creating VMD domain symlinks before invoking pci_bus_add_devices.

---
  drivers/pci/controller/vmd.c | 6 +++---

  1 file changed, 3 insertions(+), 3 deletions(-)
drivers/pci/controller/vmd.c +2 -3
index 87b7856f375a..3f208c5f9ec9 100644
@@ -961,12 +961,12 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
       list_for_each_entry(child, &vmd->bus->children, node)
           pcie_bus_configure_settings(child);
   +    WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
+                   "domain"), "Can't create symlink to domain\n");
+
       pci_bus_add_devices(vmd->bus);
         vmd_acpi_end();
-
-    WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
-                   "domain"), "Can't create symlink to domain\n");
       return 0;
   }

#3 lore raw
In reply to Jiwei Sun
Quoted 10 lines · Jiwei Sun

From: Jiwei Sun <[email protected]>

During booting into the kernel, the following error message appears:

(udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: Unable to get real path for '/sys/bus/pci/drivers/vmd/0000:c7:00.5/domain/device'' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: /dev/nvme1n1 is not attached to Intel(R) RAID controller.' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: No OROM/EFI properties for /dev/nvme1n1' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: no RAID superblock on /dev/nvme1n1.' (udev-worker)[2149]: nvme1n1: Process '/sbin/mdadm -I /dev/nvme1n1' failed with exit code 1.

Quoted 6 lines

This symptom prevents the OS from booting successfully.

After a NVMe disk is probed/added by the nvme driver, the udevd executes some rule scripts by invoking mdadm command to detect if there is a mdraid associated with this NVMe disk. The mdadm determines if one NVMe devce is connected to a particular VMD domain by checking the

device

Quoted 29 lines

domain symlink. Here is the root cause:

Thread A Thread B Thread mdadm
vmd_enable_domain
pci_bus_add_devices
__driver_probe_device
...
work_on_cpu
schedule_work_on
: wakeup Thread B
nvme_probe
: wakeup scan_work
to scan nvme disk
and add nvme disk
then wakeup udevd
: udevd executes
mdadm command
flush_work main
: wait for nvme_probe done ...
__driver_probe_device find_driver_devices
: probe next nvme device : 1) Detect the domain
... symlink; 2) Find the
... domain symlink from
... vmd sysfs; 3) The
... domain symlink is not
... created yet, failed
sysfs_create_link
: create domain symlink

Quoted 7 lines

sysfs_create_link is invoked at the end of vmd_enable_domain. However, this implementation introduces a timing issue, where mdadm might fail to retrieve the vmd symlink path because the symlink has not been created yet.

Fix the issue by creating VMD domain symlinks before invoking pci_bus_add_devices.

Add "()" after function names in subject and commit log.

Quoted 11 lines
---
 drivers/pci/controller/vmd.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/pci/controller/vmd.c +0 -0
index 87b7856f375a..3f208c5f9ec9 100644

drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 965

Quoted 5 lines
drivers/pci/controller/vmd.c +2 -0
@@ -961,12 +961,12 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	list_for_each_entry(child, &vmd->bus->children, node)
 		pcie_bus_configure_settings(child);
 
+	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
+			       "domain"), "Can't create symlink to domain\n");

Seems OK to me. IIUC, this "domain" link is created in the directory of the VMD pci_dev, and it points to the directory of the new "root bus" behind the VMD.

Since it's unrelated to the *devices* on that new root bus, I would probably move this even earlier, so it's with the other code that sets up the new root bus, e.g., somewhere around vmd_attach_resources().

Quoted 12 lines
drivers/pci/controller/vmd.c +0 -3
 	pci_bus_add_devices(vmd->bus);
 
 	vmd_acpi_end();
-
-	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
-			       "domain"), "Can't create symlink to domain\n");
 	return 0;
 }
 
Signature · 1 line
2.27.0