PCI: vmd: Create domain symlink before pci_bus_add_devices()
Subsystems (3)
PATCH v2 1/1 PCI: vmd: Create domain symlink before pci_bus_add_devices()
From: Jiwei Sun <[email protected]>
During booting into the kernel, the following error message appears:
(udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: Unable to get real path for '/sys/bus/pci/drivers/vmd/0000:c7:00.5/domain/device'' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: /dev/nvme1n1 is not attached to Intel(R) RAID controller.' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: No OROM/EFI properties for /dev/nvme1n1' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: no RAID superblock on /dev/nvme1n1.' (udev-worker)[2149]: nvme1n1: Process '/sbin/mdadm -I /dev/nvme1n1' failed with exit code 1.
This symptom prevents the OS from booting successfully.
After a NVMe disk is probed/added by the nvme driver, the udevd executes some rule scripts by invoking mdadm command to detect if there is a mdraid associated with this NVMe disk. The mdadm determines if one NVMe devce is connected to a particular VMD domain by checking the domain symlink. Here is the root cause:
Thread A Thread B Thread mdadmvmd_enable_domain
pci_bus_add_devices
__driver_probe_device
...
work_on_cpu
schedule_work_on
: wakeup Thread B
nvme_probe
: wakeup scan_work
to scan nvme disk
and add nvme disk
then wakeup udevd
: udevd executes
mdadm command
flush_work main
: wait for nvme_probe done ...
__driver_probe_device find_driver_devices
: probe next nvme device : 1) Detect the domain
... symlink; 2) Find the
... domain symlink from
... vmd sysfs; 3) The
... domain symlink is not
... created yet, failed
sysfs_create_link
: create domain symlink
sysfs_create_link() is invoked at the end of vmd_enable_domain(). However, this implementation introduces a timing issue, where mdadm might fail to retrieve the vmd symlink path because the symlink has not been created yet.
Fix the issue by creating VMD domain symlinks before invoking pci_bus_add_devices().
---
v2 changes:
- Add "()" after function names in subject and commit log
- Move sysfs_create_link() after vmd_attach_resources()
--- drivers/pci/controller/vmd.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/pci/controller/vmd.c +3 -3
@@ -925,6 +925,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)dev_set_msi_domain (&vmd->bus->dev,dev_get_msi_domain (&vmd->dev->dev)); +WARN (sysfs_create_link (&vmd->dev->dev.kobj, &vmd->bus->dev.kobj, +"domain" ),"Can't create symlink to domain\n" ); +vmd_acpi_begin ();pci_scan_child_bus (vmd->bus);
@@ -964,9 +967,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)pci_bus_add_devices (vmd->bus);vmd_acpi_end (); - -WARN (sysfs_create_link (&vmd->dev->dev.kobj, &vmd->bus->dev.kobj, -"domain" ),"Can't create symlink to domain\n" );return 0 ; }
Signature · 1 line
2.27.0
Quoted 69 lines · Jiwei Sun
From: Jiwei Sun <[email protected]>
During booting into the kernel, the following error message appears:
(udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: Unable to get real path for '/sys/bus/pci/drivers/vmd/0000:c7:00.5/domain/device'' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: /dev/nvme1n1 is not attached to Intel(R) RAID controller.' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: No OROM/EFI properties for /dev/nvme1n1' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm -I /dev/nvme1n1'(err) 'mdadm: no RAID superblock on /dev/nvme1n1.' (udev-worker)[2149]: nvme1n1: Process '/sbin/mdadm -I /dev/nvme1n1' failed with exit code 1.This symptom prevents the OS from booting successfully.
After a NVMe disk is probed/added by the nvme driver, the udevd executes some rule scripts by invoking mdadm command to detect if there is a mdraid associated with this NVMe disk. The mdadm determines if one NVMe devce is connected to a particular VMD domain by checking the domain symlink. Here is the root cause:
Thread A Thread B Thread mdadm
vmd_enable_domain
pci_bus_add_devices
__driver_probe_device
...
work_on_cpu
schedule_work_on
: wakeup Thread B
nvme_probe
: wakeupscan_work
to scan nvme disk
and add nvme disk
then wakeup udevd
: udevd executes
mdadm command
flush_workmain
: wait fornvme_probedone ...
__driver_probe_devicefind_driver_devices
: probe next nvme device : 1) Detect the domain
... symlink; 2) Find the
... domain symlink from
... vmd sysfs; 3) The
... domain symlink is not
... created yet, failed
sysfs_create_link
: create domain symlink
sysfs_create_link()is invoked at the end ofvmd_enable_domain(). However, this implementation introduces a timing issue, where mdadm might fail to retrieve the vmd symlink path because the symlink has not been created yet.Fix the issue by creating VMD domain symlinks before invoking
pci_bus_add_devices().Signed-off-by Jiwei Sun <[email protected]>Suggested-by Adrian Huang <[email protected]>---
v2 changes:
- Add "()" after function names in subject and commit log
- Movesysfs_create_link()aftervmd_attach_resources()--- drivers/pci/controller/vmd.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)drivers/pci/controller/vmd.c +0
-0
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 930
Quoted 6 lines
drivers/pci/controller/vmd.c +3
-0@@ -925,6 +925,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)dev_set_msi_domain (&vmd->bus->dev,dev_get_msi_domain (&vmd->dev->dev)); +WARN (sysfs_create_link (&vmd->dev->dev.kobj, &vmd->bus->dev.kobj, +"domain" ),"Can't create symlink to domain\n" ); +
I think you should move the sysfs_remove_link() line in vmd_remove() down as well.
Paul
Quoted 13 lines
drivers/pci/controller/vmd.c +0
-3vmd_acpi_begin ();pci_scan_child_bus (vmd->bus);@@ -964,9 +967,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)pci_bus_add_devices (vmd->bus);vmd_acpi_end (); - -WARN (sysfs_create_link (&vmd->dev->dev.kobj, &vmd->bus->dev.kobj, -"domain" ),"Can't create symlink to domain\n" );return 0 ; }
Quoted 65 lines · Paul M Stillwell Jr
Quoted 64 lines · Jiwei Sun
From: Jiwei Sun <[email protected]>
During booting into the kernel, the following error message appears:
(udev-worker)[2149]: nvme1n1: '
/sbin/mdadm-I /dev/nvme1n1'(err) 'mdadm: Unable to get real path for '/sys/bus/pci/drivers/vmd/0000:c7:00.5/domain/device'' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm-I /dev/nvme1n1'(err) 'mdadm:/dev/nvme1n1is not attached to Intel(R) RAID controller.' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm-I /dev/nvme1n1'(err) 'mdadm: No OROM/EFI properties for/dev/nvme1n1' (udev-worker)[2149]: nvme1n1: '/sbin/mdadm-I /dev/nvme1n1'(err) 'mdadm: no RAID superblock on/dev/nvme1n1.' (udev-worker)[2149]: nvme1n1: Process '/sbin/mdadm-I/dev/nvme1n1' failed with exit code 1.This symptom prevents the OS from booting successfully.
After a NVMe disk is probed/added by the nvme driver, the udevd executes some rule scripts by invoking mdadm command to detect if there is a mdraid associated with this NVMe disk. The mdadm determines if one NVMe devce is connected to a particular VMD domain by checking the domain symlink. Here is the root cause:
Thread A Thread B Thread mdadm
vmd_enable_domain
pci_bus_add_devices
__driver_probe_device
...
work_on_cpu
schedule_work_on
: wakeup Thread B
nvme_probe
: wakeupscan_work
to scan nvme disk
and add nvme disk
then wakeup udevd
: udevd executes mdadm command flush_work main
: wait fornvme_probedone ... __driver_probe_devicefind_driver_devices: probe next nvme device : 1) Detect the domain ... symlink; 2) Find the ... domain symlink from ... vmd sysfs; 3) The ... domain symlink is not ... created yet, failedsysfs_create_link
: create domain symlink
sysfs_create_link()is invoked at the end ofvmd_enable_domain(). However, this implementation introduces a timing issue, where mdadm might fail to retrieve the vmd symlink path because the symlink has not been created yet.Fix the issue by creating VMD domain symlinks before invoking
pci_bus_add_devices().Signed-off-by Jiwei Sun <[email protected]>Suggested-by Adrian Huang <[email protected]>---
v2 changes:
- Add "()" after function names in subject and commit log
- Movesysfs_create_link()aftervmd_attach_resources()---
drivers/pci/controller/vmd.c| 6 +++---1 file changed, 3 insertions(+), 3 deletions(-)
Quoted 12 lines
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 929
Quoted 10 lines
drivers/pci/controller/vmd.c +2
-0@@ -925,6 +925,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) dev_set_msi_domain(&vmd->bus->dev, dev_get_msi_domain(&vmd->dev->dev)); + WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj, + "domain"), "Can't create symlink to domain\n"); +I think you should move the
sysfs_remove_link()line invmd_remove()down as well.
Indeed, thanks for your suggestion. I will modify it in v3 patch.
Thanks,
Regards,
Jiwei
Quoted 14 lines
Paul
Quoted 11 lines
drivers/pci/controller/vmd.c +0
-3vmd_acpi_begin(); pci_scan_child_bus(vmd->bus);@@ -964,9 +967,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) pci_bus_add_devices(vmd->bus); vmd_acpi_end(); - - WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj, - "domain"), "Can't create symlink to domain\n"); return 0; }