Skip to contentSkip to conversation

PM domain change on unbound devices warning on ipmi_si unload

PATCH Joe Lawrence 5 participants 10 messages Jan 28, 2016 – Feb 3, 2016 Driver core

lore
Share & tools
lore
#0 lore raw

Starting in 4.5-rc1, I noticed this warning on ipmi_si driver removal:

% modprobe ipmi_si
% rmmod ipmi_si

bus: 'platform': driver_probe_device: matched device IPI0001:00 with driver ipmi_si
bus: 'platform': really_probe: probing driver ipmi_si with device IPI0001:00 ipmi_si IPI0001:00: ipmi_si: probing via ACPI
ipmi_si IPI0001:00: [io 0x0ca2-0x0ca3] regsize 1 spacing 1 irq 0
ipmi_si: Adding ACPI-specified kcs state machine
driver: 'ipmi_si': driver_bound: bound to device 'IPI0001:00'
bus: 'platform': really_probe: bound device IPI0001:00 to driver ipmi_si
IPMI System Interface driver.
ipmi_si: probing via SMBIOS
ipmi_si: SMBIOS: io 0xda2 regsize 1 spacing 1 irq 0
ipmi_si: Adding SMBIOS-specified kcs state machine
ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0 Registering platform device 'ipmi_bmc.0674.66'. Parent at platform
driver: 'ipmi': driver_bound: bound to device 'ipmi_bmc.0674.66'
ipmi_si IPI0001:00: Found new BMC (man_id: 0x000077, prod_id: 0x0674, dev_id: 0x42) ipmi_si IPI0001:00: IPMI kcs interface initialized

------------[ cut here ]------------
WARNING: CPU: 39 PID: 3678 at drivers/base/power/common.c:150 dev_pm_domain_set+0x52/0x60()
PM domains can only be changed for unbound devices
[ ... snip ... ]
CPU: 39 PID: 3678 Comm: rmmod Tainted: G           OE   4.5.0-rc1+ #57
Hardware name: Stratus ftServer 6800/G7LYY, BIOS BIOS Version 8.1:61 09/10/2015
0000000000000000 000000003883b68d ffff8820334d7d30 ffffffff8132caf0
ffff8820334d7d78 ffff8820334d7d68 ffffffff8107f1b6 ffff8810351eca00
0000000000000000 0000000000000001 0000000001c2a330 0000000001c29010
Call Trace:
[<ffffffff8132caf0>] dump_stack+0x44/0x64
[<ffffffff8107f1b6>] warn_slowpath_common+0x86/0xc0
[<ffffffff8107f24c>] warn_slowpath_fmt+0x5c/0x80
[<ffffffff8145e152>] dev_pm_domain_set+0x52/0x60
[<ffffffff813a38ae>] acpi_dev_pm_detach+0x3f/0x84
[<ffffffff8145e0d7>] dev_pm_domain_detach+0x27/0x30
[<ffffffff814575f8>] platform_drv_remove+0x38/0x40
[<ffffffff814557ba>] __device_release_driver+0x9a/0x140
[<ffffffff81455968>] driver_detach+0xb8/0xc0
[<ffffffff814547b5>] bus_remove_driver+0x55/0xd0
[<ffffffff814560cc>] driver_unregister+0x2c/0x50
[<ffffffff814576b2>] platform_driver_unregister+0x12/0x20
[<ffffffffa02586c9>] cleanup_ipmi_si+0x29/0xa0 [ipmi_si]
[<ffffffff81102100>] SyS_delete_module+0x190/0x220
[<ffffffff8167ffee>] entry_SYSCALL_64_fastpath+0x12/0x71
---[ end trace 671ca97b9ac15462 ]---

My platform has two BMCs (perhaps this is messing with a refcount somewhere), but I wonder about the ordering of this code:

__device_release_driver(struct device *dev)

  drv->remove(dev);
    [ platform_drv_remove ]
      ...
      dev_pm_domain_detach
        device_is_bound
          return dev->p && klist_node_attached(&dev->p->knode_driver)
  ...
  klist_remove(&dev->p->knode_driver);

Is the klist_remove at the bottom of __device_release_driver necessary to satisfy the earlier check in dev_pm_domain_detach's device_is_bound assertion? If so, could these be out of order?

This is core driver code, so I'm assuming it's not something as simple as the following (which avoided the warning on unload at least). Any suggestions or extra debugging ideas welcome! This occurs on every unload, so I'd be glad to test real solutions :)

Thanks,

-- Joe

-->8--

drivers/base/dd.c +1 -1
index c4da2df..bba54e1 100644
@@ -756,6 +756,7 @@ static void __device_release_driver(struct device *dev)
 
 		pm_runtime_put_sync(dev);
 
+		klist_remove(&dev->p->knode_driver);
 		if (dev->bus && dev->bus->remove)
 			dev->bus->remove(dev);
 		else if (drv->remove)
@@ -767,7 +768,6 @@ static void __device_release_driver(struct device *dev)
 			dev->pm_domain->dismiss(dev);
 		pm_runtime_reinit(dev);
 
-		klist_remove(&dev->p->knode_driver);
 		device_pm_check_callbacks(dev);
 		if (dev->bus)
 			blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
#1 lore raw
Quoted 4 lines · Joe Lawrence

Starting in 4.5-rc1, I noticed this warning on ipmi_si driver removal:

% modprobe ipmi_si
% rmmod ipmi_si

Yes, I had just noticed this yesterday and I was going to try to trace it down today. I had assumed it was an IPMI driver problem, but after looking at it a bit, I don't think it is. I'll try to bisect to see what caused this, nothing obvious jumps out

-corey

Quoted 93 lines

bus: 'platform': driver_probe_device: matched device IPI0001:00 with driver ipmi_si
bus: 'platform': really_probe: probing driver ipmi_si with device IPI0001:00 ipmi_si IPI0001:00: ipmi_si: probing via ACPI
ipmi_si IPI0001:00: [io 0x0ca2-0x0ca3] regsize 1 spacing 1 irq 0
ipmi_si: Adding ACPI-specified kcs state machine
driver: 'ipmi_si': driver_bound: bound to device 'IPI0001:00'
bus: 'platform': really_probe: bound device IPI0001:00 to driver ipmi_si
IPMI System Interface driver.
ipmi_si: probing via SMBIOS
ipmi_si: SMBIOS: io 0xda2 regsize 1 spacing 1 irq 0
ipmi_si: Adding SMBIOS-specified kcs state machine
ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0 Registering platform device 'ipmi_bmc.0674.66'. Parent at platform
driver: 'ipmi': driver_bound: bound to device 'ipmi_bmc.0674.66'
ipmi_si IPI0001:00: Found new BMC (man_id: 0x000077, prod_id: 0x0674, dev_id: 0x42) ipmi_si IPI0001:00: IPMI kcs interface initialized

------------[ cut here ]------------
WARNING: CPU: 39 PID: 3678 at drivers/base/power/common.c:150 dev_pm_domain_set+0x52/0x60()
PM domains can only be changed for unbound devices
[ ... snip ... ]
CPU: 39 PID: 3678 Comm: rmmod Tainted: G           OE   4.5.0-rc1+ #57
Hardware name: Stratus ftServer 6800/G7LYY, BIOS BIOS Version 8.1:61 09/10/2015
0000000000000000 000000003883b68d ffff8820334d7d30 ffffffff8132caf0
ffff8820334d7d78 ffff8820334d7d68 ffffffff8107f1b6 ffff8810351eca00
0000000000000000 0000000000000001 0000000001c2a330 0000000001c29010
Call Trace:
[<ffffffff8132caf0>] dump_stack+0x44/0x64
[<ffffffff8107f1b6>] warn_slowpath_common+0x86/0xc0
[<ffffffff8107f24c>] warn_slowpath_fmt+0x5c/0x80
[<ffffffff8145e152>] dev_pm_domain_set+0x52/0x60
[<ffffffff813a38ae>] acpi_dev_pm_detach+0x3f/0x84
[<ffffffff8145e0d7>] dev_pm_domain_detach+0x27/0x30
[<ffffffff814575f8>] platform_drv_remove+0x38/0x40
[<ffffffff814557ba>] __device_release_driver+0x9a/0x140
[<ffffffff81455968>] driver_detach+0xb8/0xc0
[<ffffffff814547b5>] bus_remove_driver+0x55/0xd0
[<ffffffff814560cc>] driver_unregister+0x2c/0x50
[<ffffffff814576b2>] platform_driver_unregister+0x12/0x20
[<ffffffffa02586c9>] cleanup_ipmi_si+0x29/0xa0 [ipmi_si]
[<ffffffff81102100>] SyS_delete_module+0x190/0x220
[<ffffffff8167ffee>] entry_SYSCALL_64_fastpath+0x12/0x71
---[ end trace 671ca97b9ac15462 ]---

My platform has two BMCs (perhaps this is messing with a refcount somewhere), but I wonder about the ordering of this code:

__device_release_driver(struct device *dev)

   drv->remove(dev);
     [ platform_drv_remove ]
       ...
       dev_pm_domain_detach
         device_is_bound
           return dev->p && klist_node_attached(&dev->p->knode_driver)
   ...
   klist_remove(&dev->p->knode_driver);

Is the klist_remove at the bottom of __device_release_driver necessary to satisfy the earlier check in dev_pm_domain_detach's device_is_bound assertion? If so, could these be out of order?

This is core driver code, so I'm assuming it's not something as simple as the following (which avoided the warning on unload at least). Any suggestions or extra debugging ideas welcome! This occurs on every unload, so I'd be glad to test real solutions :)

Thanks,

-- Joe

-->8--

drivers/base/dd.c +1 -1
index c4da2df..bba54e1 100644
@@ -756,6 +756,7 @@ static void __device_release_driver(struct device *dev)
 
  		pm_runtime_put_sync(dev);
 
+		klist_remove(&dev->p->knode_driver);
  		if (dev->bus && dev->bus->remove)
  			dev->bus->remove(dev);
  		else if (drv->remove)
@@ -767,7 +768,6 @@ static void __device_release_driver(struct device *dev)
  			dev->pm_domain->dismiss(dev);
  		pm_runtime_reinit(dev);
 
-		klist_remove(&dev->p->knode_driver);
  		device_pm_check_callbacks(dev);
  		if (dev->bus)
  			blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
#2 lore raw

Tomeu, you added that check in

[989561de9b5112999475b406557d9c7e9e59c041] PM / Domains: add setter for dev.pm_domain

and either something is wrong in the platform device handling or elsewhere in the device code, if that check is valid.

-corey

Quoted 108 lines · Corey Minyard
Quoted 4 lines · Joe Lawrence

Starting in 4.5-rc1, I noticed this warning on ipmi_si driver removal:

% modprobe ipmi_si
% rmmod ipmi_si

Yes, I had just noticed this yesterday and I was going to try to trace it down today. I had assumed it was an IPMI driver problem, but after looking at it a bit, I don't think it is. I'll try to bisect to see what caused this, nothing obvious jumps out

-corey

Quoted 95 lines

bus: 'platform': driver_probe_device: matched device IPI0001:00 with driver ipmi_si
bus: 'platform': really_probe: probing driver ipmi_si with device IPI0001:00 ipmi_si IPI0001:00: ipmi_si: probing via ACPI
ipmi_si IPI0001:00: [io 0x0ca2-0x0ca3] regsize 1 spacing 1 irq 0
ipmi_si: Adding ACPI-specified kcs state machine
driver: 'ipmi_si': driver_bound: bound to device 'IPI0001:00'
bus: 'platform': really_probe: bound device IPI0001:00 to driver ipmi_si
IPMI System Interface driver.
ipmi_si: probing via SMBIOS
ipmi_si: SMBIOS: io 0xda2 regsize 1 spacing 1 irq 0
ipmi_si: Adding SMBIOS-specified kcs state machine
ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0 Registering platform device 'ipmi_bmc.0674.66'. Parent at platform
driver: 'ipmi': driver_bound: bound to device 'ipmi_bmc.0674.66'
ipmi_si IPI0001:00: Found new BMC (man_id: 0x000077, prod_id: 0x0674, dev_id: 0x42) ipmi_si IPI0001:00: IPMI kcs interface initialized

------------[ cut here ]------------
WARNING: CPU: 39 PID: 3678 at drivers/base/power/common.c:150 dev_pm_domain_set+0x52/0x60()
PM domains can only be changed for unbound devices
[ ... snip ... ]
CPU: 39 PID: 3678 Comm: rmmod Tainted: G           OE 4.5.0-rc1+ #57
Hardware name: Stratus ftServer 6800/G7LYY, BIOS BIOS Version 8.1:61 09/10/2015
0000000000000000 000000003883b68d ffff8820334d7d30 ffffffff8132caf0
ffff8820334d7d78 ffff8820334d7d68 ffffffff8107f1b6 ffff8810351eca00
0000000000000000 0000000000000001 0000000001c2a330 0000000001c29010
Call Trace:
[<ffffffff8132caf0>] dump_stack+0x44/0x64
[<ffffffff8107f1b6>] warn_slowpath_common+0x86/0xc0
[<ffffffff8107f24c>] warn_slowpath_fmt+0x5c/0x80
[<ffffffff8145e152>] dev_pm_domain_set+0x52/0x60
[<ffffffff813a38ae>] acpi_dev_pm_detach+0x3f/0x84
[<ffffffff8145e0d7>] dev_pm_domain_detach+0x27/0x30
[<ffffffff814575f8>] platform_drv_remove+0x38/0x40
[<ffffffff814557ba>] __device_release_driver+0x9a/0x140
[<ffffffff81455968>] driver_detach+0xb8/0xc0
[<ffffffff814547b5>] bus_remove_driver+0x55/0xd0
[<ffffffff814560cc>] driver_unregister+0x2c/0x50
[<ffffffff814576b2>] platform_driver_unregister+0x12/0x20
[<ffffffffa02586c9>] cleanup_ipmi_si+0x29/0xa0 [ipmi_si]
[<ffffffff81102100>] SyS_delete_module+0x190/0x220
[<ffffffff8167ffee>] entry_SYSCALL_64_fastpath+0x12/0x71
---[ end trace 671ca97b9ac15462 ]---

My platform has two BMCs (perhaps this is messing with a refcount somewhere), but I wonder about the ordering of this code:

__device_release_driver(struct device *dev)

   drv->remove(dev);
     [ platform_drv_remove ]
       ...
       dev_pm_domain_detach
         device_is_bound
           return dev->p && klist_node_attached(&dev->p->knode_driver)
   ...
   klist_remove(&dev->p->knode_driver);

Is the klist_remove at the bottom of __device_release_driver necessary to satisfy the earlier check in dev_pm_domain_detach's device_is_bound assertion? If so, could these be out of order?

This is core driver code, so I'm assuming it's not something as simple as the following (which avoided the warning on unload at least). Any suggestions or extra debugging ideas welcome! This occurs on every unload, so I'd be glad to test real solutions :)

Thanks,

-- Joe

-->8--

drivers/base/dd.c +1 -0
index c4da2df..bba54e1 100644
@@ -756,6 +756,7 @@ static void __device_release_driver(struct device 
 *dev)
 
          pm_runtime_put_sync(dev);
 
+        klist_remove(&dev->p->knode_driver);
          if (dev->bus && dev->bus->remove)
              dev->bus->remove(dev);
          else if (drv->remove)
drivers/base/dd.c +0 -1
@@ -767,7 +768,6 @@ static void __device_release_driver(struct device 
 *dev)
              dev->pm_domain->dismiss(dev);
          pm_runtime_reinit(dev);
 
-        klist_remove(&dev->p->knode_driver);
          device_pm_check_callbacks(dev);
          if (dev->bus)

blocking_notifier_call_chain(&dev->bus->p->bus_notifier,

#3 lore raw
Quoted 9 lines · Corey Minyard

Tomeu, you added that check in

[989561de9b5112999475b406557d9c7e9e59c041] PM / Domains: add setter for dev.pm_domain

and either something is wrong in the platform device handling or elsewhere in the device code, if
that check is valid.

FYI, I'm hitting the exact same error on shutdown on one of my boxes.

Please Cc me on updates.

-- Steve

[53591.087861] kvm: exiting hardware virtualization
[53591.104798] ------------[ cut here ]------------
[53591.110058] WARNING: CPU: 0 PID: 1 at /home/rostedt/work/git/linux-trace.git/drivers/base/power/common.c:150 dev_pm_domain_set+0x82/0x90()
[53591.123716] PM domains can only be changed for unbound devices
[53591.130158] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables xt_CHECKSUM iptable_mangle bridge stp llc bluetooth lockd grace snd_hda_codec_hdmi snd_hda_codec_r
ealtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hwdep snd_hda_core vhost_net tun vhost x86_pkg_temp_thermal iTCO_wdt snd_seq snd_seq_device snd_pcm macvtap coretemp me
i_me iTCO_vendor_support hp_wmi rfkill sparse_keymap macvlan kvm_intel snd_timer mei lpc_ich snd i2c_i801 soundcore mfd_core kvm irqbypass acpi_cpufreq serio_raw wmi uinput crc32_p
clmul i915 crc32c_intel i2c_algo_bit e1000e drm_kms_helper ptp drm pps_core i2c_core video sunrpc
[53591.190440] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 4.5.0-rc1-test+ #155
[53591.198453] Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v02.05 05/07/2012
[53591.208031]  ffff8801195abc90 ffff8801195abc90 ffffffff8143ef63 ffff8801195abcd8
[53591.216168]  ffff8801195abcc8 ffffffff810acad6 ffff8801192d2328 0000000000000000
[53591.224299]  0000000000000001 ffff8801192d2388 00000000fee1dead ffff8801195abd28
[53591.232429] Call Trace:
[53591.235483]  [<ffffffff8143ef63>] dump_stack+0x44/0x61
[53591.241232]  [<ffffffff810acad6>] warn_slowpath_common+0x86/0xc0
[53591.247842]  [<ffffffff810acb5c>] warn_slowpath_fmt+0x4c/0x50
[53591.254193]  [<ffffffff810acb15>] ? warn_slowpath_fmt+0x5/0x50
[53591.260625]  [<ffffffff8157cf42>] dev_pm_domain_set+0x82/0x90
[53591.266970]  [<ffffffffa03f255e>] mei_me_remove+0xee/0x120 [mei_me]
[53591.273842]  [<ffffffff8147fe96>] pci_device_shutdown+0x36/0x70
[53591.280373]  [<ffffffff81570680>] device_shutdown+0xe0/0x1e0
[53591.286642]  [<ffffffff810d6df6>] kernel_restart_prepare+0x36/0x40
[53591.293439]  [<ffffffff810d6f62>] kernel_restart+0x12/0x60
[53591.299543]  [<ffffffff810d72ae>] SYSC_reboot+0x1ce/0x1f0
[53591.305562]  [<ffffffffa0008077>] ? 0xffffffffa0008077
[53591.311323]  [<ffffffff811b2e60>] ? stack_trace_call+0x40/0x60
[53591.317780]  [<ffffffffa0008077>] ? 0xffffffffa0008077
[53591.323552]  [<ffffffff8143eeb4>] ? _atomic_dec_and_lock+0x44/0xaf
[53591.330374]  [<ffffffff812a302b>] ? iput+0xbb/0x2c0
[53591.335880]  [<ffffffff810d7325>] ? SyS_reboot+0x5/0x10
[53591.341748]  [<ffffffff81003044>] ? lockdep_sys_exit_thunk+0x12/0x14
[53591.348743]  [<ffffffff810d732e>] SyS_reboot+0xe/0x10
[53591.354440]  [<ffffffff8185ee32>] entry_SYSCALL_64_fastpath+0x12/0x76
[53591.361534] ---[ end trace 63b298fc6d5920e4 ]---
[53591.377714] e1000e: EEE TX LPI TIMER: 00000011
[53591.468245] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[53591.487321] reboot: Restarting system
[53591.491679] reboot: machine restart
#4 lore raw
Quoted 49 lines · Steven Rostedt
Quoted 9 lines · Corey Minyard

Tomeu, you added that check in

[989561de9b5112999475b406557d9c7e9e59c041] PM / Domains: add setter for dev.pm_domain

and either something is wrong in the platform device handling or elsewhere in the device code, if
that check is valid.

FYI, I'm hitting the exact same error on shutdown on one of my boxes.

Please Cc me on updates.

-- Steve

[53591.087861] kvm: exiting hardware virtualization
[53591.104798] ------------[ cut here ]------------
[53591.110058] WARNING: CPU: 0 PID: 1 at /home/rostedt/work/git/linux-trace.git/drivers/base/power/common.c:150 dev_pm_domain_set+0x82/0x90()
[53591.123716] PM domains can only be changed for unbound devices
[53591.130158] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables xt_CHECKSUM iptable_mangle bridge stp llc bluetooth lockd grace snd_hda_codec_hdmi snd_hda_codec_r
ealtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hwdep snd_hda_core vhost_net tun vhost x86_pkg_temp_thermal iTCO_wdt snd_seq snd_seq_device snd_pcm macvtap coretemp me
i_me iTCO_vendor_support hp_wmi rfkill sparse_keymap macvlan kvm_intel snd_timer mei lpc_ich snd i2c_i801 soundcore mfd_core kvm irqbypass acpi_cpufreq serio_raw wmi uinput crc32_p
clmul i915 crc32c_intel i2c_algo_bit e1000e drm_kms_helper ptp drm pps_core i2c_core video sunrpc
[53591.190440] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 4.5.0-rc1-test+ #155
[53591.198453] Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v02.05 05/07/2012
[53591.208031]  ffff8801195abc90 ffff8801195abc90 ffffffff8143ef63 ffff8801195abcd8
[53591.216168]  ffff8801195abcc8 ffffffff810acad6 ffff8801192d2328 0000000000000000
[53591.224299]  0000000000000001 ffff8801192d2388 00000000fee1dead ffff8801195abd28
[53591.232429] Call Trace:
[53591.235483]  [<ffffffff8143ef63>] dump_stack+0x44/0x61
[53591.241232]  [<ffffffff810acad6>] warn_slowpath_common+0x86/0xc0
[53591.247842]  [<ffffffff810acb5c>] warn_slowpath_fmt+0x4c/0x50
[53591.254193]  [<ffffffff810acb15>] ? warn_slowpath_fmt+0x5/0x50
[53591.260625]  [<ffffffff8157cf42>] dev_pm_domain_set+0x82/0x90
[53591.266970]  [<ffffffffa03f255e>] mei_me_remove+0xee/0x120 [mei_me]
[53591.273842]  [<ffffffff8147fe96>] pci_device_shutdown+0x36/0x70
[53591.280373]  [<ffffffff81570680>] device_shutdown+0xe0/0x1e0
[53591.286642]  [<ffffffff810d6df6>] kernel_restart_prepare+0x36/0x40
[53591.293439]  [<ffffffff810d6f62>] kernel_restart+0x12/0x60
[53591.299543]  [<ffffffff810d72ae>] SYSC_reboot+0x1ce/0x1f0
[53591.305562]  [<ffffffffa0008077>] ? 0xffffffffa0008077
[53591.311323]  [<ffffffff811b2e60>] ? stack_trace_call+0x40/0x60
[53591.317780]  [<ffffffffa0008077>] ? 0xffffffffa0008077
[53591.323552]  [<ffffffff8143eeb4>] ? _atomic_dec_and_lock+0x44/0xaf
[53591.330374]  [<ffffffff812a302b>] ? iput+0xbb/0x2c0
[53591.335880]  [<ffffffff810d7325>] ? SyS_reboot+0x5/0x10
Quoted 9 lines
[53591.341748]  [<ffffffff81003044>] ? lockdep_sys_exit_thunk+0x12/0x14
[53591.348743]  [<ffffffff810d732e>] SyS_reboot+0xe/0x10
[53591.354440]  [<ffffffff8185ee32>] entry_SYSCALL_64_fastpath+0x12/0x76
[53591.361534] ---[ end trace 63b298fc6d5920e4 ]---
[53591.377714] e1000e: EEE TX LPI TIMER: 00000011
[53591.468245] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[53591.487321] reboot: Restarting system
[53591.491679] reboot: machine restart

Looks like Rafael adjusted for the platform shutdown case here:

https://lkml.org/lkml/2016/1/11/515

Perhaps that would applicable to the pci shutdown as well?

-- Joe

#5 lore raw
Quoted 60 lines · Joe Lawrence
Quoted 49 lines · Steven Rostedt
Quoted 9 lines · Corey Minyard

Tomeu, you added that check in

[989561de9b5112999475b406557d9c7e9e59c041] PM / Domains: add setter for dev.pm_domain

and either something is wrong in the platform device handling or elsewhere in the device code, if
that check is valid.

FYI, I'm hitting the exact same error on shutdown on one of my boxes.

Please Cc me on updates.

-- Steve

[53591.087861] kvm: exiting hardware virtualization
[53591.104798] ------------[ cut here ]------------
[53591.110058] WARNING: CPU: 0 PID: 1 at /home/rostedt/work/git/linux-trace.git/drivers/base/power/common.c:150 dev_pm_domain_set+0x82/0x90()
[53591.123716] PM domains can only be changed for unbound devices
[53591.130158] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables xt_CHECKSUM iptable_mangle bridge stp llc bluetooth lockd grace snd_hda_codec_hdmi snd_hda_codec_r
ealtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hwdep snd_hda_core vhost_net tun vhost x86_pkg_temp_thermal iTCO_wdt snd_seq snd_seq_device snd_pcm macvtap coretemp me
i_me iTCO_vendor_support hp_wmi rfkill sparse_keymap macvlan kvm_intel snd_timer mei lpc_ich snd i2c_i801 soundcore mfd_core kvm irqbypass acpi_cpufreq serio_raw wmi uinput crc32_p
clmul i915 crc32c_intel i2c_algo_bit e1000e drm_kms_helper ptp drm pps_core i2c_core video sunrpc
[53591.190440] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 4.5.0-rc1-test+ #155
[53591.198453] Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v02.05 05/07/2012
[53591.208031]  ffff8801195abc90 ffff8801195abc90 ffffffff8143ef63 ffff8801195abcd8
[53591.216168]  ffff8801195abcc8 ffffffff810acad6 ffff8801192d2328 0000000000000000
[53591.224299]  0000000000000001 ffff8801192d2388 00000000fee1dead ffff8801195abd28
[53591.232429] Call Trace:
[53591.235483]  [<ffffffff8143ef63>] dump_stack+0x44/0x61
[53591.241232]  [<ffffffff810acad6>] warn_slowpath_common+0x86/0xc0
[53591.247842]  [<ffffffff810acb5c>] warn_slowpath_fmt+0x4c/0x50
[53591.254193]  [<ffffffff810acb15>] ? warn_slowpath_fmt+0x5/0x50
[53591.260625]  [<ffffffff8157cf42>] dev_pm_domain_set+0x82/0x90
[53591.266970]  [<ffffffffa03f255e>] mei_me_remove+0xee/0x120 [mei_me]
[53591.273842]  [<ffffffff8147fe96>] pci_device_shutdown+0x36/0x70
[53591.280373]  [<ffffffff81570680>] device_shutdown+0xe0/0x1e0
[53591.286642]  [<ffffffff810d6df6>] kernel_restart_prepare+0x36/0x40
[53591.293439]  [<ffffffff810d6f62>] kernel_restart+0x12/0x60
[53591.299543]  [<ffffffff810d72ae>] SYSC_reboot+0x1ce/0x1f0
[53591.305562]  [<ffffffffa0008077>] ? 0xffffffffa0008077
[53591.311323]  [<ffffffff811b2e60>] ? stack_trace_call+0x40/0x60
[53591.317780]  [<ffffffffa0008077>] ? 0xffffffffa0008077
[53591.323552]  [<ffffffff8143eeb4>] ? _atomic_dec_and_lock+0x44/0xaf
[53591.330374]  [<ffffffff812a302b>] ? iput+0xbb/0x2c0
[53591.335880]  [<ffffffff810d7325>] ? SyS_reboot+0x5/0x10
Quoted 9 lines
[53591.341748]  [<ffffffff81003044>] ? lockdep_sys_exit_thunk+0x12/0x14
[53591.348743]  [<ffffffff810d732e>] SyS_reboot+0xe/0x10
[53591.354440]  [<ffffffff8185ee32>] entry_SYSCALL_64_fastpath+0x12/0x76
[53591.361534] ---[ end trace 63b298fc6d5920e4 ]---
[53591.377714] e1000e: EEE TX LPI TIMER: 00000011
[53591.468245] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[53591.487321] reboot: Restarting system
[53591.491679] reboot: machine restart
Quoted 5 lines

Looks like Rafael adjusted for the platform shutdown case here:

https://lkml.org/lkml/2016/1/11/515

Perhaps that would applicable to the pci shutdown as well?

To the MEI driver's shutdown rather. mei_me_remove() clears the pm_domain pointer which is sort of questionable, but then the warning may be overkill for this case.

Thanks,
Rafael

#6 lore raw
Quoted 63 lines · Rafael J. Wysocki
Quoted 62 lines · Joe Lawrence
Quoted 49 lines · Steven Rostedt
Quoted 9 lines · Corey Minyard

Tomeu, you added that check in

[989561de9b5112999475b406557d9c7e9e59c041] PM / Domains: add setter for dev.pm_domain

and either something is wrong in the platform device handling or elsewhere in the device code, if
that check is valid.

FYI, I'm hitting the exact same error on shutdown on one of my boxes.

Please Cc me on updates.

-- Steve

[53591.087861] kvm: exiting hardware virtualization
[53591.104798] ------------[ cut here ]------------
[53591.110058] WARNING: CPU: 0 PID: 1 at /home/rostedt/work/git/linux-trace.git/drivers/base/power/common.c:150 dev_pm_domain_set+0x82/0x90()
[53591.123716] PM domains can only be changed for unbound devices
[53591.130158] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables xt_CHECKSUM iptable_mangle bridge stp llc bluetooth lockd grace snd_hda_codec_hdmi snd_hda_codec_r
ealtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hwdep snd_hda_core vhost_net tun vhost x86_pkg_temp_thermal iTCO_wdt snd_seq snd_seq_device snd_pcm macvtap coretemp me
i_me iTCO_vendor_support hp_wmi rfkill sparse_keymap macvlan kvm_intel snd_timer mei lpc_ich snd i2c_i801 soundcore mfd_core kvm irqbypass acpi_cpufreq serio_raw wmi uinput crc32_p
clmul i915 crc32c_intel i2c_algo_bit e1000e drm_kms_helper ptp drm pps_core i2c_core video sunrpc
[53591.190440] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 4.5.0-rc1-test+ #155
[53591.198453] Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v02.05 05/07/2012
[53591.208031]  ffff8801195abc90 ffff8801195abc90 ffffffff8143ef63 ffff8801195abcd8
[53591.216168]  ffff8801195abcc8 ffffffff810acad6 ffff8801192d2328 0000000000000000
[53591.224299]  0000000000000001 ffff8801192d2388 00000000fee1dead ffff8801195abd28
[53591.232429] Call Trace:
[53591.235483]  [<ffffffff8143ef63>] dump_stack+0x44/0x61
[53591.241232]  [<ffffffff810acad6>] warn_slowpath_common+0x86/0xc0
[53591.247842]  [<ffffffff810acb5c>] warn_slowpath_fmt+0x4c/0x50
[53591.254193]  [<ffffffff810acb15>] ? warn_slowpath_fmt+0x5/0x50
[53591.260625]  [<ffffffff8157cf42>] dev_pm_domain_set+0x82/0x90
[53591.266970]  [<ffffffffa03f255e>] mei_me_remove+0xee/0x120 [mei_me]
[53591.273842]  [<ffffffff8147fe96>] pci_device_shutdown+0x36/0x70
[53591.280373]  [<ffffffff81570680>] device_shutdown+0xe0/0x1e0
[53591.286642]  [<ffffffff810d6df6>] kernel_restart_prepare+0x36/0x40
[53591.293439]  [<ffffffff810d6f62>] kernel_restart+0x12/0x60
[53591.299543]  [<ffffffff810d72ae>] SYSC_reboot+0x1ce/0x1f0
[53591.305562]  [<ffffffffa0008077>] ? 0xffffffffa0008077
[53591.311323]  [<ffffffff811b2e60>] ? stack_trace_call+0x40/0x60
[53591.317780]  [<ffffffffa0008077>] ? 0xffffffffa0008077
[53591.323552]  [<ffffffff8143eeb4>] ? _atomic_dec_and_lock+0x44/0xaf
[53591.330374]  [<ffffffff812a302b>] ? iput+0xbb/0x2c0
[53591.335880]  [<ffffffff810d7325>] ? SyS_reboot+0x5/0x10
Quoted 9 lines
[53591.341748]  [<ffffffff81003044>] ? lockdep_sys_exit_thunk+0x12/0x14
[53591.348743]  [<ffffffff810d732e>] SyS_reboot+0xe/0x10
[53591.354440]  [<ffffffff8185ee32>] entry_SYSCALL_64_fastpath+0x12/0x76
[53591.361534] ---[ end trace 63b298fc6d5920e4 ]---
[53591.377714] e1000e: EEE TX LPI TIMER: 00000011
[53591.468245] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[53591.487321] reboot: Restarting system
[53591.491679] reboot: machine restart

Looks like Rafael adjusted for the platform shutdown case here:

Quoted 7 lines
Quoted 3 lines

https://lkml.org/lkml/2016/1/11/515

Perhaps that would applicable to the pci shutdown as well?

To the MEI driver's shutdown rather. mei_me_remove() clears the pm_domain pointer which is sort of questionable, but then the warning may be overkill for this case.

Since MEI device is running its own power management, in some cases we are using PM domains just to avoid going through the PCI runtime pm handlers to avoid going to D3. IIRC we couldn't use PCI_DEV_FLAGS_NO_D3 as this not unique for runtime handlers I think this requirement is unique to the MEI device, so I'm not sure it's worth to push it into the PCI layer, maybe we just quite the warning somehow.

Tomas

#7 lore raw
Quoted 72 lines · Tomas Winkler
Quoted 63 lines · Rafael J. Wysocki
Quoted 62 lines · Joe Lawrence
Quoted 49 lines · Steven Rostedt
Quoted 9 lines · Corey Minyard

Tomeu, you added that check in

[989561de9b5112999475b406557d9c7e9e59c041] PM / Domains: add setter for dev.pm_domain

and either something is wrong in the platform device handling or elsewhere in the device code, if
that check is valid.

FYI, I'm hitting the exact same error on shutdown on one of my boxes.

Please Cc me on updates.

-- Steve

[53591.087861] kvm: exiting hardware virtualization
[53591.104798] ------------[ cut here ]------------
[53591.110058] WARNING: CPU: 0 PID: 1 at /home/rostedt/work/git/linux-trace.git/drivers/base/power/common.c:150 dev_pm_domain_set+0x82/0x90()
[53591.123716] PM domains can only be changed for unbound devices
[53591.130158] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables xt_CHECKSUM iptable_mangle bridge stp llc bluetooth lockd grace snd_hda_codec_hdmi snd_hda_codec_r
ealtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hwdep snd_hda_core vhost_net tun vhost x86_pkg_temp_thermal iTCO_wdt snd_seq snd_seq_device snd_pcm macvtap coretemp me
i_me iTCO_vendor_support hp_wmi rfkill sparse_keymap macvlan kvm_intel snd_timer mei lpc_ich snd i2c_i801 soundcore mfd_core kvm irqbypass acpi_cpufreq serio_raw wmi uinput crc32_p
clmul i915 crc32c_intel i2c_algo_bit e1000e drm_kms_helper ptp drm pps_core i2c_core video sunrpc
[53591.190440] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 4.5.0-rc1-test+ #155
[53591.198453] Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v02.05 05/07/2012
[53591.208031]  ffff8801195abc90 ffff8801195abc90 ffffffff8143ef63 ffff8801195abcd8
[53591.216168]  ffff8801195abcc8 ffffffff810acad6 ffff8801192d2328 0000000000000000
[53591.224299]  0000000000000001 ffff8801192d2388 00000000fee1dead ffff8801195abd28
[53591.232429] Call Trace:
[53591.235483]  [<ffffffff8143ef63>] dump_stack+0x44/0x61
[53591.241232]  [<ffffffff810acad6>] warn_slowpath_common+0x86/0xc0
[53591.247842]  [<ffffffff810acb5c>] warn_slowpath_fmt+0x4c/0x50
[53591.254193]  [<ffffffff810acb15>] ? warn_slowpath_fmt+0x5/0x50
[53591.260625]  [<ffffffff8157cf42>] dev_pm_domain_set+0x82/0x90
[53591.266970]  [<ffffffffa03f255e>] mei_me_remove+0xee/0x120 [mei_me]
[53591.273842]  [<ffffffff8147fe96>] pci_device_shutdown+0x36/0x70
[53591.280373]  [<ffffffff81570680>] device_shutdown+0xe0/0x1e0
[53591.286642]  [<ffffffff810d6df6>] kernel_restart_prepare+0x36/0x40
[53591.293439]  [<ffffffff810d6f62>] kernel_restart+0x12/0x60
[53591.299543]  [<ffffffff810d72ae>] SYSC_reboot+0x1ce/0x1f0
[53591.305562]  [<ffffffffa0008077>] ? 0xffffffffa0008077
[53591.311323]  [<ffffffff811b2e60>] ? stack_trace_call+0x40/0x60
[53591.317780]  [<ffffffffa0008077>] ? 0xffffffffa0008077
[53591.323552]  [<ffffffff8143eeb4>] ? _atomic_dec_and_lock+0x44/0xaf
[53591.330374]  [<ffffffff812a302b>] ? iput+0xbb/0x2c0
[53591.335880]  [<ffffffff810d7325>] ? SyS_reboot+0x5/0x10
Quoted 9 lines
[53591.341748]  [<ffffffff81003044>] ? lockdep_sys_exit_thunk+0x12/0x14
[53591.348743]  [<ffffffff810d732e>] SyS_reboot+0xe/0x10
[53591.354440]  [<ffffffff8185ee32>] entry_SYSCALL_64_fastpath+0x12/0x76
[53591.361534] ---[ end trace 63b298fc6d5920e4 ]---
[53591.377714] e1000e: EEE TX LPI TIMER: 00000011
[53591.468245] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[53591.487321] reboot: Restarting system
[53591.491679] reboot: machine restart

Looks like Rafael adjusted for the platform shutdown case here:

Quoted 7 lines
Quoted 3 lines

https://lkml.org/lkml/2016/1/11/515

Perhaps that would applicable to the pci shutdown as well?

To the MEI driver's shutdown rather. mei_me_remove() clears the pm_domain pointer which is sort of questionable, but then the warning may be overkill for this case.

Quoted 7 lines

Since MEI device is running its own power management, in some cases we are using PM domains just to avoid going through the PCI runtime pm handlers to avoid going to D3. IIRC we couldn't use PCI_DEV_FLAGS_NO_D3 as this not unique for runtime handlers I think this requirement is unique to the MEI device, so I'm not sure it's worth to push it into the PCI layer, maybe we just quite the warning somehow.

Does this help: https://patchwork.kernel.org/patch/8170821/ ?

Thanks,
Rafael