um: MSI parent domain conversion
cover um: MSI parent domain conversion
The initial implementation of PCI/MSI interrupt domains in the hierarchical interrupt domain model used a shortcut by providing a global PCI/MSI domain.
This works because the PCI/MSI[X] hardware is standardized and uniform, but it violates the basic design principle of hierarchical interrupt domains:
Each hardware block involved in the interrupt delivery chain should have a separate interrupt domain.
For PCI/MSI[X], the interrupt controller is per PCI device and not a global made-up entity.
Unsurprisingly, the shortcut turned out to have downsides as it does not allow dynamic allocation of interrupt vectors after initialization and it prevents supporting IMS on PCI. For further details, see:
https://lore.kernel.org/lkml/[email protected]/
The solution is implementing per device MSI domains, this means the entities which provide global PCI/MSI domain so far have to implement MSI parent domain functionality instead.
This series converts the um's driver to implement MSI parent domain.
arch/um/drivers/Kconfig | 1 + arch/um/drivers/virt-pci.c | 46 +++++++++++++++++++------------------- 2 files changed, 24 insertions(+), 23 deletions(-)
Signature · 1 line
2.39.5
PATCH 1/1 um: virt-pci: Switch to msi_create_parent_irq_domain()
Move away from the legacy MSI domain setup, switch to use msi_create_parent_irq_domain().
--- arch/um/drivers/Kconfig | 1 + arch/um/drivers/virt-pci.c | 46 +++++++++++++++++++------------------- 2 files changed, 24 insertions(+), 23 deletions(-)
arch/um/drivers/Kconfig +1 -0
@@ -160,6 +160,7 @@ config UML_RTC config UML_PCI bool select FORCE_PCI + select IRQ_MSI_LIB select UML_IOMEM_EMULATION select UML_DMA_EMULATION select PCI_MSI
arch/um/drivers/virt-pci.c +23 -23
81 lines not on this page · raw message
Signature · 1 line
2.39.5
Johannes Berg
maintainerRe: [PATCH 1/1] um: virt-pci: Switch to msi_create_parent_irq_domain()
Quoted 2 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().
I applied this on 6.16-rc1 (plus a handful of local patches), and got a lockdep warning and kernel crash with this change:
================================
WARNING: inconsistent lock state
6.16.0-rc1-g05358285a53b #1 Tainted: G O
--------------------------------
inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage.
modprobe/462 [HC0[0]:SC0[0]:HE0:SE1] takes:0000000061b3fd30 (&irq_desc_lock_class){?...}-{2:2}, at: __setup_irq+0x4ac/0x12d5 {IN-HARDIRQ-W} state was registered at:
save_stack_trace+0x55/0x62 stack_trace_save+0x81/0xa4 save_trace+0xc7/0x354 mark_lock+0x17d/0x4d9 __lock_acquire+0x670/0x1a60 lock_acquire+0x172/0x30e _raw_spin_lock+0xa4/0x107 handle_edge_irq+0x2e/0x2a6 handle_irq_desc+0x182/0x1a0 generic_handle_irq+0x1f/0x23 irq_event_handler+0xbb/0xc8 deliver_time_travel_irqs+0x254/0x2c3 unblock_signals.part.0+0x34/0x11c unblock_signals+0x1f/0x26 um_set_signals+0x26/0x40 time_travel_update_time_rel+0xeb/0xff time_travel_ndelay+0x10/0x16 virtio_pcidev_send_cmd+0x693/0x73a virtio_pcidev_cfgspace_read+0xbc/0x129 um_pci_cfgspace_read+0xde/0xf5 __raw_readl+0xe6/0xfc pci_generic_config_read+0xfc/0x133 pci_bus_read_config_dword+0xe0/0x18f pci_bus_generic_read_dev_vendor_id+0x2e/0x2b4 pci_bus_read_dev_vendor_id+0xbf/0xd6 pci_scan_single_device+0xbc/0x218 pci_scan_slot+0x277/0x3d6 pci_scan_child_bus_extend+0xa3/0x5be pci_scan_child_bus+0x12/0x18 pci_rescan_bus+0x16/0x41 um_pci_device_register+0x1b2/0x1f6 virtio_pcidev_virtio_probe+0x4f2/0x5ee virtio_dev_probe+0x53e/0x639 really_probe+0x3f6/0x8d0 __driver_probe_device+0x373/0x3f3 driver_probe_device+0x72/0x12c __driver_attach+0x35d/0x3ae bus_for_each_dev+0x136/0x184 driver_attach+0x4e/0x5a bus_add_driver+0x343/0x63c driver_register+0x2e4/0x3a1 __register_virtio_driver+0x13e/0x14e 0x600066bf do_one_initcall+0x13b/0x46e 0x60002e8a kernel_init+0x29/0x1f0 new_thread_handler+0xf2/0x119 irq event stamp: 118040 hardirqs last enabled at (118039): [<00000000603cc7af>] kasan_quarantine_put+0x1b9/0x1f3 hardirqs last disabled at (118040): [<0000000060f3c278>] _raw_spin_lock_irqsave+0x6d/0x16d softirqs last enabled at (117512): [<00000000600ac1db>] handle_softirqs+0x749/0x8c5 softirqs last disabled at (117339): [<0000000060f3db56>] __do_softirq+0x12/0x18
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&irq_desc_lock_class);
<Interrupt>
lock(&irq_desc_lock_class);*** DEADLOCK ***
3 locks held by modprobe/462:
#0: 000000006518a1a8 (&dev->mutex){....}-{4:4}, at: device_lock+0x16/0x1e #1: 0000000061b3fe58 (&desc->request_mutex){+.+.}-{4:4}, at: __setup_irq+0x332/0x12d5 #2: 0000000061b3fd30 (&irq_desc_lock_class){?...}-{2:2}, at: __setup_irq+0x4ac/0x12d5
stack backtrace: CPU: 0 UID: 0 PID: 462 Comm: modprobe Tainted: G O 6.16.0-rc1-g05358285a53b #1 VOLUNTARY Tainted: [O]=OOT_MODULE Stack: 66454828 6119fd60 61cb6fa0 00000000 ffffff00 6119fd60 612dd005 6015081f 61cb6ff0 60066af1 00000000 6005180e Call Trace: [<6005180e>] ? _printk+0x0/0xdc [<6006d7f5>] show_stack+0x1bd/0x1d4 [<6015081f>] ? __print_lock_name+0x0/0xb7 [<60066af1>] dump_stack_lvl+0xab/0x12f [<6005180e>] ? _printk+0x0/0xdc [<60066b8f>] dump_stack+0x1a/0x20 [<60154818>] print_usage_bug+0x309/0x334 [<60154843>] ? valid_state+0x0/0x98 [<601548c6>] valid_state+0x83/0x98 [<6014f895>] ? hlock_class+0x0/0xcd [<60154a9e>] mark_lock+0x1c3/0x4d9 [<60071f82>] ? segv+0x336/0x71b [<602e2a12>] ? _end_441+0x9c/0x16c [<60071c4c>] ? segv+0x0/0x71b [<601548db>] ? mark_lock+0x0/0x4d9 [<60154e0a>] mark_held_locks+0x56/0x70 [<60154db4>] ? mark_held_locks+0x0/0x70 [<60155086>] lockdep_hardirqs_on_prepare+0x262/0x319 [<6008b3ef>] ? um_set_signals+0x0/0x40 [<60288534>] trace_hardirqs_on+0x1b4/0x1d6 [<6006cfb8>] unblock_signals_trace+0x47/0x5d [<6008af06>] sig_handler_common+0xa1/0xe4 [<6008b5d1>] sig_handler+0xcb/0xea [<6008ae45>] hard_handler+0x18/0x24 Modules linked in: iwlwifi(O+) iwlmei(O) cfg80211(O) meisim(O) vlab_control(O) compat(O) Pid: 462, comm: modprobe Tainted: G O 6.16.0-rc1-g05358285a53b RIP: 0033:0x0 RSP: 00000000724ff038 EFLAGS: 00010246 RAX: 0000100000000000 RBX: 000000006aa4c400 RCX: 0000000000000000 RDX: 000000000c280950 RSI: 0000000000000001 RDI: 000000006aa4c400 RBP: 00000000724ff050 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000061404a40 R13: 000000006a813030 R14: 0000000061b3fc98 R15: 0000000000000001 Kernel panic - not syncing: Kernel mode fault at addr 0x0, ip 0x0 CPU: 0 UID: 0 PID: 462 Comm: modprobe Tainted: G O 6.16.0-rc1-g05358285a53b #1 VOLUNTARY Tainted: [O]=OOT_MODULE Stack: 6018488e 61b3fc98 6a813198 724ff070 607b3a44 61b3fc80 61b3fca8 724ff0a0 60186c6e 0d502609 61b3fc80 61b3fca8 Call Trace: [<6018488e>] ? irq_chip_unmask_parent+0x8d/0x98 [<607b3a44>] pci_irq_unmask_msix+0xa3/0x147 [<60186c6e>] unmask_irq+0xd1/0x138 [<60186d52>] irq_enable+0x7d/0x151 [<6018705a>] irq_startup+0x234/0x29e [<60180e06>] __setup_irq+0xdad/0x12d5 [<733ea37a>] ? iwl_pcie_msix_isr+0x0/0x6 [iwlwifi] [<60181598>] request_threaded_irq+0x26a/0x2e5 [<733e79c3>] ? iwl_pcie_irq_rx_msix_handler+0x0/0x55f [iwlwifi] [<60188f5c>] devm_request_threaded_irq+0xe6/0x18b [<733e79c3>] ? iwl_pcie_irq_rx_msix_handler+0x0/0x55f [iwlwifi] [<733ea37a>] ? iwl_pcie_msix_isr+0x0/0x6 [iwlwifi] [<7340d595>] iwl_pci_gen1_2_probe+0x15b3/0x2a0f [iwlwifi] [<60087977>] ? virtio_pcidev_bar_copy_from+0xb1/0xd3 [<7340bfe2>] ? iwl_pci_gen1_2_probe+0x0/0x2a0f [iwlwifi] [<6008fefe>] ? copy_sc_from_user+0x4a0/0x632 [<6008b3e8>] ? unblock_signals+0x1f/0x26 [<6008b415>] ? um_set_signals+0x26/0x40 [<600859fa>] ? um_pci_bar_read+0x0/0x111 [<733ddc30>] iwl_pci_probe+0x281/0x2b1 [iwlwifi] [<6008b3e8>] ? unblock_signals+0x1f/0x26 [<733dd9af>] ? iwl_pci_probe+0x0/0x2b1 [iwlwifi] [<60f3c65d>] ? _raw_spin_unlock_irqrestore+0x9f/0x11a [<6079c55d>] pci_device_probe+0x1bb/0x324 [<60832c93>] really_probe+0x3f6/0x8d0 [<6083289d>] ? really_probe+0x0/0x8d0 [<603cbea0>] ? __kasan_check_read+0x0/0x22 [<608334e0>] __driver_probe_device+0x373/0x3f3 [<603cbec2>] ? __kasan_check_write+0x0/0x25 [<603cbea0>] ? __kasan_check_read+0x0/0x22 [<60833696>] driver_probe_device+0x72/0x12c [<60833a15>] ? __driver_attach+0x0/0x3ae [<60eda91e>] ? klist_next+0x0/0x2d5 [<60833d72>] __driver_attach+0x35d/0x3ae [<60833a15>] ? __driver_attach+0x0/0x3ae [<60eda91e>] ? klist_next+0x0/0x2d5 [<6082d714>] bus_for_each_dev+0x136/0x184 [<60eda021>] ? klist_add_tail+0x84/0x94 [<6082d5de>] ? bus_for_each_dev+0x0/0x184 [<60830f1c>] driver_attach+0x4e/0x5a [<6082fc3c>] bus_add_driver+0x343/0x63c [<60835a96>] driver_register+0x2e4/0x3a1 [<607993b6>] __pci_register_driver+0x249/0x25d [<733de1d6>] iwl_pci_register_driver+0x30/0x5c [iwlwifi] [<7208514e>] __init_backport+0x14e/0x1000 [iwlwifi] [<72085000>] ? __init_backport+0x0/0x1000 [iwlwifi] [<60067ede>] do_one_initcall+0x13b/0x46e [<603c96bc>] ? poison_kmalloc_redzone+0x76/0x89 [<60067da3>] ? do_one_initcall+0x0/0x46e [<603cac13>] ? __asan_register_globals+0x6a/0x85 [<601bbe3c>] do_init_module+0x2f5/0x7fd [<601bbb47>] ? do_init_module+0x0/0x7fd [<601c12fb>] load_module+0x492e/0x4df4 [<603e1938>] ? __kernel_read+0x0/0x3b7 [<601bc9cd>] ? load_module+0x0/0x4df4 [<603e1cef>] ? kernel_read+0x0/0x5b [<603e1d3a>] ? kernel_read+0x4b/0x5b [<6048d18e>] ? kernel_read_file+0x406/0x466 [<60156d68>] ? __lock_acquire+0x1838/0x1a60 [<601c1d0a>] init_module_from_file+0x11a/0x160 [<601c1bf0>] ? init_module_from_file+0x0/0x160 [<6008b3e8>] ? unblock_signals+0x1f/0x26 [<60f3c40d>] ? _raw_spin_unlock+0x0/0xa6 [<60f3c40d>] ? _raw_spin_unlock+0x0/0xa6 [<601c2020>] idempotent_init_module+0x2d0/0x59c [<60f3bcd6>] ? _raw_spin_lock+0x0/0x107 [<601c1d50>] ? idempotent_init_module+0x0/0x59c [<6043dc3a>] ? __fget_light+0xd1/0x13f [<601c23d6>] sys_finit_module+0xb2/0xe7 [<601c2324>] ? sys_finit_module+0x0/0xe7 [<60073ed9>] handle_syscall+0x2cc/0x38b [<6008dcca>] userspace+0x677/0x727 [<6006bb49>] ? interrupt_end+0x0/0x11d [<6006b9b1>] fork_handler+0xa9/0xb4
johannes
Re: [PATCH 1/1] um: virt-pci: Switch to msi_create_parent_irq_domain()
Quoted 6 lines · Johannes Berg
Quoted 2 lines · Nam Cao
Move away from the legacy MSI domain setup, switch to use
msi_create_parent_irq_domain().I applied this on 6.16-rc1 (plus a handful of local patches), and got a lockdep warning and kernel crash with this change:
Thanks for testing! The later kernel crash should be fixed with:
arch/um/drivers/virt-pci.c +0 -1
@@ -407,7 +407,6 @@ static const struct irq_domain_ops um_pci_inner_domain_ops = {#define UM_PCI_MSI_FLAGS_REQUIRED (MSI_FLAG_USE_DEF_DOM_OPS | \ MSI_FLAG_USE_DEF_CHIP_OPS | \ - MSI_FLAG_PCI_MSI_MASK_PARENT | \ MSI_FLAG_NO_AFFINITY)#define UM_PCI_MSI_FLAGS_SUPPORTED (MSI_GENERIC_FLAGS_MASK | \ MSI_FLAG_PCI_MSIX)
I have no immediate idea about the lockdep warn, I don't see how the MSI_FLAG_PCI_MSI_MASK_PARENT flag can be related to that. Probably there's another problem, let me stare at it..
Nam
Johannes Berg
maintainerRe: [PATCH 1/1] um: virt-pci: Switch to msi_create_parent_irq_domain()
arch/um/drivers/virt-pci.c static const struct irq_domain_ops um_pci_inner_domain_ops = { · line 410
Quoted 13 lines · Nam Cao
Thanks for testing! The later kernel crash should be fixed with:
arch/um/drivers/virt-pci.c +0
-1@@ -407,7 +407,6 @@ static const struct irq_domain_ops um_pci_inner_domain_ops = {#define UM_PCI_MSI_FLAGS_REQUIRED (MSI_FLAG_USE_DEF_DOM_OPS | \ MSI_FLAG_USE_DEF_CHIP_OPS | \ - MSI_FLAG_PCI_MSI_MASK_PARENT | \ MSI_FLAG_NO_AFFINITY)
Can confirm that.
Quoted 3 lines
I have no immediate idea about the lockdep warn, I don't see how the
MSI_FLAG_PCI_MSI_MASK_PARENTflag can be related to that. Probably there's another problem, let me stare at it..
Seems to go away as well, perhaps it's due to taking some wrong path with the flag just prior to the crash?
IOW, works for me with the above change with no crash and no lockdep warning either.
johannes
Re: [PATCH 1/1] um: virt-pci: Switch to msi_create_parent_irq_domain()
Quoted 10 lines · Johannes Berg
Quoted 9 lines · Nam Cao
Thanks for testing! The later kernel crash should be fixed with:
arch/um/drivers/virt-pci.c +0
-0@@ -407,7 +407,6 @@ static const struct irq_domain_ops um_pci_inner_domain_ops = {
Quoted 10 lines
arch/um/drivers/virt-pci.c static const struct irq_domain_ops um_pci_inner_domain_ops = { · line 410
Quoted 4 lines
arch/um/drivers/virt-pci.c +0
-1#define UM_PCI_MSI_FLAGS_REQUIRED (MSI_FLAG_USE_DEF_DOM_OPS | \ MSI_FLAG_USE_DEF_CHIP_OPS | \ - MSI_FLAG_PCI_MSI_MASK_PARENT | \ MSI_FLAG_NO_AFFINITY)Can confirm that.
Quoted 3 lines
I have no immediate idea about the lockdep warn, I don't see how the
MSI_FLAG_PCI_MSI_MASK_PARENTflag can be related to that. Probably there's another problem, let me stare at it..
My understanding of UM is limited, but I think I get it. We dereference a NULL pointer and get SIGSEGV. We go into that code path and lockdep yells.
With the NULL pointer deref removed, we won't go into that path, so it should be good.
Quoted 5 lines
Seems to go away as well, perhaps it's due to taking some wrong path with the flag just prior to the crash?
IOW, works for me with the above change with no crash and no lockdep warning either.
I will send v2.
Nam
Re: [PATCH 1/1] um: virt-pci: Switch to msi_create_parent_irq_domain()
Quoted 5 lines · Nam Cao
My understanding of UM is limited, but I think I get it. We dereference a NULL pointer and get SIGSEGV. We go into that code path and lockdep yells.
With the NULL pointer deref removed, we won't go into that path, so it should be good.
By "that code path" I mean SIGSEGV handler.
Nam