Skip to contentSkip to conversation

"irq/matrix: Spread interrupts on allocation" breaks nouveau in mainline kernel

DISCUSSION Lyude Paul 5 participants 16 messages Jan 23, 2018 – Jan 25, 2018

lore
Share & tools
lore

Lyude Paul

author
#0 lore raw

Hi! Sorry to be the bearer of bad news, but this patch actually seems to break suspending and resuming with nouveau on my machine:

[   29.694755] PM: suspend entry (deep)
[   29.694773] PM: Syncing filesystems ... done.
[   29.696203] Freezing user space processes ... (elapsed 0.001 seconds) done.
[   29.697442] OOM killer disabled.
[   29.697448] Freezing remaining freezable tasks ... (elapsed 0.000 seconds)
done.
[   29.698232] Suspending console(s) (use no_console_suspend to debug)
[   29.698993] serial 00:05: disabled
[   29.708227] sd 4:0:0:0: [sda] Synchronizing SCSI cache
[   29.708428] sd 4:0:0:0: [sda] Stopping disk
[   30.614581] ACPI: Preparing to enter system sleep state S3
[   30.917726] PM: Saving platform NVS memory
[   30.917736] Disabling non-boot CPUs ...
[   30.925616] smpboot: CPU 1 is now offline
[   30.936915] smpboot: CPU 2 is now offline
[   30.952824] smpboot: CPU 3 is now offline
[   30.964764] smpboot: CPU 4 is now offline
[   30.980663] smpboot: CPU 5 is now offline
[   30.992692] smpboot: CPU 6 is now offline
[   31.002572] smpboot: CPU 7 is now offline
[   31.003130] ACPI: Low-level resume complete
[   31.003180] PM: Restoring platform NVS memory
[   31.003578] WARNING: CPU: 0 PID: 11523 at kernel/smp.c:291
smp_call_function_single+0xdc/0xe0
[   31.003578] Modules linked in: nouveau video mxm_wmi i2c_algo_bit ttm
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm vfat fat usbhid
crc32_pclmul i2c_piix4 i2c_core shpchp k10temp wmi acpi_cpufreq crc32c_intel
r8169 mii xhci_pci xhci_hcd w83627hf_wdt
[   31.003590] CPU: 0 PID: 11523 Comm: rtcwake Not tainted 4.15.0-rc8nouveau-
clockgating+ #1
[   31.003591] Hardware name: MSI MS-7A39/A320M GAMING PRO (MS-7A39), BIOS 1.60
09/19/2017
[   31.003592] RIP: 0010:smp_call_function_single+0xdc/0xe0
[   31.003593] RSP: 0018:ffffc900004a3c40 EFLAGS: 00010046
[   31.003594] RAX: 0000000000000000 RBX: ffffc900004a3cdc RCX: 0000000000000001
[   31.003594] RDX: ffffc900004a3c98 RSI: ffffffff8137a180 RDI: 0000000000000000
[   31.003595] RBP: ffffc900004a3c70 R08: 0000000000000001 R09: 0000000000010000
[   31.003595] R10: ffffc900004a3c98 R11: 0000000000000000 R12: 0000000000000000
[   31.003596] R13: 0000000001000000 R14: ffffc900004a3d0c R15: 0000000000000000
[   31.003597] FS:  00007f03bee93540(0000) GS:ffff88021ae00000(0000)
knlGS:0000000000000000
[   31.003597] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   31.003598] CR2: 00007fffb6673008 CR3: 000000020ddd4000 CR4: 00000000003406f0
[   31.003598] Call Trace:
[   31.003603]  ? rdmsr_safe_on_cpu+0x4b/0x70
[   31.003604]  rdmsr_safe_on_cpu+0x4b/0x70
[   31.003606]  get_block_address.isra.0+0x6e/0xe0
[   31.003607]  mce_amd_feature_init+0x63/0x2c0
[   31.003609]  mce_syscore_resume+0x1e/0x30
[   31.003611]  syscore_resume+0x4b/0x170
[   31.003613]  suspend_devices_and_enter+0x608/0x7e0
[   31.003614]  pm_suspend+0x315/0x380
[   31.003615]  state_store+0x7d/0xe0
[   31.003618]  kernfs_fop_write+0xfa/0x180
[   31.003620]  __vfs_write+0x23/0x130
[   31.003623]  ? SYSC_newfstat+0x29/0x40
[   31.003625]  ? _cond_resched+0x15/0x40
[   31.003626]  vfs_write+0xad/0x1a0
[   31.003627]  SyS_write+0x42/0x90
[   31.003629]  entry_SYSCALL_64_fastpath+0x24/0x87
[   31.003630] RIP: 0033:0x7f03be9ae8f4
[   31.003631] RSP: 002b:00007ffe6bf825f8 EFLAGS: 00000246
[   31.003632] Code: fe ff ff 8b 55 e8 83 e2 01 74 0a f3 90 8b 55 e8 83 e2 01 75
f6 48 83 c4 28 41 5a 5d 49 8d 62 f8 c3 8b 05 58 b6 48 01 85 c0 75 86 <0f> ff eb
82 0f 1f 44 00 00 f6 46 18 01 75 15 c7 46 18 01 00 00 
[   31.003648] ---[ end trace 19fa2f7781ed5237 ]---
[   31.004025] Enabling non-boot CPUs ...
[   31.004052] x86: Booting SMP configuration:
[   31.004052] smpboot: Booting Node 0 Processor 1 APIC 0x1
[   31.006368]  cache: parent cpu1 should not be sleeping
[   31.006442] microcode: CPU1: patch_level=0x08001129
[   31.006509] CPU1 is up
[   31.006525] smpboot: Booting Node 0 Processor 2 APIC 0x2
[   31.008832]  cache: parent cpu2 should not be sleeping
[   31.008894] microcode: CPU2: patch_level=0x08001129
[   31.008966] CPU2 is up
[   31.008975] smpboot: Booting Node 0 Processor 3 APIC 0x3
[   31.011264]  cache: parent cpu3 should not be sleeping
[   31.011329] microcode: CPU3: patch_level=0x08001129
[   31.011404] CPU3 is up
[   31.011413] smpboot: Booting Node 0 Processor 4 APIC 0x8
[   31.013833]  cache: parent cpu4 should not be sleeping
[   31.013903] microcode: CPU4: patch_level=0x08001129
[   31.014025] CPU4 is up
[   31.014036] smpboot: Booting Node 0 Processor 5 APIC 0x9
[   31.016354]  cache: parent cpu5 should not be sleeping
[   31.016421] microcode: CPU5: patch_level=0x08001129
[   31.016534] CPU5 is up
[   31.016544] smpboot: Booting Node 0 Processor 6 APIC 0xa
[   31.018857]  cache: parent cpu6 should not be sleeping
[   31.018930] microcode: CPU6: patch_level=0x08001129
[   31.019047] CPU6 is up
[   31.019057] smpboot: Booting Node 0 Processor 7 APIC 0xb
[   31.021376]  cache: parent cpu7 should not be sleeping
[   31.021444] microcode: CPU7: patch_level=0x08001129
[   31.021579] CPU7 is up
[   31.022166] ACPI: Waking up from system sleep state S3
[   31.070791] usb usb1: root hub lost power or was reset
[   31.070794] usb usb2: root hub lost power or was reset
[   31.071628] serial 00:05: activated
[   31.080265] sd 4:0:0:0: [sda] Starting disk
[   31.126099] hpet_rtc_timer_reinit: 68 callbacks suppressed
[   31.126099] hpet1: lost 2 rtc interrupts
[   31.160913] r8169 0000:1e:00.0 enp30s0: link down
[   31.255563] do_IRQ: 1.35 No irq handler for vector
[   31.379537] ata6: SATA link down (SStatus 0 SControl 300)
[   31.379558] ata1: SATA link down (SStatus 0 SControl 300)
[   31.380306] ata2: SATA link down (SStatus 0 SControl 300)
[   31.435705] ata9: SATA link down (SStatus 0 SControl 300)
[   31.589932] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   31.590320] ata5.00: configured for UDMA/133
[   31.610043] usb 1-4: reset low-speed USB device number 2 using xhci_hcd
[   32.226138] usb 1-5: reset low-speed USB device number 3 using xhci_hcd
[   33.257867] nouveau 0000:22:00.0: DRM: EVO timeout
[   34.237185] r8169 0000:1e:00.0 enp30s0: link up
[   35.257880] nouveau 0000:22:00.0: DRM: base-0: timeout
[   37.258334] nouveau 0000:22:00.0: DRM: base-0: timeout
[   37.276084] OOM killer enabled.
[   37.276612] Restarting tasks ... done.
[   37.277722] PM: suspend exit

I haven't yet actually investigated why it does this, but a bisect of master led me to here.

author
#1 lore raw

JFYI: I confirmed this patch is definitely broken. I'm seeing nouveau get assigned the same MSI vector as another device on the system, which would explain why interrupts suddenly stop working. I'll keep looking into it further tomorrow.

Quoted 138 lines · Lyude Paul

Hi! Sorry to be the bearer of bad news, but this patch actually seems to break suspending and resuming with nouveau on my machine:

[   29.694755] PM: suspend entry (deep)
[   29.694773] PM: Syncing filesystems ... done.
[   29.696203] Freezing user space processes ... (elapsed 0.001 seconds) done.
[   29.697442] OOM killer disabled.
[   29.697448] Freezing remaining freezable tasks ... (elapsed 0.000 seconds)
done.
[   29.698232] Suspending console(s) (use no_console_suspend to debug)
[   29.698993] serial 00:05: disabled
[   29.708227] sd 4:0:0:0: [sda] Synchronizing SCSI cache
[   29.708428] sd 4:0:0:0: [sda] Stopping disk
[   30.614581] ACPI: Preparing to enter system sleep state S3
[   30.917726] PM: Saving platform NVS memory
[   30.917736] Disabling non-boot CPUs ...
[   30.925616] smpboot: CPU 1 is now offline
[   30.936915] smpboot: CPU 2 is now offline
[   30.952824] smpboot: CPU 3 is now offline
[   30.964764] smpboot: CPU 4 is now offline
[   30.980663] smpboot: CPU 5 is now offline
[   30.992692] smpboot: CPU 6 is now offline
[   31.002572] smpboot: CPU 7 is now offline
[   31.003130] ACPI: Low-level resume complete
[   31.003180] PM: Restoring platform NVS memory
[   31.003578] WARNING: CPU: 0 PID: 11523 at kernel/smp.c:291
smp_call_function_single+0xdc/0xe0
[   31.003578] Modules linked in: nouveau video mxm_wmi i2c_algo_bit ttm

drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm vfat fat usbhid
crc32_pclmul i2c_piix4 i2c_core shpchp k10temp wmi acpi_cpufreq crc32c_intel r8169 mii xhci_pci xhci_hcd w83627hf_wdt

[   31.003590] CPU: 0 PID: 11523 Comm: rtcwake Not tainted 4.15.0-rc8nouveau-
clockgating+ #1
[   31.003591] Hardware name: MSI MS-7A39/A320M GAMING PRO (MS-7A39), BIOS
1.60
09/19/2017
[   31.003592] RIP: 0010:smp_call_function_single+0xdc/0xe0
[   31.003593] RSP: 0018:ffffc900004a3c40 EFLAGS: 00010046
[   31.003594] RAX: 0000000000000000 RBX: ffffc900004a3cdc RCX:
0000000000000001
[   31.003594] RDX: ffffc900004a3c98 RSI: ffffffff8137a180 RDI:
0000000000000000
[   31.003595] RBP: ffffc900004a3c70 R08: 0000000000000001 R09:
0000000000010000
[   31.003595] R10: ffffc900004a3c98 R11: 0000000000000000 R12:
0000000000000000
[   31.003596] R13: 0000000001000000 R14: ffffc900004a3d0c R15:
0000000000000000
[   31.003597] FS:  00007f03bee93540(0000) GS:ffff88021ae00000(0000)
knlGS:0000000000000000
[   31.003597] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   31.003598] CR2: 00007fffb6673008 CR3: 000000020ddd4000 CR4:
00000000003406f0
[   31.003598] Call Trace:
[   31.003603]  ? rdmsr_safe_on_cpu+0x4b/0x70
[   31.003604]  rdmsr_safe_on_cpu+0x4b/0x70
[   31.003606]  get_block_address.isra.0+0x6e/0xe0
[   31.003607]  mce_amd_feature_init+0x63/0x2c0
[   31.003609]  mce_syscore_resume+0x1e/0x30
[   31.003611]  syscore_resume+0x4b/0x170
[   31.003613]  suspend_devices_and_enter+0x608/0x7e0
[   31.003614]  pm_suspend+0x315/0x380
[   31.003615]  state_store+0x7d/0xe0
[   31.003618]  kernfs_fop_write+0xfa/0x180
[   31.003620]  __vfs_write+0x23/0x130
[   31.003623]  ? SYSC_newfstat+0x29/0x40
[   31.003625]  ? _cond_resched+0x15/0x40
[   31.003626]  vfs_write+0xad/0x1a0
[   31.003627]  SyS_write+0x42/0x90
[   31.003629]  entry_SYSCALL_64_fastpath+0x24/0x87
[   31.003630] RIP: 0033:0x7f03be9ae8f4
[   31.003631] RSP: 002b:00007ffe6bf825f8 EFLAGS: 00000246
[   31.003632] Code: fe ff ff 8b 55 e8 83 e2 01 74 0a f3 90 8b 55 e8 83 e2 01

75
f6 48 83 c4 28 41 5a 5d 49 8d 62 f8 c3 8b 05 58 b6 48 01 85 c0 75 86 <0f> ff eb
82 0f 1f 44 00 00 f6 46 18 01 75 15 c7 46 18 01 00 00

[   31.003648] ---[ end trace 19fa2f7781ed5237 ]---
[   31.004025] Enabling non-boot CPUs ...
[   31.004052] x86: Booting SMP configuration:
[   31.004052] smpboot: Booting Node 0 Processor 1 APIC 0x1
[   31.006368]  cache: parent cpu1 should not be sleeping
[   31.006442] microcode: CPU1: patch_level=0x08001129
[   31.006509] CPU1 is up
[   31.006525] smpboot: Booting Node 0 Processor 2 APIC 0x2
[   31.008832]  cache: parent cpu2 should not be sleeping
[   31.008894] microcode: CPU2: patch_level=0x08001129
[   31.008966] CPU2 is up
[   31.008975] smpboot: Booting Node 0 Processor 3 APIC 0x3
[   31.011264]  cache: parent cpu3 should not be sleeping
[   31.011329] microcode: CPU3: patch_level=0x08001129
[   31.011404] CPU3 is up
[   31.011413] smpboot: Booting Node 0 Processor 4 APIC 0x8
[   31.013833]  cache: parent cpu4 should not be sleeping
[   31.013903] microcode: CPU4: patch_level=0x08001129
[   31.014025] CPU4 is up
[   31.014036] smpboot: Booting Node 0 Processor 5 APIC 0x9
[   31.016354]  cache: parent cpu5 should not be sleeping
[   31.016421] microcode: CPU5: patch_level=0x08001129
[   31.016534] CPU5 is up
[   31.016544] smpboot: Booting Node 0 Processor 6 APIC 0xa
[   31.018857]  cache: parent cpu6 should not be sleeping
[   31.018930] microcode: CPU6: patch_level=0x08001129
[   31.019047] CPU6 is up
[   31.019057] smpboot: Booting Node 0 Processor 7 APIC 0xb
[   31.021376]  cache: parent cpu7 should not be sleeping
[   31.021444] microcode: CPU7: patch_level=0x08001129
[   31.021579] CPU7 is up
[   31.022166] ACPI: Waking up from system sleep state S3
[   31.070791] usb usb1: root hub lost power or was reset
[   31.070794] usb usb2: root hub lost power or was reset
[   31.071628] serial 00:05: activated
[   31.080265] sd 4:0:0:0: [sda] Starting disk
[   31.126099] hpet_rtc_timer_reinit: 68 callbacks suppressed
[   31.126099] hpet1: lost 2 rtc interrupts
[   31.160913] r8169 0000:1e:00.0 enp30s0: link down
[   31.255563] do_IRQ: 1.35 No irq handler for vector
[   31.379537] ata6: SATA link down (SStatus 0 SControl 300)
[   31.379558] ata1: SATA link down (SStatus 0 SControl 300)
[   31.380306] ata2: SATA link down (SStatus 0 SControl 300)
[   31.435705] ata9: SATA link down (SStatus 0 SControl 300)
[   31.589932] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   31.590320] ata5.00: configured for UDMA/133
[   31.610043] usb 1-4: reset low-speed USB device number 2 using xhci_hcd
[   32.226138] usb 1-5: reset low-speed USB device number 3 using xhci_hcd
[   33.257867] nouveau 0000:22:00.0: DRM: EVO timeout
[   34.237185] r8169 0000:1e:00.0 enp30s0: link up
[   35.257880] nouveau 0000:22:00.0: DRM: base-0: timeout
[   37.258334] nouveau 0000:22:00.0: DRM: base-0: timeout
[   37.276084] OOM killer enabled.
[   37.276612] Restarting tasks ... done.
[   37.277722] PM: suspend exit

I haven't yet actually investigated why it does this, but a bisect of master led
me to here.

#14 lore raw
In reply to Lyude Paul
Quoted 5 lines · Lyude Paul

Hi! Sorry to be the bearer of bad news, but this patch actually seems to break suspending and resuming with nouveau on my machine:

[   31.003578] WARNING: CPU: 0 PID: 11523 at kernel/smp.c:291

smp_call_function_single+0xdc/0xe0

This warning has absolutely no relationship to that patch.

Quoted 9 lines
[   31.003592] RIP: 0010:smp_call_function_single+0xdc/0xe0
[   31.003603]  ? rdmsr_safe_on_cpu+0x4b/0x70
[   31.003604]  rdmsr_safe_on_cpu+0x4b/0x70
[   31.003606]  get_block_address.isra.0+0x6e/0xe0
[   31.003607]  mce_amd_feature_init+0x63/0x2c0
[   31.003609]  mce_syscore_resume+0x1e/0x30
[   31.003611]  syscore_resume+0x4b/0x170
[   31.003613]  suspend_devices_and_enter+0x608/0x7e0
[   31.003614]  pm_suspend+0x315/0x380
Quoted 8 lines
[   31.003615]  state_store+0x7d/0xe0
[   31.003618]  kernfs_fop_write+0xfa/0x180
[   31.003620]  __vfs_write+0x23/0x130
[   31.003623]  ? SYSC_newfstat+0x29/0x40
[   31.003625]  ? _cond_resched+0x15/0x40
[   31.003626]  vfs_write+0xad/0x1a0
[   31.003627]  SyS_write+0x42/0x90
[   31.003629]  entry_SYSCALL_64_fastpath+0x24/0x87

Borislav?

Thanks,

tglx

#2 lore raw
In reply to Lyude Paul
Quoted 4 lines · Lyude Paul

JFYI: I confirmed this patch is definitely broken. I'm seeing nouveau get assigned the same MSI vector as another device on the system, which would explain why interrupts suddenly stop working. I'll keep looking into it further tomorrow.

How did you determine that it is the same MSI vector as another device?

Thanks,

tglx

#15 lore raw
In reply to Thomas Gleixner
Quoted 28 lines · Thomas Gleixner
Quoted 5 lines · Lyude Paul

Hi! Sorry to be the bearer of bad news, but this patch actually seems to break suspending and resuming with nouveau on my machine:

[   31.003578] WARNING: CPU: 0 PID: 11523 at kernel/smp.c:291

smp_call_function_single+0xdc/0xe0

This warning has absolutely no relationship to that patch.

Quoted 9 lines
[   31.003592] RIP: 0010:smp_call_function_single+0xdc/0xe0
[   31.003603]  ? rdmsr_safe_on_cpu+0x4b/0x70
[   31.003604]  rdmsr_safe_on_cpu+0x4b/0x70
[   31.003606]  get_block_address.isra.0+0x6e/0xe0
[   31.003607]  mce_amd_feature_init+0x63/0x2c0
[   31.003609]  mce_syscore_resume+0x1e/0x30
[   31.003611]  syscore_resume+0x4b/0x170
[   31.003613]  suspend_devices_and_enter+0x608/0x7e0
[   31.003614]  pm_suspend+0x315/0x380
Quoted 8 lines
[   31.003615]  state_store+0x7d/0xe0
[   31.003618]  kernfs_fop_write+0xfa/0x180
[   31.003620]  __vfs_write+0x23/0x130
[   31.003623]  ? SYSC_newfstat+0x29/0x40
[   31.003625]  ? _cond_resched+0x15/0x40
[   31.003626]  vfs_write+0xad/0x1a0
[   31.003627]  SyS_write+0x42/0x90
[   31.003629]  entry_SYSCALL_64_fastpath+0x24/0x87
Quoted 1 line

Borislav?

Yeah

cfee4f6f0b20 ("x86/mce/AMD: Read MSRs on the CPU allocating the threshold blocks")

Yazen CCed.

Non-core banks are accessible only on the node-base CPU or something like that. But we can't send IPIs with IRQs off, thus the warning.

Yazen, I'm thinking this whole get_block_address() dancing can be simplified by creating a data structure containing *all* MCi_MISC* addresses once during boot and then using it instead of reading it from the MSRs each time.

Also - and I'm wishfully thinking simple here - that structure could be global as I'd venture a guess that all MISC addresses are the same system-wide and not node-specific.

But I might be missing something here.

Hmmm.

Signature · 4 lines
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.
author
#3 lore raw
In reply to Thomas Gleixner

Hi, please ignore the warning: it happens before and after the regressing commit (I didn't actually mean to include it on the log I gave here, whoops). As for how I determined nouveau is getting assigned the same IRQ vector as another device, I checked using /sys/kernel/debug/irq. Additionally; when nouveau does initialize properly after resume (e.g. after reverting this patch) I see it get assigned a seperate vector from the other devices.

Quoted 14 lines · Thomas Gleixner
Quoted 5 lines · Lyude Paul

JFYI: I confirmed this patch is definitely broken. I'm seeing nouveau get assigned the same MSI vector as another device on the system, which would explain why interrupts suddenly stop working. I'll keep looking into it further
tomorrow.

How did you determine that it is the same MSI vector as another device?

Thanks,

tglx

Signature · 2 lines
Cheers,
	Lyude Paul
#4 lore raw
Quoted 10 lines

-----Original Message-----
From: [email protected] [mailto:linux-kernel- [email protected]] On Behalf Of Lyude Paul
Sent: Wednesday, January 24, 2018 12:49 PM
To: Thomas Gleixner <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; linux- [email protected]
Subject: Re: "irq/matrix: Spread interrupts on allocation" breaks nouveau in mainline kernel

Quoted 7 lines

Hi, please ignore the warning: it happens before and after the regressing commit (I didn't actually mean to include it on the log I gave here, whoops). As for how I determined nouveau is getting assigned the same IRQ vector as another device, I checked using /sys/kernel/debug/irq. Additionally; when nouveau does initialize properly after resume (e.g. after reverting this patch) I see it get assigned a seperate vector from the other devices.

+Boris. This thread seems to have split.

Lyude,
Does the warning show on mainline or does it only show when bisecting?

Sorry, I'm not sure what you mean by "it happens before and after the regressing commit".

Boris,
In any case, I like your idea on saving the block addresses. I can look into this.

Thanks,
Yazen

author
#5 lore raw
Quoted 20 lines · Yazen
Quoted 11 lines

-----Original Message-----
From: [email protected] [mailto:linux-kernel- [email protected]] On Behalf Of Lyude Paul
Sent: Wednesday, January 24, 2018 12:49 PM
To: Thomas Gleixner <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; linux- [email protected]
Subject: Re: "irq/matrix: Spread interrupts on allocation" breaks nouveau in
mainline kernel

Quoted 8 lines

Hi, please ignore the warning: it happens before and after the regressing commit (I didn't actually mean to include it on the log I gave here, whoops).
As for how I determined nouveau is getting assigned the same IRQ vector as another device, I checked using /sys/kernel/debug/irq. Additionally; when nouveau does initialize properly after resume (e.g. after reverting this patch) I see it get assigned a seperate vector from the other devices.

Quoted 7 lines

+Boris. This thread seems to have split.

Lyude,
Does the warning show on mainline or does it only show when bisecting?

Sorry, I'm not sure what you mean by "it happens before and after the regressing commit".

Sorry about that! Let me clarify a little bit: this is a problem that shows up on mainline. Normally when we suspend the GPU in nouveau, we free the IRQs it's using before going into suspend (drivers/gpu/drm/nouveau/nvkm/subdev/pci/base.c:88), then reserve IRQs again on resume (drivers/gpu/drm/nouveau/nvkm/subdev/pci/base.c:134). Since this patch got pushed to mainline, the IRQ we get from request_irq() ends up having the same MSI vector as another device on the system:

Before suspend, nouveau's IRQ allocation:

    handler:  handle_edge_irq
    device:   0000:22:00.0
    status:   0x00000000
    istate:   0x00000000
    ddepth:   0
    wdepth:   0
    dstate:   0x01400200
                IRQD_ACTIVATED
                IRQD_IRQ_STARTED
                IRQD_SINGLE_TARGET
    node:     0
    affinity: 0-7
    effectiv: 1
    pending:  
    domain:  PCI-MSI-2
     hwirq:   0x1100000
     chip:    PCI-MSI
      flags:   0x10
                 IRQCHIP_SKIP_SET_WAKE
     parent:
        domain:  VECTOR
         hwirq:   0x2f
         chip:    APIC
          flags:   0x0
         Vector:    35
         Target:     1

    After resume and allocating the interrupt for nouveau again, we get a message
    from the kernel saying: 

    [  217.150787] do_IRQ: 1.35 No irq handler for vector

    As well, nouveau ends up getting no interrupts from the card and as a result
    fails to come back up:

    [  219.153049] nouveau 0000:22:00.0: DRM: EVO timeout
    [  220.226254] r8169 0000:1e:00.0 enp30s0: link up
    [  221.153054] nouveau 0000:22:00.0: DRM: base-0: timeout
    [  223.153528] nouveau 0000:22:00.0: DRM: base-0: timeout

    If we look through all of the other IRQ allocations, we'll find that now two
    devices have the MSI vector 35:

    nouveau:
    handler:  handle_edge_irq
    device:   0000:22:00.0
    status:   0x00000000
    istate:   0x00000000
    ddepth:   0
    wdepth:   0
    dstate:   0x01400200
                IRQD_ACTIVATED
                IRQD_IRQ_STARTED
                IRQD_SINGLE_TARGET
    node:     0
    affinity: 0-7
    effectiv: 1
    pending:  
    domain:  PCI-MSI-2
     hwirq:   0x1100000
     chip:    PCI-MSI
      flags:   0x10
                 IRQCHIP_SKIP_SET_WAKE
     parent:
        domain:  VECTOR
         hwirq:   0x2f
         chip:    APIC
          flags:   0x0
         Vector:    35
         Target:     1

    and the PCI bridge (00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD]
    Family 17h (Models 00h-0fh) PCIe GPP Bridge):

        handler:  handle_edge_irq
        device:   0000:00:01.3
        status:   0x00000000
        istate:   0x00000000
        ddepth:   0
        wdepth:   0
        dstate:   0x03400200
                    IRQD_ACTIVATED
                    IRQD_IRQ_STARTED
                    IRQD_SINGLE_TARGET
        node:     0
        affinity: 0-7
        effectiv: 0
        pending:  
        domain:  PCI-MSI-2
         hwirq:   0x5800
         chip:    PCI-MSI
          flags:   0x10
                     IRQCHIP_SKIP_SET_WAKE
         parent:
            domain:  VECTOR
             hwirq:   0x19
             chip:    APIC
              flags:   0x0
             Vector:    35
             Target:     0

    hope this helps clarify, I will keep looking at this from my end as well
    > 
Quoted 7 lines

Boris,
In any case, I like your idea on saving the block addresses. I can look into this.

Thanks,
Yazen

Signature · 2 lines
Cheers,
	Lyude Paul
author
#6 lore raw
Quoted 144 lines · Lyude Paul
Quoted 24 lines · Yazen
Quoted 12 lines

-----Original Message-----
From: [email protected] [mailto:linux-kernel- [email protected]] On Behalf Of Lyude Paul
Sent: Wednesday, January 24, 2018 12:49 PM
To: Thomas Gleixner <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; linux- [email protected]
Subject: Re: "irq/matrix: Spread interrupts on allocation" breaks nouveau
in
mainline kernel

Quoted 11 lines

Hi, please ignore the warning: it happens before and after the regressing
commit (I didn't actually mean to include it on the log I gave here, whoops).
As for how I determined nouveau is getting assigned the same IRQ vector as
another device, I checked using /sys/kernel/debug/irq. Additionally; when
nouveau does initialize properly after resume (e.g. after reverting this patch) I see it get assigned a seperate vector from the other devices.

Quoted 7 lines

+Boris. This thread seems to have split.

Lyude,
Does the warning show on mainline or does it only show when bisecting?

Sorry, I'm not sure what you mean by "it happens before and after the regressing commit".

Sorry about that! Let me clarify a little bit: this is a problem that shows up
on mainline. Normally when we suspend the GPU in nouveau, we free the IRQs it's using before going into suspend (drivers/gpu/drm/nouveau/nvkm/subdev/pci/base.c:88), then reserve IRQs again on resume (drivers/gpu/drm/nouveau/nvkm/subdev/pci/base.c:134). Since this patch got pushed to mainline, the IRQ we get from request_irq() ends up having
the same MSI vector as another device on the system:

Before suspend, nouveau's IRQ allocation:

    handler:  handle_edge_irq
    device:   0000:22:00.0
    status:   0x00000000
    istate:   0x00000000
    ddepth:   0
    wdepth:   0
    dstate:   0x01400200
                IRQD_ACTIVATED
                IRQD_IRQ_STARTED
                IRQD_SINGLE_TARGET
    node:     0
    affinity: 0-7
    effectiv: 1
    pending:  
    domain:  PCI-MSI-2
     hwirq:   0x1100000
     chip:    PCI-MSI
      flags:   0x10
                 IRQCHIP_SKIP_SET_WAKE
     parent:
        domain:  VECTOR
         hwirq:   0x2f
         chip:    APIC
          flags:   0x0
         Vector:    35
         Target:     1

    After resume and allocating the interrupt for nouveau again, we get a

message
from the kernel saying:

[ 217.150787] do_IRQ: 1.35 No irq handler for vector

As well, nouveau ends up getting no interrupts from the card and as a
result

    fails to come back up:

    [  219.153049] nouveau 0000:22:00.0: DRM: EVO timeout
    [  220.226254] r8169 0000:1e:00.0 enp30s0: link up
    [  221.153054] nouveau 0000:22:00.0: DRM: base-0: timeout
    [  223.153528] nouveau 0000:22:00.0: DRM: base-0: timeout

    If we look through all of the other IRQ allocations, we'll find that now
two
    devices have the MSI vector 35:

    nouveau:
    handler:  handle_edge_irq
    device:   0000:22:00.0
    status:   0x00000000
    istate:   0x00000000
    ddepth:   0
    wdepth:   0
    dstate:   0x01400200
                IRQD_ACTIVATED
                IRQD_IRQ_STARTED
                IRQD_SINGLE_TARGET
    node:     0
    affinity: 0-7
    effectiv: 1
    pending:  
    domain:  PCI-MSI-2
     hwirq:   0x1100000
     chip:    PCI-MSI
      flags:   0x10
                 IRQCHIP_SKIP_SET_WAKE
     parent:
        domain:  VECTOR
         hwirq:   0x2f
         chip:    APIC
          flags:   0x0
         Vector:    35
         Target:     1

    and the PCI bridge (00:01.3 PCI bridge: Advanced Micro Devices, Inc.
[AMD]
    Family 17h (Models 00h-0fh) PCIe GPP Bridge):

        handler:  handle_edge_irq
        device:   0000:00:01.3
        status:   0x00000000
        istate:   0x00000000
        ddepth:   0
        wdepth:   0
        dstate:   0x03400200
                    IRQD_ACTIVATED
                    IRQD_IRQ_STARTED
                    IRQD_SINGLE_TARGET
        node:     0
        affinity: 0-7
        effectiv: 0
        pending:  
        domain:  PCI-MSI-2
         hwirq:   0x5800
         chip:    PCI-MSI
          flags:   0x10
                     IRQCHIP_SKIP_SET_WAKE
         parent:
            domain:  VECTOR
Quoted 8 lines
             hwirq:   0x19
             chip:    APIC
              flags:   0x0
             Vector:    35
             Target:     0

    hope this helps clarify, I will keep looking at this from my end as well
    > 

Almost forgot to mention: I came across this patch because reverting it locally on the mainline kernel makes request_irq() behave normally (it doesn't attempt to allocate the same vector twice anymore) and nouveau starts doing suspend/resume correctly again

Quoted 8 lines
Quoted 8 lines

Boris,
In any case, I like your idea on saving the block addresses. I can look into
this.

Thanks,
Yazen

Signature · 2 lines
Cheers,
	Lyude Paul
#7 lore raw
Quoted 4 lines · Lyude Paul

Almost forgot to mention: I came across this patch because reverting it locally on the mainline kernel makes request_irq() behave normally (it doesn't attempt to allocate the same vector twice anymore) and nouveau starts doing suspend/resume correctly again

Ah, someone already hunted down my resume woes. Yup, reverting $subject fixed up my sole reason to use nouveau (to be able to _resume_ as well as suspend:).  If anyone needs a lab rat, just holler.

-Mike

#9 lore raw
In reply to Lyude Paul
Quoted 7 lines · Lyude Paul

Sorry about that! Let me clarify a little bit: this is a problem that shows up on mainline. Normally when we suspend the GPU in nouveau, we free the IRQs it's using before going into suspend (drivers/gpu/drm/nouveau/nvkm/subdev/pci/base.c:88), then reserve IRQs again on resume (drivers/gpu/drm/nouveau/nvkm/subdev/pci/base.c:134). Since this patch got pushed to mainline, the IRQ we get from request_irq() ends up having the same MSI vector as another device on the system:

It's not the same.

Quoted 8 lines
    nouveau:
     parent:
        domain:  VECTOR
         hwirq:   0x2f
         chip:    APIC
          flags:   0x0
         Vector:    35
         Target:     1

Vector 35 on CPU1

Quoted 4 lines

After resume and allocating the interrupt for nouveau again, we get a message from the kernel saying:

[ 217.150787] do_IRQ: 1.35 No irq handler for vector

That's because there is a pending irq on the old vector for unknown reasons.

Quoted 11 lines
    As well, nouveau ends up getting no interrupts from the card and as a result
    fails to come back up:

    [  219.153049] nouveau 0000:22:00.0: DRM: EVO timeout
    [  220.226254] r8169 0000:1e:00.0 enp30s0: link up
    [  221.153054] nouveau 0000:22:00.0: DRM: base-0: timeout
    [  223.153528] nouveau 0000:22:00.0: DRM: base-0: timeout

    If we look through all of the other IRQ allocations, we'll find that now two
    devices have the MSI vector 35:
Quoted 8 lines
    nouveau:
     parent:
        domain:  VECTOR
         hwirq:   0x2f
         chip:    APIC
          flags:   0x0
         Vector:    35
         Target:     1

Vector 35 on CPU1

Quoted 10 lines

and the PCI bridge (00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge):

parent:
domain: VECTOR
hwirq: 0x19
chip: APIC
flags: 0x0
Vector: 35
Target: 0

Vector 35 on CPU0. Same vector but different CPUs. So it's NOT the same thing.

The real issue is something completely different and the revert of this patch merily papers over the underlying problem. I'm pretty sure that you can trigger this even with the revert in place. Do the following before suspend:

echo 2 >/proc/irq/$NOUVEAUIRQ/smp_affinity_list

Then do suspend/resume and you should end up with the same situation.

I can't tell from your dmesg, but I'm pretty confident that

Quoted 1 line
    [  217.150787] do_IRQ: 1.35 No irq handler for vector

happens _before_ the nouveau driver requests the irq again. Can please you add some printk to the code in question to verify that?

Thanks,

tglx

author
#10 lore raw

I think you are right, apologies. Glad to know this isn't a regression in the IRQ handling code :). It looks like our nouveau problems are probably coming from the fact that we don't just leave IRQs setup through suspend/resume which as far as I can tell, is probably not the correct thing to do.

Going to get some patches onto the mailing list for this, thanks for the help!

Quoted 90 lines · Thomas Gleixner
Quoted 9 lines · Lyude Paul

Sorry about that! Let me clarify a little bit: this is a problem that shows up
on mainline. Normally when we suspend the GPU in nouveau, we free the IRQs it's using before going into suspend (drivers/gpu/drm/nouveau/nvkm/subdev/pci/base.c:88), then reserve IRQs again on resume (drivers/gpu/drm/nouveau/nvkm/subdev/pci/base.c:134). Since this patch got pushed to mainline, the IRQ we get from request_irq() ends up having
the same MSI vector as another device on the system:

It's not the same.

Quoted 8 lines
    nouveau:
     parent:
        domain:  VECTOR
         hwirq:   0x2f
         chip:    APIC
          flags:   0x0
         Vector:    35
         Target:     1

Vector 35 on CPU1

Quoted 5 lines

After resume and allocating the interrupt for nouveau again, we get a
message
from the kernel saying:

[ 217.150787] do_IRQ: 1.35 No irq handler for vector

That's because there is a pending irq on the old vector for unknown reasons.

Quoted 13 lines

As well, nouveau ends up getting no interrupts from the card and as a
result

    fails to come back up:

    [  219.153049] nouveau 0000:22:00.0: DRM: EVO timeout
    [  220.226254] r8169 0000:1e:00.0 enp30s0: link up
    [  221.153054] nouveau 0000:22:00.0: DRM: base-0: timeout
    [  223.153528] nouveau 0000:22:00.0: DRM: base-0: timeout

    If we look through all of the other IRQ allocations, we'll find that now
two
    devices have the MSI vector 35:
Quoted 8 lines
    nouveau:
     parent:
        domain:  VECTOR
         hwirq:   0x2f
         chip:    APIC
          flags:   0x0
         Vector:    35
         Target:     1

Vector 35 on CPU1

Quoted 11 lines

and the PCI bridge (00:01.3 PCI bridge: Advanced Micro Devices, Inc.
[AMD]

    Family 17h (Models 00h-0fh) PCIe GPP Bridge):

         parent:
            domain:  VECTOR
             hwirq:   0x19
             chip:    APIC
              flags:   0x0
             Vector:    35
             Target:     0

Vector 35 on CPU0. Same vector but different CPUs. So it's NOT the same thing.

The real issue is something completely different and the revert of this patch merily papers over the underlying problem. I'm pretty sure that you can trigger this even with the revert in place. Do the following before suspend:

echo 2 >/proc/irq/$NOUVEAUIRQ/smp_affinity_list

Then do suspend/resume and you should end up with the same situation.

I can't tell from your dmesg, but I'm pretty confident that

Quoted 1 line
    [  217.150787] do_IRQ: 1.35 No irq handler for vector

happens _before_ the nouveau driver requests the irq again. Can please you add some printk to the code in question to verify that?

Thanks,

tglx

author
#8 lore raw
In reply to Mike Galbraith

Will cc you with some patches in a bit, think I have this problem figured out :)

Quoted 12 lines · Mike Galbraith
Quoted 5 lines · Lyude Paul

Almost forgot to mention: I came across this patch because reverting it locally on the mainline kernel makes request_irq() behave normally (it doesn't
attempt to allocate the same vector twice anymore) and nouveau starts doing suspend/resume correctly again

Ah, someone already hunted down my resume woes. Yup, reverting $subject fixed up my sole reason to use nouveau (to be able to _resume_ as well as suspend:). If anyone needs a lab rat, just holler.

-Mike

#11 lore raw
In reply to Lyude Paul
Quoted 4 lines · Lyude Paul

I think you are right, apologies. Glad to know this isn't a regression in the IRQ handling code :). It looks like our nouveau problems are probably coming from the fact that we don't just leave IRQs setup through suspend/resume which as far as I can tell, is probably not the correct thing to do.

If you tear down the interrupt, then you have to make sure that it's completely masked and disabled on the device side (including MSI).

Thanks,

tglx

author
#12 lore raw
Quoted 11 lines · Thomas Gleixner
Quoted 6 lines · Lyude Paul

I think you are right, apologies. Glad to know this isn't a regression in the
IRQ handling code :). It looks like our nouveau problems are probably coming from the fact that we don't just leave IRQs setup through suspend/resume which
as far as I can tell, is probably not the correct thing to do.

If you tear down the interrupt, then you have to make sure that it's completely masked and disabled on the device side (including MSI).

Does this only need to be done if we handle irq_request()/irq_free() ourselves, or can we skip some of these steps if we let the kernel handle disabling/enabling IRQs during s/r?

Quoted 4 lines

Thanks,

tglx

#13 lore raw
Quoted 10 lines · Lyude Paul
Quoted 9 lines · Thomas Gleixner
Quoted 6 lines · Lyude Paul

I think you are right, apologies. Glad to know this isn't a regression in the
IRQ handling code :). It looks like our nouveau problems are probably coming from the fact that we don't just leave IRQs setup through suspend/resume which
as far as I can tell, is probably not the correct thing to do.

Quoted 5 lines
Quoted 2 lines

If you tear down the interrupt, then you have to make sure that it's completely masked and disabled on the device side (including MSI).

Does this only need to be done if we handle irq_request()/irq_free() ourselves, or can we skip some of these steps if we let the kernel handle disabling/enabling IRQs during s/r?

If you do not free the interrupt on suspend, then the core does the right thing. Though you should not inflict an interrupt storm in that case either :)

Thanks,

tglx