Device hang when offlining a CPU due to IRQ misrouting
Hi there,
I'm seeing a driver hang with 2.6.22-rc3 while being slightly stupid about offlining CPUs. I suspect that this problem extends beyond a particular machine, as I've been able to replicate it with an IBM x3650 and an IBM x3755. This is what I'm doing:
1) I tie an IRQ to a particular CPU via /proc/irq/XXX/smp_affinity (IRQ 4341 is the network card and we're picking on CPU1 in this example):
echo 2 > /proc/irq/4341/smp_affinity
2) I then take CPU1 offline:
echo 0 > /sys/devices/system/cpu/cpu1/online
3) The kernel prints this:
[ 1101.968040] Breaking affinity for irq 4341 [ 1102.074019] CPU 1 is now offline [ 1102.081593] lockdep: not fixing up alternatives. [ 1112.886919] nfs: server 9.47.66.169 not responding, still trying
After step 2 the system never sees interrupts from the network card and remains hung like that until CPU1 is brought back up. It looks as though the kernel is trying to reroute the IRQ (or so I'm assuming from the "Breaking affinity" message), but this doesn't ever happen, so the the kernel stops seeing interrupts from the device.
Granted, one should not be offlining the CPU that is currently designated to handle an IRQ, but I suspect that the kernel ought at a minimum to reject the offlining or route the IRQ to any online CPU instead of screwing things up.
There exists a similar scenario. Set the IRQ affinity to a bunch of CPUs, watch /proc/interrupts to see which CPU is actually servicing the interrupts, then offline that CPU. The kernel does not reroute the IRQ to any of the other CPUs and the device also hangs.
The furthest that I've dug is that it works on 2.6.17 and is broken in 2.6.22-rc3 and 2.6.21. Will git-bisect further, but I wanted to know if anyone else has seen this sort of problem. afaik, this seems to happen with both IOAPIC and MSI interrupts, possibly more.
--D
Quoted 26 lines · Darrick J. Wong
Hi there,
I'm seeing a driver hang with 2.6.22-rc3 while being slightly stupid about offlining CPUs. I suspect that this problem extends beyond a particular machine, as I've been able to replicate it with an IBM x3650 and an IBM x3755. This is what I'm doing:
1) I tie an IRQ to a particular CPU via
/proc/irq/XXX/smp_affinity(IRQ 4341 is the network card and we're picking on CPU1 in this example):
echo 2 >/proc/irq/4341/smp_affinity2) I then take CPU1 offline:
echo 0 >/sys/devices/system/cpu/cpu1/online3) The kernel prints this:
[ 1101.968040] Breaking affinity for irq 4341 [ 1102.074019] CPU 1 is now offline [ 1102.081593] lockdep: not fixing up alternatives. [ 1112.886919] nfs: server 9.47.66.169 not responding, still tryingAfter step 2 the system never sees interrupts from the network card and remains hung like that until CPU1 is brought back up. It looks as though the kernel is trying to reroute the IRQ (or so I'm assuming from the "Breaking affinity" message), but this doesn't ever happen, so the the kernel stops seeing interrupts from the device.
Quoted 4 lines
Granted, one should not be offlining the CPU that is currently designated to handle an IRQ, but I suspect that the kernel ought at a minimum to reject the offlining or route the IRQ to any online CPU instead of screwing things up.
I agree.
Quoted 9 lines
There exists a similar scenario. Set the IRQ affinity to a bunch of CPUs, watch
/proc/interruptsto see which CPU is actually servicing the interrupts, then offline that CPU. The kernel does not reroute the IRQ to any of the other CPUs and the device also hangs.The furthest that I've dug is that it works on 2.6.17 and is broken in 2.6.22-rc3 and 2.6.21. Will git-bisect further, but I wanted to know if anyone else has seen this sort of problem. afaik, this seems to happen with both IOAPIC and MSI interrupts, possibly more.
Thanks for the bug report. I'm chuckling because I just submitted a patch to count that whole code path as broken, based on code review. It is trying to do something that the hardware can not reliably accomplish.
Now I am surprised you were seeing this with MSI as well because the hardware should theoretically work in that case. However the irq_fixup code has enough issues that I wouldn't be surprised if it was just doing something stupid and wrong.
Eric
Quoted 10 lines · Darrick J. Wong
Hi there,
I'm seeing a driver hang with 2.6.22-rc3 while being slightly stupid about offlining CPUs. I suspect that this problem extends beyond a particular machine, as I've been able to replicate it with an IBM x3650 and an IBM x3755. This is what I'm doing:
1) I tie an IRQ to a particular CPU via
/proc/irq/XXX/smp_affinity(IRQ 4341 is the network card and we're picking on CPU1 in this example):
echo 2 >/proc/irq/4341/smp_affinity
Darrick, I see a kernel bug in this area(which is already filled with bugs, and I am looking into ways to fix them). Are you making sure that between step-1 and step-2, that interrupts actually started arriving at cpu1?
i.e., do step-1 and wait till the irq's start hitting at cpu1. At this point do step-2 and let us know if you still hit this bug?
Quoted 21 lines
2) I then take CPU1 offline:
echo 0 >/sys/devices/system/cpu/cpu1/online3) The kernel prints this:
[ 1101.968040] Breaking affinity for irq 4341 [ 1102.074019] CPU 1 is now offline [ 1102.081593] lockdep: not fixing up alternatives. [ 1112.886919] nfs: server 9.47.66.169 not responding, still tryingAfter step 2 the system never sees interrupts from the network card and remains hung like that until CPU1 is brought back up. It looks as though the kernel is trying to reroute the IRQ (or so I'm assuming from the "Breaking affinity" message), but this doesn't ever happen, so the the kernel stops seeing interrupts from the device.
Granted, one should not be offlining the CPU that is currently designated to handle an IRQ, but I suspect that the kernel ought at a minimum to reject the offlining or route the IRQ to any online CPU instead of screwing things up.
Quoted 4 lines
There exists a similar scenario. Set the IRQ affinity to a bunch of CPUs, watch
/proc/interruptsto see which CPU is actually servicing the interrupts, then offline that CPU. The kernel does not reroute the IRQ to any of the other CPUs and the device also hangs.
Is this a theory or did you observe this problem happening?
thanks,
suresh
Quoted 5 lines
The furthest that I've dug is that it works on 2.6.17 and is broken in 2.6.22-rc3 and 2.6.21. Will git-bisect further, but I wanted to know if anyone else has seen this sort of problem. afaik, this seems to happen with both IOAPIC and MSI interrupts, possibly more.
Quoted 6 lines · Suresh B
Darrick, I see a kernel bug in this
area(which is already filled with bugs, and I am looking into ways to fix them). Are you making sure that between step-1 and step-2, that interrupts actually started arriving at cpu1?i.e., do step-1 and wait till the irq's start hitting at cpu1. At this point do step-2 and let us know if you still hit this bug?
Yes, the bug only happens after CPU1 begins to receive interrupts.
Quoted 6 lines
Quoted 4 lines
There exists a similar scenario. Set the IRQ affinity to a bunch of CPUs, watch
/proc/interruptsto see which CPU is actually servicing the interrupts, then offline that CPU. The kernel does not reroute the IRQ to any of the other CPUs and the device also hangs.Is this a theory or did you observe this problem happening?
Nope, I've observed this situation too.
--D
Quoted 10 lines · Darrick J. Wong
Quoted 6 lines · Suresh B
Darrick, I see a kernel bug in this
area(which is already filled with bugs, and I am looking into ways to fix them). Are you making sure that between step-1 and step-2, that interrupts actually started arriving at cpu1?i.e., do step-1 and wait till the irq's start hitting at cpu1. At this point do step-2 and let us know if you still hit this bug?
Yes, the bug only happens after CPU1 begins to receive interrupts.
I see. Your system should have 4 or 8 logical cpu's right. So you must be using logical flat mode, right?
When this bug happens, what does /proc/irq/<irq-no>/smp_affinity show?
thanks,
suresh
Quoted 2 lines · Suresh B
I see. Your system should have 4 or 8 logical cpu's right. So you must be using logical flat mode, right?
I believe so. The system has two Xeon 5150s with an Intel 5000 chipset of some sort.
Quoted 1 line
When this bug happens, what does /proc/irq/<irq-no>/smp_affinity show?
root@elm3a188:~# cat /proc/irq/114/smp_affinity 02
--D
Quoted 11 lines · Darrick J. Wong
Quoted 2 lines · Suresh B
I see. Your system should have 4 or 8 logical cpu's right. So you must be using logical flat mode, right?
I believe so. The system has two Xeon 5150s with an Intel 5000 chipset of some sort.
Quoted 1 line
When this bug happens, what does /proc/irq/<irq-no>/smp_affinity show?
root@elm3a188:~# cat /proc/irq/114/smp_affinity 02
Ok. What this shows is that fixup_irqs() failed to move the irq properly. Ideally we should see cpu_online_map here (i.e., 0xfd).
So most likely __assign_irq_vector() failed for some reason and I am puzzled for the reason...
Does this problem happen only under certain stress or something simple, like
boot the kernel
echo 2 > /proc/irq/114/smp_affinity
wait for irq to hit the cpu1.
echo 0 > /sys/devices/system/cpu/cpu1/online
will immmd trigger this?
thanks,
suresh
Quoted 8 lines · Suresh B
Does this problem happen only under certain stress or something simple, like
boot the kernel
echo 2 >/proc/irq/114/smp_affinity
wait for irq to hit the cpu1.
echo 0 >/sys/devices/system/cpu/cpu1/onlinewill immmd trigger this?
The system is not under any stress at all.
--D
Quoted 4 lines · Darrick J. Wong
Quoted 2 lines · Suresh B
Does this problem happen only under certain stress or something simple, like
Quoted 8 lines
Quoted 6 lines
boot the kernel
echo 2 >/proc/irq/114/smp_affinity
wait for irq to hit the cpu1.
echo 0 >/sys/devices/system/cpu/cpu1/onlinewill immmd trigger this?
The system is not under any stress at all.
Can you send us your system's dmesg aswell as output of /proc/interrupts?
thanks,
suresh
Quoted 1 line · Suresh B
Can you send us your system's dmesg aswell as output of
/proc/interrupts?
http://sweaglesw.net/~djwong/docs/dmesg
http://sweaglesw.net/~djwong/docs/interrupts
--D
Quoted 6 lines · Darrick J. Wong
Quoted 1 line · Suresh B
Can you send us your system's dmesg aswell as output of
/proc/interrupts?http://sweaglesw.net/~djwong/docs/dmesg
http://sweaglesw.net/~djwong/docs/interrupts
Didn't find anything wrong in that information. Can you try this appended debug patch and see if you see this error msg in dmesg, when you hit the bug? Thanks.
arch/x86_64/kernel/io_apic.c +3 -0
@@ -720,10 +720,13 @@ static int assign_irq_vector(int irq, cpumask_t mask) {int err;unsigned long flags; +int cpu =smp_processor_id ();spin_lock_irqsave (&vector_lock, flags); err =__assign_irq_vector (irq, mask);spin_unlock_irqrestore (&vector_lock, flags); +if (err && !cpu_isset (cpu, cpu_online_map)) +printk ("assigning irq to a vector failed : %d\n" , err);return err; }
Quoted 11 lines · Suresh B
Quoted 6 lines · Darrick J. Wong
Quoted 1 line · Suresh B
Can you send us your system's dmesg aswell as output of
/proc/interrupts?http://sweaglesw.net/~djwong/docs/dmesg
http://sweaglesw.net/~djwong/docs/interruptsDidn't find anything wrong in that information. Can you try this appended debug patch and see if you see this error msg in dmesg, when you hit the bug? Thanks.
I don't see that message.
--D
Quoted 6 lines · Darrick J. Wong
Quoted 5 lines · Suresh B
Quoted 4 lines · Darrick J. Wong
Quoted 1 line · Suresh B
Can you send us your system's dmesg aswell as output of
/proc/interrupts?
Quoted 8 lines
Quoted 6 lines
Didn't find anything wrong in that information. Can you try this appended debug patch and see if you see this error msg in dmesg, when you hit the bug? Thanks.
I don't see that message.
Weird. Then the bug can only happen if for some reason, "mask = map" didn't happen in fixup_irqs(). Can you send us the disassembly of the fixup_irqs()?
thanks,
suresh
Quoted 3 lines · Suresh B
Weird. Then the bug can only happen if for some reason, "mask = map" didn't happen in
fixup_irqs(). Can you send us the disassembly of thefixup_irqs()?
Attached.
--D
(gdb) disassemble fixup_irqs
Dump of assembler code for function fixup_irqs:
0xffffffff8020bf50 <fixup_irqs+0>: push %rbp 0xffffffff8020bf51 <fixup_irqs+1>: mov %rsp,%rbp 0xffffffff8020bf54 <fixup_irqs+4>: push %r13 0xffffffff8020bf56 <fixup_irqs+6>: xor %r13d,%r13d 0xffffffff8020bf59 <fixup_irqs+9>: push %r12 0xffffffff8020bf5b <fixup_irqs+11>: push %rbx 0xffffffff8020bf5c <fixup_irqs+12>: sub $0x28,%rsp 0xffffffff8020bf60 <fixup_irqs+16>: mov %rdi,0xffffffffffffffc0(%rbp) 0xffffffff8020bf64 <fixup_irqs+20>: mov %rsi,0xffffffffffffffc8(%rbp) 0xffffffff8020bf68 <fixup_irqs+24>: jmp 0xffffffff8020bf73 <fixup_irqs+35> 0xffffffff8020bf6a <fixup_irqs+26>: inc %r13d 0xffffffff8020bf6d <fixup_irqs+29>: cmp $0x2,%r13d 0xffffffff8020bf71 <fixup_irqs+33>: je 0xffffffff8020bf6a <fixup_irqs+26> 0xffffffff8020bf73 <fixup_irqs+35>: mov %r13d,%r12d 0xffffffff8020bf76 <fixup_irqs+38>: lea 0xffffffffffffffd0(%rbp),%rbx 0xffffffff8020bf7a <fixup_irqs+42>: lea 0xffffffffffffffc0(%rbp),%rdx 0xffffffff8020bf7e <fixup_irqs+46>: shl $0x8,%r12 0xffffffff8020bf82 <fixup_irqs+50>: mov $0x80,%ecx 0xffffffff8020bf87 <fixup_irqs+55>: lea 0xffffffff805505f8(%r12),%rsi 0xffffffff8020bf8f <fixup_irqs+63>: mov %rbx,%rdi 0xffffffff8020bf92 <fixup_irqs+66>: callq 0xffffffff802fb606 <__bitmap_and> 0xffffffff8020bf97 <fixup_irqs+71>: mov %rbx,%rdi 0xffffffff8020bf9a <fixup_irqs+74>: callq 0xffffffff802fc6ad <__any_online_cpu> 0xffffffff8020bf9f <fixup_irqs+79>: add $0xffffffffffffff80,%eax 0xffffffff8020bfa2 <fixup_irqs+82>: jne 0xffffffff8020bfc5 <fixup_irqs+117> 0xffffffff8020bfa4 <fixup_irqs+84>: mov %r13d,%esi 0xffffffff8020bfa7 <fixup_irqs+87>: mov $0xffffffff804a52b0,%rdi 0xffffffff8020bfae <fixup_irqs+94>: xor %eax,%eax 0xffffffff8020bfb0 <fixup_irqs+96>: callq 0xffffffff80233d28 <printk> 0xffffffff8020bfb5 <fixup_irqs+101>: mov 0xffffffffffffffc0(%rbp),%rax 0xffffffff8020bfb9 <fixup_irqs+105>: mov %rax,0xffffffffffffffd0(%rbp) 0xffffffff8020bfbd <fixup_irqs+109>: mov 0xffffffffffffffc8(%rbp),%rax 0xffffffff8020bfc1 <fixup_irqs+113>: mov %rax,0xffffffffffffffd8(%rbp) 0xffffffff8020bfc5 <fixup_irqs+117>: mov 0xffffffff80550588(%r12),%rax 0xffffffff8020bfcd <fixup_irqs+125>: mov 0x58(%rax),%rax 0xffffffff8020bfd1 <fixup_irqs+129>: test %rax,%rax 0xffffffff8020bfd4 <fixup_irqs+132>: je 0xffffffff8020bfe5 <fixup_irqs+149> 0xffffffff8020bfd6 <fixup_irqs+134>: mov 0xffffffffffffffd0(%rbp),%rsi 0xffffffff8020bfda <fixup_irqs+138>: mov 0xffffffffffffffd8(%rbp),%rdx 0xffffffff8020bfde <fixup_irqs+142>: mov %r13d,%edi 0xffffffff8020bfe1 <fixup_irqs+145>: callq *%rax 0xffffffff8020bfe3 <fixup_irqs+147>: jmp 0xffffffff8020c013 <fixup_irqs+195> 0xffffffff8020bfe5 <fixup_irqs+149>: cmpq $0x0,0xffffffff805505a8(%r12) 0xffffffff8020bfee <fixup_irqs+158>: je 0xffffffff8020c013 <fixup_irqs+195> 0xffffffff8020bff0 <fixup_irqs+160>: mov 5181486(%rip),%eax # 0xffffffff806fd024 <warned.11720> 0xffffffff8020bff6 <fixup_irqs+166>: inc %eax 0xffffffff8020bff8 <fixup_irqs+168>: mov %eax,5181478(%rip) # 0xffffffff806fd024 <warned.11720> 0xffffffff8020bffe <fixup_irqs+174>: dec %eax 0xffffffff8020c000 <fixup_irqs+176>: jne 0xffffffff8020c013 <fixup_irqs+195> 0xffffffff8020c002 <fixup_irqs+178>: mov %r13d,%esi 0xffffffff8020c005 <fixup_irqs+181>: mov $0xffffffff804a52ce,%rdi 0xffffffff8020c00c <fixup_irqs+188>: xor %eax,%eax 0xffffffff8020c00e <fixup_irqs+190>: callq 0xffffffff80233d28 <printk> 0xffffffff8020c013 <fixup_irqs+195>: lea 0x1(%r13),%eax 0xffffffff8020c017 <fixup_irqs+199>: cmp $0x10ff,%eax 0xffffffff8020c01c <fixup_irqs+204>: jbe 0xffffffff8020bf6a <fixup_irqs+26> 0xffffffff8020c022 <fixup_irqs+210>: callq 0xffffffff8024e46e <trace_hardirqs_on>
0xffffffff8020c027 <fixup_irqs+215>: sti0xffffffff8020c028 <fixup_irqs+216>: mov $0x418958,%edi0xffffffff8020c02d <fixup_irqs+221>: callq 0xffffffff803018cf <__const_udelay>0xffffffff8020c032 <fixup_irqs+226>: cli
0xffffffff8020c033 <fixup_irqs+227>: callq 0xffffffff8024cf31 <trace_hardirqs_off> 0xffffffff8020c038 <fixup_irqs+232>: add $0x28,%rsp 0xffffffff8020c03c <fixup_irqs+236>: pop %rbx 0xffffffff8020c03d <fixup_irqs+237>: pop %r12 0xffffffff8020c03f <fixup_irqs+239>: pop %r13
0xffffffff8020c041 <fixup_irqs+241>: leaveq0xffffffff8020c042 <fixup_irqs+242>: retq
Quoted 7 lines · Darrick J. Wong
Quoted 3 lines · Suresh B
Weird. Then the bug can only happen if for some reason, "mask = map" didn't happen in
fixup_irqs(). Can you send us the disassembly of thefixup_irqs()?Attached.
hmm.. Darrick, can't find anything wrong in there.
I am very much puzzled and the main thing I am confused about is, that how come "/proc/irq/<irq#-hung>/smp_affinity" is still pointing at the old offlined cpu, while calls to set_affinity() with cpu_online_map mask in fixup_irqs() don't show any failure..
As you have the failing system, you need to do more detective work and help me out. Can you try this debug patch and send across the dmesg after the bug happens and also can you try different compiler to see if something changes..
arch/x86_64/kernel/irq.c +4 -2
@@ -152,9 +152,11 @@ void fixup_irqs(cpumask_t map)printk ("Breaking affinity for irq %i\n" , irq); mask = map; } -if (irq_desc[irq].chip->set_affinity) +if (irq_desc[irq].chip->set_affinity) { +printk ("calling set affinity for %i, with mask %lx\n" , + irq,cpus_addr (mask)[0 ]); irq_desc[irq].chip->set_affinity (irq, mask); -else if (irq_desc[irq].action && !(warned++)) + }else if (irq_desc[irq].action && !(warned++))printk ("Cannot set affinity for irq %i\n" , irq); }
Quoted 4 lines · Suresh B
As you have the failing system, you need to do more detective work and help me out. Can you try this debug patch and send across the dmesg after the bug happens and also can you try different compiler to see if something changes..
Hrm, I just updated to -rc5. Interrupts being handled by the IOAPIC don't suffer from this problem, but MSI interrupts are still affected. I added a few printks to the kernel to figure out what IRQ affinity masks were being passed around and saw this:
[ 256.298773] Breaking affinity for irq 4341 [ 256.298774] irq=4341 affinity=2 mask=d <call to set_affinity> [ 256.298787] irq=4341 affinity=d
<ethernet on irq 4341 stops working>
I'll keep digging, but at least it appears that the problem has been shrunk down to something the MSI code.
--D
Quoted 12 lines · Darrick J. Wong
Quoted 4 lines · Suresh B
As you have the failing system, you need to do more detective work and help me out. Can you try this debug patch and send across the dmesg after the bug happens and also can you try different compiler to see if something changes..
Hrm, I just updated to -rc5. Interrupts being handled by the IOAPIC don't suffer from this problem, but MSI interrupts are still affected. I added a few printks to the kernel to figure out what IRQ affinity masks were being passed around and saw this:
Quoted 5 lines
[ 256.298773] Breaking affinity for irq 4341 [ 256.298774] irq=4341 affinity=2 mask=d <call to set_affinity> [ 256.298787] irq=4341 affinity=d<ethernet on irq 4341 stops working>
And just to make sure, at this point, your MSI irq 4341 affinity (/proc/irq/4341/smp_affinity) still points to '2'?
Quoted 2 lines
I'll keep digging, but at least it appears that the problem has been shrunk down to something the MSI code.
thanks,
suresh
Quoted 6 lines · Suresh B
Quoted 3 lines
<call to
set_affinity>[ 256.298787] irq=4341 affinity=d<ethernet on irq 4341 stops working>
And just to make sure, at this point, your MSI irq 4341 affinity (
/proc/irq/4341/smp_affinity) still points to '2'?
Actually, it's 0xD. From the kernel's perspective the mask has been updated (and I even stuck a printk into set_msi_irq_affinity to verify that the writes are happening) but ... the hardware doesn't seem to reflect this. I also tried putting read_msi_msg right afterwards to compare contents, though it complained about all the MSIs _except_ for 4341. (Of course, I could just be way off on the effectiveness of that.)
--D
Quoted 9 lines · Darrick J. Wong
Quoted 6 lines · Suresh B
Quoted 3 lines
<call to
set_affinity>[ 256.298787] irq=4341 affinity=d<ethernet on irq 4341 stops working>
And just to make sure, at this point, your MSI irq 4341 affinity (
/proc/irq/4341/smp_affinity) still points to '2'?
Quoted 7 lines
Actually, it's
0xD. From the kernel's perspective the mask has been updated (and I even stuck a printk intoset_msi_irq_affinityto verify that the writes are happening) but ... the hardware doesn't seem to reflect this. I also tried puttingread_msi_msgright afterwards to compare contents, though it complained about all the MSIs _except_ for 4341. (Of course, I could just be way off on the effectiveness of that.)
The fact that MSI interrupts are having problems is odd. It is possible that we still have a bug in there somewhere but msi interrupts should be safe to migrate outside of irq context (no known hardware bugs). As we can actually synchronize with the irq source and eliminate all of the migration races.
The non-msi case requires hitting a hardware race that is rare enough you should not normally have problems.
Eric
Quoted 19 lines · Eric W. Biederman
Quoted 9 lines · Darrick J. Wong
Quoted 6 lines · Suresh B
Quoted 3 lines
<call to
set_affinity>[ 256.298787] irq=4341 affinity=d<ethernet on irq 4341 stops working>
And just to make sure, at this point, your MSI irq 4341 affinity (
/proc/irq/4341/smp_affinity) still points to '2'?Quoted 7 lines
Actually, it's
0xD. From the kernel's perspective the mask has been updated (and I even stuck a printk intoset_msi_irq_affinityto verify that the writes are happening) but ... the hardware doesn't seem to reflect this. I also tried puttingread_msi_msgright afterwards to compare contents, though it complained about all the MSIs _except_ for 4341. (Of course, I could just be way off on the effectiveness of that.)
Quoted 8 lines
The fact that MSI interrupts are having problems is odd. It is possible that we still have a bug in there somewhere but msi interrupts should be safe to migrate outside of irq context (no known hardware bugs). As we can actually synchronize with the irq source and eliminate all of the migration races.
The non-msi case requires hitting a hardware race that is rare enough you should not normally have problems.
Yep. But Darrick's seems to say, problem happens consistently.
Anyhow, Darrick there is a general bug in this area, can you try this and see if it helps?
arch/x86_64/kernel/irq.c +19 -1
@@ -144,17 +144,35 @@ void fixup_irqs(cpumask_t map)for (irq =0 ; irq <NR_IRQS ; irq++) {cpumask_t mask; +int break_affinity =0 ; +int set_affinity =1 ; +if (irq ==2 )continue ; +/* irq's are disabled at this point */ +spin_lock (&irq_desc[irq].lock); +cpus_and (mask, irq_desc[irq].affinity, map);if (any_online_cpu (mask) ==NR_CPUS ) { -printk ("Breaking affinity for irq %i\n" , irq); + break_affinity =1 ; mask = map; } + + irq_desc[irq].chip->mask (irq); +if (irq_desc[irq].chip->set_affinity) irq_desc[irq].chip->set_affinity (irq, mask);else if (irq_desc[irq].action && !(warned++)) + set_affinity =0 ; + + irq_desc[irq].chip->unmask (irq); + +spin_unlock (&irq_desc[irq].lock); + +if (break_affinity && set_affinity) +printk ("Broke affinity for irq %i\n" , irq); +else if (!set_affinity)printk ("Cannot set affinity for irq %i\n" , irq); }
Quoted 26 lines · Suresh B
Quoted 25 lines · Eric W. Biederman
Quoted 9 lines · Darrick J. Wong
Quoted 6 lines · Suresh B
Quoted 3 lines
<call to
set_affinity>[ 256.298787] irq=4341 affinity=d<ethernet on irq 4341 stops working>
And just to make sure, at this point, your MSI irq 4341 affinity (
/proc/irq/4341/smp_affinity) still points to '2'?Quoted 7 lines
Actually, it's
0xD. From the kernel's perspective the mask has been updated (and I even stuck a printk intoset_msi_irq_affinityto verify that the writes are happening) but ... the hardware doesn't seem to reflect this. I also tried puttingread_msi_msgright afterwards to compare contents, though it complained about all the MSIs _except_ for 4341. (Of course, I could just be way off on the effectiveness of that.)The fact that MSI interrupts are having problems is odd. It is possible that we still have a bug in there somewhere but msi interrupts should be safe to migrate outside of irq context (no known hardware bugs). As we can actually synchronize with the irq source and eliminate all of the migration races.
Quoted 7 lines
Quoted 2 lines
The non-msi case requires hitting a hardware race that is rare enough you should not normally have problems.
Yep. But Darrick's seems to say, problem happens consistently.
Anyhow, Darrick there is a general bug in this area, can you try this and see if it helps?
There are several general bugs in this area. But yes your patch should help things, especially for MSI where masking the irq before migration is required. Adding locking the proper locking and masking should make things quite a bit more how set_affinity is expected to be called.
I just gave up on fixing these things because we can't eliminate the races, so the real problem is the existence of this code path with it's unsupportable semantics in the first place.
Quoted 13 lines
arch/x86_64/kernel/irq.c +3
-0@@ -144,17 +144,35 @@ void fixup_irqs(cpumask_t map)for (irq =0 ; irq <NR_IRQS ; irq++) {cpumask_t mask; +int break_affinity =0 ; +int set_affinity =1 ; +if (irq ==2 )continue ;
arch/x86_64/kernel/irq.c void fixup_irqs(cpumask_t map) · line 160
Quoted 10 lines
arch/x86_64/kernel/irq.c +4
-1+/* irq's are disabled at this point */ +spin_lock (&irq_desc[irq].lock); +cpus_and (mask, irq_desc[irq].affinity, map);if (any_online_cpu (mask) ==NR_CPUS ) { -printk ("Breaking affinity for irq %i\n" , irq); + break_affinity =1 ; mask = map; }
We should really express the "any_online_cpu(mask) == NR_CPUS" test as:
"cpus_empty(mask)" it would be much clearer.
Further we should skip the migration if "cpus_equal(mask, irq_desc[irq].affinity)" or "!irq_has_action(irq)" because no one has called request_irq.
Quoted 17 lines
arch/x86_64/kernel/irq.c +11
-0+ irq_desc[irq].chip->mask (irq); +if (irq_desc[irq].chip->set_affinity) irq_desc[irq].chip->set_affinity (irq, mask);else if (irq_desc[irq].action && !(warned++)) + set_affinity =0 ; + + irq_desc[irq].chip->unmask (irq); + +spin_unlock (&irq_desc[irq].lock); + +if (break_affinity && set_affinity) +printk ("Broke affinity for irq %i\n" , irq); +else if (!set_affinity)printk ("Cannot set affinity for irq %i\n" , irq); }
Quoted 2 lines · Suresh B
Anyhow, Darrick there is a general bug in this area, can you try this and see if it helps?
Er... that instantly locked up the system.
--D
Quoted 5 lines · Darrick J. Wong
Quoted 2 lines · Suresh B
Anyhow, Darrick there is a general bug in this area, can you try this and see if it helps?
Er... that instantly locked up the system.
hmm.. Please try this instead. This is intended only for debug. Based on your test results, we can comeup with a more decent fix.
arch/x86_64/kernel/irq.c +21 -1
@@ -144,17 +144,37 @@ void fixup_irqs(cpumask_t map)for (irq =0 ; irq <NR_IRQS ; irq++) {cpumask_t mask; +int break_affinity =0 ; +int set_affinity =1 ; +if (irq ==2 )continue ; +/* irq's are disabled at this point */ +spin_lock (&irq_desc[irq].lock); +cpus_and (mask, irq_desc[irq].affinity, map);if (any_online_cpu (mask) ==NR_CPUS ) { -printk ("Breaking affinity for irq %i\n" , irq); + break_affinity =1 ; mask = map; } + +if (irq_desc[irq].chip->mask) + irq_desc[irq].chip->mask (irq); +if (irq_desc[irq].chip->set_affinity) irq_desc[irq].chip->set_affinity (irq, mask);else if (irq_desc[irq].action && !(warned++)) + set_affinity =0 ; + +if (irq_desc[irq].chip->unmask) + irq_desc[irq].chip->unmask (irq); + +spin_unlock (&irq_desc[irq].lock); + +if (break_affinity && set_affinity) +printk ("Broke affinity for irq %i\n" , irq); +else if (!set_affinity)printk ("Cannot set affinity for irq %i\n" , irq); }
Quoted 2 lines · Suresh B
hmm.. Please try this instead. This is intended only for debug. Based on your test results, we can comeup with a more decent fix.
This fixes the problem! Hurrah!
--D
Quoted 2 lines · Darrick J. Wong
This fixes the problem! Hurrah!
Great! Andrew, please include the appended patch in -mm.
----
Subject: [patch] x86_64, irq: use mask/unmask and proper locking in fixup_irqs
From: Suresh Siddha <[email protected]>
Force irq migration path during cpu offline, is not using proper locks and irq_chip mask/unmask routines. This will result in some races(especially the device generating the interrupt can see some inconsistent state, resulting in issues like stuck irq,..).
Appended patch fixes the issue by taking proper lock and encapsulating irq_chip set_affinity() with a mask() before and an unmask() after.
This fixes a MSI irq stuck issue reported by Darrick Wong.
There are several more general bugs in this area(irq migration in the process context). For example,
1. Possibility of missing edge triggered irq.
2. Reliable method of migrating level triggered irq in the process context.
We plan to look and close these in the near future.
---
arch/x86_64/kernel/irq.c +27 -3
@@ -144,17 +144,41 @@ void fixup_irqs(cpumask_t map)for (irq =0 ; irq <NR_IRQS ; irq++) {cpumask_t mask; +int break_affinity =0 ; +int set_affinity =1 ; +if (irq ==2 )continue ; +/* interrupt's are disabled at this point */ +spin_lock (&irq_desc[irq].lock); + +if (!irq_has_action (irq) || +cpus_equal (irq_desc[irq].affinity, map)) { +spin_unlock (&irq_desc[irq].lock); +continue ; + } +cpus_and (mask, irq_desc[irq].affinity, map); -if (any_online_cpu (mask) ==NR_CPUS ) { -printk ("Breaking affinity for irq %i\n" , irq); +if (cpus_empty (mask)) { + break_affinity =1 ; mask = map; } + + irq_desc[irq].chip->mask (irq); +if (irq_desc[irq].chip->set_affinity) irq_desc[irq].chip->set_affinity (irq, mask); -else if (irq_desc[irq].action && !(warned++)) +else if (!(warned++)) + set_affinity =0 ; + + irq_desc[irq].chip->unmask (irq); + +spin_unlock (&irq_desc[irq].lock); + +if (break_affinity && set_affinity) +printk ("Broke affinity for irq %i\n" , irq); +else if (!set_affinity)printk ("Cannot set affinity for irq %i\n" , irq); }
Quoted 21 lines · Suresh B
Quoted 2 lines · Darrick J. Wong
This fixes the problem! Hurrah!
Great! Andrew, please include the appended patch in -mm.
----
Subject: [patch]x86_64, irq: use mask/unmask and proper locking infixup_irqs
From: Suresh Siddha <[email protected]>Force irq migration path during cpu offline, is not using proper locks and
irq_chipmask/unmask routines. This will result in some races(especially the device generating the interrupt can see some inconsistent state, resulting in issues like stuck irq,..).Appended patch fixes the issue by taking proper lock and encapsulating
irq_chipset_affinity()with amask()before and anunmask()after.This fixes a MSI irq stuck issue reported by Darrick Wong.
Quoted 7 lines
There are several more general bugs in this area(irq migration in the process context). For example,
1. Possibility of missing edge triggered irq.
2. Reliable method of migrating level triggered irq in the process context.We plan to look and close these in the near future.
This patch breaks hibernation on my Turion 64 X2 - based testbox (HPC nx6325).
_cpu_down() just hangs as though there were a deadlock in there, 100% of the time.
Greetings,
Rafael
Signature · 1 line
"Premature optimization is the root of all evil." - Donald Knuth
Quoted 28 lines · On Sun, 24 Jun 2007 01:54:52 +0200 "Rafael J. Wysocki" <[email protected]> wrote:
Quoted 27 lines · Suresh B
Quoted 2 lines · Darrick J. Wong
This fixes the problem! Hurrah!
Great! Andrew, please include the appended patch in -mm.
----
Subject: [patch]x86_64, irq: use mask/unmask and proper locking infixup_irqs
From: Suresh Siddha <[email protected]>Force irq migration path during cpu offline, is not using proper locks and
irq_chipmask/unmask routines. This will result in some races(especially the device generating the interrupt can see some inconsistent state, resulting in issues like stuck irq,..).Appended patch fixes the issue by taking proper lock and encapsulating
irq_chipset_affinity()with amask()before and anunmask()after.This fixes a MSI irq stuck issue reported by Darrick Wong.
There are several more general bugs in this area(irq migration in the process context). For example,
1. Possibility of missing edge triggered irq.
2. Reliable method of migrating level triggered irq in the process context.
Quoted 7 lines
Quoted 1 line
We plan to look and close these in the near future.
This patch breaks hibernation on my Turion 64 X2 - based testbox (HPC nx6325).
_cpu_down()just hangs as though there were a deadlock in there, 100% of the time.
Thanks, I dropped it.
Quoted 33 lines · Andrew Morton
Quoted 9 lines · On Sun, 24 Jun 2007 01:54:52 +0200 "Rafael J. Wysocki" <[email protected]> wrote:
Quoted 8 lines · Suresh B
Quoted 2 lines · Darrick J. Wong
This fixes the problem! Hurrah!
Great! Andrew, please include the appended patch in -mm.
----
Subject: [patch]x86_64, irq: use mask/unmask and proper locking in
fixup_irqsQuoted 21 lines
Quoted 20 lines
From: Suresh Siddha <[email protected]>
Force irq migration path during cpu offline, is not using proper locks and
irq_chipmask/unmask routines. This will result in some races(especially the device generating the interrupt can see some inconsistent state, resulting in issues like stuck irq,..).Appended patch fixes the issue by taking proper lock and encapsulating
irq_chipset_affinity()with amask()before and anunmask()after.This fixes a MSI irq stuck issue reported by Darrick Wong.
There are several more general bugs in this area(irq migration in the process context). For example,
1. Possibility of missing edge triggered irq.
2. Reliable method of migrating level triggered irq in the process context.We plan to look and close these in the near future.
Quoted 7 lines
Quoted 5 lines
This patch breaks hibernation on my Turion 64 X2 - based testbox (HPC nx6325).
_cpu_down()just hangs as though there were a deadlock in there, 100% of the time.Thanks, I dropped it.
Hmm. It looks like Siddha sent the wrong version of the patch. The working tested version had an additional test to ensure the mask and unmask methods were implemented.
i.e.
+ if (irq_desc[irq].chip->mask)
+ irq_desc[irq].chip->mask(irq);
and
+ if (irq_desc[irq].chip->unmask)
+ irq_desc[irq].chip->unmask(irq);
+
Siddha think you can resend the correct version.
Rafael. Think you can add those two ifs and see if you test bed box works?
I'm still not convinced that we can make fixup_irqs work in general but if we aren't going to yank it we should at least make it consistent with the rest of the code.
Eric
Quoted 14 lines · Eric W. Biederman
Hmm. It looks like Siddha sent the wrong version of the patch. The working tested version had an additional test to ensure the mask and unmask methods were implemented.
i.e.
(quoted diff) +2
-0+ if (irq_desc[irq].chip->mask) + irq_desc[irq].chip->mask(irq);and
arch/x86_64/kernel/irq.c +3
-0+if (irq_desc[irq].chip->unmask) + irq_desc[irq].chip->unmask (irq); +
Quoted 1 line
Siddha think you can resend the correct version.
Eric, In this version, I added the irq_has_action() check and hence removed the check which ensures the presence for mask/unmask. My tests showed that it was working fine. May be I am missing something.
Quoted 7 lines
Rafael. Think you can add those two ifs and see if you test bed box works?
I'm still not convinced that we can make
fixup_irqswork in general but if we aren't going to yank it we should at least make it consistent with the rest of the code.
I agree.
thanks,
suresh
Quoted 56 lines · Eric W. Biederman
Quoted 33 lines · Andrew Morton
Quoted 9 lines · On Sun, 24 Jun 2007 01:54:52 +0200 "Rafael J. Wysocki" <[email protected]> wrote:
Quoted 8 lines · Suresh B
Quoted 2 lines · Darrick J. Wong
This fixes the problem! Hurrah!
Great! Andrew, please include the appended patch in -mm.
----
Subject: [patch]x86_64, irq: use mask/unmask and proper locking in
fixup_irqsQuoted 21 lines
Quoted 20 lines
From: Suresh Siddha <[email protected]>
Force irq migration path during cpu offline, is not using proper locks and
irq_chipmask/unmask routines. This will result in some races(especially the device generating the interrupt can see some inconsistent state, resulting in issues like stuck irq,..).Appended patch fixes the issue by taking proper lock and encapsulating
irq_chipset_affinity()with amask()before and anunmask()after.This fixes a MSI irq stuck issue reported by Darrick Wong.
There are several more general bugs in this area(irq migration in the process context). For example,
1. Possibility of missing edge triggered irq.
2. Reliable method of migrating level triggered irq in the process context.We plan to look and close these in the near future.
Quoted 7 lines
Quoted 5 lines
This patch breaks hibernation on my Turion 64 X2 - based testbox (HPC nx6325).
_cpu_down()just hangs as though there were a deadlock in there, 100% of the time.Thanks, I dropped it.
Hmm. It looks like Siddha sent the wrong version of the patch. The working tested version had an additional test to ensure the mask and unmask methods were implemented.
i.e.
(quoted diff) +2
-0+ if (irq_desc[irq].chip->mask) + irq_desc[irq].chip->mask(irq);and
(quoted diff) +3
-0+ if (irq_desc[irq].chip->unmask) + irq_desc[irq].chip->unmask(irq); +
Quoted 4 lines
Siddha think you can resend the correct version.
Rafael. Think you can add those two ifs and see if you test bed box works?
Yes, that helps.
For reference I'm appending the complete patch that I have tested.
Greetings,
Rafael
--- arch/x86_64/kernel/irq.c | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-)
arch/x86_64/kernel/irq.c +29 -3
@@ -144,17 +144,43 @@ void fixup_irqs(cpumask_t map)for (irq =0 ; irq <NR_IRQS ; irq++) {cpumask_t mask; +int break_affinity =0 ; +int set_affinity =1 ; +if (irq ==2 )continue ; +/* interrupt's are disabled at this point */ +spin_lock (&irq_desc[irq].lock); + +if (!irq_has_action (irq) || +cpus_equal (irq_desc[irq].affinity, map)) { +spin_unlock (&irq_desc[irq].lock); +continue ; + } +cpus_and (mask, irq_desc[irq].affinity, map); -if (any_online_cpu (mask) ==NR_CPUS ) { -printk ("Breaking affinity for irq %i\n" , irq); +if (cpus_empty (mask)) { + break_affinity =1 ; mask = map; } + +if (irq_desc[irq].chip->mask) + irq_desc[irq].chip->mask (irq); +if (irq_desc[irq].chip->set_affinity) irq_desc[irq].chip->set_affinity (irq, mask); -else if (irq_desc[irq].action && !(warned++)) +else if (!(warned++)) + set_affinity =0 ; + +if (irq_desc[irq].chip->unmask) + irq_desc[irq].chip->unmask (irq); + +spin_unlock (&irq_desc[irq].lock); + +if (break_affinity && set_affinity) +printk ("Broke affinity for irq %i\n" , irq); +else if (!set_affinity)printk ("Cannot set affinity for irq %i\n" , irq); }
Quoted 4 lines · Rafael J. Wysocki
This patch breaks hibernation on my Turion 64 X2 - based testbox (HPC nx6325).
_cpu_down()just hangs as though there were a deadlock in there, 100% of the time.
Does the patch at this URL work for you?
http://marc.info/?l=linux-kernel&m=118228358826737&w=2
thanks,
suresh
Quoted 9 lines · Suresh B
Quoted 4 lines · Rafael J. Wysocki
This patch breaks hibernation on my Turion 64 X2 - based testbox (HPC nx6325).
_cpu_down()just hangs as though there were a deadlock in there, 100% of the time.Does the patch at this URL work for you?
Yes, it does.
Greetings,
Rafael
Signature · 1 line
"Premature optimization is the root of all evil." - Donald Knuth