Skip to contentSkip to conversation

Device hang when offlining a CPU due to IRQ misrouting

DISCUSSION Darrick J. Wong 5 participants 32 messages Jun 1, 2007 – Jun 24, 2007

lore
Share & tools
lore
#0 lore raw

Hi there,

I'm seeing a driver hang with 2.6.22-rc3 while being slightly stupid about offlining CPUs. I suspect that this problem extends beyond a particular machine, as I've been able to replicate it with an IBM x3650 and an IBM x3755. This is what I'm doing:

1) I tie an IRQ to a particular CPU via /proc/irq/XXX/smp_affinity (IRQ 4341 is the network card and we're picking on CPU1 in this example):
echo 2 > /proc/irq/4341/smp_affinity

2) I then take CPU1 offline:
echo 0 > /sys/devices/system/cpu/cpu1/online

3) The kernel prints this:

[ 1101.968040] Breaking affinity for irq 4341
[ 1102.074019] CPU 1 is now offline
[ 1102.081593] lockdep: not fixing up alternatives.
[ 1112.886919] nfs: server 9.47.66.169 not responding, still trying

After step 2 the system never sees interrupts from the network card and remains hung like that until CPU1 is brought back up. It looks as though the kernel is trying to reroute the IRQ (or so I'm assuming from the "Breaking affinity" message), but this doesn't ever happen, so the the kernel stops seeing interrupts from the device.

Granted, one should not be offlining the CPU that is currently designated to handle an IRQ, but I suspect that the kernel ought at a minimum to reject the offlining or route the IRQ to any online CPU instead of screwing things up.

There exists a similar scenario. Set the IRQ affinity to a bunch of CPUs, watch /proc/interrupts to see which CPU is actually servicing the interrupts, then offline that CPU. The kernel does not reroute the IRQ to any of the other CPUs and the device also hangs.

The furthest that I've dug is that it works on 2.6.17 and is broken in 2.6.22-rc3 and 2.6.21. Will git-bisect further, but I wanted to know if anyone else has seen this sort of problem. afaik, this seems to happen with both IOAPIC and MSI interrupts, possibly more.

--D

#1 lore raw
Quoted 26 lines · Darrick J. Wong

Hi there,

I'm seeing a driver hang with 2.6.22-rc3 while being slightly stupid about offlining CPUs. I suspect that this problem extends beyond a particular machine, as I've been able to replicate it with an IBM x3650 and an IBM x3755. This is what I'm doing:

1) I tie an IRQ to a particular CPU via /proc/irq/XXX/smp_affinity (IRQ 4341 is the network card and we're picking on CPU1 in this example):
echo 2 > /proc/irq/4341/smp_affinity

2) I then take CPU1 offline:
echo 0 > /sys/devices/system/cpu/cpu1/online

3) The kernel prints this:

[ 1101.968040] Breaking affinity for irq 4341
[ 1102.074019] CPU 1 is now offline
[ 1102.081593] lockdep: not fixing up alternatives.
[ 1112.886919] nfs: server 9.47.66.169 not responding, still trying

After step 2 the system never sees interrupts from the network card and remains hung like that until CPU1 is brought back up. It looks as though the kernel is trying to reroute the IRQ (or so I'm assuming from the "Breaking affinity" message), but this doesn't ever happen, so the the kernel stops seeing interrupts from the device.

Quoted 4 lines

Granted, one should not be offlining the CPU that is currently designated to handle an IRQ, but I suspect that the kernel ought at a minimum to reject the offlining or route the IRQ to any online CPU instead of screwing things up.

I agree.

Quoted 9 lines

There exists a similar scenario. Set the IRQ affinity to a bunch of CPUs, watch /proc/interrupts to see which CPU is actually servicing the interrupts, then offline that CPU. The kernel does not reroute the IRQ to any of the other CPUs and the device also hangs.

The furthest that I've dug is that it works on 2.6.17 and is broken in 2.6.22-rc3 and 2.6.21. Will git-bisect further, but I wanted to know if anyone else has seen this sort of problem. afaik, this seems to happen with both IOAPIC and MSI interrupts, possibly more.

Thanks for the bug report. I'm chuckling because I just submitted a patch to count that whole code path as broken, based on code review. It is trying to do something that the hardware can not reliably accomplish.

Now I am surprised you were seeing this with MSI as well because the hardware should theoretically work in that case. However the irq_fixup code has enough issues that I wouldn't be surprised if it was just doing something stupid and wrong.

Eric

#2 lore raw
In reply to Darrick J. Wong
Quoted 10 lines · Darrick J. Wong

Hi there,

I'm seeing a driver hang with 2.6.22-rc3 while being slightly stupid about offlining CPUs. I suspect that this problem extends beyond a particular machine, as I've been able to replicate it with an IBM x3650 and an IBM x3755. This is what I'm doing:

1) I tie an IRQ to a particular CPU via /proc/irq/XXX/smp_affinity (IRQ 4341 is the network card and we're picking on CPU1 in this example):
echo 2 > /proc/irq/4341/smp_affinity

Darrick, I see a kernel bug in this area(which is already filled with bugs, and I am looking into ways to fix them). Are you making sure that between step-1 and step-2, that interrupts actually started arriving at cpu1?

i.e., do step-1 and wait till the irq's start hitting at cpu1. At this point do step-2 and let us know if you still hit this bug?

Quoted 21 lines

2) I then take CPU1 offline:
echo 0 > /sys/devices/system/cpu/cpu1/online

3) The kernel prints this:

[ 1101.968040] Breaking affinity for irq 4341
[ 1102.074019] CPU 1 is now offline
[ 1102.081593] lockdep: not fixing up alternatives.
[ 1112.886919] nfs: server 9.47.66.169 not responding, still trying

After step 2 the system never sees interrupts from the network card and remains hung like that until CPU1 is brought back up. It looks as though the kernel is trying to reroute the IRQ (or so I'm assuming from the "Breaking affinity" message), but this doesn't ever happen, so the the kernel stops seeing interrupts from the device.

Granted, one should not be offlining the CPU that is currently designated to handle an IRQ, but I suspect that the kernel ought at a minimum to reject the offlining or route the IRQ to any online CPU instead of screwing things up.

Quoted 4 lines

There exists a similar scenario. Set the IRQ affinity to a bunch of CPUs, watch /proc/interrupts to see which CPU is actually servicing the interrupts, then offline that CPU. The kernel does not reroute the IRQ to any of the other CPUs and the device also hangs.

Is this a theory or did you observe this problem happening?

thanks,
suresh

Quoted 5 lines

The furthest that I've dug is that it works on 2.6.17 and is broken in 2.6.22-rc3 and 2.6.21. Will git-bisect further, but I wanted to know if anyone else has seen this sort of problem. afaik, this seems to happen with both IOAPIC and MSI interrupts, possibly more.

#3 lore raw
Quoted 6 lines · Suresh B

Darrick, I see a kernel bug in this area(which is already filled with bugs, and I am looking into ways to fix them). Are you making sure that between step-1 and step-2, that interrupts actually started arriving at cpu1?

i.e., do step-1 and wait till the irq's start hitting at cpu1. At this point do step-2 and let us know if you still hit this bug?

Yes, the bug only happens after CPU1 begins to receive interrupts.

Quoted 6 lines
Quoted 4 lines

There exists a similar scenario. Set the IRQ affinity to a bunch of CPUs, watch /proc/interrupts to see which CPU is actually servicing the interrupts, then offline that CPU. The kernel does not reroute the IRQ to any of the other CPUs and the device also hangs.

Is this a theory or did you observe this problem happening?

Nope, I've observed this situation too.

--D

#4 lore raw
Quoted 10 lines · Darrick J. Wong
Quoted 6 lines · Suresh B

Darrick, I see a kernel bug in this area(which is already filled with bugs, and I am looking into ways to fix them). Are you making sure that between step-1 and step-2, that interrupts actually started arriving at cpu1?

i.e., do step-1 and wait till the irq's start hitting at cpu1. At this point do step-2 and let us know if you still hit this bug?

Yes, the bug only happens after CPU1 begins to receive interrupts.

I see. Your system should have 4 or 8 logical cpu's right. So you must be using logical flat mode, right?

When this bug happens, what does /proc/irq/<irq-no>/smp_affinity show?

thanks,
suresh

#5 lore raw
Quoted 2 lines · Suresh B

I see. Your system should have 4 or 8 logical cpu's right. So you must be using logical flat mode, right?

I believe so. The system has two Xeon 5150s with an Intel 5000 chipset of some sort.

Quoted 1 line

When this bug happens, what does /proc/irq/<irq-no>/smp_affinity show?

root@elm3a188:~# cat /proc/irq/114/smp_affinity 
02

--D

#6 lore raw
Quoted 11 lines · Darrick J. Wong
Quoted 2 lines · Suresh B

I see. Your system should have 4 or 8 logical cpu's right. So you must be using logical flat mode, right?

I believe so. The system has two Xeon 5150s with an Intel 5000 chipset of some sort.

Quoted 1 line

When this bug happens, what does /proc/irq/<irq-no>/smp_affinity show?

root@elm3a188:~# cat /proc/irq/114/smp_affinity 
02

Ok. What this shows is that fixup_irqs() failed to move the irq properly. Ideally we should see cpu_online_map here (i.e., 0xfd).

So most likely __assign_irq_vector() failed for some reason and I am puzzled for the reason...

Does this problem happen only under certain stress or something simple, like

boot the kernel
echo 2 > /proc/irq/114/smp_affinity
wait for irq to hit the cpu1.
echo 0 > /sys/devices/system/cpu/cpu1/online

will immmd trigger this?

thanks,
suresh

#7 lore raw
Quoted 8 lines · Suresh B

Does this problem happen only under certain stress or something simple, like

boot the kernel
echo 2 > /proc/irq/114/smp_affinity
wait for irq to hit the cpu1.
echo 0 > /sys/devices/system/cpu/cpu1/online

will immmd trigger this?

The system is not under any stress at all.

--D

#8 lore raw
Quoted 4 lines · Darrick J. Wong
Quoted 2 lines · Suresh B

Does this problem happen only under certain stress or something simple, like

Quoted 8 lines
Quoted 6 lines

boot the kernel
echo 2 > /proc/irq/114/smp_affinity
wait for irq to hit the cpu1.
echo 0 > /sys/devices/system/cpu/cpu1/online

will immmd trigger this?

The system is not under any stress at all.

Can you send us your system's dmesg aswell as output of /proc/interrupts?

thanks,
suresh

#10 lore raw
Quoted 6 lines · Darrick J. Wong
Quoted 1 line · Suresh B

Can you send us your system's dmesg aswell as output of /proc/interrupts?

http://sweaglesw.net/~djwong/docs/dmesg
http://sweaglesw.net/~djwong/docs/interrupts

Didn't find anything wrong in that information. Can you try this appended debug patch and see if you see this error msg in dmesg, when you hit the bug? Thanks.

arch/x86_64/kernel/io_apic.c +3 -0
index d8bfe31..3409c1f 100644
@@ -720,10 +720,13 @@ static int assign_irq_vector(int irq, cpumask_t mask)
 {
 	int err;
 	unsigned long flags;
+	int cpu = smp_processor_id();
 
 	spin_lock_irqsave(&vector_lock, flags);
 	err = __assign_irq_vector(irq, mask);
 	spin_unlock_irqrestore(&vector_lock, flags);
+	if (err && !cpu_isset(cpu, cpu_online_map))
+		printk("assigning irq to a vector failed : %d\n", err);
 	return err;
 }
 
#12 lore raw
Quoted 6 lines · Darrick J. Wong
Quoted 5 lines · Suresh B
Quoted 4 lines · Darrick J. Wong
Quoted 1 line · Suresh B

Can you send us your system's dmesg aswell as output of /proc/interrupts?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

http://sweaglesw.net/~djwong/docs/dmesg
http://sweaglesw.net/~djwong/docs/interrupts

Didn't find anything wrong in that information. Can you try this appended debug patch and see if you see this error msg in dmesg, when you hit the bug? Thanks.

I don't see that message.

Weird. Then the bug can only happen if for some reason, "mask = map" didn't happen in fixup_irqs(). Can you send us the disassembly of the fixup_irqs()?

thanks,
suresh

#13 lore raw
Quoted 3 lines · Suresh B

Weird. Then the bug can only happen if for some reason, "mask = map" didn't happen in fixup_irqs(). Can you send us the disassembly of the fixup_irqs()?

Attached.

--D

(gdb) disassemble fixup_irqs
Dump of assembler code for function fixup_irqs:

0xffffffff8020bf50 <fixup_irqs+0>:	push   %rbp
0xffffffff8020bf51 <fixup_irqs+1>:	mov    %rsp,%rbp
0xffffffff8020bf54 <fixup_irqs+4>:	push   %r13
0xffffffff8020bf56 <fixup_irqs+6>:	xor    %r13d,%r13d
0xffffffff8020bf59 <fixup_irqs+9>:	push   %r12
0xffffffff8020bf5b <fixup_irqs+11>:	push   %rbx
0xffffffff8020bf5c <fixup_irqs+12>:	sub    $0x28,%rsp
0xffffffff8020bf60 <fixup_irqs+16>:	mov    %rdi,0xffffffffffffffc0(%rbp)
0xffffffff8020bf64 <fixup_irqs+20>:	mov    %rsi,0xffffffffffffffc8(%rbp)
0xffffffff8020bf68 <fixup_irqs+24>:	jmp    0xffffffff8020bf73 <fixup_irqs+35>
0xffffffff8020bf6a <fixup_irqs+26>:	inc    %r13d
0xffffffff8020bf6d <fixup_irqs+29>:	cmp    $0x2,%r13d
0xffffffff8020bf71 <fixup_irqs+33>:	je     0xffffffff8020bf6a <fixup_irqs+26>
0xffffffff8020bf73 <fixup_irqs+35>:	mov    %r13d,%r12d
0xffffffff8020bf76 <fixup_irqs+38>:	lea    0xffffffffffffffd0(%rbp),%rbx
0xffffffff8020bf7a <fixup_irqs+42>:	lea    0xffffffffffffffc0(%rbp),%rdx
0xffffffff8020bf7e <fixup_irqs+46>:	shl    $0x8,%r12
0xffffffff8020bf82 <fixup_irqs+50>:	mov    $0x80,%ecx
0xffffffff8020bf87 <fixup_irqs+55>:	lea    0xffffffff805505f8(%r12),%rsi
0xffffffff8020bf8f <fixup_irqs+63>:	mov    %rbx,%rdi
0xffffffff8020bf92 <fixup_irqs+66>:	callq  0xffffffff802fb606 <__bitmap_and>
0xffffffff8020bf97 <fixup_irqs+71>:	mov    %rbx,%rdi
0xffffffff8020bf9a <fixup_irqs+74>:	callq  0xffffffff802fc6ad <__any_online_cpu>
0xffffffff8020bf9f <fixup_irqs+79>:	add    $0xffffffffffffff80,%eax
0xffffffff8020bfa2 <fixup_irqs+82>:	jne    0xffffffff8020bfc5 <fixup_irqs+117>
0xffffffff8020bfa4 <fixup_irqs+84>:	mov    %r13d,%esi
0xffffffff8020bfa7 <fixup_irqs+87>:	mov    $0xffffffff804a52b0,%rdi
0xffffffff8020bfae <fixup_irqs+94>:	xor    %eax,%eax
0xffffffff8020bfb0 <fixup_irqs+96>:	callq  0xffffffff80233d28 <printk>
0xffffffff8020bfb5 <fixup_irqs+101>:	mov    0xffffffffffffffc0(%rbp),%rax
0xffffffff8020bfb9 <fixup_irqs+105>:	mov    %rax,0xffffffffffffffd0(%rbp)
0xffffffff8020bfbd <fixup_irqs+109>:	mov    0xffffffffffffffc8(%rbp),%rax
0xffffffff8020bfc1 <fixup_irqs+113>:	mov    %rax,0xffffffffffffffd8(%rbp)
0xffffffff8020bfc5 <fixup_irqs+117>:	mov    0xffffffff80550588(%r12),%rax
0xffffffff8020bfcd <fixup_irqs+125>:	mov    0x58(%rax),%rax
0xffffffff8020bfd1 <fixup_irqs+129>:	test   %rax,%rax
0xffffffff8020bfd4 <fixup_irqs+132>:	je     0xffffffff8020bfe5 <fixup_irqs+149>
0xffffffff8020bfd6 <fixup_irqs+134>:	mov    0xffffffffffffffd0(%rbp),%rsi
0xffffffff8020bfda <fixup_irqs+138>:	mov    0xffffffffffffffd8(%rbp),%rdx
0xffffffff8020bfde <fixup_irqs+142>:	mov    %r13d,%edi
0xffffffff8020bfe1 <fixup_irqs+145>:	callq  *%rax
0xffffffff8020bfe3 <fixup_irqs+147>:	jmp    0xffffffff8020c013 <fixup_irqs+195>
0xffffffff8020bfe5 <fixup_irqs+149>:	cmpq   $0x0,0xffffffff805505a8(%r12)
0xffffffff8020bfee <fixup_irqs+158>:	je     0xffffffff8020c013 <fixup_irqs+195>
0xffffffff8020bff0 <fixup_irqs+160>:	mov    5181486(%rip),%eax        # 0xffffffff806fd024 <warned.11720>
0xffffffff8020bff6 <fixup_irqs+166>:	inc    %eax
0xffffffff8020bff8 <fixup_irqs+168>:	mov    %eax,5181478(%rip)        # 0xffffffff806fd024 <warned.11720>
0xffffffff8020bffe <fixup_irqs+174>:	dec    %eax
0xffffffff8020c000 <fixup_irqs+176>:	jne    0xffffffff8020c013 <fixup_irqs+195>
0xffffffff8020c002 <fixup_irqs+178>:	mov    %r13d,%esi
0xffffffff8020c005 <fixup_irqs+181>:	mov    $0xffffffff804a52ce,%rdi
0xffffffff8020c00c <fixup_irqs+188>:	xor    %eax,%eax
0xffffffff8020c00e <fixup_irqs+190>:	callq  0xffffffff80233d28 <printk>
0xffffffff8020c013 <fixup_irqs+195>:	lea    0x1(%r13),%eax
0xffffffff8020c017 <fixup_irqs+199>:	cmp    $0x10ff,%eax
0xffffffff8020c01c <fixup_irqs+204>:	jbe    0xffffffff8020bf6a <fixup_irqs+26>
0xffffffff8020c022 <fixup_irqs+210>:	callq  0xffffffff8024e46e <trace_hardirqs_on>

0xffffffff8020c027 <fixup_irqs+215>: sti
0xffffffff8020c028 <fixup_irqs+216>: mov $0x418958,%edi
0xffffffff8020c02d <fixup_irqs+221>: callq 0xffffffff803018cf <__const_udelay>
0xffffffff8020c032 <fixup_irqs+226>: cli

0xffffffff8020c033 <fixup_irqs+227>:	callq  0xffffffff8024cf31 <trace_hardirqs_off>
0xffffffff8020c038 <fixup_irqs+232>:	add    $0x28,%rsp
0xffffffff8020c03c <fixup_irqs+236>:	pop    %rbx
0xffffffff8020c03d <fixup_irqs+237>:	pop    %r12
0xffffffff8020c03f <fixup_irqs+239>:	pop    %r13

0xffffffff8020c041 <fixup_irqs+241>: leaveq
0xffffffff8020c042 <fixup_irqs+242>: retq

#14 lore raw
Quoted 7 lines · Darrick J. Wong
Quoted 3 lines · Suresh B

Weird. Then the bug can only happen if for some reason, "mask = map" didn't happen in fixup_irqs(). Can you send us the disassembly of the fixup_irqs()?

Attached.

hmm.. Darrick, can't find anything wrong in there.

I am very much puzzled and the main thing I am confused about is, that how come "/proc/irq/<irq#-hung>/smp_affinity" is still pointing at the old offlined cpu, while calls to set_affinity() with cpu_online_map mask in fixup_irqs() don't show any failure..

As you have the failing system, you need to do more detective work and help me out. Can you try this debug patch and send across the dmesg after the bug happens and also can you try different compiler to see if something changes..

arch/x86_64/kernel/irq.c +4 -2
index 3eaceac..fc2a576 100644
@@ -152,9 +152,11 @@ void fixup_irqs(cpumask_t map)
 			printk("Breaking affinity for irq %i\n", irq);
 			mask = map;
 		}
-		if (irq_desc[irq].chip->set_affinity)
+		if (irq_desc[irq].chip->set_affinity) {
+			printk("calling set affinity for %i, with mask %lx\n",
+				irq, cpus_addr(mask)[0]);
 			irq_desc[irq].chip->set_affinity(irq, mask);
-		else if (irq_desc[irq].action && !(warned++))
+		} else if (irq_desc[irq].action && !(warned++))
 			printk("Cannot set affinity for irq %i\n", irq);
 	}
 
#15 lore raw
Quoted 4 lines · Suresh B

As you have the failing system, you need to do more detective work and help me out. Can you try this debug patch and send across the dmesg after the bug happens and also can you try different compiler to see if something changes..

Hrm, I just updated to -rc5. Interrupts being handled by the IOAPIC don't suffer from this problem, but MSI interrupts are still affected. I added a few printks to the kernel to figure out what IRQ affinity masks were being passed around and saw this:

[  256.298773] Breaking affinity for irq 4341
[  256.298774] irq=4341 affinity=2 mask=d
<call to set_affinity>
[  256.298787] irq=4341 affinity=d

<ethernet on irq 4341 stops working>

I'll keep digging, but at least it appears that the problem has been shrunk down to something the MSI code.

--D

#16 lore raw
Quoted 12 lines · Darrick J. Wong
Quoted 4 lines · Suresh B

As you have the failing system, you need to do more detective work and help me out. Can you try this debug patch and send across the dmesg after the bug happens and also can you try different compiler to see if something changes..

Hrm, I just updated to -rc5. Interrupts being handled by the IOAPIC don't suffer from this problem, but MSI interrupts are still affected. I added a few printks to the kernel to figure out what IRQ affinity masks were being passed around and saw this:

Quoted 5 lines
[  256.298773] Breaking affinity for irq 4341
[  256.298774] irq=4341 affinity=2 mask=d
<call to set_affinity>
[  256.298787] irq=4341 affinity=d

<ethernet on irq 4341 stops working>

And just to make sure, at this point, your MSI irq 4341 affinity (/proc/irq/4341/smp_affinity) still points to '2'?

Quoted 2 lines

I'll keep digging, but at least it appears that the problem has been shrunk down to something the MSI code.

thanks,
suresh

#17 lore raw
Quoted 6 lines · Suresh B
Quoted 3 lines

<call to set_affinity>

[  256.298787] irq=4341 affinity=d

<ethernet on irq 4341 stops working>

And just to make sure, at this point, your MSI irq 4341 affinity (/proc/irq/4341/smp_affinity) still points to '2'?

Actually, it's 0xD. From the kernel's perspective the mask has been updated (and I even stuck a printk into set_msi_irq_affinity to verify that the writes are happening) but ... the hardware doesn't seem to reflect this. I also tried putting read_msi_msg right afterwards to compare contents, though it complained about all the MSIs _except_ for 4341. (Of course, I could just be way off on the effectiveness of that.)

--D

#18 lore raw
Quoted 9 lines · Darrick J. Wong
Quoted 6 lines · Suresh B
Quoted 3 lines

<call to set_affinity>

[  256.298787] irq=4341 affinity=d

<ethernet on irq 4341 stops working>

And just to make sure, at this point, your MSI irq 4341 affinity (/proc/irq/4341/smp_affinity) still points to '2'?

Quoted 7 lines

Actually, it's 0xD. From the kernel's perspective the mask has been updated (and I even stuck a printk into set_msi_irq_affinity to verify that the writes are happening) but ... the hardware doesn't seem to reflect this. I also tried putting read_msi_msg right afterwards to compare contents, though it complained about all the MSIs _except_ for 4341. (Of course, I could just be way off on the effectiveness of that.)

The fact that MSI interrupts are having problems is odd. It is possible that we still have a bug in there somewhere but msi interrupts should be safe to migrate outside of irq context (no known hardware bugs). As we can actually synchronize with the irq source and eliminate all of the migration races.

The non-msi case requires hitting a hardware race that is rare enough you should not normally have problems.

Eric

#19 lore raw
Quoted 19 lines · Eric W. Biederman
Quoted 9 lines · Darrick J. Wong
Quoted 6 lines · Suresh B
Quoted 3 lines

<call to set_affinity>

[  256.298787] irq=4341 affinity=d

<ethernet on irq 4341 stops working>

And just to make sure, at this point, your MSI irq 4341 affinity (/proc/irq/4341/smp_affinity) still points to '2'?

Quoted 7 lines

Actually, it's 0xD. From the kernel's perspective the mask has been updated (and I even stuck a printk into set_msi_irq_affinity to verify that the writes are happening) but ... the hardware doesn't seem to reflect this. I also tried putting read_msi_msg right afterwards to compare contents, though it complained about all the MSIs _except_ for 4341. (Of course, I could just be way off on the effectiveness of that.)

Quoted 8 lines

The fact that MSI interrupts are having problems is odd. It is possible that we still have a bug in there somewhere but msi interrupts should be safe to migrate outside of irq context (no known hardware bugs). As we can actually synchronize with the irq source and eliminate all of the migration races.

The non-msi case requires hitting a hardware race that is rare enough you should not normally have problems.

Yep. But Darrick's seems to say, problem happens consistently.

Anyhow, Darrick there is a general bug in this area, can you try this and see if it helps?

arch/x86_64/kernel/irq.c +19 -1
index 3eaceac..a0e11c9 100644
@@ -144,17 +144,35 @@ void fixup_irqs(cpumask_t map)
 
 	for (irq = 0; irq < NR_IRQS; irq++) {
 		cpumask_t mask;
+		int break_affinity = 0;
+		int set_affinity = 1;
+
 		if (irq == 2)
 			continue;
 
+		/* irq's are disabled at this point */
+		spin_lock(&irq_desc[irq].lock);
+
 		cpus_and(mask, irq_desc[irq].affinity, map);
 		if (any_online_cpu(mask) == NR_CPUS) {
-			printk("Breaking affinity for irq %i\n", irq);
+			break_affinity = 1;
 			mask = map;
 		}
+
+		irq_desc[irq].chip->mask(irq);
+
 		if (irq_desc[irq].chip->set_affinity)
 			irq_desc[irq].chip->set_affinity(irq, mask);
 		else if (irq_desc[irq].action && !(warned++))
+			set_affinity = 0;
+
+		irq_desc[irq].chip->unmask(irq);
+
+		spin_unlock(&irq_desc[irq].lock);
+
+		if (break_affinity && set_affinity)
+			printk("Broke affinity for irq %i\n", irq);
+		else if (!set_affinity)
 			printk("Cannot set affinity for irq %i\n", irq);
 	}
 
#20 lore raw
Quoted 26 lines · Suresh B
Quoted 25 lines · Eric W. Biederman
Quoted 9 lines · Darrick J. Wong
Quoted 6 lines · Suresh B
Quoted 3 lines

<call to set_affinity>

[  256.298787] irq=4341 affinity=d

<ethernet on irq 4341 stops working>

And just to make sure, at this point, your MSI irq 4341 affinity (/proc/irq/4341/smp_affinity) still points to '2'?

Quoted 7 lines

Actually, it's 0xD. From the kernel's perspective the mask has been updated (and I even stuck a printk into set_msi_irq_affinity to verify that the writes are happening) but ... the hardware doesn't seem to reflect this. I also tried putting read_msi_msg right afterwards to compare contents, though it complained about all the MSIs _except_ for 4341. (Of course, I could just be way off on the effectiveness of that.)

The fact that MSI interrupts are having problems is odd. It is possible that we still have a bug in there somewhere but msi interrupts should be safe to migrate outside of irq context (no known hardware bugs). As we can actually synchronize with the irq source and eliminate all of the migration races.

Quoted 7 lines
Quoted 2 lines

The non-msi case requires hitting a hardware race that is rare enough you should not normally have problems.

Yep. But Darrick's seems to say, problem happens consistently.

Anyhow, Darrick there is a general bug in this area, can you try this and see if it helps?

There are several general bugs in this area. But yes your patch should help things, especially for MSI where masking the irq before migration is required. Adding locking the proper locking and masking should make things quite a bit more how set_affinity is expected to be called.

I just gave up on fixing these things because we can't eliminate the races, so the real problem is the existence of this code path with it's unsupportable semantics in the first place.

Quoted 13 lines
arch/x86_64/kernel/irq.c +3 -0
index 3eaceac..a0e11c9 100644
@@ -144,17 +144,35 @@ void fixup_irqs(cpumask_t map)
 
 	for (irq = 0; irq < NR_IRQS; irq++) {
 		cpumask_t mask;
+		int break_affinity = 0;
+		int set_affinity = 1;
+
 		if (irq == 2)
 			continue;

arch/x86_64/kernel/irq.c void fixup_irqs(cpumask_t map) · line 160

Quoted 10 lines
arch/x86_64/kernel/irq.c +4 -1
 
+		/* irq's are disabled at this point */
+		spin_lock(&irq_desc[irq].lock);
+
 		cpus_and(mask, irq_desc[irq].affinity, map);
 		if (any_online_cpu(mask) == NR_CPUS) {
-			printk("Breaking affinity for irq %i\n", irq);
+			break_affinity = 1;
 			mask = map;
 		}

We should really express the "any_online_cpu(mask) == NR_CPUS" test as:
"cpus_empty(mask)" it would be much clearer.

Further we should skip the migration if "cpus_equal(mask, irq_desc[irq].affinity)" or "!irq_has_action(irq)" because no one has called request_irq.

Quoted 17 lines
arch/x86_64/kernel/irq.c +11 -0
+		irq_desc[irq].chip->mask(irq);
+
 		if (irq_desc[irq].chip->set_affinity)
 			irq_desc[irq].chip->set_affinity(irq, mask);
 		else if (irq_desc[irq].action && !(warned++))
+			set_affinity = 0;
+
+		irq_desc[irq].chip->unmask(irq);
+
+		spin_unlock(&irq_desc[irq].lock);
+
+		if (break_affinity && set_affinity)
+			printk("Broke affinity for irq %i\n", irq);
+		else if (!set_affinity)
 			printk("Cannot set affinity for irq %i\n", irq);
 	}
 
#22 lore raw
Quoted 5 lines · Darrick J. Wong
Quoted 2 lines · Suresh B

Anyhow, Darrick there is a general bug in this area, can you try this and see if it helps?

Er... that instantly locked up the system.

hmm.. Please try this instead. This is intended only for debug. Based on your test results, we can comeup with a more decent fix.

arch/x86_64/kernel/irq.c +21 -1
index 3eaceac..3997679 100644
@@ -144,17 +144,37 @@ void fixup_irqs(cpumask_t map)
 
 	for (irq = 0; irq < NR_IRQS; irq++) {
 		cpumask_t mask;
+		int break_affinity = 0;
+		int set_affinity = 1;
+
 		if (irq == 2)
 			continue;
 
+		/* irq's are disabled at this point */
+		spin_lock(&irq_desc[irq].lock);
+
 		cpus_and(mask, irq_desc[irq].affinity, map);
 		if (any_online_cpu(mask) == NR_CPUS) {
-			printk("Breaking affinity for irq %i\n", irq);
+			break_affinity = 1;
 			mask = map;
 		}
+
+		if (irq_desc[irq].chip->mask)
+			irq_desc[irq].chip->mask(irq);
+
 		if (irq_desc[irq].chip->set_affinity)
 			irq_desc[irq].chip->set_affinity(irq, mask);
 		else if (irq_desc[irq].action && !(warned++))
+			set_affinity = 0;
+
+		if (irq_desc[irq].chip->unmask)
+			irq_desc[irq].chip->unmask(irq);
+
+		spin_unlock(&irq_desc[irq].lock);
+
+		if (break_affinity && set_affinity)
+			printk("Broke affinity for irq %i\n", irq);
+		else if (!set_affinity)
 			printk("Cannot set affinity for irq %i\n", irq);
 	}
 
#23 lore raw
Quoted 2 lines · Suresh B

hmm.. Please try this instead. This is intended only for debug. Based on your test results, we can comeup with a more decent fix.

This fixes the problem! Hurrah!

--D

#24 lore raw
Quoted 2 lines · Darrick J. Wong

This fixes the problem! Hurrah!

Great! Andrew, please include the appended patch in -mm.

----
Subject: [patch] x86_64, irq: use mask/unmask and proper locking in fixup_irqs
From: Suresh Siddha <[email protected]>

Force irq migration path during cpu offline, is not using proper locks and irq_chip mask/unmask routines. This will result in some races(especially the device generating the interrupt can see some inconsistent state, resulting in issues like stuck irq,..).

Appended patch fixes the issue by taking proper lock and encapsulating irq_chip set_affinity() with a mask() before and an unmask() after.

This fixes a MSI irq stuck issue reported by Darrick Wong.

There are several more general bugs in this area(irq migration in the process context). For example,

1. Possibility of missing edge triggered irq.
2. Reliable method of migrating level triggered irq in the process context.

We plan to look and close these in the near future.

---

arch/x86_64/kernel/irq.c +27 -3
index 3eaceac..55b2733 100644
@@ -144,17 +144,41 @@ void fixup_irqs(cpumask_t map)
 
 	for (irq = 0; irq < NR_IRQS; irq++) {
 		cpumask_t mask;
+		int break_affinity = 0;
+		int set_affinity = 1;
+
 		if (irq == 2)
 			continue;
 
+		/* interrupt's are disabled at this point */
+		spin_lock(&irq_desc[irq].lock);
+
+		if (!irq_has_action(irq) ||
+		    cpus_equal(irq_desc[irq].affinity, map)) {
+			spin_unlock(&irq_desc[irq].lock);
+			continue;
+		}
+
 		cpus_and(mask, irq_desc[irq].affinity, map);
-		if (any_online_cpu(mask) == NR_CPUS) {
-			printk("Breaking affinity for irq %i\n", irq);
+		if (cpus_empty(mask)) {
+			break_affinity = 1;
 			mask = map;
 		}
+
+		irq_desc[irq].chip->mask(irq);
+
 		if (irq_desc[irq].chip->set_affinity)
 			irq_desc[irq].chip->set_affinity(irq, mask);
-		else if (irq_desc[irq].action && !(warned++))
+		else if (!(warned++))
+			set_affinity = 0;
+
+		irq_desc[irq].chip->unmask(irq);
+
+		spin_unlock(&irq_desc[irq].lock);
+
+		if (break_affinity && set_affinity)
+			printk("Broke affinity for irq %i\n", irq);
+		else if (!set_affinity)
 			printk("Cannot set affinity for irq %i\n", irq);
 	}
 
#25 lore raw
Quoted 21 lines · Suresh B
Quoted 2 lines · Darrick J. Wong

This fixes the problem! Hurrah!

Great! Andrew, please include the appended patch in -mm.

----
Subject: [patch] x86_64, irq: use mask/unmask and proper locking in fixup_irqs
From: Suresh Siddha <[email protected]>

Force irq migration path during cpu offline, is not using proper locks and irq_chip mask/unmask routines. This will result in some races(especially the device generating the interrupt can see some inconsistent state, resulting in issues like stuck irq,..).

Appended patch fixes the issue by taking proper lock and encapsulating irq_chip set_affinity() with a mask() before and an unmask() after.

This fixes a MSI irq stuck issue reported by Darrick Wong.

Quoted 7 lines

There are several more general bugs in this area(irq migration in the process context). For example,

1. Possibility of missing edge triggered irq.
2. Reliable method of migrating level triggered irq in the process context.

We plan to look and close these in the near future.

This patch breaks hibernation on my Turion 64 X2 - based testbox (HPC nx6325).

_cpu_down() just hangs as though there were a deadlock in there, 100% of the time.

Greetings,
Rafael

Signature · 1 line
"Premature optimization is the root of all evil." - Donald Knuth
#26 lore raw
Quoted 28 lines · On Sun, 24 Jun 2007 01:54:52 +0200 "Rafael J. Wysocki" <[email protected]> wrote:
Quoted 27 lines · Suresh B
Quoted 2 lines · Darrick J. Wong

This fixes the problem! Hurrah!

Great! Andrew, please include the appended patch in -mm.

----
Subject: [patch] x86_64, irq: use mask/unmask and proper locking in fixup_irqs
From: Suresh Siddha <[email protected]>

Force irq migration path during cpu offline, is not using proper locks and irq_chip mask/unmask routines. This will result in some races(especially the device generating the interrupt can see some inconsistent state, resulting in issues like stuck irq,..).

Appended patch fixes the issue by taking proper lock and encapsulating irq_chip set_affinity() with a mask() before and an unmask() after.

This fixes a MSI irq stuck issue reported by Darrick Wong.

There are several more general bugs in this area(irq migration in the process context). For example,

1. Possibility of missing edge triggered irq.
2. Reliable method of migrating level triggered irq in the process context.

Quoted 7 lines
Quoted 1 line

We plan to look and close these in the near future.

This patch breaks hibernation on my Turion 64 X2 - based testbox (HPC nx6325).

_cpu_down() just hangs as though there were a deadlock in there, 100% of the time.

Thanks, I dropped it.

#27 lore raw
Quoted 33 lines · Andrew Morton
Quoted 9 lines · On Sun, 24 Jun 2007 01:54:52 +0200 "Rafael J. Wysocki" <[email protected]> wrote:
Quoted 8 lines · Suresh B
Quoted 2 lines · Darrick J. Wong

This fixes the problem! Hurrah!

Great! Andrew, please include the appended patch in -mm.

----
Subject: [patch] x86_64, irq: use mask/unmask and proper locking in

fixup_irqs

Quoted 21 lines
Quoted 20 lines

From: Suresh Siddha <[email protected]>

Force irq migration path during cpu offline, is not using proper locks and irq_chip mask/unmask routines. This will result in some races(especially the device generating the interrupt can see some inconsistent state, resulting in issues like stuck irq,..).

Appended patch fixes the issue by taking proper lock and encapsulating irq_chip set_affinity() with a mask() before and an unmask() after.

This fixes a MSI irq stuck issue reported by Darrick Wong.

There are several more general bugs in this area(irq migration in the process context). For example,

1. Possibility of missing edge triggered irq.
2. Reliable method of migrating level triggered irq in the process context.

We plan to look and close these in the near future.

Quoted 7 lines
Quoted 5 lines

This patch breaks hibernation on my Turion 64 X2 - based testbox (HPC nx6325).

_cpu_down() just hangs as though there were a deadlock in there, 100% of the time.

Thanks, I dropped it.

Hmm. It looks like Siddha sent the wrong version of the patch. The working tested version had an additional test to ensure the mask and unmask methods were implemented.

i.e.
+ if (irq_desc[irq].chip->mask)
+ irq_desc[irq].chip->mask(irq);
and

+ if (irq_desc[irq].chip->unmask)
+ irq_desc[irq].chip->unmask(irq);
+

Siddha think you can resend the correct version.

Rafael. Think you can add those two ifs and see if you test bed box works?

I'm still not convinced that we can make fixup_irqs work in general but if we aren't going to yank it we should at least make it consistent with the rest of the code.

Eric

#28 lore raw
Quoted 14 lines · Eric W. Biederman

Hmm. It looks like Siddha sent the wrong version of the patch. The working tested version had an additional test to ensure the mask and unmask methods were implemented.

i.e.

(quoted diff) +2 -0
+		if (irq_desc[irq].chip->mask)
+			irq_desc[irq].chip->mask(irq);

and

arch/x86_64/kernel/irq.c +3 -0
+		if (irq_desc[irq].chip->unmask)
+			irq_desc[irq].chip->unmask(irq);
+
Quoted 1 line

Siddha think you can resend the correct version.

Eric, In this version, I added the irq_has_action() check and hence removed the check which ensures the presence for mask/unmask. My tests showed that it was working fine. May be I am missing something.

Quoted 7 lines

Rafael. Think you can add those two ifs and see if you test bed box works?

I'm still not convinced that we can make fixup_irqs work in general but if we aren't going to yank it we should at least make it consistent with the rest of the code.

I agree.

thanks,
suresh

#29 lore raw
In reply to Eric W. Biederman
Quoted 56 lines · Eric W. Biederman
Quoted 33 lines · Andrew Morton
Quoted 9 lines · On Sun, 24 Jun 2007 01:54:52 +0200 "Rafael J. Wysocki" <[email protected]> wrote:
Quoted 8 lines · Suresh B
Quoted 2 lines · Darrick J. Wong

This fixes the problem! Hurrah!

Great! Andrew, please include the appended patch in -mm.

----
Subject: [patch] x86_64, irq: use mask/unmask and proper locking in

fixup_irqs

Quoted 21 lines
Quoted 20 lines

From: Suresh Siddha <[email protected]>

Force irq migration path during cpu offline, is not using proper locks and irq_chip mask/unmask routines. This will result in some races(especially the device generating the interrupt can see some inconsistent state, resulting in issues like stuck irq,..).

Appended patch fixes the issue by taking proper lock and encapsulating irq_chip set_affinity() with a mask() before and an unmask() after.

This fixes a MSI irq stuck issue reported by Darrick Wong.

There are several more general bugs in this area(irq migration in the process context). For example,

1. Possibility of missing edge triggered irq.
2. Reliable method of migrating level triggered irq in the process context.

We plan to look and close these in the near future.

Quoted 7 lines
Quoted 5 lines

This patch breaks hibernation on my Turion 64 X2 - based testbox (HPC nx6325).

_cpu_down() just hangs as though there were a deadlock in there, 100% of the time.

Thanks, I dropped it.

Hmm. It looks like Siddha sent the wrong version of the patch. The working tested version had an additional test to ensure the mask and unmask methods were implemented.

i.e.

(quoted diff) +2 -0
+		if (irq_desc[irq].chip->mask)
+			irq_desc[irq].chip->mask(irq);

and

(quoted diff) +3 -0
+		if (irq_desc[irq].chip->unmask)
+			irq_desc[irq].chip->unmask(irq);
+
Quoted 4 lines

Siddha think you can resend the correct version.

Rafael. Think you can add those two ifs and see if you test bed box works?

Yes, that helps.

For reference I'm appending the complete patch that I have tested.

Greetings,
Rafael

---
 arch/x86_64/kernel/irq.c |   32 +++++++++++++++++++++++++++++---
 1 file changed, 29 insertions(+), 3 deletions(-)
arch/x86_64/kernel/irq.c +29 -3
Index: linux-2.6.22-rc5/arch/x86_64/kernel/irq.c
===================================================================
@@ -144,17 +144,43 @@ void fixup_irqs(cpumask_t map)
 
 	for (irq = 0; irq < NR_IRQS; irq++) {
 		cpumask_t mask;
+		int break_affinity = 0;
+		int set_affinity = 1;
+
 		if (irq == 2)
 			continue;
 
+		/* interrupt's are disabled at this point */
+		spin_lock(&irq_desc[irq].lock);
+
+		if (!irq_has_action(irq) ||
+		    cpus_equal(irq_desc[irq].affinity, map)) {
+			spin_unlock(&irq_desc[irq].lock);
+			continue;
+		}
+
 		cpus_and(mask, irq_desc[irq].affinity, map);
-		if (any_online_cpu(mask) == NR_CPUS) {
-			printk("Breaking affinity for irq %i\n", irq);
+		if (cpus_empty(mask)) {
+			break_affinity = 1;
 			mask = map;
 		}
+
+		if (irq_desc[irq].chip->mask)
+			irq_desc[irq].chip->mask(irq);
+
 		if (irq_desc[irq].chip->set_affinity)
 			irq_desc[irq].chip->set_affinity(irq, mask);
-		else if (irq_desc[irq].action && !(warned++))
+		else if (!(warned++))
+			set_affinity = 0;
+
+		if (irq_desc[irq].chip->unmask)
+			irq_desc[irq].chip->unmask(irq);
+
+		spin_unlock(&irq_desc[irq].lock);
+
+		if (break_affinity && set_affinity)
+			printk("Broke affinity for irq %i\n", irq);
+		else if (!set_affinity)
 			printk("Cannot set affinity for irq %i\n", irq);
 	}
 
#31 lore raw
Quoted 9 lines · Suresh B
Quoted 4 lines · Rafael J. Wysocki

This patch breaks hibernation on my Turion 64 X2 - based testbox (HPC nx6325).

_cpu_down() just hangs as though there were a deadlock in there, 100% of the time.

Does the patch at this URL work for you?

http://marc.info/?l=linux-kernel&m=118228358826737&w=2

Yes, it does.

Greetings,
Rafael

Signature · 1 line
"Premature optimization is the root of all evil." - Donald Knuth