Support vector domain on ia64

Hi,

Here is a series of patches for ia64 vector domain. By these patches, we can use more than 256 irqs. The patchset is based on existing x86-64 vector domain code. This is for 2.6.22-rc5 and I tested them on my ia64 box.

Changes from previous patchset:

- rebase to 2.6.22-rc5

- define NR_IRQS as (NR_VECTORS + (32 * NR_CPUS))
NR_VECTROS equals 256

- bind gsi to irq
If irq number is changed for each enable/disable, suspend/resume may cause some problems. To avoid this problem, bind each gsi to specific irq.

This patchset is just a draft, so there is following TBD, I think.

o How to migrate irq

In migration, we don't handle the vector for the pending irqs. Probably we should manage it.

There are following 13 patches for ia64 vector domain.

  [PATCH take2 1/13]  Remove block structure for locking in iosapic.c
  [PATCH take2 2/13]  Remove duplicated members in iosapic_rte_info
  [PATCH take2 3/13]  Use per iosapic lock for indirect iosapic register access
  [PATCH take2 4/13]  Cleanup lock order in iosapic_register_intr
  [PATCH take2 5/13]  Use dynamic irq for iosapic interrupts
  [PATCH take2 6/13]  Fix invalid irq vector assumption for iosapic
  [PATCH take2 7/13]  Check if irq is sharable
  [PATCH take2 8/13]  Add mapping table between irq and vector
  [PATCH take2 9/13]  Add support for vector domain

[PATCH take2 10/13] Support irq migration across domain [PATCH take2 11/13] Enable percpu vector domain for IA64_GENERIC [PATCH take2 12/13] Enable percpu vector domain for IA64_DIG [PATCH take2 13/13] Bind gsi to irq

For more details, please refer to the header of each patch.

Any comments or questions are welcome.

Thanks.
Yasuaki Ishimatsu

There are a few whitespace issues (<space><tab>) amongst these patches (I think in parts 6 & 13). Here's the summary of the broken bits:

< + unsigned int gsi_base; /* GSI base */
< + unsigned short num_rte; /* # of RTEs on this IOSAPIC */
---

< + return vector;
---

                 from include/linux/interrupt.h:11,
                 from include/asm-ia64/mca.h:16,
                 from arch/ia64/kernel/asm-offsets.c:15:

Fix for this is just:

index 8efb6e1..91abd1b 100644
@@ -248,7 +248,7 @@ void __setup_vector_irq(int cpu)
 	}
 }
 
-#if defined(CONFIG_IA64_GENERIC) || defined(CONFIG_IA64_DIG)
+#if defined(CONFIG_SMP) && (defined(CONFIG_IA64_GENERIC) || defined(CONFIG_IA64_DIG))
 static enum vector_domain_type {
 	VECTOR_DOMAIN_NONE,
 	VECTOR_DOMAIN_PERCPU

-Tony

This one is a direct consequence of the new definition of NR_IRQS:

#define NR_IRQS (NR_VECTORS + 32 * NR_CPUS))

With a large NR_CPUS value, this gets too big. Do we really need to scale it with the number of cpus? I don't think this is the right thing to do. While large cpu count systems may also have a large number of I/O devices, the two parameters aren't strongly connected.

We could prevent it blowing up by doing:

/* NR_IRQS is limited by HARDIRQ_BITS */
#if (NR_VECTORS + 32 * NR_CPUS)) < 16363
#define NR_IRQS (NR_VECTORS + 32 * NR_CPUS))
#else
#define NR_IRQS 16383
#endif

But that looks rather ugly and still fails to build because of overflow of the percpu area. This gets big because of include/linux/kernel_stat.h:

DECLARE_PERCPU(struct kernel_stat, kstat);

With the current allocation of percpu stuff, it looks like we can push NR_IRQS up to around 7.5K, but that would leave no space for other additions to percpu space. So if large systems are going to need as many as 7.5K IRQs, then we'll also need to do something about kstat.

-Tony

Sorry. I'll search the whitespace in my patchset and fix them by posting take3 patchset for supporting vector domain.

I tested tiger_defconfig only. So I didn't notice this problem.
I confirmed that my patchset makes this problem and your following patch fixed it.

http://www.spinics.net/lists/linux-ia64/msg03352.html

Thanks tony,

I have never seen these messages. I will check them.

Regards,
Yasuaki Ishimatsu

I think if the number of CPUs is small, the value of NR_IRQS depends on the value of NR_CPUS. But, if the number of CPUs is large, this relation becomes weak. Because, in general machine composition, the number of I/O devices doesn't increase proportionally even if the number of CPUs increases. So, this threshold such as 16383 is too large. I'd like to define NR_IRQS as follows.

#if (NR_VECTORS + 32 * NR_CPUS)) < 1024
#define NR_IRQS (NR_VECTORS + 32 * NR_CPUS))
#else
#define NR_IRQS 1024
#endif

And, I will make NR_IRQS a tunable parameter which user can define as boot parameter.

Thanks,
Yasuaki Ishimatsu

Hi Tony,

I tried it on my 4-socket Montecito machine, but didn't happen. Could you send me your .config file and entire stack trace ?

Thanks,
Yasuaki Ishimatsu

This seems to be an intermittent problem. I just got my serial console working again to get you a full stack trace, and the first time I booted the kernel with your vector domain patch it didn't crash.

Second boot did crash. Here's the stack trace part of the console log:

CPU 9: synchronized ITC with CPU 0 (last diff 0 cycles, maxerr 159 cycles) CPU 10: synchronized ITC with CPU 0 (last diff 0 cycles, maxerr 40 cycles) CPU 11: synchronized ITC with CPU 0 (last diff 0 cycles, maxerr 161 cycles)

Unable to handle kernel paging request at virtual address a000008100974780
swapper[0]: Oops 8813272891392 [1]
Modules linked in:

Pid: 0, CPU 12, comm: swapper
psr : 0000101008022018 ifs : 8000000000000389 ip : [<a000000100011c50>] Not tainted
ip is at ia64_handle_irq+0x190/0x2a0
unat: 0000000000000000 pfs : 03e0000000000389 rsc : 0000000000000003
rnat: 0000000000000000 bsps: 0000000000000000 pr : 00000000000095a5
ldrs: 0000000000000000 ccv : 0000000000000fff fpsr: 0009804c0270033f csd : 0000000000000000 ssd : 0000000000000000
b0 : a000000100011b10 b6 : a000000100003320 b7 : e00000007fb1bde0 f6 : 000000000000000000000 f7 : 1003e0000000155557000 f8 : 000000000000000000000 f9 : 000000000000000000000 f10 : 000000000000000000000 f11 : 000000000000000000000 r1 : a000000100c2db30 r2 : 0000000000010000 r3 : 0000000000000000 r8 : 00000000000000fd r9 : 00000000000000fc r10 : 0000000000000000 r11 : 0009804c0270033f r12 : e000000180eafc60 r13 : e000000180ea0000 r14 : ffffffffffff4b00 r15 : ffffffffffff0428 r16 : 00000000ffffffff r17 : a000000100a45c58 r18 : e00000007fe131b0 r19 : e00000007fb1bde0 r20 : a000000100a4b6a0 r21 : a000000100970000 r22 : a000008100974780 r23 : 0000007fffffff80 r24 : a000000100974800 r25 : a000000100a2ef30 r26 : e0000001800c004c r27 : 000000000000004c r28 : e0000001800c0000 r29 : ffffffffffff0000 r30 : e0000001800d0000 r31 : a000000100a44ea0

Call Trace:
 [<a000000100012bb0>] show_stack+0x50/0xa0
                                spà00000180eaf830 bspà00000180ea0d28
 [<a000000100013480>] show_regs+0x820/0x840
                                spà00000180eafa00 bspà00000180ea0ce0
 [<a0000001000370c0>] die+0x1a0/0x280
                                spà00000180eafa00 bspà00000180ea0c98
 [<a000000100061230>] ia64_do_page_fault+0x810/0x900
                                spà00000180eafa00 bspà00000180ea0c38
 [<a00000010000bcc0>] ia64_leave_kernel+0x0/0x270
                                spà00000180eafa90 bspà00000180ea0c38
 [<a000000100011c50>] ia64_handle_irq+0x190/0x2a0
                                spà00000180eafc60 bspà00000180ea0be8
 [<a00000010000bcc0>] ia64_leave_kernel+0x0/0x270
                                spà00000180eafc60 bspà00000180ea0be8
 [<a0000001000541d0>] unlock_ipi_calllock+0x30/0x60
                                spà00000180eafe30 bspà00000180ea0bd0
 [<a000000100055fb0>] start_secondary+0x2d0/0x580
                                spà00000180eafe30 bspà00000180ea0b80
 [<a0000001000089e0>] __end_ivt_text+0x6c0/0x6f0
                                spà00000180eafe30 bspà00000180ea0b80
Kernel panic - not syncing: Aiee, killing interrupt handler!

The bit of ia64_handle_irq where we crashed looks like this:

a000000100011c30:       02 80 00 34 10 10       [MII]       ld4 r16=[r26]
a000000100011c36:       00 00 00 02 00 e0                   nop.i 0x0;;
a000000100011c3c:       02 c1 7d 53                         dep.z r23=r16,7,32
a000000100011c40:       03 00 00 00 01 00       [MII]       nop.m 0x0
a000000100011c46:       70 02 40 00 42 c0                   mov r39=r16;;
a000000100011c4c:       72 c1 00 80                         add r22=r23,r24;;
a000000100011c50:       0d 70 00 2c 18 10       [MFI]       ld8 r14=[r22]

r22 has the very bad value of 0xa000008100974780 ... so we die.

-Tony

One more note on that ... my test system often kicks out some

"Unexpected irq vector 0x13 on CPU xx!"

messages as it boots. Looking at the logs of previous boots, I see between zero and two such messages in the last few dozen boots, When the kernel with the vector-domain patch applied successfully booted this morning, this was one of the times where there were no unexpected interrupts.

Since the crash looks like it occurred while dereferencing the "desc = irq_desq + irq" in generic_handle_irq(), I think this is most likely related.

-Tony