Skip to contentSkip to conversation

take7: vector sharing (Large I/O system support)

PATCH Kenji Kaneshige 4 participants 9 messages Feb 28, 2005 – Mar 2, 2005

lore
Share & tools
lore
#0 lore raw

Hi Tony,

I had a good discussion about vector sharing patch with Ashok Raj (Thank you Ashok!!) and I made change to it based on the comments from him.

Summary of Changes are:
- Removed sharable flag from iosapic_itr_info structure
- Made NR_RTE_CACHE_ENTRIES configurable

Attached patch is against 2.6.11-rc5.

Thanks,
Kenji Kaneshige

Current ia64 linux cannot handle greater than 184 interrupt sources because of the lack of vectors. The following patch enables ia64 linux to handle greater than 184 interrupt sources by allowing the same vector number to be shared by multiple IOSAPIC's RTEs. The design of this patch is besed on "Intel(R) Itanium(R) Processor Family Interrupt Architecture Guide".

Even if you don't have a large I/O system, you can see the behavior of vector sharing by changing IOSAPIC_LAST_DEVICE_VECTOR to fewer value.

---

 linux-2.6.11-rc5-kanesige/arch/ia64/Kconfig           |    8 
 linux-2.6.11-rc5-kanesige/arch/ia64/kernel/iosapic.c  |  356 +++++++++++++-----
 linux-2.6.11-rc5-kanesige/arch/ia64/kernel/irq_ia64.c |   16 
 linux-2.6.11-rc5-kanesige/include/asm-ia64/hw_irq.h   |    1 
 4 files changed, 291 insertions(+), 90 deletions(-)
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~vector_sharing +8 -0
diff -puN arch/ia64/Kconfig~vector_sharing arch/ia64/Kconfig
@@ -318,6 +318,14 @@ config ACPI_DEALLOCATE_IRQ
 	depends on IOSAPIC && EXPERIMENTAL
 	default y
 
+config NR_RTE_CACHES
+	int "Number of RTE cache entries"
+	depends on IOSAPIC
+	default "256"
+	help
+	  If your system panic with the message "out of rte cache
+	  entries!(shortage: XX)", please set this to a larger value.
+
 source "drivers/firmware/Kconfig"
 
 source "fs/Kconfig.binfmt"
arch/ia64/kernel/iosapic.c renamed from arch/ia64/kernel/iosapic.c~vector_sharing +269 -87

571 lines not on this page · raw message

arch/ia64/kernel/irq_ia64.c renamed from arch/ia64/kernel/irq_ia64.c~vector_sharing +13 -3
diff -puN arch/ia64/kernel/irq_ia64.c~vector_sharing arch/ia64/kernel/irq_ia64.c
@@ -63,20 +63,30 @@ EXPORT_SYMBOL(isa_irq_to_vector_map);
 static unsigned long ia64_vector_mask[BITS_TO_LONGS(IA64_NUM_DEVICE_VECTORS)];
 
 int
-assign_irq_vector (int irq)
+assign_irq_vector_nopanic (int irq)
 {
 	int pos, vector;
  again:
 	pos = find_first_zero_bit(ia64_vector_mask, IA64_NUM_DEVICE_VECTORS);
 	vector = IA64_FIRST_DEVICE_VECTOR + pos;
 	if (vector > IA64_LAST_DEVICE_VECTOR)
-		/* XXX could look for sharable vectors instead of panic'ing... */
-		panic("assign_irq_vector: out of interrupt vectors!");
+		return -1;
 	if (test_and_set_bit(pos, ia64_vector_mask))
 		goto again;
 	return vector;
 }
 
+int
+assign_irq_vector (int irq)
+{
+	int vector = assign_irq_vector_nopanic(irq);
+
+	if (vector < 0)
+		panic("assign_irq_vector: out of interrupt vectors!");
+
+	return vector;
+}
+
 void
 free_irq_vector (int vector)
 {
include/asm-ia64/hw_irq.h renamed from include/asm-ia64/hw_irq.h~vector_sharing +1 -0
diff -puN include/asm-ia64/hw_irq.h~vector_sharing include/asm-ia64/hw_irq.h
@@ -81,6 +81,7 @@ extern __u8 isa_irq_to_vector_map[16];
 
 extern struct hw_interrupt_type irq_type_ia64_lsapic;	/* CPU-internal interrupt controller */
 
+extern int assign_irq_vector_nopanic (int irq); /* allocate a free vector without panic */
 extern int assign_irq_vector (int irq);	/* allocate a free vector */
 extern void free_irq_vector (int vector);
 extern void ia64_send_ipi (int cpu, int vector, int delivery_mode, int redirect);

_

#1 lore raw
Quoted 41 lines · Kenji Kaneshige

Hi Tony,

I had a good discussion about vector sharing patch with Ashok Raj (Thank you Ashok!!) and I made change to it based on the comments from him.

Summary of Changes are:
- Removed sharable flag from iosapic_itr_info structure
- Made NR_RTE_CACHE_ENTRIES configurable

Attached patch is against 2.6.11-rc5.

Thanks,
Kenji Kaneshige

Current ia64 linux cannot handle greater than 184 interrupt sources because of the lack of vectors. The following patch enables ia64 linux to handle greater than 184 interrupt sources by allowing the same vector number to be shared by multiple IOSAPIC's RTEs. The design of this patch is besed on "Intel(R) Itanium(R) Processor Family Interrupt Architecture Guide".

Even if you don't have a large I/O system, you can see the behavior of vector sharing by changing IOSAPIC_LAST_DEVICE_VECTOR to fewer value.

---

 linux-2.6.11-rc5-kanesige/arch/ia64/Kconfig           |    8 
 linux-2.6.11-rc5-kanesige/arch/ia64/kernel/iosapic.c  |  356 +++++++++++++-----
 linux-2.6.11-rc5-kanesige/arch/ia64/kernel/irq_ia64.c |   16 
 linux-2.6.11-rc5-kanesige/include/asm-ia64/hw_irq.h   |    1 
 4 files changed, 291 insertions(+), 90 deletions(-)

diff -puN arch/ia64/Kconfig~vector_sharing arch/ia64/Kconfig

arch/ia64/Kconfig config ACPI_DEALLOCATE_IRQ · line 327

Quoted 10 lines
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~vector_sharing +7 -0
@@ -318,6 +318,14 @@ config ACPI_DEALLOCATE_IRQ
 	depends on IOSAPIC && EXPERIMENTAL
 	default y
 
+config NR_RTE_CACHES
+	int "Number of RTE cache entries"
+	depends on IOSAPIC
+	default "256"
+	help
+	  If your system panic with the message "out of rte cache
+	  entries!(shortage: XX)", please set this to a larger value.

This screams for dynamic allocation of the underlying structures, no?

#2 lore raw
In reply to Kenji Kaneshige

Hi Cristoph,

Quoted 10 lines

arch/ia64/Kconfig config ACPI_DEALLOCATE_IRQ · line 327

Quoted 7 lines
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~vector_sharing +7 -0
+config NR_RTE_CACHES
+	int "Number of RTE cache entries"
+	depends on IOSAPIC
+	default "256"
+	help
+	  If your system panic with the message "out of rte cache
+	  entries!(shortage: XX)", please set this to a larger value.

This screams for dynamic allocation of the underlying structures, no?

Some number (NR_RTE_CACHES) of iosapic_rte_info structures are statically allocated because this structure might be needed before kmalloc is initialized. The panic message mentioned above means we run out of these statically allocated structures.

Thanks,
Kenji Kaneshige

Quoted 62 lines · Christoph Hellwig
Quoted 41 lines · Kenji Kaneshige

Hi Tony,

I had a good discussion about vector sharing patch with Ashok Raj (Thank you Ashok!!) and I made change to it based on the comments from him.

Summary of Changes are:
- Removed sharable flag from iosapic_itr_info structure
- Made NR_RTE_CACHE_ENTRIES configurable

Attached patch is against 2.6.11-rc5.

Thanks,
Kenji Kaneshige

Current ia64 linux cannot handle greater than 184 interrupt sources because of the lack of vectors. The following patch enables ia64 linux to handle greater than 184 interrupt sources by allowing the same vector number to be shared by multiple IOSAPIC's RTEs. The design of this patch is besed on "Intel(R) Itanium(R) Processor Family Interrupt Architecture Guide".

Even if you don't have a large I/O system, you can see the behavior of vector sharing by changing IOSAPIC_LAST_DEVICE_VECTOR to fewer value.

---

linux-2.6.11-rc5-kanesige/arch/ia64/Kconfig           |    8 
linux-2.6.11-rc5-kanesige/arch/ia64/kernel/iosapic.c  |  356 +++++++++++++-----
linux-2.6.11-rc5-kanesige/arch/ia64/kernel/irq_ia64.c |   16 
linux-2.6.11-rc5-kanesige/include/asm-ia64/hw_irq.h   |    1 
4 files changed, 291 insertions(+), 90 deletions(-)

diff -puN arch/ia64/Kconfig~vector_sharing arch/ia64/Kconfig

arch/ia64/Kconfig config ACPI_DEALLOCATE_IRQ · line 327

Quoted 10 lines
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~vector_sharing +7 -0
@@ -318,6 +318,14 @@ config ACPI_DEALLOCATE_IRQ
 	depends on IOSAPIC && EXPERIMENTAL
 	default y
 
+config NR_RTE_CACHES
+	int "Number of RTE cache entries"
+	depends on IOSAPIC
+	default "256"
+	help
+	  If your system panic with the message "out of rte cache
+	  entries!(shortage: XX)", please set this to a larger value.

This screams for dynamic allocation of the underlying structures, no?

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
In reply to Kenji Kaneshige
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Kenji Kaneshige

Kenji> Hi Cristoph,

Quoted 10 lines
Quoted 10 lines

arch/ia64/Kconfig config ACPI_DEALLOCATE_IRQ · line 327

Quoted 7 lines
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~vector_sharing +7 -0
+config NR_RTE_CACHES
+	int "Number of RTE cache entries"
+	depends on IOSAPIC
+	default "256"
+	help
+	  If your system panic with the message "out of rte cache
+	  entries!(shortage: XX)", please set this to a larger value.

This screams for dynamic allocation of the underlying structures, no?

Kenji> Some number (NR_RTE_CACHES) of iosapic_rte_info structures are Kenji> statically allocated because this structure might be needed before Kenji> kmalloc is initialized. The panic message mentioned above means we Kenji> run out of these statically allocated structures.

The problem is naming, I think. Those aren't "cache entries", they are statically preallocated entries. How about changing the config option name to:

config NR_PREALLOCATED_RTE_ENTRIES

and the help message to something along the lines of:

The I/O SAPIC code needs a small number of statically preallocated RTE entries so that it can work even before kmalloc has been initialized. The default value for this option should normally be sufficient, but if you get a panic of the form "out of preallocated RTE entries!", that's a sign that this value needs to be increased.

--david

#4 lore raw
In reply to Kenji Kaneshige
Quoted 14 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Kenji Kaneshige

Kenji> Hi Cristoph,

Quoted 10 lines
Quoted 10 lines

arch/ia64/Kconfig config ACPI_DEALLOCATE_IRQ · line 327

Quoted 7 lines
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~vector_sharing +7 -0
+config NR_RTE_CACHES
+	int "Number of RTE cache entries"
+	depends on IOSAPIC
+	default "256"
+	help
+	  If your system panic with the message "out of rte cache
+	  entries!(shortage: XX)", please set this to a larger value.

This screams for dynamic allocation of the underlying structures, no?

Quoted 8 lines

Kenji> Some number (NR_RTE_CACHES) of iosapic_rte_info structures are Kenji> statically allocated because this structure might be needed before Kenji> kmalloc is initialized. The panic message mentioned above means we Kenji> run out of these statically allocated structures.

The problem is naming, I think. Those aren't "cache entries", they are statically preallocated entries. How about changing the config option name to:

Even for early allocations what speaks against alloc_bootmem?

#5 lore raw
In reply to Kenji Kaneshige
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line · Christoph Hellwig

The problem is naming, I think. Those aren't "cache entries", they are statically preallocated entries. How about changing the config option name to:

Christoph> Even for early allocations what speaks against alloc_bootmem?

I don't recall anymore. Last time I checked, it wasn't possible, but that was a _very_ long time ago.

	--david
#6 lore raw
In reply to Kenji Kaneshige
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines

The problem is naming, I think. Those aren't "cache entries", they are statically preallocated entries. How about changing the config option name to:

Even for early allocations what speaks against alloc_bootmem?

I didn't use alloc_bootmem because I thought alloc_bootmem was very simple allocator and we should not use it to allocate arbitrary size (not PAGE_SIZE * n) of memory so frequently. But I don't know about it very much and my understanding might be incorrect.

Can I use alloc_bootmem as well as kmalloc to allocate arbitrary size of memory?

Thanks,
Kenji Kaneshige

#7 lore raw
In reply to Kenji Kaneshige
Quoted 11 lines · Kenji Kaneshige
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines

The problem is naming, I think. Those aren't "cache entries", they are statically preallocated entries. How about changing the config option name to:

Even for early allocations what speaks against alloc_bootmem?

Quoted 8 lines

I didn't use alloc_bootmem because I thought alloc_bootmem was very simple allocator and we should not use it to allocate arbitrary size (not PAGE_SIZE * n) of memory so frequently. But I don't know about it very much and my understanding might be incorrect.

Can I use alloc_bootmem as well as kmalloc to allocate arbitrary size of memory?

alloc_bootmem is pretty stupid so, yes you have to be carefull to avoid too much wastage.

#8 lore raw
In reply to Kenji Kaneshige
Quoted 2 lines

Can I use alloc_bootmem as well as kmalloc to allocate arbitrary size of memory?

The decision on which to use is based on when in the boot sequence you need to do the allocation. Before "free_all_bootmem()" [or on NUMA systems "free_all_bootmem_node()"] you must use alloc_bootmem(), afterwards you must use kmalloc() [or vmalloc(), or alloc_pages()].

If the features of the allocation you need to make demand the semantics of a particular allocator, then you will have to fix your code to make the allocation during a period of the boot sequence when the allocator you want to use is valid.

alloc_bootmem() is not so good at small allocations (especially of objects that you plan to free again) ... for very large allocations it may be the only choice (as it gets called before fragmentation sets in).

-Tony