take7: vector sharing (Large I/O system support)
Hi Tony,
I had a good discussion about vector sharing patch with Ashok Raj (Thank you Ashok!!) and I made change to it based on the comments from him.
Summary of Changes are:
- Removed sharable flag from iosapic_itr_info structure
- Made NR_RTE_CACHE_ENTRIES configurable
Attached patch is against 2.6.11-rc5.
Thanks,
Kenji Kaneshige
Current ia64 linux cannot handle greater than 184 interrupt sources because of the lack of vectors. The following patch enables ia64 linux to handle greater than 184 interrupt sources by allowing the same vector number to be shared by multiple IOSAPIC's RTEs. The design of this patch is besed on "Intel(R) Itanium(R) Processor Family Interrupt Architecture Guide".
Even if you don't have a large I/O system, you can see the behavior of vector sharing by changing IOSAPIC_LAST_DEVICE_VECTOR to fewer value.
---
linux-2.6.11-rc5-kanesige/arch/ia64/Kconfig | 8 linux-2.6.11-rc5-kanesige/arch/ia64/kernel/iosapic.c | 356 +++++++++++++----- linux-2.6.11-rc5-kanesige/arch/ia64/kernel/irq_ia64.c | 16 linux-2.6.11-rc5-kanesige/include/asm-ia64/hw_irq.h | 1 4 files changed, 291 insertions(+), 90 deletions(-)
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~vector_sharing +8 -0
@@ -318,6 +318,14 @@ config ACPI_DEALLOCATE_IRQ depends on IOSAPIC && EXPERIMENTAL default y +config NR_RTE_CACHES + int "Number of RTE cache entries" + depends on IOSAPIC + default "256" + help + If your system panic with the message "out of rte cache + entries!(shortage: XX)", please set this to a larger value. + source "drivers/firmware/Kconfig" source "fs/Kconfig.binfmt"
arch/ia64/kernel/iosapic.c renamed from arch/ia64/kernel/iosapic.c~vector_sharing +269 -87
571 lines not on this page · raw message
arch/ia64/kernel/irq_ia64.c renamed from arch/ia64/kernel/irq_ia64.c~vector_sharing +13 -3
@@ -63,20 +63,30 @@ EXPORT_SYMBOL(isa_irq_to_vector_map);static unsigned long ia64_vector_mask[BITS_TO_LONGS (IA64_NUM_DEVICE_VECTORS )];int -assign_irq_vector (int irq) +assign_irq_vector_nopanic (int irq) {int pos, vector; again: pos =find_first_zero_bit (ia64_vector_mask,IA64_NUM_DEVICE_VECTORS ); vector =IA64_FIRST_DEVICE_VECTOR + pos;if (vector >IA64_LAST_DEVICE_VECTOR ) -/* XXX could look for sharable vectors instead of panic'ing... */ -panic ("assign_irq_vector: out of interrupt vectors!" ); +return -1 ;if (test_and_set_bit (pos, ia64_vector_mask)) goto again;return vector; } +int +assign_irq_vector (int irq) +{ +int vector =assign_irq_vector_nopanic (irq); + +if (vector <0 ) +panic ("assign_irq_vector: out of interrupt vectors!" ); + +return vector; +} +void free_irq_vector (int vector) {
include/asm-ia64/hw_irq.h renamed from include/asm-ia64/hw_irq.h~vector_sharing +1 -0
@@ -81,6 +81,7 @@ extern __u8 isa_irq_to_vector_map[16];extern struct hw_interrupt_type irq_type_ia64_lsapic;/* CPU-internal interrupt controller */ +extern int assign_irq_vector_nopanic (int irq);/* allocate a free vector without panic */ extern int assign_irq_vector (int irq);/* allocate a free vector */ extern void free_irq_vector (int vector);extern void ia64_send_ipi (int cpu,int vector,int delivery_mode,int redirect);
_
Quoted 41 lines · Kenji Kaneshige
Hi Tony,
I had a good discussion about vector sharing patch with Ashok Raj (Thank you Ashok!!) and I made change to it based on the comments from him.
Summary of Changes are:
- Removed sharable flag fromiosapic_itr_infostructure
- MadeNR_RTE_CACHE_ENTRIESconfigurableAttached patch is against 2.6.11-rc5.
Thanks,
Kenji KaneshigeCurrent ia64 linux cannot handle greater than 184 interrupt sources because of the lack of vectors. The following patch enables ia64 linux to handle greater than 184 interrupt sources by allowing the same vector number to be shared by multiple IOSAPIC's RTEs. The design of this patch is besed on "Intel(R) Itanium(R) Processor Family Interrupt Architecture Guide".
Even if you don't have a large I/O system, you can see the behavior of vector sharing by changing
IOSAPIC_LAST_DEVICE_VECTORto fewer value.Signed-off-by Kenji Kaneshige <[email protected]>---
linux-2.6.11-rc5-kanesige/arch/ia64/Kconfig | 8 linux-2.6.11-rc5-kanesige/arch/ia64/kernel/iosapic.c | 356 +++++++++++++----- linux-2.6.11-rc5-kanesige/arch/ia64/kernel/irq_ia64.c | 16 linux-2.6.11-rc5-kanesige/include/asm-ia64/hw_irq.h | 1 4 files changed, 291 insertions(+), 90 deletions(-)diff -puN
arch/ia64/Kconfig~vector_sharingarch/ia64/Kconfig
arch/ia64/Kconfig config ACPI_DEALLOCATE_IRQ · line 327
Quoted 10 lines
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~vector_sharing +7
-0@@ -318,6 +318,14 @@ config ACPI_DEALLOCATE_IRQ depends on IOSAPIC && EXPERIMENTAL default y +config NR_RTE_CACHES + int "Number of RTE cache entries" + depends on IOSAPIC + default "256" + help + If your system panic with the message "out of rte cache + entries!(shortage: XX)", please set this to a larger value.
This screams for dynamic allocation of the underlying structures, no?
Hi Cristoph,
Quoted 10 lines
arch/ia64/Kconfig config ACPI_DEALLOCATE_IRQ · line 327
Quoted 7 lines
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~vector_sharing +7
-0+config NR_RTE_CACHES + int "Number of RTE cache entries" + depends on IOSAPIC + default "256" + help + If your system panic with the message "out of rte cache + entries!(shortage: XX)", please set this to a larger value.This screams for dynamic allocation of the underlying structures, no?
Some number (NR_RTE_CACHES) of iosapic_rte_info structures are statically allocated because this structure might be needed before kmalloc is initialized. The panic message mentioned above means we run out of these statically allocated structures.
Thanks,
Kenji Kaneshige
Quoted 62 lines · Christoph Hellwig
Quoted 41 lines · Kenji Kaneshige
Hi Tony,
I had a good discussion about vector sharing patch with Ashok Raj (Thank you Ashok!!) and I made change to it based on the comments from him.
Summary of Changes are:
- Removed sharable flag fromiosapic_itr_infostructure
- MadeNR_RTE_CACHE_ENTRIESconfigurableAttached patch is against 2.6.11-rc5.
Thanks,
Kenji KaneshigeCurrent ia64 linux cannot handle greater than 184 interrupt sources because of the lack of vectors. The following patch enables ia64 linux to handle greater than 184 interrupt sources by allowing the same vector number to be shared by multiple IOSAPIC's RTEs. The design of this patch is besed on "Intel(R) Itanium(R) Processor Family Interrupt Architecture Guide".
Even if you don't have a large I/O system, you can see the behavior of vector sharing by changing
IOSAPIC_LAST_DEVICE_VECTORto fewer value.Signed-off-by Kenji Kaneshige <[email protected]>---
linux-2.6.11-rc5-kanesige/arch/ia64/Kconfig | 8 linux-2.6.11-rc5-kanesige/arch/ia64/kernel/iosapic.c | 356 +++++++++++++----- linux-2.6.11-rc5-kanesige/arch/ia64/kernel/irq_ia64.c | 16 linux-2.6.11-rc5-kanesige/include/asm-ia64/hw_irq.h | 1 4 files changed, 291 insertions(+), 90 deletions(-)diff -puN
arch/ia64/Kconfig~vector_sharingarch/ia64/Kconfigarch/ia64/Kconfig config ACPI_DEALLOCATE_IRQ · line 327
Quoted 10 lines
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~vector_sharing +7
-0@@ -318,6 +318,14 @@ config ACPI_DEALLOCATE_IRQ depends on IOSAPIC && EXPERIMENTAL default y +config NR_RTE_CACHES + int "Number of RTE cache entries" + depends on IOSAPIC + default "256" + help + If your system panic with the message "out of rte cache + entries!(shortage: XX)", please set this to a larger value.This screams for dynamic allocation of the underlying structures, no?
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Kenji Kaneshige
Kenji> Hi Cristoph,
Quoted 10 lines
Quoted 10 lines
arch/ia64/Kconfig config ACPI_DEALLOCATE_IRQ · line 327
Quoted 7 lines
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~vector_sharing +7
-0+config NR_RTE_CACHES + int "Number of RTE cache entries" + depends on IOSAPIC + default "256" + help + If your system panic with the message "out of rte cache + entries!(shortage: XX)", please set this to a larger value.This screams for dynamic allocation of the underlying structures, no?
Kenji> Some number (NR_RTE_CACHES) of iosapic_rte_info structures are Kenji> statically allocated because this structure might be needed before Kenji> kmalloc is initialized. The panic message mentioned above means we Kenji> run out of these statically allocated structures.
The problem is naming, I think. Those aren't "cache entries", they are statically preallocated entries. How about changing the config option name to:
config NR_PREALLOCATED_RTE_ENTRIES
and the help message to something along the lines of:
The I/O SAPIC code needs a small number of statically preallocated RTE entries so that it can work even before kmalloc has been initialized. The default value for this option should normally be sufficient, but if you get a panic of the form "out of preallocated RTE entries!", that's a sign that this value needs to be increased.
--david
Quoted 14 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Kenji Kaneshige
Kenji> Hi Cristoph,
Quoted 10 lines
Quoted 10 lines
arch/ia64/Kconfig config ACPI_DEALLOCATE_IRQ · line 327
Quoted 7 lines
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~vector_sharing +7
-0+config NR_RTE_CACHES + int "Number of RTE cache entries" + depends on IOSAPIC + default "256" + help + If your system panic with the message "out of rte cache + entries!(shortage: XX)", please set this to a larger value.This screams for dynamic allocation of the underlying structures, no?
Quoted 8 lines
Kenji> Some number (
NR_RTE_CACHES) ofiosapic_rte_infostructures are Kenji> statically allocated because this structure might be needed before Kenji> kmalloc is initialized. The panic message mentioned above means we Kenji> run out of these statically allocated structures.The problem is naming, I think. Those aren't "cache entries", they are statically preallocated entries. How about changing the config option name to:
Even for early allocations what speaks against alloc_bootmem?
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line · Christoph Hellwig
The problem is naming, I think. Those aren't "cache entries", they are statically preallocated entries. How about changing the config option name to:
Christoph> Even for early allocations what speaks against alloc_bootmem?
I don't recall anymore. Last time I checked, it wasn't possible, but that was a _very_ long time ago.
--david
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines
The problem is naming, I think. Those aren't "cache entries", they are statically preallocated entries. How about changing the config option name to:
Even for early allocations what speaks against
alloc_bootmem?
I didn't use alloc_bootmem because I thought alloc_bootmem was very simple allocator and we should not use it to allocate arbitrary size (not PAGE_SIZE * n) of memory so frequently. But I don't know about it very much and my understanding might be incorrect.
Can I use alloc_bootmem as well as kmalloc to allocate arbitrary size of memory?
Thanks,
Kenji Kaneshige
Quoted 11 lines · Kenji Kaneshige
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines
The problem is naming, I think. Those aren't "cache entries", they are statically preallocated entries. How about changing the config option name to:
Even for early allocations what speaks against
alloc_bootmem?
Quoted 8 lines
I didn't use
alloc_bootmembecause I thoughtalloc_bootmemwas very simple allocator and we should not use it to allocate arbitrary size (notPAGE_SIZE* n) of memory so frequently. But I don't know about it very much and my understanding might be incorrect.Can I use
alloc_bootmemas well as kmalloc to allocate arbitrary size of memory?
alloc_bootmem is pretty stupid so, yes you have to be carefull to avoid too much wastage.
Quoted 2 lines
Can I use
alloc_bootmemas well as kmalloc to allocate arbitrary size of memory?
The decision on which to use is based on when in the boot sequence you need to do the allocation. Before "free_all_bootmem()" [or on NUMA systems "free_all_bootmem_node()"] you must use alloc_bootmem(), afterwards you must use kmalloc() [or vmalloc(), or alloc_pages()].
If the features of the allocation you need to make demand the semantics of a particular allocator, then you will have to fix your code to make the allocation during a period of the boot sequence when the allocator you want to use is valid.
alloc_bootmem() is not so good at small allocations (especially of objects that you plan to free again) ... for very large allocations it may be the only choice (as it gets called before fragmentation sets in).
-Tony