Add config option for batched hcalls
Subsystems (2)
PATCH 1/1 Add config option for batched hcalls
Add a config option for the (batched) MULTITCE and BULK_REMOVE h-calls.
By default, these options are on and are beneficial for performance and
throughput reasons. If disabled, the code will fall back to using less
optimal TCE and REMOVE hcalls. The ability to easily disable these
options is useful for some of the PREEMPT_RT related investigation and work occurring on Power.
---
arch/powerpc/platforms/pseries/Kconfig +20 -0
@@ -81,3 +81,23 @@ config DTL which are accessible through a debugfs file. Say N if you are unsure. + +config BULK_REMOVE + bool "Enable BULK_REMOVE" + depends on PPC_PSERIES + default y + help + Enable the BULK_REMOVE option for the hash page code. + This relies on a "hcall-bulk" firmware feature, and + should be enabled for performance throughput. + +config MULTITCE + bool "Enable MultiTCE" + depends on PPC_PSERIES + default y + help + Enable the Multi-TCE code, allowing a single hcall to + update multiple TCE entries at one time. This relies + on a "hcall-multi-tce" firmware feature, and should be + enabled for performance throughput. +
arch/powerpc/platforms/pseries/firmware.c +4 -0
@@ -51,9 +51,13 @@ firmware_features_table[FIRMWARE_MAX_FEATURES] = { {FW_FEATURE_VIO, "hcall-vio"}, {FW_FEATURE_RDMA, "hcall-rdma"}, {FW_FEATURE_LLAN, "hcall-lLAN"}, +#if defined(CONFIG_BULK_REMOVE) {FW_FEATURE_BULK_REMOVE, "hcall-bulk"}, +#endif {FW_FEATURE_XDABR, "hcall-xdabr"}, +#if defined(CONFIG_MULTITCE) {FW_FEATURE_MULTITCE, "hcall-multi-tce"}, +#endif {FW_FEATURE_SPLPAR, "hcall-splpar"}, };
/* Build up the firmware features bitmask using the contents of
Quoted 8 lines · Will Schmidt
Add a config option for the (batched) MULTITCE and
BULK_REMOVEh-calls.By default, these options are on and are beneficial for performance and
throughput reasons. If disabled, the code will fall back to using less
optimal TCE and REMOVE hcalls. The ability to easily disable these
options is useful for some of thePREEMPT_RTrelated investigation and work occurring on Power.
Hi,
I can see why it's useful to enable and disable, but these are all runtime-checked, wouldn't it be more useful to add a bootarg to handle it instead of adding some new config options that pretty much everyone will always go with the defaults on?
The bits are set early, but from looking at where they're used, there doesn't seem to be any harm in disabling them later on when a bootarg is convenient to parse and deal with?
It has the benefit of easier on/off testing, if that has any value for production debug down the road.
-Olof
Quoted 17 lines · Olof Johansson
Quoted 8 lines · Will Schmidt
Add a config option for the (batched) MULTITCE and
BULK_REMOVEh-calls.By default, these options are on and are beneficial for performance and
throughput reasons. If disabled, the code will fall back to using less
optimal TCE and REMOVE hcalls. The ability to easily disable these
options is useful for some of thePREEMPT_RTrelated investigation and work occurring on Power.Hi,
I can see why it's useful to enable and disable, but these are all runtime-checked, wouldn't it be more useful to add a bootarg to handle it instead of adding some new config options that pretty much everyone will always go with the defaults on?
Quoted 6 lines
The bits are set early, but from looking at where they're used, there doesn't seem to be any harm in disabling them later on when a bootarg is convenient to parse and deal with?
It has the benefit of easier on/off testing, if that has any value for production debug down the road.
Hi Olof,
Thats a good idea, let me poke at this a bit more, see if I can get
bootargs for this.
Thanks,
-Will
Quoted 4 lines
-Olof
PATCH v2 1/1 Add kernel parameter to disable batched hcalls
In reply to Olof JohanssonThis introduces a pair of kernel parameters that can be used to disable the MULTITCE and BULK_REMOVE h-calls.
By default, those hcalls are enabled, active, and good for throughput and performance. The ability to disable them will be useful for some of the PREEMPT_RT related investigation and work occurring on Power.
---
v2 - Per feedback from Olof, the code is reworked to utilize kernel parameter runtime checks, rather than CONFIG options.
- Added relevant change to kernel-parameters.txt
Documentation/kernel-parameters.txt +8 -0
@@ -426,6 +426,10 @@ and is between 256 and 4096 characters. It is defined in the file bttv.pll= See Documentation/video4linux/bttv/Insmod-options bttv.tuner= and Documentation/video4linux/bttv/CARDLIST + bulk_remove=off [PPC] This parameter disables the use of the pSeries + firmware feature for flushing multiple hpte entries + at a time. + BusLogic= [HW,SCSI] See drivers/scsi/BusLogic.c, comment before function BusLogic_ParseDriverOptions().
@@ -1499,6 +1503,10 @@ and is between 256 and 4096 characters. It is defined in the file mtdparts= [MTD] See drivers/mtd/cmdlinepart.c. + multitce=off [PPC] This parameter disables the use of the pSeries + firmware feature for updating multiple TCE entries + at a time. + onenand.bdry= [HW,MTD] Flex-OneNAND Boundary Configuration Format: [die0_boundary][,die0_lock][,die1_boundary][,die1_lock]
arch/powerpc/platforms/pseries/iommu.c +16 -0
@@ -625,3 +625,19 @@ void iommu_init_early_pSeries(void)set_pci_dma_ops (&dma_iommu_ops); } +static int __init disable_multitce (char *str) +{ +if (strcmp (str,"off" )==0 ) { +if (firmware_has_feature (FW_FEATURE_LPAR )) { +if (firmware_has_feature (FW_FEATURE_MULTITCE )) { +printk (KERN_INFO "Disabling MULTITCE firmware feature\n" ); + ppc_md.tce_build = tce_build_pSeriesLP; + ppc_md.tce_free = tce_free_pSeriesLP; + powerpc_firmware_features &= ~FW_FEATURE_MULTITCE ; + } + } + } +return 1 ; +} + +__setup ("multitce=" ,disable_multitce);
arch/powerpc/platforms/pseries/lpar.c +13 -0
@@ -599,6 +599,19 @@ static void pSeries_lpar_flush_hash_range(unsigned long number, int local)spin_unlock_irqrestore (&pSeries_lpar_tlbie_lock, flags); } +static int __init disable_bulk_remove (char *str) +{ +if (strcmp (str,"off" )==0 ) { +if (firmware_has_feature (FW_FEATURE_BULK_REMOVE )) { +printk (KERN_INFO "Disabling BULK_REMOVE firmware feature" ); + powerpc_firmware_features &= ~FW_FEATURE_BULK_REMOVE ; + } + } +return 1 ; +} + +__setup ("bulk_remove=" ,disable_bulk_remove); +void __init hpte_init_lpar (void ) { ppc_md.hpte_invalidate = pSeries_lpar_hpte_invalidate;
Re: [PATCH 1/1 v2 ] Add kernel parameter to disable batched hcalls
Nice. I've got minor nits below, and you might also want to run the patch through checkpatch and fix up some of the whitespace warnings.
-Olof
Quoted 58 lines · Will Schmidt
This introduces a pair of kernel parameters that can be used to disable the MULTITCE and
BULK_REMOVEh-calls.By default, those hcalls are enabled, active, and good for throughput and performance. The ability to disable them will be useful for some of the
PREEMPT_RTrelated investigation and work occurring on Power.Signed-off-by Will Schmidt <[email protected]>cc Olof Johansson <[email protected]>cc Anton Blanchard <[email protected]>cc Benjamin Herrenschmidt <[email protected]>---
v2 - Per feedback from Olof, the code is reworked to utilize kernel parameter runtime checks, rather than CONFIG options.
- Added relevant change to kernel-parameters.txtDocumentation/kernel-parameters.txt +8
-0@@ -426,6 +426,10 @@ and is between 256 and 4096 characters. It is defined in the file bttv.pll= See Documentation/video4linux/bttv/Insmod-options bttv.tuner= and Documentation/video4linux/bttv/CARDLIST + bulk_remove=off [PPC] This parameter disables the use of the pSeries + firmware feature for flushing multiple hpte entries + at a time. + BusLogic= [HW,SCSI] See drivers/scsi/BusLogic.c, comment before function BusLogic_ParseDriverOptions().@@ -1499,6 +1503,10 @@ and is between 256 and 4096 characters. It is defined in the file mtdparts= [MTD] See drivers/mtd/cmdlinepart.c. + multitce=off [PPC] This parameter disables the use of the pSeries + firmware feature for updating multiple TCE entries + at a time. + onenand.bdry= [HW,MTD] Flex-OneNAND Boundary Configuration Format: [die0_boundary][,die0_lock][,die1_boundary][,die1_lock]arch/powerpc/platforms/pseries/iommu.c +2
-0@@ -625,3 +625,19 @@ void iommu_init_early_pSeries(void)set_pci_dma_ops (&dma_iommu_ops); } +static int __init disable_multitce (char *str) +{
arch/powerpc/platforms/pseries/iommu.c void iommu_init_early_pSeries(void) · line 639
Quoted 10 lines
arch/powerpc/platforms/pseries/iommu.c +10
-0+if (strcmp (str,"off" )==0 ) { +if (firmware_has_feature (FW_FEATURE_LPAR )) { +if (firmware_has_feature (FW_FEATURE_MULTITCE )) { +printk (KERN_INFO "Disabling MULTITCE firmware feature\n" ); + ppc_md.tce_build = tce_build_pSeriesLP; + ppc_md.tce_free = tce_free_pSeriesLP; + powerpc_firmware_features &= ~FW_FEATURE_MULTITCE ; + } + } + }
I personally prefer to keep cases like these in one if statement to save indentation:
if (strcmp(str, "off") == 0 &&
firmware_has_feature(FW_FEATURE_LPAR) &&
firmware_has_feature(FW_FEATURE_MULTITCE)) {
<...>
}Quoted 10 lines
arch/powerpc/platforms/pseries/iommu.c +4
-0+return 1 ; +} + +__setup ("multitce=" ,disable_multitce);arch/powerpc/platforms/pseries/lpar.c +0
-0@@ -599,6 +599,19 @@ static void pSeries_lpar_flush_hash_range(unsigned long number, int local)spin_unlock_irqrestore (&pSeries_lpar_tlbie_lock, flags);
arch/powerpc/platforms/pseries/lpar.c static void pSeries_lpar_flush_hash_range(unsigned long number, int local) · line 609
Quoted 10 lines
arch/powerpc/platforms/pseries/lpar.c +8
-0} +static int __init disable_bulk_remove (char *str) +{ +if (strcmp (str,"off" )==0 ) { +if (firmware_has_feature (FW_FEATURE_BULK_REMOVE )) { +printk (KERN_INFO "Disabling BULK_REMOVE firmware feature" ); + powerpc_firmware_features &= ~FW_FEATURE_BULK_REMOVE ; + } + }
Same here.
Quoted 9 lines
arch/powerpc/platforms/pseries/lpar.c +5
-0+return 1 ; +} + +__setup ("bulk_remove=" ,disable_bulk_remove); +void __init hpte_init_lpar (void ) { ppc_md.hpte_invalidate = pSeries_lpar_hpte_invalidate;
Re: [PATCH 1/1 v3] Add kernel parameter to disable batched hcalls
This introduces a pair of kernel parameters that can be used to disable the MULTITCE and BULK_REMOVE h-calls.
By default, those hcalls are enabled, active, and good for throughput and performance. The ability to disable them will be useful for some of the PREEMPT_RT related investigation and work occurring on Power.
---
v2 - Per feedback from Olof, the code is reworked to utilize kernel
parameter runtime checks, rather than CONFIG options.
- Added relevant change to kernel-parameters.txt
v3 - ran through checkpatch and fixed whitespace issues.
- consolidated the if() staircases to save indentation.
Documentation/kernel-parameters.txt +8 -0
@@ -426,6 +426,10 @@ and is between 256 and 4096 characters. It is defined in the file bttv.pll= See Documentation/video4linux/bttv/Insmod-options bttv.tuner= and Documentation/video4linux/bttv/CARDLIST + bulk_remove=off [PPC] This parameter disables the use of the pSeries + firmware feature for flushing multiple hpte entries + at a time. + BusLogic= [HW,SCSI] See drivers/scsi/BusLogic.c, comment before function BusLogic_ParseDriverOptions().
@@ -1499,6 +1503,10 @@ and is between 256 and 4096 characters. It is defined in the file mtdparts= [MTD] See drivers/mtd/cmdlinepart.c. + multitce=off [PPC] This parameter disables the use of the pSeries + firmware feature for updating multiple TCE entries + at a time. + onenand.bdry= [HW,MTD] Flex-OneNAND Boundary Configuration Format: [die0_boundary][,die0_lock][,die1_boundary][,die1_lock]
arch/powerpc/platforms/pseries/iommu.c +14 -0
@@ -625,3 +625,17 @@ void iommu_init_early_pSeries(void)set_pci_dma_ops (&dma_iommu_ops); } +static int __init disable_multitce (char *str) +{ +if (strcmp (str,"off" ) ==0 && +firmware_has_feature (FW_FEATURE_LPAR ) && +firmware_has_feature (FW_FEATURE_MULTITCE )) { +printk (KERN_INFO "Disabling MULTITCE firmware feature\n" ); + ppc_md.tce_build = tce_build_pSeriesLP; + ppc_md.tce_free = tce_free_pSeriesLP; + powerpc_firmware_features &= ~FW_FEATURE_MULTITCE ; + } +return 1 ; +} + +__setup ("multitce=" , disable_multitce);
arch/powerpc/platforms/pseries/lpar.c +12 -0
@@ -599,6 +599,18 @@ static void pSeries_lpar_flush_hash_range(unsigned long number, int local)spin_unlock_irqrestore (&pSeries_lpar_tlbie_lock, flags); } +static int __init disable_bulk_remove (char *str) +{ +if (strcmp (str,"off" ) ==0 && +firmware_has_feature (FW_FEATURE_BULK_REMOVE )) { +printk (KERN_INFO "Disabling BULK_REMOVE firmware feature" ); + powerpc_firmware_features &= ~FW_FEATURE_BULK_REMOVE ; + } +return 1 ; +} + +__setup ("bulk_remove=" , disable_bulk_remove); +void __init hpte_init_lpar (void ) { ppc_md.hpte_invalidate = pSeries_lpar_hpte_invalidate;