Skip to contentSkip to conversation

Add config option for batched hcalls

PATCH In review Will Schmidt 2 participants 6 messages Sep 24, 2010 – Sep 29, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Add config option for batched hcalls

Add a config option for the (batched) MULTITCE and BULK_REMOVE h-calls.

By default, these options are on and are beneficial for performance and
throughput reasons. If disabled, the code will fall back to using less
optimal TCE and REMOVE hcalls. The ability to easily disable these
options is useful for some of the PREEMPT_RT related investigation and work occurring on Power.

cc Anton Blanchard <[email protected]>
cc Benjamin Herrenschmidt <[email protected]>

---

arch/powerpc/platforms/pseries/Kconfig +20 -0
index f0e6f28..0b5e6a9 100644
@@ -81,3 +81,23 @@ config DTL
 	  which are accessible through a debugfs file.
 
 	  Say N if you are unsure.
+
+config BULK_REMOVE
+	bool "Enable BULK_REMOVE"
+	depends on PPC_PSERIES
+	default y
+	help
+	  Enable the BULK_REMOVE option for the hash page code.
+	  This relies on a "hcall-bulk" firmware feature, and
+	  should be enabled for performance throughput.
+
+config MULTITCE
+	bool "Enable MultiTCE"
+	depends on PPC_PSERIES
+	default y
+	help
+	  Enable the Multi-TCE code, allowing a single hcall to
+	  update multiple TCE entries at one time.  This relies
+	  on a "hcall-multi-tce" firmware feature, and should be
+	  enabled for performance throughput.
+
arch/powerpc/platforms/pseries/firmware.c +4 -0
index 0a4d8c..4327064 100644
@@ -51,9 +51,13 @@ firmware_features_table[FIRMWARE_MAX_FEATURES] = {
 	{FW_FEATURE_VIO,		"hcall-vio"},
 	{FW_FEATURE_RDMA,		"hcall-rdma"},
 	{FW_FEATURE_LLAN,		"hcall-lLAN"},
+#if defined(CONFIG_BULK_REMOVE)
 	{FW_FEATURE_BULK_REMOVE,	"hcall-bulk"},
+#endif
 	{FW_FEATURE_XDABR,		"hcall-xdabr"},
+#if defined(CONFIG_MULTITCE)
 	{FW_FEATURE_MULTITCE,		"hcall-multi-tce"},
+#endif
 	{FW_FEATURE_SPLPAR,		"hcall-splpar"},
 };
 

/* Build up the firmware features bitmask using the contents of

#1 lore raw
Quoted 8 lines · Will Schmidt

Add a config option for the (batched) MULTITCE and BULK_REMOVE h-calls.

By default, these options are on and are beneficial for performance and
throughput reasons. If disabled, the code will fall back to using less
optimal TCE and REMOVE hcalls. The ability to easily disable these
options is useful for some of the PREEMPT_RT related investigation and work occurring on Power.

Hi,

I can see why it's useful to enable and disable, but these are all runtime-checked, wouldn't it be more useful to add a bootarg to handle it instead of adding some new config options that pretty much everyone will always go with the defaults on?

The bits are set early, but from looking at where they're used, there doesn't seem to be any harm in disabling them later on when a bootarg is convenient to parse and deal with?

It has the benefit of easier on/off testing, if that has any value for production debug down the road.

-Olof

#2 lore raw
Quoted 17 lines · Olof Johansson
Quoted 8 lines · Will Schmidt

Add a config option for the (batched) MULTITCE and BULK_REMOVE h-calls.

By default, these options are on and are beneficial for performance and
throughput reasons. If disabled, the code will fall back to using less
optimal TCE and REMOVE hcalls. The ability to easily disable these
options is useful for some of the PREEMPT_RT related investigation and work occurring on Power.

Hi,

I can see why it's useful to enable and disable, but these are all runtime-checked, wouldn't it be more useful to add a bootarg to handle it instead of adding some new config options that pretty much everyone will always go with the defaults on?

Quoted 6 lines

The bits are set early, but from looking at where they're used, there doesn't seem to be any harm in disabling them later on when a bootarg is convenient to parse and deal with?

It has the benefit of easier on/off testing, if that has any value for production debug down the road.

Hi Olof,
Thats a good idea, let me poke at this a bit more, see if I can get
bootargs for this.

Thanks,
-Will

Quoted 4 lines

-Olof

#3 lore raw

PATCH v2 1/1 Add kernel parameter to disable batched hcalls

In reply to Olof Johansson

This introduces a pair of kernel parameters that can be used to disable the MULTITCE and BULK_REMOVE h-calls.

By default, those hcalls are enabled, active, and good for throughput and performance. The ability to disable them will be useful for some of the PREEMPT_RT related investigation and work occurring on Power.

cc Olof Johansson <[email protected]>
cc Anton Blanchard <[email protected]>
cc Benjamin Herrenschmidt <[email protected]>

---

v2 - Per feedback from Olof, the code is reworked to utilize kernel parameter runtime checks, rather than CONFIG options.
- Added relevant change to kernel-parameters.txt

Documentation/kernel-parameters.txt +8 -0
index e2c7487..5c40801 100644
@@ -426,6 +426,10 @@ and is between 256 and 4096 characters. It is defined in the file
 	bttv.pll=	See Documentation/video4linux/bttv/Insmod-options
 	bttv.tuner=	and Documentation/video4linux/bttv/CARDLIST
 
+	bulk_remove=off	[PPC]  This parameter disables the use of the pSeries
+			firmware feature for flushing multiple hpte entries
+			at a time.
+
 	BusLogic=	[HW,SCSI]
 			See drivers/scsi/BusLogic.c, comment before function
 			BusLogic_ParseDriverOptions().
@@ -1499,6 +1503,10 @@ and is between 256 and 4096 characters. It is defined in the file
 	mtdparts=	[MTD]
 			See drivers/mtd/cmdlinepart.c.
 
+	multitce=off	[PPC]  This parameter disables the use of the pSeries
+			firmware feature for updating multiple TCE entries
+			at a time.
+
 	onenand.bdry=	[HW,MTD] Flex-OneNAND Boundary Configuration
 
 			Format: [die0_boundary][,die0_lock][,die1_boundary][,die1_lock]
arch/powerpc/platforms/pseries/iommu.c +16 -0
index 902987d..e174a2f 100644
@@ -625,3 +625,19 @@ void iommu_init_early_pSeries(void)
 	set_pci_dma_ops(&dma_iommu_ops);
 }
 
+static int __init disable_multitce(char *str)
+{
+	if (strcmp(str,"off")==0) {
+		if (firmware_has_feature(FW_FEATURE_LPAR)) {
+			if (firmware_has_feature(FW_FEATURE_MULTITCE)) {
+				printk(KERN_INFO "Disabling MULTITCE firmware feature\n");
+				ppc_md.tce_build = tce_build_pSeriesLP;
+				ppc_md.tce_free	 = tce_free_pSeriesLP;
+				powerpc_firmware_features &= ~FW_FEATURE_MULTITCE;
+			}
+		}
+	}
+	return 1;
+}
+
+__setup("multitce=",disable_multitce);
arch/powerpc/platforms/pseries/lpar.c +13 -0
index 0707653..82d15e7 100644
@@ -599,6 +599,19 @@ static void pSeries_lpar_flush_hash_range(unsigned long number, int local)
 		spin_unlock_irqrestore(&pSeries_lpar_tlbie_lock, flags);
 }
 
+static int __init disable_bulk_remove(char *str)
+{
+	if (strcmp(str,"off")==0) {
+		if (firmware_has_feature(FW_FEATURE_BULK_REMOVE)) {
+			printk(KERN_INFO "Disabling BULK_REMOVE firmware feature");
+			powerpc_firmware_features &= ~FW_FEATURE_BULK_REMOVE;
+		}
+	}
+	return 1;
+}
+
+__setup("bulk_remove=",disable_bulk_remove);
+
 void __init hpte_init_lpar(void)
 {
 	ppc_md.hpte_invalidate	= pSeries_lpar_hpte_invalidate;
#4 lore raw

Re: [PATCH 1/1 v2 ] Add kernel parameter to disable batched hcalls

Nice. I've got minor nits below, and you might also want to run the patch through checkpatch and fix up some of the whitespace warnings.

-Olof

Quoted 58 lines · Will Schmidt

This introduces a pair of kernel parameters that can be used to disable the MULTITCE and BULK_REMOVE h-calls.

By default, those hcalls are enabled, active, and good for throughput and performance. The ability to disable them will be useful for some of the PREEMPT_RT related investigation and work occurring on Power.

cc Olof Johansson <[email protected]>
cc Anton Blanchard <[email protected]>
cc Benjamin Herrenschmidt <[email protected]>

---

v2 - Per feedback from Olof, the code is reworked to utilize kernel parameter runtime checks, rather than CONFIG options.
- Added relevant change to kernel-parameters.txt

Documentation/kernel-parameters.txt +8 -0
index e2c7487..5c40801 100644
@@ -426,6 +426,10 @@ and is between 256 and 4096 characters. It is defined in the file
 	bttv.pll=	See Documentation/video4linux/bttv/Insmod-options
 	bttv.tuner=	and Documentation/video4linux/bttv/CARDLIST
 
+	bulk_remove=off	[PPC]  This parameter disables the use of the pSeries
+			firmware feature for flushing multiple hpte entries
+			at a time.
+
 	BusLogic=	[HW,SCSI]
 			See drivers/scsi/BusLogic.c, comment before function
 			BusLogic_ParseDriverOptions().
@@ -1499,6 +1503,10 @@ and is between 256 and 4096 characters. It is defined in the file
 	mtdparts=	[MTD]
 			See drivers/mtd/cmdlinepart.c.
 
+	multitce=off	[PPC]  This parameter disables the use of the pSeries
+			firmware feature for updating multiple TCE entries
+			at a time.
+
 	onenand.bdry=	[HW,MTD] Flex-OneNAND Boundary Configuration
 
 			Format: [die0_boundary][,die0_lock][,die1_boundary][,die1_lock]
arch/powerpc/platforms/pseries/iommu.c +2 -0
index 902987d..e174a2f 100644
@@ -625,3 +625,19 @@ void iommu_init_early_pSeries(void)
 	set_pci_dma_ops(&dma_iommu_ops);
 }
 
+static int __init disable_multitce(char *str)
+{

arch/powerpc/platforms/pseries/iommu.c void iommu_init_early_pSeries(void) · line 639

Quoted 10 lines
arch/powerpc/platforms/pseries/iommu.c +10 -0
+	if (strcmp(str,"off")==0) {
+		if (firmware_has_feature(FW_FEATURE_LPAR)) {
+			if (firmware_has_feature(FW_FEATURE_MULTITCE)) {
+				printk(KERN_INFO "Disabling MULTITCE firmware feature\n");
+				ppc_md.tce_build = tce_build_pSeriesLP;
+				ppc_md.tce_free	 = tce_free_pSeriesLP;
+				powerpc_firmware_features &= ~FW_FEATURE_MULTITCE;
+			}
+		}
+	}

I personally prefer to keep cases like these in one if statement to save indentation:

	if (strcmp(str, "off") == 0 &&
	    firmware_has_feature(FW_FEATURE_LPAR) &&
	    firmware_has_feature(FW_FEATURE_MULTITCE)) {
		<...>
	}
Quoted 10 lines
arch/powerpc/platforms/pseries/iommu.c +4 -0
+	return 1;
+}
+
+__setup("multitce=",disable_multitce);
arch/powerpc/platforms/pseries/lpar.c +0 -0
index 0707653..82d15e7 100644
@@ -599,6 +599,19 @@ static void pSeries_lpar_flush_hash_range(unsigned long number, int local)
 		spin_unlock_irqrestore(&pSeries_lpar_tlbie_lock, flags);

arch/powerpc/platforms/pseries/lpar.c static void pSeries_lpar_flush_hash_range(unsigned long number, int local) · line 609

Quoted 10 lines
arch/powerpc/platforms/pseries/lpar.c +8 -0
 }
 
+static int __init disable_bulk_remove(char *str)
+{
+	if (strcmp(str,"off")==0) {
+		if (firmware_has_feature(FW_FEATURE_BULK_REMOVE)) {
+			printk(KERN_INFO "Disabling BULK_REMOVE firmware feature");
+			powerpc_firmware_features &= ~FW_FEATURE_BULK_REMOVE;
+		}
+	}

Same here.

Quoted 9 lines
arch/powerpc/platforms/pseries/lpar.c +5 -0
+	return 1;
+}
+
+__setup("bulk_remove=",disable_bulk_remove);
+
 void __init hpte_init_lpar(void)
 {
 	ppc_md.hpte_invalidate	= pSeries_lpar_hpte_invalidate;
#5 lore raw

Re: [PATCH 1/1 v3] Add kernel parameter to disable batched hcalls

This introduces a pair of kernel parameters that can be used to disable the MULTITCE and BULK_REMOVE h-calls.

By default, those hcalls are enabled, active, and good for throughput and performance. The ability to disable them will be useful for some of the PREEMPT_RT related investigation and work occurring on Power.

cc Olof Johansson <[email protected]>
cc Anton Blanchard <[email protected]>
cc Benjamin Herrenschmidt <[email protected]>

---

v2 - Per feedback from Olof, the code is reworked to utilize kernel
parameter runtime checks, rather than CONFIG options.
- Added relevant change to kernel-parameters.txt

v3 - ran through checkpatch and fixed whitespace issues.
- consolidated the if() staircases to save indentation.

Documentation/kernel-parameters.txt +8 -0
index e2c7487..5c40801 100644
@@ -426,6 +426,10 @@ and is between 256 and 4096 characters. It is defined in the file
 	bttv.pll=	See Documentation/video4linux/bttv/Insmod-options
 	bttv.tuner=	and Documentation/video4linux/bttv/CARDLIST
 
+	bulk_remove=off	[PPC]  This parameter disables the use of the pSeries
+			firmware feature for flushing multiple hpte entries
+			at a time.
+
 	BusLogic=	[HW,SCSI]
 			See drivers/scsi/BusLogic.c, comment before function
 			BusLogic_ParseDriverOptions().
@@ -1499,6 +1503,10 @@ and is between 256 and 4096 characters. It is defined in the file
 	mtdparts=	[MTD]
 			See drivers/mtd/cmdlinepart.c.
 
+	multitce=off	[PPC]  This parameter disables the use of the pSeries
+			firmware feature for updating multiple TCE entries
+			at a time.
+
 	onenand.bdry=	[HW,MTD] Flex-OneNAND Boundary Configuration
 
 			Format: [die0_boundary][,die0_lock][,die1_boundary][,die1_lock]
arch/powerpc/platforms/pseries/iommu.c +14 -0
index 902987d..e174a2f 100644
@@ -625,3 +625,17 @@ void iommu_init_early_pSeries(void)
 	set_pci_dma_ops(&dma_iommu_ops);
 }
 
+static int __init disable_multitce(char *str)
+{
+	if (strcmp(str, "off") == 0 &&
+	    firmware_has_feature(FW_FEATURE_LPAR) &&
+	    firmware_has_feature(FW_FEATURE_MULTITCE)) {
+		printk(KERN_INFO "Disabling MULTITCE firmware feature\n");
+		ppc_md.tce_build = tce_build_pSeriesLP;
+		ppc_md.tce_free	 = tce_free_pSeriesLP;
+		powerpc_firmware_features &= ~FW_FEATURE_MULTITCE;
+	}
+	return 1;
+}
+
+__setup("multitce=", disable_multitce);
arch/powerpc/platforms/pseries/lpar.c +12 -0
index 0707653..82d15e7 100644
@@ -599,6 +599,18 @@ static void pSeries_lpar_flush_hash_range(unsigned long number, int local)
 		spin_unlock_irqrestore(&pSeries_lpar_tlbie_lock, flags);
 }
 
+static int __init disable_bulk_remove(char *str)
+{
+	if (strcmp(str, "off") == 0 &&
+	    firmware_has_feature(FW_FEATURE_BULK_REMOVE)) {
+			printk(KERN_INFO "Disabling BULK_REMOVE firmware feature");
+			powerpc_firmware_features &= ~FW_FEATURE_BULK_REMOVE;
+	}
+	return 1;
+}
+
+__setup("bulk_remove=", disable_bulk_remove);
+
 void __init hpte_init_lpar(void)
 {
 	ppc_md.hpte_invalidate	= pSeries_lpar_hpte_invalidate;