Skip to contentSkip to conversation

x86: UV BAU fix for non-consecutive nasids

PATCH In review Cliff Wickman 2 participants 3 messages May 10, 2011 – May 13, 2011

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 x86: UV BAU fix for non-consecutive nasids

From: Cliff Wickman <[email protected]>

(patch v2 has improved changelog and correction of a broken line)

This is a fix for the SGI Altix-UV Broadcast Assist Unit code, which is used for TLB flushing.

Certain hardware configurations (that customers are ordering) cause nasids (numa address space id's) to be non-consecutive. Specifically, once you have more than 4 blades in a IRU (Individual Rack Unit - or 1/2 rack) but less than the maximum of 16, the nasid numbering becomes non-consecutive. This currently results in a 'catastrophic error' (CATERR) detected by the firmware during OS boot. The BAU is generating an 'INTD' request that is targeting a non-existent nasid value.
Such configurations may also occur when a blade is configured off because of hardware errors. (There is one UV hub per blade.)

This patch is required to support such configurations.

The problem with the tlb_uv.c code is that is using the consecutive hub numbers as indices to the BAU distribution bit map. These are simply the ordinal position of the hub or blade within its partition. It should be using physical node numbers (pnodes), which correspond to the physical nasid values. Use of the hub number only works as long as the nasids in the partition are consecutive and increase with a stride of 1.

This patch changes the index to be the pnode number, thus allowing nasids to be non-consecutive.
It also provides a table in local memory for each cpu to translate target cpu number to target pnode and nasid.
And it improves naming to properly reflect 'node' and 'uvhub' versus 'nasid'.

Diffed against 2.6.39-rc6

---
 arch/x86/include/asm/uv/uv_bau.h |   17 +++++--
 arch/x86/platform/uv/tlb_uv.c    |   92 ++++++++++++++++++++++++++-------------
 2 files changed, 76 insertions(+), 33 deletions(-)
arch/x86/include/asm/uv/uv_bau.h +13 -4
Index: linux/arch/x86/include/asm/uv/uv_bau.h
===================================================================
@@ -94,6 +94,8 @@
 /* after this # consecutive successes, bump up the throttle if it was lowered */
 #define COMPLETE_THRESHOLD 5
 
+#define UV_LB_SUBNODEID 0x10
+
 /*
  * number of entries in the destination side payload queue
  */
@@ -124,7 +126,7 @@
  * The distribution specification (32 bytes) is interpreted as a 256-bit
  * distribution vector. Adjacent bits correspond to consecutive even numbered
  * nodeIDs. The result of adding the index of a given bit to the 15-bit
- * 'base_dest_nodeid' field of the header corresponds to the
+ * 'base_dest_nasid' field of the header corresponds to the
  * destination nodeID associated with that specified bit.
  */
 struct bau_target_uvhubmask {
@@ -176,7 +178,7 @@ struct bau_msg_payload {
 struct bau_msg_header {
 	unsigned int dest_subnodeid:6;	/* must be 0x10, for the LB */
 	/* bits 5:0 */
-	unsigned int base_dest_nodeid:15; /* nasid of the */
+	unsigned int base_dest_nasid:15; /* nasid of the */
 	/* bits 20:6 */			  /* first bit in uvhub map */
 	unsigned int command:8;	/* message type */
 	/* bits 28:21 */
@@ -378,6 +380,10 @@ struct ptc_stats {
 	unsigned long d_rcanceled; /* number of messages canceled by resets */
 };
 
+struct hub_and_pnode {
+	short uvhub;
+	short pnode;
+};
 /*
  * one per-cpu; to locate the software tables
  */
@@ -399,10 +405,12 @@ struct bau_control {
 	int baudisabled;
 	int set_bau_off;
 	short cpu;
+	short osnode;
 	short uvhub_cpu;
 	short uvhub;
 	short cpus_in_socket;
 	short cpus_in_uvhub;
+	short partition_base_pnode;
 	unsigned short message_number;
 	unsigned short uvhub_quiesce;
 	short socket_acknowledge_count[DEST_Q_SIZE];
@@ -422,15 +430,16 @@ struct bau_control {
 	int congested_period;
 	cycles_t period_time;
 	long period_requests;
+	struct hub_and_pnode *target_hub_and_pnode;
 };
 
 static inline int bau_uvhub_isset(int uvhub, struct bau_target_uvhubmask *dstp)
 {
 	return constant_test_bit(uvhub, &dstp->bits[0]);
 }
-static inline void bau_uvhub_set(int uvhub, struct bau_target_uvhubmask *dstp)
+static inline void bau_uvhub_set(int pnode, struct bau_target_uvhubmask *dstp)
 {
-	__set_bit(uvhub, &dstp->bits[0]);
+	__set_bit(pnode, &dstp->bits[0]);
 }
 static inline void bau_uvhubs_clear(struct bau_target_uvhubmask *dstp,
 				    int nbits)
arch/x86/platform/uv/tlb_uv.c +63 -29

198 lines not on this page · raw message

#1 lore raw
Quoted 3 lines · Cliff Wickman

From: Cliff Wickman <[email protected]>

(patch v2 has improved changelog and correction of a broken line)

but i did not point out just a single broken line - there's literally a dozen in your patch, still!

*All* of these measures i mentioned in my previous mail:

- eliminate many repetitive strings
- reduce indentation by the introduction of helper inlines
- sensible shortening of variable/field/definition names

... applies to this particular case and will solve the problem.

This whole file is still utterly ugly and i'm not surprised at all it's showing bugs. Functions are still *way* too large and the code is hard to read.

If you don't know how to simplify a particular function then please ask but i see half a dozen things that could be done to clean up any of these big functions, just at a first quick glance. Those patterns of code ugliness repeat all over the whole file!

I dont mind if you do it as delta patches after this fix (so that the fix stays mergable) but we need to see progress in this code first, the pain in this file has been going on for *way* too long.

Thanks,

Ingo

#2 lore raw

[tip:x86/urgent] x86: Fix UV BAU for non-consecutive nasids

In reply to Cliff Wickman
Commit-ID:  77ed23f8d995a01cd8101d84351b567bf5177a30
Gitweb:     http://git.kernel.org/tip/77ed23f8d995a01cd8101d84351b567bf5177a30
Author:     Cliff Wickman <[email protected]>
AuthorDate: Tue, 10 May 2011 08:26:43 -0500
Committer:  Ingo Molnar <[email protected]>
CommitDate: Thu, 12 May 2011 23:45:42 +0200

x86: Fix UV BAU for non-consecutive nasids

This is a fix for the SGI Altix-UV Broadcast Assist Unit code, which is used for TLB flushing.

Certain hardware configurations (that customers are ordering) cause nasids (numa address space id's) to be non-consecutive. Specifically, once you have more than 4 blades in a IRU (Individual Rack Unit - or 1/2 rack) but less than the maximum of 16, the nasid numbering becomes non-consecutive. This currently results in a 'catastrophic error' (CATERR) detected by the firmware during OS boot. The BAU is generating an 'INTD' request that is targeting a non-existent nasid value. Such configurations may also occur when a blade is configured off because of hardware errors. (There is one UV hub per blade.)

This patch is required to support such configurations.

The problem with the tlb_uv.c code is that is using the consecutive hub numbers as indices to the BAU distribution bit map. These are simply the ordinal position of the hub or blade within its partition. It should be using physical node numbers (pnodes), which correspond to the physical nasid values. Use of the hub number only works as long as the nasids in the partition are consecutive and increase with a stride of 1.

This patch changes the index to be the pnode number, thus allowing nasids to be non-consecutive.
It also provides a table in local memory for each cpu to translate target cpu number to target pnode and nasid. And it improves naming to properly reflect 'node' and 'uvhub' versus 'nasid'.

---
 arch/x86/include/asm/uv/uv_bau.h |   17 +++++--
 arch/x86/platform/uv/tlb_uv.c    |   92 ++++++++++++++++++++++++++------------
 2 files changed, 76 insertions(+), 33 deletions(-)
arch/x86/include/asm/uv/uv_bau.h +13 -4
index 3e094af..130f1ee 100644
@@ -94,6 +94,8 @@
 /* after this # consecutive successes, bump up the throttle if it was lowered */
 #define COMPLETE_THRESHOLD 5
 
+#define UV_LB_SUBNODEID 0x10
+
 /*
  * number of entries in the destination side payload queue
  */
@@ -124,7 +126,7 @@
  * The distribution specification (32 bytes) is interpreted as a 256-bit
  * distribution vector. Adjacent bits correspond to consecutive even numbered
  * nodeIDs. The result of adding the index of a given bit to the 15-bit
- * 'base_dest_nodeid' field of the header corresponds to the
+ * 'base_dest_nasid' field of the header corresponds to the
  * destination nodeID associated with that specified bit.
  */
 struct bau_target_uvhubmask {
@@ -176,7 +178,7 @@ struct bau_msg_payload {
 struct bau_msg_header {
 	unsigned int dest_subnodeid:6;	/* must be 0x10, for the LB */
 	/* bits 5:0 */
-	unsigned int base_dest_nodeid:15; /* nasid of the */
+	unsigned int base_dest_nasid:15; /* nasid of the */
 	/* bits 20:6 */			  /* first bit in uvhub map */
 	unsigned int command:8;	/* message type */
 	/* bits 28:21 */
@@ -378,6 +380,10 @@ struct ptc_stats {
 	unsigned long d_rcanceled; /* number of messages canceled by resets */
 };
 
+struct hub_and_pnode {
+	short uvhub;
+	short pnode;
+};
 /*
  * one per-cpu; to locate the software tables
  */
@@ -399,10 +405,12 @@ struct bau_control {
 	int baudisabled;
 	int set_bau_off;
 	short cpu;
+	short osnode;
 	short uvhub_cpu;
 	short uvhub;
 	short cpus_in_socket;
 	short cpus_in_uvhub;
+	short partition_base_pnode;
 	unsigned short message_number;
 	unsigned short uvhub_quiesce;
 	short socket_acknowledge_count[DEST_Q_SIZE];
@@ -422,15 +430,16 @@ struct bau_control {
 	int congested_period;
 	cycles_t period_time;
 	long period_requests;
+	struct hub_and_pnode *target_hub_and_pnode;
 };
 
 static inline int bau_uvhub_isset(int uvhub, struct bau_target_uvhubmask *dstp)
 {
 	return constant_test_bit(uvhub, &dstp->bits[0]);
 }
-static inline void bau_uvhub_set(int uvhub, struct bau_target_uvhubmask *dstp)
+static inline void bau_uvhub_set(int pnode, struct bau_target_uvhubmask *dstp)
 {
-	__set_bit(uvhub, &dstp->bits[0]);
+	__set_bit(pnode, &dstp->bits[0]);
 }
 static inline void bau_uvhubs_clear(struct bau_target_uvhubmask *dstp,
 				    int nbits)
arch/x86/platform/uv/tlb_uv.c +63 -29

198 lines not on this page · raw message