Skip to contentSkip to conversation

opensm: Reduce heap consumption by multicast routing tables (MFTs)

PATCH Superseded Hal Rosenstock 2 participants 2 messages Oct 15, 2009 – Oct 16, 2009 1 message not in the archive InfiniBand

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 opensm: Reduce heap consumption by multicast routing tables (MFTs)

Heap memory consumption by the unicast and multicast routing tables can be reduced.

This patch is analagous to the previous patch doing this for the unicast routing tables (LFTs).

Using valgrind --tool=massif (for heap profiling), there are couple of place ->38.75% (11,206,656B) 0x43267E: osm_switch_new (osm_switch.c:134) ->12.89% (3,728,256B) 0x40F8C9: osm_mcast_tbl_init (osm_mcast_tbl.c:96)

osm_mcast_tbl_init (osm_mcast_tbl.c:96):
p_tbl->p_mask_tbl = malloc(p_tbl->num_entries *
(IB_MCAST_POSITION_MAX +
1) * IB_MCAST_MASK_SIZE / 8);

num_entries above is set based on the switch's MulticastFDBCap (indicated in it's SM class SwitchInfo attribute).

MFTs are only be increased in size and are never reduced in size. If a realloc for MFT fails, it is treated as a fatal error and OpenSM is exited.

---
Changes since v3:
Renamed mft_size to mft_depth and added description in osm_mcast_tbl.h Removed vestigial realloc mask tbl call in osm_dump.c Simplified max_mlid determination in alloc_mfts
Added return value to osm_mcast_tbl_realloc_mask_tbl
Added return value to alloc_mfts
Handle alloc_mfts failure in osm_mcast_mgr_process/process_mgroups Renamed osm_mcast_tbl_realloc_mask_tbl to osm_mcast_tbl_realloc In osm_mcast_tbl_realloc, simplified mft_depth calculation

Changes since v2:
MFT allocation during routing preparation rather than on table access

Changes since v1:
MFT allocation based on actual MLID requests

opensm/include/opensm/osm_mcast_tbl.h +38 -5
index 710d199..0e07d95 100644
@@ -1,6 +1,6 @@
 /*
  * Copyright (c) 2004, 2005 Voltaire, Inc. All rights reserved.
- * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved.
+ * Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved.
  * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.
  *
  * This software is available to you under a choice of one of two
@@ -74,6 +74,7 @@ typedef struct osm_mcast_fwdbl {
 	int16_t max_block_in_use;
 	uint16_t num_entries;
 	uint16_t max_mlid_ho;
+	uint16_t mft_depth;
 	uint16_t(*p_mask_tbl)[][IB_MCAST_POSITION_MAX];
 } osm_mcast_tbl_t;
 /*
@@ -97,7 +98,11 @@ typedef struct osm_mcast_fwdbl {
 *	max_mlid_ho
 *		Maximum MLID value (host order).
 *
-*	pp_mask_tbl
+*	mft_depth
+*		Number of MLIDs in the currently allocated multicast port mask
+*		table.
+*
+*	p_mask_tbl
 *		Pointer to a two dimensional array of port_masks for this switch.
 *		The first dimension is MLID, the second dimension is mask position.
 *		This pointer is null for switches that do not support multicast.
@@ -114,8 +119,8 @@ typedef struct osm_mcast_fwdbl {
 *
 * SYNOPSIS
 */
-ib_api_status_t osm_mcast_tbl_init(IN osm_mcast_tbl_t * p_tbl,
-				   IN uint8_t num_ports, IN uint16_t capacity);
+void osm_mcast_tbl_init(IN osm_mcast_tbl_t * p_tbl, IN uint8_t num_ports,
+			IN uint16_t capacity);
 /*
 * PARAMETERS
 *	num_ports
@@ -126,7 +131,7 @@ ib_api_status_t osm_mcast_tbl_init(IN osm_mcast_tbl_t * p_tbl,
 *		by this switch.
 *
 * RETURN VALUE
-*	IB_SUCCESS on success.
+*	None.
 *
 * NOTES
 *
@@ -158,6 +163,34 @@ void osm_mcast_tbl_delete(IN osm_mcast_tbl_t ** pp_tbl);
 * SEE ALSO
 *********/
 
+/****f* OpenSM: Forwarding Table/osm_mcast_tbl_realloc
+* NAME
+*	osm_mcast_tbl_realloc
+*
+* DESCRIPTION
+*	This function reallocates the multicast port mask table if necessary.
+*
+* SYNOPSIS
+*/
+int
+osm_mcast_tbl_realloc(IN osm_mcast_tbl_t * p_tbl, IN uintn_t mlid_offset);
+/*
+* PARAMETERS
+*
+*	p_tbl
+*		[in] Pointer to the Multicast Forwarding Table object.
+*
+*	mlid_offset
+*		[in] Offset of MLID being accessed.
+*
+* RETURN VALUE
+*	Returns 0 on success and non-zero value otherwise.
+*
+* NOTES
+*
+* SEE ALSO
+*/
+
 /****f* OpenSM: Forwarding Table/osm_mcast_tbl_destroy
 * NAME
 *	osm_mcast_tbl_destroy
opensm/opensm/osm_mcast_mgr.c +42 -0

60 lines not on this page · raw message

opensm/opensm/osm_mcast_tbl.c +49 -24

138 lines not on this page · raw message

opensm/opensm/osm_switch.c +2 -5
index ed0bc66..223791c 100644
@@ -136,9 +136,8 @@ osm_switch_t *osm_switch_new(IN osm_node_t * p_node,
 
 	memset(p_sw->p_prof, 0, sizeof(*p_sw->p_prof) * num_ports);
 
-	if (osm_mcast_tbl_init(&p_sw->mcast_tbl, osm_node_get_num_physp(p_node),
-			       cl_ntoh16(p_si->mcast_cap)))
-		goto err;
+	osm_mcast_tbl_init(&p_sw->mcast_tbl, osm_node_get_num_physp(p_node),
+			   cl_ntoh16(p_si->mcast_cap));
 
 	for (port_num = 0; port_num < num_ports; port_num++)
 		osm_port_prof_construct(&p_sw->p_prof[port_num]);
@@ -508,7 +507,6 @@ static int alloc_lft(IN osm_switch_t * p_sw, uint16_t lids)
 		p_sw->lft = new_lft;
 		p_sw->lft_size = lft_size;
 	}
-
 	return 0;
 }
 
@@ -549,7 +547,6 @@ int osm_switch_prepare_path_rebuild(IN osm_switch_t * p_sw, IN uint16_t max_lids
 		p_sw->num_hops = max_lids + 1;
 	}
 	p_sw->max_lid_ho = max_lids;
-
 	return 0;
 }
 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

Sasha,

Quoted 34 lines · Hal Rosenstock

Heap memory consumption by the unicast and multicast routing tables can be reduced.

This patch is analagous to the previous patch doing this for the unicast routing tables (LFTs).

Using valgrind --tool=massif (for heap profiling), there are couple of place ->38.75% (11,206,656B) 0x43267E: osm_switch_new (osm_switch.c:134) ->12.89% (3,728,256B) 0x40F8C9: osm_mcast_tbl_init (osm_mcast_tbl.c:96)

osm_mcast_tbl_init (osm_mcast_tbl.c:96):
       p_tbl->p_mask_tbl = malloc(p_tbl->num_entries * (IB_MCAST_POSITION_MAX +
                                   1) * IB_MCAST_MASK_SIZE / 8);

num_entries above is set based on the switch's MulticastFDBCap (indicated in it's SM class SwitchInfo attribute).

MFTs are only be increased in size and are never reduced in size. If a realloc for MFT fails, it is treated as a fatal error and OpenSM is exited.

---
Changes since v3:
Renamed mft_size to mft_depth and added description in osm_mcast_tbl.h Removed vestigial realloc mask tbl call in osm_dump.c Simplified max_mlid determination in alloc_mfts
Added return value to osm_mcast_tbl_realloc_mask_tbl
Added return value to alloc_mfts
Handle alloc_mfts failure in osm_mcast_mgr_process/process_mgroups Renamed osm_mcast_tbl_realloc_mask_tbl to osm_mcast_tbl_realloc In osm_mcast_tbl_realloc, simplified mft_depth calculation

Quoted 6 lines

Changes since v2:
MFT allocation during routing preparation rather than on table access

Changes since v1:
MFT allocation based on actual MLID requests

Please ignore this. I'll resubmit once we get past the mcast tbl change for max LID host order.

-- Hal

<snip...>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html