Skip to contentSkip to conversation

opensm: Reduce heap consumption by unicast routing tables (LFTs)

PATCH Hal Rosenstock 2 participants 2 messages Sep 30, 2009 – Oct 1, 2009 4 messages not in the archive InfiniBand

lore
Share & tools
lore

Hal Rosenstock

author
#3 lore raw
In reply to a message not in the archive

Hi Sasha,

Somehow this message didn't make it to the general list.

Quoted 7 lines

Hi Hal,

Quoted 4 lines · Hal Rosenstock

Heap memory consumption by the unicast and multicast routing tables can be reduced.

Quoted 8 lines
Quoted 3 lines

Using valgrind --tool=massif (for heap profiling), there are couple of places that consume most of the heap memory:
->38.75% (11,206,656B) 0x43267E: osm_switch_new (osm_switch.c:134)

What fabric size was used for such measurements?

Assuming that maximal LFT size is less than 50K you may need > 200 switches in order to eat almost 11M memory.

Yes, this was done with around 200 switches.

Quoted 1 line

And what about memory consumption for lid matrices?

This didn't show up in the heap profiling. My understanding is that LID matrices are already optimized. The arrays of ports are allocated for each LID (in host order) on demand:

cl_status_t osm_switch_set_hops(...)

{
       ...
       if (!p_sw->hops[lid_ho]) {
               p_sw->hops[lid_ho] = malloc(p_sw->num_ports);
               if (!p_sw->hops[lid_ho])
                       return -1;
               memset(p_sw->hops[lid_ho], OSM_NO_PATH,
p_sw->num_ports);
       }
       ...
}
Quoted 9 lines
Quoted 7 lines

->12.89% (3,728,256B) 0x40F8C9: osm_mcast_tbl_init (osm_mcast_tbl.c:96)

osm_switch_new (osm_switch.c:108):
       p_sw->lft = malloc(IB_LID_UCAST_END_HO + 1);

From ib_types.h
 #define IB_LID_UCAST_END_HO 0xBFFF

Which is 49152 bytes per switch, not so terrible IMHO.

For embedded systems, 50KB per switch is not so small. Also, a similar thing happens with MFTs, so these 50KB turn into something even bigger.

Quoted 11 lines
Quoted 4 lines

The LFT can be allocated in smaller chunks. If there is a LID that exeeds the current LFT size, LFT is reallocated with an increased size.

Is such chunk-driven flow really needed?

After subnet discovery and LIDs  assignment you know exactly how many lid entries are needed for LFT. So would not it be better, faster and simpler to just prealloc switches' LFT buffers for this required and already optimal size at beginning of the LFT calculation phase?

Yes, that seems like a better approach.

Quoted 10 lines
Quoted 1 line

This reduces performance and increases memory fragmentation,

Also code's size, complexity and maintainability.

Quoted 3 lines

so this
tradeoff is made optional based on new build and config options (see below).

And assuming that we will have a simpler approach, will we need those config options?

The config options would not be needed but build options might be. That would depend on whether you want the current behavior (allocation during osm_switch_new) preserved or not. Do you ?

-- Hal

<snip...>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
In reply to a message not in the archive
Quoted 4 lines · Hal Rosenstock

The config options would not be needed but build options might be. That would depend on whether you want the current behavior (allocation during osm_switch_new) preserved or not. Do you ?

I don't see yet any reason to keep this (assuming that LFT buffer allocation is moving to ucast_mgr), as well as a build option.

Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html