Add the ability to preload the destination GID and LID caches
PATCH 10/15 Add the ability to preload the destination GID and LID caches
Preloading of these caches is supported via a file which is produced by OpenSM by the dump_pr plugin which contains sufficient SA PathRecord information. Details on this file format and configuring OpenSM for this are found in dump_pr_notes.txt in dump_pr.
File format is specified in ibacm_opts.cfg as follows:path_rec_fnt full_opensm_v1
To preload these caches, ibacm service is started with -p option which can also take optional path_rec_file which defaults to ACM_CONF_DIR/ibacm_path_records.dump
--- ibacm_opts.cfg | 7 ++ linux/osd.h | 2 + man/ibacm.1 | 18 ++++- src/acm.c | 274 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- src/acme.c | 7 ++ 5 files changed, 299 insertions(+), 9 deletions(-)
ibacm_opts.cfg +7 -0
@@ -143,3 +143,10 @@ min_mtu 2048 min_rate 10 +# path_rec_fmt: +# Indicates format of optional path records file for preloading ACM cache. +# Supported formats are: +# full_opensm_v1 - OpenSM "full" path records dump file format (version 1) + +path_rec_fmt full_opensm_v1 +
linux/osd.h +2 -0
@@ -1,5 +1,6 @@/* * Copyright (c) 2009 Intel Corporation. All rights reserved. +* Copyright (c) 2013 Mellanox Technologies LTD. All rights reserved. * * This software is available to you under the OpenFabrics.org BSD license * below:
@@ -57,6 +58,7 @@#define ACM_CONF_DIR SYSCONFDIR "/" RDMADIR#define ACM_ADDR_FILE "ibacm_addr.cfg"#define ACM_OPTS_FILE "ibacm_opts.cfg" +#define ACM_PATH_REC_FILE "ibacm_path_records.dump"#define LIB_DESTRUCTOR __attribute__((destructor))#define CDECL_FUNC
man/ibacm.1 +16 -2
@@ -1,10 +1,10 @@ -.TH "ibacm" 1 "2013-06-15" "ibacm" "ibacm" ibacm +.TH "ibacm" 1 "2013-06-18" "ibacm" "ibacm" ibacm .SH NAME ibacm \- address and route resolution services for InfiniBand. .SH SYNOPSIS .sp .nf -\fIibacm\fR [-D] [-P] [-A addr_file] [-O option_file] +\fIibacm\fR [-D] [-P] [-A addr_file] [-O option_file] [-p [path_rec_file]] .fi .SH "DESCRIPTION" The IB ACM implements and provides a framework for name,
@@ -46,6 +46,9 @@ address configuration file .TP \-O option_file option configuration file +.TP +\-p [path_rec_file] +optional path records file for preloading ACM cache .SH "QUICK START GUIDE" 1. Prerequisites: libibverbs and libibumad must be installed. The IB stack should be running with IPoIB configured.
@@ -145,5 +148,16 @@ request is received from a different QPN than a cached request. limited to 4. .P - The number of multicast groups that an endpoint can support is limited to 2. +.P +The ibacm contains several internal caches. These include caches for GID +and LID destination addresses. These caches can be optionally +preloaded. ibacm supports the OpenSM dump_pr plugin "full" PathRecord +format which is used to preload these caches. +The file format is specified in the ibacm_opts.cfg file via the +path_rec_fmt parameter which should be set to full_opensm_v1 for this file +format which is the default. See dump_pr.notes.txt in dump_pr for +more information on both this file format and how to configure OpenSM +to output this file. By default, if the -p option is specified, the default +file used to preload these caches is ACM_CONF_DIR/ibacm_path_record.dump .SH "SEE ALSO" ibacm(7), ib_acme(1), rdma_cm(7)
src/acm.c +267 -7
353 lines not on this page · raw message
src/acme.c +7 -0
@@ -244,6 +244,13 @@ static void gen_opts_temp(FILE *f)fprintf (f,"\n" );fprintf (f,"min_rate 10\n" );fprintf (f,"\n" ); +fprintf (f,"# path_rec_fmt:\n" ); +fprintf (f,"# Indicates format of optional path records file for preloading ACM cache.\n" ); +fprintf (f,"# Supported formats are:\n" ); +fprintf (f,"# full_opensm_v1 - OpenSM \"full\" path records dump file format (version 1)\n" ); +fprintf (f,"\n" ); +fprintf (f,"path_rec_fmt full_opensm_v1\n" ); +fprintf (f,"\n" ); }static int open_dir (void )
Signature · 6 lines
1.7.8.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines
Preloading of these caches is supported via a file which is produced by OpenSM by the
dump_prplugin which contains sufficient SA PathRecord information. Details on this file format and configuring OpenSM for this are found in dump_pr_notes.txt indump_pr.
Quoted 6 lines
File format is specified in ibacm_opts.cfg as follows:
path_rec_fntfull_opensm_v1To preload these caches, ibacm service is started with -p option which can also take optional
path_rec_filewhich defaults to ACM_CONF_DIR/ibacm_path_records.dump
I would rather use the config file versus adding command line parameters.
This series will preload the cache, but it does not change the actual protocol ibacm uses for updates. I want to verify that this is your intent. I see within this patch that you override the address and route timeout values set by the user. I think the code should honor those settings.
I wonder if it would make sense to treat file loading as a 'protocol' rather than a preload of the cache. By changing the timeout value, this is essentially what occurs. If the cached data can timeout, then there's an option for what ibacm should do. If it falls back to its configured address/route protocols, then this is truly a preload of the cache. An alternate option would be for ibacm to see if the input files had been updated, and if so, reload its cache.
One drawback I see with using this as a simple preload mechanism is that the entire cache would timeout at once if cache timeouts are enabled.
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Sean
Quoted 6 lines
Preloading of these caches is supported via a file which is produced by OpenSM by the
dump_prplugin which contains sufficient SA PathRecord information. Details on this file format and configuring OpenSM for this are found in dump_pr_notes.txt indump_pr.
Quoted 8 lines
Quoted 6 lines
File format is specified in ibacm_opts.cfg as follows:
path_rec_fntfull_opensm_v1To preload these caches, ibacm service is started with -p option which can also take optional
path_rec_filewhich defaults to ACM_CONF_DIR/ibacm_path_records.dumpI would rather use the config file versus adding command line parameters.
So add a new preload option in the config file for this ?
Quoted 2 lines
This series will preload the cache, but it does not change the actual protocol ibacm uses for updates. I want to verify that this is your intent.
Yes, that is my intent.
Quoted 2 lines
I see within this patch that you override the address and route timeout values set by the user. I think the code should honor those settings.
OK. The user can configure no address and/or route timeout if that is what he wants so this mode of operation is possible currently without this.
Quoted 2 lines
I wonder if it would make sense to treat file loading as a 'protocol' rather than a preload of the cache. By changing the timeout value, this is essentially what occurs.
Didn't you just write above not to change those timeout values ?
Quoted 1 line
If the cached data can timeout, then there's an option for what ibacm should do. If it falls back to its
configured
Quoted 1 line
address/route protocols, then this is truly a preload of the cache.
That's what would happen now (for node not in cache or on timeout).
Quoted 2 lines
An alternate option would be for ibacm to see if the input files had been updated, and if so, reload its cache.
Yes, that seems like another valid alternative.
Quoted 1 line
One drawback I see with using this as a simple preload mechanism is that the entire cache would timeout at once if cache timeouts are enabled.
You're talking about the way it works as currently proposed in this patch, right ?
-- Hal
Quoted 2 lines
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
To preload these caches, ibacm service is started with -p option which can also take optional
path_rec_filewhich defaults to ACM_CONF_DIR/ibacm_path_records.dumpI would rather use the config file versus adding command line parameters.
So add a new preload option in the config file for this ?
Please add entries in the config file that can be used to specify the names of the data files.
Quoted 10 lines
Quoted 1 line
This series will preload the cache, but it does not change the actual
protocol ibacm uses for updates.
Quoted 1 line
I want to verify that this is your intent.
Yes, that is my intent.
Quoted 1 line
I see within this patch that you override the address and route timeout
values set by the user.
Quoted 1 line
I think the code should honor those settings.
Quoted 8 lines
OK. The user can configure no address and/or route timeout if that is what he wants so this mode of operation is possible currently without this.
Quoted 1 line
I wonder if it would make sense to treat file loading as a 'protocol' rather
than a preload of the cache.
Quoted 1 line
By changing the timeout value, this is essentially what occurs.
Didn't you just write above not to change those timeout values ?
My preference is that the timeout values not be changed. I was commenting that since the patch did change the timeout values, it essentially disabled the protocols used for updating the cache.
Quoted 5 lines
Quoted 1 line
One drawback I see with using this as a simple preload mechanism is that the
entire cache would timeout at once if cache timeouts are enabled.
You're talking about the way it works as currently proposed in this patch, right ?
I was talking about the proposed patch, but removing the change to the timeout.
For this series, please remove the timeout changes and treat the changes as you intended, which is to simply preload the cache. If we ever want to update the cache by reading from a file, that can come as a separate change.
Thanks,
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html