Skip to contentSkip to conversation

Add the ability to preload the destination GID and LID caches

PATCH In review Hal Rosenstock 2 participants 4 messages Jun 21, 2013 – Jun 26, 2013 3 messages not in the archive InfiniBand

lore
Share & tools
lore
#0 lore raw

PATCH 10/15 Add the ability to preload the destination GID and LID caches

Preloading of these caches is supported via a file which is produced by OpenSM by the dump_pr plugin which contains sufficient SA PathRecord information. Details on this file format and configuring OpenSM for this are found in dump_pr_notes.txt in dump_pr.

File format is specified in ibacm_opts.cfg as follows:
path_rec_fnt full_opensm_v1

To preload these caches, ibacm service is started with -p option which can also take optional path_rec_file which defaults to ACM_CONF_DIR/ibacm_path_records.dump

---
 ibacm_opts.cfg |    7 ++
 linux/osd.h    |    2 +
 man/ibacm.1    |   18 ++++-
 src/acm.c      |  274 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 src/acme.c     |    7 ++
 5 files changed, 299 insertions(+), 9 deletions(-)
ibacm_opts.cfg +7 -0
index 87f31d9..a15f317 100644
@@ -143,3 +143,10 @@ min_mtu 2048
 
 min_rate 10
 
+# path_rec_fmt:
+# Indicates format of optional path records file for preloading ACM cache.
+# Supported formats are:
+# full_opensm_v1 - OpenSM "full" path records dump file format (version 1)
+
+path_rec_fmt full_opensm_v1
+
linux/osd.h +2 -0
index c8278aa..0b6884f 100644
@@ -1,5 +1,6 @@
 /*
  * Copyright (c) 2009 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2013 Mellanox Technologies LTD. All rights reserved.
  *
  * This software is available to you under the OpenFabrics.org BSD license
  * below:
@@ -57,6 +58,7 @@
 #define ACM_CONF_DIR  SYSCONFDIR "/" RDMADIR
 #define ACM_ADDR_FILE "ibacm_addr.cfg"
 #define ACM_OPTS_FILE "ibacm_opts.cfg"
+#define ACM_PATH_REC_FILE "ibacm_path_records.dump"
 
 #define LIB_DESTRUCTOR __attribute__((destructor))
 #define CDECL_FUNC
man/ibacm.1 +16 -2
index 35b79c6..382b4c0 100644
@@ -1,10 +1,10 @@
-.TH "ibacm" 1 "2013-06-15" "ibacm" "ibacm" ibacm
+.TH "ibacm" 1 "2013-06-18" "ibacm" "ibacm" ibacm
 .SH NAME
 ibacm \- address and route resolution services for InfiniBand.
 .SH SYNOPSIS
 .sp
 .nf
-\fIibacm\fR [-D] [-P] [-A addr_file] [-O option_file]
+\fIibacm\fR [-D] [-P] [-A addr_file] [-O option_file] [-p [path_rec_file]]
 .fi
 .SH "DESCRIPTION"
 The IB ACM implements and provides a framework for name,
@@ -46,6 +46,9 @@ address configuration file
 .TP
 \-O option_file
 option configuration file
+.TP
+\-p [path_rec_file]
+optional path records file for preloading ACM cache
 .SH "QUICK START GUIDE"
 1. Prerequisites: libibverbs and libibumad must be installed.
 The IB stack should be running with IPoIB configured.
@@ -145,5 +148,16 @@ request is received from a different QPN than a cached request.
 limited to 4.
 .P
 - The number of multicast groups that an endpoint can support is limited to 2.
+.P
+The ibacm contains several internal caches.  These include caches for GID
+and LID destination addresses.  These caches can be optionally
+preloaded. ibacm supports the OpenSM dump_pr plugin "full" PathRecord 
+format which is used to preload these caches.
+The file format is specified in the ibacm_opts.cfg file via the
+path_rec_fmt parameter which should be set to full_opensm_v1 for this file
+format which is the default.  See dump_pr.notes.txt in dump_pr for
+more information on both this file format and how to configure OpenSM
+to output this file.  By default, if the -p option is specified, the default
+file used to preload these caches is ACM_CONF_DIR/ibacm_path_record.dump
 .SH "SEE ALSO"
 ibacm(7), ib_acme(1), rdma_cm(7)
src/acm.c +267 -7

353 lines not on this page · raw message

src/acme.c +7 -0
index d7ad25f..ea44eb94 100644
@@ -244,6 +244,13 @@ static void gen_opts_temp(FILE *f)
 	fprintf(f, "\n");
 	fprintf(f, "min_rate 10\n");
 	fprintf(f, "\n");
+	fprintf(f, "# path_rec_fmt:\n");
+	fprintf(f, "# Indicates format of optional path records file for preloading ACM cache.\n");
+	fprintf(f, "# Supported formats are:\n");
+	fprintf(f, "# full_opensm_v1 - OpenSM \"full\" path records dump file format (version 1)\n");
+	fprintf(f, "\n");
+	fprintf(f, "path_rec_fmt full_opensm_v1\n");
+	fprintf(f, "\n");
 }
 
 static int open_dir(void)
Signature · 6 lines
1.7.8.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#2 lore raw
In reply to a message not in the archive
Quoted 6 lines

Preloading of these caches is supported via a file which is produced by OpenSM by the dump_pr plugin which contains sufficient SA PathRecord information. Details on this file format and configuring OpenSM for this are found in dump_pr_notes.txt in dump_pr.

Quoted 6 lines

File format is specified in ibacm_opts.cfg as follows:
path_rec_fnt full_opensm_v1

To preload these caches, ibacm service is started with -p option which can also take optional path_rec_file which defaults to ACM_CONF_DIR/ibacm_path_records.dump

I would rather use the config file versus adding command line parameters.

This series will preload the cache, but it does not change the actual protocol ibacm uses for updates. I want to verify that this is your intent. I see within this patch that you override the address and route timeout values set by the user. I think the code should honor those settings.

I wonder if it would make sense to treat file loading as a 'protocol' rather than a preload of the cache. By changing the timeout value, this is essentially what occurs. If the cached data can timeout, then there's an option for what ibacm should do. If it falls back to its configured address/route protocols, then this is truly a preload of the cache. An alternate option would be for ibacm to see if the input files had been updated, and if so, reload its cache.

One drawback I see with using this as a simple preload mechanism is that the entire cache would timeout at once if cache timeouts are enabled.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
In reply to a message not in the archive
Quoted 6 lines · Sean
Quoted 6 lines

Preloading of these caches is supported via a file which is produced by OpenSM by the dump_pr plugin which contains sufficient SA PathRecord information. Details on this file format and configuring OpenSM for this are found in dump_pr_notes.txt in dump_pr.

Quoted 8 lines
Quoted 6 lines

File format is specified in ibacm_opts.cfg as follows:
path_rec_fnt full_opensm_v1

To preload these caches, ibacm service is started with -p option which can also take optional path_rec_file which defaults to ACM_CONF_DIR/ibacm_path_records.dump

I would rather use the config file versus adding command line parameters.

So add a new preload option in the config file for this ?

Quoted 2 lines

This series will preload the cache, but it does not change the actual protocol ibacm uses for updates. I want to verify that this is your intent.

Yes, that is my intent.

Quoted 2 lines

I see within this patch that you override the address and route timeout values set by the user. I think the code should honor those settings.

OK. The user can configure no address and/or route timeout if that is what he wants so this mode of operation is possible currently without this.

Quoted 2 lines

I wonder if it would make sense to treat file loading as a 'protocol' rather than a preload of the cache. By changing the timeout value, this is essentially what occurs.

Didn't you just write above not to change those timeout values ?

Quoted 1 line

If the cached data can timeout, then there's an option for what ibacm should do. If it falls back to its

configured

Quoted 1 line

address/route protocols, then this is truly a preload of the cache.

That's what would happen now (for node not in cache or on timeout).

Quoted 2 lines

An alternate option would be for ibacm to see if the input files had been updated, and if so, reload its cache.

Yes, that seems like another valid alternative.

Quoted 1 line

One drawback I see with using this as a simple preload mechanism is that the entire cache would timeout at once if cache timeouts are enabled.

You're talking about the way it works as currently proposed in this patch, right ?

-- Hal

Quoted 2 lines

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
In reply to a message not in the archive
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

To preload these caches, ibacm service is started with -p option which can also take optional path_rec_file which defaults to ACM_CONF_DIR/ibacm_path_records.dump

I would rather use the config file versus adding command line parameters.

So add a new preload option in the config file for this ?

Please add entries in the config file that can be used to specify the names of the data files.

Quoted 10 lines
Quoted 1 line

This series will preload the cache, but it does not change the actual

protocol ibacm uses for updates.

Quoted 1 line

I want to verify that this is your intent.

Yes, that is my intent.

Quoted 1 line

I see within this patch that you override the address and route timeout

values set by the user.

Quoted 1 line

I think the code should honor those settings.

Quoted 8 lines

OK. The user can configure no address and/or route timeout if that is what he wants so this mode of operation is possible currently without this.

Quoted 1 line

I wonder if it would make sense to treat file loading as a 'protocol' rather

than a preload of the cache.

Quoted 1 line

By changing the timeout value, this is essentially what occurs.

Didn't you just write above not to change those timeout values ?

My preference is that the timeout values not be changed. I was commenting that since the patch did change the timeout values, it essentially disabled the protocols used for updating the cache.

Quoted 5 lines
Quoted 1 line

One drawback I see with using this as a simple preload mechanism is that the

entire cache would timeout at once if cache timeouts are enabled.

You're talking about the way it works as currently proposed in this patch, right ?

I was talking about the proposed patch, but removing the change to the timeout.

For this series, please remove the timeout changes and treat the changes as you intended, which is to simply preload the cache. If we ever want to update the cache by reading from a file, that can come as a separate change.

Thanks,
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html