opensm: Add a rate based mechanism for SMP transactions
PATCH v2 1/1 opensm: Add a rate based mechanism for SMP transactions
In order to better handle non responsive SMAs (when link is physically up but the SMA does not respond), a rate based mechanism for SMPs is added to better enable forward progress in a more timely fashion. So rather than wait for timeouts and outstanding wire SMPs to drop below some configured value, there is also a periodic rate for transaction based SMPs. These rate based SMPs are capped at a configured maximum value.
Two new options are added for this:rate_based_smp_usecs indicates the number of microseconds between rate based SMPs.max_rate_based_smps indicates the maximum number of rate based SMPs supported. When this limit is reached, rate based SMPs are no longer sent (until the number of outstanding ones drops below this limit).
The rate based SMP mechanism can be disabled by setting rate_based_smp_usecs to 0. This is equivalent to the (current) algorithm prior to this change.
By default, this mechanism is disabled.
Also, the maximum max_wire_smps is reduced to 0x3FFFFFFF from 0x7FFFFFFF so the sum of max_wire_smps and max_rate_based_smps does not wrap.
---
Changes from v1:
Algorithm change is isolated to vl15_poller rather than involving the vendor layer.
opensm/include/opensm/osm_base.h +25 -1
@@ -1,6 +1,6 @@/* * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved. *
@@ -440,6 +440,30 @@ BEGIN_C_DECLS*/ #define OSM_DEFAULT_SMP_MAX_ON_WIRE 4/***********/ +/****d* OpenSM: Base/OSM_DEFAULT_SMP_RATE_MAX +* NAME +* OSM_DEFAULT_SMP_RATE_MAX +* +* DESCRIPTION +* Specifies the default maximum number of outstanding rate based SMPs. +* +* SYNOPSIS +*/ +#define OSM_DEFAULT_SMP_RATE_MAX 100 +/***********/ +/****d* OpenSM: Base/OSM_DEFAULT_SMP_RATE +* NAME +* OSM_DEFAULT_SMP_RATE +* +* DESCRIPTION +* Specifies the default rate (in usec) for rate based SMPs. +* The default rate is 0. A value of 0 (or EVENT_NO_TIMEOUT) +* disables the rate based SMP mechanism. +* +* SYNOPSIS +*/ +#define OSM_DEFAULT_SMP_RATE 0 +/***********/ /****d* OpenSM: Base/OSM_SM_DEFAULT_QP0_RCV_SIZE * NAME * OSM_SM_DEFAULT_QP0_RCV_SIZE
opensm/include/opensm/osm_subnet.h +11 -1
@@ -1,6 +1,6 @@/* * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * Copyright (c) 2009 System Fabric Works, Inc. All rights reserved.
@@ -149,6 +149,8 @@ typedef struct osm_subn_opt {ib_net16_t m_key_lease_period;uint32_t sweep_interval;uint32_t max_wire_smps; +uint32_t max_rate_based_smps; +uint32_t rate_based_smp_usecs;uint32_t transaction_timeout;uint32_t transaction_retries;uint8_t sm_priority;
@@ -264,6 +266,14 @@ typedef struct osm_subn_opt {* max_wire_smps * The maximum number of SMPs sent in parallel. Default is 4. * +* max_rate_based_smps +* The maximum number of rate based SMPs allowed to be outstanding. +* Default is 1000. +* +* rate_based_smp_usecs +* The wait time in usec for rate based SMPs. Default is 0 +* (disabled). +* * transaction_timeout * The maximum time in milliseconds allowed for a transaction * to complete. Default is 200.
opensm/include/opensm/osm_vl15intf.h +20 -2
@@ -1,6 +1,6 @@/* * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two
@@ -117,6 +117,8 @@ typedef struct osm_vl15 {osm_thread_state_t thread_state;osm_vl15_state_t state;uint32_t max_wire_smps; +uint32_t max_rate_based_smps; +uint32_t rate_based_smp_usecs;cl_event_t signal;cl_thread_t poller;cl_qlist_t rfifo;
@@ -137,6 +139,12 @@ typedef struct osm_vl15 {* max_wire_smps * Maximum number of VL15 MADs allowed on the wire at one time. * +* max_rate_based_smps +* Maximum number of rate based SMPs allowed to be outstanding. +* +* rate_based_smp_usecs +* Wait time in usec for rate based SMPs. +* * signal * Event on which the poller sleeps. *
@@ -243,7 +251,9 @@ void osm_vl15_destroy(IN osm_vl15_t * p_vl15, IN struct osm_mad_pool *p_pool);*/ ib_api_status_t osm_vl15_init(IN osm_vl15_t * p_vl15, IN osm_vendor_t * p_vend, IN osm_log_t * p_log, IN osm_stats_t * p_stats, - IN int32_t max_wire_smps); + IN int32_t max_wire_smps, + IN int32_t max_rate_based_smps, + IN uint32_t rate_based_smp_usecs);/* * PARAMETERS * p_vl15
@@ -261,6 +271,14 @@ ib_api_status_t osm_vl15_init(IN osm_vl15_t * p_vl15, IN osm_vendor_t * p_vend,* max_wire_smps * [in] Maximum number of MADs allowed on the wire at one time. * +* max_rate_based_smps +* [in] Maximum number of rate based SMPs allowed to be +* outstanding. +* +* rate_based_smp_usecs +* [in] Wait time in usec for rate based SMPs. +* +* * RETURN VALUES * IB_SUCCESS if the VL15 object was initialized successfully. *
opensm/opensm/main.c +3 -3
@@ -1,6 +1,6 @@/* * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2009 HNR Consulting. All rights reserved. * Copyright (c) 2009 System Fabric Works, Inc. All rights reserved.
@@ -743,8 +743,8 @@ int main(int argc, char *argv[])case 'n' : opt.max_wire_smps =strtoul (optarg,NULL ,0 );if (opt.max_wire_smps ==0 || - opt.max_wire_smps >0x7FFFFFFF ) - opt.max_wire_smps =0x7FFFFFFF ; + opt.max_wire_smps >0x3FFFFFFF ) + opt.max_wire_smps =0x3FFFFFFF ;printf (" Max wire smp's = %d\n" , opt.max_wire_smps);break ;
opensm/opensm/osm_opensm.c +4 -2
@@ -1,6 +1,6 @@/* * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two
@@ -392,7 +392,9 @@ ib_api_status_t osm_opensm_init(IN osm_opensm_t * p_osm, status =osm_vl15_init (&p_osm->vl15, p_osm->p_vendor, &p_osm->log, &p_osm->stats, - p_opt->max_wire_smps); + p_opt->max_wire_smps, + p_opt->max_rate_based_smps, + p_opt->rate_based_smp_usecs);if (status !=IB_SUCCESS ) goto Exit;
opensm/opensm/osm_subnet.c +22 -3
@@ -1,6 +1,6 @@/* * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * Copyright (c) 2009 System Fabric Works, Inc. All rights reserved.
@@ -297,6 +297,8 @@ static const opt_rec_t opt_tbl[] = { { "m_key_lease_period", OPT_OFFSET(m_key_lease_period), opts_parse_net16, NULL, 1 }, { "sweep_interval", OPT_OFFSET(sweep_interval), opts_parse_uint32, NULL, 1 }, { "max_wire_smps", OPT_OFFSET(max_wire_smps), opts_parse_uint32, NULL, 1 }, + { "max_rate_based_smps", OPT_OFFSET(max_rate_based_smps), opts_parse_uint32, NULL, 1 }, + { "rate_based_smp_usecs", OPT_OFFSET(rate_based_smp_usecs), opts_parse_uint32, NULL, 1 }, { "console", OPT_OFFSET(console), opts_parse_charp, NULL, 0 }, { "console_port", OPT_OFFSET(console_port), opts_parse_uint16, NULL, 0 }, { "transaction_timeout", OPT_OFFSET(transaction_timeout), opts_parse_uint32, NULL, 0 },
@@ -670,6 +672,8 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * p_opt) p_opt->m_key_lease_period =0 ; p_opt->sweep_interval =OSM_DEFAULT_SWEEP_INTERVAL_SECS ; p_opt->max_wire_smps =OSM_DEFAULT_SMP_MAX_ON_WIRE ; + p_opt->max_rate_based_smps =OSM_DEFAULT_SMP_RATE_MAX ; + p_opt->rate_based_smp_usecs =OSM_DEFAULT_SMP_RATE ; p_opt->console =strdup (OSM_DEFAULT_CONSOLE ); p_opt->console_port =OSM_DEFAULT_CONSOLE_PORT ; p_opt->transaction_timeout =OSM_DEFAULT_TRANS_TIMEOUT_MILLISEC ;
@@ -1066,14 +1070,22 @@ int osm_subn_verify_config(IN osm_subn_opt_t * p_opts) }if (p_opts->max_wire_smps ==0 ) - p_opts->max_wire_smps =0x7FFFFFFF ; -else if (p_opts->max_wire_smps >0x7FFFFFFF ) { + p_opts->max_wire_smps =0x3FFFFFFF ; +else if (p_opts->max_wire_smps >0x3FFFFFFF ) {log_report (" Invalid Cached Option Value: max_wire_smps = %u," " Using Default: %u\n" , p_opts->max_wire_smps,OSM_DEFAULT_SMP_MAX_ON_WIRE ); p_opts->max_wire_smps =OSM_DEFAULT_SMP_MAX_ON_WIRE ; } +if (p_opts->max_rate_based_smps >0x3FFFFFFF ) { +log_report (" Invalid Cached Option Value:" +" max_rate_based_smps = %u, Using Default: %u" , + p_opts->max_rate_based_smps, +OSM_DEFAULT_SMP_RATE_MAX ); + p_opts->max_rate_based_smps =OSM_DEFAULT_SMP_RATE_MAX ; + } + if (strcmp(p_opts->console, OSM_DISABLE_CONSOLE) && strcmp(p_opts->console, OSM_LOCAL_CONSOLE)#ifdef ENABLE_OSM_CONSOLE_SOCKET
@@ -1482,6 +1494,11 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t * p_opts)"#\n# TIMING AND THREADING OPTIONS\n#\n" "# Maximum number of SMPs sent in parallel\n" "max_wire_smps %u\n\n" +"# Maximum number of rate based SMPs allowed to be outstanding\n" +"max_rate_based_smps %u\n\n" +"# The rate in [usec] at which rate based SMPs are sent\n" +"# A value of 0 disables the rate based SMP mechanism\n" +"rate_based_smp_usecs %u\n\n" "# The maximum time in [msec] allowed for a transaction to complete\n" "transaction_timeout %u\n\n" "# The maximum number of retries allowed for a transaction to complete\n"
@@ -1494,6 +1511,8 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t * p_opts)"# Use a single thread for handling SA queries\n" "single_thread %s\n\n" , p_opts->max_wire_smps, + p_opts->max_rate_based_smps, + p_opts->rate_based_smp_usecs, p_opts->transaction_timeout, p_opts->transaction_retries, p_opts->max_msg_fifo_timeout,
opensm/opensm/osm_vl15intf.c +19 -6
@@ -1,7 +1,7 @@/* * Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved. * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2006,2009 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two
@@ -113,6 +113,8 @@ static void vl15_poller(IN void *p_ptr)osm_madw_t *p_madw;osm_vl15_t *p_vl = p_ptr;cl_qlist_t *p_fifo; +int32_t max_smps = p_vl->max_wire_smps; +int32_t max_smps2 = p_vl->max_wire_smps + p_vl->max_rate_based_smps;OSM_LOG_ENTER (p_vl->p_log);
@@ -155,17 +157,22 @@ static void vl15_poller(IN void *p_ptr) status =cl_event_wait_on (&p_vl->signal,EVENT_NO_TIMEOUT , TRUE); -while (p_vl->p_stats->qp0_mads_outstanding_on_wire >= - (int32_t ) p_vl->max_wire_smps && +while (p_vl->p_stats->qp0_mads_outstanding_on_wire >= max_smps && p_vl->thread_state ==OSM_THREAD_STATE_RUN ) { status =cl_event_wait_on (&p_vl->signal, -EVENT_NO_TIMEOUT , TRUE); -if (status !=CL_SUCCESS ) { + p_vl->rate_based_smp_usecs, + TRUE); +if (status ==CL_TIMEOUT ) { +if (max_smps < max_smps2) + max_smps++; +break ; + }else if (status !=CL_SUCCESS ) {OSM_LOG (p_vl->p_log,OSM_LOG_ERROR ,"ERR 3E02: " "Event wait failed (%s)\n" ,CL_STATUS_MSG (status));break ; } + max_smps = p_vl->max_wire_smps; } }
@@ -236,7 +243,9 @@ void osm_vl15_destroy(IN osm_vl15_t * p_vl, IN struct osm_mad_pool *p_pool) ib_api_status_t osm_vl15_init(IN osm_vl15_t * p_vl, IN osm_vendor_t * p_vend, IN osm_log_t * p_log, IN osm_stats_t * p_stats, - IN int32_t max_wire_smps) + IN int32_t max_wire_smps, + IN int32_t max_rate_based_smps, + IN uint32_t rate_based_smp_usecs) {ib_api_status_t status =IB_SUCCESS ;
@@ -246,6 +255,10 @@ ib_api_status_t osm_vl15_init(IN osm_vl15_t * p_vl, IN osm_vendor_t * p_vend, p_vl->p_log = p_log; p_vl->p_stats = p_stats; p_vl->max_wire_smps = max_wire_smps; + p_vl->max_rate_based_smps = rate_based_smp_usecs && max_rate_based_smps ? + max_rate_based_smps :0 ; + p_vl->rate_based_smp_usecs = rate_based_smp_usecs && max_rate_based_smps ? + rate_based_smp_usecs :EVENT_NO_TIMEOUT ; status =cl_event_init (&p_vl->signal, FALSE);if (status !=IB_SUCCESS )
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Hal,
Quoted 15 lines · Hal Rosenstock
In order to better handle non responsive SMAs (when link is physically up but the SMA does not respond), a rate based mechanism for SMPs is added to better enable forward progress in a more timely fashion. So rather than wait for timeouts and outstanding wire SMPs to drop below some configured value, there is also a periodic rate for transaction based SMPs. These rate based SMPs are capped at a configured maximum value.
Two new options are added for this:
rate_based_smp_usecsindicates the number of microseconds between rate based SMPs.max_rate_based_smpsindicates the maximum number of rate based SMPs supported. When this limit is reached, rate based SMPs are no longer sent (until the number of outstanding ones drops below this limit).
Quoted 4 lines
The rate based SMP mechanism can be disabled by setting
rate_based_smp_usecsto 0. This is equivalent to the (current) algorithm prior to this change.By default, this mechanism is disabled.
I would strongly suggest to rework the terminology here to make things clearer.
Actually we don't have here any "rate based" SMPs, but instead two mad injection limits (regular and timedout) and timeout value (which BTW likely should be a function of --timeout parameter). Isn't it?
Sasha
Quoted 333 lines
Also, the maximum
max_wire_smpsis reduced to0x3FFFFFFFfrom0x7FFFFFFFso the sum ofmax_wire_smpsandmax_rate_based_smpsdoes not wrap.Signed-off-by Hal Rosenstock <[email protected]>---
Changes from v1:
Algorithm change is isolated tovl15_pollerrather than involving the vendor layer.opensm/include/opensm/osm_base.h +25
-1@@ -1,6 +1,6 @@/* * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved. * @@ -440,6 +440,30 @@ BEGIN_C_DECLS*/ #define OSM_DEFAULT_SMP_MAX_ON_WIRE 4/***********/ +/****d* OpenSM: Base/OSM_DEFAULT_SMP_RATE_MAX +* NAME +* OSM_DEFAULT_SMP_RATE_MAX +* +* DESCRIPTION +* Specifies the default maximum number of outstanding rate based SMPs. +* +* SYNOPSIS +*/ +#define OSM_DEFAULT_SMP_RATE_MAX 100 +/***********/ +/****d* OpenSM: Base/OSM_DEFAULT_SMP_RATE +* NAME +* OSM_DEFAULT_SMP_RATE +* +* DESCRIPTION +* Specifies the default rate (in usec) for rate based SMPs. +* The default rate is 0. A value of 0 (or EVENT_NO_TIMEOUT) +* disables the rate based SMP mechanism. +* +* SYNOPSIS +*/ +#define OSM_DEFAULT_SMP_RATE 0 +/***********/ /****d* OpenSM: Base/OSM_SM_DEFAULT_QP0_RCV_SIZE * NAME * OSM_SM_DEFAULT_QP0_RCV_SIZE opensm/include/opensm/osm_subnet.h +11
-1@@ -1,6 +1,6 @@/* * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * Copyright (c) 2009 System Fabric Works, Inc. All rights reserved. @@ -149,6 +149,8 @@ typedef struct osm_subn_opt {ib_net16_t m_key_lease_period;uint32_t sweep_interval;uint32_t max_wire_smps; +uint32_t max_rate_based_smps; +uint32_t rate_based_smp_usecs;uint32_t transaction_timeout;uint32_t transaction_retries;uint8_t sm_priority;@@ -264,6 +266,14 @@ typedef struct osm_subn_opt {* max_wire_smps * The maximum number of SMPs sent in parallel. Default is 4. * +* max_rate_based_smps +* The maximum number of rate based SMPs allowed to be outstanding. +* Default is 1000. +* +* rate_based_smp_usecs +* The wait time in usec for rate based SMPs. Default is 0 +* (disabled). +* * transaction_timeout * The maximum time in milliseconds allowed for a transaction * to complete. Default is 200. opensm/include/opensm/osm_vl15intf.h +20
-2@@ -1,6 +1,6 @@/* * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two @@ -117,6 +117,8 @@ typedef struct osm_vl15 {osm_thread_state_t thread_state;osm_vl15_state_t state;uint32_t max_wire_smps; +uint32_t max_rate_based_smps; +uint32_t rate_based_smp_usecs;cl_event_t signal;cl_thread_t poller;cl_qlist_t rfifo;@@ -137,6 +139,12 @@ typedef struct osm_vl15 {* max_wire_smps * Maximum number of VL15 MADs allowed on the wire at one time. * +* max_rate_based_smps +* Maximum number of rate based SMPs allowed to be outstanding. +* +* rate_based_smp_usecs +* Wait time in usec for rate based SMPs. +* * signal * Event on which the poller sleeps. * @@ -243,7 +251,9 @@ void osm_vl15_destroy(IN osm_vl15_t * p_vl15, IN struct osm_mad_pool *p_pool);*/ ib_api_status_t osm_vl15_init(IN osm_vl15_t * p_vl15, IN osm_vendor_t * p_vend, IN osm_log_t * p_log, IN osm_stats_t * p_stats, - IN int32_t max_wire_smps); + IN int32_t max_wire_smps, + IN int32_t max_rate_based_smps, + IN uint32_t rate_based_smp_usecs);/* * PARAMETERS * p_vl15 @@ -261,6 +271,14 @@ ib_api_status_t osm_vl15_init(IN osm_vl15_t * p_vl15, IN osm_vendor_t * p_vend,* max_wire_smps * [in] Maximum number of MADs allowed on the wire at one time. * +* max_rate_based_smps +* [in] Maximum number of rate based SMPs allowed to be +* outstanding. +* +* rate_based_smp_usecs +* [in] Wait time in usec for rate based SMPs. +* +* * RETURN VALUES * IB_SUCCESS if the VL15 object was initialized successfully. * opensm/opensm/main.c +3
-3@@ -1,6 +1,6 @@/* * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2009 HNR Consulting. All rights reserved. * Copyright (c) 2009 System Fabric Works, Inc. All rights reserved. @@ -743,8 +743,8 @@ int main(int argc, char *argv[])case 'n' : opt.max_wire_smps =strtoul (optarg,NULL ,0 );if (opt.max_wire_smps ==0 || - opt.max_wire_smps >0x7FFFFFFF ) - opt.max_wire_smps =0x7FFFFFFF ; + opt.max_wire_smps >0x3FFFFFFF ) + opt.max_wire_smps =0x3FFFFFFF ;printf (" Max wire smp's = %d\n" , opt.max_wire_smps);break ;opensm/opensm/osm_opensm.c +4
-2@@ -1,6 +1,6 @@/* * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two @@ -392,7 +392,9 @@ ib_api_status_t osm_opensm_init(IN osm_opensm_t * p_osm, status =osm_vl15_init (&p_osm->vl15, p_osm->p_vendor, &p_osm->log, &p_osm->stats, - p_opt->max_wire_smps); + p_opt->max_wire_smps, + p_opt->max_rate_based_smps, + p_opt->rate_based_smp_usecs);if (status !=IB_SUCCESS ) goto Exit;opensm/opensm/osm_subnet.c +22
-3@@ -1,6 +1,6 @@/* * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * Copyright (c) 2009 System Fabric Works, Inc. All rights reserved. @@ -297,6 +297,8 @@ static const opt_rec_t opt_tbl[] = { { "m_key_lease_period", OPT_OFFSET(m_key_lease_period), opts_parse_net16, NULL, 1 }, { "sweep_interval", OPT_OFFSET(sweep_interval), opts_parse_uint32, NULL, 1 }, { "max_wire_smps", OPT_OFFSET(max_wire_smps), opts_parse_uint32, NULL, 1 }, + { "max_rate_based_smps", OPT_OFFSET(max_rate_based_smps), opts_parse_uint32, NULL, 1 }, + { "rate_based_smp_usecs", OPT_OFFSET(rate_based_smp_usecs), opts_parse_uint32, NULL, 1 }, { "console", OPT_OFFSET(console), opts_parse_charp, NULL, 0 }, { "console_port", OPT_OFFSET(console_port), opts_parse_uint16, NULL, 0 }, { "transaction_timeout", OPT_OFFSET(transaction_timeout), opts_parse_uint32, NULL, 0 },@@ -670,6 +672,8 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * p_opt) p_opt->m_key_lease_period =0 ; p_opt->sweep_interval =OSM_DEFAULT_SWEEP_INTERVAL_SECS ; p_opt->max_wire_smps =OSM_DEFAULT_SMP_MAX_ON_WIRE ; + p_opt->max_rate_based_smps =OSM_DEFAULT_SMP_RATE_MAX ; + p_opt->rate_based_smp_usecs =OSM_DEFAULT_SMP_RATE ; p_opt->console =strdup (OSM_DEFAULT_CONSOLE ); p_opt->console_port =OSM_DEFAULT_CONSOLE_PORT ; p_opt->transaction_timeout =OSM_DEFAULT_TRANS_TIMEOUT_MILLISEC ;@@ -1066,14 +1070,22 @@ int osm_subn_verify_config(IN osm_subn_opt_t * p_opts) }if (p_opts->max_wire_smps ==0 ) - p_opts->max_wire_smps =0x7FFFFFFF ; -else if (p_opts->max_wire_smps >0x7FFFFFFF ) { + p_opts->max_wire_smps =0x3FFFFFFF ; +else if (p_opts->max_wire_smps >0x3FFFFFFF ) {log_report (" Invalid Cached Option Value: max_wire_smps = %u," " Using Default: %u\n" , p_opts->max_wire_smps,OSM_DEFAULT_SMP_MAX_ON_WIRE ); p_opts->max_wire_smps =OSM_DEFAULT_SMP_MAX_ON_WIRE ; } +if (p_opts->max_rate_based_smps >0x3FFFFFFF ) { +log_report (" Invalid Cached Option Value:" +" max_rate_based_smps = %u, Using Default: %u" , + p_opts->max_rate_based_smps, +OSM_DEFAULT_SMP_RATE_MAX ); + p_opts->max_rate_based_smps =OSM_DEFAULT_SMP_RATE_MAX ; + } + if (strcmp(p_opts->console, OSM_DISABLE_CONSOLE) && strcmp(p_opts->console, OSM_LOCAL_CONSOLE)#ifdef ENABLE_OSM_CONSOLE_SOCKET @@ -1482,6 +1494,11 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t * p_opts)"#\n# TIMING AND THREADING OPTIONS\n#\n" "# Maximum number of SMPs sent in parallel\n" "max_wire_smps %u\n\n" +"# Maximum number of rate based SMPs allowed to be outstanding\n" +"max_rate_based_smps %u\n\n" +"# The rate in [usec] at which rate based SMPs are sent\n" +"# A value of 0 disables the rate based SMP mechanism\n" +"rate_based_smp_usecs %u\n\n" "# The maximum time in [msec] allowed for a transaction to complete\n" "transaction_timeout %u\n\n" "# The maximum number of retries allowed for a transaction to complete\n" @@ -1494,6 +1511,8 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t * p_opts)"# Use a single thread for handling SA queries\n" "single_thread %s\n\n" , p_opts->max_wire_smps, + p_opts->max_rate_based_smps, + p_opts->rate_based_smp_usecs, p_opts->transaction_timeout, p_opts->transaction_retries, p_opts->max_msg_fifo_timeout,opensm/opensm/osm_vl15intf.c +15
-6@@ -1,7 +1,7 @@/* * Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved. * Copyright (c) 2004-2009 Voltaire, Inc. All rights reserved. -* Copyright (c) 2002-2006,2009 Mellanox Technologies LTD. All rights reserved. +* Copyright (c) 2002-2010 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two @@ -113,6 +113,8 @@ static void vl15_poller(IN void *p_ptr)osm_madw_t *p_madw;osm_vl15_t *p_vl = p_ptr;cl_qlist_t *p_fifo; +int32_t max_smps = p_vl->max_wire_smps; +int32_t max_smps2 = p_vl->max_wire_smps + p_vl->max_rate_based_smps;OSM_LOG_ENTER (p_vl->p_log);@@ -155,17 +157,22 @@ static void vl15_poller(IN void *p_ptr) status =cl_event_wait_on (&p_vl->signal,EVENT_NO_TIMEOUT , TRUE); -while (p_vl->p_stats->qp0_mads_outstanding_on_wire >= - (int32_t ) p_vl->max_wire_smps && +while (p_vl->p_stats->qp0_mads_outstanding_on_wire >= max_smps && p_vl->thread_state ==OSM_THREAD_STATE_RUN ) { status =cl_event_wait_on (&p_vl->signal, -EVENT_NO_TIMEOUT , TRUE); -if (status !=CL_SUCCESS ) { + p_vl->rate_based_smp_usecs, + TRUE); +if (status ==CL_TIMEOUT ) { +if (max_smps < max_smps2) + max_smps++; +break ; + }else if (status !=CL_SUCCESS ) {OSM_LOG (p_vl->p_log,OSM_LOG_ERROR ,"ERR 3E02: " "Event wait failed (%s)\n" ,CL_STATUS_MSG (status));break ; } + max_smps = p_vl->max_wire_smps; } }@@ -236,7 +243,9 @@ void osm_vl15_destroy(IN osm_vl15_t * p_vl, IN struct osm_mad_pool *p_pool) ib_api_status_t osm_vl15_init(IN osm_vl15_t * p_vl, IN osm_vendor_t * p_vend, IN osm_log_t * p_log, IN osm_stats_t * p_stats, - IN int32_t max_wire_smps) + IN int32_t max_wire_smps, + IN int32_t max_rate_based_smps, + IN uint32_t rate_based_smp_usecs) {ib_api_status_t status =IB_SUCCESS ;
opensm/opensm/osm_vl15intf.c ib_api_status_t osm_vl15_init(IN osm_vl15_t * p_vl, IN osm_vendor_t * p_vend, · line 264
Quoted 10 lines
opensm/opensm/osm_vl15intf.c +4
-0@@ -246,6 +255,10 @@ ib_api_status_t osm_vl15_init(IN osm_vl15_t * p_vl, IN osm_vendor_t * p_vend, p_vl->p_log = p_log; p_vl->p_stats = p_stats; p_vl->max_wire_smps = max_wire_smps; + p_vl->max_rate_based_smps = rate_based_smp_usecs && max_rate_based_smps ? + max_rate_based_smps :0 ; + p_vl->rate_based_smp_usecs = rate_based_smp_usecs && max_rate_based_smps ? + rate_based_smp_usecs :EVENT_NO_TIMEOUT ; status =cl_event_init (&p_vl->signal, FALSE);if (status !=IB_SUCCESS )
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Sasha,
Quoted 21 lines · Sasha Khapyorsky
Hi Hal,
Quoted 18 lines · Hal Rosenstock
In order to better handle non responsive SMAs (when link is physically up but the SMA does not respond), a rate based mechanism for SMPs is added to better enable forward progress in a more timely fashion. So rather than wait for timeouts and outstanding wire SMPs to drop below some configured value, there is also a periodic rate for transaction based SMPs. These rate based SMPs are capped at a configured maximum value.
Two new options are added for this:
rate_based_smp_usecsindicates the number of microseconds between rate based SMPs.max_rate_based_smpsindicates the maximum number of rate based SMPs supported. When this limit is reached, rate based SMPs are no longer sent (until the number of outstanding ones drops below this limit).The rate based SMP mechanism can be disabled by setting
rate_based_smp_usecsto 0. This is equivalent to the (current) algorithm prior to this change.
Quoted 6 lines
Quoted 1 line
By default, this mechanism is disabled.
I would strongly suggest to rework the terminology here to make things clearer.
Actually we don't have here any "rate based" SMPs,
Doesn't the timeout value pace the second limit of SMPs ? If so, in my mind, that is rate based.
Quoted 3 lines
but instead two mad
injection limits (regular and timedout) and timeout value (which BTW likely should be a function of --timeout parameter). Isn't it?
The separate timeout for this provides finer control over pacing the higher SMP limit rather than basing it on the transaction timeout. If it is a function of the transaction timeout as you propose above, is there admin control over it ? If there is, then there is another config param to express this anyhow. If there isn't, then what hard coded function do you think is appropriate ?
-- Hal
Quoted 1 line
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): AANLkTimE6n2e6z1pUXQPmqX5GrdLVl2kp7bbPipDfhy2-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 26 lines · Hal Rosenstock
Hi Sasha,
Quoted 23 lines · Sasha Khapyorsky
Hi Hal,
Quoted 19 lines · Hal Rosenstock
In order to better handle non responsive SMAs (when link is physically up but the SMA does not respond), a rate based mechanism for SMPs is added to better enable forward progress in a more timely fashion. So rather than wait for timeouts and outstanding wire SMPs to drop below some configured value, there is also a periodic rate for transaction based SMPs. These rate based SMPs are capped at a configured maximum value.
Two new options are added for this:
rate_based_smp_usecsindicates the number of microseconds between rate based SMPs.max_rate_based_smpsindicates the maximum number of rate based SMPs supported. When this limit is reached, rate based SMPs are no longer sent (until the number of outstanding ones drops below this limit).The rate based SMP mechanism can be disabled by setting
rate_based_smp_usecsto 0. This is equivalent to the (current) algorithm prior to this change.By default, this mechanism is disabled.
Quoted 7 lines
Quoted 4 lines
I would strongly suggest to rework the terminology here to make things clearer.
Actually we don't have here any "rate based" SMPs,
Doesn't the timeout value pace the second limit of SMPs ? If so, in my mind, that is rate based.
All those SMPs are equivalent in processing, etc., the only difference is in MADs injection mechanism. So trying to differentiate MADs itself seems confused for me.
Quoted 9 lines
Quoted 3 lines
but instead two mad
injection limits (regular and timedout) and timeout value (which BTW likely should be a function of --timeout parameter). Isn't it?The separate timeout for this provides finer control over pacing the higher SMP limit rather than basing it on the transaction timeout. If it is a function of the transaction timeout as you propose above, is there admin control over it ? If there is, then there is another config param to express this anyhow.
No problem to have this configurable for finer control, but in case when requested smps_on_wire_limit_low < smps_on_wire_limit_high we could want to have some reasonable default value for the timeout.
Quoted 2 lines
If there isn't, then what hard
coded function do you think is appropriate ?
timeout * retries ?
Sasha
Quoted 5 lines
-- Hal
Quoted 1 line
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 30 lines · Sasha Khapyorsky
Quoted 29 lines · Hal Rosenstock
Hi Sasha,
Quoted 26 lines · Sasha Khapyorsky
Hi Hal,
Quoted 15 lines · Hal Rosenstock
In order to better handle non responsive SMAs (when link is physically up but the SMA does not respond), a rate based mechanism for SMPs is added to better enable forward progress in a more timely fashion. So rather than wait for timeouts and outstanding wire SMPs to drop below some configured value, there is also a periodic rate for transaction based SMPs. These rate based SMPs are capped at a configured maximum value.
Two new options are added for this:
rate_based_smp_usecsindicates the number of microseconds between rate based SMPs.max_rate_based_smpsindicates the maximum number of rate based SMPs supported. When this limit is reached, rate based SMPs are no longer sent (until the number of outstanding ones drops below this limit).Quoted 4 lines
The rate based SMP mechanism can be disabled by setting
rate_based_smp_usecsto 0. This is equivalent to the (current) algorithm prior to this change.By default, this mechanism is disabled.
I would strongly suggest to rework the terminology here to make things clearer.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Actually we don't have here any "rate based" SMPs,
Doesn't the timeout value pace the second limit of SMPs ? If so, in my mind, that is rate based.
All those SMPs are equivalent in processing, etc., the only difference is in MADs injection mechanism.
And that difference in the MAD injection mechanism is rate based.
Quoted 1 line
So trying to differentiate MADs itself seems confused for me.
I'm still not following why you say this.
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
but instead two mad
injection limits (regular and timedout) and timeout value (which BTW likely should be a function of --timeout parameter). Isn't it?The separate timeout for this provides finer control over pacing the higher SMP limit rather than basing it on the transaction timeout. If it is a function of the transaction timeout as you propose above, is there admin control over it ? If there is, then there is another config param to express this anyhow.
Quoted 3 lines
No problem to have this configurable for finer control, but in case when requested
smps_on_wire_limit_low<smps_on_wire_limit_highwe could want to have some reasonable default value for the timeout.
The default in the proposed patch is that this mechanism is disabled. Are you saying to change this to have it enabled with a default timeout ? I think it's better to leave the default disabled so the default is to behave same as today.
-- Hal
Quoted 13 lines
Quoted 2 lines
If there isn't, then what hard
coded function do you think is appropriate ?timeout * retries ?
Sasha
Quoted 5 lines
-- Hal
Quoted 1 line
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): AANLkTin1X450bG2WvV7y8wanYSBH4JyNgNiCsj2iugL8-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 5 lines · Hal Rosenstock
Quoted 3 lines
All those SMPs are equivalent in processing, etc., the only difference is in MADs injection mechanism.
And that difference in the MAD injection mechanism is rate based.
IMHO it is not "rate based", but rather timeout driven.
Quoted 16 lines
Quoted 2 lines
So trying to differentiate MADs itself seems confused for me.
I'm still not following why you say this.
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
but instead two mad
injection limits (regular and timedout) and timeout value (which BTW likely should be a function of --timeout parameter). Isn't it?The separate timeout for this provides finer control over pacing the higher SMP limit rather than basing it on the transaction timeout. If it is a function of the transaction timeout as you propose above, is there admin control over it ? If there is, then there is another config param to express this anyhow.
Quoted 7 lines
Quoted 3 lines
No problem to have this configurable for finer control, but in case when requested
smps_on_wire_limit_low<smps_on_wire_limit_highwe could want to have some reasonable default value for the timeout.The default in the proposed patch is that this mechanism is disabled. Are you saying to change this to have it enabled with a default timeout ?
No, meaning that the default case is both limits are equivalent. The timeout is less matter in this case.
Sasha
Quoted 19 lines
I think it's better to leave the default disabled so the default is to behave same as today.
-- Hal
Quoted 13 lines
Quoted 2 lines
If there isn't, then what hard
coded function do you think is appropriate ?timeout * retries ?
Sasha
Quoted 5 lines
-- Hal
Quoted 1 line
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Sasha Khapyorsky
Quoted 5 lines · Hal Rosenstock
Quoted 3 lines
All those SMPs are equivalent in processing, etc., the only difference is in MADs injection mechanism.
And that difference in the MAD injection mechanism is rate based.
IMHO it is not "rate based", but rather timeout driven.
rate = 1 / timeout so I think "hairs" are being split here but I'll change it if you insist.
Quoted 21 lines
Quoted 21 lines
Quoted 3 lines
So trying to differentiate MADs itself seems confused for me.
I'm still not following why you say this.
Quoted 14 lines
Quoted 10 lines
Quoted 4 lines
but instead two mad
injection limits (regular and timedout) and timeout value (which BTW likely should be a function of --timeout parameter). Isn't it?The separate timeout for this provides finer control over pacing the higher SMP limit rather than basing it on the transaction timeout. If it is a function of the transaction timeout as you propose above, is there admin control over it ? If there is, then there is another config param to express this anyhow.
No problem to have this configurable for finer control, but in case when requested
smps_on_wire_limit_low<smps_on_wire_limit_highwe could want to have some reasonable default value for the timeout.
Quoted 5 lines
Quoted 3 lines
The default in the proposed patch is that this mechanism is disabled. Are you saying to change this to have it enabled with a default timeout ?
No, meaning that the default case is both limits are equivalent.
Equivalent in what way ?
Quoted 1 line
The timeout is less matter in this case.
So 3 configurables rather than 2 ? max_wire_timeout_smps (for the second limit), max_wire_timeout (along the lines of what you propose), and mechanism enable/disable (default disabled).
-- Hal
Quoted 22 lines
Sasha
Quoted 19 lines
I think it's better to leave the default disabled so the default is to behave same as today.
-- Hal
Quoted 13 lines
Quoted 2 lines
If there isn't, then what hard
coded function do you think is appropriate ?timeout * retries ?
Sasha
Quoted 5 lines
-- Hal
Quoted 1 line
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): AANLkTinOqJRYEev6Iz0EJyuAN_N8bP1OcSEFMprMkY4Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 8 lines · Hal Rosenstock
Quoted 6 lines
Quoted 4 lines
The default in the proposed patch is that this mechanism is disabled. Are you saying to change this to have it enabled with a default timeout ?
No, meaning that the default case is both limits are equivalent.
Equivalent in what way ?
By value.
Quoted 6 lines
Quoted 2 lines
The timeout is less matter in this case.
So 3 configurables rather than 2 ?
max_wire_timeout_smps(for the second limit),max_wire_timeout(along the lines of what you propose), and mechanism enable/disable (default disabled).
maxsmps (already exists)
maxsmps2 (default - = maxsmps, so the mechanism is disabled) maxsmps_timeout (optional for both disable/enable cases)
Something like this... Maybe with better names :)
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html