opensm/osm_state_mgr.c: force heavy sweep when fabric consists of single switch

PATCH In review Yevgeny Kliteynik 4 participants 18 messages Nov 3, 2009 – Nov 13, 2009 16 messages not in the archive InfiniBand

lore

Share & tools

lore

Yevgeny Kliteynik

author

2009-11-03 10:26 UTC #0 lore raw

PATCH 1/1 opensm/osm_state_mgr.c: force heavy sweep when fabric consists of single switch

Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep

Signed-off-by Yevgeny Kliteynik <[email protected]>

---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)

opensm/opensm/osm_state_mgr.c +10 -5

index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
 	 * Otherwise, this is probably our first discovery pass
 	 * or we are connected in loopback. In both cases do a
 	 * heavy sweep.
-	 * Note: If we are connected in loopback we want a heavy
-	 * sweep, since we will not be getting any traps if there is
-	 * a lost connection.
+	 * Note the following:
+	 * 1. If we are connected in loopback we want a heavy sweep, since we
+	 *    will not be getting any traps if there is a lost connection.
+	 * 2. If we are in DISCOVERING state - this means it is either in
+	 *    initializing or wake up from STANDBY - run the heavy sweep.
+	 * 3. If there is only one node in the fabric, and this node is a
+	 *    switch, and OSM runs on top of it, there might be a race when
+	 *    OSM starts running before the external ports are up - run the
+	 *    heavy sweep.
 	 */
-	/*  if we are in DISCOVERING state - this means it is either in
-	 *  initializing or wake up from STANDBY - run the heavy sweep */
 	if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+	    && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
 	    && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
 	    && sm->p_subn->opt.force_heavy_sweep == FALSE
 	    && sm->p_subn->force_heavy_sweep == FALSE

Signature · 6 lines

1.5.1.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4AF0056A.5030503-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

Sasha Khapyorsky

2009-11-03 22:12 UTC #2 lore raw

Quoted 10 lines · Yevgeny Kliteynik

Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep

Could such race happen when there are more than one node in a fabric?

Sasha

Quoted 42 lines

Signed-off-by Yevgeny Kliteynik <[email protected]>

---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)

opensm/opensm/osm_state_mgr.c +10 -5

index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
 	 * Otherwise, this is probably our first discovery pass
 	 * or we are connected in loopback. In both cases do a
 	 * heavy sweep.
-	 * Note: If we are connected in loopback we want a heavy
-	 * sweep, since we will not be getting any traps if there is
-	 * a lost connection.
+	 * Note the following:
+	 * 1. If we are connected in loopback we want a heavy sweep, since we
+	 *    will not be getting any traps if there is a lost connection.
+	 * 2. If we are in DISCOVERING state - this means it is either in
+	 *    initializing or wake up from STANDBY - run the heavy sweep.
+	 * 3. If there is only one node in the fabric, and this node is a
+	 *    switch, and OSM runs on top of it, there might be a race when
+	 *    OSM starts running before the external ports are up - run the
+	 *    heavy sweep.
 	 */
-	/*  if we are in DISCOVERING state - this means it is either in
-	 *  initializing or wake up from STANDBY - run the heavy sweep */
 	if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+	    && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
 	    && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
 	    && sm->p_subn->opt.force_heavy_sweep == FALSE
 	    && sm->p_subn->force_heavy_sweep == FALSE

Signature · 6 lines

1.5.1.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Yevgeny Kliteynik

author

2009-11-04 09:47 UTC #3 lore raw

Quoted 12 lines · Sasha Khapyorsky

Quoted 10 lines · Yevgeny Kliteynik
Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep

Quoted 1 line

Could such race happen when there are more than one node in a fabric?

I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.

So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?

Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.

The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.

Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.

-- Yevgeny

Quoted 38 lines

Sasha

Quoted 36 lines

Signed-off-by Yevgeny Kliteynik <[email protected]>

---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)

opensm/opensm/osm_state_mgr.c +10 -5

index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
 	 * Otherwise, this is probably our first discovery pass
 	 * or we are connected in loopback. In both cases do a
 	 * heavy sweep.
-	 * Note: If we are connected in loopback we want a heavy
-	 * sweep, since we will not be getting any traps if there is
-	 * a lost connection.
+	 * Note the following:
+	 * 1. If we are connected in loopback we want a heavy sweep, since we
+	 *    will not be getting any traps if there is a lost connection.
+	 * 2. If we are in DISCOVERING state - this means it is either in
+	 *    initializing or wake up from STANDBY - run the heavy sweep.
+	 * 3. If there is only one node in the fabric, and this node is a
+	 *    switch, and OSM runs on top of it, there might be a race when
+	 *    OSM starts running before the external ports are up - run the
+	 *    heavy sweep.
 	 */
-	/*  if we are in DISCOVERING state - this means it is either in
-	 *  initializing or wake up from STANDBY - run the heavy sweep */
 	if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+	    && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
 	    && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
 	    && sm->p_subn->opt.force_heavy_sweep == FALSE
 	    && sm->p_subn->force_heavy_sweep == FALSE

Signature · 1 line

1.5.1.4

Quoted 6 lines

Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4AF14DCD.3010407-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

Line Holen

2009-11-04 11:36 UTC #5 lore raw

Quoted 42 lines · Yevgeny Kliteynik

Quoted 12 lines · Sasha Khapyorsky
Quoted 10 lines · Yevgeny Kliteynik
Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep
Quoted 1 line
Could such race happen when there are more than one node in a fabric?
I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?
Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.
The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.

Quoted 1 line

-- Yevgeny

At least for the 3.2 branch there is a general race regardless of where the SM is running. I haven't checked the current master, but I cannot recall seeing any patches related to this so I assume the race is still there.

There is a window between SM discovering a switch and clearing PSC for the same switch. The SM will not detect a state change on the switch ports during this time.

I have a patch for the 3.2 branch that I can merge into master.

Line

Quoted 41 lines

Sasha

Quoted 38 lines

Signed-off-by Yevgeny Kliteynik <[email protected]>

---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)

opensm/opensm/osm_state_mgr.c +2 -3

index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
      * Otherwise, this is probably our first discovery pass
      * or we are connected in loopback. In both cases do a
      * heavy sweep.
-     * Note: If we are connected in loopback we want a heavy
-     * sweep, since we will not be getting any traps if there is
-     * a lost connection.
+     * Note the following:
+     * 1. If we are connected in loopback we want a heavy sweep,

since we

opensm/opensm/osm_state_mgr.c +8 -2

+     *    will not be getting any traps if there is a lost connection.
+     * 2. If we are in DISCOVERING state - this means it is either in
+     *    initializing or wake up from STANDBY - run the heavy sweep.
+     * 3. If there is only one node in the fabric, and this node is a
+     *    switch, and OSM runs on top of it, there might be a race when
+     *    OSM starts running before the external ports are up - run the
+     *    heavy sweep.
      */
-    /*  if we are in DISCOVERING state - this means it is either in
-     *  initializing or wake up from STANDBY - run the heavy sweep */
     if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+        && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
         && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
         && sm->p_subn->opt.force_heavy_sweep == FALSE
         && sm->p_subn->force_heavy_sweep == FALSE

Signature · 1 line

1.5.1.4

Quoted 11 lines

Quoted 6 lines

Quoted 5 lines

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4AF16740.3080600-UdXhSnd/[email protected]

Yevgeny Kliteynik

author

2009-11-04 15:54 UTC #7 lore raw

Quoted 43 lines · Line Holen

Quoted 40 lines · Yevgeny Kliteynik
Quoted 11 lines · Sasha Khapyorsky
Quoted 10 lines · Yevgeny Kliteynik
Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep
Quoted 1 line
Could such race happen when there are more than one node in a fabric?
I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?
Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.
The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.
Quoted 1 line
-- Yevgeny

Quoted 8 lines

At least for the 3.2 branch there is a general race regardless of where the SM is running. I haven't checked the current master, but I cannot recall seeing any patches related to this so I assume the race is still there.
There is a window between SM discovering a switch and clearing PSC for the same switch. The SM will not detect a state change on the switch ports during this time.

If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?

Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?

Quoted 1 line

I have a patch for the 3.2 branch that I can merge into master.

Sure, that would be nice :)

-- Yevgeny

Quoted 53 lines

Line

Quoted 49 lines

Quoted 45 lines

Sasha

Quoted 43 lines

Signed-off-by Yevgeny Kliteynik <[email protected]>

---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)

opensm/opensm/osm_state_mgr.c +2 -3

index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
      * Otherwise, this is probably our first discovery pass
      * or we are connected in loopback. In both cases do a
      * heavy sweep.
-     * Note: If we are connected in loopback we want a heavy
-     * sweep, since we will not be getting any traps if there is
-     * a lost connection.
+     * Note the following:
+     * 1. If we are connected in loopback we want a heavy sweep,

since we

opensm/opensm/osm_state_mgr.c +8 -2

+     *    will not be getting any traps if there is a lost connection.
+     * 2. If we are in DISCOVERING state - this means it is either in
+     *    initializing or wake up from STANDBY - run the heavy sweep.
+     * 3. If there is only one node in the fabric, and this node is a
+     *    switch, and OSM runs on top of it, there might be a race when
+     *    OSM starts running before the external ports are up - run the
+     *    heavy sweep.
      */
-    /*  if we are in DISCOVERING state - this means it is either in
-     *  initializing or wake up from STANDBY - run the heavy sweep */
     if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+        && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
         && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
         && sm->p_subn->opt.force_heavy_sweep == FALSE
         && sm->p_subn->force_heavy_sweep == FALSE

Signature · 6 lines

1.5.1.4

-- 
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4AF1A3CA.9070902-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

Line Holen

2009-11-04 17:42 UTC #9 lore raw

Quoted 49 lines · Yevgeny Kliteynik

Quoted 48 lines · Line Holen
Quoted 40 lines · Yevgeny Kliteynik
Quoted 11 lines · Sasha Khapyorsky
Quoted 10 lines · Yevgeny Kliteynik
Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep
Quoted 1 line
Could such race happen when there are more than one node in a fabric?
I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?
Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.
The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.
Quoted 1 line
-- Yevgeny
At least for the 3.2 branch there is a general race regardless of where the SM is running. I haven't checked the current master, but I cannot recall seeing any patches related to this so I assume the race is still there.

Quoted 8 lines

Quoted 3 lines
There is a window between SM discovering a switch and clearing PSC for the same switch. The SM will not detect a state change on the switch ports during this time.
If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?

I think the switch shall send a trap whenever it sets the PSC bit. Once set I believe it will not send another trap until it is reset. Or do I misinterpret the spec ?

Quoted 56 lines

Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?

Quoted 1 line

I have a patch for the 3.2 branch that I can merge into master.

Sure, that would be nice :)

-- Yevgeny

Quoted 45 lines

Line

Quoted 43 lines

Sasha

Quoted 41 lines

Signed-off-by Yevgeny Kliteynik <[email protected]>

---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)

opensm/opensm/osm_state_mgr.c +2 -3

index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
      * Otherwise, this is probably our first discovery pass
      * or we are connected in loopback. In both cases do a
      * heavy sweep.
-     * Note: If we are connected in loopback we want a heavy
-     * sweep, since we will not be getting any traps if there is
-     * a lost connection.
+     * Note the following:
+     * 1. If we are connected in loopback we want a heavy sweep,

since we
+ * will not be getting any traps if there is a lost
connection.

opensm/opensm/osm_state_mgr.c +4 -0

+     * 2. If we are in DISCOVERING state - this means it is either in
+     *    initializing or wake up from STANDBY - run the heavy sweep.
+     * 3. If there is only one node in the fabric, and this node is a
+     *    switch, and OSM runs on top of it, there might be a race

when
+ * OSM starts running before the external ports are up -
run the

opensm/opensm/osm_state_mgr.c +2 -2

+     *    heavy sweep.
      */
-    /*  if we are in DISCOVERING state - this means it is either in
-     *  initializing or wake up from STANDBY - run the heavy sweep */
     if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+        && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
         && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
         && sm->p_subn->opt.force_heavy_sweep == FALSE
         && sm->p_subn->force_heavy_sweep == FALSE

Signature · 1 line

1.5.1.4

Quoted 17 lines

Quoted 12 lines

Quoted 10 lines

Quoted 6 lines

Signature · 4 lines

To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4AF1BD1C.4090703-UdXhSnd/[email protected]

Yevgeny Kliteynik

author

2009-11-04 18:39 UTC #11 lore raw

Quoted 52 lines · Line Holen

Quoted 51 lines · Yevgeny Kliteynik
Quoted 50 lines · Line Holen
Quoted 40 lines · Yevgeny Kliteynik
Quoted 11 lines · Sasha Khapyorsky
Quoted 10 lines · Yevgeny Kliteynik
Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep
Quoted 1 line
Could such race happen when there are more than one node in a fabric?
I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?
Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.
The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.
Quoted 1 line
-- Yevgeny
At least for the 3.2 branch there is a general race regardless of where the SM is running. I haven't checked the current master, but I cannot recall seeing any patches related to this so I assume the race is still there.
There is a window between SM discovering a switch and clearing PSC for the same switch. The SM will not detect a state change on the switch ports during this time.

Quoted 8 lines

Quoted 4 lines
If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?
I think the switch shall send a trap whenever it sets the PSC bit. Once set I believe it will not send another trap until it is reset. Or do I misinterpret the spec ?

I may be wrong, but I thought that this is how things work:
- port state changes
- switch turns on PSC bit and starts sending traps
- SM gets the trap, sends trap repress
- switch gets trap repress and stops sending traps
- PSC is still on
- port state changes again (the same or any other port)
- switch turns on PSC bit (which doesn't matter as PSC is already on) and starts sending traps again
- etc...

Anyway, I'll double-check this issue.

-- Yevgeny

Quoted 66 lines

Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?

Quoted 1 line

I have a patch for the 3.2 branch that I can merge into master.

Sure, that would be nice :)

-- Yevgeny

Quoted 56 lines

Line

Quoted 53 lines

Quoted 49 lines

Sasha

Quoted 47 lines

Signed-off-by Yevgeny Kliteynik <[email protected]>

---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)

opensm/opensm/osm_state_mgr.c +2 -3

index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
      * Otherwise, this is probably our first discovery pass
      * or we are connected in loopback. In both cases do a
      * heavy sweep.
-     * Note: If we are connected in loopback we want a heavy
-     * sweep, since we will not be getting any traps if there is
-     * a lost connection.
+     * Note the following:
+     * 1. If we are connected in loopback we want a heavy sweep,

since we
+ * will not be getting any traps if there is a lost
connection.

opensm/opensm/osm_state_mgr.c +4 -0

+     * 2. If we are in DISCOVERING state - this means it is either in
+     *    initializing or wake up from STANDBY - run the heavy sweep.
+     * 3. If there is only one node in the fabric, and this node is a
+     *    switch, and OSM runs on top of it, there might be a race

when
+ * OSM starts running before the external ports are up -
run the

opensm/opensm/osm_state_mgr.c +2 -2

+     *    heavy sweep.
      */
-    /*  if we are in DISCOVERING state - this means it is either in
-     *  initializing or wake up from STANDBY - run the heavy sweep */
     if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+        && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
         && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
         && sm->p_subn->opt.force_heavy_sweep == FALSE
         && sm->p_subn->force_heavy_sweep == FALSE

Signature · 7 lines

1.5.1.4

-- 
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 6 lines

Quoted 4 lines

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4AF1CA61.2020007-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

Yevgeny Kliteynik

author

2009-11-05 07:29 UTC #13 lore raw

Quoted 73 lines · Yevgeny Kliteynik

Quoted 52 lines · Line Holen
Quoted 51 lines · Yevgeny Kliteynik
Quoted 50 lines · Line Holen
Quoted 40 lines · Yevgeny Kliteynik
Quoted 11 lines · Sasha Khapyorsky
Quoted 10 lines · Yevgeny Kliteynik
Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep
Quoted 1 line
Could such race happen when there are more than one node in a fabric?
I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?
Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.
The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.
Quoted 1 line
-- Yevgeny
At least for the 3.2 branch there is a general race regardless of where the SM is running. I haven't checked the current master, but I cannot recall seeing any patches related to this so I assume the race is still there.
There is a window between SM discovering a switch and clearing PSC for the same switch. The SM will not detect a state change on the switch ports during this time.
Quoted 8 lines
Quoted 4 lines
If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?
I think the switch shall send a trap whenever it sets the PSC bit. Once set I believe it will not send another trap until it is reset. Or do I misinterpret the spec ?
I may be wrong, but I thought that this is how things work:
- port state changes
- switch turns on PSC bit and starts sending traps
- SM gets the trap, sends trap repress
- switch gets trap repress and stops sending traps
- PSC is still on
- port state changes again (the same or any other port)
- switch turns on PSC bit (which doesn't matter as PSC is already on) and starts sending traps again
- etc...

Quoted 1 line

Anyway, I'll double-check this issue.

Yep, verified.
Switch sends traps regardless the PSC bit status.
Also, the spec doesn't link them together:

o14-5.1.1: If a switch supports Traps (PortInfo:
CapabilityMask.IsTrap-Supported is one), its SMA shall send trap 128 to the SM indicated by the PortInfo:MasterSMLID under any condition that would cause SwitchInfo:PortStateChange to be set to one. (See 14.2.5.4 SwitchInfo on page 827.)

-- Yevgeny

Quoted 72 lines

-- Yevgeny
Quoted 70 lines
Quoted 70 lines
Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?
Quoted 1 line
I have a patch for the 3.2 branch that I can merge into master.
Sure, that would be nice :)
-- Yevgeny
Quoted 60 lines
Line
Quoted 57 lines
Quoted 53 lines
Sasha
Quoted 51 lines
Signed-off-by Yevgeny Kliteynik <[email protected]>
---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)
opensm/opensm/osm_state_mgr.c +2 -3
index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
      * Otherwise, this is probably our first discovery pass
      * or we are connected in loopback. In both cases do a
      * heavy sweep.
-     * Note: If we are connected in loopback we want a heavy
-     * sweep, since we will not be getting any traps if there is
-     * a lost connection.
+     * Note the following:
+     * 1. If we are connected in loopback we want a heavy sweep,
since we
+ * will not be getting any traps if there is a lost
connection.
+ * 2. If we are in DISCOVERING state - this means it is
either in
+ * initializing or wake up from STANDBY - run the heavy
sweep.
+ * 3. If there is only one node in the fabric, and this node
is a
+ * switch, and OSM runs on top of it, there might be a race
when
+ * OSM starts running before the external ports are up -
run the
opensm/opensm/osm_state_mgr.c +1 -2
+     *    heavy sweep.
      */
-    /*  if we are in DISCOVERING state - this means it is either in
-     *  initializing or wake up from STANDBY - run the heavy 
sweep */
     if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+        && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
         && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
         && sm->p_subn->opt.force_heavy_sweep == FALSE
         && sm->p_subn->force_heavy_sweep == FALSE
Signature · 7 lines
1.5.1.4

-- 
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 12 lines

Quoted 6 lines

Quoted 4 lines

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4AF27EDB.6070604-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

Eli Dorfman (Voltaire)

2009-11-08 14:30 UTC #15 lore raw

Quoted 81 lines · Yevgeny Kliteynik

Quoted 74 lines · Yevgeny Kliteynik
Quoted 53 lines · Line Holen
Quoted 52 lines · Yevgeny Kliteynik
Quoted 51 lines · Line Holen
Quoted 41 lines · Yevgeny Kliteynik
Quoted 11 lines · Sasha Khapyorsky
Quoted 10 lines · Yevgeny Kliteynik
Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep
Quoted 2 lines
Could such race happen when there are more than one node in a fabric?
I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?
Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.
The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.
Quoted 1 line
-- Yevgeny
At least for the 3.2 branch there is a general race regardless of where the SM is running. I haven't checked the current master, but I cannot recall seeing any patches related to this so I assume the race is still there.
There is a window between SM discovering a switch and clearing PSC for the same switch. The SM will not detect a state change on the switch ports during this time.
Quoted 8 lines
Quoted 4 lines
If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?
I think the switch shall send a trap whenever it sets the PSC bit. Once set I believe it will not send another trap until it is reset. Or do I misinterpret the spec ?
I may be wrong, but I thought that this is how things work:
- port state changes
- switch turns on PSC bit and starts sending traps
- SM gets the trap, sends trap repress
- switch gets trap repress and stops sending traps
- PSC is still on
- port state changes again (the same or any other port)
- switch turns on PSC bit (which doesn't matter as PSC is already on) and starts sending traps again
- etc...
Quoted 1 line
Anyway, I'll double-check this issue.
Yep, verified.
Switch sends traps regardless the PSC bit status.
Also, the spec doesn't link them together:

Quoted 6 lines

o14-5.1.1: If a switch supports Traps (PortInfo:
CapabilityMask.IsTrap-Supported is one), its SMA
shall send trap 128 to the SM indicated by the PortInfo:MasterSMLID
under any condition that would cause SwitchInfo:PortStateChange to be set
to one. (See 14.2.5.4 SwitchInfo on page 827.)

Trap will be sent according to the SMLID. After first bring up the SMLID is not set yet and trap will not be sent.
In that case the opensm would discover the change only by PSC bit.
For IS3 chips the PSC bit and/or trap were set only after one or more ports changed their state, so I don't understand how can the SM discover PSC bit set while all ports are down. Or is this a change in IS4?

Eli

Quoted 84 lines

-- Yevgeny
Quoted 82 lines
-- Yevgeny
Quoted 79 lines
Quoted 77 lines
Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?
Quoted 1 line
I have a patch for the 3.2 branch that I can merge into master.
Sure, that would be nice :)
-- Yevgeny
Quoted 62 lines
Line
Quoted 59 lines
Quoted 54 lines
Sasha
Quoted 52 lines
Signed-off-by Yevgeny Kliteynik <[email protected]>
---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)
opensm/opensm/osm_state_mgr.c +2 -3
index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
      * Otherwise, this is probably our first discovery pass
      * or we are connected in loopback. In both cases do a
      * heavy sweep.
-     * Note: If we are connected in loopback we want a heavy
-     * sweep, since we will not be getting any traps if there is
-     * a lost connection.
+     * Note the following:
+     * 1. If we are connected in loopback we want a heavy sweep,
since we
+ * will not be getting any traps if there is a lost
connection.
+ * 2. If we are in DISCOVERING state - this means it is
either in
+ * initializing or wake up from STANDBY - run the heavy
sweep.
+ * 3. If there is only one node in the fabric, and this
node is a
+ * switch, and OSM runs on top of it, there might be a race
when
+ * OSM starts running before the external ports are up -
run the
opensm/opensm/osm_state_mgr.c +1 -1
+     *    heavy sweep.
      */
-    /*  if we are in DISCOVERING state - this means it is
either in
- * initializing or wake up from STANDBY - run the heavy
sweep */
     if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+        && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
         && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
         && sm->p_subn->opt.force_heavy_sweep == FALSE
         && sm->p_subn->force_heavy_sweep == FALSE
Signature · 7 lines
1.5.1.4

-- 
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 10 lines

Quoted 5 lines

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Yevgeny Kliteynik

author

2009-11-09 08:18 UTC #17 lore raw

Quoted 86 lines · Eli Dorfman (Voltaire)

Quoted 78 lines · Yevgeny Kliteynik
Quoted 72 lines · Yevgeny Kliteynik
Quoted 53 lines · Line Holen
Quoted 52 lines · Yevgeny Kliteynik
Quoted 51 lines · Line Holen
Quoted 41 lines · Yevgeny Kliteynik
Quoted 11 lines · Sasha Khapyorsky
Quoted 10 lines · Yevgeny Kliteynik
Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep
Quoted 2 lines
Could such race happen when there are more than one node in a fabric?
I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?
Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.
The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.
Quoted 1 line
-- Yevgeny
At least for the 3.2 branch there is a general race regardless of where the SM is running. I haven't checked the current master, but I cannot recall seeing any patches related to this so I assume the race is still there.
There is a window between SM discovering a switch and clearing PSC for the same switch. The SM will not detect a state change on the switch ports during this time.
Quoted 7 lines
Quoted 4 lines
If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?
I think the switch shall send a trap whenever it sets the PSC bit. Once set I believe it will not send another trap until it is reset. Or do I misinterpret the spec ?
I may be wrong, but I thought that this is how things work:
- port state changes
- switch turns on PSC bit and starts sending traps
- SM gets the trap, sends trap repress
- switch gets trap repress and stops sending traps
- PSC is still on
- port state changes again (the same or any other port)
- switch turns on PSC bit (which doesn't matter as PSC is already on) and starts sending traps again
- etc...
Quoted 1 line
Anyway, I'll double-check this issue.
Yep, verified.
Switch sends traps regardless the PSC bit status.
Also, the spec doesn't link them together:
Quoted 6 lines
o14-5.1.1: If a switch supports Traps (PortInfo:
CapabilityMask.IsTrap-Supported is one), its SMA
shall send trap 128 to the SM indicated by the PortInfo:MasterSMLID
under any condition that would cause SwitchInfo:PortStateChange to be set
to one. (See 14.2.5.4 SwitchInfo on page 827.)

Quoted 3 lines

Trap will be sent according to the SMLID. After first bring up the SMLID is not set yet and trap will not be sent.
In that case the opensm would discover the change only by PSC bit.
For IS3 chips the PSC bit and/or trap were set only after one or more ports changed their state, so I don't understand how can the SM discover PSC bit set while all ports are down. Or is this a change in IS4?

It can happen when SM runs on the switch, not not host. In this case if all ports are going down, SM will see them all down and it will see PSC bit on.

-- Yevgeny

Quoted 93 lines

Eli
Quoted 81 lines
-- Yevgeny
Quoted 79 lines
-- Yevgeny
Quoted 77 lines
Quoted 76 lines
Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?
Quoted 1 line
I have a patch for the 3.2 branch that I can merge into master.
Sure, that would be nice :)
-- Yevgeny
Quoted 61 lines
Line
Quoted 59 lines
Quoted 54 lines
Sasha
Quoted 52 lines
Signed-off-by Yevgeny Kliteynik <[email protected]>
---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)
opensm/opensm/osm_state_mgr.c +2 -3
index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
      * Otherwise, this is probably our first discovery pass
      * or we are connected in loopback. In both cases do a
      * heavy sweep.
-     * Note: If we are connected in loopback we want a heavy
-     * sweep, since we will not be getting any traps if there is
-     * a lost connection.
+     * Note the following:
+     * 1. If we are connected in loopback we want a heavy sweep,
since we
+ * will not be getting any traps if there is a lost
connection.
+ * 2. If we are in DISCOVERING state - this means it is
either in
+ * initializing or wake up from STANDBY - run the heavy
sweep.
+ * 3. If there is only one node in the fabric, and this
node is a
+ * switch, and OSM runs on top of it, there might be a race
when
+ * OSM starts running before the external ports are up -
run the
opensm/opensm/osm_state_mgr.c +1 -1
+     *    heavy sweep.
      */
-    /*  if we are in DISCOVERING state - this means it is
either in
- * initializing or wake up from STANDBY - run the heavy
sweep */
     if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+        && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
         && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
         && sm->p_subn->opt.force_heavy_sweep == FALSE
         && sm->p_subn->force_heavy_sweep == FALSE
Signature · 7 lines
1.5.1.4

-- 
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Quoted 9 lines
Quoted 5 lines
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4AF7D040.2060807-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

Eli Dorfman (Voltaire)

2009-11-09 10:42 UTC #19 lore raw

Quoted 96 lines · Yevgeny Kliteynik

Quoted 87 lines · Eli Dorfman (Voltaire)
Quoted 78 lines · Yevgeny Kliteynik
Quoted 72 lines · Yevgeny Kliteynik
Quoted 53 lines · Line Holen
Quoted 52 lines · Yevgeny Kliteynik
Quoted 51 lines · Line Holen
Quoted 41 lines · Yevgeny Kliteynik
Quoted 11 lines · Sasha Khapyorsky
Quoted 10 lines · Yevgeny Kliteynik
Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep
Quoted 2 lines
Could such race happen when there are more than one node in a fabric?
I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?
Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.
The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.
Quoted 1 line
-- Yevgeny
At least for the 3.2 branch there is a general race regardless of where the SM is running. I haven't checked the current master, but I cannot recall seeing any patches related to this so I assume the race is still there.
There is a window between SM discovering a switch and clearing PSC for the same switch. The SM will not detect a state change on the switch ports during this time.
Quoted 7 lines
Quoted 4 lines
If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?
I think the switch shall send a trap whenever it sets the PSC bit. Once set I believe it will not send another trap until it is reset. Or do I misinterpret the spec ?
I may be wrong, but I thought that this is how things work:
- port state changes
- switch turns on PSC bit and starts sending traps
- SM gets the trap, sends trap repress
- switch gets trap repress and stops sending traps
- PSC is still on
- port state changes again (the same or any other port)
- switch turns on PSC bit (which doesn't matter as PSC is already on) and starts sending traps again
- etc...
Quoted 1 line
Anyway, I'll double-check this issue.
Yep, verified.
Switch sends traps regardless the PSC bit status.
Also, the spec doesn't link them together:
Quoted 7 lines
o14-5.1.1: If a switch supports Traps (PortInfo:
CapabilityMask.IsTrap-Supported is one), its SMA
shall send trap 128 to the SM indicated by the PortInfo:MasterSMLID
under any condition that would cause SwitchInfo:PortStateChange to
be set
to one. (See 14.2.5.4 SwitchInfo on page 827.)
Quoted 7 lines
Trap will be sent according to the SMLID. After first bring up the SMLID is not set yet and trap will not be sent.
In that case the opensm would discover the change only by PSC bit. For IS3 chips the PSC bit and/or trap were set only after one or more ports changed their state, so I don't understand how can the SM discover PSC bit set while all ports are down. Or is this a change in IS4?

Quoted 3 lines

It can happen when SM runs on the switch, not not host. In this case if all ports are going down, SM will see them all down and it will see PSC bit on.

So this patch is only for SM running on a switch which is the only node in the fabric? I don't see the race when there is more than one switch - please explain.
Also AFAIK the PSC bit is set only after any physical port state change.
So if we clear the PSC bit and only then get PortInfo we will still catch any new state change. right?

Eli

Quoted 105 lines

-- Yevgeny
Quoted 101 lines
Eli
Quoted 93 lines
-- Yevgeny
Quoted 87 lines
-- Yevgeny
Quoted 79 lines
Quoted 78 lines
Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?
Quoted 1 line
I have a patch for the 3.2 branch that I can merge into master.
Sure, that would be nice :)
-- Yevgeny
Quoted 63 lines
Line
Quoted 61 lines
Quoted 56 lines
Sasha
Quoted 54 lines
Signed-off-by Yevgeny Kliteynik <[email protected]>
---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)
opensm/opensm/osm_state_mgr.c +2 -3
index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
      * Otherwise, this is probably our first discovery pass
      * or we are connected in loopback. In both cases do a
      * heavy sweep.
-     * Note: If we are connected in loopback we want a heavy
-     * sweep, since we will not be getting any traps if there is
-     * a lost connection.
+     * Note the following:
+     * 1. If we are connected in loopback we want a heavy sweep,
since we
+ * will not be getting any traps if there is a lost
connection.
+ * 2. If we are in DISCOVERING state - this means it is
either in
+ * initializing or wake up from STANDBY - run the heavy
sweep.
+ * 3. If there is only one node in the fabric, and this
node is a
+ * switch, and OSM runs on top of it, there might be a
race
when
+ * OSM starts running before the external ports are up -
run the
opensm/opensm/osm_state_mgr.c +1 -1
+     *    heavy sweep.
      */
-    /*  if we are in DISCOVERING state - this means it is
either in
- * initializing or wake up from STANDBY - run the heavy
sweep */
     if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+        && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
         && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
         && sm->p_subn->opt.force_heavy_sweep == FALSE
         && sm->p_subn->force_heavy_sweep == FALSE
Signature · 8 lines
1.5.1.4

-- 
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at 
http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Yevgeny Kliteynik

author

2009-11-09 11:09 UTC #21 lore raw

Quoted 95 lines · Eli Dorfman (Voltaire)

Quoted 94 lines · Yevgeny Kliteynik
Quoted 93 lines · Eli Dorfman (Voltaire)
Quoted 78 lines · Yevgeny Kliteynik
Quoted 72 lines · Yevgeny Kliteynik
Quoted 53 lines · Line Holen
Quoted 52 lines · Yevgeny Kliteynik
Quoted 51 lines · Line Holen
Quoted 41 lines · Yevgeny Kliteynik
Quoted 11 lines · Sasha Khapyorsky
Quoted 10 lines · Yevgeny Kliteynik
Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep
Quoted 2 lines
Could such race happen when there are more than one node in a fabric?
I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?
Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.
The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.
Quoted 1 line
-- Yevgeny
At least for the 3.2 branch there is a general race regardless of where the SM is running. I haven't checked the current master, but I cannot recall seeing any patches related to this so I assume the race is still there.
There is a window between SM discovering a switch and clearing PSC for the same switch. The SM will not detect a state change on the switch ports during this time.
Quoted 7 lines
Quoted 4 lines
If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?
I think the switch shall send a trap whenever it sets the PSC bit. Once set I believe it will not send another trap until it is reset. Or do I misinterpret the spec ?
I may be wrong, but I thought that this is how things work:
- port state changes
- switch turns on PSC bit and starts sending traps
- SM gets the trap, sends trap repress
- switch gets trap repress and stops sending traps
- PSC is still on
- port state changes again (the same or any other port)
- switch turns on PSC bit (which doesn't matter as PSC is already on) and starts sending traps again
- etc...
Quoted 1 line
Anyway, I'll double-check this issue.
Yep, verified.
Switch sends traps regardless the PSC bit status.
Also, the spec doesn't link them together:
Quoted 7 lines
o14-5.1.1: If a switch supports Traps (PortInfo:
CapabilityMask.IsTrap-Supported is one), its SMA
shall send trap 128 to the SM indicated by the PortInfo:MasterSMLID
under any condition that would cause SwitchInfo:PortStateChange to
be set
to one. (See 14.2.5.4 SwitchInfo on page 827.)
Trap will be sent according to the SMLID. After first bring up the SMLID is not set yet and trap will not be sent.
In that case the opensm would discover the change only by PSC bit. For IS3 chips the PSC bit and/or trap were set only after one or more ports changed their state, so I don't understand how can the SM discover PSC bit set while all ports are down. Or is this a change in IS4?

Quoted 6 lines

Quoted 3 lines
It can happen when SM runs on the switch, not not host. In this case if all ports are going down, SM will see them all down and it will see PSC bit on.
So this patch is only for SM running on a switch which is the only node in the fabric? I don't see the race when there is more than one switch - please explain.

Quoting from above:

The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover.

Quoted 1 line

Also AFAIK the PSC bit is set only after any physical port state change.

Yes, but it is set only once.
Meanwhile, the ports can change from up to down,
then SM discovers them, and then from down to up.

-- Yevgeny

Quoted 107 lines

So if we clear the PSC bit and only then get PortInfo we will still catch any new state change. right?
Eli
Quoted 100 lines
-- Yevgeny
Quoted 98 lines
Eli
Quoted 91 lines
-- Yevgeny
Quoted 85 lines
-- Yevgeny
Quoted 77 lines
Quoted 77 lines
Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?
Quoted 1 line
I have a patch for the 3.2 branch that I can merge into master.
Sure, that would be nice :)
-- Yevgeny
Quoted 62 lines
Line
Quoted 60 lines
Quoted 55 lines
Sasha
Quoted 53 lines
Signed-off-by Yevgeny Kliteynik <[email protected]>
---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)
opensm/opensm/osm_state_mgr.c +2 -3
index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
      * Otherwise, this is probably our first discovery pass
      * or we are connected in loopback. In both cases do a
      * heavy sweep.
-     * Note: If we are connected in loopback we want a heavy
-     * sweep, since we will not be getting any traps if there is
-     * a lost connection.
+     * Note the following:
+     * 1. If we are connected in loopback we want a heavy sweep,
since we
+ * will not be getting any traps if there is a lost
connection.
+ * 2. If we are in DISCOVERING state - this means it is
either in
+ * initializing or wake up from STANDBY - run the heavy
sweep.
+ * 3. If there is only one node in the fabric, and this
node is a
+ * switch, and OSM runs on top of it, there might be a
race
when
+ * OSM starts running before the external ports are up -
run the
opensm/opensm/osm_state_mgr.c +1 -1
+     *    heavy sweep.
      */
-    /*  if we are in DISCOVERING state - this means it is
either in
- * initializing or wake up from STANDBY - run the heavy
sweep */
     if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+        && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
         && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
         && sm->p_subn->opt.force_heavy_sweep == FALSE
         && sm->p_subn->force_heavy_sweep == FALSE
Signature · 7 lines
1.5.1.4

-- 
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4AF7F864.6030809-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

Eli Dorfman (Voltaire)

2009-11-09 13:54 UTC #23 lore raw

Quoted 105 lines · Yevgeny Kliteynik

Quoted 96 lines · Eli Dorfman (Voltaire)
Quoted 95 lines · Yevgeny Kliteynik
Quoted 94 lines · Eli Dorfman (Voltaire)
Quoted 78 lines · Yevgeny Kliteynik
Quoted 72 lines · Yevgeny Kliteynik
Quoted 53 lines · Line Holen
Quoted 52 lines · Yevgeny Kliteynik
Quoted 51 lines · Line Holen
Quoted 41 lines · Yevgeny Kliteynik
Quoted 11 lines · Sasha Khapyorsky
Quoted 10 lines · Yevgeny Kliteynik
Always do heavy sweep when there is only one node in the fabric, and this node is a switch, and SM runs on top of it - there may be a race when OSM starts running before the external ports are ports are up, or if they went through reset while SM was starting.
In this race switch brings up the ports and turns on the PSC bit, but OSM might get PortInfo before SwitchInfo, and it might see all ports as down, but PSC bit on. If that happens, OSM turns off PSC bit, and it will never see external ports again - it won't perform any heavy sweep, only light sweep
Quoted 2 lines
Could such race happen when there are more than one node in a fabric?
I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?
Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.
The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.
Quoted 1 line
-- Yevgeny
At least for the 3.2 branch there is a general race regardless of where the SM is running. I haven't checked the current master, but I cannot recall seeing any patches related to this so I assume the race is still there.
There is a window between SM discovering a switch and clearing PSC for the same switch. The SM will not detect a state change on the switch ports during this time.
Quoted 7 lines
Quoted 4 lines
If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?
I think the switch shall send a trap whenever it sets the PSC bit. Once set I believe it will not send another trap until it is reset. Or do I misinterpret the spec ?
I may be wrong, but I thought that this is how things work:
- port state changes
- switch turns on PSC bit and starts sending traps
- SM gets the trap, sends trap repress
- switch gets trap repress and stops sending traps
- PSC is still on
- port state changes again (the same or any other port)
- switch turns on PSC bit (which doesn't matter as PSC is already on) and starts sending traps again
- etc...
Quoted 1 line
Anyway, I'll double-check this issue.
Yep, verified.
Switch sends traps regardless the PSC bit status.
Also, the spec doesn't link them together:
Quoted 8 lines
o14-5.1.1: If a switch supports Traps (PortInfo:
CapabilityMask.IsTrap-Supported is one), its SMA shall send trap 128 to the SM indicated by the
PortInfo:MasterSMLID
under any condition that would cause SwitchInfo:PortStateChange to
be set
to one. (See 14.2.5.4 SwitchInfo on page 827.)
Trap will be sent according to the SMLID. After first bring up the SMLID is not set yet and trap will not be sent.
In that case the opensm would discover the change only by PSC bit. For IS3 chips the PSC bit and/or trap were set only after one or more ports changed their state, so I don't understand how can the SM discover PSC bit set while all ports are down. Or is this a change in IS4?
Quoted 7 lines
Quoted 3 lines
It can happen when SM runs on the switch, not not host. In this case if all ports are going down, SM will see them all down and it will see PSC bit on.
So this patch is only for SM running on a switch which is the only node in the fabric?
I don't see the race when there is more than one switch - please explain.

Quoted 5 lines

Quoting from above:
The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover.

I saw that but I don't understand how this can happen.
If PSC bit is set after *every* port state change and
SM clears PSC bit before reading PortInfo from the switch, then there is no race condition. As I mentioned before for IS3 switches that is correct.
Is there a different behavior with IS4 switches?

Quoted 4 lines

Quoted 2 lines
Also AFAIK the PSC bit is set only after any physical port state change.
Yes, but it is set only once.

PSC bit should be set after *every* port state change.

Eli

Quoted 127 lines

Meanwhile, the ports can change from up to down,
then SM discovers them, and then from down to up.
-- Yevgeny
Quoted 120 lines
So if we clear the PSC bit and only then get PortInfo we will still catch any new state change.
right?
Eli
Quoted 107 lines
-- Yevgeny
Quoted 105 lines
Eli
Quoted 97 lines
-- Yevgeny
Quoted 90 lines
-- Yevgeny
Quoted 82 lines
Quoted 82 lines
Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?
Quoted 1 line
I have a patch for the 3.2 branch that I can merge into master.
Sure, that would be nice :)
-- Yevgeny
Quoted 67 lines
Line
Quoted 65 lines
Quoted 59 lines
Sasha
Quoted 57 lines
Signed-off-by Yevgeny Kliteynik <[email protected]>
---
 opensm/opensm/osm_state_mgr.c |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)
opensm/opensm/osm_state_mgr.c +0 -2
index 4303d6e..537c855 100644
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
      * Otherwise, this is probably our first discovery pass
      * or we are connected in loopback. In both cases do a
      * heavy sweep.
-     * Note: If we are connected in loopback we want a heavy
-     * sweep, since we will not be getting any traps if
there is
opensm/opensm/osm_state_mgr.c +2 -1
-     * a lost connection.
+     * Note the following:
+     * 1. If we are connected in loopback we want a heavy
sweep,
since we
+ * will not be getting any traps if there is a lost
connection.
+ * 2. If we are in DISCOVERING state - this means it is
either in
+ * initializing or wake up from STANDBY - run the heavy
sweep.
+ * 3. If there is only one node in the fabric, and this
node is a
+ * switch, and OSM runs on top of it, there might be a
race
when
+ * OSM starts running before the external ports are
up -
run the
opensm/opensm/osm_state_mgr.c +1 -1
+     *    heavy sweep.
      */
-    /*  if we are in DISCOVERING state - this means it is
either in
- * initializing or wake up from STANDBY - run the heavy
sweep */
     if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+        && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
         && sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
         && sm->p_subn->opt.force_heavy_sweep == FALSE
         && sm->p_subn->force_heavy_sweep == FALSE
Signature · 8 lines
1.5.1.4

-- 
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at
http://vger.kernel.org/majordomo-info.html
Signature · 5 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at 
http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Yevgeny Kliteynik

author

2009-11-11 09:15 UTC #25 lore raw

Quoted 106 lines · Eli Dorfman (Voltaire)

Quoted 105 lines · Yevgeny Kliteynik
Quoted 96 lines · Eli Dorfman (Voltaire)
Quoted 95 lines · Yevgeny Kliteynik
Quoted 94 lines · Eli Dorfman (Voltaire)
Quoted 78 lines · Yevgeny Kliteynik
Quoted 72 lines · Yevgeny Kliteynik
Quoted 53 lines · Line Holen
Quoted 52 lines · Yevgeny Kliteynik
Quoted 51 lines · Line Holen
Quoted 41 lines · Yevgeny Kliteynik
Quoted 11 lines · Sasha Khapyorsky
Quoted 10 lines · Yevgeny Kliteynik
Always do heavy sweep when there is only one node in the
fabric, and this node is a switch, and SM runs on top of it -
there may be a race when OSM starts running before the
external ports are ports are up, or if they went through
reset while SM was starting.
In this race switch brings up the ports and turns on the
PSC bit, but OSM might get PortInfo before SwitchInfo, and it
might see all ports as down, but PSC bit on. If that happens,
OSM turns off PSC bit, and it will never see external ports
again - it won't perform any heavy sweep, only light sweep
Quoted 2 lines
Could such race happen when there are more than one node in a fabric?
I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?
Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.
The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.
Quoted 1 line
-- Yevgeny
At least for the 3.2 branch there is a general race regardless of where the SM is running. I haven't checked the current master, but I cannot recall seeing any patches related to this so I assume the race is still there.
There is a window between SM discovering a switch and clearing PSC for the same switch. The SM will not detect a state change on the switch ports during this time.
Quoted 7 lines
Quoted 4 lines
If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?
I think the switch shall send a trap whenever it sets the PSC bit. Once set I believe it will not send another trap until it is reset. Or do I misinterpret the spec ?
I may be wrong, but I thought that this is how things work:
- port state changes
- switch turns on PSC bit and starts sending traps
- SM gets the trap, sends trap repress
- switch gets trap repress and stops sending traps
- PSC is still on
- port state changes again (the same or any other port)
- switch turns on PSC bit (which doesn't matter as PSC is already on) and starts sending traps again
- etc...
Quoted 1 line
Anyway, I'll double-check this issue.
Yep, verified.
Switch sends traps regardless the PSC bit status.
Also, the spec doesn't link them together:
Quoted 8 lines
o14-5.1.1: If a switch supports Traps (PortInfo:
CapabilityMask.IsTrap-Supported is one), its SMA shall send trap 128 to the SM indicated by the
PortInfo:MasterSMLID
under any condition that would cause SwitchInfo:PortStateChange to
be set
to one. (See 14.2.5.4 SwitchInfo on page 827.)
Trap will be sent according to the SMLID. After first bring up the SMLID is not set yet and trap will not be sent.
In that case the opensm would discover the change only by PSC bit. For IS3 chips the PSC bit and/or trap were set only after one or more ports changed their state, so I don't understand how can the SM discover PSC bit set while all ports are down. Or is this a change in IS4?
Quoted 6 lines
Quoted 3 lines
It can happen when SM runs on the switch, not not host. In this case if all ports are going down, SM will see them all down and it will see PSC bit on.
So this patch is only for SM running on a switch which is the only node in the fabric?
I don't see the race when there is more than one switch - please explain.
Quoting from above:

Quoted 7 lines

Quoted 3 lines
The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover.
I saw that but I don't understand how this can happen. If PSC bit is set after *every* port state change and SM clears PSC bit before reading PortInfo from the switch,

osm_node_info_rcv.c, ni_rcv_process_switch():
I see in the code that SM receives NodeInfo, then it requests SwitchInfo and right after that it requests PortInfo for all the ports w/o waiting for the SwitchInfo response. In addition to that, if it happens during the first master sweep, SM LID is not configured yet, or configured to the wrong value, so no traps will be received by the SM.

-- Yevgeny

Quoted 136 lines

then there is no race condition.
As I mentioned before for IS3 switches that is correct. Is there a different behavior with IS4 switches?
Quoted 2 lines
Quoted 1 line
Also AFAIK the PSC bit is set only after any physical port state change.
Yes, but it is set only once.
PSC bit should be set after *every* port state change.
Eli
Quoted 124 lines
Meanwhile, the ports can change from up to down,
then SM discovers them, and then from down to up.
-- Yevgeny
Quoted 118 lines
So if we clear the PSC bit and only then get PortInfo we will still catch any new state change.
right?
Eli
Quoted 106 lines
-- Yevgeny
Quoted 104 lines
Eli
Quoted 96 lines
-- Yevgeny
Quoted 89 lines
-- Yevgeny
Quoted 81 lines
Quoted 81 lines
Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?
Quoted 1 line
I have a patch for the 3.2 branch that I can merge into master.
Sure, that would be nice :)
-- Yevgeny
Quoted 66 lines
Line
Quoted 64 lines
Quoted 59 lines
Sasha
Quoted 57 lines
Signed-off-by: Yevgeny Kliteynik <[email protected]>
---
opensm/opensm/osm_state_mgr.c | 15 ++++++++++-----
1 files changed, 10 insertions(+), 5 deletions(-)
diff --git a/opensm/opensm/osm_state_mgr.c
b/opensm/opensm/osm_state_mgr.c
index 4303d6e..537c855 100644
--- a/opensm/opensm/osm_state_mgr.c
+++ b/opensm/opensm/osm_state_mgr.c
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
* Otherwise, this is probably our first discovery pass
* or we are connected in loopback. In both cases do a
* heavy sweep.
- * Note: If we are connected in loopback we want a heavy
- * sweep, since we will not be getting any traps if
there is
- * a lost connection.
+ * Note the following:
+ * 1. If we are connected in loopback we want a heavy
sweep,
since we
+ * will not be getting any traps if there is a lost
connection.
+ * 2. If we are in DISCOVERING state - this means it is
either in
+ * initializing or wake up from STANDBY - run the heavy
sweep.
+ * 3. If there is only one node in the fabric, and this
node is a
+ * switch, and OSM runs on top of it, there might be a
race
when
+ * OSM starts running before the external ports are
up -
run the
+ * heavy sweep.
*/
- /* if we are in DISCOVERING state - this means it is
either in
- * initializing or wake up from STANDBY - run the heavy
sweep */
if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+ && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
&& sm->p_subn->sm_state != IB_SMINFO_STATE_DISCOVERING
&& sm->p_subn->opt.force_heavy_sweep == FALSE
&& sm->p_subn->force_heavy_sweep == FALSE
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at
http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4AFA80A5.8080809-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

Eli Dorfman (Voltaire)

2009-11-12 08:05 UTC #27 lore raw

Quoted 121 lines · Yevgeny Kliteynik

Quoted 112 lines · Eli Dorfman (Voltaire)
Quoted 111 lines · Yevgeny Kliteynik
Quoted 101 lines · Eli Dorfman (Voltaire)
Quoted 100 lines · Yevgeny Kliteynik
Quoted 99 lines · Eli Dorfman (Voltaire)
Quoted 83 lines · Yevgeny Kliteynik
Quoted 77 lines · Yevgeny Kliteynik
Quoted 57 lines · Line Holen
Quoted 56 lines · Yevgeny Kliteynik
Quoted 55 lines · Line Holen
Quoted 41 lines · Yevgeny Kliteynik
Quoted 13 lines · Sasha Khapyorsky
On 12:26 Tue 03 Nov , Yevgeny Kliteynik wrote:
Always do heavy sweep when there is only one node in the
fabric, and this node is a switch, and SM runs on top of it -
there may be a race when OSM starts running before the
external ports are ports are up, or if they went through
reset while SM was starting.
In this race switch brings up the ports and turns on the
PSC bit, but OSM might get PortInfo before SwitchInfo, and it
might see all ports as down, but PSC bit on. If that happens,
OSM turns off PSC bit, and it will never see external ports
again - it won't perform any heavy sweep, only light sweep
Could such race happen when there are more than one node in a
fabric?
I think that my description of the race was misleading. The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover. I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can this race happen on some *other* switch and not the one SM is running on?
Well, I don't know. I have a hunch that it can't, but I couldn't prove it to myself yet.
The race on the managed switch is a special case because SM always sees port 0, and always gets responses to its SMP queries. On any other switch, if the ports were reset, SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down, and by the time SM got SwitchInfo with PSC bit the port was already up, so SM won't start discovery beyond this port. But this race would be fixed on the next heavy sweep, when SM will discover this port that it missed the previous time, whereas race on managed switch is fatal - SM won't ever do any heavy sweep.
Quoted 1 line
-- Yevgeny
At least for the 3.2 branch there is a general race regardless of
where the SM is running. I haven't checked the current master, but
I cannot recall seeing any patches related to this so I assume the race is still there.
There is a window between SM discovering a switch and clearing PSC
for the same switch. The SM will not detect a state change on the
switch ports during this time.
Quoted 8 lines
Quoted 4 lines
If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?
I think the switch shall send a trap whenever it sets the PSC bit. Once set I believe it will not send another trap until it is reset.
Or do I misinterpret the spec ?
I may be wrong, but I thought that this is how things work:
- port state changes
- switch turns on PSC bit and starts sending traps
- SM gets the trap, sends trap repress
- switch gets trap repress and stops sending traps
- PSC is still on
- port state changes again (the same or any other port)
- switch turns on PSC bit (which doesn't matter as PSC is already on) and starts sending traps again
- etc...
Quoted 1 line
Anyway, I'll double-check this issue.
Yep, verified.
Switch sends traps regardless the PSC bit status.
Also, the spec doesn't link them together:
Quoted 8 lines
o14-5.1.1: If a switch supports Traps (PortInfo:
CapabilityMask.IsTrap-Supported is one), its SMA shall send trap 128 to the SM indicated by the
PortInfo:MasterSMLID
under any condition that would cause SwitchInfo:PortStateChange to
be set
to one. (See 14.2.5.4 SwitchInfo on page 827.)
Trap will be sent according to the SMLID. After first bring up the SMLID is not set yet and trap will not be sent.
In that case the opensm would discover the change only by PSC bit. For IS3 chips the PSC bit and/or trap were set only after one or more ports changed their state, so I don't understand how can the SM discover PSC bit set while all ports are down. Or is this a change in IS4?
Quoted 7 lines
Quoted 3 lines
It can happen when SM runs on the switch, not not host. In this case if all ports are going down, SM will see them all down and it will see PSC bit on.
So this patch is only for SM running on a switch which is the only node in the fabric?
I don't see the race when there is more than one switch - please explain.
Quoting from above:
Quoted 7 lines
Quoted 3 lines
The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover.
I saw that but I don't understand how this can happen. If PSC bit is set after *every* port state change and SM clears PSC bit before reading PortInfo from the switch,

Quoted 7 lines

osm_node_info_rcv.c, ni_rcv_process_switch():
I see in the code that SM receives NodeInfo, then it requests SwitchInfo and right after that it requests PortInfo for all the ports w/o waiting for the SwitchInfo response. In addition to that, if it happens during the first master sweep, SM LID is not configured yet, or configured to the wrong value, so no traps will be received by the SM.

I agree that trap will not be received but still PSC bit in SwitchInfo will be set for any port transition. See also the spec:
"It is set to one anytime the PortState component in the
PortInfo of any ports transitions from Down to Initialize,
Initialize to Down, Armed to Down, or Active to Down as
a result of link state machine logic. Changes in Portstate
resulting from SubnSet() do not change this bit.
This bit is cleared by writing one, writing zero is
ignored."

The problem is that switch info PSC bit is cleared too late - in osm_ucast_mgr.c, ucast_mgr_set_fwd_top() and not earlier (i.e. in osm_sw_info_rcv.c, osm_si_rcv_process_existing()).
The PSC bit should be cleared before SM reads the PortInfo from that switch.
In that case any PSC change will trigger discovery of new ports even if trap is missed.

Eli

Quoted 152 lines

-- Yevgeny
Quoted 148 lines
then there is no race condition.
As I mentioned before for IS3 switches that is correct. Is there a different behavior with IS4 switches?
Quoted 3 lines
Quoted 2 lines
Also AFAIK the PSC bit is set only after any physical port state change.
Yes, but it is set only once.
PSC bit should be set after *every* port state change.
Eli
Quoted 130 lines
Meanwhile, the ports can change from up to down,
then SM discovers them, and then from down to up.
-- Yevgeny
Quoted 124 lines
So if we clear the PSC bit and only then get PortInfo we will still catch any new state change.
right?
Eli
Quoted 111 lines
-- Yevgeny
Quoted 109 lines
Eli
Quoted 101 lines
-- Yevgeny
Quoted 94 lines
-- Yevgeny
Quoted 86 lines
Quoted 86 lines
Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?
Quoted 1 line
I have a patch for the 3.2 branch that I can merge into master.
Sure, that would be nice :)
-- Yevgeny
Quoted 70 lines
Line
Quoted 68 lines
Quoted 62 lines
Sasha
Signed-off-by: Yevgeny Kliteynik <[email protected]>
---
opensm/opensm/osm_state_mgr.c | 15 ++++++++++-----
1 files changed, 10 insertions(+), 5 deletions(-)
diff --git a/opensm/opensm/osm_state_mgr.c
b/opensm/opensm/osm_state_mgr.c
index 4303d6e..537c855 100644
--- a/opensm/opensm/osm_state_mgr.c
+++ b/opensm/opensm/osm_state_mgr.c
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
* Otherwise, this is probably our first discovery pass
* or we are connected in loopback. In both cases do a
* heavy sweep.
- * Note: If we are connected in loopback we want a heavy
- * sweep, since we will not be getting any traps if
there is
- * a lost connection.
+ * Note the following:
+ * 1. If we are connected in loopback we want a heavy
sweep,
since we
+ * will not be getting any traps if there is a lost
connection.
+ * 2. If we are in DISCOVERING state - this means it is
either in
+ * initializing or wake up from STANDBY - run the
heavy
sweep.
+ * 3. If there is only one node in the fabric, and this
node is a
+ * switch, and OSM runs on top of it, there might
be a
race
when
+ * OSM starts running before the external ports are
up -
run the
+ * heavy sweep.
*/
- /* if we are in DISCOVERING state - this means it is
either in
- * initializing or wake up from STANDBY - run the heavy
sweep */
if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+ && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
&& sm->p_subn->sm_state !=
IB_SMINFO_STATE_DISCOVERING
&& sm->p_subn->opt.force_heavy_sweep == FALSE
&& sm->p_subn->force_heavy_sweep == FALSE
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at
http://vger.kernel.org/majordomo-info.html
Signature · 5 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at
http://vger.kernel.org/majordomo-info.html
Signature · 5 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at 
http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Yevgeny Kliteynik

author

2009-11-12 12:50 UTC #29 lore raw

Quoted 138 lines · Eli Dorfman (Voltaire)

Quoted 119 lines · Yevgeny Kliteynik
Quoted 110 lines · Eli Dorfman (Voltaire)
Quoted 109 lines · Yevgeny Kliteynik
Quoted 108 lines · Eli Dorfman (Voltaire)
Quoted 100 lines · Yevgeny Kliteynik
Quoted 92 lines · Eli Dorfman (Voltaire)
Quoted 83 lines · Yevgeny Kliteynik
Quoted 77 lines · Yevgeny Kliteynik
Quoted 57 lines · Line Holen
Quoted 56 lines · Yevgeny Kliteynik
Quoted 55 lines · Line Holen
Quoted 42 lines · Yevgeny Kliteynik
Sasha Khapyorsky wrote:
On 12:26 Tue 03 Nov , Yevgeny Kliteynik wrote:
Always do heavy sweep when there is only one node in the
fabric, and this node is a switch, and SM runs on top of it -
there may be a race when OSM starts running before the
external ports are ports are up, or if they went through
reset while SM was starting.
In this race switch brings up the ports and turns on the
PSC bit, but OSM might get PortInfo before SwitchInfo, and it
might see all ports as down, but PSC bit on. If that happens,
OSM turns off PSC bit, and it will never see external ports
again - it won't perform any heavy sweep, only light sweep
Could such race happen when there are more than one node in a
fabric?
I think that my description of the race was misleading.
The race can happen on *any* fabric when SM runs on switch.
But when it does happen, SM thinks that the whole subnet
is just one switch - that's what it managed to discover.
I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can
this race happen on some *other* switch and not the one
SM is running on?
Well, I don't know. I have a hunch that it can't, but I
couldn't prove it to myself yet.
The race on the managed switch is a special case because
SM always sees port 0, and always gets responses to its
SMP queries. On any other switch, if the ports were reset,
SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down,
and by the time SM got SwitchInfo with PSC bit the port
was already up, so SM won't start discovery beyond this
port. But this race would be fixed on the next heavy sweep,
when SM will discover this port that it missed the previous
time, whereas race on managed switch is fatal - SM won't
ever do any heavy sweep.
-- Yevgeny
At least for the 3.2 branch there is a general race regardless of
where the SM is running. I haven't checked the current master, but
I cannot recall seeing any patches related to this so I assume the race is still there.
There is a window between SM discovering a switch and clearing PSC
for the same switch. The SM will not detect a state change on the
switch ports during this time.
Quoted 8 lines
Quoted 4 lines
If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?
I think the switch shall send a trap whenever it sets the PSC bit. Once set I believe it will not send another trap until it is reset.
Or do I misinterpret the spec ?
I may be wrong, but I thought that this is how things work:
- port state changes
- switch turns on PSC bit and starts sending traps
- SM gets the trap, sends trap repress
- switch gets trap repress and stops sending traps
- PSC is still on
- port state changes again (the same or any other port)
- switch turns on PSC bit (which doesn't matter as PSC is already on) and starts sending traps again
- etc...
Quoted 1 line
Anyway, I'll double-check this issue.
Yep, verified.
Switch sends traps regardless the PSC bit status.
Also, the spec doesn't link them together:
Quoted 8 lines
o14-5.1.1: If a switch supports Traps (PortInfo:
CapabilityMask.IsTrap-Supported is one), its SMA shall send trap 128 to the SM indicated by the
PortInfo:MasterSMLID
under any condition that would cause SwitchInfo:PortStateChange to
be set
to one. (See 14.2.5.4 SwitchInfo on page 827.)
Quoted 7 lines
Trap will be sent according to the SMLID. After first bring up the SMLID is not set yet and trap will not be sent.
In that case the opensm would discover the change only by PSC bit. For IS3 chips the PSC bit and/or trap were set only after one or more ports changed their state, so I don't understand how can the SM discover PSC bit set while all ports are down. Or is this a change in IS4?
Quoted 3 lines
It can happen when SM runs on the switch, not not host. In this case if all ports are going down, SM will see them all down and it will see PSC bit on.
So this patch is only for SM running on a switch which is the only node in the fabric?
I don't see the race when there is more than one switch - please explain.
Quoted 8 lines
Quoted 5 lines
Quoting from above:
The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover.
I saw that but I don't understand how this can happen. If PSC bit is set after *every* port state change and SM clears PSC bit before reading PortInfo from the switch,
Quoted 7 lines
osm_node_info_rcv.c, ni_rcv_process_switch():
I see in the code that SM receives NodeInfo, then it requests SwitchInfo and right after that it requests PortInfo for all the ports w/o waiting for the SwitchInfo response. In addition to that, if it happens during the first master sweep, SM LID is not configured yet, or configured to the wrong value, so no traps will be received by the SM.
I agree that trap will not be received but still PSC bit in SwitchInfo will be set for any port transition. See also the spec:
"It is set to one anytime the PortState component in the
PortInfo of any ports transitions from Down to Initialize,
Initialize to Down, Armed to Down, or Active to Down as
a result of link state machine logic. Changes in Portstate
resulting from SubnSet() do not change this bit.
This bit is cleared by writing one, writing zero is
ignored."

Quoted 3 lines

The problem is that switch info PSC bit is cleared too late - in osm_ucast_mgr.c, ucast_mgr_set_fwd_top() and not earlier (i.e. in osm_sw_info_rcv.c, osm_si_rcv_process_existing()).
The PSC bit should be cleared before SM reads the PortInfo from that switch.

This may be true, but do you really want to require Set(SwitchInfo) *completion* before getting other PortInfo MADs from this switch?
First of all, this means that you will have more than one Set(SwitchInfo) for the switch (during the discovery you don't know what is the LFT top). Second, it will slow down the whole discovery - you will introduce lots of new barriers that will damage the whole asynchronous chain reaction of the discovery.

-- Yevgeny

Quoted 152 lines

In that case any PSC change will trigger discovery of new ports even if trap is missed.
Eli
Quoted 147 lines
-- Yevgeny
Quoted 145 lines
then there is no race condition.
As I mentioned before for IS3 switches that is correct. Is there a different behavior with IS4 switches?
Quoted 3 lines
Quoted 2 lines
Also AFAIK the PSC bit is set only after any physical port state change.
Yes, but it is set only once.
PSC bit should be set after *every* port state change.
Eli
Quoted 129 lines
Meanwhile, the ports can change from up to down,
then SM discovers them, and then from down to up.
-- Yevgeny
Quoted 123 lines
So if we clear the PSC bit and only then get PortInfo we will still catch any new state change.
right?
Eli
Quoted 110 lines
-- Yevgeny
Quoted 108 lines
Eli
Quoted 100 lines
-- Yevgeny
Quoted 93 lines
-- Yevgeny
Quoted 85 lines
Quoted 85 lines
Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?
Quoted 1 line
I have a patch for the 3.2 branch that I can merge into master.
Sure, that would be nice :)
-- Yevgeny
Quoted 70 lines
Line
Quoted 68 lines
Sasha
Signed-off-by: Yevgeny Kliteynik <[email protected]>
---
opensm/opensm/osm_state_mgr.c | 15 ++++++++++-----
1 files changed, 10 insertions(+), 5 deletions(-)
diff --git a/opensm/opensm/osm_state_mgr.c
b/opensm/opensm/osm_state_mgr.c
index 4303d6e..537c855 100644
--- a/opensm/opensm/osm_state_mgr.c
+++ b/opensm/opensm/osm_state_mgr.c
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
* Otherwise, this is probably our first discovery pass
* or we are connected in loopback. In both cases do a
* heavy sweep.
- * Note: If we are connected in loopback we want a heavy
- * sweep, since we will not be getting any traps if
there is
- * a lost connection.
+ * Note the following:
+ * 1. If we are connected in loopback we want a heavy
sweep,
since we
+ * will not be getting any traps if there is a lost
connection.
+ * 2. If we are in DISCOVERING state - this means it is
either in
+ * initializing or wake up from STANDBY - run the
heavy
sweep.
+ * 3. If there is only one node in the fabric, and this
node is a
+ * switch, and OSM runs on top of it, there might
be a
race
when
+ * OSM starts running before the external ports are
up -
run the
+ * heavy sweep.
*/
- /* if we are in DISCOVERING state - this means it is
either in
- * initializing or wake up from STANDBY - run the heavy
sweep */
if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+ && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
&& sm->p_subn->sm_state !=
IB_SMINFO_STATE_DISCOVERING
&& sm->p_subn->opt.force_heavy_sweep == FALSE
&& sm->p_subn->force_heavy_sweep == FALSE
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at
http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at
http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4AFC04A8.1040808-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

Eli Dorfman

2009-11-13 08:24 UTC #31 lore raw

Quoted 180 lines · Yevgeny Kliteynik

Quoted 172 lines · Eli Dorfman (Voltaire)
Quoted 151 lines · Yevgeny Kliteynik
Quoted 141 lines · Eli Dorfman (Voltaire)
Quoted 139 lines · Yevgeny Kliteynik
Quoted 126 lines · Eli Dorfman (Voltaire)
Quoted 124 lines · Yevgeny Kliteynik
Quoted 121 lines · Eli Dorfman (Voltaire)
Quoted 101 lines · Yevgeny Kliteynik
Quoted 93 lines · Yevgeny Kliteynik
Quoted 74 lines · Line Holen
Quoted 68 lines · Yevgeny Kliteynik
Quoted 65 lines · Line Holen
On 11/ 4/09 10:47 AM, Yevgeny Kliteynik wrote:
Sasha Khapyorsky wrote:
On 12:26 Tue 03 Nov , Yevgeny Kliteynik wrote:
Always do heavy sweep when there is only one node in the
fabric, and this node is a switch, and SM runs on top of it
-
there may be a race when OSM starts running before the
external ports are ports are up, or if they went through
reset while SM was starting.
In this race switch brings up the ports and turns on the
PSC bit, but OSM might get PortInfo before SwitchInfo, and
it
might see all ports as down, but PSC bit on. If that
happens,
OSM turns off PSC bit, and it will never see external ports
again - it won't perform any heavy sweep, only light sweep
Could such race happen when there are more than one node in a
fabric?
I think that my description of the race was misleading.
The race can happen on *any* fabric when SM runs on switch.
But when it does happen, SM thinks that the whole subnet
is just one switch - that's what it managed to discover.
I've actually seen it happening.
So the patch fixes this particular case.
So the next question that you would probably ask is can
this race happen on some *other* switch and not the one
SM is running on?
Well, I don't know. I have a hunch that it can't, but I
couldn't prove it to myself yet.
The race on the managed switch is a special case because
SM always sees port 0, and always gets responses to its
SMP queries. On any other switch, if the ports were reset,
SM won't get any response until the ports are up again.
Perhaps there might be a case where SM got some port as down,
and by the time SM got SwitchInfo with PSC bit the port
was already up, so SM won't start discovery beyond this
port. But this race would be fixed on the next heavy sweep,
when SM will discover this port that it missed the previous
time, whereas race on managed switch is fatal - SM won't
ever do any heavy sweep.
-- Yevgeny
At least for the 3.2 branch there is a general race
regardless of
where the SM is running. I haven't checked the current
master, but
I cannot recall seeing any patches related to this so I assume
the race is still there.
There is a window between SM discovering a switch and
clearing PSC
for the same switch. The SM will not detect a state change on
the
switch ports during this time.
Quoted 4 lines
If the port changes state during that period, the switch issues new trap 128, which (I think) should cause SM to re-discover the fabric once this discovery cycle is over. Is this correct?
Quoted 5 lines
I think the switch shall send a trap whenever it sets the PSC bit.
Once set I believe it will not send another trap until it is reset.
Or do I misinterpret the spec ?
I may be wrong, but I thought that this is how things work:
- port state changes
- switch turns on PSC bit and starts sending traps
- SM gets the trap, sends trap repress
- switch gets trap repress and stops sending traps
- PSC is still on
- port state changes again (the same or any other port)
- switch turns on PSC bit (which doesn't matter as PSC is already on) and starts sending traps again
- etc...
Quoted 1 line
Anyway, I'll double-check this issue.
Yep, verified.
Switch sends traps regardless the PSC bit status.
Also, the spec doesn't link them together:
Quoted 9 lines
o14-5.1.1: If a switch supports Traps (PortInfo:
CapabilityMask.IsTrap-Supported is one), its SMA
shall send trap 128 to the SM indicated by the PortInfo:MasterSMLID
under any condition that would cause SwitchInfo:PortStateChange to
be set
to one. (See 14.2.5.4 SwitchInfo on page 827.)
Trap will be sent according to the SMLID. After first bring up the SMLID is not set yet and trap will not be sent.
In that case the opensm would discover the change only by PSC bit. For IS3 chips the PSC bit and/or trap were set only after one or more
ports changed their state, so I don't understand how can the SM discover PSC bit set while all ports are down. Or is this a change in
IS4?
Quoted 8 lines
Quoted 3 lines
It can happen when SM runs on the switch, not not host. In this case if all ports are going down, SM will see them all down and it will see PSC bit on.
So this patch is only for SM running on a switch which is the only node in the fabric?
I don't see the race when there is more than one switch - please explain.
Quoting from above:
Quoted 7 lines
Quoted 3 lines
The race can happen on *any* fabric when SM runs on switch. But when it does happen, SM thinks that the whole subnet is just one switch - that's what it managed to discover.
I saw that but I don't understand how this can happen. If PSC bit is set after *every* port state change and SM clears PSC bit before reading PortInfo from the switch,
Quoted 7 lines
osm_node_info_rcv.c, ni_rcv_process_switch():
I see in the code that SM receives NodeInfo, then it requests SwitchInfo and right after that it requests PortInfo for all the ports w/o waiting for the SwitchInfo response. In addition to that, if it happens during the first master sweep, SM LID is not configured yet, or configured to the wrong value, so no traps will be received by the SM.
I agree that trap will not be received but still PSC bit in SwitchInfo will be set for any port transition.
See also the spec:
"It is set to one anytime the PortState component in the PortInfo of any ports transitions from Down to Initialize, Initialize to Down, Armed to Down, or Active to Down as
a result of link state machine logic. Changes in Portstate resulting from SubnSet() do not change this bit.
This bit is cleared by writing one, writing zero is ignored."
Quoted 6 lines
The problem is that switch info PSC bit is cleared too late - in osm_ucast_mgr.c, ucast_mgr_set_fwd_top()
and not earlier (i.e. in osm_sw_info_rcv.c, osm_si_rcv_process_existing()).
The PSC bit should be cleared before SM reads the PortInfo from that switch.

Quoted 3 lines

This may be true, but do you really want to require Set(SwitchInfo) *completion* before getting other PortInfo MADs from this switch?

I don't see any other good alternative and I'm sure this will solve all race conditions.
Most of the Set(SwitchInfo) request will be sent in parallel to other discovery MAD PortInfo, NodeInfo, etc.

Quoted 3 lines

First of all, this means that you will have more than one Set(SwitchInfo) for the switch (during the discovery you don't know what is the LFT top).

That is correct but number of switches is small and compared to all the MADs that are sent during discovery
this is a very small penalty.

Quoted 4 lines

Second, it will slow down
the whole discovery - you will introduce lots of new barriers that will damage the whole asynchronous chain reaction of the discovery.

not necessarily.
I suggest that when PSC bit is set in SwitchInfo response we will send Set(SwitchInfo) to clear the PSC bit and
in SwitchInfo response send Get(PortInfo) - this requires adding another flag to the mad context.
If PSC bit is not set or SM is not in Master state Get(PortInfo) will be done in the immediately.

what do you think?

Eli

Quoted 173 lines

-- Yevgeny
Quoted 168 lines
In that case any PSC change will trigger discovery of new ports even if trap is missed.
Eli
Quoted 157 lines
-- Yevgeny
Quoted 155 lines
then there is no race condition.
As I mentioned before for IS3 switches that is correct. Is there a different behavior with IS4 switches?
Quoted 4 lines
Quoted 2 lines
Also AFAIK the PSC bit is set only after any physical port state change.
Yes, but it is set only once.
PSC bit should be set after *every* port state change.
Eli
Quoted 137 lines
Meanwhile, the ports can change from up to down,
then SM discovers them, and then from down to up.
-- Yevgeny
Quoted 131 lines
So if we clear the PSC bit and only then get PortInfo we will still catch any new state change.
right?
Eli
Quoted 118 lines
-- Yevgeny
Quoted 116 lines
Eli
Quoted 102 lines
-- Yevgeny
Quoted 94 lines
-- Yevgeny
Quoted 85 lines
Quoted 85 lines
Or perhaps the more serious problem happens when SM LID is not configured yet on the switch, hence the trap is not going to the right place?
Quoted 1 line
I have a patch for the 3.2 branch that I can merge into master.
Sure, that would be nice :)
-- Yevgeny
Quoted 73 lines
Line
Sasha
Signed-off-by: Yevgeny Kliteynik
<[email protected]>
---
opensm/opensm/osm_state_mgr.c | 15 ++++++++++-----
1 files changed, 10 insertions(+), 5 deletions(-)
diff --git a/opensm/opensm/osm_state_mgr.c
b/opensm/opensm/osm_state_mgr.c
index 4303d6e..537c855 100644
--- a/opensm/opensm/osm_state_mgr.c
+++ b/opensm/opensm/osm_state_mgr.c
@@ -1062,13 +1062,18 @@ static void do_sweep(osm_sm_t * sm)
* Otherwise, this is probably our first discovery pass
* or we are connected in loopback. In both cases do a
* heavy sweep.
- * Note: If we are connected in loopback we want a
heavy
- * sweep, since we will not be getting any traps if
there is
- * a lost connection.
+ * Note the following:
+ * 1. If we are connected in loopback we want a heavy
sweep,
since we
+ * will not be getting any traps if there is a lost
connection.
+ * 2. If we are in DISCOVERING state - this means it is
either in
+ * initializing or wake up from STANDBY - run the
heavy
sweep.
+ * 3. If there is only one node in the fabric, and this
node is a
+ * switch, and OSM runs on top of it, there might
be a
race
when
+ * OSM starts running before the external ports are
up -
run the
+ * heavy sweep.
*/
- /* if we are in DISCOVERING state - this means it is
either in
- * initializing or wake up from STANDBY - run the
heavy
sweep */
if (cl_qmap_count(&sm->p_subn->sw_guid_tbl)
+ && cl_qmap_count(&sm->p_subn->node_guid_tbl) != 1
&& sm->p_subn->sm_state !=
IB_SMINFO_STATE_DISCOVERING
&& sm->p_subn->opt.force_heavy_sweep == FALSE
&& sm->p_subn->force_heavy_sweep == FALSE
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at
http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe
linux-rdma" in
the body of a message to [email protected]
More majordomo info at
http://vger.kernel.org/majordomo-info.html
Quoted 6 lines
Quoted 6 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Quoted 6 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 694d48600911130024vd803e5fhf835690742f14ba7-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Sasha Khapyorsky

2009-11-13 15:20 UTC #33 lore raw

Quoted 24 lines · Eli Dorfman

Quoted 22 lines · Yevgeny Kliteynik
Quoted 21 lines · Eli Dorfman (Voltaire)
Quoted 19 lines · Yevgeny Kliteynik
Quoted 17 lines · Eli Dorfman (Voltaire)
Quoted 15 lines · Yevgeny Kliteynik
Quoted 13 lines · Eli Dorfman (Voltaire)
Quoted 11 lines · Yevgeny Kliteynik
Quoted 9 lines · Eli Dorfman (Voltaire)
Quoted 7 lines · Yevgeny Kliteynik
Quoted 5 lines · Yevgeny Kliteynik
Quoted 3 lines · Line Holen
Quoted 1 line · Yevgeny Kliteynik

Quoted 7 lines

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Line Holen wrote:
On 11/ 4/09 10:47 AM, Yevgeny Kliteynik wrote:
Sasha Khapyorsky wrote:
On 12:26 Tue 03 Nov , Yevgeny Kliteynik wrote:

This is a nice and productive discussion. But please remove a context which is not related to the current point - it helps to follow thread.

[snip]

Quoted 9 lines

Quoted 6 lines
Quoted 2 lines
The PSC bit should be cleared before SM reads the PortInfo from that switch.
This may be true, but do you really want to require Set(SwitchInfo) *completion* before getting other PortInfo MADs from this switch?
I don't see any other good alternative and I'm sure this will solve all race conditions.

I agree with Eli - if we want to prevent PSC bit loss (which can be relevant for any subnet/SM topology), we need to clear it first and than to fetch PortInfo. Obviously this may have some performance penalty, but proper discovery is more important.

Quoted 4 lines

I suggest that when PSC bit is set in SwitchInfo response we will send Set(SwitchInfo) to clear the PSC bit and
in SwitchInfo response send Get(PortInfo) - this requires adding another flag to the mad context.

Another flag or reusing one of existing flag.

Quoted 4 lines

If PSC bit is not set or SM is not in Master state Get(PortInfo) will be done in the immediately.
what do you think?

Another potential issue is to avoid a secondary trap triggered sweep for cases when PSC change is already cached.

Basically looks like a right direction for me.

Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html