Skip to contentSkip to conversation

sctp: Don't add the shutdown timer if its already been added

PATCH Superseded Neil Horman 2 participants 7 messages Mar 8, 2016 – Apr 28, 2020

Subsystems (2)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sctp: Don't add the shutdown timer if its already been added

This BUG halt was recently reported:

PID: 2879 TASK: c16adaa0 CPU: 1 COMMAND: "sctpn"
#0 [f418dd28] crash_kexec at c04a7d8c
#1 [f418dd7c] oops_end at c0863e02
#2 [f418dd90] do_invalid_op at c040aaca
#3 [f418de28] error_code (via invalid_op) at c08631a5

    EAX: f34baac0  EBX: 00000090  ECX: f418deb0  EDX: f5542950  EBP: 00000000
    DS:  007b      ESI: f34ba800  ES:  007b      EDI: f418dea0  GS:  00e0
    CS:  0060      EIP: c046fa5e  ERR: ffffffff  EFLAGS: 00010286
 #4 [f418de5c] add_timer at c046fa5e
 #5 [f418de68] sctp_do_sm at f8db8c77 [sctp]
 #6 [f418df30] sctp_primitive_SHUTDOWN at f8dcc1b5 [sctp]
 #7 [f418df48] inet_shutdown at c080baf9
 #8 [f418df5c] sys_shutdown at c079eedf
 #9 [f418df70] sys_socketcall at c079fe88
    EAX: ffffffda  EBX: 0000000d  ECX: bfceea90  EDX: 0937af98
    DS:  007b      ESI: 0000000c  ES:  007b      EDI: b7150ae4
    SS:  007b      ESP: bfceea7c  EBP: bfceeaa8  GS:  0033
    CS:  0073      EIP: b775c424  ERR: 00000066  EFLAGS: 00000282

It appears that the side effect that starts the shutdown timer was processed multiple times, which can happen as multiple paths can trigger it. This of course leads to the BUG halt in add_timer getting called.

Fix seems pretty straightforward, just check before the timer is added if its already been started. If it has mod the timer instead to min(current expiration, new expiration)

Its been tested but not confirmed to fix the problem, as the issue has only occured in production environments where test kernels are enjoined from being installed. It appears to be a sane fix to me though.

CC Vlad Yasevich <[email protected]>
CC "David S. Miller" <[email protected]>
---
 net/sctp/sm_sideeffect.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
net/sctp/sm_sideeffect.c +18 -3
index b5327bb..2a3a71a 100644
@@ -1480,9 +1480,24 @@ static int sctp_cmd_interpreter(sctp_event_t event_type,
 			timeout = asoc->timeouts[cmd->obj.to];
 			BUG_ON(!timeout);
 
-			timer->expires = jiffies + timeout;
-			sctp_association_hold(asoc);
-			add_timer(timer);
+			/*
+			 * SCTP has a hard time with timer starts.  Because we process
+			 * timer starts as side effects, it can be hard to tell if we
+			 * have already started a timer or not, which leads to BUG
+			 * halts when we call add_timer. So here, instead of just starting
+			 * a timer, we warn if the timer is already started, and just mod
+			 * the timer with the shorter of the two expiration times
+			 */
+			if (WARN_ON_ONCE(timer_pending(timer))) {
+				if (time_after(timer->expires, (jiffies + timeout))) {
+					timer->expires = jiffies+timeout;
+					mod_timer(timer, timer->expires);
+				}
+			} else {
+				timer->expires = jiffies + timeout;
+				sctp_association_hold(asoc);
+				mod_timer(timer, timer->expires);
+			}
 			break;
 
 		case SCTP_CMD_TIMER_RESTART:
Signature · 1 line
2.5.0
#1 lore raw
Quoted 80 lines · Neil Horman

This BUG halt was recently reported:

PID: 2879 TASK: c16adaa0 CPU: 1 COMMAND: "sctpn"
#0 [f418dd28] crash_kexec at c04a7d8c
#1 [f418dd7c] oops_end at c0863e02
#2 [f418dd90] do_invalid_op at c040aaca
#3 [f418de28] error_code (via invalid_op) at c08631a5

    EAX: f34baac0  EBX: 00000090  ECX: f418deb0  EDX: f5542950  EBP: 00000000
    DS:  007b      ESI: f34ba800  ES:  007b      EDI: f418dea0  GS:  00e0
    CS:  0060      EIP: c046fa5e  ERR: ffffffff  EFLAGS: 00010286
 #4 [f418de5c] add_timer at c046fa5e
 #5 [f418de68] sctp_do_sm at f8db8c77 [sctp]
 #6 [f418df30] sctp_primitive_SHUTDOWN at f8dcc1b5 [sctp]
 #7 [f418df48] inet_shutdown at c080baf9
 #8 [f418df5c] sys_shutdown at c079eedf
 #9 [f418df70] sys_socketcall at c079fe88
    EAX: ffffffda  EBX: 0000000d  ECX: bfceea90  EDX: 0937af98
    DS:  007b      ESI: 0000000c  ES:  007b      EDI: b7150ae4
    SS:  007b      ESP: bfceea7c  EBP: bfceeaa8  GS:  0033
    CS:  0073      EIP: b775c424  ERR: 00000066  EFLAGS: 00000282

It appears that the side effect that starts the shutdown timer was processed multiple times, which can happen as multiple paths can trigger it. This of course leads to the BUG halt in add_timer getting called.

Fix seems pretty straightforward, just check before the timer is added if its already been started. If it has mod the timer instead to min(current expiration, new expiration)

Its been tested but not confirmed to fix the problem, as the issue has only occured in production environments where test kernels are enjoined from being installed. It appears to be a sane fix to me though.

CC Vlad Yasevich <[email protected]>
CC "David S. Miller" <[email protected]>
---
 net/sctp/sm_sideeffect.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
net/sctp/sm_sideeffect.c +18 -3
index b5327bb..2a3a71a 100644
@@ -1480,9 +1480,24 @@ static int sctp_cmd_interpreter(sctp_event_t event_type,
 			timeout = asoc->timeouts[cmd->obj.to];
 			BUG_ON(!timeout);
 
-			timer->expires = jiffies + timeout;
-			sctp_association_hold(asoc);
-			add_timer(timer);
+			/*
+			 * SCTP has a hard time with timer starts.  Because we process
+			 * timer starts as side effects, it can be hard to tell if we
+			 * have already started a timer or not, which leads to BUG
+			 * halts when we call add_timer. So here, instead of just starting
+			 * a timer, we warn if the timer is already started, and just mod
+			 * the timer with the shorter of the two expiration times
+			 */
+			if (WARN_ON_ONCE(timer_pending(timer))) {
+				if (time_after(timer->expires, (jiffies + timeout))) {
+					timer->expires = jiffies+timeout;
+					mod_timer(timer, timer->expires);
+				}
+			} else {
+				timer->expires = jiffies + timeout;
+				sctp_association_hold(asoc);
+				mod_timer(timer, timer->expires);
+			}
 			break;
 
 		case SCTP_CMD_TIMER_RESTART:
Signature · 6 lines
2.5.0

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Ping, any thoughts here?
Neil

#2 lore raw
In reply to Neil Horman
Quoted 50 lines · Neil Horman

This BUG halt was recently reported:

PID: 2879 TASK: c16adaa0 CPU: 1 COMMAND: "sctpn"
#0 [f418dd28] crash_kexec at c04a7d8c
#1 [f418dd7c] oops_end at c0863e02
#2 [f418dd90] do_invalid_op at c040aaca
#3 [f418de28] error_code (via invalid_op) at c08631a5

    EAX: f34baac0  EBX: 00000090  ECX: f418deb0  EDX: f5542950  EBP: 00000000
    DS:  007b      ESI: f34ba800  ES:  007b      EDI: f418dea0  GS:  00e0
    CS:  0060      EIP: c046fa5e  ERR: ffffffff  EFLAGS: 00010286
 #4 [f418de5c] add_timer at c046fa5e
 #5 [f418de68] sctp_do_sm at f8db8c77 [sctp]
 #6 [f418df30] sctp_primitive_SHUTDOWN at f8dcc1b5 [sctp]
 #7 [f418df48] inet_shutdown at c080baf9
 #8 [f418df5c] sys_shutdown at c079eedf
 #9 [f418df70] sys_socketcall at c079fe88
    EAX: ffffffda  EBX: 0000000d  ECX: bfceea90  EDX: 0937af98
    DS:  007b      ESI: 0000000c  ES:  007b      EDI: b7150ae4
    SS:  007b      ESP: bfceea7c  EBP: bfceeaa8  GS:  0033
    CS:  0073      EIP: b775c424  ERR: 00000066  EFLAGS: 00000282

It appears that the side effect that starts the shutdown timer was processed multiple times, which can happen as multiple paths can trigger it. This of course leads to the BUG halt in add_timer getting called.

Fix seems pretty straightforward, just check before the timer is added if its already been started. If it has mod the timer instead to min(current expiration, new expiration)

Its been tested but not confirmed to fix the problem, as the issue has only occured in production environments where test kernels are enjoined from being installed. It appears to be a sane fix to me though.

CC Vlad Yasevich <[email protected]>
CC "David S. Miller" <[email protected]>
---
 net/sctp/sm_sideeffect.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
net/sctp/sm_sideeffect.c +0 -2
index b5327bb..2a3a71a 100644
@@ -1480,9 +1480,24 @@ static int sctp_cmd_interpreter(sctp_event_t event_type,
 			timeout = asoc->timeouts[cmd->obj.to];
 			BUG_ON(!timeout);
 
-			timer->expires = jiffies + timeout;
-			sctp_association_hold(asoc);

net/sctp/sm_sideeffect.c static int sctp_cmd_interpreter(sctp_event_t event_type, · line 1491

Quoted 10 lines
net/sctp/sm_sideeffect.c +9 -1
-			add_timer(timer);
+			/*
+			 * SCTP has a hard time with timer starts.  Because we process
+			 * timer starts as side effects, it can be hard to tell if we
+			 * have already started a timer or not, which leads to BUG
+			 * halts when we call add_timer. So here, instead of just starting
+			 * a timer, we warn if the timer is already started, and just mod
+			 * the timer with the shorter of the two expiration times
+			 */
+			if (WARN_ON_ONCE(timer_pending(timer))) {

I'm not sure if this WARN is needed. As you said, this can happen. The user will have the trace logged but no action will be needed, seems to be just noise then.

Quoted 20 lines
net/sctp/sm_sideeffect.c +9 -0
+				if (time_after(timer->expires, (jiffies + timeout))) {
+					timer->expires = jiffies+timeout;
+					mod_timer(timer, timer->expires);
+				}
+			} else {
+				timer->expires = jiffies + timeout;
+				sctp_association_hold(asoc);
+				mod_timer(timer, timer->expires);
+			}
 			break;
 
 		case SCTP_CMD_TIMER_RESTART:
Signature · 6 lines
2.5.0

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#3 lore raw
In reply to Neil Horman
Quoted 49 lines · Marcelo Ricardo Leitner
Quoted 48 lines · Neil Horman

This BUG halt was recently reported:

PID: 2879 TASK: c16adaa0 CPU: 1 COMMAND: "sctpn"
#0 [f418dd28] crash_kexec at c04a7d8c
#1 [f418dd7c] oops_end at c0863e02
#2 [f418dd90] do_invalid_op at c040aaca
#3 [f418de28] error_code (via invalid_op) at c08631a5

    EAX: f34baac0  EBX: 00000090  ECX: f418deb0  EDX: f5542950  EBP: 00000000
    DS:  007b      ESI: f34ba800  ES:  007b      EDI: f418dea0  GS:  00e0
    CS:  0060      EIP: c046fa5e  ERR: ffffffff  EFLAGS: 00010286
 #4 [f418de5c] add_timer at c046fa5e
 #5 [f418de68] sctp_do_sm at f8db8c77 [sctp]
 #6 [f418df30] sctp_primitive_SHUTDOWN at f8dcc1b5 [sctp]
 #7 [f418df48] inet_shutdown at c080baf9
 #8 [f418df5c] sys_shutdown at c079eedf
 #9 [f418df70] sys_socketcall at c079fe88
    EAX: ffffffda  EBX: 0000000d  ECX: bfceea90  EDX: 0937af98
    DS:  007b      ESI: 0000000c  ES:  007b      EDI: b7150ae4
    SS:  007b      ESP: bfceea7c  EBP: bfceeaa8  GS:  0033
    CS:  0073      EIP: b775c424  ERR: 00000066  EFLAGS: 00000282

It appears that the side effect that starts the shutdown timer was processed multiple times, which can happen as multiple paths can trigger it. This of course leads to the BUG halt in add_timer getting called.

Fix seems pretty straightforward, just check before the timer is added if its already been started. If it has mod the timer instead to min(current expiration, new expiration)

Its been tested but not confirmed to fix the problem, as the issue has only occured in production environments where test kernels are enjoined from being installed. It appears to be a sane fix to me though.

CC Vlad Yasevich <[email protected]>
CC "David S. Miller" <[email protected]>
---
 net/sctp/sm_sideeffect.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
net/sctp/sm_sideeffect.c +0 -0
index b5327bb..2a3a71a 100644
@@ -1480,9 +1480,24 @@ static int sctp_cmd_interpreter(sctp_event_t event_type,
 			timeout = asoc->timeouts[cmd->obj.to];
 			BUG_ON(!timeout);
 
Quoted 17 lines

net/sctp/sm_sideeffect.c static int sctp_cmd_interpreter(sctp_event_t event_type, · line 1491

Quoted 12 lines
net/sctp/sm_sideeffect.c +9 -3
-			timer->expires = jiffies + timeout;
-			sctp_association_hold(asoc);
-			add_timer(timer);
+			/*
+			 * SCTP has a hard time with timer starts.  Because we process
+			 * timer starts as side effects, it can be hard to tell if we
+			 * have already started a timer or not, which leads to BUG
+			 * halts when we call add_timer. So here, instead of just starting
+			 * a timer, we warn if the timer is already started, and just mod
+			 * the timer with the shorter of the two expiration times
+			 */
+			if (WARN_ON_ONCE(timer_pending(timer))) {

I'm not sure if this WARN is needed. As you said, this can happen. The user will have the trace logged but no action will be needed, seems to be just noise then.

Thats a good point, I'll remove it and repost, thanks! Neil

Quoted 25 lines
Quoted 20 lines
net/sctp/sm_sideeffect.c +9 -0
+				if (time_after(timer->expires, (jiffies + timeout))) {
+					timer->expires = jiffies+timeout;
+					mod_timer(timer, timer->expires);
+				}
+			} else {
+				timer->expires = jiffies + timeout;
+				sctp_association_hold(asoc);
+				mod_timer(timer, timer->expires);
+			}
 			break;
 
 		case SCTP_CMD_TIMER_RESTART:
Signature · 6 lines
2.5.0

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw

PATCH 1/1 sctp: Don't add the shutdown timer if its already been added

In reply to Neil Horman

This BUG halt was reported a while back, but the patch somehow got missed:

PID: 2879 TASK: c16adaa0 CPU: 1 COMMAND: "sctpn"
#0 [f418dd28] crash_kexec at c04a7d8c
#1 [f418dd7c] oops_end at c0863e02
#2 [f418dd90] do_invalid_op at c040aaca
#3 [f418de28] error_code (via invalid_op) at c08631a5

    EAX: f34baac0  EBX: 00000090  ECX: f418deb0  EDX: f5542950  EBP: 00000000
    DS:  007b      ESI: f34ba800  ES:  007b      EDI: f418dea0  GS:  00e0
    CS:  0060      EIP: c046fa5e  ERR: ffffffff  EFLAGS: 00010286
 #4 [f418de5c] add_timer at c046fa5e
 #5 [f418de68] sctp_do_sm at f8db8c77 [sctp]
 #6 [f418df30] sctp_primitive_SHUTDOWN at f8dcc1b5 [sctp]
 #7 [f418df48] inet_shutdown at c080baf9
 #8 [f418df5c] sys_shutdown at c079eedf
 #9 [f418df70] sys_socketcall at c079fe88
    EAX: ffffffda  EBX: 0000000d  ECX: bfceea90  EDX: 0937af98
    DS:  007b      ESI: 0000000c  ES:  007b      EDI: b7150ae4
    SS:  007b      ESP: bfceea7c  EBP: bfceeaa8  GS:  0033
    CS:  0073      EIP: b775c424  ERR: 00000066  EFLAGS: 00000282

It appears that the side effect that starts the shutdown timer was processed multiple times, which can happen as multiple paths can trigger it. This of course leads to the BUG halt in add_timer getting called.

Fix seems pretty straightforward, just check before the timer is added if its already been started. If it has mod the timer instead to min(current expiration, new expiration)

Its been tested but not confirmed to fix the problem, as the issue has only occured in production environments where test kernels are enjoined from being installed. It appears to be a sane fix to me though. Also, recentely, Jere found a reproducer posted on list to confirm that this resolves the issues

CC Vlad Yasevich <[email protected]>
CC "David S. Miller" <[email protected]>
CC Jere Leppänen <[email protected]>
---
 net/sctp/sm_sideeffect.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
net/sctp/sm_sideeffect.c +18 -3
index 2bc29463e1dc..43a256b5a5f0 100644
@@ -1523,9 +1523,24 @@ static int sctp_cmd_interpreter(enum sctp_event_type event_type,
 			timeout = asoc->timeouts[cmd->obj.to];
 			BUG_ON(!timeout);
 
-			timer->expires = jiffies + timeout;
-			sctp_association_hold(asoc);
-			add_timer(timer);
+			/*
+			 * SCTP has a hard time with timer starts.  Because we process
+			 * timer starts as side effects, it can be hard to tell if we
+			 * have already started a timer or not, which leads to BUG
+			 * halts when we call add_timer. So here, instead of just starting
+			 * a timer, if the timer is already started, and just mod
+			 * the timer with the shorter of the two expiration times
+			 */
+			if (timer_pending(timer)) {
+				if (time_after(timer->expires, (jiffies + timeout))) {
+					timer->expires = jiffies+timeout;
+					mod_timer(timer, timer->expires);
+				}
+			} else {
+				timer->expires = jiffies + timeout;
+				sctp_association_hold(asoc);
+				mod_timer(timer, timer->expires);
+			}
 			break;
 
 		case SCTP_CMD_TIMER_RESTART:
Signature · 1 line
2.25.3
#5 lore raw
In reply to Neil Horman
Quoted 64 lines · Neil Horman

This BUG halt was reported a while back, but the patch somehow got missed:

PID: 2879 TASK: c16adaa0 CPU: 1 COMMAND: "sctpn"
#0 [f418dd28] crash_kexec at c04a7d8c
#1 [f418dd7c] oops_end at c0863e02
#2 [f418dd90] do_invalid_op at c040aaca
#3 [f418de28] error_code (via invalid_op) at c08631a5

    EAX: f34baac0  EBX: 00000090  ECX: f418deb0  EDX: f5542950  EBP: 00000000
    DS:  007b      ESI: f34ba800  ES:  007b      EDI: f418dea0  GS:  00e0
    CS:  0060      EIP: c046fa5e  ERR: ffffffff  EFLAGS: 00010286
 #4 [f418de5c] add_timer at c046fa5e
 #5 [f418de68] sctp_do_sm at f8db8c77 [sctp]
 #6 [f418df30] sctp_primitive_SHUTDOWN at f8dcc1b5 [sctp]
 #7 [f418df48] inet_shutdown at c080baf9
 #8 [f418df5c] sys_shutdown at c079eedf
 #9 [f418df70] sys_socketcall at c079fe88
    EAX: ffffffda  EBX: 0000000d  ECX: bfceea90  EDX: 0937af98
    DS:  007b      ESI: 0000000c  ES:  007b      EDI: b7150ae4
    SS:  007b      ESP: bfceea7c  EBP: bfceeaa8  GS:  0033
    CS:  0073      EIP: b775c424  ERR: 00000066  EFLAGS: 00000282

It appears that the side effect that starts the shutdown timer was processed multiple times, which can happen as multiple paths can trigger it. This of course leads to the BUG halt in add_timer getting called.

Fix seems pretty straightforward, just check before the timer is added if its already been started. If it has mod the timer instead to min(current expiration, new expiration)

Its been tested but not confirmed to fix the problem, as the issue has only occured in production environments where test kernels are enjoined from being installed. It appears to be a sane fix to me though. Also, recentely, Jere found a reproducer posted on list to confirm that this resolves the issues

CC Vlad Yasevich <[email protected]>
CC "David S. Miller" <[email protected]>
CC Jere Leppänen <[email protected]>
---
 net/sctp/sm_sideeffect.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
net/sctp/sm_sideeffect.c +8 -3
index 2bc29463e1dc..43a256b5a5f0 100644
@@ -1523,9 +1523,24 @@ static int sctp_cmd_interpreter(enum sctp_event_type event_type,
 			timeout = asoc->timeouts[cmd->obj.to];
 			BUG_ON(!timeout);
 
-			timer->expires = jiffies + timeout;
-			sctp_association_hold(asoc);
-			add_timer(timer);
+			/*
+			 * SCTP has a hard time with timer starts.  Because we process
+			 * timer starts as side effects, it can be hard to tell if we
+			 * have already started a timer or not, which leads to BUG
+			 * halts when we call add_timer. So here, instead of just starting
+			 * a timer, if the timer is already started, and just mod
+			 * the timer with the shorter of the two expiration times
+			 */

net/sctp/sm_sideeffect.c static int sctp_cmd_interpreter(enum sctp_event_type event_type, · line 1543

Quoted 10 lines
net/sctp/sm_sideeffect.c +10 -0
+			if (timer_pending(timer)) {
+				if (time_after(timer->expires, (jiffies + timeout))) {
+					timer->expires = jiffies+timeout;
+					mod_timer(timer, timer->expires);
+				}
+			} else {
+				timer->expires = jiffies + timeout;
+				sctp_association_hold(asoc);
+				mod_timer(timer, timer->expires);
+			}

Jere, can you then update it to use timer_reduce(), as you had suggested?

Quoted 6 lines
net/sctp/sm_sideeffect.c +0 -0
 			break;
 
 		case SCTP_CMD_TIMER_RESTART:
Signature · 1 line
2.25.3
#6 lore raw
In reply to Neil Horman
Quoted 70 lines · Marcelo Ricardo Leitner
Quoted 69 lines · Neil Horman

This BUG halt was reported a while back, but the patch somehow got missed:

PID: 2879 TASK: c16adaa0 CPU: 1 COMMAND: "sctpn"
#0 [f418dd28] crash_kexec at c04a7d8c
#1 [f418dd7c] oops_end at c0863e02
#2 [f418dd90] do_invalid_op at c040aaca
#3 [f418de28] error_code (via invalid_op) at c08631a5

    EAX: f34baac0  EBX: 00000090  ECX: f418deb0  EDX: f5542950  EBP: 00000000
    DS:  007b      ESI: f34ba800  ES:  007b      EDI: f418dea0  GS:  00e0
    CS:  0060      EIP: c046fa5e  ERR: ffffffff  EFLAGS: 00010286
 #4 [f418de5c] add_timer at c046fa5e
 #5 [f418de68] sctp_do_sm at f8db8c77 [sctp]
 #6 [f418df30] sctp_primitive_SHUTDOWN at f8dcc1b5 [sctp]
 #7 [f418df48] inet_shutdown at c080baf9
 #8 [f418df5c] sys_shutdown at c079eedf
 #9 [f418df70] sys_socketcall at c079fe88
    EAX: ffffffda  EBX: 0000000d  ECX: bfceea90  EDX: 0937af98
    DS:  007b      ESI: 0000000c  ES:  007b      EDI: b7150ae4
    SS:  007b      ESP: bfceea7c  EBP: bfceeaa8  GS:  0033
    CS:  0073      EIP: b775c424  ERR: 00000066  EFLAGS: 00000282

It appears that the side effect that starts the shutdown timer was processed multiple times, which can happen as multiple paths can trigger it. This of course leads to the BUG halt in add_timer getting called.

Fix seems pretty straightforward, just check before the timer is added if its already been started. If it has mod the timer instead to min(current expiration, new expiration)

Its been tested but not confirmed to fix the problem, as the issue has only occured in production environments where test kernels are enjoined from being installed. It appears to be a sane fix to me though. Also, recentely, Jere found a reproducer posted on list to confirm that this resolves the issues

CC Vlad Yasevich <[email protected]>
CC "David S. Miller" <[email protected]>
CC Jere Leppänen <[email protected]>
---
 net/sctp/sm_sideeffect.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
net/sctp/sm_sideeffect.c +13 -3
index 2bc29463e1dc..43a256b5a5f0 100644
@@ -1523,9 +1523,24 @@ static int sctp_cmd_interpreter(enum sctp_event_type event_type,
 			timeout = asoc->timeouts[cmd->obj.to];
 			BUG_ON(!timeout);
 
-			timer->expires = jiffies + timeout;
-			sctp_association_hold(asoc);
-			add_timer(timer);
+			/*
+			 * SCTP has a hard time with timer starts.  Because we process
+			 * timer starts as side effects, it can be hard to tell if we
+			 * have already started a timer or not, which leads to BUG
+			 * halts when we call add_timer. So here, instead of just starting
+			 * a timer, if the timer is already started, and just mod
+			 * the timer with the shorter of the two expiration times
+			 */
+			if (timer_pending(timer)) {
+				if (time_after(timer->expires, (jiffies + timeout))) {
+					timer->expires = jiffies+timeout;
+					mod_timer(timer, timer->expires);
+				}
Quoted 9 lines

net/sctp/sm_sideeffect.c static int sctp_cmd_interpreter(sctp_event_t event_type, · line 1500

Quoted 5 lines
net/sctp/sm_sideeffect.c +5 -0
+			} else {
+				timer->expires = jiffies + timeout;
+				sctp_association_hold(asoc);
+				mod_timer(timer, timer->expires);
+			}

Jere, can you then update it to use timer_reduce(), as you had suggested?

Sorry, I didn't read past all the reproducer gorp, I can take care of this. Neil

Quoted 7 lines
Quoted 6 lines
net/sctp/sm_sideeffect.c +0 -0
 			break;
 
 		case SCTP_CMD_TIMER_RESTART:
Signature · 1 line
2.25.3