time, ntp: Do not update time_state in middle of leap second [v3]
PATCH 1/1 time, ntp: Do not update time_state in middle of leap second [v3]
During leap second insertion testing it was noticed that a small window exists where the time_state could be reset such that time_state = TIME_OK, which then causes the leap second to not occur, or causes the entire leap second state machine to fail with time_state = TIME_INS at the end of the leap second.
The test did the following in userspace:
tx.modes = ADJ_STATUS;
tx.status = STA_INS;
/* send leap second request */
ret = adjtimex(&tx);
/* Check adjtimex output every half second */
now = tx.time.tv_sec;
while (now < next_leap+2) {
char buf[26];
ret = adjtimex(&tx);
ctime_r(&tx.time.tv_sec, buf);
buf[strlen(buf)-1] = 0; /*remove trailing\n */
printf("%s + %6ld us\t%s\n",
buf,
tx.time.tv_usec,
time_state_str(ret));
now = tx.time.tv_sec;
/* Sleep for another half second */
ts.tv_sec = 0;
ts.tv_nsec = NSEC_PER_SEC/2;
clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL);
}which was intended to mimic the insertion of a leap second. A successful run of the test would result in the time_state transitioning from TIME_OK to TIME_INS, then to TIME_OOP when the leap second was inserted, and then to TIME_WAIT when the leap second was completed. While running this code failures were seen in which the time_state remained TIME_INS, even though the leap second had occurred.
After some investigation it was noted that the test contained a small error:
the test does not reinitialize tx.status and reissues the STA_INS every 1/2 second. As a result of this broken test, the following failure was noticed (the output below is a mix of kernel messages and the output from the test program, the remaining annotations are printk's in the code and my own additional notes):
[ 942.952833] time_state [1] change from TIME_OK to TIME_INS
Fri Feb 13 18:59:51 2015 + 318126 us TIME_INS Fri Feb 13 18:59:51 2015 + 818167 us TIME_INS Fri Feb 13 18:59:52 2015 + 318208 us TIME_INS Fri Feb 13 18:59:52 2015 + 818248 us TIME_INS Fri Feb 13 18:59:53 2015 + 318290 us TIME_INS Fri Feb 13 18:59:53 2015 + 818331 us TIME_INS Fri Feb 13 18:59:54 2015 + 318372 us TIME_INS Fri Feb 13 18:59:54 2015 + 818413 us TIME_INS Fri Feb 13 18:59:55 2015 + 318454 us TIME_INS Fri Feb 13 18:59:55 2015 + 818495 us TIME_INS Fri Feb 13 18:59:56 2015 + 318534 us TIME_INS Fri Feb 13 18:59:56 2015 + 818575 us TIME_INS Fri Feb 13 18:59:57 2015 + 318617 us TIME_INS Fri Feb 13 18:59:57 2015 + 818660 us TIME_INS Fri Feb 13 18:59:58 2015 + 318702 us TIME_INS Fri Feb 13 18:59:58 2015 + 818744 us TIME_INS Fri Feb 13 18:59:59 2015 + 318785 us TIME_INS Fri Feb 13 18:59:59 2015 + 818837 us TIME_INS
[ 952.953143] time_state [4] change from TIME_INS to TIME_OOP [ 952.953150] Clock: inserting leap second 23:59:60 UTC [ 953.299905] process_adj_status: insert_leap_sec[1223] setting time_state back to TIME_OK [1, 1] <<< adjtimex() call every 1/2 second [ 953.299913] time_state [9] change from TIME_OOP to TIME_OK
Fri Feb 13 18:59:59 2015 + 318878 us TIME_OK
Fri Feb 13 18:59:59 2015 + 818931 us TIME_OK
[ 954.064237] time_state [1] change from TIME_OK to TIME_INS
Fri Feb 13 19:00:00 2015 + 318972 us TIME_INS Fri Feb 13 19:00:00 2015 + 819012 us TIME_INS Fri Feb 13 19:00:01 2015 + 319051 us TIME_INS Fri Feb 13 19:00:01 2015 + 819089 us TIME_INS Fri Feb 13 19:00:02 2015 + 319128 us TIME_INS
As previously stated, the time_state remains TIME_INS even though the leap second has already occurred @ 952.953150.
The test was changed to reset tx.status to 0 in the loop, and the test then succeeded with a 100% rate with the time state ending in TIME_WAIT.
While this is highly unlikely to ever happen in the real world it is still something we should protect against, as breaking the state machine is bad.
If the time_state == TIME_OOP (ie, the leap second is in progress) do not allow an external update to time_state in process_adj_status(). This will prevent external adjtimex() calls from breaking the leap second state machine.
[v2]: Only block time_state change when TIME_OOP [v3]: Write a much more detailed explanation of the bug.
--- kernel/time/ntp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/time/ntp.c +2 -1
@@ -535,7 +535,8 @@ void ntp_notify_cmos_timer(void) { }static inline void process_adj_status (struct timex *txc,struct timespec64 *ts) {if ((time_status &STA_PLL ) && !(txc->status &STA_PLL )) { - time_state =TIME_OK ; +if (time_state !=TIME_OOP ) + time_state =TIME_OK ; time_status =STA_UNSYNC ;/* restart PPS frequency calibration */ pps_reset_freq_interval ();
Signature · 1 line
1.7.9.3
Quoted 96 lines · Prarit Bhargava
During leap second insertion testing it was noticed that a small window exists where the
time_statecould be reset such thattime_state=TIME_OK, which then causes the leap second to not occur, or causes the entire leap second state machine to fail withtime_state=TIME_INSat the end of the leap second.The test did the following in userspace:
tx.modes = ADJ_STATUS; tx.status = STA_INS; /* send leap second request */ ret = adjtimex(&tx); /* Check adjtimex output every half second */ now = tx.time.tv_sec; while (now < next_leap+2) { char buf[26]; ret = adjtimex(&tx); ctime_r(&tx.time.tv_sec, buf); buf[strlen(buf)-1] = 0; /*remove trailing\n */ printf("%s + %6ld us\t%s\n", buf, tx.time.tv_usec, time_state_str(ret)); now = tx.time.tv_sec; /* Sleep for another half second */ ts.tv_sec = 0; ts.tv_nsec = NSEC_PER_SEC/2; clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL); }which was intended to mimic the insertion of a leap second. A successful run of the test would result in the
time_statetransitioning fromTIME_OKtoTIME_INS, then toTIME_OOPwhen the leap second was inserted, and then toTIME_WAITwhen the leap second was completed. While running this code failures were seen in which thetime_stateremainedTIME_INS, even though the leap second had occurred.After some investigation it was noted that the test contained a small error:
the test does not reinitialize tx.status and reissues theSTA_INSevery 1/2 second. As a result of this broken test, the following failure was noticed (the output below is a mix of kernel messages and the output from the test program, the remaining annotations are printk's in the code and my own additional notes):[ 942.952833] time_state [1] change from TIME_OK to TIME_INSFri Feb 13 18:59:51 2015 + 318126 us TIME_INS Fri Feb 13 18:59:51 2015 + 818167 us TIME_INS Fri Feb 13 18:59:52 2015 + 318208 us TIME_INS Fri Feb 13 18:59:52 2015 + 818248 us TIME_INS Fri Feb 13 18:59:53 2015 + 318290 us TIME_INS Fri Feb 13 18:59:53 2015 + 818331 us TIME_INS Fri Feb 13 18:59:54 2015 + 318372 us TIME_INS Fri Feb 13 18:59:54 2015 + 818413 us TIME_INS Fri Feb 13 18:59:55 2015 + 318454 us TIME_INS Fri Feb 13 18:59:55 2015 + 818495 us TIME_INS Fri Feb 13 18:59:56 2015 + 318534 us TIME_INS Fri Feb 13 18:59:56 2015 + 818575 us TIME_INS Fri Feb 13 18:59:57 2015 + 318617 us TIME_INS Fri Feb 13 18:59:57 2015 + 818660 us TIME_INS Fri Feb 13 18:59:58 2015 + 318702 us TIME_INS Fri Feb 13 18:59:58 2015 + 818744 us TIME_INS Fri Feb 13 18:59:59 2015 + 318785 us TIME_INS Fri Feb 13 18:59:59 2015 + 818837 us TIME_INS[ 952.953143] time_state [4] change from TIME_INS to TIME_OOP [ 952.953150] Clock: inserting leap second 23:59:60 UTC [ 953.299905] process_adj_status: insert_leap_sec[1223] setting time_state back to TIME_OK [1, 1] <<< adjtimex() call every 1/2 second [ 953.299913] time_state [9] change from TIME_OOP to TIME_OKFri Feb 13 18:59:59 2015 + 318878 us
TIME_OK
Fri Feb 13 18:59:59 2015 + 818931 usTIME_OK[ 954.064237] time_state [1] change from TIME_OK to TIME_INSFri Feb 13 19:00:00 2015 + 318972 us TIME_INS Fri Feb 13 19:00:00 2015 + 819012 us TIME_INS Fri Feb 13 19:00:01 2015 + 319051 us TIME_INS Fri Feb 13 19:00:01 2015 + 819089 us TIME_INS Fri Feb 13 19:00:02 2015 + 319128 us TIME_INSAs previously stated, the
time_stateremainsTIME_INSeven though the leap second has already occurred @ 952.953150.The test was changed to reset tx.status to 0 in the loop, and the test then succeeded with a 100% rate with the time state ending in
TIME_WAIT.While this is highly unlikely to ever happen in the real world it is still something we should protect against, as breaking the state machine is bad.
Quoted 7 lines
If the
time_state==TIME_OOP(ie, the leap second is in progress) do not allow an external update totime_stateinprocess_adj_status(). This will prevent externaladjtimex()calls from breaking the leap second state machine.[v2]: Only block
time_statechange whenTIME_OOP[v3]: Write a much more detailed explanation of the bug.
Ok, thanks for the more verbose explanation. Although this is more a history of what you've seen rather then the crux of the change.
To distill this down just a bit, the point is the usual mode for NTP time_state machine looks like:
TIME_OK -> TIME_INS -> TIME_OOP
| | v v TIME_DEL ------------> TIME_WAIT -(back)-> TIME_OK
(hopefully the ascii art survives here)
Now, from any of these states, currently if adjtimex is called w/ the STA_PLL bit cleared (after STA_PLL was set), we reset back to TIME_OK, effectively cancelling any transitions. (You'll have to imagine a line from any of the states back to TIME_OK, since that's going to be too ugly to do in ascii)
Your patch is trying to remove the line back from TIME_OOP back to TIME_OK. Basically stopping the ability to reset the ntp state during a leapsecond.
I do get that the behavior seen was strange due to a bug in the test code which caused unexpected cancellation of state, but I'm not sure if we should change the behavior to enforce that cancellation not be possible. I could imagine some logic which really wants to reset the state, which just by chance lands during a leap second, and the application is confused since the state change didn't occur as expected.
So I guess I'm not seeing that the state machine is actually "broken" in this case that you've outlined. If you can articulate better why the OOP -> OK transition is truly invalid, I'd be interested in hearing, but I'm not sure I want to risk a behavioral change unless there's wide agreement.
thanks
-john
Quoted 31 lines · John Stultz
Ok, thanks for the more verbose explanation. Although this is more a history of what you've seen rather then the crux of the change.
To distill this down just a bit, the point is the usual mode for NTP
time_statemachine looks like:
TIME_OK->TIME_INS->TIME_OOP| | v v TIME_DEL ------------> TIME_WAIT -(back)-> TIME_OK(hopefully the ascii art survives here)
Now, from any of these states, currently if adjtimex is called w/ the
STA_PLLbit cleared (afterSTA_PLLwas set), we reset back toTIME_OK, effectively cancelling any transitions. (You'll have to imagine a line from any of the states back toTIME_OK, since that's going to be too ugly to do in ascii)Your patch is trying to remove the line back from
TIME_OOPback toTIME_OK. Basically stopping the ability to reset the ntp state during a leapsecond.I do get that the behavior seen was strange due to a bug in the test code which caused unexpected cancellation of state, but I'm not sure if we should change the behavior to enforce that cancellation not be possible. I could imagine some logic which really wants to reset the state, which just by chance lands during a leap second, and the application is confused since the state change didn't occur as expected.
Quoted 5 lines
So I guess I'm not seeing that the state machine is actually "broken" in this case that you've outlined. If you can articulate better why the OOP -> OK transition is truly invalid, I'd be interested in hearing, but I'm not sure I want to risk a behavioral change unless there's wide agreement.
I think the only real problem occurs when the adjtimex is called in the the TIME_OOP state with STA_PLL cleared _and_ STA_INS set. In this case the state machine is reset to TIME_OK but goes back to TIME_INS on the next second_overflow, potentially causing another false leap second to be inserted on the following midnight.
The state machine is meant to only go back to TIME_INS once STA_INS is cleared and then set again - this is what the TIME_WAIT state is for.
In fact, I don't see a reason why the STA_PLL -> !STA_PLL transition should ever set the time_state to TIME_OK.
- When the STA_INS/STA_DEL flag is removed from the status, the state machine will end up in TIME_OK from any state.
- When STA_INS/STA_DEL is set in
the status, the state mchine will transition from TIME_OK to TIME_INS/TIME_DEL anyway.
I think the "time_status = TIME_OK" should be just dropped.
It has been added by eea83d896e318bda54be2d2770d2c5d6668d11db (ntp: NTP4 user space bits update) and it's not clear why. Roman?
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, SUSE CZ
Quoted 2 lines · Jiri Bohac
I think the only real problem occurs when the adjtimex is called in the the
TIME_OOPstate
... or the TIME_WAIT state ...
Quoted 23 lines
with
STA_PLLcleared _and_STA_INSset.
In this case the state machine is reset toTIME_OKbut goes back toTIME_INSon the nextsecond_overflow, potentially causing another false leap second to be inserted on the following midnight.The state machine is meant to only go back to
TIME_INSonceSTA_INSis cleared and then set again - this is what theTIME_WAITstate is for.In fact, I don't see a reason why the
STA_PLL-> !STA_PLL transition should ever set thetime_statetoTIME_OK.
- When the STA_INS/STA_DEL flag is removed from the status, the state machine will end up inTIME_OKfrom any state.
- When STA_INS/STA_DEL is set in
the status, the state mchine will transition fromTIME_OKto TIME_INS/TIME_DEL anyway.I think the "
time_status=TIME_OK" should be just dropped.It has been added by
eea83d896e318bda54be2d2770d2c5d6668d11db(ntp: NTP4 user space bits update) and it's not clear why. Roman?
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, SUSE CZ
Quoted 31 lines · John Stultz
Quoted 8 lines · Prarit Bhargava
which was intended to mimic the insertion of a leap second. A successful run of the test would result in the
time_statetransitioning fromTIME_OKtoTIME_INS, then toTIME_OOPwhen the leap second was inserted, and then toTIME_WAITwhen the leap second was completed. While running this code failures were seen in which thetime_stateremainedTIME_INS, even though the leap second had occurred.Ok, thanks for the more verbose explanation. Although this is more a history of what you've seen rather then the crux of the change.
To distill this down just a bit, the point is the usual mode for NTP
time_statemachine looks like:
TIME_OK->TIME_INS->TIME_OOP| | v v TIME_DEL ------------> TIME_WAIT -(back)-> TIME_OK(hopefully the ascii art survives here)
Now, from any of these states, currently if adjtimex is called w/ the
STA_PLLbit cleared (afterSTA_PLLwas set), we reset back toTIME_OK, effectively cancelling any transitions. (You'll have to imagine a line from any of the states back toTIME_OK, since that's going to be too ugly to do in ascii)
Quoted 3 lines
Your patch is trying to remove the line back from
TIME_OOPback toTIME_OK. Basically stopping the ability to reset the ntp state during a leapsecond.
Correct.
Quoted 8 lines
I do get that the behavior seen was strange due to a bug in the test code which caused unexpected cancellation of state, but I'm not sure if we should change the behavior to enforce that cancellation not be possible. I could imagine some logic which really wants to reset the state, which just by chance lands during a leap second, and the application is confused since the state change didn't occur as expected.
I think setting it in the middle of the leap second should be a NOOP. We all know how fragile this code has been in the past and allowing a state transition at that particular time isn't a good idea given the outcome that the state may remain TIME_INS.
Quoted 6 lines
So I guess I'm not seeing that the state machine is actually "broken" in this case that you've outlined. If you can articulate better why the OOP -> OK transition is truly invalid, I'd be interested in hearing, but I'm not sure I want to risk a behavioral change unless there's wide agreement.
I understand -- After thinking about it from your point of view I agree that calling it "broken" is not right. Perhaps a better way of looking at it is, as you also point out, if OOP -> OK is truly valid.
P.
Quoted 4 lines
thanks
-john
Hi,
I'm trying to understand what exactly is going on here...
Quoted 28 lines · Prarit Bhargava
The test did the following in userspace:
tx.modes = ADJ_STATUS; tx.status = STA_INS; /* send leap second request */ ret = adjtimex(&tx); /* Check adjtimex output every half second */ now = tx.time.tv_sec; while (now < next_leap+2) { char buf[26]; ret = adjtimex(&tx); ctime_r(&tx.time.tv_sec, buf); buf[strlen(buf)-1] = 0; /*remove trailing\n */ printf("%s + %6ld us\t%s\n", buf, tx.time.tv_usec, time_state_str(ret)); now = tx.time.tv_sec; /* Sleep for another half second */ ts.tv_sec = 0; ts.tv_nsec = NSEC_PER_SEC/2; clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL); }
Quoted 3 lines
After some investigation it was noted that the test contained a small error:
the test does not reinitialize tx.status and reissues theSTA_INSevery 1/2 second.
Prarit, can you explain who sets the STA_PLL flag, so that process_adj_status() detects a STA_PLL->!STA_PLL transition and goes to the branch that sets time_state = TIME_OK?
Is that ntpd running in parallel with your test program? If that is the case, you would eventually end up oscilating between the the TIME_INS and TIME_OK states anyway, even with your patch. ntpd will clear the STA_INS flag after midnight, the state machine will transition from TIME_WAIT to TIME_OK and your test program will set STA_INS again (fighting with ntpd which will clear the flag from time to time) ... right?
Thanks,
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, SUSE CZ
Quoted 33 lines · Jiri Bohac
Hi,
I'm trying to understand what exactly is going on here...
Quoted 28 lines · Prarit Bhargava
The test did the following in userspace:
tx.modes = ADJ_STATUS; tx.status = STA_INS; /* send leap second request */ ret = adjtimex(&tx); /* Check adjtimex output every half second */ now = tx.time.tv_sec; while (now < next_leap+2) { char buf[26]; ret = adjtimex(&tx); ctime_r(&tx.time.tv_sec, buf); buf[strlen(buf)-1] = 0; /*remove trailing\n */ printf("%s + %6ld us\t%s\n", buf, tx.time.tv_usec, time_state_str(ret)); now = tx.time.tv_sec; /* Sleep for another half second */ ts.tv_sec = 0; ts.tv_nsec = NSEC_PER_SEC/2; clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL); }
Quoted 7 lines
Quoted 3 lines
After some investigation it was noted that the test contained a small error:
the test does not reinitialize tx.status and reissues theSTA_INSevery 1/2 second.Prarit, can you explain who sets the
STA_PLLflag, so thatprocess_adj_status()detects a STA_PLL->!STA_PLL transition and goes to the branch that setstime_state=TIME_OK?
Jiri,
The test being run is:
https://github.com/johnstultz-work/timetests/blob/master/leap-a-day.c
prior to commit
https://github.com/johnstultz-work/timetests/commit/be4526e8b5d48cd108a8d2cf7f5c8fd763acf421
Quoted 2 lines
Is that ntpd running in parallel with your test program? If that
No -- ntpd is disabled (chronyd in the case of systemd + current Fedora).
P.
Quoted 7 lines · Prarit Bhargava
Quoted 3 lines · Jiri Bohac
Prarit, can you explain who sets the
STA_PLLflag, so thatprocess_adj_status()detects a STA_PLL->!STA_PLL transition and goes to the branch that setstime_state=TIME_OK?Jiri,
Quoted 7 lines
The test being run is:
https://github.com/johnstultz-work/timetests/blob/master/leap-a-day.c
prior to commit
https://github.com/johnstultz-work/timetests/commit/be4526e8b5d48cd108a8d2cf7f5c8fd763acf421
I can't make sense of the output of your test:
Quoted 14 lines · Prarit Bhargava
[ 942.952833] time_state [1] change from TIME_OK to TIME_INSFri Feb 13 18:59:51 2015 + 318126 us TIME_INS Fri Feb 13 18:59:51 2015 + 818167 us TIME_INS Fri Feb 13 18:59:52 2015 + 318208 us TIME_INS Fri Feb 13 18:59:52 2015 + 818248 us TIME_INS Fri Feb 13 18:59:53 2015 + 318290 us TIME_INS Fri Feb 13 18:59:53 2015 + 818331 us TIME_INS Fri Feb 13 18:59:54 2015 + 318372 us TIME_INS Fri Feb 13 18:59:54 2015 + 818413 us TIME_INS Fri Feb 13 18:59:55 2015 + 318454 us TIME_INS Fri Feb 13 18:59:55 2015 + 818495 us TIME_INS Fri Feb 13 18:59:56 2015 + 318534 us TIME_INS Fri Feb 13 18:59:56 2015 + 818575 us TIME_INS
Why did the test program print the above lines? It's supposed to sleep until 3 seconds prior to the midnight:
/* Wake up 3 seconds before leap */
ts.tv_sec = next_leap - 3;
ts.tv_nsec = 0;
while(clock_nanosleep(CLOCK_REALTIME, TIMER_ABSTIME, &ts, NULL))
printf("Something woke us up, returning to sleep\n");Quoted 7 lines
Fri Feb 13 18:59:57 2015 + 318617 us TIME_INS Fri Feb 13 18:59:57 2015 + 818660 us TIME_INS Fri Feb 13 18:59:58 2015 + 318702 us TIME_INS Fri Feb 13 18:59:58 2015 + 818744 us TIME_INS Fri Feb 13 18:59:59 2015 + 318785 us TIME_INS Fri Feb 13 18:59:59 2015 + 818837 us TIME_INS
Quoted 5 lines
[ 952.953143] time_state [4] change from TIME_INS to TIME_OOP [ 952.953150] Clock: inserting leap second 23:59:60 UTC [ 953.299905] process_adj_status: insert_leap_sec[1223] setting time_state back to TIME_OK [1, 1] <<< adjtimex() call every 1/2 second [ 953.299913] time_state [9] change from TIME_OOP to TIME_OK
2) The only place where the test program sets STA_PLL is in clear_time_state(); It clears it right after that.
clear_time_state() is not called inside the while "(now < next_leap+2)" loop, except in the SIGINT/SIGKILL handler. Did you send signals to the program at this point?
If not, I can't understand how the status went from STA_PLL to !STA_PLL and thus why time_state went to TIME_OK
Signature · 2 lines
Jiri Bohac <[email protected]> SUSE Labs, SUSE CZ