Skip to contentSkip to conversation

time, ntp: Do not update time_state in middle of leap second [v3]

PATCH In review Prarit Bhargava 3 participants 8 messages Feb 12, 2015 – Feb 20, 2015 Timekeeping, clocksource core, NTP, alarmtimer

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 time, ntp: Do not update time_state in middle of leap second [v3]

During leap second insertion testing it was noticed that a small window exists where the time_state could be reset such that time_state = TIME_OK, which then causes the leap second to not occur, or causes the entire leap second state machine to fail with time_state = TIME_INS at the end of the leap second.

The test did the following in userspace:

        tx.modes = ADJ_STATUS;
        tx.status = STA_INS;

	/* send leap second request */
        ret = adjtimex(&tx);

        /* Check adjtimex output every half second */
        now = tx.time.tv_sec;
        while (now < next_leap+2) {
                char buf[26];
                ret = adjtimex(&tx);

                ctime_r(&tx.time.tv_sec, buf);
                buf[strlen(buf)-1] = 0; /*remove trailing\n */

                printf("%s + %6ld us\t%s\n",
                                buf,
                                tx.time.tv_usec,
                                time_state_str(ret));
                now = tx.time.tv_sec;
                /* Sleep for another half second */
                ts.tv_sec = 0;
                ts.tv_nsec = NSEC_PER_SEC/2;
                clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL);
        }

which was intended to mimic the insertion of a leap second. A successful run of the test would result in the time_state transitioning from TIME_OK to TIME_INS, then to TIME_OOP when the leap second was inserted, and then to TIME_WAIT when the leap second was completed. While running this code failures were seen in which the time_state remained TIME_INS, even though the leap second had occurred.

After some investigation it was noted that the test contained a small error:
the test does not reinitialize tx.status and reissues the STA_INS every 1/2 second. As a result of this broken test, the following failure was noticed (the output below is a mix of kernel messages and the output from the test program, the remaining annotations are printk's in the code and my own additional notes):

[  942.952833] time_state [1] change from TIME_OK to TIME_INS
Fri Feb 13 18:59:51 2015 + 318126 us    TIME_INS
Fri Feb 13 18:59:51 2015 + 818167 us    TIME_INS
Fri Feb 13 18:59:52 2015 + 318208 us    TIME_INS
Fri Feb 13 18:59:52 2015 + 818248 us    TIME_INS
Fri Feb 13 18:59:53 2015 + 318290 us    TIME_INS
Fri Feb 13 18:59:53 2015 + 818331 us    TIME_INS
Fri Feb 13 18:59:54 2015 + 318372 us    TIME_INS
Fri Feb 13 18:59:54 2015 + 818413 us    TIME_INS
Fri Feb 13 18:59:55 2015 + 318454 us    TIME_INS
Fri Feb 13 18:59:55 2015 + 818495 us    TIME_INS
Fri Feb 13 18:59:56 2015 + 318534 us    TIME_INS
Fri Feb 13 18:59:56 2015 + 818575 us    TIME_INS
Fri Feb 13 18:59:57 2015 + 318617 us    TIME_INS
Fri Feb 13 18:59:57 2015 + 818660 us    TIME_INS
Fri Feb 13 18:59:58 2015 + 318702 us    TIME_INS
Fri Feb 13 18:59:58 2015 + 818744 us    TIME_INS
Fri Feb 13 18:59:59 2015 + 318785 us    TIME_INS
Fri Feb 13 18:59:59 2015 + 818837 us    TIME_INS
[  952.953143] time_state [4] change from TIME_INS to TIME_OOP
[  952.953150] Clock: inserting leap second 23:59:60 UTC
[  953.299905] process_adj_status: insert_leap_sec[1223] setting time_state back
to TIME_OK [1, 1]   <<< adjtimex() call every 1/2 second
[  953.299913] time_state [9] change from TIME_OOP to TIME_OK

Fri Feb 13 18:59:59 2015 + 318878 us TIME_OK
Fri Feb 13 18:59:59 2015 + 818931 us TIME_OK

[  954.064237] time_state [1] change from TIME_OK to TIME_INS
Fri Feb 13 19:00:00 2015 + 318972 us    TIME_INS
Fri Feb 13 19:00:00 2015 + 819012 us    TIME_INS
Fri Feb 13 19:00:01 2015 + 319051 us    TIME_INS
Fri Feb 13 19:00:01 2015 + 819089 us    TIME_INS
Fri Feb 13 19:00:02 2015 + 319128 us    TIME_INS

As previously stated, the time_state remains TIME_INS even though the leap second has already occurred @ 952.953150.

The test was changed to reset tx.status to 0 in the loop, and the test then succeeded with a 100% rate with the time state ending in TIME_WAIT.

While this is highly unlikely to ever happen in the real world it is still something we should protect against, as breaking the state machine is bad.

If the time_state == TIME_OOP (ie, the leap second is in progress) do not allow an external update to time_state in process_adj_status(). This will prevent external adjtimex() calls from breaking the leap second state machine.

[v2]: Only block time_state change when TIME_OOP [v3]: Write a much more detailed explanation of the bug.

Cc John Stultz <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Miroslav Lichvar <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 kernel/time/ntp.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/time/ntp.c +2 -1
index 28bf91c..6ff5cd5 100644
@@ -535,7 +535,8 @@ void ntp_notify_cmos_timer(void) { }
 static inline void process_adj_status(struct timex *txc, struct timespec64 *ts)
 {
 	if ((time_status & STA_PLL) && !(txc->status & STA_PLL)) {
-		time_state = TIME_OK;
+		if (time_state != TIME_OOP)
+			time_state = TIME_OK;
 		time_status = STA_UNSYNC;
 		/* restart PPS frequency calibration */
 		pps_reset_freq_interval();
Signature · 1 line
1.7.9.3
#1 lore raw
Quoted 96 lines · Prarit Bhargava

During leap second insertion testing it was noticed that a small window exists where the time_state could be reset such that time_state = TIME_OK, which then causes the leap second to not occur, or causes the entire leap second state machine to fail with time_state = TIME_INS at the end of the leap second.

The test did the following in userspace:

        tx.modes = ADJ_STATUS;
        tx.status = STA_INS;

        /* send leap second request */
        ret = adjtimex(&tx);

        /* Check adjtimex output every half second */
        now = tx.time.tv_sec;
        while (now < next_leap+2) {
                char buf[26];
                ret = adjtimex(&tx);

                ctime_r(&tx.time.tv_sec, buf);
                buf[strlen(buf)-1] = 0; /*remove trailing\n */

                printf("%s + %6ld us\t%s\n",
                                buf,
                                tx.time.tv_usec,
                                time_state_str(ret));
                now = tx.time.tv_sec;
                /* Sleep for another half second */
                ts.tv_sec = 0;
                ts.tv_nsec = NSEC_PER_SEC/2;
                clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL);
        }

which was intended to mimic the insertion of a leap second. A successful run of the test would result in the time_state transitioning from TIME_OK to TIME_INS, then to TIME_OOP when the leap second was inserted, and then to TIME_WAIT when the leap second was completed. While running this code failures were seen in which the time_state remained TIME_INS, even though the leap second had occurred.

After some investigation it was noted that the test contained a small error:
the test does not reinitialize tx.status and reissues the STA_INS every 1/2 second. As a result of this broken test, the following failure was noticed (the output below is a mix of kernel messages and the output from the test program, the remaining annotations are printk's in the code and my own additional notes):

[  942.952833] time_state [1] change from TIME_OK to TIME_INS
Fri Feb 13 18:59:51 2015 + 318126 us    TIME_INS
Fri Feb 13 18:59:51 2015 + 818167 us    TIME_INS
Fri Feb 13 18:59:52 2015 + 318208 us    TIME_INS
Fri Feb 13 18:59:52 2015 + 818248 us    TIME_INS
Fri Feb 13 18:59:53 2015 + 318290 us    TIME_INS
Fri Feb 13 18:59:53 2015 + 818331 us    TIME_INS
Fri Feb 13 18:59:54 2015 + 318372 us    TIME_INS
Fri Feb 13 18:59:54 2015 + 818413 us    TIME_INS
Fri Feb 13 18:59:55 2015 + 318454 us    TIME_INS
Fri Feb 13 18:59:55 2015 + 818495 us    TIME_INS
Fri Feb 13 18:59:56 2015 + 318534 us    TIME_INS
Fri Feb 13 18:59:56 2015 + 818575 us    TIME_INS
Fri Feb 13 18:59:57 2015 + 318617 us    TIME_INS
Fri Feb 13 18:59:57 2015 + 818660 us    TIME_INS
Fri Feb 13 18:59:58 2015 + 318702 us    TIME_INS
Fri Feb 13 18:59:58 2015 + 818744 us    TIME_INS
Fri Feb 13 18:59:59 2015 + 318785 us    TIME_INS
Fri Feb 13 18:59:59 2015 + 818837 us    TIME_INS
[  952.953143] time_state [4] change from TIME_INS to TIME_OOP
[  952.953150] Clock: inserting leap second 23:59:60 UTC
[  953.299905] process_adj_status: insert_leap_sec[1223] setting time_state back
to TIME_OK [1, 1]   <<< adjtimex() call every 1/2 second
[  953.299913] time_state [9] change from TIME_OOP to TIME_OK

Fri Feb 13 18:59:59 2015 + 318878 us TIME_OK
Fri Feb 13 18:59:59 2015 + 818931 us TIME_OK

[  954.064237] time_state [1] change from TIME_OK to TIME_INS
Fri Feb 13 19:00:00 2015 + 318972 us    TIME_INS
Fri Feb 13 19:00:00 2015 + 819012 us    TIME_INS
Fri Feb 13 19:00:01 2015 + 319051 us    TIME_INS
Fri Feb 13 19:00:01 2015 + 819089 us    TIME_INS
Fri Feb 13 19:00:02 2015 + 319128 us    TIME_INS

As previously stated, the time_state remains TIME_INS even though the leap second has already occurred @ 952.953150.

The test was changed to reset tx.status to 0 in the loop, and the test then succeeded with a 100% rate with the time state ending in TIME_WAIT.

While this is highly unlikely to ever happen in the real world it is still something we should protect against, as breaking the state machine is bad.

Quoted 7 lines

If the time_state == TIME_OOP (ie, the leap second is in progress) do not allow an external update to time_state in process_adj_status(). This will prevent external adjtimex() calls from breaking the leap second state machine.

[v2]: Only block time_state change when TIME_OOP [v3]: Write a much more detailed explanation of the bug.

Ok, thanks for the more verbose explanation. Although this is more a history of what you've seen rather then the crux of the change.

To distill this down just a bit, the point is the usual mode for NTP time_state machine looks like:

TIME_OK -> TIME_INS -> TIME_OOP

  |                       |
  v                       v
TIME_DEL ------------> TIME_WAIT  -(back)-> TIME_OK

(hopefully the ascii art survives here)

Now, from any of these states, currently if adjtimex is called w/ the STA_PLL bit cleared (after STA_PLL was set), we reset back to TIME_OK, effectively cancelling any transitions. (You'll have to imagine a line from any of the states back to TIME_OK, since that's going to be too ugly to do in ascii)

Your patch is trying to remove the line back from TIME_OOP back to TIME_OK. Basically stopping the ability to reset the ntp state during a leapsecond.

I do get that the behavior seen was strange due to a bug in the test code which caused unexpected cancellation of state, but I'm not sure if we should change the behavior to enforce that cancellation not be possible. I could imagine some logic which really wants to reset the state, which just by chance lands during a leap second, and the application is confused since the state change didn't occur as expected.

So I guess I'm not seeing that the state machine is actually "broken" in this case that you've outlined. If you can articulate better why the OOP -> OK transition is truly invalid, I'd be interested in hearing, but I'm not sure I want to risk a behavioral change unless there's wide agreement.

thanks
-john

#2 lore raw
Quoted 31 lines · John Stultz

Ok, thanks for the more verbose explanation. Although this is more a history of what you've seen rather then the crux of the change.

To distill this down just a bit, the point is the usual mode for NTP time_state machine looks like:

TIME_OK -> TIME_INS -> TIME_OOP

  |                       |
  v                       v
TIME_DEL ------------> TIME_WAIT  -(back)-> TIME_OK

(hopefully the ascii art survives here)

Now, from any of these states, currently if adjtimex is called w/ the STA_PLL bit cleared (after STA_PLL was set), we reset back to TIME_OK, effectively cancelling any transitions. (You'll have to imagine a line from any of the states back to TIME_OK, since that's going to be too ugly to do in ascii)

Your patch is trying to remove the line back from TIME_OOP back to TIME_OK. Basically stopping the ability to reset the ntp state during a leapsecond.

I do get that the behavior seen was strange due to a bug in the test code which caused unexpected cancellation of state, but I'm not sure if we should change the behavior to enforce that cancellation not be possible. I could imagine some logic which really wants to reset the state, which just by chance lands during a leap second, and the application is confused since the state change didn't occur as expected.

Quoted 5 lines

So I guess I'm not seeing that the state machine is actually "broken" in this case that you've outlined. If you can articulate better why the OOP -> OK transition is truly invalid, I'd be interested in hearing, but I'm not sure I want to risk a behavioral change unless there's wide agreement.

I think the only real problem occurs when the adjtimex is called in the the TIME_OOP state with STA_PLL cleared _and_ STA_INS set. In this case the state machine is reset to TIME_OK but goes back to TIME_INS on the next second_overflow, potentially causing another false leap second to be inserted on the following midnight.

The state machine is meant to only go back to TIME_INS once STA_INS is cleared and then set again - this is what the TIME_WAIT state is for.

In fact, I don't see a reason why the STA_PLL -> !STA_PLL transition should ever set the time_state to TIME_OK.
- When the STA_INS/STA_DEL flag is removed from the status, the state machine will end up in TIME_OK from any state.
- When STA_INS/STA_DEL is set in
the status, the state mchine will transition from TIME_OK to TIME_INS/TIME_DEL anyway.

I think the "time_status = TIME_OK" should be just dropped.

It has been added by eea83d896e318bda54be2d2770d2c5d6668d11db (ntp: NTP4 user space bits update) and it's not clear why. Roman?

Signature · 2 lines
Jiri Bohac <[email protected]>
SUSE Labs, SUSE CZ
#3 lore raw
Quoted 2 lines · Jiri Bohac

I think the only real problem occurs when the adjtimex is called in the the TIME_OOP state

... or the TIME_WAIT state ...

Quoted 23 lines

with STA_PLL cleared _and_ STA_INS set.
In this case the state machine is reset to TIME_OK but goes back to TIME_INS on the next second_overflow, potentially causing another false leap second to be inserted on the following midnight.

The state machine is meant to only go back to TIME_INS once STA_INS is cleared and then set again - this is what the TIME_WAIT state is for.

In fact, I don't see a reason why the STA_PLL -> !STA_PLL transition should ever set the time_state to TIME_OK.
- When the STA_INS/STA_DEL flag is removed from the status, the state machine will end up in TIME_OK from any state.
- When STA_INS/STA_DEL is set in
the status, the state mchine will transition from TIME_OK to TIME_INS/TIME_DEL anyway.

I think the "time_status = TIME_OK" should be just dropped.

It has been added by eea83d896e318bda54be2d2770d2c5d6668d11db (ntp: NTP4 user space bits update) and it's not clear why. Roman?

Signature · 2 lines
Jiri Bohac <[email protected]>
SUSE Labs, SUSE CZ
#4 lore raw
In reply to John Stultz
Quoted 31 lines · John Stultz
Quoted 8 lines · Prarit Bhargava

which was intended to mimic the insertion of a leap second. A successful run of the test would result in the time_state transitioning from TIME_OK to TIME_INS, then to TIME_OOP when the leap second was inserted, and then to TIME_WAIT when the leap second was completed. While running this code failures were seen in which the time_state remained TIME_INS, even though the leap second had occurred.

Ok, thanks for the more verbose explanation. Although this is more a history of what you've seen rather then the crux of the change.

To distill this down just a bit, the point is the usual mode for NTP time_state machine looks like:

TIME_OK -> TIME_INS -> TIME_OOP

  |                       |
  v                       v
TIME_DEL ------------> TIME_WAIT  -(back)-> TIME_OK

(hopefully the ascii art survives here)

Now, from any of these states, currently if adjtimex is called w/ the STA_PLL bit cleared (after STA_PLL was set), we reset back to TIME_OK, effectively cancelling any transitions. (You'll have to imagine a line from any of the states back to TIME_OK, since that's going to be too ugly to do in ascii)

Quoted 3 lines

Your patch is trying to remove the line back from TIME_OOP back to TIME_OK. Basically stopping the ability to reset the ntp state during a leapsecond.

Correct.

Quoted 8 lines

I do get that the behavior seen was strange due to a bug in the test code which caused unexpected cancellation of state, but I'm not sure if we should change the behavior to enforce that cancellation not be possible. I could imagine some logic which really wants to reset the state, which just by chance lands during a leap second, and the application is confused since the state change didn't occur as expected.

I think setting it in the middle of the leap second should be a NOOP. We all know how fragile this code has been in the past and allowing a state transition at that particular time isn't a good idea given the outcome that the state may remain TIME_INS.

Quoted 6 lines

So I guess I'm not seeing that the state machine is actually "broken" in this case that you've outlined. If you can articulate better why the OOP -> OK transition is truly invalid, I'd be interested in hearing, but I'm not sure I want to risk a behavioral change unless there's wide agreement.

I understand -- After thinking about it from your point of view I agree that calling it "broken" is not right. Perhaps a better way of looking at it is, as you also point out, if OOP -> OK is truly valid.

P.

Quoted 4 lines

thanks
-john

#5 lore raw
In reply to Prarit Bhargava

Hi,

I'm trying to understand what exactly is going on here...

Quoted 28 lines · Prarit Bhargava

The test did the following in userspace:

        tx.modes = ADJ_STATUS;
        tx.status = STA_INS;

	/* send leap second request */
        ret = adjtimex(&tx);

        /* Check adjtimex output every half second */
        now = tx.time.tv_sec;
        while (now < next_leap+2) {
                char buf[26];
                ret = adjtimex(&tx);

                ctime_r(&tx.time.tv_sec, buf);
                buf[strlen(buf)-1] = 0; /*remove trailing\n */

                printf("%s + %6ld us\t%s\n",
                                buf,
                                tx.time.tv_usec,
                                time_state_str(ret));
                now = tx.time.tv_sec;
                /* Sleep for another half second */
                ts.tv_sec = 0;
                ts.tv_nsec = NSEC_PER_SEC/2;
                clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL);
        }
Quoted 3 lines

After some investigation it was noted that the test contained a small error:
the test does not reinitialize tx.status and reissues the STA_INS every 1/2 second.

Prarit, can you explain who sets the STA_PLL flag, so that process_adj_status() detects a STA_PLL->!STA_PLL transition and goes to the branch that sets time_state = TIME_OK?

Is that ntpd running in parallel with your test program? If that is the case, you would eventually end up oscilating between the the TIME_INS and TIME_OK states anyway, even with your patch. ntpd will clear the STA_INS flag after midnight, the state machine will transition from TIME_WAIT to TIME_OK and your test program will set STA_INS again (fighting with ntpd which will clear the flag from time to time) ... right?

Thanks,

Signature · 2 lines
Jiri Bohac <[email protected]>
SUSE Labs, SUSE CZ
#6 lore raw
Quoted 33 lines · Jiri Bohac

Hi,

I'm trying to understand what exactly is going on here...

Quoted 28 lines · Prarit Bhargava

The test did the following in userspace:

        tx.modes = ADJ_STATUS;
        tx.status = STA_INS;

	/* send leap second request */
        ret = adjtimex(&tx);

        /* Check adjtimex output every half second */
        now = tx.time.tv_sec;
        while (now < next_leap+2) {
                char buf[26];
                ret = adjtimex(&tx);

                ctime_r(&tx.time.tv_sec, buf);
                buf[strlen(buf)-1] = 0; /*remove trailing\n */

                printf("%s + %6ld us\t%s\n",
                                buf,
                                tx.time.tv_usec,
                                time_state_str(ret));
                now = tx.time.tv_sec;
                /* Sleep for another half second */
                ts.tv_sec = 0;
                ts.tv_nsec = NSEC_PER_SEC/2;
                clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL);
        }
Quoted 7 lines
Quoted 3 lines

After some investigation it was noted that the test contained a small error:
the test does not reinitialize tx.status and reissues the STA_INS every 1/2 second.

Prarit, can you explain who sets the STA_PLL flag, so that process_adj_status() detects a STA_PLL->!STA_PLL transition and goes to the branch that sets time_state = TIME_OK?

Jiri,

The test being run is:

https://github.com/johnstultz-work/timetests/blob/master/leap-a-day.c

prior to commit

https://github.com/johnstultz-work/timetests/commit/be4526e8b5d48cd108a8d2cf7f5c8fd763acf421

Quoted 2 lines

Is that ntpd running in parallel with your test program? If that

No -- ntpd is disabled (chronyd in the case of systemd + current Fedora).

P.

#7 lore raw
Quoted 7 lines · Prarit Bhargava
Quoted 3 lines · Jiri Bohac

Prarit, can you explain who sets the STA_PLL flag, so that process_adj_status() detects a STA_PLL->!STA_PLL transition and goes to the branch that sets time_state = TIME_OK?

Jiri,

Quoted 7 lines

The test being run is:

https://github.com/johnstultz-work/timetests/blob/master/leap-a-day.c

prior to commit

https://github.com/johnstultz-work/timetests/commit/be4526e8b5d48cd108a8d2cf7f5c8fd763acf421

I can't make sense of the output of your test:

Quoted 14 lines · Prarit Bhargava
[  942.952833] time_state [1] change from TIME_OK to TIME_INS
Fri Feb 13 18:59:51 2015 + 318126 us    TIME_INS
Fri Feb 13 18:59:51 2015 + 818167 us    TIME_INS
Fri Feb 13 18:59:52 2015 + 318208 us    TIME_INS
Fri Feb 13 18:59:52 2015 + 818248 us    TIME_INS
Fri Feb 13 18:59:53 2015 + 318290 us    TIME_INS
Fri Feb 13 18:59:53 2015 + 818331 us    TIME_INS
Fri Feb 13 18:59:54 2015 + 318372 us    TIME_INS
Fri Feb 13 18:59:54 2015 + 818413 us    TIME_INS
Fri Feb 13 18:59:55 2015 + 318454 us    TIME_INS
Fri Feb 13 18:59:55 2015 + 818495 us    TIME_INS
Fri Feb 13 18:59:56 2015 + 318534 us    TIME_INS
Fri Feb 13 18:59:56 2015 + 818575 us    TIME_INS

Why did the test program print the above lines? It's supposed to sleep until 3 seconds prior to the midnight:

	/* Wake up 3 seconds before leap */
	ts.tv_sec = next_leap - 3;
	ts.tv_nsec = 0;
	while(clock_nanosleep(CLOCK_REALTIME, TIMER_ABSTIME, &ts, NULL))
		printf("Something woke us up, returning to sleep\n");
Quoted 7 lines
Fri Feb 13 18:59:57 2015 + 318617 us    TIME_INS
Fri Feb 13 18:59:57 2015 + 818660 us    TIME_INS
Fri Feb 13 18:59:58 2015 + 318702 us    TIME_INS
Fri Feb 13 18:59:58 2015 + 818744 us    TIME_INS
Fri Feb 13 18:59:59 2015 + 318785 us    TIME_INS
Fri Feb 13 18:59:59 2015 + 818837 us    TIME_INS
Quoted 5 lines
[  952.953143] time_state [4] change from TIME_INS to TIME_OOP
[  952.953150] Clock: inserting leap second 23:59:60 UTC
[  953.299905] process_adj_status: insert_leap_sec[1223] setting time_state back
to TIME_OK [1, 1]   <<< adjtimex() call every 1/2 second
[  953.299913] time_state [9] change from TIME_OOP to TIME_OK

2) The only place where the test program sets STA_PLL is in clear_time_state(); It clears it right after that.

clear_time_state() is not called inside the while "(now < next_leap+2)" loop, except in the SIGINT/SIGKILL handler. Did you send signals to the program at this point?

If not, I can't understand how the status went from STA_PLL to !STA_PLL and thus why time_state went to TIME_OK

Signature · 2 lines
Jiri Bohac <[email protected]>
SUSE Labs, SUSE CZ