Skip to contentSkip to conversation

Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

tip:timers/urgent Linus Torvalds 5 participants 14 messages Apr 25, 2018 – Apr 26, 2018 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 3 lines · tip-bot for Thomas Gleixner

As stated in the pull request for the unification of CLOCK_MONOTONIC and CLOCK_BOOTTIME, it was clear that we might have to revert the change.

I'm ok with the revert., but I do want the revert message to talk about *exactly* what broke.

I don't see the reported watchdog things, and I run systemd and I ran these patches.

So it's not just "systemd uses this for a watchdog". It must be something specific that made it trigger for some people and not others, and it's that specific thing that we should document so that we know why this simplification failed.

Linus

#2 lore raw
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · tip-bot for Thomas Gleixner

As stated in the pull request for the unification of CLOCK_MONOTONIC and CLOCK_BOOTTIME, it was clear that we might have to revert the change.

I'm ok with the revert., but I do want the revert message to talk about *exactly* what broke.

I don't see the reported watchdog things, and I run systemd and I ran these patches.

With suspend/resume?

Quoted 4 lines

So it's not just "systemd uses this for a watchdog". It must be something specific that made it trigger for some people and not others, and it's that specific thing that we should document so that we know why this simplification failed.

Fair enough. Do the reporters have more information or do I have to chase it down myself?

Thanks,

tglx

#3 lore raw
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines · Linus Torvalds

I don't see the reported watchdog things, and I run systemd and I ran these patches.

With suspend/resume?

Yes. I run them on my laptop, and while I don't do a _lot_ of suspend/resume, I do some. My laptop right now is running a "just after rc1" kernel, and has been for the last week, and has a couple of suspends in that time. Type "closed my laptop, took it elsewhere, opened it again", so more than a few minutes.

Quoted 2 lines

Fair enough. Do the reporters have more information or do I have to chase it down myself?

I think reporters should just say what distro they are running, and systemd versions etc, and how they debugged this. Maybe the pattern is obvious.

For the record, I just run plain F27, obviously with my own kernel (and a tweaked config, but it approaches just being a "make localmodconfig" of the distro config).

Linus

#4 lore raw
Quoted 17 lines · Linus Torvalds
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines · Linus Torvalds

I don't see the reported watchdog things, and I run systemd and I ran these patches.

With suspend/resume?

Yes. I run them on my laptop, and while I don't do a _lot_ of suspend/resume, I do some. My laptop right now is running a "just after rc1" kernel, and has been for the last week, and has a couple of suspends in that time. Type "closed my laptop, took it elsewhere, opened it again", so more than a few minutes.

Quoted 2 lines

Fair enough. Do the reporters have more information or do I have to chase it down myself?

Quoted 7 lines

I think reporters should just say what distro they are running, and systemd versions etc, and how they debugged this. Maybe the pattern is obvious.

For the record, I just run plain F27, obviously with my own kernel (and a tweaked config, but it approaches just being a "make localmodconfig" of the distro config).

So... I do have debian 8.10. With 4.17-rc1, I get "networking disabled" message from network manager (and broken network), about half of the time.

But there's more visible symptom: suspend/resume with 4.16, machine is okay after resume. With 4.17-rc1, just after resume, display starts dimming and screensaver will kick in. MATE desktop.

Best regards,
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#5 lore raw
Quoted 34 lines · CEST Pavel Machek

--J2SCkAp4GZ/dPZZf
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Quoted 1 line · Linus Torvalds

On Wed, Apr 25, 2018 at 12:09 PM, Thomas Gleixner <[email protected]> wr=

ote:

Quoted 5 lines
Quoted 5 lines · Linus Torvalds
Quoted 3 lines

I don't see the reported watchdog things, and I run systemd and I ran these patches.

Quoted 9 lines
Quoted 1 line

With suspend/resume?

=20
Yes. I run them on my laptop, and while I don't do a _lot_ of suspend/resume, I do some. My laptop right now is running a "just after rc1" kernel, and has been for the last week, and has a couple of suspends in that time. Type "closed my laptop, took it elsewhere, opened it again", so more than a few minutes.
=20

Quoted 1 line

Fair enough. Do the reporters have more information or do I have to cha=

se

Quoted 9 lines
Quoted 1 line

it down myself?

=20
I think reporters should just say what distro they are running, and systemd versions etc, and how they debugged this. Maybe the pattern is obvious.
=20
For the record, I just run plain F27, obviously with my own kernel (and a tweaked config, but it approaches just being a "make localmodconfig" of the distro config).

Quoted 1 line

So... I do have debian 8.10.

Does that use systemd at all?

Quoted 7 lines

With 4.17-rc1, I get "networking
disabled" message from network manager (and broken network), about half of the time.

But there's more visible symptom: suspend/resume with 4.16, machine is okay after resume. With 4.17-rc1, just after resume, display starts dimming and screensaver will kick in. MATE desktop.

Yeah, why not. It thinks that you've been inactive for the duration of system sleep. :-)

Thanks,
Rafael

#6 lore raw

Hi!

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

For the record, I just run plain F27, obviously with my own kernel (and a tweaked config, but it approaches just being a "make localmodconfig" of the distro config).

So... I do have debian 8.10.

Does that use systemd at all?

pavel@duo:/data/l/linux$ systemd --version
systemd 215
+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ
-SECCOMP -APPARMOR
Quoted 10 lines
Quoted 7 lines

With 4.17-rc1, I get "networking
disabled" message from network manager (and broken network), about half of the time.

But there's more visible symptom: suspend/resume with 4.16, machine is okay after resume. With 4.17-rc1, just after resume, display starts dimming and screensaver will kick in. MATE desktop.

Yeah, why not. It thinks that you've been inactive for the duration of system sleep. :-)

Well, its rather annoying. "Wake up!" (hitting a key) "OOh, I'm slowly waking up.." (few seconds, wakeup is not instant) "and now I'm going back to sleep".

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#7 lore raw
Quoted 12 lines · CEST Pavel Machek

Hi!

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

For the record, I just run plain F27, obviously with my own kernel (and a tweaked config, but it approaches just being a "make localmodconfig" of the distro config).

So... I do have debian 8.10.

Does that use systemd at all?

pavel@duo:/data/l/linux$ systemd --version
systemd 215

Review

Quoted 2 lines
(quoted diff) +1 -1
+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ
-SECCOMP -APPARMOR

OK

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

With 4.17-rc1, I get "networking
disabled" message from network manager (and broken network), about half of the time.

I'm not sure if NM is affected by systemd, though.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

But there's more visible symptom: suspend/resume with 4.16, machine is okay after resume. With 4.17-rc1, just after resume, display starts dimming and screensaver will kick in. MATE desktop.

Yeah, why not. It thinks that you've been inactive for the duration of system sleep. :-)

Well, its rather annoying.

Yes, it is, no question about that.

Quoted 3 lines

"Wake up!" (hitting a key) "OOh, I'm slowly
waking up.." (few seconds, wakeup is not instant) "and now I'm going back to sleep".

I just wanted to say that you could expect that if the clock used by it was counting over suspend/resume.

#8 lore raw
In reply to Linus Torvalds
Quoted 17 lines · CEST Linus Torvalds
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines · Linus Torvalds

I don't see the reported watchdog things, and I run systemd and I ran these patches.

With suspend/resume?

Yes. I run them on my laptop, and while I don't do a _lot_ of suspend/resume, I do some. My laptop right now is running a "just after rc1" kernel, and has been for the last week, and has a couple of suspends in that time. Type "closed my laptop, took it elsewhere, opened it again", so more than a few minutes.

Quoted 2 lines

Fair enough. Do the reporters have more information or do I have to chase it down myself?

Quoted 7 lines

I think reporters should just say what distro they are running, and systemd versions etc, and how they debugged this. Maybe the pattern is obvious.

For the record, I just run plain F27, obviously with my own kernel (and a tweaked config, but it approaches just being a "make localmodconfig" of the distro config).

FWIW, I run (64-bit) OpenSUSE 42.3 on the failing machine (Acer Aspire S5) with my own kernel (custom config based on the distro one).

The problem here is that the machine doesn't resume from S2R. It appears to just hang somewhere around where we return to user space. However, to trigger the issue it needs to stay in suspend for several minutes (not sure how many exactly). If it is just back-to-back suspend/resume, it will resume correctly.

Thanks,
Rafael

#9 lore raw
Quoted 7 lines · On Wed, 25 Apr 2018 22:57:12 +0200 "Rafael J. Wysocki" <[email protected]>…

FWIW, I run (64-bit) OpenSUSE 42.3 on the failing machine (Acer Aspire S5) with my own kernel (custom config based on the distro one).

The problem here is that the machine doesn't resume from S2R. It appears to just hang somewhere around where we return to user space. However, to trigger the issue it needs to stay in suspend for several minutes (not sure how many exactly). If it is just back-to-back suspend/resume, it will resume correctly.

I guess the question that needs to be asked is, have you tried a kernel with this commit reverted and it works? I'd play a bit going back and forth between two kernels, one with the commit and one without and no other differences, to make sure that is indeed the problem commit.

-- Steve

#10 lore raw
Quoted 6 lines · CEST Steven Rostedt
Quoted 3 lines · On Wed, 25 Apr 2018 22:57:12 +0200 "Rafael J. Wysocki" <[email protected]>…

FWIW, I run (64-bit) OpenSUSE 42.3 on the failing machine (Acer Aspire S5) with my own kernel (custom config based on the distro one).

Quoted 7 lines
Quoted 4 lines

The problem here is that the machine doesn't resume from S2R. It appears to just hang somewhere around where we return to user space. However, to trigger the issue it needs to stay in suspend for several minutes (not sure how many exactly). If it is just back-to-back suspend/resume, it will resume correctly.

I guess the question that needs to be asked is, have you tried a kernel with this commit reverted and it works?

Yes, I have.

Quoted 3 lines

I'd play a bit going back and
forth between two kernels, one with the commit and one without and no other differences, to make sure that is indeed the problem commit.

I've done that.

It is the problem commit.

Thanks,
Rafael

#11 lore raw
In reply to Thomas Gleixner
Quoted 15 lines · CEST Thomas Gleixner
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · tip-bot for Thomas Gleixner

As stated in the pull request for the unification of CLOCK_MONOTONIC and CLOCK_BOOTTIME, it was clear that we might have to revert the change.

I'm ok with the revert., but I do want the revert message to talk about *exactly* what broke.

I don't see the reported watchdog things, and I run systemd and I ran these patches.

With suspend/resume?

Quoted 7 lines
Quoted 4 lines

So it's not just "systemd uses this for a watchdog". It must be something specific that made it trigger for some people and not others, and it's that specific thing that we should document so that we know why this simplification failed.

Fair enough. Do the reporters have more information or do I have to chase it down myself?

Well, if CLOCK_MONOTONIC did not count when the system was suspended and now it does that due to the change in question, then this is a significant difference in behavior likely to confuse things even in principle.

While it may not confuse things on some systems after all, does this really matter?

Thanks,
Rafael

#12 lore raw
Quoted 21 lines · CEST Rafael J. Wysocki
Quoted 20 lines · CEST Thomas Gleixner
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · tip-bot for Thomas Gleixner

As stated in the pull request for the unification of CLOCK_MONOTONIC and CLOCK_BOOTTIME, it was clear that we might have to revert the change.

I'm ok with the revert., but I do want the revert message to talk about *exactly* what broke.

I don't see the reported watchdog things, and I run systemd and I ran these patches.

With suspend/resume?

Quoted 4 lines

So it's not just "systemd uses this for a watchdog". It must be something specific that made it trigger for some people and not others, and it's that specific thing that we should document so that we know why this simplification failed.

Quoted 6 lines
Quoted 2 lines

Fair enough. Do the reporters have more information or do I have to chase it down myself?

Well, if CLOCK_MONOTONIC did not count when the system was suspended and now it does that due to the change in question, then this is a significant difference in behavior likely to confuse things even in principle.

FWIW, here's a list of issues reported and narrowed down to the commit in question so far I could find:

* systemd kills daemons on resume, after >WatchdogSec seconds of suspending (Genki Sky). [Verified that that's because systemd uses CLOCK_MONOTONIC and expects it to not include the suspend time.]

* systemd-journald misbehaves after resume:
systemd-journald[7266]: File /var/log/journal/016627c3c4784cd4812d4b7e96a34226/system.journal corrupted or uncleanly shut down, renaming and replacing.
( Mike Galbraith).

* NetworkManager reports "networking disabled" and networking is broken after resume 50% of the time (Pavel). [May be because of systemd.]

* MATE desktop dims the display and starts the screensaver right after system resume (Pavel).

* Full system hang during resume (me). [May be due to systemd or NM or both.]

I'm not sure if all of the above can be connected to one specific thing (other than the fact that things get confused if they expect CLOCK_MONOTONIC to not include system suspend time).

#13 lore raw
Quoted 14 lines · Rafael J. Wysocki

FWIW, here's a list of issues reported and narrowed down to the commit in question so far I could find:

* systemd kills daemons on resume, after >WatchdogSec seconds of suspending (Genki Sky). [Verified that that's because systemd uses CLOCK_MONOTONIC and expects it to not include the suspend time.]

* systemd-journald misbehaves after resume:
systemd-journald[7266]: File /var/log/journal/016627c3c4784cd4812d4b7e96a34226/system.journal corrupted or uncleanly shut down, renaming and replacing.
( Mike Galbraith).

* NetworkManager reports "networking disabled" and networking is broken after resume 50% of the time (Pavel). [May be because of systemd.]

Quoted 8 lines

* MATE desktop dims the display and starts the screensaver right after system resume (Pavel).

* Full system hang during resume (me). [May be due to systemd or NM or both.]

I'm not sure if all of the above can be connected to one specific thing (other than the fact that things get confused if they expect CLOCK_MONOTONIC to not include system suspend time).

Thanks for collecting the info. I'll amend the changelog with those.

Thanks,

tglx

Tested-by
#14 lore raw
In reply to a message not in the archive
Quoted 27 lines · tip-bot for Thomas Gleixner
Commit-ID:  757d5a9b3052a4eebd029f207f1e1df59bef7b7d
Gitweb:     https://git.kernel.org/tip/757d5a9b3052a4eebd029f207f1e1df59bef7b7d
Author:     Thomas Gleixner <[email protected]>
AuthorDate: Wed, 25 Apr 2018 15:33:38 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Wed, 25 Apr 2018 15:35:08 +0200

Revert: Unify CLOCK_MONOTONIC and CLOCK_BOOTTIME

Revert commits

92af4dcb4e1c ("tracing: Unify the "boot" and "mono" tracing clocks") 127bfa5f4342 ("hrtimer: Unify MONOTONIC and BOOTTIME clock behavior") 7250a4047aa6 ("posix-timers: Unify MONOTONIC and BOOTTIME clock behavior") d6c7270e913d ("timekeeping: Remove boot time specific code") f2d6fdbfd238 ("Input: Evdev - unify MONOTONIC and BOOTTIME clock behavior") d6ed449afdb3 ("timekeeping: Make the MONOTONIC clock behave like the BOOTTIME clock") 72199320d49d ("timekeeping: Add the new CLOCK_MONOTONIC_ACTIVE clock")

As stated in the pull request for the unification of CLOCK_MONOTONIC and CLOCK_BOOTTIME, it was clear that we might have to revert the change.

As reported by several folks systemd and other applications rely on the documented behaviour of CLOCK_MONOTONIC on Linux and break with the above changes. After resume daemons time out and other timeout related issues are observed.

Quoted 5 lines

It's sad, that these problems were neither catched in -next nor by those folks who expressed interest in this change.

Both screensaver problem and network problems are gone on X60 after I applied the revert. (I did not do too many tests, but it was normally too easy to reproduce.)

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html