tsc clocksource bug fix
cover tsc clocksource bug fix
We find some benchmarks drop a lot on tip/sched/core on many Intel boxes, like oltp, tbench, hackbench etc. and bisected the commit 5d33b883ae cause this regression. Due to this commit, the clocksource was changed to hpet from tsc even tsc will be set CLOCK_SOURCE_VALID_FOR_HRES later in clocksource_watchdog.
Tim Chen said the hpet reading cost much. That cause this regression.
This patchset fixed this bug by re-select clocksource after this flag set on tsc.
BTW,
If the tsc is marked as constant and nonstop, could we set it as system clocksource when do tsc register? w/o checking it on clocksource_watchdog?
regards!
Alex
PATCH 1/3 clocksource: clean up clocksource_select
After clocksource_find_best() introduced, it is impossible to get into some code path. so clean them up.
--- kernel/time/clocksource.c | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-)
kernel/time/clocksource.c +1 -15
@@ -582,26 +582,12 @@ static void __clocksource_select(bool skipcur)if (!best)return ; -/* Check for the override clocksource. */ list_for_each_entry (cs, &clocksource_list, list) {if (skipcur && cs == curr_clocksource)continue ;if (strcmp (cs->name, override_name) !=0 )continue ; -/* -* Check to make sure we don't switch to a non-highres -* capable clocksource if the tick code is in oneshot -* mode (highres or nohz) -*/ -if (!(cs->flags &CLOCK_SOURCE_VALID_FOR_HRES ) && oneshot) { -/* Override clocksource cannot be used. */ -printk (KERN_WARNING "Override clocksource %s is not " -"HRT compatible. Cannot switch while in " -"HRT/NOHZ mode\n" , cs->name); - override_name[0 ] =0 ; - }else -/* Override clocksource can be used. */ - best = cs; + best = cs;break ; }
Signature · 1 line
1.7.12
Re: [PATCH 1/3] clocksource: clean up clocksource_select
Quoted 2 lines · Alex Shi
After
clocksource_find_best()introduced, it is impossible to get into some code path. so clean them up.
That's wrong.
Quoted 10 lines
Signed-off-by Alex Shi <[email protected]>--- kernel/time/clocksource.c | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-)kernel/time/clocksource.c +0
-0
kernel/time/clocksource.c static void __clocksource_select(bool skipcur) · line 589
Quoted 9 lines
kernel/time/clocksource.c +0
-1@@ -582,26 +582,12 @@ static void __clocksource_select(bool skipcur)if (!best)return ; -/* Check for the override clocksource. */ list_for_each_entry (cs, &clocksource_list, list) {if (skipcur && cs == curr_clocksource)continue ;if (strcmp (cs->name, override_name) !=0 )continue ;
We need this check and it is completely unrelated to the problem you're trying to solve.
Assume the following:
System boots with clocksource A and switches into highres mode. Now clocksource B gets registered and B is not highres capable.
clocksource_find_best() selects again A, but we have clocksource=B on the kernel command line to override the kernel decision.
By removing the check, you install he non highres capable clocksource B and kill the machine.
Quoted 7 lines
kernel/time/clocksource.c +0
-7-/* -* Check to make sure we don't switch to a non-highres -* capable clocksource if the tick code is in oneshot -* mode (highres or nohz) -*/ -if (!(cs->flags &CLOCK_SOURCE_VALID_FOR_HRES ) && oneshot) { -/* Override clocksource cannot be used. */
kernel/time/clocksource.c static void __clocksource_select(bool skipcur) · line 592
Quoted 10 lines
kernel/time/clocksource.c +1
-7-printk (KERN_WARNING "Override clocksource %s is not " -"HRT compatible. Cannot switch while in " -"HRT/NOHZ mode\n" , cs->name); - override_name[0 ] =0 ; - }else -/* Override clocksource can be used. */ - best = cs; + best = cs;break ; }
Thanks,
tglx
Re: [PATCH 1/3] clocksource: clean up clocksource_select
Quoted 9 lines
We need this check and it is completely unrelated to the problem you're trying to solve.
Assume the following:
System boots with clocksource A and switches into highres mode. Now clocksource B gets registered and B is not highres capable.
Quoted 7 lines
clocksource_find_best()selects again A, but we have clocksource=B on the kernel command line to override the kernel decision.By removing the check, you install he non highres capable clocksource B and kill the machine.
You'r right. my bad, Sorry!
BTW, why we allow user override a second best clocksource? I mean user can override the tsc with hpet. because undetected unstable tsc?
Signature · 2 lines
Thanks
AlexRe: [PATCH 1/3] clocksource: clean up clocksource_select
Quoted 12 lines · Alex Shi
Quoted 12 lines
We need this check and it is completely unrelated to the problem you're trying to solve.
Assume the following:
System boots with clocksource A and switches into highres mode. Now clocksource B gets registered and B is not highres capable.
clocksource_find_best()selects again A, but we have clocksource=B on the kernel command line to override the kernel decision.
Quoted 8 lines
Quoted 3 lines
By removing the check, you install he non highres capable clocksource B and kill the machine.
You'r right. my bad, Sorry!
BTW, why we allow user override a second best clocksource? I mean user can override the tsc with hpet. because undetected unstable tsc?
The user can decide to override with clocksource=jiffies if he wants for testing purposes. That wont switch into highres ever if done from the kernel command line. Now we have a sysfs interface, so we need to sanity check the user override against highres.
The override is quite useful to test hpet, pmtimer on a machine which would always select TSC.
Thanks,
tglx
Re: [PATCH 1/3] clocksource: clean up clocksource_select
Quoted 9 lines · Thomas Gleixner
Quoted 2 lines
Quoted 2 lines
BTW, why we allow user override a second best clocksource? I mean user can override the tsc with hpet. because undetected unstable tsc?
The user can decide to override with clocksource=jiffies if he wants for testing purposes. That wont switch into highres ever if done from the kernel command line. Now we have a sysfs interface, so we need to sanity check the user override against highres.
The override is quite useful to test hpet, pmtimer on a machine which would always select TSC.
Got it. thanks a lot for friendly explanation!
Signature · 2 lines
Thanks
AlexPATCH 2/3 clockesource: set override clocksource
In reply to Alex ShiShrink the mutex region. And save a clocksource_select action if set clocksource is same as current clocksource.
--- kernel/time/clocksource.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
kernel/time/clocksource.c +7 -5
@@ -885,13 +885,15 @@ static ssize_t sysfs_override_clocksource(struct device *dev, {size_t ret; -mutex_lock (&clocksource_mutex); - ret =sysfs_get_uname (buf, override_name, count); -if (ret >=0 ) -clocksource_select (); +if (ret >=0 ) { +if (!strcmp (curr_clocksource->name, override_name)) +return ret; -mutex_unlock (&clocksource_mutex); +mutex_lock (&clocksource_mutex); +clocksource_select (); +mutex_unlock (&clocksource_mutex); + }return ret; }
Signature · 1 line
1.7.12
Re: [PATCH 2/3] clockesource: set override clocksource
Quoted 2 lines · Alex Shi
Shrink the mutex region. And save a
clocksource_selectaction if set clocksource is same as current clocksource.
Again, how is that related to the issue described in 0/3 ?
That's an optimization and not a regression fix. And it's wrong as well.
Quoted 10 lines
Signed-off-by Alex Shi <[email protected]>--- kernel/time/clocksource.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)kernel/time/clocksource.c +0
-0
kernel/time/clocksource.c static ssize_t sysfs_override_clocksource(struct device *dev, · line 890
Quoted 10 lines
kernel/time/clocksource.c +2
-4@@ -885,13 +885,15 @@ static ssize_t sysfs_override_clocksource(struct device *dev, {size_t ret; -mutex_lock (&clocksource_mutex); - ret =sysfs_get_uname (buf, override_name, count); -if (ret >=0 ) -clocksource_select (); +if (ret >=0 ) { +if (!strcmp (curr_clocksource->name, override_name))
What if you get preempted in the middle of sysfs_get_uname() or in the middle of strcmp() and some other code triggers a clocksource_select() while you are off the CPU?
That might end up in a half filled override_name buffer or accessing memory which might have been freed already because curr_clocksource changed and the old driver was unloaded.
Not pretty.
If at all we can do:
- mutex_lock(&clocksource_mutex);
-
- ret = sysfs_get_uname(buf, override_name, count);
+ ret = sysfs_get_uname(buf, tmp_buf, count);
+ if (ret < 0)
+ return ret;
+ mutex_lock(&clocksource_mutex);
+ if (strcmp(override_name, tmp_buf) != 0) {
+ memcpy(override_name, tmp_buf, sizeof(tmp_buf));
+ clocksource_select();
+ }
mutex_unlock(&clocksource_mutex);
return ret;
}
Thanks,
tglx
PATCH 3/3 clocksource: fix can not set tsc as clocksource bug
In reply to Alex Shicommit 5d33b883aed81c6fbcd09c6f7c3619eee850a7e2
clocksource: Always verify highres capability
This commit will reject a clock to be system clocksource if it has no CLOCK_SOURCE_VALID_FOR_HRES flags. Then the tsc to be rejected as clocksource, because this flag for tsc is set in clocksource_watchdog which run after the tsc register.
TSC registered in tsc_refine_calibration_work() and started the watchdog at that time.
This patch re-select the clocksource after we make sure the tsc has this flag. Fixed this bug.
--- kernel/time/clocksource.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/time/clocksource.c +7 -1
@@ -308,6 +308,8 @@ static void clocksource_watchdog(unsigned long data)* transition into high-res mode: */ tick_clock_notify (); +if (finished_booting) +schedule_work (&watchdog_work); } }
@@ -404,6 +406,7 @@ static void clocksource_dequeue_watchdog(struct clocksource *cs)spin_unlock_irqrestore (&watchdog_lock, flags); } +static void clocksource_select (void );static int clocksource_watchdog_kthread (void *data) {struct clocksource *cs, *tmp;
@@ -412,11 +415,14 @@ static int clocksource_watchdog_kthread(void *data)mutex_lock (&clocksource_mutex);spin_lock_irqsave (&watchdog_lock, flags); -list_for_each_entry_safe (cs, tmp, &watchdog_list, wd_list) +list_for_each_entry_safe (cs, tmp, &watchdog_list, wd_list) {if (cs->flags &CLOCK_SOURCE_UNSTABLE ) {list_del_init (&cs->wd_list);list_add (&cs->wd_list, &unstable); } +if (cs->flags &CLOCK_SOURCE_VALID_FOR_HRES ) +clocksource_select (); + }/* Check if the watchdog timer needs to be stopped. */ clocksource_stop_watchdog ();spin_unlock_irqrestore (&watchdog_lock, flags);
Signature · 1 line
1.7.12
Re: [PATCH 3/3] clocksource: fix can not set tsc as clocksource bug
Quoted 5 lines · Alex Shi
commit
5d33b883aed81c6fbcd09c6f7c3619eee850a7e2
clocksource: Always verify highres capabilityThis commit will reject a clock to be system clocksource if it has no
CLOCK_SOURCE_VALID_FOR_HRESflags. Then the tsc to be rejected as
No. It rejects the clocksource if the system is in oneshot mode and the clocksource does not support HIGHRES.
So at boot time, the tick device mode is PERIODIC and the clocksource is set to jiffies.
In clocksource_done_booting() we select the best clocksource from the registered list. We are still in PERIODIC mode, so the selection in clocksource_find_best() should grab TSC whether the HIGHRES_VALID flag is set or not. The relevant check in find_best() is:
if (oneshot && !(cs->flags & CLOCK_SOURCE_VALID_FOR_HRES))
continue; And oneshot should be definitely false at that point. So we don't care about the HRES flag at all.
So if we select TSC from clocksource_done_booting() that prevents the system from switching into highres mode as long as the VALID_FOR_HRES flag is not set by the watchdog. If it gets set then tick_clock_notify() tells the tick code to reevaluate.
So now the question is why you are observing that HPET is selected in the first place.
Can you add instrumentation to the code and provide the data please? I try to reproduce myself. What's the environment you're using?
kernel/time/clocksource.c static void clocksource_watchdog(unsigned long data) · line 312
Quoted 10 lines
kernel/time/clocksource.c +2
-0@@ -308,6 +308,8 @@ static void clocksource_watchdog(unsigned long data)* transition into high-res mode: */ tick_clock_notify (); +if (finished_booting) +schedule_work (&watchdog_work);
This only makes sense, when the clocksource which gets the FLAG set has the highest rating, but was not selected because the system was in ONESHOT mode already.
And I can't see why that should suddenly happen.
Quoted 5 lines
kernel/time/clocksource.c +0
-0static int clocksource_watchdog_kthread (void *data) {struct clocksource *cs, *tmp;@@ -412,11 +415,14 @@ static int clocksource_watchdog_kthread(void *data)
kernel/time/clocksource.c static int clocksource_watchdog_kthread(void *data) · line 424
Quoted 10 lines
kernel/time/clocksource.c +3
-1mutex_lock (&clocksource_mutex);spin_lock_irqsave (&watchdog_lock, flags); -list_for_each_entry_safe (cs, tmp, &watchdog_list, wd_list) +list_for_each_entry_safe (cs, tmp, &watchdog_list, wd_list) {if (cs->flags &CLOCK_SOURCE_UNSTABLE ) {list_del_init (&cs->wd_list);list_add (&cs->wd_list, &unstable); } +if (cs->flags &CLOCK_SOURCE_VALID_FOR_HRES ) +clocksource_select ();
Unlikely, but if we have 3 watched clocksources which have the HRES bit set, you'd call 3 times clocksource_select().
Thanks,
tglx
Re: [PATCH 3/3] clocksource: fix can not set tsc as clocksource bug
Quoted 29 lines · Thomas Gleixner
Quoted 5 lines · Alex Shi
commit
5d33b883aed81c6fbcd09c6f7c3619eee850a7e2
clocksource: Always verify highres capabilityThis commit will reject a clock to be system clocksource if it has no
CLOCK_SOURCE_VALID_FOR_HRESflags. Then the tsc to be rejected asNo. It rejects the clocksource if the system is in oneshot mode and the clocksource does not support HIGHRES.
So at boot time, the tick device mode is PERIODIC and the clocksource is set to jiffies.
In
clocksource_done_booting()we select the best clocksource from the registered list. We are still in PERIODIC mode, so the selection inclocksource_find_best()should grab TSC whether theHIGHRES_VALIDflag is set or not. The relevant check infind_best()is:if (oneshot && !(cs->flags & CLOCK_SOURCE_VALID_FOR_HRES)) continue;And oneshot should be definitely false at that point. So we don't care about the HRES flag at all.
So if we select TSC from
clocksource_done_booting()that prevents the system from switching into highres mode as long as theVALID_FOR_HRESflag is not set by the watchdog. If it gets set thentick_clock_notify()tells the tick code to reevaluate.
Quoted 5 lines
So now the question is why you are observing that HPET is selected in the first place.
Can you add instrumentation to the code and provide the data please? I try to reproduce myself. What's the environment you're using?
Ok. Figured it out.
Boot.
start tsc revalidation()
register(hpet)
register(pmtimer);
clocksource_done_booting()
select HPET
switch to oneshot
tsc_revalidation()
register(TSC)
clocksource_select()
keep HPET because TSC is not yet validated by the watchdog
watchdog validates TSC, but that does not make TSC the system clocksource.
So, the old code was actually installing TSC over HPET even when TSC was not yet qualified for HRES. Kinda worked :)
So yes, we need somthing like your patch to fix that.
Quoted 15 lines
kernel/time/clocksource.c static void clocksource_watchdog(unsigned long data) · line 312
Quoted 10 lines
kernel/time/clocksource.c +2
-0@@ -308,6 +308,8 @@ static void clocksource_watchdog(unsigned long data)* transition into high-res mode: */ tick_clock_notify (); +if (finished_booting) +schedule_work (&watchdog_work);This only makes sense, when the clocksource which gets the FLAG set has the highest rating, but was not selected because the system was in ONESHOT mode already.
Quoted 1 line
And I can't see why that should suddenly happen.
Now I can :)
Quoted 5 lines
Quoted 5 lines
kernel/time/clocksource.c +0
-0static int clocksource_watchdog_kthread (void *data) {struct clocksource *cs, *tmp;@@ -412,11 +415,14 @@ static int clocksource_watchdog_kthread(void *data)
Quoted 13 lines
kernel/time/clocksource.c static int clocksource_watchdog_kthread(void *data) · line 424
Quoted 10 lines
kernel/time/clocksource.c +3
-1mutex_lock (&clocksource_mutex);spin_lock_irqsave (&watchdog_lock, flags); -list_for_each_entry_safe (cs, tmp, &watchdog_list, wd_list) +list_for_each_entry_safe (cs, tmp, &watchdog_list, wd_list) {if (cs->flags &CLOCK_SOURCE_UNSTABLE ) {list_del_init (&cs->wd_list);list_add (&cs->wd_list, &unstable); } +if (cs->flags &CLOCK_SOURCE_VALID_FOR_HRES ) +clocksource_select ();Unlikely, but if we have 3 watched clocksources which have the HRES bit set, you'd call 3 times
clocksource_select().
Also the reselect must be called outside the watchdog_lock region.
Thanks,
tglx
PATCH 1/1 clocksource: Reselect clocksource when watchdog validated
Up to commit 5d33b883a (clocksource: Always verify highres capability) we had no sanity check when selecting a clocksource, which prevented that a non highres capable clocksource is used when the system already switched to highres/nohz mode.
The new sanity check works as Alex and Tim found out. It prevents the TSC from being used. This happens because on x86 the boot process looks like this:
tsc_start_freqency_validation(TSC); clocksource_register(HPET); clocksource_done_booting();
clocksource_select()
Selects HPET which is valid for high-res
switch_to_highres();
clocksource_register(TSC);
TSC is not selected, because it is not yet flagged as VALID_HIGH_RES
clocksource_watchdog()
Validates TSC for highres, but that does not make TSC the current clocksource.
Before the sanity check was added, we installed TSC unvalidated which worked most of the time. If the TSC was really detected as unstable, then the unstable logic removed it and installed HPET again.
The sanity check is correct and needed. So the watchdog needs to kick a reselection of the clocksource, when it qualifies TSC as a valid high res clocksource.
To solve this, we mark the clocksource which got the flag CLOCK_SOURCE_VALID_FOR_HRES set by the watchdog with an new flag CLOCK_SOURCE_RESELECT and trigger the watchdog thread. The watchdog thread evaluates the flag and invokes clocksource_select() when set.
To avoid that the clocksource_done_booting() code, which is about to install the first real clocksource anyway, needs to go through clocksource_select and tick_oneshot_notify() pointlessly, split out the clocksource_watchdog_kthread() list walk code and invoke the select/notify only when called from clocksource_watchdog_kthread().
So clocksource_done_booting() can utilize the same splitout code without the select/notify invocation and the clocksource_mutex unlock/relock dance.
--- include/linux/clocksource.h | 1 kernel/time/clocksource.c | 57 ++++++++++++++++++++++++++++++++------------ 2 files changed, 43 insertions(+), 15 deletions(-)
include/linux/clocksource.h +1 -0
@@ -210,6 +210,7 @@ struct clocksource {#define CLOCK_SOURCE_VALID_FOR_HRES 0x20#define CLOCK_SOURCE_UNSTABLE 0x40#define CLOCK_SOURCE_SUSPEND_NONSTOP 0x80 +#define CLOCK_SOURCE_RESELECT 0x100/* simplify initialization of mask field */ #define CLOCKSOURCE_MASK (bits) (cycle_t)((bits) < 64 ? ((1ULL<<(bits))-1) : -1)
kernel/time/clocksource.c +42 -15
122 lines not on this page · raw message
Re: [PATCH] clocksource: Reselect clocksource when watchdog validated
Quoted 8 lines · Thomas Gleixner
Reported-by Alex Shi <[email protected]>Cc Hans Peter Anvin <[email protected]>Cc Tim Chen <[email protected]>Cc Andi Kleen <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Ingo Molnar <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>
Glad to the problem fixed. Thanks!
Signature · 2 lines
Thanks
Alextip-bot for Thomas Gleixner
merge bot[tip:timers/core] clocksource: Reselect clocksource when watchdog validated high-res capability
In reply to Thomas Gleixnerclocksource: Reselect clocksource when watchdog validated high-res capability
Commit-ID: 332962f2c88868ed3cdab466870baaa34dd58612 Gitweb: http://git.kernel.org/tip/332962f2c88868ed3cdab466870baaa34dd58612 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 4 Jul 2013 22:46:45 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Fri, 5 Jul 2013 11:09:28 +0200
clocksource: Reselect clocksource when watchdog validated high-res capability
Up to commit 5d33b883a (clocksource: Always verify highres capability) we had no sanity check when selecting a clocksource, which prevented that a non highres capable clocksource is used when the system already switched to highres/nohz mode.
The new sanity check works as Alex and Tim found out. It prevents the TSC from being used. This happens because on x86 the boot process looks like this:
tsc_start_freqency_validation(TSC); clocksource_register(HPET); clocksource_done_booting();
clocksource_select()
Selects HPET which is valid for high-res
switch_to_highres();
clocksource_register(TSC);
TSC is not selected, because it is not yet flagged as VALID_HIGH_RES
clocksource_watchdog()
Validates TSC for highres, but that does not make TSC the current clocksource.
Before the sanity check was added, we installed TSC unvalidated which worked most of the time. If the TSC was really detected as unstable, then the unstable logic removed it and installed HPET again.
The sanity check is correct and needed. So the watchdog needs to kick a reselection of the clocksource, when it qualifies TSC as a valid high res clocksource.
To solve this, we mark the clocksource which got the flag CLOCK_SOURCE_VALID_FOR_HRES set by the watchdog with an new flag CLOCK_SOURCE_RESELECT and trigger the watchdog thread. The watchdog thread evaluates the flag and invokes clocksource_select() when set.
To avoid that the clocksource_done_booting() code, which is about to install the first real clocksource anyway, needs to go through clocksource_select and tick_oneshot_notify() pointlessly, split out the clocksource_watchdog_kthread() list walk code and invoke the select/notify only when called from clocksource_watchdog_kthread().
So clocksource_done_booting() can utilize the same splitout code without the select/notify invocation and the clocksource_mutex unlock/relock dance.
--- include/linux/clocksource.h | 1 + kernel/time/clocksource.c | 57 +++++++++++++++++++++++++++++++++------------ 2 files changed, 43 insertions(+), 15 deletions(-)
include/linux/clocksource.h +1 -0
@@ -210,6 +210,7 @@ struct clocksource {#define CLOCK_SOURCE_VALID_FOR_HRES 0x20#define CLOCK_SOURCE_UNSTABLE 0x40#define CLOCK_SOURCE_SUSPEND_NONSTOP 0x80 +#define CLOCK_SOURCE_RESELECT 0x100/* simplify initialization of mask field */ #define CLOCKSOURCE_MASK (bits) (cycle_t)((bits) < 64 ? ((1ULL<<(bits))-1) : -1)
kernel/time/clocksource.c +42 -15
122 lines not on this page · raw message
Re: [PATCH 3/3] clocksource: fix can not set tsc as clocksource bug
In reply to Thomas GleixnerQuoted 18 lines · Thomas Gleixner
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
static int
clocksource_watchdog_kthread(void *data){kernel/time/clocksource.c +0
-0struct clocksource *cs, *tmp;@@ -412,11 +415,14 @@ static int clocksource_watchdog_kthread(void *data)Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
kernel/time/clocksource.c +1
-1mutex_lock (&clocksource_mutex);spin_lock_irqsave (&watchdog_lock, flags); -list_for_each_entry_safe (cs, tmp, &watchdog_list, wd_list) +list_for_each_entry_safe (cs, tmp, &watchdog_list, wd_list) {if (cs->flags &CLOCK_SOURCE_UNSTABLE ) {Quoted 8 lines
Quoted 5 lines
kernel/time/clocksource.c static int clocksource_watchdog_kthread(void *data) · line 424
Quoted 5 lines
kernel/time/clocksource.c +2
-0list_del_init (&cs->wd_list);list_add (&cs->wd_list, &unstable); } +if (cs->flags &CLOCK_SOURCE_VALID_FOR_HRES ) +clocksource_select ();Unlikely, but if we have 3 watched clocksources which have the HRES bit set, you'd call 3 times
clocksource_select().
Quoted 1 line
Also the reselect must be called outside the
watchdog_lockregion.
Sorry for stupid, the watchdog_lock used protect watchdog related resource. but clocksource_select doesn't touch them. So, I know it isn't necessary to put reselect under this lock. Just don't know why the reselect *must* be called outside of it?
Signature · 2 lines
Thanks
AlexRe: [PATCH 3/3] clocksource: fix can not set tsc as clocksource bug
Quoted 19 lines · Alex Shi
Quoted 18 lines · Thomas Gleixner
Quoted 18 lines
Quoted 18 lines
Quoted 5 lines
Quoted 5 lines
static int
clocksource_watchdog_kthread(void *data){kernel/time/clocksource.c +0
-0struct clocksource *cs, *tmp;@@ -412,11 +415,14 @@ static int clocksource_watchdog_kthread(void *data)Quoted 10 lines
kernel/time/clocksource.c static int clocksource_watchdog_kthread(void *data) · line 424
Quoted 10 lines
kernel/time/clocksource.c +3
-1mutex_lock (&clocksource_mutex);spin_lock_irqsave (&watchdog_lock, flags); -list_for_each_entry_safe (cs, tmp, &watchdog_list, wd_list) +list_for_each_entry_safe (cs, tmp, &watchdog_list, wd_list) {if (cs->flags &CLOCK_SOURCE_UNSTABLE ) {list_del_init (&cs->wd_list);list_add (&cs->wd_list, &unstable); } +if (cs->flags &CLOCK_SOURCE_VALID_FOR_HRES ) +clocksource_select ();Unlikely, but if we have 3 watched clocksources which have the HRES bit set, you'd call 3 times
clocksource_select().
Quoted 6 lines
Quoted 1 line
Also the reselect must be called outside the
watchdog_lockregion.Sorry for stupid, the
watchdog_lockused protect watchdog related resource. butclocksource_selectdoesn't touch them. So, I know it isn't necessary to put reselect under this lock. Just don't know why the reselect *must* be called outside of it?
clocksource_select()
timekeeping_notify()
stop_machine()
get_online_cpus()
might_sleep()
mutex_lock()
So we need to be in preemptible context when we call clocksource_select().
Thanks,
tglx
Re: [PATCH 3/3] clocksource: fix can not set tsc as clocksource bug
Quoted 8 lines · Thomas Gleixner
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Unlikely, but if we have 3 watched clocksources which have the HRES bit set, you'd call 3 times
clocksource_select().Also the reselect must be called outside the
watchdog_lockregion.Sorry for stupid, the
watchdog_lockused protect watchdog related resource. butclocksource_selectdoesn't touch them. So, I know it isn't necessary to put reselect under this lock. Just don't know why the reselect *must* be called outside of it?
Quoted 8 lines
clocksource_select()
timekeeping_notify()
stop_machine()
get_online_cpus()
might_sleep()
mutex_lock()So we need to be in preemptible context when we call
clocksource_select().
understand! many thanks!
Signature · 2 lines
Thanks
AlexQuoted 2 lines · Alex Shi
If the tsc is marked as constant and nonstop, could we set it as system clocksource when do tsc register? w/o checking it on
clocksource_watchdog?
I'd not do that; the BIOS can still screw you over, we need some validation.
That said; we do need means to disable the clocksource watchdog -- although I suppose Frederic might already have provided this for this NOHZ efforts when I wasn't looking.
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Alex Shi
If the tsc is marked as constant and nonstop, could we set it as system clocksource when do tsc register? w/o checking it on
clocksource_watchdog?I'd not do that; the BIOS can still screw you over, we need some validation.
I see. thanks!
Quoted 5 lines
That said; we do need means to disable the clocksource watchdog -- although I suppose Frederic might already have provided this for this NOHZ efforts when I wasn't looking.
Signature · 2 lines
Thanks
Alex2013/7/4 Peter Zijlstra <[email protected]>:
Quoted 10 lines · Alex Shi
Quoted 3 lines
If the tsc is marked as constant and nonstop, could we set it as system clocksource when do tsc register? w/o checking it on
clocksource_watchdog?I'd not do that; the BIOS can still screw you over, we need some validation.
That said; we do need means to disable the clocksource watchdog -- although I suppose Frederic might already have provided this for this NOHZ efforts when I wasn't looking.
Nope, I haven't touched that. I prefer not to fiddle with unstable clocksource for now :)
As for unstable TSCs, if sched_clock_tick() needs to be fed, we simply don't stop the tick.
Quoted 5 lines · Frederic Weisbecker
Nope, I haven't touched that. I prefer not to fiddle with unstable clocksource for now :)
As for unstable TSCs, if
sched_clock_tick()needs to be fed, we simply don't stop the tick.
Not entirely the same thing; I thought the clocksource watchdog was ran even when we have a 'stable' TSC, just to make sure it stays stable. There's known cases where the BIOS f*cks us over and wrecks TSC sync.
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
Nope, I haven't touched that. I prefer not to fiddle with unstable clocksource for now :)
As for unstable TSCs, if
sched_clock_tick()needs to be fed, we simply don't stop the tick.Not entirely the same thing; I thought the clocksource watchdog was ran even when we have a 'stable' TSC, just to make sure it stays stable. There's known cases where the BIOS f*cks us over and wrecks TSC sync.
See arch/x86/kernel/tsc.c
We disable the watchdog for the TSC when tsc_clocksource_reliable is set.
tsc_clocksource_reliable is set when:
- you add tsc=reliable to the kernel command line
- boot_cpu_has(X86_FEATURE_TSC_RELIABLE)
X86_FEATURE_TSC_RELIABLE is a software flag, set by vmware and moorsetown. So all other machines keep the watchdog enabled.
- On Geode LX (OLPC)
Thanks,
tglx
Quoted 8 lines · Thomas Gleixner
See
arch/x86/kernel/tsc.cWe disable the watchdog for the TSC when
tsc_clocksource_reliableis set.
tsc_clocksource_reliableis set when:- you add tsc=reliable to the kernel command line
Ah, I didn't know about that one, useful.
Quoted 4 lines
-
boot_cpu_has(X86_FEATURE_TSC_RELIABLE)
X86_FEATURE_TSC_RELIABLEis a software flag, set by vmware and moorsetown. So all other machines keep the watchdog enabled.
Right.. I knew it was enabled on my machines even though they normally have usable TSC.
Quoted 12 lines · Peter Zijlstra
Quoted 8 lines · Thomas Gleixner
See
arch/x86/kernel/tsc.cWe disable the watchdog for the TSC when
tsc_clocksource_reliableis set.
tsc_clocksource_reliableis set when:- you add tsc=reliable to the kernel command line
Ah, I didn't know about that one, useful.
Quoted 7 lines
Quoted 4 lines
-
boot_cpu_has(X86_FEATURE_TSC_RELIABLE)
X86_FEATURE_TSC_RELIABLEis a software flag, set by vmware and moorsetown. So all other machines keep the watchdog enabled.Right.. I knew it was enabled on my machines even though they normally have usable TSC.
Yeah, but our well justified paranoia still prevents us from trusting these CPU flags. Maybe some day BIOS is going to be replaced by something useful. You know: Hope springs eternal....
Quoted 3 lines · Thomas Gleixner
Yeah, but our well justified paranoia still prevents us from trusting these CPU flags. Maybe some day BIOS is going to be replaced by something useful. You know: Hope springs eternal....
Not in the next 10 yrs at least if one took a look at the overengineered, obese at birth and braindead crap by the name of UEFI.
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Quoted 7 lines · Borislav Petkov
Quoted 3 lines · Thomas Gleixner
Yeah, but our well justified paranoia still prevents us from trusting these CPU flags. Maybe some day BIOS is going to be replaced by something useful. You know: Hope springs eternal....
Not in the next 10 yrs at least if one took a look at the overengineered, obese at birth and braindead crap by the name of UEFI.
Good news! 10 years is way less than eternity and just before retirement :)
Quoted 2 lines · Thomas Gleixner
Good news! 10 years is way less than eternity and just before retirement :)
You know that after the 10 years they'll come up with an even uglier platform-differentiation-fiddle-with-dong-while-smoking-crack-crap which will even replace the OS, right?
Hmm, I'm wondering what would be faster: wait *at least* 10 more years or get an old mainboard and start experimenting with coreboot...
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Quoted 18 lines · Thomas Gleixner
Quoted 17 lines · Peter Zijlstra
Quoted 8 lines · Thomas Gleixner
See
arch/x86/kernel/tsc.cWe disable the watchdog for the TSC when
tsc_clocksource_reliableis set.
tsc_clocksource_reliableis set when:- you add tsc=reliable to the kernel command line
Ah, I didn't know about that one, useful.
Quoted 4 lines
-
boot_cpu_has(X86_FEATURE_TSC_RELIABLE)
X86_FEATURE_TSC_RELIABLEis a software flag, set by vmware and moorsetown. So all other machines keep the watchdog enabled.
Quoted 6 lines
Quoted 2 lines
Right.. I knew it was enabled on my machines even though they normally have usable TSC.
Yeah, but our well justified paranoia still prevents us from trusting these CPU flags. Maybe some day BIOS is going to be replaced by something useful. You know: Hope springs eternal....
Oh quite agreed. Its just that at several times I've wanted to disable the thing. Now I know you can do using the kernel cmdline. Previously I had to wreck code -- not that much harder really :-)
Quoted 5 lines · Alex Shi
We find some benchmarks drop a lot on tip/sched/core on many Intel boxes, like oltp, tbench, hackbench etc. and bisected the commit
5d33b883aecause this regression. Due to this commit, the clocksource was changed to hpet from tsc even tsc will be setCLOCK_SOURCE_VALID_FOR_HRESlater inclocksource_watchdog.
5d33b883ae is not in tip/sched/core. So what are you testing and bisecting?
Thanks,
tglx
Davidlohr Bueso
Acked-by John StultzQuoted 10 lines · Thomas Gleixner
Quoted 5 lines · Alex Shi
We find some benchmarks drop a lot on tip/sched/core on many Intel boxes, like oltp, tbench, hackbench etc. and bisected the commit
5d33b883aecause this regression. Due to this commit, the clocksource was changed to hpet from tsc even tsc will be setCLOCK_SOURCE_VALID_FOR_HRESlater inclocksource_watchdog.
5d33b883aeis not in tip/sched/core. So what are you testing and bisecting?
I think he's referring to:
commit 5d33b883aed81c6fbcd09c6f7c3619eee850a7e2
Author: Thomas Gleixner <[email protected]> Date: Thu Apr 25 20:31:43 2013 +0000
clocksource: Always verify highres capability
If a clocksource has a (wrong) high rating, but can't be used as a timebase for oneshot tick mode, it is unconditionally selected even when the system is already in oneshot tick mode. This causes full system failure.
Verify the clocksource selection against the oneshot mode.
Quoted 9 lines · Davidlohr Bueso
Quoted 8 lines · Thomas Gleixner
Quoted 5 lines · Alex Shi
We find some benchmarks drop a lot on tip/sched/core on many Intel boxes, like oltp, tbench, hackbench etc. and bisected the commit
5d33b883aecause this regression. Due to this commit, the clocksource was changed to hpet from tsc even tsc will be setCLOCK_SOURCE_VALID_FOR_HRESlater inclocksource_watchdog.
Quoted 6 lines
Quoted 2 lines
5d33b883aeis not in tip/sched/core. So what are you testing and bisecting?I think he's referring to:
commit
5d33b883aed81c6fbcd09c6f7c3619eee850a7e2
I know what he is referring to. He explicitly mentions this commit:
Quoted 1 line
Quoted 1 line
Quoted 1 line
like oltp, tbench, hackbench etc. and bisected the commit
5d33b883ae
What I was pointing out that he was referring to tip sched/core at the same time
Quoted 1 line
Quoted 1 line
Quoted 1 line
We find some benchmarks drop a lot on tip/sched/core on many Intel boxes,
And that branch does NOT have that commit included. So how can you see a regression on a branch caused by a commit NOT included into that branch?
The offending commit is in tip timers/core and not in tip sched/core. What I'm wanted to say is, that we need a proper description of problems and not some random association.
It tricked you to assume, that I'm not able to figure it out myself :)
See? These things are complex and subtle, so we need precise descriptions and not some sloppy semi correct data.
I'm well aware of the issue and with Peters help I got a reasonable explanation for it. A proper fix is about to be sent out.
Thanks,
tglx
Quoted 9 lines · Thomas Gleixner
Quoted 1 line
Quoted 1 line
Quoted 1 line
We find some benchmarks drop a lot on tip/sched/core on many Intel boxes,
And that branch does NOT have that commit included. So how can you see a regression on a branch caused by a commit NOT included into that branch?
The offending commit is in tip timers/core and not in tip sched/core. What I'm wanted to say is, that we need a proper description of problems and not some random association.
Quoted 8 lines
It tricked you to assume, that I'm not able to figure it out myself :)
See? These things are complex and subtle, so we need precise descriptions and not some sloppy semi correct data.
I'm well aware of the issue and with Peters help I got a reasonable explanation for it. A proper fix is about to be sent out.
Ingo had merged your branch into sched/core. :)
commit f9bed7021710a3e45c331f7d7781de914cc1b939
Merge: 7e76057 67dd331
Author: Ingo Molnar <[email protected]>
Date: Wed May 29 11:21:59 2013 +0200
Merge branch 'timers/urgent'
...
commit 7d194f78bde64ec813c1ed8291181bdd61515e78
Merge: 0298bf7 1eaff67
Author: Ingo Molnar <[email protected]>
Date: Tue May 28 09:53:41 2013 +0200
Merge branch 'timers/core'
commit 0298bf70644d7334bec16ae47f3aa58f4f883b59
Merge: cc662fa 2938d27
Author: Ingo Molnar <[email protected]>
Date: Tue May 28 09:49:51 2013 +0200
Merge branch 'timers/urgent'
Signature · 2 lines
Thanks
AlexQuoted 19 lines · Alex Shi
Quoted 9 lines · Thomas Gleixner
Quoted 1 line
Quoted 1 line
Quoted 1 line
We find some benchmarks drop a lot on tip/sched/core on many Intel boxes,
And that branch does NOT have that commit included. So how can you see a regression on a branch caused by a commit NOT included into that branch?
The offending commit is in tip timers/core and not in tip sched/core. What I'm wanted to say is, that we need a proper description of problems and not some random association.
Quoted 8 lines
It tricked you to assume, that I'm not able to figure it out myself :)
See? These things are complex and subtle, so we need precise descriptions and not some sloppy semi correct data.
I'm well aware of the issue and with Peters help I got a reasonable explanation for it. A proper fix is about to be sent out.
Quoted 8 lines
Ingo had merged your branch into sched/core. :)
commit
f9bed7021710a3e45c331f7d7781de914cc1b939
Merge: 7e76057 67dd331
Author: Ingo Molnar <[email protected]>
Date: Wed May 29 11:21:59 2013 +0200Merge branch 'timers/urgent'
Not really.
tip$ git branch --contains f9bed70217
* master
tip$ git branch --contains 5d33b883ae
* master
timers/core
So you are testing tip/master not tip/sched/core.
Thanks,
tglx
Quoted 11 lines · Thomas Gleixner
Quoted 9 lines
Quoted 9 lines
Ingo had merged your branch into sched/core. :)
commit
f9bed7021710a3e45c331f7d7781de914cc1b939
Merge: 7e76057 67dd331
Author: Ingo Molnar <[email protected]>
Date: Wed May 29 11:21:59 2013 +0200Merge branch 'timers/urgent'
Not really.
Quoted 8 lines
tip$ git branch --contains
f9bed70217
* mastertip$ git branch --contains
5d33b883ae
* master
timers/coreSo you are testing tip/master not tip/sched/core.
You'r right. I mixed them. sorry.
Signature · 2 lines
Thanks
Alex