Highres / dynticks drop in replacement for 2.6.19-rc5-mm1
Subsystems (8)
cover Highres / dynticks drop in replacement for 2.6.19-rc5-mm1
Andrew,
this is a drop in replacement for the following patches in 2.6.19-rc5-mm1:
hrtimers-state-tracking.patch
up to
acpi-verify-lapic-timer-fix.patch
The patch set is taking all the changes made during the -mm testing into account and merged them back at the appropriate places.
Changes vs. the initially reviewed patch set:
- Trivial compile / Kconfig fixes
- command line paramater fixups
- apic timer / ACPI (C states) broadcasting fixups
- Disabled apic timer for high res on UP systems due to the unsolvable brokeness of BIOS supplied C state functionality
- APIC code cleanup in i386
- reworked APIC timer calibration
Dropped from the rc5-mm1 patch conglomerate:
- the naive attempt to detect the local APIC timer brokeness in C2 state due to circular dependency on interrupts (where the local APIC timer interrupt might be the only active one). The problem is detectable, but it needs more thought and the gathered information/experience is not lost ! Replaced by brute force for now.
Some annotations for making the review simpler:
hrtimers-state-tracking.patch
callback state trivial fix
hrtimers-clean-up-callback-tracking.patch
no changes, kept for linearity
hrtimers-move-and-add-documentation.patch
no changes, kept for linearity
clockevents-core.patch
One off bug fixed
inlcude and compile fixes
broadcast support
acpi-include-apic-h.patch
new
acpi-keep-track-of-timer-broadcast.patch
new
acpi-add-hres-dyntick-broadcast-support.patch
new
i386-cleanup-apic.patch
new, no functional changes
clockevents-drivers-for-i386.patch
broadcast fixups
pm-timer-allow-early-access.patch
new, no functional changes
i386-lapic-calibrate-timer.patch
new
high-res-timers-core.patch
trivial fixups
gtod-mark-tsc-unusable-for-highres-timers.patch
no changes, kept for linearity
dynticks-core.patch
trivial fixups
dynticks-add-nohz-stats-to-proc-stat.patch
no changes, patch fuzz due to prior patches
dynticks-i386-arch-code.patch
no changes, patch fuzz due to prior patches
dynticks-i386-nmi-fix.patch
new
high-res-timers-dynticks-enable-i386-support.patch
no changes, patch fuzz due to prior patches
debugging-feature-timer-stats.patch
no changes, patch fuzz due to prior patches
Thanks,
tglx
--
PATCH 1/19 hrtimers: state tracking
From: Thomas Gleixner <[email protected]>
Reintroduce ktimers feature "optimized away" by the ktimers review process:
multiple hrtimer states to enable the running of hrtimers without holding the cpu-base-lock.
(The "optimized" rbtree hack carried only 2 states worth of information and we need 4 for high resolution timers and dynamic ticks.)
Build-fixes-from: Andrew Morton <[email protected]>
include/linux/hrtimer.h +35 -1
@@ -40,6 +40,34 @@ enum hrtimer_restart { HRTIMER_RESTART, /* Timer must be restarted */ }; +/* +* Bit values to track state of the timer +* +* Possible states: +* +* 0x00 inactive +* 0x01 enqueued into rbtree +* 0x02 callback function running +* 0x03 callback function running and enqueued +* (was requeued on another CPU) +* +* The "callback function running and enqueued" status is only possible on +* SMP. It happens for example when a posix timer expired and the callback +* queued a signal. Between dropping the lock which protects the posix timer +* and reacquiring the base lock of the hrtimer, another CPU can deliver the +* signal and rearm the timer. We have to preserve the callback running state, +* as otherwise the timer could be removed before the softirq code finishes the +* the handling of the timer. +* +* The HRTIMER_STATE_ENQUEUE bit is always or'ed to the current state to +* preserve the HRTIMER_STATE_CALLBACK bit in the above scenario. +* +* All state transitions are protected by cpu_base->lock. +*/ +#define HRTIMER_STATE_INACTIVE 0x00 +#define HRTIMER_STATE_ENQUEUED 0x01 +#define HRTIMER_STATE_CALLBACK 0x02 +/** * struct hrtimer - the basic hrtimer structure * @node: red black tree node for time ordered insertion
@@ -48,6 +76,7 @@ enum hrtimer_restart {* which the timer is based. * @function: timer expiry callback function * @base: pointer to the timer base (per cpu and per clock) +* @state: state information (See bit values above) * * The hrtimer structure must be initialized by init_hrtimer_#CLOCKTYPE() */
@@ -56,6 +85,7 @@ struct hrtimer {ktime_t expires;enum hrtimer_restart (*function)(struct hrtimer *);struct hrtimer_clock_base *base; +unsigned long state; };/**
@@ -141,9 +171,13 @@ extern int hrtimer_get_res(const clockidextern ktime_t hrtimer_get_next_event (void );#endif +/* +* A timer is active, when it is enqueued into the rbtree or the callback +* function is running. +*/ static inline int hrtimer_active (const struct hrtimer *timer) { -return rb_parent (&timer->node) != &timer->node; +return timer->state !=HRTIMER_STATE_INACTIVE ; }/* Forward a hrtimer so it expires after now: */
kernel/hrtimer.c +21 -8
@@ -235,6 +235,12 @@ lock_hrtimer_base(const struct hrtimer *#endif /* !CONFIG_SMP */ +static inline int hrtimer_is_queued (struct hrtimer *timer) +{ +return timer->state !=HRTIMER_STATE_INACTIVE && + timer->state !=HRTIMER_STATE_CALLBACK ; +} +/* * Functions for the union type storage format of ktime_t which are * too large for inlining:
@@ -385,6 +391,11 @@ static void enqueue_hrtimer(struct hrtim*/ rb_link_node (&timer->node, parent, link);rb_insert_color (&timer->node, &base->active); +/* +* HRTIMER_STATE_ENQUEUED is or'ed to the current state to preserve the +* state of a possibly running callback. +*/ + timer->state |=HRTIMER_STATE_ENQUEUED ;if (!base->first || timer->expires.tv64 < rb_entry(base->first, struct hrtimer, node)->expires.tv64)
@@ -397,7 +408,8 @@ static void enqueue_hrtimer(struct hrtim* Caller must hold the base lock. */ static void __remove_hrtimer (struct hrtimer *timer, -struct hrtimer_clock_base *base) +struct hrtimer_clock_base *base, +unsigned long newstate) {/* * Remove the timer from the rbtree and replace the
@@ -406,7 +418,7 @@ static void __remove_hrtimer(struct hrtiif (base->first == &timer->node) base->first =rb_next (&timer->node);rb_erase (&timer->node, &base->active); -rb_set_parent (&timer->node, &timer->node); + timer->state = newstate; }/*
@@ -415,8 +427,8 @@ static void __remove_hrtimer(struct hrtistatic inline int remove_hrtimer (struct hrtimer *timer,struct hrtimer_clock_base *base) { -if (hrtimer_active (timer)) { -__remove_hrtimer (timer, base); +if (hrtimer_is_queued (timer)) { +__remove_hrtimer (timer, base,HRTIMER_STATE_INACTIVE );return 1 ; }return 0 ;
@@ -488,7 +500,7 @@ int hrtimer_try_to_cancel(struct hrtimer base =lock_hrtimer_base (timer, &flags); -if (base->cpu_base->curr_timer != timer) +if (!(timer->state &HRTIMER_STATE_CALLBACK )) ret =remove_hrtimer (timer, base);unlock_hrtimer_base (timer, &flags);
@@ -593,7 +605,6 @@ void hrtimer_init(struct hrtimer *timer, clock_id =CLOCK_MONOTONIC ; timer->base = &cpu_base->clock_base[clock_id]; -rb_set_parent (&timer->node, &timer->node); }EXPORT_SYMBOL_GPL (hrtimer_init);
@@ -644,13 +655,14 @@ static inline void run_hrtimer_queue(str fn = timer->function;set_curr_timer (cpu_base, timer); -__remove_hrtimer (timer, base); +__remove_hrtimer (timer, base,HRTIMER_STATE_CALLBACK );spin_unlock_irq (&cpu_base->lock); restart =fn (timer);spin_lock_irq (&cpu_base->lock); + timer->state &= ~HRTIMER_STATE_CALLBACK ;if (restart !=HRTIMER_NORESTART ) {BUG_ON (hrtimer_active (timer));enqueue_hrtimer (timer, base);
@@ -821,7 +833,8 @@ static void migrate_hrtimer_list(structwhile ((node =rb_first (&old_base->active))) { timer = rb_entry(node, struct hrtimer, node); -__remove_hrtimer (timer, old_base); +BUG_ON (timer->state &HRTIMER_STATE_CALLBACK ); +__remove_hrtimer (timer, old_base,HRTIMER_STATE_INACTIVE ); timer->base = new_base;enqueue_hrtimer (timer, new_base); }
--
Re: [patch 01/19] hrtimers: state tracking
Quoted 18 lines
include/linux/hrtimer.h +17
-0+/* +* Bit values to track state of the timer +* +* Possible states: +* +* 0x00 inactive +* 0x01 enqueued into rbtree +* 0x02 callback function running +* 0x03 callback function running and enqueued +* (was requeued on another CPU) +* +* The "callback function running and enqueued" status is only possible on +* SMP. It happens for example when a posix timer expired and the callback +* queued a signal. Between dropping the lock which protects the posix timer +* and reacquiring the base lock of the hrtimer, another CPU can deliver the +* signal and rearm the timer. We have to preserve the callback running state, +* as otherwise the timer could be removed before the softirq code finishes the
include/linux/hrtimer.h enum hrtimer_restart { · line 69
Quoted 10 lines
include/linux/hrtimer.h +10
-0+* the handling of the timer. +* +* The HRTIMER_STATE_ENQUEUE bit is always or'ed to the current state to +* preserve the HRTIMER_STATE_CALLBACK bit in the above scenario. +* +* All state transitions are protected by cpu_base->lock. +*/ +#define HRTIMER_STATE_INACTIVE 0x00 +#define HRTIMER_STATE_ENQUEUED 0x01 +#define HRTIMER_STATE_CALLBACK 0x02
where is the define for 0x03?
kernel/hrtimer.c lock_hrtimer_base(const struct hrtimer * · line 242
Quoted 6 lines
kernel/hrtimer.c +5
-0+static inline int hrtimer_is_queued (struct hrtimer *timer) +{ +return timer->state !=HRTIMER_STATE_INACTIVE && + timer->state !=HRTIMER_STATE_CALLBACK ; +}
the state things are either bits or they're not. If they're bits, you probably want to make this a bitcheck instead...
kernel/hrtimer.c static void enqueue_hrtimer(struct hrtim · line 398
Quoted 6 lines
kernel/hrtimer.c +5
-0rb_insert_color (&timer->node, &base->active); +/* +* HRTIMER_STATE_ENQUEUED is or'ed to the current state to preserve the +* state of a possibly running callback. +*/ + timer->state |=HRTIMER_STATE_ENQUEUED ;
ok so it IS a bit thing, see comment about hrtimer_is_queued() not being a bit check then...
kernel/hrtimer.c int hrtimer_try_to_cancel(struct hrtimer · line 504
Quoted 3 lines
kernel/hrtimer.c +1
-1-if (base->cpu_base->curr_timer != timer) +if (!(timer->state &HRTIMER_STATE_CALLBACK )) ret =remove_hrtimer (timer, base);
if there is a hrtimer_is_queued() inline, might as well make a hrtimer_is_running() inline as well
otherwise lookes ok; if you fix these few comments:
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Re: [patch 01/19] hrtimers: state tracking
Quoted 40 lines · On Fri, 10 Nov 2006 10:19:48 +0100 Arjan van de Ven <[email protected]> wrote:
Quoted 18 lines
include/linux/hrtimer.h +17
-0+/* +* Bit values to track state of the timer +* +* Possible states: +* +* 0x00 inactive +* 0x01 enqueued into rbtree +* 0x02 callback function running +* 0x03 callback function running and enqueued +* (was requeued on another CPU) +* +* The "callback function running and enqueued" status is only possible on +* SMP. It happens for example when a posix timer expired and the callback +* queued a signal. Between dropping the lock which protects the posix timer +* and reacquiring the base lock of the hrtimer, another CPU can deliver the +* signal and rearm the timer. We have to preserve the callback running state, +* as otherwise the timer could be removed before the softirq code finishes the include/linux/hrtimer.h enum hrtimer_restart { · line 69
Quoted 10 lines
include/linux/hrtimer.h +10
-0+* the handling of the timer. +* +* The HRTIMER_STATE_ENQUEUE bit is always or'ed to the current state to +* preserve the HRTIMER_STATE_CALLBACK bit in the above scenario. +* +* All state transitions are protected by cpu_base->lock. +*/ +#define HRTIMER_STATE_INACTIVE 0x00 +#define HRTIMER_STATE_ENQUEUED 0x01 +#define HRTIMER_STATE_CALLBACK 0x02where is the define for
0x03?kernel/hrtimer.c lock_hrtimer_base(const struct hrtimer * · line 242
Quoted 6 lines
kernel/hrtimer.c +5
-0+static inline int hrtimer_is_queued (struct hrtimer *timer) +{ +return timer->state !=HRTIMER_STATE_INACTIVE && + timer->state !=HRTIMER_STATE_CALLBACK ; +}the state things are either bits or they're not. If they're bits, you probably want to make this a bitcheck instead...
Quoted 10 lines
kernel/hrtimer.c static void enqueue_hrtimer(struct hrtim · line 398
Quoted 6 lines
kernel/hrtimer.c +5
-0rb_insert_color (&timer->node, &base->active); +/* +* HRTIMER_STATE_ENQUEUED is or'ed to the current state to preserve the +* state of a possibly running callback. +*/ + timer->state |=HRTIMER_STATE_ENQUEUED ;ok so it IS a bit thing, see comment about
hrtimer_is_queued()not being a bit check then...
eek. I exhaustively went over that confusion in my initial (and lengthy) review of these patches.
I don't think we ever saw a point-by-point reply. What got lost?
Re: [patch 01/19] hrtimers: state tracking
Quoted 9 lines · Andrew Morton
Quoted 4 lines
ok so it IS a bit thing, see comment about
hrtimer_is_queued()not being a bit check then...eek. I exhaustively went over that confusion in my initial (and lengthy) review of these patches.
I don't think we ever saw a point-by-point reply. What got lost?
I added comments in the defines and in the code as you requested. Obviously not enough comments.
tglx
Re: [patch 01/19] hrtimers: state tracking
In reply to Thomas GleixnerHi,
Quoted 5 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Reintroduce ktimers feature "optimized away" by the ktimers review process:
multiple hrtimer states to enable the running of hrtimers without holding the cpu-base-lock.
They were "optimized away" for a reason...
Quoted 2 lines
(The "optimized" rbtree hack carried only 2 states worth of information and we need 4 for high resolution timers and dynamic ticks.)
If you need further flags for dynticks, then to do it conditionally, but keep this as is (at least for now), keep it small for the simple stuff. As others have noted your usage is confusing, something like this hard to maintain - every time a flag is added/changed, almost every user has to checked to insure the state machine stays correct. Keep the basic states separate and if it the flag field should be needed for other reason, it should be easy enough to convert. This is not needed for an initial merge. (Same goes for the next patch.)
bye, Roman
PATCH 2/19 hrtimers: clean up callback tracking
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Reintroduce ktimers feature "optimized away" by the ktimers review process:
remove the curr_timer pointer from the cpu-base and use the hrtimer state.
include/linux/hrtimer.h +0 -1
@@ -136,7 +136,6 @@ struct hrtimer_cpu_base {spinlock_t lock;struct lock_class_key lock_key;struct hrtimer_clock_base clock_base[HRTIMER_MAX_CLOCK_BASES ]; -struct hrtimer *curr_timer; };/*
kernel/hrtimer.c +1 -9
@@ -150,8 +150,6 @@ static void hrtimer_get_softirq_time(str*/ #ifdef CONFIG_SMP -#define set_curr_timer (b, t) do { (b)->curr_timer = (t); } while (0) -/* * We are using hashed locking: holding per_cpu(hrtimer_bases)[n].lock * means that all timers which are tied to this base via timer->base are
@@ -205,7 +203,7 @@ switch_hrtimer_base(struct hrtimer *time* completed. There is no conflict as we hold the lock until * the timer is enqueued. */ -if (unlikely (base->cpu_base->curr_timer == timer)) +if (unlikely (timer->state &HRTIMER_STATE_CALLBACK ))return base;/* See the comment in lock_timer_base() */
@@ -219,8 +217,6 @@ switch_hrtimer_base(struct hrtimer *time#else /* CONFIG_SMP */ -#define set_curr_timer (b, t) do { } while (0) -static inline struct hrtimer_clock_base *lock_hrtimer_base (const struct hrtimer *timer,unsigned long *flags) {
@@ -654,7 +650,6 @@ static inline void run_hrtimer_queue(strbreak ; fn = timer->function; -set_curr_timer (cpu_base, timer);__remove_hrtimer (timer, base,HRTIMER_STATE_CALLBACK );spin_unlock_irq (&cpu_base->lock);
@@ -668,7 +663,6 @@ static inline void run_hrtimer_queue(strenqueue_hrtimer (timer, base); } } -set_curr_timer (cpu_base,NULL );spin_unlock_irq (&cpu_base->lock); }
@@ -855,8 +849,6 @@ static void migrate_hrtimers(int cpu)spin_lock (&old_base->lock);for (i =0 ; i <HRTIMER_MAX_CLOCK_BASES ; i++) { -BUG_ON (old_base->curr_timer); -migrate_hrtimer_list (&old_base->clock_base[i], &new_base->clock_base[i]); }
--
Re: [patch 02/19] hrtimers: clean up callback tracking
Quoted 13 lines · Thomas Gleixner
plain text document attachment
(hrtimers-clean-up-callback-tracking.patch)
From: Thomas Gleixner <[email protected]>Reintroduce ktimers feature "optimized away" by the ktimers review process:
remove thecurr_timerpointer from the cpu-base and use the hrtimer state.Signed-off-by Thomas Gleixner <[email protected]>Signed-off-by Ingo Molnar <[email protected]>Index:
linux-2.6.19-rc5-mm1/include/linux/hrtimer.h
===================================================================
--- linux-2.6.19-rc5-mm1.orig/include/l
kernel/hrtimer.c switch_hrtimer_base(struct hrtimer *time · line 207
Quoted 3 lines
kernel/hrtimer.c +1
-1-if (unlikely (base->cpu_base->curr_timer == timer)) +if (unlikely (timer->state &HRTIMER_STATE_CALLBACK ))return base;
this also could use the hrtimer_is_running() inline from the [01/19] review
otherwise looks ok:
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
PATCH 3/19 hrtimers: Move and add documentation
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Move the initial hrtimer.txt document to the new directory "Documentation/hrtimer"
Add design notes for the high resolution timer and dynamic tick functionality.
Documentation/hrtimer/highres.txt new file +249 -0
@@ -0,0 +1,249 @@ +High resolution timers and dynamic ticks design notes +----------------------------------------------------- + +Further information can be found in the paper of the OLS 2006 talk "hrtimers +and beyond". The paper is part of the OLS 2006 Proceedings Volume 1, which can +be found on the OLS website: +http://www.linuxsymposium.org/2006/linuxsymposium_procv1.pdf + +The slides to this talk are available from: +http://tglx.de/projects/hrtimers/ols2006-hrtimers.pdf + +The slides contain five figures (pages 2, 15, 18, 20, 22), which illustrate the +changes in the time(r) related Linux subsystems. Figure #1 (p. 2) shows the +design of the Linux time(r) system before hrtimers and other building blocks +got merged into mainline. + +Note: the paper and the slides are talking about "clock event source", while we +switched to the name "clock event devices" in meantime. + +The design contains the following basic building blocks: + +- hrtimer base infrastructure +- timeofday and clock source management +- clock event management +- high resolution timer functionality +- dynamic ticks + + +hrtimer base infrastructure +--------------------------- + +The hrtimer base infrastructure was merged into the 2.6.16 kernel. Details of +the base implementation are covered in Documentation/hrtimer/hrtimer.txt. See +also figure #2 (OLS slides p. 15) + +The main differences to the timer wheel, which holds the armed timer_list type +timers are: + - time ordered enqueueing into a rb-tree + - independent of ticks (the processing is based on nanoseconds) + + +timeofday and clock source management +------------------------------------- + +John Stultz's Generic Time Of Day (GTOD) framework moves a large portion of +code out of the architecture-specific areas into a generic management +framework, as illustrated in figure #3 (OLS slides p. 18). The architecture +specific portion is reduced to the low level hardware details of the clock +sources, which are registered in the framework and selected on a quality based +decision. The low level code provides hardware setup and readout routines and +initializes data structures, which are used by the generic time keeping code to +convert the clock ticks to nanosecond based time values. All other time keeping +related functionality is moved into the generic code. The GTOD base patch got +merged into the 2.6.18 kernel. + +Further information about the Generic Time Of Day framework is available in the +OLS 2005 Proceedings Volume 1: +http://www.linuxsymposium.org/2005/linuxsymposium_procv1.pdf + +The paper "We Are Not Getting Any Younger: A New Approach to Time and +Timers" was written by J. Stultz, D.V. Hart, & N. Aravamudan. + +Figure #3 (OLS slides p.18) illustrates the transformation. + + +clock event management +---------------------- + +While clock sources provide read access to the monotonically increasing time +value, clock event devices are used to schedule the next event +interrupt(s). The next event is currently defined to be periodic, with its +period defined at compile time. The setup and selection of the event device +for various event driven functionalities is hardwired into the architecture +dependent code. This results in duplicated code across all architectures and +makes it extremely difficult to change the configuration of the system to use +event interrupt devices other than those already built into the +architecture. Another implication of the current design is that it is necessary +to touch all the architecture-specific implementations in order to provide new +functionality like high resolution timers or dynamic ticks. + +The clock events subsystem tries to address this problem by providing a generic +solution to manage clock event devices and their usage for the various clock +event driven kernel functionalities. The goal of the clock event subsystem is +to minimize the clock event related architecture dependent code to the pure +hardware related handling and to allow easy addition and utilization of new +clock event devices. It also minimizes the duplicated code across the +architectures as it provides generic functionality down to the interrupt +service handler, which is almost inherently hardware dependent. + +Clock event devices are registered either by the architecture dependent boot +code or at module insertion time. Each clock event device fills a data +structure with clock-specific property parameters and callback functions. The +clock event management decides, by using the specified property parameters, the +set of system functions a clock event device will be used to support. This +includes the distinction of per-CPU and per-system global event devices. + +System-level global event devices are used for the Linux periodic tick. Per-CPU +event devices are used to provide local CPU functionality such as process +accounting, profiling, and high resolution timers. + +The management layer assignes one or more of the folliwing functions to a clock +event device: + - system global periodic tick (jiffies update) + - cpu local update_process_times + - cpu local profiling + - cpu local next event interrupt (non periodic mode) + +The clock event device delegates the selection of those timer interrupt related +functions completely to the management layer. The clock management layer stores +a function pointer in the device description structure, which has to be called +from the hardware level handler. This removes a lot of duplicated code from the +architecture specific timer interrupt handlers and hands the control over the +clock event devices and the assignment of timer interrupt related functionality +to the core code. + +The clock event layer API is rather small. Aside from the clock event device +registration interface it provides functions to schedule the next event +interrupt, clock event device notification service and support for suspend and +resume. + +The framework adds about 700 lines of code which results in a 2KB increase of +the kernel binary size. The conversion of i386 removes about 100 lines of +code. The binary size decrease is in the range of 400 byte. We believe that the +increase of flexibility and the avoidance of duplicated code across +architectures justifies the slight increase of the binary size. + +The conversion of an architecture has no functional impact, but allows to +utilize the high resolution and dynamic tick functionalites without any change +to the clock event device and timer interrupt code. After the conversion the +enabling of high resolution timers and dynamic ticks is simply provided by +adding the kernel/time/Kconfig file to the architecture specific Kconfig and +adding the dynamic tick specific calls to the idle routine (a total of 3 lines +added to the idle function and the Kconfig file) + +Figure #4 (OLS slides p.20) illustrates the transformation. + + +high resolution timer functionality +----------------------------------- + +During system boot it is not possible to use the high resolution timer +functionality, while making it possible would be difficult and would serve no +useful function. The initialization of the clock event device framework, the +clock source framework (GTOD) and hrtimers itself has to be done and +appropriate clock sources and clock event devices have to be registered before +the high resolution functionality can work. Up to the point where hrtimers are +initialized, the system works in the usual low resolution periodic mode. The +clock source and the clock event device layers provide notification functions +which inform hrtimers about availability of new hardware. hrtimers validates +the usability of the registered clock sources and clock event devices before +switching to high resolution mode. This ensures also that a kernel which is +configured for high resolution timers can run on a system which lacks the +necessary hardware support. + +The high resolution timer code does not support SMP machines which have only +global clock event devices. The support of such hardware would involve IPI +calls when an interrupt happens. The overhead would be much larger than the +benefit. This is the reason why we currently disable high resolution and +dynamic ticks on i386 SMP systems which stop the local APIC in C3 power +state. A workaround is available as an idea, but the problem has not been +tackled yet. + +The time ordered insertion of timers provides all the infrastructure to decide +whether the event device has to be reprogrammed when a timer is added. The +decision is made per timer base and synchronized across per-cpu timer bases in +a support function. The design allows the system to utilize separate per-CPU +clock event devices for the per-CPU timer bases, but currently only one +reprogrammable clock event device per-CPU is utilized. + +When the timer interrupt happens, the next event interrupt handler is called +from the clock event distribution code and moves expired timers from the +red-black tree to a separate double linked list and invokes the softirq +handler. An additional mode field in the hrtimer structure allows the system to +execute callback functions directly from the next event interrupt handler. This +is restricted to code which can safely be executed in the hard interrupt +context. This applies, for example, to the common case of a wakeup function as +used by nanosleep. The advantage of executing the handler in the interrupt +context is the avoidance of up to two context switches - from the interrupted +context to the softirq and to the task which is woken up by the expired +timer. + +Once a system has switched to high resolution mode, the periodic tick is +switched off. This disables the per system global periodic clock event device - +e.g. the PIT on i386 SMP systems. + +The periodic tick functionality is provided by an per-cpu hrtimer. The callback +function is executed in the next event interrupt context and updates jiffies +and calls update_process_times and profiling. The implementation of the hrtimer +based periodic tick is designed to be extended with dynamic tick functionality. +This allows to use a single clock event device to schedule high resolution +timer and periodic events (jiffies tick, profiling, process accounting) on UP +systems. This has been proved to work with the PIT on i386 and the Incrementer +on PPC. + +The softirq for running the hrtimer queues and executing the callbacks has been +separated from the tick bound timer softirq to allow accurate delivery of high +resolution timer signals which are used by itimer and POSIX interval +timers. The execution of this softirq can still be delayed by other softirqs, +but the overall latencies have been significantly improved by this separation. + +Figure #5 (OLS slides p.22) illustrates the transformation. + + +dynamic ticks +------------- + +Dynamic ticks are the logical consequence of the hrtimer based periodic tick +replacement (sched_tick). The functionality of the sched_tick hrtimer is +extended by three functions: + +- hrtimer_stop_sched_tick +- hrtimer_restart_sched_tick +- hrtimer_update_jiffies + +hrtimer_stop_sched_tick() is called when a CPU goes into idle state. The code +evaluates the next scheduled timer event (from both hrtimers and the timer +wheel) and in case that the next event is further away than the next tick it +reprograms the sched_tick to this future event, to allow longer idle sleeps +without worthless interruption by the periodic tick. The function is also +called when an interrupt happens during the idle period, which does not cause a +reschedule. The call is necessary as the interrupt handler might have armed a +new timer whose expiry time is before the time which was identified as the +nearest event in the previous call to hrtimer_stop_sched_tick. + +hrtimer_restart_sched_tick() is called when the CPU leaves the idle state before +it calls schedule(). hrtimer_restart_sched_tick() resumes the periodic tick, +which is kept active until the next call to hrtimer_stop_sched_tick(). + +hrtimer_update_jiffies() is called from irq_enter() when an interrupt happens +in the idle period to make sure that jiffies are up to date and the interrupt +handler has not to deal with an eventually stale jiffy value. + +The dynamic tick feature provides statistical values which are exported to +userspace via /proc/stats and can be made available for enhanced power +management control. + +The implementation leaves room for further development like full tickless +systems, where the time slice is controlled by the scheduler, variable +frequency profiling, and a complete removal of jiffies in the future. + + +Aside the current initial submission of i386 support, the patchset has been +extended to x86_64 and ARM already. Initial (work in progress) support is also +available for MIPS and PowerPC. + + Thomas, Ingo + + +
Documentation/hrtimer/hrtimers.txt new file +178 -0
178 lines not on this page · raw message
Documentation/hrtimers.txt deleted +0 -178
178 lines not on this page · raw message
--
PATCH 4/19 Add a framework to manage clock event devices.
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
We have two types of clock event devices:
- global events (one device per system)
- local events (one device per cpu)
We assign the various time(r) related interrupts to those devices:
- global tick (advances jiffies)
- update process times (per cpu)
- profiling (per cpu)
- next timer events (per cpu)
Architectures register their clock event devices, with specific capability bits set, and the framework code assigns the appropriate event handler to the event device. The functionality is assigned via an event handler to avoid runtime evalutation of the assigned function bits.
This allows to control the clock event devices without the architectures having to worry about the details of function assignment. This is also a preliminary for high resolution timers and dynamic ticks to allow the core code to control the clock functionality without intrusive changes to the architecture code.
For x86 based systems the code provides the ability to broadcast timer events. This is necessary due to the fact, that the per CPU local APIC timers are stopped in power saving states.
When high resolution timers and dynamic ticks are disabled, there is no change in the behaviour of the system.
include/linux/clockchips.h new file +143 -0
@@ -0,0 +1,143 @@ +/* linux/include/linux/clockchips.h +* +* This file contains the structure definitions for clockchips. +* +* If you are not a clockchip, or the time of day code, you should +* not be including this file! +*/ +#ifndef _LINUX_CLOCKCHIPS_H +#define _LINUX_CLOCKCHIPS_H + +#ifdef CONFIG_GENERIC_CLOCKEVENTS + +#include <linux/clocksource.h> +#include <linux/interrupt.h> + +struct clock_event_device ; + +/* Clock event mode commands */ +enum clock_event_mode { +CLOCK_EVT_PERIODIC , +CLOCK_EVT_ONESHOT , +CLOCK_EVT_SHUTDOWN , +}; + +/* +* Clock event capability flags: +* +* CAP_TICK: The event source should be used for the periodic tick +* CAP_UPDATE: The event source handler should call update_process_times() +* CAP_PROFILE: The event source handler should call profile_tick() +* CAP_NEXTEVT: The event source can be reprogrammed in oneshot mode and is +* a per cpu event source. +* +* The capability flags are used to select the appropriate handler for an event +* source. On an i386 UP system the PIT can serve all of the functionalities, +* while on a SMP system the PIT is solely used for the periodic tick and the +* local APIC timers are used for UPDATE / PROFILE / NEXTEVT. To avoid the run +* time query of those flags, the clock events layer assigns the appropriate +* event handler function, which contains only the selected calls, to the +* event. +*/ +#define CLOCK_CAP_TICK 0x000001 +#define CLOCK_CAP_UPDATE 0x000002 +#define CLOCK_CAP_PROFILE 0x000004 +#ifdef CONFIG_HIGH_RES_TIMERS +# define CLOCK_CAP_NEXTEVT 0x000008 +#else +# define CLOCK_CAP_NEXTEVT 0x000000 +#endif + +#define CLOCK_BASE_CAPS_MASK (CLOCK_CAP_TICK | CLOCK_CAP_PROFILE | \ + CLOCK_CAP_UPDATE) +#define CLOCK_CAPS_MASK (CLOCK_BASE_CAPS_MASK | CLOCK_CAP_NEXTEVT) + +/** +* struct clock_event_device - clock event descriptor +* +* @name: ptr to clock event name +* @capabilities: capabilities of the event chip +* @max_delta_ns: maximum delta value in ns +* @min_delta_ns: minimum delta value in ns +* @mult: nanosecond to cycles multiplier +* @shift: nanoseconds to cycles divisor (power of two) +* @set_next_event: set next event +* @set_mode: set mode function +* @suspend: suspend function (optional) +* @resume: resume function (optional) +* @evthandler: Assigned by the framework to be called by the low +* level handler of the event source +*/ +struct clock_event_device { +const char *name; +unsigned int capabilities; +unsigned long max_delta_ns; +unsigned long min_delta_ns; +unsigned long mult; +int shift; +void (*set_next_event)(unsigned long evt, +struct clock_event_device *); +void (*set_mode)(enum clock_event_mode mode, +struct clock_event_device *); +void (*event_handler)(struct pt_regs *regs); +}; + +/* +* Calculate a multiplication factor for scaled math, which is used to convert +* nanoseconds based values to clock ticks: +* +* clock_ticks = (nanoseconds * factor) >> shift. +* +* div_sc is the rearranged equation to calculate a factor from a given clock +* ticks / nanoseconds ratio: +* +* factor = (clock_ticks << shift) / nanoseconds +*/ +static inline unsigned long div_sc (unsigned long ticks,unsigned long nsec, +int shift) +{ +uint64_t tmp = ((uint64_t )ticks) << shift; + +do_div (tmp, nsec); +return (unsigned long ) tmp; +} + +/* Clock event layer functions */ +extern int register_local_clockevent (struct clock_event_device *); +extern int register_global_clockevent (struct clock_event_device *); +extern unsigned long clockevent_delta2ns (unsigned long latch, +struct clock_event_device *evt); +extern void clockevents_init (void ); + +extern int clockevents_init_next_event (void ); +extern int clockevents_set_next_event (ktime_t expires,int force); +extern int clockevents_next_event_available (void ); +extern void clockevents_resume_events (void ); + +#ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST +extern void clockevents_set_broadcast (struct clock_event_device *evt, +int broadcast); +extern void clockevents_set_global_broadcast (struct clock_event_device *evt, +int broadcast); +extern int clockevents_register_broadcast (void (*fun)(cpumask_t *mask)); +#else +static inline void clockevents_set_broadcast (struct clock_event_device *evt, +int broadcast) +{ +} +#endif + +#else + +# define clockevents_init () do { } while(0) +# define clockevents_resume_events () do { } while(0) + +struct clock_event_device ; +static inline void clockevents_set_broadcast (struct clock_event_device *evt, +int broadcast) +{ +} + +#endif + +#endif
include/linux/hrtimer.h +3 -0
@@ -144,6 +144,9 @@ struct hrtimer_cpu_base {* is expired in the next softirq when the clock was advanced. */ #define clock_was_set () do { } while (0) +#define hrtimer_clock_notify () do { } while (0) +extern ktime_t ktime_get (void ); +extern ktime_t ktime_get_real (void );/* Exported timer functions: */
init/main.c +2 -0
@@ -37,6 +37,7 @@#include <linux/moduleparam.h> #include <linux/kallsyms.h> #include <linux/writeback.h> +#include <linux/clockchips.h> #include <linux/cpu.h> #include <linux/cpuset.h> #include <linux/efi.h>
@@ -532,6 +533,7 @@ asmlinkage void __init start_kernel(voidrcu_init ();init_IRQ ();pidhash_init (); +clockevents_init ();init_timers ();hrtimers_init ();softirq_init ();
kernel/hrtimer.c +4 -2
@@ -31,6 +31,7 @@* For licencing details see kernel-base/COPYING */ +#include <linux/clockchips.h> #include <linux/cpu.h> #include <linux/module.h> #include <linux/percpu.h>
@@ -46,7 +47,7 @@* * returns the time in ktime_t format */ -static ktime_t ktime_get (void ) +ktime_t ktime_get (void ) {struct timespec now;
@@ -60,7 +61,7 @@ static ktime_t ktime_get(void)* * returns the time in ktime_t format */ -static ktime_t ktime_get_real (void ) +ktime_t ktime_get_real (void ) {struct timespec now;
@@ -299,6 +300,7 @@ static unsigned long ktime_divns(const k*/ void hrtimer_notify_resume (void ) { +clockevents_resume_events ();clock_was_set (); }
kernel/time/Makefile +2 -0
@@ -1 +1,3 @@ obj-y += ntp.o clocksource.o jiffies.o + +obj-$(CONFIG_GENERIC_CLOCKEVENTS) += clockevents.o
kernel/time/clockevents.c new file +757 -0
757 lines not on this page · raw message
--
Re: [patch 04/19] Add a framework to manage clock event devices.
Quoted 7 lines · Thomas Gleixner
+ * struct
clock_event_device- clock event descriptor
+ *
+ * @name: ptr to clock event name
+ * @capabilities: capabilities of the event chip
+ * @max_delta_ns: maximum delta value in ns
+ * @min_delta_ns: minimum delta value in ns
+ * @mult: nanosecond to cycles multiplier
Quoted 8 lines
+ * @shift: nanoseconds to cycles divisor (power of two)
+ * @set_next_event: set next event
+ * @set_mode: set mode function
+ * @suspend: suspend function (optional)
+ * @resume: resume function (optional)
+ * @evthandler: Assigned by the framework to be called by the low
+ * level handler of the event source
+ */
it would be nice if the datastructure was "pure"; eg entirely owned by the source (and could be made const), however the only way I can see that done is by having a private duplicate datastructure in each clock driver... which is way overkill for one single function pointer ;(
well you could do
struct clock_event_device
{
const struct clock_ops *ops;
void *instance;
evthndlr_t *evthandler;
}that way if you have, say, 3 hpet channels you can use the same "ops" (or maybe "props") structure for all three, but still register the per channel state as well..
you maybe also want to have a "costs" member, so that you can pick the cheapest available timer..
include/linux/clockchips.h line 82
Quoted 12 lines
include/linux/clockchips.h new file +12
-0+struct clock_event_device { +const char *name; +unsigned int capabilities; +unsigned long max_delta_ns; +unsigned long min_delta_ns; +unsigned long mult; +int shift; +void (*set_next_event)(unsigned long evt, +struct clock_event_device *); +void (*set_mode)(enum clock_event_mode mode, +struct clock_event_device *); +void (*event_handler)(struct pt_regs *regs);
is pt_regs really really needed here? We got rid of it in most places (and made it a per tast struct thing), I wonder if it can be made to go away here too.
kernel/time/clockevents.c line 201
Quoted 8 lines
kernel/time/clockevents.c new file +8
-0+/* +* Start up an event device +*/ +static void startup_event (struct clock_event_device *evt,unsigned int caps) +{ +int mode; + +if (caps ==CLOCK_CAP_NEXTEVT )
isn't caps a bitfield ? if so, shouldn't this be a & ?
kernel/time/clockevents.c line 516
Quoted 11 lines
+ */
kernel/time/clockevents.c new file +10
-0+int clockevents_set_next_event (ktime_t expires,int force) +{ +struct local_events *devices = &__get_cpu_var (local_eventdevices); +struct clock_event_device *nextevt = devices->nextevt; +int64_t delta =ktime_to_ns (ktime_sub (expires,ktime_get ())); + +if (delta <=0 && !force) { + devices->expires_next.tv64 =KTIME_MAX ; +return -ETIME ; + }
hmmm so if I set a timer 10 nsec in the future, and then I get an interrupt, I suddenly get an infinite time timer? Sounds more like a case of "please just run it right away"
Quoted 4 lines
+ * Resume the cpu local clock events
+ */kernel/time/clockevents.c new file +2
-0+static void clockevents_resume_local_events (void *arg) +{
kernel/time/clockevents.c line 701
Quoted 10 lines
kernel/time/clockevents.c new file +10
-0+struct local_events *devices = &__get_cpu_var (local_eventdevices); +int i; + +for (i =0 ; i < devices->installed; i++) { +if (devices->events[i].real_caps) +startup_event (devices->events[i].event, + devices->events[i].real_caps); + } +touch_softlockup_watchdog (); +}
what is this watchdog touching for?
kernel/time/clockevents.c line 733
Quoted 6 lines
kernel/time/clockevents.c new file +6
-0+static int clockevents_cpu_notify (struct notifier_block *self, +unsigned long action,void *hcpu) +{ +switch (action) { +case CPU_UP_PREPARE : +break ;
don't you want to start the per cpu timer in such a case?
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Re: [patch 04/19] Add a framework to manage clock event devices.
In reply to Thomas GleixnerHi,
Quoted 18 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
We have two types of clock event devices:
- global events (one device per system)
- local events (one device per cpu)We assign the various time(r) related interrupts to those devices:
- global tick (advances jiffies)
- update process times (per cpu)
- profiling (per cpu)
- next timer events (per cpu)Architectures register their clock event devices, with specific capability bits set, and the framework code assigns the appropriate event handler to the event device. The functionality is assigned via an event handler to avoid runtime evalutation of the assigned function bits.
Quoted 5 lines
This allows to control the clock event devices without the architectures having to worry about the details of function assignment. This is also a preliminary for high resolution timers and dynamic ticks to allow the core code to control the clock functionality without intrusive changes to the architecture code.
I have a few problems with this code and I'd really prefer some more arch maintainers would look at this (i.e. post it to the arch ml). It's basically limited to only one global and one per cpu timer, are you sure this enough? Large systems may have several timer.
Even for cases I'm interested in I have no idea how to make use of it, e.g. I have a somewhat limited timer, which can't be reprogrammed without losing accuracy, but I have two (or maybe more) of them, so I can use one as general tick timer and its interrupt can be disabled as needed and a second timer can be used for dynamic timer events.
Something else I want to use separate timer is for kernel profiling, currently events started from the timer tick are basically invisible, so I'd like to start profiling on a different timer with a different frequency.
Currently high resolution timer are used for quite a lot once enabled, but I would like to see the option to limit them, i.e. use a low resolution timer for standard tasks (e.g. itimer, nanosleep) and provide a separate high resolution posix timer to user space.
This should give some idea of the background with which I'm looking at this code and I'm trying to find an answer to the question, how generic this really is and how usable this is beyond dynamic ticks.
Quoted 23 lines
include/linux/clockchips.h new file +23
-0+struct clock_event_device { +const char *name; +unsigned int capabilities; +unsigned long max_delta_ns; +unsigned long min_delta_ns; +unsigned long mult; +int shift; +void (*set_next_event)(unsigned long evt, +struct clock_event_device *); +void (*set_mode)(enum clock_event_mode mode, +struct clock_event_device *); +void (*event_handler)(struct pt_regs *regs); +}; + +/* +* Calculate a multiplication factor for scaled math, which is used to convert +* nanoseconds based values to clock ticks: +* +* clock_ticks = (nanoseconds * factor) >> shift. +* +* div_sc is the rearranged equation to calculate a factor from a given clock +* ticks / nanoseconds ratio: +*
include/linux/clockchips.h line 103
Quoted 10 lines
include/linux/clockchips.h new file +10
-0+* factor = (clock_ticks << shift) / nanoseconds +*/ +static inline unsigned long div_sc (unsigned long ticks,unsigned long nsec, +int shift) +{ +uint64_t tmp = ((uint64_t )ticks) << shift; + +do_div (tmp, nsec); +return (unsigned long ) tmp; +}
One possible problem in this area: the nsec2cycle multiplier is mostly constant AFAICT, where as the clock source cycle2nsec isn't (especially if controlled via ntp). This means this could produce slightly wrong results, the larger the longer the period is between timer interrupts.
Quoted 18 lines
kernel/time/clockevents.c new file +18
-0+ +#define MAX_CLOCK_EVENTS 4 +#define GLOBAL_CLOCK_EVENT MAX_CLOCK_EVENTS + +struct event_descr { +struct clock_event_device *event; +unsigned int mode; +unsigned int real_caps; +struct irqaction action; +}; + +struct local_events { +int installed; +struct event_descr events[MAX_CLOCK_EVENTS ]; +struct clock_event_device *nextevt; +ktime_t expires_next; +}; +
[...]
Quoted 89 lines
kernel/time/clockevents.c new file +89
-0+static void handle_tick (struct pt_regs *regs) +{ +write_seqlock (&xtime_lock); +do_timer (1 ); +write_sequnlock (&xtime_lock); +} + +/* +* Bootup and lowres handler: ticks and update_process_times +*/ +static void handle_tick_update (struct pt_regs *regs) +{ +write_seqlock (&xtime_lock); +do_timer (1 ); +write_sequnlock (&xtime_lock); + +update_process_times (user_mode (regs)); +} + +/* +* Bootup and lowres handler: ticks and profileing +*/ +static void handle_tick_profile (struct pt_regs *regs) +{ +write_seqlock (&xtime_lock); +do_timer (1 ); +write_sequnlock (&xtime_lock); + +profile_tick (CPU_PROFILING ); +} + +/* +* Bootup and lowres handler: ticks, update_process_times and profiling +*/ +static void handle_tick_update_profile (struct pt_regs *regs) +{ +write_seqlock (&xtime_lock); +do_timer (1 ); +write_sequnlock (&xtime_lock); + +update_process_times (user_mode (regs)); +profile_tick (CPU_PROFILING ); +} + +/* +* Bootup and lowres handler: update_process_times +*/ +static void handle_update (struct pt_regs *regs) +{ +update_process_times (user_mode (regs)); +} + +/* +* Bootup and lowres handler: update_process_times and profiling +*/ +static void handle_update_profile (struct pt_regs *regs) +{ +update_process_times (user_mode (regs)); +profile_tick (CPU_PROFILING ); +} + +/* +* Bootup and lowres handler: profiling +*/ +static void handle_profile (struct pt_regs *regs) +{ +profile_tick (CPU_PROFILING ); +} + +/* +* Noop handler when we shut down an event device +*/ +static void handle_noop (struct pt_regs *regs) +{ +} + +/* +* Lookup table for bootup and lowres event assignment +* +* The event handler is choosen by the capability flags of the clock event +* device. +*/ +static void __read_mostly *event_handlers[] = { + handle_noop,/* 0: No capability selected */ + handle_tick,/* 1: Tick only */ + handle_update,/* 2: Update process times */ + handle_tick_update,/* 3: Tick + update process times */ + handle_profile,/* 4: Profiling int */ + handle_tick_profile,/* 5: Tick + Profiling int */
kernel/time/clockevents.c line 193
Quoted 10 lines
kernel/time/clockevents.c new file +9
-0+ handle_update_profile,/* 6: Update process times + +profiling */ + handle_tick_update_profile,/* 7: Tick + update process times + +profiling */ +#ifdef CONFIG_HIGH_RES_TIMERS + hrtimer_interrupt, /* 8: Reprogrammable event device */ +#endif +}; +
What's the point with all these little helper functions? This looks rather inflexible to me. The generic code has to know about all timer events and even provide helper to handle all variation between them? There might be a small performance benefit in doing this, usually I'm all for it, but I don't think this one is really worth it. A simple list of active events would be much simpler, e.g. something like this is what I have in mind:
source->time = gettime();
source->timeout += MAX_TIMEOUT;
for_each_handler() {
if (source->time >= timer->timeout) {
...
source->timeout = min(source->timeout, timer->timeout);
}
}
reprogram_timer();The interrupt system already maintains a list of handler, so something like this should be reasonably possible without any extra overhead. AFAICT it could even simplify your hrtimer code by keeping realtime and monotonic hrtimer separate. Right now you have one loop for HRTIMER_MAX_CLOCK_BASES in the hrtimer code and another in the clock code only to calculate the next timeout.
The generic interrupt system already has a concept of per cpu interrupts, IMO it's worth it to check out how it can be used for clock events. IMO the core clock code should have almost no idea of per cpu clocks.
Basically I have these problems with this code (maybe there is a reason for all this, but I don't see it from the existing documentation):
- generic code shouldn't care about the number of used hw timer
- generic code shouldn't care about how these timers are used
- the various clients shouldn't know about each other (e.g. the sched_tick code in the hrtimer source is just ugly).
In this context I still don't understand why you insist on the strict separation of clock events and sources, these _are_ related and separating it like this prevents more flexible usage, e.g. to have multiple clock sources and timer (and in many cases they are even the same).
kernel/time/clockevents.c line 215
Quoted 7 lines
kernel/time/clockevents.c new file +7
-0+/* +* Setup an event device. Assign an handler and start it up +*/ +static void setup_event (struct event_descr *descr, +struct clock_event_device *evt,unsigned int caps) +{ +void *handler = event_handlers[caps];
These flags aren't really capabilities but intended usage. Every clock event driver has already its possible usage hardcoded (sometimes even with ugly ifdefs, e.g. lapic).
Capabilities would be something like:
- cpu local/global
- reprogrammable/fixed timer
Based on this the generic code should dynamically decide how to use and attach the clients.
bye, Roman
PATCH 5/19 ACPI: Include apic.h
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
apic.h does not get included on UP compiles.
That way the APICTIMER_STOPS_ON_C3 is not there and UP boxen have no support for timer broadcasting. This was never noticed, because the lapic timer is only used for profiling on UP.
drivers/acpi/processor_idle.c +10 -0
@@ -40,6 +40,16 @@#include <linux/sched.h> /* need_resched() */ #include <linux/latency.h> +/* +* Include the apic definitions for x86 to have the APIC timer related defines +* available also for UP (on SMP it gets magically included via linux/smp.h). +* asm/acpi.h is not an option, as it would require more include magic. Also +* creating an empty asm-ia64/apic.h would just trade pest vs. cholera. +*/ +#ifdef CONFIG_X86 +#include <asm/apic.h> +#endif +#include <asm/io.h> #include <asm/uaccess.h>
--
PATCH 6/19 ACPI: Keep track of timer broadcast
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
This is a preperatory patch for highres/dyntick:
- replace the big #ifdef ARCH_APICTIMER_STOPS_ON_C3 hackery by functions
- remove the double switch in the power verify function (in the worst case we switched ipi to apic and 20usec later apic to ipi)
- keep track of the the state which stops local APIC timer
drivers/acpi/processor_idle.c renamed from drivers/acpi/processor_idle.c~acpi-keep-track-of-timer-broadcast +48 -19
@@ -246,6 +246,49 @@ static void acpi_cstate_enter(struct acp } } +#ifdef ARCH_APICTIMER_STOPS_ON_C3 + +/* +* Some BIOS implementations switch to C3 in the published C2 state. This seems +* to be a common problem on AMD boxen. +*/ +static void acpi_timer_check_state (int state,struct acpi_processor *pr, +struct acpi_processor_cx *cx) +{ +struct acpi_processor_power *pwr = &pr->power; + +/* +* Check, if one of the previous states already marked the lapic +* unstable +*/ +if (pwr->timer_broadcast_on_state < state) +return ; + +if (cx->type ==ACPI_STATE_C3 || + boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) { + pr->power.timer_broadcast_on_state = state; +return ; + } +} + +static void acpi_propagate_timer_broadcast (struct acpi_processor *pr) +{ +cpumask_t mask =cpumask_of_cpu (pr->id); + +if (pr->power.timer_broadcast_on_state <INT_MAX ) +on_each_cpu (switch_APIC_timer_to_ipi, &mask,1 ,1 ); +else +on_each_cpu (switch_ipi_to_APIC_timer, &mask,1 ,1 ); +} + +#else + +static void acpi_timer_check_state (int state,struct acpi_processor *pr, +struct acpi_processor_cx *cstate) { } +static void acpi_propagate_timer_broadcast (struct acpi_processor *pr) { } + +#endif +static void acpi_processor_idle (void ) {struct acpi_processor *pr =NULL ;
@@ -912,11 +955,7 @@ static int acpi_processor_power_verify(sunsigned int i;unsigned int working =0 ; -#ifdef ARCH_APICTIMER_STOPS_ON_C3 -int timer_broadcast =0 ; -cpumask_t mask =cpumask_of_cpu (pr->id); -on_each_cpu (switch_ipi_to_APIC_timer, &mask,1 ,1 ); -#endif + pr->power.timer_broadcast_on_state =INT_MAX ;for (i =1 ; i <ACPI_PROCESSOR_MAX_POWER ; i++) {struct acpi_processor_cx *cx = &pr->power.states[i];
@@ -928,21 +967,14 @@ static int acpi_processor_power_verify(scase ACPI_STATE_C2 :acpi_processor_power_verify_c2 (cx); -#ifdef ARCH_APICTIMER_STOPS_ON_C3 -/* Some AMD systems fake C3 as C2, but still -have timer troubles */ -if (cx->valid && - boot_cpu_data.x86_vendor ==X86_VENDOR_AMD ) - timer_broadcast++; -#endif +if (cx->valid) +acpi_timer_check_state (i, pr, cx);break ;case ACPI_STATE_C3 :acpi_processor_power_verify_c3 (pr, cx); -#ifdef ARCH_APICTIMER_STOPS_ON_C3 if (cx->valid) - timer_broadcast++; -#endif +acpi_timer_check_state (i, pr, cx);break ; }
@@ -950,10 +982,7 @@ static int acpi_processor_power_verify(s working++; } -#ifdef ARCH_APICTIMER_STOPS_ON_C3 -if (timer_broadcast) -on_each_cpu (switch_APIC_timer_to_ipi, &mask,1 ,1 ); -#endif +acpi_propagate_timer_broadcast (pr);return (working); }
include/acpi/processor.h renamed from include/acpi/processor.h~acpi-keep-track-of-timer-broadcast +1 -0
@@ -79,6 +79,7 @@ struct acpi_processor_power {u32 bm_activity;int count;struct acpi_processor_cx states[ACPI_PROCESSOR_MAX_POWER ]; +int timer_broadcast_on_state; };/* Performance Management */
_
--
Re: [patch 06/19] ACPI: Keep track of timer broadcast
On Thu, 2006-11-09 at 23:38 +0000, Thomas Gleixner wrote:
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
PATCH 7/19 ACPI: Add state propagation for dynamic broadcasting
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
This is a preparatory patch for high resolution timers and dynticks.
The local APIC timer is fast and per CPU, but it gets stopped in C3 state. On some broken systems, especially AMD based ones it gets stopped in C2. This also affects akpm's jinxed VAIO.
The broadcast function informs the local APIC management code that a state which stops the local APIC is going to be entered/exited. This switches the local APIC timer to the PIT broadcast mode. The clockevents layer takes care of the distribution of events.
The lapic_timer_idle_broadcast() function is an empty inline for now, which will be replaced by the later clockevents patches for the affected architectures.
drivers/acpi/processor_idle.c +20 -0
@@ -281,11 +281,27 @@ static void acpi_propagate_timer_broadca on_each_cpu(switch_ipi_to_APIC_timer, &mask, 1, 1); } +/* Power(C) State timer broadcast control */ +static void acpi_state_timer_broadcast (struct acpi_processor *pr, +struct acpi_processor_cx *cx, +int broadcast) +{ +int state = cx - pr->power.states; + +if (state >= pr->power.timer_broadcast_on_state) +lapic_timer_idle_broadcast (broadcast); +} +#else static void acpi_timer_check_state (int state,struct acpi_processor *pr,struct acpi_processor_cx *cstate) { }static void acpi_propagate_timer_broadcast (struct acpi_processor *pr) { } +static void acpi_state_timer_broadcast (struct acpi_processor *pr, +struct acpi_processor_cx *cx, +int broadcast) +{ +}#endif
@@ -431,6 +447,7 @@ static void acpi_processor_idle(void)/* Get start time (ticks) */ t1 =inl (acpi_fadt.xpm_tmr_blk.address);/* Invoke C2 */ +acpi_state_timer_broadcast (pr, cx,1 );acpi_cstate_enter (cx);/* Get end time (ticks) */ t2 =inl (acpi_fadt.xpm_tmr_blk.address);
@@ -445,6 +462,7 @@ static void acpi_processor_idle(void)/* Compute time (ticks) that we were actually asleep */ sleep_ticks =ticks_elapsed (t1, t2) - cx->latency_ticks -C2_OVERHEAD ; +acpi_state_timer_broadcast (pr, cx,0 );break ;case ACPI_STATE_C3 :
@@ -467,6 +485,7 @@ static void acpi_processor_idle(void)/* Get start time (ticks) */ t1 =inl (acpi_fadt.xpm_tmr_blk.address);/* Invoke C3 */ +acpi_state_timer_broadcast (pr, cx,1 );acpi_cstate_enter (cx);/* Get end time (ticks) */ t2 =inl (acpi_fadt.xpm_tmr_blk.address);
@@ -487,6 +506,7 @@ static void acpi_processor_idle(void)/* Compute time (ticks) that we were actually asleep */ sleep_ticks =ticks_elapsed (t1, t2) - cx->latency_ticks -C3_OVERHEAD ; +acpi_state_timer_broadcast (pr, cx,0 );break ;default :
include/asm-i386/apic.h +1 -0
@@ -113,6 +113,7 @@ extern void setup_secondary_APIC_clock (extern int APIC_init_uniprocessor (void );extern void disable_APIC_timer (void );extern void enable_APIC_timer (void ); +static inline void lapic_timer_idle_broadcast (int broadcast) { }extern void enable_NMI_through_LVT0 (void * dummy);
include/asm-x86_64/apic.h +1 -0
@@ -84,6 +84,7 @@ extern int APIC_init_uniprocessor (void)extern void disable_APIC_timer (void );extern void enable_APIC_timer (void );extern void clustered_apic_check (void ); +static inline void lapic_timer_idle_broadcast (int broadcast) { }extern void setup_APIC_extened_lvt (unsigned char lvt_off,unsigned char vector,unsigned char msg_type,unsigned char mask);
--
Re: [patch 07/19] ACPI: Add state propagation for dynamic broadcasting
Review
Quoted 5 lines · Thomas Gleixner
plain text document attachment
(acpi-add-hres-dyntick-broadcast-support.patch)(quoted diff) +2
-0+ lapic_timer_idle_broadcast(broadcast); +}
is this really lapic specific?
anyway Acked-by: Arjan van de Ven <[email protected]>
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
PATCH 8/19 i386: cleanup apic code
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
The apic code is quite unstructured and missing a lot of comments.
- Restructure the code into helper functions, timer, setup/shutdown, interrupt and power management blocks.
- Fixup comments.
- Namespace fixups
- Inline helpers for version and is_integrated
- Combine the ack_bad_irq functions
No functional changes.
arch/i386/kernel/apic.c +1064 -1029
@@ -44,6 +44,13 @@#include "io_ports.h" /* +* Sanity check +*/ +#if (SPURIOUS_APIC_VECTOR &0x0F ) !=0x0F +# error SPURIOUS_APIC_VECTOR definition error +#endif + +/* * cpu_mask that denotes the CPUs that needs timer interrupt coming in as * IPIs in place of local APIC timers */
@@ -51,561 +58,559 @@ static cpumask_t timer_bcast_ipi;/* * Knob to control our willingness to enable the local APIC. +* +* -1=force-disable, +1=force-enable */ -static int enable_local_apic__initdata =0 ;/* -1=force-disable, +1=force-enable */ - -static inline void lapic_disable (void ) -{ - enable_local_apic =-1 ; -clear_bit (X86_FEATURE_APIC , boot_cpu_data.x86_capability); -} - -static inline void lapic_enable (void ) -{ - enable_local_apic =1 ; -} +static int enable_local_apic__initdata =0 ;/* -* Debug level +* Debug level, exported for io_apic.c */ int apic_verbosity; -static void apic_pm_activate (void ); -static int modern_apic (void ) + +/* Using APIC to generate smp_local_timer_interrupt? */ +int using_apic_timer__read_mostly =0 ; + +/* Local APIC was disabled by the BIOS and enabled by the kernel */ +static int enabled_via_apicbase; + +/* +* Get the LAPIC version +*/ +static inline int lapic_get_version (void ) { -unsigned int lvr, version; -/* AMD systems use old APIC versions, so check the CPU */ -if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD && - boot_cpu_data.x86 >=0xf ) -return 1 ; - lvr =apic_read (APIC_LVR ); - version =GET_APIC_VERSION (lvr); -return version >=0x14 ; +return GET_APIC_VERSION (apic_read (APIC_LVR )); }/* -* 'what should we do if we get a hw irq event on an illegal vector'. -* each architecture has to answer this themselves. +* Check, if the APIC is integrated or a seperate chip */ -void ack_bad_irq (unsigned int irq) +static inline int lapic_is_integrated (void ) { -printk ("unexpected IRQ trap at vector %02x\n" , irq); -/* -* Currently unexpected vectors happen only on SMP and APIC. -* We _must_ ack these because every local APIC has only N -* irq slots per priority level, and a 'hanging, unacked' IRQ -* holds up an irq slot - in excessive cases (when multiple -* unexpected vectors occur) that might lock up the APIC -* completely. -* But only ack when the APIC is enabled -AK -*/ -if (cpu_has_apic) -ack_APIC_irq (); +return APIC_INTEGRATED (lapic_get_version ()); } -void __init apic_intr_init (void ) +/* +* Check, whether this is a modern or a first generation APIC +*/ +static int modern_apic (void ) { -#ifdef CONFIG_SMP -smp_intr_init (); -#endif -/* self generated IPI for local APIC timer */ -set_intr_gate (LOCAL_TIMER_VECTOR , apic_timer_interrupt); - -/* IPI vectors for APIC spurious and error interrupts */ -set_intr_gate (SPURIOUS_APIC_VECTOR , spurious_interrupt); -set_intr_gate (ERROR_APIC_VECTOR , error_interrupt); - -/* thermal monitor LVT interrupt */ -#ifdef CONFIG_X86_MCE_P4THERMAL -set_intr_gate (THERMAL_APIC_VECTOR , thermal_interrupt); -#endif +/* AMD systems use old APIC versions, so check the CPU */ +if (boot_cpu_data.x86_vendor ==X86_VENDOR_AMD && + boot_cpu_data.x86 >=0xf ) +return 1 ; +return lapic_get_version () >=0x14 ; } -/* Using APIC to generate smp_local_timer_interrupt? */ -int using_apic_timer__read_mostly =0 ; - -static int enabled_via_apicbase; - +/** +* enable_NMI_through_LVT0 - enable NMI through local vector table 0 +*/ void enable_NMI_through_LVT0 (void * dummy) { -unsigned int v, ver; +unsigned int v =APIC_DM_NMI ; - ver =apic_read (APIC_LVR ); - ver =GET_APIC_VERSION (ver); - v =APIC_DM_NMI ;/* unmask and set to NMI */ -if (!APIC_INTEGRATED (ver))/* 82489DX */ +/* Level triggered for 82489DX */ +if (!lapic_is_integrated ()) v |=APIC_LVT_LEVEL_TRIGGER ;apic_write_around (APIC_LVT0 , v); } +/** +* get_physical_broadcast - Get number of physical broadcast IDs +*/ int get_physical_broadcast (void ) { -if (modern_apic ()) -return 0xff ; -else -return 0xf ; +return modern_apic () ?0xff :0xf ; } -int get_maxlvt (void ) +/** +* lapic_get_maxlvt - get the maximum number of local vector table entries +*/ +int lapic_get_maxlvt (void ) { -unsigned int v, ver, maxlvt; +unsigned int v =apic_read (APIC_LVR ); - v =apic_read (APIC_LVR ); - ver =GET_APIC_VERSION (v);/* 82489DXs do not report # of LVT entries. */ - maxlvt =APIC_INTEGRATED (ver) ?GET_APIC_MAXLVT (v) :2 ; -return maxlvt; +return APIC_INTEGRATED (GET_APIC_VERSION (v)) ?GET_APIC_MAXLVT (v) :2 ; } -void clear_local_APIC (void ) +/* +* Local APIC timer +*/ + +/* +* This part sets up the APIC 32 bit clock in LVTT1, with HZ interrupts +* per second. We assume that the caller has already set up the local +* APIC. +* +* The APIC timer is not exactly sync with the external timer chip, it +* closely follows bus clocks. +*/ + +/* +* The timer chip is already set up at HZ interrupts per second here, +* but we do not accept timer interrupts yet. We only allow the BP +* to calibrate. +*/ +static unsigned int __devinit get_8254_timer_count (void ) { -int maxlvt; -unsigned long v; +unsigned long flags; - maxlvt =get_maxlvt (); +unsigned int count; -/* -* Masking an LVT entry can trigger a local APIC error -* if the vector is zero. Mask LVTERR first to prevent this. -*/ -if (maxlvt >=3 ) { - v =ERROR_APIC_VECTOR ;/* any non-zero vector will do */ -apic_write_around (APIC_LVTERR , v |APIC_LVT_MASKED ); - } -/* -* Careful: we have to set masks only first to deassert -* any level-triggered sources. -*/ - v =apic_read (APIC_LVTT ); -apic_write_around (APIC_LVTT , v |APIC_LVT_MASKED ); - v =apic_read (APIC_LVT0 ); -apic_write_around (APIC_LVT0 , v |APIC_LVT_MASKED ); - v =apic_read (APIC_LVT1 ); -apic_write_around (APIC_LVT1 , v |APIC_LVT_MASKED ); -if (maxlvt >=4 ) { - v =apic_read (APIC_LVTPC ); -apic_write_around (APIC_LVTPC , v |APIC_LVT_MASKED ); - } +spin_lock_irqsave (&i8253_lock, flags); -/* lets not touch this if we didn't frob it */ -#ifdef CONFIG_X86_MCE_P4THERMAL -if (maxlvt >=5 ) { - v =apic_read (APIC_LVTTHMR ); -apic_write_around (APIC_LVTTHMR , v |APIC_LVT_MASKED ); - } -#endif -/* -* Clean APIC state for other OSs: -*/ -apic_write_around (APIC_LVTT ,APIC_LVT_MASKED ); -apic_write_around (APIC_LVT0 ,APIC_LVT_MASKED ); -apic_write_around (APIC_LVT1 ,APIC_LVT_MASKED ); -if (maxlvt >=3 ) -apic_write_around (APIC_LVTERR ,APIC_LVT_MASKED ); -if (maxlvt >=4 ) -apic_write_around (APIC_LVTPC ,APIC_LVT_MASKED ); +outb_p (0x00 ,PIT_MODE ); + count =inb_p (PIT_CH0 ); + count |=inb_p (PIT_CH0 ) <<8 ; -#ifdef CONFIG_X86_MCE_P4THERMAL -if (maxlvt >=5 ) -apic_write_around (APIC_LVTTHMR ,APIC_LVT_MASKED ); -#endif - v =GET_APIC_VERSION (apic_read (APIC_LVR )); -if (APIC_INTEGRATED (v)) {/* !82489DX */ -if (maxlvt >3 )/* Due to Pentium errata 3AP and 11AP. */ -apic_write (APIC_ESR ,0 ); -apic_read (APIC_ESR ); - } -} +spin_unlock_irqrestore (&i8253_lock, flags); -void __init connect_bsp_APIC (void ) -{ -if (pic_mode) { -/* -* Do not trust the local APIC being empty at bootup. -*/ -clear_local_APIC (); -/* -* PIC mode, enable APIC mode in the IMCR, i.e. -* connect BSP's local APIC to INT and NMI lines. -*/ -apic_printk (APIC_VERBOSE ,"leaving PIC mode, " -"enabling APIC mode.\n" ); -outb (0x70 ,0x22 ); -outb (0x01 ,0x23 ); - } -enable_apic_mode (); +return count; } -void disconnect_bsp_APIC (int virt_wire_setup) +/* next tick in 8254 can be caught by catching timer wraparound */ +static void __devinit wait_8254_wraparound (void ) { -if (pic_mode) { -/* -* Put the board back into PIC mode (has an effect -* only on certain older boards). Note that APIC -* interrupts, including IPIs, won't work beyond -* this point! The only exception are INIT IPIs. -*/ -apic_printk (APIC_VERBOSE ,"disabling APIC mode, " -"entering PIC mode.\n" ); -outb (0x70 ,0x22 ); -outb (0x00 ,0x23 ); - } -else { -/* Go back to Virtual Wire compatibility mode */ -unsigned long value; +unsigned int curr_count, prev_count; -/* For the spurious interrupt use vector F, and enable it */ - value =apic_read (APIC_SPIV ); - value &= ~APIC_VECTOR_MASK ; - value |=APIC_SPIV_APIC_ENABLED ; - value |=0xf ; -apic_write_around (APIC_SPIV , value); + curr_count =get_8254_timer_count (); +do { + prev_count = curr_count; + curr_count =get_8254_timer_count (); -if (!virt_wire_setup) { -/* For LVT0 make it edge triggered, active high, external and enabled */ - value =apic_read (APIC_LVT0 ); - value &= ~(APIC_MODE_MASK |APIC_SEND_PENDING | -APIC_INPUT_POLARITY |APIC_LVT_REMOTE_IRR | -APIC_LVT_LEVEL_TRIGGER |APIC_LVT_MASKED ); - value |=APIC_LVT_REMOTE_IRR |APIC_SEND_PENDING ; - value =SET_APIC_DELIVERY_MODE (value,APIC_MODE_EXTINT ); -apic_write_around (APIC_LVT0 , value); - } -else { -/* Disable LVT0 */ -apic_write_around (APIC_LVT0 ,APIC_LVT_MASKED ); - } +/* workaround for broken Mercury/Neptune */ +if (prev_count >= curr_count +0x100 ) + curr_count =get_8254_timer_count (); -/* For LVT1 make it edge triggered, active high, nmi and enabled */ - value =apic_read (APIC_LVT1 ); - value &= ~( -APIC_MODE_MASK |APIC_SEND_PENDING | -APIC_INPUT_POLARITY |APIC_LVT_REMOTE_IRR | -APIC_LVT_LEVEL_TRIGGER |APIC_LVT_MASKED ); - value |=APIC_LVT_REMOTE_IRR |APIC_SEND_PENDING ; - value =SET_APIC_DELIVERY_MODE (value,APIC_MODE_NMI ); -apic_write_around (APIC_LVT1 , value); - } + }while (prev_count >= curr_count); } -void disable_local_APIC (void ) +/* +* Default initialization for 8254 timers. If we use other timers like HPET, +* we override this later +*/ +void (*wait_timer_tick)(void )__devinitdata = wait_8254_wraparound; + +/* +* This function sets up the local APIC timer, with a timeout of +* 'clocks' APIC bus clock. During calibration we actually call +* this function twice on the boot CPU, once with a bogus timeout +* value, second time for real. The other (noncalibrating) CPUs +* call this function only once, with the real, calibrated value. +* +* We do reads before writes even if unnecessary, to get around the +* P5 APIC double write bug. +*/ + +#define APIC_DIVISOR 16 + +static void __setup_APIC_LVTT (unsigned int clocks) { -unsigned long value; +unsigned int lvtt_value, tmp_value; +int cpu =smp_processor_id (); -clear_local_APIC (); + lvtt_value =APIC_LVT_TIMER_PERIODIC |LOCAL_TIMER_VECTOR ; +if (!lapic_is_integrated ()) + lvtt_value |=SET_APIC_TIMER_BASE (APIC_TIMER_BASE_DIV ); + +if (cpu_isset (cpu, timer_bcast_ipi)) + lvtt_value |=APIC_LVT_MASKED ; + +apic_write_around (APIC_LVTT , lvtt_value);/* -* Disable APIC (implies clearing of registers -* for 82489DX!). +* Divide PICLK by 16 */ - value =apic_read (APIC_SPIV ); - value &= ~APIC_SPIV_APIC_ENABLED ; -apic_write_around (APIC_SPIV , value); + tmp_value =apic_read (APIC_TDCR ); +apic_write_around (APIC_TDCR , (tmp_value + & ~(APIC_TDR_DIV_1 |APIC_TDR_DIV_TMBASE )) + |APIC_TDR_DIV_16 ); -if (enabled_via_apicbase) { -unsigned int l, h; -rdmsr (MSR_IA32_APICBASE , l, h); - l &= ~MSR_IA32_APICBASE_ENABLE ; -wrmsr (MSR_IA32_APICBASE , l, h); - } +apic_write_around (APIC_TMICT , clocks/APIC_DIVISOR ); } -/* -* This is to verify that we're looking at a real local APIC. -* Check these against your board if the CPUs aren't getting -* started for no apparent reason. -*/ -int __init verify_local_APIC (void ) +static void __devinit setup_APIC_timer (unsigned int clocks) { -unsigned int reg0, reg1; +unsigned long flags; + +local_irq_save (flags);/* -* The version register is read-only in a real APIC. +* Wait for IRQ0's slice: */ - reg0 =apic_read (APIC_LVR ); -apic_printk (APIC_DEBUG ,"Getting VERSION: %x\n" , reg0); -apic_write (APIC_LVR , reg0 ^APIC_LVR_MASK ); - reg1 =apic_read (APIC_LVR ); -apic_printk (APIC_DEBUG ,"Getting VERSION: %x\n" , reg1); +wait_timer_tick (); + +__setup_APIC_LVTT (clocks); + +local_irq_restore (flags); +} + +/* +* In this function we calibrate APIC bus clocks to the external +* timer. Unfortunately we cannot use jiffies and the timer irq +* to calibrate, since some later bootup code depends on getting +* the first irq? Ugh. +* +* We want to do the calibration only once since we +* want to have local timer irqs syncron. CPUs connected +* by the same APIC bus have the very same bus frequency. +* And we want to have irqs off anyways, no accidental +* APIC irq that way. +*/ + +static int __init calibrate_APIC_clock (void ) +{ +unsigned long long t1 =0 , t2 =0 ; +long tt1, tt2; +long result; +int i; +const int LOOPS =HZ /10 ; + +apic_printk (APIC_VERBOSE ,"calibrating APIC timer ...\n" );/* -* The two version reads above should print the same -* numbers. If the second one is different, then we -* poke at a non-APIC. +* Put whatever arbitrary (but long enough) timeout +* value into the APIC clock, we just want to get the +* counter running for calibration. */ -if (reg1 != reg0) -return 0 ; +__setup_APIC_LVTT (1000000000 );/* -* Check if the version looks reasonably. +* The timer chip counts down to zero. Let's wait +* for a wraparound to start exact measurement: +* (the current tick might have been already half done) */ - reg1 =GET_APIC_VERSION (reg0); -if (reg1 ==0x00 || reg1 ==0xff ) -return 0 ; - reg1 =get_maxlvt (); -if (reg1 <0x02 || reg1 ==0xff ) -return 0 ; + +wait_timer_tick ();/* -* The ID register is read/write in a real APIC. +* We wrapped around just now. Let's start: */ - reg0 =apic_read (APIC_ID ); -apic_printk (APIC_DEBUG ,"Getting ID: %x\n" , reg0); +if (cpu_has_tsc) +rdtscll (t1); + tt1 =apic_read (APIC_TMCCT );/* -* The next two are just to see if we have sane values. -* They're only really relevant if we're in Virtual Wire -* compatibility mode, but most boxes are anymore. +* Let's wait LOOPS wraprounds: */ - reg0 =apic_read (APIC_LVT0 ); -apic_printk (APIC_DEBUG ,"Getting LVT0: %x\n" , reg0); - reg1 =apic_read (APIC_LVT1 ); -apic_printk (APIC_DEBUG ,"Getting LVT1: %x\n" , reg1); +for (i =0 ; i <LOOPS ; i++) +wait_timer_tick (); -return 1 ; -} + tt2 =apic_read (APIC_TMCCT ); +if (cpu_has_tsc) +rdtscll (t2); -void __init sync_Arb_IDs (void ) -{ -/* Unsupported on P4 - see Intel Dev. Manual Vol. 3, Ch. 8.6.1 -And not needed on AMD */ -if (modern_apic ()) -return ;/* -* Wait for idle. +* The APIC bus clock counter is 32 bits only, it +* might have overflown, but note that we use signed +* longs, thus no extra care needed. +* +* underflown to be exact, as the timer counts down ;) */ -apic_wait_icr_idle (); -apic_printk (APIC_DEBUG ,"Synchronizing Arb IDs.\n" ); -apic_write_around (APIC_ICR ,APIC_DEST_ALLINC |APIC_INT_LEVELTRIG - |APIC_DM_INIT ); + result = (tt1-tt2)*APIC_DIVISOR /LOOPS ; + +if (cpu_has_tsc) +apic_printk (APIC_VERBOSE ,"..... CPU clock speed is " +"%ld.%04ld MHz.\n" , + ((long )(t2-t1)/LOOPS )/(1000000 /HZ ), + ((long )(t2-t1)/LOOPS )%(1000000 /HZ )); + +apic_printk (APIC_VERBOSE ,"..... host bus clock speed is " +"%ld.%04ld MHz.\n" , + result/(1000000 /HZ ), + result%(1000000 /HZ )); + +return result; } -extern void __error_in_apic_c (void ); +static unsigned int calibration_result; -/* -* An initial setup of the virtual wire mode. -*/ -void __init init_bsp_APIC (void ) +void __init setup_boot_APIC_clock (void ) { -unsigned long value, ver; +unsigned long flags; +apic_printk (APIC_VERBOSE ,"Using local APIC timer interrupts.\n" ); + using_apic_timer =1 ; + +local_irq_save (flags); + calibration_result =calibrate_APIC_clock ();/* -* Don't do the setup now if we have a SMP BIOS as the -* through-I/O-APIC virtual wire mode might be active. +* Now set up the timer for real. */ -if (smp_found_config || !cpu_has_apic) -return ; +setup_APIC_timer (calibration_result); + +local_irq_restore (flags); +} - value =apic_read (APIC_LVR ); - ver =GET_APIC_VERSION (value); +void __devinit setup_secondary_APIC_clock (void ) +{ +setup_APIC_timer (calibration_result); +} -/* -* Do not trust the local APIC being empty at bootup. -*/ -clear_local_APIC (); +void disable_APIC_timer (void ) +{ +if (using_apic_timer) { +unsigned long v; -/* -* Enable APIC. -*/ - value =apic_read (APIC_SPIV ); - value &= ~APIC_VECTOR_MASK ; - value |=APIC_SPIV_APIC_ENABLED ; - -/* This bit is reserved on P4/Xeon and should be cleared */ -if ((boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ) && (boot_cpu_data.x86 ==15 )) - value &= ~APIC_SPIV_FOCUS_DISABLED ; -else - value |=APIC_SPIV_FOCUS_DISABLED ; - value |=SPURIOUS_APIC_VECTOR ; -apic_write_around (APIC_SPIV , value); + v =apic_read (APIC_LVTT ); +/* +* When an illegal vector value (0-15) is written to an LVT +* entry and delivery mode is Fixed, the APIC may signal an +* illegal vector error, with out regard to whether the mask +* bit is set or whether an interrupt is actually seen on +* input. +* +* Boot sequence might call this function when the LVTT has +* '0' vector value. So make sure vector field is set to +* valid value. +*/ + v |= (APIC_LVT_MASKED |LOCAL_TIMER_VECTOR ); +apic_write_around (APIC_LVTT , v); + } +} -/* -* Set up the virtual wire mode. -*/ -apic_write_around (APIC_LVT0 ,APIC_DM_EXTINT ); - value =APIC_DM_NMI ; -if (!APIC_INTEGRATED (ver))/* 82489DX */ - value |=APIC_LVT_LEVEL_TRIGGER ; -apic_write_around (APIC_LVT1 , value); +void enable_APIC_timer (void ) +{ +int cpu =smp_processor_id (); + +if (using_apic_timer && !cpu_isset (cpu, timer_bcast_ipi)) { +unsigned long v; + + v =apic_read (APIC_LVTT ); +apic_write_around (APIC_LVTT , v & ~APIC_LVT_MASKED ); + } } -void __devinit setup_local_APIC (void ) +void switch_APIC_timer_to_ipi (void *cpumask) { -unsigned long oldvalue, value, ver, maxlvt; -int i, j; +cpumask_t mask = *(cpumask_t *)cpumask; +int cpu =smp_processor_id (); -/* Pound the ESR really hard over the head with a big hammer - mbligh */ -if (esr_disable) { -apic_write (APIC_ESR ,0 ); -apic_write (APIC_ESR ,0 ); -apic_write (APIC_ESR ,0 ); -apic_write (APIC_ESR ,0 ); +if (cpu_isset (cpu, mask) && + !cpu_isset (cpu, timer_bcast_ipi)) { +disable_APIC_timer (); +cpu_set (cpu, timer_bcast_ipi); } +} +EXPORT_SYMBOL (switch_APIC_timer_to_ipi); - value =apic_read (APIC_LVR ); - ver =GET_APIC_VERSION (value); +void switch_ipi_to_APIC_timer (void *cpumask) +{ +cpumask_t mask = *(cpumask_t *)cpumask; +int cpu =smp_processor_id (); + +if (cpu_isset (cpu, mask) && +cpu_isset (cpu, timer_bcast_ipi)) { +cpu_clear (cpu, timer_bcast_ipi); +enable_APIC_timer (); + } +} +EXPORT_SYMBOL (switch_ipi_to_APIC_timer); -if ((SPURIOUS_APIC_VECTOR &0x0f ) !=0x0f ) -__error_in_apic_c (); +/* +* Local timer interrupt handler. It does both profiling and +* process statistics/rescheduling. +*/ +inline void smp_local_timer_interrupt (void ) +{ +profile_tick (CPU_PROFILING ); +#ifdef CONFIG_SMP +update_process_times (user_mode_vm (get_irq_regs ())); +#endif /* -* Double-check whether this APIC is really registered. +* We take the 'long' return path, and there every subsystem +* grabs the apropriate locks (kernel lock/ irq lock). +* +* we might want to decouple profiling from the 'long path', +* and do the profiling totally in assembly. +* +* Currently this isn't too much of an issue (performance wise), +* we can take more than 100K local irqs per second on a 100 MHz P5. */ -if (!apic_id_registered ()) -BUG (); +} + +/* +* Local APIC timer interrupt. This is the most natural way for doing +* local interrupts, but local timer interrupts can be emulated by +* broadcast interrupts too. [in case the hw doesn't support APIC timers] +* +* [ if a single-CPU system runs an SMP kernel then we call the local +* interrupt as well. Thus we cannot inline the local irq ... ] +*/ + +fastcall void smp_apic_timer_interrupt(struct pt_regs *regs) +{ +struct pt_regs *old_regs =set_irq_regs (regs); +int cpu =smp_processor_id ();/* -* Intel recommends to set DFR, LDR and TPR before enabling -* an APIC. See e.g. "AP-388 82489DX User's Manual" (Intel -* document number 292116). So here it goes... +* the NMI deadlock-detector uses this. */ -init_apic_ldr (); +per_cpu (irq_stat, cpu).apic_timer_irqs++;/* -* Set Task Priority to 'accept all'. We never change this -* later on. +* NOTE! We'd better ACK the irq immediately, +* because timer handling can be slow. */ - value =apic_read (APIC_TASKPRI ); - value &= ~APIC_TPRI_MASK ; -apic_write_around (APIC_TASKPRI , value); - +ack_APIC_irq ();/* -* After a crash, we no longer service the interrupts and a pending -* interrupt from previous kernel might still have ISR bit set. -* -* Most probably by now CPU has serviced that pending interrupt and -* it might not have done the ack_APIC_irq() because it thought, -* interrupt came from i8259 as ExtInt. LAPIC did not get EOI so it -* does not clear the ISR bit and cpu thinks it has already serivced -* the interrupt. Hence a vector might get locked. It was noticed -* for timer irq (vector 0x31). Issue an extra EOI to clear ISR. +* update_process_times() expects us to have done irq_enter(). +* Besides, if we don't timer interrupts ignore the global +* interrupt lock, which is the WrongThing (tm) to do. */ -for (i =APIC_ISR_NR -1 ; i >=0 ; i--) { - value =apic_read (APIC_ISR + i*0x10 ); -for (j =31 ; j >=0 ; j--) { -if (value & (1 <<j)) -ack_APIC_irq (); - } - } +irq_enter (); +smp_local_timer_interrupt (); +irq_exit (); +set_irq_regs (old_regs); +} -/* -* Now that we are all set up, enable the APIC -*/ - value =apic_read (APIC_SPIV ); - value &= ~APIC_VECTOR_MASK ; -/* -* Enable APIC -*/ - value |=APIC_SPIV_APIC_ENABLED ; +#ifndef CONFIG_SMP +static void up_apic_timer_interrupt_call (void ) +{ +int cpu =smp_processor_id ();/* -* Some unknown Intel IO/APIC (or APIC) errata is biting us with -* certain networking cards. If high frequency interrupts are -* happening on a particular IOAPIC pin, plus the IOAPIC routing -* entry is masked/unmasked at a high rate as well then sooner or -* later IOAPIC line gets 'stuck', no more interrupts are received -* from the device. If focus CPU is disabled then the hang goes -* away, oh well :-( -* -* [ This bug can be reproduced easily with a level-triggered -* PCI Ne2000 networking cards and PII/PIII processors, dual -* BX chipset. ] -*/ -/* -* Actually disabling the focus CPU check just makes the hang less -* frequent as it makes the interrupt distributon model be more -* like LRU than MRU (the short-term load is more even across CPUs). -* See also the comment in end_level_ioapic_irq(). --macro +* the NMI deadlock-detector uses this. */ -#if 1 -/* Enable focus processor (bit==0) */ - value &= ~APIC_SPIV_FOCUS_DISABLED ; +per_cpu (irq_stat, cpu).apic_timer_irqs++; + +smp_local_timer_interrupt (); +} +#endif + +void smp_send_timer_broadcast_ipi (void ) +{ +cpumask_t mask; + +cpus_and (mask, cpu_online_map, timer_bcast_ipi); +if (!cpus_empty (mask)) { +#ifdef CONFIG_SMP +send_IPI_mask (mask,LOCAL_TIMER_VECTOR );#else -/* Disable focus processor (bit==1) */ - value |=APIC_SPIV_FOCUS_DISABLED ; +/* +* We can directly call the apic timer interrupt handler +* in UP case. Minus all irq related functions +*/ +up_apic_timer_interrupt_call ();#endif -/* -* Set spurious IRQ vector -*/ - value |=SPURIOUS_APIC_VECTOR ; -apic_write_around (APIC_SPIV , value); + } +} + +int setup_profiling_timer (unsigned int multiplier) +{ +return -EINVAL ; +} + +/* +* Local APIC start and shutdown +*/ + +/** +* clear_local_APIC - shutdown the local APIC +* +* This is called, when a CPU is disabled and before rebooting, so the state of +* the local APIC has no dangling leftovers. Also used to cleanout any BIOS +* leftovers during boot. +*/ +void clear_local_APIC (void ) +{ +int maxlvt =lapic_get_maxlvt (); +unsigned long v;/* -* Set up LVT0, LVT1: -* -* set up through-local-APIC on the BP's LINT0. This is not -* strictly necessery in pure symmetric-IO mode, but sometimes -* we delegate interrupts to the 8259A. +* Masking an LVT entry can trigger a local APIC error +* if the vector is zero. Mask LVTERR first to prevent this. */ +if (maxlvt >=3 ) { + v =ERROR_APIC_VECTOR ;/* any non-zero vector will do */ +apic_write_around (APIC_LVTERR , v |APIC_LVT_MASKED ); + }/* -* TODO: set up through-local-APIC from through-I/O-APIC? --macro +* Careful: we have to set masks only first to deassert +* any level-triggered sources. */ - value =apic_read (APIC_LVT0 ) &APIC_LVT_MASKED ; -if (!smp_processor_id () && (pic_mode || !value)) { - value =APIC_DM_EXTINT ; -apic_printk (APIC_VERBOSE ,"enabled ExtINT on CPU#%d\n" , -smp_processor_id ()); - }else { - value =APIC_DM_EXTINT |APIC_LVT_MASKED ; -apic_printk (APIC_VERBOSE ,"masked ExtINT on CPU#%d\n" , -smp_processor_id ()); + v =apic_read (APIC_LVTT ); +apic_write_around (APIC_LVTT , v |APIC_LVT_MASKED ); + v =apic_read (APIC_LVT0 ); +apic_write_around (APIC_LVT0 , v |APIC_LVT_MASKED ); + v =apic_read (APIC_LVT1 ); +apic_write_around (APIC_LVT1 , v |APIC_LVT_MASKED ); +if (maxlvt >=4 ) { + v =apic_read (APIC_LVTPC ); +apic_write_around (APIC_LVTPC , v |APIC_LVT_MASKED ); } -apic_write_around (APIC_LVT0 , value); +/* lets not touch this if we didn't frob it */ +#ifdef CONFIG_X86_MCE_P4THERMAL +if (maxlvt >=5 ) { + v =apic_read (APIC_LVTTHMR ); +apic_write_around (APIC_LVTTHMR , v |APIC_LVT_MASKED ); + } +#endif /* -* only the BP should see the LINT1 NMI signal, obviously. +* Clean APIC state for other OSs: */ -if (!smp_processor_id ()) - value =APIC_DM_NMI ; -else - value =APIC_DM_NMI |APIC_LVT_MASKED ; -if (!APIC_INTEGRATED (ver))/* 82489DX */ - value |=APIC_LVT_LEVEL_TRIGGER ; -apic_write_around (APIC_LVT1 , value); - -if (APIC_INTEGRATED (ver) && !esr_disable) {/* !82489DX */ - maxlvt =get_maxlvt (); -if (maxlvt >3 )/* Due to the Pentium erratum 3AP. */ -apic_write (APIC_ESR ,0 ); - oldvalue =apic_read (APIC_ESR ); +apic_write_around (APIC_LVTT ,APIC_LVT_MASKED ); +apic_write_around (APIC_LVT0 ,APIC_LVT_MASKED ); +apic_write_around (APIC_LVT1 ,APIC_LVT_MASKED ); +if (maxlvt >=3 ) +apic_write_around (APIC_LVTERR ,APIC_LVT_MASKED ); +if (maxlvt >=4 ) +apic_write_around (APIC_LVTPC ,APIC_LVT_MASKED ); - value =ERROR_APIC_VECTOR ;// enables sending errors -apic_write_around (APIC_LVTERR , value); -/* -* spec says clear errors after enabling vector. -*/ +#ifdef CONFIG_X86_MCE_P4THERMAL +if (maxlvt >=5 ) +apic_write_around (APIC_LVTTHMR ,APIC_LVT_MASKED ); +#endif +/* Integrated APIC (!82489DX) ? */ +if (lapic_is_integrated ()) {if (maxlvt >3 ) +/* Clear ESR due to Pentium errata 3AP and 11AP */ apic_write (APIC_ESR ,0 ); - value =apic_read (APIC_ESR ); -if (value != oldvalue) -apic_printk (APIC_VERBOSE ,"ESR value before enabling " -"vector: 0x%08lx after: 0x%08lx\n" , - oldvalue, value); - }else { -if (esr_disable) -/* -* Something untraceble is creating bad interrupts on -* secondary quads ... for the moment, just leave the -* ESR disabled - we can't do anything useful with the -* errors anyway - mbligh -*/ -printk ("Leaving ESR disabled.\n" ); -else -printk ("No ESR for 82489DX.\n" ); +apic_read (APIC_ESR ); } +} -setup_apic_nmi_watchdog (NULL ); -apic_pm_activate (); +/** +* disable_local_APIC - clear and disable the local APIC +*/ +void disable_local_APIC (void ) +{ +unsigned long value; + +clear_local_APIC (); + +/* +* Disable APIC (implies clearing of registers +* for 82489DX!). +*/ + value =apic_read (APIC_SPIV ); + value &= ~APIC_SPIV_APIC_ENABLED ; +apic_write_around (APIC_SPIV , value); + +/* +* When LAPIC was disabled by the BIOS and enabled by the kernel, +* restore the disabled state. +*/ +if (enabled_via_apicbase) { +unsigned int l, h; + +rdmsr (MSR_IA32_APICBASE , l, h); + l &= ~MSR_IA32_APICBASE_ENABLE ; +wrmsr (MSR_IA32_APICBASE , l, h); + } }/* -* If Linux enabled the LAPIC against the BIOS default -* disable it down before re-entering the BIOS on shutdown. -* Otherwise the BIOS may get confused and not power-off. -* Additionally clear all LVT entries before disable_local_APIC +* If Linux enabled the LAPIC against the BIOS default disable it down before +* re-entering the BIOS on shutdown. Otherwise the BIOS may get confused and +* not power-off. Additionally clear all LVT entries before disable_local_APIC * for the case where Linux didn't enable the LAPIC. */ void lapic_shutdown (void )
@@ -624,666 +629,519 @@ void lapic_shutdown(void)local_irq_restore (flags); } -#ifdef CONFIG_PM - -static struct { -int active; -/* r/w apic fields */ -unsigned int apic_id; -unsigned int apic_taskpri; -unsigned int apic_ldr; -unsigned int apic_dfr; -unsigned int apic_spiv; -unsigned int apic_lvtt; -unsigned int apic_lvtpc; -unsigned int apic_lvt0; -unsigned int apic_lvt1; -unsigned int apic_lvterr; -unsigned int apic_tmict; -unsigned int apic_tdcr; -unsigned int apic_thmr; -} apic_pm_state; - -static int lapic_suspend (struct sys_device *dev,pm_message_t state) +/* +* This is to verify that we're looking at a real local APIC. +* Check these against your board if the CPUs aren't getting +* started for no apparent reason. +*/ +int __init verify_local_APIC (void ) { -unsigned long flags; +unsigned int reg0, reg1; -if (!apic_pm_state.active) +/* +* The version register is read-only in a real APIC. +*/ + reg0 =apic_read (APIC_LVR ); +apic_printk (APIC_DEBUG ,"Getting VERSION: %x\n" , reg0); +apic_write (APIC_LVR , reg0 ^APIC_LVR_MASK ); + reg1 =apic_read (APIC_LVR ); +apic_printk (APIC_DEBUG ,"Getting VERSION: %x\n" , reg1); + +/* +* The two version reads above should print the same +* numbers. If the second one is different, then we +* poke at a non-APIC. +*/ +if (reg1 != reg0)return 0 ; - apic_pm_state.apic_id =apic_read (APIC_ID ); - apic_pm_state.apic_taskpri =apic_read (APIC_TASKPRI ); - apic_pm_state.apic_ldr =apic_read (APIC_LDR ); - apic_pm_state.apic_dfr =apic_read (APIC_DFR ); - apic_pm_state.apic_spiv =apic_read (APIC_SPIV ); - apic_pm_state.apic_lvtt =apic_read (APIC_LVTT ); - apic_pm_state.apic_lvtpc =apic_read (APIC_LVTPC ); - apic_pm_state.apic_lvt0 =apic_read (APIC_LVT0 ); - apic_pm_state.apic_lvt1 =apic_read (APIC_LVT1 ); - apic_pm_state.apic_lvterr =apic_read (APIC_LVTERR ); - apic_pm_state.apic_tmict =apic_read (APIC_TMICT ); - apic_pm_state.apic_tdcr =apic_read (APIC_TDCR ); - apic_pm_state.apic_thmr =apic_read (APIC_LVTTHMR ); - -local_irq_save (flags); -disable_local_APIC (); -local_irq_restore (flags); -return 0 ; -} - -static int lapic_resume (struct sys_device *dev) -{ -unsigned int l, h; -unsigned long flags; - -if (!apic_pm_state.active) -return 0 ; - -local_irq_save (flags); -/* -* Make sure the APICBASE points to the right address -* -* FIXME! This will be wrong if we ever support suspend on -* SMP! We'll need to do this as part of the CPU restore! +* Check if the version looks reasonably. */ -rdmsr (MSR_IA32_APICBASE , l, h); - l &= ~MSR_IA32_APICBASE_BASE ; - l |=MSR_IA32_APICBASE_ENABLE | mp_lapic_addr; -wrmsr (MSR_IA32_APICBASE , l, h); - -apic_write (APIC_LVTERR ,ERROR_APIC_VECTOR |APIC_LVT_MASKED ); -apic_write (APIC_ID , apic_pm_state.apic_id); -apic_write (APIC_DFR , apic_pm_state.apic_dfr); -apic_write (APIC_LDR , apic_pm_state.apic_ldr); -apic_write (APIC_TASKPRI , apic_pm_state.apic_taskpri); -apic_write (APIC_SPIV , apic_pm_state.apic_spiv); -apic_write (APIC_LVT0 , apic_pm_state.apic_lvt0); -apic_write (APIC_LVT1 , apic_pm_state.apic_lvt1); -apic_write (APIC_LVTTHMR , apic_pm_state.apic_thmr); -apic_write (APIC_LVTPC , apic_pm_state.apic_lvtpc); -apic_write (APIC_LVTT , apic_pm_state.apic_lvtt); -apic_write (APIC_TDCR , apic_pm_state.apic_tdcr); -apic_write (APIC_TMICT , apic_pm_state.apic_tmict); -apic_write (APIC_ESR ,0 ); -apic_read (APIC_ESR ); -apic_write (APIC_LVTERR , apic_pm_state.apic_lvterr); -apic_write (APIC_ESR ,0 ); -apic_read (APIC_ESR ); -local_irq_restore (flags); -return 0 ; -} - -/* -* This device has no shutdown method - fully functioning local APICs -* are needed on every CPU up until machine_halt/restart/poweroff. -*/ - -static struct sysdev_class lapic_sysclass = { -set_kset_name ("lapic" ), - .resume = lapic_resume, - .suspend = lapic_suspend, -}; - -static struct sys_device device_lapic = { - .id =0 , - .cls = &lapic_sysclass, -}; - -static void __devinit apic_pm_activate (void ) -{ - apic_pm_state.active =1 ; -} - -static int __init init_lapic_sysfs (void ) -{ -int error; - -if (!cpu_has_apic) + reg1 =GET_APIC_VERSION (reg0); +if (reg1 ==0x00 || reg1 ==0xff ) +return 0 ; + reg1 =lapic_get_maxlvt (); +if (reg1 <0x02 || reg1 ==0xff )return 0 ; -/* XXX: remove suspend/resume procs if !apic_pm_state.active? */ - - error =sysdev_class_register (&lapic_sysclass); -if (!error) - error =sysdev_register (&device_lapic); -return error; -} -device_initcall (init_lapic_sysfs); - -#else /* CONFIG_PM */ - -static void apic_pm_activate (void ) { } - -#endif /* CONFIG_PM */ - -/* -* Detect and enable local APICs on non-SMP boards. -* Original code written by Keir Fraser. -*/ - -static int __init apic_set_verbosity (char *str) -{ -if (strcmp ("debug" , str) ==0 ) - apic_verbosity =APIC_DEBUG ; -else if (strcmp ("verbose" , str) ==0 ) - apic_verbosity =APIC_VERBOSE ; -return 1 ; -} - -__setup ("apic=" , apic_set_verbosity); - -static int __init detect_init_APIC (void ) -{ -u32 h, l, features; - -/* Disabled by kernel option? */ -if (enable_local_apic <0 ) -return -1 ; - -switch (boot_cpu_data.x86_vendor) { -case X86_VENDOR_AMD : -if ((boot_cpu_data.x86 ==6 && boot_cpu_data.x86_model >1 ) || - (boot_cpu_data.x86 ==15 )) -break ; - goto no_apic; -case X86_VENDOR_INTEL : -if (boot_cpu_data.x86 ==6 || boot_cpu_data.x86 ==15 || - (boot_cpu_data.x86 ==5 && cpu_has_apic)) -break ; - goto no_apic; -default : - goto no_apic; - } - -if (!cpu_has_apic) { -/* -* Over-ride BIOS and try to enable the local -* APIC only if "lapic" specified. -*/ -if (enable_local_apic <=0 ) { -printk ("Local APIC disabled by BIOS -- " -"you can enable it with \"lapic\"\n" ); -return -1 ; - } -/* -* Some BIOSes disable the local APIC in the -* APIC_BASE MSR. This can only be done in -* software for Intel P6 or later and AMD K7 -* (Model > 1) or later. -*/ -rdmsr (MSR_IA32_APICBASE , l, h); -if (!(l &MSR_IA32_APICBASE_ENABLE )) { -printk ("Local APIC disabled by BIOS -- reenabling.\n" ); - l &= ~MSR_IA32_APICBASE_BASE ; - l |=MSR_IA32_APICBASE_ENABLE |APIC_DEFAULT_PHYS_BASE ; -wrmsr (MSR_IA32_APICBASE , l, h); - enabled_via_apicbase =1 ; - } - } -/* -* The APIC feature bit should now be enabled -* in `cpuid' -*/ - features =cpuid_edx (1 ); -if (!(features & (1 <<X86_FEATURE_APIC ))) { -printk ("Could not enable APIC!\n" ); -return -1 ; - } -set_bit (X86_FEATURE_APIC , boot_cpu_data.x86_capability); - mp_lapic_addr =APIC_DEFAULT_PHYS_BASE ; - -/* The BIOS may have set up the APIC at some other address */ -rdmsr (MSR_IA32_APICBASE , l, h); -if (l &MSR_IA32_APICBASE_ENABLE ) - mp_lapic_addr = l &MSR_IA32_APICBASE_BASE ; - -if (nmi_watchdog !=NMI_NONE ) - nmi_watchdog =NMI_LOCAL_APIC ; - -printk ("Found and enabled local APIC!\n" ); - -apic_pm_activate (); - -return 0 ; - -no_apic: -printk ("No local APIC present or hardware disabled\n" ); -return -1 ; -} - -void __init init_apic_mappings (void ) -{ -unsigned long apic_phys;/* -* If no local APIC can be found then set up a fake all -* zeroes page to simulate the local APIC and another -* one for the IO-APIC. +* The ID register is read/write in a real APIC. */ -if (!smp_found_config &&detect_init_APIC ()) { - apic_phys = (unsigned long )alloc_bootmem_pages (PAGE_SIZE ); - apic_phys =__pa (apic_phys); - }else - apic_phys = mp_lapic_addr; - -set_fixmap_nocache (FIX_APIC_BASE , apic_phys); -printk (KERN_DEBUG "mapped APIC to %08lx (%08lx)\n" ,APIC_BASE , - apic_phys); + reg0 =apic_read (APIC_ID ); +apic_printk (APIC_DEBUG ,"Getting ID: %x\n" , reg0);/* -* Fetch the APIC ID of the BSP in case we have a -* default configuration (or the MP table is broken). +* The next two are just to see if we have sane values. +* They're only really relevant if we're in Virtual Wire +* compatibility mode, but most boxes are anymore. */ -if (boot_cpu_physical_apicid ==-1U ) - boot_cpu_physical_apicid =GET_APIC_ID (apic_read (APIC_ID )); - -#ifdef CONFIG_X86_IO_APIC - { -unsigned long ioapic_phys, idx =FIX_IO_APIC_BASE_0 ; -int i; - -for (i =0 ; i < nr_ioapics; i++) { -if (smp_found_config) { - ioapic_phys = mp_ioapics[i].mpc_apicaddr; -if (!ioapic_phys) { -printk (KERN_ERR -"WARNING: bogus zero IO-APIC " -"address found in MPTABLE, " -"disabling IO/APIC support!\n" ); - smp_found_config =0 ; - skip_ioapic_setup =1 ; - goto fake_ioapic_page; - } - }else { -fake_ioapic_page: - ioapic_phys = (unsigned long ) -alloc_bootmem_pages (PAGE_SIZE ); - ioapic_phys =__pa (ioapic_phys); - } -set_fixmap_nocache (idx, ioapic_phys); -printk (KERN_DEBUG "mapped IOAPIC to %08lx (%08lx)\n" , -__fix_to_virt (idx), ioapic_phys); - idx++; - } - } -#endif -} - -/* -* This part sets up the APIC 32 bit clock in LVTT1, with HZ interrupts -* per second. We assume that the caller has already set up the local -* APIC. -* -* The APIC timer is not exactly sync with the external timer chip, it -* closely follows bus clocks. -*/ - -/* -* The timer chip is already set up at HZ interrupts per second here, -* but we do not accept timer interrupts yet. We only allow the BP -* to calibrate. -*/ -static unsigned int __devinit get_8254_timer_count (void ) -{ -unsigned long flags; - -unsigned int count; - -spin_lock_irqsave (&i8253_lock, flags); - -outb_p (0x00 ,PIT_MODE ); - count =inb_p (PIT_CH0 ); - count |=inb_p (PIT_CH0 ) <<8 ; - -spin_unlock_irqrestore (&i8253_lock, flags); - -return count; -} - -/* next tick in 8254 can be caught by catching timer wraparound */ -static void __devinit wait_8254_wraparound (void ) -{ -unsigned int curr_count, prev_count; - - curr_count =get_8254_timer_count (); -do { - prev_count = curr_count; - curr_count =get_8254_timer_count (); - -/* workaround for broken Mercury/Neptune */ -if (prev_count >= curr_count +0x100 ) - curr_count =get_8254_timer_count (); - - }while (prev_count >= curr_count); -} - -/* -* Default initialization for 8254 timers. If we use other timers like HPET, -* we override this later -*/ -void (*wait_timer_tick)(void )__devinitdata = wait_8254_wraparound; - -/* -* This function sets up the local APIC timer, with a timeout of -* 'clocks' APIC bus clock. During calibration we actually call -* this function twice on the boot CPU, once with a bogus timeout -* value, second time for real. The other (noncalibrating) CPUs -* call this function only once, with the real, calibrated value. -* -* We do reads before writes even if unnecessary, to get around the -* P5 APIC double write bug. -*/ - -#define APIC_DIVISOR 16 - -static void __setup_APIC_LVTT (unsigned int clocks) -{ -unsigned int lvtt_value, tmp_value, ver; -int cpu =smp_processor_id (); - - ver =GET_APIC_VERSION (apic_read (APIC_LVR )); - lvtt_value =APIC_LVT_TIMER_PERIODIC |LOCAL_TIMER_VECTOR ; -if (!APIC_INTEGRATED (ver)) - lvtt_value |=SET_APIC_TIMER_BASE (APIC_TIMER_BASE_DIV ); - -if (cpu_isset (cpu, timer_bcast_ipi)) - lvtt_value |=APIC_LVT_MASKED ; + reg0 =apic_read (APIC_LVT0 ); +apic_printk (APIC_DEBUG ,"Getting LVT0: %x\n" , reg0); + reg1 =apic_read (APIC_LVT1 ); +apic_printk (APIC_DEBUG ,"Getting LVT1: %x\n" , reg1); -apic_write_around (APIC_LVTT , lvtt_value); +return 1 ; +} +/** +* sync_Arb_IDs - synchronize APIC bus arbitration IDs +*/ +void __init sync_Arb_IDs (void ) +{/* -* Divide PICLK by 16 +* Unsupported on P4 - see Intel Dev. Manual Vol. 3, Ch. 8.6.1 And not +* needed on AMD. */ - tmp_value =apic_read (APIC_TDCR ); -apic_write_around (APIC_TDCR , (tmp_value - & ~(APIC_TDR_DIV_1 |APIC_TDR_DIV_TMBASE )) - |APIC_TDR_DIV_16 ); +if (modern_apic ()) +return ; +/* +* Wait for idle. +*/ +apic_wait_icr_idle (); -apic_write_around (APIC_TMICT , clocks/APIC_DIVISOR ); +apic_printk (APIC_DEBUG ,"Synchronizing Arb IDs.\n" ); +apic_write_around (APIC_ICR ,APIC_DEST_ALLINC |APIC_INT_LEVELTRIG + |APIC_DM_INIT ); } -static void __devinit setup_APIC_timer (unsigned int clocks) +/* +* An initial setup of the virtual wire mode. +*/ +void __init init_bsp_APIC (void ) { -unsigned long flags; +unsigned long value; -local_irq_save (flags); +/* +* Don't do the setup now if we have a SMP BIOS as the +* through-I/O-APIC virtual wire mode might be active. +*/ +if (smp_found_config || !cpu_has_apic) +return ;/* -* Wait for IRQ0's slice: +* Do not trust the local APIC being empty at bootup. */ -wait_timer_tick (); +clear_local_APIC (); -__setup_APIC_LVTT (clocks); +/* +* Enable APIC. +*/ + value =apic_read (APIC_SPIV ); + value &= ~APIC_VECTOR_MASK ; + value |=APIC_SPIV_APIC_ENABLED ; -local_irq_restore (flags); +/* This bit is reserved on P4/Xeon and should be cleared */ +if ((boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ) && + (boot_cpu_data.x86 ==15 )) + value &= ~APIC_SPIV_FOCUS_DISABLED ; +else + value |=APIC_SPIV_FOCUS_DISABLED ; + value |=SPURIOUS_APIC_VECTOR ; +apic_write_around (APIC_SPIV , value); + +/* +* Set up the virtual wire mode. +*/ +apic_write_around (APIC_LVT0 ,APIC_DM_EXTINT ); + value =APIC_DM_NMI ; +if (!lapic_is_integrated ())/* 82489DX */ + value |=APIC_LVT_LEVEL_TRIGGER ; +apic_write_around (APIC_LVT1 , value); } -/* -* In this function we calibrate APIC bus clocks to the external -* timer. Unfortunately we cannot use jiffies and the timer irq -* to calibrate, since some later bootup code depends on getting -* the first irq? Ugh. -* -* We want to do the calibration only once since we -* want to have local timer irqs syncron. CPUs connected -* by the same APIC bus have the very same bus frequency. -* And we want to have irqs off anyways, no accidental -* APIC irq that way. +/** +* setup_local_APIC - setup the local APIC */ - -static int __init calibrate_APIC_clock (void ) +void __devinit setup_local_APIC (void ) { -unsigned long long t1 =0 , t2 =0 ; -long tt1, tt2; -long result; -int i; -const int LOOPS =HZ /10 ; +unsigned long oldvalue, value, maxlvt, integrated; +int i, j; -apic_printk (APIC_VERBOSE ,"calibrating APIC timer ...\n" ); +/* Pound the ESR really hard over the head with a big hammer - mbligh */ +if (esr_disable) { +apic_write (APIC_ESR ,0 ); +apic_write (APIC_ESR ,0 ); +apic_write (APIC_ESR ,0 ); +apic_write (APIC_ESR ,0 ); + } + + integrated =lapic_is_integrated ();/* -* Put whatever arbitrary (but long enough) timeout -* value into the APIC clock, we just want to get the -* counter running for calibration. +* Double-check whether this APIC is really registered. */ -__setup_APIC_LVTT (1000000000 ); +if (!apic_id_registered ()) +BUG ();/* -* The timer chip counts down to zero. Let's wait -* for a wraparound to start exact measurement: -* (the current tick might have been already half done) +* Intel recommends to set DFR, LDR and TPR before enabling +* an APIC. See e.g. "AP-388 82489DX User's Manual" (Intel +* document number 292116). So here it goes... */ - -wait_timer_tick (); +init_apic_ldr ();/* -* We wrapped around just now. Let's start: +* Set Task Priority to 'accept all'. We never change this +* later on. */ -if (cpu_has_tsc) -rdtscll (t1); - tt1 =apic_read (APIC_TMCCT ); + value =apic_read (APIC_TASKPRI ); + value &= ~APIC_TPRI_MASK ; +apic_write_around (APIC_TASKPRI , value);/* -* Let's wait LOOPS wraprounds: +* After a crash, we no longer service the interrupts and a pending +* interrupt from previous kernel might still have ISR bit set. +* +* Most probably by now CPU has serviced that pending interrupt and +* it might not have done the ack_APIC_irq() because it thought, +* interrupt came from i8259 as ExtInt. LAPIC did not get EOI so it +* does not clear the ISR bit and cpu thinks it has already serivced +* the interrupt. Hence a vector might get locked. It was noticed +* for timer irq (vector 0x31). Issue an extra EOI to clear ISR. */ -for (i =0 ; i <LOOPS ; i++) -wait_timer_tick (); +for (i =APIC_ISR_NR -1 ; i >=0 ; i--) { + value =apic_read (APIC_ISR + i*0x10 ); +for (j =31 ; j >=0 ; j--) { +if (value & (1 <<j)) +ack_APIC_irq (); + } + } - tt2 =apic_read (APIC_TMCCT ); -if (cpu_has_tsc) -rdtscll (t2); +/* +* Now that we are all set up, enable the APIC +*/ + value =apic_read (APIC_SPIV ); + value &= ~APIC_VECTOR_MASK ; +/* +* Enable APIC +*/ + value |=APIC_SPIV_APIC_ENABLED ;/* -* The APIC bus clock counter is 32 bits only, it -* might have overflown, but note that we use signed -* longs, thus no extra care needed. +* Some unknown Intel IO/APIC (or APIC) errata is biting us with +* certain networking cards. If high frequency interrupts are +* happening on a particular IOAPIC pin, plus the IOAPIC routing +* entry is masked/unmasked at a high rate as well then sooner or +* later IOAPIC line gets 'stuck', no more interrupts are received +* from the device. If focus CPU is disabled then the hang goes +* away, oh well :-( * -* underflown to be exact, as the timer counts down ;) +* [ This bug can be reproduced easily with a level-triggered +* PCI Ne2000 networking cards and PII/PIII processors, dual +* BX chipset. ] +*/ +/* +* Actually disabling the focus CPU check just makes the hang less +* frequent as it makes the interrupt distributon model be more +* like LRU than MRU (the short-term load is more even across CPUs). +* See also the comment in end_level_ioapic_irq(). --macro */ - result = (tt1-tt2)*APIC_DIVISOR /LOOPS ; - -if (cpu_has_tsc) -apic_printk (APIC_VERBOSE ,"..... CPU clock speed is " -"%ld.%04ld MHz.\n" , - ((long )(t2-t1)/LOOPS )/(1000000 /HZ ), - ((long )(t2-t1)/LOOPS )%(1000000 /HZ )); - -apic_printk (APIC_VERBOSE ,"..... host bus clock speed is " -"%ld.%04ld MHz.\n" , - result/(1000000 /HZ ), - result%(1000000 /HZ )); - -return result; -} - -static unsigned int calibration_result; - -void __init setup_boot_APIC_clock (void ) -{ -unsigned long flags; -apic_printk (APIC_VERBOSE ,"Using local APIC timer interrupts.\n" ); - using_apic_timer =1 ; - -local_irq_save (flags); +/* Enable focus processor (bit==0) */ + value &= ~APIC_SPIV_FOCUS_DISABLED ; - calibration_result =calibrate_APIC_clock ();/* -* Now set up the timer for real. +* Set spurious IRQ vector */ -setup_APIC_timer (calibration_result); + value |=SPURIOUS_APIC_VECTOR ; +apic_write_around (APIC_SPIV , value); -local_irq_restore (flags); -} +/* +* Set up LVT0, LVT1: +* +* set up through-local-APIC on the BP's LINT0. This is not +* strictly necessery in pure symmetric-IO mode, but sometimes +* we delegate interrupts to the 8259A. +*/ +/* +* TODO: set up through-local-APIC from through-I/O-APIC? --macro +*/ + value =apic_read (APIC_LVT0 ) &APIC_LVT_MASKED ; +if (!smp_processor_id () && (pic_mode || !value)) { + value =APIC_DM_EXTINT ; +apic_printk (APIC_VERBOSE ,"enabled ExtINT on CPU#%d\n" , +smp_processor_id ()); + }else { + value =APIC_DM_EXTINT |APIC_LVT_MASKED ; +apic_printk (APIC_VERBOSE ,"masked ExtINT on CPU#%d\n" , +smp_processor_id ()); + } +apic_write_around (APIC_LVT0 , value); -void __devinit setup_secondary_APIC_clock (void ) -{ -setup_APIC_timer (calibration_result); -} +/* +* only the BP should see the LINT1 NMI signal, obviously. +*/ +if (!smp_processor_id ()) + value =APIC_DM_NMI ; +else + value =APIC_DM_NMI |APIC_LVT_MASKED ; +if (!integrated)/* 82489DX */ + value |=APIC_LVT_LEVEL_TRIGGER ; +apic_write_around (APIC_LVT1 , value); -void disable_APIC_timer (void ) -{ -if (using_apic_timer) { -unsigned long v; +if (integrated && !esr_disable) {/* !82489DX */ + maxlvt =lapic_get_maxlvt (); +if (maxlvt >3 )/* Due to the Pentium erratum 3AP. */ +apic_write (APIC_ESR ,0 ); + oldvalue =apic_read (APIC_ESR ); - v =apic_read (APIC_LVTT ); +/* enables sending errors */ + value =ERROR_APIC_VECTOR ; +apic_write_around (APIC_LVTERR , value);/* -* When an illegal vector value (0-15) is written to an LVT -* entry and delivery mode is Fixed, the APIC may signal an -* illegal vector error, with out regard to whether the mask -* bit is set or whether an interrupt is actually seen on input. -* -* Boot sequence might call this function when the LVTT has -* '0' vector value. So make sure vector field is set to -* valid value. +* spec says clear errors after enabling vector. */ - v |= (APIC_LVT_MASKED |LOCAL_TIMER_VECTOR ); -apic_write_around (APIC_LVTT , v); +if (maxlvt >3 ) +apic_write (APIC_ESR ,0 ); + value =apic_read (APIC_ESR ); +if (value != oldvalue) +apic_printk (APIC_VERBOSE ,"ESR value before enabling " +"vector: 0x%08lx after: 0x%08lx\n" , + oldvalue, value); + }else { +if (esr_disable) +/* +* Something untraceble is creating bad interrupts on +* secondary quads ... for the moment, just leave the +* ESR disabled - we can't do anything useful with the +* errors anyway - mbligh +*/ +printk (KERN_INFO "Leaving ESR disabled.\n" ); +else +printk (KERN_INFO "No ESR for 82489DX.\n" ); } + +setup_apic_nmi_watchdog (NULL ); +apic_pm_activate (); } -void enable_APIC_timer (void ) +/* +* Detect and initialize APIC +*/ +static int __init detect_init_APIC (void ) { -int cpu =smp_processor_id (); +u32 h, l, features; -if (using_apic_timer && - !cpu_isset (cpu, timer_bcast_ipi)) { -unsigned long v; +/* Disabled by kernel option? */ +if (enable_local_apic <0 ) +return -1 ; - v =apic_read (APIC_LVTT ); -apic_write_around (APIC_LVTT , v & ~APIC_LVT_MASKED ); +switch (boot_cpu_data.x86_vendor) { +case X86_VENDOR_AMD : +if ((boot_cpu_data.x86 ==6 && boot_cpu_data.x86_model >1 ) || + (boot_cpu_data.x86 ==15 )) +break ; + goto no_apic; +case X86_VENDOR_INTEL : +if (boot_cpu_data.x86 ==6 || boot_cpu_data.x86 ==15 || + (boot_cpu_data.x86 ==5 && cpu_has_apic)) +break ; + goto no_apic; +default : + goto no_apic; } -} - -void switch_APIC_timer_to_ipi (void *cpumask) -{ -cpumask_t mask = *(cpumask_t *)cpumask; -int cpu =smp_processor_id (); -if (cpu_isset (cpu, mask) && - !cpu_isset (cpu, timer_bcast_ipi)) { -disable_APIC_timer (); -cpu_set (cpu, timer_bcast_ipi); +if (!cpu_has_apic) { +/* +* Over-ride BIOS and try to enable the local APIC only if +* "lapic" specified. +*/ +if (enable_local_apic <=0 ) { +printk (KERN_INFO "Local APIC disabled by BIOS -- " +"you can enable it with \"lapic\"\n" ); +return -1 ; + } +/* +* Some BIOSes disable the local APIC in the APIC_BASE +* MSR. This can only be done in software for Intel P6 or later +* and AMD K7 (Model > 1) or later. +*/ +rdmsr (MSR_IA32_APICBASE , l, h); +if (!(l &MSR_IA32_APICBASE_ENABLE )) { +printk (KERN_INFO +"Local APIC disabled by BIOS -- reenabling.\n" ); + l &= ~MSR_IA32_APICBASE_BASE ; + l |=MSR_IA32_APICBASE_ENABLE |APIC_DEFAULT_PHYS_BASE ; +wrmsr (MSR_IA32_APICBASE , l, h); + enabled_via_apicbase =1 ; + } } -} -EXPORT_SYMBOL (switch_APIC_timer_to_ipi); +/* +* The APIC feature bit should now be enabled +* in `cpuid' +*/ + features =cpuid_edx (1 ); +if (!(features & (1 <<X86_FEATURE_APIC ))) { +printk (KERN_WARNING "Could not enable APIC!\n" ); +return -1 ; + } +set_bit (X86_FEATURE_APIC , boot_cpu_data.x86_capability); + mp_lapic_addr =APIC_DEFAULT_PHYS_BASE ; -void switch_ipi_to_APIC_timer (void *cpumask) -{ -cpumask_t mask = *(cpumask_t *)cpumask; -int cpu =smp_processor_id (); +/* The BIOS may have set up the APIC at some other address */ +rdmsr (MSR_IA32_APICBASE , l, h); +if (l &MSR_IA32_APICBASE_ENABLE ) + mp_lapic_addr = l &MSR_IA32_APICBASE_BASE ; -if (cpu_isset (cpu, mask) && -cpu_isset (cpu, timer_bcast_ipi)) { -cpu_clear (cpu, timer_bcast_ipi); -enable_APIC_timer (); - } -} -EXPORT_SYMBOL (switch_ipi_to_APIC_timer); +if (nmi_watchdog !=NMI_NONE ) + nmi_watchdog =NMI_LOCAL_APIC ; -#undef APIC_DIVISOR +printk (KERN_INFO "Found and enabled local APIC!\n" ); -/* -* Local timer interrupt handler. It does both profiling and -* process statistics/rescheduling. -* -* We do profiling in every local tick, statistics/rescheduling -* happen only every 'profiling multiplier' ticks. The default -* multiplier is 1 and it can be changed by writing the new multiplier -* value into /proc/profile. -*/ +apic_pm_activate (); -inline void smp_local_timer_interrupt (void ) -{ -profile_tick (CPU_PROFILING ); -#ifdef CONFIG_SMP -update_process_times (user_mode_vm (get_irq_regs ())); -#endif +return 0 ; -/* -* We take the 'long' return path, and there every subsystem -* grabs the apropriate locks (kernel lock/ irq lock). -* -* we might want to decouple profiling from the 'long path', -* and do the profiling totally in assembly. -* -* Currently this isn't too much of an issue (performance wise), -* we can take more than 100K local irqs per second on a 100 MHz P5. -*/ +no_apic: +printk (KERN_INFO "No local APIC present or hardware disabled\n" ); +return -1 ; } -/* -* Local APIC timer interrupt. This is the most natural way for doing -* local interrupts, but local timer interrupts can be emulated by -* broadcast interrupts too. [in case the hw doesn't support APIC timers] -* -* [ if a single-CPU system runs an SMP kernel then we call the local -* interrupt as well. Thus we cannot inline the local irq ... ] +/** +* init_apic_mappings - initialize APIC mappings */ - -fastcall void smp_apic_timer_interrupt(struct pt_regs *regs) +void __init init_apic_mappings (void ) { -struct pt_regs *old_regs =set_irq_regs (regs); -int cpu =smp_processor_id (); +unsigned long apic_phys;/* -* the NMI deadlock-detector uses this. +* If no local APIC can be found then set up a fake all +* zeroes page to simulate the local APIC and another +* one for the IO-APIC. */ -per_cpu (irq_stat, cpu).apic_timer_irqs++; +if (!smp_found_config &&detect_init_APIC ()) { + apic_phys = (unsigned long )alloc_bootmem_pages (PAGE_SIZE ); + apic_phys =__pa (apic_phys); + }else + apic_phys = mp_lapic_addr; + +set_fixmap_nocache (FIX_APIC_BASE , apic_phys); +printk (KERN_DEBUG "mapped APIC to %08lx (%08lx)\n" ,APIC_BASE , + apic_phys);/* -* NOTE! We'd better ACK the irq immediately, -* because timer handling can be slow. -*/ -ack_APIC_irq (); -/* -* update_process_times() expects us to have done irq_enter(). -* Besides, if we don't timer interrupts ignore the global -* interrupt lock, which is the WrongThing (tm) to do. +* Fetch the APIC ID of the BSP in case we have a +* default configuration (or the MP table is broken). */ -irq_enter (); -smp_local_timer_interrupt (); -irq_exit (); -set_irq_regs (old_regs); +if (boot_cpu_physical_apicid ==-1U ) + boot_cpu_physical_apicid =GET_APIC_ID (apic_read (APIC_ID )); + +#ifdef CONFIG_X86_IO_APIC + { +unsigned long ioapic_phys, idx =FIX_IO_APIC_BASE_0 ; +int i; + +for (i =0 ; i < nr_ioapics; i++) { +if (smp_found_config) { + ioapic_phys = mp_ioapics[i].mpc_apicaddr; +if (!ioapic_phys) { +printk (KERN_ERR +"WARNING: bogus zero IO-APIC " +"address found in MPTABLE, " +"disabling IO/APIC support!\n" ); + smp_found_config =0 ; + skip_ioapic_setup =1 ; + goto fake_ioapic_page; + } + }else { +fake_ioapic_page: + ioapic_phys = (unsigned long ) +alloc_bootmem_pages (PAGE_SIZE ); + ioapic_phys =__pa (ioapic_phys); + } +set_fixmap_nocache (idx, ioapic_phys); +printk (KERN_DEBUG "mapped IOAPIC to %08lx (%08lx)\n" , +__fix_to_virt (idx), ioapic_phys); + idx++; + } + } +#endif } -#ifndef CONFIG_SMP -static void up_apic_timer_interrupt_call (void ) +/* +* This initializes the IO-APIC and APIC hardware if this is +* a UP kernel. +*/ +int __init APIC_init_uniprocessor (void ) { -int cpu =smp_processor_id (); +if (enable_local_apic <0 ) +clear_bit (X86_FEATURE_APIC , boot_cpu_data.x86_capability); + +if (!smp_found_config && !cpu_has_apic) +return -1 ;/* -* the NMI deadlock-detector uses this. +* Complain if the BIOS pretends there is one. */ -per_cpu (irq_stat, cpu).apic_timer_irqs++; +if (!cpu_has_apic && +APIC_INTEGRATED (apic_version[boot_cpu_physical_apicid])) { +printk (KERN_ERR "BIOS bug, local APIC #%d not detected!...\n" , + boot_cpu_physical_apicid); +clear_bit (X86_FEATURE_APIC , boot_cpu_data.x86_capability); +return -1 ; + } -smp_local_timer_interrupt (); -} +verify_local_APIC (); + +connect_bsp_APIC (); + +/* +* Hack: In case of kdump, after a crash, kernel might be booting +* on a cpu with non-zero lapic id. But boot_cpu_physical_apicid +* might be zero if read from MP tables. Get it from LAPIC. +*/ +#ifdef CONFIG_CRASH_DUMP + boot_cpu_physical_apicid =GET_APIC_ID (apic_read (APIC_ID ));#endif + phys_cpu_present_map =physid_mask_of_physid (boot_cpu_physical_apicid); -void smp_send_timer_broadcast_ipi (void ) -{ -cpumask_t mask; +setup_local_APIC (); -cpus_and (mask, cpu_online_map, timer_bcast_ipi); -if (!cpus_empty (mask)) { -#ifdef CONFIG_SMP -send_IPI_mask (mask,LOCAL_TIMER_VECTOR ); -#else -/* -* We can directly call the apic timer interrupt handler -* in UP case. Minus all irq related functions -*/ -up_apic_timer_interrupt_call (); +#ifdef CONFIG_X86_IO_APIC +if (smp_found_config) +if (!skip_ioapic_setup && nr_ioapics) +setup_IO_APIC ();#endif - } +setup_boot_APIC_clock (); + +return 0 ; +} + +/* +* APIC command line parameters +*/ +static int __init parse_lapic (char *arg) +{ + enable_local_apic =1 ; +return 0 ; +} +early_param ("lapic" , parse_lapic); + +static int __init parse_nolapic (char *arg) +{ + enable_local_apic =-1 ; +clear_bit (X86_FEATURE_APIC , boot_cpu_data.x86_capability); +return 0 ; } +early_param ("nolapic" , parse_nolapic); -int setup_profiling_timer (unsigned int multiplier) +static int __init apic_set_verbosity (char *str) { -return -EINVAL ; +if (strcmp ("debug" , str) ==0 ) + apic_verbosity =APIC_DEBUG ; +else if (strcmp ("verbose" , str) ==0 ) + apic_verbosity =APIC_VERBOSE ; +return 1 ; } +__setup ("apic=" , apic_set_verbosity); + + +/* +* Local APIC interrupts +*/ +/* * This interrupt should _never_ happen with our APIC/SMP architecture */
@@ -1302,15 +1160,14 @@ fastcall void smp_spurious_interrupt(strack_APIC_irq ();/* see sw-dev-man vol 3, chapter 7.4.13.5 */ -printk (KERN_INFO "spurious APIC interrupt on CPU#%d, should never happen.\n" , -smp_processor_id ()); +printk (KERN_INFO "spurious APIC interrupt on CPU#%d, " +"should never happen.\n" ,smp_processor_id ());irq_exit (); }/* * This interrupt should never happen with our APIC/SMP architecture */ - fastcall void smp_error_interrupt(struct pt_regs *regs) {unsigned long v, v1;
@@ -1334,69 +1191,247 @@ fastcall void smp_error_interrupt(struct7: Illegal register address */ printk (KERN_DEBUG "APIC error on CPU%d: %02lx(%02lx)\n" , -smp_processor_id (), v , v1); +smp_processor_id (), v , v1);irq_exit (); }/* -* This initializes the IO-APIC and APIC hardware if this is -* a UP kernel. +* Initialize APIC interrupts */ -int __init APIC_init_uniprocessor (void ) +void __init apic_intr_init (void ) { -if (enable_local_apic <0 ) -clear_bit (X86_FEATURE_APIC , boot_cpu_data.x86_capability); +#ifdef CONFIG_SMP +smp_intr_init (); +#endif +/* self generated IPI for local APIC timer */ +set_intr_gate (LOCAL_TIMER_VECTOR , apic_timer_interrupt); -if (!smp_found_config && !cpu_has_apic) -return -1 ; +/* IPI vectors for APIC spurious and error interrupts */ +set_intr_gate (SPURIOUS_APIC_VECTOR , spurious_interrupt); +set_intr_gate (ERROR_APIC_VECTOR , error_interrupt); -/* -* Complain if the BIOS pretends there is one. -*/ -if (!cpu_has_apic &&APIC_INTEGRATED (apic_version[boot_cpu_physical_apicid])) { -printk (KERN_ERR "BIOS bug, local APIC #%d not detected!...\n" , - boot_cpu_physical_apicid); -clear_bit (X86_FEATURE_APIC , boot_cpu_data.x86_capability); -return -1 ; +/* thermal monitor LVT interrupt */ +#ifdef CONFIG_X86_MCE_P4THERMAL +set_intr_gate (THERMAL_APIC_VECTOR , thermal_interrupt); +#endif +} + +/** +* connect_bsp_APIC - attach the APIC to the interrupt system +*/ +void __init connect_bsp_APIC (void ) +{ +if (pic_mode) { +/* +* Do not trust the local APIC being empty at bootup. +*/ +clear_local_APIC (); +/* +* PIC mode, enable APIC mode in the IMCR, i.e. connect BSP's +* local APIC to INT and NMI lines. +*/ +apic_printk (APIC_VERBOSE ,"leaving PIC mode, " +"enabling APIC mode.\n" ); +outb (0x70 ,0x22 ); +outb (0x01 ,0x23 ); } +enable_apic_mode (); +} -verify_local_APIC (); +/** +* disconnect_bsp_APIC - detach the APIC from the interrupt system +* @virt_wire_setup: indicates, whether virtual wire mode is selected +* +* Virtual wire mode is necessary to deliver legacy interrupts even when the +* APIC is disabled. +*/ +void disconnect_bsp_APIC (int virt_wire_setup) +{ +if (pic_mode) { +/* +* Put the board back into PIC mode (has an effect only on +* certain older boards). Note that APIC interrupts, including +* IPIs, won't work beyond this point! The only exception are +* INIT IPIs. +*/ +apic_printk (APIC_VERBOSE ,"disabling APIC mode, " +"entering PIC mode.\n" ); +outb (0x70 ,0x22 ); +outb (0x00 ,0x23 ); + }else { +/* Go back to Virtual Wire compatibility mode */ +unsigned long value; -connect_bsp_APIC (); +/* For the spurious interrupt use vector F, and enable it */ + value =apic_read (APIC_SPIV ); + value &= ~APIC_VECTOR_MASK ; + value |=APIC_SPIV_APIC_ENABLED ; + value |=0xf ; +apic_write_around (APIC_SPIV , value); -/* -* Hack: In case of kdump, after a crash, kernel might be booting -* on a cpu with non-zero lapic id. But boot_cpu_physical_apicid -* might be zero if read from MP tables. Get it from LAPIC. -*/ -#ifdef CONFIG_CRASH_DUMP - boot_cpu_physical_apicid =GET_APIC_ID (apic_read (APIC_ID )); -#endif - phys_cpu_present_map =physid_mask_of_physid (boot_cpu_physical_apicid); +if (!virt_wire_setup) { +/* +* For LVT0 make it edge triggered, active high, +* external and enabled +*/ + value =apic_read (APIC_LVT0 ); + value &= ~(APIC_MODE_MASK |APIC_SEND_PENDING | +APIC_INPUT_POLARITY |APIC_LVT_REMOTE_IRR | +APIC_LVT_LEVEL_TRIGGER |APIC_LVT_MASKED ); + value |=APIC_LVT_REMOTE_IRR |APIC_SEND_PENDING ; + value =SET_APIC_DELIVERY_MODE (value,APIC_MODE_EXTINT ); +apic_write_around (APIC_LVT0 , value); + }else { +/* Disable LVT0 */ +apic_write_around (APIC_LVT0 ,APIC_LVT_MASKED ); + } -setup_local_APIC (); +/* +* For LVT1 make it edge triggered, active high, nmi and +* enabled +*/ + value =apic_read (APIC_LVT1 ); + value &= ~( +APIC_MODE_MASK |APIC_SEND_PENDING | +APIC_INPUT_POLARITY |APIC_LVT_REMOTE_IRR | +APIC_LVT_LEVEL_TRIGGER |APIC_LVT_MASKED ); + value |=APIC_LVT_REMOTE_IRR |APIC_SEND_PENDING ; + value =SET_APIC_DELIVERY_MODE (value,APIC_MODE_NMI ); +apic_write_around (APIC_LVT1 , value); + } +} -#ifdef CONFIG_X86_IO_APIC -if (smp_found_config) -if (!skip_ioapic_setup && nr_ioapics) -setup_IO_APIC (); -#endif -setup_boot_APIC_clock (); +/* +* Power management +*/ +#ifdef CONFIG_PM + +static struct { +int active; +/* r/w apic fields */ +unsigned int apic_id; +unsigned int apic_taskpri; +unsigned int apic_ldr; +unsigned int apic_dfr; +unsigned int apic_spiv; +unsigned int apic_lvtt; +unsigned int apic_lvtpc; +unsigned int apic_lvt0; +unsigned int apic_lvt1; +unsigned int apic_lvterr; +unsigned int apic_tmict; +unsigned int apic_tdcr; +unsigned int apic_thmr; +} apic_pm_state; + +static int lapic_suspend (struct sys_device *dev,pm_message_t state) +{ +unsigned long flags; + +if (!apic_pm_state.active) +return 0 ; + + apic_pm_state.apic_id =apic_read (APIC_ID ); + apic_pm_state.apic_taskpri =apic_read (APIC_TASKPRI ); + apic_pm_state.apic_ldr =apic_read (APIC_LDR ); + apic_pm_state.apic_dfr =apic_read (APIC_DFR ); + apic_pm_state.apic_spiv =apic_read (APIC_SPIV ); + apic_pm_state.apic_lvtt =apic_read (APIC_LVTT ); + apic_pm_state.apic_lvtpc =apic_read (APIC_LVTPC ); + apic_pm_state.apic_lvt0 =apic_read (APIC_LVT0 ); + apic_pm_state.apic_lvt1 =apic_read (APIC_LVT1 ); + apic_pm_state.apic_lvterr =apic_read (APIC_LVTERR ); + apic_pm_state.apic_tmict =apic_read (APIC_TMICT ); + apic_pm_state.apic_tdcr =apic_read (APIC_TDCR ); + apic_pm_state.apic_thmr =apic_read (APIC_LVTTHMR ); +local_irq_save (flags); +disable_local_APIC (); +local_irq_restore (flags);return 0 ; } -static int __init parse_lapic (char *arg) +static int lapic_resume (struct sys_device *dev) { -lapic_enable (); +unsigned int l, h; +unsigned long flags; + +if (!apic_pm_state.active) +return 0 ; + +local_irq_save (flags); + +/* +* Make sure the APICBASE points to the right address +* +* FIXME! This will be wrong if we ever support suspend on +* SMP! We'll need to do this as part of the CPU restore! +*/ +rdmsr (MSR_IA32_APICBASE , l, h); + l &= ~MSR_IA32_APICBASE_BASE ; + l |=MSR_IA32_APICBASE_ENABLE | mp_lapic_addr; +wrmsr (MSR_IA32_APICBASE , l, h); + +apic_write (APIC_LVTERR ,ERROR_APIC_VECTOR |APIC_LVT_MASKED ); +apic_write (APIC_ID , apic_pm_state.apic_id); +apic_write (APIC_DFR , apic_pm_state.apic_dfr); +apic_write (APIC_LDR , apic_pm_state.apic_ldr); +apic_write (APIC_TASKPRI , apic_pm_state.apic_taskpri); +apic_write (APIC_SPIV , apic_pm_state.apic_spiv); +apic_write (APIC_LVT0 , apic_pm_state.apic_lvt0); +apic_write (APIC_LVT1 , apic_pm_state.apic_lvt1); +apic_write (APIC_LVTTHMR , apic_pm_state.apic_thmr); +apic_write (APIC_LVTPC , apic_pm_state.apic_lvtpc); +apic_write (APIC_LVTT , apic_pm_state.apic_lvtt); +apic_write (APIC_TDCR , apic_pm_state.apic_tdcr); +apic_write (APIC_TMICT , apic_pm_state.apic_tmict); +apic_write (APIC_ESR ,0 ); +apic_read (APIC_ESR ); +apic_write (APIC_LVTERR , apic_pm_state.apic_lvterr); +apic_write (APIC_ESR ,0 ); +apic_read (APIC_ESR ); +local_irq_restore (flags);return 0 ; } -early_param ("lapic" , parse_lapic); -static int __init parse_nolapic (char *arg) +/* +* This device has no shutdown method - fully functioning local APICs +* are needed on every CPU up until machine_halt/restart/poweroff. +*/ + +static struct sysdev_class lapic_sysclass = { +set_kset_name ("lapic" ), + .resume = lapic_resume, + .suspend = lapic_suspend, +}; + +static struct sys_device device_lapic = { + .id =0 , + .cls = &lapic_sysclass, +}; + +static void __devinit apic_pm_activate (void ) { -lapic_disable (); -return 0 ; + apic_pm_state.active =1 ; } -early_param ("nolapic" , parse_nolapic); +static int __init init_lapic_sysfs (void ) +{ +int error; + +if (!cpu_has_apic) +return 0 ; +/* XXX: remove suspend/resume procs if !apic_pm_state.active? */ + + error =sysdev_class_register (&lapic_sysclass); +if (!error) + error =sysdev_register (&device_lapic); +return error; +} +device_initcall (init_lapic_sysfs); + +#else /* CONFIG_PM */ + +static void apic_pm_activate (void ) { } + +#endif /* CONFIG_PM */
arch/i386/kernel/io_apic.c +1 -1
@@ -1599,7 +1599,7 @@ void /*__init*/ print_local_APIC(void * v =apic_read (APIC_LVR );printk (KERN_INFO "... APIC VERSION: %08x\n" , v); ver =GET_APIC_VERSION (v); - maxlvt =get_maxlvt (); + maxlvt =lapic_get_maxlvt (); v =apic_read (APIC_TASKPRI );printk (KERN_DEBUG "... APIC TASKPRI: %08x (%02x)\n" , v, v &APIC_TPRI_MASK );
arch/i386/kernel/irq.c +18 -4
@@ -10,7 +10,6 @@* io_apic.c.) */ -#include <asm/uaccess.h> #include <linux/module.h> #include <linux/seq_file.h> #include <linux/interrupt.h>
@@ -19,19 +18,34 @@#include <linux/cpu.h> #include <linux/delay.h> +#include <asm/apic.h> +#include <asm/uaccess.h> +DEFINE_PER_CPU (irq_cpustat_t, irq_stat)____cacheline_internodealigned_in_smp ;EXPORT_PER_CPU_SYMBOL (irq_stat); -#ifndef CONFIG_X86_LOCAL_APIC /* * 'what should we do if we get a hw irq event on an illegal vector'. * each architecture has to answer this themselves. */ void ack_bad_irq (unsigned int irq) { -printk ("unexpected IRQ trap at vector %02x\n" , irq); -} +printk (KERN_ERR "unexpected IRQ trap at vector %02x\n" , irq); + +#ifdef CONFIG_X86_LOCAL_APIC +/* +* Currently unexpected vectors happen only on SMP and APIC. +* We _must_ ack these because every local APIC has only N +* irq slots per priority level, and a 'hanging, unacked' IRQ +* holds up an irq slot - in excessive cases (when multiple +* unexpected vectors occur) that might lock up the APIC +* completely. +* But only ack when the APIC is enabled -AK +*/ +if (cpu_has_apic) +ack_APIC_irq ();#endif +}#ifdef CONFIG_4KSTACKS /*
arch/i386/kernel/smpboot.c +2 -2
@@ -744,7 +744,7 @@ wakeup_secondary_cpu(int logical_apicid,/* * Due to the Pentium erratum 3AP. */ - maxlvt =get_maxlvt (); + maxlvt =lapic_get_maxlvt ();if (maxlvt >3 ) {apic_read_around (APIC_SPIV );apic_write (APIC_ESR ,0 );
@@ -834,7 +834,7 @@ wakeup_secondary_cpu(int phys_apicid, un*/ Dprintk ("#startup loops: %d.\n" , num_starts); - maxlvt =get_maxlvt (); + maxlvt =lapic_get_maxlvt ();for (j =1 ; j <= num_starts; j++) {Dprintk ("Sending STARTUP #%d.\n" ,j);
include/asm-i386/apic.h +1 -1
@@ -95,7 +95,7 @@ static inline void ack_APIC_irq(void)extern void (*wait_timer_tick)(void ); -extern int get_maxlvt (void ); +extern int lapic_get_maxlvt (void );extern void clear_local_APIC (void );extern void connect_bsp_APIC (void );extern void disconnect_bsp_APIC (int virt_wire_setup);
--
Re: [patch 08/19] i386: cleanup apic code
Quoted 5 lines
/*
* Knob to control our willingness to enable the local APIC.
+ *
+ * -1=force-disable, +1=force-enable
mind doing 2 defines for these? Makes things more readable I suspect
arch/i386/kernel/apic.c static cpumask_t timer_bcast_ipi; · line 138
Quoted 3 lines
arch/i386/kernel/apic.c +1
-1-return maxlvt; +return APIC_INTEGRATED (GET_APIC_VERSION (v)) ?GET_APIC_MAXLVT (v) :2 ; }
why not use lapic_is_integrated() here?
Review
Quoted 3 lines
\
(quoted diff) +2
-0+ if (cpu_has_tsc) + apic_printk(APIC_VERBOSE, "..... CPU clock speed is "
please put "approximated at" or something here; or people will call supportlines if they bought a 3.4Ghz processor and this shows 3.39999Ghz
Quoted 1 line
+EXPORT_SYMBOL(switch_APIC_timer_to_ipi);
why is this exported at all? Modules really shouldn't be touching apic level details....
this patch is extremely difficult to review because diff has made a mess out of it ;(
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Re: [patch 08/19] i386: cleanup apic code
Quoted 7 lines · Arjan van de Ven
Quoted 5 lines
/*
* Knob to control our willingness to enable the local APIC.
+ *
+ * -1=force-disable, +1=force-enablemind doing 2 defines for these? Makes things more readable I suspect
Yep.
Quoted 5 lines
arch/i386/kernel/apic.c static cpumask_t timer_bcast_ipi; · line 138
Quoted 3 lines
arch/i386/kernel/apic.c +1
-1-return maxlvt; +return APIC_INTEGRATED (GET_APIC_VERSION (v)) ?GET_APIC_MAXLVT (v) :2 ; }why not use
lapic_is_integrated()here?
oops.
Quoted 6 lines
Review
Quoted 3 lines
\
(quoted diff) +2
-0+ if (cpu_has_tsc) + apic_printk(APIC_VERBOSE, "..... CPU clock speed is "please put "approximated at" or something here; or people will call supportlines if they bought a 3.4Ghz processor and this shows 3.39999Ghz
:)
Quoted 4 lines
Quoted 1 line
+EXPORT_SYMBOL(switch_APIC_timer_to_ipi);
why is this exported at all? Modules really shouldn't be touching apic level details....
This is exported for ACPI to handle the C3 stops lapic hell.
tglx
Re: [patch 08/19] i386: cleanup apic code
Quoted 6 lines · Thomas Gleixner
Quoted 4 lines
Quoted 1 line
+EXPORT_SYMBOL(switch_APIC_timer_to_ipi);
why is this exported at all? Modules really shouldn't be touching apic level details....
This is exported for ACPI to handle the C3 stops lapic hell.
sounds that's so internal that it truely deserves to be a _GPL export
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
PATCH 9/19 i386: Convert to clock event devices
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Add clockevent drivers for i386: lapic (local) and PIT (global). Update the timer IRQ to call into the PIT driver's event handler and the lapic-timer IRQ to call into the lapic clockevent driver. The assignement of timer functionality is delegated to the core framework code and replaces the compile and runtime evalution in do_timer_interrupt_hook()
Use the clockevents broadcast support and implement the lapic_broadcast function for ACPI.
Build-fixes-from: Andrew Morton <[email protected]>
arch/i386/Kconfig +8 -0
@@ -18,6 +18,14 @@ config GENERIC_TIME bool default y +config GENERIC_CLOCKEVENTS + bool + default y + +config GENERIC_CLOCKEVENTS_BROADCAST + bool + default y + config LOCKDEP_SUPPORT bool default y
arch/i386/kernel/apic.c +162 -116
439 lines not on this page · raw message
arch/i386/kernel/i8253.c +84 -8
118 lines not on this page · raw message
arch/i386/kernel/time.c +0 -50
74 lines not on this page · raw message
include/asm-i386/i8253.h +20 -0
@@ -1,6 +1,26 @@#ifndef __ASM_I8253_H__#define __ASM_I8253_H__ +#include <linux/clockchips.h> +extern spinlock_t i8253_lock; +#ifdef CONFIG_HPET_TIMER +extern struct clock_event_device *global_clock_event; +#else +extern struct clock_event_device pit_clockevent; +# define global_clock_event (&pit_clockevent) +#endif + +/** +* pit_interrupt_hook - hook into timer tick +* @regs: standard registers from interrupt +* +* Call the global clock event handler. +**/ +static inline void pit_interrupt_hook (struct pt_regs *regs) +{ + global_clock_event->event_handler (regs); +} +#endif /* __ASM_I8253_H__ */
include/asm-i386/mach-default/do_timer.h +4 -22
@@ -1,36 +1,18 @@/* defines for inline arch setup functions */ +#include <linux/clockchips.h> -#include <asm/apic.h> #include <asm/i8259.h> +#include <asm/i8253.h> /** * do_timer_interrupt_hook - hook into timer tick -* @regs: standard registers from interrupt * -* Description: -* This hook is called immediately after the timer interrupt is ack'd. -* It's primary purpose is to allow architectures that don't possess -* individual per CPU clocks (like the CPU APICs supply) to broadcast the -* timer interrupt as a means of triggering reschedules etc. +* Call the pit clock event handler. see asm/i8253.h **/ static inline void do_timer_interrupt_hook (void ) { -do_timer (1 ); -#ifndef CONFIG_SMP -update_process_times (user_mode_vm (get_irq_regs ())); -#endif -/* -* In the SMP case we use the local APIC timer interrupt to do the -* profiling, except when we simulate SMP mode on a uniprocessor -* system, in that case we have to call the local interrupt handler. -*/ -#ifndef CONFIG_X86_LOCAL_APIC -profile_tick (CPU_PROFILING ); -#else -if (!using_apic_timer) -smp_local_timer_interrupt (); -#endif +pit_interrupt_hook (get_irq_regs ()); }
include/asm-i386/mach-voyager/do_timer.h +10 -5
@@ -1,13 +1,18 @@/* defines for inline arch setup functions */ +#include <linux/clockchips.h> +#include <asm/voyager.h> +#include <asm/i8253.h> +/** +* do_timer_interrupt_hook - hook into timer tick +* @regs: standard registers from interrupt +* +* Call the pit clock event handler. see asm/i8253.h +**/ static inline void do_timer_interrupt_hook (void ) { -do_timer (1 ); -#ifndef CONFIG_SMP -update_process_times (user_mode_vm (irq_regs)); -#endif - +pit_interrupt_hook (get_irq_regs ());voyager_timer_interrupt (); }
Documentation/kernel-parameters.txt +5 -0
@@ -748,6 +748,11 @@ and is between 256 and 4096 characters. lapic [IA-32,APIC] Enable the local APIC even if BIOS disabled it. + lapictimer [IA-32,APIC] Enable the local APIC timer on UP + systems for high resulution timers and dynticks. + This only has an effect when the local APIC is + available. It does not imply the "lapic" option. + lasi= [HW,SCSI] PARISC LASI driver for the 53c700 chip Format: addr:<io>,irq:<irq>
arch/i386/kernel/smpboot.c +1 -4
@@ -438,9 +438,7 @@ static void __devinit smp_callin(void)/* * Save our processor parameters */ -smp_store_cpu_info (cpuid); - -disable_APIC_timer (); +smp_store_cpu_info (cpuid);/* * Allow the master to continue.
@@ -557,7 +555,6 @@ static void __devinit start_secondary(voenable_NMI_through_LVT0 (NULL );enable_8259A_irq (0 ); } -enable_APIC_timer ();/* * low-memory mappings have been cleared, flush them from * the local TLBs too.
include/asm-i386/apic.h +1 -4
@@ -111,13 +111,10 @@ extern void smp_local_timer_interrupt (vextern void setup_boot_APIC_clock (void );extern void setup_secondary_APIC_clock (void );extern int APIC_init_uniprocessor (void ); -extern void disable_APIC_timer (void ); -extern void enable_APIC_timer (void ); -static inline void lapic_timer_idle_broadcast (int broadcast) { } +extern void lapic_timer_idle_broadcast (int broadcast);extern void enable_NMI_through_LVT0 (void * dummy); -void smp_send_timer_broadcast_ipi (void );void switch_APIC_timer_to_ipi (void *cpumask);void switch_ipi_to_APIC_timer (void *cpumask);#define ARCH_APICTIMER_STOPS_ON_C3 1
include/asm-i386/mpspec.h +0 -1
@@ -23,7 +23,6 @@ extern struct mpc_config_intsrc mp_irqsextern int mpc_default_type;extern unsigned long mp_lapic_addr;extern int pic_mode; -extern int using_apic_timer;#ifdef CONFIG_ACPI extern void mp_register_lapic (u8 id,u8 enabled);
--
Re: [patch 09/19] i386: Convert to clock event devices
arch/i386/kernel/apic.c void __init setup_boot_APIC_clock(void) · line 453
Quoted 6 lines · Thomas Gleixner
- * Local timer interrupt handler. It does both profiling and
- * process statistics/rescheduling.
+ * The guts of the apic timer interrupt
*/arch/i386/kernel/apic.c +1
-1-inline void smp_local_timer_interrupt (void ) +fastcall void local_apic_timer_interrupt(struct pt_regs *regs)
please don't add more "fastcall"; CONFIG_REGPARM makes that the default anyway
arch/i386/kernel/i8253.c line 109
Quoted 12 lines
arch/i386/kernel/i8253.c +12
-0+void __init setup_pit_timer (void ) +{ + pit_clockevent.mult =div_sc (CLOCK_TICK_RATE ,NSEC_PER_SEC ,32 ); + pit_clockevent.max_delta_ns = +clockevent_delta2ns (0x7FFF , &pit_clockevent); + pit_clockevent.min_delta_ns = +clockevent_delta2ns (0xF , &pit_clockevent); +register_global_clockevent (&pit_clockevent); +#ifdef CONFIG_HPET_TIMER + global_clock_event = &pit_clockevent; +#endif +}
ok this ifdef looks really really weird to me. Why does PIT code depend on CONFIG_HPET ? HPET is mostly a runtime property!
other than that Acked-by: Arjan van de Ven <[email protected]>
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
PATCH 10/19 PM_timer: allow early access and move externs to a header file
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Allow early access to the power management timer by exposing the verified read function and providing a helper function which checks the pmtmr_ioport variable and returns either the pm timer readout or 0 in case the pm timer is not available.
Create a new header file and replace also the ifdef'ed extern definition in arch/i386/kernel/acpi/boot.c
This is a preperatory patch for the rework of the local apic timer calibration.
No functional changes.
arch/i386/kernel/acpi/boot.c +1 -4
@@ -25,6 +25,7 @@#include <linux/init.h> #include <linux/acpi.h> +#include <linux/acpi_pmtmr.h> #include <linux/efi.h> #include <linux/cpumask.h> #include <linux/module.h>
@@ -702,10 +703,6 @@ static int __init acpi_parse_hpet(unsign#define acpi_parse_hpet NULL#endif -#ifdef CONFIG_X86_PM_TIMER -extern u32 pmtmr_ioport; -#endif -static int __init acpi_parse_fadt (unsigned long phys,unsigned long size) {struct fadt_descriptor *fadt =NULL ;
drivers/clocksource/acpi_pm.c +9 -8
@@ -16,15 +16,13 @@* This file is licensed under the GPL v2. */ +#include <linux/acpi_pmtmr.h> #include <linux/clocksource.h> #include <linux/errno.h> #include <linux/init.h> #include <linux/pci.h> #include <asm/io.h> -/* Number of PMTMR ticks expected during calibration run */ -#define PMTMR_TICKS_PER_SEC 3579545 -/* * The I/O port the PMTMR resides at. * The location is detected during setup_arch(),
@@ -32,15 +30,13 @@*/ u32 pmtmr_ioport__read_mostly ; -#define ACPI_PM_MASK CLOCKSOURCE_MASK(24)/* limit it to 24 bits */ -static inline u32 read_pmtmr (void ) {/* mask the output to 24 bits */ return inl (pmtmr_ioport) &ACPI_PM_MASK ; } -static cycle_t acpi_pm_read_verified (void ) +u32 acpi_pm_read_verified (void ) {u32 v1 =0 , v2 =0 , v3 =0 ;
@@ -57,7 +53,12 @@ static cycle_t acpi_pm_read_verified(voi }while (unlikely ((v1 > v2 && v1 < v3) || (v2 > v3 && v2 < v1) || (v3 > v1 && v3 < v2))); -return (cycle_t )v2; +return v2; +} + +static cycle_t acpi_pm_read_slow (void ) +{ +return (cycle_t )acpi_pm_read_verified (); }static cycle_t acpi_pm_read (void )
@@ -87,7 +88,7 @@ __setup("acpi_pm_good", acpi_pm_good_setstatic inline void acpi_pm_need_workaround (void ) { - clocksource_acpi_pm.read = acpi_pm_read_verified; + clocksource_acpi_pm.read = acpi_pm_read_slow; clocksource_acpi_pm.rating =110 ; }
include/linux/acpi_pmtmr.h new file +38 -0
@@ -0,0 +1,38 @@ +#ifndef _ACPI_PMTMR_H_ +#define _ACPI_PMTMR_H_ + +#include <linux/clocksource.h> + +/* Number of PMTMR ticks expected during calibration run */ +#define PMTMR_TICKS_PER_SEC 3579545 + +/* limit it to 24 bits */ +#define ACPI_PM_MASK CLOCKSOURCE_MASK(24) + +/* Overrun value */ +#define ACPI_PM_OVRRUN 1<<24 + +#ifdef CONFIG_X86_PM_TIMER + +extern u32 acpi_pm_read_verified (void ); +extern u32 pmtmr_ioport; + +static inline u32 acpi_pm_read_early (void ) +{ +if (!pmtmr_ioport) +return 0 ; +/* mask the output to 24 bits */ +return acpi_pm_read_verified (); +} + +#else + +static inline u32 acpi_pm_read_early (void ) +{ +return 0 ; +} + +#endif + +#endif +
--
Re: [patch 10/19] PM_timer: allow early access and move externs to a header file
include/linux/acpi_pmtmr.h line 13
Quoted 2 lines · Thomas Gleixner
include/linux/acpi_pmtmr.h new file +2
-0+/* Overrun value */ +#define ACPI_PM_OVRRUN 1<<24
technically the PM timer can be either 24 or 32 bits and you can find out at runtime; 24 is safe value though...
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
PATCH 11/19 i386: Rework local APIC calibration
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
The local apic timer calibration has two problem cases:
1. The calibration is based on readout of the PIT/HPET timer to detect the wrap of the periodic tick. It happens that a box gets stuck in the calibration loop due to a PIT with a broken readout function.
2. CoreDuo boxen show a sporadic PIT runs too slow defect, which results in a wrong lapic calibration. The PIT goes back to normal operation once the lapic timer is switched to periodic mode.
Rework the code to address both problems:
- Make the calibration interrupt driven. This removes the wait_timer_tick magic hackery from lapic.c and time_hpet.c. The clockevents framework allows easy substitution of the global tick event handler for the calibration. This is more accurate than monitoring jiffies. At this point of the boot process, nothing disturbes the interrupt delivery, so the results are very accurate.
- Verify the calibration against the PM timer, when available by using the early access function. When the measured calibration period is outside of an one percent window, then the lapic timer calibration is adjusted to the pm timer result.
- Verify the calibration by running the lapic timer with the calibration handler. Disable lapic timer in case of deviation.
This also removes the "synchronization" of the local apic timer to the global tick. This synchronization never worked, as there is no way to synchronize PIT(HPET) and local APIC timer. The synchronization by waiting for the tick just alignes the local APIC timer for the first events, but later the events drift away due to the different clocks. Removing the "sync" is just randomizing the asynchronous behaviour at setup time.
arch/i386/kernel/apic.c +204 -130
@@ -26,6 +26,7 @@#include <linux/sysdev.h> #include <linux/cpu.h> #include <linux/clockchips.h> +#include <linux/acpi_pmtmr.h> #include <linux/module.h> #include <asm/atomic.h>
@@ -163,64 +164,8 @@ int lapic_get_maxlvt(void)* Local APIC timer */ -/* -* This part sets up the APIC 32 bit clock in LVTT1, with HZ interrupts -* per second. We assume that the caller has already set up the local -* APIC. -* -* The APIC timer is not exactly sync with the external timer chip, it -* closely follows bus clocks. -*/ - -/* -* FIXME: Move this to i8253.h. There is no need to keep the access to -* the PIT scattered all around the place -tglx -*/ - -/* -* The timer chip is already set up at HZ interrupts per second here, -* but we do not accept timer interrupts yet. We only allow the BP -* to calibrate. -*/ -static unsigned int __devinit get_8254_timer_count (void ) -{ -unsigned long flags; - -unsigned int count; - -spin_lock_irqsave (&i8253_lock, flags); - -outb_p (0x00 ,PIT_MODE ); - count =inb_p (PIT_CH0 ); - count |=inb_p (PIT_CH0 ) <<8 ; - -spin_unlock_irqrestore (&i8253_lock, flags); - -return count; -} - -/* next tick in 8254 can be caught by catching timer wraparound */ -static void __devinit wait_8254_wraparound (void ) -{ -unsigned int curr_count, prev_count; - - curr_count =get_8254_timer_count (); -do { - prev_count = curr_count; - curr_count =get_8254_timer_count (); - -/* workaround for broken Mercury/Neptune */ -if (prev_count >= curr_count +0x100 ) - curr_count =get_8254_timer_count (); - - }while (prev_count >= curr_count); -} - -/* -* Default initialization for 8254 timers. If we use other timers like HPET, -* we override this later -*/ -void (*wait_timer_tick)(void )__devinitdata = wait_8254_wraparound; +/* Clock divisor is set to 16 */ +#define APIC_DIVISOR 16/* * This function sets up the local APIC timer, with a timeout of
@@ -232,9 +177,6 @@ void (*wait_timer_tick)(void) __devinitd* We do reads before writes even if unnecessary, to get around the * P5 APIC double write bug. */ - -#define APIC_DIVISOR 16 -static void __setup_APIC_LVTT (unsigned int clocks,int oneshot,int irqen) {unsigned int lvtt_value, tmp_value;
@@ -312,112 +254,244 @@ static void __devinit setup_APIC_timer(v }/* -* In this function we calibrate APIC bus clocks to the external -* timer. Unfortunately we cannot use jiffies and the timer irq -* to calibrate, since some later bootup code depends on getting -* the first irq? Ugh. +* In this functions we calibrate APIC bus clocks to the external timer. +* +* We want to do the calibration only once since we want to have local timer +* irqs syncron. CPUs connected by the same APIC bus have the very same bus +* frequency. * -* TODO: Fix this rather than saying "Ugh" -tglx +* This was previously done by reading the PIT/HPET and waiting for a wrap +* around to find out, that a tick has elapsed. I have a box, where the PIT +* readout is broken, so it never gets out of the wait loop again. This was +* also reported by others. * -* We want to do the calibration only once since we -* want to have local timer irqs syncron. CPUs connected -* by the same APIC bus have the very same bus frequency. -* And we want to have irqs off anyways, no accidental -* APIC irq that way. +* Monitoring the jiffies value is inaccurate and the clockevents +* infrastructure allows us to do a simple substitution of the interrupt +* handler. +* +* The calibration routine also uses the pm_timer when possible, as the PIT +* happens to run way too slow (factor 2.3 on my VAIO CoreDuo, which goes +* back to normal later in the boot process). */ -static int __init calibrate_APIC_clock (void ) +#define LAPIC_CAL_LOOPS (HZ/10) + +static __initdata volatile int lapic_cal_loops =-1 ; +static __initdata long lapic_cal_t1, lapic_cal_t2; +static __initdata unsigned long long lapic_cal_tsc1, lapic_cal_tsc2; +static __initdata unsigned long lapic_cal_pm1, lapic_cal_pm2; +static __initdata unsigned long lapic_cal_j1, lapic_cal_j2; + +/* +* Temporary interrupt handler. +*/ +static void __init lapic_cal_handler (struct pt_regs *regs) { -unsigned long long t1 =0 , t2 =0 ; -long tt1, tt2; -long result; -int i; -const int LOOPS =HZ /10 ; +unsigned long long tsc =0 ; +long tapic =apic_read (APIC_TMCCT ); +unsigned long pm =acpi_pm_read_early (); -apic_printk (APIC_VERBOSE ,"calibrating APIC timer ...\n" ); +if (cpu_has_tsc) +rdtscll (tsc); -/* -* Put whatever arbitrary (but long enough) timeout -* value into the APIC clock, we just want to get the -* counter running for calibration. -*/ -__setup_APIC_LVTT (1000000000 ,0 ,0 ); +switch (lapic_cal_loops++) { +case 0 : + lapic_cal_t1 = tapic; + lapic_cal_tsc1 = tsc; + lapic_cal_pm1 = pm; + lapic_cal_j1 = jiffies; +break ; -/* -* The timer chip counts down to zero. Let's wait -* for a wraparound to start exact measurement: -* (the current tick might have been already half done) -*/ +case LAPIC_CAL_LOOPS : + lapic_cal_t2 = tapic; + lapic_cal_tsc2 = tsc; +if (pm < lapic_cal_pm1) + pm +=ACPI_PM_OVRRUN ; + lapic_cal_pm2 = pm; + lapic_cal_j2 = jiffies; +break ; + } +} -wait_timer_tick (); +/* +* Setup the boot APIC +* +* Calibrate and verify the result. +*/ +void __init setup_boot_APIC_clock (void ) +{ +struct clock_event_device *levt = &__get_cpu_var (lapic_events); +const long pm_100ms =PMTMR_TICKS_PER_SEC /10 ; +const long pm_thresh = pm_100ms/100 ; +void (*real_handler)(struct pt_regs *regs); +unsigned long deltaj; +long delta, deltapm; +cpumask_t cpumask; -/* -* We wrapped around just now. Let's start: -*/ -if (cpu_has_tsc) -rdtscll (t1); - tt1 =apic_read (APIC_TMCCT ); +apic_printk (APIC_VERBOSE ,"Using local APIC timer interrupts.\n" +"calibrating APIC timer ...\n" ); + +/* Register broadcast function */ +clockevents_register_broadcast (lapic_timer_broadcast);/* -* Let's wait LOOPS wraprounds: +* Enable the apic timer next event capability only for +* SMP and on UP, when requested via commandline */ -for (i =0 ; i <LOOPS ; i++) -wait_timer_tick (); +if (num_possible_cpus () >1 || enable_local_apic_timer) + lapic_clockevent.capabilities |=CLOCK_CAP_NEXTEVT ; - tt2 =apic_read (APIC_TMCCT ); -if (cpu_has_tsc) -rdtscll (t2); +local_irq_disable (); + +/* Replace the global interrupt handler */ + real_handler = global_clock_event->event_handler; + global_clock_event->event_handler = lapic_cal_handler;/* -* The APIC bus clock counter is 32 bits only, it -* might have overflown, but note that we use signed -* longs, thus no extra care needed. -* -* underflown to be exact, as the timer counts down ;) +* Setup the APIC counter to 1e9. There is no way the lapic +* can underflow in the 100ms detection time frame */ +__setup_APIC_LVTT (1000000000 ,0 ,0 ); + +/* Let the interrupts run */ +local_irq_enable (); + +while (lapic_cal_loops <=LAPIC_CAL_LOOPS ); + +local_irq_disable (); + +/* Restore the real event handler */ + global_clock_event->event_handler = real_handler; + +/* Build delta t1-t2 as apic timer counts down */ + delta = lapic_cal_t1 - lapic_cal_t2; +apic_printk (APIC_VERBOSE ,"... lapic delta = %ld\n" , delta); + +/* Check, if the PM timer is available */ + deltapm = lapic_cal_pm2 - lapic_cal_pm1; +apic_printk (APIC_VERBOSE ,"... PM timer delta = %ld\n" , deltapm); - result = (tt1-tt2)*APIC_DIVISOR /LOOPS ; +if (deltapm) { +unsigned long mult; +u64 res; + + mult =clocksource_hz2mult (PMTMR_TICKS_PER_SEC ,22 ); + +if (deltapm > (pm_100ms - pm_thresh) && + deltapm < (pm_100ms + pm_thresh)) { +apic_printk (APIC_VERBOSE ,"... PM timer result ok\n" ); + }else { + res = (((u64 ) deltapm) * mult) >>22 ; +do_div (res,1000000 ); +printk (KERN_WARNING "APIC calibration not consistent " +"with PM Timer: %ldms instead of 100ms\n" , + (long )res); +/* Correct the lapic counter value */ + res = (((u64 ) delta ) * pm_100ms); +do_div (res, deltapm); +printk (KERN_INFO "APIC delta adjusted to PM-Timer: " +"%lu (%ld)\n" , (unsigned long ) res, delta); + delta = (long ) res; + } + }/* Calculate the scaled math multiplication factor */ - lapic_clockevent.mult =div_sc (tt1-tt2,TICK_NSEC *LOOPS ,32 ); + lapic_clockevent.mult =div_sc (delta,TICK_NSEC *LAPIC_CAL_LOOPS ,32 ); lapic_clockevent.max_delta_ns =clockevent_delta2ns (0x7FFFFF , &lapic_clockevent); lapic_clockevent.min_delta_ns =clockevent_delta2ns (0xF , &lapic_clockevent); -apic_printk (APIC_VERBOSE ,"..... tt1-tt2 %ld\n" , tt1 - tt2); + calibration_result = (delta *APIC_DIVISOR ) /LAPIC_CAL_LOOPS ; + +apic_printk (APIC_VERBOSE ,"..... delta %ld\n" , delta);apic_printk (APIC_VERBOSE ,"..... mult: %ld\n" , lapic_clockevent.mult); -apic_printk (APIC_VERBOSE ,"..... calibration result: %ld\n" , result); +apic_printk (APIC_VERBOSE ,"..... calibration result: %u\n" , + calibration_result); -if (cpu_has_tsc) +if (cpu_has_tsc) { + delta = (long )(lapic_cal_tsc2 - lapic_cal_tsc1);apic_printk (APIC_VERBOSE ,"..... CPU clock speed is " -"%ld.%04ld MHz.\n" , - ((long )(t2-t1)/LOOPS )/(1000000 /HZ ), - ((long )(t2-t1)/LOOPS )%(1000000 /HZ )); +"%ld.%04ld MHz.\n" , + (delta /LAPIC_CAL_LOOPS ) / (1000000 /HZ ), + (delta /LAPIC_CAL_LOOPS ) % (1000000 /HZ )); + }apic_printk (APIC_VERBOSE ,"..... host bus clock speed is " -"%ld.%04ld MHz.\n" , - result/(1000000 /HZ ), - result%(1000000 /HZ )); +"%u.%04u MHz.\n" , + calibration_result / (1000000 /HZ ), + calibration_result % (1000000 /HZ )); -return result; -} -void __init setup_boot_APIC_clock (void ) -{ -unsigned long flags; -apic_printk (APIC_VERBOSE ,"Using local APIC timer interrupts.\n" ); - using_apic_timer =1 ; - -local_irq_save (flags); +apic_printk (APIC_VERBOSE ,"... verify APIC timer\n" ); - calibration_result =calibrate_APIC_clock ();/* -* Now set up the timer for real. +* Start LAPIC timer and verify that the calculated factor is correct */ setup_APIC_timer (); -local_irq_restore (flags); +/* Replace the lapic interrupt handler */ + real_handler = levt->event_handler; + levt->event_handler = lapic_cal_handler; + lapic_cal_loops =-1 ; + +/* Let the interrupts run */ +local_irq_enable (); + +while (lapic_cal_loops <=LAPIC_CAL_LOOPS ); + +local_irq_disable (); + +/* Restore the real event handler */ + levt->event_handler = real_handler; + +local_irq_enable (); + +/* Jiffies delta */ + deltaj = lapic_cal_j2 - lapic_cal_j1; +apic_printk (APIC_VERBOSE ,"... jiffies delta = %lu\n" , deltaj); + +/* Check, if the PM timer is available */ + deltapm = lapic_cal_pm2 - lapic_cal_pm1; +apic_printk (APIC_VERBOSE ,"... PM timer delta = %ld\n" , deltapm); + +if (deltapm) { +if (deltapm > (pm_100ms - pm_thresh) && + deltapm < (pm_100ms + pm_thresh)) { +apic_printk (APIC_VERBOSE ,"... PM timer result ok\n" ); +/* Check, if the jiffies result is consistent */ +if (deltaj <LAPIC_CAL_LOOPS -2 || + deltaj >LAPIC_CAL_LOOPS +2 ) { +/* +* Not sure, what we can do about this one. +* When high resultion timers are active +* and the lapic timer does not stop in C3 +* we are fine. Otherwise more trouble might +* be waiting. -- tglx +*/ +printk (KERN_WARNING "Global event device %s " +"has wrong frequency " +"(%lu ticks instead of %d)\n" , + global_clock_event->name, deltaj, +LAPIC_CAL_LOOPS ); + } +return ; + } + }else { +/* Check, if the jiffies result is consistent */ +if (deltaj >=LAPIC_CAL_LOOPS -2 && + deltaj <=LAPIC_CAL_LOOPS +2 ) { +apic_printk (APIC_VERBOSE ,"... jiffies result ok\n" ); +return ; + } + } + +printk (KERN_WARNING +"APIC timer disabled due to verification failure.\n" ); +local_irq_disable (); + cpumask =cpumask_of_cpu (smp_processor_id ()); +switch_APIC_timer_to_ipi (&cpumask); +local_irq_enable (); }void __devinit setup_secondary_APIC_clock (void )
arch/i386/kernel/time_hpet.c +0 -22
@@ -43,23 +43,6 @@ static void hpet_writel(unsigned long d,writel (d, hpet_virt_address + a); } -#ifdef CONFIG_X86_LOCAL_APIC -/* -* HPET counters dont wrap around on every tick. They just change the -* comparator value and continue. Next tick can be caught by checking -* for a change in the comparator value. Used in apic.c. -*/ -static void __devinit wait_hpet_tick (void ) -{ -unsigned int start_cmp_val, end_cmp_val; - - start_cmp_val =hpet_readl (HPET_T0_CMP ); -do { - end_cmp_val =hpet_readl (HPET_T0_CMP ); - }while (start_cmp_val == end_cmp_val); -} -#endif -static int hpet_timer_stop_set_go (unsigned long tick) {unsigned int cfg;
@@ -213,11 +196,6 @@ int __init hpet_enable(void) hpet_alloc(&hd); }#endif - -#ifdef CONFIG_X86_LOCAL_APIC -if (hpet_use_timer) - wait_timer_tick = wait_hpet_tick; -#endif return 0 ; }
include/asm-i386/apic.h +0 -2
@@ -93,8 +93,6 @@ static inline void ack_APIC_irq(void)apic_write_around (APIC_EOI ,0 ); } -extern void (*wait_timer_tick)(void ); -extern int lapic_get_maxlvt (void );extern void clear_local_APIC (void );extern void connect_bsp_APIC (void );
--
Re: [patch 11/19] i386: Rework local APIC calibration
Quoted 2 lines · Thomas Gleixner
plain text document attachment (i386-lapic-calibrate-timer.patch)
From: Thomas Gleixner <[email protected]>
One question: why do the irq measurement at all if pmtimer is available?
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Re: [patch 11/19] i386: Rework local APIC calibration
Quoted 6 lines · Arjan van de Ven
Quoted 2 lines · Thomas Gleixner
plain text document attachment (i386-lapic-calibrate-timer.patch)
From: Thomas Gleixner <[email protected]>One question: why do the irq measurement at all if pmtimer is available?
Good point. OTOH this works everywhere.
tglx
Re: [patch 11/19] i386: Rework local APIC calibration
In reply to Arjan van de VenQuoted 6 lines · Arjan van de Ven
Quoted 2 lines · Thomas Gleixner
plain text document attachment (i386-lapic-calibrate-timer.patch)
From: Thomas Gleixner <[email protected]>One question: why do the irq measurement at all if pmtimer is available?
the pmtimer read will always return zero on platforms where there's no pm-timer clock - in this case the irq measurement is what calibrates.
Ingo
PATCH 12/19 high-res timers: core
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Add the core bits of high-res timers support.
The design makes use of the existing hrtimers subsystem which manages a per-CPU and per-clock tree of timers, and the clockevents framework, which provides a standard API to request programmable clock events from. The core code does not have to know about the clock details - it makes use of clockevents_set_next_event().
Once the preliminaries for high resolution mode (a continous time source for time keeping and a reprogrammable clock event device) are available, the hrtimer code is switched to high resolution mode. The per-cpu clock event devices are switched into one shot mode and on SMP systems an eventually available global clock event device (e.g. PIT on i386) is switched off. The periodic tick, which updates jiffies and calls update_process_times and profiling, is provided by a per-cpu hrtimer. The callback function is executed in the timer interrupt context. The hrtimer based implementation of the periodic tick is designed to be extended with dynamic tick functionality.
The impact to non-high-res architectures is intended to be minimal.
More detailed information is available in Documentation/hrtimer/highres.txt
Build-fixes-from: Valdis.Kletnieks <[email protected]>
Documentation/kernel-parameters.txt +4 -0
@@ -594,6 +594,10 @@ and is between 256 and 4096 characters. highmem otherwise. This also works to reduce highmem size on bigger boxes. + highres= [KNL] Enable/disable high resolution timer mode. + Valid parameters: "on", "off" + Default: "on" + hisax= [HW,ISDN] See Documentation/isdn/README.HiSax.
include/linux/hrtimer.h +95 -2
163 lines not on this page · raw message
include/linux/interrupt.h +4 -1
@@ -236,7 +236,10 @@ enumNET_TX_SOFTIRQ ,NET_RX_SOFTIRQ ,BLOCK_SOFTIRQ , - TASKLET_SOFTIRQ +TASKLET_SOFTIRQ , +#ifdef CONFIG_HIGH_RES_TIMERS + HRTIMER_SOFTIRQ, +#endif };/* softirq mask and active fields moved to irq_cpustat_t in
include/linux/ktime.h +1 -2
@@ -261,8 +261,7 @@ static inline u64 ktime_to_ns(const ktim* idea of the (in)accuracy of timers. Timer values are rounded up to * this resolution values. */ -#define KTIME_REALTIME_RES (ktime_t){ .tv64 = TICK_NSEC } -#define KTIME_MONOTONIC_RES (ktime_t){ .tv64 = TICK_NSEC } +#define KTIME_LOW_RES (ktime_t){ .tv64 = TICK_NSEC }/* Get the monotonic time in timespec format: */ extern void ktime_get_ts (struct timespec *ts);
kernel/hrtimer.c +646 -27
825 lines not on this page · raw message
kernel/itimer.c +1 -1
@@ -136,7 +136,7 @@ enum hrtimer_restart it_real_fn(struct hsend_group_sig_info (SIGALRM ,SEND_SIG_PRIV , sig->tsk);if (sig->it_real_incr.tv64 !=0 ) { -hrtimer_forward (timer, timer->base->softirq_time, +hrtimer_forward (timer,hrtimer_cb_get_time (timer), sig->it_real_incr);return HRTIMER_RESTART ; }
kernel/posix-timers.c +1 -1
@@ -356,7 +356,7 @@ static enum hrtimer_restart posix_timer_if (timr->it.real.interval.tv64 !=0 ) { timr->it_overrun +=hrtimer_forward (timer, - timer->base->softirq_time, +hrtimer_cb_get_time (timer), timr->it.real.interval); ret =HRTIMER_RESTART ; ++timr->it_requeue_pending;
kernel/time/Kconfig new file +11 -0
@@ -0,0 +1,11 @@ +# +# Timer subsystem related configuration options +# +config HIGH_RES_TIMERS + bool "High Resolution Timer Support" + depends on GENERIC_TIME && GENERIC_CLOCKEVENTS + help + This option enables high resolution timer support. If your + hardware is not capable then this option only increases + the size of the kernel image. +
kernel/timer.c +1 -0
@@ -1048,6 +1048,7 @@ static void update_wall_time(void)if (change_clocksource ()) { clock->error =0 ; clock->xtime_nsec =0 ; +hrtimer_clock_notify ();clocksource_calculate_interval (clock, tick_nsec); } }
--
Re: [patch 12/19] high-res timers: core
Quoted 8 lines · Thomas Gleixner
+ * hrtimer callback modes:
+ *
+ *HRTIMER_CB_SOFTIRQ: Callback must run in softirq context
+ *HRTIMER_CB_IRQSAFE: Callback may run in hardirq context
+ *HRTIMER_CB_IRQSAFE_NO_RESTART: Callback may run in hardirq context and
+ * does not restart the timer
+ *HRTIMER_CB_IRQSAFE_NO_SOFTIRQ: Callback must run in softirq context
+ * Special mode for tick emultation
This naming is treacherous (or the comment is wrong); NO_SOFTIRQ suggests "can't run in softirq" but your comment says "must run in softirq".. which is it?
kernel/hrtimer.c static unsigned long ktime_divns(const k · line 463
Quoted 11 lines
kernel/hrtimer.c +11
-0+/** +* hrtimer_clock_notify - A clock source or a clock event has been installed +* +* Notify the per cpu softirqs to recheck the clock sources and events +*/ +void hrtimer_clock_notify (void ) +{ +int i; + +if (hrtimer_hres_enabled) { +for_each_possible_cpu (i)
hmm. possible or online or ..
If you fix the comment/define:
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
PATCH 13/19 GTOD: Mark TSC unusable for highres timers
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
The TSC is too unstable and unreliable to be used with high resolution timers. The automatic detection of TSC unstability fails once we switched to high resolution mode, because the tick emulation would use the TSC as reference. This results in a circular dependency. Mark it unusable for high res upfront.
arch/i386/kernel/tsc.c renamed from arch/i386/kernel/tsc.c~gtod-mark-tsc-unusable-for-highres-timers +14 -1
@@ -459,10 +459,23 @@ static int __init init_tsc_clocksource(v current_tsc_khz = tsc_khz; clocksource_tsc.mult =clocksource_khz2mult (current_tsc_khz, clocksource_tsc.shift); +#ifndef CONFIG_HIGH_RES_TIMERS /* lower the rating if we already know its unstable: */ if (check_tsc_unstable ()) clocksource_tsc.rating =0 ; - +#else +/* +* Mark TSC unsuitable for high resolution timers. TSC has so +* many pitfalls: frequency changes, stop in idle ... When we +* switch to high resolution mode we can not longer detect a +* firmware caused frequency change, as the emulated tick uses +* TSC as reference. This results in a circular dependency. +* Switch only to high resolution mode, if pm_timer or such +* is available. +*/ + clocksource_tsc.rating =50 ; + clocksource_tsc.is_continuous =0 ; +#endif init_timer (&verify_tsc_freq_timer); verify_tsc_freq_timer.function = verify_tsc_freq; verify_tsc_freq_timer.expires =
_
--
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 31 lines · Thomas Gleixner
plain text document attachment
(gtod-mark-tsc-unusable-for-highres-timers.patch)
From: Thomas Gleixner <[email protected]>The TSC is too unstable and unreliable to be used with high resolution timers. The automatic detection of TSC unstability fails once we switched to high resolution mode, because the tick emulation would use the TSC as reference. This results in a circular dependency. Mark it unusable for high res upfront.
[[email protected]: updated for i386-time-avoid-pit-smp-lockups.patch]Signed-off-by Thomas Gleixner <[email protected]>Signed-off-by Ingo Molnar <[email protected]>diff -puN
arch/i386/kernel/tsc.c~gtod-mark-tsc-unusable-for-highres-timersarch/i386/kernel/tsc.carch/i386/kernel/tsc.c renamed from arch/i386/kernel/tsc.c~gtod-mark-tsc-unusable-for-highres-timers +7
-1@@ -459,10 +459,23 @@ static int __init init_tsc_clocksource(v current_tsc_khz = tsc_khz; clocksource_tsc.mult =clocksource_khz2mult (current_tsc_khz, clocksource_tsc.shift); +#ifndef CONFIG_HIGH_RES_TIMERS /* lower the rating if we already know its unstable: */ if (check_tsc_unstable ()) clocksource_tsc.rating =0 ; - +#else +/* +* Mark TSC unsuitable for high resolution timers. TSC has so +* many pitfalls: frequency changes, stop in idle ... When we +* switch to high resolution mode we can not longer detect a +* firmware caused frequency change, as the emulated tick uses
arch/i386/kernel/tsc.c static int __init init_tsc_clocksource(v · line 481
Quoted 10 lines
arch/i386/kernel/tsc.c renamed from arch/i386/kernel/tsc.c~gtod-mark-tsc-unusable-for-highres-timers +7
-0+* TSC as reference. This results in a circular dependency. +* Switch only to high resolution mode, if pm_timer or such +* is available. +*/ + clocksource_tsc.rating =50 ; + clocksource_tsc.is_continuous =0 ; +#endif init_timer (&verify_tsc_freq_timer); verify_tsc_freq_timer.function = verify_tsc_freq; verify_tsc_freq_timer.expires =
Hmmm. I wish this patch was unnecessary, but I don't see an easy solution.
Mind adding a warning so users know why a system that might use the TSC normally does not use the TSC w/ highres timers?
Otherwise looks ok.
thanks
-john
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
current_tsc_khz = tsc_khz;
Quoted 19 lines
Quoted 19 lines
arch/i386/kernel/tsc.c renamed from arch/i386/kernel/tsc.c~gtod-mark-tsc-unusable-for-highres-timers +13
-1clocksource_tsc.mult =clocksource_khz2mult (current_tsc_khz, clocksource_tsc.shift); +#ifndef CONFIG_HIGH_RES_TIMERS /* lower the rating if we already know its unstable: */ if (check_tsc_unstable ()) clocksource_tsc.rating =0 ; - +#else +/* +* Mark TSC unsuitable for high resolution timers. TSC has so +* many pitfalls: frequency changes, stop in idle ... When we +* switch to high resolution mode we can not longer detect a +* firmware caused frequency change, as the emulated tick uses +* TSC as reference. This results in a circular dependency. +* Switch only to high resolution mode, if pm_timer or such +* is available. +*/ + clocksource_tsc.rating =50 ; + clocksource_tsc.is_continuous =0 ;
Quoted 8 lines
arch/i386/kernel/tsc.c static int __init init_tsc_clocksource(v · line 481
Quoted 4 lines
arch/i386/kernel/tsc.c renamed from arch/i386/kernel/tsc.c~gtod-mark-tsc-unusable-for-highres-timers +1
-0+#endif init_timer (&verify_tsc_freq_timer); verify_tsc_freq_timer.function = verify_tsc_freq; verify_tsc_freq_timer.expires =Hmmm. I wish this patch was unnecessary, but I don't see an easy solution.
Very sad. This will make a lot of people unhappy, even to the point where they might prefer disabling noidlehz over super slow gettimeofday. I assume you at least have a suitable command line option for that, right?
Can we get a summary on which systems the TSC is considered unstable? Normally we assume if it's stable enough for gettimeofday it should be stable enough for longer delays too.
-Andi
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 10 lines · Andi Kleen
Quoted 6 lines
arch/i386/kernel/tsc.c static int __init init_tsc_clocksource(v · line 481
Quoted 2 lines
arch/i386/kernel/tsc.c renamed from arch/i386/kernel/tsc.c~gtod-mark-tsc-unusable-for-highres-timers +0
-0verify_tsc_freq_timer.function = verify_tsc_freq; verify_tsc_freq_timer.expires =Hmmm. I wish this patch was unnecessary, but I don't see an easy solution.
Very sad. This will make a lot of people unhappy, even to the point where they might prefer disabling noidlehz over super slow gettimeofday. I assume you at least have a suitable command line option for that, right?
Yes it is sad. And the sadest part is that AMD and Intel have been asked to fix that more than 5 years ago. They did not get their brain straight and now we are the dimwits.
Quoted 3 lines
Can we get a summary on which systems the TSC is considered unstable? Normally we assume if it's stable enough for gettimeofday it should be stable enough for longer delays too.
TSC is simply a nightmare:
- Frequency changes with CPU clock
- Unsynced across CPUs
- Stops in C3, which makes it completely unusable
Once you take away periodic interrupts it is simply broken. AMD and Intel can run in circels, it does not get better.
tglx
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 20 lines · On Fri, 10 Nov 2006 09:10:06 +0100 Thomas Gleixner <[email protected]> wrote:
Quoted 10 lines · Andi Kleen
Quoted 6 lines
arch/i386/kernel/tsc.c static int __init init_tsc_clocksource(v · line 481
Quoted 2 lines
arch/i386/kernel/tsc.c renamed from arch/i386/kernel/tsc.c~gtod-mark-tsc-unusable-for-highres-timers +0
-0verify_tsc_freq_timer.function = verify_tsc_freq; verify_tsc_freq_timer.expires =Hmmm. I wish this patch was unnecessary, but I don't see an easy solution.
Very sad. This will make a lot of people unhappy, even to the point where they might prefer disabling noidlehz over super slow gettimeofday. I assume you at least have a suitable command line option for that, right?
Yes it is sad. And the sadest part is that AMD and Intel have been asked to fix that more than 5 years ago. They did not get their brain straight and now we are the dimwits.
Quoted 3 lines
Can we get a summary on which systems the TSC is considered unstable? Normally we assume if it's stable enough for gettimeofday it should be stable enough for longer delays too.
Quoted 9 lines
TSC is simply a nightmare:
- Frequency changes with CPU clock
- Unsynced across CPUs
- Stops in C3, which makes it completely unusableOnce you take away periodic interrupts it is simply broken. AMD and Intel can run in circels, it does not get better.
What is the actual problem? verify_tsc_freq()?
If so, could that function use the PIT/pmtimer/etc for working out if the TSC is bust, rather than directly using jiffies?
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 2 lines · Andrew Morton
If so, could that function use the PIT/pmtimer/etc for working out if the TSC is bust, rather than directly using jiffies?
there's no realiable way to figure out the TSC is bust: some CPUs have a slight 'skew' between cores for example. On some systems the TSC might skew between sockets. A CPU might break its TSC only once some powersaving mode has been activated - which might be long after bootup. The whole TSC business is a nightmare and cannot be supported reliably. AFAIK Windows doesnt use it, so it's a continuous minefield for new hardware to break.
We should wait until CPU makers get their act together and implement a TSC variant that is /architecturally promised/ to have constant frequency (system bus frequency or whatever) and which never stops.
Ingo
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 6 lines · On Fri, 10 Nov 2006 09:57:28 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 2 lines · Andrew Morton
If so, could that function use the PIT/pmtimer/etc for working out if the TSC is bust, rather than directly using jiffies?
Quoted 7 lines
there's no realiable way to figure out the TSC is bust: some CPUs have a slight 'skew' between cores for example. On some systems the TSC might skew between sockets. A CPU might break its TSC only once some powersaving mode has been activated - which might be long after bootup. The whole TSC business is a nightmare and cannot be supported reliably. AFAIK Windows doesnt use it, so it's a continuous minefield for new hardware to break.
But that's different.
We're limping along in a semi-OK fashion with the TSC. But now Thomas is proposing that we effectively kill it off for all x86 because of hrtimers.
And afaict the reason for that is that we're using jiffies to determine if the TSC has gone bad, and that test is getting false positives.
Quoted 4 lines
We should wait until CPU makers get their act together and implement a TSC variant that is /architecturally promised/ to have constant frequency (system bus frequency or whatever) and which never stops.
That'll hurt the big machines rather a lot, won't it?
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 5 lines
But that's different.
We're limping along in a semi-OK fashion with the TSC. But now Thomas is proposing that we effectively kill it off for all x86 because of hrtimers.
I'm totally against that.
Quoted 2 lines
And afaict the reason for that is that we're using jiffies to determine if the TSC has gone bad, and that test is getting false positives.
The i386 clocksource had always trouble with that. e.g. I have a box where the TSC works perfectly fine on a 64bit kernel, but since the new i386 clocksource code is in it always insists on disabling it shortly after boot. My guess is that some of the checks in there are just broken and need to be fixed.
Quoted 7 lines
Quoted 5 lines
We should wait until CPU makers get their act together and implement a TSC variant that is /architecturally promised/ to have constant frequency (system bus frequency or whatever) and which never stops.
That'll hurt the big machines rather a lot, won't it?
It's unrealistic and short term it will cause extreme pain in many workloads which are gettimeofday intensive (networking, databases etc.)
-Andi
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 6 lines · Andi Kleen
Quoted 4 lines
But that's different.
We're limping along in a semi-OK fashion with the TSC. But now Thomas is proposing that we effectively kill it off for all x86 because of hrtimers.
I'm totally against that.
I'm working on that. The general disable is indeed overkill. All I need to prevent is to switch over to highres/dyntick in case that there is no fallback (e.g. pm_timer) available. Else I end up in a circular dependency as the emulated tick depends on the monotonic clock.
Quoted 8 lines
Quoted 2 lines
And afaict the reason for that is that we're using jiffies to determine if the TSC has gone bad, and that test is getting false positives.
The i386 clocksource had always trouble with that. e.g. I have a box where the TSC works perfectly fine on a 64bit kernel, but since the new i386 clocksource code is in it always insists on disabling it shortly after boot. My guess is that some of the checks in there are just broken and need to be fixed.
It's the unconditional mark_unstable call in ACPI C2 state. /me looks.
tglx
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
And afaict the reason for that is that we're using jiffies to determine if the TSC has gone bad, and that test is getting false positives.
The i386 clocksource had always trouble with that. e.g. I have a box where the TSC works perfectly fine on a 64bit kernel, but since the new i386 clocksource code is in it always insists on disabling it shortly after boot.
shortly after boot means in user space here, not during the first idling.
Quoted 4 lines
Quoted 2 lines
My guess is that some of the checks in there are just broken and need to be fixed.
It's the unconditional
mark_unstablecall in ACPI C2 state. /me looks.
The system doesn't support C2 states. It's an older single socket Athlon 64 with VIA chipset. I haven't looked in detail on why it fails.
-Andi
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 9 lines · Andi Kleen
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
And afaict the reason for that is that we're using jiffies to determine if the TSC has gone bad, and that test is getting false positives.
The i386 clocksource had always trouble with that. e.g. I have a box where the TSC works perfectly fine on a 64bit kernel, but since the new i386 clocksource code is in it always insists on disabling it shortly after boot.
shortly after boot means in user space here, not during the first idling.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
My guess is that some of the checks in there are just broken and need to be fixed.
It's the unconditional
mark_unstablecall in ACPI C2 state. /me looks.The system doesn't support C2 states. It's an older single socket Athlon 64 with VIA chipset. I haven't looked in detail on why it fails.
Does it have cpu freqency changing ?
tglx
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 10 lines · Thomas Gleixner
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
My guess is that some of the checks in there are just broken and need to be fixed.
It's the unconditional
mark_unstablecall in ACPI C2 state. /me looks.The system doesn't support C2 states. It's an older single socket Athlon 64 with VIA chipset. I haven't looked in detail on why it fails.
Does it have cpu freqency changing ?
Yep. But only OS controlled one (powernow).
Most likely it happens when ondemand starts doing its thing.
-Andi
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 8 lines · Andi Kleen
Quoted 6 lines · Thomas Gleixner
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
My guess is that some of the checks in there are just broken and need to be fixed.
It's the unconditional
mark_unstablecall in ACPI C2 state. /me looks.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
The system doesn't support C2 states. It's an older single socket Athlon 64 with VIA chipset. I haven't looked in detail on why it fails.
Does it have cpu freqency changing ?
Yep. But only OS controlled one (powernow).
Most likely it happens when ondemand starts doing its thing.
Yes, thats one of the criterias the tsc clocksource is using. I'm looking into that right now.
tglx
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
In reply to Andrew MortonQuoted 2 lines
We're limping along in a semi-OK fashion with the TSC.
that's because we fake it a heck of a lot; like after C3 we just make the kernel guestimate how much to progress it so that it has just enough ductape on it to not totally fall apart ;(
There's no easy answer. We can keep trying to ductape the TSC everywhere it sort of breaks (cpu frequency changes on older chips, C3 idle (which old kernels hit less often just because of the constant timer ticks), cross cpu drifts and offsets etc etc).
What that would need at minimum is
1) a per cpu "offset" that gets added to whatever we read from rdtsc instruction
2) a per cpu "multiplier" or something that gets applied to tsc deltas
3) all code that gets to mop up where TSC breaks (cpuspeed and C3 power states) use "other timers" to adjust the offset/multiplier values on a per cpu basis, rather than "hardware TSC".
I suspect that is enough to mostly keep it limping along. It's not cheap, but it moves the costs mostly to the places where the hardware can't do it, so if you want to call gettimeofday() in a tight loop at least you don't pay the hpet tax. (only an add and maybe a mul but those are cheap and effectively unavoidable if we want to keep the illusion alive)
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 6 lines · Arjan van de Ven
Quoted 2 lines
We're limping along in a semi-OK fashion with the TSC.
that's because we fake it a heck of a lot; like after C3 we just make the kernel guestimate how much to progress it so that it has just enough ductape on it to not totally fall apart ;(
Do we? Where? AFAIK we just do some resetting after cpu frequency changes, but on C3 TSC is just disabled globally.
That is better than it sounds.
Most systems don't have C3 right now. And on those that have (laptops) it tends to be not that critical because they normally don't run workload where gettimeofday() is really time critical (and nobody expects them to be particularly fast anyways)
[... proposal for per CPU TSC state snipped ...]
All is being worked on.
-Andi
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 6 lines
Do we? Where? AFAIK we just do some resetting after cpu frequency changes, but on C3 TSC is just disabled globally.
That is better than it sounds.
is it?
Quoted 5 lines
Most systems don't have C3 right now. And on those that have (laptops) it tends to be not that critical because they normally don't run workload where
gettimeofday()is really time critical (and nobody expects them to be particularly fast anyways)
and that got changed when the blade people decided to start using laptop processors ......
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 7 lines · Arjan van de Ven
Quoted 4 lines
Most systems don't have C3 right now. And on those that have (laptops) it tends to be not that critical because they normally don't run workload where
gettimeofday()is really time critical (and nobody expects them to be particularly fast anyways)and that got changed when the blade people decided to start using laptop processors ......
and some systems disable the lapic in C2 already: BIOSs started doing lowlevel-C3 in their C2 functionality and lie to the OS about it.
Ingo
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
In reply to Arjan van de VenQuoted 8 lines · Arjan van de Ven
Quoted 5 lines
Most systems don't have C3 right now. And on those that have (laptops) it tends to be not that critical because they normally don't run workload where
gettimeofday()is really time critical (and nobody expects them to be particularly fast anyways)and that got changed when the blade people decided to start using laptop processors ......
Well those will be handled eventually. Currently they just have a slower gettimeofday.
But the majority of systems is not impacted.
BTW if someone really wants to have fast gettimeofday on a blade they can just disable C3 and force TSC.
-Andi
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
In reply to Ingo MolnarQuoted 8 lines · Ingo Molnar
Quoted 2 lines · Andrew Morton
If so, could that function use the PIT/pmtimer/etc for working out if the TSC is bust, rather than directly using jiffies?
there's no realiable way to figure out the TSC is bust: some CPUs have a slight 'skew' between cores for example.
We find this out by black listing them. I got that working reliably as far as I know.
The main cases I know where we can't use it right now is:
- AMD >1 core
* when clock ramping is disabled it gets a little better, but on multi socket it is still broken
* also it varies in frequency here which has to be handled
+ There is a little issue here that the frequency takes some unpredictible time to stabilize after the frequency change. AFAIK the error is too small to cause problems though.
- Some Intel NUMA systems (IBM x4xx, Unisys ES7000, ScaleMP)
* handled by detecting multiple Apic Clusters
- Intel systems with C3
* stops in C3. disable here
- a few P4 dual cores seem to lose TSC synchronization when overclocked (or most likely overvolted) and running out of Spec
* I chose to ignore this case. User fault. They can set command line options.
- We had one Intel BIOS which misprogrammed the FSB dividers
* Got fixed by BIOS update. Also it was a obscure case that can be handled with command line options.
I don't see how this is changing much with dyntimers. The only difference that should be there is that you require TSC stability for a longer time (instead of only HZ), but normally when the TSC is instable it already causes trouble in the current setup.
You're probably overreacting to something. Maybe one of the old bugs? (I had a typo in the Intel C3 detection for a long time that broke a lot of Intel laptops)
Quoted 4 lines
On some systems the TSC might
skew between sockets. A CPU might break its TSC only once some powersaving mode has been activated - which might be long after bootup. The whole TSC business is a nightmare and cannot be supported reliably.
I disagree.
Quoted 2 lines
AFAIK Windows doesnt use it, so it's a continuous minefield for new hardware to break.
Not true.
Quoted 3 lines
We should wait until CPU makers get their act together and implement a TSC variant that is /architecturally promised/ to have constant frequency (system bus frequency or whatever)
Intel already has that (modulo totally broken BIOS and overclocking). TSC is running always at highest P-state and usually synchronized too.
AMD is getting there.
Quoted 1 line
and which never stops.
That's unrealistic unfortunately any time soon. All the CPU vendors are pushing for much more aggressive power saving and this basically means turning off the CPU completely in the deeper sleep states.
-Andi
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
In reply to Ingo MolnarAr Gwe, 2006-11-10 am 09:57 +0100, ysgrifennodd Ingo Molnar:
Quoted 2 lines
AFAIK Windows doesnt use it, so it's a continuous minefield for new hardware to break.
Windows uses it extensively especially games. The AMD desync upset a lot of Windows gamers.
Quoted 3 lines
We should wait until CPU makers get their act together and implement a TSC variant that is /architecturally promised/ to have constant frequency (system bus frequency or whatever) and which never stops.
This will never happen for the really big boxes, light is just too slow... Our current TSC handling is not perfect but the TSC is often quite usable.
If hrtimer needs and requires we stop TSC support then we should delay the merge of HRTIMERS until these new processors are out and common ;)
Alan
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 6 lines · Alan Cox
Ar Gwe, 2006-11-10 am 09:57 +0100, ysgrifennodd Ingo Molnar:
Quoted 2 lines
AFAIK Windows doesnt use it, so it's a continuous minefield for new hardware to break.
Windows uses it extensively especially games. The AMD desync upset a lot of Windows gamers.
well, i meant the Windows kernel itself, not applications. (maybe the Windows kernel uses it on SMP systems where the TSC /used to be/ pretty stable, i dont know)
Quoted 6 lines
Quoted 3 lines
We should wait until CPU makers get their act together and implement a TSC variant that is /architecturally promised/ to have constant frequency (system bus frequency or whatever) and which never stops.
This will never happen for the really big boxes, light is just too slow... [...]
that's not a problem - time goes as fast as light [by definition] :-)
Quoted 1 line
If hrtimer needs and requires we stop TSC support [...]
no, it doesnt, so there's no real friction here. We just observed that in the past 10 years no generally working TSC-based gettimeofday was written (and i wrote the first version of it for the Pentium, so the blame is on me too), and that we might be better off without it. If someone can pull off a working TSC-based gettimeofday() implementation then there's no objection from us.
Ingo
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
In reply to Alan CoxQuoted 7 lines · Alan Cox
Ar Gwe, 2006-11-10 am 09:57 +0100, ysgrifennodd Ingo Molnar:
Quoted 1 line
We should wait until CPU makers get their act together and implement a TSC variant that is /architecturally promised/ to have constant frequency (system bus frequency or whatever) and which never stops.
This will never happen for the really big boxes, light is just too slow... Our current TSC handling is not perfect but the TSC is often quite usable.
This hypothetical clock wouldn't have to run full speed, would it? You could have a 1MHz clock distributed across even a large system fairly easily.
Wouldn't that be good enough?
Chris
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
In reply to Ingo MolnarHi!
Quoted 6 lines
Quoted 2 lines
If so, could that function use the PIT/pmtimer/etc for working out if the TSC is bust, rather than directly using jiffies?
there's no realiable way to figure out the TSC is bust: some CPUs have a slight 'skew' between cores for example. On some systems the TSC might skew between sockets. A CPU might break its TSC only once some
But we could still do a whitelist?
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 9 lines · Pavel Machek
Quoted 7 lines
Quoted 2 lines
If so, could that function use the PIT/pmtimer/etc for working out if the TSC is bust, rather than directly using jiffies?
there's no realiable way to figure out the TSC is bust: some CPUs have a slight 'skew' between cores for example. On some systems the TSC might skew between sockets. A CPU might break its TSC only once some
But we could still do a whitelist?
we could, but it would have to be almost empty right now :-) Reason:
even on systems that have (hardware-initialized) 'perfect' TSCs and which do not support any frequency scaling or power-saving mode, our current TSC initialization on SMP systems introduces a small (1-2 usecs) skew.
but even that limited set of systems is now mostly obsolete: no multi-core CPU based system i'm aware of would qualify. I have written user-space testcode for TSC and gettimeofday warps, see:
http://redhat.com/~mingo/time-warp-test/time-warp-test.c
no SMP system i have passes at the moment, running 2.6.17/18:
--------------------------------------
jupiter:~> ./time-warp-test
4 CPUs, running 4 parallel test-tasks.
checking for time-warps via:
- read time stamp counter (RDTSC) instruction (cycle resolution)
- gettimeofday (TOD) syscall (usec resolution)
[...]
new TSC-warp maximum: -6392 cycles, 0000294e1f3b6100 -> 0000294e1f3b4808
| # of TSC-warps:183606 |
--------------------------------------
venus:~> ./time-warp-test
4 CPUs, running 4 parallel test-tasks.
[...]
new TSC-warp maximum: -1328 cycles, 00001d9549c6c738 -> 00001d9549c6c208
| # of TSC-warps:332510 |
--------------------------------------
neptune:~> ./time-warp-test
2 CPUs, running 2 parallel test-tasks.
[...]
new TSC-warp maximum: -332 cycles, 0000005e00b1b89e -> 0000005e00b1b752
| # of TSC-warps:340 |
[and i'm lazy to turn on the 8-way now, but that has TSC warps too.]
so i'd love to see non-warping time, but after 10 years of trying i'm not holding my breath.
Ingo
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 6 lines
we could, but it would have to be almost empty right now :-) Reason:
even on systems that have (hardware-initialized) 'perfect' TSCs and which do not support any frequency scaling or power-saving mode, our current TSC initialization on SMP systems introduces a small (1-2 usecs) skew.
On Intel we don't sync the TSC anymore and on most systems users seem to be happy at least. And on multicore AMD it is drifting anyways and usually turned off.
-Andi
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 7 lines · Andi Kleen
Quoted 5 lines
we could, but it would have to be almost empty right now :-) Reason:
even on systems that have (hardware-initialized) 'perfect' TSCs and which do not support any frequency scaling or power-saving mode, our current TSC initialization on SMP systems introduces a small (1-2 usecs) skew.On Intel we don't sync the TSC anymore [...]
yeah, after i reported this a few months ago ;-)
Ingo
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
In reply to Ingo MolnarHi!
Quoted 11 lines
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines
If so, could that function use the PIT/pmtimer/etc for working out if the TSC is bust, rather than directly using jiffies?
there's no realiable way to figure out the TSC is bust: some CPUs have a slight 'skew' between cores for example. On some systems the TSC might skew between sockets. A CPU might break its TSC only once some
But we could still do a whitelist?
we could, but it would have to be almost empty right now :-) Reason:
Well, if it would contain at least 50% of the UP machines... that would be reasonably long list for a start.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 4 lines · Pavel Machek
Quoted 1 line
we could, but it would have to be almost empty right now :-) Reason:
Well, if it would contain at least 50% of the UP machines... that would be reasonably long list for a start.
which 50%? Does it include those where the TSC slows down due a thermal event SMM?
Ingo
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
In reply to Thomas GleixnerAhoj!
Tahle debata (lkml) by se Vam mohla hodit...
Pavel
Quoted 39 lines · Thomas Gleixner
Quoted 10 lines · Andi Kleen
Quoted 6 lines
arch/i386/kernel/tsc.c static int __init init_tsc_clocksource(v · line 481
Quoted 2 lines
arch/i386/kernel/tsc.c renamed from arch/i386/kernel/tsc.c~gtod-mark-tsc-unusable-for-highres-timers +0
-0verify_tsc_freq_timer.function = verify_tsc_freq; verify_tsc_freq_timer.expires =Hmmm. I wish this patch was unnecessary, but I don't see an easy solution.
Very sad. This will make a lot of people unhappy, even to the point where they might prefer disabling noidlehz over super slow gettimeofday. I assume you at least have a suitable command line option for that, right?
Yes it is sad. And the sadest part is that AMD and Intel have been asked to fix that more than 5 years ago. They did not get their brain straight and now we are the dimwits.
Quoted 3 lines
Can we get a summary on which systems the TSC is considered unstable? Normally we assume if it's stable enough for gettimeofday it should be stable enough for longer delays too.
TSC is simply a nightmare:
- Frequency changes with CPU clock
- Unsynced across CPUs
- Stops in C3, which makes it completely unusableOnce you take away periodic interrupts it is simply broken. AMD and Intel can run in circels, it does not get better.
tglx
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
In reply to Andi KleenQuoted 5 lines · Andi Kleen
Very sad. This will make a lot of people unhappy, even to the point where they might prefer disabling noidlehz over super slow gettimeofday. I assume you at least have a suitable command line option for that, right?
Can we get a summary on which systems the TSC is considered unstable?
the part where it stops in idle...
(the rest is fixed in recent enough hw)
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 8 lines · Arjan van de Ven
Quoted 5 lines · Andi Kleen
Very sad. This will make a lot of people unhappy, even to the point where they might prefer disabling noidlehz over super slow gettimeofday. I assume you at least have a suitable command line option for that, right?
Can we get a summary on which systems the TSC is considered unstable?
the part where it stops in idle...
That is handled by if (intel && C3 available) disable
-Andi
Re: [patch 13/19] GTOD: Mark TSC unusable for highres timers
Quoted 11 lines · Andi Kleen
Quoted 8 lines · Arjan van de Ven
Quoted 5 lines · Andi Kleen
Very sad. This will make a lot of people unhappy, even to the point where they might prefer disabling noidlehz over super slow gettimeofday. I assume you at least have a suitable command line option for that, right?
Can we get a summary on which systems the TSC is considered unstable?
the part where it stops in idle...
That is handled by if (intel && C3 available) disable
I'm not so sure it doesn't stop on AMD; the ACPI spec at least allows it; just that I've seen few AMD CPUs that actually have C3, but that could be a matter of time.
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
PATCH 14/19 dynticks: core code
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
dynamic ticks core code.
This is an extension to the per-cpu sched_tick timer of the high resolution timer functionality. The sched_tick timer is reprogrammed to a longer timeout before going idle, when no timer events are due in the next tick. The periodic tick is resumed when the CPU leaves the idle state. If a non-timer IRQ hits the idle task jiffies are updated from irq_enter before calling the interrupt code, otherwise the interrupt handler would eventually deal with a stale jiffy value.
The per-cpu idle statistics information can be used to optimize power management decisions.
More detailed information is available in Documentation/hrtimer/highres.txt
include/linux/hardirq.h +11 -1
@@ -106,6 +106,16 @@ static inline void account_system_vtime(* always balanced, so the interrupted value of ->hardirq_context * will always be restored. */ +#define __irq_enter () \ + do { \ + account_system_vtime(current); \ + add_preempt_count(HARDIRQ_OFFSET); \ + trace_hardirq_enter(); \ + } while (0) + +/* +* Enter irq context (on NO_HZ, update jiffies): +*/ extern void irq_enter (void );/*
@@ -123,7 +133,7 @@ extern void irq_enter(void);*/ extern void irq_exit (void ); -#define nmi_enter () do { lockdep_off(); irq_enter(); } while (0) +#define nmi_enter () do { lockdep_off(); __irq_enter(); } while (0)#define nmi_exit () do { __irq_exit(); lockdep_on(); } while (0)#endif /* LINUX_HARDIRQ_H */
include/linux/hrtimer.h +34 -0
@@ -22,6 +22,7 @@#include <linux/list.h> #include <linux/wait.h> +struct seq_file ;struct hrtimer_clock_base ;struct hrtimer_cpu_base ;
@@ -178,6 +179,17 @@ struct hrtimer_clock_base {* in the softirq. * @sched_timer: hrtimer to schedule the periodic tick in high * resolution mode +* @nr_events: Total number of timer interrupt events +* @idle_tick: Store the last idle tick expiry time when the tick +* timer is modified for idle sleeps. This is necessary +* to resume the tick timer operation in the timeline +* when the CPU returns from idle +* @tick_stopped: Indicator that the idle tick has been stopped +* @idle_jiffies: jiffies at the entry to idle for idle time accounting +* @idle_calls: Total number of idle calls +* @idle_sleeps: Number of idle calls, where the sched tick was stopped +* @idle_entrytime: Time when the idle call was entered +* @idle_sleeptime: Sum of the time slept in idle with sched tick stopped */ struct hrtimer_cpu_base {spinlock_t lock;
@@ -189,6 +201,16 @@ struct hrtimer_cpu_base {unsigned long check_clocks;struct list_head cb_pending;struct hrtimer sched_timer; +unsigned long nr_events; +#endif +#ifdef CONFIG_NO_HZ +ktime_t idle_tick; +int tick_stopped; +unsigned long idle_jiffies; +unsigned long idle_calls; +unsigned long idle_sleeps; +ktime_t idle_entrytime; +ktime_t idle_sleeptime;#endif };
@@ -295,6 +317,18 @@ extern void hrtimer_run_queues(void);/* Resume notification */ void hrtimer_notify_resume (void ); +#ifdef CONFIG_NO_HZ +extern void hrtimer_stop_sched_tick (void ); +extern void hrtimer_restart_sched_tick (void ); +extern void hrtimer_update_jiffies (void ); +extern void show_no_hz_stats (struct seq_file *p); +#else +static inline void hrtimer_stop_sched_tick (void ) { } +static inline void hrtimer_restart_sched_tick (void ) { } +static inline void hrtimer_update_jiffies (void ) { } +static inline void show_no_hz_stats (struct seq_file *p) { } +#endif +/* Bootup initialization: */ extern void __init hrtimers_init (void );
kernel/hrtimer.c +254 -6
318 lines not on this page · raw message
kernel/softirq.c +11 -3
@@ -278,9 +278,11 @@ EXPORT_SYMBOL(do_softirq);*/ void irq_enter (void ) { -account_system_vtime (current); -add_preempt_count (HARDIRQ_OFFSET ); -trace_hardirq_enter (); +__irq_enter (); +#ifdef CONFIG_NO_HZ +if (idle_cpu (smp_processor_id ())) +hrtimer_update_jiffies (); +#endif }#ifdef __ARCH_IRQ_EXIT_IRQS_DISABLED
@@ -299,6 +301,12 @@ void irq_exit(void)sub_preempt_count (IRQ_EXIT_OFFSET );if (!in_interrupt () &&local_softirq_pending ())invoke_softirq (); + +#ifdef CONFIG_NO_HZ +/* Make sure that timer wheel updates are propagated */ +if (!in_interrupt () &&idle_cpu (smp_processor_id ()) && !need_resched ()) +hrtimer_stop_sched_tick (); +#endif preempt_enable_no_resched (); }
kernel/time/Kconfig +7 -0
@@ -9,3 +9,10 @@ config HIGH_RES_TIMERS hardware is not capable then this option only increases the size of the kernel image. +config NO_HZ + bool "Tickless System (Dynamic Ticks)" + depends on HIGH_RES_TIMERS + help + This option enables a tickless system: timer interrupts will + only trigger on an as-needed basis both when the system is + busy and when the system is idle.
kernel/timer.c +1 -1
@@ -462,7 +462,7 @@ static inline void __run_timers(tvec_basspin_unlock_irq (&base->lock); } -#ifdef CONFIG_NO_IDLE_HZ +#if defined(CONFIG_NO_IDLE_HZ ) || defined(CONFIG_NO_HZ )/* * Find out when the next timer event is due to happen. This * is used on S/390 to stop all activity when a cpus is idle.
--
PATCH 15/19 dyntick: add nohz stats to /proc/stat
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Add nohz stats to /proc/stat.
fs/proc/proc_misc.c renamed from fs/proc/proc_misc.c~dynticks-add-nohz-stats-to-proc-stat +2 -0
@@ -527,6 +527,8 @@ static int show_stat(struct seq_file *p,nr_running (), nr_iowait()); +show_no_hz_stats (p); +return 0 ; }
_
--
PATCH 16/19 dynticks: i386 arch code
In reply to Thomas GleixnerFrom: Ingo Molnar <[email protected]>
Prepare i386 for dyntick: idle handler callbacks.
arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c~dynticks-i386-arch-code +2 -0
@@ -168,6 +168,7 @@ void cpu_idle(void)/* endless idle loop with no priority at all */ while (1 ) { +hrtimer_stop_sched_tick ();while (!need_resched ()) {void (*idle)(void );
@@ -186,6 +187,7 @@ void cpu_idle(void)__get_cpu_var (irq_stat).idle_timestamp = jiffies;idle (); } +hrtimer_restart_sched_tick ();preempt_enable_no_resched ();schedule ();preempt_disable ();
_
--
PATCH 17/19 dynticks: Fix nmi watchdog
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
The NMI watchdog implementation assumes that the local APIC timer interrupt is happening. This assumption is not longer true when high resolution timers and dynamic ticks come into play, as they may switch off the local APIC timer completely. Take the PIT/HPET interrupts into account too, to avoid false positives.
arch/i386/kernel/nmi.c +7 -2
@@ -23,6 +23,7 @@#include <linux/dmi.h> #include <linux/kprobes.h> #include <linux/cpumask.h> +#include <linux/kernel_stat.h> #include <asm/smp.h> #include <asm/nmi.h>
@@ -920,9 +921,13 @@ __kprobes int nmi_watchdog_tick(struct pcpu_clear (cpu, backtrace_mask); } - sum =per_cpu (irq_stat, cpu).apic_timer_irqs; +/* +* Take the local apic timer and PIT/HPET into account. We don't +* know which one is active, when we have highres/dyntick on +*/ + sum =per_cpu (irq_stat, cpu).apic_timer_irqs +kstat_irqs (0 ); -/* if the apic timer isn't firing, this cpu isn't doing much */ +/* if the none of the timers isn't firing, this cpu isn't doing much */ if (!touched && last_irq_sums[cpu] == sum) {/* * Ayiee, looks like this CPU is stuck ...
--
PATCH 18/19 high-res timers, dynticks: enable i386 support
In reply to Thomas GleixnerFrom: Ingo Molnar <[email protected]>
Enable high-res timers and dyntick on i386.
arch/i386/Kconfig +2 -0
@@ -82,6 +82,8 @@ source "init/Kconfig" menu "Processor type and features" +source "kernel/time/Kconfig" + config SMP bool "Symmetric multi-processing support" ---help---
--
PATCH 19/19 debugging feature: timer stats
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Add /proc/timer_stats support: debugging feature to profile timer expiration. Both the starting site, process/PID and the expiration function is captured. This allows the quick identification of timer event sources in a system.
Sample output:
# echo 1 > /proc/tstats
# cat /proc/tstats
Timerstats sample period: 3.888770 s
12, 0 swapper hrtimer_stop_sched_tick (hrtimer_sched_tick)
15, 1 swapper hcd_submit_urb (rh_timer_func)
4, 959 kedac schedule_timeout (process_timeout)
1, 0 swapper page_writeback_init (wb_timer_fn)
28, 0 swapper hrtimer_stop_sched_tick (hrtimer_sched_tick)
22, 2948 IRQ 4 tty_flip_buffer_push (delayed_work_timer_fn)
3, 3100 bash schedule_timeout (process_timeout)
1, 1 swapper queue_delayed_work_on (delayed_work_timer_fn)
1, 1 swapper queue_delayed_work_on (delayed_work_timer_fn)
1, 1 swapper neigh_table_init_no_netlink (neigh_periodic_timer)
1, 2292 ip __netdev_watchdog_up (dev_watchdog)
1, 23 events/1 do_cache_clean (delayed_work_timer_fn)
90 total events, 30.0 events/secDocumentation/hrtimer/timer_stats.txt new file +68 -0
@@ -0,0 +1,68 @@ +timer_stats - timer usage statistics +------------------------------------ + +timer_stats is a debugging facility to make the timer (ab)usage in a Linux +system visible to kernel and userspace developers. It is not intended for +production usage as it adds significant overhead to the (hr)timer code and the +(hr)timer data structures. + +timer_stats should be used by kernel and userspace developers to verify that +their code does not make unduly use of timers. This helps to avoid unnecessary +wakeups, which should be avoided to optimize power consumption. + +It can be enabled by CONFIG_TIMER_STATS in the "Kernel hacking" configuration +section. + +timer_stats collects information about the timer events which are fired in a +Linux system over a sample period: + +- the pid of the task(process) which initialized the timer +- the name of the process which initialized the timer +- the function where the timer was intialized +- the callback function which is associated to the timer +- the number of events (callbacks) + +timer_stats adds an entry to /proc: /proc/timer_stats + +This entry is used to control the statistics functionality and to read out the +sampled information. + +The timer_stats functionality is inactive on bootup. + +To activate a sample period issue: +# echo 1 >/proc/timer_stats + +To stop a sample period issue: +# echo 0 >/proc/timer_stats + +The statistics can be retrieved by: +# cat /proc/timer_stats + +The readout of /proc/timer_stats automatically disables sampling. The sampled +information is kept until a new sample period is started. This allows multiple +readouts. + +Sample output of /proc/timer_stats: + +Timerstats sample period: 3.888770 s + 12, 0 swapper hrtimer_stop_sched_tick (hrtimer_sched_tick) + 15, 1 swapper hcd_submit_urb (rh_timer_func) + 4, 959 kedac schedule_timeout (process_timeout) + 1, 0 swapper page_writeback_init (wb_timer_fn) + 28, 0 swapper hrtimer_stop_sched_tick (hrtimer_sched_tick) + 22, 2948 IRQ 4 tty_flip_buffer_push (delayed_work_timer_fn) + 3, 3100 bash schedule_timeout (process_timeout) + 1, 1 swapper queue_delayed_work_on (delayed_work_timer_fn) + 1, 1 swapper queue_delayed_work_on (delayed_work_timer_fn) + 1, 1 swapper neigh_table_init_no_netlink (neigh_periodic_timer) + 1, 2292 ip __netdev_watchdog_up (dev_watchdog) + 1, 23 events/1 do_cache_clean (delayed_work_timer_fn) +90 total events, 30.0 events/sec + +The first column is the number of events, the second column the pid, the third +column is the name of the process. The forth column shows the function which +initialized the timer and in parantheses the callback function which was +executed on expiry. + + Thomas, Ingo +
include/linux/hrtimer.h +52 -1
70 lines not on this page · raw message
include/linux/timer.h +49 -0
67 lines not on this page · raw message
kernel/hrtimer.c +26 -0
@@ -972,6 +972,18 @@ static inline void hrtimer_resume_jiffy_#endif /* CONFIG_HIGH_RES_TIMERS */ +#ifdef CONFIG_TIMER_STATS +void __timer_stats_hrtimer_set_start_info (struct hrtimer *timer,void *addr) +{ +if (timer->start_site) +return ; + + timer->start_site = addr; +memcpy (timer->start_comm, current->comm,TASK_COMM_LEN ); + timer->start_pid = current->pid; +} +#endif +/* * Timekeeping resumed notification */
@@ -1140,6 +1152,7 @@ remove_hrtimer(struct hrtimer *timer, st* reprogramming happens in the interrupt handler. This is a * rare case and less expensive than a smp call. */ +timer_stats_hrtimer_clear_start_info (timer); reprogram = base->cpu_base == &__get_cpu_var (hrtimer_bases);__remove_hrtimer (timer, base,HRTIMER_STATE_INACTIVE , reprogram);
@@ -1188,6 +1201,8 @@ hrtimer_start(struct hrtimer *timer, kti } timer->expires = tim; +timer_stats_hrtimer_set_start_info (timer); +enqueue_hrtimer (timer, new_base, base == new_base);unlock_hrtimer_base (timer, &flags);
@@ -1320,6 +1335,12 @@ void hrtimer_init(struct hrtimer *timer, timer->base = &cpu_base->clock_base[clock_id];hrtimer_init_timer_hres (timer); + +#ifdef CONFIG_TIMER_STATS + timer->start_site =NULL ; + timer->start_pid =-1 ; +memset (timer->start_comm,0 ,TASK_COMM_LEN ); +#endif }EXPORT_SYMBOL_GPL (hrtimer_init);
@@ -1400,6 +1421,7 @@ void hrtimer_interrupt(struct pt_regs *r__remove_hrtimer (timer, base,HRTIMER_STATE_CALLBACK ,0 ); +timer_stats_account_hrtimer (timer);if (timer->function (timer) !=HRTIMER_NORESTART ) {BUG_ON (timer->state !=HRTIMER_STATE_CALLBACK );
@@ -1442,6 +1464,8 @@ static void run_hrtimer_softirq(struct s timer =list_entry (cpu_base->cb_pending.next, struct hrtimer, cb_entry); +timer_stats_account_hrtimer (timer); + fn = timer->function;__remove_hrtimer (timer, timer->base,HRTIMER_STATE_CALLBACK ,0 );spin_unlock_irq (&cpu_base->lock);
@@ -1498,6 +1522,8 @@ static inline void run_hrtimer_queue(strif (base->softirq_time.tv64 <= timer->expires.tv64)break ; +timer_stats_account_hrtimer (timer); + fn = timer->function;__remove_hrtimer (timer, base,HRTIMER_STATE_CALLBACK ,0 );spin_unlock_irq (&cpu_base->lock);
kernel/time/Makefile +2 -1
@@ -1,3 +1,4 @@ obj-y += ntp.o clocksource.o jiffies.o -obj-$(CONFIG_GENERIC_CLOCKEVENTS) += clockevents.o +obj-$(CONFIG_GENERIC_CLOCKEVENTS) += clockevents.o +obj-$(CONFIG_TIMER_STATS) += timer_stats.o
kernel/time/timer_stats.c new file +244 -0
244 lines not on this page · raw message
kernel/timer.c +27 -2
@@ -34,6 +34,7 @@#include <linux/cpu.h> #include <linux/syscalls.h> #include <linux/delay.h> +#include <linux/kallsyms.h> #include <asm/uaccess.h> #include <asm/unistd.h>
@@ -133,6 +134,18 @@ static void internal_add_timer(tvec_baselist_add_tail (&timer->entry, vec); } +#ifdef CONFIG_TIMER_STATS +void __timer_stats_timer_set_start_info (struct timer_list *timer,void *addr) +{ +if (timer->start_site) +return ; + + timer->start_site = addr; +memcpy (timer->start_comm, current->comm,TASK_COMM_LEN ); + timer->start_pid = current->pid; +} +#endif +/** * init_timer - initialize a timer. * @timer: the timer to be initialized
@@ -144,11 +157,16 @@ void fastcall init_timer(struct timer_li { timer->entry.next =NULL ; timer->base =__raw_get_cpu_var (tvec_bases); +#ifdef CONFIG_TIMER_STATS + timer->start_site =NULL ; + timer->start_pid =-1 ; +memset (timer->start_comm,0 ,TASK_COMM_LEN ); +#endif }EXPORT_SYMBOL (init_timer);static inline void detach_timer (struct timer_list *timer, -int clear_pending) +int clear_pending) {struct list_head *entry = &timer->entry;
@@ -195,6 +213,7 @@ int __mod_timer(struct timer_list *timerunsigned long flags;int ret =0 ; +timer_stats_timer_set_start_info (timer);BUG_ON (!timer->function); base =lock_timer_base (timer, &flags);
@@ -245,6 +264,7 @@ void add_timer_on(struct timer_list *timtvec_base_t *base =per_cpu (tvec_bases, cpu);unsigned long flags; +timer_stats_timer_set_start_info (timer);BUG_ON (timer_pending (timer) || !timer->function);spin_lock_irqsave (&base->lock, flags); timer->base = base;
@@ -277,6 +297,7 @@ int mod_timer(struct timer_list *timer, {BUG_ON (!timer->function); +timer_stats_timer_set_start_info (timer);/* * This is a common optimization triggered by the * networking code - if the timer is re-modified
@@ -307,6 +328,7 @@ int del_timer(struct timer_list *timer)unsigned long flags;int ret =0 ; +timer_stats_timer_clear_start_info (timer);if (timer_pending (timer)) { base =lock_timer_base (timer, &flags);if (timer_pending (timer)) {
@@ -440,6 +462,8 @@ static inline void __run_timers(tvec_bas fn = timer->function; data = timer->data; +timer_stats_account_timer (timer); +set_running_timer (base, timer);detach_timer (timer,1 );spin_unlock_irq (&base->lock);
@@ -1128,7 +1152,8 @@ static void run_timer_softirq(struct sof {tvec_base_t *base =__get_cpu_var (tvec_bases); -hrtimer_run_queues (); +hrtimer_run_queues (); +if (time_after_eq (jiffies, base->timer_jiffies))__run_timers (base); }
kernel/workqueue.c +4 -2
@@ -122,7 +122,7 @@ int fastcall queue_work(struct workqueue }EXPORT_SYMBOL_GPL (queue_work); -static void delayed_work_timer_fn (unsigned long __data) +void delayed_work_timer_fn (unsigned long __data) {struct work_struct *work = (struct work_struct *)__data;struct workqueue_struct *wq = work->wq_data;
@@ -143,11 +143,12 @@ static void delayed_work_timer_fn(unsign* Returns 0 if @work was already on a queue, non-zero otherwise. */ int fastcall queue_delayed_work(struct workqueue_struct *wq, -struct work_struct *work,unsigned long delay) +struct work_struct *work,unsigned long delay) {int ret =0 ;struct timer_list *timer = &work->timer; +timer_stats_timer_set_start_info (&work->timer);if (!test_and_set_bit (0 , &work->pending)) {BUG_ON (timer_pending (timer));BUG_ON (!list_empty (&work->entry));
@@ -489,6 +490,7 @@ EXPORT_SYMBOL(schedule_work);*/ int fastcall schedule_delayed_work(struct work_struct *work, unsigned long delay) { +timer_stats_timer_set_start_info (&work->timer);return queue_delayed_work (keventd_wq, work, delay); }EXPORT_SYMBOL (schedule_delayed_work);
lib/Kconfig.debug +11 -0
@@ -125,6 +125,17 @@ config SCHEDSTATS application, you can say N to avoid the very slight overhead this adds. +config TIMER_STATS + bool "Collect kernel timers statistics" + depends on DEBUG_KERNEL && PROC_FS + help + If you say Y here, additional code will be inserted into the + timer routines to collect statistics about kernel timers being + reprogrammed. The statistics can be read from /proc/timer_stats. + The statistics collection is started by writing 1 to /proc/timer_stats, + writing 0 stops it. This feature is useful to collect information + about timer usage patterns in kernel and userspace. + config DEBUG_SLAB bool "Debug slab memory allocations" depends on DEBUG_KERNEL && SLAB
--
Hi,
Quoted 7 lines · Thomas Gleixner
Andrew,
this is a drop in replacement for the following patches in 2.6.19-rc5-mm1:
hrtimers-state-tracking.patch
up to
acpi-verify-lapic-timer-fix.patch
There is still the gtod-exponential-update_wall_time patch before that, I explained previously why it's wrong and how to fix this properly. Andrew, please drop this one.
http://www.ussg.iu.edu/hypermail/linux/kernel/0609.3/1320.html
http://www.ussg.iu.edu/hypermail/linux/kernel/0609.3/1303.html
Something I also wanted to mention about the OLS paper: It's an interesting read and answers a few question, but not all. It concentrates very much on the past (previous and current implementations), what I'm missing are more details on how it can be used in the future. IMO it's very important information regarding merging, i.e. how can this be applied to our various architectures. This is were have my doubts and more questions about it later.
The paper stresses the point that it provides a generic infrastructure, but as such it also brings some amazing complexities. Dedicated implementations often have the advantage to be simpler and faster (I'm not saying that current ones are). How does your implementation keep the source and runtime complexities under control? Such generic frameworks have the tendency to grow - new requirements have to be met and thus complexity further increases.
bye, Roman