2.6.24-git2: Oracle 11g VKTM process enters R state on startup and is unkillable [still broken in 2.6.25-rc1]
Subsystems (6)
Quoted 59 lines · Alessandro Suardi
I finally had a bit of time to try out different kernel versions to find
out where this began... and it's in 2.6.24-git2.What happens: Oracle 11g starts up and forks a number of so
called background processes. Starting in 2.6.24-git2 the VKTM process never fully completes its initialization but gets in R state, never accumulating CPU, and can't be straced/gdb'd/killed.Sysrq-T reports for VKTM looks like this
Feb 9 16:10:46 sandman kernel: ======================= Feb 9 16:10:46 sandman kernel: oracle R running 2684 2258 1 Feb 9 16:10:46 sandman kernel: f591dfb0 00200086 f6bbc3c4 f6863cc0 c010547a 00000000 b794f62c b7b70600 Feb 9 16:10:46 sandman kernel: b79453dc f591d000 c0103caa b794f62c b7943708 b79453e4 b7b70600 b79453dc Feb 9 16:10:46 sandman kernel: bfb0dd5c b79500b0 0000007b 0000007b c0320000 ffffffff 0e072d7a 00000073 Feb 9 16:10:46 sandman kernel: Call Trace: Feb 9 16:10:46 sandman kernel: [<c010547a>] ? do_IRQ+0xac/0xc1 Feb 9 16:10:46 sandman kernel: [<c0103caa>] work_resched+0x5/0x16 Feb 9 16:10:46 sandman kernel: [<c0320000>] ? pci_setup+0xb3/0x104 Feb 9 16:10:46 sandman kernel: =======================2.6.24-git1 is okay
2.6.24-git2 is bad
...
2.6.24-git20 is badOnly differences in kernel .config between -git1 and -git2 are
[asuardi@sandman src]$ diff .config-2.6.24-git[12] 3,4c3,4
< # Linux kernel version: 2.6.24-git1
< # Sat Jan 26 01:04:43 2008
---Quoted 2 lines
# Linux kernel version: 2.6.24-git2
# Sat Jan 26 12:10:15 2008121a122,123
Quoted 2 lines
CONFIG_CLASSIC_RCU=y
#CONFIG_PREEMPT_RCUis not set187a190
Quoted 1 line
#
CONFIG_RCU_TRACEis not set230a234
Quoted 1 line
#
CONFIG_SCHED_HRTICKis not set755a760
Quoted 1 line
#
CONFIG_PATA_NINJA32is not set
1807a1813Quoted 1 line
#
CONFIG_LATENCYTOPis not setSymptom is similar to what Rafael reported here
http://www.ussg.iu.edu/hypermail/linux/kernel/0801.3/4114.html
and similarly VKTM attempts to run at elevated priority as normal user process (Oracle kernel binary is not setuid root).
Quoted 8 lines
Peter Zijlstra's patches mentioned in the above thread, at
http://programming.kicks-ass.net/kernel-patches/sched-rt-group ,
do not appear to be in -git20 yet.
I'm available for further testing. Thanks, ciao,
Only to add that 2.6.25-rc1 is still broken.
thanks,
--alessandro
"We act as though comfort and luxury were the chief requirements
of life, when all that we need to make us really happy is something to be enthusiastic about."
(Charles Kingsley)
Quoted 50 lines · Alessandro Suardi
Quoted 49 lines · Alessandro Suardi
I finally had a bit of time to try out different kernel versions to find
out where this began... and it's in 2.6.24-git2.What happens: Oracle 11g starts up and forks a number of so
called background processes. Starting in 2.6.24-git2 the VKTM process never fully completes its initialization but gets in R state, never accumulating CPU, and can't be straced/gdb'd/killed.Sysrq-T reports for VKTM looks like this
Feb 9 16:10:46 sandman kernel: ======================= Feb 9 16:10:46 sandman kernel: oracle R running 2684 2258 1 Feb 9 16:10:46 sandman kernel: f591dfb0 00200086 f6bbc3c4 f6863cc0 c010547a 00000000 b794f62c b7b70600 Feb 9 16:10:46 sandman kernel: b79453dc f591d000 c0103caa b794f62c b7943708 b79453e4 b7b70600 b79453dc Feb 9 16:10:46 sandman kernel: bfb0dd5c b79500b0 0000007b 0000007b c0320000 ffffffff 0e072d7a 00000073 Feb 9 16:10:46 sandman kernel: Call Trace: Feb 9 16:10:46 sandman kernel: [<c010547a>] ? do_IRQ+0xac/0xc1 Feb 9 16:10:46 sandman kernel: [<c0103caa>] work_resched+0x5/0x16 Feb 9 16:10:46 sandman kernel: [<c0320000>] ? pci_setup+0xb3/0x104 Feb 9 16:10:46 sandman kernel: =======================2.6.24-git1 is okay
2.6.24-git2 is bad
...
2.6.24-git20 is badOnly differences in kernel .config between -git1 and -git2 are
[asuardi@sandman src]$ diff .config-2.6.24-git[12] 3,4c3,4
< # Linux kernel version: 2.6.24-git1
< # Sat Jan 26 01:04:43 2008
---Quoted 2 lines
# Linux kernel version: 2.6.24-git2
# Sat Jan 26 12:10:15 2008121a122,123
Quoted 2 lines
CONFIG_CLASSIC_RCU=y
#CONFIG_PREEMPT_RCUis not set187a190
Quoted 1 line
#
CONFIG_RCU_TRACEis not set230a234
Quoted 1 line
#
CONFIG_SCHED_HRTICKis not set755a760
Quoted 1 line
#
CONFIG_PATA_NINJA32is not set
1807a1813
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
#
CONFIG_LATENCYTOPis not setSymptom is similar to what Rafael reported here
http://www.ussg.iu.edu/hypermail/linux/kernel/0801.3/4114.html
and similarly VKTM attempts to run at elevated priority as normal user process (Oracle kernel binary is not setuid root).
Yes, I think this is the same problem. Please try to unset CONFIG_GROUP_SCHED and see if that helps.
Quoted 10 lines
Quoted 8 lines
Peter Zijlstra's patches mentioned in the above thread, at
http://programming.kicks-ass.net/kernel-patches/sched-rt-group ,
do not appear to be in -git20 yet.
I'm available for further testing. Thanks, ciao,
Only to add that 2.6.25-rc1 is still broken.
Yes, it is.
Thanks,
Rafael
Quoted 4 lines · Rafael J. Wysocki
Quoted 3 lines · Alessandro Suardi
Quoted 3 lines
2.6.24-git1 is okay
2.6.24-git2 is bad
Ok, that's git ID's
b47711bfbcd4eb77ca61ef0162487b20e023ae55 2.6.24-git1 9b73e76f3cf63379dcf45fcd4f112f5812418d0a 2.6.24-git2
so if you get a git tree, you can do
gitk b47711b..9b73e76
to see what happened in there.
However, the obvious candidates are the scheduler or the ocfs2 merge, and the latter is only relevant in case you use ocfs2, of course.
The rest of it tends to be the DVB and SCSI updates.
But it would be great if you could do a bisect and verify. Just do
git bisect start git bisect good b47711bfbcd4eb77ca61ef0162487b20e023ae55 git bisect bad 9b73e76f3cf63379dcf45fcd4f112f5812418d0a
and off you go..
Linus
Quoted 24 lines · Linus Torvalds
Quoted 4 lines · Rafael J. Wysocki
Quoted 3 lines · Alessandro Suardi
Quoted 3 lines
2.6.24-git1 is okay
2.6.24-git2 is badOk, that's git ID's
b47711bfbcd4eb77ca61ef0162487b20e023ae55 2.6.24-git1 9b73e76f3cf63379dcf45fcd4f112f5812418d0a 2.6.24-git2so if you get a git tree, you can do
gitk b47711b..9b73e76to see what happened in there.
However, the obvious candidates are the scheduler or the ocfs2 merge, and the latter is only relevant in case you use ocfs2, of course.
The rest of it tends to be the DVB and SCSI updates.
Quoted 7 lines
But it would be great if you could do a bisect and verify. Just do
git bisect start git bisect good b47711bfbcd4eb77ca61ef0162487b20e023ae55 git bisect bad 9b73e76f3cf63379dcf45fcd4f112f5812418d0aand off you go..
Well, I've already bisected that down to commit 6f505b16425a51270058e4a93441fe64de3dd435 "sched: rt group scheduling" and provided a simple test case. Moreover, there are patches from Peter that fix the problem, but they are lost somewhere in the way from him to you (please see
http://lkml.org/lkml/2008/2/5/535 and http://lkml.org/lkml/2008/2/6/320).
Thanks,
Rafael
Quoted 6 lines · Rafael J. Wysocki
Well, I've already bisected that down to commit
6f505b16425a51270058e4a93441fe64de3dd435"sched: rt group scheduling" and provided a simple test case. Moreover, there are patches from Peter that fix the problem, but they are lost somewhere in the way from him to you (please see http://lkml.org/lkml/2008/2/5/535 and
http://lkml.org/lkml/2008/2/6/320).
no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:
git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.git
combo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.
this all was complicated by the fact that people found interactivity of the group scheduler to be not up to their expectations (the observed latencies would go up linearly with the number of UIDs in the system). And your positive test results depended on the presence of those (at that time, still half-baken) changes. That issue turned out to be due to a .24-era design bug in the group scheduler and it took Peter a longer time to straighten it out. (he flattened the rbtree to maintain perfect weights and to get latencies back within the target.)
So what fixed your thing was simply not pullable into -rc1, as -rc1 was being cooled down at around last Friday already. Sorry about that! The short-term emergency fix would be to turn off the group scheduler altogether. (or, if we are lucky, tomorrow i can send the pull request for the changes below.)
Ingo
------------------>
Peter Zijlstra (15):
hrtimer: more hrtimer_init_sleeper() fallout.
sched: fair-group: separate tg->shares from task_group_lock
sched: fix incorrect irq lock usage in normalize_rt_tasks()
sched: rt-group: deal with PI
sched: rt-group: interface
sched: rt-group: make rt groups scheduling configurable
sched: rt-group: clean up the ifdeffery
sched: rt-group: refure unrunnable tasks
sched: rt-group: synchonised bandwidth period
sched: rt-group: smp balancing
sched: cleanup old and rarely used 'debug' features.
sched: fair-group scheduling vs latency
sched: fair-group: de-couple load-balancing from the rb-trees
sched: fair-group: single RQ approach
sched: remove sysctl_sched_batch_wakeup_granularity
Steven Rostedt (1):
rcu: add support for dynamic ticks and preempt rcu
Documentation/sched-rt-group.txt | 59 +++ include/linux/cgroup_subsys.h | 2 +- include/linux/hardirq.h | 10 + include/linux/init_task.h | 3 + include/linux/rcuclassic.h | 3 + include/linux/rcupreempt.h | 22 ++ include/linux/sched.h | 27 +- init/Kconfig | 23 +- kernel/rcupreempt.c | 224 +++++++++++- kernel/rtmutex.c | 5 +- kernel/sched.c | 766 +++++++++++++++++++++++++++++--------- kernel/sched_debug.c | 1 - kernel/sched_fair.c | 381 ++++++++++--------- kernel/sched_rt.c | 262 ++++++++++--- kernel/softirq.c | 1 + kernel/sysctl.c | 43 +-- kernel/time/tick-sched.c | 8 +- kernel/user.c | 78 ++++- 18 files changed, 1444 insertions(+), 474 deletions(-) create mode 100644 Documentation/sched-rt-group.txt
Documentation/sched-rt-group.txt new file +59 -0
@@ -0,0 +1,59 @@ + + +Real-Time group scheduling. + +The problem space: + +In order to schedule multiple groups of realtime tasks each group must +be assigned a fixed portion of the CPU time available. Without a minimum +guarantee a realtime group can obviously fall short. A fuzzy upper limit +is of no use since it cannot be relied upon. Which leaves us with just +the single fixed portion. + +CPU time is divided by means of specifying how much time can be spent +running in a given period. Say a frame fixed realtime renderer must +deliver 25 frames a second, which yields a period of 0.04s. Now say +it will also have to play some music and respond to input, leaving it +with around 80% for the graphics. We can then give this group a runtime +of 0.8 * 0.04s = 0.032s. + +This way the graphics group will have a 0.04s period with a 0.032s runtime +limit. + +Now if the audio thread needs to refill the DMA buffer every 0.005s, but +needs only about 3% CPU time to do so, it can do with a 0.03 * 0.005s += 0.00015s. + + +The Interface: + +system wide: + +/proc/sys/kernel/sched_rt_period_ms +/proc/sys/kernel/sched_rt_runtime_us + +CONFIG_FAIR_USER_SCHED + +/sys/kernel/uids/<uid>/cpu_rt_runtime_us + +or + +CONFIG_FAIR_CGROUP_SCHED + +/cgroup/<cgroup>/cpu.rt_runtime_us + +[ time is specified in us because the interface is s32; this gives an + operating range of ~35m to 1us ] + +The period takes values in [ 1, INT_MAX ], runtime in [ -1, INT_MAX - 1 ]. + +A runtime of -1 specifies runtime == period, ie. no limit. + +New groups get the period from /proc/sys/kernel/sched_rt_period_us and +a runtime of 0. + +Settings are constrained to: + + \Sum_{i} runtime_{i} / global_period <= global_runtime / global_period + +in order to keep the configuration schedulable.
include/linux/cgroup_subsys.h +1 -1
@@ -25,7 +25,7 @@ SUBSYS(ns)/* */ -#ifdef CONFIG_FAIR_CGROUP_SCHED +#ifdef CONFIG_CGROUP_SCHED SUBSYS(cpu_cgroup)#endif
include/linux/hardirq.h +10 -0
@@ -109,6 +109,14 @@ static inline void account_system_vtime(struct task_struct *tsk) }#endif +#if defined(CONFIG_PREEMPT_RCU ) && defined(CONFIG_NO_HZ ) +extern void rcu_irq_enter (void ); +extern void rcu_irq_exit (void ); +#else +# define rcu_irq_enter () do { } while (0) +# define rcu_irq_exit () do { } while (0) +#endif /* CONFIG_PREEMPT_RCU */ +/* * It is safe to do non-atomic ops on ->hardirq_context, * because NMI handlers may not preempt and the ops are
@@ -117,6 +125,7 @@ static inline void account_system_vtime(struct task_struct *tsk)*/ #define __irq_enter () \ do { \ + rcu_irq_enter(); \ account_system_vtime(current); \ add_preempt_count(HARDIRQ_OFFSET); \ trace_hardirq_enter(); \
@@ -135,6 +144,7 @@ extern void irq_enter(void);trace_hardirq_exit (); \account_system_vtime (current); \sub_preempt_count (HARDIRQ_OFFSET ); \ +rcu_irq_exit (); \ }while (0 )/*
include/linux/init_task.h +3 -0
@@ -151,6 +151,9 @@ extern struct group_info init_groups; .cpus_allowed = CPU_MASK_ALL, \ .mm = NULL, \ .active_mm = &init_mm, \ + .se = { \ + .group_node = LIST_HEAD_INIT(tsk.se.group_node), \ + }, \ .rt = { \ .run_list = LIST_HEAD_INIT(tsk.rt.run_list), \ .time_slice = HZ, \
include/linux/rcuclassic.h +3 -0
@@ -160,5 +160,8 @@ extern void rcu_restart_cpu(int cpu);extern long rcu_batches_completed (void );extern long rcu_batches_completed_bh (void ); +#define rcu_enter_nohz () do { } while (0) +#define rcu_exit_nohz () do { } while (0) +#endif /* __KERNEL__ */ #endif /* __LINUX_RCUCLASSIC_H */
include/linux/rcupreempt.h +22 -0
@@ -82,5 +82,27 @@ extern struct rcupreempt_trace *rcupreempt_trace_cpu(int cpu);struct softirq_action ; +#ifdef CONFIG_NO_HZ +DECLARE_PER_CPU (long, dynticks_progress_counter); + +static inline void rcu_enter_nohz (void ) +{ +__get_cpu_var (dynticks_progress_counter)++; +WARN_ON (__get_cpu_var (dynticks_progress_counter) &0x1 ); +mb (); +} + +static inline void rcu_exit_nohz (void ) +{ +mb (); +__get_cpu_var (dynticks_progress_counter)++; +WARN_ON (!(__get_cpu_var (dynticks_progress_counter) &0x1 )); +} + +#else /* CONFIG_NO_HZ */ +#define rcu_enter_nohz () do { } while (0) +#define rcu_exit_nohz () do { } while (0) +#endif /* CONFIG_NO_HZ */ +#endif /* __KERNEL__ */ #endif /* __LINUX_RCUPREEMPT_H */
include/linux/sched.h +20 -7
@@ -590,7 +590,7 @@ struct user_struct {struct hlist_node uidhash_node;uid_t uid; -#ifdef CONFIG_FAIR_USER_SCHED +#ifdef CONFIG_USER_SCHED struct task_group *tg;#ifdef CONFIG_SYSFS struct kobject kobj;
@@ -917,6 +917,7 @@ struct load_weight {struct sched_entity {struct load_weight load;/* for load-balancing */ struct rb_node run_node; +struct list_head group_node;unsigned int on_rq;u64 exec_start;
@@ -973,7 +974,7 @@ struct sched_rt_entity {unsigned long timeout;int nr_cpus_allowed; -#ifdef CONFIG_FAIR_GROUP_SCHED +#ifdef CONFIG_RT_GROUP_SCHED struct sched_rt_entity *parent;/* rq on which this entity is (to be) queued: */ struct rt_rq *rt_rq;
@@ -1536,13 +1537,10 @@ extern void sched_idle_next(void);extern unsigned int sysctl_sched_latency;extern unsigned int sysctl_sched_min_granularity;extern unsigned int sysctl_sched_wakeup_granularity; -extern unsigned int sysctl_sched_batch_wakeup_granularity;extern unsigned int sysctl_sched_child_runs_first;extern unsigned int sysctl_sched_features;extern unsigned int sysctl_sched_migration_cost;extern unsigned int sysctl_sched_nr_migrate; -extern unsigned int sysctl_sched_rt_period; -extern unsigned int sysctl_sched_rt_ratio;#if defined(CONFIG_FAIR_GROUP_SCHED ) && defined(CONFIG_SMP )extern unsigned int sysctl_sched_min_bal_int_shares;extern unsigned int sysctl_sched_max_bal_int_shares;
@@ -1552,6 +1550,12 @@ int sched_nr_latency_handler(struct ctl_table *table, int write, struct file *file, void __user *buffer, size_t *length, loff_t *ppos);#endif +extern unsigned int sysctl_sched_rt_period; +extern int sysctl_sched_rt_runtime; + +int sched_rt_handler (struct ctl_table *table,int write, +struct file *filp,void __user *buffer,size_t *lenp, +loff_t *ppos);extern unsigned int sysctl_sched_compat_yield;
@@ -2027,16 +2031,25 @@ extern int sched_mc_power_savings, sched_smt_power_savings;extern void normalize_rt_tasks (void ); -#ifdef CONFIG_FAIR_GROUP_SCHED +#ifdef CONFIG_GROUP_SCHED extern struct task_group init_task_group;extern struct task_group *sched_create_group (void );extern void sched_destroy_group (struct task_group *tg);extern void sched_move_task (struct task_struct *tsk); +#ifdef CONFIG_FAIR_GROUP_SCHED extern int sched_group_set_shares (struct task_group *tg,unsigned long shares);extern unsigned long sched_group_shares (struct task_group *tg); - +#endif +#ifdef CONFIG_RT_GROUP_SCHED +extern int sched_group_set_rt_runtime (struct task_group *tg, +long rt_runtime_us); +extern long sched_group_rt_runtime (struct task_group *tg); +extern int sched_group_set_rt_period (struct task_group *tg, +long rt_period_us); +extern long sched_group_rt_period (struct task_group *tg); +#endif #endif #ifdef CONFIG_TASK_XACCT
init/Kconfig +17 -6
@@ -311,25 +311,36 @@ config CPUSETS Say N if unsure. -config FAIR_GROUP_SCHED - bool "Fair group CPU scheduler" +config GROUP_SCHED + bool "Group CPU scheduler" default y help This feature lets CPU scheduler recognize task groups and control CPU bandwidth allocation to such task groups. +config FAIR_GROUP_SCHED + bool "Group scheduling for SCHED_OTHER" + depends on GROUP_SCHED + default y + +config RT_GROUP_SCHED + bool "Group scheduling for SCHED_RR/FIFO" + depends on EXPERIMENTAL + depends on GROUP_SCHED + default n + choice - depends on FAIR_GROUP_SCHED + depends on GROUP_SCHED prompt "Basis for grouping tasks" - default FAIR_USER_SCHED + default USER_SCHED -config FAIR_USER_SCHED +config USER_SCHED bool "user id" help This option will choose userid as the basis for grouping tasks, thus providing equal CPU bandwidth to each user. -config FAIR_CGROUP_SCHED +config CGROUP_SCHED bool "Control groups" depends on CGROUPS help
kernel/rcupreempt.c +220 -4
262 lines not on this page · raw message
kernel/rtmutex.c +4 -1
@@ -630,9 +630,12 @@ rt_mutex_slowlock(struct rt_mutex *lock, int state,set_current_state (state);/* Setup the timer, when timeout != NULL */ -if (unlikely (timeout)) +if (unlikely (timeout)) {hrtimer_start (&timeout->timer, timeout->timer.expires,HRTIMER_MODE_ABS ); +if (!hrtimer_active (&timeout->timer)) + timeout->task =NULL ; + }for (;;) {/* Try to acquire the lock: */
kernel/sched.c +594 -172
1,190 lines not on this page · raw message
kernel/sched_debug.c +0 -1
@@ -214,7 +214,6 @@ static int sched_debug_show(struct seq_file *m, void *v)PN (sysctl_sched_latency);PN (sysctl_sched_min_granularity);PN (sysctl_sched_wakeup_granularity); -PN (sysctl_sched_batch_wakeup_granularity);PN (sysctl_sched_child_runs_first);P (sysctl_sched_features);#undef PN
kernel/sched_fair.c +195 -186
593 lines not on this page · raw message
kernel/sched_rt.c +205 -57
372 lines not on this page · raw message
kernel/softirq.c +1 -0
@@ -313,6 +313,7 @@ void irq_exit(void)/* Make sure that timer wheel updates are propagated */ if (!in_interrupt () &&idle_cpu (smp_processor_id ()) && !need_resched ())tick_nohz_stop_sched_tick (); +rcu_irq_exit ();#endif preempt_enable_no_resched (); }
kernel/sysctl.c +16 -27
61 lines not on this page · raw message
kernel/time/tick-sched.c +3 -5
@@ -192,7 +192,6 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time)void tick_nohz_stop_sched_tick (void ) {unsigned long seq, last_jiffies, next_jiffies, delta_jiffies, flags; -unsigned long rt_jiffies;struct tick_sched *ts;ktime_t last_update, expires, now;struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev;
@@ -245,10 +244,6 @@ void tick_nohz_stop_sched_tick(void) next_jiffies =get_next_timer_interrupt (last_jiffies); delta_jiffies = next_jiffies - last_jiffies; - rt_jiffies =rt_needs_cpu (cpu); -if (rt_jiffies && rt_jiffies < delta_jiffies) - delta_jiffies = rt_jiffies; -if (rcu_needs_cpu (cpu)) delta_jiffies =1 ;/*
@@ -282,6 +277,7 @@ void tick_nohz_stop_sched_tick(void) ts->idle_tick = ts->sched_timer.expires; ts->tick_stopped =1 ; ts->idle_jiffies = last_jiffies; +rcu_enter_nohz (); }/*
@@ -375,6 +371,8 @@ void tick_nohz_restart_sched_tick(void)return ; } +rcu_exit_nohz (); +/* Update jiffies first */ select_nohz_load_balancer (0 ); now =ktime_get ();
kernel/user.c +71 -7
130 lines not on this page · raw message
Quoted 10 lines · Ingo Molnar
no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:
git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.gitcombo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.
ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is fec13e45305d69fd0bd23b30bd05a0a42cf341f8.
Ingo
kernel/sched.c +5 -0
@@ -219,6 +219,10 @@ static void start_rt_bandwidth(struct rtif (rt_b->rt_runtime ==RUNTIME_INF )return ; +if (hrtimer_active (&rt_b->rt_period_timer)) +return ; + +spin_lock (&rt_b->rt_runtime_lock);for (;;) {if (hrtimer_active (&rt_b->rt_period_timer))break ;
@@ -229,6 +233,7 @@ static void start_rt_bandwidth(struct rt rt_b->rt_period_timer.expires, HRTIMER_MODE_ABS); } +spin_unlock (&rt_b->rt_runtime_lock); }#ifdef CONFIG_RT_GROUP_SCHED
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Ingo Molnar
no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:
git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.git
Quoted 8 lines
Quoted 3 lines
combo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.
ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is
fec13e45305d69fd0bd23b30bd05a0a42cf341f8.
With the previous patch and this patch applied, the issue is not reproducible here.
Thanks,
Rafael
Quoted 25 lines
Index:
linux-2.6/kernel/sched.c
===================================================================kernel/sched.c +5
-0@@ -219,6 +219,10 @@ static void start_rt_bandwidth(struct rtif (rt_b->rt_runtime ==RUNTIME_INF )return ; +if (hrtimer_active (&rt_b->rt_period_timer)) +return ; + +spin_lock (&rt_b->rt_runtime_lock);for (;;) {if (hrtimer_active (&rt_b->rt_period_timer))break ;@@ -229,6 +233,7 @@ static void start_rt_bandwidth(struct rt rt_b->rt_period_timer.expires, HRTIMER_MODE_ABS); } +spin_unlock (&rt_b->rt_runtime_lock); }#ifdef CONFIG_RT_GROUP_SCHED
Quoted 20 lines · Rafael J. Wysocki
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Ingo Molnar
no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:
git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.gitQuoted 8 lines
Quoted 3 lines
combo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.
ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is
fec13e45305d69fd0bd23b30bd05a0a42cf341f8.
Quoted 5 lines
With the previous patch and this patch applied, the issue is not reproducible here.
Thanks,
Rafael
The problem is fixed for me as well with the previous patch + the patch
below, VKTM now enters S state and Oracle shuts down properly again.
Thanks !
Quoted 22 lines
Quoted 22 lines
Index:
linux-2.6/kernel/sched.c
===================================================================kernel/sched.c +5
-0@@ -219,6 +219,10 @@ static void start_rt_bandwidth(struct rtif (rt_b->rt_runtime ==RUNTIME_INF )return ; +if (hrtimer_active (&rt_b->rt_period_timer)) +return ; + +spin_lock (&rt_b->rt_runtime_lock);for (;;) {if (hrtimer_active (&rt_b->rt_period_timer))break ;@@ -229,6 +233,7 @@ static void start_rt_bandwidth(struct rt rt_b->rt_period_timer.expires, HRTIMER_MODE_ABS); } +spin_unlock (&rt_b->rt_runtime_lock); }
Quoted 1 line
kernel/sched.c static void start_rt_bandwidth(struct rt · line 239
Quoted 1 line
kernel/sched.c +0
-0#ifdef CONFIG_RT_GROUP_SCHED
--alessandro
"We act as though comfort and luxury were the chief requirements
of life, when all that we need to make us really happy is something to be enthusiastic about."
(Charles Kingsley)
Quoted 11 lines · Alessandro Suardi
Quoted 7 lines
Quoted 4 lines
ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is
fec13e45305d69fd0bd23b30bd05a0a42cf341f8.With the previous patch and this patch applied, the issue is not reproducible here.
The problem is fixed for me as well with the previous patch + the patch below, VKTM now enters S state and Oracle shuts down properly again.
thanks alot for testing this. The scheduler queue is now looking good in testing, will probably send a pull request to Linus later today.
Ingo
Quoted 15 lines · Rafael J. Wysocki
Quoted 14 lines · Ingo Molnar
Quoted 10 lines · Ingo Molnar
no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:
git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.gitcombo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.
Quoted 7 lines
Quoted 4 lines
ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is
fec13e45305d69fd0bd23b30bd05a0a42cf341f8.With the previous patch and this patch applied, the issue is not reproducible here.
Did you enable CONFIG_RT_GROUP_SCHED (it defaults to n)?
If you didn't, could you try with it set to y?
Quoted 22 lines · Peter Zijlstra
Quoted 20 lines · Rafael J. Wysocki
Quoted 18 lines · Ingo Molnar
Quoted 10 lines · Ingo Molnar
no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:
git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.gitcombo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.
ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is
fec13e45305d69fd0bd23b30bd05a0a42cf341f8.
Quoted 6 lines
Quoted 2 lines
With the previous patch and this patch applied, the issue is not reproducible here.
Did you enable
CONFIG_RT_GROUP_SCHED(it defaults to n)?If you didn't, could you try with it set to y?
I just rebuilt 2.6.25-rc1-git2 with Ingo's patch and your patch on top,
and the Oracle VKTM issue is still gone even with
[asuardi@sandman ~]$ grep GROUP_SCHED
/share/src/linux-2.6.25-rc1-git2-orafix/.configCONFIG_GROUP_SCHED=yCONFIG_FAIR_GROUP_SCHED=yCONFIG_RT_GROUP_SCHED=y
# CONFIG_CGROUP_SCHED is not set
so it's good for me.
Or is it necessary to also enable CONFIG_CGROUP_SCHED and retest ?
--alessandro
"We act as though comfort and luxury were the chief requirements
of life, when all that we need to make us really happy is something to be enthusiastic about."
(Charles Kingsley)
Quoted 40 lines · Alessandro Suardi
Quoted 22 lines · Peter Zijlstra
Quoted 20 lines · Rafael J. Wysocki
Quoted 18 lines · Ingo Molnar
Quoted 10 lines · Ingo Molnar
no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:
git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.gitcombo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.
ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is
fec13e45305d69fd0bd23b30bd05a0a42cf341f8.Quoted 6 lines
Quoted 2 lines
With the previous patch and this patch applied, the issue is not reproducible here.
Did you enable
CONFIG_RT_GROUP_SCHED(it defaults to n)?If you didn't, could you try with it set to y?
I just rebuilt 2.6.25-rc1-git2 with Ingo's patch and your patch on top,
and the Oracle VKTM issue is still gone even with[asuardi@sandman ~]$ grep
GROUP_SCHED
/share/src/linux-2.6.25-rc1-git2-orafix/.configCONFIG_GROUP_SCHED=yCONFIG_FAIR_GROUP_SCHED=yCONFIG_RT_GROUP_SCHED=y
#CONFIG_CGROUP_SCHEDis not set
Quoted 3 lines
so it's good for me.
Or is it necessary to also enable
CONFIG_CGROUP_SCHEDand retest ?
No that should be quite all-right. Thanks for testing!
Quoted 22 lines · Peter Zijlstra
Quoted 20 lines · Rafael J. Wysocki
Quoted 18 lines · Ingo Molnar
Quoted 10 lines · Ingo Molnar
no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:
git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.gitcombo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.
ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is
fec13e45305d69fd0bd23b30bd05a0a42cf341f8.
Quoted 6 lines
Quoted 2 lines
With the previous patch and this patch applied, the issue is not reproducible here.
Did you enable
CONFIG_RT_GROUP_SCHED(it defaults to n)?If you didn't, could you try with it set to y?
Tested with CONFIG_RT_GROUP_SCHED set and it also works as expected.
Thanks,
Rafael