Skip to contentSkip to conversation

2.6.24-git2: Oracle 11g VKTM process enters R state on startup and is unkillable [still broken in 2.6.25-rc1]

DISCUSSION Alessandro Suardi 5 participants 13 messages Feb 11, 2008 – Feb 13, 2008

Subsystems (6)
Documentation Scheduler cgroup RCU +2 more subsystems
lore
Share & tools
lore
#0 lore raw
Quoted 59 lines · Alessandro Suardi

I finally had a bit of time to try out different kernel versions to find
out where this began... and it's in 2.6.24-git2.

What happens: Oracle 11g starts up and forks a number of so
called background processes. Starting in 2.6.24-git2 the VKTM process never fully completes its initialization but gets in R state, never accumulating CPU, and can't be straced/gdb'd/killed.

Sysrq-T reports for VKTM looks like this

Feb  9 16:10:46 sandman kernel:  =======================
Feb  9 16:10:46 sandman kernel: oracle        R running   2684  2258      1
Feb  9 16:10:46 sandman kernel:        f591dfb0 00200086 f6bbc3c4
f6863cc0 c010547a 00000000 b794f62c b7b70600
Feb  9 16:10:46 sandman kernel:        b79453dc f591d000 c0103caa
b794f62c b7943708 b79453e4 b7b70600 b79453dc
Feb  9 16:10:46 sandman kernel:        bfb0dd5c b79500b0 0000007b
0000007b c0320000 ffffffff 0e072d7a 00000073
Feb  9 16:10:46 sandman kernel: Call Trace:
Feb  9 16:10:46 sandman kernel:  [<c010547a>] ? do_IRQ+0xac/0xc1
Feb  9 16:10:46 sandman kernel:  [<c0103caa>] work_resched+0x5/0x16
Feb  9 16:10:46 sandman kernel:  [<c0320000>] ? pci_setup+0xb3/0x104
Feb  9 16:10:46 sandman kernel:  =======================

2.6.24-git1 is okay
2.6.24-git2 is bad
...
2.6.24-git20 is bad

Only differences in kernel .config between -git1 and -git2 are

[asuardi@sandman src]$ diff .config-2.6.24-git[12] 3,4c3,4
< # Linux kernel version: 2.6.24-git1
< # Sat Jan 26 01:04:43 2008
---

Quoted 2 lines

# Linux kernel version: 2.6.24-git2
# Sat Jan 26 12:10:15 2008

121a122,123

Quoted 2 lines

CONFIG_CLASSIC_RCU=y
# CONFIG_PREEMPT_RCU is not set

187a190

Quoted 1 line

# CONFIG_RCU_TRACE is not set

230a234

Quoted 1 line

# CONFIG_SCHED_HRTICK is not set

755a760

Quoted 1 line

# CONFIG_PATA_NINJA32 is not set

1807a1813

Quoted 1 line

# CONFIG_LATENCYTOP is not set

Symptom is similar to what Rafael reported here

http://www.ussg.iu.edu/hypermail/linux/kernel/0801.3/4114.html

and similarly VKTM attempts to run at elevated priority as normal user process (Oracle kernel binary is not setuid root).

Quoted 8 lines

Peter Zijlstra's patches mentioned in the above thread, at

http://programming.kicks-ass.net/kernel-patches/sched-rt-group ,

do not appear to be in -git20 yet.

I'm available for further testing. Thanks, ciao,

Only to add that 2.6.25-rc1 is still broken.

thanks,

--alessandro

"We act as though comfort and luxury were the chief requirements
of life, when all that we need to make us really happy is something to be enthusiastic about."

(Charles Kingsley)

#1 lore raw
Quoted 50 lines · Alessandro Suardi
Quoted 49 lines · Alessandro Suardi

I finally had a bit of time to try out different kernel versions to find
out where this began... and it's in 2.6.24-git2.

What happens: Oracle 11g starts up and forks a number of so
called background processes. Starting in 2.6.24-git2 the VKTM process never fully completes its initialization but gets in R state, never accumulating CPU, and can't be straced/gdb'd/killed.

Sysrq-T reports for VKTM looks like this

Feb  9 16:10:46 sandman kernel:  =======================
Feb  9 16:10:46 sandman kernel: oracle        R running   2684  2258      1
Feb  9 16:10:46 sandman kernel:        f591dfb0 00200086 f6bbc3c4
f6863cc0 c010547a 00000000 b794f62c b7b70600
Feb  9 16:10:46 sandman kernel:        b79453dc f591d000 c0103caa
b794f62c b7943708 b79453e4 b7b70600 b79453dc
Feb  9 16:10:46 sandman kernel:        bfb0dd5c b79500b0 0000007b
0000007b c0320000 ffffffff 0e072d7a 00000073
Feb  9 16:10:46 sandman kernel: Call Trace:
Feb  9 16:10:46 sandman kernel:  [<c010547a>] ? do_IRQ+0xac/0xc1
Feb  9 16:10:46 sandman kernel:  [<c0103caa>] work_resched+0x5/0x16
Feb  9 16:10:46 sandman kernel:  [<c0320000>] ? pci_setup+0xb3/0x104
Feb  9 16:10:46 sandman kernel:  =======================

2.6.24-git1 is okay
2.6.24-git2 is bad
...
2.6.24-git20 is bad

Only differences in kernel .config between -git1 and -git2 are

[asuardi@sandman src]$ diff .config-2.6.24-git[12] 3,4c3,4
< # Linux kernel version: 2.6.24-git1
< # Sat Jan 26 01:04:43 2008
---

Quoted 2 lines

# Linux kernel version: 2.6.24-git2
# Sat Jan 26 12:10:15 2008

121a122,123

Quoted 2 lines

CONFIG_CLASSIC_RCU=y
# CONFIG_PREEMPT_RCU is not set

187a190

Quoted 1 line

# CONFIG_RCU_TRACE is not set

230a234

Quoted 1 line

# CONFIG_SCHED_HRTICK is not set

755a760

Quoted 1 line

# CONFIG_PATA_NINJA32 is not set

1807a1813

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

# CONFIG_LATENCYTOP is not set

Symptom is similar to what Rafael reported here

http://www.ussg.iu.edu/hypermail/linux/kernel/0801.3/4114.html

and similarly VKTM attempts to run at elevated priority as normal user process (Oracle kernel binary is not setuid root).

Yes, I think this is the same problem. Please try to unset CONFIG_GROUP_SCHED and see if that helps.

Quoted 10 lines
Quoted 8 lines

Peter Zijlstra's patches mentioned in the above thread, at

http://programming.kicks-ass.net/kernel-patches/sched-rt-group ,

do not appear to be in -git20 yet.

I'm available for further testing. Thanks, ciao,

Only to add that 2.6.25-rc1 is still broken.

Yes, it is.

Thanks,
Rafael

#2 lore raw
Quoted 4 lines · Rafael J. Wysocki
Quoted 3 lines · Alessandro Suardi
Quoted 3 lines

2.6.24-git1 is okay
2.6.24-git2 is bad

Ok, that's git ID's

	b47711bfbcd4eb77ca61ef0162487b20e023ae55 2.6.24-git1
	9b73e76f3cf63379dcf45fcd4f112f5812418d0a 2.6.24-git2

so if you get a git tree, you can do

	gitk b47711b..9b73e76

to see what happened in there.

However, the obvious candidates are the scheduler or the ocfs2 merge, and the latter is only relevant in case you use ocfs2, of course.

The rest of it tends to be the DVB and SCSI updates.

But it would be great if you could do a bisect and verify. Just do

	git bisect start
	git bisect good b47711bfbcd4eb77ca61ef0162487b20e023ae55
	git bisect bad 9b73e76f3cf63379dcf45fcd4f112f5812418d0a

and off you go..

Linus

#3 lore raw
Quoted 24 lines · Linus Torvalds
Quoted 4 lines · Rafael J. Wysocki
Quoted 3 lines · Alessandro Suardi
Quoted 3 lines

2.6.24-git1 is okay
2.6.24-git2 is bad

Ok, that's git ID's

	b47711bfbcd4eb77ca61ef0162487b20e023ae55 2.6.24-git1
	9b73e76f3cf63379dcf45fcd4f112f5812418d0a 2.6.24-git2

so if you get a git tree, you can do

	gitk b47711b..9b73e76

to see what happened in there.

However, the obvious candidates are the scheduler or the ocfs2 merge, and the latter is only relevant in case you use ocfs2, of course.

The rest of it tends to be the DVB and SCSI updates.

Quoted 7 lines

But it would be great if you could do a bisect and verify. Just do

	git bisect start
	git bisect good b47711bfbcd4eb77ca61ef0162487b20e023ae55
	git bisect bad 9b73e76f3cf63379dcf45fcd4f112f5812418d0a

and off you go..

Well, I've already bisected that down to commit 6f505b16425a51270058e4a93441fe64de3dd435 "sched: rt group scheduling" and provided a simple test case. Moreover, there are patches from Peter that fix the problem, but they are lost somewhere in the way from him to you (please see
http://lkml.org/lkml/2008/2/5/535 and http://lkml.org/lkml/2008/2/6/320).

Thanks,
Rafael

#4 lore raw
Quoted 6 lines · Rafael J. Wysocki

Well, I've already bisected that down to commit 6f505b16425a51270058e4a93441fe64de3dd435 "sched: rt group scheduling" and provided a simple test case. Moreover, there are patches from Peter that fix the problem, but they are lost somewhere in the way from him to you (please see http://lkml.org/lkml/2008/2/5/535 and
http://lkml.org/lkml/2008/2/6/320).

no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:

git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.git

combo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.

this all was complicated by the fact that people found interactivity of the group scheduler to be not up to their expectations (the observed latencies would go up linearly with the number of UIDs in the system). And your positive test results depended on the presence of those (at that time, still half-baken) changes. That issue turned out to be due to a .24-era design bug in the group scheduler and it took Peter a longer time to straighten it out. (he flattened the rbtree to maintain perfect weights and to get latencies back within the target.)

So what fixed your thing was simply not pullable into -rc1, as -rc1 was being cooled down at around last Friday already. Sorry about that! The short-term emergency fix would be to turn off the group scheduler altogether. (or, if we are lucky, tomorrow i can send the pull request for the changes below.)

Ingo

------------------>
Peter Zijlstra (15):
hrtimer: more hrtimer_init_sleeper() fallout.
sched: fair-group: separate tg->shares from task_group_lock
sched: fix incorrect irq lock usage in normalize_rt_tasks()
sched: rt-group: deal with PI
sched: rt-group: interface
sched: rt-group: make rt groups scheduling configurable
sched: rt-group: clean up the ifdeffery
sched: rt-group: refure unrunnable tasks
sched: rt-group: synchonised bandwidth period
sched: rt-group: smp balancing
sched: cleanup old and rarely used 'debug' features.
sched: fair-group scheduling vs latency
sched: fair-group: de-couple load-balancing from the rb-trees
sched: fair-group: single RQ approach
sched: remove sysctl_sched_batch_wakeup_granularity

Steven Rostedt (1):
rcu: add support for dynamic ticks and preempt rcu

 Documentation/sched-rt-group.txt |   59 +++
 include/linux/cgroup_subsys.h    |    2 +-
 include/linux/hardirq.h          |   10 +
 include/linux/init_task.h        |    3 +
 include/linux/rcuclassic.h       |    3 +
 include/linux/rcupreempt.h       |   22 ++
 include/linux/sched.h            |   27 +-
 init/Kconfig                     |   23 +-
 kernel/rcupreempt.c              |  224 +++++++++++-
 kernel/rtmutex.c                 |    5 +-
 kernel/sched.c                   |  766 +++++++++++++++++++++++++++++---------
 kernel/sched_debug.c             |    1 -
 kernel/sched_fair.c              |  381 ++++++++++---------
 kernel/sched_rt.c                |  262 ++++++++++---
 kernel/softirq.c                 |    1 +
 kernel/sysctl.c                  |   43 +--
 kernel/time/tick-sched.c         |    8 +-
 kernel/user.c                    |   78 ++++-
 18 files changed, 1444 insertions(+), 474 deletions(-)
 create mode 100644 Documentation/sched-rt-group.txt
Documentation/sched-rt-group.txt new file +59 -0
new file mode 100644
index 0000000..1c6332f
@@ -0,0 +1,59 @@
+
+
+Real-Time group scheduling.
+
+The problem space:
+
+In order to schedule multiple groups of realtime tasks each group must
+be assigned a fixed portion of the CPU time available. Without a minimum
+guarantee a realtime group can obviously fall short. A fuzzy upper limit
+is of no use since it cannot be relied upon. Which leaves us with just
+the single fixed portion.
+
+CPU time is divided by means of specifying how much time can be spent
+running in a given period. Say a frame fixed realtime renderer must
+deliver 25 frames a second, which yields a period of 0.04s. Now say
+it will also have to play some music and respond to input, leaving it
+with around 80% for the graphics. We can then give this group a runtime
+of 0.8 * 0.04s = 0.032s.
+
+This way the graphics group will have a 0.04s period with a 0.032s runtime
+limit.
+
+Now if the audio thread needs to refill the DMA buffer every 0.005s, but
+needs only about 3% CPU time to do so, it can do with a 0.03 * 0.005s
+= 0.00015s.
+
+
+The Interface:
+
+system wide:
+
+/proc/sys/kernel/sched_rt_period_ms
+/proc/sys/kernel/sched_rt_runtime_us
+
+CONFIG_FAIR_USER_SCHED
+
+/sys/kernel/uids/<uid>/cpu_rt_runtime_us
+
+or
+
+CONFIG_FAIR_CGROUP_SCHED
+
+/cgroup/<cgroup>/cpu.rt_runtime_us
+
+[ time is specified in us because the interface is s32; this gives an
+  operating range of ~35m to 1us ]
+
+The period takes values in [ 1, INT_MAX ], runtime in [ -1, INT_MAX - 1 ].
+
+A runtime of -1 specifies runtime == period, ie. no limit.
+
+New groups get the period from /proc/sys/kernel/sched_rt_period_us and
+a runtime of 0.
+
+Settings are constrained to:
+
+   \Sum_{i} runtime_{i} / global_period <= global_runtime / global_period
+
+in order to keep the configuration schedulable.
include/linux/cgroup_subsys.h +1 -1
index 228235c..ac6aad9 100644
@@ -25,7 +25,7 @@ SUBSYS(ns)
 
 /* */
 
-#ifdef CONFIG_FAIR_CGROUP_SCHED
+#ifdef CONFIG_CGROUP_SCHED
 SUBSYS(cpu_cgroup)
 #endif
 
include/linux/hardirq.h +10 -0
index 2961ec7..4982998 100644
@@ -109,6 +109,14 @@ static inline void account_system_vtime(struct task_struct *tsk)
 }
 #endif
 
+#if defined(CONFIG_PREEMPT_RCU) && defined(CONFIG_NO_HZ)
+extern void rcu_irq_enter(void);
+extern void rcu_irq_exit(void);
+#else
+# define rcu_irq_enter() do { } while (0)
+# define rcu_irq_exit() do { } while (0)
+#endif /* CONFIG_PREEMPT_RCU */
+
 /*
  * It is safe to do non-atomic ops on ->hardirq_context,
  * because NMI handlers may not preempt and the ops are
@@ -117,6 +125,7 @@ static inline void account_system_vtime(struct task_struct *tsk)
  */
 #define __irq_enter()					\
 	do {						\
+		rcu_irq_enter();			\
 		account_system_vtime(current);		\
 		add_preempt_count(HARDIRQ_OFFSET);	\
 		trace_hardirq_enter();			\
@@ -135,6 +144,7 @@ extern void irq_enter(void);
 		trace_hardirq_exit();			\
 		account_system_vtime(current);		\
 		sub_preempt_count(HARDIRQ_OFFSET);	\
+		rcu_irq_exit();				\
 	} while (0)
 
 /*
include/linux/init_task.h +3 -0
index 1f74e1d..37a6f5b 100644
@@ -151,6 +151,9 @@ extern struct group_info init_groups;
 	.cpus_allowed	= CPU_MASK_ALL,					\
 	.mm		= NULL,						\
 	.active_mm	= &init_mm,					\
+	.se		= {						\
+		.group_node 	= LIST_HEAD_INIT(tsk.se.group_node),	\
+	},								\
 	.rt		= {						\
 		.run_list	= LIST_HEAD_INIT(tsk.rt.run_list),	\
 		.time_slice	= HZ, 					\
include/linux/rcuclassic.h +3 -0
index 4d66242..b3dccd6 100644
@@ -160,5 +160,8 @@ extern void rcu_restart_cpu(int cpu);
 extern long rcu_batches_completed(void);
 extern long rcu_batches_completed_bh(void);
 
+#define rcu_enter_nohz()	do { } while (0)
+#define rcu_exit_nohz()		do { } while (0)
+
 #endif /* __KERNEL__ */
 #endif /* __LINUX_RCUCLASSIC_H */
include/linux/rcupreempt.h +22 -0
index 60c2a03..01152ed 100644
@@ -82,5 +82,27 @@ extern struct rcupreempt_trace *rcupreempt_trace_cpu(int cpu);
 
 struct softirq_action;
 
+#ifdef CONFIG_NO_HZ
+DECLARE_PER_CPU(long, dynticks_progress_counter);
+
+static inline void rcu_enter_nohz(void)
+{
+	__get_cpu_var(dynticks_progress_counter)++;
+	WARN_ON(__get_cpu_var(dynticks_progress_counter) & 0x1);
+	mb();
+}
+
+static inline void rcu_exit_nohz(void)
+{
+	mb();
+	__get_cpu_var(dynticks_progress_counter)++;
+	WARN_ON(!(__get_cpu_var(dynticks_progress_counter) & 0x1));
+}
+
+#else /* CONFIG_NO_HZ */
+#define rcu_enter_nohz()	do { } while (0)
+#define rcu_exit_nohz()		do { } while (0)
+#endif /* CONFIG_NO_HZ */
+
 #endif /* __KERNEL__ */
 #endif /* __LINUX_RCUPREEMPT_H */
include/linux/sched.h +20 -7
index 00e1441..3052436 100644
@@ -590,7 +590,7 @@ struct user_struct {
 	struct hlist_node uidhash_node;
 	uid_t uid;
 
-#ifdef CONFIG_FAIR_USER_SCHED
+#ifdef CONFIG_USER_SCHED
 	struct task_group *tg;
 #ifdef CONFIG_SYSFS
 	struct kobject kobj;
@@ -917,6 +917,7 @@ struct load_weight {
 struct sched_entity {
 	struct load_weight	load;		/* for load-balancing */
 	struct rb_node		run_node;
+	struct list_head	group_node;
 	unsigned int		on_rq;
 
 	u64			exec_start;
@@ -973,7 +974,7 @@ struct sched_rt_entity {
 	unsigned long timeout;
 	int nr_cpus_allowed;
 
-#ifdef CONFIG_FAIR_GROUP_SCHED
+#ifdef CONFIG_RT_GROUP_SCHED
 	struct sched_rt_entity	*parent;
 	/* rq on which this entity is (to be) queued: */
 	struct rt_rq		*rt_rq;
@@ -1536,13 +1537,10 @@ extern void sched_idle_next(void);
 extern unsigned int sysctl_sched_latency;
 extern unsigned int sysctl_sched_min_granularity;
 extern unsigned int sysctl_sched_wakeup_granularity;
-extern unsigned int sysctl_sched_batch_wakeup_granularity;
 extern unsigned int sysctl_sched_child_runs_first;
 extern unsigned int sysctl_sched_features;
 extern unsigned int sysctl_sched_migration_cost;
 extern unsigned int sysctl_sched_nr_migrate;
-extern unsigned int sysctl_sched_rt_period;
-extern unsigned int sysctl_sched_rt_ratio;
 #if defined(CONFIG_FAIR_GROUP_SCHED) && defined(CONFIG_SMP)
 extern unsigned int sysctl_sched_min_bal_int_shares;
 extern unsigned int sysctl_sched_max_bal_int_shares;
@@ -1552,6 +1550,12 @@ int sched_nr_latency_handler(struct ctl_table *table, int write,
 		struct file *file, void __user *buffer, size_t *length,
 		loff_t *ppos);
 #endif
+extern unsigned int sysctl_sched_rt_period;
+extern int sysctl_sched_rt_runtime;
+
+int sched_rt_handler(struct ctl_table *table, int write,
+		struct file *filp, void __user *buffer, size_t *lenp,
+		loff_t *ppos);
 
 extern unsigned int sysctl_sched_compat_yield;
 
@@ -2027,16 +2031,25 @@ extern int sched_mc_power_savings, sched_smt_power_savings;
 
 extern void normalize_rt_tasks(void);
 
-#ifdef CONFIG_FAIR_GROUP_SCHED
+#ifdef CONFIG_GROUP_SCHED
 
 extern struct task_group init_task_group;
 
 extern struct task_group *sched_create_group(void);
 extern void sched_destroy_group(struct task_group *tg);
 extern void sched_move_task(struct task_struct *tsk);
+#ifdef CONFIG_FAIR_GROUP_SCHED
 extern int sched_group_set_shares(struct task_group *tg, unsigned long shares);
 extern unsigned long sched_group_shares(struct task_group *tg);
-
+#endif
+#ifdef CONFIG_RT_GROUP_SCHED
+extern int sched_group_set_rt_runtime(struct task_group *tg,
+				      long rt_runtime_us);
+extern long sched_group_rt_runtime(struct task_group *tg);
+extern int sched_group_set_rt_period(struct task_group *tg,
+				      long rt_period_us);
+extern long sched_group_rt_period(struct task_group *tg);
+#endif
 #endif
 
 #ifdef CONFIG_TASK_XACCT
init/Kconfig +17 -6
index 824d48c..dcef8b5 100644
@@ -311,25 +311,36 @@ config CPUSETS
 
 	  Say N if unsure.
 
-config FAIR_GROUP_SCHED
-	bool "Fair group CPU scheduler"
+config GROUP_SCHED
+	bool "Group CPU scheduler"
 	default y
 	help
 	  This feature lets CPU scheduler recognize task groups and control CPU
 	  bandwidth allocation to such task groups.
 
+config FAIR_GROUP_SCHED
+	bool "Group scheduling for SCHED_OTHER"
+	depends on GROUP_SCHED
+	default y
+
+config RT_GROUP_SCHED
+	bool "Group scheduling for SCHED_RR/FIFO"
+	depends on EXPERIMENTAL
+	depends on GROUP_SCHED
+	default n
+
 choice
-	depends on FAIR_GROUP_SCHED
+	depends on GROUP_SCHED
 	prompt "Basis for grouping tasks"
-	default FAIR_USER_SCHED
+	default USER_SCHED
 
-config FAIR_USER_SCHED
+config USER_SCHED
 	bool "user id"
 	help
 	  This option will choose userid as the basis for grouping
 	  tasks, thus providing equal CPU bandwidth to each user.
 
-config FAIR_CGROUP_SCHED
+config CGROUP_SCHED
 	bool "Control groups"
  	depends on CGROUPS
  	help
kernel/rcupreempt.c +220 -4

262 lines not on this page · raw message

kernel/rtmutex.c +4 -1
index 0deef71..6522ae5 100644
@@ -630,9 +630,12 @@ rt_mutex_slowlock(struct rt_mutex *lock, int state,
 	set_current_state(state);
 
 	/* Setup the timer, when timeout != NULL */
-	if (unlikely(timeout))
+	if (unlikely(timeout)) {
 		hrtimer_start(&timeout->timer, timeout->timer.expires,
 			      HRTIMER_MODE_ABS);
+		if (!hrtimer_active(&timeout->timer))
+			timeout->task = NULL;
+	}
 
 	for (;;) {
 		/* Try to acquire the lock: */
kernel/sched.c +594 -172

1,190 lines not on this page · raw message

kernel/sched_debug.c +0 -1
index 4b5e24c..8b2c1fd 100644
@@ -214,7 +214,6 @@ static int sched_debug_show(struct seq_file *m, void *v)
 	PN(sysctl_sched_latency);
 	PN(sysctl_sched_min_granularity);
 	PN(sysctl_sched_wakeup_granularity);
-	PN(sysctl_sched_batch_wakeup_granularity);
 	PN(sysctl_sched_child_runs_first);
 	P(sysctl_sched_features);
 #undef PN
kernel/sched_fair.c +195 -186

593 lines not on this page · raw message

kernel/sched_rt.c +205 -57

372 lines not on this page · raw message

kernel/softirq.c +1 -0
index 5b3aea5..31e9f2a 100644
@@ -313,6 +313,7 @@ void irq_exit(void)
 	/* Make sure that timer wheel updates are propagated */
 	if (!in_interrupt() && idle_cpu(smp_processor_id()) && !need_resched())
 		tick_nohz_stop_sched_tick();
+	rcu_irq_exit();
 #endif
 	preempt_enable_no_resched();
 }
kernel/sysctl.c +16 -27

61 lines not on this page · raw message

kernel/time/tick-sched.c +3 -5
index fa9bb73..769deb9 100644
@@ -192,7 +192,6 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time)
 void tick_nohz_stop_sched_tick(void)
 {
 	unsigned long seq, last_jiffies, next_jiffies, delta_jiffies, flags;
-	unsigned long rt_jiffies;
 	struct tick_sched *ts;
 	ktime_t last_update, expires, now;
 	struct clock_event_device *dev = __get_cpu_var(tick_cpu_device).evtdev;
@@ -245,10 +244,6 @@ void tick_nohz_stop_sched_tick(void)
 	next_jiffies = get_next_timer_interrupt(last_jiffies);
 	delta_jiffies = next_jiffies - last_jiffies;
 
-	rt_jiffies = rt_needs_cpu(cpu);
-	if (rt_jiffies && rt_jiffies < delta_jiffies)
-		delta_jiffies = rt_jiffies;
-
 	if (rcu_needs_cpu(cpu))
 		delta_jiffies = 1;
 	/*
@@ -282,6 +277,7 @@ void tick_nohz_stop_sched_tick(void)
 			ts->idle_tick = ts->sched_timer.expires;
 			ts->tick_stopped = 1;
 			ts->idle_jiffies = last_jiffies;
+			rcu_enter_nohz();
 		}
 
 		/*
@@ -375,6 +371,8 @@ void tick_nohz_restart_sched_tick(void)
 		return;
 	}
 
+	rcu_exit_nohz();
+
 	/* Update jiffies first */
 	select_nohz_load_balancer(0);
 	now = ktime_get();
kernel/user.c +71 -7

130 lines not on this page · raw message

#5 lore raw
Quoted 10 lines · Ingo Molnar

no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:

   git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.git

combo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.

ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is fec13e45305d69fd0bd23b30bd05a0a42cf341f8.

Ingo

kernel/sched.c +5 -0
Index: linux-2.6/kernel/sched.c
===================================================================
@@ -219,6 +219,10 @@ static void start_rt_bandwidth(struct rt
 	if (rt_b->rt_runtime == RUNTIME_INF)
 		return;
 
+	if (hrtimer_active(&rt_b->rt_period_timer))
+		return;
+
+	spin_lock(&rt_b->rt_runtime_lock);
 	for (;;) {
 		if (hrtimer_active(&rt_b->rt_period_timer))
 			break;
@@ -229,6 +233,7 @@ static void start_rt_bandwidth(struct rt
 			      rt_b->rt_period_timer.expires,
 			      HRTIMER_MODE_ABS);
 	}
+	spin_unlock(&rt_b->rt_runtime_lock);
 }
 
 #ifdef CONFIG_RT_GROUP_SCHED
#6 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Ingo Molnar

no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:

   git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.git
Quoted 8 lines
Quoted 3 lines

combo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.

ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is fec13e45305d69fd0bd23b30bd05a0a42cf341f8.

With the previous patch and this patch applied, the issue is not reproducible here.

Thanks,
Rafael

Quoted 25 lines

Index: linux-2.6/kernel/sched.c
===================================================================

kernel/sched.c +5 -0
@@ -219,6 +219,10 @@ static void start_rt_bandwidth(struct rt
 	if (rt_b->rt_runtime == RUNTIME_INF)
 		return;
 
+	if (hrtimer_active(&rt_b->rt_period_timer))
+		return;
+
+	spin_lock(&rt_b->rt_runtime_lock);
 	for (;;) {
 		if (hrtimer_active(&rt_b->rt_period_timer))
 			break;
@@ -229,6 +233,7 @@ static void start_rt_bandwidth(struct rt
 			      rt_b->rt_period_timer.expires,
 			      HRTIMER_MODE_ABS);
 	}
+	spin_unlock(&rt_b->rt_runtime_lock);
 }
 
 #ifdef CONFIG_RT_GROUP_SCHED
#7 lore raw
Quoted 20 lines · Rafael J. Wysocki
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Ingo Molnar

no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:

   git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.git
Quoted 8 lines
Quoted 3 lines

combo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.

ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is fec13e45305d69fd0bd23b30bd05a0a42cf341f8.

Quoted 5 lines

With the previous patch and this patch applied, the issue is not reproducible here.

Thanks,
Rafael

The problem is fixed for me as well with the previous patch + the patch
below, VKTM now enters S state and Oracle shuts down properly again.

Thanks !

Quoted 22 lines
Quoted 22 lines

Index: linux-2.6/kernel/sched.c
===================================================================

kernel/sched.c +5 -0
@@ -219,6 +219,10 @@ static void start_rt_bandwidth(struct rt
      if (rt_b->rt_runtime == RUNTIME_INF)
              return;
 
+     if (hrtimer_active(&rt_b->rt_period_timer))
+             return;
+
+     spin_lock(&rt_b->rt_runtime_lock);
      for (;;) {
              if (hrtimer_active(&rt_b->rt_period_timer))
                      break;
@@ -229,6 +233,7 @@ static void start_rt_bandwidth(struct rt
                            rt_b->rt_period_timer.expires,
                            HRTIMER_MODE_ABS);
      }
+     spin_unlock(&rt_b->rt_runtime_lock);
 }
 
Quoted 1 line

kernel/sched.c static void start_rt_bandwidth(struct rt · line 239

Quoted 1 line
kernel/sched.c +0 -0
 #ifdef CONFIG_RT_GROUP_SCHED

--alessandro

"We act as though comfort and luxury were the chief requirements
of life, when all that we need to make us really happy is something to be enthusiastic about."

(Charles Kingsley)

#8 lore raw
Quoted 11 lines · Alessandro Suardi
Quoted 7 lines
Quoted 4 lines

ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is fec13e45305d69fd0bd23b30bd05a0a42cf341f8.

With the previous patch and this patch applied, the issue is not reproducible here.

The problem is fixed for me as well with the previous patch + the patch below, VKTM now enters S state and Oracle shuts down properly again.

thanks alot for testing this. The scheduler queue is now looking good in testing, will probably send a pull request to Linus later today.

Ingo

#9 lore raw
In reply to Rafael J. Wysocki
Quoted 15 lines · Rafael J. Wysocki
Quoted 14 lines · Ingo Molnar
Quoted 10 lines · Ingo Molnar

no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:

   git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.git

combo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.

Quoted 7 lines
Quoted 4 lines

ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is fec13e45305d69fd0bd23b30bd05a0a42cf341f8.

With the previous patch and this patch applied, the issue is not reproducible here.

Did you enable CONFIG_RT_GROUP_SCHED (it defaults to n)?

If you didn't, could you try with it set to y?

#10 lore raw
Quoted 22 lines · Peter Zijlstra
Quoted 20 lines · Rafael J. Wysocki
Quoted 18 lines · Ingo Molnar
Quoted 10 lines · Ingo Molnar

no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:

   git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.git

combo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.

ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is fec13e45305d69fd0bd23b30bd05a0a42cf341f8.

Quoted 6 lines
Quoted 2 lines

With the previous patch and this patch applied, the issue is not reproducible here.

Did you enable CONFIG_RT_GROUP_SCHED (it defaults to n)?

If you didn't, could you try with it set to y?

I just rebuilt 2.6.25-rc1-git2 with Ingo's patch and your patch on top,
and the Oracle VKTM issue is still gone even with

[asuardi@sandman ~]$ grep GROUP_SCHED
/share/src/linux-2.6.25-rc1-git2-orafix/.config
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
# CONFIG_CGROUP_SCHED is not set

so it's good for me.

Or is it necessary to also enable CONFIG_CGROUP_SCHED and retest ?

--alessandro

"We act as though comfort and luxury were the chief requirements
of life, when all that we need to make us really happy is something to be enthusiastic about."

(Charles Kingsley)

#11 lore raw
Quoted 40 lines · Alessandro Suardi
Quoted 22 lines · Peter Zijlstra
Quoted 20 lines · Rafael J. Wysocki
Quoted 18 lines · Ingo Molnar
Quoted 10 lines · Ingo Molnar

no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:

   git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.git

combo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.

ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is fec13e45305d69fd0bd23b30bd05a0a42cf341f8.

Quoted 6 lines
Quoted 2 lines

With the previous patch and this patch applied, the issue is not reproducible here.

Did you enable CONFIG_RT_GROUP_SCHED (it defaults to n)?

If you didn't, could you try with it set to y?

I just rebuilt 2.6.25-rc1-git2 with Ingo's patch and your patch on top,
and the Oracle VKTM issue is still gone even with

[asuardi@sandman ~]$ grep GROUP_SCHED
/share/src/linux-2.6.25-rc1-git2-orafix/.config
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
# CONFIG_CGROUP_SCHED is not set

Quoted 3 lines

so it's good for me.

Or is it necessary to also enable CONFIG_CGROUP_SCHED and retest ?

No that should be quite all-right. Thanks for testing!

#12 lore raw
In reply to Peter Zijlstra
Quoted 22 lines · Peter Zijlstra
Quoted 20 lines · Rafael J. Wysocki
Quoted 18 lines · Ingo Molnar
Quoted 10 lines · Ingo Molnar

no, they were not lost, they just didnt pass QA here (they crashed on a particularly hard to debug 8-way box i have) and Peter worked on that queue of fixes up until today to get it really correct. Could you check:

   git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched.git

combo patch below as well - whichever you prefer. The shortlog can be found below as well - but i dont yet consider this pullable, i'd like it to see pass a full night of randconfig tests on my test-systems.

ok, we just found the reason for the 8-way crash, the delta fix from Peter is below if any of you have tried the previous combo patch. Updated sched.git as well, new HEAD is fec13e45305d69fd0bd23b30bd05a0a42cf341f8.

Quoted 6 lines
Quoted 2 lines

With the previous patch and this patch applied, the issue is not reproducible here.

Did you enable CONFIG_RT_GROUP_SCHED (it defaults to n)?

If you didn't, could you try with it set to y?

Tested with CONFIG_RT_GROUP_SCHED set and it also works as expected.

Thanks,
Rafael