Skip to contentSkip to conversation

oom-kill: give the dying task a higher priority

RFC In review Luis Claudio R. Goncalves 8 participants 55 messages May 27, 2010 – Jun 3, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 oom-kill: give the dying task a higher priority

Hello,

Could you please review this patch?

The idea behind it is quite simple: give the dying task a higher priority so that it can be scheduled sooner and die to free memory.

oom-kill: give the dying task a higher priority

In a system under heavy load it was observed that even after the oom-killer selects a task to die, the task may take a long time to die.

Right before sending a SIGKILL to the selected task the oom-killer increases the task priority so that it can exit quickly, freeing memory. That is accomplished by:

/*
* We give our sacrificial lamb high priority and access to
* all the memory it needs. That way it should be able to
* exit() and clear out its resources quickly... */
p->rt.time_slice = HZ;
set_tsk_thread_flag(p, TIF_MEMDIE);

It sounds plausible giving the dying task an even higher priority to be sure it will be scheduled sooner and free the desired memory.

mm/oom_kill.c +4 -0
index b68e802..8047309 100644
@@ -382,6 +382,8 @@ static void dump_header(struct task_struct *p, gfp_t gfp_mask, int order,
  */
 static void __oom_kill_task(struct task_struct *p, int verbose)
 {
+	struct sched_param param;
+
 	if (is_global_init(p)) {
 		WARN_ON(1);
 		printk(KERN_WARNING "tried to kill init!\n");
@@ -413,6 +415,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 */
 	p->rt.time_slice = HZ;
 	set_tsk_thread_flag(p, TIF_MEMDIE);
+	param.sched_priority = MAX_RT_PRIO-1;
+	sched_setscheduler(p, SCHED_FIFO, &param);
 
 	force_sig(SIGKILL, p);
 }

Thanks,
Luis

Signature · 8 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw
Quoted 3 lines · Luis Claudio R. Goncalves

It sounds plausible giving the dying task an even higher priority to be sure it will be scheduled sooner and free the desired memory.

As usual, I can't really comment the changes in oom logic, just minor nits...

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 421

Quoted 8 lines
mm/oom_kill.c +2 -0
@@ -413,6 +415,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 */
 	p->rt.time_slice = HZ;
 	set_tsk_thread_flag(p, TIF_MEMDIE);
+	param.sched_priority = MAX_RT_PRIO-1;
+	sched_setscheduler(p, SCHED_FIFO, &param);
 
 	force_sig(SIGKILL, p);

Probably sched_setscheduler_nocheck() makes more sense.

Minor, but perhaps it would be a bit better to send SIGKILL first, then raise its prio.

Oleg.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw

Hi Luis,

Quoted 15 lines · Luis Claudio R. Goncalves
Quoted 3 lines

It sounds plausible giving the dying task an even higher priority to be sure it will be scheduled sooner and free the desired memory.

As usual, I can't really comment the changes in oom logic, just minor nits...

Quoted 7 lines
mm/oom_kill.c +2 -0
@@ -413,6 +415,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 */
 	p->rt.time_slice = HZ;
 	set_tsk_thread_flag(p, TIF_MEMDIE);
+	param.sched_priority = MAX_RT_PRIO-1;
+	sched_setscheduler(p, SCHED_FIFO, &param);
 
Quoted 6 lines

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 421

Quoted 1 line
mm/oom_kill.c +0 -0
 	force_sig(SIGKILL, p);

Probably sched_setscheduler_nocheck() makes more sense.

Minor, but perhaps it would be a bit better to send SIGKILL first, then raise its prio.

I have no objection too. but I don't think Oleg's pointed thing is minor. Please send updated patch.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw

On Fri, May 28, 2010 at 11:54:07AM +0900, KOSAKI Motohiro wrote:

| Hi Luis,
| 
| > On 05/27, Luis Claudio R. Goncalves wrote:
| > >
| > > It sounds plausible giving the dying task an even higher priority to be
| > > sure it will be scheduled sooner and free the desired memory.
| > 
| > As usual, I can't really comment the changes in oom logic, just minor
| > nits...
| > 
| > > @@ -413,6 +415,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
| > >  	 */
| > >  	p->rt.time_slice = HZ;
| > >  	set_tsk_thread_flag(p, TIF_MEMDIE);
| > > +	param.sched_priority = MAX_RT_PRIO-1;
| > > +	sched_setscheduler(p, SCHED_FIFO, &param);
| > >
| > >  	force_sig(SIGKILL, p);
| > 
| > Probably sched_setscheduler_nocheck() makes more sense.
| > 
| > Minor, but perhaps it would be a bit better to send SIGKILL first,
| > then raise its prio.
| 
| I have no objection too. but I don't think Oleg's pointed thing is minor.
| Please send updated patch.
| 
| Thanks.

This version of the patch addresses the suggestions from Oleg Nesterov and Kosaki Motohiro.

Thanks again for reviewing the patch.

oom-kill: give the dying task a higher priority (v2)

In a system under heavy load it was observed that even after the oom-killer selects a task to die, the task may take a long time to die.

Right before sending a SIGKILL to the task selected by the oom-killer this task has it's priority increased so that it can exit() exit soon, freeing memory. That is accomplished by:

/*
* We give our sacrificial lamb high priority and access to
* all the memory it needs. That way it should be able to
* exit() and clear out its resources quickly... */
p->rt.time_slice = HZ;
set_tsk_thread_flag(p, TIF_MEMDIE);

It sounds plausible giving the dying task an even higher priority to be sure it will be scheduled sooner and free the desired memory. Oleg Nesterov pointed out it would be interesting sending the signal before increasing the task priority.

mm/oom_kill.c +4 -1
index b68e802..d352b3e 100644
@@ -382,6 +382,8 @@ static void dump_header(struct task_struct *p, gfp_t gfp_mask, int order,
  */
 static void __oom_kill_task(struct task_struct *p, int verbose)
 {
+	struct sched_param param;
+
 	if (is_global_init(p)) {
 		WARN_ON(1);
 		printk(KERN_WARNING "tried to kill init!\n");
@@ -413,8 +415,9 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 */
 	p->rt.time_slice = HZ;
 	set_tsk_thread_flag(p, TIF_MEMDIE);
-
 	force_sig(SIGKILL, p);
+	param.sched_priority = MAX_RT_PRIO-1;
+	sched_setscheduler_nocheck(p, SCHED_FIFO, &param);
 }
 
 static int oom_kill_task(struct task_struct *p)
Signature · 2 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]
#4 lore raw
Quoted 10 lines · Luis Claudio R. Goncalves
mm/oom_kill.c +2 -0
@@ -382,6 +382,8 @@ static void dump_header(struct task_struct *p, gfp_t gfp_mask, int order,
  */
 static void __oom_kill_task(struct task_struct *p, int verbose)
 {
+	struct sched_param param;
+
 	if (is_global_init(p)) {
 		WARN_ON(1);
 		printk(KERN_WARNING "tried to kill init!\n");

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 422

Quoted 9 lines
mm/oom_kill.c +2 -1
@@ -413,8 +415,9 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 */
 	p->rt.time_slice = HZ;
 	set_tsk_thread_flag(p, TIF_MEMDIE);
-
 	force_sig(SIGKILL, p);
+	param.sched_priority = MAX_RT_PRIO-1;
+	sched_setscheduler_nocheck(p, SCHED_FIFO, &param);
 }
 

I would like to understand the visible benefits of this patch. Have you seen an OOM kill tasked really get bogged down. Should this task really be competing with other important tasks for run time?

Signature · 8 lines
	Three Cheers,
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#5 lore raw
Quoted 17 lines · Luis Claudio R. Goncalves
Quoted 16 lines
mm/oom_kill.c +2 -1
@@ -382,6 +382,8 @@ static void dump_header(struct task_struct *p, gfp_t gfp_mask, int order,
  */
 static void __oom_kill_task(struct task_struct *p, int verbose)
 {
+	struct sched_param param;
+
 	if (is_global_init(p)) {
 		WARN_ON(1);
 		printk(KERN_WARNING "tried to kill init!\n");
@@ -413,8 +415,9 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 */
 	p->rt.time_slice = HZ;
 	set_tsk_thread_flag(p, TIF_MEMDIE);
-
 	force_sig(SIGKILL, p);
Quoted 8 lines

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 422

Quoted 4 lines
mm/oom_kill.c +2 -0
+	param.sched_priority = MAX_RT_PRIO-1;
+	sched_setscheduler_nocheck(p, SCHED_FIFO, &param);
 }
 

I would like to understand the visible benefits of this patch. Have you seen an OOM kill tasked really get bogged down. Should this task really be competing with other important tasks for run time?

What you mean important? Until OOM victim task exit completely, the system have no memory. all of important task can't do anything.

In almost kernel subsystems, automatically priority boost is really bad idea because it may break RT task's deterministic behavior. but OOM is one of exception. The deterministic was alread broken by memory starvation.

That's the reason I acked it.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw

Hi, Kosaki.

Quoted 26 lines · KOSAKI Motohiro
Quoted 17 lines · Luis Claudio R. Goncalves
Quoted 16 lines
mm/oom_kill.c +2 -1
@@ -382,6 +382,8 @@ static void dump_header(struct task_struct *p, gfp_t gfp_mask, int order,
   */
  static void __oom_kill_task(struct task_struct *p, int verbose)
  {
+   struct sched_param param;
+
     if (is_global_init(p)) {
             WARN_ON(1);
             printk(KERN_WARNING "tried to kill init!\n");
@@ -413,8 +415,9 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
      */
     p->rt.time_slice = HZ;
     set_tsk_thread_flag(p, TIF_MEMDIE);
-
     force_sig(SIGKILL, p);
Quoted 8 lines

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 422

Quoted 4 lines
mm/oom_kill.c +2 -0
+   param.sched_priority = MAX_RT_PRIO-1;
+   sched_setscheduler_nocheck(p, SCHED_FIFO, &param);
  }
 

I would like to understand the visible benefits of this patch. Have you seen an OOM kill tasked really get bogged down. Should this task really be competing with other important tasks for run time?

Quoted 6 lines

What you mean important? Until OOM victim task exit completely, the system have no memory. all of important task can't do anything.

In almost kernel subsystems, automatically priority boost is really bad idea because it may break RT task's deterministic behavior. but OOM is one of exception. The deterministic was alread broken by memory starvation.

Yes or No.

IMHO, normally RT tasks shouldn't use dynamic allocation(ie, non-deterministic functions or system calls) in place which is needed deterministic. So memory starvation might not break real-time deterministic.

Quoted 11 lines

That's the reason I acked it.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#7 lore raw

Hi

Quoted 37 lines

Hi, Kosaki.

Quoted 26 lines · KOSAKI Motohiro
Quoted 17 lines · Luis Claudio R. Goncalves
Quoted 16 lines
(quoted diff) +0 -0
@@ -382,6 +382,8 @@ static void dump_header(struct task_struct *p, gfp_t gfp_mask, int order,

A */
A static void __oom_kill_task(struct task_struct *p, int verbose) A {
+ A struct sched_param param;
+

A  A  if (is_global_init(p)) {
A  A  A  A  A  A  WARN_ON(1);
A  A  A  A  A  A  printk(KERN_WARNING "tried to kill init!\n");
(quoted diff) +0 -0
@@ -413,8 +415,9 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
A  A  A */
A  A  p->rt.time_slice = HZ;
A  A  set_tsk_thread_flag(p, TIF_MEMDIE);

-
A A force_sig(SIGKILL, p);

Quoted 8 lines
Quoted 4 lines

+ A param.sched_priority = MAX_RT_PRIO-1;
+ A sched_setscheduler_nocheck(p, SCHED_FIFO, &param); A }

I would like to understand the visible benefits of this patch. Have you seen an OOM kill tasked really get bogged down. Should this task really be competing with other important tasks for run time?

Quoted 6 lines

What you mean important? Until OOM victim task exit completely, the system have no memory. all of important task can't do anything.

In almost kernel subsystems, automatically priority boost is really bad idea because it may break RT task's deterministic behavior. but OOM is one of exception. The deterministic was alread broken by memory starvation.

Quoted 6 lines

Yes or No.

IMHO, normally RT tasks shouldn't use dynamic allocation(ie, non-deterministic functions or system calls) in place which is needed deterministic. So memory starvation might not break real-time deterministic.

I think It's impossible. Normally RT task use mlock and it prevent almost page allocation. but every syscall internally call kmalloc(). They can't avoid it practically.

How do you perfectly avoid dynamic allocation?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#8 lore raw
Quoted 46 lines · KOSAKI Motohiro

Hi

Quoted 37 lines

Hi, Kosaki.

Quoted 26 lines · KOSAKI Motohiro
Quoted 17 lines · Luis Claudio R. Goncalves
Quoted 16 lines
mm/oom_kill.c +2 -1
@@ -382,6 +382,8 @@ static void dump_header(struct task_struct *p, gfp_t gfp_mask, int order,
   */
  static void __oom_kill_task(struct task_struct *p, int verbose)
  {
+   struct sched_param param;
+
     if (is_global_init(p)) {
             WARN_ON(1);
             printk(KERN_WARNING "tried to kill init!\n");
@@ -413,8 +415,9 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
      */
     p->rt.time_slice = HZ;
     set_tsk_thread_flag(p, TIF_MEMDIE);
-
     force_sig(SIGKILL, p);
Quoted 8 lines

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 422

Quoted 4 lines
mm/oom_kill.c +2 -0
+   param.sched_priority = MAX_RT_PRIO-1;
+   sched_setscheduler_nocheck(p, SCHED_FIFO, &param);
  }
 

I would like to understand the visible benefits of this patch. Have you seen an OOM kill tasked really get bogged down. Should this task really be competing with other important tasks for run time?

Quoted 6 lines

What you mean important? Until OOM victim task exit completely, the system have no memory. all of important task can't do anything.

In almost kernel subsystems, automatically priority boost is really bad idea because it may break RT task's deterministic behavior. but OOM is one of exception. The deterministic was alread broken by memory starvation.

Quoted 6 lines

Yes or No.

IMHO, normally RT tasks shouldn't use dynamic allocation(ie, non-deterministic functions or system calls) in place which is needed deterministic. So memory starvation might not break real-time deterministic.

Quoted 5 lines

I think It's impossible. Normally RT task use mlock and it prevent almost page allocation. but every syscall internally call kmalloc(). They can't avoid it practically.

How do you perfectly avoid dynamic allocation?

RT Task

void non-RT-function()
{
system call();
buffer = malloc();
memset(buffer);
}
/*
* We make sure this function must be executed in some millisecond */
void RT-function()
{
some calculation(); <- This doesn't have no dynamic characteristic
}
int main()
{
non-RT-function();
/* This function make sure RT-function cannot preempt by others */ set_RT_max_high_priority();
RT-function A();
set_normal_priority();
non-RT-function();
}

We don't want realtime in whole function of the task. What we want is just RT-function A.
Of course, current Linux cannot make perfectly sure RT-functionA can not preempt by others.
That's because some interrupt or exception happen. But RT-function A doesn't related to any dynamic characteristic. What can justify to preempt RT-function A by other processes?

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#9 lore raw
Quoted 25 lines

RT Task

void non-RT-function()

{
   system call();
   buffer = malloc();
   memset(buffer);
}

/*
* We make sure this function must be executed in some millisecond */
void RT-function()
{
some calculation(); <- This doesn't have no dynamic characteristic
}
int main()

{
   non-RT-function();
   /* This function make sure RT-function cannot preempt by others */
   set_RT_max_high_priority();
   RT-function A();
   set_normal_priority();
   non-RT-function();
}
Quoted 7 lines

We don't want realtime in whole function of the task. What we want is just RT-function A.
Of course, current Linux cannot make perfectly sure RT-functionA can not preempt by others.
That's because some interrupt or exception happen. But RT-function A doesn't related to any dynamic characteristic. What can justify to preempt RT-function A by other processes?

As far as my observation, RT-function always have some syscall. because pure calculation doesn't need deterministic guarantee. But _if_ you are really using such priority design. I'm ok maximum NonRT priority instead maximum RT priority too.

Luis, NonRT high priority break your use case? and if yes, can you please explain the reason?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#10 lore raw
Quoted 33 lines · KOSAKI Motohiro
Quoted 25 lines

RT Task

void non-RT-function()
{
   system call();
   buffer = malloc();
   memset(buffer);
}
/*
 * We make sure this function must be executed in some millisecond */
void RT-function()
{
   some calculation(); <- This doesn't have no dynamic characteristic }
int main()
{
   non-RT-function();
   /* This function make sure RT-function cannot preempt by others */ set_RT_max_high_priority();
   RT-function A();
   set_normal_priority();
   non-RT-function();
}

Quoted 7 lines

We don't want realtime in whole function of the task. What we want is just RT-function A.
Of course, current Linux cannot make perfectly sure RT-functionA can not preempt by others.
That's because some interrupt or exception happen. But RT-function A doesn't related to any dynamic characteristic. What can justify to preempt RT-function A by other processes?

Quoted 4 lines

As far as my observation, RT-function always have some syscall. because pure calculation doesn't need deterministic guarantee. But _if_ you are really using such priority design. I'm ok maximum NonRT priority instead maximum RT priority too.

Hmm. It's just example. but it would be not good exmaple. Let's change it with this.

void RT-function()

{
     int result = some calculation(); <- This doesn't have no dynamic

characteristic
*mmap_base = result; <-- mmap_base is mapped by GPIO device.
}

Could we allow preemption of this RT function due to other task's memory pressure?
Of course, Linux is not Hard RT featured OS, I think. So I thinks it is a policy problem.
If we think system memory pressure is more important than RT task and we _all_ agree such policy, we can allow it.

But I don't hope it.

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#11 lore raw
In reply to KOSAKI Motohiro

On Fri, May 28, 2010 at 02:59:02PM +0900, KOSAKI Motohiro wrote:

| > RT Task
| > 
| > void non-RT-function()
| > {
| >    system call();
| >    buffer = malloc();
| >    memset(buffer);
| > }
| > /*
| >  * We make sure this function must be executed in some millisecond
| >  */
| > void RT-function()
| > {
| >    some calculation(); <- This doesn't have no dynamic characteristic
| > }
| > int main()
| > {
| >    non-RT-function();
| >    /* This function make sure RT-function cannot preempt by others */
| >    set_RT_max_high_priority();
| >    RT-function A();
| >    set_normal_priority();
| >    non-RT-function();
| > }
| > 
| > We don't want realtime in whole function of the task. What we want is
| > just RT-function A.
| > Of course, current Linux cannot make perfectly sure RT-functionA can
| > not preempt by others.
| > That's because some interrupt or exception happen. But RT-function A
| > doesn't related to any dynamic characteristic. What can justify to
| > preempt RT-function A by other processes?
| 
| As far as my observation, RT-function always have some syscall. because pure
| calculation doesn't need deterministic guarantee. But _if_ you are really
| using such priority design. I'm ok maximum NonRT priority instead maximum
| RT priority too.

I confess I failed to distinguish memcg OOM and system OOM and used "in case of OOM kill the selected task the faster you can" as the guideline. If the exit code path is short that shouldn't be a problem.

Maybe the right way to go would be giving the dying task the biggest priority inside that memcg to be sure that it will be the next process from that memcg to be scheduled. Would that be reasonable?

| Luis, NonRT high priority break your use case? and if yes, can you please | explain the reason?

Most of my tests are in the realtime land, usually with premmpt_rt kernels. In this case, an RT priority will be usually necessary. But that is not the general case and I agree that a smoother (but not slower) solution could be devised by just increasing the dying process'priority to be high enough for the process set (memcg) it belongs to.

What is the impact of a memcg OOM for the system as a whole? That should be factored in the equation too.

Luis

Signature · 8 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#12 lore raw
Quoted 39 lines · Luis Claudio R. Goncalves

On Fri, May 28, 2010 at 02:59:02PM +0900, KOSAKI Motohiro wrote:

| > RT Task
| > 
| > void non-RT-function()
| > {
| >    system call();
| >    buffer = malloc();
| >    memset(buffer);
| > }
| > /*
| >  * We make sure this function must be executed in some millisecond
| >  */
| > void RT-function()
| > {
| >    some calculation(); <- This doesn't have no dynamic characteristic
| > }
| > int main()
| > {
| >    non-RT-function();
| >    /* This function make sure RT-function cannot preempt by others */
| >    set_RT_max_high_priority();
| >    RT-function A();
| >    set_normal_priority();
| >    non-RT-function();
| > }
| > 
| > We don't want realtime in whole function of the task. What we want is
| > just RT-function A.
| > Of course, current Linux cannot make perfectly sure RT-functionA can
| > not preempt by others.
| > That's because some interrupt or exception happen. But RT-function A
| > doesn't related to any dynamic characteristic. What can justify to
| > preempt RT-function A by other processes?
| 
| As far as my observation, RT-function always have some syscall. because pure
| calculation doesn't need deterministic guarantee. But _if_ you are really
| using such priority design. I'm ok maximum NonRT priority instead maximum
| RT priority too.
Quoted 7 lines

I confess I failed to distinguish memcg OOM and system OOM and used "in case of OOM kill the selected task the faster you can" as the guideline. If the exit code path is short that shouldn't be a problem.

Maybe the right way to go would be giving the dying task the biggest priority inside that memcg to be sure that it will be the next process from that memcg to be scheduled. Would that be reasonable?

Hmm. I can't understand your point.
What do you mean failing distinguish memcg and system OOM?

We already have been distinguish it by mem_cgroup_out_of_memory. (but we have to enable CONFIG_CGROUP_MEM_RES_CTLR).
So task selected in select_bad_process is one out of memcg's tasks when memcg have a memory pressure.

Isn't it enough?

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#13 lore raw
Quoted 8 lines · MinChan Kim
Quoted 7 lines

I confess I failed to distinguish memcg OOM and system OOM and used "in case of OOM kill the selected task the faster you can" as the guideline. If the exit code path is short that shouldn't be a problem.

Maybe the right way to go would be giving the dying task the biggest priority inside that memcg to be sure that it will be the next process from that memcg to be scheduled. Would that be reasonable?

Quoted 8 lines

Hmm. I can't understand your point.
What do you mean failing distinguish memcg and system OOM?

We already have been distinguish it by mem_cgroup_out_of_memory. (but we have to enable CONFIG_CGROUP_MEM_RES_CTLR).
So task selected in select_bad_process is one out of memcg's tasks when memcg have a memory pressure.

We have a routine to help figure out if the task belongs to the memory cgroup that cause the OOM. The OOM entry from memory cgroup is different from a regular one.

Signature · 8 lines
	Three Cheers,
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#14 lore raw
Quoted 19 lines · Balbir Singh
Quoted 8 lines · MinChan Kim
Quoted 7 lines

I confess I failed to distinguish memcg OOM and system OOM and used "in case of OOM kill the selected task the faster you can" as the guideline. If the exit code path is short that shouldn't be a problem.

Maybe the right way to go would be giving the dying task the biggest priority inside that memcg to be sure that it will be the next process from that memcg to be scheduled. Would that be reasonable?

Quoted 8 lines

Hmm. I can't understand your point.
What do you mean failing distinguish memcg and system OOM?

We already have been distinguish it by mem_cgroup_out_of_memory. (but we have to enable CONFIG_CGROUP_MEM_RES_CTLR).
So task selected in select_bad_process is one out of memcg's tasks when memcg have a memory pressure.

Quoted 3 lines

We have a routine to help figure out if the task belongs to the memory cgroup that cause the OOM. The OOM entry from memory cgroup is different from a regular one.

I meant it.
My english is poor. "out of" isn't proper.

Quoted 4 lines
Signature · 3 lines
-- 
	Three Cheers,
	Balbir
Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#15 lore raw
In reply to Minchan Kim

On Fri, May 28, 2010 at 11:06:23PM +0900, Minchan Kim wrote:
| On Fri, May 28, 2010 at 09:53:05AM -0300, Luis Claudio R. Goncalves wrote:
| > On Fri, May 28, 2010 at 02:59:02PM +0900, KOSAKI Motohiro wrote:
...

| > | As far as my observation, RT-function always have some syscall. because pure
| > | calculation doesn't need deterministic guarantee. But _if_ you are really
| > | using such priority design. I'm ok maximum NonRT priority instead maximum
| > | RT priority too.
| > 
| > I confess I failed to distinguish memcg OOM and system OOM and used "in
| > case of OOM kill the selected task the faster you can" as the guideline.
| > If the exit code path is short that shouldn't be a problem.
| > 
| > Maybe the right way to go would be giving the dying task the biggest
| > priority inside that memcg to be sure that it will be the next process from
| > that memcg to be scheduled. Would that be reasonable?
| 
| Hmm. I can't understand your point. 
| What do you mean failing distinguish memcg and system OOM?
| 
| We already have been distinguish it by mem_cgroup_out_of_memory.
| (but we have to enable CONFIG_CGROUP_MEM_RES_CTLR). 
| So task selected in select_bad_process is one out of memcg's tasks when 
| memcg have a memory pressure. 

The approach of giving the highest priority to the dying task makes sense in a system wide OOM situation. I though that would also be good for the memcg OOM case.

After Balbir Singh's comment, I understand that in a memcg OOM the dying task should have a priority just above the priority of the main task of that memcg, in order to avoid interfering in the rest of the system.

That is the point where I failed to distinguish between memcg and system OOM.

Should I pursue that new idea of looking for the right priority inside the memcg or is it overkill? I really don't have a clear view of the impact of a memcg OOM on system performance - don't know if it is better to solve the issue sooner (highest RT priority) or leave it to be solved later (highest prio on the memcg). I have the impression the general case points to the simpler solution.

Luis

Signature · 8 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#16 lore raw
Quoted 33 lines · Luis Claudio R. Goncalves

On Fri, May 28, 2010 at 11:06:23PM +0900, Minchan Kim wrote:
| On Fri, May 28, 2010 at 09:53:05AM -0300, Luis Claudio R. Goncalves wrote:
| > On Fri, May 28, 2010 at 02:59:02PM +0900, KOSAKI Motohiro wrote:
...

| > | As far as my observation, RT-function always have some syscall. because pure
| > | calculation doesn't need deterministic guarantee. But _if_ you are really
| > | using such priority design. I'm ok maximum NonRT priority instead maximum
| > | RT priority too.
| > 
| > I confess I failed to distinguish memcg OOM and system OOM and used "in
| > case of OOM kill the selected task the faster you can" as the guideline.
| > If the exit code path is short that shouldn't be a problem.
| > 
| > Maybe the right way to go would be giving the dying task the biggest
| > priority inside that memcg to be sure that it will be the next process from
| > that memcg to be scheduled. Would that be reasonable?
| 
| Hmm. I can't understand your point. 
| What do you mean failing distinguish memcg and system OOM?
| 
| We already have been distinguish it by mem_cgroup_out_of_memory.
| (but we have to enable CONFIG_CGROUP_MEM_RES_CTLR). 
| So task selected in select_bad_process is one out of memcg's tasks when 
| memcg have a memory pressure. 

The approach of giving the highest priority to the dying task makes sense in a system wide OOM situation. I though that would also be good for the memcg OOM case.

After Balbir Singh's comment, I understand that in a memcg OOM the dying task should have a priority just above the priority of the main task of that memcg, in order to avoid interfering in the rest of the system.

Quoted 8 lines

That is the point where I failed to distinguish between memcg and system OOM.

Should I pursue that new idea of looking for the right priority inside the memcg or is it overkill? I really don't have a clear view of the impact of a memcg OOM on system performance - don't know if it is better to solve the issue sooner (highest RT priority) or leave it to be solved later (highest prio on the memcg). I have the impression the general case points to the simpler solution.

I think highest RT proirity ins't good solution.
As I mentiond, Some RT functions don't want to be preempted by other processes which cause memory pressure. It makes RT task broken.

On the other hand, normal processes don't have a requirement of RT. But it isn't a big problem that it lost little time slice, I think. So how about raising max normal priority?
but I am not sure this is right solution.
Let's listen other's opinion.
I believe Peter have a good idea.

Quoted 6 lines

Luis

Signature · 2 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]
Signature · 2 lines
Kind regards,
Minchan Kim
#17 lore raw
Quoted 3 lines · Minchan Kim

I think highest RT proirity ins't good solution.
As I mentiond, Some RT functions don't want to be preempted by other processes which cause memory pressure. It makes RT task broken.

All the patches I've seen use MAX_RT_PRIO-1, which is actually FIFO-1, which is the lowest RT priority.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#18 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Minchan Kim

I think highest RT proirity ins't good solution.
As I mentiond, Some RT functions don't want to be preempted by other processes which cause memory pressure. It makes RT task broken.

All the patches I've seen use MAX_RT_PRIO-1, which is actually FIFO-1, which is the lowest RT priority.

Stupid me. I confused that until now.
That's exactly what I want.

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#19 lore raw
In reply to Minchan Kim

On Sat, May 29, 2010 at 12:12:49AM +0900, Minchan Kim wrote:

| On Fri, May 28, 2010 at 11:36:17AM -0300, Luis Claudio R. Goncalves wrote:
| > On Fri, May 28, 2010 at 11:06:23PM +0900, Minchan Kim wrote:
| > | On Fri, May 28, 2010 at 09:53:05AM -0300, Luis Claudio R. Goncalves wrote:
| > | > On Fri, May 28, 2010 at 02:59:02PM +0900, KOSAKI Motohiro wrote:
| > ...
| > | > | As far as my observation, RT-function always have some syscall. because pure
| > | > | calculation doesn't need deterministic guarantee. But _if_ you are really
| > | > | using such priority design. I'm ok maximum NonRT priority instead maximum
| > | > | RT priority too.
| > | > 
| > | > I confess I failed to distinguish memcg OOM and system OOM and used "in
| > | > case of OOM kill the selected task the faster you can" as the guideline.
| > | > If the exit code path is short that shouldn't be a problem.
| > | > 
| > | > Maybe the right way to go would be giving the dying task the biggest
| > | > priority inside that memcg to be sure that it will be the next process from
| > | > that memcg to be scheduled. Would that be reasonable?
| > | 
| > | Hmm. I can't understand your point. 
| > | What do you mean failing distinguish memcg and system OOM?
| > | 
| > | We already have been distinguish it by mem_cgroup_out_of_memory.
| > | (but we have to enable CONFIG_CGROUP_MEM_RES_CTLR). 
| > | So task selected in select_bad_process is one out of memcg's tasks when 
| > | memcg have a memory pressure. 
| > 
| > The approach of giving the highest priority to the dying task makes sense
| > in a system wide OOM situation. I though that would also be good for the
| > memcg OOM case.
| > 
| > After Balbir Singh's comment, I understand that in a memcg OOM the dying
| > task should have a priority just above the priority of the main task of
| > that memcg, in order to avoid interfering in the rest of the system.
| > 
| > That is the point where I failed to distinguish between memcg and system OOM.
| > 
| > Should I pursue that new idea of looking for the right priority inside the
| > memcg or is it overkill? I really don't have a clear view of the impact of
| > a memcg OOM on system performance - don't know if it is better to solve the
| > issue sooner (highest RT priority) or leave it to be solved later (highest
| > prio on the memcg). I have the impression the general case points to the
| > simpler solution.
| 
| I think highest RT proirity ins't good solution.
| As I mentiond, Some RT functions don't want to be preempted by other processes
| which cause memory pressure. It makes RT task broken.

For the RT case, if you reached a system OOM situation, your determinism has already been hurt. If the memcg OOM happens on the same memcg your RT task is - what will probably be the case most of time - again, the determinism has deteriorated. For both these cases, giving the dying task SCHED_FIFO MAX_RT_PRIO-1 means a faster recovery.

I don't know what is the system-wide latency effect of a memcg OOM, if any, or if it would affect an RT task running on another memcg. That is the case where a more careful priority selection could be necessary.

| On the other hand, normal processes don't have a requirement of RT.
| But it isn't a big problem that it lost little time slice, I think. 
| So how about raising max normal priority? 
| but I am not sure this is right solution.
| Let's listen other's opinion. 
| I believe Peter have a good idea.

Thanks again for helping to discuss this idea.

Luis

Signature · 8 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#20 lore raw
Quoted 48 lines · Luis Claudio R. Goncalves

On Sat, May 29, 2010 at 12:12:49AM +0900, Minchan Kim wrote:

| On Fri, May 28, 2010 at 11:36:17AM -0300, Luis Claudio R. Goncalves wrote:
| > On Fri, May 28, 2010 at 11:06:23PM +0900, Minchan Kim wrote:
| > | On Fri, May 28, 2010 at 09:53:05AM -0300, Luis Claudio R. Goncalves wrote:
| > | > On Fri, May 28, 2010 at 02:59:02PM +0900, KOSAKI Motohiro wrote:
| > ...
| > | > | As far as my observation, RT-function always have some syscall. because pure
| > | > | calculation doesn't need deterministic guarantee. But _if_ you are really
| > | > | using such priority design. I'm ok maximum NonRT priority instead maximum
| > | > | RT priority too.
| > | > 
| > | > I confess I failed to distinguish memcg OOM and system OOM and used "in
| > | > case of OOM kill the selected task the faster you can" as the guideline.
| > | > If the exit code path is short that shouldn't be a problem.
| > | > 
| > | > Maybe the right way to go would be giving the dying task the biggest
| > | > priority inside that memcg to be sure that it will be the next process from
| > | > that memcg to be scheduled. Would that be reasonable?
| > | 
| > | Hmm. I can't understand your point. 
| > | What do you mean failing distinguish memcg and system OOM?
| > | 
| > | We already have been distinguish it by mem_cgroup_out_of_memory.
| > | (but we have to enable CONFIG_CGROUP_MEM_RES_CTLR). 
| > | So task selected in select_bad_process is one out of memcg's tasks when 
| > | memcg have a memory pressure. 
| > 
| > The approach of giving the highest priority to the dying task makes sense
| > in a system wide OOM situation. I though that would also be good for the
| > memcg OOM case.
| > 
| > After Balbir Singh's comment, I understand that in a memcg OOM the dying
| > task should have a priority just above the priority of the main task of
| > that memcg, in order to avoid interfering in the rest of the system.
| > 
| > That is the point where I failed to distinguish between memcg and system OOM.
| > 
| > Should I pursue that new idea of looking for the right priority inside the
| > memcg or is it overkill? I really don't have a clear view of the impact of
| > a memcg OOM on system performance - don't know if it is better to solve the
| > issue sooner (highest RT priority) or leave it to be solved later (highest
| > prio on the memcg). I have the impression the general case points to the
| > simpler solution.
| 
| I think highest RT proirity ins't good solution.
| As I mentiond, Some RT functions don't want to be preempted by other processes
| which cause memory pressure. It makes RT task broken.
Quoted 5 lines

For the RT case, if you reached a system OOM situation, your determinism has already been hurt. If the memcg OOM happens on the same memcg your RT task is - what will probably be the case most of time - again, the determinism has deteriorated. For both these cases, giving the dying task SCHED_FIFO MAX_RT_PRIO-1 means a faster recovery.

What I want to say is that determinisic has no relation with OOM. Why is some RT task affected by other process's OOM?

Of course, if system has no memory, it is likely to slow down RT task. But it's just only thought. If some task scheduled just is exit, we don't need to raise OOMed task's priority.

But raising min rt priority on your patch was what I want. It doesn't preempt any RT task.

So until now, I have made noise about your patch. Really, sorry for that.
I don't have any objection on raising priority part from now on.

Thanks, Luis.

Signature · 2 lines
Kind regards,
Minchan Kim
#21 lore raw

On Sat, May 29, 2010 at 12:45:49AM +0900, Minchan Kim wrote:
| On Fri, May 28, 2010 at 12:28:42PM -0300, Luis Claudio R. Goncalves wrote:
| > On Sat, May 29, 2010 at 12:12:49AM +0900, Minchan Kim wrote:
...

| > | I think highest RT proirity ins't good solution.
| > | As I mentiond, Some RT functions don't want to be preempted by other processes
| > | which cause memory pressure. It makes RT task broken.
| > 
| > For the RT case, if you reached a system OOM situation, your determinism has
| > already been hurt. If the memcg OOM happens on the same memcg your RT task
| > is - what will probably be the case most of time - again, the determinism
| > has deteriorated. For both these cases, giving the dying task SCHED_FIFO
| > MAX_RT_PRIO-1 means a faster recovery.
| 
| What I want to say is that determinisic has no relation with OOM. 
| Why is some RT task affected by other process's OOM?
| 
| Of course, if system has no memory, it is likely to slow down RT task. 
| But it's just only thought. If some task scheduled just is exit, we don't need
| to raise OOMed task's priority.
| 
| But raising min rt priority on your patch was what I want.
| It doesn't preempt any RT task.
| 
| So until now, I have made noise about your patch.
| Really, sorry for that. 
| I don't have any objection on raising priority part from now on. 

This is the third version of the patch, factoring in your input along with Peter's comment. Basically the same patch, but using the lowest RT priority to boost the dying task.

Thanks again for reviewing and commenting.
Luis

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

mm/oom_kill.c +4 -0
index 84bbba2..2b0204f 100644
@@ -266,6 +266,8 @@ static struct task_struct *select_bad_process(unsigned long *ppoints)
  */
 static void __oom_kill_task(struct task_struct *p, int verbose)
 {
+	struct sched_param param;
+
 	if (is_global_init(p)) {
 		WARN_ON(1);
 		printk(KERN_WARNING "tried to kill init!\n");
@@ -288,6 +290,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 * exit() and clear out its resources quickly...
 	 */
 	p->time_slice = HZ;
+	param.sched_priority = MAX_RT_PRIO-10;
+	sched_setscheduler(p, SCHED_FIFO, &param);
 	set_tsk_thread_flag(p, TIF_MEMDIE);
 
 	force_sig(SIGKILL, p);
Signature · 8 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#22 lore raw

Hi

Quoted 6 lines

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

Almostly acceptable to me. but I have two requests,

- need 1) force_sig() 2)sched_setscheduler() order as Oleg mentioned
- don't boost priority if it's in mem_cgroup_out_of_memory()

Can you accept this? if not, can you please explain the reason?

Thanks.

Quoted 27 lines
mm/oom_kill.c +4 -0
index 84bbba2..2b0204f 100644
@@ -266,6 +266,8 @@ static struct task_struct *select_bad_process(unsigned long *ppoints)
  */
 static void __oom_kill_task(struct task_struct *p, int verbose)
 {
+	struct sched_param param;
+
 	if (is_global_init(p)) {
 		WARN_ON(1);
 		printk(KERN_WARNING "tried to kill init!\n");
@@ -288,6 +290,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 * exit() and clear out its resources quickly...
 	 */
 	p->time_slice = HZ;
+	param.sched_priority = MAX_RT_PRIO-10;
+	sched_setscheduler(p, SCHED_FIFO, &param);
 	set_tsk_thread_flag(p, TIF_MEMDIE);
 
 	force_sig(SIGKILL, p);
Signature · 2 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author Reviewed-by KOSAKI Motohiro
#23 lore raw

On Sat, May 29, 2010 at 12:59:09PM +0900, KOSAKI Motohiro wrote:

| Hi
| 
| > oom-killer: give the dying task rt priority (v3)
| > 
| > Give the dying task RT priority so that it can be scheduled quickly and die,
| > freeing needed memory.
| > 
| > Signed-off-by: Luis Claudio R. Goncalves <[email protected]>
| 
| Almostly acceptable to me. but I have two requests, 
| 
| - need 1) force_sig() 2)sched_setscheduler() order as Oleg mentioned
| - don't boost priority if it's in mem_cgroup_out_of_memory()
| 
| Can you accept this? if not, can you please explain the reason?
| 
| Thanks.

The last patch I posted was the wrong patch from my queue. Sorry for the confusion. Here is the last version of the patch, including the suggestions from Oleg, Peter and Kosaki Motohiro:

oom-kill: give the dying task a higher priority (v4)

In a system under heavy load it was observed that even after the oom-killer selects a task to die, the task may take a long time to die.

Right before sending a SIGKILL to the task selected by the oom-killer this task has it's priority increased so that it can exit() exit soon, freeing memory. That is accomplished by:

/*
* We give our sacrificial lamb high priority and access to
* all the memory it needs. That way it should be able to
* exit() and clear out its resources quickly... */
p->rt.time_slice = HZ;
set_tsk_thread_flag(p, TIF_MEMDIE);

It sounds plausible giving the dying task an even higher priority to be sure it will be scheduled sooner and free the desired memory. It was suggested on LKML using SCHED_FIFO:1, the lowest RT priority so that this task won't interfere with any running RT task.

Another good suggestion, implemented here, was to avoid boosting the dying task priority in case of mem_cgroup OOM.

mm/oom_kill.c +17 -7
index 709aedf..6a25293 100644
@@ -380,7 +380,8 @@ static void dump_header(struct task_struct *p, gfp_t gfp_mask, int order,
  * flag though it's unlikely that  we select a process with CAP_SYS_RAW_IO
  * set.
  */
-static void __oom_kill_task(struct task_struct *p, int verbose)
+static void __oom_kill_task(struct task_struct *p, struct mem_cgroup *mem,
+								int verbose)
 {
 	if (is_global_init(p)) {
 		WARN_ON(1);
@@ -413,11 +414,20 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 */
 	p->rt.time_slice = HZ;
 	set_tsk_thread_flag(p, TIF_MEMDIE);
-
 	force_sig(SIGKILL, p);
+	/*
+	 * If this is a system OOM (not a memcg OOM), speed up the recovery
+	 * by boosting the dying task priority to the lowest FIFO priority.
+	 * That helps with the recovery and avoids interfering with RT tasks.
+	 */
+	if (mem == NULL) {
+		struct sched_param param;
+		param.sched_priority = 1;
+		sched_setscheduler_nocheck(p, SCHED_FIFO, &param);
+	}
 }
 
-static int oom_kill_task(struct task_struct *p)
+static int oom_kill_task(struct task_struct *p, struct mem_cgroup *mem)
 {
 	/* WARNING: mm may not be dereferenced since we did not obtain its
 	 * value from get_task_mm(p).  This is OK since all we need to do is
@@ -430,7 +440,7 @@ static int oom_kill_task(struct task_struct *p)
 	if (!p->mm || p->signal->oom_adj == OOM_DISABLE)
 		return 1;
 
-	__oom_kill_task(p, 1);
+	__oom_kill_task(p, mem, 1);
 
 	return 0;
 }
@@ -449,7 +459,7 @@ static int oom_kill_process(struct task_struct *p, gfp_t gfp_mask, int order,
 	 * its children or threads, just set TIF_MEMDIE so it can die quickly
 	 */
 	if (p->flags & PF_EXITING) {
-		__oom_kill_task(p, 0);
+		__oom_kill_task(p, mem, 0);
 		return 0;
 	}
 
@@ -462,10 +472,10 @@ static int oom_kill_process(struct task_struct *p, gfp_t gfp_mask, int order,
 			continue;
 		if (mem && !task_in_mem_cgroup(c, mem))
 			continue;
-		if (!oom_kill_task(c))
+		if (!oom_kill_task(c, mem))
 			return 0;
 	}
-	return oom_kill_task(p);
+	return oom_kill_task(p, mem);
 }
 
 #ifdef CONFIG_CGROUP_MEM_RES_CTLR
Signature · 8 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#24 lore raw
In reply to KOSAKI Motohiro

Hi, Kosaki.

Quoted 7 lines · KOSAKI Motohiro

Hi

Quoted 5 lines

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

Quoted 6 lines
Quoted 1 line

Almostly acceptable to me. but I have two requests,

- need 1) force_sig() 2)sched_setscheduler() order as Oleg mentioned
- don't boost priority if it's in mem_cgroup_out_of_memory()

Why do you want to not boost priority if it's path of memcontrol?

If it's path of memcontrol and CONFIG_CGROUP_MEM_RES_CTLR is enabled, mem_cgroup_out_of_memory will select victim task in memcg. So __oom_kill_task's target task would be in memcg, I think.

As you and memcg guys don't complain this, I would be missing something. Could you explain it? :)

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#25 lore raw

Hi

Quoted 15 lines

Hi, Kosaki.

Quoted 11 lines · KOSAKI Motohiro

Hi

Quoted 6 lines

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

Almostly acceptable to me. but I have two requests,

Quoted 8 lines
Quoted 2 lines

- need 1) force_sig() 2)sched_setscheduler() order as Oleg mentioned
- don't boost priority if it's in mem_cgroup_out_of_memory()

Why do you want to not boost priority if it's path of memcontrol?

If it's path of memcontrol and CONFIG_CGROUP_MEM_RES_CTLR is enabled, mem_cgroup_out_of_memory will select victim task in memcg. So __oom_kill_task's target task would be in memcg, I think.

Yep.
But priority boost naturally makes CPU starvation for out of the group processes.
It seems to break cgroup's isolation concept.

Quoted 2 lines

As you and memcg guys don't complain this, I would be missing something. Could you explain it? :)

So, My points are,

1) Usually priority boost is wrong idea. It have various side effect, but system wide OOM is one of exception. In such case, all tasks aren't runnable, then, the downside is acceptable.
2) memcg have OOM notification mechanism. If the admin need priority boost, they can do it by their OOM-daemon.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#26 lore raw
Quoted 34 lines · KOSAKI Motohiro

Hi

Quoted 15 lines

Hi, Kosaki.

Quoted 11 lines · KOSAKI Motohiro

Hi

Quoted 6 lines

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

Almostly acceptable to me. but I have two requests,

Quoted 8 lines
Quoted 2 lines

- need 1) force_sig() 2)sched_setscheduler() order as Oleg mentioned
- don't boost priority if it's in mem_cgroup_out_of_memory()

Why do you want to not boost priority if it's path of memcontrol?

If it's path of memcontrol and CONFIG_CGROUP_MEM_RES_CTLR is enabled, mem_cgroup_out_of_memory will select victim task in memcg. So __oom_kill_task's target task would be in memcg, I think.

Yep.
But priority boost naturally makes CPU starvation for out of the group processes.
It seems to break cgroup's isolation concept.

Quoted 2 lines

As you and memcg guys don't complain this, I would be missing something. Could you explain it? :)

Quoted 7 lines

So, My points are,

1) Usually priority boost is wrong idea. It have various side effect, but system wide OOM is one of exception. In such case, all tasks aren't runnable, then, the downside is acceptable.
2) memcg have OOM notification mechanism. If the admin need priority boost, they can do it by their OOM-daemon.

Is it possible kill the hogging task immediately when the daemon send kill signal?
I mean we can make OOM daemon higher priority than others and it can send signal to normal process. but when is normal process exited after receiving kill signal from OOM daemon? Maybe it's when killed task is executed by scheduler. It's same problem again, I think.

Kame, Do you have an idea?

Quoted 4 lines

Thanks.

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#27 lore raw
Quoted 44 lines · On Mon, 31 May 2010 16:05:48 +0900 Minchan Kim <[email protected]> wrote:
Quoted 34 lines · KOSAKI Motohiro

Hi

Quoted 15 lines

Hi, Kosaki.

Quoted 11 lines · KOSAKI Motohiro

Hi

Quoted 6 lines

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

Almostly acceptable to me. but I have two requests,

Quoted 8 lines
Quoted 2 lines

- need 1) force_sig() 2)sched_setscheduler() order as Oleg mentioned
- don't boost priority if it's in mem_cgroup_out_of_memory()

Why do you want to not boost priority if it's path of memcontrol?

If it's path of memcontrol and CONFIG_CGROUP_MEM_RES_CTLR is enabled, mem_cgroup_out_of_memory will select victim task in memcg. So __oom_kill_task's target task would be in memcg, I think.

Yep.
But priority boost naturally makes CPU starvation for out of the group processes.
It seems to break cgroup's isolation concept.

Quoted 2 lines

As you and memcg guys don't complain this, I would be missing something. Could you explain it? :)

Quoted 7 lines

So, My points are,

1) Usually priority boost is wrong idea. It have various side effect, but A system wide OOM is one of exception. In such case, all tasks aren't A runnable, then, the downside is acceptable.
2) memcg have OOM notification mechanism. If the admin need priority boost, A they can do it by their OOM-daemon.

Quoted 9 lines

Is it possible kill the hogging task immediately when the daemon send kill signal?
I mean we can make OOM daemon higher priority than others and it can send signal to normal process. but when is normal process exited after receiving kill signal from OOM daemon? Maybe it's when killed task is executed by scheduler. It's same problem again, I think.

Kame, Do you have an idea?

This is just an idea and I have no implementaion, yet.

With memcg, oom situation can be recovered by "enlarging limit temporary". Then, what the daemon has to do is

1. send signal (kill or other signal to abort for coredump.)
2. move a problematic task to a jail if necessary.
3. enlarge limit for indicating "Go"
4. After stabilization, reduce the limit.

This is the fastest. Admin has to think of extra-room or jails and the daemon should be enough clever. But in most case, I think this works well.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#28 lore raw
Quoted 61 lines · KAMEZAWA Hiroyuki
Quoted 44 lines · On Mon, 31 May 2010 16:05:48 +0900 Minchan Kim <[email protected]> wrote:
Quoted 34 lines · KOSAKI Motohiro

Hi

Quoted 15 lines

Hi, Kosaki.

Quoted 11 lines · KOSAKI Motohiro

Hi

Quoted 6 lines

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

Almostly acceptable to me. but I have two requests,

Quoted 8 lines
Quoted 2 lines

- need 1) force_sig() 2)sched_setscheduler() order as Oleg mentioned
- don't boost priority if it's in mem_cgroup_out_of_memory()

Why do you want to not boost priority if it's path of memcontrol?

If it's path of memcontrol and CONFIG_CGROUP_MEM_RES_CTLR is enabled, mem_cgroup_out_of_memory will select victim task in memcg. So __oom_kill_task's target task would be in memcg, I think.

Yep.
But priority boost naturally makes CPU starvation for out of the group processes.
It seems to break cgroup's isolation concept.

Quoted 2 lines

As you and memcg guys don't complain this, I would be missing something. Could you explain it? :)

Quoted 7 lines

So, My points are,

1) Usually priority boost is wrong idea. It have various side effect, but system wide OOM is one of exception. In such case, all tasks aren't runnable, then, the downside is acceptable.
2) memcg have OOM notification mechanism. If the admin need priority boost, they can do it by their OOM-daemon.

Quoted 9 lines

Is it possible kill the hogging task immediately when the daemon send kill signal?
I mean we can make OOM daemon higher priority than others and it can send signal to normal process. but when is normal process exited after receiving kill signal from OOM daemon? Maybe it's when killed task is executed by scheduler. It's same problem again, I think.

Kame, Do you have an idea?

This is just an idea and I have no implementaion, yet.

With memcg, oom situation can be recovered by "enlarging limit temporary". Then, what the daemon has to do is

Quoted 7 lines

 1. send signal (kill or other signal to abort for coredump.)
 2. move a problematic task to a jail if necessary.
 3. enlarge limit for indicating "Go"
 4. After stabilization, reduce the limit.

This is the fastest. Admin has to think of extra-room or jails and the daemon should be enough clever. But in most case, I think this works well.

I think it is very hard that how much we have to make extra-room since we can't expect how many tasks are stuck to allocate memory. But tend to agree that system-wide OOM problem is more important than memcg's one.
And memcg's guy doesn't seem to have any problem. So I am not against this patch any more.

Thanks, Kosaki and Kame.

Quoted 4 lines

Thanks,
-Kame

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#29 lore raw
In reply to Luis Claudio R. Goncalves
Quoted 56 lines · Luis Claudio R. Goncalves

On Sat, May 29, 2010 at 12:45:49AM +0900, Minchan Kim wrote:
| On Fri, May 28, 2010 at 12:28:42PM -0300, Luis Claudio R. Goncalves wrote:
| > On Sat, May 29, 2010 at 12:12:49AM +0900, Minchan Kim wrote:
...

| > | I think highest RT proirity ins't good solution.
| > | As I mentiond, Some RT functions don't want to be preempted by other processes
| > | which cause memory pressure. It makes RT task broken.
| > 
| > For the RT case, if you reached a system OOM situation, your determinism has
| > already been hurt. If the memcg OOM happens on the same memcg your RT task
| > is - what will probably be the case most of time - again, the determinism
| > has deteriorated. For both these cases, giving the dying task SCHED_FIFO
| > MAX_RT_PRIO-1 means a faster recovery.
| 
| What I want to say is that determinisic has no relation with OOM. 
| Why is some RT task affected by other process's OOM?
| 
| Of course, if system has no memory, it is likely to slow down RT task. 
| But it's just only thought. If some task scheduled just is exit, we don't need
| to raise OOMed task's priority.
| 
| But raising min rt priority on your patch was what I want.
| It doesn't preempt any RT task.
| 
| So until now, I have made noise about your patch.
| Really, sorry for that. 
| I don't have any objection on raising priority part from now on. 

This is the third version of the patch, factoring in your input along with Peter's comment. Basically the same patch, but using the lowest RT priority to boost the dying task.

Thanks again for reviewing and commenting.
Luis

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

mm/oom_kill.c +2 -0
index 84bbba2..2b0204f 100644
@@ -266,6 +266,8 @@ static struct task_struct *select_bad_process(unsigned long *ppoints)
  */
 static void __oom_kill_task(struct task_struct *p, int verbose)
 {
+	struct sched_param param;
+
 	if (is_global_init(p)) {
 		WARN_ON(1);
 		printk(KERN_WARNING "tried to kill init!\n");

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 293

Quoted 4 lines
mm/oom_kill.c +1 -0
@@ -288,6 +290,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 * exit() and clear out its resources quickly...
 	 */
 	p->time_slice = HZ;
+	param.sched_priority = MAX_RT_PRIO-10;

I can't understand your point, still.
Why you put the priority as "MAX_RT_PRIO - 10"?
What I and peter mentioned was "1" which is lowest RT priority.

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 294

Quoted 1 line
mm/oom_kill.c +1 -0
+	sched_setscheduler(p, SCHED_FIFO, &param);

Why do you change sched_setscheduler_nocheck with sched_set_scheduler? It means you can't boost prioity if current context doesn't have permission. Is it a your intention?

Quoted 7 lines
mm/oom_kill.c +0 -0
 	set_tsk_thread_flag(p, TIF_MEMDIE);
 
 	force_sig(SIGKILL, p);
Signature · 2 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]
Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#30 lore raw
In reply to Luis Claudio R. Goncalves
Quoted 56 lines · On Fri, 28 May 2010 13:48:26 -0300 "Luis Claudio R. Goncalves"…

On Sat, May 29, 2010 at 12:45:49AM +0900, Minchan Kim wrote:
| On Fri, May 28, 2010 at 12:28:42PM -0300, Luis Claudio R. Goncalves wrote:
| > On Sat, May 29, 2010 at 12:12:49AM +0900, Minchan Kim wrote:
...

| > | I think highest RT proirity ins't good solution.
| > | As I mentiond, Some RT functions don't want to be preempted by other processes
| > | which cause memory pressure. It makes RT task broken.
| > 
| > For the RT case, if you reached a system OOM situation, your determinism has
| > already been hurt. If the memcg OOM happens on the same memcg your RT task
| > is - what will probably be the case most of time - again, the determinism
| > has deteriorated. For both these cases, giving the dying task SCHED_FIFO
| > MAX_RT_PRIO-1 means a faster recovery.
| 
| What I want to say is that determinisic has no relation with OOM. 
| Why is some RT task affected by other process's OOM?
| 
| Of course, if system has no memory, it is likely to slow down RT task. 
| But it's just only thought. If some task scheduled just is exit, we don't need
| to raise OOMed task's priority.
| 
| But raising min rt priority on your patch was what I want.
| It doesn't preempt any RT task.
| 
| So until now, I have made noise about your patch.
| Really, sorry for that. 
| I don't have any objection on raising priority part from now on. 

This is the third version of the patch, factoring in your input along with Peter's comment. Basically the same patch, but using the lowest RT priority to boost the dying task.

Thanks again for reviewing and commenting.
Luis

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

mm/oom_kill.c +2 -0
index 84bbba2..2b0204f 100644
@@ -266,6 +266,8 @@ static struct task_struct *select_bad_process(unsigned long *ppoints)
  */
 static void __oom_kill_task(struct task_struct *p, int verbose)
 {
+	struct sched_param param;
+
 	if (is_global_init(p)) {
 		WARN_ON(1);
 		printk(KERN_WARNING "tried to kill init!\n");

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 296

Quoted 7 lines
mm/oom_kill.c +2 -0
@@ -288,6 +290,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 * exit() and clear out its resources quickly...
 	 */
 	p->time_slice = HZ;
+	param.sched_priority = MAX_RT_PRIO-10;
+	sched_setscheduler(p, SCHED_FIFO, &param);
 	set_tsk_thread_flag(p, TIF_MEMDIE);
 

BTW, how about the other threads which share mm_struct ?

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#31 lore raw

Hi, Kame.

Quoted 25 lines · KAMEZAWA Hiroyuki
Quoted 23 lines · On Fri, 28 May 2010 13:48:26 -0300 "Luis Claudio R. Goncalves"…

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

mm/oom_kill.c +2 -0
index 84bbba2..2b0204f 100644
@@ -266,6 +266,8 @@ static struct task_struct *select_bad_process(unsigned long *ppoints)
   */
  static void __oom_kill_task(struct task_struct *p, int verbose)
  {
+     struct sched_param param;
+
       if (is_global_init(p)) {
               WARN_ON(1);
               printk(KERN_WARNING "tried to kill init!\n");
@@ -288,6 +290,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
        * exit() and clear out its resources quickly...
Quoted 8 lines

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 296

Quoted 6 lines
mm/oom_kill.c +2 -0
        */
       p->time_slice = HZ;
+     param.sched_priority = MAX_RT_PRIO-10;
+     sched_setscheduler(p, SCHED_FIFO, &param);
       set_tsk_thread_flag(p, TIF_MEMDIE);
 

BTW, how about the other threads which share mm_struct ?

Could you elaborate your intention? :)

Quoted 5 lines

Thanks,
-Kame

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#32 lore raw
Quoted 38 lines · On Mon, 31 May 2010 14:01:03 +0900 Minchan Kim <[email protected]> wrote:

Hi, Kame.

Quoted 25 lines · KAMEZAWA Hiroyuki
Quoted 23 lines · On Fri, 28 May 2010 13:48:26 -0300 "Luis Claudio R. Goncalves"…

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

mm/oom_kill.c +0 -0
index 84bbba2..2b0204f 100644
@@ -266,6 +266,8 @@ static struct task_struct *select_bad_process(unsigned long *ppoints)

A */
A static void __oom_kill_task(struct task_struct *p, int verbose) A {
+ A A struct sched_param param;
+

A  A  A  if (is_global_init(p)) {
A  A  A  A  A  A  A  WARN_ON(1);
A  A  A  A  A  A  A  printk(KERN_WARNING "tried to kill init!\n");
mm/oom_kill.c +0 -0
@@ -288,6 +290,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)

A A A A * exit() and clear out its resources quickly...

Quoted 8 lines
Quoted 6 lines
A  A  A  A */
A  A  A  p->time_slice = HZ;
+ A  A  param.sched_priority = MAX_RT_PRIO-10;
+ A  A  sched_setscheduler(p, SCHED_FIFO, &param);

A A A set_tsk_thread_flag(p, TIF_MEMDIE);

BTW, how about the other threads which share mm_struct ?

Quoted 2 lines

Could you elaborate your intention? :)

IIUC, the purpose of rising priority is to accerate dying thread to exit() for freeing memory AFAP. But to free memory, exit, all threads which share mm_struct should exit, too. I'm sorry if I miss something.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#33 lore raw
Quoted 41 lines · KAMEZAWA Hiroyuki
Quoted 38 lines · On Mon, 31 May 2010 14:01:03 +0900 Minchan Kim <[email protected]> wrote:

Hi, Kame.

Quoted 33 lines · KAMEZAWA Hiroyuki
Quoted 22 lines · On Fri, 28 May 2010 13:48:26 -0300 "Luis Claudio R. Goncalves"…

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

mm/oom_kill.c +2 -0
index 84bbba2..2b0204f 100644
@@ -266,6 +266,8 @@ static struct task_struct *select_bad_process(unsigned long *ppoints)
   */
  static void __oom_kill_task(struct task_struct *p, int verbose)
  {
+     struct sched_param param;
+
       if (is_global_init(p)) {
               WARN_ON(1);
               printk(KERN_WARNING "tried to kill init!\n");

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 296

Quoted 7 lines
mm/oom_kill.c +2 -0
@@ -288,6 +290,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
        * exit() and clear out its resources quickly...
        */
       p->time_slice = HZ;
+     param.sched_priority = MAX_RT_PRIO-10;
+     sched_setscheduler(p, SCHED_FIFO, &param);
       set_tsk_thread_flag(p, TIF_MEMDIE);
 

BTW, how about the other threads which share mm_struct ?

Quoted 6 lines
Quoted 2 lines

Could you elaborate your intention? :)

IIUC, the purpose of rising priority is to accerate dying thread to exit() for freeing memory AFAP. But to free memory, exit, all threads which share mm_struct should exit, too. I'm sorry if I miss something.

How do we kill only some thread and what's the benefit of it? I think when if some thread receives KILL signal, the process include the thread will be killed.

I think

Quoted 6 lines

Thanks,
-Kame

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#34 lore raw
Quoted 46 lines · On Mon, 31 May 2010 14:46:05 +0900 Minchan Kim <[email protected]> wrote:
Quoted 44 lines · KAMEZAWA Hiroyuki
Quoted 40 lines · On Mon, 31 May 2010 14:01:03 +0900 Minchan Kim <[email protected]> wrote:

Hi, Kame.

Quoted 25 lines · KAMEZAWA Hiroyuki
Quoted 23 lines · On Fri, 28 May 2010 13:48:26 -0300 "Luis Claudio R. Goncalves"…

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

mm/oom_kill.c +0 -0
index 84bbba2..2b0204f 100644
@@ -266,6 +266,8 @@ static struct task_struct *select_bad_process(unsigned long *ppoints)

A */
A static void __oom_kill_task(struct task_struct *p, int verbose) A {
+ A A struct sched_param param;
+

A  A  A  if (is_global_init(p)) {
A  A  A  A  A  A  A  WARN_ON(1);
A  A  A  A  A  A  A  printk(KERN_WARNING "tried to kill init!\n");
mm/oom_kill.c +0 -0
@@ -288,6 +290,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)

A A A A * exit() and clear out its resources quickly...

Quoted 8 lines
Quoted 6 lines
A  A  A  A */
A  A  A  p->time_slice = HZ;
+ A  A  param.sched_priority = MAX_RT_PRIO-10;
+ A  A  sched_setscheduler(p, SCHED_FIFO, &param);

A A A set_tsk_thread_flag(p, TIF_MEMDIE);

BTW, how about the other threads which share mm_struct ?

Could you elaborate your intention? :)

Quoted 8 lines
Quoted 3 lines

IIUC, the purpose of rising priority is to accerate dying thread to exit() for freeing memory AFAP. But to free memory, exit, all threads which share mm_struct should exit, too. I'm sorry if I miss something.

How do we kill only some thread and what's the benefit of it? I think when if some thread receives KILL signal, the process include the thread will be killed.

yes, so, if you want a _process_ die quickly, you have to acceralte the whole threads on a process. Acceralating a thread in a process is not big help.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#35 lore raw
Quoted 53 lines · KAMEZAWA Hiroyuki
Quoted 50 lines · On Mon, 31 May 2010 14:46:05 +0900 Minchan Kim <[email protected]> wrote:
Quoted 47 lines · KAMEZAWA Hiroyuki
Quoted 38 lines · On Mon, 31 May 2010 14:01:03 +0900 Minchan Kim <[email protected]> wrote:

Hi, Kame.

Quoted 25 lines · KAMEZAWA Hiroyuki
Quoted 23 lines · On Fri, 28 May 2010 13:48:26 -0300 "Luis Claudio R. Goncalves"…

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

mm/oom_kill.c +2 -0
index 84bbba2..2b0204f 100644
@@ -266,6 +266,8 @@ static struct task_struct *select_bad_process(unsigned long *ppoints)
   */
  static void __oom_kill_task(struct task_struct *p, int verbose)
  {
+     struct sched_param param;
+
       if (is_global_init(p)) {
               WARN_ON(1);
               printk(KERN_WARNING "tried to kill init!\n");
@@ -288,6 +290,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
        * exit() and clear out its resources quickly...
Quoted 8 lines

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 296

Quoted 6 lines
mm/oom_kill.c +2 -0
        */
       p->time_slice = HZ;
+     param.sched_priority = MAX_RT_PRIO-10;
+     sched_setscheduler(p, SCHED_FIFO, &param);
       set_tsk_thread_flag(p, TIF_MEMDIE);
 

BTW, how about the other threads which share mm_struct ?

Quoted 2 lines

Could you elaborate your intention? :)

IIUC, the purpose of rising priority is to accerate dying thread to exit() for freeing memory AFAP. But to free memory, exit, all threads which share mm_struct should exit, too. I'm sorry if I miss something.

Quoted 6 lines
Quoted 4 lines

How do we kill only some thread and what's the benefit of it? I think when if some thread receives  KILL signal, the process include the thread will be killed.

yes, so, if you want a _process_ die quickly, you have to acceralte the whole threads on a process. Acceralating a thread in a process is not big help.

Yes.

I see the code.
oom_kill_process is called by

1. mem_cgroup_out_of_memory
2. __out_of_memory
3. out_of_memory

(1,2) calls select_bad_process which select victim task in processes by do_each_process.
But 3 isn't In case of CONSTRAINT_MEMORY_POLICY, it kills current. In only the case, couldn't we pass task of process, not one of thread?

Quoted 5 lines

Thanks,
-Kame

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#36 lore raw
Quoted 72 lines · On Mon, 31 May 2010 15:09:41 +0900 Minchan Kim <[email protected]> wrote:
Quoted 53 lines · KAMEZAWA Hiroyuki
Quoted 50 lines · On Mon, 31 May 2010 14:46:05 +0900 Minchan Kim <[email protected]> wrote:
Quoted 47 lines · KAMEZAWA Hiroyuki
Quoted 38 lines · On Mon, 31 May 2010 14:01:03 +0900 Minchan Kim <[email protected]> wrote:

Hi, Kame.

Quoted 25 lines · KAMEZAWA Hiroyuki
Quoted 23 lines · On Fri, 28 May 2010 13:48:26 -0300 "Luis Claudio R. Goncalves"…

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

mm/oom_kill.c +0 -0
index 84bbba2..2b0204f 100644
@@ -266,6 +266,8 @@ static struct task_struct *select_bad_process(unsigned long *ppoints)

A */
A static void __oom_kill_task(struct task_struct *p, int verbose) A {
+ A A struct sched_param param;
+

A  A  A  if (is_global_init(p)) {
A  A  A  A  A  A  A  WARN_ON(1);
A  A  A  A  A  A  A  printk(KERN_WARNING "tried to kill init!\n");
mm/oom_kill.c +0 -0
@@ -288,6 +290,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)

A A A A * exit() and clear out its resources quickly...

Quoted 8 lines
Quoted 6 lines
A  A  A  A */
A  A  A  p->time_slice = HZ;
+ A  A  param.sched_priority = MAX_RT_PRIO-10;
+ A  A  sched_setscheduler(p, SCHED_FIFO, &param);

A A A set_tsk_thread_flag(p, TIF_MEMDIE);

BTW, how about the other threads which share mm_struct ?

Quoted 2 lines

Could you elaborate your intention? :)

IIUC, the purpose of rising priority is to accerate dying thread to exit() for freeing memory AFAP. But to free memory, exit, all threads which share mm_struct should exit, too. I'm sorry if I miss something.

Quoted 6 lines
Quoted 4 lines

How do we kill only some thread and what's the benefit of it? I think when if some thread receives A KILL signal, the process include the thread will be killed.

yes, so, if you want a _process_ die quickly, you have to acceralte the whole threads on a process. Acceralating a thread in a process is not big help.

Yes.

I see the code.
oom_kill_process is called by

1. mem_cgroup_out_of_memory
2. __out_of_memory
3. out_of_memory

Quoted 5 lines

(1,2) calls select_bad_process which select victim task in processes by do_each_process.
But 3 isn't In case of CONSTRAINT_MEMORY_POLICY, it kills current. In only the case, couldn't we pass task of process, not one of thread?

Hmm, my point is that priority-acceralation is against a thread, not against a process. So, most of threads in memory-eater will not gain high priority even with this patch and works slowly.
I have no objections to this patch. I just want to confirm the purpose. If this patch is for accelating exiting process by SIGKILL, it seems not enough.
If an explanation as "acceralating all thread's priority in a process seems overkill" is given in changelog or comment, it's ok to me.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#37 lore raw
Quoted 81 lines · KAMEZAWA Hiroyuki
Quoted 72 lines · On Mon, 31 May 2010 15:09:41 +0900 Minchan Kim <[email protected]> wrote:
Quoted 53 lines · KAMEZAWA Hiroyuki
Quoted 50 lines · On Mon, 31 May 2010 14:46:05 +0900 Minchan Kim <[email protected]> wrote:
Quoted 47 lines · KAMEZAWA Hiroyuki
Quoted 38 lines · On Mon, 31 May 2010 14:01:03 +0900 Minchan Kim <[email protected]> wrote:

Hi, Kame.

Quoted 25 lines · KAMEZAWA Hiroyuki
Quoted 23 lines · On Fri, 28 May 2010 13:48:26 -0300 "Luis Claudio R. Goncalves"…

oom-killer: give the dying task rt priority (v3)

Give the dying task RT priority so that it can be scheduled quickly and die, freeing needed memory.

mm/oom_kill.c +2 -0
index 84bbba2..2b0204f 100644
@@ -266,6 +266,8 @@ static struct task_struct *select_bad_process(unsigned long *ppoints)
   */
  static void __oom_kill_task(struct task_struct *p, int verbose)
  {
+     struct sched_param param;
+
       if (is_global_init(p)) {
               WARN_ON(1);
               printk(KERN_WARNING "tried to kill init!\n");
@@ -288,6 +290,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
        * exit() and clear out its resources quickly...
Quoted 8 lines

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 296

Quoted 6 lines
mm/oom_kill.c +2 -0
        */
       p->time_slice = HZ;
+     param.sched_priority = MAX_RT_PRIO-10;
+     sched_setscheduler(p, SCHED_FIFO, &param);
       set_tsk_thread_flag(p, TIF_MEMDIE);
 

BTW, how about the other threads which share mm_struct ?

Quoted 2 lines

Could you elaborate your intention? :)

IIUC, the purpose of rising priority is to accerate dying thread to exit() for freeing memory AFAP. But to free memory, exit, all threads which share mm_struct should exit, too. I'm sorry if I miss something.

Quoted 6 lines
Quoted 4 lines

How do we kill only some thread and what's the benefit of it? I think when if some thread receives  KILL signal, the process include the thread will be killed.

yes, so, if you want a _process_ die quickly, you have to acceralte the whole threads on a process. Acceralating a thread in a process is not big help.

Yes.

I see the code.
oom_kill_process is called by

1. mem_cgroup_out_of_memory
2. __out_of_memory
3. out_of_memory

Quoted 5 lines

(1,2) calls select_bad_process which select victim task in processes by do_each_process.
But 3 isn't In case of  CONSTRAINT_MEMORY_POLICY, it kills current. In only the case, couldn't we pass task of process, not one of thread?

Quoted 7 lines

Hmm, my point is that priority-acceralation is against a thread, not against a process. So, most of threads in memory-eater will not gain high priority even with this patch and works slowly.
I have no objections to this patch. I just want to confirm the purpose. If this patch is for accelating exiting process by SIGKILL, it seems not enough.
If an explanation as "acceralating all thread's priority in a process seems overkill" is given in changelog or comment, it's ok to me.

Okay. I got your point.
Kame's concern is proper.

Couldn't we raise priorities of whole threads of the task killed?

Quoted 5 lines

Thanks,
-Kame

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#38 lore raw
In reply to KAMEZAWA Hiroyuki

On Mon, May 31, 2010 at 03:51:02PM +0900, KAMEZAWA Hiroyuki wrote:

| On Mon, 31 May 2010 15:09:41 +0900
| Minchan Kim <[email protected]> wrote:
| > On Mon, May 31, 2010 at 2:54 PM, KAMEZAWA Hiroyuki
| > <[email protected]> wrote:
...
| > >> > IIUC, the purpose of rising priority is to accerate dying thread to exit()
| > >> > for freeing memory AFAP. But to free memory, exit, all threads which share
| > >> > mm_struct should exit, too. I'm sorry if I miss something.
| > >>
| > >> How do we kill only some thread and what's the benefit of it?
| > >> I think when if some thread receives  KILL signal, the process include
| > >> the thread will be killed.
| > >>
| > > yes, so, if you want a _process_ die quickly, you have to acceralte the whole
| > > threads on a process. Acceralating a thread in a process is not big help.
| > 
| > Yes.
| > 
| > I see the code.
| > oom_kill_process is called by
| > 
| > 1. mem_cgroup_out_of_memory
| > 2. __out_of_memory
| > 3. out_of_memory
| > 
| > 
| > (1,2) calls select_bad_process which select victim task in processes
| > by do_each_process.
| > But 3 isn't In case of  CONSTRAINT_MEMORY_POLICY, it kills current.
| > In only the case, couldn't we pass task of process, not one of thread?
| > 
| 
| Hmm, my point is that priority-acceralation is against a thread, not against a process.
| So, most of threads in memory-eater will not gain high priority even with this patch
| and works slowly.

This is a good point...

| I have no objections to this patch. I just want to confirm the purpose. If this patch | is for accelating exiting process by SIGKILL, it seems not enough.

I understand (from the comments in the code) the badness calculation gives more points to the siblings in a thread that have their own mm. I wonder if what you are describing is not a corner case.

Again, your idea sounds like an interesting refinement to the patch. I am just not sure this change should implemented now or in a second round of changes.

| If an explanation as "acceralating all thread's priority in a process seems overkill" | is given in changelog or comment, it's ok to me.

If my understanding of badness() is right, I wouldn't be ashamed of saying that it seems to be _a bit_ overkill. But I may be wrong in my interpretation.

While re-reading the code I noticed that in select_bad_process() we can eventually bump on an already dying task, case in which we just wait for the task to die and avoid killing other tasks. Maybe we could boost the priority of the dying task here too.

Luis

Signature · 8 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#39 lore raw
Quoted 12 lines · On Mon, 31 May 2010 10:52:27 -0300 "Luis Claudio R. Goncalves"…

| If an explanation as "acceralating all thread's priority in a process seems overkill" | is given in changelog or comment, it's ok to me.

If my understanding of badness() is right, I wouldn't be ashamed of saying that it seems to be _a bit_ overkill. But I may be wrong in my interpretation.

While re-reading the code I noticed that in select_bad_process() we can eventually bump on an already dying task, case in which we just wait for the task to die and avoid killing other tasks. Maybe we could boost the priority of the dying task here too.

yes, nice catch.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#40 lore raw

On Tue, Jun 01, 2010 at 08:50:06AM +0900, KAMEZAWA Hiroyuki wrote:

| On Mon, 31 May 2010 10:52:27 -0300
| "Luis Claudio R. Goncalves" <[email protected]> wrote:
| 
| > | If an explanation as "acceralating all thread's priority in a process seems overkill"
| > | is given in changelog or comment, it's ok to me.
| > 
| > If my understanding of badness() is right, I wouldn't be ashamed of saying
| > that it seems to be _a bit_ overkill. But I may be wrong in my
| > interpretation.
| > 
| > While re-reading the code I noticed that in select_bad_process() we can
| > eventually bump on an already dying task, case in which we just wait for
| > the task to die and avoid killing other tasks. Maybe we could boost the
| > priority of the dying task here too.
| > 
| yes, nice catch.

Here is a more complete version of the patch, boosting priority on the three exit points of the OOM-killer. I also avoid touching the priority if the task is already an RT task. The patch:

oom-kill: give the dying task a higher priority (v5)

In a system under heavy load it was observed that even after the oom-killer selects a task to die, the task may take a long time to die.

Right before sending a SIGKILL to the task selected by the oom-killer this task has it's priority increased so that it can exit() exit soon, freeing memory. That is accomplished by:

/*
* We give our sacrificial lamb high priority and access to
* all the memory it needs. That way it should be able to
* exit() and clear out its resources quickly... */
p->rt.time_slice = HZ;
set_tsk_thread_flag(p, TIF_MEMDIE);

It sounds plausible giving the dying task an even higher priority to be sure it will be scheduled sooner and free the desired memory. It was suggested on LKML using SCHED_FIFO:1, the lowest RT priority so that this task won't interfere with any running RT task.

If the dying task is already an RT task, leave it untouched.

Another good suggestion, implemented here, was to avoid boosting the dying task priority in case of mem_cgroup OOM.

mm/oom_kill.c +30 -10
index 709aedf..67e18ca 100644
@@ -52,6 +52,22 @@ static int has_intersects_mems_allowed(struct task_struct *tsk)
 	return 0;
 }
 
+/*
+ * If this is a system OOM (not a memcg OOM) and the task selected to be
+ * killed is not already running at high (RT) priorities, speed up the
+ * recovery by boosting the dying task to the lowest FIFO priority.
+ * That helps with the recovery and avoids interfering with RT tasks.
+ */
+static void boost_dying_task_prio(struct task_struct *p,
+					struct mem_cgroup *mem)
+{
+	if ((mem == NULL) && !rt_task(p)) {
+		struct sched_param param;
+		param.sched_priority = 1;
+		sched_setscheduler_nocheck(p, SCHED_FIFO, &param);
+	}
+}
+
 /**
  * badness - calculate a numeric value for how bad this task has been
  * @p: task struct of which task we should calculate
@@ -277,8 +293,10 @@ static struct task_struct *select_bad_process(unsigned long *ppoints,
 		 * blocked waiting for another task which itself is waiting
 		 * for memory. Is there a better alternative?
 		 */
-		if (test_tsk_thread_flag(p, TIF_MEMDIE))
+		if (test_tsk_thread_flag(p, TIF_MEMDIE)) {
+			boost_dying_task_prio(p, mem);
 			return ERR_PTR(-1UL);
+		}
 
 		/*
 		 * This is in the process of releasing memory so wait for it
@@ -291,9 +309,10 @@ static struct task_struct *select_bad_process(unsigned long *ppoints,
 		 * Otherwise we could get an easy OOM deadlock.
 		 */
 		if (p->flags & PF_EXITING) {
-			if (p != current)
+			if (p != current) {
+				boost_dying_task_prio(p, mem);
 				return ERR_PTR(-1UL);
-
+			}
 			chosen = p;
 			*ppoints = ULONG_MAX;
 		}
@@ -380,7 +399,8 @@ static void dump_header(struct task_struct *p, gfp_t gfp_mask, int order,
  * flag though it's unlikely that  we select a process with CAP_SYS_RAW_IO
  * set.
  */
-static void __oom_kill_task(struct task_struct *p, int verbose)
+static void __oom_kill_task(struct task_struct *p, struct mem_cgroup *mem,
+								int verbose)
 {
 	if (is_global_init(p)) {
 		WARN_ON(1);
@@ -413,11 +433,11 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 */
 	p->rt.time_slice = HZ;
 	set_tsk_thread_flag(p, TIF_MEMDIE);
-
 	force_sig(SIGKILL, p);
+	boost_dying_task_prio(p, mem);
 }
 
-static int oom_kill_task(struct task_struct *p)
+static int oom_kill_task(struct task_struct *p, struct mem_cgroup *mem)
 {
 	/* WARNING: mm may not be dereferenced since we did not obtain its
 	 * value from get_task_mm(p).  This is OK since all we need to do is
@@ -430,7 +450,7 @@ static int oom_kill_task(struct task_struct *p)
 	if (!p->mm || p->signal->oom_adj == OOM_DISABLE)
 		return 1;
 
-	__oom_kill_task(p, 1);
+	__oom_kill_task(p, mem, 1);
 
 	return 0;
 }
@@ -449,7 +469,7 @@ static int oom_kill_process(struct task_struct *p, gfp_t gfp_mask, int order,
 	 * its children or threads, just set TIF_MEMDIE so it can die quickly
 	 */
 	if (p->flags & PF_EXITING) {
-		__oom_kill_task(p, 0);
+		__oom_kill_task(p, mem, 0);
 		return 0;
 	}
 
@@ -462,10 +482,10 @@ static int oom_kill_process(struct task_struct *p, gfp_t gfp_mask, int order,
 			continue;
 		if (mem && !task_in_mem_cgroup(c, mem))
 			continue;
-		if (!oom_kill_task(c))
+		if (!oom_kill_task(c, mem))
 			return 0;
 	}
-	return oom_kill_task(p);
+	return oom_kill_task(p, mem);
 }
 
 #ifdef CONFIG_CGROUP_MEM_RES_CTLR
Signature · 2 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]
#41 lore raw
Quoted 58 lines · Luis Claudio R. Goncalves

oom-kill: give the dying task a higher priority (v5)

In a system under heavy load it was observed that even after the oom-killer selects a task to die, the task may take a long time to die.

Right before sending a SIGKILL to the task selected by the oom-killer this task has it's priority increased so that it can exit() exit soon, freeing memory. That is accomplished by:

/*
* We give our sacrificial lamb high priority and access to
* all the memory it needs. That way it should be able to
* exit() and clear out its resources quickly... */

mm/oom_kill.c +0 -0
 	p->rt.time_slice = HZ;
 	set_tsk_thread_flag(p, TIF_MEMDIE);

It sounds plausible giving the dying task an even higher priority to be sure it will be scheduled sooner and free the desired memory. It was suggested on LKML using SCHED_FIFO:1, the lowest RT priority so that this task won't interfere with any running RT task.

If the dying task is already an RT task, leave it untouched.

Another good suggestion, implemented here, was to avoid boosting the dying task priority in case of mem_cgroup OOM.

mm/oom_kill.c +16 -0
index 709aedf..67e18ca 100644
@@ -52,6 +52,22 @@ static int has_intersects_mems_allowed(struct task_struct *tsk)
 	return 0;
 }
 
+/*
+ * If this is a system OOM (not a memcg OOM) and the task selected to be
+ * killed is not already running at high (RT) priorities, speed up the
+ * recovery by boosting the dying task to the lowest FIFO priority.
+ * That helps with the recovery and avoids interfering with RT tasks.
+ */
+static void boost_dying_task_prio(struct task_struct *p,
+					struct mem_cgroup *mem)
+{
+	if ((mem == NULL) && !rt_task(p)) {
+		struct sched_param param;
+		param.sched_priority = 1;
+		sched_setscheduler_nocheck(p, SCHED_FIFO, &param);
+	}
+}
+
 /**
  * badness - calculate a numeric value for how bad this task has been
  * @p: task struct of which task we should calculate
@@ -277,8 +293,10 @@ static struct task_struct *select_bad_process(unsigned long *ppoints,
 		 * blocked waiting for another task which itself is waiting

mm/oom_kill.c static struct task_struct *select_bad_process(unsigned long *ppoints, · line 302

Quoted 10 lines
mm/oom_kill.c +3 -1
 		 * for memory. Is there a better alternative?
 		 */
-		if (test_tsk_thread_flag(p, TIF_MEMDIE))
+		if (test_tsk_thread_flag(p, TIF_MEMDIE)) {
+			boost_dying_task_prio(p, mem);
 			return ERR_PTR(-1UL);
+		}
 
 		/*
 		 * This is in the process of releasing memory so wait for it

That's unnecessary, if p already has TIF_MEMDIE set, then boost_dying_task_prio(p) has already been called.

mm/oom_kill.c static struct task_struct *select_bad_process(unsigned long *ppoints, · line 318

Quoted 13 lines
mm/oom_kill.c +3 -2
@@ -291,9 +309,10 @@ static struct task_struct *select_bad_process(unsigned long *ppoints,
 		 * Otherwise we could get an easy OOM deadlock.
 		 */
 		if (p->flags & PF_EXITING) {
-			if (p != current)
+			if (p != current) {
+				boost_dying_task_prio(p, mem);
 				return ERR_PTR(-1UL);
-
+			}
 			chosen = p;
 			*ppoints = ULONG_MAX;
 		}

This has the potential to actually make it harder to free memory if p is waiting to acquire a writelock on mm->mmap_sem in the exit path while the thread holding mm->mmap_sem is trying to run.

#42 lore raw
Quoted 9 lines
Quoted 9 lines
mm/oom_kill.c +2 -2
@@ -291,9 +309,10 @@ static struct task_struct *select_bad_process(unsigned long *ppoints,
 		 * Otherwise we could get an easy OOM deadlock.
 		 */
 		if (p->flags & PF_EXITING) {
-			if (p != current)
+			if (p != current) {
+				boost_dying_task_prio(p, mem);
 				return ERR_PTR(-1UL);
-
Quoted 8 lines

mm/oom_kill.c static struct task_struct *select_bad_process(unsigned long *ppoints, · line 318

Quoted 4 lines
mm/oom_kill.c +1 -0
+			}
 			chosen = p;
 			*ppoints = ULONG_MAX;
 		}

This has the potential to actually make it harder to free memory if p is waiting to acquire a writelock on mm->mmap_sem in the exit path while the thread holding mm->mmap_sem is trying to run.

if p is waiting, changing prio have no effect. It continue tol wait to release mmap_sem.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#43 lore raw

On Wed, Jun 02, 2010 at 10:54:01PM +0900, KOSAKI Motohiro wrote:

| > > @@ -291,9 +309,10 @@ static struct task_struct *select_bad_process(unsigned long *ppoints,
| > >  		 * Otherwise we could get an easy OOM deadlock.
| > >  		 */
| > >  		if (p->flags & PF_EXITING) {
| > > -			if (p != current)
| > > +			if (p != current) {
| > > +				boost_dying_task_prio(p, mem);
| > >  				return ERR_PTR(-1UL);
| > > -
| > > +			}
| > >  			chosen = p;
| > >  			*ppoints = ULONG_MAX;
| > >  		}
| > 
| > This has the potential to actually make it harder to free memory if p is 
| > waiting to acquire a writelock on mm->mmap_sem in the exit path while the 
| > thread holding mm->mmap_sem is trying to run.
| 
| if p is waiting, changing prio have no effect. It continue tol wait to release mmap_sem.

Ok, that was not a good idea after all :)

But I understand the !rt_task(p) test is necessary to avoid decrementing the priority of an eventual RT task selected to die. Though it may also be a corner case in badness().

Luis

Signature · 8 lines
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ]
[ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#44 lore raw
In reply to KOSAKI Motohiro
Quoted 18 lines · KOSAKI Motohiro
Quoted 9 lines
Quoted 9 lines
mm/oom_kill.c +2 -2
@@ -291,9 +309,10 @@ static struct task_struct *select_bad_process(unsigned long *ppoints,
 		 * Otherwise we could get an easy OOM deadlock.
 		 */
 		if (p->flags & PF_EXITING) {
-			if (p != current)
+			if (p != current) {
+				boost_dying_task_prio(p, mem);
 				return ERR_PTR(-1UL);
-
Quoted 8 lines

mm/oom_kill.c static struct task_struct *select_bad_process(unsigned long *ppoints, · line 318

Quoted 4 lines
mm/oom_kill.c +1 -0
+			}
 			chosen = p;
 			*ppoints = ULONG_MAX;
 		}

This has the potential to actually make it harder to free memory if p is waiting to acquire a writelock on mm->mmap_sem in the exit path while the thread holding mm->mmap_sem is trying to run.

Quoted 2 lines

if p is waiting, changing prio have no effect. It continue tol wait to release mmap_sem.

And that can reduce the runtime of the thread holding a writelock on mm->mmap_sem, making the exit actually take longer than without the patch if its priority is significantly higher, especially on smaller machines.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#45 lore raw
Quoted 20 lines · KOSAKI Motohiro
Quoted 19 lines
Quoted 18 lines
Quoted 4 lines
mm/oom_kill.c +0 -0
@@ -291,9 +309,10 @@ static struct task_struct *select_bad_process(unsigned long *ppoints,
 		 * Otherwise we could get an easy OOM deadlock.
 		 */

mm/oom_kill.c static struct task_struct *select_bad_process(unsigned long *ppoints, · line 318

Quoted 10 lines
mm/oom_kill.c +3 -2
 		if (p->flags & PF_EXITING) {
-			if (p != current)
+			if (p != current) {
+				boost_dying_task_prio(p, mem);
 				return ERR_PTR(-1UL);
-
+			}
 			chosen = p;
 			*ppoints = ULONG_MAX;
 		}

This has the potential to actually make it harder to free memory if p is waiting to acquire a writelock on mm->mmap_sem in the exit path while the thread holding mm->mmap_sem is trying to run.

Quoted 6 lines
Quoted 2 lines

if p is waiting, changing prio have no effect. It continue tol wait to release mmap_sem.

And that can reduce the runtime of the thread holding a writelock on mm->mmap_sem, making the exit actually take longer than without the patch if its priority is significantly higher, especially on smaller machines.

If p need mmap_sem, p is going to sleep to wait mmap_sem. if p doesn't, quickly exit is good thing. In other word, task fairness is not our goal when oom occur.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#46 lore raw
Quoted 23 lines · KOSAKI Motohiro
Quoted 23 lines · KOSAKI Motohiro
Quoted 21 lines
Quoted 10 lines
Quoted 10 lines
mm/oom_kill.c +2 -2
@@ -291,9 +309,10 @@ static struct task_struct *select_bad_process(unsigned long *ppoints,
                  * Otherwise we could get an easy OOM deadlock.
                  */
                 if (p->flags & PF_EXITING) {
-                       if (p != current)
+                       if (p != current) {
+                               boost_dying_task_prio(p, mem);
                                 return ERR_PTR(-1UL);
-
Quoted 8 lines

mm/oom_kill.c static struct task_struct *select_bad_process(unsigned long *ppoints, · line 318

Quoted 4 lines
mm/oom_kill.c +1 -0
+                       }
                         chosen = p;
                         *ppoints = ULONG_MAX;
                 }

This has the potential to actually make it harder to free memory if p is waiting to acquire a writelock on mm->mmap_sem in the exit path while the thread holding mm->mmap_sem is trying to run.

if p is waiting, changing prio have no effect. It continue tol wait to release mmap_sem.

Quoted 8 lines
Quoted 3 lines

And that can reduce the runtime of the thread holding a writelock on mm->mmap_sem, making the exit actually take longer than without the patch if its priority is significantly higher, especially on smaller machines.

If p need mmap_sem, p is going to sleep to wait mmap_sem. if p doesn't, quickly exit is good thing. In other word, task fairness is not our goal when oom occur.

Tend to agree. I didn't agree boosting of whole threads' priority.

Task fairness VS system hang is trade off. task fairness is best effort but system hang is critical.
Also, we have tried to it.

        /*
         * We give our sacrificial lamb high priority and access to
         * all the memory it needs. That way it should be able to
         * exit() and clear out its resources quickly...
         */
        p->rt.time_slice = HZ;
        set_tsk_thread_flag(p, TIF_MEMDIE);

But I think above code is meaningless unless p use SCHED_RR. So boosting of lowest RT priority with FIFO is to meet above comment's goal, I think.

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#47 lore raw
In reply to David Rientjes
Quoted 4 lines · David Rientjes

And that can reduce the runtime of the thread holding a writelock on mm->mmap_sem, making the exit actually take longer than without the patch if its priority is significantly higher, especially on smaller machines.

/me smells an inversion... on -rt we solved those ;-)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#48 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines

And that can reduce the runtime of the thread holding a writelock on mm->mmap_sem, making the exit actually take longer than without the patch if its priority is significantly higher, especially on smaller machines.

/me smells an inversion... on -rt we solved those ;-)

Right, but I don't see how increasing an oom killed tasks priority to a divine priority doesn't impact the priorities of other tasks which may be blocking the exit of that task, namely a coredumper or holder of mm->mmap_sem. This patch also doesn't address how it negatively impacts the priorities of jobs running in different cpusets (although sharing the same cpus) because one cpuset is oom.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#49 lore raw
In reply to Luis Claudio R. Goncalves
Quoted 42 lines · Luis Claudio R. Goncalves

On Mon, May 31, 2010 at 03:51:02PM +0900, KAMEZAWA Hiroyuki wrote:

| On Mon, 31 May 2010 15:09:41 +0900
| Minchan Kim <[email protected]> wrote:
| > On Mon, May 31, 2010 at 2:54 PM, KAMEZAWA Hiroyuki
| > <[email protected]> wrote:
...
| > >> > IIUC, the purpose of rising priority is to accerate dying thread to exit()
| > >> > for freeing memory AFAP. But to free memory, exit, all threads which share
| > >> > mm_struct should exit, too. I'm sorry if I miss something.
| > >>
| > >> How do we kill only some thread and what's the benefit of it?
| > >> I think when if some thread receives  KILL signal, the process include
| > >> the thread will be killed.
| > >>
| > > yes, so, if you want a _process_ die quickly, you have to acceralte the whole
| > > threads on a process. Acceralating a thread in a process is not big help.
| >
| > Yes.
| >
| > I see the code.
| > oom_kill_process is called by
| >
| > 1. mem_cgroup_out_of_memory
| > 2. __out_of_memory
| > 3. out_of_memory
| >
| >
| > (1,2) calls select_bad_process which select victim task in processes
| > by do_each_process.
| > But 3 isn't In case of  CONSTRAINT_MEMORY_POLICY, it kills current.
| > In only the case, couldn't we pass task of process, not one of thread?
| >
|
| Hmm, my point is that priority-acceralation is against a thread, not against a process.
| So, most of threads in memory-eater will not gain high priority even with this patch
| and works slowly.

This is a good point...

| I have no objections to this patch. I just want to confirm the purpose. If this patch | is for accelating exiting process by SIGKILL, it seems not enough.

Quoted 7 lines

I understand (from the comments in the code) the badness calculation gives more points to the siblings in a thread that have their own mm. I wonder if what you are describing is not a corner case.

Again, your idea sounds like an interesting refinement to the patch. I am just not sure this change should implemented now or in a second round of changes.

First of all, I think your patch is first. That's because I am not sure this logic is effective.

/*
* We give our sacrificial lamb high priority and access to
* all the memory it needs. That way it should be able to
* exit() and clear out its resources quickly... */
p->rt.time_slice = HZ;

Peter changed it in fa717060f1ab.
Now if we change rt.time_slice as HZ, it means the task have high priority? I am not a scheduler expert. but as I looked through scheduler code, rt.time_slice is only related to RT scheduler. so if we uses CFS, it doesn't make task high priority.
Perter, Right?

If it is right, I think Luis patch will fix it.

Secondly, as Kame pointed out, we have to raise whole thread's priority to kill victim process for reclaiming pages. But I think it has deadlock problem.
If we raise whole threads's priority and some thread has dependency of other thread which is blocked, it makes system deadlock. So I think it's not easy part.

If this part is really big problem, we should consider it more carefully.

Quoted 4 lines

| If an explanation as "acceralating all thread's priority in a process seems overkill" | is given in changelog or comment, it's ok to me.

Quoted 8 lines

If my understanding of badness() is right, I wouldn't be ashamed of saying that it seems to be _a bit_ overkill. But I may be wrong in my interpretation.

While re-reading the code I noticed that in select_bad_process() we can eventually bump on an already dying task, case in which we just wait for the task to die and avoid killing other tasks. Maybe we could boost the priority of the dying task here too.

Yes. It is good where we boost priority of task, I think.

Quoted 7 lines

Luis
--
[ Luis Claudio R. Goncalves                    Bass - Gospel - RT ] [ Fingerprint: 4FDD B8C4 3C59 34BD 8BE9  2696 7203 D980 A448 C8F8 ]

Signature · 8 lines
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#50 lore raw
Quoted 3 lines · Minchan Kim

Secondly, as Kame pointed out, we have to raise whole thread's priority to kill victim process for reclaiming pages. But I think it has deadlock problem.

Agreed, this has the potential to actually increase the amount of time for an oom killed task to fully exit: the exit path takes mm->mmap_sem on exit and if that is held by another thread waiting for the oom killed task to exit (i.e. reclaim has failed and the oom killer becomes a no-op because it sees an already killed task) then there's a livelock. That's always been a problem, but is compounded with increasing the priority of a task not holding mm->mmap_sem if the thread holding the writelock actually isn't looking for memory but simply doesn't get a chance to release because it fails to run.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#51 lore raw
In reply to KOSAKI Motohiro
Quoted 26 lines · KOSAKI Motohiro
Quoted 17 lines · Luis Claudio R. Goncalves
Quoted 16 lines
mm/oom_kill.c +2 -1
@@ -382,6 +382,8 @@ static void dump_header(struct task_struct *p, gfp_t gfp_mask, int order,
  */
 static void __oom_kill_task(struct task_struct *p, int verbose)
 {
+	struct sched_param param;
+
 	if (is_global_init(p)) {
 		WARN_ON(1);
 		printk(KERN_WARNING "tried to kill init!\n");
@@ -413,8 +415,9 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 */
 	p->rt.time_slice = HZ;
 	set_tsk_thread_flag(p, TIF_MEMDIE);
-
 	force_sig(SIGKILL, p);
Quoted 8 lines

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 422

Quoted 4 lines
mm/oom_kill.c +2 -0
+	param.sched_priority = MAX_RT_PRIO-1;
+	sched_setscheduler_nocheck(p, SCHED_FIFO, &param);
 }
 

I would like to understand the visible benefits of this patch. Have you seen an OOM kill tasked really get bogged down. Should this task really be competing with other important tasks for run time?

Quoted 7 lines

What you mean important? Until OOM victim task exit completely, the system have no memory. all of important task can't do anything.

In almost kernel subsystems, automatically priority boost is really bad idea because it may break RT task's deterministic behavior. but OOM is one of exception. The deterministic was alread broken by memory starvation.

I am still not convinced, specially if we are running under mem cgroup. Even setting SCHED_FIFO does not help, you could have other things like cpusets that might restrict the CPUs you can run on, or any other policy and we could end up contending anyway with other SCHED_FIFO tasks.

Quoted 1 line

That's the reason I acked it.

If we could show faster recovery from OOM or anything else, I would be more convinced.

Signature · 8 lines
	Three Cheers,
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#52 lore raw
Quoted 36 lines · KOSAKI Motohiro
Quoted 27 lines
Quoted 18 lines · Luis Claudio R. Goncalves
Quoted 16 lines
mm/oom_kill.c +2 -1
@@ -382,6 +382,8 @@ static void dump_header(struct task_struct *p, gfp_t gfp_mask, int order,
  */
 static void __oom_kill_task(struct task_struct *p, int verbose)
 {
+	struct sched_param param;
+
 	if (is_global_init(p)) {
 		WARN_ON(1);
 		printk(KERN_WARNING "tried to kill init!\n");
@@ -413,8 +415,9 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 */
 	p->rt.time_slice = HZ;
 	set_tsk_thread_flag(p, TIF_MEMDIE);
-
 	force_sig(SIGKILL, p);
Quoted 8 lines

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 422

Quoted 4 lines
mm/oom_kill.c +2 -0
+	param.sched_priority = MAX_RT_PRIO-1;
+	sched_setscheduler_nocheck(p, SCHED_FIFO, &param);
 }
 

I would like to understand the visible benefits of this patch. Have you seen an OOM kill tasked really get bogged down. Should this task really be competing with other important tasks for run time?

Quoted 7 lines

What you mean important? Until OOM victim task exit completely, the system have no memory. all of important task can't do anything.

In almost kernel subsystems, automatically priority boost is really bad idea because it may break RT task's deterministic behavior. but OOM is one of exception. The deterministic was alread broken by memory starvation.

Quoted 5 lines

I am still not convinced, specially if we are running under mem cgroup. Even setting SCHED_FIFO does not help, you could have other things like cpusets that might restrict the CPUs you can run on, or any other policy and we could end up contending anyway with other SCHED_FIFO tasks.

Ah, right you are. I had missed mem-cgroup.
But I think memcgroup also don't need following two boost. Can we get rid of it?

	p->rt.time_slice = HZ;
	set_tsk_thread_flag(p, TIF_MEMDIE);

I mean we need distinguish global oom and memcg oom, perhapls.

Quoted 4 lines
Quoted 1 line

That's the reason I acked it.

If we could show faster recovery from OOM or anything else, I would be more convinced.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#53 lore raw
In reply to Balbir Singh
Quoted 11 lines · On Fri, 28 May 2010 11:57:01 +0530 Balbir Singh <[email protected]>…

I am still not convinced, specially if we are running under mem cgroup. Even setting SCHED_FIFO does not help, you could have other things like cpusets that might restrict the CPUs you can run on, or any other policy and we could end up contending anyway with other SCHED_FIFO tasks.

Quoted 1 line

That's the reason I acked it.

If we could show faster recovery from OOM or anything else, I would be more convinced.

Off topic.

1. Run a daemon in the highest RT priority.
2. disable OOM for a mem cgroup.
3. The daemon register oom-event-notifier of the mem cgroup.

When OOM happens.
4. The daemon receive a event, and then,
a) enlarge limit
or
b) kill a task
or
c) enlarge limit temporary and kill a task, later, reduce limit again.

This is the fastest and promissing operation for memcg users.

memcg's oom slowdown happens just because it's limited by a user configuration not by the system. That's a point to be considered.
The oom situation can be _immediaterly_ fixed up by enlarge limit as emergency mode.

If you has to wait for the end of a task, there will be delay, it's unavoidable.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#54 lore raw
In reply to Luis Claudio R. Goncalves

mm/oom_kill.c static void __oom_kill_task(struct task_struct *p, int verbose) · line 420

Quoted 2 lines · Luis Claudio R. Goncalves
mm/oom_kill.c +2 -0
+       param.sched_priority = MAX_RT_PRIO-1;
+       sched_setscheduler_nocheck(p, SCHED_FIFO, &param);

Argh, so you got me confused as well.

the sched_param ones are userspace values, so you should be using 1.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>