Skip to contentSkip to conversation

Page writeback broken after resume: wb_timer lost

PATCH In review Peter Lundkvist 7 participants 14 messages May 20, 2006 – Jun 22, 2006 Memory management

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Page writeback broken after resume: wb_timer lost

Hi,
I have noticed for some time that nr_dirty never drops but increases except when VM pressure forces it down. This only occurs after a resume, never on a freshly booted system.

It seems the wb_timer is lost when the timer function is trying to start a frozen pdflush thread, and this occurs during suspend or resume.

I have included a patch which work for me. Don't know if the test also should include a check for freezing to be safe, ie
if ( !frozen(..) && !freezing(..) )

mm/pdflush.c +10 -6
diff -ru linux-2.6.17.org/mm/pdflush.c linux-2.6.17/mm/pdflush.c
@@ -213,12 +213,16 @@
 		struct pdflush_work *pdf;
 
 		pdf = list_entry(pdflush_list.next, struct pdflush_work, list);
-		list_del_init(&pdf->list);
-		if (list_empty(&pdflush_list))
-			last_empty_jifs = jiffies;
-		pdf->fn = fn;
-		pdf->arg0 = arg0;
-		wake_up_process(pdf->who);
+		if (!frozen(pdf->who)) {
+			list_del_init(&pdf->list);
+			if (list_empty(&pdflush_list))
+				last_empty_jifs = jiffies;
+			pdf->fn = fn;
+			pdf->arg0 = arg0;
+			wake_up_process(pdf->who);
+		}
+		else
+			ret = -1;
 		spin_unlock_irqrestore(&pdflush_lock, flags);
 	}
 	return ret;
Signature · 1 line
Peter Lundkvist
#1 lore raw
Quoted 32 lines · Peter Lundkvist

Hi,
I have noticed for some time that nr_dirty never drops but increases except when VM pressure forces it down. This only occurs after a resume, never on a freshly booted system.

It seems the wb_timer is lost when the timer function is trying to start a frozen pdflush thread, and this occurs during suspend or resume.

I have included a patch which work for me. Don't know if the test also should include a check for freezing to be safe, ie
if ( !frozen(..) && !freezing(..) )

diff -ru linux-2.6.17.org/mm/pdflush.c linux-2.6.17/mm/pdflush.c

mm/pdflush.c +3 -6
@@ -213,12 +213,16 @@
 		struct pdflush_work *pdf;
 
 		pdf = list_entry(pdflush_list.next, struct pdflush_work, list);
-		list_del_init(&pdf->list);
-		if (list_empty(&pdflush_list))
-			last_empty_jifs = jiffies;
-		pdf->fn = fn;
-		pdf->arg0 = arg0;
-		wake_up_process(pdf->who);
+		if (!frozen(pdf->who)) {
+			list_del_init(&pdf->list);
+			if (list_empty(&pdflush_list))

mm/pdflush.c line 228

Quoted 10 lines
mm/pdflush.c +7 -0
+				last_empty_jifs = jiffies;
+			pdf->fn = fn;
+			pdf->arg0 = arg0;
+			wake_up_process(pdf->who);
+		}
+		else
+			ret = -1;
 		spin_unlock_irqrestore(&pdflush_lock, flags);
 	}
 	return ret;

Maybe the code over in page-writeback.c should just rearm the timee within the timer handler rather than waiting for a pdflush thread to do it. I'll think about that.

But the main questions is: what on earth is going on here? We've taken a kernel thread and we've done a wake_up_process() on it, but because it was in a frozen state it just never gets to run, even after the resume. Presumably it goes back into interruptible sleep after the resume. We took it off the list (in the expectation that it'd run again) so we've lost control of it.

Pavel, Rafael: this amounts to a lost wakeup. What's the story?

#2 lore raw

Hi!

Quoted 11 lines
Quoted 11 lines

I have noticed for some time that nr_dirty never drops but increases except when VM pressure forces it down. This only occurs after a resume, never on a freshly booted system.

It seems the wb_timer is lost when the timer function is trying to start a frozen pdflush thread, and this occurs during suspend or resume.

I have included a patch which work for me. Don't know if the test also should include a check for freezing to be safe, ie
if ( !frozen(..) && !freezing(..) )

Yep, I have seen this too. Sync took *way* too long and I believe I lost some data because of this problem.

Quoted 10 lines

Maybe the code over in page-writeback.c should just rearm the timee within the timer handler rather than waiting for a pdflush thread to do it. I'll think about that.

But the main questions is: what on earth is going on here? We've taken a kernel thread and we've done a wake_up_process() on it, but because it was in a frozen state it just never gets to run, even after the resume. Presumably it goes back into interruptible sleep after the resume. We took it off the list (in the expectation that it'd run again) so we've lost control of it.

I guess you should not try to wake up process while it is frozen. Such wakeups are likely to get lost. Should we add some BUG_ON() somewhere?

...we have to eat some wakeups, because we fake some.

Or perhaps we should do WARN_ON(frozen(current)) just after schedule() below?

Quoted 1 line

Pavel, Rafael: this amounts to a lost wakeup. What's the story?

Pavel

Refrigerator looks like this:

/* Refrigerator is place where frozen processes are stored :-). */ void refrigerator(void)

{
        /* Hmm, should we be allowed to suspend when there are
realtime
           processes around? */
        long save;
        save = current->state;
        pr_debug("%s entered refrigerator\n", current->comm);
        printk("=");

        frozen_process(current);
        spin_lock_irq(&current->sighand->siglock);
        recalc_sigpending(); /* We sent fake signal, clean it up */
        spin_unlock_irq(&current->sighand->siglock);

        while (frozen(current)) {
                current->state = TASK_UNINTERRUPTIBLE;
                schedule();
        }
        pr_debug("%s left refrigerator\n", current->comm);
        current->state = save;
}
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#3 lore raw
Quoted 25 lines · Pavel Machek

Refrigerator looks like this:

/* Refrigerator is place where frozen processes are stored :-). */ void refrigerator(void)

 {
         /* Hmm, should we be allowed to suspend when there are
 realtime
            processes around? */
         long save;
         save = current->state;
         pr_debug("%s entered refrigerator\n", current->comm);
         printk("=");

         frozen_process(current);
         spin_lock_irq(&current->sighand->siglock);
         recalc_sigpending(); /* We sent fake signal, clean it up */
         spin_unlock_irq(&current->sighand->siglock);

         while (frozen(current)) {
                 current->state = TASK_UNINTERRUPTIBLE;
                 schedule();
         }
         pr_debug("%s left refrigerator\n", current->comm);
         current->state = save;
Quoted 1 line

}

Well that's a crock, isn't it?

Peter, does this fix it?

From: Andrew Morton <[email protected]>

pdflush is carefully designed to ensure that all wakeups have some corresponding work to do - if a woken-up pdflush thread discovers that it hasn't been given any work to do then this is considered an error.

That all broke when swsusp came along - because a timer-delivered wakeup to a frozen pdflush thread will just get lost. This causes the pdflush thread to get lost as well: the writeback timer is supposed to be re-armed by pdflush in process context, but pdflush doesn't execute the callout which does this.

Fix that up by ignoring the return value from try_to_freeze(): jsut proceed, see if we have any work pending and only go back to sleep if that is not the case.

---

 mm/pdflush.c |    9 ++-------
 1 files changed, 2 insertions(+), 7 deletions(-)
mm/pdflush.c renamed from mm/pdflush.c~pdflush-handle-resume-wakeups +2 -7
diff -puN mm/pdflush.c~pdflush-handle-resume-wakeups mm/pdflush.c
@@ -104,13 +104,8 @@ static int __pdflush(struct pdflush_work
 		list_move(&my_work->list, &pdflush_list);
 		my_work->when_i_went_to_sleep = jiffies;
 		spin_unlock_irq(&pdflush_lock);
-
 		schedule();
-		if (try_to_freeze()) {
-			spin_lock_irq(&pdflush_lock);
-			continue;
-		}
-
+		try_to_freeze();
 		spin_lock_irq(&pdflush_lock);
 		if (!list_empty(&my_work->list)) {
 			printk("pdflush: bogus wakeup!\n");
@@ -118,7 +113,7 @@ static int __pdflush(struct pdflush_work
 			continue;
 		}
 		if (my_work->fn == NULL) {
-			printk("pdflush: NULL work function\n");
+			printk("pflush: resuming\n");
 			continue;
 		}
 		spin_unlock_irq(&pdflush_lock);

_

#4 lore raw
Quoted 3 lines · Andrew Morton

Peter, does this fix it?

Yes, it does fix the problem.

Hopefully we'll see this fix in the next stable update, because of the risk of data loss.

Quoted 54 lines

From: Andrew Morton <[email protected]>

pdflush is carefully designed to ensure that all wakeups have some corresponding work to do - if a woken-up pdflush thread discovers that it hasn't been given any work to do then this is considered an error.

That all broke when swsusp came along - because a timer-delivered wakeup to a frozen pdflush thread will just get lost. This causes the pdflush thread to get lost as well: the writeback timer is supposed to be re-armed by pdflush in process context, but pdflush doesn't execute the callout which does this.

Fix that up by ignoring the return value from try_to_freeze(): jsut proceed, see if we have any work pending and only go back to sleep if that is not the case.

---

 mm/pdflush.c |    9 ++-------
 1 files changed, 2 insertions(+), 7 deletions(-)

diff -puN mm/pdflush.c~pdflush-handle-resume-wakeups mm/pdflush.c

mm/pdflush.c renamed from mm/pdflush.c~pdflush-handle-resume-wakeups +2 -7
@@ -104,13 +104,8 @@ static int __pdflush(struct pdflush_work
 		list_move(&my_work->list, &pdflush_list);
 		my_work->when_i_went_to_sleep = jiffies;
 		spin_unlock_irq(&pdflush_lock);
-
 		schedule();
-		if (try_to_freeze()) {
-			spin_lock_irq(&pdflush_lock);
-			continue;
-		}
-
+		try_to_freeze();
 		spin_lock_irq(&pdflush_lock);
 		if (!list_empty(&my_work->list)) {
 			printk("pdflush: bogus wakeup!\n");
@@ -118,7 +113,7 @@ static int __pdflush(struct pdflush_work
 			continue;
 		}
 		if (my_work->fn == NULL) {
-			printk("pdflush: NULL work function\n");
+			printk("pflush: resuming\n");
 			continue;
 		}
 		spin_unlock_irq(&pdflush_lock);

_

#5 lore raw
In reply to Andrew Morton

Hi!

Quoted 12 lines

Well that's a crock, isn't it?

Peter, does this fix it?

From: Andrew Morton <[email protected]>

pdflush is carefully designed to ensure that all wakeups have some corresponding work to do - if a woken-up pdflush thread discovers that it hasn't been given any work to do then this is considered an error.

Quoted 8 lines

That all broke when swsusp came along - because a timer-delivered wakeup to a frozen pdflush thread will just get lost. This causes the pdflush thread to get lost as well: the writeback timer is supposed to be re-armed by pdflush in process context, but pdflush doesn't execute the callout which does this.

Fix that up by ignoring the return value from try_to_freeze(): jsut proceed, see if we have any work pending and only go back to sleep if that is not the case.

Looks okay to me.

Pavel
(who wonders what are the other places where we have similar problems)

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#6 lore raw
In reply to Andrew Morton
Quoted 11 lines · Andrew Morton

From: Andrew Morton <[email protected]>

pdflush is carefully designed to ensure that all wakeups have some corresponding work to do - if a woken-up pdflush thread discovers that it hasn't been given any work to do then this is considered an error.

That all broke when swsusp came along - because a timer-delivered wakeup to a frozen pdflush thread will just get lost. This causes the pdflush thread to get lost as well: the writeback timer is supposed to be re-armed by pdflush in process context, but pdflush doesn't execute the callout which does this.

Quoted 6 lines

Fix that up by ignoring the return value from try_to_freeze(): jsut proceed, see if we have any work pending and only go back to sleep if that is not the case.

I've tested this patch for about a week now, by applying it to the 2.6.17-rc3 kernel on my laptop, which I've been using for more than a month now. This patch seems to cure the mysterious symptoms reported in February:

http://lkml.org/lkml/2006/2/6/167
http://lkml.org/lkml/2006/2/6/170
http://lkml.org/lkml/2006/2/13/424
etc.

Actually I didn't remember to check "Dirty:" in /proc/meminfo, but when I "sync"ed at the end of my workday, just prior to swsupending it, sync returned immediately. with unpatched 2.6.17-rc3, sync would take half a minute. Maybe Mark can give this patch a spin to check if it cures his problem, too. (I still use vmware, so vmware was not the culprit.)

Thanks,
Johannes

Quoted 37 lines

---

 mm/pdflush.c |    9 ++-------
 1 files changed, 2 insertions(+), 7 deletions(-)

diff -puN mm/pdflush.c~pdflush-handle-resume-wakeups mm/pdflush.c

mm/pdflush.c renamed from mm/pdflush.c~pdflush-handle-resume-wakeups +2 -7
@@ -104,13 +104,8 @@ static int __pdflush(struct pdflush_work
 		list_move(&my_work->list, &pdflush_list);
 		my_work->when_i_went_to_sleep = jiffies;
 		spin_unlock_irq(&pdflush_lock);
-
 		schedule();
-		if (try_to_freeze()) {
-			spin_lock_irq(&pdflush_lock);
-			continue;
-		}
-
+		try_to_freeze();
 		spin_lock_irq(&pdflush_lock);
 		if (!list_empty(&my_work->list)) {
 			printk("pdflush: bogus wakeup!\n");
@@ -118,7 +113,7 @@ static int __pdflush(struct pdflush_work
 			continue;
 		}
 		if (my_work->fn == NULL) {
-			printk("pdflush: NULL work function\n");
+			printk("pflush: resuming\n");
 			continue;
 		}
 		spin_unlock_irq(&pdflush_lock);

_

#7 lore raw

Hi.

Sorry for coming late to the party. I've only just seen this thread, and have had reports of it too.

My concern is, shouldn't we be dealing with the cause rather than just one symptom (and as Pavel rightly wondered, assuming there aren't more)? I'm not sure that I have a solution, but I think the point is worth raising again.

Do we want something like adding the process's task struct to timer data, and get the timer code to delay firing timers for frozen processes?

Regards,

Nigel

Signature · 4 lines
Nigel, Michelle and Alisdair Cunningham
5 Mitchell Street
Cobden 3266
Victoria, Australia
#8 lore raw
In reply to Johannes Stezenbach
Quoted 30 lines · Johannes Stezenbach
Quoted 11 lines · Andrew Morton

From: Andrew Morton <[email protected]>

pdflush is carefully designed to ensure that all wakeups have some corresponding work to do - if a woken-up pdflush thread discovers that it hasn't been given any work to do then this is considered an error.

That all broke when swsusp came along - because a timer-delivered wakeup to a frozen pdflush thread will just get lost. This causes the pdflush thread to get lost as well: the writeback timer is supposed to be re-armed by pdflush in process context, but pdflush doesn't execute the callout which does this.

Quoted 6 lines

Fix that up by ignoring the return value from try_to_freeze(): jsut proceed, see if we have any work pending and only go back to sleep if that is not the case.

I've tested this patch for about a week now, by applying it to the 2.6.17-rc3 kernel on my laptop, which I've been using for more than a month now. This patch seems to cure the mysterious symptoms reported in February:

http://lkml.org/lkml/2006/2/6/167
http://lkml.org/lkml/2006/2/6/170
http://lkml.org/lkml/2006/2/13/424
etc.

Quoted 6 lines

Actually I didn't remember to check "Dirty:" in /proc/meminfo, but when I "sync"ed at the end of my workday, just prior to swsupending it, sync returned immediately. with unpatched 2.6.17-rc3, sync would take half a minute. Maybe Mark can give this patch a spin to check if it cures his problem, too. (I still use vmware, so vmware was not the culprit.)

I just gave it a try here. With or without a suspend/resume cycle after boot, the "sync" time is much quicker. But the Dirty count in /proc/meminfo still shows very huge (eg. 600MB) values that never really get smaller until I type "sync". But that subsequent "sync" only takes a couple of seconds now, rather than 10-20 seconds like before.

Dunno what that all means -- I'm still keeping my little daemon around to do periodic "sync" calls for safety.

Cheers

#9 lore raw
Quoted 35 lines · Mark Lord
Quoted 34 lines · Johannes Stezenbach
Quoted 11 lines · Andrew Morton

From: Andrew Morton <[email protected]>

pdflush is carefully designed to ensure that all wakeups have some corresponding work to do - if a woken-up pdflush thread discovers that it hasn't been given any work to do then this is considered an error.

That all broke when swsusp came along - because a timer-delivered wakeup to a frozen pdflush thread will just get lost. This causes the pdflush thread to get lost as well: the writeback timer is supposed to be re-armed by pdflush in process context, but pdflush doesn't execute the callout which does this.

Quoted 6 lines

Fix that up by ignoring the return value from try_to_freeze(): jsut proceed, see if we have any work pending and only go back to sleep if that is not the case.

I've tested this patch for about a week now, by applying it to the 2.6.17-rc3 kernel on my laptop, which I've been using for more than a month now. This patch seems to cure the mysterious symptoms reported in February:

http://lkml.org/lkml/2006/2/6/167
http://lkml.org/lkml/2006/2/6/170
http://lkml.org/lkml/2006/2/13/424
etc.

Actually I didn't remember to check "Dirty:" in /proc/meminfo, but when I "sync"ed at the end of my workday, just prior to swsupending it, sync returned immediately. with unpatched 2.6.17-rc3, sync would take half a minute

...

Quoted 5 lines

I just gave it a try here. With or without a suspend/resume cycle after boot, the "sync" time is much quicker. But the Dirty count in /proc/meminfo still shows very huge (eg. 600MB) values that never really get smaller until I type "sync". But that subsequent "sync" only takes a couple of seconds now, rather than 10-20 seconds like before.

..

Yup, behaviour is *definitely* much better now. I'm not sure why the /proc/meminfo "Dirty" count lags behind reality, but the disk is being kept much more up-to-date than without this patch.

Thanks!

#10 lore raw
Quoted 11 lines · On Tue, 20 Jun 2006 23:38:57 -0400 Mark Lord <[email protected]> wrote:
Quoted 6 lines

I just gave it a try here. With or without a suspend/resume cycle after boot,
the "sync" time is much quicker. But the Dirty count in /proc/meminfo still shows very huge (eg. 600MB) values that never really get smaller until I type "sync". But that subsequent "sync" only takes a couple of seconds now, rather than 10-20 seconds like before.

..

Yup, behaviour is *definitely* much better now. I'm not sure why the /proc/meminfo "Dirty" count lags behind reality, but the disk is being kept much more up-to-date than without this patch.

Are you able to come up with a foolproof set of steps which would allow the laggy-dirtiness to be reproduced by yours truly?

#11 lore raw
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Tue, 20 Jun 2006 23:38:57 -0400 Mark Lord <[email protected]> wrote:
Quoted 5 lines

I just gave it a try here. With or without a suspend/resume cycle after boot, the "sync" time is much quicker. But the Dirty count in /proc/meminfo still shows very huge (eg. 600MB) values that never really get smaller until I type "sync". But that subsequent "sync" only takes a couple of seconds now, rather than 10-20 seconds like before.

Quoted 8 lines
Quoted 5 lines

..

Yup, behaviour is *definitely* much better now. I'm not sure why the /proc/meminfo "Dirty" count lags behind reality, but the disk is being kept much more up-to-date than without this patch.

Are you able to come up with a foolproof set of steps which would allow the laggy-dirtiness to be reproduced by yours truly?

Heh.. don't I wish!

The best is still as described originally:

http://lkml.org/lkml/2006/2/6/170

Basically, "cat" a ton of huge files together into a single new one, and then watch /proc/meminfo to see what happens. For me, the count there still just hangs at some big number like 500MB until I type "sync", at which point it (nearly) instantly now goes to zero.

Previous to this patch, the "sync" actually resulted in a ton of disk writes, but now those happen on the tail end of the "cat" command, as they should.

My kernel .config is available from http://rtr.ca/dell_i9300/kernel/latest/

Cheers

#12 lore raw
Quoted 30 lines · On Wed, 21 Jun 2006 00:10:55 -0400 Mark Lord <[email protected]> wrote:
Quoted 9 lines · Andrew Morton
Quoted 6 lines · On Tue, 20 Jun 2006 23:38:57 -0400 Mark Lord <[email protected]> wrote:
Quoted 6 lines

I just gave it a try here. With or without a suspend/resume cycle after boot,
the "sync" time is much quicker. But the Dirty count in /proc/meminfo still shows very huge (eg. 600MB) values that never really get smaller until I type "sync". But that subsequent "sync" only takes a couple of seconds now, rather than 10-20 seconds like before.

Quoted 8 lines
Quoted 5 lines

..

Yup, behaviour is *definitely* much better now. I'm not sure why the /proc/meminfo "Dirty" count lags behind reality, but the disk is being kept much more up-to-date than without this patch.

Are you able to come up with a foolproof set of steps which would allow the laggy-dirtiness to be reproduced by yours truly?

Heh.. don't I wish!

The best is still as described originally:

http://lkml.org/lkml/2006/2/6/170

Basically, "cat" a ton of huge files together into a single new one, and then watch /proc/meminfo to see what happens. For me, the count there still just hangs at some big number like 500MB until I type "sync", at which point it (nearly) instantly now goes to zero.

Quoted 4 lines

Previous to this patch, the "sync" actually resulted in a ton of disk writes, but now those happen on the tail end of the "cat" command, as they should.

My kernel .config is available from http://rtr.ca/dell_i9300/kernel/latest/

Is that after a suspend/resume, or does it happen after a reboot?

Are you sure all the dirty memory doesn't get autocleaned after 30-60 seconds?

#13 lore raw
Quoted 2 lines · Andrew Morton

Is that after a suspend/resume, or does it happen after a reboot?

Definitely after a suspend/resume (RAM).
It's been too long for me to remember about after a reboot.

Quoted 2 lines

Are you sure all the dirty memory doesn't get autocleaned after 30-60 seconds?

No, it does not get autocleaned (without this patch) after 30-60 *minutes*, let alone 30-60 seconds. Nor with or without laptop-mode enabled/disabled. Nor anything else suggested. Except for this patch.

Cheers