Skip to contentSkip to conversation

Good way to free as much memory as possible under 2.5.34?

DISCUSSION Pavel Machek 7 participants 15 messages Sep 13, 2002 – Sep 14, 2002

lore
Share & tools
lore
#0 lore raw

Hi!

/*
* Try to free as much memory as possible, but do not OOM-kill anyone *
* Notice: all userland should be stopped at this point, or livelock
is possible.
*/

This worked before -rmap came in, but it does not free anything now. What needs to be done to fix it?

Pavel

static void free_some_memory(void)

{
        printk("Freeing memory: ");
        while
(try_to_free_pages(&contig_page_data.node_zones[ZONE_HIGHMEM], GFP_KSWAPD, 0))
                printk(".");
        printk("|\n");
}
Signature · 2 lines
Worst form of spam? Adding advertisment signatures ala sourceforge.net.
What goes next? Inserting advertisment *into* email?
#1 lore raw
Quoted 9 lines · Pavel Machek

/*
* Try to free as much memory as possible, but do not OOM-kill anyone *
* Notice: all userland should be stopped at this point, or livelock
is possible.
*/

This worked before -rmap came in, but it does not free anything now. What needs to be done to fix it?

Actually, it still worked when -rmap came in, but it stopped working when the LRU lists were made to be per-zone...

Quoted 8 lines

static void free_some_memory(void)

{
        printk("Freeing memory: ");
        while
(try_to_free_pages(&contig_page_data.node_zones[ZONE_HIGHMEM], GFP_KSWAPD, 0))
                printk(".");
        printk("|\n");
}

Why don't you just allocate memory ?

To prevent the OOM kill you can just check for a variable in the OOM slow path. No need to rely on any particular behaviour of the VM.

regards,

Rik

Signature · 5 lines
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/

Spamtraps of the month:  [email protected] [email protected]
#2 lore raw

Hi!

Quoted 22 lines
Quoted 9 lines

/*
* Try to free as much memory as possible, but do not OOM-kill anyone *
* Notice: all userland should be stopped at this point, or livelock
is possible.
*/

This worked before -rmap came in, but it does not free anything now. What needs to be done to fix it?

Actually, it still worked when -rmap came in, but it stopped working when the LRU lists were made to be per-zone...

Quoted 8 lines

static void free_some_memory(void)

{
        printk("Freeing memory: ");
        while
(try_to_free_pages(&contig_page_data.node_zones[ZONE_HIGHMEM], GFP_KSWAPD, 0))
                printk(".");
        printk("|\n");
}
Quoted 5 lines

Why don't you just allocate memory ?

To prevent the OOM kill you can just check for a variable in the OOM slow path. No need to rely on any particular behaviour of the VM.

Allocating memory is pain because I have to free it afterwards. Yep I have such code, but it is ugly. try_to_free_pages() really seems like cleaner solution to me... if you only tell me how to fix it :-).

If it is not easy to fix, I can reintroduce memory eating code; but I'd hate to special-case SWSUSP in VM. Otoh GFP_FAIL or something like that might help me, right?
Pavel

Signature · 2 lines
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.
#3 lore raw
Quoted 3 lines · Pavel Machek

Allocating memory is pain because I have to free it afterwards. Yep I have such code, but it is ugly. try_to_free_pages() really seems like cleaner solution to me... if you only tell me how to fix it :-).

"Fixing" the VM just so it behaves the way swsuspend wants is out. If swsuspend relies on all other subsystems playing nicely, I think it should be removed from the kernel.

I suspect only very few people will use swsuspend, so it should not be intrusive.

regards,

Rik

Signature · 5 lines
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/

Spamtraps of the month:  [email protected] [email protected]
#4 lore raw
Quoted 2 lines · Rik van Riel

I suspect only very few people will use swsuspend, so it should not be intrusive.

I don't think so.

#5 lore raw
Quoted 6 lines · Xavier Bestel
Quoted 2 lines · Rik van Riel

I suspect only very few people will use swsuspend, so it should not be intrusive.

I don't think so.

You think many people will use it, or you think swsuspend should be intrusive and have code in all other kernel subsystems ? ;)

Rik

Signature · 5 lines
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/

Spamtraps of the month:  [email protected] [email protected]
#6 lore raw
Quoted 11 lines · Rik van Riel
Quoted 6 lines · Xavier Bestel
Quoted 2 lines · Rik van Riel

I suspect only very few people will use swsuspend, so it should not be intrusive.

I don't think so.

You think many people will use it, or you think swsuspend should be intrusive and have code in all other kernel subsystems ? ;)

I think people will use it. Either for laptops (when ACPI support will match APM) or for desktop. Personnaly I'd use it for both if it were functionnal.

#7 lore raw
In reply to Rik van Riel
Quoted 11 lines · Rik van Riel
Quoted 3 lines · Pavel Machek

Allocating memory is pain because I have to free it afterwards. Yep I have such code, but it is ugly. try_to_free_pages() really seems like cleaner solution to me... if you only tell me how to fix it :-).

"Fixing" the VM just so it behaves the way swsuspend wants is out. If swsuspend relies on all other subsystems playing nicely, I think it should be removed from the kernel.

Yup. Martin Bligh is cooking up a multi-page allocation API, so when that's in place, swsusp need only do:

	LIST_HEAD(foo);
	alloc_many_pages(&foo, nr_pages, __GFP_HIGHMEM|__GFP_WAIT);
	free_many_pages(&foo);

So I suggest you do something local for the while, plan to use that later.

(Actually, the implementation would probably have a heart attack if you asked for 100,000 pages so you may need to sit in a loop there; we'll see).

#8 lore raw
Quoted 3 lines · Andrew Morton

So I suggest you do something local for the while, plan to use that later. (Actually, the implementation would probably have a heart attack if you asked for 100,000 pages so you may need to sit in a loop there; we'll see).

Actually, that's probably going to trip the NMI oopser.

Cheers,
Bill

#9 lore raw
In reply to Andrew Morton

Hi!

Quoted 16 lines
Quoted 8 lines
Quoted 3 lines

Allocating memory is pain because I have to free it afterwards. Yep I have such code, but it is ugly. try_to_free_pages() really seems like cleaner solution to me... if you only tell me how to fix it :-).

"Fixing" the VM just so it behaves the way swsuspend wants is out. If swsuspend relies on all other subsystems playing nicely, I think it should be removed from the kernel.

Yup. Martin Bligh is cooking up a multi-page allocation API, so when that's in place, swsusp need only do:

	LIST_HEAD(foo);
	alloc_many_pages(&foo, nr_pages, __GFP_HIGHMEM|__GFP_WAIT);
	free_many_pages(&foo);
Quoted 5 lines

So I suggest you do something local for the while, plan to use that later.

(Actually, the implementation would probably have a heart attack if you asked for 100,000 pages so you may need to sit in a loop there; we'll see).

If nr_pages is > than number of pages really freable will it return NULL or stall the calling process forever?

If it will return NULL I'm happy to use that... Otherwise its not too usable because I do not want to OOM the machine.
Pavel

Signature · 2 lines
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.
#10 lore raw
Quoted 19 lines · Pavel Machek

Hi!

Quoted 16 lines
Quoted 8 lines
Quoted 3 lines

Allocating memory is pain because I have to free it afterwards. Yep I have such code, but it is ugly. try_to_free_pages() really seems like cleaner solution to me... if you only tell me how to fix it :-).

"Fixing" the VM just so it behaves the way swsuspend wants is out. If swsuspend relies on all other subsystems playing nicely, I think it should be removed from the kernel.

Yup. Martin Bligh is cooking up a multi-page allocation API, so when that's in place, swsusp need only do:

      LIST_HEAD(foo);
      alloc_many_pages(&foo, nr_pages, __GFP_HIGHMEM|__GFP_WAIT);
      free_many_pages(&foo);
Quoted 8 lines
Quoted 5 lines

So I suggest you do something local for the while, plan to use that later.

(Actually, the implementation would probably have a heart attack if you asked for 100,000 pages so you may need to sit in a loop there; we'll see).

If nr_pages is > than number of pages really freable will it return NULL or stall the calling process forever?

Good point. In current Linus tree, it could be oom-killed. Later, lack of __GFP_FS will prevent that. But it could loop forever. I guess you'll need to drop the __GFP_WAIT and yield yourself, let kswapd free the next batch of pages.

We need to do something more robust than this. I'll cook something up.

#11 lore raw
In reply to Rik van Riel

Hi Rik,

Quoted 11 lines · Rik van Riel
Quoted 3 lines · Pavel Machek

Allocating memory is pain because I have to free it afterwards. Yep I have such code, but it is ugly. try_to_free_pages() really seems like cleaner solution to me... if you only tell me how to fix it :-).

"Fixing" the VM just so it behaves the way swsuspend wants is out. If swsuspend relies on all other subsystems playing nicely, I think it should be removed from the kernel.

I suspect only very few people will use swsuspend, so it should not be intrusive.

By now, it's only used by a minority (those, who get it going reliably), but I bet, things change, when 2.6 is out. I would love and even celebrate once swsuspend is working via nbd for my diskless setups. I consider this as a real quantum leap (from a usability/energy saving point of view).

The question is: why is the VM not able to fulfill such a simple need in a clean way?

Quoted 3 lines

regards,

Rik

Regards,
Hans-Peter

#12 lore raw
Quoted 2 lines · Hans-Peter Jansen

The question is: why is the VM not able to fulfill such a simple need in a clean way?

No. The question is: "why does swsuspend need to stick its fingers into every other subsystem of the kernel, instead of trying to remain vaguely modular ?"

If you have an answer, please let me know.

Rik

Signature · 5 lines
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/

Spamtraps of the month:  [email protected] [email protected]
#13 lore raw

Hi!

Quoted 6 lines
Quoted 2 lines

The question is: why is the VM not able to fulfill such a simple need in a clean way?

No. The question is: "why does swsuspend need to stick its fingers into every other subsystem of the kernel, instead of trying to remain vaguely modular ?"

I'm not trying to stick swsusp fingers anywhere. It was suggested to me to hack VM and I'm trying to avoid that.
Pavel

Signature · 2 lines
Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.
#14 lore raw
In reply to Rik van Riel
Quoted 10 lines · Rik van Riel
Quoted 7 lines · Pavel Machek

/*
* Try to free as much memory as possible, but do not OOM-kill anyone *
* Notice: all userland should be stopped at this point, or livelock
is possible.
*/

Quoted 5 lines
Quoted 2 lines

This worked before -rmap came in, but it does not free anything now. What needs to be done to fix it?

Actually, it still worked when -rmap came in, but it stopped working when the LRU lists were made to be per-zone...

hmm, I missed that. Yes, the zone balancing in try_to_free_pages() will see that all zones are above ->pages_high and will just return.

Quoted 10 lines
Quoted 8 lines

static void free_some_memory(void)

{
        printk("Freeing memory: ");
        while
(try_to_free_pages(&contig_page_data.node_zones[ZONE_HIGHMEM], GFP_KSWAPD, 0))
                printk(".");
        printk("|\n");
}

Why don't you just allocate memory ?

That would work.

Quoted 4 lines

To prevent the OOM kill you can just check for a variable in the OOM slow path. No need to rely on any particular behaviour of the VM.

Sure.

I'm not sure why swsusp needs "half of memory to be free"? What's the story there?

I'd recommend that you sit in a loop, allocating pages with an allocation mode of

	__GFP_HIGHMEM | __GFP_WAIT

This will give you all the easily-reclaimable pages in the machine, without trashing the page reserves (no __GFP_HIGH).

String all the pages together via page->list and when you have "enough", free them all again.