2.6.22 -mm merge plans

Quoted 1 line · Andrew Morton

proper-prototype-for-hugetlb_get_unmapped_area.patch

...

Quoted 1 line

convert-hugetlbfs-to-use-vm_ops-fault.patch

...

Quoted 1 line

get_unmapped_area-handles-map_fixed-in-hugetlbfs.patch

...

Quoted 1 line

get_unmapped_area-doesnt-need-hugetlbfs-hacks-anymore.patch

...

Quoted 1 line

Will merge.

I've gone over these again and all are still good. The same holds for the get_unmapped_area() series in general where I've reviewed it for hugetlb relevance.

-- wli

Neil Brown

2007-05-01 00:09 UTC #15 lore raw

nfsd/md patches Re: 2.6.22 -mm merge plans

Quoted 9 lines · On Monday April 30, [email protected] wrote:

remove-nfs4_acl_add_ace.patch the-nfsv2-nfsv3-server-does-not-handle-zero-length-write.patch knfsd-rename-sk_defer_lock-to-sk_lock.patch nfsd-nfs4state-remove-unnecessary-daemonize-call.patch rpc-add-wrapper-for-svc_reserve-to-account-for-checksum.patch
nfsd things - will merge after checking with Neil.

All acked, though that last one won't fix any oopses like the comment hopes for - I really should look into that.

Quoted 6 lines

drivers-mdc-use-array_size-macro-when-appropriate.patch md-cleanup-use-seq_release_private-where-appropriate.patch md-remove-broken-sigkill-support.patch
Will merge after checking with Neil

NAK on md-remove-broken-sigkill-support.patch - I'll follow up the original mail.

ACK on the other two.

Quoted 5 lines

md-dm-reduce-stack-usage-with-stacked-block-devices.patch
Will we ever fix this?

I think we have several votes for "just merge it". I don't think there are known problems with it.

NeilBrown

Christoph Hellwig

2007-05-01 09:08 UTC #16 lore raw

Re: nfsd/md patches Re: 2.6.22 -mm merge plans

apropos nfsd patches, what's the merge plans for my two export ops patch series?

Andrew Morton

author maintainer

2007-05-01 09:15 UTC #17 lore raw

Re: nfsd/md patches Re: 2.6.22 -mm merge plans

Quoted 2 lines · On Tue, 1 May 2007 10:08:43 +0100 Christoph Hellwig <[email protected]> wrote:

apropos nfsd patches, what's the merge plans for my two export ops patch series?

box:/usr/src/25/patches> grep -l '^From:.*hch' $(cat-series ../series ) dvb_en_50221-convert-to-kthread-api.patch simplify-the-stacktrace-code.patch vfs-remove-superflous-sb-==-null-checks.patch nameic-remove-utterly-outdated-comment.patch move-die-notifier-handling-to-common-code.patch merge-compat_ioctlh-into-compat_ioctlc.patch cleanup-compat-ioctl-handling.patch kprobes-use-hlist_for_each_entry.patch kprobes-codingstyle-cleanups.patch kprobes-kretprobes-simplifcations.patch

I give up. Where are they hiding?

Christoph Hellwig

2007-05-01 09:21 UTC #18 lore raw

Re: nfsd/md patches Re: 2.6.22 -mm merge plans

Quoted 4 lines · Andrew Morton

Quoted 2 lines · On Tue, 1 May 2007 10:08:43 +0100 Christoph Hellwig <[email protected]> wrote:
apropos nfsd patches, what's the merge plans for my two export ops patch series?

This question was directed to Neil, sorry.

Quoted 12 lines

box:/usr/src/25/patches> grep -l '^From:.*hch' $(cat-series ../series ) dvb_en_50221-convert-to-kthread-api.patch simplify-the-stacktrace-code.patch vfs-remove-superflous-sb-==-null-checks.patch nameic-remove-utterly-outdated-comment.patch move-die-notifier-handling-to-common-code.patch merge-compat_ioctlh-into-compat_ioctlc.patch cleanup-compat-ioctl-handling.patch kprobes-use-hlist_for_each_entry.patch kprobes-codingstyle-cleanups.patch kprobes-kretprobes-simplifcations.patch

Quoted 1 line

I give up. Where are they hiding?

Good question :) I sent them to the nfs list.

Neil Brown

2007-05-01 09:52 UTC #19 lore raw

Re: nfsd/md patches Re: 2.6.22 -mm merge plans

Quoted 2 lines · On Tuesday May 1, [email protected] wrote:

apropos nfsd patches, what's the merge plans for my two export ops patch series?

Still sitting in my tree - I've had my mind on other things (nfs-utils, portmap....) and let them slip - sorry.

I think also there was an unanswered question about the second series (there first I am completely happy with).

Quoted 4 lines

Date: Fri, 30 Mar 2007 13:34:53 +1000
My only question involves motivation.

Quoted 9 lines

You say "less complex", but to me it just looks "different" - though being very familiar with the original, that might be a biased view. Can you say more about how it is less complex?
Maybe the extension to generic 64bit support will make that clear...
But then generic 64bit support should just be an independent set of helper functions that can be plugged in to the export_operations structure.

It think I programmed myself to use a reply to that to be my wake_up to forwarded them on, and forgot to register a timeout handler....

NeilBrown

Christoph Hellwig

2007-05-01 10:15 UTC #20 lore raw

Re: nfsd/md patches Re: 2.6.22 -mm merge plans

Quoted 9 lines · Neil Brown

Quoted 2 lines · On Tuesday May 1, [email protected] wrote:
apropos nfsd patches, what's the merge plans for my two export ops patch series?
Still sitting in my tree - I've had my mind on other things (nfs-utils, portmap....) and let them slip - sorry.
I think also there was an unanswered question about the second series (there first I am completely happy with).

A sorry, this mail got somewhere lost. I'll reply on the nfs list because we have a little more context there. (and due to the subscribers only policy I can't crosspost unfortunately)

Trond Myklebust

2007-05-01 14:29 UTC #21 lore raw

Re: nfsd/md patches Re: 2.6.22 -mm merge plans

Quoted 8 lines · Christoph Hellwig

Quoted 7 lines · Neil Brown
Quoted 2 lines · On Tuesday May 1, [email protected] wrote:
apropos nfsd patches, what's the merge plans for my two export ops patch series?
Still sitting in my tree - I've had my mind on other things (nfs-utils, portmap....) and let them slip - sorry.

Quoted 6 lines

Quoted 2 lines
I think also there was an unanswered question about the second series (there first I am completely happy with).
A sorry, this mail got somewhere lost. I'll reply on the nfs list because we have a little more context there. (and due to the subscribers only policy I can't crosspost unfortunately)

I though we lifted the subscribers only policy quite a while back. There should be nothing preventing you from cross-posting.

Cheers
Trond

Rik van Riel

2007-05-01 00:54 UTC #22 lore raw

MADV_FREE functionality

Quoted 10 lines · Andrew Morton

lazy-freeing-of-memory-through-madv_free.patch lazy-freeing-of-memory-through-madv_free-vs-mm-madvise-avoid-exclusive-mmap_sem.patch restore-madv_dontneed-to-its-original-linux-behaviour.patch
I think the MADV_FREE changes need more work:
We need crystal-clear statements regarding the present functionality, the new functionality and how these relate to the spec and to implmentations in other OS'es. Once we have that info we are in a position to work out whether the code can be merged as-is, or if additional changes are needed.

There are two MADV variants that free pages, both do the exact same thing with mapped file pages, but both do something slightly different with anonymous pages.

MADV_DONTNEED will unmap file pages and free anonymous pages. When a process accesses anonymous memory at an address that was zapped with MADV_DONTNEED, it will return fresh zero filled pages.

MADV_FREE will unmap file pages. MADV_FREE on anonymous pages is interpreted as a signal that the application no longer needs the data in the pages, and they can be thrown away if the kernel needs the memory for something else. However, if the process accesses the memory again before the kernel needs it, the process will simply get the original pages back. If the kernel needed the memory first, the process will get a fresh zero filled page like with MADV_DONTNEED.

In short:
- both MADV_FREE and MADV_DONTNEED only unmap file pages
- after MADV_DONTNEED the application will always get back fresh zero filled anonymous pages when accessing the memory
- after MADV_FREE the application can either get back the original data (without a page fault) or zero filled anonymous memory

The Linux MADV_DONTNEED behavior is not POSIX compliant. POSIX says that with MADV_DONTNEED the application's data will be preserved.

Currently glibc simply ignores POSIX_MADV_DONTNEED requests from applications on Linux. Changing the behaviour which some Linux applications may rely on might not be the best idea.

If you want POSIX_MADV_DONTNEED behaviour added, please let me know and I'll whip up a patch.

Quoted 3 lines

Because right now, I don't know where we are with respect to these things and I doubt if many of our users know either. How can Michael write a manpage for this is we don't tell him what it all does?

If you need any additional information, please let me know.

If you still think the MADV_FREE patches themselves should not be merged yet, can we at least merge the #defines, so the Fedora kernel can get the MADV_FREE functionality?

Again, I'd be more than willing to whip up a patch for that.

Signature · 9 lines

Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andrew Morton

author maintainer

2007-05-01 01:18 UTC #23 lore raw

Re: MADV_FREE functionality

Quoted 40 lines · On Mon, 30 Apr 2007 20:54:02 -0400 Rik van Riel <[email protected]> wrote:

Quoted 10 lines · Andrew Morton
lazy-freeing-of-memory-through-madv_free.patch lazy-freeing-of-memory-through-madv_free-vs-mm-madvise-avoid-exclusive-mmap_sem.patch restore-madv_dontneed-to-its-original-linux-behaviour.patch
I think the MADV_FREE changes need more work:
We need crystal-clear statements regarding the present functionality, the new functionality and how these relate to the spec and to implmentations in other OS'es. Once we have that info we are in a position to work out whether the code can be merged as-is, or if additional changes are needed.
There are two MADV variants that free pages, both do the exact same thing with mapped file pages, but both do something slightly different with anonymous pages.
MADV_DONTNEED will unmap file pages and free anonymous pages. When a process accesses anonymous memory at an address that was zapped with MADV_DONTNEED, it will return fresh zero filled pages.
MADV_FREE will unmap file pages. MADV_FREE on anonymous pages is interpreted as a signal that the application no longer needs the data in the pages, and they can be thrown away if the kernel needs the memory for something else. However, if the process accesses the memory again before the kernel needs it, the process will simply get the original pages back. If the kernel needed the memory first, the process will get a fresh zero filled page like with MADV_DONTNEED.
In short:
- both MADV_FREE and MADV_DONTNEED only unmap file pages
- after MADV_DONTNEED the application will always get back fresh zero filled anonymous pages when accessing the memory
- after MADV_FREE the application can either get back the original data (without a page fault) or zero filled anonymous memory

Quoted 8 lines

The Linux MADV_DONTNEED behavior is not POSIX compliant. POSIX says that with MADV_DONTNEED the application's data will be preserved.
Currently glibc simply ignores POSIX_MADV_DONTNEED requests from applications on Linux. Changing the behaviour which some Linux applications may rely on might not be the best idea.

OK, thanks. I stuck that in the changelog.

Michael, do you think that's enough to finalise a manpage?

Quoted 1 line

If you need any additional information, please let me know.

The patch doesn't update the various comments in madvise.c at all, which is a surprise. Could you please check that they are all accurate and complete?

Also, where did we end up with the Solaris compatibility?

The patch I have at present retains MADV_FREE=0x05 for sparc and sparc64 which should be good.

Did we decide that the Solaris and Linux implementations of MADV_FREE are compatible?

What about the Solaris and Linux MADV_DONTNEED implementations?

Rik van Riel

2007-05-01 01:23 UTC #24 lore raw

Re: MADV_FREE functionality

Quoted 4 lines · Andrew Morton

Quoted 1 line
If you need any additional information, please let me know.
The patch doesn't update the various comments in madvise.c at all, which is a surprise. Could you please check that they are all accurate and complete?

I'll take a look.

Quoted 7 lines

Also, where did we end up with the Solaris compatibility?
The patch I have at present retains MADV_FREE=0x05 for sparc and sparc64 which should be good.
Did we decide that the Solaris and Linux implementations of MADV_FREE are compatible?

Yes, the Linux, Solaris and FreeBSD implementations of MADV_FREE appear to have equivalent semantics.

Quoted 1 line

What about the Solaris and Linux MADV_DONTNEED implementations?

This was never, and is still not, the same. Linux will throw away the data in anonymous pages while POSIX says we should simply move the data to swap. I assume Solaris and FreeBSD will move the data to swap instead of throwing it away.

For file backed pages I suspect they all behave the same.

This is the reason that inside glibc, POSIX_MADV_DONTNEED is a noop.

Signature · 9 lines

Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Jakub Jelinek

2007-05-01 07:13 UTC #25 lore raw

Re: MADV_FREE functionality

Quoted 13 lines · Andrew Morton

Quoted 13 lines
In short:
- both MADV_FREE and MADV_DONTNEED only unmap file pages
- after MADV_DONTNEED the application will always get back fresh zero filled anonymous pages when accessing the memory
- after MADV_FREE the application can either get back the original data (without a page fault) or zero filled anonymous memory
The Linux MADV_DONTNEED behavior is not POSIX compliant. POSIX says that with MADV_DONTNEED the application's data will be preserved.

Quoted 6 lines

Quoted 4 lines
Currently glibc simply ignores POSIX_MADV_DONTNEED requests from applications on Linux. Changing the behaviour which some Linux applications may rely on might not be the best idea.
OK, thanks. I stuck that in the changelog.

FYI, Solaris man page on MADV_FREE says:

MADV_FREE
Tells the kernel that contents in the specified address range are no longer important and the range will be overwritten. When there is demand for memory, the system will free pages associated with the speci- fied address range. In this instance, the next time a page in the address range is referenced, it will con- tail all zeroes. Otherwise, it will contain the data that was there prior to the MADV_FREE call. References made to the address range will not make the system read from backing store (swap space) until the page is modified again.

This value cannot be used on mappings that have under- lying file objects.

The last paragraph seems to be just about the operation being undefined, madvise MADV_FREE on MAP_SHARED file mapping returns 0 rather than flagging an error.

FreeBSD man page:

MADV_FREE Gives the VM system the freedom to free pages, and tells
the system that information in the specified page range is no longer important. This is an efficient way of allowing malloc(3) to free pages anywhere in the address space, while keeping the address space valid. The next time that the page is referenced, the page might be demand zeroed, or might contain the data that was there before the MADV_FREE call. References made to that address space range will not make the VM system page the information back in from backing store until the page is modified again.

Quoted 7 lines

Also, where did we end up with the Solaris compatibility?
The patch I have at present retains MADV_FREE=0x05 for sparc and sparc64 which should be good.
Did we decide that the Solaris and Linux implementations of MADV_FREE are compatible?

SPARC Solaris binary compatibility in Linux is in really bad shape, madvise in Solaris is implemented using memcntl syscall (at least according to truss(1)) and that syscall is
systbl.S: .word solaris_unimplemented /* memcntl 131 */
When/if anyone decides to put more effort into the Solaris binary compatibility (I'm quite doubtful anyone will), codes which don't match can be simply translated into other codes, ignored etc., we can't use sys_madvise to implement memcntl syscall anyway. While Solaris MADV_FREE is the same as Linux MADV_FREE proposed by Rik (except perhaps the documented undefined behavior with file mappings, on
#include <sys/mman.h>
#include <unistd.h>
#include <fcntl.h>

int
main (void)
{
getpid ();
int fd = open ("test", O_RDWR);
void *p = mmap (0, 8192, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); memset (p, ' ', 8192);
madvise (p, 8192, MADV_FREE);
return 0;
}
on Solaris the spaces actually made it into the file), MADV_DONTNEED is not, but that doesn't really matter except for arch/sparc*/solaris/ layer if anyone cares. We certainly can't change current MADV_DONTNEED behavior, all we can do is implement a new MADV_* code with a different behavior and let glibc translate POSIX_MADV_* codes on posix_madvise to the Linux specific MADV_*.

Jakub

Ulrich Drepper

2007-05-01 01:23 UTC #26 lore raw

Re: MADV_FREE functionality

In reply to Rik van Riel

Quoted 4 lines · Rik van Riel

Quoted 3 lines · Andrew Morton
Because right now, I don't know where we are with respect to these things and I doubt if many of our users know either. How can Michael write a manpage for this is we don't tell him what it all does?

I think we've been very clear before and Rik's description here puts it all nicely in one place. If you're worried about semantics you can rest assured, it is all sound. If this is what is holding up the patch then add it to your collection. Only if you have technical objections should you hold it off. The patch makes sense (and has been validated by being implemented in the same way on other OSes) and it is really needed.

Stefan Richter

2007-05-01 01:39 UTC #27 lore raw

Quoted 4 lines · Andrew Morton

sbp2-include-fixes.patch
ieee1394-iso-needs-schedh.patch
For Stephan

They were merged some hours ago.

Signature · 9 lines

Stefan Richter
-=====-=-=== -=-= ----=
http://arcgraph.de/sr/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dmitry Torokhov

2007-05-01 02:30 UTC #28 lore raw

Re: 2.6.22 -mm merge plans (RE: input)

Quoted 10 lines · Andrew Morton

input-convert-from-class-devices-to-standard-devices.patch input-evdev-implement-proper-locking.patch mousedev-fix.patch
mousedev-fix-2.patch
Dmitry will merge these once Greg has merged the preparatory work. Except these patches make the Vaio-of-doom crash in obscure circumstances, and we weren't able to fix that?

Would like to keep cooking in your tree till we get your Vaio going, if you don't mind.

Quoted 1 line

wistron_btns-add-led-support.patch

Will review once again and apply.

Quoted 3 lines

input-ff-add-ff_raw-effect.patch
input-phantom-add-a-new-driver.patch

It looks like Phanotom will not be using input layer...

Quoted 4 lines

input-rfkill-add-support-for-input-key-to-control-wireless-radio.patch
Will resend to davem once the preparatory bits are merged by Greg.

You mean me, right? I need to do some locking changes that DaveM pointed out.

Signature · 7 lines

Dmitry

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Jiri Slaby

2007-05-01 08:14 UTC #29 lore raw

Re: 2.6.22 -mm merge plans (RE: input)

Dmitry Torokhov napsal(a):

Quoted 6 lines · Andrew Morton

Quoted 3 lines
input-ff-add-ff_raw-effect.patch
input-phantom-add-a-new-driver.patch
It looks like Phanotom will not be using input layer...

Yes, I have a new version, planning to test it tomorrow when I reach the device and then post it.

You don't want it in input/misc in that case, right? If yes, Andrew, please drop both.

thanks,

Signature · 4 lines

http://www.fi.muni.cz/~xslaby/            Jiri Slaby
faculty of informatics, masaryk university, brno, cz
e-mail: jirislaby gmail com, gpg pubkey fingerprint:
B674 9967 0407 CE62 ACC8  22A0 32CC 55C3 39D4 7A7E

Dmitry Torokhov

2007-05-01 12:05 UTC #30 lore raw

Re: 2.6.22 -mm merge plans (RE: input)

Quoted 6 lines · Jiri Slaby

Dmitry Torokhov napsal(a):
Quoted 5 lines · Andrew Morton
Quoted 3 lines
input-ff-add-ff_raw-effect.patch
input-phantom-add-a-new-driver.patch

Quoted 6 lines

Quoted 1 line
It looks like Phanotom will not be using input layer...
Yes, I have a new version, planning to test it tomorrow when I reach the device and then post it.
You don't want it in input/misc in that case, right?

Correct. Input/misc is only visible if CONFIG_INPUT is selected. But if Phantom is not using input layer it should not depend on CONFIG_INPUT. I'd put it into drivers/misc...

Signature · 1 line

Dmitry

Andy Whitcroft

2007-05-01 08:11 UTC #31 lore raw

Re: 2.6.22 -mm merge plans -- pfn_valid_within

Quoted 8 lines · Andrew Morton

add-pfn_valid_within-helper-for-sub-max_order-hole-detection.patch anti-fragmentation-switch-over-to-pfn_valid_within.patch lumpy-move-to-using-pfn_valid_within.patch
More Mel things, and linkage between Mel-things and lumpy reclaim. It's here where the patch ordering gets into a mess and things won't improve if moveable-zones and lumpy-reclaim get deferred. Such a deferral would limit my ability to queue more MM changes for 2.6.23.

The first of these is really a cleanup and should slide into the stack before Mobility and Lumpy. The other two should then join their respective stacks anti-fragmentation-... to Mobility and lumpy-... to Lumpy. I would not expect them to increase linkage that way.

-apw

Andrew Morton

author maintainer

2007-05-01 08:19 UTC #32 lore raw

Re: 2.6.22 -mm merge plans -- pfn_valid_within

Quoted 11 lines · On Tue, 01 May 2007 09:11:18 +0100 Andy Whitcroft <[email protected]> wrote:

Quoted 8 lines · Andrew Morton
add-pfn_valid_within-helper-for-sub-max_order-hole-detection.patch anti-fragmentation-switch-over-to-pfn_valid_within.patch lumpy-move-to-using-pfn_valid_within.patch
More Mel things, and linkage between Mel-things and lumpy reclaim. It's here where the patch ordering gets into a mess and things won't improve if moveable-zones and lumpy-reclaim get deferred. Such a deferral would limit my ability to queue more MM changes for 2.6.23.

Quoted 5 lines

The first of these is really a cleanup and should slide into the stack before Mobility and Lumpy. The other two should then join their respective stacks anti-fragmentation-... to Mobility and lumpy-... to Lumpy. I would not expect them to increase linkage that way.

yup, that improved things a bit, thanks.

Christoph Hellwig

2007-05-01 08:42 UTC #33 lore raw

"partical" kthread conversion

In reply to Eric W. Biederman

Quoted 8 lines · Andrew Morton

macintosh-mediabay-convert-to-kthread-api.patch macintosh-adb-convert-to-the-kthread-api.patch macintosh-therm_pm72c-partially-convert-to-kthread-api.patch powerpc-pseries-rtasd-convert-to-kthread-api.patch powerpc-pseries-eeh-convert-to-kthread-api.patch
Will send to paulus (I already did - does Paul not handle the macintosh driver?)

Please don't send out the partial kthread conversions, as they're not that helpful. Depending on the way we'll let the API evolve a kthread_create/run not paired by a kthread_stop might be actually harmful.

Please only send along patches that are paired or always built in so that they don't require stopping at all.

Btw, many of the drivers above should probably go to benh.

There's probably a few more patches falling into this category, these were just the first one the stick into my eye.

Andrew Morton

author maintainer

2007-05-01 08:51 UTC #34 lore raw

Re: "partical" kthread conversion

Quoted 14 lines · On Tue, 1 May 2007 09:42:45 +0100 Christoph Hellwig <[email protected]> wrote:

Quoted 8 lines · Andrew Morton
macintosh-mediabay-convert-to-kthread-api.patch macintosh-adb-convert-to-the-kthread-api.patch macintosh-therm_pm72c-partially-convert-to-kthread-api.patch powerpc-pseries-rtasd-convert-to-kthread-api.patch powerpc-pseries-eeh-convert-to-kthread-api.patch
Will send to paulus (I already did - does Paul not handle the macintosh driver?)
Please don't send out the partial kthread conversions, as they're not that helpful. Depending on the way we'll let the API evolve a kthread_create/run not paired by a kthread_stop might be actually harmful.

Quoted 7 lines

Please only send along patches that are paired or always built in so that they don't require stopping at all.
Btw, many of the drivers above should probably go to benh.
There's probably a few more patches falling into this category, these were just the first one the stick into my eye.

Yes, I think I'll probably drop all of them - I've completely lost track of which ones are complete, which ones need more work, etc.

I might send ia64-sn-xpc-convert-to-use-kthread-api.patch+fixes off to Tony, as people put quite a bit of review and test effort into that one.

Dean Nelson

2007-05-02 14:01 UTC #35 lore raw

Re: "partical" kthread conversion

Quoted 21 lines · Andrew Morton

Quoted 19 lines · On Tue, 1 May 2007 09:42:45 +0100 Christoph Hellwig <[email protected]> wrote:
Quoted 8 lines · Andrew Morton
macintosh-mediabay-convert-to-kthread-api.patch macintosh-adb-convert-to-the-kthread-api.patch macintosh-therm_pm72c-partially-convert-to-kthread-api.patch powerpc-pseries-rtasd-convert-to-kthread-api.patch powerpc-pseries-eeh-convert-to-kthread-api.patch
Will send to paulus (I already did - does Paul not handle the macintosh driver?)
Please don't send out the partial kthread conversions, as they're not that helpful. Depending on the way we'll let the API evolve a kthread_create/run not paired by a kthread_stop might be actually harmful.
Please only send along patches that are paired or always built in so that they don't require stopping at all.
Btw, many of the drivers above should probably go to benh.

Quoted 8 lines

Quoted 2 lines
There's probably a few more patches falling into this category, these were just the first one the stick into my eye.
Yes, I think I'll probably drop all of them - I've completely lost track of which ones are complete, which ones need more work, etc.
I might send ia64-sn-xpc-convert-to-use-kthread-api.patch+fixes off to Tony, as people put quite a bit of review and test effort into that one.

Andrew, I would recommend holding off on sending these xpc patches to Tony as the kthread_run()s aren't paired with kthread_stop()s yet. I need to generate an additional patch after I've first sorted out how best to deal with kthread_stop()'ng XPC's pool of kthreads with Eric.

Thanks,
Dean

Eric W. Biederman

2007-05-02 14:46 UTC #36 lore raw

Re: "partical" kthread conversion

Quoted 6 lines · Dean Nelson

Quoted 5 lines · Andrew Morton
Quoted 2 lines
There's probably a few more patches falling into this category, these were just the first one the stick into my eye.
Yes, I think I'll probably drop all of them - I've completely lost track of which ones are complete, which ones need more work, etc.

Andrew as far as dropping them. If all you have is one of my dinky patches that changes things to use kthread_run feel free, because of the general necessity of calling kthread_stop I'm going to have to rework those anyway, and I still have the originals.

If there is something more the we probably want to keep the patch because someone has actually looked at it and done something useful.

I'm just now starting to work my way through them all again paying a little closer attention, so I can do a thorough conversion.

Quoted 7 lines

Quoted 2 lines
I might send ia64-sn-xpc-convert-to-use-kthread-api.patch+fixes off to Tony, as people put quite a bit of review and test effort into that one.
Andrew, I would recommend holding off on sending these xpc patches to Tony as the kthread_run()s aren't paired with kthread_stop()s yet. I need to generate an additional patch after I've first sorted out how best to deal with kthread_stop()'ng XPC's pool of kthreads with Eric.

Ok. Dean gve me a couple of a day or so. I think I have just worked through how to directly create kthreads without too much pain. We are still going to need kthreadd for spawning for a bit because I don't expect all architectures to change over immediately, but I think things can be done in a fairly simple low risk manner.

The changes to the kernel_thread replacement aren't going to be too
bad, pretty much just adding a couple of parameters. It is
copy_thread where things get sticky.

If we can spawn threads fast enough we don't need a thread pool, I would rather do that.

Eric

Dean Nelson

2007-05-02 15:37 UTC #37 lore raw

Re: "partical" kthread conversion

Quoted 17 lines · Eric W. Biederman

Quoted 8 lines · Dean Nelson
Quoted 2 lines · Andrew Morton
I might send ia64-sn-xpc-convert-to-use-kthread-api.patch+fixes off to Tony, as people put quite a bit of review and test effort into that one.
Andrew, I would recommend holding off on sending these xpc patches to Tony as the kthread_run()s aren't paired with kthread_stop()s yet. I need to generate an additional patch after I've first sorted out how best to deal with kthread_stop()'ng XPC's pool of kthreads with Eric.
Ok. Dean gve me a couple of a day or so. I think I have just worked through how to directly create kthreads without too much pain. We are still going to need kthreadd for spawning for a bit because I don't expect all architectures to change over immediately, but I think things can be done in a fairly simple low risk manner.

Quoted 6 lines

The changes to the kernel_thread replacement aren't going to be too
bad, pretty much just adding a couple of parameters. It is
copy_thread where things get sticky.
If we can spawn threads fast enough we don't need a thread pool, I would rather do that.

I'd typed up some questions for you about the new patch I need to create which I'd just sent to you, so I won't repeat them here.

Before proceeding to far with your above changes, you might wait to see the proposal that Robin Holt is putting together for a kthread pool. I'm not sure how spawning a thread (which involves allocation of the task_struct amongst other things, plus scheduling) can beat a wake_up() of an already existing thread for cost time-wise.

Dean

Eric W. Biederman

2007-05-02 15:49 UTC #38 lore raw

Re: "partical" kthread conversion

Quoted 8 lines · Dean Nelson

I'd typed up some questions for you about the new patch I need to create which I'd just sent to you, so I won't repeat them here.
Before proceeding to far with your above changes, you might wait to see the proposal that Robin Holt is putting together for a kthread pool. I'm not sure how spawning a thread (which involves allocation of the task_struct amongst other things, plus scheduling) can beat a wake_up() of an already existing thread for cost time-wise.

A reasonable point, although if you don't happen to sleep in the allocations I suspect time wise it's pretty much a wash.

I have some other reasons I might need the capability of clone a thread from a non-parent process, and it has the potential to simplify some things in the kthread case so I'm going to finish investigating, since I believe I have figured out a path to that target.

Eric

Andrew Morton

author maintainer

2007-05-02 19:35 UTC #39 lore raw

Re: "partical" kthread conversion

Quoted 6 lines · On Wed, 02 May 2007 08:45:54 -0600 [email protected] (Eric W. Biederman)…

Quoted 4 lines · Dean Nelson
Quoted 3 lines · Andrew Morton
Quoted 2 lines
There's probably a few more patches falling into this category, these were just the first one the stick into my eye.

Quoted 8 lines

Quoted 2 lines
Quoted 2 lines
Yes, I think I'll probably drop all of them - I've completely lost track of which ones are complete, which ones need more work, etc.
Andrew as far as dropping them. If all you have is one of my dinky patches that changes things to use kthread_run feel free, because of the general necessity of calling kthread_stop I'm going to have to rework those anyway, and I still have the originals.

I gave up and dropped them all - let's have another run at it. Possibly some of them were complete and didn't deserve dropping, in which case you can send them straight back at me.

Eric W. Biederman

2007-05-02 20:39 UTC #40 lore raw

Re: "partical" kthread conversion

Quoted 3 lines · Andrew Morton

I gave up and dropped them all - let's have another run at it. Possibly some of them were complete and didn't deserve dropping, in which case you can send them straight back at me.

Sounds like a plan.

Eric

Nick Piggin

2007-05-01 08:44 UTC #41 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 8 lines · Andrew Morton

mm-simplify-filemap_nopage.patch mm-fix-fault-vs-invalidate-race-for-linear-mappings.patch mm-merge-populate-and-nopage-into-fault-fixes-nonlinear.patch mm-merge-nopfn-into-fault.patch convert-hugetlbfs-to-use-vm_ops-fault.patch mm-remove-legacy-cruft.patch mm-debug-check-for-the-fault-vs-invalidate-race.patch

Quoted 4 lines

mm-fix-clear_page_dirty_for_io-vs-fault-race.patch
Miscish MM changes. Will merge, dependent upon what still applies and works if the moveable-zone patches get stalled.

These fix some bugs in the core vm, at least the former one we have seen numerous people hitting in production...

I don't suppose you mean these are logically dependant on new features sitting below them in your patch stack, just that you don't want to spend time fixing a lot of rejects? If so, I can help fix those up, but I don't think there is anything major, IIRC the biggest annoyance is just that changing some GFP_types throws some big hunks.

So, do you or anyone else have any problems with these patches going in 2.6.22? I haven't had much feedback for a while, but I was under the impression that people are more-or-less happy with them?

mm-fix-fault-vs-invalidate-race-for-linear-mappings.patch

This patch fixes the core filemap_nopage vs invalidate_inode_pages2 race by having filemap_nopage return a locked page to do_no_page, and removes the fairly complex (and inadequate) truncate_count synchronisation logic.

There were concerns that we could do this more cheaply, but I think it is important to start with a base that is simple and more likely to be correct and build on that. My testing didn't show any obvious problems with performance.

mm-merge-populate-and-nopage-into-fault-fixes-nonlinear.patch mm-merge-nopfn-into-fault.patch
etc.

These move ->nopage, ->populate, ->nopfn (and soon, ->page_mkwrite) into a single, unified interface. Although this strictly closes some similar holes in nonlinear faults as well, they are very uncommon, so I wouldn't be so upset if these aren't merged in 2.6.22 (I don't see any reason not to, but at least they don't fix major bugs).

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andrew Morton

author maintainer

2007-05-01 08:54 UTC #42 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 13 lines · On Tue, 01 May 2007 18:44:07 +1000 Nick Piggin <[email protected]> wrote:

Quoted 8 lines
mm-simplify-filemap_nopage.patch mm-fix-fault-vs-invalidate-race-for-linear-mappings.patch mm-merge-populate-and-nopage-into-fault-fixes-nonlinear.patch mm-merge-nopfn-into-fault.patch convert-hugetlbfs-to-use-vm_ops-fault.patch mm-remove-legacy-cruft.patch mm-debug-check-for-the-fault-vs-invalidate-race.patch
Quoted 4 lines
mm-fix-clear_page_dirty_for_io-vs-fault-race.patch
Miscish MM changes. Will merge, dependent upon what still applies and works if the moveable-zone patches get stalled.

Quoted 6 lines

These fix some bugs in the core vm, at least the former one we have seen numerous people hitting in production...
I don't suppose you mean these are logically dependant on new features sitting below them in your patch stack, just that you don't want to spend time fixing a lot of rejects?

It'll probably be OK - I just haven't checked yet. I'm fairly handy at fixing rejects nowadays ;)

Nobody seems to be taking up this opportunity to provide us with review and test results on the antifrag patches.

Hugh Dickins

2007-05-01 19:31 UTC #43 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 17 lines · Nick Piggin

Quoted 8 lines · Andrew Morton
mm-simplify-filemap_nopage.patch mm-fix-fault-vs-invalidate-race-for-linear-mappings.patch mm-merge-populate-and-nopage-into-fault-fixes-nonlinear.patch mm-merge-nopfn-into-fault.patch convert-hugetlbfs-to-use-vm_ops-fault.patch mm-remove-legacy-cruft.patch mm-debug-check-for-the-fault-vs-invalidate-race.patch
Quoted 4 lines
mm-fix-clear_page_dirty_for_io-vs-fault-race.patch
Miscish MM changes. Will merge, dependent upon what still applies and works if the moveable-zone patches get stalled.
These fix some bugs in the core vm, at least the former one we have seen numerous people hitting in production...

...

Quoted 12 lines

So, do you or anyone else have any problems with these patches going in 2.6.22? I haven't had much feedback for a while, but I was under the impression that people are more-or-less happy with them?
mm-fix-fault-vs-invalidate-race-for-linear-mappings.patch
This patch fixes the core filemap_nopage vs invalidate_inode_pages2 race by having filemap_nopage return a locked page to do_no_page, and removes the fairly complex (and inadequate) truncate_count synchronisation logic.

Quoted 4 lines

There were concerns that we could do this more cheaply, but I think it is important to start with a base that is simple and more likely to be correct and build on that. My testing didn't show any obvious problems with performance.

I don't see _problems_ with performance, but I do consistently see the same kind of ~5% degradation in lmbench fork, exec, sh, mmap latency and page fault tests on SMP, several machines, just as I did last year.

I'm assuming this patch is the one responsible: at 2.6.20-rc4 time you posted a set of 10 and a set of 7 patches I tried in versus out; at 2.6.21-rc3-mm2 time you had a group of patches in -mm I tried in versus out; with similar results.

I did check the graphs on test.kernel.org, I couldn't see any bad behaviour there that correlated with this work; though each -mm has such a variety of new work in it, it's very hard to attribute. And nobody else has reported any regression from your patches.

I'm inclined to write it off as poorer performance in some micro- benchmarks, against which we offset the improved understandabilty of holding the page lock over the file fault.

But I was quite disappointed when mm-fix-fault-vs-invalidate-race-for-linear-mappings-fix.patch appeared, putting double unmap_mapping_range calls in. Certainly you were wrong to take the one out, but a pity to end up with two.

Your comment says/said:
The nopage vs invalidate race fix patch did not take care of truncating private COW pages. Mind you, I'm pretty sure this was previously racy even for regular truncate, not to mention vmtruncate_range.

vmtruncate_range (holepunch) was deficient I agree, and though we can now take out your second unmap_mapping_range there, that's only because I've slipped one into shmem_truncate_range. In due course it needs to be properly handled by noting the range in shmem inode info.

(I think you couldn't take that approach, noting invalid range in ->mapping while invalidating, because NFS has/had some cases of invalidate_whatever without i_mutex?)

But I'm pretty sure (to use your words!) regular truncate was not racy
before: I believe Andrea's sequence count was handling that case fine, without a second unmap_mapping_range.

Well, I guess I've come to accept that, expensive as unmap_mapping_range may be, truncating files while they're mmap'ed is perverse behaviour:
perhaps even deserving such punishment.

But it is a shame, and leaves me wondering what you gained with the page lock there.

One thing gained is ease of understanding, and if your later patches build an edifice upon the knowledge of holding that page lock while faulting, I've no wish to undermine that foundation.

Quoted 10 lines

mm-merge-populate-and-nopage-into-fault-fixes-nonlinear.patch mm-merge-nopfn-into-fault.patch
etc.
These move ->nopage, ->populate, ->nopfn (and soon, ->page_mkwrite) into a single, unified interface. Although this strictly closes some similar holes in nonlinear faults as well, they are very uncommon, so I wouldn't be so upset if these aren't merged in 2.6.22 (I don't see any reason not to, but at least they don't fix major bugs).

I don't have an opinion on these, but I think BenH and others were strongly in favour, with various people waiting upon them.

Hugh

Nick Piggin

2007-05-02 03:08 UTC #44 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 11 lines · Hugh Dickins

Quoted 4 lines · Nick Piggin
There were concerns that we could do this more cheaply, but I think it is important to start with a base that is simple and more likely to be correct and build on that. My testing didn't show any obvious problems with performance.
I don't see _problems_ with performance, but I do consistently see the same kind of ~5% degradation in lmbench fork, exec, sh, mmap latency and page fault tests on SMP, several machines, just as I did last year.

OK. I did run some tests at one stage which didn't show a regression on my P4, however I don't know that they were statistically significant. I'll try a couple more runs and post numbers.

Quoted 23 lines

I'm assuming this patch is the one responsible: at 2.6.20-rc4 time you posted a set of 10 and a set of 7 patches I tried in versus out; at 2.6.21-rc3-mm2 time you had a group of patches in -mm I tried in versus out; with similar results.
I did check the graphs on test.kernel.org, I couldn't see any bad behaviour there that correlated with this work; though each -mm has such a variety of new work in it, it's very hard to attribute. And nobody else has reported any regression from your patches.
I'm inclined to write it off as poorer performance in some micro- benchmarks, against which we offset the improved understandabilty of holding the page lock over the file fault.
But I was quite disappointed when mm-fix-fault-vs-invalidate-race-for-linear-mappings-fix.patch appeared, putting double unmap_mapping_range calls in. Certainly
you were wrong to take the one out, but a pity to end up with two.
Your comment says/said:
The nopage vs invalidate race fix patch did not take care of truncating private COW pages. Mind you, I'm pretty sure this was previously racy even for regular truncate, not to mention vmtruncate_range.

Quoted 8 lines

vmtruncate_range (holepunch) was deficient I agree, and though we can now take out your second unmap_mapping_range there, that's only because I've slipped one into shmem_truncate_range. In due course it needs to be properly handled by noting the range in shmem inode info.
(I think you couldn't take that approach, noting invalid range in ->mapping while invalidating, because NFS has/had some cases of invalidate_whatever without i_mutex?)

Sorry, I didn't parse this? But I wonder whether it is better to do it in vmtruncate_range than the filesystem? Private COWed pages are not really a filesystem "thing"...

Quoted 3 lines

But I'm pretty sure (to use your words!) regular truncate was not racy
before: I believe Andrea's sequence count was handling that case fine, without a second unmap_mapping_range.

OK, I think you're right. I _think_ it should also be OK with the lock_page version as well: we should not be able to have any pages after the first unmap_mapping_range call, because of the i_size write. So if we have no pages, there is nothing to 'cow' from.

Quoted 10 lines

Well, I guess I've come to accept that, expensive as unmap_mapping_range may be, truncating files while they're mmap'ed is perverse behaviour:
perhaps even deserving such punishment.
But it is a shame, and leaves me wondering what you gained with the page lock there.
One thing gained is ease of understanding, and if your later patches build an edifice upon the knowledge of holding that page lock while faulting, I've no wish to undermine that foundation.

It also fixes a bug, doesn't it? ;)

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Nick Piggin

2007-05-02 09:15 UTC #45 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 12 lines · Nick Piggin

Quoted 10 lines · Hugh Dickins
Quoted 4 lines · Nick Piggin
There were concerns that we could do this more cheaply, but I think it is important to start with a base that is simple and more likely to be correct and build on that. My testing didn't show any obvious problems with performance.

Quoted 8 lines

Quoted 3 lines
I don't see _problems_ with performance, but I do consistently see the same kind of ~5% degradation in lmbench fork, exec, sh, mmap latency and page fault tests on SMP, several machines, just as I did last year.
OK. I did run some tests at one stage which didn't show a regression on my P4, however I don't know that they were statistically significant. I'll try a couple more runs and post numbers.

I didn't have enough time tonight to get means/stddev, etc, but the runs are pretty stable.

Patch tested was just the lock page one.

SMP kernel, tasks bound to 1 CPU:

P4 Xeon

         pagefault   fork          exec
2.6.21   1.67-1.69   140.7-142.0   449.5-460.8
+patch   1.75-1.77   144.0-145.5   456.2-463.0

So it's taken on nearly 5% on pagefault, but looks like less than 2% on fork, so not as bad as your numbers (phew).

         pagefault   fork          exec
2.6.21   1.49-1.51   164.6-170.8   741.8-760.3
+patch   1.71-1.73   175.2-180.8   780.5-794.2

Bigger hit there.

Page faults can be improved a tiny bit by not using a test and clear op in unlock_page (less barriers for the G5).

I don't think that's really a blocker problem for a merge, but I wonder what we can do to improve it. Lockless pagecache shaves quite a bit of straight line find_get_page performance there.

Going to a non-sleeping lock might be one way to go in the long term, but it would require quite a lot of restructuring.

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hugh Dickins

2007-05-02 14:00 UTC #46 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 17 lines · Nick Piggin

Quoted 16 lines · Hugh Dickins
But I was quite disappointed when mm-fix-fault-vs-invalidate-race-for-linear-mappings-fix.patch appeared, putting double unmap_mapping_range calls in. Certainly you were wrong to take the one out, but a pity to end up with two.
Your comment says/said:
The nopage vs invalidate race fix patch did not take care of truncating private COW pages. Mind you, I'm pretty sure this was previously racy even for regular truncate, not to mention vmtruncate_range.
vmtruncate_range (holepunch) was deficient I agree, and though we can now take out your second unmap_mapping_range there, that's only because I've slipped one into shmem_truncate_range. In due course it needs to be properly handled by noting the range in shmem inode info.

Quoted 5 lines

Quoted 3 lines
(I think you couldn't take that approach, noting invalid range in ->mapping while invalidating, because NFS has/had some cases of invalidate_whatever without i_mutex?)
Sorry, I didn't parse this?

I meant that i_size is used to protect against truncation races, but we have no equivalent inval_start,inval_end in the struct inode or struct address_space, such as could be used for similar protection against races while invalidating.

And that IIRC there are places where NFS was doing the invalidation without i_mutex: so there could be concurrent invalidations, so one inval_start,inval_end in the structure wouldn't be enough anyway.

Quoted 3 lines

But I wonder whether it is better to do
it in vmtruncate_range than the filesystem? Private COWed pages are not really a filesystem "thing"...

It wasn't the thought of private COWed pages which made me put a second unmap_mapping_range in shmem_truncate_range, it was its own internal file<->swap consistency which needed that (as a quick fix). The real fix to be having a trunc_start,trunc_end or whatever in the shmem_inode_info (assuming it's not wanted in the common inode:
might be if holepunch spreads e.g. it's been mentioned with fallocate).

Re private COWed pages and holepunch: Miklos and I agree that really it would be better for holepunch _not_ to remove them - but that's rather off-topic.

More on-topic, since you suggest doing more within vmtruncate_range than the filesystem: no, I'm afraid that's misdesigned, and I want to move almost all of it into the filesystem ->truncate_range. Because, if what vmtruncate_range is doing before it gets to the filesystem isn't to be just a waste of time, the filesystem needs to know what's going on in advance - just as notify_change warns the filesystem about a coming truncation. But easier than inventing some new notification is to move it all into the filesystem, with unmap_mapping_range+truncate_inode_pages_range its library helpers.

Quoted 9 lines

Quoted 4 lines
But I'm pretty sure (to use your words!) regular truncate was not racy
before: I believe Andrea's sequence count was handling that case fine, without a second unmap_mapping_range.
OK, I think you're right. I _think_ it should also be OK with the lock_page version as well: we should not be able to have any pages after the first unmap_mapping_range call, because of the i_size write. So if we have no pages, there is nothing to 'cow' from.

I'd be delighted if you can remove those later unmap_mapping_ranges. As I recall, the important thing for the copy pages is to be holding the page lock (or whatever other serialization) on the copied page still while the copy page is inserted into pagetable: that looks to be so in your __do_fault.

Quoted 8 lines

Quoted 6 lines
But it is a shame, and leaves me wondering what you gained with the page lock there.
One thing gained is ease of understanding, and if your later patches build an edifice upon the knowledge of holding that page lock while faulting, I've no wish to undermine that foundation.
It also fixes a bug, doesn't it? ;)

Well, I'd come to think that perhaps the bugs would be solved by that second unmap_mapping_range alone, so the pagelock changes just a misleading diversion.

I'm not sure how I feel about that: calling unmap_mapping_range a second time feels such a cheat, but if (big if) it does solve the races, and the pagelock method is as expensive as your numbers now suggest...

Hugh

Nick Piggin

2007-05-03 01:32 UTC #47 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 1 line · Hugh Dickins

On Wed, 2 May 2007, Nick Piggin wrote:

[snip]

Quoted 9 lines

More on-topic, since you suggest doing more within vmtruncate_range than the filesystem: no, I'm afraid that's misdesigned, and I want to move almost all of it into the filesystem ->truncate_range. Because, if what vmtruncate_range is doing before it gets to the filesystem isn't to be just a waste of time, the filesystem needs to know what's going on in advance - just as notify_change warns the filesystem about a coming truncation. But easier than inventing some new notification is to move it all into the filesystem, with unmap_mapping_range+truncate_inode_pages_range its library helpers.

Well I would prefer it to follow the same pattern as regular truncate. I don't think it is misdesigned to call the filesystem _first_, but I think if you do that then the filesystem should call the vm to prepare / finish truncate, rather than open code calls to unmap itself.

Quoted 10 lines

Quoted 8 lines
Quoted 3 lines
But I'm pretty sure (to use your words!) regular truncate was not racy
before: I believe Andrea's sequence count was handling that case fine, without a second unmap_mapping_range.
OK, I think you're right. I _think_ it should also be OK with the lock_page version as well: we should not be able to have any pages after the first unmap_mapping_range call, because of the i_size write. So if we have no pages, there is nothing to 'cow' from.

Quoted 5 lines

I'd be delighted if you can remove those later unmap_mapping_ranges. As I recall, the important thing for the copy pages is to be holding the page lock (or whatever other serialization) on the copied page still while the copy page is inserted into pagetable: that looks to be so in your __do_fault.

Yeah, I think my thought process went wrong on those... I'll revisit.

Quoted 10 lines

Quoted 8 lines
Quoted 6 lines
But it is a shame, and leaves me wondering what you gained with the page lock there.
One thing gained is ease of understanding, and if your later patches build an edifice upon the knowledge of holding that page lock while faulting, I've no wish to undermine that foundation.
It also fixes a bug, doesn't it? ;)

Quoted 8 lines

Well, I'd come to think that perhaps the bugs would be solved by that second unmap_mapping_range alone, so the pagelock changes just a misleading diversion.
I'm not sure how I feel about that: calling unmap_mapping_range a second time feels such a cheat, but if (big if) it does solve the races, and the pagelock method is as expensive as your numbers now suggest...

Well aside from being terribly ugly, it means we can still drop the dirty bit where we'd otherwise rather not, so I don't think we can do that.

I think there may be some way we can do this without taking the page lock, and I was going to look at it, but I think it is quite neat to just lock the page...

I don't think performance is _that_ bad. On the P4 it is a couple of % on the microbenchmarks. The G5 is worse, but even then I don't think it is I'll try to improve that and get back to you.

The problem is that lock/unlock_page is expensive on powerpc, and if we improve that, we improve more than just the fault handler...

The attached patch gets performance up a bit by avoiding some barriers and some cachelines:

         pagefault   fork          exec
2.6.21   1.49-1.51   164.6-170.8   741.8-760.3
+patch   1.71-1.73   175.2-180.8   780.5-794.2
+patch2  1.61-1.63   169.8-175.0   748.6-757.0

So that brings the fork/exec hits down to much less than 5%, and would likely speed up other things that lock the page, like write or page reclaim.

I think we could get further performance improvement by implementing arch specific bitops for lock/unlock operations, so we don't need to use things like smb_mb__before_clear_bit() if they aren't needed or full barriers in the test_and_set_bit().

Signature · 1 line

SUSE Labs, Novell Inc.

include/linux/page-flags.h +2 -0

Index: linux-2.6/include/linux/page-flags.h
===================================================================
@@ -91,6 +91,8 @@
 #define PG_nosave_free		18	/* Used for system suspend/resume */
 #define PG_buddy		19	/* Page is free, on buddy lists */
 
+#define PG_waiters		20	/* Page has PG_locked waiters */
+
 /* PG_owner_priv_1 users should have descriptive aliases */
 #define PG_checked		PG_owner_priv_1 /* Used by some filesystems */

include/linux/pagemap.h +2 -2

Index: linux-2.6/include/linux/pagemap.h
===================================================================
@@ -141,7 +141,7 @@
 static inline void lock_page(struct page *page)
 {
 	might_sleep();
-	if (TestSetPageLocked(page))
+	if (unlikely(TestSetPageLocked(page)))
 		__lock_page(page);
 }

@@ -152,7 +152,7 @@
 static inline void lock_page_nosync(struct page *page)
 {
 	might_sleep();
-	if (TestSetPageLocked(page))
+	if (unlikely(TestSetPageLocked(page)))
 		__lock_page_nosync(page);
 }

mm/filemap.c +11 -4

Index: linux-2.6/mm/filemap.c
===================================================================
@@ -532,11 +532,13 @@
  */
 void fastcall unlock_page(struct page *page)
 {
+	VM_BUG_ON(!PageLocked(page));
 	smp_mb__before_clear_bit();
-	if (!TestClearPageLocked(page))
-		BUG();
-	smp_mb__after_clear_bit(); 
-	wake_up_page(page, PG_locked);
+	ClearPageLocked(page);
+	if (unlikely(test_bit(PG_waiters, &page->flags))) {
+		clear_bit(PG_waiters, &page->flags);
+		wake_up_page(page, PG_locked);
+	}
 }
 EXPORT_SYMBOL(unlock_page);

@@ -568,6 +570,11 @@
 {
 	DEFINE_WAIT_BIT(wait, &page->flags, PG_locked);
 
+	set_bit(PG_waiters, &page->flags);
+	if (unlikely(!TestSetPageLocked(page))) {
+		clear_bit(PG_waiters, &page->flags);
+		return;
+	}
 	__wait_on_bit_lock(page_waitqueue(page), &wait, sync_page,
 							TASK_UNINTERRUPTIBLE);
 }

Christoph Hellwig

2007-05-03 10:37 UTC #48 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 9 lines · Nick Piggin

The attached patch gets performance up a bit by avoiding some barriers and some cachelines:
G5
         pagefault   fork          exec
2.6.21   1.49-1.51   164.6-170.8   741.8-760.3
(quoted diff) +2 -0
+patch   1.71-1.73   175.2-180.8   780.5-794.2
+patch2  1.61-1.63   169.8-175.0   748.6-757.0

Quoted 3 lines

So that brings the fork/exec hits down to much less than 5%, and would likely speed up other things that lock the page, like write or page reclaim.

Is that every fork/exec or just under certain cicumstances? A 5% regression on every fork/exec is not acceptable.

Nick Piggin

2007-05-03 12:56 UTC #49 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 11 lines · Christoph Hellwig

Quoted 9 lines · Nick Piggin
The attached patch gets performance up a bit by avoiding some barriers and some cachelines:
G5
pagefault fork exec
2.6.21 1.49-1.51 164.6-170.8 741.8-760.3
(quoted diff) +2 -0
+patch   1.71-1.73   175.2-180.8   780.5-794.2
+patch2  1.61-1.63   169.8-175.0   748.6-757.0

Quoted 7 lines

Quoted 3 lines
So that brings the fork/exec hits down to much less than 5%, and would likely speed up other things that lock the page, like write or page reclaim.
Is that every fork/exec or just under certain cicumstances? A 5% regression on every fork/exec is not acceptable.

Well after patch2, G5 fork is 3% and exec is 1%, I'd say the P4 numbers will be improved as well with that patch. Then if we have specific lock/unlock bitops, I hope it should reduce that further.

The overhead that is there should just be coming from the extra overhead in the file backed fault handler. For noop fork/execs, I think that tends to be more pronounced, it is hard to see any difference on any non-micro benchmark.

The other thing is that I think there could be some cache effects happening -- for example the exec numbers on the 2nd line are disproportionately large.

It definitely isn't a good thing to drop performance anywhere though, so I'll keep looking for improvements.

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Nick Piggin

2007-05-04 09:23 UTC #50 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 9 lines · Nick Piggin

Quoted 2 lines · Christoph Hellwig
Is that every fork/exec or just under certain cicumstances? A 5% regression on every fork/exec is not acceptable.
Well after patch2, G5 fork is 3% and exec is 1%, I'd say the P4 numbers will be improved as well with that patch. Then if we have specific lock/unlock bitops, I hope it should reduce that further.

OK, with the races and missing barriers fixed from the previous patch, plus the attached one added (+patch3), numbers are better again (I'm not sure if I have the ppc barriers correct though).

These ops could also be put to use in bit spinlocks, buffer lock, and probably a few other places too.

2.6.21   1.49-1.51   164.6-170.8   741.8-760.3
+patch   1.71-1.73   175.2-180.8   780.5-794.2
+patch2  1.61-1.63   169.8-175.0   748.6-757.0
+patch3  1.54-1.57   165.6-170.9   748.5-757.5

So fault performance goes to under 5%, fork is in the noise, exec is still up 1%, but maybe that's noise or cache effects again.

Signature · 1 line

SUSE Labs, Novell Inc.

include/asm-powerpc/bitops.h +39 -0

Index: linux-2.6/include/asm-powerpc/bitops.h
===================================================================
@@ -87,6 +87,24 @@
 	: "cc" );
 }
 
+static __inline__ void clear_bit_unlock(int nr, volatile unsigned long *addr)
+{
+	unsigned long old;
+	unsigned long mask = BITOP_MASK(nr);
+	unsigned long *p = ((unsigned long *)addr) + BITOP_WORD(nr);
+
+	__asm__ __volatile__(
+	LWSYNC_ON_SMP
+"1:"	PPC_LLARX "%0,0,%3	# clear_bit_unlock\n"
+	"andc	%0,%0,%2\n"
+	PPC405_ERR77(0,%3)
+	PPC_STLCX "%0,0,%3\n"
+	"bne-	1b"
+	: "=&r" (old), "+m" (*p)
+	: "r" (mask), "r" (p)
+	: "cc" );
+}
+
 static __inline__ void change_bit(int nr, volatile unsigned long *addr)
 {
 	unsigned long old;

@@ -126,6 +144,27 @@
 	return (old & mask) != 0;
 }
 
+static __inline__ int test_and_set_bit_lock(unsigned long nr,
+				       volatile unsigned long *addr)
+{
+	unsigned long old, t;
+	unsigned long mask = BITOP_MASK(nr);
+	unsigned long *p = ((unsigned long *)addr) + BITOP_WORD(nr);
+
+	__asm__ __volatile__(
+"1:"	PPC_LLARX "%0,0,%3		# test_and_set_bit_lock\n"
+	"or	%1,%0,%2 \n"
+	PPC405_ERR77(0,%3)
+	PPC_STLCX "%1,0,%3 \n"
+	"bne-	1b"
+	ISYNC_ON_SMP
+	: "=&r" (old), "=&r" (t)
+	: "r" (mask), "r" (p)
+	: "cc", "memory");
+
+	return (old & mask) != 0;
+}
+
 static __inline__ int test_and_clear_bit(unsigned long nr,
 					 volatile unsigned long *addr)
 {

include/linux/pagemap.h +7 -2

Index: linux-2.6/include/linux/pagemap.h
===================================================================
@@ -136,13 +136,18 @@
 extern void FASTCALL(__wait_on_page_locked(struct page *page));
 extern void FASTCALL(unlock_page(struct page *page));
 
+static inline int trylock_page(struct page *page)
+{
+	return (likely(!TestSetPageLocked_Lock(page)));
+}
+
 /*
  * lock_page may only be called if we have the page's inode pinned.
  */
 static inline void lock_page(struct page *page)
 {
 	might_sleep();
-	if (unlikely(TestSetPageLocked(page)))
+	if (!trylock_page(page))
 		__lock_page(page);
 }

@@ -153,7 +158,7 @@
 static inline void lock_page_nosync(struct page *page)
 {
 	might_sleep();
-	if (unlikely(TestSetPageLocked(page)))
+	if (!trylock_page(page))
 		__lock_page_nosync(page);
 }

drivers/scsi/sg.c +1 -1

Index: linux-2.6/drivers/scsi/sg.c
===================================================================
@@ -1734,7 +1734,7 @@
                  */
 		flush_dcache_page(pages[i]);
 		/* ?? Is locking needed? I don't think so */
-		/* if (TestSetPageLocked(pages[i]))
+		/* if (!trylock_page(pages[i]))
 		   goto out_unlock; */
         }

fs/cifs/file.c +1 -1

Index: linux-2.6/fs/cifs/file.c
===================================================================
@@ -1229,7 +1229,7 @@
 
 			if (first < 0)
 				lock_page(page);
-			else if (TestSetPageLocked(page))
+			else if (!trylock_page(page))
 				break;
 
 			if (unlikely(page->mapping != mapping)) {

fs/jbd/commit.c +1 -1

Index: linux-2.6/fs/jbd/commit.c
===================================================================
@@ -64,7 +64,7 @@
 		goto nope;
 
 	/* OK, it's a truncated page */
-	if (TestSetPageLocked(page))
+	if (!trylock_page(page))
 		goto nope;
 
 	page_cache_get(page);

fs/jbd2/commit.c +1 -1

Index: linux-2.6/fs/jbd2/commit.c
===================================================================
@@ -64,7 +64,7 @@
 		goto nope;
 
 	/* OK, it's a truncated page */
-	if (TestSetPageLocked(page))
+	if (!trylock_page(page))
 		goto nope;
 
 	page_cache_get(page);

fs/xfs/linux-2.6/xfs_aops.c +2 -2

Index: linux-2.6/fs/xfs/linux-2.6/xfs_aops.c
===================================================================
@@ -601,7 +601,7 @@
 			} else
 				pg_offset = PAGE_CACHE_SIZE;
 
-			if (page->index == tindex && !TestSetPageLocked(page)) {
+			if (page->index == tindex && trylock_page(page)) {
 				len = xfs_probe_page(page, pg_offset, mapped);
 				unlock_page(page);
 			}

@@ -685,7 +685,7 @@
 
 	if (page->index != tindex)
 		goto fail;
-	if (TestSetPageLocked(page))
+	if (!trylock_page(page))
 		goto fail;
 	if (PageWriteback(page))
 		goto fail_unlock_page;

include/linux/page-flags.h +4 -0

Index: linux-2.6/include/linux/page-flags.h
===================================================================
@@ -116,8 +116,12 @@
 		set_bit(PG_locked, &(page)->flags)
 #define TestSetPageLocked(page)		\
 		test_and_set_bit(PG_locked, &(page)->flags)
+#define TestSetPageLocked_Lock(page)		\
+		test_and_set_bit_lock(PG_locked, &(page)->flags)
 #define ClearPageLocked(page)		\
 		clear_bit(PG_locked, &(page)->flags)
+#define ClearPageLocked_Unlock(page)		\
+		clear_bit_unlock(PG_locked, &(page)->flags)
 #define TestClearPageLocked(page)	\
 		test_and_clear_bit(PG_locked, &(page)->flags)

mm/memory.c +1 -1

Index: linux-2.6/mm/memory.c
===================================================================
@@ -1550,7 +1550,7 @@
 	 * not dirty accountable.
 	 */
 	if (PageAnon(old_page)) {
-		if (!TestSetPageLocked(old_page)) {
+		if (trylock_page(old_page)) {
 			reuse = can_share_swap_page(old_page);
 			unlock_page(old_page);
 		}

mm/migrate.c +2 -2

Index: linux-2.6/mm/migrate.c
===================================================================
@@ -569,7 +569,7 @@
 	 * establishing additional references. We are the only one
 	 * holding a reference to the new page at this point.
 	 */
-	if (TestSetPageLocked(newpage))
+	if (!trylock_page(newpage))
 		BUG();
 
 	/* Prepare mapping for the new page.*/

@@ -621,7 +621,7 @@
 		goto move_newpage;
 
 	rc = -EAGAIN;
-	if (TestSetPageLocked(page)) {
+	if (!trylock_page(page)) {
 		if (!force)
 			goto move_newpage;
 		lock_page(page);

mm/rmap.c +1 -1

Index: linux-2.6/mm/rmap.c
===================================================================
@@ -426,7 +426,7 @@
 			referenced += page_referenced_anon(page);
 		else if (is_locked)
 			referenced += page_referenced_file(page);
-		else if (TestSetPageLocked(page))
+		else if (!trylock_page(page))
 			referenced++;
 		else {
 			if (page->mapping)

mm/shmem.c +2 -2

Index: linux-2.6/mm/shmem.c
===================================================================
@@ -1155,7 +1155,7 @@
 		}
 
 		/* We have to do this with page locked to prevent races */
-		if (TestSetPageLocked(swappage)) {
+		if (!trylock_page(swappage)) {
 			shmem_swp_unmap(entry);
 			spin_unlock(&info->lock);
 			wait_on_page_locked(swappage);

@@ -1214,7 +1214,7 @@
 		shmem_swp_unmap(entry);
 		filepage = find_get_page(mapping, idx);
 		if (filepage &&
-		    (!PageUptodate(filepage) || TestSetPageLocked(filepage))) {
+		    (!PageUptodate(filepage) || !trylock_page(filepage))) {
 			spin_unlock(&info->lock);
 			wait_on_page_locked(filepage);
 			page_cache_release(filepage);

mm/swap.c +1 -1

Index: linux-2.6/mm/swap.c
===================================================================
@@ -412,7 +412,7 @@
 	for (i = 0; i < pagevec_count(pvec); i++) {
 		struct page *page = pvec->pages[i];
 
-		if (PagePrivate(page) && !TestSetPageLocked(page)) {
+		if (PagePrivate(page) && trylock_page(page)) {
 			if (PagePrivate(page))
 				try_to_release_page(page, 0);
 			unlock_page(page);

mm/swap_state.c +1 -1

Index: linux-2.6/mm/swap_state.c
===================================================================
@@ -252,7 +252,7 @@
  */
 static inline void free_swap_cache(struct page *page)
 {
-	if (PageSwapCache(page) && !TestSetPageLocked(page)) {
+	if (PageSwapCache(page) && trylock_page(page)) {
 		remove_exclusive_swap_page(page);
 		unlock_page(page);
 	}

mm/swapfile.c +1 -1

Index: linux-2.6/mm/swapfile.c
===================================================================
@@ -401,7 +401,7 @@
 	if (p) {
 		if (swap_entry_free(p, swp_offset(entry)) == 1) {
 			page = find_get_page(&swapper_space, entry.val);
-			if (page && unlikely(TestSetPageLocked(page))) {
+			if (page && unlikely(!trylock_page(page))) {
 				page_cache_release(page);
 				page = NULL;
 			}

mm/truncate.c +2 -2

Index: linux-2.6/mm/truncate.c
===================================================================
@@ -185,7 +185,7 @@
 			if (page_index > next)
 				next = page_index;
 			next++;
-			if (TestSetPageLocked(page))
+			if (!trylock_page(page))
 				continue;
 			if (PageWriteback(page)) {
 				unlock_page(page);

@@ -291,7 +291,7 @@
 			pgoff_t index;
 			int lock_failed;
 
-			lock_failed = TestSetPageLocked(page);
+			lock_failed = !trylock_page(page);
 
 			/*
 			 * We really shouldn't be looking at the ->index of an

mm/vmscan.c +2 -2

Index: linux-2.6/mm/vmscan.c
===================================================================
@@ -466,7 +466,7 @@
 		page = lru_to_page(page_list);
 		list_del(&page->lru);
 
-		if (TestSetPageLocked(page))
+		if (!trylock_page(page))
 			goto keep;
 
 		VM_BUG_ON(PageActive(page));

@@ -538,7 +538,7 @@
 				 * A synchronous write - probably a ramdisk.  Go
 				 * ahead and try to reclaim the page.
 				 */
-				if (TestSetPageLocked(page))
+				if (!trylock_page(page))
 					goto keep;
 				if (PageDirty(page) || PageWriteback(page))
 					goto keep_locked;

Nick Piggin

2007-05-04 09:43 UTC #51 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 27 lines · Nick Piggin

Quoted 11 lines · Nick Piggin
Quoted 2 lines · Christoph Hellwig
Is that every fork/exec or just under certain cicumstances? A 5% regression on every fork/exec is not acceptable.
Well after patch2, G5 fork is 3% and exec is 1%, I'd say the P4 numbers will be improved as well with that patch. Then if we have specific lock/unlock bitops, I hope it should reduce that further.
OK, with the races and missing barriers fixed from the previous patch, plus the attached one added (+patch3), numbers are better again (I'm not sure if I have the ppc barriers correct though).
These ops could also be put to use in bit spinlocks, buffer lock, and probably a few other places too.
2.6.21 1.49-1.51 164.6-170.8 741.8-760.3
(quoted diff) +3 -0
+patch   1.71-1.73   175.2-180.8   780.5-794.2
+patch2  1.61-1.63   169.8-175.0   748.6-757.0
+patch3  1.54-1.57   165.6-170.9   748.5-757.5

Quoted 2 lines

So fault performance goes to under 5%, fork is in the noise, exec is still up 1%, but maybe that's noise or cache effects again.

OK, with my new lock/unlock_page, dd if=large (bigger than RAM) sparse file of=/dev/null with an experimentally optimal block size (32K) goes from 626MB/s to 683MB/s on 2 CPU G5 booted with maxcpus=1.

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Benjamin Herrenschmidt

2007-05-08 03:03 UTC #52 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 2 lines · Nick Piggin

These ops could also be put to use in bit spinlocks, buffer lock, and probably a few other places too.

Ok, the performance hit seems to be under control (especially with the bigger benchmark showing actual improvements).

There's a little bogon with the PG_waiters bit that you already know about but appart from that it should be ok.

I must say I absolutely _LOVE_ the bitops with explicit _lock/_unlock semantics. That should allow us to remove a whole bunch of dodgy barriers and smp_mb__before_whatever_magic_crap() things we have all over the place by providing precisely the expected semantics for bit locks.

There are quite a few people who've been trying to do bit locks and I've always been very worried by how easy it is to get the barriers wrong (or too much barriers in the fast path) with these.

There are a couple of things we might want to think about regarding the actual API to bit locks... the API you propose is simple, but it might not fit some of the most exotic usage requirements, which typically are related to manipulating other bits along with the lock bit.

We might just ignore them though. In the case of the page lock, it's only hitting the slow path, and I would expect other usage scenarii to be similar.

Cheers,
Ben.

Hugh Dickins

2007-05-03 12:24 UTC #53 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 6 lines · Nick Piggin

The problem is that lock/unlock_page is expensive on powerpc, and if we improve that, we improve more than just the fault handler...
The attached patch gets performance up a bit by avoiding some barriers and some cachelines:

There's a strong whiff of raciness about this...
but I could very easily be wrong.

Quoted 23 lines

Index: linux-2.6/mm/filemap.c
===================================================================

mm/filemap.c +6 -4

@@ -532,11 +532,13 @@
  */
 void fastcall unlock_page(struct page *page)
 {
+	VM_BUG_ON(!PageLocked(page));
 	smp_mb__before_clear_bit();
-	if (!TestClearPageLocked(page))
-		BUG();
-	smp_mb__after_clear_bit(); 
-	wake_up_page(page, PG_locked);
+	ClearPageLocked(page);
+	if (unlikely(test_bit(PG_waiters, &page->flags))) {
+		clear_bit(PG_waiters, &page->flags);
+		wake_up_page(page, PG_locked);
+	}
 }
 EXPORT_SYMBOL(unlock_page);

mm/filemap.c __lock_page (diff -p would tell us!) · line 574

Quoted 5 lines

mm/filemap.c +2 -0

@@ -568,6 +570,11 @@ __lock_page (diff -p would tell us!)
 {
 	DEFINE_WAIT_BIT(wait, &page->flags, PG_locked);
 
+	set_bit(PG_waiters, &page->flags);
+	if (unlikely(!TestSetPageLocked(page))) {

What happens if another cpu is coming through __lock_page at the same time, did its set_bit, now finds PageLocked, and so proceeds to the __wait_on_bit_lock? But this cpu now clears PG_waiters, so this task's unlock_page won't wake the other?

mm/filemap.c line 580

Quoted 6 lines

mm/filemap.c +3 -0

+		clear_bit(PG_waiters, &page->flags);
+		return;
+	}
 	__wait_on_bit_lock(page_waitqueue(page), &wait, sync_page,
 							TASK_UNINTERRUPTIBLE);
 }

Nick Piggin

2007-05-03 12:43 UTC #54 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 39 lines · Hugh Dickins

Quoted 5 lines · Nick Piggin
The problem is that lock/unlock_page is expensive on powerpc, and if we improve that, we improve more than just the fault handler...
The attached patch gets performance up a bit by avoiding some barriers and some cachelines:
There's a strong whiff of raciness about this...
but I could very easily be wrong.
Quoted 26 lines
Index: linux-2.6/mm/filemap.c
===================================================================
mm/filemap.c +6 -4
@@ -532,11 +532,13 @@
 */
 void fastcall unlock_page(struct page *page)
 {
+	VM_BUG_ON(!PageLocked(page));
 	smp_mb__before_clear_bit();
-	if (!TestClearPageLocked(page))
-		BUG();
-	smp_mb__after_clear_bit(); 
-	wake_up_page(page, PG_locked);
+	ClearPageLocked(page);
+	if (unlikely(test_bit(PG_waiters, &page->flags))) {
+		clear_bit(PG_waiters, &page->flags);
+		wake_up_page(page, PG_locked);
+	}
 }
 EXPORT_SYMBOL(unlock_page);
 
@@ -568,6 +570,11 @@ __lock_page (diff -p would tell us!)
 {
 	DEFINE_WAIT_BIT(wait, &page->flags, PG_locked);
 

Quoted 8 lines

mm/filemap.c line 574
Quoted 2 lines
mm/filemap.c +2 -0
+	set_bit(PG_waiters, &page->flags);
+	if (unlikely(!TestSetPageLocked(page))) {
What happens if another cpu is coming through __lock_page at the same time, did its set_bit, now finds PageLocked, and so proceeds to the __wait_on_bit_lock? But this cpu now clears PG_waiters, so this task's unlock_page won't wake the other?

You're right, we can't clear the bit here. Doubt it mattered much anyway?

BTW. I also forgot an smp_mb__after_clear_bit() before the wake_up_page above... that barrier is in the slow path as well though, so it shouldn't matter either.

Quoted 10 lines

Quoted 8 lines

mm/filemap.c +3 -0

+		clear_bit(PG_waiters, &page->flags);
+		return;
+	}
 	__wait_on_bit_lock(page_waitqueue(page), &wait, sync_page,
 							TASK_UNINTERRUPTIBLE);
 }

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hugh Dickins

2007-05-03 12:58 UTC #55 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 12 lines · Nick Piggin

Quoted 11 lines
mm/filemap.c __lock_page (diff -p would tell us!) · line 574
Quoted 6 lines
mm/filemap.c +2 -0
@@ -568,6 +570,11 @@ __lock_page (diff -p would tell us!)
 {
 DEFINE_WAIT_BIT(wait, &page->flags, PG_locked);
 
+	set_bit(PG_waiters, &page->flags);
+	if (unlikely(!TestSetPageLocked(page))) {
What happens if another cpu is coming through __lock_page at the same time, did its set_bit, now finds PageLocked, and so proceeds to the __wait_on_bit_lock? But this cpu now clears PG_waiters, so this task's unlock_page won't wake the other?

Quoted 1 line

You're right, we can't clear the bit here. Doubt it mattered much anyway?

Ah yes, that's a good easy answer. In fact, just remove this whole test and block (we already tried TestSetPageLocked outside just a short while ago, so this repeat won't often save anything).

Quoted 4 lines

BTW. I also forgot an smp_mb__after_clear_bit() before the wake_up_page above... that barrier is in the slow path as well though, so it shouldn't matter either.

I vaguely wondered how such barriers had managed to dissolve away, but cranking my brain up to think about barriers takes far too long.

Quoted 6 lines

mm/filemap.c line 580

Quoted 6 lines

mm/filemap.c +3 -0

+		clear_bit(PG_waiters, &page->flags);
+		return;
+	}
 __wait_on_bit_lock(page_waitqueue(page), &wait, sync_page,
       TASK_UNINTERRUPTIBLE);
 }

Nick Piggin

2007-05-03 13:08 UTC #56 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 14 lines · Hugh Dickins

Quoted 12 lines · Nick Piggin
Quoted 11 lines
mm/filemap.c __lock_page (diff -p would tell us!) · line 574
Quoted 6 lines
mm/filemap.c +2 -0
@@ -568,6 +570,11 @@ __lock_page (diff -p would tell us!)
 {
 DEFINE_WAIT_BIT(wait, &page->flags, PG_locked);
 
+	set_bit(PG_waiters, &page->flags);
+	if (unlikely(!TestSetPageLocked(page))) {
What happens if another cpu is coming through __lock_page at the same time, did its set_bit, now finds PageLocked, and so proceeds to the __wait_on_bit_lock? But this cpu now clears PG_waiters, so this task's unlock_page won't wake the other?

Quoted 6 lines

Quoted 1 line
You're right, we can't clear the bit here. Doubt it mattered much anyway?
Ah yes, that's a good easy answer. In fact, just remove this whole test and block (we already tried TestSetPageLocked outside just a short while ago, so this repeat won't often save anything).

Yeah, I was getting too clever for my own boots :)

I think the patch has merit though. Unfortunate that it uses another page flag, however it seemed to have quite a bit speedup on unlock_page (probably from both the barriers and an extra random cacheline load (from the hash)).

I guess it has to get good results from more benchmarks...

Quoted 7 lines

Quoted 3 lines
BTW. I also forgot an smp_mb__after_clear_bit() before the wake_up_page above... that barrier is in the slow path as well though, so it shouldn't matter either.
I vaguely wondered how such barriers had managed to dissolve away, but cranking my brain up to think about barriers takes far too long.

That barrier was one too many :)

However I believe the fastpath barrier can go away because the PG_locked operation is depending on the same cacheline as PG_waiters.

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andrew Morton

author maintainer

2007-05-03 16:52 UTC #57 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 4 lines · On Thu, 03 May 2007 11:32:23 +1000 Nick Piggin <[email protected]> wrote:

void fastcall unlock_page(struct page *page)
{
mm/filemap.c +1 -0
+	VM_BUG_ON(!PageLocked(page));
 	smp_mb__before_clear_bit();

mm/filemap.c line 542

Quoted 10 lines

mm/filemap.c +5 -4

-	if (!TestClearPageLocked(page))
-		BUG();
-	smp_mb__after_clear_bit(); 
-	wake_up_page(page, PG_locked);
+	ClearPageLocked(page);
+	if (unlikely(test_bit(PG_waiters, &page->flags))) {
+		clear_bit(PG_waiters, &page->flags);
+		wake_up_page(page, PG_locked);
+	}
 }

Why is that significantly faster than plain old wake_up_page(), which tests waitqueue_active()?

Nick Piggin

2007-05-04 04:16 UTC #58 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 13 lines · Andrew Morton

Quoted 10 lines · On Thu, 03 May 2007 11:32:23 +1000 Nick Piggin <[email protected]> wrote:
void fastcall unlock_page(struct page *page)
{
mm/filemap.c +1 -0
+	VM_BUG_ON(!PageLocked(page));
	smp_mb__before_clear_bit();
mm/filemap.c +2 -4
-	if (!TestClearPageLocked(page))
-		BUG();
-	smp_mb__after_clear_bit(); 
-	wake_up_page(page, PG_locked);
+	ClearPageLocked(page);
+	if (unlikely(test_bit(PG_waiters, &page->flags))) {

Quoted 8 lines

Quoted 4 lines
mm/filemap.c +3 -0
+		clear_bit(PG_waiters, &page->flags);
+		wake_up_page(page, PG_locked);
+	}
}
Why is that significantly faster than plain old wake_up_page(), which tests waitqueue_active()?

Because it needs fewer barriers and doesn't touch random a random hash cacheline in the fastpath.

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Nick Piggin

2007-05-09 12:34 UTC #59 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 12 lines · Hugh Dickins

Quoted 8 lines · Nick Piggin
Quoted 3 lines
But I'm pretty sure (to use your words!) regular truncate was not racy
before: I believe Andrea's sequence count was handling that case fine, without a second unmap_mapping_range.
OK, I think you're right. I _think_ it should also be OK with the lock_page version as well: we should not be able to have any pages after the first unmap_mapping_range call, because of the i_size write. So if we have no pages, there is nothing to 'cow' from.

Quoted 5 lines

I'd be delighted if you can remove those later unmap_mapping_ranges. As I recall, the important thing for the copy pages is to be holding the page lock (or whatever other serialization) on the copied page still while the copy page is inserted into pagetable: that looks to be so in your __do_fault.

Hmm, on second thoughts, I think I was right the first time, and do need the unmap after the pages are truncated. With the lock_page code, after the first unmap, we can get new ptes mapping pages, and subsequently they can be COWed and then the original pte zapped before the truncate loop checks it.

However, I wonder if we can't test mapping_mapped before the spinlock, which would make most truncates cheaper?

Signature · 1 line

SUSE Labs, Novell Inc.

mm/filemap.c +1 -2

Index: linux-2.6/mm/filemap.c
===================================================================
@@ -2579,8 +2579,7 @@
 	if (rw == WRITE) {
 		write_len = iov_length(iov, nr_segs);
 		end = (offset + write_len - 1) >> PAGE_CACHE_SHIFT;
-	       	if (mapping_mapped(mapping))
-			unmap_mapping_range(mapping, offset, write_len, 0);
+		unmap_mapping_range(mapping, offset, write_len, 0);
 	}
 
 	retval = filemap_write_and_wait(mapping);

mm/memory.c +3 -0

Index: linux-2.6/mm/memory.c
===================================================================
@@ -1956,6 +1956,9 @@
 	pgoff_t hba = holebegin >> PAGE_SHIFT;
 	pgoff_t hlen = (holelen + PAGE_SIZE - 1) >> PAGE_SHIFT;
 
+	if (!mapping_mapped(mapping))
+		return;
+
 	/* Check for overflow. */
 	if (sizeof(holelen) > sizeof(hlen)) {
 		long long holeend =

Hugh Dickins

2007-05-09 14:28 UTC #60 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 18 lines · Nick Piggin

Quoted 11 lines · Hugh Dickins
Quoted 8 lines · Nick Piggin
Quoted 3 lines
But I'm pretty sure (to use your words!) regular truncate was not racy
before: I believe Andrea's sequence count was handling that case fine, without a second unmap_mapping_range.
OK, I think you're right. I _think_ it should also be OK with the lock_page version as well: we should not be able to have any pages after the first unmap_mapping_range call, because of the i_size write. So if we have no pages, there is nothing to 'cow' from.
Quoted 5 lines
I'd be delighted if you can remove those later unmap_mapping_ranges. As I recall, the important thing for the copy pages is to be holding the page lock (or whatever other serialization) on the copied page still while the copy page is inserted into pagetable: that looks to be so in your __do_fault.

Quoted 5 lines

Hmm, on second thoughts, I think I was right the first time, and do need the unmap after the pages are truncated. With the lock_page code, after the first unmap, we can get new ptes mapping pages, and subsequently they can be COWed and then the original pte zapped before the truncate loop checks it.

The filesystem (or page cache) allows pages beyond i_size to come in there? That wasn't a problem before, was it? But now it is?

Quoted 3 lines

However, I wonder if we can't test mapping_mapped before the spinlock, which would make most truncates cheaper?

Slightly cheaper, yes, though I doubt it'd be much in comparison with actually doing any work in unmap_mapping_range or truncate_inode_pages. Suspect you'd need a barrier of some kind between the i_size_write and the mapping_mapped test? But that's a change we could have made at any time if we'd bothered, it's not really the issue here.

Hugh

Nick Piggin

2007-05-09 14:45 UTC #61 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 28 lines · Hugh Dickins

Quoted 19 lines · Nick Piggin
Quoted 11 lines · Hugh Dickins
Quoted 8 lines · Nick Piggin
Quoted 3 lines
But I'm pretty sure (to use your words!) regular truncate was not racy
before: I believe Andrea's sequence count was handling that case fine, without a second unmap_mapping_range.
OK, I think you're right. I _think_ it should also be OK with the lock_page version as well: we should not be able to have any pages after the first unmap_mapping_range call, because of the i_size write. So if we have no pages, there is nothing to 'cow' from.
Quoted 5 lines
I'd be delighted if you can remove those later unmap_mapping_ranges. As I recall, the important thing for the copy pages is to be holding the page lock (or whatever other serialization) on the copied page still while the copy page is inserted into pagetable: that looks to be so in your __do_fault.
Quoted 5 lines
Hmm, on second thoughts, I think I was right the first time, and do need the unmap after the pages are truncated. With the lock_page code, after the first unmap, we can get new ptes mapping pages, and subsequently they can be COWed and then the original pte zapped before the truncate loop checks it.

Quoted 2 lines

The filesystem (or page cache) allows pages beyond i_size to come in there? That wasn't a problem before, was it? But now it is?

The filesystem still doesn't, but if i_size is updated after the page is returned, we can have a problem that was previously taken care of with the truncate_count but now isn't.

Quoted 6 lines

Quoted 2 lines
However, I wonder if we can't test mapping_mapped before the spinlock, which would make most truncates cheaper?
Slightly cheaper, yes, though I doubt it'd be much in comparison with actually doing any work in unmap_mapping_range or truncate_inode_pages.

But if we're supposing the common case for truncate is unmapped mappings, then the main cost there will be the locking, which I'm trying to avoid. Hopefully with this patch, most truncate workloads would get faster, even though truncate mapped files is going to be unavoidably slower.

Quoted 2 lines

Suspect you'd need a barrier of some kind between the i_size_write and the mapping_mapped test?

The unmap_mapping_range that runs after the truncate_inode_pages should run in the correct order, I believe.

Quoted 2 lines

But that's a change we could have made at
any time if we'd bothered, it's not really the issue here.

I don't see how you could, because you need to increment truncate_count.

But I believe this is fixing the issue, even if it does so in a peripheral manner, because it avoids the added cost for unmapped files.

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hugh Dickins

2007-05-09 15:38 UTC #62 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 7 lines · Nick Piggin

Quoted 3 lines
The filesystem (or page cache) allows pages beyond i_size to come in there? That wasn't a problem before, was it? But now it is?
The filesystem still doesn't, but if i_size is updated after the page is returned, we can have a problem that was previously taken care of with the truncate_count but now isn't.

But... I thought the page lock was now taking care of that in your scheme? truncate_inode_pages has to wait for the page lock, then it finds the page is mapped and... ahh, it finds the copiee page is not mapped, so doesn't do its own little unmap_mapping_range, and the copied page squeaks through. Drat.

I really think the truncate_count solution worked better, for truncation anyway. There may be persuasive reasons you need the page lock for invalidation: I gave up on trying to understand the required behaviour(s) for invalidation.

So, bring back (the original use of, not my tree marker use of) truncate_count? Hmm, you probably don't want to do that, because there was some pleasure in removing the strange barriers associated with it.

A second unmap_mapping_range is just one line of code - but it sure feels like a defeat to me, calling the whole exercise into question. (But then, you'd be right to say my perfectionism made it impossible for me to come up with any solution to the invalidation issues.)

Quoted 5 lines

Quoted 2 lines
Suspect you'd need a barrier of some kind between the i_size_write and the mapping_mapped test?
The unmap_mapping_range that runs after the truncate_inode_pages should run in the correct order, I believe.

Yes, if there's going to be that backup call, the first won't really need a barrier.

Quoted 4 lines

Quoted 2 lines
But that's a change we could have made at
any time if we'd bothered, it's not really the issue here.
I don't see how you could, because you need to increment truncate_count.

Though indeed we did so, I don't see that we needed to increment truncate_count in that case (nobody could be coming through do_no_page on that file, when there are no mappings of it).

Quoted 2 lines

But I believe this is fixing the issue, even if it does so in a peripheral manner, because it avoids the added cost for unmapped files.

It's a small improvement to your common case, I agree.

Hugh

Nick Piggin

2007-05-09 22:24 UTC #63 lore raw

Re: 2.6.22 -mm merge plans -- vm bugfixes

Quoted 26 lines · Hugh Dickins

Quoted 6 lines · Nick Piggin
Quoted 2 lines
The filesystem (or page cache) allows pages beyond i_size to come in there? That wasn't a problem before, was it? But now it is?
The filesystem still doesn't, but if i_size is updated after the page is returned, we can have a problem that was previously taken care of with the truncate_count but now isn't.
But... I thought the page lock was now taking care of that in your scheme? truncate_inode_pages has to wait for the page lock, then it finds the page is mapped and... ahh, it finds the copiee page is not mapped, so doesn't do its own little unmap_mapping_range, and the copied page squeaks through. Drat.
I really think the truncate_count solution worked better, for truncation anyway. There may be persuasive reasons you need the page lock for invalidation: I gave up on trying to understand the required behaviour(s) for invalidation.
So, bring back (the original use of, not my tree marker use of) truncate_count? Hmm, you probably don't want to do that, because there was some pleasure in removing the strange barriers associated with it.

Quoted 4 lines

A second unmap_mapping_range is just one line of code - but it sure feels like a defeat to me, calling the whole exercise into question. (But then, you'd be right to say my perfectionism made it impossible for me to come up with any solution to the invalidation issues.)

Well we could bring back the truncate_count, but I think that sucks because that's moving work into the page fault handler in order to avoid a bit of work when truncating mapped files.

Quoted 9 lines

Quoted 4 lines
Quoted 2 lines
But that's a change we could have made at
any time if we'd bothered, it's not really the issue here.
I don't see how you could, because you need to increment truncate_count.
Though indeed we did so, I don't see that we needed to increment truncate_count in that case (nobody could be coming through do_no_page on that file, when there are no mappings of it).

Of course :P

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Christoph Hellwig

2007-05-01 08:46 UTC #64 lore raw

pcmcia ioctl removal

Quoted 1 line

pcmcia-delete-obsolete-pcmcia_ioctl-feature.patch

...

Quoted 1 line

Dominik is busy. Will probably re-review and send these direct to Linus.

The patch above is the removal of cardmgr support. While I'd love to see this cruft gone it definitively needs maintainer judgement on whether they time has come that no one relies on cardmgr anymore.

Russell King

2007-05-01 08:56 UTC #65 lore raw

Re: pcmcia ioctl removal

Quoted 9 lines · Christoph Hellwig

Quoted 1 line
pcmcia-delete-obsolete-pcmcia_ioctl-feature.patch
...
Quoted 1 line
Dominik is busy. Will probably re-review and send these direct to Linus.
The patch above is the removal of cardmgr support. While I'd love to see this cruft gone it definitively needs maintainer judgement on whether they time has come that no one relies on cardmgr anymore.

And I still run and use a platform where the GUI issues cardmgr ioctls. A recent kernel upgrade (from 2.6.9ish to something more recent) broke the "eject" GUI button applet due to the fscking with the cardmgr ioctls, and it thinks the wireless card is always plugged in (and therefore the signal strength meter remains.)

With all the ioctls gone I'll probably loose the signal strength meter.

And no, I don't have the resources (read: code) to fix and rebuild userspace since I didn't snarf them when the CVS server was alive a few years back.

That's the problem with API changes - things always break.

Signature · 9 lines

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Willy Tarreau

2007-05-01 08:57 UTC #66 lore raw

Re: pcmcia ioctl removal

Hi Christoph,

Quoted 9 lines · Christoph Hellwig

Quoted 1 line
pcmcia-delete-obsolete-pcmcia_ioctl-feature.patch
...
Quoted 1 line
Dominik is busy. Will probably re-review and send these direct to Linus.
The patch above is the removal of cardmgr support. While I'd love to see this cruft gone it definitively needs maintainer judgement on whether they time has come that no one relies on cardmgr anymore.

Well, I've not followed evolutions in this area for a long time. Here's what I get on my notebook :

willy@wtap:~$ uname -r
2.6.20-wt3-wtap
willy@wtap:~$ ps auxw|grep card   
root      1216  0.0  0.0     0    0 ?        S<   Apr28   0:00 [pccardd]
root      1221  0.0  0.0     0    0 ?        S<   Apr28   0:00 [pccardd]
root      1244  0.0  0.0     0    0 ?        S<   Apr28   0:00 [pccardd]
root      1251  0.0  0.0     0    0 ?        Ss   Apr28   0:00 /sbin/cardmgr

What's the new recommended way of using PCMCIA cards when cardmgr is gone ?

Thanks,
Willy

Andrew Morton

author maintainer

2007-05-01 09:08 UTC #67 lore raw

Re: pcmcia ioctl removal

Quoted 16 lines · On Tue, 1 May 2007 10:57:10 +0200 Willy Tarreau <[email protected]> wrote:

Hi Christoph,
Quoted 9 lines · Christoph Hellwig
Quoted 1 line
pcmcia-delete-obsolete-pcmcia_ioctl-feature.patch
...
Quoted 1 line
Dominik is busy. Will probably re-review and send these direct to Linus.
The patch above is the removal of cardmgr support. While I'd love to see this cruft gone it definitively needs maintainer judgement on whether they time has come that no one relies on cardmgr anymore.
Well, I've not followed evolutions in this area for a long time. Here's what I get on my notebook :

Quoted 8 lines

willy@wtap:~$ uname -r
2.6.20-wt3-wtap
willy@wtap:~$ ps auxw|grep card   
root      1216  0.0  0.0     0    0 ?        S<   Apr28   0:00 [pccardd]
root      1221  0.0  0.0     0    0 ?        S<   Apr28   0:00 [pccardd]
root      1244  0.0  0.0     0    0 ?        S<   Apr28   0:00 [pccardd]
root      1251  0.0  0.0     0    0 ?        Ss   Apr28   0:00 /sbin/cardmgr

Yes, that seems premature. feature-removal.txt is pretty useless for getting poeple off old tools. If we're ever to make this migration we'll need loud and scary printks coming out of the kernel. Probably it'll take another year or two to get there *once* we've done that.

Adrian Bunk

2007-05-01 14:46 UTC #68 lore raw

Re: pcmcia ioctl removal

Quoted 27 lines · Andrew Morton

Quoted 16 lines · On Tue, 1 May 2007 10:57:10 +0200 Willy Tarreau <[email protected]> wrote:
Hi Christoph,
Quoted 9 lines · Christoph Hellwig
Quoted 1 line
pcmcia-delete-obsolete-pcmcia_ioctl-feature.patch
...
Quoted 1 line
Dominik is busy. Will probably re-review and send these direct to Linus.
The patch above is the removal of cardmgr support. While I'd love to see this cruft gone it definitively needs maintainer judgement on whether they time has come that no one relies on cardmgr anymore.
Well, I've not followed evolutions in this area for a long time. Here's what I get on my notebook :
Quoted 8 lines
willy@wtap:~$ uname -r
2.6.20-wt3-wtap
willy@wtap:~$ ps auxw|grep card   
root      1216  0.0  0.0     0    0 ?        S<   Apr28   0:00 [pccardd]
root      1221  0.0  0.0     0    0 ?        S<   Apr28   0:00 [pccardd]
root      1244  0.0  0.0     0    0 ?        S<   Apr28   0:00 [pccardd]
root      1251  0.0  0.0     0    0 ?        Ss   Apr28   0:00 /sbin/cardmgr

Quoted 4 lines

Yes, that seems premature. feature-removal.txt is pretty useless for getting poeple off old tools. If we're ever to make this migration we'll need loud and scary printks coming out of the kernel. Probably it'll take another year or two to get there *once* we've done that.

You already said the same two years ago, and you forwarded a patch implementing exactly this nearly two years ago:

commit c352ec8ab87b065cd2edda171811f49ac7d0d5cd

Author: Dominik Brodowski <[email protected]>
Date:   Tue Sep 13 01:25:03 2005 -0700

[PATCH] pcmcia: warn on IOCTL usage

More visible user information of scheduled feature removal.

Signed-off-by Dominik Brodowski <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

Signed-off-by Linus Torvalds <[email protected]>

drivers/pcmcia/pcmcia_ioctl.c +12 -0

index 39ba640..80969f7 100644
@@ -376,6 +376,7 @@ static int ds_open(struct inode *inode, struct file *file)
     socket_t i = iminor(inode);
     struct pcmcia_socket *s;
     user_info_t *user;
+    static int warning_printed = 0;
 
     ds_dbg(0, "ds_open(socket %d)\n", i);

@@ -407,6 +408,17 @@ static int ds_open(struct inode *inode, struct file *file)
     s->user = user;
     file->private_data = user;
 
+    if (!warning_printed) {
+	    printk(KERN_INFO "pcmcia: Detected deprecated PCMCIA ioctl "
+			"usage.\n");
+	    printk(KERN_INFO "pcmcia: This interface will soon be removed from "
+			"the kernel; please expect breakage unless you upgrade "
+			"to new tools.\n");
+	    printk(KERN_INFO "pcmcia: see http://www.kernel.org/pub/linux/"
+			"utils/kernel/pcmcia/pcmcia.html for details.\n");
+	    warning_printed = 1;
+    }
+
     if (s->pcmcia_state.present)
 	queue_event(user, CS_EVENT_CARD_INSERTION);
     return 0;

cu
Adrian

Signature · 10 lines

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Robert P. J. Day

2007-05-01 09:16 UTC #69 lore raw

Re: pcmcia ioctl removal

In reply to Willy Tarreau

Quoted 9 lines · Christoph Hellwig

Quoted 1 line
pcmcia-delete-obsolete-pcmcia_ioctl-feature.patch
...
Quoted 1 line
Dominik is busy. Will probably re-review and send these direct to Linus.
The patch above is the removal of cardmgr support. While I'd love to see this cruft gone it definitively needs maintainer judgement on whether they time has come that no one relies on cardmgr anymore.

since i was the one who submitted the original patch to remove that stuff, let me make an observation.

when i submitted a patch to remove, for instance, the traffic shaper since it's clearly obsolete, i was told -- in no uncertain terms -- that that couldn't be done since there had been no warning about its impending removal.

fair enough, i can accept that.

on the other hand, the features removal file contains the following:

...

What:   PCMCIA control ioctl (needed for pcmcia-cs [cardmgr, cardctl])
When:   November 2005

...

in other words, the PCMCIA ioctl feature *has* been listed as obsolete for quite some time, and is already a *year and a half* overdue for removal.

in short, it's annoying to take the position that stuff can't be deleted without warning, then turn around and be reluctant to remove stuff for which *more than ample warning* has already been given. doing that just makes a joke of the features removal file, and makes you wonder what its purpose is in the first place.

a little consistency would be nice here, don't you think?

rday

Signature · 13 lines

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry
Waterloo, Ontario, CANADA

http://fsdev.net/wiki/index.php?title=Main_Page
========================================================================

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Willy Tarreau

2007-05-01 09:44 UTC #70 lore raw

Re: pcmcia ioctl removal

Quoted 33 lines · Robert P. J. Day

Quoted 9 lines · Christoph Hellwig
Quoted 1 line
pcmcia-delete-obsolete-pcmcia_ioctl-feature.patch
...
Quoted 1 line
Dominik is busy. Will probably re-review and send these direct to Linus.
The patch above is the removal of cardmgr support. While I'd love to see this cruft gone it definitively needs maintainer judgement on whether they time has come that no one relies on cardmgr anymore.
since i was the one who submitted the original patch to remove that stuff, let me make an observation.
when i submitted a patch to remove, for instance, the traffic shaper since it's clearly obsolete, i was told -- in no uncertain terms -- that that couldn't be done since there had been no warning about its impending removal.
fair enough, i can accept that.
on the other hand, the features removal file contains the following:
...
What:   PCMCIA control ioctl (needed for pcmcia-cs [cardmgr, cardctl])
When:   November 2005
...
in other words, the PCMCIA ioctl feature *has* been listed as obsolete for quite some time, and is already a *year and a half* overdue for removal.

Quoted 7 lines

in short, it's annoying to take the position that stuff can't be deleted without warning, then turn around and be reluctant to remove stuff for which *more than ample warning* has already been given. doing that just makes a joke of the features removal file, and makes you wonder what its purpose is in the first place.
a little consistency would be nice here, don't you think?

No, it just shows how useless this file is. What is needed is a big warning during usage, not a file that nobody reads. Facts are :

- 90% of people here do not even know that this file exists
- 80% of the people who know about it do not consult it on a regular basis
- 80% of those who consult it on a regular basis are not concerned
- 75% of statistics are invented

=> only 20% of 20% of 10% of those who read LKML know that one feature
they are concerned about will soon be removed = 0.4% of LKML readers.

If you put a warning in kernel messages (as I've seen for a long time about tcpdump using obsolete AF_PACKET), close to 100% of the users of the obsolete code who are likely to change their kernels will notice it.

I'm sorry for your patch which may get delayed a lot. You would spend fewer time stuffing warnings in areas affected by scheduled removal.

BTW, I'm not even against the end of cardmgr support, it's just that I don't know what the alternative is, and I suspect that many users do not either. A big warning would have brought them to google who would have provided them with suggestions for alternatives.

Willy

Robert P. J. Day

2007-05-01 10:16 UTC #71 lore raw

Re: pcmcia ioctl removal

Quoted 1 line · Willy Tarreau

On Tue, May 01, 2007 at 05:16:13AM -0400, Robert P. J. Day wrote:

... snip ...

Quoted 10 lines

Quoted 10 lines
in other words, the PCMCIA ioctl feature *has* been listed as obsolete for quite some time, and is already a *year and a half* overdue for removal.
in short, it's annoying to take the position that stuff can't be deleted without warning, then turn around and be reluctant to remove stuff for which *more than ample warning* has already been given. doing that just makes a joke of the features removal file, and makes you wonder what its purpose is in the first place.

Quoted 3 lines

Quoted 1 line
a little consistency would be nice here, don't you think?
No, it just shows how useless this file is.

agreed. it's mildly entertaining to have watched this raging discussion over the last few days regarding bugs and emails and bugzilla and adrian's regressions, while the one feature that's meant to track aging and removable kernel features is essentially valueless, and no one seems to care.

Quoted 2 lines

What is needed is a big warning during usage, not a file that nobody reads.

agreed there as well. but short of that, it would still be nice if people took a minute, perused the feature removal file, and at least brought it up-to-date. if it's going to have any value, then:

1) all proposed removal dates should be reviewed to make sure they're still meaningful,

2) stuff that's overdue for removal should be either removed, or have its expiry date brought forward, and

3) stuff in the kernel tree that is understood to be obsolete or nearly so should have an entry added to that file, so that the clock can at least *start* ticking for that stuff, and you can at least say you *tried* to warn current users.

as a start, i posted last month the results of running the simple command:

  $ grep -iw obsolete $(find . -name Kconfig\*)

and some of what was printed is clearly misleading. (don't worry, tilman -- we're not going to reopen that whole isdn4linux thing. :-)

i mean, what of the following is actually obsolete:

* traffic policing
* IP6 Userspace queueing via NETLINK
* IP Userspace queueing via NETLINK
* ebt: ulog support
* Traffic Shaper

and so on (and there's that legacy PM thing as well).

Quoted 1 line

I'm sorry for your patch which may get delayed a lot.

obviously, leaving stuff like that in the kernel doesn't actually *hurt* anything but, yeah, it's a tad annoying to invest a few minutes to do some janitor work based on what should be killable, submit the patch, then have people freak out about how that is still an essential feature.

bottom line: if you want janitor folks to help out with cleanup, make sure they know what can legitimately be cleaned, and stop wasting peoples' time.

rday

p.s. now if there were only a way to, say, tag various kernel features as "obsolete" or "deprecated" ... :-)

Signature · 13 lines

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry
Waterloo, Ontario, CANADA

http://fsdev.net/wiki/index.php?title=Main_Page
========================================================================

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Gabriel C

2007-05-01 10:28 UTC #72 lore raw

Re: pcmcia ioctl removal

Quoted 54 lines · Willy Tarreau

Quoted 35 lines · Robert P. J. Day
Quoted 11 lines · Christoph Hellwig
Quoted 2 lines
pcmcia-delete-obsolete-pcmcia_ioctl-feature.patch
...
Quoted 2 lines
Dominik is busy. Will probably re-review and send these direct to Linus.
The patch above is the removal of cardmgr support. While I'd love to see this cruft gone it definitively needs maintainer judgement on whether they time has come that no one relies on cardmgr anymore.
since i was the one who submitted the original patch to remove that stuff, let me make an observation.
when i submitted a patch to remove, for instance, the traffic shaper since it's clearly obsolete, i was told -- in no uncertain terms -- that that couldn't be done since there had been no warning about its impending removal.
fair enough, i can accept that.
on the other hand, the features removal file contains the following:
...
What:   PCMCIA control ioctl (needed for pcmcia-cs [cardmgr, cardctl])
When:   November 2005
...
in other words, the PCMCIA ioctl feature *has* been listed as obsolete for quite some time, and is already a *year and a half* overdue for removal.
Quoted 8 lines
in short, it's annoying to take the position that stuff can't be deleted without warning, then turn around and be reluctant to remove stuff for which *more than ample warning* has already been given. doing that just makes a joke of the features removal file, and makes you wonder what its purpose is in the first place.
a little consistency would be nice here, don't you think?
No, it just shows how useless this file is. What is needed is a big warning during usage, not a file that nobody reads. Facts are :
- 90% of people here do not even know that this file exists
- 80% of the people who know about it do not consult it on a regular basis
- 80% of those who consult it on a regular basis are not concerned
- 75% of statistics are invented

Quoted 9 lines

=> only 20% of 20% of 10% of those who read LKML know that one feature
they are concerned about will soon be removed = 0.4% of LKML readers.
If you put a warning in kernel messages (as I've seen for a long time about tcpdump using obsolete AF_PACKET), close to 100% of the users of the obsolete code who are likely to change their kernels will notice it.

Hmm ? There is already a fat warning in dmesg for a long time now.

snip

...

pcmcia: Detected deprecated PCMCIA ioctl usage from process: discover.
pcmcia: This interface will soon be removed from the kernel; please expect breakage unless you upgrade to new tools.
pcmcia: see http://www.kernel.org/pub/linux/utils/kernel/pcmcia/pcmcia.html for details.

...

Quoted 12 lines

I'm sorry for your patch which may get delayed a lot. You would spend fewer time stuffing warnings in areas affected by scheduled removal.
BTW, I'm not even against the end of cardmgr support, it's just that I don't know what the alternative is, and I suspect that many users do not either. A big warning would have brought them to google who would have provided them with suggestions for alternatives.
Willy

Regards,

Gabriel

Willy Tarreau

2007-05-01 10:52 UTC #73 lore raw

Re: pcmcia ioctl removal

Quoted 74 lines · Gabriel C

Quoted 56 lines · Willy Tarreau
Quoted 36 lines · Robert P. J. Day
Quoted 5 lines · Christoph Hellwig
Quoted 2 lines
pcmcia-delete-obsolete-pcmcia_ioctl-feature.patch
...
Quoted 7 lines
Quoted 3 lines
Dominik is busy. Will probably re-review and send these direct to Linus.
The patch above is the removal of cardmgr support. While I'd love to see this cruft gone it definitively needs maintainer judgement on whether they time has come that no one relies on cardmgr anymore.
since i was the one who submitted the original patch to remove that stuff, let me make an observation.
when i submitted a patch to remove, for instance, the traffic shaper since it's clearly obsolete, i was told -- in no uncertain terms -- that that couldn't be done since there had been no warning about its impending removal.
fair enough, i can accept that.
on the other hand, the features removal file contains the following:
...
What:   PCMCIA control ioctl (needed for pcmcia-cs [cardmgr, cardctl])
When:   November 2005
...
in other words, the PCMCIA ioctl feature *has* been listed as obsolete for quite some time, and is already a *year and a half* overdue for removal.
Quoted 8 lines
in short, it's annoying to take the position that stuff can't be deleted without warning, then turn around and be reluctant to remove stuff for which *more than ample warning* has already been given. doing that just makes a joke of the features removal file, and makes you wonder what its purpose is in the first place.
a little consistency would be nice here, don't you think?
No, it just shows how useless this file is. What is needed is a big warning during usage, not a file that nobody reads. Facts are :
- 90% of people here do not even know that this file exists
- 80% of the people who know about it do not consult it on a regular basis
- 80% of those who consult it on a regular basis are not concerned
- 75% of statistics are invented
Quoted 9 lines
=> only 20% of 20% of 10% of those who read LKML know that one feature
they are concerned about will soon be removed = 0.4% of LKML readers.
If you put a warning in kernel messages (as I've seen for a long time about tcpdump using obsolete AF_PACKET), close to 100% of the users of the obsolete code who are likely to change their kernels will notice it.
Hmm ? There is already a fat warning in dmesg for a long time now.
snip
...

Quoted 7 lines

pcmcia: Detected deprecated PCMCIA ioctl usage from process: discover.
pcmcia: This interface will soon be removed from the kernel; please expect breakage unless you upgrade to new tools.
pcmcia: see
http://www.kernel.org/pub/linux/utils/kernel/pcmcia/pcmcia.html for details.
...

Oh you're terribly right ! I have grepped my logs and found it lying there too in the middle of the memory probe messages and the IO probe ones. I never noticed it. I know it's my fault, but I think that "fat warning" is not really what could characterize it though, because of the context verbose around it :

pcmcia: parent PCI bridge Memory window: 0x90000000 - 0x903fffff
pcmcia: parent PCI bridge Memory window: 0x30000000 - 0x3dffffff
pccard: PCMCIA card inserted into slot 0
cs: memory probe 0x90000000-0x903fffff: excluding 0x90000000-0x9003ffff 0x90080000-0x900bffff 0x90100000-0x9013ffff 0x90180000-0x901bffff 0x90200000-0x9023ffff 0x90280000-0x902bffff 0x90300000-0x9033ffff 0x90380000-0x903bffff
pcmcia: registering new device pcmcia0.0
pcmcia: Detected deprecated PCMCIA ioctl usage from process: cardmgr.
pcmcia: This interface will soon be removed from the kernel; please expect breakage unless you upgrade to new tools.
pcmcia: see http://www.kernel.org/pub/linux/utils/kernel/pcmcia/pcmcia.html for details.
cs: IO port probe 0xc00-0xcff: clean.
cs: IO port probe 0x820-0x8ff: clean.
cs: IO port probe 0x800-0x80f: clean.
cs: IO port probe 0x3e0-0x4ff: clean.
cs: IO port probe 0x100-0x3af: excluding 0x140-0x14f 0x378-0x37f
cs: IO port probe 0xa00-0xaff: clean.

Now I have the URL for the details ;-)

BTW, I thing we should standardize on some formating to display messages about deprecated/obsolete code. Maybe something like this would be more noticeable :

cs: memory probe 0x90000000-0x903fffff: excluding 0x90000000-0x9003ffff 0x90080000-0x900bffff 0x90100000-0x9013ffff 0x90180000-0x901bffff 0x90200000-0x9023ffff 0x90280000-0x902bffff 0x90300000-0x9033ffff 0x90380000-0x903bffff
pcmcia: registering new device pcmcia0.0
WARNING !!! Detected DEPRECATED PCMCIA ioctl usage from process: cardmgr.
WARNING !!! This process may stop working past November 2005.
WARNING !!! see http://www.kernel.org/pub/linux/utils/kernel/pcmcia/pcmcia.html for details.
cs: IO port probe 0xc00-0xcff: clean.

Regards,
Willy

Jan Engelhardt

2007-05-01 10:12 UTC #74 lore raw

Re: pcmcia ioctl removal

In reply to Robert P. J. Day

Quoted 12 lines · Robert P. J. Day

on the other hand, the features removal file contains the following:
...
What:   PCMCIA control ioctl (needed for pcmcia-cs [cardmgr, cardctl])
When:   November 2005
...
in other words, the PCMCIA ioctl feature *has* been listed as obsolete for quite some time, and is already a *year and a half* overdue for removal.

Quoted 7 lines

in short, it's annoying to take the position that stuff can't be deleted without warning, then turn around and be reluctant to remove stuff for which *more than ample warning* has already been given. doing that just makes a joke of the features removal file, and makes you wonder what its purpose is in the first place.
a little consistency would be nice here, don't you think?

I think this could raise their attention...

init/Makefile
obj-y += obsolete.o

init/obsolete.c:
static __init int obsolete_init(void)

{
	printk("\e[1;31m""

The following stuff is gonna get removed \e[5;37m SOON: \e[0m
- cardmgr
- foobar
- bweebol

");
	schedule_timeout(3 * HZ);
	return;
}

static __exit void obsolete_exit(void) {}

Jan

Signature · 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Willy Tarreau

2007-05-01 11:00 UTC #75 lore raw

Re: pcmcia ioctl removal

Quoted 37 lines · Jan Engelhardt

Quoted 12 lines · Robert P. J. Day
on the other hand, the features removal file contains the following:
...
What:   PCMCIA control ioctl (needed for pcmcia-cs [cardmgr, cardctl])
When:   November 2005
...
in other words, the PCMCIA ioctl feature *has* been listed as obsolete for quite some time, and is already a *year and a half* overdue for removal.
Quoted 7 lines
in short, it's annoying to take the position that stuff can't be deleted without warning, then turn around and be reluctant to remove stuff for which *more than ample warning* has already been given. doing that just makes a joke of the features removal file, and makes you wonder what its purpose is in the first place.
a little consistency would be nice here, don't you think?
I think this could raise their attention...
init/Makefile
obj-y += obsolete.o
init/obsolete.c:
static __init int obsolete_init(void)
{
	printk("\e[1;31m""
The following stuff is gonna get removed \e[5;37m SOON: \e[0m
- cardmgr
- foobar
- bweebol

Quoted 6 lines

");
	schedule_timeout(3 * HZ);
	return;
}
static __exit void obsolete_exit(void) {}

There's something I like here : the fact that all features are centralized and not hidden in the noise. Clearly we need some standard inside the kernel to manage obsolete code as well as we currently do by hand.

Willy

Konstantin Münning

2007-05-01 12:06 UTC #76 lore raw

Re: pcmcia ioctl removal

Quoted 40 lines · Willy Tarreau

Quoted 39 lines · Jan Engelhardt
Quoted 11 lines · Robert P. J. Day
on the other hand, the features removal file contains the following:
...
What:   PCMCIA control ioctl (needed for pcmcia-cs [cardmgr, cardctl])
When:   November 2005
...
in other words, the PCMCIA ioctl feature *has* been listed as obsolete for quite some time, and is already a *year and a half* overdue for removal.
Quoted 7 lines
in short, it's annoying to take the position that stuff can't be deleted without warning, then turn around and be reluctant to remove stuff for which *more than ample warning* has already been given. doing that just makes a joke of the features removal file, and makes you wonder what its purpose is in the first place.
a little consistency would be nice here, don't you think?
I think this could raise their attention...
init/Makefile
obj-y += obsolete.o
init/obsolete.c:
static __init int obsolete_init(void)
{
	printk("\e[1;31m""
The following stuff is gonna get removed \e[5;37m SOON: \e[0m
- cardmgr
- foobar
- bweebol
");
	schedule_timeout(3 * HZ);
	return;
}

Quoted 7 lines

Quoted 1 line
static __exit void obsolete_exit(void) {}
There's something I like here : the fact that all features are centralized and not hidden in the noise. Clearly we need some standard inside the kernel to manage obsolete code as well as we currently do by hand.
Willy

What about something like the tainted flag which status can be displayed
easily? And even better when a list of the used obsolete features can
be displayed as well on request? This way you don't need to search the logs. A standardized obsolete function like the one above could do all the job.

Just my 2 cents.

Signature · 7 lines

Konstantin Munning

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Rogan Dawes

2007-05-01 13:56 UTC #77 lore raw

Re: pcmcia ioctl removal

In reply to Willy Tarreau

Quoted 40 lines · Willy Tarreau

Quoted 39 lines · Jan Engelhardt
Quoted 11 lines · Robert P. J. Day
on the other hand, the features removal file contains the following:
...
What:   PCMCIA control ioctl (needed for pcmcia-cs [cardmgr, cardctl])
When:   November 2005
...
in other words, the PCMCIA ioctl feature *has* been listed as obsolete for quite some time, and is already a *year and a half* overdue for removal.
Quoted 7 lines
in short, it's annoying to take the position that stuff can't be deleted without warning, then turn around and be reluctant to remove stuff for which *more than ample warning* has already been given. doing that just makes a joke of the features removal file, and makes you wonder what its purpose is in the first place.
a little consistency would be nice here, don't you think?
I think this could raise their attention...
init/Makefile
obj-y += obsolete.o
init/obsolete.c:
static __init int obsolete_init(void)
{
	printk("\e[1;31m""
The following stuff is gonna get removed \e[5;37m SOON: \e[0m
- cardmgr
- foobar
- bweebol
");
	schedule_timeout(3 * HZ);
	return;
}

Quoted 7 lines

Quoted 1 line
static __exit void obsolete_exit(void) {}
There's something I like here : the fact that all features are centralized and not hidden in the noise. Clearly we need some standard inside the kernel to manage obsolete code as well as we currently do by hand.
Willy

The difference between this function and the PCAP/TCPDUMP warning is that the warning only showed up when the obsolete functionality was actually used.

Maybe a mechanism to automatically increase the severity of reporting as the removal date approaches would be an idea? i.e. for each new kernel that you build leading up the the removal date, a severity is calculated based on the time until official removal, and then, depending on the severity, the message can be logged in various ways.

Rogan

Russell King

2007-05-01 19:10 UTC #78 lore raw

Re: pcmcia ioctl removal

In reply to Jan Engelhardt

Quoted 10 lines · Jan Engelhardt

init/obsolete.c:
static __init int obsolete_init(void)
{
	printk("\e[1;31m""
The following stuff is gonna get removed \e[5;37m SOON: \e[0m
- cardmgr
- foobar
- bweebol

Quoted 4 lines

");
	schedule_timeout(3 * HZ);
	return;
}

The kernel console isn't VT102 compatible. It doesn't understand any escape codes, at all. Neither does sysklogd. So the above will just end up as rubbish on your console.

Signature · 9 lines

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Jan Engelhardt

2007-05-01 20:41 UTC #79 lore raw

Re: pcmcia ioctl removal

Quoted 11 lines · Russell King

Quoted 10 lines · Jan Engelhardt
init/obsolete.c:
static __init int obsolete_init(void)
{
	printk("\e[1;31m""
The following stuff is gonna get removed \e[5;37m SOON: \e[0m
- cardmgr
- foobar
- bweebol

Quoted 8 lines

Quoted 4 lines
");
	schedule_timeout(3 * HZ);
	return;
}
The kernel console isn't VT102 compatible. It doesn't understand any escape codes, at all. Neither does sysklogd. So the above will just end up as rubbish on your console.

It will (should) at least show up as nicely as in the C source code. With escape codes, but largely readable.
If someone knows how to directly spew it to tty0 (the current active console - most likely tty1), the better.
Anyway, I just wanted to point out how to really highlight it for the user to see. Although then there would be the distros who obscure it with funky bootsplash screens. But hopefully, their users would not need to care too much for old stuff (gets updated through the distro's update mechanism)

Jan

Signature · 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Pavel Machek

2007-05-09 12:54 UTC #80 lore raw

Re: pcmcia ioctl removal

Hi!

Quoted 9 lines

Quoted 1 line
pcmcia-delete-obsolete-pcmcia_ioctl-feature.patch
...
Quoted 1 line
Dominik is busy. Will probably re-review and send these direct to Linus.
The patch above is the removal of cardmgr support. While I'd love to see this cruft gone it definitively needs maintainer judgement on whether they time has come that no one relies on cardmgr anymore.

I remember needing cardmgr few months ago on sa-1100 arm system. I'm not sure this is obsolete-enough to kill.

Pavel

Signature · 8 lines

(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Robert P. J. Day

2007-05-09 13:00 UTC #81 lore raw

Re: pcmcia ioctl removal

Quoted 8 lines · Pavel Machek

Hi!
Quoted 6 lines
Quoted 1 line
pcmcia-delete-obsolete-pcmcia_ioctl-feature.patch
...
Quoted 1 line
Dominik is busy. Will probably re-review and send these direct to Linus.

Quoted 7 lines

Quoted 4 lines
The patch above is the removal of cardmgr support. While I'd love to see this cruft gone it definitively needs maintainer judgement on whether they time has come that no one relies on cardmgr anymore.
I remember needing cardmgr few months ago on sa-1100 arm system. I'm not sure this is obsolete-enough to kill.

in that case, someone really should update feature-removal-schedule.txt, which currently reads:

What:   PCMCIA control ioctl (needed for pcmcia-cs [cardmgr, cardctl])
When:   November 2005

...

rday

Signature · 12 lines

========================================================================
Robert P. J. Day Linux Consulting, Training and Annoying Kernel
Pedantry Waterloo, Ontario, CANADA

http://fsdev.net/wiki/index.php?title=Main_Page
========================================================================

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Adrian Bunk

2007-05-09 13:03 UTC #82 lore raw

Re: pcmcia ioctl removal

In reply to Pavel Machek

Quoted 6 lines · Pavel Machek

Hi!
Quoted 4 lines
Quoted 1 line
pcmcia-delete-obsolete-pcmcia_ioctl-feature.patch
...

Quoted 8 lines

Quoted 5 lines
Quoted 1 line
Dominik is busy. Will probably re-review and send these direct to Linus.
The patch above is the removal of cardmgr support. While I'd love to see this cruft gone it definitively needs maintainer judgement on whether they time has come that no one relies on cardmgr anymore.
I remember needing cardmgr few months ago on sa-1100 arm system. I'm not sure this is obsolete-enough to kill.

Why didn't pcmciautils work?

Quoted 1 line

Pavel

cu
Adrian

Signature · 10 lines

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Romano Giannetti

2007-05-09 19:11 UTC #83 lore raw

Re: pcmcia ioctl removal

Quoted 7 lines · Adrian Bunk

On Wed, May 09, 2007 at 12:54:16PM +0000, Pavel Machek wrote:
relies on cardmgr anymore.
Quoted 3 lines
I remember needing cardmgr few months ago on sa-1100 arm system. I'm not sure this is obsolete-enough to kill.
Why didn't pcmciautils work?

I have had a problem until 2.6.20 was out with pcmciautils (it did not recognise the second function of multi-functions pcmcia cards that needed a firmware .cis file), and the only way to use it was with cardmgr, way after Nov 2005 :-).

Now it is solved (modulo that sometime the pcmcia modem is ttyS1, sometime ttyS2, but that's another history --- and probably my fault). But I wonder if similar problems are hidden away... what about put the ioctls under a normally-disabled option and let a kernel out with it?

Romano

--
La presente comunicaciA3n tiene carA!cter confidencial y es para el exclusivo uso del destinatario indicado en la misma. Si Ud. no es el destinatario indicado, le informamos que cualquier forma de distribuciA3n, reproducciA3n o uso de esta comunicaciA3n y/o de la informaciA3n contenida en la misma estA!n estrictamente prohibidos por la ley. Si Ud. ha recibido esta comunicaciA3n por error, por favor, notifA-quelo inmediatamente al remitente contestando a este mensaje y proceda a continuaciA3n a destruirlo. Gracias por su colaboraciA3n.

This communication contains confidential information. It is for the exclusive use of the intended addressee. If you are not the intended addressee, please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited by law. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this message. Thank you for your cooperation.

Adrian Bunk

2007-05-10 12:40 UTC #84 lore raw

Re: pcmcia ioctl removal

Quoted 14 lines · Romano Giannetti

Quoted 7 lines · Adrian Bunk
On Wed, May 09, 2007 at 12:54:16PM +0000, Pavel Machek wrote:
relies on cardmgr anymore.
Quoted 3 lines
I remember needing cardmgr few months ago on sa-1100 arm system. I'm not sure this is obsolete-enough to kill.
Why didn't pcmciautils work?
I have had a problem until 2.6.20 was out with pcmciautils (it did not recognise the second function of multi-functions pcmcia cards that needed a firmware .cis file), and the only way to use it was with cardmgr, way after Nov 2005 :-).

Quoted 4 lines

Now it is solved (modulo that sometime the pcmcia modem is ttyS1, sometime ttyS2, but that's another history --- and probably my fault). But I wonder if similar problems are hidden away... what about put the ioctls under a normally-disabled option and let a kernel out with it?

It already prints a runtime warning to the user since 2005. And people won't notice a changed default when using "make oldconfig".

Are there any known known regressions left?
Otherwise, the best way for getting problem reports for pcmciautils is to remove the ioctl (that's an experience from similar cases in other parts of the kernel)...

Quoted 1 line

Romano

cu
Adrian

Signature · 10 lines

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Christoph Hellwig

2007-05-01 08:48 UTC #85 lore raw

pci hotplug patches

Quoted 11 lines

fix-gregkh-pci-pci-remove-the-broken-pci_multithread_probe-option.patch remove-pci_dac_dma_-apis.patch round_up-macro-cleanup-in-drivers-pci.patch pcie-remove-spin_lock_unlocked.patch cpqphp-partially-convert-to-use-the-kthread-api.patch ibmphp-partially-convert-to-use-the-kthreads-api.patch cpci_hotplug-partially-convert-to-use-the-kthread-api.patch msi-fix-arm-compile.patch support-pci-mcfg-space-on-intel-i915-bridges.patch pci-syscallc-switch-to-refcounting-api.patch

Quoted 3 lines

Stuff to (various levels of re-)send to Greg for the PCI tree. I'll probably drop the kthread patches as they seemed a bit half-baked and I've lost track of which ones have which levels of baking.

All the partially kthread conversion were superceed with full conversion from me. I've only got feedback from the cpci maintainer, and he acked my patch together with a simple fix from him.

Greg KH

2007-05-02 03:57 UTC #86 lore raw

Re: pci hotplug patches

Quoted 11 lines · Christoph Hellwig

Quoted 11 lines
fix-gregkh-pci-pci-remove-the-broken-pci_multithread_probe-option.patch remove-pci_dac_dma_-apis.patch round_up-macro-cleanup-in-drivers-pci.patch pcie-remove-spin_lock_unlocked.patch cpqphp-partially-convert-to-use-the-kthread-api.patch ibmphp-partially-convert-to-use-the-kthreads-api.patch cpci_hotplug-partially-convert-to-use-the-kthread-api.patch msi-fix-arm-compile.patch support-pci-mcfg-space-on-intel-i915-bridges.patch pci-syscallc-switch-to-refcounting-api.patch

Quoted 7 lines

Quoted 3 lines
Stuff to (various levels of re-)send to Greg for the PCI tree. I'll probably drop the kthread patches as they seemed a bit half-baked and I've lost track of which ones have which levels of baking.
All the partially kthread conversion were superceed with full conversion from me. I've only got feedback from the cpci maintainer, and he acked my patch together with a simple fix from him.

Hm, I'm no longer the PCI Hotplug maintainer, so that's why I haven't added them to my tree. It would probably be best for everyone involved to send them to her instead :)

thanks,

greg k-h

Christoph Hellwig

2007-05-13 20:59 UTC #87 lore raw

Re: pci hotplug patches

Quoted 3 lines · Greg KH

Hm, I'm no longer the PCI Hotplug maintainer, so that's why I haven't added them to my tree. It would probably be best for everyone involved to send them to her instead :)

FYI: MAINTAINERS still lists you as the maintainer of the cpqphp driver.

Greg KH

2007-05-14 11:48 UTC #88 lore raw

Re: pci hotplug patches

Quoted 6 lines · Christoph Hellwig

Quoted 3 lines · Greg KH
Hm, I'm no longer the PCI Hotplug maintainer, so that's why I haven't added them to my tree. It would probably be best for everyone involved to send them to her instead :)
FYI: MAINTAINERS still lists you as the maintainer of the cpqphp driver.

Ick, I'll go fix that up, I don't even have the hardware anymore (donated it to a local university...)

thanks,

greg k-h

Christoph Hellwig

2007-05-01 08:54 UTC #89 lore raw

cache-pipe-buf-page-address-for-non-highmem-arch.patch

Quoted 1 line

cache-pipe-buf-page-address-for-non-highmem-arch.patch

I still don't like this one at all. If page_address on x86_64 is too slow we should fix the root cause.

Andrew Morton

author maintainer

2007-05-01 09:04 UTC #90 lore raw

Re: cache-pipe-buf-page-address-for-non-highmem-arch.patch

Quoted 4 lines · On Tue, 1 May 2007 09:54:31 +0100 Christoph Hellwig <[email protected]> wrote:

Quoted 1 line
cache-pipe-buf-page-address-for-non-highmem-arch.patch
I still don't like this one at all. If page_address on x86_64 is too slow we should fix the root cause.

Fair enough, it is a bit of an ugly thing. And I see no measurements there on what the overall speedup was for any workload.

Ken, which memory model was in use? sparsemem?

Andi, what are the prospects of speeding any of that up?

Andi Kleen

2007-05-01 11:31 UTC #91 lore raw

Re: cache-pipe-buf-page-address-for-non-highmem-arch.patch

Quoted 1 line

Andi, what are the prospects of speeding any of that up?

Good, see my original replies. Also there are Christoph's vmemmap patches which also promised some speedup.

-Andi

Ken Chen

2007-05-03 03:48 UTC #92 lore raw

Re: cache-pipe-buf-page-address-for-non-highmem-arch.patch

Quoted 4 lines · Andrew Morton

Fair enough, it is a bit of an ugly thing. And I see no measurements there on what the overall speedup was for any workload.
Ken, which memory model was in use? sparsemem?

discontigmem with config_numa on.

Christoph Hellwig

2007-05-01 08:55 UTC #93 lore raw

consolidate-generic_writepages-and-mpage_writepages.patch

Quoted 3 lines

consolidate-generic_writepages-and-mpage_writepages.patch
Might merge. I forget what happened to this.

ACK from me on this one.

Pekka Enberg

2007-05-01 09:17 UTC #94 lore raw

Quoted 1 line · Andrew Morton

revoke-special-mmap-handling.patch

[snip]

Quoted 2 lines

Hold. This is tricky stuff and I don't think we've seen sufficient reviewing, testing and acking yet?

Agreed. While Peter and Nick have done some review of the patches, I would really like VFS maintainers to review them before merge. Christoph, have you had the chance to take a look at it?

Christoph Hellwig

2007-05-01 09:24 UTC #95 lore raw

Quoted 11 lines · Pekka Enberg

Quoted 1 line · Andrew Morton
revoke-special-mmap-handling.patch
[snip]
Quoted 2 lines
Hold. This is tricky stuff and I don't think we've seen sufficient reviewing, testing and acking yet?
Agreed. While Peter and Nick have done some review of the patches, I would really like VFS maintainers to review them before merge. Christoph, have you had the chance to take a look at it?

Not so far, but it's on my long list of highly useful things I want to review.

Peter Zijlstra

2007-05-01 09:37 UTC #96 lore raw

In reply to Pekka Enberg

Quoted 11 lines · Pekka Enberg

Quoted 1 line · Andrew Morton
revoke-special-mmap-handling.patch
[snip]
Quoted 2 lines
Hold. This is tricky stuff and I don't think we've seen sufficient reviewing, testing and acking yet?
Agreed. While Peter and Nick have done some review of the patches, I would really like VFS maintainers to review them before merge. Christoph, have you had the chance to take a look at it?

I'll have another look at it; also, I'll try to work through Mel's patches once again.

Andi Kleen

2007-05-01 11:22 UTC #97 lore raw

In reply to Pekka Enberg

Quoted 11 lines · Pekka Enberg

Quoted 1 line · Andrew Morton
revoke-special-mmap-handling.patch
[snip]
Quoted 2 lines
Hold. This is tricky stuff and I don't think we've seen sufficient reviewing, testing and acking yet?
Agreed. While Peter and Nick have done some review of the patches, I would really like VFS maintainers to review them before merge. Christoph, have you had the chance to take a look at it?

Also have the cache performance concerns raised on the original review been addressed?

-Andi

Pekka Enberg

2007-05-01 17:12 UTC #98 lore raw

Hi Andi,

Quoted 2 lines · Andi Kleen

Also have the cache performance concerns raised on the original review been addressed?

I am only aware of the fget_light() related issues Eric Dumazet raised but it's fixed. If you're thinking of something else, could you please remind me what it is?

Mel Gorman

2007-05-01 10:16 UTC #99 lore raw

fragmentation avoidance Re: 2.6.22 -mm merge plans

In reply to Andy Whitcroft

On (30/04/07 16:20), Andrew Morton didst pronounce:

Quoted 15 lines

add-apply_to_page_range-which-applies-a-function-to-a-pte-range.patch add-apply_to_page_range-which-applies-a-function-to-a-pte-range-fix.patch safer-nr_node_ids-and-nr_node_ids-determination-and-initial.patch use-zvc-counters-to-establish-exact-size-of-dirtyable-pages.patch proper-prototype-for-hugetlb_get_unmapped_area.patch mm-remove-gcc-workaround.patch slab-ensure-cache_alloc_refill-terminates.patch mm-more-rmap-checking.patch mm-make-read_cache_page-synchronous.patch fs-buffer-dont-pageuptodate-without-page-locked.patch allow-oom_adj-of-saintly-processes.patch introduce-config_has_dma.patch mm-slabc-proper-prototypes.patch mm-detach_vmas_to_be_unmapped-fix.patch

Quoted 1 line

Misc MM things. Will merge.

After Andy's mail, I am guessing that the patch below is also going here in the stack as a cleanup.

add-pfn_valid_within-helper-for-sub-max_order-hole-detection.patch

Quoted 5 lines

add-a-bitmap-that-is-used-to-track-flags-affecting-a-block-of-pages.patch add-__gfp_movable-for-callers-to-flag-allocations-from-high-memory-that-may-be-migrated.patch split-the-free-lists-for-movable-and-unmovable-allocations.patch choose-pages-from-the-per-cpu-list-based-on-migration-type.patch add-a-configure-option-to-group-pages-by-mobility.patch

Quoted 8 lines

drain-per-cpu-lists-when-high-order-allocations-fail.patch move-free-pages-between-lists-on-steal.patch group-short-lived-and-reclaimable-kernel-allocations.patch group-high-order-atomic-allocations.patch do-not-group-pages-by-mobility-type-on-low-memory-systems.patch bias-the-placement-of-kernel-pages-at-lower-pfns.patch be-more-agressive-about-stealing-when-migrate_reclaimable-allocations-fallback.patch fix-corruption-of-memmap-on-ia64-sparsemem-when-mem_section-is-not-a-power-of-2.patch

Plus the patch below from Andy's pfn_valid_within() series would be here:

anti-fragmentation-switch-over-to-pfn_valid_within.patch

These patches are the grouping pages by mobility patches. They get tested every time someone boots the machine from the perspective that they affect the page allocator. It is working to keep fragmentation problems to a minimum and being exercised. We have beaten it heavily here on tests with a variety of machines using the system that drives test.kernel.org for both functionality and performance testing. That covers x86, x86_64, ppc64 and occasionally IA64. Granted, there are corner-case machines out there or we'd never receive bug reports at all.

They are currently being reviewed by Christoph Lameter. His feedback in the linux-mm thread "Antifrag patchset comments" has given me a TODO list which I'm currently working through. So far, there has been no fundamental mistake in my opinion and the additional work is logical extensions.

The closest thing to a fundamental mistake was grouping pages by MAX_ORDER_NR_PAGES instead of an arbitrary order. What I did was fine for x86_64, i386 and ppc64 but not as useful for IA64 with 1GB worth of memory in MAX_ORDER_NR_PAGES. I also missed some temporary allocations as picked up in Christophs review.

Quoted 10 lines

create-the-zone_movable-zone.patch allow-huge-page-allocations-to-use-gfp_high_movable.patch x86-specify-amount-of-kernel-memory-at-boot-time.patch ppc-and-powerpc-specify-amount-of-kernel-memory-at-boot-time.patch x86_64-specify-amount-of-kernel-memory-at-boot-time.patch ia64-specify-amount-of-kernel-memory-at-boot-time.patch add-documentation-for-additional-boot-parameter-and-sysctl.patch handle-kernelcore=-boot-parameter-in-common-code-to-avoid-boot-problem-on-ia64.patch
Mel's moveable-zone work.

These patches are what creates ZONE_MOVABLE. The last 6 patches should be collapsed into a single patch:

handle-kernelcore=-generic

I believe Yasunori Goto is looking at these from the perspective of memory hot-remove and has caught a few bugs in the past. Goto-san may be able to comment on whether they have been reviewed recently.

The main complexity is in one function in patch one which determines where the PFN is in each node for ZONE_MOVABLE. Getting that right so that the requested amount of kernel memory spread as evenly as possible is just not straight-forward.

Quoted 11 lines

I don't believe that this has had sufficient review and I'm sure that it hasn't had sufficient third-party testing. Most of the approbations thus far have consisted of people liking the overall idea, based on the changelogs and multi-year-old discussions.
For such a large and core change I'd have expected more detailed reviewing effort and more third-party testing. And I STILL haven't made time to review the code in detail myself.
So I'm a bit uncomfortable with moving ahead with these changes.

Ok. It is getting reviewed by Christoph and I'm going through the TODO items it yielded. Andy has also been regularly reviewing them which is probably why they have had less public errors than you might expect from something like this. Christoph may like to comment more here.

Quoted 3 lines

<snip>
lumpy-reclaim-v4.patch

And I guess this patch also moves here

lumpy-move-to-using-pfn_valid_within.patch

Quoted 4 lines

This is in a similar situation to the moveable-zone work. Sounds great on paper, but it needs considerable third-party testing and review. It is a major change to core MM and, we hope, a significant advance. On paper.

Andy will probably comment more here. Like the fragmentation stuff, we have beaten this heavily in tests.

I'm not sure of it's review situation.

Quoted 5 lines

More Mel things, and linkage between Mel-things and lumpy reclaim. It's here where the patch ordering gets into a mess and things won't improve if moveable-zones and lumpy-reclaim get deferred. Such a deferral would limit my ability to queue more MM changes for 2.6.23.

This is where the three patches were originally. From the other thread, I am assuming these are sorted out.

Quoted 8 lines

<snip>
bias-the-location-of-pages-freed-for-min_free_kbytes-in-the-same-max_order_nr_pages-blocks.patch remove-page_group_by_mobility.patch dont-group-high-order-atomic-allocations.patch
More moveable-zone work.

This is the MIGRATE_RESERVE patch and two patches that back out parts of the grouping pages by mobility stack. If possible, these patches should move to the end of that stack. To fix the ordering, would it be helpful to provide a fresh stack based on 2.6.21? That would delete 4 patches in all. The two that introduce configuration items and highorder atomic groupings and these two patches that subsequently remove them.

Quoted 6 lines

<SNIP>
slub-exploit-page-mobility-to-increase-allocation-order.patch
Slub entanglement with moveable-zones. Will merge if moveable-zones is merged.

Well, grouping pages by mobility is what it really depends on. The ZONE_MOVABLE is not required for SLUB. However, I get the point and agree with it. If the rest of SLUB gets merged, this patch could be moved to the end of the grouping by mobility stack.

Signature · 9 lines

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andy Whitcroft

2007-05-01 13:02 UTC #100 lore raw

Re: 2.6.22 -mm merge plans -- lumpy reclaim

Mel Gorman wrote:

<snip>

Quoted 4 lines

Quoted 1 line
lumpy-reclaim-v4.patch
And I guess this patch also moves here

Quoted 8 lines

lumpy-move-to-using-pfn_valid_within.patch
Quoted 3 lines
This is in a similar situation to the moveable-zone work. Sounds great on paper, but it needs considerable third-party testing and review. It is a major change to core MM and, we hope, a significant advance. On paper.
Andy will probably comment more here. Like the fragmentation stuff, we have beaten this heavily in tests.

With this stack the basic functionality for Lumpy reclaim is complete. Better integration with kswapd is desirable, but IMO that should be a separate change.

In testing it has produced significant improvements the likelyhood of reclaiming a page (reclaim effectiveness) at very high orders (where the likelyhood of success is least), and effectiveness at lower orders should be better again. In general -mm testing lumpy is triggered for any stalled allocation above order-0; it is common to see stack allocations triggering lumpy under higher load. kswapd also now utilises lumpy when required.

As Mel has indicated a lot of automated testing has been done on these patches. As reclaim is only entered when low on memory, our testing focuses on triggering pushing the system to a heavily fragmented state where reclaim is used heavily. This testing has not shown any regressions and shows improved effectiveness particularly under load.

Effectiveness for regular reclaim is based on random distributions, as such it is only likely to successfully reclaim pages at lower orders. Lumpy reclaim improves on this by actively targeting reclaim on areas at the orders required and so succeeds at significantly higher order. Very high order allocations require better layout, from the mobility patches.

I have some primitive stats patches which we have used performance testing. Perhaps those could be brought up to date to provide better visibility into lumpy's operation. Again this would be a separate patch.

Quoted 1 line

I'm not sure of it's review situation.

As lumpy reclaim and grouping-by-mobility are complementary patch sets (in that they both assist at the highest order) we work pretty closely and I generally pass all my patches past Mel before general release. Early versions were based on patches from Peter Zijlstra who also reviewed earlier versions if memory serves. The changes since then have been reviewed by Mel and Andrew Morton only to my knowledge.

Perhaps Peter would have some time to take a look over the latest stack as it appears in -mm when that releases; ping me for a patch kit if you want it before then :).

<snip>

-apw

Peter Zijlstra

Acked-by

2007-05-01 18:03 UTC #101 lore raw

Re: 2.6.22 -mm merge plans -- lumpy reclaim

Quoted 3 lines · Andy Whitcroft

Perhaps Peter would have some time to take a look over the latest stack as it appears in -mm when that releases; ping me for a patch kit if you want it before then :).

Lumpy-reclaim -v7, as per the roll-up provided privately;

Code is looking good, I like what you did to it :-)

Acked-by Peter Zijlstra <[email protected]>

Andrew Morton

author maintainer

2007-05-01 19:00 UTC #102 lore raw

Re: 2.6.22 -mm merge plans -- lumpy reclaim

Quoted 3 lines · On Tue, 01 May 2007 14:02:41 +0100 Andy Whitcroft <[email protected]> wrote:

I have some primitive stats patches which we have used performance testing. Perhaps those could be brought up to date to provide better visibility into lumpy's operation. Again this would be a separate patch.

Feel free to add new counters in /proc/vmstat - perhaps per-order success and fail rates? Monitoring the ratio between those would show how effective lumpiness is being, perhaps.

It's always nice to see what's going on in there.

Christoph Lameter

2007-05-01 14:54 UTC #103 lore raw

Re: fragmentation avoidance Re: 2.6.22 -mm merge plans

In reply to Mel Gorman

Quoted 11 lines · Mel Gorman

anti-fragmentation-switch-over-to-pfn_valid_within.patch
These patches are the grouping pages by mobility patches. They get tested every time someone boots the machine from the perspective that they affect the page allocator. It is working to keep fragmentation problems to a minimum and being exercised. We have beaten it heavily here on tests with a variety of machines using the system that drives test.kernel.org for both functionality and performance testing. That covers x86, x86_64, ppc64 and occasionally IA64. Granted, there are corner-case machines out there or we'd never receive bug reports at all.

Quoted 4 lines

They are currently being reviewed by Christoph Lameter. His feedback in the linux-mm thread "Antifrag patchset comments" has given me a TODO list which I'm currently working through. So far, there has been no fundamental mistake in my opinion and the additional work is logical extensions.

I think we really urgently need a defragmentation solution in Linux in order to support higher page allocations for various purposes. SLUB f.e. would benefit from it and the large blocksize patches are not reasonable without such a method.

However, the current code is not up to the task. I did not see a clean categorization of allocations nor a consistent handling of those. The cleanup work that would have to be done throughout the kernel is not there. It is spotty. There seems to be a series of heuristic driving this thing (I have to agree with Nick there). The temporary allocations that were missed are just a few that I found. The review of the rest of the kernel was not done. Mel said that he fixed up locations that showed up to be a problem in testing. That is another issue: Too much focus on testing instead of conceptual cleanness and clean code in the kernel. It looks like this is geared for a specific series of tests on specific platforms and also to a particular allocation size (max order sized huge pages).

There are major technical problems with

1. Large Scale allocs. Multiple MAX_ORDER blocks as required by the antifrag patches may not exist on all platforms. Thus the antifrag patches will not be able to generate their MAX_ORDER sections. We could reduce MAX_ORDER on some platforms but that would have other implications like limiting the highest order allocation.

2. Small huge page size support. F.e. IA64 can support down to page size huge pages. The antifrag patches handle huge page in a special way. They are categorized as movable. Small huge pages may therefore contaminate the movable area.

3. Defining the size of ZONE_MOVABLE. This was done to guarantee availability of movable memory but the practical effect is to guarantee that we panic when too many unreclaimable allocations have been done.

I have already said during the review that IMHO the patches are not ready for merging. They are currently more like a prototype that explores ideas. The generalization steps are not done.

How we could make progress:

1. Develop a useful categorization of allocations in the kernel whose utility goes beyond the antifrag patches. I.e. length of the objects existence and the method of reclaim could be useful in various contexts.

2. Have statistics of these various allocations.

3. Page allocator should gather statistics on how memory was allocated in the various categories.

4. The available data can then be used to driver more intelligent reclaim and develop methods of antifrag or defragmentation.

Mel Gorman

2007-05-01 19:00 UTC #104 lore raw

Re: fragmentation avoidance Re: 2.6.22 -mm merge plans

Quoted 18 lines · Christoph Lameter

Quoted 11 lines · Mel Gorman
anti-fragmentation-switch-over-to-pfn_valid_within.patch
These patches are the grouping pages by mobility patches. They get tested every time someone boots the machine from the perspective that they affect the page allocator. It is working to keep fragmentation problems to a minimum and being exercised. We have beaten it heavily here on tests with a variety of machines using the system that drives test.kernel.org for both functionality and performance testing. That covers x86, x86_64, ppc64 and occasionally IA64. Granted, there are corner-case machines out there or we'd never receive bug reports at all.
Quoted 4 lines
They are currently being reviewed by Christoph Lameter. His feedback in the linux-mm thread "Antifrag patchset comments" has given me a TODO list which I'm currently working through. So far, there has been no fundamental mistake in my opinion and the additional work is logical extensions.

Quoted 5 lines

I think we really urgently need a defragmentation solution in Linux in order to support higher page allocations for various purposes. SLUB f.e. would benefit from it and the large blocksize patches are not reasonable without such a method.

I continue to maintain that anti-fragmentation is a pre-requisite for any defragmentation mechanism to be effective without trashing overall performance. If allocation success rates are low when everything possible has been reclaimed as is the case without fragmentation avoidance, then defragmentation will not help unless the the 1:1 phys:virt mappings is broken which incurs its own considerable set of problems.

Quoted 4 lines

However, the current code is not up to the task. I did not see a clean categorization of allocations nor a consistent handling of those. The cleanup work that would have to be done throughout the kernel is not there.

The choice of mobility marker to use in each case was deliberate (even if I have made mistakes but what else is review for?). The choice by default is UNMOVABLE as it's the safe choice even if may be sub-optimal. The description of the mobility types may not be the clearest. For example, buffers were placed beside page cache in MOVABLE because they can both be reclaimed in the same fashion - I consider moving it to disk to be as "movable" as any other definition of the word but in your world movable always means page migration which has led to some confusion. They could have been separated out as MOVABLE and BUFFERS for a conceptually cleaner split but it did not seem necessary because the more types there are, the bigger the memory and performance footprint becomes. Additional flag groupings like GFP_BUFFERS could be defined that alias to MOVABLE if you felt it would make the code clearer but functionally, the behaviour remains the same. This is similar to your feedback on the treatment of GFP_TEMPORARY.

There can be as many alias mobility types as you wish but if more "real" types are required, you can have as you want as long as NR_PAGEBLOCK_BITS is increased properly and allocflags_to_migratetype() is able to translate GFP flags to the appropriate mobility type. It increases the performance and memory footprint though.

Quoted 4 lines

It is spotty. There seems to be a series of heuristic driving this thing (I have to agree with Nick there). The temporary allocations that were missed are just a few that I found. The review of the rest of the kernel was not done.

The review for temporary allocations was aimed at catching the most common callers, not every single one of them because a full review of every caller is a large undertaking. If anything, it makes more sense to do a review of all callers at the end when the core mechanism is finished. The default to treat them as UNMOVABLE is sensible.

Quoted 3 lines

Mel said that he fixed up locations that showed up to
be a problem in testing. That is another issue: Too much focus on testing instead of conceptual cleanness and clean code in the kernel.

The patches started as a thought experiment of what "should work". They were then tested to find flaws in the model and the results were fed back in. How is that a disadvantage exactly?

Quoted 4 lines

It looks
like this is geared for a specific series of tests on specific platforms and also to a particular allocation size (max order sized huge pages).

Some series of tests had to be chosen and one combination was chosen that was known to be particularly hostile to external fragmentation - i.e. large numbers of kernel cache allocations at the same time as page cache allocations. No one has suggested an alternative test that would be more suitable. The platforms used were x86, x86_64 and ppc64 which are not exactly insignificant platforms. At the time, I didn't have an IA64 machine and franky the one I have now does not always boot so testing is not as thorough.

Huge page sized pages were chosen because they were the hardest allocation to satisfy. If they could be allocated successfully, it stood to reason that smaller allocations at least as well.

Hugepages and MAX_ORDER pages were close to the same size on x86, x86_64 and ppc64 which is why that figure was chosen. I point out that while IA64 can specify hugepagesz= to change the hugepage size, it's not documented in Documentation/kernel-parameters.txt or I might have spotted this sooner.

These decisions were not random.

Quoted 7 lines

There are major technical problems with
1. Large Scale allocs. Multiple MAX_ORDER blocks as required by the antifrag patches may not exist on all platforms. Thus the antifrag patches will not be able to generate their MAX_ORDER sections. We could reduce MAX_ORDER on some platforms but that would have other implications like limiting the highest order allocation.

MAX_ORDER was a sensible choice on the three initial platforms. However, it is not a fundamental value in the mechanism and is an easy assumption to break. I've included a patch below based on your review that choses a size based on the value of HPAGE_SHIFT. It took 45 minutes to cobble together so it's rough looking and I might have missed something but it has passed stress tests on x86 without difficulty. Here is the dmesg output

[    0.000000] Built 1 zonelists, mobility grouping on at order 5. Total pages: 16224

Voila, grouping on order 5 instead of 10 (I used 5 instead of HPAGE_SHIFT for testing purposes).

The order used can be any value >= 2 and < MAX_ORDER.

Quoted 4 lines

2. Small huge page size support. F.e. IA64 can support down to page size huge pages. The antifrag patches handle huge page in a special way. They are categorized as movable. Small huge pages may therefore contaminate the movable area.

They are only categorised as movable when a sysctl is set. This has to be the deliberate choice of the administrator and its intention was to allow hugepages to be alloced from ZONE_MOVABLE. This was to allow flexible sizing of the hugepage pool when that zone is configured until such time as hugepages were really movable in 100% of situations.

Quoted 5 lines

3. Defining the size of ZONE_MOVABLE. This was done to guarantee availability of movable memory but the practical effect is to guarantee that we panic when too many unreclaimable allocations have been done.

The size of ZONE_MOVABLE is determined at boot time and it is not required for grouping page by mobility to be effective. Presumably by an administrator that has identified the problem that is fixed by having this zone available. Furthermore, it would be done with the understanding of what it means for OOM situations if the partition is made too small. The expectation is that he has a solid understanding of his workload before using this option.

Quoted 11 lines

I have already said during the review that IMHO the patches are not ready for merging. They are currently more like a prototype that explores ideas. The generalization steps are not done.
How we could make progress:
1. Develop a useful categorization of allocations in the kernel whose utility goes beyond the antifrag patches. I.e. length of the objects existence and the method of reclaim could be useful in various contexts.

The length of objects existence is something I am wary of because it puts a big burden on the caller of the page allocator. The method of reclaim is already implied by the existing categorisations. What may be missing is clear documentation

UNMOVABLE - You can't reclaim it

RECLAIMABLE - You need the help of another subsystem to reclaim objects
within the page before the page is reclaimed or the allocation is short-lived. Even when reclaimable, there is no guarantee that reclaim will succeed.

MOVABLE - The page is directly reclaimable by kswapd or it may be
migrated. Being able to reclaim is guaranteed except where mlock() is involved. mlock pages need to be migrated.

You've defined these better yourself in your review. Arguably, RECLAIMABLE should be separate from TEMPORARY and page buffers should be away from MOVABLE but this did not appear necessary when tested.

If this breakout is found to be required, it is trivial to implement.

Quoted 5 lines

2. Have statistics of these various allocations.
3. Page allocator should gather statistics on how memory was allocated in the various categories.

Statistics gathering has been done before and it can be done again. They were used earlier in the development of the patches and then I stopped bringing them forward in the belief they would not be of general interest. In a large part, they helped define the current mobility types. Gathering statistics again is not a fundamental problem.

Quoted 3 lines

4. The available data can then be used to driver more intelligent reclaim and develop methods of antifrag or defragmentation.

Once that data is available, it would help show how successfully fragmentation avoidance as it currently stands and how it can be improved. The lack of the statistics today does not seem a blocking issue because there are no users of fragmentation avoidance that blow up if it's not effective.

Patch for breaking the MAX_ORDER grouping is as follows. Again, it's 45 minutes coding so maybe I missed something but it survived a quick stress testing.

Not signed off due to incompleteness (e.g. should use a constant if the hugepage size is known at compile time, nr_pages_pageblock should be __read_mostly, not checked everywhere etc) and lack of full regression testing and verification. If I hadn't bothered updating comments or printks, the patch would be fairly small.

include/linux/pageblock-flags.h +5 -1

diff -rup -X /usr/src/patchset-0.6/bin//dontdiff linux-2.6.21-rc7-mm2-004_temporary/include/linux/pageblock-flags.h linux-2.6.21-rc7-mm2-005_group_arbitrary/include/linux/pageblock-flags.h
@@ -1,6 +1,6 @@
 /*
  * Macros for manipulating and testing flags related to a
- * MAX_ORDER_NR_PAGES block of pages.
+ * large contiguous block of pages.
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by

@@ -35,6 +35,10 @@ enum pageblock_bits {
 	NR_PAGEBLOCK_BITS
 };
 
+/* Each pages_per_mobility_block of pages has NR_PAGEBLOCK_BITS */
+extern unsigned long nr_pages_pageblock;
+extern int pageblock_order;
+
 /* Forward declaration */
 struct page;

mm/page_alloc.c +46 ~~-20~~

170 lines not on this page · raw message

Signature · 9 lines

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andrew Morton

author maintainer

2007-05-01 18:57 UTC #105 lore raw

Re: fragmentation avoidance Re: 2.6.22 -mm merge plans

In reply to Mel Gorman

Quoted 1 line · On Tue, 1 May 2007 11:16:51 +0100 [email protected] (Mel Gorman) wrote:

OK, I did all the reorganisation which you recommended.

Quoted 4 lines

Ok. It is getting reviewed by Christoph and I'm going through the TODO items it yielded. Andy has also been regularly reviewing them which is probably why they have had less public errors than you might expect from something like this.

Great. I'm a bit behind on my linux-mm reading.

Quoted 1 line

Christoph may like to comment more here.

That would be helpful.

Yasunori Goto

2007-05-07 13:07 UTC #106 lore raw

Re: fragmentation avoidance Re: 2.6.22 -mm merge plans

In reply to Mel Gorman

Sorry for late response. I went on a vacation in last week. And I'm in the mountain of a ton of unread mail now....

Quoted 10 lines

Quoted 1 line
Mel's moveable-zone work.
These patches are what creates ZONE_MOVABLE. The last 6 patches should be collapsed into a single patch:
handle-kernelcore=-generic
I believe Yasunori Goto is looking at these from the perspective of memory hot-remove and has caught a few bugs in the past. Goto-san may be able to comment on whether they have been reviewed recently.

Hmm, I don't think my review is enough.
To be precise, I'm just one user/tester of ZONE_MOVABLE.
I have tried to make memory remove patches with Mel-san's ZONE_MOVABLE patch. And the bugs are things that I found in its work. (I'll post these patches in a few days.)

Quoted 6 lines

The main complexity is in one function in patch one which determines where the PFN is in each node for ZONE_MOVABLE. Getting that right so that the requested amount of kernel memory spread as evenly as possible is just not straight-forward.
From memory-hotplug view, ZONE_MOVABLE should be aligned by section

size. But MAX_ORDER alignment is enough for others...

Bye.

Signature · 8 lines

Yasunori Goto 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andi Kleen

2007-05-01 11:19 UTC #107 lore raw

In reply to Mathieu Desnoyers

Quoted 1 line · Andrew Morton

Static markers. Will merge.

There don't seem to be any users of this. How do you know it hasn't already bitrotted?

It seems quite overcomplicated to me. Has the complexity been justified?

Quoted 2 lines

Will merge the rustyvisor.

IMHO the user code still doesn't belong into Documentation. Also it needs another review round I guess. And some beta testing by more people.

Quoted 3 lines

Hopefully Wu will be coming up with a much simpler best-of-readahead patch soon. I don't think we can get these patches over the hump and they are somewhat costly to maintain.

Didn't he have one already? There was a relatively simple readahead patch recently, although it was unclear what dependencies it needed. IMHO this work has much potential so i hope the benchmarking-review process can be done quickly.

-Andi

Mathieu Desnoyers

2007-05-01 22:08 UTC #108 lore raw

Hi Andi,

Quoted 7 lines · * Andi Kleen ([email protected]) wrote:

Quoted 1 line · Andrew Morton
Static markers. Will merge.
There don't seem to be any users of this. How do you know it hasn't already bitrotted?

See the detailed explanation at :
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc7/2.6.21-rc7-mm2/broken-out/linux-kernel-markers-kconfig-menus.patch

Major points :

It is currently used as an instrumentation infrastructure for the LTTng tracer at IBM, Google, Autodesk, Sony, MontaVista and deployed in WindRiver products. The SystemTAP project also plan to use this type of infrastructure to trace sites hard to instrument. The Linux Kernel Markers has the support of Frank C. Eigler, author of their current marker alternative (which he wishes to drop in order to adopt the markers infrastructure as soon as it hits mainline).

Quoting Jim Keniston <[email protected]> :

"kprobes remains a vital foundation for SystemTap. But markers are attactive as an alternate source of trace/debug info. Here's why:
[...]"

Quoted 2 lines

It seems quite overcomplicated to me. Has the complexity been justified?

To summarize the document pointed at the URL above, where the full the key goals of the markers, showing the rationale being the most important design choices :
- Almost non perceivable impact on production machines when compiled in but markers are "disabled".
- Use a separate section to keep the data to minimize d-cache trashing.
- Put the code (stack setup and function call) in unlikely branches of the if() condition to minimize i-cache impact.
- Since it is required to allow instrumentation of variables within the body of a function, accept the impact on compiler's optimizations and let it keep the variables "live" sometimes longer than required. It is up to the person who puts the marker in the code to choose the location that will have a small impact in this aspect.
- Allow per-architecture optimized versions which removes the need for a d-cache based branch (patch a "load immediate" instruction instead). It minimized the d-cache impact of the disabled markers.
- Accept the cost of an unlikely branch at the marker site because the gcc compiler does not give the ability to put "nops" instead of a branch generated from C code. Keep this in mind for future per-architecture optimizations.
- Instrumentation of challenging kernel sites
- Instrumentation such as the one provided in the already existing Lock dependency checker (lockdep) and instrumentation of trap handlers implies being reentrant for such context. Therefore, the implementation must be lock-free and update the state in an atomic fashion (rcu-style). It must also let the programmer who describes a marker site the ability to specify what is forbidden in the probe that will be connected to the marker : can it generate a trap ? Can it call lockdep (irq disable, take any type of lock), can it call printk ? This is why flags can be passed to the _MARK() marker, while the MARK() marker has the default flags.

Please tell me if I forgot to explain the rationale behind some implementation detail and I will be happy to explain in more depth.

Regards,

Mathieu

Signature · 3 lines

Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Andi Kleen

2007-05-02 10:44 UTC #109 lore raw

Quoted 7 lines

It is currently used as an instrumentation infrastructure for the LTTng tracer at IBM, Google, Autodesk, Sony, MontaVista and deployed in WindRiver products. The SystemTAP project also plan to use this type of infrastructure to trace sites hard to instrument. The Linux Kernel Markers has the support of Frank C. Eigler, author of their current marker alternative (which he wishes to drop in order to adopt the markers infrastructure as soon as it hits mainline).

All of the above don't use mainline kernels.
That doesn't constitute using it.

Quoted 5 lines

Quoting Jim Keniston <[email protected]> :
"kprobes remains a vital foundation for SystemTap. But markers are attactive as an alternate source of trace/debug info. Here's why:
[...]"

Talk is cheap. Do they have working code to use it?

Quoted 3 lines

- Allow per-architecture optimized versions which removes the need for a d-cache based branch (patch a "load immediate" instruction instead). It minimized the d-cache impact of the disabled markers.

That's a good idea in general, but should be generalized (available independently), not hidden in your subsystem. I know a couple of places who could use this successfully.

Quoted 4 lines

- Accept the cost of an unlikely branch at the marker site because the gcc compiler does not give the ability to put "nops" instead of a branch generated from C code. Keep this in mind for future per-architecture optimizations.

See upcomming paravirt code for a way to do this.

Quoted 11 lines

- Instrumentation of challenging kernel sites
- Instrumentation such as the one provided in the already existing Lock dependency checker (lockdep) and instrumentation of trap handlers implies being reentrant for such context. Therefore, the implementation must be lock-free and update the state in an atomic fashion (rcu-style). It must also let the programmer who describes a marker site the ability to specify what is forbidden in the probe that will be connected to the marker : can it generate a trap ? Can it call lockdep (irq disable, take any type of lock), can it call printk ? This is why flags can be passed to the _MARK() marker, while the MARK() marker has the default flags.

Why can't you just generally forbid probes from doing all of this? It would greatly simplify your code, wouldn't it?

Keep it simple please.

Quoted 2 lines

Please tell me if I forgot to explain the rationale behind some implementation detail and I will be happy to explain in more depth.

Having lots of flags to do things differently optionally normally starts up all warning lights of early over design. While Linux has this sometimes it is generally only in mature old subsystems. But when something is freshly merged it shouldn't be like this. That is because code tends to grow more complicated over its livetime and when it is already complicated at the beginning it will eventually fall over (you can study current slab as a poster child of this)

-Andi

Frank Ch. Eigler

2007-05-02 16:38 UTC #110 lore raw

Quoted 7 lines · Andi Kleen

[...] The SystemTAP project also plan to use this type of
Quoted 3 lines
infrastructure to trace sites hard to instrument. The Linux Kernel Markers has the support of Frank C. Eigler, author of their current marker alternative [...]
All of the above don't use mainline kernels.
That doesn't constitute using it.

Systemtap does run on mainline kernels.

Quoted 5 lines

Quoted 3 lines
"kprobes remains a vital foundation for SystemTap. But markers are attactive as an alternate source of trace/debug info. Here's why:
[...]"
Talk is cheap. Do they have working code to use it? [...]

We had been waiting on the chicken & egg semaphore. LTTNG has working code yesterday (months ago); systemtap will have it "tomorrow" (a week or few).

- FChE

Andrew Morton

author maintainer

2007-05-02 16:48 UTC #111 lore raw

In reply to Andi Kleen

Quoted 9 lines · On Wed, 2 May 2007 12:44:13 +0200 Andi Kleen <[email protected]> wrote:

Quoted 7 lines
It is currently used as an instrumentation infrastructure for the LTTng tracer at IBM, Google, Autodesk, Sony, MontaVista and deployed in WindRiver products. The SystemTAP project also plan to use this type of infrastructure to trace sites hard to instrument. The Linux Kernel Markers has the support of Frank C. Eigler, author of their current marker alternative (which he wishes to drop in order to adopt the markers infrastructure as soon as it hits mainline).
All of the above don't use mainline kernels.

That's because they have to add a markers patch!

Quoted 1 line

That doesn't constitute using it.

Andi, there was a huge amount of discussion about all this in September last year (subjects: *markers* and *LTTng*). The outcome of all that was, I believe, that the kernel should have a static marker infrastructure.

Christoph Hellwig

2007-05-02 17:29 UTC #112 lore raw

Quoted 5 lines · Andrew Morton

Quoted 1 line
That doesn't constitute using it.
Andi, there was a huge amount of discussion about all this in September last year (subjects: *markers* and *LTTng*). The outcome of all that was, I believe, that the kernel should have a static marker infrastructure.

Only when it's actually useable. A prerequisite for merging it is having an actual trace transport infrastructure aswell as a few actually useful tracing modules in the kernel tree.

Let this count as a vote to merge the markers once we have the infrastructure above ready, it'll be very useful then.

Mathieu Desnoyers

2007-05-02 20:41 UTC #113 lore raw

Quoted 7 lines · * Christoph Hellwig ([email protected]) wrote:

Quoted 5 lines · Andrew Morton
Quoted 1 line
That doesn't constitute using it.
Andi, there was a huge amount of discussion about all this in September last year (subjects: *markers* and *LTTng*). The outcome of all that was, I believe, that the kernel should have a static marker infrastructure.

Quoted 6 lines

Only when it's actually useable. A prerequisite for merging it is having an actual trace transport infrastructure aswell as a few actually useful tracing modules in the kernel tree.
Let this count as a vote to merge the markers once we have the infrastructure above ready, it'll be very useful then.

Hi Christoph,

The idea is the following : either we integrate the infrastructure for instrumentation / data serialization / buffer management / extraction of data to user space in multiple different steps, which makes code review easier for you guys, or we bring the main pieces of the LTTng project altogether with the Linux Kernel Markers, which would result in a bigger change.

Based on the premise that discussing about logically distinct pieces of infrastructure is easier and can be done more thoroughly when done separately, we decided to submit the markers first, with the other pieces planned in a near future.

I agree that it would be very useful to have the full tracing stack available in the Linux kernel, but we inevitably face the argument :
"this change is too big" if we submit all LTTng modules at once or the argument : "we want the whole tracing stack, not just part of it" if we don't.

This is why we chose to push the tracing infrastructure chunk by chunk :
to make code review and criticism more efficient.

Regards,

Mathieu

Signature · 3 lines

Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Andrew Morton

author maintainer

2007-05-02 20:54 UTC #114 lore raw

Quoted 29 lines · On Wed, 2 May 2007 16:36:27 -0400 Mathieu Desnoyers…

Quoted 7 lines · * Christoph Hellwig ([email protected]) wrote:
Quoted 5 lines · Andrew Morton
Quoted 1 line
That doesn't constitute using it.
Andi, there was a huge amount of discussion about all this in September last year (subjects: *markers* and *LTTng*). The outcome of all that was, I believe, that the kernel should have a static marker infrastructure.
Quoted 6 lines
Only when it's actually useable. A prerequisite for merging it is having an actual trace transport infrastructure aswell as a few actually useful tracing modules in the kernel tree.
Let this count as a vote to merge the markers once we have the infrastructure above ready, it'll be very useful then.
Hi Christoph,
The idea is the following : either we integrate the infrastructure for instrumentation / data serialization / buffer management / extraction of data to user space in multiple different steps, which makes code review easier for you guys, or we bring the main pieces of the LTTng project altogether with the Linux Kernel Markers, which would result in a bigger change.
Based on the premise that discussing about logically distinct pieces of infrastructure is easier and can be done more thoroughly when done separately, we decided to submit the markers first, with the other pieces planned in a near future.

Quoted 9 lines

I agree that it would be very useful to have the full tracing stack available in the Linux kernel, but we inevitably face the argument :
"this change is too big" if we submit all LTTng modules at once or the argument : "we want the whole tracing stack, not just part of it" if we don't.
This is why we chose to push the tracing infrastructure chunk by chunk :
to make code review and criticism more efficient.

I didn't know that this was the plan.

The problem I have with this is that once we've merged one part, we're committed to merging the other parts even though we haven't seen them yet.

What happens if there's a revolt over the next set of patches? Do we remove the core markers patches again? We end up in a cant-go-forward, cant-go-backward situation.

I thought the existing code was useful as-is for several projects, without requiring additional patching to core kernel. If such additional patching _is_ needed to make the markers code useful then I agree that we should continue to buffer the markers code in -mm until the use-markers-for-something patches have been eyeballed.

In which case we have:

For 2.6.22

linux-kernel-markers-kconfig-menus.patch linux-kernel-markers-architecture-independant-code.patch linux-kernel-markers-powerpc-optimization.patch linux-kernel-markers-i386-optimization.patch markers-add-instrumentation-markers-menus-to-avr32.patch linux-kernel-markers-non-optimized-architectures.patch markers-alpha-and-avr32-supportadd-alpha-markerh-add-arm26-markerh.patch linux-kernel-markers-documentation.patch
#
markers-define-the-linker-macro-extra_rwdata.patch markers-use-extra_rwdata-in-architectures.patch
# some-grammatical-fixups-and-additions-to-atomich-kernel-doc.patch no-longer-include-asm-kdebugh.patch

Hold.

Mathieu Desnoyers

2007-05-02 23:11 UTC #115 lore raw

Quoted 51 lines · * Andrew Morton ([email protected]) wrote:

Quoted 29 lines · On Wed, 2 May 2007 16:36:27 -0400 Mathieu Desnoyers…
Quoted 7 lines · * Christoph Hellwig ([email protected]) wrote:
Quoted 5 lines · Andrew Morton
Quoted 1 line
That doesn't constitute using it.
Andi, there was a huge amount of discussion about all this in September last year (subjects: *markers* and *LTTng*). The outcome of all that was, I believe, that the kernel should have a static marker infrastructure.
Quoted 6 lines
Only when it's actually useable. A prerequisite for merging it is having an actual trace transport infrastructure aswell as a few actually useful tracing modules in the kernel tree.
Let this count as a vote to merge the markers once we have the infrastructure above ready, it'll be very useful then.
Hi Christoph,
The idea is the following : either we integrate the infrastructure for instrumentation / data serialization / buffer management / extraction of data to user space in multiple different steps, which makes code review easier for you guys, or we bring the main pieces of the LTTng project altogether with the Linux Kernel Markers, which would result in a bigger change.
Based on the premise that discussing about logically distinct pieces of infrastructure is easier and can be done more thoroughly when done separately, we decided to submit the markers first, with the other pieces planned in a near future.
Quoted 9 lines
I agree that it would be very useful to have the full tracing stack available in the Linux kernel, but we inevitably face the argument :
"this change is too big" if we submit all LTTng modules at once or the argument : "we want the whole tracing stack, not just part of it" if we don't.
This is why we chose to push the tracing infrastructure chunk by chunk :
to make code review and criticism more efficient.
I didn't know that this was the plan.
The problem I have with this is that once we've merged one part, we're committed to merging the other parts even though we haven't seen them yet.
What happens if there's a revolt over the next set of patches? Do we remove the core markers patches again? We end up in a cant-go-forward, cant-go-backward situation.

Quoted 6 lines

I thought the existing code was useful as-is for several projects, without requiring additional patching to core kernel. If such additional patching _is_ needed to make the markers code useful then I agree that we should continue to buffer the markers code in -mm until the use-markers-for-something patches have been eyeballed.

My statement was probably not clear enough. The actual marker code is useful as-is without any further kernel patching required : SystemTAP is an example where they use external modules to load probes that can connect either to markers or through kprobes. LTTng, in its current state, has a mostly modular core that also uses the markers.

Although some, like Christoph and myself, think that it would benefit to the kernel community to have a common infrastructure for more than just markers (meaning common serialization and buffering mechanism), it does not change the fact that the markers, being in mainline, are usable by projects through additional kernel modules.

If we are looking at current "potential users" that are already in mainline, we could change blktrace to make it use the markers.

Mathieu

Quoted 41 lines

In which case we have:
atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-alpha.patch atomich-complete-atomic_long-operations-in-asm-generic.patch atomich-i386-type-safety-fix.patch atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-ia64.patch atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-mips.patch atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-parisc.patch atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-powerpc.patch atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-sparc64.patch atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-x86_64.patch atomich-atomic_add_unless-as-inline-remove-systemh-atomich-circular-dependency.patch local_t-architecture-independant-extension.patch local_t-alpha-extension.patch
local_t-i386-extension.patch
local_t-ia64-extension.patch
local_t-mips-extension.patch
local_t-parisc-cleanup.patch
local_t-powerpc-extension.patch
local_t-sparc64-cleanup.patch
local_t-x86_64-extension.patch
For 2.6.22
linux-kernel-markers-kconfig-menus.patch linux-kernel-markers-architecture-independant-code.patch linux-kernel-markers-powerpc-optimization.patch linux-kernel-markers-i386-optimization.patch markers-add-instrumentation-markers-menus-to-avr32.patch linux-kernel-markers-non-optimized-architectures.patch markers-alpha-and-avr32-supportadd-alpha-markerh-add-arm26-markerh.patch linux-kernel-markers-documentation.patch
#
markers-define-the-linker-macro-extra_rwdata.patch markers-use-extra_rwdata-in-architectures.patch
# some-grammatical-fixups-and-additions-to-atomich-kernel-doc.patch no-longer-include-asm-kdebugh.patch
Hold.

Signature · 3 lines

Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Andrew Morton

author maintainer

2007-05-02 23:22 UTC #116 lore raw

Quoted 16 lines · On Wed, 2 May 2007 19:11:04 -0400 Mathieu Desnoyers…

Quoted 9 lines
I didn't know that this was the plan.
The problem I have with this is that once we've merged one part, we're committed to merging the other parts even though we haven't seen them yet.
What happens if there's a revolt over the next set of patches? Do we remove the core markers patches again? We end up in a cant-go-forward, cant-go-backward situation.
Quoted 6 lines
I thought the existing code was useful as-is for several projects, without requiring additional patching to core kernel. If such additional patching _is_ needed to make the markers code useful then I agree that we should continue to buffer the markers code in -mm until the use-markers-for-something patches have been eyeballed.

Quoted 5 lines

My statement was probably not clear enough. The actual marker code is useful as-is without any further kernel patching required : SystemTAP is an example where they use external modules to load probes that can connect either to markers or through kprobes. LTTng, in its current state, has a mostly modular core that also uses the markers.

OK, that's what I thought.

Quoted 8 lines

Although some, like Christoph and myself, think that it would benefit to the kernel community to have a common infrastructure for more than just markers (meaning common serialization and buffering mechanism), it does not change the fact that the markers, being in mainline, are usable by projects through additional kernel modules.
If we are looking at current "potential users" that are already in mainline, we could change blktrace to make it use the markers.

That'd be a useful demonstration.

Mathieu Desnoyers

2007-05-03 15:09 UTC #117 lore raw

Quoted 10 lines · * Andrew Morton ([email protected]) wrote:

Quoted 8 lines
Although some, like Christoph and myself, think that it would benefit to the kernel community to have a common infrastructure for more than just markers (meaning common serialization and buffering mechanism), it does not change the fact that the markers, being in mainline, are usable by projects through additional kernel modules.
If we are looking at current "potential users" that are already in mainline, we could change blktrace to make it use the markers.
That'd be a useful demonstration.

Here is a proof of concept patch, for demonstration purpose, of moving blktrace to the markers.

A few remarks : this patch has the positive effect of removing some code from the block io tracing hot paths, minimizing the i-cache impact in a system where the io tracing is compiled in but inactive.

It also moves the blk tracing code from a header (and therefore from the body of the instrumented functions) to a separate C file.

There, as soon as one device has to be traced, every devices have to fall into the tracing function call. This is slower than the previous inline function which tested the condition quickly. If it becomes a show stopper, it could be fixed by having the possibility to test a supplementary condition, dependant of the marker context, at the marker site, just after the enable/disable test.

It does not make the code smaller, since I left all the specialized tracing functions for requests, bio, generic, remap, which would go away once a generic infrastructure is in place to serialize the information passed to the marker. This is mostly why I consider it a proof a concept.

Patch named "markers-port-blktrace-to-markers.patch", can be placed after the marker patches in the 2.6.21-rc7-mm2 series.

Signed-off-by Mathieu Desnoyers <[email protected]>

block/elevator.c +3 -3

Index: linux-2.6-lttng/block/elevator.c
===================================================================
@@ -32,7 +32,7 @@
 #include <linux/init.h>
 #include <linux/compiler.h>
 #include <linux/delay.h>
-#include <linux/blktrace_api.h>
+#include <linux/marker.h>
 #include <linux/hash.h>
 
 #include <asm/uaccess.h>

@@ -571,7 +571,7 @@
 	unsigned ordseq;
 	int unplug_it = 1;
 
-	blk_add_trace_rq(q, rq, BLK_TA_INSERT);
+	MARK(blk_request_insert, "%p %p", q, rq);
 
 	rq->q = q;

@@ -757,7 +757,7 @@
 			 * not be passed by new incoming requests
 			 */
 			rq->cmd_flags |= REQ_STARTED;
-			blk_add_trace_rq(q, rq, BLK_TA_ISSUE);
+			MARK(blk_request_issue, "%p %p", q, rq);
 		}
 
 		if (!q->boundary_rq || q->boundary_rq == rq) {

block/ll_rw_blk.c +14 ~~-13~~

Index: linux-2.6-lttng/block/ll_rw_blk.c
===================================================================
@@ -28,6 +28,7 @@
 #include <linux/task_io_accounting_ops.h>
 #include <linux/interrupt.h>
 #include <linux/cpu.h>
+#include <linux/marker.h>
 #include <linux/blktrace_api.h>
 #include <linux/fault-inject.h>

@@ -1551,7 +1552,7 @@
 
 	if (!test_and_set_bit(QUEUE_FLAG_PLUGGED, &q->queue_flags)) {
 		mod_timer(&q->unplug_timer, jiffies + q->unplug_delay);
-		blk_add_trace_generic(q, NULL, 0, BLK_TA_PLUG);
+		MARK(blk_plug_device, "%p %p %d", q, NULL, 0);
 	}
 }

@@ -1617,7 +1618,7 @@
 	 * devices don't necessarily have an ->unplug_fn defined
 	 */
 	if (q->unplug_fn) {
-		blk_add_trace_pdu_int(q, BLK_TA_UNPLUG_IO, NULL,
+		MARK(blk_pdu_unplug_io, "%p %p %d", q, NULL,
 					q->rq.count[READ] + q->rq.count[WRITE]);
 
 		q->unplug_fn(q);

@@ -1628,7 +1629,7 @@
 {
 	request_queue_t *q = container_of(work, request_queue_t, unplug_work);
 
-	blk_add_trace_pdu_int(q, BLK_TA_UNPLUG_IO, NULL,
+	MARK(blk_pdu_unplug_io, "%p %p %d", q, NULL,
 				q->rq.count[READ] + q->rq.count[WRITE]);
 
 	q->unplug_fn(q);

@@ -1638,7 +1639,7 @@
 {
 	request_queue_t *q = (request_queue_t *)data;
 
-	blk_add_trace_pdu_int(q, BLK_TA_UNPLUG_TIMER, NULL,
+	MARK(blk_pdu_unplug_timer, "%p %p %d", q, NULL,
 				q->rq.count[READ] + q->rq.count[WRITE]);
 
 	kblockd_schedule_work(&q->unplug_work);

@@ -2148,7 +2149,7 @@
 	
 	rq_init(q, rq);
 
-	blk_add_trace_generic(q, bio, rw, BLK_TA_GETRQ);
+	MARK(blk_get_request, "%p %p %d", q, bio, rw);
 out:
 	return rq;
 }

@@ -2178,7 +2179,7 @@
 		if (!rq) {
 			struct io_context *ioc;
 
-			blk_add_trace_generic(q, bio, rw, BLK_TA_SLEEPRQ);
+			MARK(blk_sleep_request, "%p %p %d", q, bio, rw);
 
 			__generic_unplug_device(q);
 			spin_unlock_irq(q->queue_lock);

@@ -2252,7 +2253,7 @@
  */
 void blk_requeue_request(request_queue_t *q, struct request *rq)
 {
-	blk_add_trace_rq(q, rq, BLK_TA_REQUEUE);
+	MARK(blk_requeue, "%p %p", q, rq);
 
 	if (blk_rq_tagged(rq))
 		blk_queue_end_tag(q, rq);

@@ -2937,7 +2938,7 @@
 			if (!ll_back_merge_fn(q, req, bio))
 				break;
 
-			blk_add_trace_bio(q, bio, BLK_TA_BACKMERGE);
+			MARK(blk_bio_backmerge, "%p %p", q, bio);
 
 			req->biotail->bi_next = bio;
 			req->biotail = bio;

@@ -2954,7 +2955,7 @@
 			if (!ll_front_merge_fn(q, req, bio))
 				break;
 
-			blk_add_trace_bio(q, bio, BLK_TA_FRONTMERGE);
+			MARK(blk_bio_frontmerge, "%p %p", q, bio);
 
 			bio->bi_next = req->bio;
 			req->bio = bio;

@@ -3184,10 +3185,10 @@
 		blk_partition_remap(bio);
 
 		if (old_sector != -1)
-			blk_add_trace_remap(q, bio, old_dev, bio->bi_sector, 
-					    old_sector);
+			MARK(blk_remap, "%p %p %u %llu %llu", q, bio, old_dev,
+					(u64)bio->bi_sector, (u64)old_sector);
 
-		blk_add_trace_bio(q, bio, BLK_TA_QUEUE);
+		MARK(blk_bio_queue, "%p %p", q, bio);
 
 		old_sector = bio->bi_sector;
 		old_dev = bio->bi_bdev->bd_dev;

@@ -3329,7 +3330,7 @@
 	int total_bytes, bio_nbytes, error, next_idx = 0;
 	struct bio *bio;
 
-	blk_add_trace_rq(req->q, req, BLK_TA_COMPLETE);
+	MARK(blk_request_complete, "%p %p", req->q, req);
 
 	/*
 	 * extend uptodate bool to allow < 0 value to be direct io error

block/Kconfig +1 -0

Index: linux-2.6-lttng/block/Kconfig
===================================================================
@@ -32,6 +32,7 @@
 	depends on SYSFS
 	select RELAY
 	select DEBUG_FS
+	select MARKERS
 	help
 	  Say Y here, if you want to be able to trace the block layer actions
 	  on a given queue. Tracing allows you to see any traffic happening

block/Makefile +1 -1

Index: linux-2.6-lttng/block/Makefile
===================================================================
@@ -9,4 +9,4 @@
 obj-$(CONFIG_IOSCHED_DEADLINE)	+= deadline-iosched.o
 obj-$(CONFIG_IOSCHED_CFQ)	+= cfq-iosched.o
 
-obj-$(CONFIG_BLK_DEV_IO_TRACE)	+= blktrace.o
+obj-$(CONFIG_BLK_DEV_IO_TRACE)	+= blktrace.o blk-probe.o

block/blk-probe.c new file +276 -0

276 lines not on this page · raw message

block/blktrace.c +19 -0

Index: linux-2.6-lttng/block/blktrace.c
===================================================================
@@ -28,6 +28,10 @@
 static DEFINE_PER_CPU(unsigned long long, blk_trace_cpu_offset) = { 0, };
 static unsigned int blktrace_seq __read_mostly = 1;
 
+/* Global reference count of probes */
+static struct mutex blk_probe_mutex;
+static int blk_probes_ref = 0;
+
 /*
  * Send out a notify message.
  */

@@ -229,6 +233,12 @@
 	blk_remove_tree(bt->dir);
 	free_percpu(bt->sequence);
 	kfree(bt);
+	mutex_lock(&blk_probe_mutex);
+	if (blk_probes_ref == 1) {
+		blk_probe_disconnect();
+		blk_probes_ref--;
+	}
+	mutex_unlock(&blk_probe_mutex);
 }
 
 static int blk_trace_remove(request_queue_t *q)

@@ -386,6 +396,14 @@
 		goto err;
 	}
 
+	/* Connect probes to markers */
+	mutex_lock(&blk_probe_mutex);
+	if (!blk_probes_ref) {
+		blk_probe_connect();
+		blk_probes_ref++;
+	}
+	mutex_unlock(&blk_probe_mutex);
+
 	return 0;
 err:
 	if (dir)

@@ -552,6 +570,7 @@
 static __init int blk_trace_init(void)
 {
 	mutex_init(&blk_tree_mutex);
+	mutex_init(&blk_probe_mutex);
 	on_each_cpu(blk_trace_check_cpu_time, NULL, 1, 1);
 	blk_trace_set_ht_offsets();

include/linux/blktrace_api.h +11 ~~-135~~

166 lines not on this page · raw message

mm/bounce.c +2 -2

Index: linux-2.6-lttng/mm/bounce.c
===================================================================
@@ -13,7 +13,7 @@
 #include <linux/init.h>
 #include <linux/hash.h>
 #include <linux/highmem.h>
-#include <linux/blktrace_api.h>
+#include <linux/marker.h>
 #include <asm/tlbflush.h>
 
 #define POOL_SIZE	64

@@ -237,7 +237,7 @@
 	if (!bio)
 		return;
 
-	blk_add_trace_bio(q, *bio_orig, BLK_TA_BOUNCE);
+	MARK(blk_bio_bounce, "%p %p", q, *bio_orig);
 
 	/*
 	 * at least one page was bounced, fill in possible non-highmem

mm/highmem.c +1 -1

Index: linux-2.6-lttng/mm/highmem.c
===================================================================
@@ -26,7 +26,7 @@
 #include <linux/init.h>
 #include <linux/hash.h>
 #include <linux/highmem.h>
-#include <linux/blktrace_api.h>
+#include <linux/marker.h>
 #include <asm/tlbflush.h>
 
 /*

fs/bio.c +2 -2

Index: linux-2.6-lttng/fs/bio.c
===================================================================
@@ -25,7 +25,7 @@
 #include <linux/module.h>
 #include <linux/mempool.h>
 #include <linux/workqueue.h>
-#include <linux/blktrace_api.h>
+#include <linux/marker.h>
 #include <scsi/sg.h>		/* for struct sg_iovec */
 
 #define BIO_POOL_SIZE 2

@@ -1081,7 +1081,7 @@
 	if (!bp)
 		return bp;
 
-	blk_add_trace_pdu_int(bdev_get_queue(bi->bi_bdev), BLK_TA_SPLIT, bi,
+	MARK(blk_pdu_split, "%p %p %d", bdev_get_queue(bi->bi_bdev), bi,
 				bi->bi_sector + first_sectors);
 
 	BUG_ON(bi->bi_vcnt != 1);

drivers/block/cciss.c +2 -2

Index: linux-2.6-lttng/drivers/block/cciss.c
===================================================================
@@ -37,7 +37,7 @@
 #include <linux/hdreg.h>
 #include <linux/spinlock.h>
 #include <linux/compat.h>
-#include <linux/blktrace_api.h>
+#include <linux/marker.h>
 #include <asm/uaccess.h>
 #include <asm/io.h>

@@ -2502,7 +2502,7 @@
 	}
 	cmd->rq->data_len = 0;
 	cmd->rq->completion_data = cmd;
-	blk_add_trace_rq(cmd->rq->q, cmd->rq, BLK_TA_COMPLETE);
+	MARK(blk_request_complete, "%p %p", cmd->rq->q, cmd->rq);
 	blk_complete_request(cmd->rq);
 }

drivers/md/dm.c +7 -7

Index: linux-2.6-lttng/drivers/md/dm.c
===================================================================
@@ -19,7 +19,7 @@
 #include <linux/slab.h>
 #include <linux/idr.h>
 #include <linux/hdreg.h>
-#include <linux/blktrace_api.h>
+#include <linux/marker.h>
 #include <linux/smp_lock.h>
 
 #define DM_MSG_PREFIX "core"

@@ -485,8 +485,8 @@
 			wake_up(&io->md->wait);
 
 		if (io->error != DM_ENDIO_REQUEUE) {
-			blk_add_trace_bio(io->md->queue, io->bio,
-					  BLK_TA_COMPLETE);
+			MARK(blk_request_complete, "%p %p",
+				io->md->queue, io->bio);
 
 			bio_endio(io->bio, io->bio->bi_size, io->error);
 		}

@@ -582,10 +582,10 @@
 	r = ti->type->map(ti, clone, &tio->info);
 	if (r == DM_MAPIO_REMAPPED) {
 		/* the bio has been remapped so dispatch it */
-
-		blk_add_trace_remap(bdev_get_queue(clone->bi_bdev), clone,
-				    tio->io->bio->bi_bdev->bd_dev, sector,
-				    clone->bi_sector);
+		MARK(blk_remap, "%p %p %u %llu %llu",
+			bdev_get_queue(clone->bi_bdev), clone,
+			(u64)tio->io->bio->bi_bdev->bd_dev, (u64)sector,
+			(u64)clone->bi_sector);
 
 		generic_make_request(clone);
 	} else if (r < 0 || r == DM_MAPIO_REQUEUE) {

Signature · 3 lines

Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Christoph Hellwig

2007-05-03 15:12 UTC #118 lore raw

block/elevator.c line 574

Quoted 2 lines · Mathieu Desnoyers

block/elevator.c +1 -1

-	blk_add_trace_rq(q, rq, BLK_TA_INSERT);
+	MARK(blk_request_insert, "%p %p", q, rq);

I don't really like the shouting MARK name very much. Can we have a less-generic, less shouting name, e.g. trace_marker? The aboe would then be:

	trace_mark(blk_request_insert, "%p %p", q, rq);

Quoted 1 line

+#define NUM_PROBES ARRAY_SIZE(probe_array)

just get rid of this and use ARRAY_SIZE diretly below.

block/blk-probe.c line 248

Quoted 4 lines

block/blk-probe.c new file +4 -0

+int blk_probe_connect(void)
+{
+	int result;
+	uint8_t i;

just use an int for for loops. it's easy to read and probably faster on most systems (it the compiler isn't smart enough and promotes it to int anyway during code generation)

block/blk-probe.c line 270

Quoted 8 lines

block/blk-probe.c new file +8 -0

+void blk_probe_disconnect(void)
+{
+	uint8_t i;
+
+	for (i = 0; i < NUM_PROBES; i++) {
+		marker_remove_probe(probe_array[i].name);
+	}
+	synchronize_sched();	/* Wait for probes to finish */

kprobes does this kind of synchronization internally, so the marker wrapper should probabl aswell.

Quoted 1 line

+static int blk_probes_ref = 0;

no need to initialize this.

block/blktrace.c line 240

Quoted 12 lines

/*
* Send out a notify message.
*/

block/blktrace.c +5 -0

@@ -229,6 +233,12 @@
 	blk_remove_tree(bt->dir);
 	free_percpu(bt->sequence);
 	kfree(bt);
+	mutex_lock(&blk_probe_mutex);
+	if (blk_probes_ref == 1) {
+		blk_probe_disconnect();
+		blk_probes_ref--;
+	}

	if (--blk_probes_ref == 0)
		blk_probe_disconnect();

would probably be a tad cleaner.

block/blktrace.c line 404

Quoted 4 lines

block/blktrace.c +4 -0

+	if (!blk_probes_ref) {
+		blk_probe_connect();
+		blk_probes_ref++;
+	}

Dito here with a:

	if (!blk_probes_ref++)
		blk_probe_connect();

also the connect in the name seems rather add, what about arm/disarm instead?

block/blktrace.c line 573

Quoted 4 lines

block/blktrace.c +1 -0

 static __init int blk_trace_init(void)
 {
 	mutex_init(&blk_tree_mutex);
+	mutex_init(&blk_probe_mutex);

both should use DEFINE_MUTEX for compile-time initialization isntead.

Also it's probably better to put the trace points into blktrace.c, that means all blktrace code can be static and self-contained. And we can probably do some additional cleanups by simplifying things later on.

Mathieu Desnoyers

2007-05-03 17:22 UTC #119 lore raw

Here is the reworked patch, except a comment :

Quoted 12 lines · * Christoph Hellwig ([email protected]) wrote:

Review
Quoted 8 lines
(quoted diff) +8 -0
+void blk_probe_disconnect(void)
+{
+	uint8_t i;
+
+	for (i = 0; i < NUM_PROBES; i++) {
+		marker_remove_probe(probe_array[i].name);
+	}
+	synchronize_sched();	/* Wait for probes to finish */
kprobes does this kind of synchronization internally, so the marker wrapper should probabl aswell.

The problem appears on heavily loaded systems. Doing 50 synchronize_sched() calls in a row can take up to a few seconds on a 4-way machine. This is why I prefer to do it in the module to which the callbacks belong.

Here is the reviewed patch. It depends on a newer version of markers I'll send to Andrew soon.

Signed-off-by Mathieu Desnoyers <[email protected]>

block/elevator.c +3 -3

Index: linux-2.6-lttng/block/elevator.c
===================================================================
@@ -32,7 +32,7 @@
 #include <linux/init.h>
 #include <linux/compiler.h>
 #include <linux/delay.h>
-#include <linux/blktrace_api.h>
+#include <linux/marker.h>
 #include <linux/hash.h>
 
 #include <asm/uaccess.h>

@@ -571,7 +571,7 @@
 	unsigned ordseq;
 	int unplug_it = 1;
 
-	blk_add_trace_rq(q, rq, BLK_TA_INSERT);
+	trace_mark(blk_request_insert, "%p %p", q, rq);
 
 	rq->q = q;

@@ -757,7 +757,7 @@
 			 * not be passed by new incoming requests
 			 */
 			rq->cmd_flags |= REQ_STARTED;
-			blk_add_trace_rq(q, rq, BLK_TA_ISSUE);
+			trace_mark(blk_request_issue, "%p %p", q, rq);
 		}
 
 		if (!q->boundary_rq || q->boundary_rq == rq) {

block/ll_rw_blk.c +14 ~~-13~~

Index: linux-2.6-lttng/block/ll_rw_blk.c
===================================================================
@@ -28,6 +28,7 @@
 #include <linux/task_io_accounting_ops.h>
 #include <linux/interrupt.h>
 #include <linux/cpu.h>
+#include <linux/marker.h>
 #include <linux/blktrace_api.h>
 #include <linux/fault-inject.h>

@@ -1551,7 +1552,7 @@
 
 	if (!test_and_set_bit(QUEUE_FLAG_PLUGGED, &q->queue_flags)) {
 		mod_timer(&q->unplug_timer, jiffies + q->unplug_delay);
-		blk_add_trace_generic(q, NULL, 0, BLK_TA_PLUG);
+		trace_mark(blk_plug_device, "%p %p %d", q, NULL, 0);
 	}
 }

@@ -1617,7 +1618,7 @@
 	 * devices don't necessarily have an ->unplug_fn defined
 	 */
 	if (q->unplug_fn) {
-		blk_add_trace_pdu_int(q, BLK_TA_UNPLUG_IO, NULL,
+		trace_mark(blk_pdu_unplug_io, "%p %p %d", q, NULL,
 					q->rq.count[READ] + q->rq.count[WRITE]);
 
 		q->unplug_fn(q);

@@ -1628,7 +1629,7 @@
 {
 	request_queue_t *q = container_of(work, request_queue_t, unplug_work);
 
-	blk_add_trace_pdu_int(q, BLK_TA_UNPLUG_IO, NULL,
+	trace_mark(blk_pdu_unplug_io, "%p %p %d", q, NULL,
 				q->rq.count[READ] + q->rq.count[WRITE]);
 
 	q->unplug_fn(q);

@@ -1638,7 +1639,7 @@
 {
 	request_queue_t *q = (request_queue_t *)data;
 
-	blk_add_trace_pdu_int(q, BLK_TA_UNPLUG_TIMER, NULL,
+	trace_mark(blk_pdu_unplug_timer, "%p %p %d", q, NULL,
 				q->rq.count[READ] + q->rq.count[WRITE]);
 
 	kblockd_schedule_work(&q->unplug_work);

@@ -2148,7 +2149,7 @@
 	
 	rq_init(q, rq);
 
-	blk_add_trace_generic(q, bio, rw, BLK_TA_GETRQ);
+	trace_mark(blk_get_request, "%p %p %d", q, bio, rw);
 out:
 	return rq;
 }

@@ -2178,7 +2179,7 @@
 		if (!rq) {
 			struct io_context *ioc;
 
-			blk_add_trace_generic(q, bio, rw, BLK_TA_SLEEPRQ);
+			trace_mark(blk_sleep_request, "%p %p %d", q, bio, rw);
 
 			__generic_unplug_device(q);
 			spin_unlock_irq(q->queue_lock);

@@ -2252,7 +2253,7 @@
  */
 void blk_requeue_request(request_queue_t *q, struct request *rq)
 {
-	blk_add_trace_rq(q, rq, BLK_TA_REQUEUE);
+	trace_mark(blk_requeue, "%p %p", q, rq);
 
 	if (blk_rq_tagged(rq))
 		blk_queue_end_tag(q, rq);

@@ -2937,7 +2938,7 @@
 			if (!ll_back_merge_fn(q, req, bio))
 				break;
 
-			blk_add_trace_bio(q, bio, BLK_TA_BACKMERGE);
+			trace_mark(blk_bio_backmerge, "%p %p", q, bio);
 
 			req->biotail->bi_next = bio;
 			req->biotail = bio;

@@ -2954,7 +2955,7 @@
 			if (!ll_front_merge_fn(q, req, bio))
 				break;
 
-			blk_add_trace_bio(q, bio, BLK_TA_FRONTMERGE);
+			trace_mark(blk_bio_frontmerge, "%p %p", q, bio);
 
 			bio->bi_next = req->bio;
 			req->bio = bio;

@@ -3184,10 +3185,10 @@
 		blk_partition_remap(bio);
 
 		if (old_sector != -1)
-			blk_add_trace_remap(q, bio, old_dev, bio->bi_sector, 
-					    old_sector);
+			trace_mark(blk_remap, "%p %p %u %llu %llu", q, bio, old_dev,
+					(u64)bio->bi_sector, (u64)old_sector);
 
-		blk_add_trace_bio(q, bio, BLK_TA_QUEUE);
+		trace_mark(blk_bio_queue, "%p %p", q, bio);
 
 		old_sector = bio->bi_sector;
 		old_dev = bio->bi_bdev->bd_dev;

@@ -3329,7 +3330,7 @@
 	int total_bytes, bio_nbytes, error, next_idx = 0;
 	struct bio *bio;
 
-	blk_add_trace_rq(req->q, req, BLK_TA_COMPLETE);
+	trace_mark(blk_request_complete, "%p %p", req->q, req);
 
 	/*
 	 * extend uptodate bool to allow < 0 value to be direct io error

block/Kconfig +1 -0

Index: linux-2.6-lttng/block/Kconfig
===================================================================
@@ -32,6 +32,7 @@
 	depends on SYSFS
 	select RELAY
 	select DEBUG_FS
+	select MARKERS
 	help
 	  Say Y here, if you want to be able to trace the block layer actions
 	  on a given queue. Tracing allows you to see any traffic happening

block/blktrace.c +280 -2

319 lines not on this page · raw message

include/linux/blktrace_api.h +11 ~~-135~~

166 lines not on this page · raw message

mm/bounce.c +2 -2

Index: linux-2.6-lttng/mm/bounce.c
===================================================================
@@ -13,7 +13,7 @@
 #include <linux/init.h>
 #include <linux/hash.h>
 #include <linux/highmem.h>
-#include <linux/blktrace_api.h>
+#include <linux/marker.h>
 #include <asm/tlbflush.h>
 
 #define POOL_SIZE	64

@@ -237,7 +237,7 @@
 	if (!bio)
 		return;
 
-	blk_add_trace_bio(q, *bio_orig, BLK_TA_BOUNCE);
+	trace_mark(blk_bio_bounce, "%p %p", q, *bio_orig);
 
 	/*
 	 * at least one page was bounced, fill in possible non-highmem

mm/highmem.c +1 -1

Index: linux-2.6-lttng/mm/highmem.c
===================================================================
@@ -26,7 +26,7 @@
 #include <linux/init.h>
 #include <linux/hash.h>
 #include <linux/highmem.h>
-#include <linux/blktrace_api.h>
+#include <linux/marker.h>
 #include <asm/tlbflush.h>
 
 /*

fs/bio.c +2 -2

Index: linux-2.6-lttng/fs/bio.c
===================================================================
@@ -25,7 +25,7 @@
 #include <linux/module.h>
 #include <linux/mempool.h>
 #include <linux/workqueue.h>
-#include <linux/blktrace_api.h>
+#include <linux/marker.h>
 #include <scsi/sg.h>		/* for struct sg_iovec */
 
 #define BIO_POOL_SIZE 2

@@ -1081,7 +1081,7 @@
 	if (!bp)
 		return bp;
 
-	blk_add_trace_pdu_int(bdev_get_queue(bi->bi_bdev), BLK_TA_SPLIT, bi,
+	trace_mark(blk_pdu_split, "%p %p %d", bdev_get_queue(bi->bi_bdev), bi,
 				bi->bi_sector + first_sectors);
 
 	BUG_ON(bi->bi_vcnt != 1);

drivers/block/cciss.c +2 -2

Index: linux-2.6-lttng/drivers/block/cciss.c
===================================================================
@@ -37,7 +37,7 @@
 #include <linux/hdreg.h>
 #include <linux/spinlock.h>
 #include <linux/compat.h>
-#include <linux/blktrace_api.h>
+#include <linux/marker.h>
 #include <asm/uaccess.h>
 #include <asm/io.h>

@@ -2502,7 +2502,7 @@
 	}
 	cmd->rq->data_len = 0;
 	cmd->rq->completion_data = cmd;
-	blk_add_trace_rq(cmd->rq->q, cmd->rq, BLK_TA_COMPLETE);
+	trace_mark(blk_request_complete, "%p %p", cmd->rq->q, cmd->rq);
 	blk_complete_request(cmd->rq);
 }

drivers/md/dm.c +7 -7

Index: linux-2.6-lttng/drivers/md/dm.c
===================================================================
@@ -19,7 +19,7 @@
 #include <linux/slab.h>
 #include <linux/idr.h>
 #include <linux/hdreg.h>
-#include <linux/blktrace_api.h>
+#include <linux/marker.h>
 #include <linux/smp_lock.h>
 
 #define DM_MSG_PREFIX "core"

@@ -485,8 +485,8 @@
 			wake_up(&io->md->wait);
 
 		if (io->error != DM_ENDIO_REQUEUE) {
-			blk_add_trace_bio(io->md->queue, io->bio,
-					  BLK_TA_COMPLETE);
+			trace_mark(blk_request_complete, "%p %p",
+				io->md->queue, io->bio);
 
 			bio_endio(io->bio, io->bio->bi_size, io->error);
 		}

@@ -582,10 +582,10 @@
 	r = ti->type->map(ti, clone, &tio->info);
 	if (r == DM_MAPIO_REMAPPED) {
 		/* the bio has been remapped so dispatch it */
-
-		blk_add_trace_remap(bdev_get_queue(clone->bi_bdev), clone,
-				    tio->io->bio->bi_bdev->bd_dev, sector,
-				    clone->bi_sector);
+		trace_mark(blk_remap, "%p %p %u %llu %llu",
+			bdev_get_queue(clone->bi_bdev), clone,
+			(u64)tio->io->bio->bi_bdev->bd_dev, (u64)sector,
+			(u64)clone->bi_sector);
 
 		generic_make_request(clone);
 	} else if (r < 0 || r == DM_MAPIO_REQUEUE) {

Signature · 3 lines

Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Christoph Hellwig

2007-05-03 17:25 UTC #120 lore raw

Quoted 8 lines · Mathieu Desnoyers

Quoted 3 lines
kprobes does this kind of synchronization internally, so the marker wrapper should probabl aswell.
The problem appears on heavily loaded systems. Doing 50 synchronize_sched() calls in a row can take up to a few seconds on a 4-way machine. This is why I prefer to do it in the module to which the callbacks belong.

We recently had a discussion on batch unreistration interface for kprobes. I'm not very happy with having so different interfaces for different kind of probe registrations.

Mathieu Desnoyers

2007-05-10 19:39 UTC #121 lore raw

Quoted 5 lines · * Christoph Hellwig ([email protected]) wrote:

Quoted 4 lines · Mathieu Desnoyers
Quoted 3 lines
kprobes does this kind of synchronization internally, so the marker wrapper should probabl aswell.

Quoted 9 lines

Quoted 4 lines
The problem appears on heavily loaded systems. Doing 50 synchronize_sched() calls in a row can take up to a few seconds on a 4-way machine. This is why I prefer to do it in the module to which the callbacks belong.
We recently had a discussion on batch unreistration interface for kprobes. I'm not very happy with having so different interfaces for different kind of probe registrations.

Ok, I've had a look at the kprobes batch registration mechanisms and.. well, it does not look well suited for the markers. Adding supplementary data structures such as linked lists of probes does not look like a good match.

However, I agree with you that providing a similar API is good.

Therefore, here is my proposal :

The goal is to do the synchronize just after we unregister the last probe handler provided by a given module. Since the unregistration functions iterate on every marker present in the kernel, we can keep a count of how many probes provided by the same module are still present. If we see that we unregistered the last probe pointing to this module, we issue a synchronize_sched().

It adds no data structure and keeps the same order of complexity as what is already there, we only have to do 2 passes in the marker structures :
the first one finds the module associated with the callback and the second disables the callbacks and keep a count of the number of callbacks associated with the module.

Mathieu

P.S.: here is the code.

Linux Kernel Markers - Architecture Independant code Provide internal synchronize_sched() in batch.

Signed-off-by Mathieu Desnoyers <[email protected]>

---
 kernel/module.c |   62 ++++++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 52 insertions(+), 10 deletions(-)

kernel/module.c +52 ~~-10~~

Index: linux-2.6-lttng/kernel/module.c
===================================================================
@@ -404,8 +404,12 @@
 }
 
 /* Sets a range of markers to a disabled state : unset the enable bit and
- * provide the empty callback. */
+ * provide the empty callback.
+ * Keep a count of other markers connected to the same module as the one
+ * provided as parameter. */
 static int marker_remove_probe_range(const char *name,
+	struct module *probe_module,
+	int *ref_count,
 	const struct __mark_marker *begin,
 	const struct __mark_marker *end)
 {

@@ -413,12 +417,17 @@
 	int found = 0;
 
 	for (iter = begin; iter < end; iter++) {
-		if (strcmp(name, iter->mdata->name) == 0) {
-			marker_set_enable(iter->enable, 0,
-				iter->mdata->flags);
-			iter->mdata->call = __mark_empty_function;
-			found++;
+		if (strcmp(name, iter->mdata->name) != 0) {
+			if (probe_module)
+				if (__module_text_address(
+					(unsigned long)iter->mdata->call)
+						== probe_module)
+					(*ref_count)++;
+			continue;
 		}
+		marker_set_enable(iter->enable, 0, iter->mdata->flags);
+		iter->mdata->call = __mark_empty_function;
+		found++;
 	}
 	return found;
 }

@@ -450,6 +459,29 @@
 	return found;
 }
 
+/* Get the module to which the probe handler's text belongs.
+ * Called with module_mutex taken.
+ * Returns NULL if the probe handler is not in a module. */
+static struct module *__marker_get_probe_module(const char *name)
+{
+	struct module *mod;
+	const struct __mark_marker *iter;
+
+	list_for_each_entry(mod, &modules, list) {
+		if (mod->taints)
+			continue;
+		for (iter = mod->markers;
+			iter < mod->markers+mod->num_markers; iter++) {
+			if (strcmp(name, iter->mdata->name) != 0)
+				continue;
+			if (iter->mdata->call)
+				return __module_text_address(
+					(unsigned long)iter->mdata->call);
+		}
+	}
+	return NULL;
+}
+
 /* Calls _marker_set_probe_range for the core markers and modules markers.
  * Marker enabling/disabling use the modlist_lock to synchronise. */
 int _marker_set_probe(int flags, const char *name, const char *format,

@@ -477,23 +509,33 @@
 EXPORT_SYMBOL_GPL(_marker_set_probe);
 
 /* Calls _marker_remove_probe_range for the core markers and modules markers.
- * Marker enabling/disabling use the modlist_lock to synchronise. */
+ * Marker enabling/disabling use the modlist_lock to synchronise.
+ * ref_count is the number of markers still connected to the same module
+ * as the one in which sits the probe handler currently removed, excluding the
+ * one currently removed. If the count is 0, we issue a synchronize_sched() to
+ * make sure the module can safely unload. */
 int marker_remove_probe(const char *name)
 {
-	struct module *mod;
+	struct module *mod, *probe_module;
 	int found = 0;
+	int ref_count = 0;
 
 	mutex_lock(&module_mutex);
+	/* In what module is the probe handler ? */
+	probe_module = __marker_get_probe_module(name);
 	/* Core kernel markers */
-	found += marker_remove_probe_range(name,
+	found += marker_remove_probe_range(name, probe_module, &ref_count,
 			__start___markers, __stop___markers);
 	/* Markers in modules. */
 	list_for_each_entry(mod, &modules, list) {
 		if (!mod->taints)
-			found += marker_remove_probe_range(name,
+			found += marker_remove_probe_range(name, probe_module,
+				&ref_count,
 				mod->markers, mod->markers+mod->num_markers);
 	}
 	mutex_unlock(&module_mutex);
+	if (!ref_count)
+		synchronize_sched();
 	return found;
 }
 EXPORT_SYMBOL_GPL(marker_remove_probe);

Signature · 3 lines

Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Christoph Hellwig

2007-05-13 21:04 UTC #122 lore raw

Quoted 53 lines · Mathieu Desnoyers

Quoted 5 lines · * Christoph Hellwig ([email protected]) wrote:
Quoted 4 lines · Mathieu Desnoyers
Quoted 3 lines
kprobes does this kind of synchronization internally, so the marker wrapper should probabl aswell.
Quoted 9 lines
Quoted 4 lines
The problem appears on heavily loaded systems. Doing 50 synchronize_sched() calls in a row can take up to a few seconds on a 4-way machine. This is why I prefer to do it in the module to which the callbacks belong.
We recently had a discussion on batch unreistration interface for kprobes. I'm not very happy with having so different interfaces for different kind of probe registrations.
Ok, I've had a look at the kprobes batch registration mechanisms and.. well, it does not look well suited for the markers. Adding supplementary data structures such as linked lists of probes does not look like a good match.
However, I agree with you that providing a similar API is good.
Therefore, here is my proposal :
The goal is to do the synchronize just after we unregister the last probe handler provided by a given module. Since the unregistration functions iterate on every marker present in the kernel, we can keep a count of how many probes provided by the same module are still present. If we see that we unregistered the last probe pointing to this module, we issue a synchronize_sched().
It adds no data structure and keeps the same order of complexity as what is already there, we only have to do 2 passes in the marker structures :
the first one finds the module associated with the callback and the second disables the callbacks and keep a count of the number of callbacks associated with the module.
Mathieu
P.S.: here is the code.
Linux Kernel Markers - Architecture Independant code Provide internal synchronize_sched() in batch.
The goal is to do the synchronize just after we unregister the last probe handler provided by a given module. Since the unregistration functions iterate on every marker present in the kernel, we can keep a count of how many probes provided by the same module are still present. If we see that we unregistered the last probe pointing to this module, we issue a synchronize_sched().

Quoted 5 lines

It adds no data structure and keeps the same order of complexity as what is already there, we only have to do 2 passes in the marker structures :
the first one finds the module associated with the callback and the second disables the callbacks and keep a count of the number of callbacks associated with the module.

Looks good to me, please incorporate this is the next round of the markers patch series.

Christoph Hellwig

2007-05-03 08:06 UTC #123 lore raw

Quoted 5 lines · Mathieu Desnoyers

My statement was probably not clear enough. The actual marker code is useful as-is without any further kernel patching required : SystemTAP is an example where they use external modules to load probes that can connect either to markers or through kprobes. LTTng, in its current state, has a mostly modular core that also uses the markers.

That just mean you have to load an enormous emount of exernal crap that replaces the missing kernel functionality. It's exactly the situation we want to avoid.

Mathieu Desnoyers

2007-05-03 14:48 UTC #124 lore raw

Quoted 11 lines · * Christoph Hellwig ([email protected]) wrote:

Quoted 5 lines · Mathieu Desnoyers
My statement was probably not clear enough. The actual marker code is useful as-is without any further kernel patching required : SystemTAP is an example where they use external modules to load probes that can connect either to markers or through kprobes. LTTng, in its current state, has a mostly modular core that also uses the markers.
That just mean you have to load an enormous emount of exernal crap that replaces the missing kernel functionality. It's exactly the situation we want to avoid.

It makes sense to use -mm to hold the hole usable infrastructure before submitting it to mainline. I will submit my core LTTng patches to Andrew in the following weeks. There is no hurry, in the LTTng perspective, to merge the markers sooner, although they could be useful to other (external) projects meanwhile.

Mathieu

Signature · 3 lines

Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Andi Kleen

2007-05-03 10:31 UTC #125 lore raw

In reply to Mathieu Desnoyers

Quoted 2 lines

If we are looking at current "potential users" that are already in mainline, we could change blktrace to make it use the markers.

Ok, but do it step by step:
- split out useful pieces like the "patched in enable/disable flags" and submit them separate with an example user or two
[I got a couple of candidates e.g. with some of the sysctls in VM or networking]
- post and merge that.
- don't implement anything initially that is not needed by blktrace
- post a minimal marker patch together with the blktrace conversion for review again on linux-kernel
- await review comments. This review would not cover the basic need of markers, just the specific implementation.
- then potentially merge incorporate review comments
- then merge
- later add features with individual review/discussion as new users in the kernel are added.

-Andi

Mathieu Desnoyers

2007-05-03 14:49 UTC #126 lore raw

Hi Andi,

This plan makes sense. I will split the "patched in enabled/disable flags" part into a separate piece (good idea!) and then submit the LTTng core to Andrew. Christoph's has a good point about wanting a usable infrastructure to go ini. Regarding your plan, I must argue that blktrace is not a general purpose tracing infrastructure, but one dedicated to block io tracing. Therefore, it makes sense to bring in the generic infrastructure first and then convert blktrace to it.

Mathieu

Quoted 20 lines · * Andi Kleen ([email protected]) wrote:

Quoted 2 lines
If we are looking at current "potential users" that are already in mainline, we could change blktrace to make it use the markers.
Ok, but do it step by step:
- split out useful pieces like the "patched in enable/disable flags" and submit them separate with an example user or two
[I got a couple of candidates e.g. with some of the sysctls in VM or networking]
- post and merge that.
- don't implement anything initially that is not needed by blktrace
- post a minimal marker patch together with the blktrace conversion for review again on linux-kernel
- await review comments. This review would not cover the basic need of markers, just the specific implementation.
- then potentially merge incorporate review comments
- then merge
- later add features with individual review/discussion as new users in the kernel are added.
-Andi

Signature · 3 lines

Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Christoph Hellwig

2007-05-03 08:09 UTC #127 lore raw

In reply to Mathieu Desnoyers

Quoted 24 lines · Andrew Morton

In which case we have:
atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-alpha.patch atomich-complete-atomic_long-operations-in-asm-generic.patch atomich-i386-type-safety-fix.patch atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-ia64.patch atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-mips.patch atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-parisc.patch atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-powerpc.patch atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-sparc64.patch atomich-add-atomic64-cmpxchg-xchg-and-add_unless-to-x86_64.patch atomich-atomic_add_unless-as-inline-remove-systemh-atomich-circular-dependency.patch local_t-architecture-independant-extension.patch local_t-alpha-extension.patch
local_t-i386-extension.patch
local_t-ia64-extension.patch
local_t-mips-extension.patch
local_t-parisc-cleanup.patch
local_t-powerpc-extension.patch
local_t-sparc64-cleanup.patch
local_t-x86_64-extension.patch
For 2.6.22

Quoted 14 lines

linux-kernel-markers-kconfig-menus.patch linux-kernel-markers-architecture-independant-code.patch linux-kernel-markers-powerpc-optimization.patch linux-kernel-markers-i386-optimization.patch markers-add-instrumentation-markers-menus-to-avr32.patch linux-kernel-markers-non-optimized-architectures.patch markers-alpha-and-avr32-supportadd-alpha-markerh-add-arm26-markerh.patch linux-kernel-markers-documentation.patch
#
markers-define-the-linker-macro-extra_rwdata.patch markers-use-extra_rwdata-in-architectures.patch
# some-grammatical-fixups-and-additions-to-atomich-kernel-doc.patch no-longer-include-asm-kdebugh.patch

This it a plan I can fully agree with.

Christoph Hellwig

2007-05-03 08:08 UTC #128 lore raw

Quoted 4 lines · Mathieu Desnoyers

The idea is the following : either we integrate the infrastructure for instrumentation / data serialization / buffer management / extraction of data to user space in multiple different steps, which makes code review easier for you guys,

the staging area for that is -mm

Andi Kleen

2007-05-02 17:49 UTC #129 lore raw

Quoted 10 lines · Andrew Morton

Quoted 8 lines · On Wed, 2 May 2007 12:44:13 +0200 Andi Kleen <[email protected]> wrote:
Quoted 7 lines
It is currently used as an instrumentation infrastructure for the LTTng tracer at IBM, Google, Autodesk, Sony, MontaVista and deployed in WindRiver products. The SystemTAP project also plan to use this type of infrastructure to trace sites hard to instrument. The Linux Kernel Markers has the support of Frank C. Eigler, author of their current marker alternative (which he wishes to drop in order to adopt the markers infrastructure as soon as it hits mainline).

Quoted 3 lines

Quoted 1 line
All of the above don't use mainline kernels.
That's because they have to add a markers patch!

I meant they use very old kernels. Their experiences don't apply to mainline bitrottyness.

Quoted 5 lines

Quoted 1 line
That doesn't constitute using it.
Andi, there was a huge amount of discussion about all this in September last year (subjects: *markers* and *LTTng*). The outcome of all that was, I believe, that the kernel should have a static marker infrastructure.

I have no problem with that in principle; just some doubts about the current proposed implementation: in particular its complexity. And also I think when something is merged it should have some users in tree.

-Andi

Tilman Schmidt

2007-05-02 21:43 UTC #130 lore raw

Am 02.05.2007 19:49 schrieb Andi Kleen:

Quoted 1 line

And also I think when something is merged it should have some users in tree.

Isn't that a circular dependency?

Signature · 3 lines

Tilman Schmidt                          E-Mail: [email protected]
Bonn, Germany
- Undetected errors are handled as if no error occurred. (IBM) -

Andi Kleen

2007-05-03 10:12 UTC #131 lore raw

Quoted 4 lines · Tilman Schmidt

Am 02.05.2007 19:49 schrieb Andi Kleen:
Quoted 1 line
And also I think when something is merged it should have some users in tree.
Isn't that a circular dependency?

The normal mode of operation is to merge the initial users and the subsystem at the same time.

-Andi

Mathieu Desnoyers

2007-05-02 17:24 UTC #132 lore raw

In reply to Andi Kleen

Quoted 11 lines · * Andi Kleen ([email protected]) wrote:

Quoted 7 lines
It is currently used as an instrumentation infrastructure for the LTTng tracer at IBM, Google, Autodesk, Sony, MontaVista and deployed in WindRiver products. The SystemTAP project also plan to use this type of infrastructure to trace sites hard to instrument. The Linux Kernel Markers has the support of Frank C. Eigler, author of their current marker alternative (which he wishes to drop in order to adopt the markers infrastructure as soon as it hits mainline).
All of the above don't use mainline kernels.
That doesn't constitute using it.

I am afraid this argument does not hold :

- These companies are not shipping their products with mainline kernels to make sure things have time to stabilize.
- They eventually get to the next version some time after it is not "head" anymore. They still want to benefit from the features of the newer versions.
- All these companies would be really happy to have a marker infrastructure in mainline so they can stop applying a separate set of patches to provide this functionality.
- Arguing the fact that "they apply their set of patches anyway" goes against the advice I have received from Greg KH, which is can be reworded as : please submit your patches to mainline instead of keeping your separate set of patches. See his various presentations about "mainlining" for more info about this.

Because of these 4 arguments, I think that these companies can be considered as users and contributors of/to mainline kernels.

Quoted 8 lines

Quoted 5 lines
Quoting Jim Keniston <[email protected]> :
"kprobes remains a vital foundation for SystemTap. But markers are attactive as an alternate source of trace/debug info. Here's why:
[...]"
Talk is cheap. Do they have working code to use it?

LTTng has been using the markers for about 6 months now. SystemTAP is waiting on the "it hits mainline" signal before they switch from their STP_MARK() markers to this infrastructure. Give them a few days and they will proceed to the change.

Quoted 8 lines

Quoted 3 lines
- Allow per-architecture optimized versions which removes the need for a d-cache based branch (patch a "load immediate" instruction instead). It minimized the d-cache impact of the disabled markers.
That's a good idea in general, but should be generalized (available independently), not hidden in your subsystem. I know a couple of places who could use this successfully.

I agree that an efficient hooking mechanism is useful to manyr; listing at least security hooks and instrumentation for tracing. What other usage scenario do you have in mind that could not fit in my marker infrastructure ? I have tried to generalize this as much as possible, but if you see, within this, a piece of infrastructure that could be taken apart and used more widely, I will be happy to submit it separately to increase its usefulness.

Quoted 7 lines

Quoted 4 lines
- Accept the cost of an unlikely branch at the marker site because the gcc compiler does not give the ability to put "nops" instead of a branch generated from C code. Keep this in mind for future per-architecture optimizations.
See upcomming paravirt code for a way to do this.

I have looked at the paravirt code in Andrew's 2.6.21-rc7-mm2. A few reasons why I do not plan to use it :

1 - It requires specific arg setup for the calls to be crafted by hand, in assembly, for each and every number of parameters and each types, for each architecture. I use a variable argument list as a parameter to my marker to make sure that a single macro can be used for markup in a generic manner.

Quoting : http://lkml.org/lkml/2007/4/4/577
"+ * Unfortunately there's no way to get gcc to generate the args setup
+ * for the call, and then allow the call itself to be generated by an
+ * inline asm. Because of this, we must do the complete arg setup and
+ * return value handling from within these macros. This is fairly
+ * cumbersome."

2 - I also provide an architecture independent "generic" version which does not depend on per-architecture assembly. From what I see, paravirt is only offered for i386 and x86_64. Are there any plans to support the other ~12 architectures ? Does it offer a architecture agnostic fallback in the cases where it is not implemented for a given architecture ?

3 - It can only alter instructions "safely" in the UP case before the other CPUs are turned on. See my arch/i386/marker.c code patcher for XMC-safe instruction patching. Marker activation must be done at runtime, when the system is fully operational.

Quoting 2.6.21 arch/i386/kernel/alternative.c
"/* Replace instructions with better alternatives for this CPU type.
This runs before SMP is initialized to avoid SMP problems with self modifying code. This implies that assymetric systems where APs have less capabilities than the boot processor are not handled. Tough. Make sure you disable such features by hand. */

void apply_alternatives(struct alt_instr *start, struct alt_instr *end)"

4 - paravirt does not offer the ability to replace a branch instruction, generated by gcc, through its mechanism. If I choose to use paravirt mechanism, I must do the stack setup and function call by hand, which has been argued in points (1) and (2). GCC must itself generate the branch instruction to jump over the function call containing the variable argument list.

Quoted 12 lines

Quoted 11 lines
- Instrumentation of challenging kernel sites
- Instrumentation such as the one provided in the already existing Lock dependency checker (lockdep) and instrumentation of trap handlers implies being reentrant for such context. Therefore, the implementation must be lock-free and update the state in an atomic fashion (rcu-style). It must also let the programmer who describes a marker site the ability to specify what is forbidden in the probe that will be connected to the marker : can it generate a trap ? Can it call lockdep (irq disable, take any type of lock), can it call printk ? This is why flags can be passed to the _MARK() marker, while the MARK() marker has the default flags.

Quoted 5 lines

Why can't you just generally forbid probes from doing all of this? It would greatly simplify your code, wouldn't it?
Keep it simple please.

An example, taken from the marker mechanism itself (no probe involved) shows how difficult it can be to "forbid all of this" :

The optimized version patches code while the system is live. This implies cross modifying code in SMP environment. It can be done safely on x86 and x86_64 by using a breakpoint during the code modification to make sure the CPU issues a serializing instruction between the moment a given CPU speculates the code execution and actually reaches it. It implies going though a trap, which does funny things such as enabling interrupts, which calls into lockdep. Therefore, adding a marker into the lockdep code cannot be done with a breakpoint-based marker on these architectures. We have to provide an alternative way to do this, less intrusive, which is exactly what the "generic" markers provide. The same applies to instrumentation of the breakpoint trap handler.

I strongly doubt that _every_ users of the markers would be comfortable with the "write your code so it does not take any lock and does everything atomically" constraint. I have done it in LTTng so I could have a fully reentrant tracer, but even then you can be limited by the nature of where you want to send the data. Richard Purdie implemented a serial port based data relay as an alternative data relay mechanism connected to LTTng; he needed a spinlock because of the semantic of his port, so he has to accept the limitation regarding the sites that can and cannot be probed. Providing an explicit declaration of site limitations make sense in this regard.

On other architectures, it is the time source which requires a read seqlock. It is not atomic in the sense that a reader can nest over a writer (if coming from NMI context) and spin forever.

I can list a lot of situations where we cannot _require_ the probe to run atomically in every aspect; so generally forbidding these actions does not seem to be a viable solution. In fact, this would be the best way to make sure the marker infrastructure is never used by early adopters because of the complexity level of writing probes, due to these "rules".

Quoted 11 lines

Quoted 2 lines
Please tell me if I forgot to explain the rationale behind some implementation detail and I will be happy to explain in more depth.
Having lots of flags to do things differently optionally normally starts up all warning lights of early over design. While Linux has this sometimes it is generally only in mature old subsystems. But when something is freshly merged it shouldn't be like this. That is because code tends to grow more complicated over its livetime and when it is already complicated at the beginning it will eventually fall over (you can study current slab as a poster child of this)

Quoted 2 lines

-Andi

Explicitly identifying "hard to instrument" sites is nothing new. It has been done in different manners in the past. Kprobes sprinkles "__kprobes" declarations before function declarations all over the place to specify which ones cannot be safely instrumented. It results in a visually less appealing source code and it limits the sites that can be probed.

The goal of the marker infrastructure is exactly to instrument those sites. Therefore, the approach "we forbid instrumentation of sites hard to instrument" misses the point of this infrastructure. We can leverage the fact that the marker is put in a context known by the programmer; it makes sense to give him the ability to specify what are the restrictions on the probes connected to this marker with some level of granularity.

Regards,

Mathieu

Signature · 3 lines

Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Rusty Russell

2007-05-02 00:31 UTC #133 lore raw

In reply to Andi Kleen

Quoted 6 lines · Andi Kleen

Quoted 1 line · Andrew Morton
Will merge the rustyvisor.
IMHO the user code still doesn't belong into Documentation. Also it needs another review round I guess. And some beta testing by more people.

Like any piece of code more review and more testing would be great. (Your earlier review was particularly useful!). But it's not clear that waiting for longer will achieve either.

Look at kvm's experience for the reverse case: it went in, then got rewritten.

As for the code in Documentation, my initial attempts tried to get around the need for a userspace part by putting everything in the kernel module. It meant you could launch a guest by writing a string to /dev/lguest (no real ABI burden there), but it's a worse solution than some user code in the kernel tree 8(

Cheers,
Rusty.

Andi Kleen

2007-05-02 10:31 UTC #134 lore raw

Quoted 11 lines · Rusty Russell

Quoted 6 lines · Andi Kleen
Quoted 1 line · Andrew Morton
Will merge the rustyvisor.
IMHO the user code still doesn't belong into Documentation. Also it needs another review round I guess. And some beta testing by more people.
Like any piece of code more review and more testing would be great. (Your earlier review was particularly useful!). But it's not clear that waiting for longer will achieve either.

Not clear to me. Release a clear lguest patchkit with documentation on l-k several times and you'll probably get both reviewers and testers. Then confidence level will rise.

Quoted 3 lines

Look at kvm's experience for the reverse case: it went in, then got rewritten.

They at least already had some user base at this point.

Quoted 5 lines

As for the code in Documentation, my initial attempts tried to get around the need for a userspace part by putting everything in the kernel module. It meant you could launch a guest by writing a string to /dev/lguest (no real ABI burden there), but it's a worse solution than some user code in the kernel tree 8(

Just put it into a separate tarball.

-Andi

Stephen Smalley

2007-05-01 13:06 UTC #135 lore raw

file capabilities and security_task_wait failure Re: 2.6.22 -mm merge plans

Quoted 6 lines · Andrew Morton

implement-file-posix-capabilities.patch file-capabilities-accomodate-future-64-bit-caps.patch return-eperm-not-echild-on-security_task_wait-failure.patch
I think we're still waiting for the security guys to work out what to do with this work.

return-eperm-not-echild-on-security_task_wait-failure.patch should be merged - it is effectively a bug fix.

On the file capabilities support, have any of the filesystem folks (cc'd) looked at the code yet?

Signature · 2 lines

Stephen Smalley
National Security Agency

Hugh Dickins

2007-05-01 14:31 UTC #136 lore raw

Re: 2.6.22 -mm merge plans: mm-more-rmap-checking

Quoted 5 lines · Andrew Morton

...
mm-more-rmap-checking.patch
...
Misc MM things. Will merge.

Would Nick mind very much if I ask you to drop this one? You did CC me ages ago, but I've only just run across it. It's a small matter, but I'd prefer it dropped for now.

Quoted 6 lines

Quoted 6 lines
Re-introduce rmap verification patches that Hugh removed when he removed PG_map_lock. PG_map_lock actually isn't needed to synchronise access to anonymous pages, because PG_locked and PTL together already do.
These checks were important in discovering and fixing a rare rmap corruption in SLES9.

It introduces some silly checks which were never in mainline, nor so far as I can tell in SLES9: I'm thinking of those
+ BUG_ON(address < vma->vm_start || address >= vma->vm_end);
There are few callsites for these rmap functions, I don't think they need to be checking their arguments in that way.

It also changes the inline page_dup_rmap (a single atomic increment) into a bugchecking out-of-line function: do we really want to slow down fork in that way, for 2.6.22 to fix a rare corruption in SLES9?

What I really like about the patch is Nick's observation that my

	/* else checking page index and mapping is racy */

is no longer true: a change we made to the do_swap_page sequence some while ago has indeed cured that raciness, and I'm happy to reintroduce the check on mapping and index in page_add_anon_rmap, and his BUG_ON(!PageLocked(page)) there (despite BUG_ONs falling out of fashion very recently).

That becomes more important when I send the patches to free up PG_swapcache, using a PAGE_MAPPING_SWAP bit instead: so I was planning to include that part of Nick's patch in that series.

Hugh

Nick Piggin

2007-05-02 01:42 UTC #137 lore raw

Re: 2.6.22 -mm merge plans: mm-more-rmap-checking

Quoted 11 lines · Hugh Dickins

Quoted 4 lines · Andrew Morton
... mm-more-rmap-checking.patch
...
Misc MM things. Will merge.
Would Nick mind very much if I ask you to drop this one? You did CC me ages ago, but I've only just run across it. It's a small matter, but I'd prefer it dropped for now.

I guess I would prefer it to go under CONFIG_DEBUG_VM. Speaking of which, it would be nice to be able to turn that on unconditionally in -rc1. Although I may have put a few too many things under it, so it might slow down too much...

Review

Quoted 11 lines

Quoted 6 lines
Quoted 6 lines
Re-introduce rmap verification patches that Hugh removed when he removed PG_map_lock. PG_map_lock actually isn't needed to synchronise access to anonymous pages, because PG_locked and PTL together already do.
These checks were important in discovering and fixing a rare rmap corruption in SLES9.
It introduces some silly checks which were never in mainline, nor so far as I can tell in SLES9: I'm thinking of those
(quoted diff) +1 -0
+	BUG_ON(address < vma->vm_start || address >= vma->vm_end);

Yes, but IIRC I put that in because there was another check in SLES9 that I actually couldn't put in, but used this one instead because it also caught the bug we saw.

Quoted 6 lines

There are few callsites for these rmap functions, I don't think they need to be checking their arguments in that way.
It also changes the inline page_dup_rmap (a single atomic increment) into a bugchecking out-of-line function: do we really want to slow down fork in that way, for 2.6.22 to fix a rare corruption in SLES9?

This was actually a rare corruption that is also in 2.6.21, and as few rmap callsites as we have, it was never noticed until the SLES9 bug check was triggered.

Quoted 7 lines

What I really like about the patch is Nick's observation that my
	/* else checking page index and mapping is racy */
is no longer true: a change we made to the do_swap_page sequence some while ago has indeed cured that raciness, and I'm happy to reintroduce the check on mapping and index in page_add_anon_rmap, and his BUG_ON(!PageLocked(page)) there (despite BUG_ONs falling out of fashion very recently).

Hmm, I didn't notice the do_swap_page change, rather just derived its safety by looking at the current state of the code (which I guess must have been post-do_swap_page change)...

Do you have a pointer to the patch, for my interest?

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hugh Dickins

2007-05-02 13:17 UTC #138 lore raw

Re: 2.6.22 -mm merge plans: mm-more-rmap-checking

Quoted 8 lines · Nick Piggin

Yes, but IIRC I put that in because there was another check in SLES9 that I actually couldn't put in, but used this one instead because it also caught the bug we saw.
...
This was actually a rare corruption that is also in 2.6.21, and as few rmap callsites as we have, it was never noticed until the SLES9 bug check was triggered.

You are being very mysterious. Please describe this bug (privately if you think it's exploitable), and let's work on the patch to fix it, rather than this "debug" patch.

Quoted 3 lines

Hmm, I didn't notice the do_swap_page change, rather just derived its safety by looking at the current state of the code (which I guess must have been post-do_swap_page change)...

Your addition of page_add_new_anon_rmap clarified the situation too.

Quoted 1 line

Do you have a pointer to the patch, for my interest?

The patch which changed do_swap_page?

commit c475a8ab625d567eacf5e30ec35d6d8704558062

Author: Hugh Dickins <[email protected]>
Date:   Tue Jun 21 17:15:12 2005 -0700

[PATCH] can_share_swap_page: use page_mapcount

Or my intended PG_swapcache to PAGE_MAPPING_SWAP patch, which does assume PageLocked in page_add_anon_rmap? Yes, I can send you its current unsplit state if you like (but have higher priorities before splitting and commenting it for posting).

Hugh

Nick Piggin

2007-05-03 00:18 UTC #139 lore raw

Re: 2.6.22 -mm merge plans: mm-more-rmap-checking

Quoted 10 lines · Hugh Dickins

Quoted 6 lines · Nick Piggin
Yes, but IIRC I put that in because there was another check in SLES9 that I actually couldn't put in, but used this one instead because it also caught the bug we saw.
... This was actually a rare corruption that is also in 2.6.21, and as few rmap callsites as we have, it was never noticed until the SLES9 bug check was triggered.

Quoted 3 lines

You are being very mysterious. Please describe this bug (privately if you think it's exploitable), and let's work on the patch to fix it, rather than this "debug" patch.

It is exec-fix-remove_arg_zero.patch in Andrew's tree, it's exploitable in that it leaks memory, but it could also release corrupted pagetables into quicklists on those architectures that have them...

Anyway, it quite likely would have gone unfixed for several more years if we didn't have the bug triggers in. Now you could argue that my patch obviously fixes all bugs in there (but I wouldn't :)), and being most complex of the few callsites, _now_ we can avoid the bug checks. However I'd prefer to keep them at least under CONFIG_DEBUG_VM.

Quoted 11 lines

Quoted 3 lines
Hmm, I didn't notice the do_swap_page change, rather just derived its safety by looking at the current state of the code (which I guess must have been post-do_swap_page change)...
Your addition of page_add_new_anon_rmap clarified the situation too.
Quoted 1 line
Do you have a pointer to the patch, for my interest?

Quoted 6 lines

The patch which changed do_swap_page?
commit c475a8ab625d567eacf5e30ec35d6d8704558062
Author: Hugh Dickins <[email protected]>
Date:   Tue Jun 21 17:15:12 2005 -0700
[PATCH] can_share_swap_page: use page_mapcount

Yeah, this one, thanks. I'm just interested.

Quoted 5 lines

Or my intended PG_swapcache to PAGE_MAPPING_SWAP patch, which does assume PageLocked in page_add_anon_rmap? Yes, I can send you its current unsplit state if you like (but have higher priorities before splitting and commenting it for posting).

I would like to see that too, but when you are ready :)

Thanks,
Nick

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Zan Lynx

2007-05-01 16:56 UTC #140 lore raw

On Mon, 2007-04-30 at 16:20 -0700, Andrew Morton wrote:
[snip]

Quoted 10 lines

Mel's moveable-zone work.
I don't believe that this has had sufficient review and I'm sure that it hasn't had sufficient third-party testing. Most of the approbations thus far have consisted of people liking the overall idea, based on the changelogs and multi-year-old discussions.
For such a large and core change I'd have expected more detailed reviewing effort and more third-party testing. And I STILL haven't made time to review the code in detail myself.

[snip]

I am a fan of this, but I hadn't really realized that it's in -mm, and that it has to be enabled with kernelcore=

Now that I am, I'm running it on my laptop with kernelcore=256M (it wouldn't boot with 128M or less, weird initscript errors and OOMs).

1 GB single-core laptops are probably not the intended test audience :) But I'll see what happens.

Signature · 1 line

Zan Lynx <[email protected]>

Hugh Dickins

2007-05-01 17:06 UTC #141 lore raw

Re: 2.6.22 -mm merge plans: mm-detach_vmas_to_be_unmapped-fix

Quoted 4 lines · Andrew Morton

...
mm-detach_vmas_to_be_unmapped-fix.patch
...
Misc MM things. Will merge.

No, I think that one is just drifting like flotsam towards mainline, because nobody at all has yet found time to look at it. And Mr Akuster appears not to have signed off on it yet. I've given it a quick look now, and it seems to be based on misdescription and misconception.

Quoted 5 lines

From: <[email protected]>
Wolfgang Wander submitted a fix to address a mmap fragmentation issue. The git patch ( 1363c3cd8603a913a27e2995dccbd70d5312d8e6 ) is somewhat different and yields different results when running Wolfgang's test case leakme.c.

Ken did a lot of the work on that I believe: I certainly wouldn't want to see this patch go in without his Ack. (I've never done any work on unmapped area heuristics, but detach_vmas_to_be_unmapped always catches my eye.)

Quoted 3 lines

IMHO, the vm start and end address are swapped in arch_unmap_area and arch_unmap_area_topdown functions.

I disagree.

Quoted 3 lines

Prior to this patch arch_unmap_area() used area->vm_start and arch_unmap_area_topdown used area->vm_end

Yes (where area is the vma being unmapped).

Quoted 6 lines

in the git patch the following change showed up.
if (mm->unmap_area == arch_unmap_area)
     addr = prev ? prev->vm_start : mm->mmap_base;
else
     addr = vma ?  vma->vm_end : mm->mmap_base;

No, that's not what showed up in the git patch: that's what the patch below is trying to change it to. The git patch said

	addr = prev ? prev->vm_end : mm->mmap_base
for the bottomup case i.e. setting the unmapped area to the
end of the vma below; and
	addr = vma ? vma->vm_start: mm->mmap_base;

for the topdown case i.e. setting the unmapped area to the beginning of the vma above.

That seems to me consistent with what was done before, but pushing the bounds out across any hole, for presumably better behaviour.

Quoted 5 lines

Using Wolfgang Wander's leakme.c test, I get the same results seen with his original "Avoiding mmap fragmentation" patch as I do after swapping the start & end address in the above code segment. The patch I submitted addresses this typo issue.

I'm pretty sure it is not a typo. I did a very hasty test with two aLLocator .c progs Wolfgang posted (one unnamed, one named leakme4.c), on x86_64, and got apparently the same successful result with and without the patch below. In my case, it's probably just slightly slowing down the algorithm, by demanding an additional find_vma() because it mispositions mm->free_area_cache to an occupied area. I don't see how it could ever be an improvement, but I've not spent long enough checking out that code.

I bet there's improvements that could be made there, but this patch looks wrong - please don't rush it into 2.6.22 (personally I'd say drop it, but I'd rather Ken takes a look).

Hugh

Quoted 13 lines

Signed-off-by Andrew Morton <[email protected]>
---
 mm/mmap.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
diff -puN mm/mmap.c~mm-detach_vmas_to_be_unmapped-fix mm/mmap.c
mm/mmap.c renamed from mm/mmap.c~mm-detach_vmas_to_be_unmapped-fix +0 -0
@@ -1723,9 +1723,9 @@ detach_vmas_to_be_unmapped(struct mm_str
 	*insertion_point = vma;

mm/mmap.c detach_vmas_to_be_unmapped(struct mm_str · line 1731

Quoted 10 lines

mm/mmap.c renamed from mm/mmap.c~mm-detach_vmas_to_be_unmapped-fix +2 -2

 	tail_vma->vm_next = NULL;
 	if (mm->unmap_area == arch_unmap_area)
-		addr = prev ? prev->vm_end : mm->mmap_base;
+		addr = prev ? prev->vm_start : mm->mmap_base;
 	else
-		addr = vma ?  vma->vm_start : mm->mmap_base;
+		addr = vma ?  vma->vm_end : mm->mmap_base;
 	mm->unmap_area(mm, addr);
 	mm->mmap_cache = NULL;		/* Kill the cache. */
 }

Hugh Dickins

2007-05-01 18:10 UTC #142 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 20 lines · Andrew Morton

i386-use-page-allocator-to-allocate-thread_info-structure.patch slub-core.patch
slub. Or part thereof. This is another patch series which got messed up by poor patch sequencing.
make-page-private-usable-in-compound-pages-v1.patch optimize-compound_head-by-avoiding-a-shared-page.patch add-virt_to_head_page-and-consolidate-code-in-slab-and-slub.patch slub-fix-object-tracking.patch slub-enable-tracking-of-full-slabs.patch slub-validation-of-slabs-metadata-and-guard-zones.patch slub-add-min_partial.patch slub-add-ability-to-list-alloc--free-callers-per-slab.patch slub-free-slabs-and-sort-partial-slab-lists-in-kmem_cache_shrink.patch slub-remove-object-activities-out-of-checking-functions.patch slub-user-documentation.patch
slub-add-slabinfo-tool.patch

Quoted 1 line

Most of the rest of slub. Will merge it all.

Merging slub already? I'm surprised. That's a very key piece of infrastructure, and I doubt it's had the exposure it needs yet.

Just what has it been widely tested on so far? x86_64. Not many of us have ia64, but I guess SGI people will have been trying it on that. Not i386, that's excluded.

Not powerpc - hmm, I thought that was known, but looking I see no ARCH_USES_SLAB_PAGE_STRUCT there: just built and tried to run it up, crashes in slab_free from pgtable_free_tlb frpm free_pte_range from free_pgd_range from free_pgtables from unmap_region form do_munmap. That's 2.6.21-rc7-mm2.

slob has a justified place at the low end, but do we want some people running with slab and some with slub? I'd expected slub to stay in 2.6.22-mm, and have all the architectures cut over to it in that time, before advancing to mainline.

I've nothing against slub in itself, though I'm wary of its cache merging (more scope for one corrupting another) (and sometimes I think Christoph spent one life uglifying slab for NUMA, then another life ripping that all out to make slub ;)

Hugh

Christoph Lameter

2007-05-01 19:25 UTC #143 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 4 lines · Hugh Dickins

Quoted 1 line
Most of the rest of slub. Will merge it all.
Merging slub already? I'm surprised. That's a very key piece of infrastructure, and I doubt it's had the exposure it needs yet.

Its not the default. Its just an alternative like SLOB. It will take some time to test with various loads in order to see if it can really replace SLAB in all scenarios.

Quoted 3 lines

Just what has it been widely tested on so far? x86_64. Not many of us have ia64, but I guess SGI people will have been trying it on that. Not i386, that's excluded.

There is an i386 patch pending and I have used it on i386 for a while.

Quoted 5 lines

Not powerpc - hmm, I thought that was known, but looking I see no ARCH_USES_SLAB_PAGE_STRUCT there: just built and tried to run it up, crashes in slab_free from pgtable_free_tlb frpm free_pte_range from free_pgd_range from free_pgtables from unmap_region form do_munmap. That's 2.6.21-rc7-mm2.

Hmmm... True I have not spend any time with that platform. We can set ARCH_USES_SLAB_PAGE_STRUCT there to switch it off. SLUB is the default for mm so I am a bit surprised that this did not surface earlier.

Quoted 2 lines

I've nothing against slub in itself, though I'm wary of its cache merging (more scope for one corrupting another) (and

Yes but then SLUB has more diagnostics etc etc than SLAB to prevent any issues. In debug mode all slabs are separate. The merge feature is very stable these days and significantly reduces cache overhead problems that plague SLAB and require it to have a complex object expiration technique. As a result I was able to rip out all timers. SLUB has no cache reaper nor any timer. Its silent if not in use.

Quoted 2 lines

sometimes I think Christoph spent one life uglifying slab for NUMA, then another life ripping that all out to make slub ;)

SLAB has a certain paradigm of doing things (queues) and I had to work within that framework. It was a group effort. SLUB is an answer to those complaints and a result of the lessons learned through years of some painful slab debugging. SLUB makes debugging extremely easy (and also the design is very simple and comprehensible). No rebuilding of the kernel. Just pop in a debug option on the command line which can even be targeted to a slab cache if we know that things break there.

Andrew Morton

author maintainer

2007-05-01 19:55 UTC #144 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 3 lines · On Tue, 1 May 2007 19:10:29 +0100 (BST) Hugh Dickins <[email protected]> wrote:

Quoted 1 line
Most of the rest of slub. Will merge it all.
Merging slub already? I'm surprised.

My thinking here is "does slub have a future". I think the answer is "yes", so we're reasonably safe getting it into mainline for the finishing work. The kernel.org kernel will still default to slab.

Does that sound wrong?

Hugh Dickins

2007-05-01 20:19 UTC #145 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 9 lines · Andrew Morton

Quoted 3 lines · On Tue, 1 May 2007 19:10:29 +0100 (BST) Hugh Dickins <[email protected]> wrote:
Quoted 1 line
Most of the rest of slub. Will merge it all.
Merging slub already? I'm surprised.
My thinking here is "does slub have a future".
I think the answer is "yes",

I think I agree with that,
though it's a judgement I'd leave to you and others.

Quoted 4 lines

so we're reasonably safe getting it into mainline for the finishing work. The kernel.org kernel will still default to slab.
Does that sound wrong?

Yes, to me it does. If it could be defaulted to on throughout the -rcs, on every architecture, then I'd say that's "finishing work"; and we'd be safe knowing we could go back to slab in a hurry if needed. But it hasn't reached that stage yet, I think.

Hugh

Andrew Morton

author maintainer

2007-05-01 20:36 UTC #146 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 17 lines · On Tue, 1 May 2007 21:19:09 +0100 (BST) Hugh Dickins <[email protected]> wrote:

Quoted 9 lines · Andrew Morton
Quoted 3 lines · On Tue, 1 May 2007 19:10:29 +0100 (BST) Hugh Dickins <[email protected]> wrote:
Quoted 1 line
Most of the rest of slub. Will merge it all.
Merging slub already? I'm surprised.
My thinking here is "does slub have a future".
I think the answer is "yes",
I think I agree with that,
though it's a judgement I'd leave to you and others.
Quoted 3 lines
so we're reasonably safe getting it into mainline for the finishing work. The kernel.org kernel will still default to slab.

Quoted 7 lines

Quoted 1 line
Does that sound wrong?
Yes, to me it does. If it could be defaulted to on throughout the -rcs, on every architecture, then I'd say that's "finishing work"; and we'd be safe knowing we could go back to slab in a hurry if needed. But it hasn't reached that stage yet, I think.

Given the current state and the current rate of development I'd expect slub to have reached the level of completion which you're describing around -rc2 or -rc3. I think we'd be pretty safe making that assumption.

This is a bit unusual but there is of course some self-interest here: the patch dependencies are getting awful and having this hanging around out-of-tree will make 2.6.23 development harder for everyone.

So on balance, given that we _do_ expect slub to have a future, I'm inclined to crash ahead with it. The worst that can happen will be a later rm mm/slub.c which would be pretty simple to do.

otoh I could do some frantic patch mangling and make it easier to carry slub out-of-tree, but do we gain much from that?

Christoph Lameter

2007-05-01 20:46 UTC #147 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 2 lines · Andrew Morton

otoh I could do some frantic patch mangling and make it easier to carry slub out-of-tree, but do we gain much from that?

Then we may loose all the slab API cleanups? Yuck. I really do not want redo those....

Andrew Morton

author maintainer

2007-05-01 21:09 UTC #148 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 7 lines · On Tue, 1 May 2007 13:46:26 -0700 (PDT) Christoph Lameter <[email protected]>…

Quoted 2 lines · Andrew Morton
otoh I could do some frantic patch mangling and make it easier to carry slub out-of-tree, but do we gain much from that?
Then we may loose all the slab API cleanups? Yuck. I really do not want redo those....

No, I meant that I'd look at splitting those patches up into one-against-mainline and one-against-slub.

Hugh Dickins

2007-05-02 12:54 UTC #149 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 4 lines · Andrew Morton

Given the current state and the current rate of development I'd expect slub to have reached the level of completion which you're describing around -rc2 or -rc3. I think we'd be pretty safe making that assumption.

Its developer does show signs of being active!

Quoted 4 lines

This is a bit unusual but there is of course some self-interest here: the patch dependencies are getting awful and having this hanging around out-of-tree will make 2.6.23 development harder for everyone.

That is a very strong argument: a somewhat worrisome argument, but a very strong one. Maintaining your sanity is important.

Quoted 4 lines

So on balance, given that we _do_ expect slub to have a future, I'm inclined to crash ahead with it. The worst that can happen will be a later rm mm/slub.c which would be pretty simple to do.

Okay. And there's been no chorus to echo my concern.

But if Linus' tree is to be better than a warehouse to avoid awkward merges, I still think we want it to default to on for all the architectures, and for most if not all -rcs.

Quoted 3 lines

otoh I could do some frantic patch mangling and make it easier to carry slub out-of-tree, but do we gain much from that?

No, keep away from that.

Hugh

Christoph Lameter

2007-05-02 17:03 UTC #150 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 3 lines · Hugh Dickins

But if Linus' tree is to be better than a warehouse to avoid awkward merges, I still think we want it to default to on for all the architectures, and for most if not all -rcs.

At some point I dream that SLUB could become the default but I thought this would take at least 6 month or so. If want to force this now then I will certainly have some busy weeks ahead.

Andrew Morton

author maintainer

2007-05-02 19:11 UTC #151 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 9 lines · On Wed, 2 May 2007 10:03:50 -0700 (PDT) Christoph Lameter <[email protected]>…

Quoted 3 lines · Hugh Dickins
But if Linus' tree is to be better than a warehouse to avoid awkward merges, I still think we want it to default to on for all the architectures, and for most if not all -rcs.
At some point I dream that SLUB could become the default but I thought this would take at least 6 month or so. If want to force this now then I will certainly have some busy weeks ahead.

s/dream/promise/ ;)

Six months sounds reasonable - I was kind of hoping for less. Make it default-to-on in 2.6.23-rc1, see how it goes.

Christoph Lameter

2007-05-02 19:42 UTC #152 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 8 lines · Andrew Morton

Quoted 3 lines
At some point I dream that SLUB could become the default but I thought this would take at least 6 month or so. If want to force this now then I will certainly have some busy weeks ahead.
s/dream/promise/ ;)
Six months sounds reasonable - I was kind of hoping for less. Make it default-to-on in 2.6.23-rc1, see how it goes.

Here is how I think the future could develop

Cycle SLAB SLUB SLOB SLxB

2.6.22 API fixes Stabilization API fixes

Major event: SLUB availability as experimental

2.6.23 API upgrades Perf. Valid. EOL

Major events: SLUB performance validation. Switch off
experimental (could even be the default) Slab allocators support targeted reclaim for at least one slab cache (dentry?) (vacate/move all objects in a slab)

2.6.24 Earliest EOL Stable - Experiments

Major events: SLUB stable. Stable targeted reclaim
for all major reclaimable slabs. Maybe experiments with another new allocator?

2.6.25 EOL default - ?

Death of SLAB. SLUB default. Hopefully new ideas on the horizon.

Sam Ravnborg

2007-05-02 19:53 UTC #153 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 19 lines · Christoph Lameter

Quoted 8 lines · Andrew Morton
Quoted 3 lines
At some point I dream that SLUB could become the default but I thought this would take at least 6 month or so. If want to force this now then I will certainly have some busy weeks ahead.
s/dream/promise/ ;)
Six months sounds reasonable - I was kind of hoping for less. Make it default-to-on in 2.6.23-rc1, see how it goes.
Here is how I think the future could develop
Cycle SLAB SLUB SLOB SLxB
2.6.22 API fixes Stabilization API fixes
Major event: SLUB availability as experimental

Quoted 7 lines

2.6.23 API upgrades Perf. Valid. EOL
Major events: SLUB performance validation. Switch off
experimental (could even be the default) Slab allocators support targeted reclaim for at least one slab cache (dentry?) (vacate/move all objects in a slab)

To facilitate this do NOT introduce CONFIG_SLAB until we decide that SLUB are default. In this way we can make CONFIG_SLUB be default and people will not continue with CONFIG_SLAB because they had it in their .config already.
Or just rename CONFIG_SLAB to CONFIG_SLAB_DEPRECATED or something.

The point is make sure that LSUB becomes default for people that does an make oldconfig (explicit or implicit).

Sam

Christoph Lameter

2007-05-02 20:14 UTC #154 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 4 lines · Sam Ravnborg

To facilitate this do NOT introduce CONFIG_SLAB until we decide that SLUB are default. In this way we can make CONFIG_SLUB be default and people will not continue with CONFIG_SLAB because they had it in their config already.

We already have CONFIG_SLAB. If you use your existing .config then you will stay with SLAB.

Quoted 2 lines

The point is make sure that LSUB becomes default for people that does an make oldconfig (explicit or implicit).

Hmmmm... We can think about that when we actually want to make SLUB the default.

Siddha, Suresh B

2007-05-02 18:52 UTC #155 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 6 lines · Hugh Dickins

Quoted 3 lines · Andrew Morton
So on balance, given that we _do_ expect slub to have a future, I'm inclined to crash ahead with it. The worst that can happen will be a later rm mm/slub.c which would be pretty simple to do.
Okay. And there's been no chorus to echo my concern.

I have been looking into "slub" recently to avoid some of the NUMA alien cache issues that we were encountering on the regular slab.

I am having some stability issues with slub on an ia64 NUMA platform and didn't have time to dig further. I am hoping to look into it soon and share the data/findings with Christoph.

We also did a quick perf collection on x86_64(atleast didn't hear any stability issues from our team on regular x86_64 SMP), that we will be sharing shortly.

Quoted 3 lines

But if Linus' tree is to be better than a warehouse to avoid awkward merges, I still think we want it to default to on for all the architectures, and for most if not all -rcs.

I will not suggest for default on at this point.

thanks,
suresh

Christoph Lameter

2007-05-02 18:58 UTC #156 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 2 lines · Suresh B

I have been looking into "slub" recently to avoid some of the NUMA alien cache issues that we were encountering on the regular slab.

Yes that is also our main concern.

Quoted 3 lines

I am having some stability issues with slub on an ia64 NUMA platform and didn't have time to dig further. I am hoping to look into it soon and share the data/findings with Christoph.

There is at least one patch on top of 2.6.21-rc7-mm2 already in mm that may be necessary for you.

Christoph Lameter

2007-05-01 21:08 UTC #157 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 4 lines · Hugh Dickins

Yes, to me it does. If it could be defaulted to on throughout the -rcs, on every architecture, then I'd say that's "finishing work"; and we'd be safe knowing we could go back to slab in a hurry if needed. But it hasn't reached that stage yet, I think.

Why would we need to go back to SLAB if we have not switched to SLUB? SLUB is marked experimental and not the default.

The only problems that I am aware of is(or was) the issue with arches modifying page struct fields of slab pages that SLUB needs for its own operations. And I thought it was all fixed since the powerpc guys were quiet and the patch was in for i386.

Hugh Dickins

2007-05-02 12:45 UTC #158 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 9 lines · Christoph Lameter

Quoted 4 lines · Hugh Dickins
Yes, to me it does. If it could be defaulted to on throughout the -rcs, on every architecture, then I'd say that's "finishing work"; and we'd be safe knowing we could go back to slab in a hurry if needed. But it hasn't reached that stage yet, I think.
Why would we need to go back to SLAB if we have not switched to SLUB? SLUB is marked experimental and not the default.

I said above that I thought SLUB ought to be defaulted to on throughout the -rcs: if we don't do that, we're not going to learn much from having it in Linus' tree.

And perhaps that line which appends "PREEMPT " to an oops report ought to append "SLUB " too, for so long as there's a choice.

Quoted 4 lines

The only problems that I am aware of is(or was) the issue with arches modifying page struct fields of slab pages that SLUB needs for its own operations. And I thought it was all fixed since the powerpc guys were quiet and the patch was in for i386.

You're forgetting your unions in struct page: in the SPLIT_PTLOCK case (NR_CPUS >= 4) the pagetable code is using spinlock_t ptl, which overlays SLUB's first_page and slab pointers.

I just tried rebuilding powerpc with the SPLIT_PTLOCK cutover edited to 8 cpus instead, and then no crash.

I presume the answer is just to extend your quicklist work to powerpc's lowest level of pagetables. The only other architecture which is using kmem_cache for them is arm26, which has "#error SMP is not supported", so won't be giving this problem.

Hugh

Christoph Lameter

2007-05-02 17:01 UTC #159 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 6 lines · Hugh Dickins

Quoted 2 lines
Why would we need to go back to SLAB if we have not switched to SLUB? SLUB is marked experimental and not the default.
I said above that I thought SLUB ought to be defaulted to on throughout the -rcs: if we don't do that, we're not going to learn much from having it in Linus' tree.

I'd rather be careful with that..... mm is enough for now. Why go to the extremes immediately. If it is an option then people can gradually start testing with it.

Quoted 8 lines

Quoted 4 lines
The only problems that I am aware of is(or was) the issue with arches modifying page struct fields of slab pages that SLUB needs for its own operations. And I thought it was all fixed since the powerpc guys were quiet and the patch was in for i386.
You're forgetting your unions in struct page: in the SPLIT_PTLOCK case (NR_CPUS >= 4) the pagetable code is using spinlock_t ptl, which overlays SLUB's first_page and slab pointers.

Uhhh.... Right. So SLUB wont work if the lowest page table block is managed via slabs.

Quoted 5 lines

I just tried rebuilding powerpc with the SPLIT_PTLOCK cutover edited to 8 cpus instead, and then no crash.
I presume the answer is just to extend your quicklist work to powerpc's lowest level of pagetables. The only other architecture

I am not sure how PowerPCs lower pagetable pages work. If they are of PAGE_SIZE then this is no problem.

Quoted 2 lines

which is using kmem_cache for them is arm26, which has "#error SMP is not supported", so won't be giving this problem.

Ahh. Good.

But these are arch specific problems. We could use ARCH_USES_SLAB_PAGE_STRUCT to disable SLUB on these platforms.

Hugh Dickins

2007-05-02 18:08 UTC #160 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 3 lines · Christoph Lameter

But these are arch specific problems. We could use ARCH_USES_SLAB_PAGE_STRUCT to disable SLUB on these platforms.

As a quick hack, sure. But every ARCH_USES_SLAB_PAGE_STRUCT diminishes the testing SLUB will get. If the idea is that we're going to support both SLAB and SLUB, some arches with one, some with another, some with either, for more than a single release, then I'm back to saying SLUB is being pushed in too early. I can understand people wanting pluggable schedulers, but pluggable slab allocators?

Hugh

Christoph Lameter

2007-05-02 18:28 UTC #161 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 5 lines · Hugh Dickins

Quoted 3 lines · Christoph Lameter
But these are arch specific problems. We could use ARCH_USES_SLAB_PAGE_STRUCT to disable SLUB on these platforms.

Quoted 7 lines

As a quick hack, sure. But every ARCH_USES_SLAB_PAGE_STRUCT diminishes the testing SLUB will get. If the idea is that we're going to support both SLAB and SLUB, some arches with one, some with another, some with either, for more than a single release, then I'm back to saying SLUB is being pushed in too early. I can understand people wanting pluggable schedulers, but pluggable slab allocators?

This is a sensitive piece of the kernel as you say and we better allow the running of two allocator for some time to make sure that it behaves in all load situations. The design is fundamentally different so its performance characteristics may diverge significantly and perhaps there will be corner cases for each where they do the best job.

I have already reworked the slab API to allow for an easy implementation of alternate slab allocators (released with 2.6.20) which only covered SLAB and SLOB. This is continuing the cleanup work and adding a third one.

Andrew Morton

author maintainer

2007-05-02 18:42 UTC #162 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 15 lines · On Wed, 2 May 2007 11:28:26 -0700 (PDT) Christoph Lameter <[email protected]>…

Quoted 5 lines · Hugh Dickins
Quoted 3 lines · Christoph Lameter
But these are arch specific problems. We could use ARCH_USES_SLAB_PAGE_STRUCT to disable SLUB on these platforms.
Quoted 7 lines
As a quick hack, sure. But every ARCH_USES_SLAB_PAGE_STRUCT diminishes the testing SLUB will get. If the idea is that we're going to support both SLAB and SLUB, some arches with one, some with another, some with either, for more than a single release, then I'm back to saying SLUB is being pushed in too early. I can understand people wanting pluggable schedulers, but pluggable slab allocators?

Quoted 5 lines

This is a sensitive piece of the kernel as you say and we better allow the running of two allocator for some time to make sure that it behaves in all load situations. The design is fundamentally different so its performance characteristics may diverge significantly and perhaps there will be corner cases for each where they do the best job.

eek. We'd need to fix those corner cases then. Our endgame here really must be rm mm/slab.c.

Christoph Lameter

2007-05-02 18:53 UTC #163 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 8 lines · Andrew Morton

Quoted 5 lines
This is a sensitive piece of the kernel as you say and we better allow the running of two allocator for some time to make sure that it behaves in all load situations. The design is fundamentally different so its performance characteristics may diverge significantly and perhaps there will be corner cases for each where they do the best job.
eek. We'd need to fix those corner cases then. Our endgame here really must be rm mm/slab.c.

First we need to discover them and I doubt that mm covers much more than development loads. I hope we can get to a point where we have SLUB be the primarily allocator soon but I would expect various performance issues to show up.

On the other hand: I am pretty sure that SLUB can replace SLOB completely given SLOBs limitations and SLUBs more efficient use of space. SLOB needs 8 bytes of overhead. SLUB needs none. We may just have to #ifdef out the debugging support to make the code be of similar size to SLOB too. SLOB is a general problem because its features are not compatible to SLAB. F.e. it does not support DESTROY_BY_RCU and does not do reclaim the right way etc etc. SLUB may turn out to be the ideal embedded slab allocator.

Christoph Lameter

2007-05-02 17:25 UTC #164 lore raw

Re: 2.6.22 -mm merge plans: slub

In reply to Christoph Lameter

Quoted 4 lines · Hugh Dickins

I presume the answer is just to extend your quicklist work to powerpc's lowest level of pagetables. The only other architecture which is using kmem_cache for them is arm26, which has "#error SMP is not supported", so won't be giving this problem.

In the meantime we would need something like this to disable SLUB in this particular configuration. Note that I have not tested this and the <= for the comparision with SPLIT_PTLOCK_CPUS may not work (Never seen such a construct in a Kconfig file but it is needed here).

PowerPC: Disable SLUB for configurations in which slab page structs are modified

PowerPC uses the slab allocator to manage the lowest level of the page table. In high cpu configurations we also use the page struct to split the page table lock. Disallow the selection of SLUB for that case.

[Not tested: I am not familiar with powerpc build procedures etc]

Signed-off-by Christoph Lameter <[email protected]>

arch/powerpc/Kconfig +13 -0

Index: linux-2.6.21-rc7-mm2/arch/powerpc/Kconfig
===================================================================
@@ -117,6 +117,19 @@ config GENERIC_BUG
 	default y
 	depends on BUG
 
+#
+# Powerpc uses the slab allocator to manage its ptes and the
+# page structs of ptes are used for splitting the page table
+# lock for configurations supporting more than SPLIT_PTLOCK_CPUS.
+#
+# In that special configuration the page structs of slabs are modified.
+# This setting disables the selection of SLUB as a slab allocator.
+#
+config ARCH_USES_SLAB_PAGE_STRUCT
+	bool
+	default y
+	depends on SPLIT_PTLOCK_CPUS <= NR_CPUS
+
 config DEFAULT_UIMAGE
 	bool
 	help

Hugh Dickins

2007-05-02 18:36 UTC #165 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 11 lines · Christoph Lameter

Quoted 4 lines · Hugh Dickins
I presume the answer is just to extend your quicklist work to powerpc's lowest level of pagetables. The only other architecture which is using kmem_cache for them is arm26, which has "#error SMP is not supported", so won't be giving this problem.
In the meantime we would need something like this to disable SLUB in this particular configuration. Note that I have not tested this and the <= for the comparision with SPLIT_PTLOCK_CPUS may not work (Never seen such a construct in a Kconfig file but it is needed here).

I'm astonished and impressed, both with Kconfig and your use of it:
that does seem to work. Though I don't dare go so far as to give the patch an ack, and don't like this way out at all. It needs a proper (quicklist) solution, and by the time that solution comes along, all the powerpc people will have CONFIG_SLAB=y in their .config, and "make oldconfig" will just perpetuate that status quo, instead of the switching over to CONFIG_SLUB=y. I think. Unless we keep changing the config option names, or go through a phase with no option.

I'd much rather be testing a quicklist patch:
I'd better give that a try.

Hugh

Quoted 27 lines

PowerPC: Disable SLUB for configurations in which slab page structs are modified
PowerPC uses the slab allocator to manage the lowest level of the page table. In high cpu configurations we also use the page struct to split the page table lock. Disallow the selection of SLUB for that case.
[Not tested: I am not familiar with powerpc build procedures etc]
Signed-off-by Christoph Lameter <[email protected]>
Index: linux-2.6.21-rc7-mm2/arch/powerpc/Kconfig
===================================================================
arch/powerpc/Kconfig +6 -0
@@ -117,6 +117,19 @@ config GENERIC_BUG
 	default y
 	depends on BUG
 
+#
+# Powerpc uses the slab allocator to manage its ptes and the
+# page structs of ptes are used for splitting the page table
+# lock for configurations supporting more than SPLIT_PTLOCK_CPUS.
+#
+# In that special configuration the page structs of slabs are modified.

arch/powerpc/Kconfig config GENERIC_BUG · line 135

Quoted 10 lines

arch/powerpc/Kconfig +7 -0

+# This setting disables the selection of SLUB as a slab allocator.
+#
+config ARCH_USES_SLAB_PAGE_STRUCT
+	bool
+	default y
+	depends on SPLIT_PTLOCK_CPUS <= NR_CPUS
+
 config DEFAULT_UIMAGE
 	bool
 	help

Christoph Lameter

2007-05-02 18:39 UTC #166 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 1 line · Hugh Dickins

I'm astonished and impressed, both with Kconfig and your use of it:

Thanks!

Quoted 2 lines

I'd much rather be testing a quicklist patch:
I'd better give that a try.

Great. But I certainly do not mind people use SLAB. I do not think that one approach should be there for all. Choice is the way to have multiple allocators compete. One reason that SLAB is so crusty is because it was the only solution for so long.

Andrew Morton

author maintainer

2007-05-02 18:57 UTC #167 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 6 lines · On Wed, 2 May 2007 11:39:20 -0700 (PDT) Christoph Lameter <[email protected]>…

Quoted 1 line · Hugh Dickins
I'm astonished and impressed, both with Kconfig and your use of it:
Thanks!

Quoted 8 lines

Quoted 2 lines
I'd much rather be testing a quicklist patch:
I'd better give that a try.
Great. But I certainly do not mind people use SLAB. I do not think that one approach should be there for all. Choice is the way to have multiple allocators compete. One reason that SLAB is so crusty is because it was the only solution for so long.

noooo, we don't want competing slab allocators, please. We should get slub working well on all architectures then remove slab completely. Having to maintain both slab.c and slub.c would be awful.

Christoph Lameter

2007-05-02 19:01 UTC #168 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 3 lines · Andrew Morton

noooo, we don't want competing slab allocators, please. We should get slub working well on all architectures then remove slab completely. Having to maintain both slab.c and slub.c would be awful.

Owww... You throw my roadmap out of the window and may create too high expectations of SLUB.

I am the one who has to maintain SLAB and SLUB it seems and I have been dealing with the trio SLAB, SLOB and SLUB for awhile now. Its okay and it will be much easier once the cleanups are in.

Pekka Enberg

2007-05-02 19:18 UTC #169 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 2 lines · Christoph Lameter

Owww... You throw my roadmap out of the window and may create too high expectations of SLUB.

Me too!

Quoted 3 lines · Christoph Lameter

I am the one who has to maintain SLAB and SLUB it seems and I have been dealing with the trio SLAB, SLOB and SLUB for awhile now. Its okay and it will be much easier once the cleanups are in.

And then there's patches such as kmemleak which would need to target all three. Plus it doesn't really make sense for users to select between three competiting implementations. Please don't take away our high hopes of getting rid of mm/slab.c Christoph =)

Pekka

Christoph Lameter

2007-05-02 19:34 UTC #170 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 9 lines · Pekka Enberg

Quoted 3 lines · Christoph Lameter
I am the one who has to maintain SLAB and SLUB it seems and I have been dealing with the trio SLAB, SLOB and SLUB for awhile now. Its okay and it will be much easier once the cleanups are in.
And then there's patches such as kmemleak which would need to target all three. Plus it doesn't really make sense for users to select between three competiting implementations. Please don't take away our high hopes of getting rid of mm/slab.c Christoph =)

SLUB supports kmemleak (actually its quite improved). Switch debugging on and try

cat /sys/slab/kmalloc-128/alloc_calls.

Christoph Lameter

2007-05-02 19:43 UTC #171 lore raw

Re: 2.6.22 -mm merge plans: slub

In reply to Pekka Enberg

Quoted 4 lines · Pekka Enberg

And then there's patches such as kmemleak which would need to target all three. Plus it doesn't really make sense for users to select between three competiting implementations. Please don't take away our high hopes of getting rid of mm/slab.c Christoph =)

You too, Brutus ...

Andrew Morton

author maintainer

2007-05-03 08:15 UTC #172 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 35 lines · On Wed, 2 May 2007 10:25:47 -0700 (PDT) Christoph Lameter <[email protected]>…

Quoted 4 lines · Hugh Dickins
I presume the answer is just to extend your quicklist work to powerpc's lowest level of pagetables. The only other architecture which is using kmem_cache for them is arm26, which has "#error SMP is not supported", so won't be giving this problem.
In the meantime we would need something like this to disable SLUB in this particular configuration. Note that I have not tested this and the <= for the comparision with SPLIT_PTLOCK_CPUS may not work (Never seen such a construct in a Kconfig file but it is needed here).
PowerPC: Disable SLUB for configurations in which slab page structs are modified
PowerPC uses the slab allocator to manage the lowest level of the page table. In high cpu configurations we also use the page struct to split the page table lock. Disallow the selection of SLUB for that case.
[Not tested: I am not familiar with powerpc build procedures etc]
Signed-off-by Christoph Lameter <[email protected]>
Index: linux-2.6.21-rc7-mm2/arch/powerpc/Kconfig
===================================================================
arch/powerpc/Kconfig +3 -0
@@ -117,6 +117,19 @@ config GENERIC_BUG
 	default y
 	depends on BUG
 
+#
+# Powerpc uses the slab allocator to manage its ptes and the
+# page structs of ptes are used for splitting the page table

arch/powerpc/Kconfig config GENERIC_BUG · line 132

Quoted 10 lines

arch/powerpc/Kconfig +10 -0

+# lock for configurations supporting more than SPLIT_PTLOCK_CPUS.
+#
+# In that special configuration the page structs of slabs are modified.
+# This setting disables the selection of SLUB as a slab allocator.
+#
+config ARCH_USES_SLAB_PAGE_STRUCT
+	bool
+	default y
+	depends on SPLIT_PTLOCK_CPUS <= NR_CPUS
+

That all seems to work as intended.

However with NR_CPUS=8 SPLIT_PTLOCK_CPUS=4, enabling SLUB=y crashes the machine early in boot.

Too early for netconsole, no serial console. Wedges up uselessly with CONFIG_XMON=n, does mysterious repeated uncontrollable exceptions with CONFIG_XMON=y. This is all fairly typical for a powerpc/G5 crash :(

However I was able to glimpse some stuff as it flew past. Crash started in flush_old_exec and ended in pgtable_free_tlb -> kmem_cache_free. I don't know how to do better than that I'm afraid, unless I'm to hunt down a PCIE serial card, perhaps.

William Lee Irwin III

2007-05-03 08:26 UTC #173 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 10 lines · Andrew Morton

That all seems to work as intended.
However with NR_CPUS=8 SPLIT_PTLOCK_CPUS=4, enabling SLUB=y crashes the machine early in boot.
Too early for netconsole, no serial console. Wedges up uselessly with CONFIG_XMON=n, does mysterious repeated uncontrollable exceptions with CONFIG_XMON=y. This is all fairly typical for a powerpc/G5 crash :( However I was able to glimpse some stuff as it flew past. Crash started in flush_old_exec and ended in pgtable_free_tlb -> kmem_cache_free. I don't know how to do better than that I'm afraid, unless I'm to hunt down a PCIE serial card, perhaps.

I've seen this crash in flush_old_exec() before. ISTR it being due to slub vs. pagetable alignment or something on that order.

-- wli

Christoph Lameter

2007-05-03 16:30 UTC #174 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 4 lines · William Lee Irwin III

I've seen this crash in flush_old_exec() before. ISTR it being due to slub vs. pagetable alignment or something on that order.
From from other discussion regarding SLAB: It may be necessary for

powerpc to set the default alignment to 8 bytes on 32 bit powerpc because it requires that alignemnt for 64 bit value. SLUB will *not* disable debugging like SLAB if you do that.

Hugh Dickins

2007-05-03 08:46 UTC #175 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 10 lines · Andrew Morton

arch/powerpc/Kconfig config GENERIC_BUG · line 132
Quoted 5 lines · On Wed, 2 May 2007 10:25:47 -0700 (PDT) Christoph Lameter <[email protected]>…
arch/powerpc/Kconfig +5 -0
+config ARCH_USES_SLAB_PAGE_STRUCT
+	bool
+	default y
+	depends on SPLIT_PTLOCK_CPUS <= NR_CPUS
+
That all seems to work as intended.

Quoted 2 lines

However with NR_CPUS=8 SPLIT_PTLOCK_CPUS=4, enabling SLUB=y crashes the machine early in boot.

I thought that if that worked as intended, you wouldn't even get the chance to choose SLUB=y? That was how it was working for me (but I realize I didn't try more than make oldconfig).

Quoted 9 lines

Too early for netconsole, no serial console. Wedges up uselessly with CONFIG_XMON=n, does mysterious repeated uncontrollable exceptions with CONFIG_XMON=y. This is all fairly typical for a powerpc/G5 crash :(
However I was able to glimpse some stuff as it flew past. Crash started in flush_old_exec and ended in pgtable_free_tlb -> kmem_cache_free. I don't know how to do better than that I'm afraid, unless I'm to hunt down a PCIE serial card, perhaps.

That sounds like what happens when SLUB's pagestruct use meets SPLIT_PTLOCK's pagestruct use. Does your .config really show CONFIG_SLUB=y together with CONFIG_ARCH_USES_SLAB_PAGE_STRUCT=y?

Hugh

Andrew Morton

author maintainer

2007-05-03 08:57 UTC #176 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 11 lines · On Thu, 3 May 2007 09:46:32 +0100 (BST) Hugh Dickins <[email protected]> wrote:

Quoted 10 lines · Andrew Morton
arch/powerpc/Kconfig config GENERIC_BUG · line 132
Quoted 5 lines · On Wed, 2 May 2007 10:25:47 -0700 (PDT) Christoph Lameter <[email protected]>…
arch/powerpc/Kconfig +5 -0
+config ARCH_USES_SLAB_PAGE_STRUCT
+	bool
+	default y
+	depends on SPLIT_PTLOCK_CPUS <= NR_CPUS
+
That all seems to work as intended.

Quoted 6 lines

Quoted 2 lines
However with NR_CPUS=8 SPLIT_PTLOCK_CPUS=4, enabling SLUB=y crashes the machine early in boot.
I thought that if that worked as intended, you wouldn't even get the chance to choose SLUB=y? That was how it was working for me (but I realize I didn't try more than make oldconfig).

Right. This can be tested on x86 without a cross-compiler:

ARCH=powerpc make mrproper
ARCH=powerpc make fooconfig

Quoted 5 lines

Quoted 5 lines
Too early for netconsole, no serial console. Wedges up uselessly with CONFIG_XMON=n, does mysterious repeated uncontrollable exceptions with CONFIG_XMON=y. This is all fairly typical for a powerpc/G5 crash :(

Quoted 9 lines

Quoted 4 lines
However I was able to glimpse some stuff as it flew past. Crash started in flush_old_exec and ended in pgtable_free_tlb -> kmem_cache_free. I don't know how to do better than that I'm afraid, unless I'm to hunt down a PCIE serial card, perhaps.
That sounds like what happens when SLUB's pagestruct use meets SPLIT_PTLOCK's pagestruct use. Does your .config really show CONFIG_SLUB=y together with CONFIG_ARCH_USES_SLAB_PAGE_STRUCT=y?

Nope.

g5:/usr/src/25> grep SLUB .config
CONFIG_SLUB=y
g5:/usr/src/25> grep SLAB .config
# CONFIG_SLAB is not set
g5:/usr/src/25> grep CPUS .config
CONFIG_NR_CPUS=8
# CONFIG_CPUSETS is not set
# CONFIG_IRQ_ALL_CPUS is not set
CONFIG_SPLIT_PTLOCK_CPUS=4

It's in http://userweb.kernel.org/~akpm/config-g5.txt

Hugh Dickins

2007-05-03 09:15 UTC #177 lore raw

Re: 2.6.22 -mm merge plans: slub

Quoted 35 lines · Andrew Morton

Quoted 14 lines · On Thu, 3 May 2007 09:46:32 +0100 (BST) Hugh Dickins <[email protected]> wrote:
Quoted 10 lines · Andrew Morton
arch/powerpc/Kconfig config GENERIC_BUG · line 132
Quoted 5 lines · On Wed, 2 May 2007 10:25:47 -0700 (PDT) Christoph Lameter <[email protected]>…
arch/powerpc/Kconfig +5 -0
+config ARCH_USES_SLAB_PAGE_STRUCT
+	bool
+	default y
+	depends on SPLIT_PTLOCK_CPUS <= NR_CPUS
+
That all seems to work as intended.
Quoted 2 lines
However with NR_CPUS=8 SPLIT_PTLOCK_CPUS=4, enabling SLUB=y crashes the machine early in boot.
Quoted 7 lines
I thought that if that worked as intended, you wouldn't even get the chance to choose SLUB=y? That was how it was working for me (but I realize I didn't try more than make oldconfig).
That sounds like what happens when SLUB's pagestruct use meets SPLIT_PTLOCK's pagestruct use. Does your .config really show CONFIG_SLUB=y together with CONFIG_ARCH_USES_SLAB_PAGE_STRUCT=y?
Nope.
g5:/usr/src/25> grep SLUB .config
CONFIG_SLUB=y
g5:/usr/src/25> grep SLAB .config
# CONFIG_SLAB is not set
g5:/usr/src/25> grep CPUS .config
CONFIG_NR_CPUS=8
# CONFIG_CPUSETS is not set
# CONFIG_IRQ_ALL_CPUS is not set
CONFIG_SPLIT_PTLOCK_CPUS=4

Quoted 1 line

It's in http://userweb.kernel.org/~akpm/config-g5.txt

Seems we're all wrong in thinking Christoph's Kconfiggery worked as intended: maybe it just works some of the time. I'm not going to hazard a guess as to how to fix it up, will resume looking at the powerpc's quicklist potential later.

Hugh

Hugh Dickins

2007-05-03 21:04 UTC #178 lore raw

Re: 2.6.22 -mm merge plans: slub on PowerPC

Quoted 5 lines · Hugh Dickins

Seems we're all wrong in thinking Christoph's Kconfiggery worked as intended: maybe it just works some of the time. I'm not going to hazard a guess as to how to fix it up, will resume looking at the powerpc's quicklist potential later.

Here's the patch I've been testing on G5, with 4k and with 64k pages, with SLAB and with SLUB. But, though it doesn't crash, the pgd kmem_cache in the 4k-page SLUB case is revealing SLUB's propensity for using highorder allocations where SLAB would stick to order 0:
under load, exec's mm_init gets page allocation failure on order 4
- SLUB's calculate_order may need some retuning. (I'd expect it to be going for order 3 actually, I'm not sure how order 4 comes about.)

I don't know how offensive Ben and Paulus may find this patch:
the kmem_cache use was nicely done and this messes it up a little.

The SLUB allocator relies on struct page fields first_page and slab, overwritten by ptl when SPLIT_PTLOCK: so the SLUB allocator cannot then be used for the lowest level of pagetable pages. This was obstructing SLUB on PowerPC, which uses kmem_caches for its pagetables. So convert its pte level to use quicklist pages (whereas pmd, pud and 64k-page pgd want partpages, so continue to use kmem_caches for pmd, pud and pgd). But to keep up appearances for pgtable_free, we still need PTE_CACHE_NUM.

Signed-off-by Hugh Dickins <[email protected]>

---

 arch/powerpc/Kconfig          |    4 ++++
 arch/powerpc/mm/init_64.c     |   17 ++++++-----------
 include/asm-powerpc/pgalloc.h |   26 +++++++++++---------------
 3 files changed, 21 insertions(+), 26 deletions(-)

arch/powerpc/Kconfig +4 -0

@@ -31,6 +31,10 @@ config MMU
 	bool
 	default y
 
+config QUICKLIST
+	bool
+	default y
+
 config GENERIC_HARDIRQS
 	bool
 	default y

arch/powerpc/mm/init_64.c +6 ~~-11~~

@@ -146,21 +146,16 @@ static void zero_ctor(void *addr, struct
 	memset(addr, 0, kmem_cache_size(cache));
 }
 
-#ifdef CONFIG_PPC_64K_PAGES
-static const unsigned int pgtable_cache_size[3] = {
-	PTE_TABLE_SIZE, PMD_TABLE_SIZE, PGD_TABLE_SIZE
-};
-static const char *pgtable_cache_name[ARRAY_SIZE(pgtable_cache_size)] = {
-	"pte_pmd_cache", "pmd_cache", "pgd_cache",
-};
-#else
 static const unsigned int pgtable_cache_size[2] = {
-	PTE_TABLE_SIZE, PMD_TABLE_SIZE
+	PGD_TABLE_SIZE, PMD_TABLE_SIZE
 };
 static const char *pgtable_cache_name[ARRAY_SIZE(pgtable_cache_size)] = {
-	"pgd_pte_cache", "pud_pmd_cache",
-};
+#ifdef CONFIG_PPC_64K_PAGES
+	"pgd_cache", "pmd_cache",
+#else
+	"pgd_cache", "pud_pmd_cache",
 #endif /* CONFIG_PPC_64K_PAGES */
+};
 
 #ifdef CONFIG_HUGETLB_PAGE
 /* Hugepages need one extra cache, initialized in hugetlbpage.c.  We

include/asm-powerpc/pgalloc.h +11 ~~-15~~

@@ -10,21 +10,15 @@
 #include <linux/slab.h>
 #include <linux/cpumask.h>
 #include <linux/percpu.h>
+#include <linux/quicklist.h>
 
 extern struct kmem_cache *pgtable_cache[];
 
-#ifdef CONFIG_PPC_64K_PAGES
-#define PTE_CACHE_NUM	0
-#define PMD_CACHE_NUM	1
-#define PGD_CACHE_NUM	2
-#define HUGEPTE_CACHE_NUM 3
-#else
-#define PTE_CACHE_NUM	0
-#define PMD_CACHE_NUM	1
-#define PUD_CACHE_NUM	1
 #define PGD_CACHE_NUM	0
+#define PUD_CACHE_NUM	1
+#define PMD_CACHE_NUM	1
 #define HUGEPTE_CACHE_NUM 2
-#endif
+#define PTE_CACHE_NUM	3	/* from quicklist rather than  kmem_cache */
 
 /*
  * This program is free software; you can redistribute it and/or

@@ -97,8 +91,7 @@ static inline void pmd_free(pmd_t *pmd)
 static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
 					  unsigned long address)
 {
-	return kmem_cache_alloc(pgtable_cache[PTE_CACHE_NUM],
-				GFP_KERNEL|__GFP_REPEAT);
+	return quicklist_alloc(0, GFP_KERNEL|__GFP_REPEAT, NULL);
 }
 
 static inline struct page *pte_alloc_one(struct mm_struct *mm,

@@ -109,7 +102,7 @@ static inline struct page *pte_alloc_one
 		
 static inline void pte_free_kernel(pte_t *pte)
 {
-	kmem_cache_free(pgtable_cache[PTE_CACHE_NUM], pte);
+	quicklist_free(0, NULL, pte);
 }
 
 static inline void pte_free(struct page *ptepage)

@@ -136,7 +129,10 @@ static inline void pgtable_free(pgtable_
 	void *p = (void *)(pgf.val & ~PGF_CACHENUM_MASK);
 	int cachenum = pgf.val & PGF_CACHENUM_MASK;
 
-	kmem_cache_free(pgtable_cache[cachenum], p);
+	if (cachenum == PTE_CACHE_NUM)
+		quicklist_free(0, NULL, p);
+	else
+		kmem_cache_free(pgtable_cache[cachenum], p);
 }
 
 extern void pgtable_free_tlb(struct mmu_gather *tlb, pgtable_free_t pgf);

@@ -153,7 +149,7 @@ extern void pgtable_free_tlb(struct mmu_
 		PUD_CACHE_NUM, PUD_TABLE_SIZE-1))
 #endif /* CONFIG_PPC_64K_PAGES */
 
-#define check_pgt_cache()	do { } while (0)
+#define check_pgt_cache()	quicklist_trim(0, NULL, 25, 16)
 
 #endif /* CONFIG_PPC64 */
 #endif /* __KERNEL__ */

Christoph Lameter

2007-05-03 21:15 UTC #179 lore raw

Re: 2.6.22 -mm merge plans: slub on PowerPC

Quoted 7 lines · Hugh Dickins

Quoted 5 lines · Hugh Dickins
Seems we're all wrong in thinking Christoph's Kconfiggery worked as intended: maybe it just works some of the time. I'm not going to hazard a guess as to how to fix it up, will resume looking at the powerpc's quicklist potential later.

Quoted 7 lines

Here's the patch I've been testing on G5, with 4k and with 64k pages, with SLAB and with SLUB. But, though it doesn't crash, the pgd kmem_cache in the 4k-page SLUB case is revealing SLUB's propensity for using highorder allocations where SLAB would stick to order 0:
under load, exec's mm_init gets page allocation failure on order 4
- SLUB's calculate_order may need some retuning. (I'd expect it to be going for order 3 actually, I'm not sure how order 4 comes about.)

There are SLUB patches pending (not in rc7-mm2 as far as I can recall) that reduce the default page order sizes to head off this issue. The defaults were initially too large (and they still default to large for testing if Mel's Antifrag work is detected to be active).

include/asm-powerpc/pgalloc.h static inline void pmd_free(pmd_t *pmd) · line 94

Quoted 3 lines

include/asm-powerpc/pgalloc.h +1 -2

-	return kmem_cache_alloc(pgtable_cache[PTE_CACHE_NUM],
-				GFP_KERNEL|__GFP_REPEAT);
+	return quicklist_alloc(0, GFP_KERNEL|__GFP_REPEAT, NULL);

__GFP_REPEAT is unusual here but this was carried over from the kmem_cache_alloc it seems. Hmm... There is some variance on how we do this between arches. Should we uniformly set or not set this flag?

clameter@schroedinger:~/software/linux-2.6.21-rc7-mm2$ grep quicklist_alloc include/asm-ia64/*
include/asm-ia64/pgalloc.h:     return quicklist_alloc(0, GFP_KERNEL, NULL);
include/asm-ia64/pgalloc.h:     return quicklist_alloc(0, GFP_KERNEL, NULL);
include/asm-ia64/pgalloc.h:     return quicklist_alloc(0, GFP_KERNEL, NULL);
include/asm-ia64/pgalloc.h:     void *pg = quicklist_alloc(0, GFP_KERNEL, NULL);
include/asm-ia64/pgalloc.h:     return quicklist_alloc(0, GFP_KERNEL, NULL);
clameter@schroedinger:~/software/linux-2.6.21-rc7-mm2$ grep quicklist_alloc arch/i386/mm/*    
arch/i386/mm/pgtable.c: pgd_t *pgd = quicklist_alloc(0, GFP_KERNEL, pgd_ctor);
clameter@schroedinger:~/software/linux-2.6.21-rc7-mm2$ grep quicklist_alloc include/asm-sparc64/*
include/asm-sparc64/pgalloc.h:  return quicklist_alloc(0, GFP_KERNEL, NULL);
include/asm-sparc64/pgalloc.h:  return quicklist_alloc(0, GFP_KERNEL, NULL);
include/asm-sparc64/pgalloc.h:  return quicklist_alloc(0, GFP_KERNEL, NULL);
include/asm-sparc64/pgalloc.h:  void *pg = quicklist_alloc(0, GFP_KERNEL, NULL);
clameter@schroedinger:~/software/linux-2.6.21-rc7-mm2$ grep quicklist_alloc include/asm-x86_64/* 
include/asm-x86_64/pgalloc.h:   return (pmd_t *)quicklist_alloc(QUICK_PT, GFP_KERNEL|__GFP_REPEAT, NULL);
include/asm-x86_64/pgalloc.h:   return (pud_t *)quicklist_alloc(QUICK_PT, GFP_KERNEL|__GFP_REPEAT, NULL);
include/asm-x86_64/pgalloc.h:   pgd_t *pgd = (pgd_t *)quicklist_alloc(QUICK_PGD,
include/asm-x86_64/pgalloc.h:   return (pte_t *)quicklist_alloc(QUICK_PT, GFP_KERNEL|__GFP_REPEAT, NULL);
include/asm-x86_64/pgalloc.h:   void *p = (void *)quicklist_alloc(QUICK_PT, GFP_KERNEL|__GFP_REPEAT, NULL);

Hugh Dickins

2007-05-03 22:41 UTC #180 lore raw

Re: 2.6.22 -mm merge plans: slub on PowerPC

Quoted 5 lines · Christoph Lameter

There are SLUB patches pending (not in rc7-mm2 as far as I can recall) that reduce the default page order sizes to head off this issue. The defaults were initially too large (and they still default to large for testing if Mel's Antifrag work is detected to be active).

Sounds good.

Quoted 7 lines

include/asm-powerpc/pgalloc.h static inline void pmd_free(pmd_t *pmd) · line 94
Quoted 3 lines
include/asm-powerpc/pgalloc.h +1 -2
-	return kmem_cache_alloc(pgtable_cache[PTE_CACHE_NUM],
-				GFP_KERNEL|__GFP_REPEAT);
+	return quicklist_alloc(0, GFP_KERNEL|__GFP_REPEAT, NULL);
__GFP_REPEAT is unusual here but this was carried over from the kmem_cache_alloc it seems. Hmm... There is some variance on how we do this between arches. Should we uniformly set or not set this flag?

Not something to get into in this patch, but it did surprise me too. I believe __GFP_REPEAT should be avoided, and I don't see justification for it here (but need to remember not to do a blind virt_to_page on the result in some places if it might return NULL - which IIRC it actually can do even if __GFP_REPEAT, when chosen for OOM kill). But I've a suspicion it got put in there for some good reason I don't know about.

Hugh

Benjamin Herrenschmidt

2007-05-04 00:25 UTC #181 lore raw

Re: 2.6.22 -mm merge plans: slub on PowerPC

Quoted 19 lines · Hugh Dickins

Quoted 5 lines · Hugh Dickins
Seems we're all wrong in thinking Christoph's Kconfiggery worked as intended: maybe it just works some of the time. I'm not going to hazard a guess as to how to fix it up, will resume looking at the powerpc's quicklist potential later.
Here's the patch I've been testing on G5, with 4k and with 64k pages, with SLAB and with SLUB. But, though it doesn't crash, the pgd kmem_cache in the 4k-page SLUB case is revealing SLUB's propensity for using highorder allocations where SLAB would stick to order 0:
under load, exec's mm_init gets page allocation failure on order 4
- SLUB's calculate_order may need some retuning. (I'd expect it to be going for order 3 actually, I'm not sure how order 4 comes about.)
I don't know how offensive Ben and Paulus may find this patch:
the kmem_cache use was nicely done and this messes it up a little.

Quoted 7 lines

The SLUB allocator relies on struct page fields first_page and slab, overwritten by ptl when SPLIT_PTLOCK: so the SLUB allocator cannot then be used for the lowest level of pagetable pages. This was obstructing SLUB on PowerPC, which uses kmem_caches for its pagetables. So convert its pte level to use quicklist pages (whereas pmd, pud and 64k-page pgd want partpages, so continue to use kmem_caches for pmd, pud and pgd). But to keep up appearances for pgtable_free, we still need PTE_CACHE_NUM.

Interesting... I'll have a look asap.

Ben.

Christoph Lameter

2007-05-04 00:54 UTC #182 lore raw

Re: 2.6.22 -mm merge plans: slub on PowerPC

Quoted 9 lines · Benjamin Herrenschmidt

Quoted 7 lines
The SLUB allocator relies on struct page fields first_page and slab, overwritten by ptl when SPLIT_PTLOCK: so the SLUB allocator cannot then be used for the lowest level of pagetable pages. This was obstructing SLUB on PowerPC, which uses kmem_caches for its pagetables. So convert its pte level to use quicklist pages (whereas pmd, pud and 64k-page pgd want partpages, so continue to use kmem_caches for pmd, pud and pgd). But to keep up appearances for pgtable_free, we still need PTE_CACHE_NUM.
Interesting... I'll have a look asap.

I would also recommend looking at removing the constructors for the remaining slabs. A constructor requires that SLUB never touch the object (same situation as is resulting from enabling debugging). So it must increase the object size in order to put the free pointer after the object. In case of a order of 2 cache this has a particularly bad effect of doubling object size. If the objects can be overwritten on free (no constructor) then we can use the first word of the object as a freepointer on kfree. Meaning we can use a hot cacheline so no cache miss. On alloc we have already touched the first cacheline which also avoids a cacheline fetch there. This is the optimal way of operation for SLUB.

Hmmm.... We could add an option to allow the use of a constructor while keeping the free pointer at the beginning of the object? Then we would have to zap the first word on alloc. Would work like quicklists.

Add SLAB_FREEPOINTER_MAY_OVERLAP?

Christoph Lameter

2007-05-03 16:45 UTC #183 lore raw

Re: 2.6.22 -mm merge plans: slub

Hmmmm...There are a gazillion configs to choose from. It works fine with cell_defconfig. If I switch to 2 processors I can enable SLUB if I switch to 4 I cannot.

I saw some other config weirdness like being unable to set SMP if SLOB is enabled with some configs. This should not work and does not work but the menus are then vanishing and one can still configure lots of processors while not having enabled SMP.

It works as far as I can tell... The rest is arch weirdness.

Ingo Molnar

Acked-by

2007-05-03 15:54 UTC #184 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 3 lines · Andrew Morton

- If replying, please be sure to cc the appropriate individuals. Please also consider rewriting the Subject: to something appropriate.

i'm wondering about swap-prefetch:

mm-implement-swap-prefetching.patch
swap-prefetch-avoid-repeating-entry.patch add-__gfp_movable-for-callers-to-flag-allocations-from-high-memory-that-may-be-migrated-swap-prefetch.patch

The swap-prefetch feature is relatively compact:

   10 files changed, 745 insertions(+), 1 deletion(-)

it is contained mostly to itself:

   mm/swap_prefetch.c            |  581 ++++++++++++++++++++++++++++++++

i've reviewed it once again and in the !CONFIG_SWAP_PREFETCH case it's a clear NOP, while in the CONFIG_SWAP_PREFETCH=y case all the feedback i've seen so far was positive. Time to have this upstream and time for a desktop-oriented distro to pick it up.

I think this has been held back way too long. It's .config selectable and it is as ready for integration as it ever is going to be. So it's a win/win scenario.

Acked-by Ingo Molnar <[email protected]>

Ingo

Michal Piotrowski

2007-05-03 16:15 UTC #185 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Hi,

Quoted 21 lines · Ingo Molnar

Quoted 3 lines · Andrew Morton
- If replying, please be sure to cc the appropriate individuals. Please also consider rewriting the Subject: to something appropriate.
i'm wondering about swap-prefetch:
mm-implement-swap-prefetching.patch
swap-prefetch-avoid-repeating-entry.patch add-__gfp_movable-for-callers-to-flag-allocations-from-high-memory-that-may-be-migrated-swap-prefetch.patch
The swap-prefetch feature is relatively compact:
   10 files changed, 745 insertions(+), 1 deletion(-)
it is contained mostly to itself:
   mm/swap_prefetch.c            |  581 ++++++++++++++++++++++++++++++++

Quoted 8 lines

i've reviewed it once again and in the !CONFIG_SWAP_PREFETCH case it's a clear NOP, while in the CONFIG_SWAP_PREFETCH=y case all the feedback i've seen so far was positive. Time to have this upstream and time for a desktop-oriented distro to pick it up.
I think this has been held back way too long. It's .config selectable and it is as ready for integration as it ever is going to be. So it's a win/win scenario.

I'm using SWAP_PREFETCH since 2.6.17-mm1 (I don't have earlier configs)
http://www.stardust.webpages.pl/files/tbf/euridica/2.6.17-mm1/mm-config
and I don't recall _any_ problems. It's very stable for me.

Quoted 4 lines

Acked-by Ingo Molnar <[email protected]>
Ingo

Regards,
Michal

Signature · 9 lines

Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Michal Piotrowski

2007-05-03 16:23 UTC #186 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 29 lines · Michal Piotrowski

Hi,
Quoted 26 lines · Ingo Molnar
Quoted 3 lines · Andrew Morton
- If replying, please be sure to cc the appropriate individuals. Please also consider rewriting the Subject: to something appropriate.
i'm wondering about swap-prefetch:
mm-implement-swap-prefetching.patch
swap-prefetch-avoid-repeating-entry.patch add-__gfp_movable-for-callers-to-flag-allocations-from-high-memory-that-may-be-migrated-swap-prefetch.patch
The swap-prefetch feature is relatively compact:
   10 files changed, 745 insertions(+), 1 deletion(-)
it is contained mostly to itself:
   mm/swap_prefetch.c            |  581 ++++++++++++++++++++++++++++++++
i've reviewed it once again and in the !CONFIG_SWAP_PREFETCH case it's a clear NOP, while in the CONFIG_SWAP_PREFETCH=y case all the feedback i've seen so far was positive. Time to have this upstream and time for a desktop-oriented distro to pick it up.

Quoted 5 lines

Quoted 3 lines
I think this has been held back way too long. It's .config selectable and it is as ready for integration as it ever is going to be. So it's a win/win scenario.
I'm using SWAP_PREFETCH since 2.6.17-mm1 (I don't have earlier configs)

since 2.6.14-mm2 :)
http://lkml.org/lkml/2005/11/11/260

Regards,
Michal

Signature · 9 lines

Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Con Kolivas

2007-05-03 22:14 UTC #187 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

In reply to Ingo Molnar

Quoted 10 lines · Ingo Molnar

Quoted 3 lines · Andrew Morton
- If replying, please be sure to cc the appropriate individuals. Please also consider rewriting the Subject: to something appropriate.
i've reviewed it once again and in the !CONFIG_SWAP_PREFETCH case it's a clear NOP, while in the CONFIG_SWAP_PREFETCH=y case all the feedback i've seen so far was positive. Time to have this upstream and time for a desktop-oriented distro to pick it up.

Quoted 5 lines

I think this has been held back way too long. It's .config selectable and it is as ready for integration as it ever is going to be. So it's a win/win scenario.
Acked-by Ingo Molnar <[email protected]>

Thank you very much for code review, ack and support!

Signature · 7 lines

-ck

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Nick Piggin

2007-05-04 07:34 UTC #188 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

In reply to Ingo Molnar

Quoted 9 lines · Ingo Molnar

Quoted 3 lines · Andrew Morton
- If replying, please be sure to cc the appropriate individuals. Please also consider rewriting the Subject: to something appropriate.
i'm wondering about swap-prefetch:

Well I had some issues with it that I don't think were fully discussed, and Andrew prompted me to say something, but it went off list for a couple of posts (my fault, sorry). Posting it below with Andrew's permission...

Quoted 8 lines

mm-implement-swap-prefetching.patch
swap-prefetch-avoid-repeating-entry.patch add-__gfp_movable-for-callers-to-flag-allocations-from-high-memory-that-may-be-migrated-swap-prefetch.patch
The swap-prefetch feature is relatively compact:
   10 files changed, 745 insertions(+), 1 deletion(-)

Quoted 7 lines

it is contained mostly to itself:
   mm/swap_prefetch.c            |  581 ++++++++++++++++++++++++++++++++
i've reviewed it once again and in the !CONFIG_SWAP_PREFETCH case it's a clear NOP, while in the CONFIG_SWAP_PREFETCH=y case all the feedback i've seen so far was positive. Time to have this upstream and time for a desktop-oriented distro to pick it up.
I think this has been held back way too long. It's .config selectable and it is as ready for integration as it ever is going to be. So it's a win/win scenario.

Being able to config all these core heuristics changes is really not that much of a positive. The fact that we might _need_ to config something out, and double the configuration range isn't too pleasing.

Here were some of my concerns, and where our discussion got up to.

Quoted 16 lines · Andrew Morton

Quoted 11 lines · On Fri, 04 May 2007 14:34:45 +1000 Nick Piggin <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
istr you had issues with swap-prefetch?
If so, now's a good time to reiterate them ;)
1) It is said to help with the updatedb overnight problem, however it actually _doesn't_ prefetch swap when there are low free pages, which is how updatedb will leave the system. So this really puzzles me how it would work. However if updatedb is causing excessive swapout, I think we should try improving use-once algorithms first, for example.

Quoted 3 lines

Yes. Perhaps it just doesn't help with the updatedb thing. Or maybe with normal system activity we get enough free pages to kick the thing off and running. Perhaps updatedb itself has a lot of rss, for example.

Could be, but I don't know. I'd think it unlikely to allow _much_ swapin, if huge amounts of the desktop have been swapped out. But maybe... as I said, nobody seems to have a recipe for these things.

Quoted 2 lines

Would be useful to see this claim substantiated with a real testcase, description of results and an explanation of how and why it worked.

Yes... and then try to first improve regular page reclaim and use-once handling.

Quoted 8 lines

Quoted 4 lines
2) It is a _highly_ speculative operation, and in workloads where periods of low and high page usage with genuinely unused anonymous / tmpfs pages, it could waste power, memory bandwidth, bus bandwidth, disk bandwidth...
Yes. I suspect that's a matter of waiting for the corner-case reporters to complain, then add more heuristics.

Ugh. Well it is a pretty fundamental problem. Basically swap-prefetch is happy to do a _lot_ of work for these things which we have already decided are least likely to be used again.

Quoted 5 lines

Quoted 2 lines
3) I haven't seen a single set of numbers out of it. Feedback seems to have mostly come from people who
Yup. But can we come up with a testcase? It's hard.

I guess it is hard firstly because swapping is quite random to start with. But I haven't even seen basic things like "make -jhuge swapstorm has no regressions".

Quoted 6 lines

Quoted 2 lines
4) If this is helpful, wouldn't it be equally important for things like mapped file pages? Seems like half a solution.
True.

Quoted 7 lines

Without thinking about it, I almost wonder if one could do a userspace implementation with something which pokes around in /proc/pid/pagemap and /proc/pid/kpagemap, perhaps with some additional interfaces added to do a swapcache read. (Give userspace the ability to get at swapcache via a regular fd?)
(otoh the akpm usersapce implementation is swapoff -a;swapon -a)

Perhaps. You may need a few indicators to see whether the system is idle... but OTOH, we've already got a lot of indicators for memory, disk usage, etc. So, maybe :)

Quoted 5 lines

Quoted 2 lines
5) New one: it is possibly going to interact badly with MADV_FREE lazy freeing. The more complex we make page reclaim, the worse IMO.
That's a bit vague. What sort of problems do you envisage?

Well MADV_FREE pages aren't technically free, are they? So it might be possible for a significant number of them to build up and prevent swap prefetch from working. Maybe.

Quoted 6 lines

Quoted 3 lines
...) I had a few issues with implementation, like interaction with
cpusets. Don't know if these are all fixed or not. I sort of gave up looking at it.
Ah yes, I remember some mention of cpusets. I forget what it was though.

I could be wrong, but IIRC there is no good way to know which cpuset to bring the page back into, (and I guess similarly it would be hard to know what container to account it to, if doing account-on-allocate).

We could hope that users of these features would be mostly disjoint sets, but that's an evil road to start heading down, where we have various core bits of mm that don't play nice together.

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Ingo Molnar

2007-05-04 08:52 UTC #189 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 5 lines · Nick Piggin

Quoted 1 line
i'm wondering about swap-prefetch:
Being able to config all these core heuristics changes is really not that much of a positive. The fact that we might _need_ to config something out, and double the configuration range isn't too pleasing.

Well, to the desktop user this is a speculative performance feature that he is willing to potentially waste CPU and IO capacity, in expectation of better performance.

On the conceptual level it is _precisely the same thing as regular file readahead_. (with the difference that to me swapahead seems to be quite a bit more intelligent than our current file readahead logic.)

This feature has no API or ABI impact at all, it's a pure performance feature. (besides the trivial sysctl to turn it runtime on/off).

Quoted 10 lines

Here were some of my concerns, and where our discussion got up to.
Quoted 4 lines
Yes. Perhaps it just doesn't help with the updatedb thing. Or maybe with normal system activity we get enough free pages to kick the thing off and running. Perhaps updatedb itself has a lot of rss, for example.
Could be, but I don't know. I'd think it unlikely to allow _much_ swapin, if huge amounts of the desktop have been swapped out. But maybe... as I said, nobody seems to have a recipe for these things.

can i take this one as a "no fundamental objection"? There are really only 2 maintainance options left:

1) either you can do it better or at least have a _very_ clearly described idea outlined about how to do it differently

2) or you should let others try it

#1 you've not done for 2-3 years since swap-prefetch was waiting for integration so it's not an option at this stage anymore. Then you are pretty much obliged to do #2. ;-)

And let me be really blunt about this, there is no option #3 to say: "I have no real better idea, I have no code, I have no time, but hey, lets not merge this because it 'could in theory' be possible to do it better" =B-)

really, we are likely be better off by risking the merge of _bad_ code (which in the swap-prefetch case is the exact opposite of the truth), than to let code stagnate. People are clearly unhappy about certain desktop aspects of swapping, and the only way out of that is to let more people hack that code. Merging code involves more people. It will cause 'noise' and could cause regressions, but at least in this case the only impact is 'performance' and the feature is trivial to disable.

The maintainance drag outside of swap_prefetch.c is essentially _zero_. If the feature doesnt work it ends up on Con's desk. If it turns out to not work at all (despite years of testing and happy users) it still only ends up on Con's desk. A clear win/win scenario for you i think :-)

Quoted 6 lines

Quoted 3 lines
Would be useful to see this claim substantiated with a real testcase, description of results and an explanation of how and why it worked.
Yes... and then try to first improve regular page reclaim and use-once handling.

agreed. Con, IIRC you wrote a testcase for this, right? Could you please send us the results of that testing?

Quoted 11 lines

Quoted 7 lines
Quoted 4 lines
2) It is a _highly_ speculative operation, and in workloads where periods of low and high page usage with genuinely unused anonymous / tmpfs pages, it could waste power, memory bandwidth, bus bandwidth, disk bandwidth...
Yes. I suspect that's a matter of waiting for the corner-case reporters to complain, then add more heuristics.
Ugh. Well it is a pretty fundamental problem. Basically swap-prefetch is happy to do a _lot_ of work for these things which we have already decided are least likely to be used again.

i see no real problem here. We've had heuristics for a _long_ time in various areas of the code. Sometimes they work, sometimes they suck.

the flow of this is really easy: distro looking for a feature edge turns it on and announces it, if the feature does not work out for users then user turns it off and complains to distro, if enough users complain then distro turns it off for next release, upstream forgets about this performance feature and eventually removes it once someone notices that it wouldnt even compile in the past 2 main releases. I see no problem here, we did that in the past too with performance features. The networking stack has literally dozens of such small tunable things which get experimented with, and whose defaults do get tuned carefully. Some of the knobs help bandwidth, some help latency.

I do not even see any risk of "splitup of mindshare" - swap-prefetch is so clearly speculative that it's not really a different view about how to do swapping (which would split the tester base, etc.), it's simply a "do you want your system to speculate about the future or not" add-on decision. Every system has a pretty clear idea about that: desktops generally want to do it, clusters generally dont want to do it.

Quoted 4 lines

Quoted 4 lines
Quoted 2 lines
3) I haven't seen a single set of numbers out of it. Feedback seems to have mostly come from people who
Yup. But can we come up with a testcase? It's hard.

i think Con has a testcase.

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
4) If this is helpful, wouldn't it be equally important for things like mapped file pages? Seems like half a solution.

[...]

Quoted 5 lines

Quoted 1 line
(otoh the akpm usersapce implementation is swapoff -a;swapon -a)
Perhaps. You may need a few indicators to see whether the system is idle... but OTOH, we've already got a lot of indicators for memory, disk usage, etc. So, maybe :)

The time has passed for this. Let others play too. Please :-)

Quoted 4 lines

I could be wrong, but IIRC there is no good way to know which cpuset to bring the page back into, (and I guess similarly it would be hard to know what container to account it to, if doing account-on-allocate).

(i think cpusets are totally uninteresting in this context: nobody in their right mind is going to use swap-prefetch on a big NUMA box. Nor can i see any fundamental impediment to making this more cpuset-aware, just like other subsystems were made cpuset-aware, once the requests from actual users came in and people started getting interested in it.)

I think the "lack of testcase and numbers" is the only valid technical objection i've seen so far. Con might be able to help us with that?

Ingo

Nick Piggin

2007-05-04 09:09 UTC #190 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 12 lines · Ingo Molnar

Quoted 6 lines · Nick Piggin
Here were some of my concerns, and where our discussion got up to.
Quoted 1 line
Yes. Perhaps it just doesn't help with the updatedb thing. Or maybe with normal system activity we get enough free pages to kick the thing off and running. Perhaps updatedb itself has a lot of rss, for example.
Could be, but I don't know. I'd think it unlikely to allow _much_ swapin, if huge amounts of the desktop have been swapped out. But maybe... as I said, nobody seems to have a recipe for these things.
can i take this one as a "no fundamental objection"? There are really only 2 maintainance options left:

Quoted 7 lines

1) either you can do it better or at least have a _very_ clearly described idea outlined about how to do it differently
2) or you should let others try it
#1 you've not done for 2-3 years since swap-prefetch was waiting for
integration so it's not an option at this stage anymore. Then you are pretty much obliged to do #2. ;-)

The burden is not on me to get someone else's feature merged. If it can be shown to work well and people's concerns addressed, then anything will get merged. The reason Linux is so good is because of what we don't merge, figuratively speaking.

I wanted to see some basic regression tests to show that it hasn't caused obvious problems, and some basic scenarios where it helps, so that we can analyse them. It is really simple, but I haven't got any since first asking.

And note that I don't think I ever explicitly "nacked" anything, just voiced my concerns. If my concerns had been addressed, then I couldn't have stopped anybody from merging anything.

Quoted 11 lines

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
2) It is a _highly_ speculative operation, and in workloads where periods of low and high page usage with genuinely unused anonymous / tmpfs pages, it could waste power, memory bandwidth, bus bandwidth, disk bandwidth...
Yes. I suspect that's a matter of waiting for the corner-case reporters to complain, then add more heuristics.
Ugh. Well it is a pretty fundamental problem. Basically swap-prefetch is happy to do a _lot_ of work for these things which we have already decided are least likely to be used again.
i see no real problem here. We've had heuristics for a _long_ time in various areas of the code. Sometimes they work, sometimes they suck.

So that's one of my issues with the code. If all you have to support a merge is anecodal evidence, then I find it interesting that you would easily discount something like this.

Quoted 11 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
4) If this is helpful, wouldn't it be equally important for things like mapped file pages? Seems like half a solution.
[...]
Quoted 3 lines
Quoted 1 line
(otoh the akpm usersapce implementation is swapoff -a;swapon -a)
Perhaps. You may need a few indicators to see whether the system is idle... but OTOH, we've already got a lot of indicators for memory, disk usage, etc. So, maybe :)
The time has passed for this. Let others play too. Please :-)

Play with what? Prefetching mmaped file pages as well? Sure.

Quoted 4 lines

Quoted 1 line
I could be wrong, but IIRC there is no good way to know which cpuset to bring the page back into, (and I guess similarly it would be hard to know what container to account it to, if doing account-on-allocate).
(i think cpusets are totally uninteresting in this context: nobody in their right mind is going to use swap-prefetch on a big NUMA box. Nor can i see any fundamental impediment to making this more cpuset-aware, just like other subsystems were made cpuset-aware, once the requests from actual users came in and people started getting interested in it.)

OK, so make it more cpuset aware. This isn't a new issue, I raised it a long time ago. And trust me, it is a nightmare to just assume that nobody will use cpusets on a small box for example (AFAIK the resource control guys are looking at doing just that).

All core VM features should play nicely with each other without *really* good reason.

Quoted 1 line

I think the "lack of testcase and numbers" is the only valid technical objection i've seen so far.

Well you're entitled to your opinion too.

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Con Kolivas

2007-05-04 12:10 UTC #191 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

In reply to Ingo Molnar

Quoted 2 lines · Ingo Molnar

agreed. Con, IIRC you wrote a testcase for this, right? Could you please send us the results of that testing?

Yes, sorry it's a crappy test app but works on 32bit. Timed with prefetch disabled and then enabled swap prefetch saves ~5 seconds on average hardware on this one test case. I had many users try this and the results were between 2 and 10 seconds, but always showed a saving on this testcase. This effect easily occurs on printing a big picture, editing a large file, compressing an iso image or whatever in real world workloads. Smaller, but much more frequent effects of this over the course of a day obviously also occur and do add up.

Signature · 1 line

-ck

#include <stdio.h>
#include <stdarg.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>
#include <time.h>

void fatal(const char *format, ...)

{
	va_list ap;

	if (format) {
		va_start(ap, format);
		vfprintf(stderr, format, ap);
		va_end(ap);
	}

	fprintf(stderr, "Fatal error - exiting\n");
	exit(1);
}

size_t get_ram(void)

{
        unsigned long ramsize;
	FILE *meminfo;
        char aux[256];

	if(!(meminfo = fopen("/proc/meminfo", "r")))
		fatal("fopen\n");

	while( !feof(meminfo) && !fscanf(meminfo, "MemTotal: %lu kB", &ramsize) )
		fgets(aux,sizeof(aux),meminfo);
	if (fclose(meminfo) == -1)
		fatal("fclose");
	return ramsize * 1000;
}

unsigned long get_usecs(struct timespec *myts)

{
	if (clock_gettime(CLOCK_REALTIME, myts))
		fatal("clock_gettime");
	return (myts->tv_sec * 1000000 + myts->tv_nsec / 1000 );
}

int main(void)

{
	unsigned long current_time, time_diff;
	struct timespec myts;
	char *buf1, *buf2, *buf3, *buf4;
	size_t size, full_size = get_ram();
	int sleep_seconds = 600;

	size = full_size * 7 / 10;
	printf("Starting first malloc of %d bytes\n", size);
	buf1 = malloc(size);
	if (buf1 == (char *)-1)
		fatal("Failed to malloc 1st buffer\n");
	memset(buf1, 0, size);
	printf("Completed first malloc and starting second malloc of %d bytes\n", full_size);

	buf2 = malloc(full_size);
	if (buf2 == (char *)-1)
		fatal("Failed to malloc 2nd buffer\n");
	memset(buf2, 0, full_size);
	buf4 = malloc(1);
	for (buf3 = buf2 + full_size; buf3 > buf2; buf3--)
		*buf4 = *buf3;
	free(buf2);
	printf("Completed second malloc and free\n");

	printf("Sleeping for %d seconds\n", sleep_seconds);
	sleep(sleep_seconds);

	printf("Important part - starting read of first malloc\n");
	time_diff = current_time = get_usecs(&myts);
	for (buf3 = buf1; buf3 < buf1 + size; buf3++)
		*buf4 = *buf3;
	current_time = get_usecs(&myts);
	free(buf4);
	free(buf1);
	printf("Completed read and freeing of first malloc\n");
	time_diff = current_time - time_diff;
	printf("Timed portion %lu microseconds\n",time_diff);

	return 0;
}

Con Kolivas

2007-05-05 08:42 UTC #192 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 4 lines · Con Kolivas

Quoted 2 lines · Ingo Molnar
agreed. Con, IIRC you wrote a testcase for this, right? Could you please send us the results of that testing?

Quoted 8 lines

Yes, sorry it's a crappy test app but works on 32bit. Timed with prefetch disabled and then enabled swap prefetch saves ~5 seconds on average hardware on this one test case. I had many users try this and the results were between 2 and 10 seconds, but always showed a saving on this testcase. This effect easily occurs on printing a big picture, editing a large file, compressing an iso image or whatever in real world workloads. Smaller, but much more frequent effects of this over the course of a day obviously also occur and do add up.

Here's a better swap prefetch tester. Instructions in file.

Machine with 2GB ram and 2GB swapfile

Prefetch disabled:
./sp_tester
Ram 2060352000 Swap 1973420000
Total ram to be malloced: 3047062000 bytes
Starting first malloc of 1523531000 bytes
Starting 1st read of first malloc
Touching this much ram takes 809 milliseconds
Starting second malloc of 1523531000 bytes
Completed second malloc and free
Sleeping for 600 seconds
Important part - starting reread of first malloc
Completed read of first malloc
Timed portion 53397 milliseconds

Enabled:
./sp_tester
Ram 2060352000 Swap 1973420000
Total ram to be malloced: 3047062000 bytes
Starting first malloc of 1523531000 bytes
Starting 1st read of first malloc
Touching this much ram takes 676 milliseconds
Starting second malloc of 1523531000 bytes
Completed second malloc and free
Sleeping for 600 seconds
Important part - starting reread of first malloc
Completed read of first malloc
Timed portion 26351 milliseconds

Note huge time difference.

Signature · 1 line

-ck

/*
sp_tester.c

Build with:
gcc -o sp_tester sp_tester.c -lrt -W -Wall -O2

How to use:
echo 1 > /proc/sys/vm/overcommit_memory
swapoff -a
swapon -a
./sp_tester

then repeat with changed conditions eg
echo 0 > /proc/sys/vm/swap_prefetch

Each Test takes 10 minutes
*/

#include <stdio.h>
#include <stdarg.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>
#include <time.h>

void fatal(const char *format, ...)

{
	va_list ap;

	if (format) {
		va_start(ap, format);
		vfprintf(stderr, format, ap);
		va_end(ap);
	}

	fprintf(stderr, "Fatal error - exiting\n");
	exit(1);
}

unsigned long ramsize, swapsize;

size_t get_ram(void)

{
	FILE *meminfo;
        char aux[256];

	if(!(meminfo = fopen("/proc/meminfo", "r")))
		fatal("fopen\n");

	while( !feof(meminfo) && !fscanf(meminfo, "MemTotal: %lu kB", &ramsize) )
		fgets(aux,sizeof(aux),meminfo);
	while( !feof(meminfo) && !fscanf(meminfo, "SwapTotal: %lu kB", &swapsize) )
		fgets(aux,sizeof(aux),meminfo);
	if (fclose(meminfo) == -1)
		fatal("fclose");
	ramsize *= 1000;
	swapsize *= 1000;
	printf("Ram %lu  Swap %lu\n", ramsize, swapsize);
	return ramsize + (swapsize / 2);
}

unsigned long get_usecs(struct timespec *myts)

{
	if (clock_gettime(CLOCK_REALTIME, myts))
		fatal("clock_gettime");
	return (myts->tv_sec * 1000000 + myts->tv_nsec / 1000 );
}

int main(void)

{
	unsigned long current_time, time_diff;
	struct timespec myts;
	char *buf1, *buf2, *buf3, *buf4;
	size_t size = get_ram();
	int sleep_seconds = 600;

	if (size > ramsize / 2 * 3)
		size = ramsize / 2 * 3;
	printf("Total ram to be malloced: %u bytes\n", size);
	size /= 2;
	printf("Starting first malloc of %u bytes\n", size);
	buf1 = malloc(size);
	buf4 = malloc(1);
	if (buf1 == (char *)-1)
		fatal("Failed to malloc 1st buffer\n");
	memset(buf1, 0, size);
	time_diff = current_time = get_usecs(&myts);
	for (buf3 = buf1; buf3 < buf1 + size; buf3++)
		*buf4 = *buf3;
	printf("Starting 1st read of first malloc\n");
	current_time = get_usecs(&myts);
	time_diff = current_time - time_diff;
	printf("Touching this much ram takes %lu milliseconds\n",time_diff / 1000);
	printf("Starting second malloc of %u bytes\n", size);

	buf2 = malloc(size);
	if (buf2 == (char *)-1)
		fatal("Failed to malloc 2nd buffer\n");
	memset(buf2, 0, size);
	for (buf3 = buf2 + size; buf3 > buf2; buf3--)
		*buf4 = *buf3;
	free(buf2);
	printf("Completed second malloc and free\n");

	printf("Sleeping for %u seconds\n", sleep_seconds);
	sleep(sleep_seconds);

	printf("Important part - starting reread of first malloc\n");
	time_diff = current_time = get_usecs(&myts);
	for (buf3 = buf1; buf3 < buf1 + size; buf3++)
		*buf4 = *buf3;
	current_time = get_usecs(&myts);
	time_diff = current_time - time_diff;
	printf("Completed read of first malloc\n");
	printf("Timed portion %lu milliseconds\n",time_diff / 1000);

	free(buf1);
	free(buf4);

	return 0;
}

Antonino Ingargiola

2007-05-06 10:13 UTC #193 lore raw

Re: [ck] Re: swap-prefetch: 2.6.22 -mm merge plans

2007/5/5, Con Kolivas <[email protected]>:
[cut]

Quoted 1 line

Here's a better swap prefetch tester. Instructions in file.

The system should be leaved totally inactive for the test duration (10m) right?

Regards,

~ Antonio

Jory A. Pratt

2007-05-06 18:23 UTC #194 lore raw

Re: [ck] Re: swap-prefetch: 2.6.22 -mm merge plans

In reply to Con Kolivas

Quoted 18 lines · Con Kolivas

Here's a better swap prefetch tester. Instructions in file.
Machine with 2GB ram and 2GB swapfile
Prefetch disabled:
./sp_tester
Ram 2060352000 Swap 1973420000
Total ram to be malloced: 3047062000 bytes
Starting first malloc of 1523531000 bytes
Starting 1st read of first malloc
Touching this much ram takes 809 milliseconds
Starting second malloc of 1523531000 bytes
Completed second malloc and free
Sleeping for 600 seconds
Important part - starting reread of first malloc
Completed read of first malloc
Timed portion 53397 milliseconds

Quoted 14 lines

Enabled:
./sp_tester
Ram 2060352000 Swap 1973420000
Total ram to be malloced: 3047062000 bytes
Starting first malloc of 1523531000 bytes
Starting 1st read of first malloc
Touching this much ram takes 676 milliseconds
Starting second malloc of 1523531000 bytes
Completed second malloc and free
Sleeping for 600 seconds
Important part - starting reread of first malloc
Completed read of first malloc
Timed portion 26351 milliseconds

echo 1 > /proc/sys/vm/overcommit_memory
swapoff -a
swapon -a
./sp_tester

Ram 1153644000 Swap 1004052000
Total ram to be malloced: 1655670000 bytes
Starting first malloc of 827835000 bytes
Starting 1st read of first malloc
Touching this much ram takes 937 milliseconds
Starting second malloc of 827835000 bytes
Completed second malloc and free
Sleeping for 600 seconds
Important part - starting reread of first malloc
Completed read of first malloc
Timed portion 15011 milliseconds

echo 0 > /proc/sys/vm/overcommit_memory
swapoff -a
swapon -a
./sp_tester

Ram 1153644000 Swap 1004052000
Total ram to be malloced: 1655670000 bytes
Starting first malloc of 827835000 bytes
Starting 1st read of first malloc
Touching this much ram takes 1125 milliseconds
Starting second malloc of 827835000 bytes
Completed second malloc and free
Sleeping for 600 seconds
Important part - starting reread of first malloc
Completed read of first malloc
Timed portion 14611 milliseconds

Con Kolivas

2007-05-09 23:28 UTC #195 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

In reply to Con Kolivas

Quoted 21 lines · Con Kolivas

Quoted 4 lines · Con Kolivas
Quoted 2 lines · Ingo Molnar
agreed. Con, IIRC you wrote a testcase for this, right? Could you please send us the results of that testing?
Quoted 8 lines
Yes, sorry it's a crappy test app but works on 32bit. Timed with prefetch disabled and then enabled swap prefetch saves ~5 seconds on average hardware on this one test case. I had many users try this and the results were between 2 and 10 seconds, but always showed a saving on this testcase. This effect easily occurs on printing a big picture, editing a large file, compressing an iso image or whatever in real world workloads. Smaller, but much more frequent effects of this over the course of a day obviously also occur and do add up.
Here's a better swap prefetch tester. Instructions in file.
Machine with 2GB ram and 2GB swapfile
Prefetch disabled:
./sp_tester

Quoted 8 lines

Timed portion 53397 milliseconds
Enabled:
./sp_tester
Timed portion 26351 milliseconds
Note huge time difference.

Well how about that? That was the difference with a swap _file_ as I said, but I went ahead and checked with a swap partition as I used to have. I didn't notice, but somewhere in the last few months, swap prefetch code itself being unchanged for a year, seems to have been broken by other changes in the vm and it doesn't even start up prefetching often and has stale swap entries in its list. Once it breaks like that it does nothing from then on. So that leaves me with a quandry now.

Do I:

1. Go ahead and find whatever breakage was introduced and fix it with hopefully a trivial change

2. Do option 1. and then implement support for yet another kernel feature (cpusets) that will be used perhaps never with swap prefetch [No Nick I don't believe you that cpusets have anything to do with normal users on a desktop ever; if it's used on a desktop it will only be by a kernel developer testing the cpusets code].

3. Dump swap prefetch forever and ignore that it ever worked and was helpful and was a lot of work to implement and so on.

Given that even if I do 1 and/or 2 it'll still be blocked from ever going to mainline I think the choice is clear.

Nick since you're personally the gatekeeper for this code, would you like to make a call? Just say 3 and put me out of my misery please.

Signature · 9 lines

-ck

P.S. Ingo, thanks (and sorry) for your involvement here.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Nick Piggin

2007-05-10 00:05 UTC #196 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 9 lines · Con Kolivas

Well how about that? That was the difference with a swap _file_ as I said, but I went ahead and checked with a swap partition as I used to have. I didn't notice, but somewhere in the last few months, swap prefetch code itself being unchanged for a year, seems to have been broken by other changes in the vm and it doesn't even start up prefetching often and has stale swap entries in its list. Once it breaks like that it does nothing from then on. So that leaves me with a quandry now.
Do I:
1. Go ahead and find whatever breakage was introduced and fix it with hopefully a trivial change
2. Do option 1. and then implement support for yet another kernel feature (cpusets) that will be used perhaps never with swap prefetch [No Nick I don't believe you that cpusets have anything to do with normal users on a desktop ever; if it's used on a desktop it will only be by a kernel developer testing the cpusets code].

Quoted 8 lines

or
3. Dump swap prefetch forever and ignore that it ever worked and was helpful and was a lot of work to implement and so on.
Given that even if I do 1 and/or 2 it'll still be blocked from ever going to mainline I think the choice is clear.
Nick since you're personally the gatekeeper for this code, would you like to make a call? Just say 3 and put me out of my misery please.

I'm not the gatekeeper and it is completely up to you whether you want to work on something or not... but I'm sure you understand where I was coming from when I suggested it doesn't get merged yet.

You may not believe this, but I agree that swap prefetching (and prefetching in general) has some potential to help desktop workloads :). But it still should go through the normal process of being tested and questioned and having a look at options for first improving existing code in those problematic cases.

Once that process happens and it is shown to work nicely, etc., then I would not be able to (or want to) keep it from getting merged.

As far as cpusets goes... if your code goes in last, then you have to make it work with what is there, as a rule. People are using cpusets for memory resource control, which would have uses on a desktop system. It is just a really bad precedent to set, having different parts of the VM not work correctly together. Even if you made them mutually exclusive CONFIG_ options, that is still not a very nice solution.

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Con Kolivas

2007-05-10 01:34 UTC #197 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 31 lines · Nick Piggin

Quoted 30 lines · Con Kolivas
Well how about that? That was the difference with a swap _file_ as I said, but I went ahead and checked with a swap partition as I used to have. I didn't notice, but somewhere in the last few months, swap prefetch code itself being unchanged for a year, seems to have been broken by other changes in the vm and it doesn't even start up prefetching often and has stale swap entries in its list. Once it breaks like that it does nothing from then on. So that leaves me with a quandry now.
Do I:
1. Go ahead and find whatever breakage was introduced and fix it with hopefully a trivial change
2. Do option 1. and then implement support for yet another kernel feature (cpusets) that will be used perhaps never with swap prefetch [No Nick I don't believe you that cpusets have anything to do with normal users on a desktop ever; if it's used on a desktop it will only be by a kernel developer testing the cpusets code].
or
3. Dump swap prefetch forever and ignore that it ever worked and was helpful and was a lot of work to implement and so on.
Given that even if I do 1 and/or 2 it'll still be blocked from ever going to mainline I think the choice is clear.

Quoted 6 lines

Quoted 2 lines
Nick since you're personally the gatekeeper for this code, would you like to make a call? Just say 3 and put me out of my misery please.
I'm not the gatekeeper and it is completely up to you whether you want to work on something or not... but I'm sure you understand where I was coming from when I suggested it doesn't get merged yet.

No matter how you spin it, you're the gatekeeper.

Quoted 5 lines

You may not believe this, but I agree that swap prefetching (and prefetching in general) has some potential to help desktop workloads :). But it still should go through the normal process of being tested and questioned and having a look at options for first improving existing code in those problematic cases.

Not this again? Proof was there ages ago that it helped and no proof that it harmed could be found yet you cunningly pretend it never existed. It's been done to death and I'm sick of this.

Quoted 9 lines

Once that process happens and it is shown to work nicely, etc., then I would not be able to (or want to) keep it from getting merged.
As far as cpusets goes... if your code goes in last, then you have to make it work with what is there, as a rule. People are using cpusets for memory resource control, which would have uses on a desktop system. It is just a really bad precedent to set, having different parts of the VM not work correctly together. Even if you made them mutually exclusive CONFIG_ options, that is still not a very nice solution.

That's as close to a 3 as I'm likely to get out of you.

Andrew you'll be relieved to know I would like you to throw swap prefetch and related patches into the bin. Thanks.

Signature · 7 lines

-ck

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Nick Piggin

2007-05-10 01:56 UTC #198 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 8 lines · Con Kolivas

Quoted 3 lines · Nick Piggin
I'm not the gatekeeper and it is completely up to you whether you want to work on something or not... but I'm sure you understand where I was coming from when I suggested it doesn't get merged yet.
No matter how you spin it, you're the gatekeeper.

If raising unaddressed issues means closing a gate, then OK. You can equally open it by answering them.

Quoted 8 lines

Quoted 5 lines
You may not believe this, but I agree that swap prefetching (and prefetching in general) has some potential to help desktop workloads :). But it still should go through the normal process of being tested and questioned and having a look at options for first improving existing code in those problematic cases.
Not this again? Proof was there ages ago that it helped and no proof that it harmed could be found yet you cunningly pretend it never existed. It's been done to death and I'm sick of this.

I said I know it can help. Do you know how many patches I have that help some workloads but are not merged? That's just the way it works.

What I have seen is it helps the case where you force out a huge amount of swap. OK, that's nice -- the base case obviously works.

You said it helped with the updatedb problem. That says we should look at why it is going bad first, and for example improve use-once algorithms. After we do that, then swap prefetching might still help, which is fine.

Quoted 11 lines

Quoted 9 lines
Once that process happens and it is shown to work nicely, etc., then I would not be able to (or want to) keep it from getting merged.
As far as cpusets goes... if your code goes in last, then you have to make it work with what is there, as a rule. People are using cpusets for memory resource control, which would have uses on a desktop system. It is just a really bad precedent to set, having different parts of the VM not work correctly together. Even if you made them mutually exclusive CONFIG_ options, that is still not a very nice solution.

Quoted 1 line

That's as close to a 3 as I'm likely to get out of you.

If you're not willing to try making it work with existing code, among other things, then yes it will be difficult to get it merged. That's not going to change.

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Ray Lee

2007-05-10 03:48 UTC #199 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 3 lines · Nick Piggin

You said it helped with the updatedb problem. That says we should look at why it is going bad first, and for example improve use-once algorithms. After we do that, then swap prefetching might still help, which is fine.

Nick, if you're volunteering to do that analysis, then great. If not, then you're just providing a airy hope with nothing to back up when or if that work would ever occur.

Further, if you or someone else *does* do that work, then guess what, we still have the option to rip out the swap prefetching code after the hypothetical use-once improvements have been proven and merged. Which, by the way, I've watched people talk about since 2.4. That was, y'know, a *while* ago.

So enough with the stop energy, okay? You're better than that.

Con? He is right about the last feature to go in needs to work gracefully with what's there now. However, it's not unheard of for authors of other sections of code to help out with incompatibilities by answering politely phrased questions for guidance. Though the intersection of users between cpusets and desktop systems seems small indeed.

Nick Piggin

2007-05-10 03:56 UTC #200 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 10 lines · Ray Lee

Quoted 3 lines · Nick Piggin
You said it helped with the updatedb problem. That says we should look at why it is going bad first, and for example improve use-once algorithms. After we do that, then swap prefetching might still help, which is fine.
Nick, if you're volunteering to do that analysis, then great. If not, then you're just providing a airy hope with nothing to back up when or if that work would ever occur.

I'd like to try helping. Tell me your problem.

Quoted 5 lines

Further, if you or someone else *does* do that work, then guess what, we still have the option to rip out the swap prefetching code after the hypothetical use-once improvements have been proven and merged. Which, by the way, I've watched people talk about since 2.4. That was, y'know, a *while* ago.

What's wrong with the use-once we have? What improvements are you talking about?

Quoted 1 line

So enough with the stop energy, okay? You're better than that.

I don't think it is about energy or being mean, I'm just stating the issues I have with it.

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Ray Lee

2007-05-10 05:52 UTC #201 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 7 lines · Nick Piggin

Quoted 6 lines · Ray Lee
Quoted 3 lines · Nick Piggin
You said it helped with the updatedb problem. That says we should look at why it is going bad first, and for example improve use-once algorithms. After we do that, then swap prefetching might still help, which is fine.

Quoted 5 lines

Quoted 3 lines
Nick, if you're volunteering to do that analysis, then great. If not, then you're just providing a airy hope with nothing to back up when or if that work would ever occur.
I'd like to try helping. Tell me your problem.

Huh? You already stated one version of it above, namely updatedb. But let's put this another way, shall we? A gedankenexperiment, if you will.

Say we have a perfect swap-out algorithm that can choose exactly what needs to be evicted to disk. ('Perfect', of course, is dependent upon one's metric, but let's go with "maximizes overall system utilization and minimizes IO wait time." Arbitrary, but hey.)

So, great, the right things got swapped out. Anything else that could have been chosen would have caused more overall IO Wait. Yay us.

So what happens when those processes that triggered the swap-outs go away? (Firefox is closed, I stop hitting my local copy of a database, whatever.) Well, currently, nothing. What happens when I switch workspaces and try to use my email program? Swap-ins.

Okay, so why didn't the system swap that stuff in preemptively? Why am I sitting there waiting for something that it could have already done in the background?

A new swap-out algorithm, be it use-once, Clock-Pro, or perfect foreknowledge isn't going to change that issue. Swap prefetch does.

Quoted 8 lines

Quoted 5 lines
Further, if you or someone else *does* do that work, then guess what, we still have the option to rip out the swap prefetching code after the hypothetical use-once improvements have been proven and merged. Which, by the way, I've watched people talk about since 2.4. That was, y'know, a *while* ago.
What's wrong with the use-once we have? What improvements are you talking about?

You said, effectively: "Use-once could be improved to deal with updatedb". I said I've been reading emails from Rik and others talking about that for four years now, and we're still talking about it. Were it merely updatedb, I'd say us userspace folk should step up and rewrite the damn thing to amortize its work. However, I and others feel it's only an example -- glaring, obviously -- of a more pervasive issue. A small issue, to be sure!, but an issue nevertheless.

In general, I/others are talking about improving the desktop experience of running too much on a RAM limited machine. (Which, in my case, is with a gig and a 2.2GHz processor.)

Or restated: the desktop experience occasionally sucks for me, and I don't think I'm alone. There may be a heuristic, completely isolated from userspace (and so isn't an API the kernel has to support! -- if it doesn't work, we can rip it out again), that may mitigate the suckiness. Let's try it.

Quoted 4 lines

Quoted 1 line
So enough with the stop energy, okay? You're better than that.
I don't think it is about energy or being mean, I'm just stating the issues I have with it.

Nick, I in no way think you're being mean, and I'm sorry if I've given you that impression. However, if you're just stating the issues you have with it, then can I assume that you won't lobby against having this experiment merged?

Nick Piggin

2007-05-10 07:04 UTC #202 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 12 lines · Ray Lee

Quoted 10 lines · Nick Piggin
Quoted 3 lines · Ray Lee
Quoted 1 line · Nick Piggin
You said it helped with the updatedb problem. That says we should
look at
Quoted 1 line
Quoted 1 line
why it is going bad first, and for example improve use-once
algorithms.
Quoted 1 line
Quoted 1 line
After we do that, then swap prefetching might still help, which is
fine.
Quoted 1 line

Quoted 8 lines

Quoted 5 lines
Quoted 3 lines
Nick, if you're volunteering to do that analysis, then great. If not, then you're just providing a airy hope with nothing to back up when or if that work would ever occur.
I'd like to try helping. Tell me your problem.
Huh? You already stated one version of it above, namely updatedb. But

So a swapping problem with updatedb should be unusual and we'd like to see if we can fix it without resorting to prefetching.

I know the theory behind swap prefetching, and I'm not saying it doesn't work, so I'll snip the rest of that.

Quoted 11 lines

Quoted 2 lines
What's wrong with the use-once we have? What improvements are you talking about?
You said, effectively: "Use-once could be improved to deal with updatedb". I said I've been reading emails from Rik and others talking about that for four years now, and we're still talking about it. Were it merely updatedb, I'd say us userspace folk should step up and rewrite the damn thing to amortize its work. However, I and others feel it's only an example -- glaring, obviously -- of a more pervasive issue. A small issue, to be sure!, but an issue nevertheless.

It isn't going to get fixed unless people complain about it. If you cover the use-once problem with swap prefetching, then it will never get fixed.

Quoted 8 lines

Quoted 2 lines
I don't think it is about energy or being mean, I'm just stating the issues I have with it.
Nick, I in no way think you're being mean, and I'm sorry if I've given you that impression. However, if you're just stating the issues you have with it, then can I assume that you won't lobby against having this experiment merged?

Anybody is free to merge anything into their kernel. And if somebody asks for my issues with the swap prefetching patch, then I'll give them :)

Signature · 7 lines

SUSE Labs, Novell Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

William Lee Irwin III

2007-05-10 07:20 UTC #203 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 1 line · Ray Lee

Quoted 1 line
Huh? You already stated one version of it above, namely updatedb. But

Quoted 4 lines · Nick Piggin

So a swapping problem with updatedb should be unusual and we'd like to see if we can fix it without resorting to prefetching.
I know the theory behind swap prefetching, and I'm not saying it doesn't work, so I'll snip the rest of that.

I've not run updatedb in years, so I have no idea what it does to a modern kernel. It used to be an unholy terror of slab fragmentation and displacing user memory. The case of streaming kernel metadata IO is probably not quite as easy as streaming file IO.

Quoted 7 lines · Ray Lee

Quoted 7 lines
You said, effectively: "Use-once could be improved to deal with updatedb". I said I've been reading emails from Rik and others talking about that for four years now, and we're still talking about it. Were it merely updatedb, I'd say us userspace folk should step up and rewrite the damn thing to amortize its work. However, I and others feel it's only an example -- glaring, obviously -- of a more pervasive issue. A small issue, to be sure!, but an issue nevertheless.

Quoted 3 lines · Nick Piggin

It isn't going to get fixed unless people complain about it. If you cover the use-once problem with swap prefetching, then it will never get fixed.

The policy people need to clean this up once and for all at some point. clameter's targeted reclaim bits for slub look like a plausible tactic, but are by no means comprehensive. Things need to attempt to eat their own tails before eating everyone else alive. Maybe we need to take hits on things such as badari's dd's to resolve the pathologies.

-- wli

Ray Lee

2007-05-10 12:34 UTC #204 lore raw

Re: swap-prefetch: 2.6.22 -mm merge plans

Quoted 7 lines · Nick Piggin

Quoted 1 line
Huh? You already stated one version of it above, namely updatedb. But
So a swapping problem with updatedb should be unusual and we'd like to see if we can fix it without resorting to prefetching.
I know the theory behind swap prefetching, and I'm not saying it doesn't work, so I'll snip the rest of that.

updatedb is only part of the problem. The other part is that the kernel has an opportunity to preemptively return some of the evicted working set to RAM before I ask for it. No fancy use-once algorithm is going to address that, so your solution is provably incomplete for my problem.

What's so hard to understand about that?

Con Kolivas

2007-05-12 04:45 UTC #205 lore raw

PATCH 1/1 mm: swap prefetch improvements