maps: pagemap, kpagemap, and related cleanups

Move the page walker code to lib/

This lets it get shared outside of proc/ and linked in only when needed.

Quoted 4 lines · Matt Mackall

Move the page walker code to lib/

This lets it get shared outside of proc/ and linked in only when needed.

I think it would be better in mm/.

So would clear_refs_pte_range, and clear_refs_write (in a more generic form), IMO.

Sweet patchset, though.

Quoted 4 lines · Nick Piggin

Quoted 2 lines · Matt Mackall

Move the page walker code to lib/

Quoted 4 lines

Quoted 2 lines

This lets it get shared outside of proc/ and linked in only when needed.

I think it would be better in mm/.

I originally was looking at putting it in mm/memory.c and possibly

converting some users there. But I decided it ought to be conditionally linked until it had some core users and I wasn't quite ready to tackle the tricky mm/ bits.

Make /proc/pid/clear_refs option under CONFIG_EMBEDDED

This interface is primarily useful for doing memory profiling and not much use on deployed embedded boxes. Make it optional. Together with /proc/pid/smaps, this save a few K.

Signed-off-by Matt Mackall <[email protected]>

Add /proc/pid/pagemap interface

This interface provides a mapping for each page in an address space to its physical page frame number, allowing precise determination of what pages are mapped and what pages are shared between processes.

fs/proc/task_mmu.c +6 -0

+#ifdef CONFIG_PROC_PAGEMAP
+struct pagemapread {
+	struct mm_struct *mm;
+	unsigned long next;
+	unsigned long *buf;
+	unsigned long pos;

This pushes binary data to the user space. Wasn't /proc supposed to be ascii-based to avoid compatibility problems (e.g., size of unsigned long changing, endianness, etc.)?

Now I could adjust these to only export u64s in some preferred endianness. But given I already need details like the page size to make any sense of it, it seems unnecessary. Also, the PFNs are fairly opaque unless you're attempting to correlate them with /proc/kpagemap.

Quoted 9 lines · Matt Mackall

Quoted 7 lines · Nick Piggin

Quoted 4 lines · Matt Mackall

Move the page walker code to lib/

This lets it get shared outside of proc/ and linked in only when needed.

Quoted 4 lines

Quoted 1 line

I think it would be better in mm/.

I originally was looking at putting it in mm/memory.c and possibly

Just put it in its own file in mm/ rather than its own file in lib. lib should be for almost-standalone stuff, IMO (ie. only using basic kernel functionality).

Apart from these users outside mm/, I don't see much point in converting things over. The page table walking API we have now is neat and simple. It takes a few lines of code, but is it a big problem?

Matt Mackall writes:

[...]

Quoted 1 line

Quoted 4 lines

Now I could adjust these to only export u64s in some preferred endianness. But given I already need details like the page size to make any sense of it, it seems unnecessary. Also, the PFNs are fairly opaque unless you're attempting to correlate them with /proc/kpagemap.

Alternatively, you can export some meta-data at the beginning of that file like /proc/profile does.

Quoted 16 lines · Nick Piggin

Quoted 14 lines · Matt Mackall

Quoted 9 lines · Nick Piggin

Quoted 4 lines · Matt Mackall

Move the page walker code to lib/

This lets it get shared outside of proc/ and linked in only when needed.

I think it would be better in mm/.

Quoted 5 lines

Quoted 1 line

I originally was looking at putting it in mm/memory.c and possibly

Just put it in its own file in mm/ rather than its own file in lib. lib should be for almost-standalone stuff, IMO (ie. only using basic kernel functionality).

Arguably that's what lib/ should be for, but it's currently largely

used to avoid linking in unused code without adding more hair to Kconfig. Which is what I'm trying to do here.

Quoted 3 lines

Apart from these users outside mm/, I don't see much point in converting things over. The page table walking API we have now is neat and simple. It takes a few lines of code, but is it a big problem?

I don't think it really qualifies as either neat or simple. It may be about as neat as walking a heterogenous tree with an inconsistent naming scheme can be, but it's still a headache. A maze of twisty little passages, all slightly different.

Quoted 8 lines · Matt Mackall

Quoted 3 lines · Nick Piggin

Just put it in its own file in mm/ rather than its own file in lib. lib should be for almost-standalone stuff, IMO (ie. only using basic kernel functionality).

Quoted 1 line

Arguably that's what lib/ should be for, but it's currently largely

I disagree. There is code everywhere that exists to provide some functionality via an API to other parts of the kernel. You don't think mm/page_alloc.c should go in lib/?

This patch series introduces /proc/pid/pagemap and /proc/kpagemap, which allow detailed run-time examination of process memory usage at a page granularity.
The first several patches whip the page-walking code introduced for /proc/pid/smaps and clear_refs into a more generic form, the next couple make those interfaces optional, and the last two introduce the new interfaces, also optional.

Quoted 7 lines · Matt Mackall

This patch series introduces /proc/pid/pagemap and /proc/kpagemap, which allow detailed run-time examination of process memory usage at a page granularity.
The first several patches whip the page-walking code introduced for /proc/pid/smaps and clear_refs into a more generic form, the next couple make those interfaces optional, and the last two introduce the new interfaces, also optional.

This solves a real-life problem for Oracle system monitoring software (specifically EM). Among the tasks it must carry out is determining per-process memory footprint of a set of cooperating tasks (i.e. Oracle processes). RSS is inadequate for this due to page sharing; this work provides sufficient information to determine what EM needs.

+ * Each entry is a pair of unsigned longs representing the
+ * corresponding physical page, the first containing the page flags
+ * and the second containing the page use count.
+ *
+ * The first 4 bytes of this file form a simple header:
+ *

+ * first byte:   0 for big endian, 1 for little
+ * second byte:  page shift (eg 12 for 4096 byte pages)
+ * third byte:   entry size in bytes (currently either 4 or 8)
+ * fourth byte:  header size

...

+ ((char *)page)[1] = PAGE_SHIFT;

+ ((char *)page)[2] = sizeof(unsigned long);

+ ((char *)page)[3] = KPMSIZE;

(quoted diff) +4 -0

+                       i = 2;
+                       pfn++;
+               }
+

Quoted 5 lines

This solves a real-life problem for Oracle system monitoring software (specifically EM). Among the tasks it must carry out is determining per-process memory footprint of a set of cooperating tasks (i.e. Oracle processes). RSS is inadequate for this due to page sharing; this work provides sufficient information to determine what EM needs.

Not a good idea to expose raw flags in this manner - it changes at the drop of a hat. We'd need to also expose the kernel's PG_foo-to-bitnumber mapping to make this viable.
Not a good idea to use page->_count: page_count() will be more stable. Otherwise OK, I guess: the interpretation of the page refcount is unlikely to change much over time.

Review

Quoted 8 lines · On Thu, 12 Apr 2007 16:10:50 -0700 William Lee Irwin III <[email protected]>…

(quoted diff) +8 -0

+       while (count > 0) {
+               chunk = min_t(size_t, count, PAGE_SIZE);
+               i = 0;
+
+               if (pfn == -1) {
+                       page[0] = 0;
+                       page[1] = 0;
+                       ((char *)page)[0] = (ntohl(1) != 1);

OK.

Quoted 1 line

+ ((char *)page)[1] = PAGE_SHIFT;

OK.

Review

Quoted 10 lines

(quoted diff) +10 -0

+               for (; i < 2 * chunk / KPMSIZE; i += 2, pfn++) {
+                       ppage = pfn_to_page(pfn);
+                       if (!ppage) {
+                               page[i] = 0;
+                               page[i + 1] = 0;
+                       } else {
+                               page[i] = ppage->flags;
+                               page[i + 1] = atomic_read(&ppage->_count);
+                       }
+               }

Not a good idea to expose raw flags in this manner - it changes at the drop of a hat. We'd need to also expose the kernel's PG_foo-to-bitnumber mapping to make this viable.

Quoted 5 lines · On Thu, 12 Apr 2007 16:10:50 -0700 William Lee Irwin III <[email protected]>…

Quoted 5 lines

This solves a real-life problem for Oracle system monitoring software (specifically EM). Among the tasks it must carry out is determining per-process memory footprint of a set of cooperating tasks (i.e. Oracle processes). RSS is inadequate for this due to page sharing; this work provides sufficient information to determine what EM needs.

Quoted 5 lines · Andrew Morton

Not a good idea to expose raw flags in this manner - it changes at the drop
of a hat. We'd need to also expose the kernel's PG_foo-to-bitnumber
mapping to make this viable.
Not a good idea to use page->_count: page_count() will be more stable. Otherwise OK, I guess: the interpretation of the page refcount is unlikely to change much over time.

EM wants to determine page_mapcount() for the most part for the purposes of determining "uniquely attributable RSS" (my ca. 2004 nomenclature) or "proportional RSS" (mpm's more recent nomenclature); as things now stand it will have to infer them by maintaining a table of pfn's and mappings thereof, but at least that can be done with it.

Quoted 9 lines · On Thu, 12 Apr 2007 16:10:50 -0700 William Lee Irwin III <[email protected]>…

Quoted 7 lines · Matt Mackall

This patch series introduces /proc/pid/pagemap and /proc/kpagemap, which allow detailed run-time examination of process memory usage at a page granularity.
The first several patches whip the page-walking code introduced for /proc/pid/smaps and clear_refs into a more generic form, the next couple make those interfaces optional, and the last two introduce the new interfaces, also optional.

Quoted 5 lines

This solves a real-life problem for Oracle system monitoring software (specifically EM). Among the tasks it must carry out is determining per-process memory footprint of a set of cooperating tasks (i.e. Oracle processes). RSS is inadequate for this due to page sharing; this work provides sufficient information to determine what EM needs.

I'm still dying to see what the human-readable output from this thing looks like.

Quoted 16 lines · Andrew Morton

Review

Quoted 8 lines · On Thu, 12 Apr 2007 16:10:50 -0700 William Lee Irwin III <[email protected]>…

(quoted diff) +8 -0

+       while (count > 0) {
+               chunk = min_t(size_t, count, PAGE_SIZE);
+               i = 0;
+
+               if (pfn == -1) {
+                       page[0] = 0;
+                       page[1] = 0;
+                       ((char *)page)[0] = (ntohl(1) != 1);

OK.

Quoted 4 lines

Quoted 1 line

+ ((char *)page)[1] = PAGE_SHIFT;

OK.

Shouldn't we just expose page size and endianness by other means? (another file or
syscall).

Quoted 3 lines

Not a good idea to expose raw flags in this manner - it changes at the drop of a hat. We'd need to also expose the kernel's PG_foo-to-bitnumber mapping to make this viable.

I don't think it is viable because that makes the flags part of the userspace ABI. I wonder what they are needed for.

Quoted 4 lines

Quoted 1 line

+ ((char *)page)[1] = PAGE_SHIFT;

OK.

Shouldn't we just expose page size and endianness by other means? (another file or syscall).

Quoted 12 lines

Quoted 10 lines

(quoted diff) +10 -0

+               for (; i < 2 * chunk / KPMSIZE; i += 2, pfn++) {
+                       ppage = pfn_to_page(pfn);
+                       if (!ppage) {
+                               page[i] = 0;
+                               page[i + 1] = 0;
+                       } else {
+                               page[i] = ppage->flags;
+                               page[i + 1] = atomic_read(&ppage->_count);
+                       }
+               }

Quoted 3 lines

Not a good idea to expose raw flags in this manner - it changes at the drop of a hat. We'd need to also expose the kernel's PG_foo-to-bitnumber mapping to make this viable.

I don't think it is viable because that makes the flags part of the userspace ABI.

I wonder what they are needed for.

Quoted 22 lines · Nick Piggin

Quoted 20 lines · Andrew Morton

Review

Quoted 8 lines · On Thu, 12 Apr 2007 16:10:50 -0700 William Lee Irwin III <[email protected]>…

(quoted diff) +8 -0

+       while (count > 0) {
+               chunk = min_t(size_t, count, PAGE_SIZE);
+               i = 0;
+
+               if (pfn == -1) {
+                       page[0] = 0;
+                       page[1] = 0;
+                       ((char *)page)[0] = (ntohl(1) != 1);

OK.

Quoted 1 line

+ ((char *)page)[1] = PAGE_SHIFT;

Quoted 4 lines

Quoted 1 line

OK.

Shouldn't we just expose page size and endianness by other means? (another file or syscall).

If I send you this file dumped from a random machine, you won't know what to make of it.

I'm planning to write a trivial server to sit on, say, my embedded target and spew this over the wire to a client.

Quoted 4 lines

Quoted 3 lines

Not a good idea to expose raw flags in this manner - it changes at the drop of a hat. We'd need to also expose the kernel's PG_foo-to-bitnumber mapping to make this viable.

Quoted 2 lines

I don't think it is viable because that makes the flags part of the userspace ABI. I wonder what they are needed for.

Basically: to show what the hell's going on in the VM.

Quoted 12 lines · On Fri, 13 Apr 2007 10:15:24 +1000 Nick Piggin <[email protected]> wrote:

Quoted 12 lines

Quoted 10 lines

(quoted diff) +10 -0

+               for (; i < 2 * chunk / KPMSIZE; i += 2, pfn++) {
+                       ppage = pfn_to_page(pfn);
+                       if (!ppage) {
+                               page[i] = 0;
+                               page[i + 1] = 0;
+                       } else {
+                               page[i] = ppage->flags;
+                               page[i + 1] = atomic_read(&ppage->_count);
+                       }
+               }

Quoted 6 lines

Quoted 3 lines

Not a good idea to expose raw flags in this manner - it changes at the drop of a hat. We'd need to also expose the kernel's PG_foo-to-bitnumber mapping to make this viable.

I don't think it is viable because that makes the flags part of the userspace ABI.

It *will* be viable. If the application wants to know if a page is dirty, it looks up "PG_dirty" in /proc/pg_foo-to-bitnumber and uses PG_dirty's numerical offset when inspecting fields in /proc/kpagemap. If correctly designed, such a monitoring application will be able to report upon page flags which we haven't even thought up yet.

Quoted 1 line

I wonder what they are needed for.

Poking deeply into the kernel to provide information about kernel state.

There are real-world needs for this, and the people who develop tools to process this information will have decent kernel understanding and will know that the file's contents may alter across kernel versions. It sure beats poking around in /dev/kmem.

I doubt if there's a sensible way in which we can prettify this interface without losing information. But we should aim to make it as robust as possible agaisnt future kenrel changes, of course.

And we should satisfy ourselves that all the required information has been made available. The fact that it will satisfy the Oracle requirement is encouraging.

Quoted 22 lines · Andrew Morton

Quoted 12 lines · On Fri, 13 Apr 2007 10:15:24 +1000 Nick Piggin <[email protected]> wrote:

Quoted 12 lines

Quoted 10 lines

(quoted diff) +10 -0

+               for (; i < 2 * chunk / KPMSIZE; i += 2, pfn++) {
+                       ppage = pfn_to_page(pfn);
+                       if (!ppage) {
+                               page[i] = 0;
+                               page[i + 1] = 0;
+                       } else {
+                               page[i] = ppage->flags;
+                               page[i + 1] = atomic_read(&ppage->_count);
+                       }
+               }

Quoted 6 lines

Quoted 3 lines

Not a good idea to expose raw flags in this manner - it changes at the drop of a hat. We'd need to also expose the kernel's PG_foo-to-bitnumber mapping to make this viable.

I don't think it is viable because that makes the flags part of the userspace ABI.

Quoted 5 lines

It *will* be viable. If the application wants to know if a page is dirty, it looks up "PG_dirty" in /proc/pg_foo-to-bitnumber and uses PG_dirty's numerical offset when inspecting fields in /proc/kpagemap. If correctly designed, such a monitoring application will be able to report upon page flags which we haven't even thought up yet.

Ooh, you wanted a _runtime_ mapping of flags, yeah then I guess that works. Still seems like a basically hit and miss affair to just use flags. What if you want to know the process mapping a page? With systemtap or something you could walk the rmap structures. What if you want to look at pages along the LRU list rather than per-pfn? What about connecting pages to inodes?

But I was going to say
that satisfying an Oracle requirement is a good reason _not_ to merge it ;)

(I joke!)

Quoted 8 lines · On Fri, 13 Apr 2007 11:14:20 +1000 Nick Piggin <[email protected]> wrote:

Quoted 5 lines · Andrew Morton

It *will* be viable. If the application wants to know if a page is dirty, it looks up "PG_dirty" in /proc/pg_foo-to-bitnumber and uses PG_dirty's numerical offset when inspecting fields in /proc/kpagemap. If correctly designed, such a monitoring application will be able to report upon page flags which we haven't even thought up yet.

Quoted 5 lines

Ooh, you wanted a _runtime_ mapping of flags, yeah then I guess that works. Still seems like a basically hit and miss affair to just use flags. What if you want to know the process mapping a page? With systemtap or something you could walk the rmap structures. What if you want to look at pages along the LRU list rather than per-pfn? What about connecting pages to inodes?

Well hang on. This isn't a tool for understanding kernel behaviour. It's a tool for understanding applciation behaviour.

So one doesn't ask "who is mapping that page" - that's a kernel developer thing.

Instead, one says "what pages are being used by my application", then, for

each of those pages "what is that page's state". So the first step is to collect all the pfns from /proc/$(pidof my-application)/pagemap and then to use those pfns to look the individual pages up in /proc/kpagemap.

If you really want to know "who is using page 123435" then you'd need to search /proc/*/pagemap. There are possibly legitimate reasons why an application developer would want to at least pertially perform such an operation ("who am I sharing with"), but I doubt if it's the common case.

Quoted 1 line

Basically: to show what the hell's going on in the VM.

kprobes / systemtap isn't good enough?

Quoted 21 lines · Andrew Morton

Quoted 8 lines · On Fri, 13 Apr 2007 11:14:20 +1000 Nick Piggin <[email protected]> wrote:

Quoted 5 lines · Andrew Morton

It *will* be viable. If the application wants to know if a page is dirty, it looks up "PG_dirty" in /proc/pg_foo-to-bitnumber and uses PG_dirty's numerical offset when inspecting fields in /proc/kpagemap. If correctly designed, such a monitoring application will be able to report upon page flags which we haven't even thought up yet.

Quoted 5 lines

Ooh, you wanted a _runtime_ mapping of flags, yeah then I guess that works. Still seems like a basically hit and miss affair to just use flags. What if you want to know the process mapping a page? With systemtap or something you could walk the rmap structures. What if you want to look at pages along the LRU list rather than per-pfn? What about connecting pages to inodes?

Well hang on. This isn't a tool for understanding kernel behaviour. It's a tool for understanding applciation behaviour.

Quoted 4 lines

So one doesn't ask "who is mapping that page" - that's a kernel developer thing.

Instead, one says "what pages are being used by my application", then, for

That includes unmapped pagecache being used by my application, doesn't it? Maybe that's too hard to do via /proc so we forget about it...

Quoted 4 lines

each of those pages "what is that page's state". So the first step is to collect all the pfns from /proc/$(pidof my-application)/pagemap and then to use those pfns to look the individual pages up in /proc/kpagemap.

OK I realise you could do it that way, but systemtap can definitely be used as a tool for understanding application behaviour in the context of the kernel, I think? The purpose for it is so that various little bits of deep kernel internals do not have to be exposed on a case by case basis.

If kprobes is simply crappy and doesn't work properly for this, then I could accept that. I'm not someone trying to get this info. So why can't it be used? (not just for kpagemap, but for clear_refs and all that gunk too).

Quoted 4 lines

If you really want to know "who is using page 123435" then you'd need to search /proc/*/pagemap. There are possibly legitimate reasons why an application developer would want to at least pertially perform such an operation ("who am I sharing with"), but I doubt if it's the common case.

Maybe. How about LRU? Reclaim performance is bad, and you want to work out which pages keep going off the end of it, or which pages keep getting written out via it, or who's pages are on the active list, forcing mine out.

Quoted 1 line

Instead, one says "what pages are being used by my application", then, for

That includes unmapped pagecache being used by my application, doesn't it? Maybe that's too hard to do via /proc so we forget about it...

Quoted 3 lines

each of those pages "what is that page's state". So the first step is to collect all the pfns from /proc/$(pidof my-application)/pagemap and then to use those pfns to look the individual pages up in /proc/kpagemap.

OK I realise you could do it that way, but systemtap can definitely be used as a tool for understanding application behaviour in the context of the kernel, I think? The purpose for it is so that various little bits of deep kernel internals do not have to be exposed on a case by case basis.

If kprobes is simply crappy and doesn't work properly for this, then I could accept that. I'm not someone trying to get this info. So why can't it be used? (not just for kpagemap, but for clear_refs and all that gunk too).

Quoted 4 lines

If you really want to know "who is using page 123435" then you'd need to search /proc/*/pagemap. There are possibly legitimate reasons why an application developer would want to at least pertially perform such an operation ("who am I sharing with"), but I doubt if it's the common case.

Maybe. How about LRU? Reclaim performance is bad, and you want to work out which pages keep going off the end of it, or which pages keep getting written out via it, or who's pages are on the active list, forcing mine out.

Quoted 2 lines · Nick Piggin

Quoted 1 line

Basically: to show what the hell's going on in the VM.

Quoted 1 line

kprobes / systemtap isn't good enough?

It's not really a good match to the kprobes model. I'm not interested in events, per se. I don't want to need to know about every single alloc/free of N different varieties integrated from boot onward to build up an image of the state of the system. Instead, I want to take snapshots of the state of the VM.

The main goal here is to be able to answer the question "where's my memory going?". Currently you can't really give a good answer to that question from userspace because of shared mappings, etc.

There are lots of secondary questions that follow on very quickly from that, like "what parts of my shared mappings are or aren't shared, and why?", "what's actually in my application's working set?" and "how much of this crap can I ditch?".

I guess one could generate an answer to the static question with systemtap, by accumulating running counts across the application lifetime and then snapshotting them. Sounds hard though.

Quoted 4 lines · On Fri, 13 Apr 2007 11:42:29 +1000 Nick Piggin <[email protected]> wrote:

Maybe. How about LRU? Reclaim performance is bad, and you want to work out which pages keep going off the end of it, or which pages keep getting written out via it, or who's pages are on the active list, forcing mine out.

I guess we have static analysis versus dynamic. The interfaces which Matt is proposing are suited to answering the question "what is my memory being used for" (static). They're unlikely to be useful for answering the question "what's happening in the VM" (dynamic). Systemtap is probably better for the dynamic analysis.

I guess one could generate an answer to the static question with systemtap, by accumulating running counts across the application lifetime and then snapshotting them. Sounds hard though.

Quoted 4 lines · Nick Piggin

If kprobes is simply crappy and doesn't work properly for this, then I could accept that. I'm not someone trying to get this info. So why can't it be used? (not just for kpagemap, but for clear_refs and all that gunk too).

kprobes is good for looking at events, but bad for looking at state. Especially metric shitloads of state.

Quoted 4 lines

Maybe. How about LRU? Reclaim performance is bad, and you want to work out which pages keep going off the end of it, or which pages keep getting written out via it, or who's pages are on the active list, forcing mine out.

Those are actually probably a good match for systemtap as they're all events.

Quoted 3 lines · Andrew Morton

I guess one could generate an answer to the static question with systemtap, by accumulating running counts across the application lifetime and then snapshotting them. Sounds hard though.

You'd have to do it from boot onward to get a complete system image. One way to look at it is that systemtap can give you the derivative of the information, and you have to integrate it.

Quoted 3 lines

I guess one could generate an answer to the static question with systemtap, by accumulating running counts across the application lifetime and then snapshotting them. Sounds hard though.

Can't you just traverse arbitrary kernel data structures at a given point in time, exactly like the /proc/ call is doing?

Quoted 8 lines · Matt Mackall

Quoted 4 lines · Nick Piggin

If kprobes is simply crappy and doesn't work properly for this, then I could accept that. I'm not someone trying to get this info. So why can't it be used? (not just for kpagemap, but for clear_refs and all that gunk too).

Quoted 2 lines

kprobes is good for looking at events, but bad for looking at state. Especially metric shitloads of state.

Why? Why is a kprobes trap significantly more expensive than a read syscall?

Quoted 8 lines

Quoted 4 lines

Maybe. How about LRU? Reclaim performance is bad, and you want to work out which pages keep going off the end of it, or which pages keep getting written out via it, or who's pages are on the active list, forcing mine out.

Those are actually probably a good match for systemtap as they're all events.

Traverse the LRU? Which files do they belong to? What process maps them?

Quoted 4 lines · On Fri, 13 Apr 2007 12:18:56 +1000 Nick Piggin <[email protected]> wrote:

Quoted 3 lines

I guess one could generate an answer to the static question with systemtap, by accumulating running counts across the application lifetime and then snapshotting them. Sounds hard though.

Quoted 2 lines

Can't you just traverse arbitrary kernel data structures at a given point in time, exactly like the /proc/ call is doing?

Do a full pagetable walk, with all the associated locking from within a systemtap script? I'd be surprised. Maybe if it's mostly hand-coded in C, perhaps.

Then you just end up with the same thing, don't you?

Quoted 13 lines · Nick Piggin

Quoted 10 lines · Matt Mackall

Quoted 4 lines · Nick Piggin

If kprobes is simply crappy and doesn't work properly for this, then I could accept that. I'm not someone trying to get this info. So why can't it be used? (not just for kpagemap, but for clear_refs and all that gunk too).

kprobes is good for looking at events, but bad for looking at state. Especially metric shitloads of state.

Quoted 2 lines

Why? Why is a kprobes trap significantly more expensive than a read syscall?

I guess I'm not clear on what you're proposing. From my understanding of kprobes (admittedly not an expert), this is hard to do and not a very good match.

Quoted 6 lines

Quoted 6 lines

Quoted 4 lines

Maybe. How about LRU? Reclaim performance is bad, and you want to work out which pages keep going off the end of it, or which pages keep getting written out via it, or who's pages are on the active list, forcing mine out.

Quoted 3 lines

Quoted 1 line

Those are actually probably a good match for systemtap as they're all events.

Traverse the LRU? Which files do they belong to? What process maps them?

-ENOPARSE.

Quoted 1 line · Andrew Morton

Then you just end up with the same thing, don't you?

Well _you_ do, because that happens to be exactly what you want. Bill ends up with something that displays page_mapcount instead. And I end up with something that traverses LRU lists rather than pfns. And none of it goes in /proc/ or linux-2.6/.

Quoted 4 lines · On Fri, 13 Apr 2007 12:18:56 +1000 Nick Piggin <[email protected]> wrote:

Quoted 3 lines

I guess one could generate an answer to the static question with systemtap, by accumulating running counts across the application lifetime and then snapshotting them. Sounds hard though.

Quoted 2 lines

Can't you just traverse arbitrary kernel data structures at a given point in time, exactly like the /proc/ call is doing?

Do a full pagetable walk, with all the associated locking from within a systemtap script? I'd be surprised. Maybe if it's mostly hand-coded in C, perhaps. Then you just end up with the same thing, don't you?

Quoted 1 line

Then you just end up with the same thing, don't you?

Well _you_ do, because that happens to be exactly what you want. Bill ends up with something that displays page_mapcount instead. And I end up with something that traverses LRU lists rather than pfns. And none of it goes in /proc/ or linux-2.6/.
So it isn't really the same thing at all.

Quoted 3 lines

Do a full pagetable walk, with all the associated locking from within a systemtap script? I'd be surprised. Maybe if it's mostly hand-coded in C, perhaps. Then you just end up with the same thing, don't you?

And my problem isn't with the hardcoded pagetable walker. Yeah, we'd probably still keep the pagetable callback walker thingy with Matt's associated cleanups (and my subsequent ones to clean it up more and move it to mm/): there are other in-kernel users for that anyway. The point is the proc API, and exposing random little parts of deep kernel internals that some people happen to find useful at the time.

(which is why we have an incredible proliferation of these things). With systemtap scripts, you could walk pagetables and print *the exact page information you want*, or you could walk pfns, or LRU, or page_tree, or walk the page tree then the rmap structures. And you can selectively cull out items you don't care about if you only care about a subset of items, based on arbitrary criteria. And you can most likely do all that more efficiently than with a conglomeration of various /proc files (assuming they even provide what you want in the first place).

Quoted 3 lines · Andrew Morton

Quoted 3 lines

Do a full pagetable walk, with all the associated locking from within a systemtap script? I'd be surprised. Maybe if it's mostly hand-coded in C, perhaps. Then you just end up with the same thing, don't you?

Quoted 6 lines · Nick Piggin

And my problem isn't with the hardcoded pagetable walker. Yeah, we'd probably still keep the pagetable callback walker thingy with Matt's associated cleanups (and my subsequent ones to clean it up more and move it to mm/): there are other in-kernel users for that anyway. The point is the proc API, and exposing random little parts of deep kernel internals that some people happen to find useful at the time.

Quoted 8 lines

(which is why we have an incredible proliferation of these things). With systemtap scripts, you could walk pagetables and print *the exact page information you want*, or you could walk pfns, or LRU, or page_tree, or walk the page tree then the rmap structures. And you can selectively cull out items you don't care about if you only care about a subset of items, based on arbitrary criteria. And you can most likely do all that more efficiently than with a conglomeration of various /proc files (assuming they even provide what you want in the first place).

The EM guys are unwilling or unable for support-oriented reasons to deal with anything but unmodified kernels as shipped by distros.

Quoted 1 line · Andrew Morton

Quoted 1 line

Then you just end up with the same thing, don't you?

Quoted 5 lines · Nick Piggin

Well _you_ do, because that happens to be exactly what you want. Bill ends up with something that displays page_mapcount instead. And I end up with something that traverses LRU lists rather than pfns. And none of it goes in /proc/ or linux-2.6/.
So it isn't really the same thing at all.

The EM guys aren't dealing with the database; they're dealing with some enterprise management thingie that does things like control how many client connections are allowed for each database instance. Unless they're doing less than I expect, and are largely something like procps on steroids and enterprise silliness.

Quoted 2 lines

The EM guys are unwilling or unable for support-oriented reasons to deal with anything but unmodified kernels as shipped by distros.

And I think major distros ship with kprobes enabled, so that is yet another reason why systemtap should be considered before adding these proc interfaces.

Ah, OK. Anyway, with kprobes/systemtap they can do whatever they like and none of us need to care in the slightest ;)

Quoted 2 lines · Nick Piggin

Ah, OK. Anyway, with kprobes/systemtap they can do whatever they like and none of us need to care in the slightest ;)

Umm, folks. systemtap basically means people compile kernel modules from an odd scripting language with embedded C snipplets into kernel modules. The kernel modules don't use normal exported APIs but use kallsysms and dwarf info to poke into every possible private bit.

Saying you don't care the slightest whether oracle will load huge amounts of code into the kernel that depends on intimate implementation details, and that you don't even have source to to debug it is not what I'd call "none of us need to care in the slightest", at least for those of you working for distributions that may actually have to debug this shit in the end.

While we're at it, what happened to the idea of tainting the kernel as soon as krpobes are placed in the kernel to at least make people aware of it?

Yeah good point ;) I just meant the wider "we".

With all the problems kprobes has, something like poking deep into kernel internals seems like a good thing to use it for instead of hardcoding that stuff into the kernel. If not, then why did we even merge it in the first place?

We could distribute some systemtap scripts, and even distribute some basic useful ones like this sort of page info in the kernel source tree.

Umm, folks. systemtap basically means people compile kernel modules from an odd scripting language with embedded C snipplets into kernel modules. The kernel modules don't use normal exported APIs but use kallsysms and dwarf info to poke into every possible private bit. Saying you don't care the slightest whether oracle will load huge

amounts of code into the kernel that depends on intimate implementation details, and that you don't even have source to to debug it is not what I'd call "none of us need to care in the slightest", at least for those of you working for distributions that may actually have to debug this shit in the end.
While we're at it, what happened to the idea of tainting the kernel as soon as krpobes are placed in the kernel to at least make people aware of it?

Quoted 6 lines · Nick Piggin

Yeah good point ;) I just meant the wider "we".

With all the problems kprobes has, something like poking deep into kernel internals seems like a good thing to use it for instead of hardcoding that stuff into the kernel. If not, then why did we even merge it in the first place?

It's very nice to poke deep into the kernel for development purposes. For example for the spu scheduler work I'm doing currently I have a module using kprobes (note the systemtap crap because it's big, bloated, in and odd language, and does a lot of really wrong things in it's runtime).

This module allows me to put probes into various places in the scheduler and writes them into a ringbuffer with timestampts allowing me to trace what's going on there. This is really neat. Unfortunately it breaks as soon as I do some major reshuffling because then the points it hooks up to are not there anymore. That's perfectly fine for my setup, because _I_ know what I have to change when it breaks, and can easily fix that. Now imagine a similar module to trace pagecache activity used by a third-party monitoring tool. We now get a major change to the pagecache (say to make it lockless), and the probes just break. In the est case it just doesn't work anymore, in the worst case it crashes the kernel. Now if the app vendor at least gave me their source I could at least fix it to not crash, but there's a fair enough chance they poke into bits that simply aren't there anymore.

Now if we have a proper user interface with real code behing it we can have a defined interface. If the interface is bad enough (or just too lowlevel) we might have the last problem of stastistic that were there once to go away aswell, but we can deal with that gracefully by declaring parts of the stats volatile and make sure people don't mess with them.

To summarize, I really love kprobes to ease my debugging work, but using it for any kind of production code is a total nightmare.

Quoted 3 lines

We could distribute some systemtap scripts, and even distribute some basic useful ones like this sort of page info in the kernel source tree.

We could not really distribute systemtap scripts with the kernel. systemtap is a bloody complicated piece of shit outside the kernel tree that breaks all the time we change kernel internals. We could provide useful kprobes modules, a proper tracing system (ltt-ng-lite) and surrounding infrastucture.

But at least make it into its own module with a debugfs interface or something. I mean, exposing a PG_name-to-nr and page count pfn and flags as a supposedly formal proc interface doesn't sound nice to me. Page flags does not tell you what is going on in the VM, it gives you a tiny window into "something". Between reading a /proc/pid/ pfn and finding the pfn's page flags, it could be used for something completely different anyway.

Quoted 9 lines · Andrew Morton

Quoted 5 lines · On Fri, 13 Apr 2007 12:18:56 +1000 Nick Piggin <[email protected]> wrote:

Quoted 4 lines

I guess one could generate an answer to the static question with systemtap,
by accumulating running counts across the application lifetime and then snapshotting them. Sounds hard though.

Quoted 7 lines

Quoted 2 lines

Can't you just traverse arbitrary kernel data structures at a given point in time, exactly like the /proc/ call is doing?

Do a full pagetable walk, with all the associated locking from within a systemtap script? I'd be surprised. Maybe if it's mostly hand-coded in C, perhaps.

It looks like you can traverse arbitrary data structures, yes.

It definitely seems like you can use some kernel functions, but the ones I saw may just be systemtap facilities. But what is so surprising about being able to call a kernel function when running in kernel context? Perhaps there is some fundamental limitation of kprobes that I don't understand.

Quoted 19 lines · Matt Mackall

Quoted 15 lines · Nick Piggin

Quoted 10 lines · Matt Mackall

Quoted 4 lines · Nick Piggin

If kprobes is simply crappy and doesn't work properly for this, then I could accept that. I'm not someone trying to get this info. So why can't it be used? (not just for kpagemap, but for clear_refs and all that gunk too).

kprobes is good for looking at events, but bad for looking at state. Especially metric shitloads of state.

Why? Why is a kprobes trap significantly more expensive than a read syscall?

Quoted 3 lines

I guess I'm not clear on what you're proposing. From my understanding of kprobes (admittedly not an expert), this is hard to do and not a very good match.

But you have an idea that it is bad for exposing lots of data. Why? (I'm not a kprobes expert either, these are not rhetorical questions)

Quoted 5 lines · Nick Piggin

It definitely seems like you can use some kernel functions, but the ones I saw may just be systemtap facilities. But what is so surprising about being able to call a kernel function when running in kernel context? Perhaps there is some fundamental limitation of kprobes that I don't understand.

The main requirement for kprobes handlers is that they can't sleep. You could definitely call a kernel function from kprobe handlers as long as the function doesn't sleep.

With systemtap scripts, you could walk pagetables and print *the exact page information you want*, or you could walk pfns, or LRU, or page_tree, or walk the page tree then the rmap structures. And you can selectively cull out items you don't care about if you only care about a subset of items, based on arbitrary criteria. And you can most likely do all that more efficiently than with a conglomeration of various /proc files (assuming they even provide what you want in the first place).

Quoted 7 lines · On Fri, 13 Apr 2007 10:15:24 +1000 Nick Piggin <[email protected]> wrote:

Quoted 4 lines

Quoted 1 line

+ ((char *)page)[1] = PAGE_SHIFT;

OK.

Shouldn't we just expose page size and endianness by other means? (another file or syscall).

I don't think so - this file exposes fairly deep kernel internals and that's unavoidable, really - it's *supposed* to do that. It is explicitly designed for monitoring kernel behaviour.

So it needs special handling by userspace. Keeping the number of files which need such special handling to a minimum will keep the number of applications which are exposed to kernel changes to a minimum.

Quoted 16 lines

Quoted 15 lines

Review

Quoted 10 lines

(quoted diff) +10 -0

+               for (; i < 2 * chunk / KPMSIZE; i += 2, pfn++) {
+                       ppage = pfn_to_page(pfn);
+                       if (!ppage) {
+                               page[i] = 0;
+                               page[i + 1] = 0;
+                       } else {
+                               page[i] = ppage->flags;
+                               page[i + 1] = atomic_read(&ppage->_count);
+                       }
+               }

Not a good idea to expose raw flags in this manner - it changes at the drop of a hat. We'd need to also expose the kernel's PG_foo-to-bitnumber mapping to make this viable.

Quoted 2 lines

I don't think it is viable because that makes the flags part of the userspace ABI.

It *will* be viable. If the application wants to know if a page is dirty, it looks up "PG_dirty" in /proc/pg_foo-to-bitnumber and uses PG_dirty's numerical offset when inspecting fields in /proc/kpagemap. If correctly designed, such a monitoring application will be able to report upon page flags which we haven't even thought up yet.

Quoted 1 line

I wonder what they are needed for.

Poking deeply into the kernel to provide information about kernel state.

There are real-world needs for this, and the people who develop tools to process this information will have decent kernel understanding and will know that the file's contents may alter across kernel versions. It sure beats poking around in /dev/kmem.

I doubt if there's a sensible way in which we can prettify this interface without losing information. But we should aim to make it as robust as possible agaisnt future kenrel changes, of course.

And we should satisfy ourselves that all the required information has been made available. The fact that it will satisfy the Oracle requirement is encouraging.

Matt, these changes make the new field in /proc/pid/smaps redundant, don't they?

Quoted 5 lines

It *will* be viable. If the application wants to know if a page is dirty, it looks up "PG_dirty" in /proc/pg_foo-to-bitnumber and uses PG_dirty's numerical offset when inspecting fields in /proc/kpagemap. If correctly designed, such a monitoring application will be able to report upon page flags which we haven't even thought up yet.

We can probably fit this in the existing (variable-sized) header.

Quoted 13 lines

Quoted 1 line

I wonder what they are needed for.

Poking deeply into the kernel to provide information about kernel state.

There are real-world needs for this, and the people who develop tools to process this information will have decent kernel understanding and will know that the file's contents may alter across kernel versions. It sure beats poking around in /dev/kmem.

I doubt if there's a sensible way in which we can prettify this interface without losing information. But we should aim to make it as robust as possible agaisnt future kenrel changes, of course.

Quoted 6 lines

And we should satisfy ourselves that all the required information has been made available. The fact that it will satisfy the Oracle requirement is encouraging.

Matt, these changes make the new field in /proc/pid/smaps redundant, don't they?

Which new field?

From /proc/kpagemap + /proc/*/pagemap, you can
basically synthesize any statistic you want, including all the existing ones. For some data, /proc/pid/smaps (or /proc/meminfo) will be considerably more efficient.

But in general, most of the statistics in smaps are basically useless for shared mappings, just like RSS. Problem is, we really don't know what statistics we want yet, or even if it can be distilled down to simple numbers anyway.

Can't you just traverse arbitrary kernel data structures at a given point in time, exactly like the /proc/ call is doing?

Quoted 7 lines · On Fri, 13 Apr 2007 11:24:36 -0500 Matt Mackall <[email protected]> wrote:

Quoted 5 lines

It *will* be viable. If the application wants to know if a page is dirty, it looks up "PG_dirty" in /proc/pg_foo-to-bitnumber and uses PG_dirty's numerical offset when inspecting fields in /proc/kpagemap. If correctly designed, such a monitoring application will be able to report upon page flags which we haven't even thought up yet.

We can probably fit this in the existing (variable-sized) header.

hm, OK..

Quoted 13 lines

Quoted 13 lines

Quoted 1 line

I wonder what they are needed for.

Poking deeply into the kernel to provide information about kernel state.

There are real-world needs for this, and the people who develop tools to process this information will have decent kernel understanding and will know that the file's contents may alter across kernel versions. It sure beats poking around in /dev/kmem.

I doubt if there's a sensible way in which we can prettify this interface without losing information. But we should aim to make it as robust as possible agaisnt future kenrel changes, of course.

Quoted 8 lines

Quoted 6 lines

And we should satisfy ourselves that all the required information has been made available. The fact that it will satisfy the Oracle requirement is encouraging.

Matt, these changes make the new field in /proc/pid/smaps redundant, don't they?

Which new field?

Referenced:

Quoted 4 lines

From /proc/kpagemap + /proc/*/pagemap, you can
basically synthesize any statistic you want, including all the existing ones. For some data, /proc/pid/smaps (or /proc/meminfo) will be considerably more efficient.

You'd need to poke clear_refs beforehand to make the referenced bits useful.

Actually, we also need to run around the ptes and collect the pte-referenced bits too. I don't think your code copes with any of that?

Quoted 4 lines

But in general, most of the statistics in smaps are basically useless for shared mappings, just like RSS. Problem is, we really don't know what statistics we want yet, or even if it can be distilled down to simple numbers anyway.

yup. But that's the whole point, really: don't prejudge what info userspace is trying to collect.

Quoted 5 lines

Quoted 4 lines

From /proc/kpagemap + /proc/*/pagemap, you can
basically synthesize any statistic you want, including all the existing ones. For some data, /proc/pid/smaps (or /proc/meminfo) will be considerably more efficient.

Quoted 4 lines

You'd need to poke clear_refs beforehand to make the referenced bits useful.

Actually, we also need to run around the ptes and collect the pte-referenced bits too. I don't think your code copes with any of that?

No, and it probably should. Perhaps dirty as well, though I've kindof lost the plot on how that works lately.

[...]

Quoted 1 line

merge it in the first place?

It's very nice to poke deep into the kernel for development purposes. For example for the spu scheduler work I'm doing currently I have a module using kprobes (note the systemtap crap because it's big, bloated, in and odd language, and does a lot of really wrong things in its runtime).

This module allows me to put probes into various places in the scheduler and writes them into a ringbuffer with timestampts allowing me to trace what's going on there. This is really neat. [...]

[...] To summarize, I really love kprobes to ease my debugging work, but using it for any kind of production code is a total nightmare.

Quoted 3 lines

We could distribute some systemtap scripts, and even distribute some basic useful ones like this sort of page info in the kernel source tree.

We could not really distribute systemtap scripts with the kernel. systemtap is a bloody complicated piece of [software]

outside the kernel tree that breaks all the time we change kernel internals. [...]

Quoted 9 lines · Nick Piggin

Quoted 8 lines · Matt Mackall

Quoted 4 lines · Nick Piggin

If kprobes is simply crappy and doesn't work properly for this, then I could accept that. I'm not someone trying to get this info. So why can't it be used? (not just for kpagemap, but for clear_refs and all that gunk too).

Quoted 5 lines

Quoted 2 lines

kprobes is good for looking at events, but bad for looking at state. Especially metric shitloads of state.

Why? Why is a kprobes trap significantly more expensive than a read syscall?

I guess I'm not clear on what you're proposing. From my understanding of kprobes (admittedly not an expert), this is hard to do and not a very good match.

Quoted 6 lines

Quoted 6 lines

Quoted 4 lines

Maybe. How about LRU? Reclaim performance is bad, and you want to work out which pages keep going off the end of it, or which pages keep getting written out via it, or who's pages are on the active list, forcing mine out.

Quoted 4 lines

Quoted 2 lines

Those are actually probably a good match for systemtap as they're all events.

Traverse the LRU? Which files do they belong to? What process maps them?

-ENOPARSE.

It may be worthwhile to remind people that it is easy to use systemtap only to the extent of automating the placement of kprobes: just to perform the function-name/source-file/line-number triplet to PC mapping. They can use embedded-C code to do all the same stuff they'd do with kprobes. They are not obligated to write any odd script code for probing logic, nor indeed use any of this really wrong runtime.

Quoted 2 lines

We could not really distribute systemtap scripts with the kernel. systemtap is a bloody complicated piece of [software]

I don't know if that should be treated a compliment to our team, for being able to work quickly on something that a full-grown kernel developer finds bloody complicated. Perhaps your information is simply outdated. Big & bloated? We have several times asked for specifics rather than smears - what about it?

Quoted 2 lines

outside the kernel tree that breaks all the time we change kernel internals. [...]

That's begging the question. If kernel folks are willing to maintain some included systemtap-related code, then by definition it would not break all the time.

Quoted 7 lines · Nick Piggin

With systemtap scripts, you could walk pagetables and print *the exact page information you want*, or you could walk pfns, or LRU, or page_tree, or walk the page tree then the rmap structures. And you can selectively cull out items you don't care about if you only care about a subset of items, based on arbitrary criteria. And you can most likely do all that more efficiently than with a conglomeration of various /proc files (assuming they even provide what you want in the first place).

Yes, but maintaining the systemtap scripts will be a nightmare, since they would be outside the kernel, and as we change our internal data structure, the scripts would become useless.

This is a fundamental problem with systemtap that we haven't been able to solve yet, because solving it would freeze various internal data structures or kernel functions. I agree that's not acceptable; which is why I don't think systemtap would be a good match for the problem we're trying to solve here.

and does a lot of really wrong things in it's runtime).

Quoted 2 lines · Christoph Hellwig

and [systemtap] does a lot of really wrong things in it's runtime). [...]

I must agree with that. Perhaps it would be good if its runtime code was posted to l-k at some point and reviewed in the standard way even when it isn't merged.