Skip to contentSkip to conversation

perf: Translating mmap2 ids into socket info?

DISCUSSION Don Zickus 4 participants 12 messages Oct 22, 2014 – Oct 23, 2014

lore
Share & tools
lore

Don Zickus

author
#0 lore raw

Hi,

A question/request came up during our cache to cache analysis. We were wondering if give an unique mmap2 id (major, minor, inode, inode generation), if it was possible to determine a cpu socket id that memory was attached to at the time of the captured perf event?

We ran into a scenario awhile back where a dcache struct was allocated on say node2 and then subsequently free'd. The memory was thought to be re-allocated on node0 for another dcache entry. It turned out the memory was still attached to node2 and was causing slowdowns.

Our cache-to-cache tool noticed the slowdown but we couldn't understand why because we had falsely assumed the memory was allocated on the local node but instead it was on the remote node.

I wasn't involved too much in this problem so the details provided above are a little hazy.

Anyway, an idea popped up that having socket information might have helped us solve/notice this problem sooner.

Thoughts? Not sure if that data is possible.

Cheers,
Don

#1 lore raw
Quoted 6 lines · Don Zickus

Hi,

A question/request came up during our cache to cache analysis. We were wondering if give an unique mmap2 id (major, minor, inode, inode generation), if it was possible to determine a cpu socket id that memory was attached to at the time of the captured perf event?

No, see below. Also socket is the wrong information, both AMD and Intel have chips with two nodes in one socket :-)

Quoted 4 lines

We ran into a scenario awhile back where a dcache struct was allocated on say node2 and then subsequently free'd. The memory was thought to be re-allocated on node0 for another dcache entry. It turned out the memory was still attached to node2 and was causing slowdowns.

Yes, kernel memory is directly addresses, you basically have a static address->node mapping, it never changes.

For instance, on my ivb-ep I can find the following in my dmesg:

[    0.000000] NUMA: Initialized distance table, cnt=2
[    0.000000] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x43fffffff] -> [mem 0x00000000-0x43fffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x43fffc000-0x43fffffff]
[    0.000000] NODE_DATA(1) allocated [mem 0x83fff9000-0x83fffcfff]
[    0.000000]  [ffffea0000000000-ffffea000edfffff] PMD -> [ffff88042fe00000-ffff88043ddfffff] on node 0
[    0.000000]  [ffffea000ee00000-ffffea001cdfffff] PMD -> [ffff88082f600000-ffff88083d5fffff] on node 1
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   [mem 0x100000000-0x83fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0008dfff]
[    0.000000]   node   0: [mem 0x00100000-0xbad28fff]
[    0.000000]   node   0: [mem 0xbaf90000-0xbafc4fff]
[    0.000000]   node   0: [mem 0xbafda000-0xbb3d3fff]
[    0.000000]   node   0: [mem 0xbdfac000-0xbdffffff]
[    0.000000]   node   0: [mem 0x100000000-0x43fffffff]
[    0.000000]   node   1: [mem 0x440000000-0x83fffffff]
[    0.000000] Initmem setup node 0 [mem 0x00001000-0x43fffffff]
[    0.000000] On node 0 totalpages: 4174137
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3981 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 10422 pages used for memmap
[    0.000000]   DMA32 zone: 762284 pages, LIFO batch:31
[    0.000000]   Normal zone: 46592 pages used for memmap
[    0.000000]   Normal zone: 3407872 pages, LIFO batch:31
[    0.000000] Initmem setup node 1 [mem 0x440000000-0x83fffffff]
[    0.000000] On node 1 totalpages: 4194304
[    0.000000]   Normal zone: 57344 pages used for memmap
[    0.000000]   Normal zone: 4194304 pages, LIFO batch:31
Quoted 3 lines

Our cache-to-cache tool noticed the slowdown but we couldn't understand why because we had falsely assumed the memory was allocated on the local node but instead it was on the remote node.

But in general, you can never say for user memory, since that has the process page table mapping in between, the user virtual address is unrelated to backing (and can change frequently and without notification).

Therefore the mmap(2) information is useless for this, it only concerns user memory.

#2 lore raw

Em Wed, Oct 22, 2014 at 06:45:10PM +0200, Peter Zijlstra escreveu:

Quoted 5 lines · Don Zickus
Quoted 3 lines

Our cache-to-cache tool noticed the slowdown but we couldn't understand why because we had falsely assumed the memory was allocated on the local node but instead it was on the remote node.

Quoted 7 lines

But in general, you can never say for user memory, since that has the process page table mapping in between, the user virtual address is unrelated to backing (and can change frequently and without notification).

Therefore the mmap(2) information is useless for this, it only concerns user memory.

So what you are saying is that it is difficult to have some sort of mechanism that an mmap moved from one node to another, when that happens, i.e. a new tracepoint for that?

- Arnaldo

#3 lore raw

Em Wed, Oct 22, 2014 at 02:42:26PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 11 lines

Em Wed, Oct 22, 2014 at 06:45:10PM +0200, Peter Zijlstra escreveu:

Quoted 9 lines · Don Zickus
Quoted 3 lines

Our cache-to-cache tool noticed the slowdown but we couldn't understand why because we had falsely assumed the memory was allocated on the local node but instead it was on the remote node.

But in general, you can never say for user memory, since that has the process page table mapping in between, the user virtual address is unrelated to backing (and can change frequently and without notification).

Quoted 6 lines
Quoted 2 lines

Therefore the mmap(2) information is useless for this, it only concerns user memory.

So what you are saying is that it is difficult to have some sort of mechanism that an mmap moved from one node to another, when that happens, i.e. a new tracepoint for that?

Humm, so, after reading a bit more, I see that we can figure out which ranges are in which NUMA nodes (like shown in dmesg), and we have the phys addr, so is this just a matter of perf tooling complement the information the kernel provides on mmap2 (and the hardware provides on those events)?

- Arnaldo

#4 lore raw
Quoted 19 lines · Arnaldo Carvalho de Melo

Em Wed, Oct 22, 2014 at 02:42:26PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 11 lines

Em Wed, Oct 22, 2014 at 06:45:10PM +0200, Peter Zijlstra escreveu:

Quoted 9 lines · Don Zickus
Quoted 3 lines

Our cache-to-cache tool noticed the slowdown but we couldn't understand why because we had falsely assumed the memory was allocated on the local node but instead it was on the remote node.

But in general, you can never say for user memory, since that has the process page table mapping in between, the user virtual address is unrelated to backing (and can change frequently and without notification).

Quoted 6 lines
Quoted 2 lines

Therefore the mmap(2) information is useless for this, it only concerns user memory.

So what you are saying is that it is difficult to have some sort of mechanism that an mmap moved from one node to another, when that happens, i.e. a new tracepoint for that?

Quoted 5 lines

Humm, so, after reading a bit more, I see that we can figure out which ranges are in which NUMA nodes (like shown in dmesg), and we have the phys addr, so is this just a matter of perf tooling complement the information the kernel provides on mmap2 (and the hardware provides on those events)?

We explicitly do not expose the phys address of user space pages. So nope.

#5 lore raw
In reply to Arnaldo Carvalho de Melo
Quoted 11 lines · Arnaldo Carvalho de Melo

Em Wed, Oct 22, 2014 at 06:45:10PM +0200, Peter Zijlstra escreveu:

Quoted 9 lines · Don Zickus
Quoted 3 lines

Our cache-to-cache tool noticed the slowdown but we couldn't understand why because we had falsely assumed the memory was allocated on the local node but instead it was on the remote node.

But in general, you can never say for user memory, since that has the process page table mapping in between, the user virtual address is unrelated to backing (and can change frequently and without notification).

Quoted 6 lines
Quoted 2 lines

Therefore the mmap(2) information is useless for this, it only concerns user memory.

So what you are saying is that it is difficult to have some sort of mechanism that an mmap moved from one node to another, when that happens, i.e. a new tracepoint for that?

Node is per page, not per mapping. Every single page in a mmap can have a different node (assuming enough nodes etc..), and it can change at relatively high frequency.

#6 lore raw
In reply to Peter Zijlstra
Quoted 11 lines · Peter Zijlstra
Quoted 6 lines · Don Zickus

Hi,

A question/request came up during our cache to cache analysis. We were wondering if give an unique mmap2 id (major, minor, inode, inode generation), if it was possible to determine a cpu socket id that memory was attached to at the time of the captured perf event?

No, see below. Also socket is the wrong information, both AMD and Intel have chips with two nodes in one socket :-)

Quoted 7 lines
Quoted 4 lines

We ran into a scenario awhile back where a dcache struct was allocated on say node2 and then subsequently free'd. The memory was thought to be re-allocated on node0 for another dcache entry. It turned out the memory was still attached to node2 and was causing slowdowns.

Yes, kernel memory is directly addresses, you basically have a static address->node mapping, it never changes.

For kernel addresses, is there a reason not to have it available in perf, especially when that knowledge is important to understanding a numa-related slowdown?

In our case, when we booted with one configuration, AIM ran fine. When we booted another way, AIM's performance dropped 50%. It was all due to the dentry lock being located on a different (now remote) numa node.

We used your dmesg approach to track down the home node in an attempt to understand what was different between the two boots. But the problem would have been obvious if perf simply listed the home node info.

Quoted 48 lines

For instance, on my ivb-ep I can find the following in my dmesg:

[    0.000000] NUMA: Initialized distance table, cnt=2
[    0.000000] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x43fffffff] -> [mem 0x00000000-0x43fffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x43fffc000-0x43fffffff]
[    0.000000] NODE_DATA(1) allocated [mem 0x83fff9000-0x83fffcfff]
[    0.000000]  [ffffea0000000000-ffffea000edfffff] PMD -> [ffff88042fe00000-ffff88043ddfffff] on node 0
[    0.000000]  [ffffea000ee00000-ffffea001cdfffff] PMD -> [ffff88082f600000-ffff88083d5fffff] on node 1
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   [mem 0x100000000-0x83fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0008dfff]
[    0.000000]   node   0: [mem 0x00100000-0xbad28fff]
[    0.000000]   node   0: [mem 0xbaf90000-0xbafc4fff]
[    0.000000]   node   0: [mem 0xbafda000-0xbb3d3fff]
[    0.000000]   node   0: [mem 0xbdfac000-0xbdffffff]
[    0.000000]   node   0: [mem 0x100000000-0x43fffffff]
[    0.000000]   node   1: [mem 0x440000000-0x83fffffff]
[    0.000000] Initmem setup node 0 [mem 0x00001000-0x43fffffff]
[    0.000000] On node 0 totalpages: 4174137
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3981 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 10422 pages used for memmap
[    0.000000]   DMA32 zone: 762284 pages, LIFO batch:31
[    0.000000]   Normal zone: 46592 pages used for memmap
[    0.000000]   Normal zone: 3407872 pages, LIFO batch:31
[    0.000000] Initmem setup node 1 [mem 0x440000000-0x83fffffff]
[    0.000000] On node 1 totalpages: 4194304
[    0.000000]   Normal zone: 57344 pages used for memmap
[    0.000000]   Normal zone: 4194304 pages, LIFO batch:31
Quoted 3 lines

Our cache-to-cache tool noticed the slowdown but we couldn't understand why because we had falsely assumed the memory was allocated on the local node but instead it was on the remote node.

But in general, you can never say for user memory, since that has the process page table mapping in between, the user virtual address is unrelated to backing (and can change frequently and without notification).

Therefore the mmap(2) information is useless for this, it only concerns user memory.

#7 lore raw
Quoted 5 lines · Joe Mario
Quoted 2 lines

Yes, kernel memory is directly addresses, you basically have a static address->node mapping, it never changes.

For kernel addresses, is there a reason not to have it available in perf, especially when that knowledge is important to understanding a numa-related slowdown?

Dunno why that isn't exposed in sysfs.

Quoted 7 lines

In our case, when we booted with one configuration, AIM ran fine. When we booted another way, AIM's performance dropped 50%. It was all due to the dentry lock being located on a different (now remote) numa node.

We used your dmesg approach to track down the home node in an attempt to understand what was different between the two boots. But the problem would have been obvious if perf simply listed the home node info.

Or if you'd used more counters that track the node interconnect traffic ;-) There are a few simple ones that count local/remote type things (offcore), but using the uncore counters you can track way more.

author
#8 lore raw
Quoted 13 lines · Peter Zijlstra
Quoted 5 lines · Joe Mario
Quoted 2 lines

Yes, kernel memory is directly addresses, you basically have a static address->node mapping, it never changes.

For kernel addresses, is there a reason not to have it available in perf, especially when that knowledge is important to understanding a numa-related slowdown?

Dunno why that isn't exposed in sysfs.

Quoted 4 lines

In our case, when we booted with one configuration, AIM ran fine. When we booted another way, AIM's performance dropped 50%. It was all due to the dentry lock being located on a different (now remote) numa node.

Quoted 7 lines
Quoted 3 lines

We used your dmesg approach to track down the home node in an attempt to understand what was different between the two boots. But the problem would have been obvious if perf simply listed the home node info.

Or if you'd used more counters that track the node interconnect traffic ;-) There are a few simple ones that count local/remote type things (offcore), but using the uncore counters you can track way more.

Ha! I have been telling myself for a year I would try to learn more about those offcore/uncore counters. Is there documentation for how to access the uncore stuff? Do I have to long hand it with 'perf record -e uncore_qpi_1/<stuff>/ foo'?

Cheers,
Don

#9 lore raw
Quoted 10 lines · Don Zickus
Quoted 3 lines · Peter Zijlstra

Or if you'd used more counters that track the node interconnect traffic ;-) There are a few simple ones that count local/remote type things (offcore), but using the uncore counters you can track way more.

Ha! I have been telling myself for a year I would try to learn more about those offcore/uncore counters. Is there documentation for how to access the uncore stuff? Do I have to long hand it with 'perf record -e uncore_qpi_1/<stuff>/ foo'?

So uncore is something unholy complicated and different for each cpu, although the newer chips seems to have limited the variation somewhat.

I would suggest you start with the offcore bits, those are in the SDM.

We have some numa cache events, but if those don't work for you you might want to look at the offcore code we have in the kernel to help construct your own events.

I Cc'ed you on a patch series cleaning up some of the recent HSW code.

#10 lore raw
In reply to Don Zickus
Quoted 4 lines · Don Zickus

Ha! I have been telling myself for a year I would try to learn more about those offcore/uncore counters. Is there documentation for how to access the uncore stuff? Do I have to long hand it with 'perf record -e uncore_qpi_1/<stuff>/ foo'?

For uncore there is some in the SDM and there are these documents; there might be more but who knows.

http://www.intel.com/Assets/PDF/designguide/323535.pdf
http://www.intel.com/content/dam/www/public/us/en/documents/design-guides/xeon-e5-2600-uncore-guide.pdf

author
#11 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Don Zickus

Ha! I have been telling myself for a year I would try to learn more about those offcore/uncore counters. Is there documentation for how to access the uncore stuff? Do I have to long hand it with 'perf record -e uncore_qpi_1/<stuff>/ foo'?

For uncore there is some in the SDM and there are these documents; there might be more but who knows.

http://www.intel.com/Assets/PDF/designguide/323535.pdf
http://www.intel.com/content/dam/www/public/us/en/documents/design-guides/xeon-e5-2600-uncore-guide.pdf

Thanks!

Cheers,
Don