Skip to contentSkip to conversation

Purpose of numa_node?

DISCUSSION Chris Snook 5 participants 11 messages Jan 31, 2008 – Feb 20, 2008

lore
Share & tools
lore
#0 lore raw

While pondering ways to optimize I/O and swapping on large NUMA machines, I noticed that the numa_node field in struct device isn't actually used anywhere.
We just have a couple dozen lines of code to conditionally create a sysfs file
that will always return -1. Is anyone even working on code to actually use this field? I think it's a good piece of information to keep track of, so I'm not suggesting we remove it, but I want to make sure I'm not stepping on toes or duplicating effort if I try to make it useful.

-- Chris

#1 lore raw
Quoted 9 lines · Chris Snook

While pondering ways to optimize I/O and swapping on large NUMA machines, I noticed that the numa_node field in struct device isn't actually used anywhere. We just have a couple dozen lines of code to conditionally
create a sysfs file that will always return -1. Is anyone even working on
code to actually use this field? I think it's a good piece of information to keep track of, so I'm not suggesting we remove it, but I want to make sure I'm not stepping on toes or duplicating effort if I try to make it useful.

It's manipulated with accessors. If you look at the users of dev_to_node()/set_dev_node() you can see where it's being used. It's primarily used in allocation paths for node locality, and the existing set_dev_node() callsites are places where node locality information already exists (ie, which node a given controller sits on). You can see this in places like PCI (pcibus_to_node()) and USB, with node allocation hints used in places like the dmapool and skb alloc paths.

The in-kernel use looks perfectly sane in that regard, though I'm not sure what the point of exporting this as a RO attribute to userspace is. Presumably someone has a tool somewhere that cares about this.

#2 lore raw
Quoted 4 lines · Paul Mundt

The in-kernel use looks perfectly sane in that regard, though I'm not sure what the point of exporting this as a RO attribute to userspace is. Presumably someone has a tool somewhere that cares about this.

The idea was to allow e.g. NUMA aware irqbalanced that directs the interrupts on the same node as the device is connected to. Don't know if it was ever actually implemented.

-Andi

#3 lore raw
In reply to Paul Mundt
Quoted 14 lines · Paul Mundt
Quoted 4 lines · Chris Snook

While pondering ways to optimize I/O and swapping on large NUMA machines, I noticed that the numa_node field in struct device isn't actually used anywhere. We just have a couple dozen lines of code to conditionally
create a sysfs file that will always return -1. Is anyone even working on
code to actually use this field? I think it's a good piece of information to keep track of, so I'm not suggesting we remove it, but I want to make sure I'm not stepping on toes or duplicating effort if I try to make it useful.

It's manipulated with accessors. If you look at the users of dev_to_node()/set_dev_node() you can see where it's being used. It's primarily used in allocation paths for node locality, and the existing set_dev_node() callsites are places where node locality information already exists (ie, which node a given controller sits on). You can see this in places like PCI (pcibus_to_node()) and USB, with node allocation hints used in places like the dmapool and skb alloc paths.

Quoted 4 lines

The in-kernel use looks perfectly sane in that regard, though I'm not sure what the point of exporting this as a RO attribute to userspace is. Presumably someone has a tool somewhere that cares about this.

I added the numa_node sysfs attribute in the beginning to make it easier to bind processes near some devices. So yes I have some user-space tool using it. It is much easier to use than the local_cpus field on large machines, especially when you use the libnuma interface to bind things, since you don't have to translate numa_node from/to cpumasks.

It works fine on regular machines such as dual opterons. However, I noticed recently that it was wrong on some quad-opteron machines (see http://marc.info/?l=linux-pci&m=119072400008538&w=2) because something is not initialized in the right order. But I haven't tested 2.6.24 on this hardware yet, and I don't know if things have changed regarding this.

Brice

#4 lore raw
Quoted 31 lines · Brice Goglin
Quoted 19 lines · Paul Mundt
Quoted 9 lines · Chris Snook

While pondering ways to optimize I/O and swapping on large NUMA machines, I noticed that the numa_node field in struct device isn't actually used anywhere. We just have a couple dozen lines of code to conditionally
create a sysfs file that will always return -1. Is anyone even working on
code to actually use this field? I think it's a good piece of information to keep track of, so I'm not suggesting we remove it, but I want to make sure I'm not stepping on toes or duplicating effort if I try to make it useful.

It's manipulated with accessors. If you look at the users of dev_to_node()/set_dev_node() you can see where it's being used. It's primarily used in allocation paths for node locality, and the existing set_dev_node() callsites are places where node locality information already exists (ie, which node a given controller sits on). You can see this in places like PCI (pcibus_to_node()) and USB, with node allocation hints used in places like the dmapool and skb alloc paths.

Quoted 4 lines

The in-kernel use looks perfectly sane in that regard, though I'm not sure what the point of exporting this as a RO attribute to userspace is. Presumably someone has a tool somewhere that cares about this.

I added the numa_node sysfs attribute in the beginning to make it easier to bind processes near some devices. So yes I have some user-space tool using it. It is much easier to use than the local_cpus field on large machines, especially when you use the libnuma interface to bind things, since you don't have to translate numa_node from/to cpumasks.

Quoted 5 lines

It works fine on regular machines such as dual opterons. However, I noticed recently that it was wrong on some quad-opteron machines (see
http://marc.info/?l=linux-pci&m=119072400008538&w=2) because something is not initialized in the right order. But I haven't tested 2.6.24 on this hardware yet, and I don't know if things have changed regarding this.

that will depend if you dsdt have _PXM for your pci root bus. otherwise you will get all -1

I have a patchset locally that it call bus_numa, can get that from pci conf space for AMD64 based machine.
so you can use that for AMD64 system without _PXM for pci root bus or even with acpi=off.

let me know if you want test it.

YH

#5 lore raw
Quoted 9 lines · Yinghai Lu
Quoted 6 lines · Brice Goglin

It works fine on regular machines such as dual opterons. However, I noticed recently that it was wrong on some quad-opteron machines (see
http://marc.info/?l=linux-pci&m=119072400008538&w=2) because something is not initialized in the right order. But I haven't tested 2.6.24 on this hardware yet, and I don't know if things have changed regarding this.

Quoted 3 lines

that will depend if you dsdt have _PXM for your pci root bus. otherwise you will get all -1

Have a look at the above link. I don't get -1. I get 0 everywhere, while I should get 1 for some devices. And if I unplug/replug a device using fakephp, numa_node becomes correct (1 instead of 0). This just looks like the code is there but things are initialized in the wrong order.

Brice

#6 lore raw
Quoted 10 lines · Brice Goglin
Quoted 9 lines · Yinghai Lu
Quoted 6 lines · Brice Goglin

It works fine on regular machines such as dual opterons. However, I noticed recently that it was wrong on some quad-opteron machines (see
http://marc.info/?l=linux-pci&m=119072400008538&w=2) because something is not initialized in the right order. But I haven't tested 2.6.24 on this hardware yet, and I don't know if things have changed regarding this.

Quoted 8 lines
Quoted 3 lines

that will depend if you dsdt have _PXM for your pci root bus. otherwise you will get all -1

Have a look at the above link. I don't get -1. I get 0 everywhere, while I should get 1 for some devices. And if I unplug/replug a device using fakephp, numa_node becomes correct (1 instead of 0). This just looks like the code is there but things are initialized in the wrong order.

do you have
...
bus 00 -> pxm 0 -> node 0
...
bus 40 -> pxm 1 -> node 1
...
bus 80 -> pxm 1 -> node 1

in your boot msg or dmesg?

if not, your dsdt doesn't have _PXM for pci root bus. or you need to ask your HW vendor to add that in their BIOS, or use my patchset.

YH

#7 lore raw
Quoted 29 lines · Yinghai Lu
Quoted 10 lines · Brice Goglin
Quoted 9 lines · Yinghai Lu
Quoted 6 lines · Brice Goglin

It works fine on regular machines such as dual opterons. However, I noticed recently that it was wrong on some quad-opteron machines (see
http://marc.info/?l=linux-pci&m=119072400008538&w=2) because something is not initialized in the right order. But I haven't tested 2.6.24 on this hardware yet, and I don't know if things have changed regarding this.

Quoted 8 lines
Quoted 3 lines

that will depend if you dsdt have _PXM for your pci root bus. otherwise you will get all -1

Have a look at the above link. I don't get -1. I get 0 everywhere, while I should get 1 for some devices. And if I unplug/replug a device using fakephp, numa_node becomes correct (1 instead of 0). This just looks like the code is there but things are initialized in the wrong order.

do you have
...
bus 00 -> pxm 0 -> node 0
...
bus 40 -> pxm 1 -> node 1
...
bus 80 -> pxm 1 -> node 1

Quoted 4 lines

in your boot msg or dmesg?

if not, your dsdt doesn't have _PXM for pci root bus. or you need to ask your HW vendor to add that in their BIOS, or use my patchset.

please try the attached patchset

please get x86.git then use quilt apply the patch

http://people.redhat.com/mingo/x86.git/README

YH

#8 lore raw
Quoted 19 lines · Yinghai Lu
Quoted 13 lines
Quoted 5 lines

Have a look at the above link. I don't get -1. I get 0 everywhere, while I should get 1 for some devices. And if I unplug/replug a device using fakephp, numa_node becomes correct (1 instead of 0). This just looks like the code is there but things are initialized in the wrong order.

do you have
...
bus 00 -> pxm 0 -> node 0
...
bus 40 -> pxm 1 -> node 1
...
bus 80 -> pxm 1 -> node 1

Quoted 5 lines

in your boot msg or dmesg?

if not, your dsdt doesn't have _PXM for pci root bus. or you need to ask your HW vendor to add that in their BIOS, or use my patchset.

Quoted 6 lines

please try the attached patchset

please get x86.git then use quilt apply the patch

http://people.redhat.com/mingo/x86.git/README

I finally managed to test this and it seems to work. I now get the following numa_node attributes:
/sys/devices/pci0000:00/0000:00:01.0/numa_node 0
/sys/devices/pci0000:00/0000:00:07.0/numa_node 0
/sys/devices/pci0000:00/0000:00:07.0/0000:38:0d.0/numa_node 0

/sys/devices/pci0000:00/0000:00:08.0/numa_node    0
/sys/devices/pci0000:00/0000:00:08.1/numa_node    0
/sys/devices/pci0000:00/0000:00:08.2/numa_node    0
/sys/devices/pci0000:00/0000:00:09.0/numa_node    0
/sys/devices/pci0000:00/0000:00:09.1/numa_node    0
/sys/devices/pci0000:00/0000:00:09.2/numa_node    0
/sys/devices/pci0000:00/0000:00:0a.0/numa_node    0

/sys/devices/pci0000:00/0000:00:0a.0/0000:22:00.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0b.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0c.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0c.0/0000:0c:00.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0c.0/0000:0c:00.0/0000:0d:00.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0d.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0d.0/0000:01:00.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0e.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0e.0/0000:17:00.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0e.0/0000:17:00.0/0000:18:00.0/numa_node 0

/sys/devices/pci0000:00/0000:00:18.0/numa_node    0
/sys/devices/pci0000:00/0000:00:18.1/numa_node    0
/sys/devices/pci0000:00/0000:00:18.2/numa_node    0
/sys/devices/pci0000:00/0000:00:18.3/numa_node    0
/sys/devices/pci0000:00/0000:00:19.0/numa_node    0
/sys/devices/pci0000:00/0000:00:19.1/numa_node    0
/sys/devices/pci0000:00/0000:00:19.2/numa_node    0
/sys/devices/pci0000:00/0000:00:19.3/numa_node    0
/sys/devices/pci0000:00/0000:00:1a.0/numa_node    0
/sys/devices/pci0000:00/0000:00:1a.1/numa_node    0
/sys/devices/pci0000:00/0000:00:1a.2/numa_node    0
/sys/devices/pci0000:00/0000:00:1a.3/numa_node    0
/sys/devices/pci0000:00/0000:00:1b.0/numa_node    0
/sys/devices/pci0000:00/0000:00:1b.1/numa_node    0
/sys/devices/pci0000:00/0000:00:1b.2/numa_node    0
/sys/devices/pci0000:00/0000:00:1b.3/numa_node    0
/sys/devices/pci0000:40/0000:40:0f.0/numa_node    1
/sys/devices/pci0000:40/0000:40:10.0/numa_node    1
/sys/devices/pci0000:40/0000:40:11.0/numa_node    1
/sys/devices/pci0000:40/0000:40:12.0/numa_node    1

/sys/devices/pci0000:40/0000:40:12.0/0000:51:00.0/numa_node 1
/sys/devices/pci0000:40/0000:40:13.0/numa_node 1

The 5 last lines above would report 0 instead of 1 with an older kernel. Everything looks correct now (0000:40 is the second PCIe bus and it is attached to socket #1).

Thanks a lot, Yinghai! Are you planning to merge these patches in the near future? 2.6.26?

Brice

PS: I saved the corresponding dmesg. If you want to look at it, please let me know.

#9 lore raw
Quoted 74 lines · Brice Goglin
Quoted 19 lines · Yinghai Lu
Quoted 13 lines
Quoted 5 lines

Have a look at the above link. I don't get -1. I get 0 everywhere, while I should get 1 for some devices. And if I unplug/replug a device using fakephp, numa_node becomes correct (1 instead of 0). This just looks like the code is there but things are initialized in the wrong order.

do you have
...
bus 00 -> pxm 0 -> node 0
...
bus 40 -> pxm 1 -> node 1
...
bus 80 -> pxm 1 -> node 1

Quoted 5 lines

in your boot msg or dmesg?

if not, your dsdt doesn't have _PXM for pci root bus. or you need to ask your HW vendor to add that in their BIOS, or use my patchset.

Quoted 6 lines

please try the attached patchset

please get x86.git then use quilt apply the patch

http://people.redhat.com/mingo/x86.git/README

I finally managed to test this and it seems to work. I now get the following numa_node attributes:
/sys/devices/pci0000:00/0000:00:01.0/numa_node 0
/sys/devices/pci0000:00/0000:00:07.0/numa_node 0
/sys/devices/pci0000:00/0000:00:07.0/0000:38:0d.0/numa_node 0

/sys/devices/pci0000:00/0000:00:08.0/numa_node    0
/sys/devices/pci0000:00/0000:00:08.1/numa_node    0
/sys/devices/pci0000:00/0000:00:08.2/numa_node    0
/sys/devices/pci0000:00/0000:00:09.0/numa_node    0
/sys/devices/pci0000:00/0000:00:09.1/numa_node    0
/sys/devices/pci0000:00/0000:00:09.2/numa_node    0
/sys/devices/pci0000:00/0000:00:0a.0/numa_node    0

/sys/devices/pci0000:00/0000:00:0a.0/0000:22:00.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0b.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0c.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0c.0/0000:0c:00.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0c.0/0000:0c:00.0/0000:0d:00.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0d.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0d.0/0000:01:00.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0e.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0e.0/0000:17:00.0/numa_node 0
/sys/devices/pci0000:00/0000:00:0e.0/0000:17:00.0/0000:18:00.0/numa_node 0

/sys/devices/pci0000:00/0000:00:18.0/numa_node    0
/sys/devices/pci0000:00/0000:00:18.1/numa_node    0
/sys/devices/pci0000:00/0000:00:18.2/numa_node    0
/sys/devices/pci0000:00/0000:00:18.3/numa_node    0
/sys/devices/pci0000:00/0000:00:19.0/numa_node    0
/sys/devices/pci0000:00/0000:00:19.1/numa_node    0
/sys/devices/pci0000:00/0000:00:19.2/numa_node    0
/sys/devices/pci0000:00/0000:00:19.3/numa_node    0
/sys/devices/pci0000:00/0000:00:1a.0/numa_node    0
/sys/devices/pci0000:00/0000:00:1a.1/numa_node    0
/sys/devices/pci0000:00/0000:00:1a.2/numa_node    0
/sys/devices/pci0000:00/0000:00:1a.3/numa_node    0
/sys/devices/pci0000:00/0000:00:1b.0/numa_node    0
/sys/devices/pci0000:00/0000:00:1b.1/numa_node    0
/sys/devices/pci0000:00/0000:00:1b.2/numa_node    0
/sys/devices/pci0000:00/0000:00:1b.3/numa_node    0
/sys/devices/pci0000:40/0000:40:0f.0/numa_node    1
/sys/devices/pci0000:40/0000:40:10.0/numa_node    1
/sys/devices/pci0000:40/0000:40:11.0/numa_node    1
/sys/devices/pci0000:40/0000:40:12.0/numa_node    1

/sys/devices/pci0000:40/0000:40:12.0/0000:51:00.0/numa_node 1
/sys/devices/pci0000:40/0000:40:13.0/numa_node 1

Quoted 6 lines

The 5 last lines above would report 0 instead of 1 with an older kernel. Everything looks correct now (0000:40 is the second PCIe bus and it is attached to socket #1).

Thanks a lot, Yinghai! Are you planning to merge these patches in the near future? 2.6.26?

Andi thought that is too hardware related...
they have stayed a while in -mm.

these patchset could be only useful
when you have several HT chains, and BIOS doesn't have pxm->node in dsdt, or doesn't allocate io resource to some of addon cards.

YH

#10 lore raw
In reply to Brice Goglin
Quoted 7 lines · Brice Goglin
 /sys/devices/pci0000:40/0000:40:0f.0/numa_node    1
 /sys/devices/pci0000:40/0000:40:10.0/numa_node    1
 /sys/devices/pci0000:40/0000:40:11.0/numa_node    1
 /sys/devices/pci0000:40/0000:40:12.0/numa_node    1

/sys/devices/pci0000:40/0000:40:12.0/0000:51:00.0/numa_node 1
/sys/devices/pci0000:40/0000:40:13.0/numa_node 1

Quoted 7 lines

The 5 last lines above would report 0 instead of 1 with an older kernel. Everything looks correct now (0000:40 is the second PCIe bus and it is attached to socket #1).

Thanks a lot, Yinghai! Are you planning to merge these patches in the near future? 2.6.26?

ingo put them in x86.git#testing

please check
http://people.redhat.com/mingo/x86.git/README
to get that.

YH