Skip to contentSkip to conversation

for_each_online_cpu broken ?

DISCUSSION Dave Jones 4 participants 12 messages Dec 8, 2005 – Dec 9, 2005

lore
Share & tools
lore

Dave Jones

author
#0 lore raw

Whilst debugging a memory leak, I hit sysrq meminfo, and got hot/cold info for CONFIG_NR_CPUS rather than 4 cpus I fixed a bug recently in mm/page_alloc.c to change this from a for_each_cpu to a for_each_online_cpu and I'm pretty certain I tested that it worked, but for reasons unknown, it's now misbehaving again.

I've only tried reproducing this on x86-64 so far.

Dave

#1 lore raw
Quoted 8 lines · Dave Jones

Whilst debugging a memory leak, I hit sysrq meminfo, and got hot/cold info for CONFIG_NR_CPUS rather than 4 cpus I fixed a bug recently in mm/page_alloc.c to change this from a for_each_cpu to a for_each_online_cpu and I'm pretty certain I tested that it worked, but for reasons unknown, it's now misbehaving again.

I've only tried reproducing this on x86-64 so far.

If the online map is wrong all kinds of things would go wrong.

Most likely your kernel doesn't have the fix.

The possible map is fixed kind of BTW in 2.6.15rc*. It was a side effect of CPU hotplug, which now uses a better algorithm to guess the number of possible CPUs. In 2.6.15 you will just get half the number of available CPUs in addition by default

-Andi

author
#2 lore raw

On Thu, Dec 08, 2005 at 06:26:32AM +0100, Andi Kleen wrote:

Hi Andi,

Quoted 8 lines
Quoted 4 lines

Whilst debugging a memory leak, I hit sysrq meminfo, and got hot/cold info for CONFIG_NR_CPUS rather than 4 cpus

I've only tried reproducing this on x86-64 so far.

If the online map is wrong all kinds of things would go wrong.

Most likely your kernel doesn't have the fix.

This was seen with a .15rc5-git1 kernel.
Is this something still living in your x86-64 patchset or -mm ?

Quoted 4 lines

The possible map is fixed kind of BTW in 2.6.15rc*. It was a side effect of CPU hotplug, which now uses a better algorithm to guess the number of possible CPUs. In 2.6.15 you will just get half the number of available CPUs in addition by default

Yep, I noticed it offers a maximum of 6 cpus on my way. As a sidenote, seems kinda funny (and wasteful maybe?), doing this on a lot of hardware that isn't hotplug capable. (Whilst I could disable cpu hotplug in my local build, this isn't an answer for a generic distro kernel).

Dave

#3 lore raw

From: Dave Jones <[email protected]>
Date: Thu, 8 Dec 2005 00:33:02 -0500

Quoted 7 lines · Andi Kleen
Quoted 5 lines

The possible map is fixed kind of BTW in 2.6.15rc*. It was a side effect of CPU hotplug, which now uses a better algorithm to guess the number of possible CPUs. In 2.6.15 you will just get half the number of available CPUs in addition by default

Quoted 5 lines

Yep, I noticed it offers a maximum of 6 cpus on my way. As a sidenote, seems kinda funny (and wasteful maybe?), doing this on a lot of hardware that isn't hotplug capable. (Whilst I could disable cpu hotplug in my local build, this isn't an answer for a generic distro kernel).

This can be dangerous btw, as some subsystem such as netfilter allocate enormous datastructures based upon the largest possible cpu number in the system.

In 2.6.16 it will use something a bit more intelligent, but overestimating the possible cpu set can be quite a waste.

#4 lore raw
Quoted 15 lines · David S. Miller

From: Dave Jones <[email protected]>
Date: Thu, 8 Dec 2005 00:33:02 -0500

Quoted 12 lines · Andi Kleen
Quoted 5 lines

The possible map is fixed kind of BTW in 2.6.15rc*. It was a side effect of CPU hotplug, which now uses a better algorithm to guess the number of possible CPUs. In 2.6.15 you will just get half the number of available CPUs in addition by default

Yep, I noticed it offers a maximum of 6 cpus on my way. As a sidenote, seems kinda funny (and wasteful maybe?), doing this on a lot of hardware that isn't hotplug capable. (Whilst I could disable cpu hotplug in my local build, this isn't an answer for a generic distro kernel).

If you can figure out a way to detect this please share. The ACPI designers unfortunately didn't think that far (they did it right for memory hotplug, but not for CPU)

I invented an ACPI extensin for it, but it's non standard so the half of CPUs is used as a default unless overwritten (additional_cpus=NUM)

Anyways I changed it earlier to 1 additional CPU by default.

Quoted 4 lines

This can be dangerous btw, as some subsystem such as netfilter allocate enormous datastructures based upon the largest possible cpu number in the system.

It is a big improvement in fact. Previously with NR_CPUS==128 (default on some distros) and CPU_HOTPLUG enabled it would always allocate several hundred KB of just standard per cpu data unnecessarily.

-Andi

#6 lore raw
In reply to Dave Jones

Hi.

Quoted 21 lines · Dave Jones

On Thu, Dec 08, 2005 at 06:26:32AM +0100, Andi Kleen wrote:

Hi Andi,

Quoted 8 lines
Quoted 4 lines

Whilst debugging a memory leak, I hit sysrq meminfo, and got hot/cold info for CONFIG_NR_CPUS rather than 4 cpus

I've only tried reproducing this on x86-64 so far.

If the online map is wrong all kinds of things would go wrong.

Most likely your kernel doesn't have the fix.

This was seen with a .15rc5-git1 kernel.
Is this something still living in your x86-64 patchset or -mm ?

Quoted 4 lines

The possible map is fixed kind of BTW in 2.6.15rc*. It was a side effect of CPU hotplug, which now uses a better algorithm to guess the number of possible CPUs. In 2.6.15 you will just get half the number of available CPUs in addition by default

Quoted 5 lines

Yep, I noticed it offers a maximum of 6 cpus on my way. As a sidenote, seems kinda funny (and wasteful maybe?), doing this on a lot of hardware that isn't hotplug capable. (Whilst I could disable cpu hotplug in my local build, this isn't an answer for a generic distro kernel).

Both suspend to disk (and suspend to ram?) implementations now depend on hotplug_cpu to enable extra cpus, so there is at least one reason for them to want hotplug support in a generic kernel.

Regards,

Nigel

author
#5 lore raw
In reply to Andi Kleen
Quoted 21 lines · Andi Kleen
Quoted 15 lines · David S. Miller

From: Dave Jones <[email protected]>
Date: Thu, 8 Dec 2005 00:33:02 -0500

Quoted 12 lines · Andi Kleen
Quoted 5 lines

The possible map is fixed kind of BTW in 2.6.15rc*. It was a side effect of CPU hotplug, which now uses a better algorithm to guess the number of possible CPUs. In 2.6.15 you will just get half the number of available CPUs in addition by default

Yep, I noticed it offers a maximum of 6 cpus on my way. As a sidenote, seems kinda funny (and wasteful maybe?), doing this on a lot of hardware that isn't hotplug capable. (Whilst I could disable cpu hotplug in my local build, this isn't an answer for a generic distro kernel).

If you can figure out a way to detect this please share. The ACPI designers unfortunately didn't think that far (they did it right for memory hotplug, but not for CPU)

Quoted 5 lines

I invented an ACPI extensin for it, but it's non standard so the half of CPUs is used as a default unless overwritten (additional_cpus=NUM)

Anyways I changed it earlier to 1 additional CPU by default.

Just guessing seems to be pretty guaranteed to give the wrong answer. I think it makes more sense to say "if your BIOS doesn't give the relevant info (as is usually the case), boot with additional_cpus)

Penalising the many for the needs of the few just seems wrong.

Dave

author
#7 lore raw
In reply to Nigel Cunningham
Quoted 9 lines · Nigel Cunningham
Quoted 5 lines

Yep, I noticed it offers a maximum of 6 cpus on my way. As a sidenote, seems kinda funny (and wasteful maybe?), doing this on a lot of hardware that isn't hotplug capable. (Whilst I could disable cpu hotplug in my local build, this isn't an answer for a generic distro kernel).

Both suspend to disk (and suspend to ram?) implementations now depend on hotplug_cpu to enable extra cpus, so there is at least one reason for them to want hotplug support in a generic kernel.

You mean suspend -> plug in a new cpu -> resume transitions ? That sounds *terrifying*.

Dave

#8 lore raw
Quoted 8 lines · Dave Jones
Quoted 6 lines · Nigel Cunningham
Quoted 5 lines

Yep, I noticed it offers a maximum of 6 cpus on my way. As a sidenote, seems kinda funny (and wasteful maybe?), doing this on a lot of hardware that isn't hotplug capable. (Whilst I could disable cpu hotplug in my local build, this isn't an answer for a generic distro kernel).

Quoted 6 lines
Quoted 3 lines

Both suspend to disk (and suspend to ram?) implementations now depend on hotplug_cpu to enable extra cpus, so there is at least one reason for them to want hotplug support in a generic kernel.

You mean suspend -> plug in a new cpu -> resume transitions ? That sounds *terrifying*.

No, they unplug all non BP CPUs before suspending. It obviously doesn't require additional CPUs though.

-Andi

#10 lore raw
In reply to Dave Jones
Quoted 2 lines

This was seen with a .15rc5-git1 kernel.
Is this something still living in your x86-64 patchset or -mm ?

Only the change for less additional CPUs.

-Andi

author
#11 lore raw
Quoted 4 lines · Andi Kleen
Quoted 2 lines

This was seen with a .15rc5-git1 kernel.
Is this something still living in your x86-64 patchset or -mm ?

Only the change for less additional CPUs.

Ok, then I'm at a loss why it's dumping state for NR_CPUS cpus. I'm away tomorrow, but I'll poke at it some more on Friday.

Dave

#9 lore raw
In reply to Dave Jones

Hi.

Quoted 8 lines · Dave Jones
Quoted 6 lines · Nigel Cunningham
Quoted 5 lines

Yep, I noticed it offers a maximum of 6 cpus on my way. As a sidenote, seems kinda funny (and wasteful maybe?), doing this on a lot of hardware that isn't hotplug capable. (Whilst I could disable cpu hotplug in my local build, this isn't an answer for a generic distro kernel).

Quoted 6 lines
Quoted 3 lines

Both suspend to disk (and suspend to ram?) implementations now depend on hotplug_cpu to enable extra cpus, so there is at least one reason for them to want hotplug support in a generic kernel.

You mean suspend -> plug in a new cpu -> resume transitions ? That sounds *terrifying*.

Andi is right, it's just a logical unplug. But having said that, I suppose extra cpus could be plugged/unplugged during a suspend to disk. Not that I've ever tried it. I have a real SMP mobo, but haven't had the opportunity to fire it up.

Regards,

Nigel