Skip to contentSkip to conversation

2.6.19-rc1: Slowdown in lmbench's fork

DISCUSSION Tim Chen 4 participants 11 messages Nov 2, 2006 – Nov 3, 2006

lore
Share & tools
lore

Tim Chen

author
#0 lore raw

After introduction of the following patch:

[PATCH] genirq: x86_64 irq: make vector_irq per cpu
http://kernel.org/git/?
p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=550f2299ac8ffaba943cf211380d3a8d3fa75301

we see fork benchmark in lmbench-3.0-a7 slowed by 11.5% on a 2 socket woodcrest machine. Similar change is seen also on other SMP Xeon machines.

When running lmbench, we have chosen the lmbench option to pin parent and child on different processor cores

Overhead of calling sched_setaffinity to place the process on processor is included in lmbench's fork time measurement. The patch may play a role in increasing this.

The two follow up patches to the original "make vector_irq per cpu" did not affect the fork time.

[PATCH] x86_64 irq: Properly update vector_irq [PATCH] x86-64: Only look at per_cpu data for online ...

Tim

#1 lore raw
Quoted 10 lines · Tim Chen

After introduction of the following patch:

[PATCH] genirq: x86_64 irq: make vector_irq per cpu
http://kernel.org/git/?
p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=550f2299ac8ffaba943cf211380d3a8d3fa75301

we see fork benchmark in lmbench-3.0-a7 slowed by 11.5% on a 2 socket woodcrest machine. Similar change is seen also on other SMP Xeon machines.

Quoted 6 lines

When running lmbench, we have chosen the lmbench option to pin parent and child on different processor cores

Overhead of calling sched_setaffinity to place the process on processor is included in lmbench's fork time measurement. The patch may play a role in increasing this.

The only think I can think of is that because data structures moved around we may be seeing some more cache misses. If we were talking normal interrupts taking a little longer I can see my change having a direct correlation. But I don't believe I touched anything in that patch that touched the IPI path.

I did add some things to the per cpu area and expanded it a little which may be what you are seeing.

That feels like a significant increase in fork times. I will think about it and holler if I can think of a productive direction to try.

My only partial guess is that it might be worth adding the per cpu variables my patch adds without any of the corresponding code changes. And see if adding variables to the per cpu area is what is causing the change.

The two tests I can see in this line are:
- to add the percpu vector_irq variable.
- to increase NR_IRQs.

My suspicion is that one of those two changes alone will change things enough that you see your lmbench slowdown. If that is the case then it is probably worth shuffling around the variables in the per cpu area to get better cache line affinity.

Eric

author
#2 lore raw
Quoted 9 lines · Eric W. Biederman

My only partial guess is that it might be worth adding the per cpu variables my patch adds without any of the corresponding code changes. And see if adding variables to the per cpu area is what is causing the change.

The two tests I can see in this line are:
- to add the percpu vector_irq variable.
- to increase NR_IRQs.

Increasing the NR_IRQs resulted in the regression. Adding the percpu vector_irq variable did not cause any changes.

Tim

#3 lore raw

Re: 2.6.19-rc1: x86_64 slowdown in lmbench's fork

Quoted 7 lines · Tim Chen
Quoted 5 lines · Eric W. Biederman

My only partial guess is that it might be worth adding the per cpu variables my patch adds without any of the corresponding code changes. And see if adding variables to the per cpu area is what is causing the change.

Quoted 6 lines
Quoted 3 lines

The two tests I can see in this line are:
- to add the percpu vector_irq variable.
- to increase NR_IRQs.

Increasing the NR_IRQs resulted in the regression.
...

What's your CONFIG_NR_CPUS setting that you are seeing such a big regression?

Quoted 1 line

Tim

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#4 lore raw

Re: 2.6.19-rc1: x86_64 slowdown in lmbench's fork

Quoted 12 lines · Adrian Bunk
Quoted 11 lines · Tim Chen
Quoted 8 lines · Eric W. Biederman

My only partial guess is that it might be worth adding the per cpu variables my patch adds without any of the corresponding code changes. And see if adding variables to the per cpu area is what is causing the change.

The two tests I can see in this line are:
- to add the percpu vector_irq variable.
- to increase NR_IRQs.

Quoted 5 lines
Quoted 2 lines

Increasing the NR_IRQs resulted in the regression.
...

What's your CONFIG_NR_CPUS setting that you are seeing such a big regression?

Also could we see the section of System.map that deals with per cpu variables.

I believe there are some counters for processes and the like just below kstat whose size increase is causing you real problems.

Ugh. I just looked at include/linux/kernel_stat.h
kstat has the per cpu irq counters and all of the cpu process time accounting so it is quite likely that we are going to be touching this structure plus the run queues and the process counts during a fork. All of which are now potentially much more spread out.

Also has anyone else reproduce this problem yet?

I don't doubt that it exists but having a few more data points or eyeballs on the problem couldn't hurt.

Eric

author
#5 lore raw

Re: 2.6.19-rc1: x86_64 slowdown in lmbench's fork

In reply to Adrian Bunk
Quoted 4 lines · Adrian Bunk

What's your CONFIG_NR_CPUS setting that you are seeing such a big regression?

CONFIG_NR_CPUS is set to 8.

Tim

#6 lore raw

Re: 2.6.19-rc1: x86_64 slowdown in lmbench's fork

Quoted 8 lines · Tim Chen
Quoted 4 lines · Adrian Bunk

What's your CONFIG_NR_CPUS setting that you are seeing such a big regression?

CONFIG_NR_CPUS is set to 8.

Ugh. This simply changes NR_IRQS from 256 to 512. Changing the size of data from 1K to 2K.

So unless there is some other array that is sized by NR_IRQs in the context switch path which could account for this in other ways. It looks like you just got unlucky.

The only hypothesis that I can seem to come up with is that maybe you are getting an extra tlb now that you didn't use to. I think the per cpu area is covered by huge pages but maybe not.

Eric

#7 lore raw

Re: 2.6.19-rc1: x86_64 slowdown in lmbench's fork

Quoted 4 lines

So unless there is some other array that is sized by NR_IRQs in the context switch path which could account for this in other ways. It looks like you just got unlucky.

TLB/cache profiling data might be useful?
My bet would be more on cache effects.

Quoted 3 lines

The only hypothesis that I can seem to come up with is that maybe you are getting an extra tlb now that you didn't use to. I think the per cpu area is covered by huge pages but maybe not.

It should be.

-Andi

#8 lore raw

Re: 2.6.19-rc1: x86_64 slowdown in lmbench's fork

Quoted 7 lines · Andi Kleen
Quoted 3 lines

So unless there is some other array that is sized by NR_IRQs in the context switch path which could account for this in other ways. It looks like you just got unlucky.

TLB/cache profiling data might be useful?
My bet would be more on cache effects.

The only way I can see that being true is if some irq was keeping the cache line warm for something in the process startup.

I have trouble seeing how adding 1K to an already 1K data structure can cause a cache fault that wasn't happening already.

Quoted 5 lines
Quoted 3 lines

The only hypothesis that I can seem to come up with is that maybe you are getting an extra tlb now that you didn't use to. I think the per cpu area is covered by huge pages but maybe not.

It should be.

Which invalidates the tlb fault hypothesis unless it happens to lie on the 2MB boundary.

Eric

#9 lore raw

Re: 2.6.19-rc1: x86_64 slowdown in lmbench's fork

Quoted 3 lines

Which invalidates the tlb fault hypothesis unless it happens to lie on the 2MB boundary.

It's in different pages than the code.

One drawback of using large TLBs is that the CPUs tend to have a lot less of them than small TLBs so they can actually thrash quicker if you're unlucky.

Only profiling can tell I guess.

-Andi

author
#10 lore raw

Re: 2.6.19-rc1: x86_64 slowdown in lmbench's fork

In reply to Andi Kleen
Quoted 8 lines · Andi Kleen
Quoted 3 lines

So unless there is some other array that is sized by NR_IRQs in the context switch path which could account for this in other ways. It looks like you just got unlucky.

TLB/cache profiling data might be useful?
My bet would be more on cache effects.

The TLB miss, cache miss and page walk profiles did not change when I measured it.

I have a suspicion that the overhead to pin parent and child processes to specific cpu had something to do with the change in time observed. Lmbench includes this overhead in the fork time it reported. I had chosen the lmbench option to set parent and child process on specific cpu.

When I skip this by picking another lmbench option to let scheduler pick the placement of parent and child process. I see that the fork time now stays unchanged with this setting. Wonder if the increase in time is in sched_setaffinity.

Tim