Skip to contentSkip to conversation

Update cacheline size on X86_GENERIC

PATCH Dave Jones 5 participants 17 messages Oct 9, 2008 – Oct 12, 2008 x86

lore
Share & tools
lore

Dave Jones

author
#0 lore raw

I just noticed that configuring a kernel to use CONFIG_X86_GENERIC
(as is typical for a distro kernel) configures it to use a 128 byte cacheline size. This made sense when that was commonplace (P4 era) but current
Intel, AMD and VIA cpus use 64 byte cachelines.

arch/x86/Kconfig.cpu renamed from arch/x86/Kconfig.cpu~ +4 -4
@@ -301,8 +301,8 @@ config X86_CPU
 # Define implied options from the CPU selection here
 config X86_L1_CACHE_BYTES
 	int
-	default "128" if GENERIC_CPU || MPSC
-	default "64" if MK8 || MCORE2
+	default "128" if MPENTIUM4 || MPSC
+	default "64" if MK8 || MCORE2 || GENERIC_CPU
 	depends on X86_64
 
 config X86_INTERNODE_CACHE_BYTES
@@ -316,10 +316,10 @@ config X86_CMPXCHG
 
 config X86_L1_CACHE_SHIFT
 	int
-	default "7" if MPENTIUM4 || X86_GENERIC || GENERIC_CPU || MPSC
+	default "7" if MPENTIUM4 || MPSC
 	default "4" if X86_ELAN || M486 || M386 || MGEODEGX1
 	default "5" if MWINCHIP3D || MWINCHIP2 || MWINCHIPC6 || MCRUSOE || MEFFICEON || MCYRIXIII || MK6 || MPENTIUMIII || MPENTIUMII || M686 || M586MMX || M586TSC || M586 || MVIAC3_2 || MGEODE_LX
-	default "6" if MK7 || MK8 || MPENTIUMM || MCORE2 || MVIAC7
+	default "6" if MK7 || MK8 || MPENTIUMM || MCORE2 || MVIAC7 || X86_GENERIC || GENERIC_CPU
 
 config X86_XADD
 	def_bool y
Signature · 1 line
http://www.codemonkey.org.uk
#1 lore raw
Quoted 5 lines · Dave Jones

I just noticed that configuring a kernel to use CONFIG_X86_GENERIC (as is typical for a distro kernel) configures it to use a 128 byte cacheline size. This made sense when that was commonplace (P4 era) but current
Intel, AMD and VIA cpus use 64 byte cachelines.

I think P4 technically did have 64 byte cachelines, but had some adjacent line prefetching. And AFAIK core2 CPUs can do similar prefetching (but maybe it's smarter and doesn't cause so much bouncing?).

Anyway, GENERIC kernel should run well on all architectures, and while going too big causes slightly increased structures sometimes, going too small could result in horrible bouncing.

Lastly, I think x86 will go to 128 byte lines in the next year or two, so maybe at this point we can just keep 128 byte alignment?

/random thoughts

Quoted 32 lines

--- linux-2.6.26.noarch/arch/x86/Kconfig.cpu~ 2008-10-09 12:59:56.000000000
-0400 +++ linux-2.6.26.noarch/arch/x86/Kconfig.cpu 2008-10-09
13:11:32.000000000 -0400 @@ -301,8 +301,8 @@ config X86_CPU
# Define implied options from the CPU selection here config X86_L1_CACHE_BYTES

arch/x86/Kconfig.cpu renamed from arch/x86/Kconfig.cpu~ +3 -3
 	int
-	default "128" if GENERIC_CPU || MPSC
-	default "64" if MK8 || MCORE2
+	default "128" if MPENTIUM4 || MPSC
+	default "64" if MK8 || MCORE2 || GENERIC_CPU
 	depends on X86_64
 
 config X86_INTERNODE_CACHE_BYTES
@@ -316,10 +316,10 @@ config X86_CMPXCHG
 
 config X86_L1_CACHE_SHIFT
 	int
-	default "7" if MPENTIUM4 || X86_GENERIC || GENERIC_CPU || MPSC
+	default "7" if MPENTIUM4 || MPSC
 	default "4" if X86_ELAN || M486 || M386 || MGEODEGX1
 	default "5" if MWINCHIP3D || MWINCHIP2 || MWINCHIPC6 || MCRUSOE ||

MEFFICEON || MCYRIXIII || MK6 || MPENTIUMIII || MPENTIUMII || M686 ||

M586MMX || M586TSC || M586 || MVIAC3_2 || MGEODE_LX -	default "6" if MK7 ||
MK8 || MPENTIUMM || MCORE2 || MVIAC7
(quoted diff) +1 -0
+	default "6" if MK7 || MK8 || MPENTIUMM || MCORE2 || MVIAC7 || X86_GENERIC

|| GENERIC_CPU

config X86_XADD
def_bool y

#2 lore raw
Quoted 9 lines · Nick Piggin
Quoted 5 lines · Dave Jones

I just noticed that configuring a kernel to use CONFIG_X86_GENERIC (as is typical for a distro kernel) configures it to use a 128 byte cacheline size. This made sense when that was commonplace (P4 era) but current
Intel, AMD and VIA cpus use 64 byte cachelines.

I think P4 technically did have 64 byte cachelines, but had some adjacent line prefetching.

The "coherency unit" on P4, which is what matters for SMP alignment purposes to avoid false sharing, is 128 bytes.

Quoted 2 lines

And AFAIK core2 CPUs can do similar prefetching (but maybe it's smarter and doesn't cause so much bouncing?).

On Core2 the coherency unit is 64bytes.

Quoted 3 lines

Anyway, GENERIC kernel should run well on all architectures, and while going too big causes slightly increased structures sometimes, going too small could result in horrible bouncing.

Exactly.

That is it costs one percent or so on TPC, but I think the fix for that is just to analyze where the problem is and size those data structures based on the runtime cache size. Some subsystems like slab do this already.

TPC is a bit of a extreme case because it is so extremly cache bound.

Overall the memory impact of the cache padding is getting less over time because more and more data is moving into the per CPU data areas.

-Andi

Signature · 1 line
[email protected]
#3 lore raw
Quoted 11 lines · Andi Kleen
Quoted 9 lines · Nick Piggin
Quoted 5 lines · Dave Jones

I just noticed that configuring a kernel to use CONFIG_X86_GENERIC (as is typical for a distro kernel) configures it to use a 128 byte cacheline size. This made sense when that was commonplace (P4 era) but current
Intel, AMD and VIA cpus use 64 byte cachelines.

I think P4 technically did have 64 byte cachelines, but had some adjacent line prefetching.

Quoted 7 lines

The "coherency unit" on P4, which is what matters for SMP alignment purposes to avoid false sharing, is 128 bytes.

Quoted 2 lines

And AFAIK core2 CPUs can do similar prefetching (but maybe it's smarter and doesn't cause so much bouncing?).

On Core2 the coherency unit is 64bytes.

OK.

Quoted 10 lines
Quoted 3 lines

Anyway, GENERIC kernel should run well on all architectures, and while going too big causes slightly increased structures sometimes, going too small could result in horrible bouncing.

Exactly.

That is it costs one percent or so on TPC, but I think the fix for that is just to analyze where the problem is and size those data structures based on the runtime cache size. Some subsystems like slab do this already.

Costs 1% on TPC? Is that 128 byte aligning data structures on Core2, or 64 byte aligning them on P4 that costs the performance?

Quoted 1 line

TPC is a bit of a extreme case because it is so extremly cache bound.

Still, it is a good canary.

Quoted 2 lines

Overall the memory impact of the cache padding is getting less over time because more and more data is moving into the per CPU data areas.

Right.

#4 lore raw
Quoted 6 lines · Nick Piggin
Quoted 6 lines
Quoted 4 lines

Anyway, GENERIC kernel should run well on all architectures, and while going too big causes slightly increased structures sometimes, going too small could result in horrible bouncing.

Exactly.

Quoted 7 lines
Quoted 4 lines

That is it costs one percent or so on TPC, but I think the fix for that is just to analyze where the problem is and size those data structures based on the runtime cache size. Some subsystems like slab do this already.

Costs 1% on TPC? Is that 128 byte aligning data structures on Core2, or 64 byte aligning them on P4 that costs the performance?

The first. BTW it was a rough number from memory, in that ballpark. Also the experiment was on older kernels, might be different now.

The second would undoubtedly be much worse.

-Andi

#15 lore raw
In reply to Nick Piggin
Quoted 7 lines · Nick Piggin

I think P4 technically did have 64 byte cachelines, but had some adjacent line prefetching. And AFAIK core2 CPUs can do similar prefetching (but maybe it's smarter and doesn't cause so much bouncing?).

Anyway, GENERIC kernel should run well on all architectures, and while going too big causes slightly increased structures sometimes, going too small could result in horrible bouncing.

Well, GENERIC really is targetted toward the commercial mainstream at the time, with the additional caveat that it shouldn't totally suck on anything that isn't so obscure it's irrelevant. It is thus a moving target. 1% on TPC doesn't count as "totally suck", especially since by now anyone who is running workloads like TPC either will have phased out their P4s or they don't care about performance at all.

Quoted 2 lines

Lastly, I think x86 will go to 128 byte lines in the next year or two, so maybe at this point we can just keep 128 byte alignment?

"x86" doesn't have a cache line size; a specific implementation will. Which particular implementation do you believe is going to 128-byte L1 cachelines?

-hpa

#16 lore raw
Quoted 9 lines · H. Peter Anvin
Quoted 7 lines · Nick Piggin

I think P4 technically did have 64 byte cachelines, but had some adjacent line prefetching. And AFAIK core2 CPUs can do similar prefetching (but maybe it's smarter and doesn't cause so much bouncing?).

Anyway, GENERIC kernel should run well on all architectures, and while going too big causes slightly increased structures sometimes, going too small could result in horrible bouncing.

Quoted 6 lines

Well, GENERIC really is targetted toward the commercial mainstream at the time, with the additional caveat that it shouldn't totally suck on anything that isn't so obscure it's irrelevant. It is thus a moving target. 1% on TPC doesn't count as "totally suck", especially since by now anyone who is running workloads like TPC either will have phased out their P4s or they don't care about performance at all.

tpc shouldn't have too false sharing these days, AFAICT (slab is rather important there, but it finds cacheline sizes at runtime). Actually I thought Andi was referring to the slowdown on 64-byte cacheline systems. But other workloads could be hurt much worse than tpc-c from false sharing I think.

Quoted 6 lines
Quoted 2 lines

Lastly, I think x86 will go to 128 byte lines in the next year or two, so maybe at this point we can just keep 128 byte alignment?

"x86" doesn't have a cache line size; a specific implementation will. Which particular implementation do you believe is going to 128-byte L1 cachelines?

Right. I thought a future implementation would. But I'm probably wrong about that, and anyway OK it wasn't such a good argument for kernel.org kernels I suppose.

#5 lore raw
In reply to Andi Kleen
Quoted 12 lines · Andi Kleen
Quoted 11 lines · Nick Piggin
Quoted 10 lines
Quoted 3 lines

Anyway, GENERIC kernel should run well on all architectures, and while going too big causes slightly increased structures sometimes, going too small could result in horrible bouncing.

Exactly.

That is it costs one percent or so on TPC, but I think the fix for that is just to analyze where the problem is and size those data structures based on the runtime cache size. Some subsystems like slab do this already.

Quoted 7 lines
Quoted 2 lines

Costs 1% on TPC? Is that 128 byte aligning data structures on Core2, or 64 byte aligning them on P4 that costs the performance?

The first. BTW it was a rough number from memory, in that ballpark. Also the experiment was on older kernels, might be different now.

The second would undoubtedly be much worse.

OK. Well I don't have a really strong opinion on what to do...

I guess there is a reasonable argument to not care about P4 so much in today's GENERIC kernel. If it is worth around 1% on tpc on a more modern architecture, that is a pretty big motivation to change it too...

#6 lore raw
Quoted 1 line

I guess there is a reasonable argument to not care about P4 so

I don't think it is. Ignoring old systems would be a mistake and the wrong signal. One of Linux's forte over the competition was always to run reasonable on older systems too.

There are millions and millions of P4s around.
And they're not that old, they're still shipping in fact. And the point of GENERIC was to be a reasonable default on all systems.

If you want to optimize for a specific CPU you're always free to compile the kernel for that. But GENERIC should be really GENERIC.

Quoted 3 lines

much in today's GENERIC kernel. If it is worth around 1% on tpc on a more modern architecture, that is a pretty big motivation to change it too...

TPC is a extreme case, it is extremly cache bound.

Besides I suspect the TPC issue could be fixed with a minimal tweaks without breaking other systems.

-Andi

Signature · 1 line
[email protected]
#7 lore raw
Quoted 5 lines · Andi Kleen
Quoted 1 line

I guess there is a reasonable argument to not care about P4 so

I don't think it is. Ignoring old systems would be a mistake and the wrong signal. One of Linux's forte over the competition was always to run reasonable on older systems too.

I think there is a reasonable argument: and that is that most multiprocessor P4 systems in production and using a GENERIC (ie. probably not custom but probably vendor compiled) kernel is not likely to be upgraded to a 2.6.28+ based GENERIC kernel.

I also think there are reasonable arguments the other way, and I personally also think it might be better to leave it 128 (even if it is unlikely, introducing a regression is not good).

Quoted 2 lines

There are millions and millions of P4s around. And they're not that old, they're still shipping in fact.

Still shipping in anything aside from 1s systems?

Quoted 7 lines

And the point of GENERIC was to be a reasonable default on all systems.

If you want to optimize for a specific CPU you're always free to compile the kernel for that. But GENERIC should be really GENERIC.

Quoted 5 lines
Quoted 3 lines

much in today's GENERIC kernel. If it is worth around 1% on tpc on a more modern architecture, that is a pretty big motivation to change it too...

TPC is a extreme case, it is extremly cache bound.

Still, 1% there is a large increase.

Quoted 2 lines

Besides I suspect the TPC issue could be fixed with a minimal tweaks without breaking other systems.

That would be nice. It would be interesting to know what is causing the slowdown.

#8 lore raw
Quoted 3 lines · Nick Piggin

I also think there are reasonable arguments the other way, and I personally also think it might be better to leave it 128 (even if it is unlikely, introducing a regression is not good).

The issue is also that the regression will be likely large. False sharing can really hurt when it hits as you know, because the penalties are so large.

Quoted 4 lines
Quoted 2 lines

There are millions and millions of P4s around. And they're not that old, they're still shipping in fact.

Still shipping in anything aside from 1s systems?

Remember the first Core2 based 4S (Tigerton) Xeon was only introduced last year and that market is quite conservative. For 2S it's a bit longer, but it wouldn't surprise me there if new systems are still shipping.

Also to be honest I doubt the theory that older systems are never upgraded to newer OS.

Quoted 2 lines

That would be nice. It would be interesting to know what is causing the slowdown.

At least that test is extremly cache footprint sensitive. A lot of the cache misses are surprisingly in hd_struct, because it runs with hundred of disks and each needs hd_struct references in the fast path. The recent introduction of fine grained per partition statistics caused a large slowdown. But I don't think kernel workloads
are normally that extremly cache sensitive.

-Andi

Signature · 1 line
[email protected]
#9 lore raw
Quoted 12 lines · Andi Kleen
Quoted 3 lines · Nick Piggin

I also think there are reasonable arguments the other way, and I personally also think it might be better to leave it 128 (even if it is unlikely, introducing a regression is not good).

The issue is also that the regression will be likely large. False sharing can really hurt when it hits as you know, because the penalties are so large.

Quoted 3 lines
Quoted 2 lines

There are millions and millions of P4s around. And they're not that old, they're still shipping in fact.

Quoted 8 lines
Quoted 1 line

Still shipping in anything aside from 1s systems?

Remember the first Core2 based 4S (Tigerton) Xeon was only introduced last year and that market is quite conservative. For 2S it's a bit longer, but it wouldn't surprise me there if new systems are still shipping.

Also to be honest I doubt the theory that older systems are never upgraded to newer OS.

Actaually, I have examples to the contrary. :-)

Thanks,
Rafael

#10 lore raw
In reply to Andi Kleen
Quoted 6 lines · Andi Kleen
Quoted 3 lines · Nick Piggin

I also think there are reasonable arguments the other way, and I personally also think it might be better to leave it 128 (even if it is unlikely, introducing a regression is not good).

The issue is also that the regression will be likely large.

Yeah, that is what I'm worried about. If it was a simple case of 1% loss on P4 for 1% gain on Core2, it would be a good change. But it might be huge losses on P4s.

Quoted 6 lines

False sharing can really hurt when it hits as you know, because the penalties are so large.

Quoted 3 lines
Quoted 2 lines

There are millions and millions of P4s around. And they're not that old, they're still shipping in fact.

Quoted 8 lines
Quoted 1 line

Still shipping in anything aside from 1s systems?

Remember the first Core2 based 4S (Tigerton) Xeon was only introduced last year and that market is quite conservative. For 2S it's a bit longer, but it wouldn't surprise me there if new systems are still shipping.

Also to be honest I doubt the theory that older systems are never upgraded to newer OS.

Yeah, fair enough.

Quoted 9 lines
Quoted 2 lines

That would be nice. It would be interesting to know what is causing the slowdown.

At least that test is extremly cache footprint sensitive. A lot of the cache misses are surprisingly in hd_struct, because it runs with hundred of disks and each needs hd_struct references in the fast path. The recent introduction of fine grained per partition statistics caused a large slowdown. But I don't think kernel workloads
are normally that extremly cache sensitive.

That's interesting. struct device is pretty big. I wonder if fields couldn't be rearranged to minimise the fastpath cacheline footprint? I guess that's already been looked at?

#11 lore raw
Quoted 11 lines
Quoted 9 lines
Quoted 2 lines

That would be nice. It would be interesting to know what is causing the slowdown.

At least that test is extremly cache footprint sensitive. A lot of the cache misses are surprisingly in hd_struct, because it runs with hundred of disks and each needs hd_struct references in the fast path. The recent introduction of fine grained per partition statistics caused a large slowdown. But I don't think kernel workloads
are normally that extremly cache sensitive.

That's interesting. struct device is pretty big. I wonder if fields

Yes it is (it actually can be easily shrunk -- see willy's recent patch to remove the struct completion from knodes), but that won't help because it will always
be larger than a cache line and it's in the middle, so the accesses to first part of it and last part of it will be separate.

Quoted 2 lines

couldn't be rearranged to minimise the fastpath cacheline footprint? I guess that's already been looked at?

Yes, but not very intensively. So far I was looking for more detailed profiling data to see the exact accesses.

Of course if you have any immediate ideas that could be tried too.

-Andi

Signature · 1 line
[email protected]
#12 lore raw
Quoted 18 lines · Andi Kleen
Quoted 11 lines
Quoted 9 lines
Quoted 2 lines

That would be nice. It would be interesting to know what is causing the slowdown.

At least that test is extremly cache footprint sensitive. A lot of the cache misses are surprisingly in hd_struct, because it runs with hundred of disks and each needs hd_struct references in the fast path. The recent introduction of fine grained per partition statistics caused a large slowdown. But I don't think kernel workloads are normally that extremly cache sensitive.

That's interesting. struct device is pretty big. I wonder if fields

Yes it is (it actually can be easily shrunk -- see willy's recent patch to remove the struct completion from knodes), but that won't help because it will always
be larger than a cache line and it's in the middle, so the accesses to first part of it and last part of it will be separate.

Quoted 7 lines
Quoted 2 lines

couldn't be rearranged to minimise the fastpath cacheline footprint? I guess that's already been looked at?

Yes, but not very intensively. So far I was looking for more detailed profiling data to see the exact accesses.

Of course if you have any immediate ideas that could be tried too.

No immediate ideas. Jens probably is a good person to cc. With direct IO workloads, hd_struct should mostly only be touched in partition remapping and IO accounting.

start_sect, nr_sects would be read for partition remapping.

*dkstats will be read to do accounting (dkstats for UP is written, but false sharing doesn't matter on UP), as does partno.

These could all go together at the top of the struct perhaps.

struct device->parent gets read as well. This might go at the top of struct device, which could come next.

stamp and in_flight are tricky, as they get both read and written often :(

Still, you might just be able to fit them into the same 64-byte cacheline as well as all the above fields.

At this point, you would want to cacheline align hd_struct. So if you want to do that dynamically, you would need to change the disk_part_tbl scheme (but at least you could test with static annotations first).

The other thing I notice is the block layer has some functions which have error paths that have BDEVNAME_SIZE size arrays for error cases, which gcc may not do well at. Probably they should go out to noinline functions.

#13 lore raw
Quoted 3 lines

No immediate ideas. Jens probably is a good person to cc. With direct IO workloads, hd_struct should mostly only be touched in partition remapping and IO accounting.

I found it doubtful if grouping the rw and ro members together was a good idea.

Quoted 1 line

At this point, you would want to cacheline align hd_struct. So if you

The problem is probably not false sharing, but simply cache misses because it's so big. I think.

-Andi

Signature · 1 line
[email protected]
#14 lore raw
Quoted 6 lines · Andi Kleen
Quoted 3 lines

No immediate ideas. Jens probably is a good person to cc. With direct IO workloads, hd_struct should mostly only be touched in partition remapping and IO accounting.

I found it doubtful if grouping the rw and ro members together was a good idea.

If all members touched in the fastpath fit into one cacheline, then it definitely is. Because if you put the rw members in another cacheline, that line is still going to bounce just the same, but then you are just going to take up one more cacheline with the ro members.

Quoted 4 lines
Quoted 1 line

At this point, you would want to cacheline align hd_struct. So if you

The problem is probably not false sharing, but simply cache misses because it's so big. I think.

If you line up all the commonly touched members into cacheline boundaries, then presumably you have to assume the start of the struct has eg. cacheline alignment. There are situations where you can almost cut down cacheline footprint of a given data structure by 2x by aligning the items (eg. if we align struct page to 64 bytes, then random access to mem_map array will be almost 1/2 the cacheline footprint). I've always been interested in whether "oltp" benefits from that (eg. define WANT_PAGE_VIRTUAL for x86-64).