Skip to contentSkip to conversation

fix CONFIG_X86_L1_CACHE_SHIFT

PATCH Manfred Spraul 5 participants 6 messages Sep 7, 2003 – Sep 9, 2003

lore
Share & tools
lore
#0 lore raw
Quoted 3 lines · Adrian

With CONFIG_M686 CONFIG_X86_L1_CACHE_SHIFT was set to 5, but a Pentium 4 requires 7.

Why requires? On x86, the cpu caches are fully coherent. A too small L1 cache shift results in false sharing on SMP, but it shouldn't cause the described problems.

And obviously: Pentium II cpus have a 32 byte cache line, increasing the L1 setting to 128 bytes only helps by chance.

My bet is that someone overwrites critical memory structures, and with more padding, the critical stuff is further away.

--
Manfred

#1 lore raw
Quoted 10 lines · Manfred Spraul
Quoted 4 lines · Adrian

With CONFIG_M686 CONFIG_X86_L1_CACHE_SHIFT was set to 5, but a Pentium 4 requires 7.

Why requires? On x86, the cpu caches are fully coherent. A too small L1 cache shift results in false sharing on SMP, but it shouldn't cause the described problems.
...

Thanks for the correction, I falsely thought CONFIG_X86_L1_CACHE_SHIFT does something different than it does.

Quoted 1 line

Manfred

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#2 lore raw
Quoted 7 lines · Adrian Bunk
Quoted 4 lines

Why requires? On x86, the cpu caches are fully coherent. A too small L1 cache shift results in false sharing on SMP, but it shouldn't cause the described problems.
...

Thanks for the correction, I falsely thought CONFIG_X86_L1_CACHE_SHIFT does something different than it does.

Were there any changes in the kernel to do with PCI MWI settings?

(MWI == memory write and invalidate)

If MWI is set incorrectly, I think PCI DMA is capable of breaking x86 cache coherence.

-- Jamie

#3 lore raw
Quoted 10 lines · Jamie Lokier
Quoted 7 lines · Adrian Bunk
Quoted 4 lines

Why requires? On x86, the cpu caches are fully coherent. A too small L1 cache shift results in false sharing on SMP, but it shouldn't cause the described problems.
...

Thanks for the correction, I falsely thought CONFIG_X86_L1_CACHE_SHIFT does something different than it does.

Were there any changes in the kernel to do with PCI MWI settings?

Yes; I've lost the specific context of the thread, but I have been working on MWI/cacheline size issues along with IvanK for a while.

It's apparently the responsibility of the OS to fill in correct PCI_CACHE_LINE_SIZE values, which in the case of generic kernels must be filled in at runtime (pci_cache_line_size) not at compile-time (SMP_CACHE_BYTES, etc.)

If you don't call pci_set_mwi() for a PCI device, which triggers the cacheline size fixups and other checks, then using Memory-Write-Invalidate (MWI) is definitely wrong. Or on an older kernel, without the latest MWI changes, you could wind up programming cacheline size to a value smaller than your current processor (again, due to generic kernels).

If a feature/device/whatever can be programmed with cacheline size at runtime, that will always be the preference. With a compile-time constant for cacheline size, you are _guaranteed_ it will be wrong in some case.

Jeff

#4 lore raw
Quoted 4 lines · Jeff Garzik

Yes; I've lost the specific context of the thread, but I have been working on MWI/cacheline size issues along with IvanK for a while.

Context: Peter experiences very bad network performance with 2.4.22 - it looks like 99% packet drop or something like that. The packet drop disappears if CONFIG_L1_CACHE_SHIFT is set to 7 (i.e. 128 byte cache line size). 2.4.21 works. The network cards are some kind of atm cards. Several systems are affected - at least Pentium II and PPro systems.

Peter: what's the exact brand and nic driver that you use? Could you try to figure out what exactly breaks? I'd use "ping -f -s 1500", perhaps together with "tcpdump -s 1500 -x" on both ends.

--
Manfred

#5 lore raw
Quoted 10 lines · Manfred Spraul

Context: Peter experiences very bad network performance with 2.4.22 - it looks like 99% packet drop or something like that. The packet drop disappears if CONFIG_L1_CACHE_SHIFT is set to 7 (i.e. 128 byte cache line size). 2.4.21 works.
The network cards are some kind of atm cards. Several systems are affected - at least Pentium II and PPro systems.

Peter: what's the exact brand and nic driver that you use? Could you try to figure out what exactly breaks? I'd use "ping -f -s 1500", perhaps together with "tcpdump -s 1500 -x" on both ends.

Meanwhile, I could verify that the problems do not occur, when I use an ethernet network adapter. All the machines have Marconi/Fore Systems Forerunner LE ATM cards (/proc/pci: ATM network controller: Integrated Device Tech IDT77211 ATM Adapter (rev 3)) with LAN emulation.

To figure out what exactly breaks, is the hard part: There is no general malfunction, I only found some particular test cases. The easiest of these
is: "wget ftp://ftp.nai.com/pub/datfiles/english/dat-4291.zip". With a vanilla 2.4.22 kernel, the data connection dies with a timeout after transferring some kbytes. When capturing the connection with tcpdump, the only thing unusual I can discover is that the time interval between 2 data packets coming in is unusually hight. I don't have the slightest idea, what is really going wrong. Transferring data from some other ftp server (e.g. ftp.kernel.org) works as usual.

As esoterical as it sounds, the whole issue is 100% reproducible and disappears with CONFIG_L1_CACHE_SHIFT set to 7. (The example with wget is only the easiest test I could find - for my purposes, the fact that sendmail and samba don't work correctly makes the kernel almost useless). "Ping" doesn't show anything unusual (no dropped packets). Since the problem did not occur with older kernels, the "CONFIG_L1_CACHE_SHIFT" setting can hardly be the real problem.

Regards,
Peter