MIPS: lib: prefetch ahead in csum_partial()
PATCH 1/1 MIPS: lib: prefetch ahead in csum_partial()
csum_partial() reads every byte of the buffer once. Cores like the 74K have no hardware prefetcher, so every cache line is a demand miss, and after a non-coherent DMA the received data is never in the cache. With GRO enabled, validating the checksum of forwarded TCP traffic made csum_partial() the top entry in the profile on a BCM4716 (74Kc at 480 MHz, no L2, ~80 cycles per DRAM miss): 13-16% of all cycles.
Prefetch the 128 byte block two iterations ahead in the main loop. Unlike memcpy(), which disables prefetching on DMA_NONCOHERENT entirely, only prefetch blocks that lie inside the buffer: the 74K does not need a post-DMA invalidate, so a line prefetched past the end could belong to a buffer the device is still writing, and the CPU would later read the stale copy.
On an Asus RT-N16 (BCM4716, bgmac, single TCP stream unless noted, iperf3 median of 3, A/B in the same boot by toggling the prefetch, two rounds each):
without with (Mbit/s) local rx / tx 220 / 194 243 / 212 routed up / down 169 / 140 187 / 154 routed, 4 streams 65 / 79 69 / 84 flow offload bidir 253 292
--- arch/mips/lib/csum_partial.S | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
arch/mips/lib/csum_partial.S +15 -0
@@ -190,6 +190,21 @@ EXPORT_SYMBOL(csum_partial) andi t2, a1, 0x40 .Lmove_128bytes: +#ifdef CONFIG_CPU_HAS_PREFETCH + /* + * Fetch the block two iterations ahead, but only while it lies + * inside the buffer: on non-coherent systems a line prefetched + * past the end could belong to a buffer the device still owns. + */ + sltiu t5, t8, 3 + bnez t5, 2f + nop + pref 0, 0x100(src) + pref 0, 0x120(src) + pref 0, 0x140(src) + pref 0, 0x160(src) +2: +#endif CSUM_BIGCHUNK(src, 0x00, sum, t0, t1, t3, t4) CSUM_BIGCHUNK(src, 0x20, sum, t0, t1, t3, t4) CSUM_BIGCHUNK(src, 0x40, sum, t0, t1, t3, t4)
Signature · 1 line
2.56.0
kernel test robot
build botkernel test robot noticed the following build errors:
Hi Rosen,
kernel test robot noticed the following build errors:
[auto build test ERROR on linus/master]
[also build test ERROR on v7.3-rc6 next-20261009]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Rosen-Penev/MIPS-lib-prefetch-ahead-in-csum_partial/20261009-235442 base: linus/master patch link: https://lore.kernel.org/r/20261010065442.23780-1-rosenp%40gmail.com patch subject: [PATCH] MIPS: lib: prefetch ahead in csum_partial() config: mips-rbtx49xx_defconfig (https://download.01.org/0day-ci/archive/20261013/[email protected]/config) compiler: mips-linux-gcc (GCC) 16.1.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261013/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
Assembler messages:
>> arch/mips/lib/csum_partial.S:202: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x100($4)'
arch/mips/lib/csum_partial.S:203: Error: opcode not supported on this processor: r4600 (mips3) pref 0,0x120($4)' arch/mips/lib/csum_partial.S:204: Error: opcode not supported on this processor: r4600 (mips3) pref 0,0x140($4)' arch/mips/lib/csum_partial.S:205: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x160($4)'
vim +202 arch/mips/lib/csum_partial.S
102
103 .text
104 .set noreorder
105 .align 5
106 LEAF(csum_partial)
107 EXPORT_SYMBOL(csum_partial)
108 move sum, zero
109 move t7, zero
110
111 sltiu t8, a1, 0x8
112 bnez t8, .Lsmall_csumcpy /* < 8 bytes to copy */
113 move t2, a1
114
115 andi t7, src, 0x1 /* odd buffer? */
116
117 .Lhword_align:
118 beqz t7, .Lword_align
119 andi t8, src, 0x2
120
121 lbu t0, (src)
122 LONG_SUBU a1, a1, 0x1
123 #ifdef __MIPSEL__
124 sll t0, t0, 8
125 #endif
126 ADDC(sum, t0)
127 PTR_ADDU src, src, 0x1
128 andi t8, src, 0x2
129
130 .Lword_align:
131 beqz t8, .Ldword_align
132 sltiu t8, a1, 56
133
134 lhu t0, (src)
135 LONG_SUBU a1, a1, 0x2
136 ADDC(sum, t0)
137 sltiu t8, a1, 56
138 PTR_ADDU src, src, 0x2
139
140 .Ldword_align:
141 bnez t8, .Ldo_end_words
142 move t8, a1
143
144 andi t8, src, 0x4
145 beqz t8, .Lqword_align
146 andi t8, src, 0x8
147
148 LOAD32 t0, 0x00(src)
149 LONG_SUBU a1, a1, 0x4
150 ADDC(sum, t0)
151 PTR_ADDU src, src, 0x4
152 andi t8, src, 0x8
153
154 .Lqword_align:
155 beqz t8, .Loword_align
156 andi t8, src, 0x10
157
158 #ifdef USE_DOUBLE
159 ld t0, 0x00(src)
160 LONG_SUBU a1, a1, 0x8
161 ADDC(sum, t0)
162 #else
163 lw t0, 0x00(src)
164 lw t1, 0x04(src)
165 LONG_SUBU a1, a1, 0x8
166 ADDC(sum, t0)
167 ADDC(sum, t1)
168 #endif
169 PTR_ADDU src, src, 0x8
170 andi t8, src, 0x10
171
172 .Loword_align:
173 beqz t8, .Lbegin_movement
174 LONG_SRL t8, a1, 0x7
175
176 #ifdef USE_DOUBLE
177 ld t0, 0x00(src)
178 ld t1, 0x08(src)
179 ADDC(sum, t0)
180 ADDC(sum, t1)
181 #else
182 CSUM_BIGCHUNK1(src, 0x00, sum, t0, t1, t3, t4)
183 #endif
184 LONG_SUBU a1, a1, 0x10
185 PTR_ADDU src, src, 0x10
186 LONG_SRL t8, a1, 0x7
187
188 .Lbegin_movement:
189 beqz t8, 1f
190 andi t2, a1, 0x40
191
192 .Lmove_128bytes:
193 #ifdef CONFIG_CPU_HAS_PREFETCH
194 /*
195 * Fetch the block two iterations ahead, but only while it lies
196 * inside the buffer: on non-coherent systems a line prefetched
197 * past the end could belong to a buffer the device still owns.
198 */
199 sltiu t5, t8, 3
200 bnez t5, 2f
201 nop
> 202 pref 0, 0x100(src)
203 pref 0, 0x120(src)
204 pref 0, 0x140(src)
205 pref 0, 0x160(src)
206 2:
207 #endif
208 CSUM_BIGCHUNK(src, 0x00, sum, t0, t1, t3, t4)
209 CSUM_BIGCHUNK(src, 0x20, sum, t0, t1, t3, t4)
210 CSUM_BIGCHUNK(src, 0x40, sum, t0, t1, t3, t4)
211 CSUM_BIGCHUNK(src, 0x60, sum, t0, t1, t3, t4)
212 LONG_SUBU t8, t8, 0x01
213 .set reorder /* DADDI_WAR */
214 PTR_ADDU src, src, 0x80
215 bnez t8, .Lmove_128bytes
216 .set noreorder
217
218 1:
219 beqz t2, 1f
220 andi t2, a1, 0x20
221
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
Quoted 9 lines · kernel test robot
config: mips-rbtx49xx_defconfig (https://download.01.org/0day-ci/archive/20261013/[email protected]/config)
compiler: mips-linux-gcc (GCC) 16.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261013/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
Quoted 7 lines
All errors (new ones prefixed by >>):
Assembler messages:
Quoted 1 line
Quoted 1 line
arch/mips/lib/csum_partial.S:202: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x100($4)'
arch/mips/lib/csum_partial.S:203: Error: opcode not supported on this processor: r4600 (mips3)pref 0,0x120($4)' arch/mips/lib/csum_partial.S:204: Error: opcode not supported on this processor: r4600 (mips3)pref 0,0x140($4)'arch/mips/lib/csum_partial.S:205: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x160($4)'
So oddly enough we're missing Tx49 support in the toolchain and we've
only managed to survive with CPU_HAS_PREFETCH enabled for CPU_TX49XX because it also implies DMA_NONCOHERENT via MACH_TX49XX -> MACH_TXX9, which in turn disables stuff in arch/mips/lib/memcpy.S.
Quite straighforward to fix in binutils since docs are available (as in:
I have a copy), but do we have any users? Geert?
Maciej
Hi Maciej,
Quoted 18 lines · Maciej W. Rozycki
Quoted 9 lines · kernel test robot
config: mips-rbtx49xx_defconfig (https://download.01.org/0day-ci/archive/20261013/[email protected]/config)
compiler: mips-linux-gcc (GCC) 16.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261013/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/Quoted 7 lines
All errors (new ones prefixed by >>):
Assembler messages:
Quoted 1 line
Quoted 1 line
arch/mips/lib/csum_partial.S:202: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x100($4)'
arch/mips/lib/csum_partial.S:203: Error: opcode not supported on this processor: r4600 (mips3)pref 0,0x120($4)' arch/mips/lib/csum_partial.S:204: Error: opcode not supported on this processor: r4600 (mips3)pref 0,0x140($4)'arch/mips/lib/csum_partial.S:205: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x160($4)'
Quoted 7 lines
So oddly enough we're missing Tx49 support in the toolchain and we've
only managed to survive withCPU_HAS_PREFETCHenabled forCPU_TX49XXbecause it also impliesDMA_NONCOHERENTviaMACH_TX49XX->MACH_TXX9, which in turn disables stuff inarch/mips/lib/memcpy.S.Quite straighforward to fix in binutils since docs are available (as in:
I have a copy), but do we have any users? Geert?
Yes, I still boot test rbtx4927 regularly.
I never touched binutils sources, though...
Gr{oetje,eeting}s,
Geert
Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected] In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Hi Geert,
Quoted 9 lines
Quoted 7 lines
So oddly enough we're missing Tx49 support in the toolchain and we've
only managed to survive withCPU_HAS_PREFETCHenabled forCPU_TX49XXbecause it also impliesDMA_NONCOHERENTviaMACH_TX49XX->MACH_TXX9, which in turn disables stuff inarch/mips/lib/memcpy.S.Quite straighforward to fix in binutils since docs are available (as in:
I have a copy), but do we have any users? Geert?Yes, I still boot test rbtx4927 regularly.
Thanks for confirming.
Quoted 1 line
I never touched binutils sources, though...
No worries, this seems straightforward: just PREF, MADD/U plus [D]MULT/U
extensions, and DERET/SDBBP debug stuff, much of which shared with the Tx39 we already support, so just a matter of sticking another flag onto the existing opcode table entries.
I'd estimate it as one day worth of engineering effort plus maybe another
one for machines to work through verification, for the binutils side and the associated changes needed for GCC, i.e. a `-march=' option update plus maybe enabling a couple of existing machine description patterns for the Tx49 too. Would you be able to run GCC verification with your system?
So I guess I could squeeze this in after the oustanding Allegrex support
review, hopefully to start this week.
Maciej
Hi Maciej,
Quoted 19 lines · Maciej W. Rozycki
Quoted 9 lines
Quoted 7 lines
So oddly enough we're missing Tx49 support in the toolchain and we've
only managed to survive withCPU_HAS_PREFETCHenabled forCPU_TX49XXbecause it also impliesDMA_NONCOHERENTviaMACH_TX49XX->MACH_TXX9, which in turn disables stuff inarch/mips/lib/memcpy.S.Quite straighforward to fix in binutils since docs are available (as in:
I have a copy), but do we have any users? Geert?Yes, I still boot test rbtx4927 regularly.
Thanks for confirming.
Quoted 1 line
I never touched binutils sources, though...
No worries, this seems straightforward: just PREF, MADD/U plus [D]MULT/U
extensions, and DERET/SDBBP debug stuff, much of which shared with the Tx39 we already support, so just a matter of sticking another flag onto the existing opcode table entries.
Quoted 5 lines
I'd estimate it as one day worth of engineering effort plus maybe another
one for machines to work through verification, for the binutils side and the associated changes needed for GCC, i.e. a `-march=' option update plus maybe enabling a couple of existing machine description patterns for the Tx49 too. Would you be able to run GCC verification with your system?
The rbtx4927 is about the slowest system I still do regular testing on. It uses a Debian 8 nfsroot (over 10 Mbps Ethernet), which is the latest Debian release still supporting the instruction subset suppported by TX49XX.
So as long as no native compilation of large packages is required, it can be done. I can also give you remote access?
Thanks!
Gr{oetje,eeting}s,
Geert
Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected] In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Hi Geert,
Quoted 7 lines
Quoted 5 lines
I'd estimate it as one day worth of engineering effort plus maybe another
one for machines to work through verification, for the binutils side and the associated changes needed for GCC, i.e. a `-march=' option update plus maybe enabling a couple of existing machine description patterns for the Tx49 too. Would you be able to run GCC verification with your system?The rbtx4927 is about the slowest system I still do regular testing on.
Speed is of no essence, I've recently run regression-testing with a R3k
DECstation, a VAXstation (with NetBSD), and previously, as a day job, with a crippled ARM board well below the capabilities of the former systems. Depending on the components chosen and the amount of breakage triggered by the tests themselves (timeouts!) it could take from several hours to a few days.
Quoted 3 lines
It uses a Debian 8 nfsroot (over 10 Mbps Ethernet), which is the latest Debian release still supporting the instruction subset suppported by TX49XX.
I run most of my target systems NFS-rooted as well (over 100Mbps FDDI for
a change). Little is required from the target environment: an SSH server and a POSIX shell, and then either an NFS client for direct build system access or rsync for remote sysroot maintenance (in which case `rm' command is needed too to discard test cases copied over, once executed).
Quoted 2 lines
So as long as no native compilation of large packages is required, it can be done. I can also give you remote access?
There is no native compilation at all, only remote execution over SSH,
driven from the build system by DejaGNU.
As for remote access, that would be an option, perhaps an easy one since
I've done it before with someone else, as long as you can offer temporary storage for the sysroot to operate the target system with (~225MiB for o32, ~285MiB for n64, indicative) and some network bandwidth.
Maciej
Quoted 7 lines · Rosen Penev
arch/mips/lib/csum_partial.S +0
-0@@ -190,6 +190,21 @@ EXPORT_SYMBOL(csum_partial) andi t2, a1, 0x40
arch/mips/lib/csum_partial.S EXPORT_SYMBOL(csum_partial) · line 201
Quoted 10 lines
arch/mips/lib/csum_partial.S +9
-0.Lmove_128bytes: +#ifdef CONFIG_CPU_HAS_PREFETCH + /* + * Fetch the block two iterations ahead, but only while it lies + * inside the buffer: on non-coherent systems a line prefetched + * past the end could belong to a buffer the device still owns. + */ + sltiu t5, t8, 3 + bnez t5, 2f + nop
I don't like this manual delay slot scheduling, but it seems like a case
for a larger clean-up of this file, so I'd be inclined to accept it.
arch/mips/lib/csum_partial.S EXPORT_SYMBOL(csum_partial) · line 205
Quoted 4 lines
arch/mips/lib/csum_partial.S +4
-0+ pref 0, 0x100(src) + pref 0, 0x120(src) + pref 0, 0x140(src) + pref 0, 0x160(src)
What about MIPSr6 though, which only has a span of -256..255 for the
immediate offset in the machine instruction here?
Maciej