PROBLEM: 2.6 kernels on x86 do not preserve FPU flags across context switches
Hi,
I am the team lead and chief VM developer for a Smaltalk implementation based on a JIT execution engine. Our customers have been seeing rare incorrect floating-point results in intensive fp applications on 2.6 kernels using various x86 compatible processors. These problems do not occur on previous kernel versons. We recently had occasion to reimplement our fp primitives to avoid severe performance problems on Xeon processors that were traced to Xeon's relatively slow implementation of fnclex and fstsw. The older implementaton would produce a result and test for a valid (non NaN, non Inf) result by examining the FPU status flags via fstsw. The newer implementation produces a result and tests its exponent for the NaN/Inf exponent. The new implementation does not show the rare incorrect floating-point results in intensive fp applications on 2.6 kernels. My conclusion is that context switches between the production of the result and the execution of the fstsw are the culprit, and that the context switch machinery fails to preserve the FPU status flags.
I don't know whether any action on your part is appropriate. The use of the FPU status flags is presumably rare on linux (I believe that neither gcc nor glibc make use of them). But "exotic" execution machinery such as runtimes for dynamic or functional languages (language implementations that may not use IEEE arithmetic and instead flag Infs and NaNs as an error) may fall foul of this issue. Since previous versions of the kernel on x86 apparently do preserve the FPU status flags perhaps its simple to preserve the old behaviour. At the very least let me suggest you document the limitation.
Sincerely,
---
Eliot Miranda ,,,^..^,,, mailto:[email protected]
VisualWorks Engineering, Cincom Smalltalk: scene not herd Tel +1 408 216 4581 3350 Scott Blvd, Bldg 36 Suite B, Santa Clara, CA 95054 USA Fax +1 408 216 4500
Quoted 4 lines · On Wed, 16 Jun 2004 [email protected] wrote:
Hi,
I am the team lead and chief VM developer for a Smaltalk implementation based on a JIT execution engine. Our customers have been seeing rare incorrect floating-point results in intensive fp applications on 2.6 kernels using various x86 compatible processors. These problems do not occur on previous kernel versons. We recently had occasion to reimplement our fp primitives to avoid severe performance problems on Xeon processors that were traced to Xeon's relatively slow implementation of fnclex and fstsw. The older implementaton would produce a result and test for a valid (non NaN, non Inf) result by examining the FPU status flags via fstsw. The newer implementation produces a result and tests its exponent for the NaN/Inf exponent. The new implementation does not show the rare incorrect floating-point results in intensive fp applications on 2.6 kernels. My conclusion is that context switches between the production of the result and the execution of the fstsw are the culprit, and that the context switch machinery fails to preserve the FPU status flags.
Quoted 8 lines
I don't know whether any action on your part is appropriate. The use of the FPU status flags is presumably rare on linux (I believe that neither gcc nor glibc make use of them). But "exotic" execution machinery such as runtimes for dynamic or functional languages (language implementations that may not use IEEE arithmetic and instead flag Infs and NaNs as an error) may fall foul of this issue. Since previous versions of the kernel on x86 apparently do preserve the FPU status flags perhaps its simple to preserve the old behaviour. At the very least let me suggest you document the limitation.
Sincerely,
---
Eliot Miranda ,,,^..^,,, mailto:[email protected]
VisualWorks Engineering, Cincom Smalltalk: scene not herd Tel +1 408 216 4581 3350 Scott Blvd, Bldg 36 Suite B, Santa Clara, CA 95054 USA Fax +1 408 216 4500
All versions of the kernels preserve FPU state, including its flags across context-switches. They use the FNSAVE/FRSTOR pair which saves and restores the entire FPU environment including its flags. This is very expensive, taking roughly 130 clocks for each operation.
What they don't do is save/restore FPU state during system calls because it is extremely wasteful of preformance. So, if there is any module loaded that (improperly) uses the FPU, the user's FPU state can get trashed.
Also, I don't imagine you know what the [Enter] key does, do you? If you hit this occasionally, somebody might be able to read your text on a conventional terminal, rather than a Windows auto-warp contraption.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
Note 96.31% of all statistics are fiction.
Quoted 16 lines · Richard B. Johnson
Quoted 15 lines · On Wed, 16 Jun 2004 [email protected] wrote:
Hi,
I am the team lead and chief VM developer for a Smaltalk implementation
based on a JIT execution engine. Our customers have been seeing rare incorrect floating-point results in intensive fp applications on 2.6 kernels using various x86 compatible processors. These problems do not occur on previous kernel versons. We recently had occasion to reimplement our fp primitives to avoid severe performance problems on Xeon processors that were traced to Xeon's relatively slow implementation of fnclex and fstsw. The older implementaton would produce a result and test for a valid (non NaN, non Inf) result by examining the FPU status flags via fstsw. The newer implementation produces a result and tests its exponent for the NaN/Inf exponent. The new implementation does not show the rare incorrect floating-point results in intensive fp applications on 2.6 kernels. My conclusion is that context switches
wow. Perfect place to install debug code.
Check _both_ exponent and flags, if they disagree, yell. That may give some useful info about cause of this.
BTW, I didn't pay attention, but some FPU-related patches were floating around just recently. Check list archives.
Quoted 13 lines
Quoted 13 lines
between the production of the result and the execution of the fstsw are the culprit, and that the context switch machinery fails to preserve the FPU status flags.
I don't know whether any action on your part is appropriate. The use of the FPU status flags is presumably rare on linux (I believe that neither gcc nor glibc make use of them). But "exotic" execution machinery such as runtimes for dynamic or functional languages (language implementations that may not use IEEE arithmetic and instead flag Infs and NaNs as an error) may fall foul of this issue. Since previous versions of the kernel on x86 apparently do preserve the FPU status flags perhaps its simple to preserve the old behaviour. At the very least let me suggest you document the limitation.
Signature · 1 line
vda