floating-point error
Hello,
We're using Oracle 9iRAC on Linux I2 for a proof of concept and are having one issue.
Sometimes the Oracle process dies with the following error message:
Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault at ip 40000000048b4562
The ip address is always the same. This happens on all of our 4 nodes as it seems randomly. I do not have other debug info as this is the only message printed. Some times for the same process the message is printed up to 4 times.
Kernel used: 2.4.18-e.25 (RedHat update) based on 2.4.18 + a lot of patches making it difficult to match with David's releases.
The error message is a printk warning in traps.c.
Any idea on what can cause it, and how we could debug more the context ? Thanks in advance,
Bruno.
Signature · 4 lines
Linux Solution Consultant Tél: +33 476 143 278 - Fax: +33 476 146 105 HP/Intel Solution Center http://hpintelco.net Hewlett-Packard Grenoble/France Des infos sur Linux? http://www.HyPer-Linux.org http://www.hp.com/linux La musique ancienne? http://www.musique-ancienne.org http://www.medieval.org
Quoted 9 lines · Bruno Cornec
Sometimes the Oracle process dies with the following error message:
Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault at ip 40000000048b4562The ip address is always the same. This happens on all of our 4 nodes as it seems randomly. I do not have other debug info as this is the only message printed. Some times for the same process the message is printed up to 4 times.
Quoted 7 lines
Kernel used: 2.4.18-e.25 (RedHat update) based on 2.4.18 + a lot of patches making it difficult to match with David's releases.
The error message is a printk warning in
traps.c.Any idea on what can cause it, and how we could debug more the context ? Thanks in advance,
It would be helpful to apply the patch that Martin Hicks sent out awhile ago which will print the isr as well as the IP. You can then decode the isr to figure out exactly what's causing the fault in your app.
Jesse
Quoted 4 lines · Jesse Barnes
It would be helpful to apply the patch that Martin Hicks sent out awhile ago which will print the isr as well as the IP. You can then decode the isr to figure out exactly what's causing the fault in your app.
I couldn't find this in the archives, but I got it out of David's 2.5 tree, and applied it for 2.4 as well. Here it is if you want to try it:
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:
# ChangeSet 1.889.308.20 -> 1.889.308.21
# arch/ia64/kernel/traps.c 1.26 -> 1.26.1.1
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 03/03/18 [email protected] 1.889.308.21
# [PATCH] ia64: print ISR for FPSWA faults
#
# Here is a simple patch to also print isr during the handling of a
# floating point assist fault.
# --------------------------------------------
#
diff -Nru a/arch/ia64/kernel/traps.c b/arch/ia64/kernel/traps.c
--- a/arch/ia64/kernel/traps.c Thu Apr 17 13:41:11 2003
+++ b/arch/ia64/kernel/traps.c Thu Apr 17 13:41:11 2003
@@ -336,8 +336,8 @@
fpu_swa_count = 0;
if ((++fpu_swa_count < 5) && !(current->thread.flags & IA64_THREAD_FPEMU_NOPRINT)) {
last_time = jiffies;
- printk(KERN_WARNING "%s(%d): floating-point assist fault at ip %016lx\n",
- current->comm, current->pid, regs->cr_iip + ia64_psr(regs)->ri);
+ printk(KERN_WARNING "%s(%d): floating-point assist fault at ip %016lx, isr %016lx\n",
+ current->comm, current->pid, regs->cr_iip + ia64_psr(regs)->ri, isr);
} exception = fp_emulate(fp_fault, bundle, ®s->cr_ipsr, ®s->ar_fpsr, &isr, ®s->pr,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Bruno Cornec
Bruno> Hello,
Bruno> We're using Oracle 9iRAC on Linux I2 for a proof of concept and are having Bruno> one issue.
Bruno> Sometimes the Oracle process dies with the following error message:
Bruno> Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault at ip 40000000048b4562
Bruno> The ip address is always the same. This happens on all of our Bruno> 4 nodes as it seems randomly. I do not have other debug info Bruno> as this is the only message printed. Some times for the same Bruno> process the message is printed up to 4 times.
Bruno> [snip...]
Bruno> Any idea on what can cause it, and how we could debug more Bruno> the context ?
The message is informational. There is probably nothing wrong with the application, at least if the message occurs only rarely (at an interval less than 4 messages every 10 seconds or so). If the message occurs all the time, then there could be a performance problem with the application.
You should be able to turn of the message with "prctl --fpemul=silent".
--david
Great, thanks Bjorn.
Jesse
Quoted 40 lines · Bjorn Helgaas
Quoted 4 lines · Jesse Barnes
It would be helpful to apply the patch that Martin Hicks sent out awhile ago which will print the isr as well as the IP. You can then decode the isr to figure out exactly what's causing the fault in your app.
I couldn't find this in the archives, but I got it out of David's 2.5 tree, and applied it for 2.4 as well. Here it is if you want to try it:
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:
# ChangeSet 1.889.308.20 -> 1.889.308.21
#arch/ia64/kernel/traps.c1.26 -> 1.26.1.1
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 03/03/18 [email protected] 1.889.308.21
# [PATCH] ia64: print ISR for FPSWA faults
#
# Here is a simple patch to also print isr during the handling of a# floating point assist fault. # -------------------------------------------- # diff -Nru a/arch/ia64/kernel/traps.c b/arch/ia64/kernel/traps.carch/ia64/kernel/traps.c +2
-2@@ -336,8 +336,8 @@ fpu_swa_count =0 ;if ((++fpu_swa_count <5 ) && !(current->thread.flags &IA64_THREAD_FPEMU_NOPRINT )) { last_time = jiffies; -printk (KERN_WARNING "%s(%d): floating-point assist fault at ip %016lx\n" , - current->comm, current->pid, regs->cr_iip +ia64_psr (regs)->ri); +printk (KERN_WARNING "%s(%d): floating-point assist fault at ip %016lx, isr %016lx\n" , + current->comm, current->pid, regs->cr_iip +ia64_psr (regs)->ri, isr); } exception = fp_emulate(fp_fault, bundle, ®s->cr_ipsr, ®s->ar_fpsr, &isr, ®s->pr,
Quoted 8 lines
Sometimes the Oracle process dies with the following error message:
Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault at ip 40000000048b4562The ip address is always the same. This happens on all of our 4 nodes as it seems randomly. I do not have other debug info as this is the only message printed. Some times for the same process the message is printed up to 4 times.
Are you certain that the message is related to the death of the process?
This message is a warning to let you know that your application has run into one of the corner cases of IEEE floating point that is not implemented in hardware by the processor (typically operations involving denormalized numbers will cause this, but there may be other cases). There is rate limiting code in the kernel to prevent this message from flooding the logs (and from becoming even more of a performance drag than taking a trap and emulating in s/w).
It is relatively normal to see this message (even multiple times from the same process), and it usually isn't fatal.
-Tony Luck
Hello,
Thanks a lot David for your quick reply (as well as others).
Quoted 6 lines · David Mosberger
Bruno> Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault at ip
40000000048b4562The message is informational. There is probably nothing wrong with the application, at least if the message occurs only rarely (at an interval less than 4 messages every 10 seconds or so).
Which is the case.
But on the other hand we're loosing Oracle connections. So I suppose that the information means something in that case.
Quoted 5 lines
If the message
occurs all the time, then there could be a performance problem with the application.You should be able to turn of the message with "prctl --fpemul=silent".
Ok, will try that tomorrow.
Best regards,
Bruno.
PS: We may also try to recompile our own kernel based on your latest patch
+ the one given in the other messages of the thread to see if it changes anything.
Signature · 4 lines
Linux Solution Consultant Tél: +33 476 143 278 - Fax: +33 476 146 105 HP/Intel Solution Center http://hpintelco.net Hewlett-Packard Grenoble/France Des infos sur Linux? http://www.HyPer-Linux.org http://www.hp.com/linux La musique ancienne? http://www.musique-ancienne.org http://www.medieval.org
Oracle has its own exception handler. It usually dumps out more information in its alert file. Look in $ORACLE_HOME/rdbms/log/*.log for more clue.
- Ken
-----Original Message-----
From: Luck, Tony
Sent: Thursday, April 17, 2003 1:40 PM
To: Bruno Cornec; [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: RE: [Linux-ia64] floating-point error
Quoted 3 lines
Sometimes the Oracle process dies with the following error message:
Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault
at ip 40000000048b4562
Quoted 3 lines
The ip address is always the same. This happens on all of our 4 nodes as it seems randomly. I do not have other debug info as this is the
only
Quoted 1 line
message printed. Some times for the same process the message is
printed
Quoted 1 line
up to 4 times.
Are you certain that the message is related to the death of the process?
This message is a warning to let you know that your application has run into
one of the corner cases of IEEE floating point that is not implemented in
hardware by the processor (typically operations involving denormalized numbers
will cause this, but there may be other cases). There is rate limiting code in
the kernel to prevent this message from flooding the logs (and from becoming
even more of a performance drag than taking a trap and emulating in s/w).
It is relatively normal to see this message (even multiple times from the
same process), and it usually isn't fatal.
-Tony Luck
_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64
we did a little shell script to watch kernel messages and do a 'ps ax' on the pid, and saw oracle proceses were not running after the message was printk'ed (at least not for much time).
the floating-poin assist fault was only trigered by oracle processes, but oracle was the only heavily used on this machines during the test.
by the way it happens on all four nodes we're running, the firmware is up to date and includes the FPSWA, but it might be affected by system load, since in some cases it would be affecting about 10 % of the oracle processes while on other tests it may be seen affecting below 1%.
(i'm not on the list)
m.
Quoted 23 lines · Tony
Quoted 8 lines
Sometimes the Oracle process dies with the following error message:
Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault at ip 40000000048b4562The ip address is always the same. This happens on all of our 4 nodes as it seems randomly. I do not have other debug info as this is the only message printed. Some times for the same process the message is printed up to 4 times.
Are you certain that the message is related to the death of the process?
This message is a warning to let you know that your application has run into one of the corner cases of IEEE floating point that is not implemented in hardware by the processor (typically operations involving denormalized numbers will cause this, but there may be other cases). There is rate limiting code in the kernel to prevent this message from flooding the logs (and from becoming even more of a performance drag than taking a trap and emulating in s/w).
It is relatively normal to see this message (even multiple times from the same process), and it usually isn't fatal.
-Tony Luck
it does not write anything about this on the alert log, and neither does it on a trace file.
m.
Quoted 6 lines · Kenneth W
Oracle has its own exception handler. It usually dumps out more information in its alert file. Look in $ORACLE_HOME/rdbms/log/*.log for more clue.
- Ken