Skip to contentSkip to conversation

floating-point error

Linux-ia64 Bruno Cornec 7 participants 10 messages Apr 17, 2003

lore
Share & tools
lore
#0 lore raw

Hello,

We're using Oracle 9iRAC on Linux I2 for a proof of concept and are having one issue.

Sometimes the Oracle process dies with the following error message:

Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault at ip 40000000048b4562

The ip address is always the same. This happens on all of our 4 nodes as it seems randomly. I do not have other debug info as this is the only message printed. Some times for the same process the message is printed up to 4 times.

Kernel used: 2.4.18-e.25 (RedHat update) based on 2.4.18 + a lot of patches making it difficult to match with David's releases.

The error message is a printk warning in traps.c.

Any idea on what can cause it, and how we could debug more the context ? Thanks in advance,

Bruno.

Signature · 4 lines
Linux Solution Consultant         Tél: +33 476 143 278 - Fax: +33 476 146 105
HP/Intel Solution Center http://hpintelco.net Hewlett-Packard Grenoble/France
Des infos sur Linux?  http://www.HyPer-Linux.org      http://www.hp.com/linux
La musique ancienne?  http://www.musique-ancienne.org http://www.medieval.org
#1 lore raw
Quoted 9 lines · Bruno Cornec

Sometimes the Oracle process dies with the following error message:

Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault at ip 40000000048b4562

The ip address is always the same. This happens on all of our 4 nodes as it seems randomly. I do not have other debug info as this is the only message printed. Some times for the same process the message is printed up to 4 times.

Quoted 7 lines

Kernel used: 2.4.18-e.25 (RedHat update) based on 2.4.18 + a lot of patches making it difficult to match with David's releases.

The error message is a printk warning in traps.c.

Any idea on what can cause it, and how we could debug more the context ? Thanks in advance,

It would be helpful to apply the patch that Martin Hicks sent out awhile ago which will print the isr as well as the IP. You can then decode the isr to figure out exactly what's causing the fault in your app.

Jesse

#2 lore raw
In reply to Bruno Cornec
Quoted 4 lines · Jesse Barnes

It would be helpful to apply the patch that Martin Hicks sent out awhile ago which will print the isr as well as the IP. You can then decode the isr to figure out exactly what's causing the fault in your app.

I couldn't find this in the archives, but I got it out of David's 2.5 tree, and applied it for 2.4 as well. Here it is if you want to try it:

# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:
# ChangeSet 1.889.308.20 -> 1.889.308.21
# arch/ia64/kernel/traps.c 1.26 -> 1.26.1.1
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 03/03/18 [email protected] 1.889.308.21
# [PATCH] ia64: print ISR for FPSWA faults
#
# Here is a simple patch to also print isr during the handling of a

# floating point assist fault.
# --------------------------------------------
#
diff -Nru a/arch/ia64/kernel/traps.c b/arch/ia64/kernel/traps.c
--- a/arch/ia64/kernel/traps.c	Thu Apr 17 13:41:11 2003
+++ b/arch/ia64/kernel/traps.c	Thu Apr 17 13:41:11 2003
@@ -336,8 +336,8 @@
 		fpu_swa_count = 0;
 	if ((++fpu_swa_count < 5) && !(current->thread.flags & IA64_THREAD_FPEMU_NOPRINT)) {
 		last_time = jiffies;
-		printk(KERN_WARNING "%s(%d): floating-point assist fault at ip %016lx\n",
-		       current->comm, current->pid, regs->cr_iip + ia64_psr(regs)->ri);
+		printk(KERN_WARNING "%s(%d): floating-point assist fault at ip %016lx, isr %016lx\n",
+		       current->comm, current->pid, regs->cr_iip + ia64_psr(regs)->ri, isr);
 	}

exception = fp_emulate(fp_fault, bundle, &regs->cr_ipsr, &regs->ar_fpsr, &isr, &regs->pr,

#3 lore raw
In reply to Bruno Cornec
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Bruno Cornec

Bruno> Hello,
Bruno> We're using Oracle 9iRAC on Linux I2 for a proof of concept and are having Bruno> one issue.

Bruno> Sometimes the Oracle process dies with the following error message:

Bruno> Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault at ip 40000000048b4562

Bruno> The ip address is always the same. This happens on all of our Bruno> 4 nodes as it seems randomly. I do not have other debug info Bruno> as this is the only message printed. Some times for the same Bruno> process the message is printed up to 4 times.

Bruno> [snip...]

Bruno> Any idea on what can cause it, and how we could debug more Bruno> the context ?

The message is informational. There is probably nothing wrong with the application, at least if the message occurs only rarely (at an interval less than 4 messages every 10 seconds or so). If the message occurs all the time, then there could be a performance problem with the application.

You should be able to turn of the message with "prctl --fpemul=silent".

	--david
#4 lore raw
In reply to Bruno Cornec

Great, thanks Bjorn.

Jesse

Quoted 40 lines · Bjorn Helgaas
Quoted 4 lines · Jesse Barnes

It would be helpful to apply the patch that Martin Hicks sent out awhile ago which will print the isr as well as the IP. You can then decode the isr to figure out exactly what's causing the fault in your app.

I couldn't find this in the archives, but I got it out of David's 2.5 tree, and applied it for 2.4 as well. Here it is if you want to try it:

# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:
# ChangeSet 1.889.308.20 -> 1.889.308.21
# arch/ia64/kernel/traps.c 1.26 -> 1.26.1.1
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 03/03/18 [email protected] 1.889.308.21
# [PATCH] ia64: print ISR for FPSWA faults
#
# Here is a simple patch to also print isr during the handling of a

# floating point assist fault.
# --------------------------------------------
#
diff -Nru a/arch/ia64/kernel/traps.c b/arch/ia64/kernel/traps.c
arch/ia64/kernel/traps.c +2 -2
@@ -336,8 +336,8 @@
 		fpu_swa_count = 0;
 	if ((++fpu_swa_count < 5) && !(current->thread.flags & IA64_THREAD_FPEMU_NOPRINT)) {
 		last_time = jiffies;
-		printk(KERN_WARNING "%s(%d): floating-point assist fault at ip %016lx\n",
-		       current->comm, current->pid, regs->cr_iip + ia64_psr(regs)->ri);
+		printk(KERN_WARNING "%s(%d): floating-point assist fault at ip %016lx, isr %016lx\n",
+		       current->comm, current->pid, regs->cr_iip + ia64_psr(regs)->ri, isr);
 	}
 
 	exception = fp_emulate(fp_fault, bundle, &regs->cr_ipsr, &regs->ar_fpsr, &isr, &regs->pr,
#5 lore raw
In reply to Bruno Cornec
Quoted 8 lines

Sometimes the Oracle process dies with the following error message:

Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault at ip 40000000048b4562

The ip address is always the same. This happens on all of our 4 nodes as it seems randomly. I do not have other debug info as this is the only message printed. Some times for the same process the message is printed up to 4 times.

Are you certain that the message is related to the death of the process?

This message is a warning to let you know that your application has run into one of the corner cases of IEEE floating point that is not implemented in hardware by the processor (typically operations involving denormalized numbers will cause this, but there may be other cases). There is rate limiting code in the kernel to prevent this message from flooding the logs (and from becoming even more of a performance drag than taking a trap and emulating in s/w).

It is relatively normal to see this message (even multiple times from the same process), and it usually isn't fatal.

-Tony Luck

#6 lore raw
In reply to Bruno Cornec

Hello,

Thanks a lot David for your quick reply (as well as others).

Quoted 6 lines · David Mosberger

Bruno> Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault at ip 40000000048b4562

The message is informational. There is probably nothing wrong with the application, at least if the message occurs only rarely (at an interval less than 4 messages every 10 seconds or so).

Which is the case.
But on the other hand we're loosing Oracle connections. So I suppose that the information means something in that case.

Quoted 5 lines

If the message
occurs all the time, then there could be a performance problem with the application.

You should be able to turn of the message with "prctl --fpemul=silent".

Ok, will try that tomorrow.

Best regards,
Bruno.

PS: We may also try to recompile our own kernel based on your latest patch
+ the one given in the other messages of the thread to see if it changes anything.

Signature · 4 lines
Linux Solution Consultant         Tél: +33 476 143 278 - Fax: +33 476 146 105
HP/Intel Solution Center http://hpintelco.net Hewlett-Packard Grenoble/France
Des infos sur Linux?  http://www.HyPer-Linux.org      http://www.hp.com/linux
La musique ancienne?  http://www.musique-ancienne.org http://www.medieval.org
#7 lore raw
In reply to Bruno Cornec

Oracle has its own exception handler. It usually dumps out more information in its alert file. Look in $ORACLE_HOME/rdbms/log/*.log for more clue.

- Ken

-----Original Message-----
From: Luck, Tony
Sent: Thursday, April 17, 2003 1:40 PM
To: Bruno Cornec; [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: RE: [Linux-ia64] floating-point error

Quoted 3 lines

Sometimes the Oracle process dies with the following error message:

Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault

at ip 40000000048b4562

Quoted 3 lines

The ip address is always the same. This happens on all of our 4 nodes as it seems randomly. I do not have other debug info as this is the

only

Quoted 1 line

message printed. Some times for the same process the message is

printed

Quoted 1 line

up to 4 times.

Are you certain that the message is related to the death of the process?

This message is a warning to let you know that your application has run into
one of the corner cases of IEEE floating point that is not implemented in
hardware by the processor (typically operations involving denormalized numbers
will cause this, but there may be other cases). There is rate limiting code in
the kernel to prevent this message from flooding the logs (and from becoming
even more of a performance drag than taking a trap and emulating in s/w).

It is relatively normal to see this message (even multiple times from the
same process), and it usually isn't fatal.

-Tony Luck

_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64

#8 lore raw
In reply to Bruno Cornec

we did a little shell script to watch kernel messages and do a 'ps ax' on the pid, and saw oracle proceses were not running after the message was printk'ed (at least not for much time).
the floating-poin assist fault was only trigered by oracle processes, but oracle was the only heavily used on this machines during the test.
by the way it happens on all four nodes we're running, the firmware is up to date and includes the FPSWA, but it might be affected by system load, since in some cases it would be affecting about 10 % of the oracle processes while on other tests it may be seen affecting below 1%.

(i'm not on the list)

m.

Quoted 23 lines · Tony
Quoted 8 lines

Sometimes the Oracle process dies with the following error message:

Apr 17 20:24:48 rx1 kernel: oracle(7148): floating-point assist fault at ip 40000000048b4562

The ip address is always the same. This happens on all of our 4 nodes as it seems randomly. I do not have other debug info as this is the only message printed. Some times for the same process the message is printed up to 4 times.

Are you certain that the message is related to the death of the process?

This message is a warning to let you know that your application has run into one of the corner cases of IEEE floating point that is not implemented in hardware by the processor (typically operations involving denormalized numbers will cause this, but there may be other cases). There is rate limiting code in the kernel to prevent this message from flooding the logs (and from becoming even more of a performance drag than taking a trap and emulating in s/w).

It is relatively normal to see this message (even multiple times from the same process), and it usually isn't fatal.

-Tony Luck

#9 lore raw
In reply to Bruno Cornec

it does not write anything about this on the alert log, and neither does it on a trace file.

m.

Quoted 6 lines · Kenneth W

Oracle has its own exception handler. It usually dumps out more information in its alert file. Look in $ORACLE_HOME/rdbms/log/*.log for more clue.

- Ken