Skip to contentSkip to conversation

Oops in 2.6.10-ac12 in kjournald (journal_commit_transaction)

DISCUSSION Ralf Hildebrandt 5 participants 10 messages Feb 15, 2005 – Feb 17, 2005

lore
Share & tools
lore
#0 lore raw

Today our mailserver froze after just one day of uptime. I was able to capture the Oops on the screen using my digital camera:

http://www.stahl.bau.tu-bs.de/~hildeb/bugreport/

Keywords: EIP is at journal_commit_transaction, process kjournald

# mount
/dev/cciss/c0d0p6 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/cciss/c0d0p5 on /boot type ext3 (rw)
/dev/shm on /var/amavis type tmpfs (rw,noatime,size=200m,mode=770,uid=104,gid=108)
Signature · 4 lines
Ralf Hildebrandt (i.A. des IT-Zentrum)          [email protected]
Charite - Universitätsmedizin Berlin            Tel.  +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin    Fax.  +49 (0)30-450 570-962
IT-Zentrum Standort CBF                 send no mail to [email protected]
#1 lore raw

Hello,

Quoted 6 lines

Today our mailserver froze after just one day of uptime. I was able to capture the Oops on the screen using my digital camera:

http://www.stahl.bau.tu-bs.de/~hildeb/bugreport/

Keywords: EIP is at journal_commit_transaction, process kjournald

I guess the system is SMP... Sadly a few lines in the beginning of the
report are missing (probably scrolled off the screen) but it seems similar like a several other oopses I've seen reported recently. Is this the first time you hit this bug?

Quoted 8 lines
# mount
/dev/cciss/c0d0p6 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/cciss/c0d0p5 on /boot type ext3 (rw)
/dev/shm on /var/amavis type tmpfs (rw,noatime,size=200m,mode=770,uid=104,gid=108)

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
#2 lore raw

* Jan Kara <[email protected]>:

Quoted 1 line

I guess the system is SMP...

Indeed it is. Dual Xeon with SMP.

Quoted 2 lines

Sadly a few lines in the beginning of the
report are missing (probably scrolled off the screen)

Yes, this sucks. I rebooted with vesafb active, no I do have 50 lines :)

Quoted 2 lines

but it seems similar like a several other oopses I've seen reported recently. Is this the first time you hit this bug?

It's actually the second time. The first time it hit the SAME box but with kernel-2.6.10 (vanilla) after 30 days of uptime. Nobody had a camera at hand, so I couldn't take a photo.

Any suggestions? I'm open to suggestions. One difference between the 2.6.10 and 2.6.10-ac12 was that 2.6.10 has no in-kernel irq balancing, while in 2.6.10-ac12 I acivated that.

Signature · 4 lines
Ralf Hildebrandt (i.A. des IT-Zentrum)          [email protected]
Charite - Universitätsmedizin Berlin            Tel.  +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin    Fax.  +49 (0)30-450 570-962
IT-Zentrum Standort CBF                 send no mail to [email protected]
#3 lore raw
Quoted 6 lines · Ralf Hildebrandt

* Jan Kara <[email protected]>:

Quoted 1 line

I guess the system is SMP...

Indeed it is. Dual Xeon with SMP.

This looks very similar (at least to me) to an OOPS I posted with 2.6.9 on 12/03/2004.
http://marc.theaimsgroup.com/?l=linux-kernel&m=110210705504716&w=2

My system is also a dual Xeon using SMP and Hyperthreading (/proc/cpuinfo shows 4 cpus).
Mine, like Ralf's, is also a mail server running postfix using ext3 for the spool directory.

Quoted 7 lines
Quoted 2 lines

but it seems similar like a several other oopses I've seen reported recently. Is this the first time you hit this bug?

It's actually the second time. The first time it hit the SAME box but with kernel-2.6.10 (vanilla) after 30 days of uptime. Nobody had a camera at hand, so I couldn't take a photo.

I've actually hit this bug (assuming it's the same) with 2.6.10 also. I had to power cycle remotely and unfortunately didn't have the serial console logging enabled when it happened with 2.6.10. I upgraded from 2.4.23 to 2.6.8.1 and crashed within a week, and continued to crash at least monthly after that. It had been running 2.4.23 for 200+ days with no problems.

Hope this helps trace it back.

Dale

#4 lore raw

* Dale Blount <[email protected]>:

Quoted 3 lines

This looks very similar (at least to me) to an OOPS I posted with 2.6.9 on 12/03/2004.
http://marc.theaimsgroup.com/?l=linux-kernel&m=110210705504716&w=2

Could be.

Quoted 2 lines

My system is also a dual Xeon using SMP and Hyperthreading (/proc/cpuinfo shows 4 cpus).

Same system here.

Quoted 2 lines

Mine, like Ralf's, is also a mail server running postfix using ext3 for the spool directory.

Same here.

Quoted 8 lines

I've actually hit this bug (assuming it's the same) with 2.6.10 also. I had to power cycle remotely and unfortunately didn't have the serial console logging enabled when it happened with 2.6.10. I upgraded from 2.4.23 to 2.6.8.1 and crashed within a week, and continued to crash at least monthly after that. It had been running 2.4.23 for 200+ days with no problems.

Hope this helps trace it back.

Me too

Signature · 4 lines
Ralf Hildebrandt (i.A. des IT-Zentrum)          [email protected]
Charite - Universitätsmedizin Berlin            Tel.  +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin    Fax.  +49 (0)30-450 570-962
IT-Zentrum Standort CBF                 send no mail to [email protected]
#5 lore raw
In reply to Dale Blount
Quoted 4 lines · Dale Blount

This looks very similar (at least to me) to an OOPS I posted with 2.6.9 on 12/03/2004.
http://marc.theaimsgroup.com/?l=linux-kernel&m=110210705504716&w=2

There have been a handful of reports - there's surely a race in there.

Unfortunately I've yet to see a report from which we can identify the offending line in the very large journal_commit_transaction() function.

The best way to do that is to ensure that the kernel was built with CONFIG_DEBUG_INFO, note the offending EIP value, then do

# gdb vmlinux
(gdb) l *0xc0<whatever>
#6 lore raw

* Andrew Morton <[email protected]>:

Quoted 4 lines

There have been a handful of reports - there's surely a race in there.

Unfortunately I've yet to see a report from which we can identify the offending line in the very large journal_commit_transaction() function.

:(

Quoted 6 lines

The best way to do that is to ensure that the kernel was built with CONFIG_DEBUG_INFO, note the offending EIP value, then do

# gdb vmlinux
(gdb) l *0xc0<whatever>

I'm rebuilding the ac12 kernel which crashed on me after just one day and will reboot it today.

Signature · 4 lines
Ralf Hildebrandt (i.A. des IT-Zentrum)          [email protected]
Charite - Universitätsmedizin Berlin            Tel.  +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin    Fax.  +49 (0)30-450 570-962
IT-Zentrum Standort CBF                 send no mail to [email protected]
#7 lore raw

* Ralf Hildebrandt <[email protected]>:

Quoted 8 lines
Quoted 5 lines

The best way to do that is to ensure that the kernel was built with CONFIG_DEBUG_INFO, note the offending EIP value, then do

# gdb vmlinux
(gdb) l *0xc0<whatever>

I'm rebuilding the ac12 kernel which crashed on me after just one day and will reboot it today.

Is it normal that the kernel with debugging enabled is not larger than the normal kernel?

#8 lore raw
Quoted 9 lines · Ralf Hildebrandt

* Ralf Hildebrandt <[email protected]>:

Quoted 6 lines
Quoted 5 lines

The best way to do that is to ensure that the kernel was built with CONFIG_DEBUG_INFO, note the offending EIP value, then do

# gdb vmlinux
(gdb) l *0xc0<whatever>
Quoted 7 lines
Quoted 2 lines

I'm rebuilding the ac12 kernel which crashed on me after just one day and will reboot it today.

Is it normal that the kernel with debugging enabled is not larger than the normal kernel?
-

No, it should be much larger. Recheck the .config file for CONFIG_DEBUG_INFO=y. Maybe you need to do 'make clean' first.

Signature · 1 line
~Randy
#9 lore raw

* Randy.Dunlap <[email protected]>:

Quoted 7 lines
Quoted 3 lines

Is it normal that the kernel with debugging enabled is not larger than the normal kernel?
-

No, it should be much larger. Recheck the .config file for CONFIG_DEBUG_INFO=y. Maybe you need to do 'make clean' first.

CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_INFO=y
# CONFIG_FRAME_POINTER is not set
CONFIG_EARLY_PRINTK=y

I built that using "make-kpkg"

make-kpkg clean
CONCURRENCY_LEVEL=4 MAKEFLAGS="CC=gcc-3.4" make-kpkg --revision=20050217 kernel_image

Signature · 4 lines
Ralf Hildebrandt (i.A. des IT-Zentrum)          [email protected]
Charite - Universitätsmedizin Berlin            Tel.  +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin    Fax.  +49 (0)30-450 570-962
IT-Zentrum Standort CBF                 send no mail to [email protected]