Skip to contentSkip to conversation

2.6.15-git7 oopses in ext3 during LTP runs

DISCUSSION Andi Kleen 8 participants 15 messages Jan 11, 2006 – Jan 12, 2006 Sparse checker

lore
Share & tools
lore

Andi Kleen

author
#0 lore raw

Running LTP with the default runfile on a 4 virtual CPU x86-64 system gives

To reproduce: run ltp 20040908 (newer one will probably work too) with runltp -p -q -l uname -r on a ext3 file system

config is x86-64 defconfig.

-Andi

----------- [cut here ] --------- [please bite here ] ---------

Kernel BUG at /home/lsrc/quilt/linux/fs/ext3/super.c:2154
invalid opcode: 0000 [1] SMP 
CPU 0 
Modules linked in:
Pid: 14055, comm: pdflush Not tainted 2.6.15-git7 #90
RIP: 0010:[<ffffffff801d998e>] <ffffffff801d998e>{ext3_write_super+20}
RSP: 0018:ffff810117fb1e08  EFLAGS: 00010202
RAX: 0000000000000001 RBX: ffff81011f9ae800 RCX: ffffffff80492060
RDX: 0000000000000000 RSI: ffff810117fb0000 RDI: ffff81011f9ae888
RBP: ffff81011f9ae888 R08: ffff810117fb0000 R09: ffff810004f219e0
R10: 0000000000000000 R11: 0000000000000002 R12: ffff81011f9ae870
R13: ffffffff80159aee R14: ffff8100cdce1d68 R15: ffffffff80146b14
FS:  0000000000000000(0000) GS:ffffffff8060d000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00002aaaaab61000 CR3: 000000011baf3000 CR4: 00000000000006e0
Process pdflush (pid: 14055, threadinfo ffff810117fb0000, task ffff81011fe5c400)
Stack: ffff81011f9ae800 ffffffff8017aa43 0000000000000064 0000000000000000 
       ffff8100cdce1d78 ffffffff801592f9 ffff81011fe5c400 ffffffff80494000 
       0000000000000000 0000000000000000 
Call Trace: <ffffffff8017aa43>{sync_supers+133} <ffffffff801592f9>{wb_kupdate+49}
       <ffffffff80159aee>{pdflush+0} <ffffffff80159c40>{pdflush+338}
       <ffffffff801592c8>{wb_kupdate+0} <ffffffff80146c7f>{kthread+203}
       <ffffffff801107ca>{child_rip+8} <ffffffff80146b14>{keventd_create_kthread+0}
       <ffffffff80146bb4>{kthread+0} <ffffffff801107c2>{child_rip+0}

Code: 0f 0b 68 e9 0d 43 80 c2 6a 08 c6 43 21 00 5b c3 49 c7 c0 ca 

RIP <ffffffff801d998e>{ext3_write_super+20} RSP <ffff810117fb1e08>
Badness in do_exit at /home/lsrc/quilt/linux/kernel/exit.c:796

Call Trace: <ffffffff80136800>{do_exit+84} <ffffffff804024dd>{_spin_unlock_irqrestore+8}
       <ffffffff80146b14>{keventd_create_kthread+0} <ffffffff801114ef>{kernel_math_error+0}
       <ffffffff80159aee>{pdflush+0} <ffffffff80111c61>{do_invalid_op+163}
       <ffffffff801d998e>{ext3_write_super+20} <ffffffff8040116a>{thread_return+0}
       <ffffffff80110611>{error_exit+0} <ffffffff80146b14>{keventd_create_kthread+0}
       <ffffffff80159aee>{pdflush+0} <ffffffff801d998e>{ext3_write_super+20}
       <ffffffff801d998a>{ext3_write_super+16} <ffffffff8017aa43>{sync_supers+133}
       <ffffffff801592f9>{wb_kupdate+49} <ffffffff80159aee>{pdflush+0}
       <ffffffff80159c40>{pdflush+338} <ffffffff801592c8>{wb_kupdate+0}
       <ffffffff80146c7f>{kthread+203} <ffffffff801107ca>{child_rip+8}
       <ffffffff80146b14>{keventd_create_kthread+0} <ffffffff80146bb4>{kthread+0}
       <ffffffff801107c2>{child_rip+0}
#1 lore raw
Quoted 10 lines · Andi Kleen

Running LTP with the default runfile on a 4 virtual CPU x86-64 system gives

To reproduce: run ltp 20040908 (newer one will probably work too) with runltp -p -q -l uname -r on a ext3 file system

config is x86-64 defconfig.

mutex_trylock() is returning the wrong value. fs/super.c:write_super() clearly took the lock.

Ingo, weren't you hitting this occasionally?

Quoted 43 lines

----------- [cut here ] --------- [please bite here ] ---------

Kernel BUG at /home/lsrc/quilt/linux/fs/ext3/super.c:2154
invalid opcode: 0000 [1] SMP 
CPU 0 
Modules linked in:
Pid: 14055, comm: pdflush Not tainted 2.6.15-git7 #90
RIP: 0010:[<ffffffff801d998e>] <ffffffff801d998e>{ext3_write_super+20}
RSP: 0018:ffff810117fb1e08  EFLAGS: 00010202
RAX: 0000000000000001 RBX: ffff81011f9ae800 RCX: ffffffff80492060
RDX: 0000000000000000 RSI: ffff810117fb0000 RDI: ffff81011f9ae888
RBP: ffff81011f9ae888 R08: ffff810117fb0000 R09: ffff810004f219e0
R10: 0000000000000000 R11: 0000000000000002 R12: ffff81011f9ae870
R13: ffffffff80159aee R14: ffff8100cdce1d68 R15: ffffffff80146b14
FS:  0000000000000000(0000) GS:ffffffff8060d000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00002aaaaab61000 CR3: 000000011baf3000 CR4: 00000000000006e0
Process pdflush (pid: 14055, threadinfo ffff810117fb0000, task ffff81011fe5c400)
Stack: ffff81011f9ae800 ffffffff8017aa43 0000000000000064 0000000000000000 
       ffff8100cdce1d78 ffffffff801592f9 ffff81011fe5c400 ffffffff80494000 
       0000000000000000 0000000000000000 
Call Trace: <ffffffff8017aa43>{sync_supers+133} <ffffffff801592f9>{wb_kupdate+49}
       <ffffffff80159aee>{pdflush+0} <ffffffff80159c40>{pdflush+338}
       <ffffffff801592c8>{wb_kupdate+0} <ffffffff80146c7f>{kthread+203}
       <ffffffff801107ca>{child_rip+8} <ffffffff80146b14>{keventd_create_kthread+0}
       <ffffffff80146bb4>{kthread+0} <ffffffff801107c2>{child_rip+0}

Code: 0f 0b 68 e9 0d 43 80 c2 6a 08 c6 43 21 00 5b c3 49 c7 c0 ca 

RIP <ffffffff801d998e>{ext3_write_super+20} RSP <ffff810117fb1e08>
Badness in do_exit at /home/lsrc/quilt/linux/kernel/exit.c:796

Call Trace: <ffffffff80136800>{do_exit+84} <ffffffff804024dd>{_spin_unlock_irqrestore+8}
       <ffffffff80146b14>{keventd_create_kthread+0} <ffffffff801114ef>{kernel_math_error+0}
       <ffffffff80159aee>{pdflush+0} <ffffffff80111c61>{do_invalid_op+163}
       <ffffffff801d998e>{ext3_write_super+20} <ffffffff8040116a>{thread_return+0}
       <ffffffff80110611>{error_exit+0} <ffffffff80146b14>{keventd_create_kthread+0}
       <ffffffff80159aee>{pdflush+0} <ffffffff801d998e>{ext3_write_super+20}
       <ffffffff801d998a>{ext3_write_super+16} <ffffffff8017aa43>{sync_supers+133}
       <ffffffff801592f9>{wb_kupdate+49} <ffffffff80159aee>{pdflush+0}
       <ffffffff80159c40>{pdflush+338} <ffffffff801592c8>{wb_kupdate+0}
       <ffffffff80146c7f>{kthread+203} <ffffffff801107ca>{child_rip+8}
       <ffffffff80146b14>{keventd_create_kthread+0} <ffffffff80146bb4>{kthread+0}
       <ffffffff801107c2>{child_rip+0}
#2 lore raw
Quoted 6 lines · Andrew Morton
Quoted 5 lines · Andi Kleen

Running LTP with the default runfile on a 4 virtual CPU x86-64 system gives

Quoted 8 lines
Quoted 5 lines

To reproduce: run ltp 20040908 (newer one will probably work too) with runltp -p -q -l uname -r on a ext3 file system

config is x86-64 defconfig.

mutex_trylock() is returning the wrong value. fs/super.c:write_super() clearly took the lock.

the conversion is buggy.

mutex_trylock has the same convention as spin_try_lock (which is the opposite of down_trylock). THe conversion forgot to add a !

fs/ext3/super.c renamed from fs/ext3/super.c~ +1 -1
@@ -2150,7 +2150,7 @@
 
 static void ext3_write_super (struct super_block * sb)
 {
-	if (mutex_trylock(&sb->s_lock) != 0)
+	if (!mutex_trylock(&sb->s_lock) != 0)
 		BUG();
 	sb->s_dirt = 0;
 }
#3 lore raw
Quoted 26 lines · Arjan van de Ven
Quoted 6 lines · Andrew Morton
Quoted 5 lines · Andi Kleen

Running LTP with the default runfile on a 4 virtual CPU x86-64 system gives

Quoted 8 lines
Quoted 5 lines

To reproduce: run ltp 20040908 (newer one will probably work too) with runltp -p -q -l uname -r on a ext3 file system

config is x86-64 defconfig.

mutex_trylock() is returning the wrong value. fs/super.c:write_super() clearly took the lock.

the conversion is buggy.

mutex_trylock has the same convention as spin_try_lock (which is the opposite of down_trylock). THe conversion forgot to add a !

fs/ext3/super.c line 2156

Quoted 8 lines
fs/ext3/super.c renamed from fs/ext3/super.c~ +1 -1
@@ -2150,7 +2150,7 @@
 
 static void ext3_write_super (struct super_block * sb)
 {
-	if (mutex_trylock(&sb->s_lock) != 0)
+	if (!mutex_trylock(&sb->s_lock) != 0)
 		BUG();
 	sb->s_dirt = 0;
 }

We expect the lock to be held on entry. Hence we expect mutex_trylock() to return zero.

#14 lore raw
In reply to Arjan van de Ven

fs/ext3/super.c line 2153

Quoted 9 lines

the conversion is buggy.

mutex_trylock has the same convention as spin_try_lock (which is the opposite of down_trylock). THe conversion forgot to add a !

static void ext3_write_super (struct super_block * sb) {

fs/ext3/super.c renamed from fs/ext3/super.c~ +1 -1
-	if (mutex_trylock(&sb->s_lock) != 0)
+	if (!mutex_trylock(&sb->s_lock) != 0)

How about a nicer...?

        if(mutex_trylock(&sb->s_lock) == 0)

Jan Engelhardt

#4 lore raw
In reply to Andrew Morton
Quoted 4 lines

15/fs/ext3/super.c~ 2006-01-11 21:54:13.000000000 +0100

Quoted 3 lines

+++ linux-2.6.15/fs/ext3/super.c 2006-01-11 21:54:13.000000000 +0100

(quoted diff) +0 -0
@@ -2150,7 +2150,7 @@
 
Quoted 10 lines

Review line 2156

Quoted 7 lines
(quoted diff) +1 -1
 static void ext3_write_super (struct super_block * sb)
 {
-	if (mutex_trylock(&sb->s_lock) != 0)
+	if (!mutex_trylock(&sb->s_lock) != 0)
 		BUG();
 	sb->s_dirt = 0;
 }

We expect the lock to be held on entry. Hence we expect mutex_trylock() to return zero.

you are correct, and the x86-64 mutex.h is buggy

include/asm-x86_64/mutex.h renamed from include/asm-x86_64/mutex.h.org +1 -1
@@ -104,7 +104,7 @@
 static inline int
 __mutex_fastpath_trylock(atomic_t *count, int (*fail_fn)(atomic_t *))
 {
-	if (likely(atomic_cmpxchg(count, 1, 0)) == 1)
+	if (likely(atomic_cmpxchg(count, 1, 0) == 1))
 		return 1;
 	else
 		return 0;

changes the asm to be the correct one for me.
This is odd/evil though..

#5 lore raw
Quoted 4 lines · Arjan van de Ven
Quoted 2 lines

We expect the lock to be held on entry. Hence we expect mutex_trylock() to return zero.

you are correct, and the x86-64 mutex.h is buggy

ahh ... indeed! And i386 trylock was buggy too.

Ingo

----

fix typo in asm-i386/mutex.h:__mutex_fastpath_trylock - noticed by Arjan van de Ven.

asm-i386/mutex.h renamed from asm-i386/mutex.h.orig +1 -1
@@ -125,7 +125,7 @@ __mutex_fastpath_trylock(atomic_t *count
 	 * the mutex state would be.
 	 */
 #ifdef __HAVE_ARCH_CMPXCHG
-	if (likely(atomic_cmpxchg(count, 1, 0)) == 1)
+	if (likely(atomic_cmpxchg(count, 1, 0) == 1))
 		return 1;
 	return 0;
 #else
#6 lore raw
In reply to Arjan van de Ven
Quoted 17 lines · Arjan van de Ven
Quoted 2 lines

We expect the lock to be held on entry. Hence we expect mutex_trylock() to return zero.

you are correct, and the x86-64 mutex.h is buggy

include/asm-x86_64/mutex.h renamed from include/asm-x86_64/mutex.h.org +1 -1
@@ -104,7 +104,7 @@
 static inline int
 __mutex_fastpath_trylock(atomic_t *count, int (*fail_fn)(atomic_t *))
 {
-	if (likely(atomic_cmpxchg(count, 1, 0)) == 1)
+	if (likely(atomic_cmpxchg(count, 1, 0) == 1))
 		return 1;
 	else
 		return 0;
Quoted 2 lines

changes the asm to be the correct one for me.
This is odd/evil though..

likely is the evil part here. What about this? Should make this bug impossible to do....

include/linux/compiler.h +2 -2
@@ -59,8 +59,8 @@ extern void __chk_io_ptr(void __iomem *)
  * specific implementations come from the above header files
  */
 
-#define likely(x)	__builtin_expect(!!(x), 1)
-#define unlikely(x)	__builtin_expect(!!(x), 0)
+#define likely(x)	(__builtin_expect(!!(x), 1))
+#define unlikely(x)	(__builtin_expect(!!(x), 0))
 
 /* Optimization barrier */
 #ifndef barrier
kernel/sched.c +3 -3
@@ -367,7 +367,7 @@ repeat_lock_task:
 	local_irq_save(*flags);
 	rq = task_rq(p);
 	spin_lock(&rq->lock);
-	if (unlikely(rq != task_rq(p))) {
+	if unlikely(rq != task_rq(p)) {
 		spin_unlock_irqrestore(&rq->lock, *flags);
 		goto repeat_lock_task;
 	}
@@ -751,7 +751,7 @@ static int recalc_task_prio(task_t *p, u
 	else
 		sleep_time = (unsigned long)__sleep_time;
 
-	if (likely(sleep_time > 0)) {
+	if likely(sleep_time > 0) {
 		/*
 		 * User tasks that sleep a long time are categorised as
 		 * idle and will get just interactive status to stay active &
@@ -876,7 +876,7 @@ static void resched_task(task_t *p)
 
 	assert_spin_locked(&task_rq(p)->lock);
 
-	if (unlikely(test_tsk_thread_flag(p, TIF_NEED_RESCHED)))
+	if unlikely(test_tsk_thread_flag(p, TIF_NEED_RESCHED))
 		return;
 
 	set_tsk_thread_flag(p, TIF_NEED_RESCHED);
Signature · 1 line
Thanks, Sharp!
#7 lore raw

From: Pavel Machek <[email protected]>
Date: Wed, 11 Jan 2006 23:40:13 +0100

Quoted 4 lines

likely is the evil part here. What about this? Should make this bug impossible to do....

This doesn't let you do:

     if (likely(y) || unlikely(x))

which I have had reason to use in the past.

#9 lore raw
In reply to Pavel Machek
Quoted 44 lines · Pavel Machek
Quoted 17 lines · Arjan van de Ven
Quoted 2 lines

We expect the lock to be held on entry. Hence we expect mutex_trylock() to return zero.

you are correct, and the x86-64 mutex.h is buggy

include/asm-x86_64/mutex.h 2006-01-11 22:25:43.000000000 +0100 renamed from include/asm-x86_64/mutex.h.org 2006-01-11 22:25:37.000000000 +0100 +1 -1
@@ -104,7 +104,7 @@
 static inline int
 __mutex_fastpath_trylock(atomic_t *count, int (*fail_fn)(atomic_t *))
 {
-     if (likely(atomic_cmpxchg(count, 1, 0)) == 1)
+     if (likely(atomic_cmpxchg(count, 1, 0) == 1))
              return 1;
      else
              return 0;
Quoted 2 lines

changes the asm to be the correct one for me.
This is odd/evil though..

likely is the evil part here. What about this? Should make this bug impossible to do....

include/linux/compiler.h +2 -2
@@ -59,8 +59,8 @@ extern void __chk_io_ptr(void __iomem *)
  * specific implementations come from the above header files
  */
 
-#define likely(x)      __builtin_expect(!!(x), 1)
-#define unlikely(x)    __builtin_expect(!!(x), 0)
+#define likely(x)      (__builtin_expect(!!(x), 1))
+#define unlikely(x)    (__builtin_expect(!!(x), 0))
 
 /* Optimization barrier */
 #ifndef barrier

kernel/sched.c repeat_lock_task: · line 370

Quoted 5 lines
kernel/sched.c +1 -1
@@ -367,7 +367,7 @@ repeat_lock_task:
        local_irq_save(*flags);
        rq = task_rq(p);
        spin_lock(&rq->lock);
-       if (unlikely(rq != task_rq(p))) {
+       if unlikely(rq != task_rq(p)) {

This is confusing to read. Why not keep the parenthesis around (unlikely(...)) ? Yes, it's an extra set of parenthesis that are not strictly needed now that you've added them to the likely/unlikely macros, but they don't do any harm either and make the code less
surprising to read... I know that I at least think *BUG* at once
when I read a line like
if unlikely(rq != task_rq(p)) {
and then when I find that it actually compiles fine I go dig for the reason for that, find the macro and see that all is well, but that just wasted a lot of time.

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#8 lore raw
In reply to David S. Miller
Quoted 11 lines · David S. Miller

From: Pavel Machek <[email protected]>
Date: Wed, 11 Jan 2006 23:40:13 +0100

Quoted 4 lines

likely is the evil part here. What about this? Should make this bug impossible to do....

This doesn't let you do:

     if (likely(y) || unlikely(x))

hm, why not? It will expand to:

   if ((__builtin_expect(!!(y), 1)) || (__builtin_expect(!!(x), 0)))

which seems correct to me. Pavel only added extra parantheses, to make the simple case more readable.

Ingo

author
#11 lore raw

Re: 2.6.15-git7 oopses in ext3 during LTP runs II - more problems

In reply to Arjan van de Ven
Quoted 15 lines · Arjan van de Ven
Quoted 4 lines

15/fs/ext3/super.c~ 2006-01-11 21:54:13.000000000 +0100

Quoted 3 lines

+++ linux-2.6.15/fs/ext3/super.c 2006-01-11 21:54:13.000000000 +0100

(quoted diff) +0 -0
@@ -2150,7 +2150,7 @@
 
Quoted 10 lines

Review line 2156

Quoted 7 lines
(quoted diff) +1 -1
 static void ext3_write_super (struct super_block * sb)
 {
-	if (mutex_trylock(&sb->s_lock) != 0)
+	if (!mutex_trylock(&sb->s_lock) != 0)
 		BUG();
 	sb->s_dirt = 0;
 }

We expect the lock to be held on entry. Hence we expect mutex_trylock() to return zero.

Quoted 1 line

you are correct, and the x86-64 mutex.h is buggy

While this patch seemed to fix LTP my desktop running the same kernel (with mutex fix) hung the mailer while sending an unrelated mail. Again on ext3.

I unfortunately don't have the backtraces anymore because I couldn't save them to disk before the reboot (and forgot to copy them to another system sorry), but they were also hanging in some JBD/ext3 functions in D, with all disk accesses hanging.

So things appear to be still broken in ext3 land.

-Andi

#12 lore raw

Re: 2.6.15-git7 oopses in ext3 during LTP runs II - more problems

Quoted 22 lines · Andi Kleen
Quoted 15 lines · Arjan van de Ven
Quoted 4 lines

15/fs/ext3/super.c~ 2006-01-11 21:54:13.000000000 +0100

Quoted 3 lines

+++ linux-2.6.15/fs/ext3/super.c 2006-01-11 21:54:13.000000000 +0100

(quoted diff) +0 -0
@@ -2150,7 +2150,7 @@
 
Quoted 10 lines

Review line 2156

Quoted 7 lines
(quoted diff) +1 -1
 static void ext3_write_super (struct super_block * sb)
 {
-	if (mutex_trylock(&sb->s_lock) != 0)
+	if (!mutex_trylock(&sb->s_lock) != 0)
 		BUG();
 	sb->s_dirt = 0;
 }

We expect the lock to be held on entry. Hence we expect mutex_trylock() to return zero.

Quoted 1 line

you are correct, and the x86-64 mutex.h is buggy

While this patch seemed to fix LTP my desktop running the same kernel (with mutex fix) hung the mailer while sending an unrelated mail. Again on ext3.

Quoted 6 lines

I unfortunately don't have the backtraces anymore because I couldn't save them to disk before the reboot (and forgot to copy them to another system sorry), but they were also hanging in some JBD/ext3 functions in D, with all disk accesses hanging.

So things appear to be still broken in ext3 land.

Are you using MD?

sata? If so, which?

author
#13 lore raw

Re: 2.6.15-git7 oopses in ext3 during LTP runs II - more problems

Quoted 3 lines · Andrew Morton

Are you using MD?

sata? If so, which?

None of both. old style IDE using VIA driver.

-Andi

#10 lore raw
In reply to Jesper Juhl
Quoted 21 lines · Jesper Juhl
Quoted 20 lines · Pavel Machek
Quoted 19 lines · Arjan van de Ven
Quoted 2 lines

We expect the lock to be held on entry. Hence we expect mutex_trylock() to return zero.

you are correct, and the x86-64 mutex.h is buggy

include/asm-x86_64/mutex.h 2006-01-11 22:25:43.000000000 +0100 renamed from include/asm-x86_64/mutex.h.org 2006-01-11 22:25:37.000000000 +0100 +1 -1
@@ -104,7 +104,7 @@
 static inline int
 __mutex_fastpath_trylock(atomic_t *count, int (*fail_fn)(atomic_t *))
 {
-     if (likely(atomic_cmpxchg(count, 1, 0)) == 1)
+     if (likely(atomic_cmpxchg(count, 1, 0) == 1))
              return 1;
      else
              return 0;

changes the asm to be the correct one for me.
This is odd/evil though..

....

Quoted 5 lines
Quoted 5 lines

+++ b/kernel/sched.c

kernel/sched.c +0 -0
@@ -367,7 +367,7 @@ repeat_lock_task:
        local_irq_save(*flags);
        rq = task_rq(p);
        spin_lock(&rq->lock);
Quoted 8 lines

kernel/sched.c repeat_lock_task: · line 370

Quoted 2 lines
kernel/sched.c +1 -1
-       if (unlikely(rq != task_rq(p))) {
+       if unlikely(rq != task_rq(p)) {

This is confusing to read. Why not keep the parenthesis around (unlikely(...)) ? Yes, it's an extra set of parenthesis that are not strictly needed now that you've added them to the likely/unlikely macros, but they don't do any harm either and make the code less
surprising to read... I know that I at least think *BUG* at once

Read the email from the start, there's very nice example why the extra ()'s are evil in mutex_fast_trylock.

Pavel

Signature · 1 line
Thanks, Sharp!