Skip to contentSkip to conversation

Question about ext3 jbd module

PATCH bibo, mao 2 participants 2 messages Jul 25, 2006

lore
Share & tools
lore

bibo, mao

author
#0 lore raw

Hi,
When I run LTP stress test on my IA64 box based on ditribution
kernel version, kernel will crash within three days, I think this problem should exist in recent kernel also, but I does not trigger this. The problem is in function journal_dirty_metadata(),
there is one sentence like this:

	struct journal_head *jh = bh2jh(bh);
Quoted 1 line

From my debug result, jh is NULL at this point, I do not know whether

it is because of contention or kernel forgets to consider NULL pointer condition.

I wrote one patch, after this patch LTP stress test does not crash, but I am not familiar filesystem, I do not know whether it is the root cause or what negative influence this patch will bring out.

Thanks
bibo,mao

fs/jbd/transaction.c 2006-07-07 02:56:32.000000000 +0800 renamed from fs/jbd/transaction.c.orig 2006-06-30 14:05:58.000000000 +0800 +0 -0
@@ -1104,13 +1104,15 @@ int journal_dirty_metadata(handle_t *han
{
       transaction_t *transaction = handle->h_transaction;
       journal_t *journal = transaction->t_journal;

- struct journal_head *jh = bh2jh(bh);
+ struct journal_head *jh;

- jbd_debug(5, "journal_head %p\n", jh);

-       JBUFFER_TRACE(jh, "entry");
       if (is_handle_aborted(handle))
               goto out;
+       jh = journal_add_journal_head(bh);
+       jbd_debug(5, "journal_head %p\n", jh);
+       JBUFFER_TRACE(jh, "entry");
+
       jbd_lock_bh_state(bh);

       /*
@@ -1154,6 +1156,7 @@ int journal_dirty_metadata(handle_t *han
       spin_unlock(&journal->j_list_lock);
out_unlock_bh:
       jbd_unlock_bh_state(bh);
+       journal_put_journal_head(jh);
out:
       JBUFFER_TRACE(jh, "exit");
       return 0;

inode02[28417]: NaT consumption 2216203124768 [1]

Modules linked in: nfs(U) nfsd(U) exportfs(U) lockd(U) nfs_acl(U) md5(U) ipv6(U) parport_pc(U) lp(U) parport(U) autofs4(U) sunrpc(U) ds(U) yenta_socket(U) pc

mcia_core(U) vfat(U) fat(U) dm_mirror(U) dm_mod(U) joydev(U) button(U) uhci_hcd(U) ehci_hcd(U) shpchp(U) e1000(U) ext3(U) jbd(U) mptscsih(U) mptsas(U) mptspi
(U) mptfc(U) mptscsi(U) mptbase(U) sd_mod(U) scsi_mod(U)

Pid: 28417, CPU 13, comm: inode02
psr : 0000121008126010 ifs : 800000000000040d ip : [<a000000200134721>] Not tainted
ip is at journal_dirty_metadata+0x2c1/0x5e0 [jbd]
unat: 0000000000000000 pfs : 0000000000000917 rsc : 0000000000000003
rnat: 0000000000000000 bsps: 0000000000000000 pr : 005965a026595569
ldrs: 0000000000000000 ccv : 0000000000060011 fpsr: 0009804c8a70033f csd : 0000000000000000 ssd : 0000000000000000
b0 : a0000002001d3350 b6 : a000000100589f20 b7 : a0000001001fee60 f6 : 1003e0000000000000000 f7 : 1003e0000000000000080 f8 : 1003e00000000000008c1 f9 : 1003effffffffffffc0a0 f10 : 100049c8d719c0533ddf0 f11 : 1003e00000000000008c1 r1 : a000000200330000 r2 : a000000200158630 r3 : a000000200158630 r8 : e0000001d885631c r9 : 0000000000000000 r10 : e0000001bede66e0 r11 : 0000000000000010 r12 : e0000001a8a67d40 r13 : e0000001a8a60000 r14 : 0000000000060011 r15 : 0000000000000000 r16 : e0000001fef76e80 r17 : e0000001e3e34b38 r18 : 0000000000000020 r19 : 0000000000060011 r20 : 00000000000e0011 r21 : 0000000000060011 r22 : 0000000000000000 r23 : 0000000000000000 r24 : 0000000000000000 r25 : e0000001e6cc8090 r26 : e0000001d88561e0 r27 : 0000000044bc2661 r28 : e0000001d8856078 r29 : 000000007c6fe61a r30 : e0000001e6cc80e8 r31 : e0000001d8856080

Call Trace:
[<a000000100016b20>] show_stack+0x80/0xa0
                               sp=e0000001a8a67750 bsp=e0000001a8a61360
[<a000000100017430>] show_regs+0x890/0x8c0
                               sp=e0000001a8a67920 bsp=e0000001a8a61318
[<a00000010003dbf0>] die+0x150/0x240
                               sp=e0000001a8a67940 bsp=e0000001a8a612d8
[<a00000010003dd20>] die_if_kernel+0x40/0x60
                               sp=e0000001a8a67940 bsp=e0000001a8a612a8
[<a00000010003f930>] ia64_fault+0x1450/0x15a0
                               sp=e0000001a8a67940 bsp=e0000001a8a61250
[<a00000010000f540>] ia64_leave_kernel+0x0/0x260
                               sp=e0000001a8a67b70 bsp=e0000001a8a61250
[<a000000200134720>] journal_dirty_metadata+0x2c0/0x5e0 [jbd]
                               sp=e0000001a8a67d40 bsp=e0000001a8a611e0
[<a0000002001d3350>] ext3_mark_iloc_dirty+0x750/0xc00 [ext3]
                               sp=e0000001a8a67d40 bsp=e0000001a8a61150
[<a0000002001d3a90>] ext3_mark_inode_dirty+0xb0/0xe0 [ext3]
                               sp=e0000001a8a67d40 bsp=e0000001a8a61128
[<a0000002001ce3d0>] ext3_new_inode+0x1670/0x1a80 [ext3]
                               sp=e0000001a8a67d60 bsp=e0000001a8a61068
[<a0000002001df5e0>] ext3_mkdir+0x120/0x940 [ext3]
                               sp=e0000001a8a67da0 bsp=e0000001a8a61008
[<a000000100148250>] vfs_mkdir+0x250/0x380
                               sp=e0000001a8a67db0 bsp=e0000001a8a60fb0
[<a0000001001484f0>] sys_mkdir+0x170/0x280
                               sp=e0000001a8a67db0 bsp=e0000001a8a60f30
[<a00000010000f3e0>] ia64_ret_from_syscall+0x0/0x20
#1 lore raw
Quoted 41 lines · mao

Hi,
When I run LTP stress test on my IA64 box based on ditribution
kernel version, kernel will crash within three days, I think this problem should exist in recent kernel also, but I does not trigger this. The problem is in function journal_dirty_metadata(), there is one sentence like this:

	struct journal_head *jh = bh2jh(bh);
Quoted 1 line

From my debug result, jh is NULL at this point, I do not know whether

it is because of contention or kernel forgets to consider NULL pointer condition.

I wrote one patch, after this patch LTP stress test does not crash, but I am not familiar filesystem, I do not know whether it is the root cause or what negative influence this patch will bring out.

Thanks
bibo,mao

fs/jbd/transaction.c 2006-07-07 02:56:32.000000000 +0800 renamed from fs/jbd/transaction.c.orig 2006-06-30 14:05:58.000000000 +0800 +5 -3
@@ -1104,13 +1104,15 @@ int journal_dirty_metadata(handle_t *han
 {
        transaction_t *transaction = handle->h_transaction;
        journal_t *journal = transaction->t_journal;
-       struct journal_head *jh = bh2jh(bh);
+       struct journal_head *jh;
 
-       jbd_debug(5, "journal_head %p\n", jh);
-       JBUFFER_TRACE(jh, "entry");
        if (is_handle_aborted(handle))
                goto out;
 
+       jh = journal_add_journal_head(bh);
+       jbd_debug(5, "journal_head %p\n", jh);
+       JBUFFER_TRACE(jh, "entry");
+
        jbd_lock_bh_state(bh);
 
        /*

fs/jbd/transaction.c 2006-07-07 02:56:32.000000000 +0800 int journal_dirty_metadata(handle_t *han · line 1162

Quoted 7 lines
fs/jbd/transaction.c 2006-07-07 02:56:32.000000000 +0800 renamed from fs/jbd/transaction.c.orig 2006-06-30 14:05:58.000000000 +0800 +1 -0
@@ -1154,6 +1156,7 @@ int journal_dirty_metadata(handle_t *han
        spin_unlock(&journal->j_list_lock);
 out_unlock_bh:
        jbd_unlock_bh_state(bh);
+       journal_put_journal_head(jh);
 out:
        JBUFFER_TRACE(jh, "exit");
        return 0;

That's a worry. We've attached a journal_head and we've done do_get_write_access() and we're now proceeding to journal the buffer as metadata but someone has presumably gone and run __journal_try_to_free_buffer() against the thing and has stolen our journal_head.

Simply reattaching a new journal_head is most likely wrong - we'll lose whatever state was supposed to be in the old one (like, which journal list this bh+jh is on).

Somewhere, somehow, that journal_head has passed through a state which permitted __journal_try_to_free_buffer() to free it while appropriate locks were not held. I wonder where.

Your diff headers claim to be against 2.6.9. Is that so?

Would it be correct to assume that there was some page replacement pressure happening at the time?

It looks like a big box - can you describe it a bit please?

Quoted 50 lines

Pid: 28417, CPU 13, comm: inode02
psr : 0000121008126010 ifs : 800000000000040d ip : [<a000000200134721>] Not tainted
ip is at journal_dirty_metadata+0x2c1/0x5e0 [jbd]
unat: 0000000000000000 pfs : 0000000000000917 rsc : 0000000000000003
rnat: 0000000000000000 bsps: 0000000000000000 pr : 005965a026595569
ldrs: 0000000000000000 ccv : 0000000000060011 fpsr: 0009804c8a70033f csd : 0000000000000000 ssd : 0000000000000000
b0 : a0000002001d3350 b6 : a000000100589f20 b7 : a0000001001fee60 f6 : 1003e0000000000000000 f7 : 1003e0000000000000080 f8 : 1003e00000000000008c1 f9 : 1003effffffffffffc0a0 f10 : 100049c8d719c0533ddf0 f11 : 1003e00000000000008c1 r1 : a000000200330000 r2 : a000000200158630 r3 : a000000200158630 r8 : e0000001d885631c r9 : 0000000000000000 r10 : e0000001bede66e0 r11 : 0000000000000010 r12 : e0000001a8a67d40 r13 : e0000001a8a60000 r14 : 0000000000060011 r15 : 0000000000000000 r16 : e0000001fef76e80 r17 : e0000001e3e34b38 r18 : 0000000000000020 r19 : 0000000000060011 r20 : 00000000000e0011 r21 : 0000000000060011 r22 : 0000000000000000 r23 : 0000000000000000 r24 : 0000000000000000 r25 : e0000001e6cc8090 r26 : e0000001d88561e0 r27 : 0000000044bc2661 r28 : e0000001d8856078 r29 : 000000007c6fe61a r30 : e0000001e6cc80e8 r31 : e0000001d8856080

Call Trace:
 [<a000000100016b20>] show_stack+0x80/0xa0
                                sp=e0000001a8a67750 bsp=e0000001a8a61360
 [<a000000100017430>] show_regs+0x890/0x8c0
                                sp=e0000001a8a67920 bsp=e0000001a8a61318
 [<a00000010003dbf0>] die+0x150/0x240
                                sp=e0000001a8a67940 bsp=e0000001a8a612d8
 [<a00000010003dd20>] die_if_kernel+0x40/0x60
                                sp=e0000001a8a67940 bsp=e0000001a8a612a8
 [<a00000010003f930>] ia64_fault+0x1450/0x15a0
                                sp=e0000001a8a67940 bsp=e0000001a8a61250
 [<a00000010000f540>] ia64_leave_kernel+0x0/0x260
                                sp=e0000001a8a67b70 bsp=e0000001a8a61250
 [<a000000200134720>] journal_dirty_metadata+0x2c0/0x5e0 [jbd]
                                sp=e0000001a8a67d40 bsp=e0000001a8a611e0
 [<a0000002001d3350>] ext3_mark_iloc_dirty+0x750/0xc00 [ext3]
                                sp=e0000001a8a67d40 bsp=e0000001a8a61150
 [<a0000002001d3a90>] ext3_mark_inode_dirty+0xb0/0xe0 [ext3]
                                sp=e0000001a8a67d40 bsp=e0000001a8a61128
 [<a0000002001ce3d0>] ext3_new_inode+0x1670/0x1a80 [ext3]
                                sp=e0000001a8a67d60 bsp=e0000001a8a61068
 [<a0000002001df5e0>] ext3_mkdir+0x120/0x940 [ext3]
                                sp=e0000001a8a67da0 bsp=e0000001a8a61008
 [<a000000100148250>] vfs_mkdir+0x250/0x380
                                sp=e0000001a8a67db0 bsp=e0000001a8a60fb0
 [<a0000001001484f0>] sys_mkdir+0x170/0x280
                                sp=e0000001a8a67db0 bsp=e0000001a8a60f30
 [<a00000010000f3e0>] ia64_ret_from_syscall+0x0/0x20