Skip to contentSkip to conversation

2.6.16-rc5-mm3: spinlock bad magic on CPU#0 on AMD64

DISCUSSION Rafael J. Wysocki 4 participants 12 messages Mar 11, 2006 – Mar 13, 2006

lore
Share & tools
lore
#0 lore raw

Hi,

With the 2.6.16-rc5-mm3 kernel w/ the patch

revert-x86_64-mm-i386-early-alignment.patch

applied I'm able to hang my box (Asus L5D, 1 CPU, x86-64 kernel) solid by running OpenOffice.org from under KDE (100% of the time but on one user account only). Before it hangs I get something like this on the serial console:

BUG: spinlock bad magic on CPU#0, soffice.bin/5293
 lock: ffff81005e174e28, .magic: 000001ff, .owner: .5).@4).06)./0, .owner_cpu: -2141827648
BUG: spinlock lockup on CPU#0, soffice.bin/5293, ffff81005e174e28

Greetings,
Rafael

#1 lore raw
Quoted 7 lines · Rafael J. Wysocki

Hi,

With the 2.6.16-rc5-mm3 kernel w/ the patch

revert-x86_64-mm-i386-early-alignment.patch

Quoted 8 lines

applied I'm able to hang my box (Asus L5D, 1 CPU, x86-64 kernel) solid by running OpenOffice.org from under KDE (100% of the time but on one user account only). Before it hangs I get something like this on the serial console:

BUG: spinlock bad magic on CPU#0, soffice.bin/5293
 lock: ffff81005e174e28, .magic: 000001ff, .owner: .5).@4).06)./0, .owner_cpu: -2141827648
BUG: spinlock lockup on CPU#0, soffice.bin/5293, ffff81005e174e28

Is it a !CONFIG_SMP kernel?

There's no stack trace?

#2 lore raw
Quoted 9 lines · Andrew Morton
Quoted 8 lines · Rafael J. Wysocki

With the 2.6.16-rc5-mm3 kernel w/ the patch

revert-x86_64-mm-i386-early-alignment.patch

applied I'm able to hang my box (Asus L5D, 1 CPU, x86-64 kernel) solid by running OpenOffice.org from under KDE (100% of the time but on one user account only). Before it hangs I get something like this on the serial console:

Quoted 6 lines
Quoted 4 lines
BUG: spinlock bad magic on CPU#0, soffice.bin/5293
 lock: ffff81005e174e28, .magic: 000001ff, .owner: .5).@4).06)./0, .owner_cpu: -2141827648
BUG: spinlock lockup on CPU#0, soffice.bin/5293, ffff81005e174e28

Is it a !CONFIG_SMP kernel?

Yes.

Quoted 1 line

There's no stack trace?

Well, probably there was one but it didn't appear on the serial console because the console loglevel was too low. I'll try to increase console_loglevel in spin_bug() and see what happens.

[BTW, looking at the relevant code, in my case .owner_cpu appears to have only the most significant bit set (??).]

#3 lore raw

[Dropped [email protected] from the Cc list, it's probably OT there]

Quoted 17 lines · Rafael J. Wysocki
Quoted 9 lines · Andrew Morton
Quoted 8 lines · Rafael J. Wysocki

With the 2.6.16-rc5-mm3 kernel w/ the patch

revert-x86_64-mm-i386-early-alignment.patch

applied I'm able to hang my box (Asus L5D, 1 CPU, x86-64 kernel) solid by running OpenOffice.org from under KDE (100% of the time but on one user account only). Before it hangs I get something like this on the serial console:

Quoted 6 lines
Quoted 4 lines
BUG: spinlock bad magic on CPU#0, soffice.bin/5293
 lock: ffff81005e174e28, .magic: 000001ff, .owner: .5).@4).06)./0, .owner_cpu: -2141827648
BUG: spinlock lockup on CPU#0, soffice.bin/5293, ffff81005e174e28

Is it a !CONFIG_SMP kernel?

Quoted 7 lines

Yes.

Quoted 1 line

There's no stack trace?

Well, probably there was one but it didn't appear on the serial console because the console loglevel was too low. I'll try to increase console_loglevel in spin_bug() and see what happens.

Done, and now it looks like this:

BUG: spinlock bad magic on CPU#0, soffice.bin/5192
 lock: ffff81005f79ae28, .magic: 000001ff, .owner: 1..1..|1. |1.|1..|1.�1..|1..1. 1./-1,

.owner_cpu: -2141838208

Call Trace: <ffffffff80210383>{__alloc_pages+99} <ffffffff802156c3>{spin_bug+195}
       <ffffffff802077dc>{_raw_spin_lock+44} <ffffffff80270a4e>{_spin_lock+30}
       <ffffffff8033712d>{journal_extend+77} <ffffffff80327255>{ext3_get_block+165}
       <ffffffff8022c2f9>{do_mpage_readpage+425} <ffffffff80270cc4>{_write_unlock_irq+20}
       <ffffffff8020cce2>{add_to_page_cache+162} <ffffffff8023fdee>{mpage_readpages+254}
       <ffffffff803271b0>{ext3_get_block+0} <ffffffff803271b0>{ext3_get_block+0}
       <ffffffff803146df>{get_cnode+95} <ffffffff8020a3bb>{get_page_from_freelist+619}
       <ffffffff80210383>{__alloc_pages+99} <ffffffff80323c1a>{ext3_readpages+26}
       <ffffffff80214030>{__do_page_cache_readahead+416} <ffffffff80213c12>{poison_obj+66}
       <ffffffff80232058>{wake_up_bit+40} <ffffffff80243152>{unlock_buffer+18}
       <ffffffff80315bb8>{reiserfs_prepare_for_journal+104}
       <ffffffff802b6ab4>{do_page_cache_readahead+100} <ffffffff80215942>{filemap_nopage+322}
       <ffffffff80208b2c>{__handle_mm_fault+1004} <ffffffff80270e7d>{_spin_unlock_irqrestore+29}
       <ffffffff8020ae99>{do_page_fault+1257} <ffffffff8026af8d>{error_exit+0}
       <ffffffff802ffb40>{reiserfs_copy_from_user_to_file_region+80}
       <ffffffff80302446>{reiserfs_file_write+6102} <ffffffff802f8f4e>{reiserfs_add_entry+1054}
       <ffffffff8033c1ff>{journal_cancel_revoke+351} <ffffffff80213c12>{poison_obj+66}
       <ffffffff80236d27>{cache_free_debugcheck+711} <ffffffff80335734>{journal_stop+772}
       <ffffffff80270f30>{_spin_unlock+16} <ffffffff802193a2>{vfs_write+226}
       <ffffffff80219c80>{sys_write+80} <ffffffff8026d234>{cstar_do_call+27}
BUG: spinlock lockup on CPU#0, soffice.bin/5192, ffff81005f79ae28
#4 lore raw
Quoted 2 lines · Rafael J. Wysocki

Done, and now it looks like this:

Helps a lot, thanks.

Quoted 17 lines
 BUG: spinlock bad magic on CPU#0, soffice.bin/5192
  lock: ffff81005f79ae28, .magic: 000001ff, .owner: 1..1..|1. |1.|1..|1.___1..|1..1. 1./-1,
 .owner_cpu: -2141838208

 Call Trace: <ffffffff80210383>{__alloc_pages+99} <ffffffff802156c3>{spin_bug+195}
        <ffffffff802077dc>{_raw_spin_lock+44} <ffffffff80270a4e>{_spin_lock+30}
        <ffffffff8033712d>{journal_extend+77} <ffffffff80327255>{ext3_get_block+165}
        <ffffffff8022c2f9>{do_mpage_readpage+425} <ffffffff80270cc4>{_write_unlock_irq+20}
        <ffffffff8020cce2>{add_to_page_cache+162} <ffffffff8023fdee>{mpage_readpages+254}
        <ffffffff803271b0>{ext3_get_block+0} <ffffffff803271b0>{ext3_get_block+0}
        <ffffffff803146df>{get_cnode+95} <ffffffff8020a3bb>{get_page_from_freelist+619}
        <ffffffff80210383>{__alloc_pages+99} <ffffffff80323c1a>{ext3_readpages+26}
        <ffffffff80214030>{__do_page_cache_readahead+416} <ffffffff80213c12>{poison_obj+66}
        <ffffffff80232058>{wake_up_bit+40} <ffffffff80243152>{unlock_buffer+18}
        <ffffffff80315bb8>{reiserfs_prepare_for_journal+104}
        <ffffffff802b6ab4>{do_page_cache_readahead+100} <ffffffff80215942>{filemap_nopage+322}
        <ffffffff80208b2c>{__handle_mm_fault+1004} <ffffffff80270e7d>{_spin_unlock_irqrestore+29}
Quoted 8 lines
        <ffffffff8020ae99>{do_page_fault+1257} <ffffffff8026af8d>{error_exit+0}
        <ffffffff802ffb40>{reiserfs_copy_from_user_to_file_region+80}
        <ffffffff80302446>{reiserfs_file_write+6102} <ffffffff802f8f4e>{reiserfs_add_entry+1054}
        <ffffffff8033c1ff>{journal_cancel_revoke+351} <ffffffff80213c12>{poison_obj+66}
        <ffffffff80236d27>{cache_free_debugcheck+711} <ffffffff80335734>{journal_stop+772}
        <ffffffff80270f30>{_spin_unlock+16} <ffffffff802193a2>{vfs_write+226}
        <ffffffff80219c80>{sys_write+80} <ffffffff8026d234>{cstar_do_call+27}
 BUG: spinlock lockup on CPU#0, soffice.bin/5192, ffff81005f79ae28

It's a pretty vile backtrace. I supposed you have CONFIG_FRAME_POINTER=n.

Still. It seems that what's happened is that we took a pagefault while reiserfs had a transaction open. The fault is against a mmapped ext3 file and we ended up in the recently-reworked ext3_get_block() which tests journal_current_handle() to work out whether we're in a write or a read. oops. The presence of reiserfs journal_info makes it decide it's a write, not a read so it starts treating a reiserfs journal_info as an ext3 one.

The code used to work OK because it was only for direct-IO, which doesn't get recurred into like this. But it got used for regular I/O in -mm.

This should fix:

fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +1 -1
@@ -830,7 +830,7 @@ ext3_direct_io_get_blocks(struct inode *
 	handle_t *handle = journal_current_handle();
 	int ret = 0;
 
-	if (!handle)
+	if (!create)
 		goto get_block;		/* A read */
 
 	if (max_blocks == 1)

_

#5 lore raw
Quoted 2 lines · Andrew Morton

It's a pretty vile backtrace. I supposed you have CONFIG_FRAME_POINTER=n.

Won't make a difference because the oops backtracer doesn't use them.

Quoted 9 lines

Still. It seems that what's happened is that we took a pagefault while reiserfs had a transaction open. The fault is against a mmapped ext3 file and we ended up in the recently-reworked ext3_get_block() which tests journal_current_handle() to work out whether we're in a write or a read. oops. The presence of reiserfs journal_info makes it decide it's a write, not a read so it starts treating a reiserfs journal_info as an ext3 one.

The code used to work OK because it was only for direct-IO, which doesn't get recurred into like this. But it got used for regular I/O in -mm.

Oops. Can this happen in more situations?

-Andi

#6 lore raw
Quoted 8 lines · Andi Kleen
Quoted 8 lines

Still. It seems that what's happened is that we took a pagefault while reiserfs had a transaction open. The fault is against a mmapped ext3 file and we ended up in the recently-reworked ext3_get_block() which tests journal_current_handle() to work out whether we're in a write or a read. oops. The presence of reiserfs journal_info makes it decide it's a write, not a read so it starts treating a reiserfs journal_info as an ext3 one.

Quoted 4 lines
Quoted 2 lines

The code used to work OK because it was only for direct-IO, which doesn't get recurred into like this. But it got used for regular I/O in -mm.

Oops. Can this happen in more situations?

I don't _think_ so, but it's pretty scary. The code's been this way for a while.

Typical scenario:

	reiserfs_journal_start		- sets current->journal_info
	  copy_from_user
	    pagefault
	      ext3_readpages()

As long as ext3_readpage[s]() doesn't try to start a transaction we're OK. And it shouldn't, if create==0.

Fortunately filemap_nopage() doesn't do atime updates. If it did, things would get messy.

We do have deadlock possibilities in there - filemap_nopage() does lock_page() inside journal_start(), whereas generic_file_write() does journal_start() inside lock_page(). Chris Mason and I have stared unhappily at that a few times. Hard to fix.

But I don't _think_ we have any more journal_start() recursions like this - ext3 tends to get pretty noisy if it detects that in unexpected places.

#7 lore raw
In reply to Andrew Morton
Quoted 49 lines · Andrew Morton
Quoted 2 lines · Rafael J. Wysocki

Done, and now it looks like this:

Helps a lot, thanks.

Quoted 17 lines
 BUG: spinlock bad magic on CPU#0, soffice.bin/5192
  lock: ffff81005f79ae28, .magic: 000001ff, .owner: 1..1..|1. |1.|1..|1.___1..|1..1. 1./-1,
 .owner_cpu: -2141838208

 Call Trace: <ffffffff80210383>{__alloc_pages+99} <ffffffff802156c3>{spin_bug+195}
        <ffffffff802077dc>{_raw_spin_lock+44} <ffffffff80270a4e>{_spin_lock+30}
        <ffffffff8033712d>{journal_extend+77} <ffffffff80327255>{ext3_get_block+165}
        <ffffffff8022c2f9>{do_mpage_readpage+425} <ffffffff80270cc4>{_write_unlock_irq+20}
        <ffffffff8020cce2>{add_to_page_cache+162} <ffffffff8023fdee>{mpage_readpages+254}
        <ffffffff803271b0>{ext3_get_block+0} <ffffffff803271b0>{ext3_get_block+0}
        <ffffffff803146df>{get_cnode+95} <ffffffff8020a3bb>{get_page_from_freelist+619}
        <ffffffff80210383>{__alloc_pages+99} <ffffffff80323c1a>{ext3_readpages+26}
        <ffffffff80214030>{__do_page_cache_readahead+416} <ffffffff80213c12>{poison_obj+66}
        <ffffffff80232058>{wake_up_bit+40} <ffffffff80243152>{unlock_buffer+18}
        <ffffffff80315bb8>{reiserfs_prepare_for_journal+104}
        <ffffffff802b6ab4>{do_page_cache_readahead+100} <ffffffff80215942>{filemap_nopage+322}
        <ffffffff80208b2c>{__handle_mm_fault+1004} <ffffffff80270e7d>{_spin_unlock_irqrestore+29}
Quoted 8 lines
        <ffffffff8020ae99>{do_page_fault+1257} <ffffffff8026af8d>{error_exit+0}
        <ffffffff802ffb40>{reiserfs_copy_from_user_to_file_region+80}
        <ffffffff80302446>{reiserfs_file_write+6102} <ffffffff802f8f4e>{reiserfs_add_entry+1054}
        <ffffffff8033c1ff>{journal_cancel_revoke+351} <ffffffff80213c12>{poison_obj+66}
        <ffffffff80236d27>{cache_free_debugcheck+711} <ffffffff80335734>{journal_stop+772}
        <ffffffff80270f30>{_spin_unlock+16} <ffffffff802193a2>{vfs_write+226}
        <ffffffff80219c80>{sys_write+80} <ffffffff8026d234>{cstar_do_call+27}
 BUG: spinlock lockup on CPU#0, soffice.bin/5192, ffff81005f79ae28

It's a pretty vile backtrace. I supposed you have CONFIG_FRAME_POINTER=n.

Still. It seems that what's happened is that we took a pagefault while reiserfs had a transaction open. The fault is against a mmapped ext3 file and we ended up in the recently-reworked ext3_get_block() which tests journal_current_handle() to work out whether we're in a write or a read. oops. The presence of reiserfs journal_info makes it decide it's a write, not a read so it starts treating a reiserfs journal_info as an ext3 one.

The code used to work OK because it was only for direct-IO, which doesn't get recurred into like this. But it got used for regular I/O in -mm.

This should fix:

fs/ext3/inode.c ext3_direct_io_get_blocks(struct inode * · line 836

Quoted 8 lines
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +1 -1
@@ -830,7 +830,7 @@ ext3_direct_io_get_blocks(struct inode *
 	handle_t *handle = journal_current_handle();
 	int ret = 0;
 
-	if (!handle)
+	if (!create)
 		goto get_block;		/* A read */
 
 	if (max_blocks == 1)

Er, it doesn't apply to either 2.6.16-rc5-mm3 or 2.6.16-rc6-mm1.

There's no function ext3_direct_io_get_blocks() there in fs/ext3/inode.c, AFAICT.

#8 lore raw
Quoted 9 lines · Rafael J. Wysocki
Quoted 9 lines

This should fix:

fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +0 -0
@@ -830,7 +830,7 @@ ext3_direct_io_get_blocks(struct inode *
 	handle_t *handle = journal_current_handle();
 	int ret = 0;
 
Quoted 7 lines

fs/ext3/inode.c ext3_direct_io_get_blocks(struct inode * · line 836

Quoted 5 lines
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +1 -1
-	if (!handle)
+	if (!create)
 		goto get_block;		/* A read */
 
 	if (max_blocks == 1)

Er, it doesn't apply to either 2.6.16-rc5-mm3 or 2.6.16-rc6-mm1.

Nope, it applies OK to rc6-mm1.

But whatever - it's only a single line?

#9 lore raw
Quoted 14 lines · Andrew Morton
Quoted 13 lines · Rafael J. Wysocki
Quoted 13 lines

This should fix:

fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +1 -1
@@ -830,7 +830,7 @@ ext3_direct_io_get_blocks(struct inode *
 	handle_t *handle = journal_current_handle();
 	int ret = 0;
 
-	if (!handle)
+	if (!create)
 		goto get_block;		/* A read */
 
Quoted 5 lines
Quoted 3 lines

fs/ext3/inode.c ext3_direct_io_get_blocks(struct inode * · line 836

Quoted 1 line
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +0 -0
 	if (max_blocks == 1)

Er, it doesn't apply to either 2.6.16-rc5-mm3 or 2.6.16-rc6-mm1.

Nope, it applies OK to rc6-mm1.

Well, this means my rc6-mm1 is different to what you have. :-)

Anyway in "my" version there's a function ext3_get_block() which reads like this:

static int ext3_get_block(struct inode *inode, sector_t iblock,
struct buffer_head *bh_result, int create)

{
        handle_t *handle = journal_current_handle();
        int ret = 0;
        unsigned max_blocks = bh_result->b_size >> inode->i_blkbits;

        if (!handle)
                goto get_block;         /* A read */

        if (max_blocks == 1)
                goto get_block;         /* A single block get */

etc.

I guess I should replace the "if (!handle)" with "if (!create)"?

#10 lore raw
Quoted 38 lines · Rafael J. Wysocki
Quoted 14 lines · Andrew Morton
Quoted 13 lines · Rafael J. Wysocki
Quoted 13 lines

This should fix:

fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +1 -1
@@ -830,7 +830,7 @@ ext3_direct_io_get_blocks(struct inode *
 	handle_t *handle = journal_current_handle();
 	int ret = 0;
 
-	if (!handle)
+	if (!create)
 		goto get_block;		/* A read */
 
Quoted 5 lines
Quoted 3 lines

fs/ext3/inode.c ext3_direct_io_get_blocks(struct inode * · line 836

Quoted 1 line
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +0 -0
 	if (max_blocks == 1)

Er, it doesn't apply to either 2.6.16-rc5-mm3 or 2.6.16-rc6-mm1.

Nope, it applies OK to rc6-mm1.

Well, this means my rc6-mm1 is different to what you have. :-)

Anyway in "my" version there's a function ext3_get_block() which reads like this:

static int ext3_get_block(struct inode *inode, sector_t iblock,
struct buffer_head *bh_result, int create)

{
        handle_t *handle = journal_current_handle();
        int ret = 0;
        unsigned max_blocks = bh_result->b_size >> inode->i_blkbits;

        if (!handle)
                goto get_block;         /* A read */

        if (max_blocks == 1)
                goto get_block;         /* A single block get */
Quoted 3 lines

etc.

I guess I should replace the "if (!handle)" with "if (!create)"?

Yes. In "-mm" ext3_get_block() == ext3_direct_io_getblocks() in mainline.

I renamed ext3_direct_io_getblocks() to ext3_get_block() (in -mm) since both of them do same thing now. (both can deal with mapping multiple blocks).

Thanks,
Badari

#11 lore raw
Quoted 41 lines · Badari Pulavarty
Quoted 40 lines · Rafael J. Wysocki
Quoted 14 lines · Andrew Morton
Quoted 13 lines · Rafael J. Wysocki
Quoted 13 lines

This should fix:

fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +1 -1
@@ -830,7 +830,7 @@ ext3_direct_io_get_blocks(struct inode *
 	handle_t *handle = journal_current_handle();
 	int ret = 0;
 
-	if (!handle)
+	if (!create)
 		goto get_block;		/* A read */
 
Quoted 5 lines
Quoted 3 lines

fs/ext3/inode.c ext3_direct_io_get_blocks(struct inode * · line 836

Quoted 1 line
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +0 -0
 	if (max_blocks == 1)

Er, it doesn't apply to either 2.6.16-rc5-mm3 or 2.6.16-rc6-mm1.

Nope, it applies OK to rc6-mm1.

Well, this means my rc6-mm1 is different to what you have. :-)

Anyway in "my" version there's a function ext3_get_block() which reads like this:

static int ext3_get_block(struct inode *inode, sector_t iblock,
struct buffer_head *bh_result, int create)

{
        handle_t *handle = journal_current_handle();
        int ret = 0;
        unsigned max_blocks = bh_result->b_size >> inode->i_blkbits;

        if (!handle)
                goto get_block;         /* A read */

        if (max_blocks == 1)
                goto get_block;         /* A single block get */

etc.

Quoted 8 lines
Quoted 1 line

I guess I should replace the "if (!handle)" with "if (!create)"?

Yes. In "-mm" ext3_get_block() == ext3_direct_io_getblocks() in mainline.

I renamed ext3_direct_io_getblocks() to ext3_get_block() (in -mm) since both of them do same thing now. (both can deal with mapping multiple blocks).

Ah, I see, thanks.

Replacing the "if (!handle)" with "if (!create)" fixed the problem for me.

Greetings,
Rafael