2.6.16-rc5-mm3: spinlock bad magic on CPU#0 on AMD64
Hi,
With the 2.6.16-rc5-mm3 kernel w/ the patch
revert-x86_64-mm-i386-early-alignment.patch
applied I'm able to hang my box (Asus L5D, 1 CPU, x86-64 kernel) solid by running OpenOffice.org from under KDE (100% of the time but on one user account only). Before it hangs I get something like this on the serial console:
BUG: spinlock bad magic on CPU#0, soffice.bin/5293 lock: ffff81005e174e28, .magic: 000001ff, .owner: .5).@4).06)./0, .owner_cpu: -2141827648 BUG: spinlock lockup on CPU#0, soffice.bin/5293, ffff81005e174e28
Greetings,
Rafael
Quoted 7 lines · Rafael J. Wysocki
Hi,
With the 2.6.16-rc5-mm3 kernel w/ the patch
revert-x86_64-mm-i386-early-alignment.patch
Quoted 8 lines
applied I'm able to hang my box (Asus L5D, 1 CPU, x86-64 kernel) solid by running OpenOffice.org from under KDE (100% of the time but on one user account only). Before it hangs I get something like this on the serial console:
BUG: spinlock bad magic on CPU#0, soffice.bin/5293 lock: ffff81005e174e28, .magic: 000001ff, .owner: .5).@4).06)./0, .owner_cpu: -2141827648 BUG: spinlock lockup on CPU#0, soffice.bin/5293, ffff81005e174e28
Is it a !CONFIG_SMP kernel?
There's no stack trace?
Quoted 9 lines · Andrew Morton
Quoted 8 lines · Rafael J. Wysocki
With the 2.6.16-rc5-mm3 kernel w/ the patch
revert-x86_64-mm-i386-early-alignment.patch
applied I'm able to hang my box (Asus L5D, 1 CPU, x86-64 kernel) solid by running OpenOffice.org from under KDE (100% of the time but on one user account only). Before it hangs I get something like this on the serial console:
Quoted 6 lines
Quoted 4 lines
BUG: spinlock bad magic on CPU#0, soffice.bin/5293 lock: ffff81005e174e28, .magic: 000001ff, .owner: .5).@4).06)./0, .owner_cpu: -2141827648 BUG: spinlock lockup on CPU#0, soffice.bin/5293, ffff81005e174e28Is it a !CONFIG_SMP kernel?
Yes.
Quoted 1 line
There's no stack trace?
Well, probably there was one but it didn't appear on the serial console because the console loglevel was too low. I'll try to increase console_loglevel in spin_bug() and see what happens.
[BTW, looking at the relevant code, in my case .owner_cpu appears to have only the most significant bit set (??).]
[Dropped [email protected] from the Cc list, it's probably OT there]
Quoted 17 lines · Rafael J. Wysocki
Quoted 9 lines · Andrew Morton
Quoted 8 lines · Rafael J. Wysocki
With the 2.6.16-rc5-mm3 kernel w/ the patch
revert-x86_64-mm-i386-early-alignment.patch
applied I'm able to hang my box (Asus L5D, 1 CPU, x86-64 kernel) solid by running OpenOffice.org from under KDE (100% of the time but on one user account only). Before it hangs I get something like this on the serial console:
Quoted 6 lines
Quoted 4 lines
BUG: spinlock bad magic on CPU#0, soffice.bin/5293 lock: ffff81005e174e28, .magic: 000001ff, .owner: .5).@4).06)./0, .owner_cpu: -2141827648 BUG: spinlock lockup on CPU#0, soffice.bin/5293, ffff81005e174e28Is it a !CONFIG_SMP kernel?
Quoted 7 lines
Yes.
Quoted 1 line
There's no stack trace?
Well, probably there was one but it didn't appear on the serial console because the console loglevel was too low. I'll try to increase
console_loglevelinspin_bug()and see what happens.
Done, and now it looks like this:
BUG: spinlock bad magic on CPU#0, soffice.bin/5192 lock: ffff81005f79ae28, .magic: 000001ff, .owner: 1..1..|1. |1.|1..|1.�1..|1..1. 1./-1,
.owner_cpu: -2141838208
Call Trace: <ffffffff80210383>{__alloc_pages+99} <ffffffff802156c3>{spin_bug+195}
<ffffffff802077dc>{_raw_spin_lock+44} <ffffffff80270a4e>{_spin_lock+30}
<ffffffff8033712d>{journal_extend+77} <ffffffff80327255>{ext3_get_block+165}
<ffffffff8022c2f9>{do_mpage_readpage+425} <ffffffff80270cc4>{_write_unlock_irq+20}
<ffffffff8020cce2>{add_to_page_cache+162} <ffffffff8023fdee>{mpage_readpages+254}
<ffffffff803271b0>{ext3_get_block+0} <ffffffff803271b0>{ext3_get_block+0}
<ffffffff803146df>{get_cnode+95} <ffffffff8020a3bb>{get_page_from_freelist+619}
<ffffffff80210383>{__alloc_pages+99} <ffffffff80323c1a>{ext3_readpages+26}
<ffffffff80214030>{__do_page_cache_readahead+416} <ffffffff80213c12>{poison_obj+66}
<ffffffff80232058>{wake_up_bit+40} <ffffffff80243152>{unlock_buffer+18}
<ffffffff80315bb8>{reiserfs_prepare_for_journal+104}
<ffffffff802b6ab4>{do_page_cache_readahead+100} <ffffffff80215942>{filemap_nopage+322}
<ffffffff80208b2c>{__handle_mm_fault+1004} <ffffffff80270e7d>{_spin_unlock_irqrestore+29}
<ffffffff8020ae99>{do_page_fault+1257} <ffffffff8026af8d>{error_exit+0}
<ffffffff802ffb40>{reiserfs_copy_from_user_to_file_region+80}
<ffffffff80302446>{reiserfs_file_write+6102} <ffffffff802f8f4e>{reiserfs_add_entry+1054}
<ffffffff8033c1ff>{journal_cancel_revoke+351} <ffffffff80213c12>{poison_obj+66}
<ffffffff80236d27>{cache_free_debugcheck+711} <ffffffff80335734>{journal_stop+772}
<ffffffff80270f30>{_spin_unlock+16} <ffffffff802193a2>{vfs_write+226}
<ffffffff80219c80>{sys_write+80} <ffffffff8026d234>{cstar_do_call+27}
BUG: spinlock lockup on CPU#0, soffice.bin/5192, ffff81005f79ae28Quoted 2 lines · Rafael J. Wysocki
Done, and now it looks like this:
Helps a lot, thanks.
Quoted 17 lines
BUG: spinlock bad magic on CPU#0, soffice.bin/5192
lock: ffff81005f79ae28, .magic: 000001ff, .owner: 1..1..|1. |1.|1..|1.___1..|1..1. 1./-1,
.owner_cpu: -2141838208
Call Trace: <ffffffff80210383>{__alloc_pages+99} <ffffffff802156c3>{spin_bug+195}
<ffffffff802077dc>{_raw_spin_lock+44} <ffffffff80270a4e>{_spin_lock+30}
<ffffffff8033712d>{journal_extend+77} <ffffffff80327255>{ext3_get_block+165}
<ffffffff8022c2f9>{do_mpage_readpage+425} <ffffffff80270cc4>{_write_unlock_irq+20}
<ffffffff8020cce2>{add_to_page_cache+162} <ffffffff8023fdee>{mpage_readpages+254}
<ffffffff803271b0>{ext3_get_block+0} <ffffffff803271b0>{ext3_get_block+0}
<ffffffff803146df>{get_cnode+95} <ffffffff8020a3bb>{get_page_from_freelist+619}
<ffffffff80210383>{__alloc_pages+99} <ffffffff80323c1a>{ext3_readpages+26}
<ffffffff80214030>{__do_page_cache_readahead+416} <ffffffff80213c12>{poison_obj+66}
<ffffffff80232058>{wake_up_bit+40} <ffffffff80243152>{unlock_buffer+18}
<ffffffff80315bb8>{reiserfs_prepare_for_journal+104}
<ffffffff802b6ab4>{do_page_cache_readahead+100} <ffffffff80215942>{filemap_nopage+322}
<ffffffff80208b2c>{__handle_mm_fault+1004} <ffffffff80270e7d>{_spin_unlock_irqrestore+29}Quoted 8 lines
<ffffffff8020ae99>{do_page_fault+1257} <ffffffff8026af8d>{error_exit+0}
<ffffffff802ffb40>{reiserfs_copy_from_user_to_file_region+80}
<ffffffff80302446>{reiserfs_file_write+6102} <ffffffff802f8f4e>{reiserfs_add_entry+1054}
<ffffffff8033c1ff>{journal_cancel_revoke+351} <ffffffff80213c12>{poison_obj+66}
<ffffffff80236d27>{cache_free_debugcheck+711} <ffffffff80335734>{journal_stop+772}
<ffffffff80270f30>{_spin_unlock+16} <ffffffff802193a2>{vfs_write+226}
<ffffffff80219c80>{sys_write+80} <ffffffff8026d234>{cstar_do_call+27}
BUG: spinlock lockup on CPU#0, soffice.bin/5192, ffff81005f79ae28It's a pretty vile backtrace. I supposed you have CONFIG_FRAME_POINTER=n.
Still. It seems that what's happened is that we took a pagefault while reiserfs had a transaction open. The fault is against a mmapped ext3 file and we ended up in the recently-reworked ext3_get_block() which tests journal_current_handle() to work out whether we're in a write or a read. oops. The presence of reiserfs journal_info makes it decide it's a write, not a read so it starts treating a reiserfs journal_info as an ext3 one.
The code used to work OK because it was only for direct-IO, which doesn't get recurred into like this. But it got used for regular I/O in -mm.
This should fix:
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +1 -1
@@ -830,7 +830,7 @@ ext3_direct_io_get_blocks(struct inode *handle_t *handle =journal_current_handle ();int ret =0 ; -if (!handle) +if (!create) goto get_block;/* A read */ if (max_blocks ==1 )
_
Quoted 2 lines · Andrew Morton
It's a pretty vile backtrace. I supposed you have
CONFIG_FRAME_POINTER=n.
Won't make a difference because the oops backtracer doesn't use them.
Quoted 9 lines
Still. It seems that what's happened is that we took a pagefault while reiserfs had a transaction open. The fault is against a mmapped ext3 file and we ended up in the recently-reworked
ext3_get_block()which testsjournal_current_handle()to work out whether we're in a write or a read. oops. The presence of reiserfsjournal_infomakes it decide it's a write, not a read so it starts treating a reiserfsjournal_infoas an ext3 one.The code used to work OK because it was only for direct-IO, which doesn't get recurred into like this. But it got used for regular I/O in -mm.
Oops. Can this happen in more situations?
-Andi
Quoted 8 lines · Andi Kleen
Quoted 8 lines
Still. It seems that what's happened is that we took a pagefault while reiserfs had a transaction open. The fault is against a mmapped ext3 file and we ended up in the recently-reworked
ext3_get_block()which testsjournal_current_handle()to work out whether we're in a write or a read. oops. The presence of reiserfsjournal_infomakes it decide it's a write, not a read so it starts treating a reiserfsjournal_infoas an ext3 one.
Quoted 4 lines
Quoted 2 lines
The code used to work OK because it was only for direct-IO, which doesn't get recurred into like this. But it got used for regular I/O in -mm.
Oops. Can this happen in more situations?
I don't _think_ so, but it's pretty scary. The code's been this way for a while.
Typical scenario:
reiserfs_journal_start - sets current->journal_info copy_from_user pagefault ext3_readpages()
As long as ext3_readpage[s]() doesn't try to start a transaction we're OK. And it shouldn't, if create==0.
Fortunately filemap_nopage() doesn't do atime updates. If it did, things would get messy.
We do have deadlock possibilities in there - filemap_nopage() does lock_page() inside journal_start(), whereas generic_file_write() does journal_start() inside lock_page(). Chris Mason and I have stared unhappily at that a few times. Hard to fix.
But I don't _think_ we have any more journal_start() recursions like this - ext3 tends to get pretty noisy if it detects that in unexpected places.
Quoted 49 lines · Andrew Morton
Quoted 2 lines · Rafael J. Wysocki
Done, and now it looks like this:
Helps a lot, thanks.
Quoted 17 lines
BUG: spinlock bad magic on CPU#0, soffice.bin/5192 lock: ffff81005f79ae28, .magic: 000001ff, .owner: 1..1..|1. |1.|1..|1.___1..|1..1. 1./-1, .owner_cpu: -2141838208 Call Trace: <ffffffff80210383>{__alloc_pages+99} <ffffffff802156c3>{spin_bug+195} <ffffffff802077dc>{_raw_spin_lock+44} <ffffffff80270a4e>{_spin_lock+30} <ffffffff8033712d>{journal_extend+77} <ffffffff80327255>{ext3_get_block+165} <ffffffff8022c2f9>{do_mpage_readpage+425} <ffffffff80270cc4>{_write_unlock_irq+20} <ffffffff8020cce2>{add_to_page_cache+162} <ffffffff8023fdee>{mpage_readpages+254} <ffffffff803271b0>{ext3_get_block+0} <ffffffff803271b0>{ext3_get_block+0} <ffffffff803146df>{get_cnode+95} <ffffffff8020a3bb>{get_page_from_freelist+619} <ffffffff80210383>{__alloc_pages+99} <ffffffff80323c1a>{ext3_readpages+26} <ffffffff80214030>{__do_page_cache_readahead+416} <ffffffff80213c12>{poison_obj+66} <ffffffff80232058>{wake_up_bit+40} <ffffffff80243152>{unlock_buffer+18} <ffffffff80315bb8>{reiserfs_prepare_for_journal+104} <ffffffff802b6ab4>{do_page_cache_readahead+100} <ffffffff80215942>{filemap_nopage+322} <ffffffff80208b2c>{__handle_mm_fault+1004} <ffffffff80270e7d>{_spin_unlock_irqrestore+29}Quoted 8 lines
<ffffffff8020ae99>{do_page_fault+1257} <ffffffff8026af8d>{error_exit+0} <ffffffff802ffb40>{reiserfs_copy_from_user_to_file_region+80} <ffffffff80302446>{reiserfs_file_write+6102} <ffffffff802f8f4e>{reiserfs_add_entry+1054} <ffffffff8033c1ff>{journal_cancel_revoke+351} <ffffffff80213c12>{poison_obj+66} <ffffffff80236d27>{cache_free_debugcheck+711} <ffffffff80335734>{journal_stop+772} <ffffffff80270f30>{_spin_unlock+16} <ffffffff802193a2>{vfs_write+226} <ffffffff80219c80>{sys_write+80} <ffffffff8026d234>{cstar_do_call+27} BUG: spinlock lockup on CPU#0, soffice.bin/5192, ffff81005f79ae28It's a pretty vile backtrace. I supposed you have
CONFIG_FRAME_POINTER=n.Still. It seems that what's happened is that we took a pagefault while reiserfs had a transaction open. The fault is against a mmapped ext3 file and we ended up in the recently-reworked
ext3_get_block()which testsjournal_current_handle()to work out whether we're in a write or a read. oops. The presence of reiserfsjournal_infomakes it decide it's a write, not a read so it starts treating a reiserfsjournal_infoas an ext3 one.The code used to work OK because it was only for direct-IO, which doesn't get recurred into like this. But it got used for regular I/O in -mm.
This should fix:
fs/ext3/inode.c ext3_direct_io_get_blocks(struct inode * · line 836
Quoted 8 lines
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +1
-1@@ -830,7 +830,7 @@ ext3_direct_io_get_blocks(struct inode *handle_t *handle =journal_current_handle ();int ret =0 ; -if (!handle) +if (!create) goto get_block;/* A read */ if (max_blocks ==1 )
Er, it doesn't apply to either 2.6.16-rc5-mm3 or 2.6.16-rc6-mm1.
There's no function ext3_direct_io_get_blocks() there in fs/ext3/inode.c, AFAICT.
Quoted 9 lines · Rafael J. Wysocki
Quoted 9 lines
This should fix:
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +0
-0@@ -830,7 +830,7 @@ ext3_direct_io_get_blocks(struct inode *handle_t *handle =journal_current_handle ();int ret =0 ;
Quoted 7 lines
fs/ext3/inode.c ext3_direct_io_get_blocks(struct inode * · line 836
Quoted 5 lines
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +1
-1-if (!handle) +if (!create) goto get_block;/* A read */ if (max_blocks ==1 )Er, it doesn't apply to either 2.6.16-rc5-mm3 or 2.6.16-rc6-mm1.
Nope, it applies OK to rc6-mm1.
But whatever - it's only a single line?
Quoted 14 lines · Andrew Morton
Quoted 13 lines · Rafael J. Wysocki
Quoted 13 lines
This should fix:
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +1
-1@@ -830,7 +830,7 @@ ext3_direct_io_get_blocks(struct inode *handle_t *handle =journal_current_handle ();int ret =0 ; -if (!handle) +if (!create) goto get_block;/* A read */
Quoted 5 lines
Quoted 3 lines
fs/ext3/inode.c ext3_direct_io_get_blocks(struct inode * · line 836
Quoted 1 line
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +0
-0if (max_blocks ==1 )Er, it doesn't apply to either 2.6.16-rc5-mm3 or 2.6.16-rc6-mm1.
Nope, it applies OK to rc6-mm1.
Well, this means my rc6-mm1 is different to what you have. :-)
Anyway in "my" version there's a function ext3_get_block() which reads like this:
static int ext3_get_block(struct inode *inode, sector_t iblock,
struct buffer_head *bh_result, int create)
{
handle_t *handle = journal_current_handle();
int ret = 0;
unsigned max_blocks = bh_result->b_size >> inode->i_blkbits;
if (!handle)
goto get_block; /* A read */
if (max_blocks == 1)
goto get_block; /* A single block get */etc.
I guess I should replace the "if (!handle)" with "if (!create)"?
Quoted 38 lines · Rafael J. Wysocki
Quoted 14 lines · Andrew Morton
Quoted 13 lines · Rafael J. Wysocki
Quoted 13 lines
This should fix:
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +1
-1@@ -830,7 +830,7 @@ ext3_direct_io_get_blocks(struct inode *handle_t *handle =journal_current_handle ();int ret =0 ; -if (!handle) +if (!create) goto get_block;/* A read */ Quoted 5 lines
Quoted 3 lines
fs/ext3/inode.c ext3_direct_io_get_blocks(struct inode * · line 836
Quoted 1 line
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +0
-0if (max_blocks ==1 )Er, it doesn't apply to either 2.6.16-rc5-mm3 or 2.6.16-rc6-mm1.
Nope, it applies OK to rc6-mm1.
Well, this means my rc6-mm1 is different to what you have. :-)
Anyway in "my" version there's a function
ext3_get_block()which reads like this:static int ext3_get_block(struct inode *inode,
sector_tiblock,
structbuffer_head*bh_result, int create){ handle_t *handle = journal_current_handle(); int ret = 0; unsigned max_blocks = bh_result->b_size >> inode->i_blkbits; if (!handle) goto get_block; /* A read */ if (max_blocks == 1) goto get_block; /* A single block get */
Quoted 3 lines
etc.
I guess I should replace the "if (!handle)" with "if (!create)"?
Yes. In "-mm" ext3_get_block() == ext3_direct_io_getblocks() in mainline.
I renamed ext3_direct_io_getblocks() to ext3_get_block() (in -mm) since both of them do same thing now. (both can deal with mapping multiple blocks).
Thanks,
Badari
Quoted 41 lines · Badari Pulavarty
Quoted 40 lines · Rafael J. Wysocki
Quoted 14 lines · Andrew Morton
Quoted 13 lines · Rafael J. Wysocki
Quoted 13 lines
This should fix:
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +1
-1@@ -830,7 +830,7 @@ ext3_direct_io_get_blocks(struct inode *handle_t *handle =journal_current_handle ();int ret =0 ; -if (!handle) +if (!create) goto get_block;/* A read */ Quoted 5 lines
Quoted 3 lines
fs/ext3/inode.c ext3_direct_io_get_blocks(struct inode * · line 836
Quoted 1 line
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-get-blocks-maping-multiple-blocks-at-a-once-journal-reentry-fix +0
-0if (max_blocks ==1 )Er, it doesn't apply to either 2.6.16-rc5-mm3 or 2.6.16-rc6-mm1.
Nope, it applies OK to rc6-mm1.
Well, this means my rc6-mm1 is different to what you have. :-)
Anyway in "my" version there's a function
ext3_get_block()which reads like this:static int ext3_get_block(struct inode *inode,
sector_tiblock,
structbuffer_head*bh_result, int create){ handle_t *handle = journal_current_handle(); int ret = 0; unsigned max_blocks = bh_result->b_size >> inode->i_blkbits; if (!handle) goto get_block; /* A read */ if (max_blocks == 1) goto get_block; /* A single block get */etc.
Quoted 8 lines
Quoted 1 line
I guess I should replace the "if (!handle)" with "if (!create)"?
Yes. In "-mm"
ext3_get_block()==ext3_direct_io_getblocks()in mainline.I renamed
ext3_direct_io_getblocks()toext3_get_block()(in -mm) since both of them do same thing now. (both can deal with mapping multiple blocks).
Ah, I see, thanks.
Replacing the "if (!handle)" with "if (!create)" fixed the problem for me.
Greetings,
Rafael