BUG() in end_page_writeback(), stack overflows and system speed decrease with XFS over USB
Hello,
my machine is running very unstable since I use XFS on an external USB harddisc (855 GByte XFS partition on 1TByte). One problem was the stack overflows caused by the large stack use of XFS, USB, SCSI and VFS in Linux 2.6.23.13. NFS on XFS caused much more stack overflows. I think I got around the stack overflows by disabling preemption, SMP and NFS in Linux, but I am not sure about it. I think that I didn't got a message from the stack overflow detection after this. I also tried a Live-CD (KNOPPIX), but there are the same problems. I exchanged some of the hardware. XFS is decreasing system performance. I use the Linux VDR with DVB-S which seems to increase the problems. I was able to record 3 high bandwidth streams in parallel before using XFS. Now it has problems to record one high bandwidth stream. The system got a little bit usable after I changed the IO scheduler to deadline. It is difficult to get a good backtrace of the kernel crash, because the backlog is not saved on the internal harddisc (reiserfs and ext3). I was able to find out that XFS triggers a BUG() in end_page_writeback() at mm/filemap.c:552:
void end_page_writeback(struct page *page)
{
if (!TestClearPageReclaim(page) || rotate_reclaimable_page(page)) {
if (!test_clear_page_writeback(page))
BUG();
}
smp_mb__after_clear_bit();
wake_up_page(page, PG_writeback);
}The backtrace looks like this (Sorry, I needed to write it down from screen and I don't have everything):
end_page_writeback()end_buffer_async_write()update_stats_wait_end()xfs_setfilesize()
xfs_???_dealloc()xfs_destroy_ioend()run_workqueue()
After searching in the code I found:
/* TODO: cleanup count and page_dirty */
It seems that page_dirty may be handled wrong and could cause the problem, but I don't know the purpose of this stuff. The same comment is in the latest source code from GIT.
After running the system for while, I was able to trigger the kernel crash by starting "sync" in the command line.
My stack traces includes often dvb_dmx_swfilter_packets(), do_IRQ()/tasklets and sys_write()/vfs_write(). I can't scroll up in most situations. Can anyone help me?
Is there an easy way to backup the data or replace the file system without kernel crash in between?
Best regards
Juergen Urban
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Juergen Urban
Hello,
my machine is running very unstable since I use XFS on an external USB harddisc (855 GByte XFS partition on 1TByte). One problem was the stack overflows caused by the large stack use of XFS, USB, SCSI and VFS in Linux 2.6.23.13. NFS on XFS caused much more stack overflows. I think I got around the stack overflows by disabling preemption, SMP and NFS in Linux, but I am not sure about it. I think that I didn't got a message from the stack overflow detection after this.
Are you on 4k stacks? To be honest I'd still expect things to be mostly ok stack-wise even if so.
Quoted 5 lines
I also tried a Live-CD (KNOPPIX), but there are the same problems. I exchanged some of the hardware. XFS is decreasing system performance. I use the Linux VDR with DVB-S which seems to increase the problems. I was able to record 3 high bandwidth streams in parallel before using XFS.
Really, you could record 3 parallel high-def TV streams to ext3 via USB? I guess I'm a little surprised...
Quoted 7 lines
Now it has problems to record one high bandwidth stream. The system got a little bit usable after I changed the IO scheduler to deadline. It is difficult to get a good backtrace of the kernel crash, because the backlog is not saved on the internal harddisc (reiserfs and ext3). I was able to find out that XFS triggers a
BUG()inend_page_writeback()atmm/filemap.c:552:void
end_page_writeback(struct page *page)
Quoted 8 lines
{
if (!TestClearPageReclaim(page) || rotate_reclaimable_page(page)) {
if (!test_clear_page_writeback(page))
BUG();
}
smp_mb__after_clear_bit();
wake_up_page(page, PG_writeback);
}Regarding the bug, if there is any way to test a kernel newer than .23, I'd start there; I don't know offhand of a bug that was fixed here, but .23 was a long time ago...
Quoted 14 lines
The backtrace looks like this (Sorry, I needed to write it down from screen and I don't have everything):
end_page_writeback()end_buffer_async_write()update_stats_wait_end()xfs_setfilesize()
xfs_???_dealloc()xfs_destroy_ioend()run_workqueue()After searching in the code I found:
/* TODO: cleanup count andpage_dirty*/
Quoted 10 lines
It seems that
page_dirtymay be handled wrong and could cause the problem, but I don't know the purpose of this stuff. The same comment is in the latest source code from GIT.
After running the system for while, I was able to trigger the kernel crash by starting "sync" in the command line.
My stack traces includes oftendvb_dmx_swfilter_packets(), do_IRQ()/tasklets and sys_write()/vfs_write(). I can't scroll up in most situations. Can anyone help me?
Is there an easy way to backup the data or replace the file system without kernel crash in between?
You should certainly be able to copy data off xfs via usb; if it's failing, I guess we'll need more info to find out why, but I'd suggest at least booting a newer livecd to do that copy and see if things fare better.
-Eric
Quoted 8 lines
Best regards
Juergen Urban_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · On Thursday 19 November 2009 19:00:16 Eric Sandeen wrote:
Quoted 9 lines · Juergen Urban
Hello,
my machine is running very unstable since I use XFS on an external USB harddisc (855 GByte XFS partition on 1TByte). One problem was the stack overflows caused by the large stack use of XFS, USB, SCSI and VFS in Linux 2.6.23.13. NFS on XFS caused much more stack overflows. I think I got around the stack overflows by disabling preemption, SMP and NFS in Linux, but I am not sure about it. I think that I didn't got a message from the stack overflow detection after this.
Quoted 2 lines
Are you on 4k stacks? To be honest I'd still expect things to be mostly ok stack-wise even if so.
No, I am using 8k stacks.
Quoted 10 lines
Quoted 6 lines
I also tried a Live-CD (KNOPPIX), but there are the same problems. I exchanged some of the hardware. XFS is decreasing system performance. I use the Linux VDR with DVB-S which seems to increase the problems. I was able to record 3 high bandwidth streams in parallel before using XFS.
Really, you could record 3 parallel high-def TV streams to ext3 via USB? I guess I'm a little surprised...
No, I meant that I was able to record 3 high bandwidth SDTV streams on the internal hard disc with ext3. Then I've got an external USB drive and formatted it with XFS, because someone told me that XFS is running stable with VDR on an internal hard disc.
Quoted 17 lines
Quoted 8 lines
Now it has problems to record one high bandwidth stream. The system got a little bit usable after I changed the IO scheduler to deadline. It is difficult to get a good backtrace of the kernel crash, because the backlog is not saved on the internal harddisc (reiserfs and ext3). I was able to find out that XFS triggers a
BUG()inend_page_writeback()atmm/filemap.c:552:void
end_page_writeback(struct page *page)Quoted 8 lines
{ if (!TestClearPageReclaim(page) || rotate_reclaimable_page(page)) { if (!test_clear_page_writeback(page)) BUG(); } smp_mb__after_clear_bit(); wake_up_page(page, PG_writeback); }
Quoted 3 lines
Regarding the bug, if there is any way to test a kernel newer than .23, I'd start there; I don't know offhand of a bug that was fixed here, but .23 was a long time ago...
Now I tried linux-2.6.31.6. My system hangs in the start scripts. Maybe this is caused by network scripts. I got the message that ehci_hcd need to be loaded before uhci_hcd and ohci_hcd. I skipped uhci_hcd and ohci_hcd in /etc/discover.conf. Now I have a higher performance with linux-2.6.23.13 and I can record 3 normal streams in parallel on the with USB and XFS. But it is still unstable. The last error what I got was in block_prepare_write (fs/buffer.c). This caused follow up errors in do_invalidate_page() called by xfs_get_blocks().
Sometimes there are file system deadlocks. I can do everything, but not access the file system. Every try to access the file system leads to a deadlock of the program. This normally happens after a kernel exception.
Quoted 26 lines
Quoted 15 lines
The backtrace looks like this (Sorry, I needed to write it down from screen and I don't have everything):
end_page_writeback()end_buffer_async_write()update_stats_wait_end()xfs_setfilesize()
xfs_???_dealloc()xfs_destroy_ioend()run_workqueue()After searching in the code I found:
/* TODO: cleanup count andpage_dirty*/Quoted 10 lines
It seems that
page_dirtymay be handled wrong and could cause the problem, but I don't know the purpose of this stuff. The same comment is in the latest source code from GIT.
After running the system for while, I was able to trigger the kernel crash by starting "sync" in the command line.
My stack traces includes oftendvb_dmx_swfilter_packets(), do_IRQ()/tasklets and sys_write()/vfs_write(). I can't scroll up in most situations. Can anyone help me?
Is there an easy way to backup the data or replace the file system without kernel crash in between?
Quoted 4 lines
You should certainly be able to copy data off xfs via usb; if it's failing, I guess we'll need more info to find out why, but I'd suggest at least booting a newer livecd to do that copy and see if things fare better.
My idea was to shrink it and create a new partition where I can copy the data. As far as I understand I need to mount it for the shrink process, so I may have the problem of kernel exceptions while shrinking.
Quoted 11 lines
-Eric
Quoted 7 lines
Best regards
Juergen Urban_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Juergen Urban
Quoted 10 lines · On Thursday 19 November 2009 19:00:16 Eric Sandeen wrote:
Quoted 9 lines · Juergen Urban
Hello,
my machine is running very unstable since I use XFS on an external USB harddisc (855 GByte XFS partition on 1TByte). One problem was the stack overflows caused by the large stack use of XFS, USB, SCSI and VFS in Linux 2.6.23.13. NFS on XFS caused much more stack overflows. I think I got around the stack overflows by disabling preemption, SMP and NFS in Linux, but I am not sure about it. I think that I didn't got a message from the stack overflow detection after this.
Quoted 4 lines
Quoted 2 lines
Are you on 4k stacks? To be honest I'd still expect things to be mostly ok stack-wise even if so.
No, I am using 8k stacks.
Hmm.
...
Quoted 11 lines
Now I tried linux-2.6.31.6. My system hangs in the start scripts. Maybe this is caused by network scripts. I got the message that
ehci_hcdneed to be loaded beforeuhci_hcdandohci_hcd. I skippeduhci_hcdandohci_hcdin/etc/discover.conf. Now I have a higher performance with linux-2.6.23.13 and I can record 3 normal streams in parallel on the with USB and XFS. But it is still unstable. The last error what I got was inblock_prepare_write(fs/buffer.c). This caused follow up errors indo_invalidate_page()called byxfs_get_blocks().
Sometimes there are file system deadlocks. I can do everything, but not access the file system. Every try to access the file system leads to a deadlock of the program. This normally happens after a kernel exception.
I guess including the kernel messages you see would help us know what might be going on.
...
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
Is there an easy way to backup the data or replace the file system without kernel crash in between?
You should certainly be able to copy data off xfs via usb; if it's failing, I guess we'll need more info to find out why, but I'd suggest at least booting a newer livecd to do that copy and see if things fare better.
My idea was to shrink it and create a new partition where I can copy the data. As far as I understand I need to mount it for the shrink process, so I may have the problem of kernel exceptions while shrinking.
You can't shrink xfs, if that's what you mean.
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 12 lines · Eric Sandeen
Quoted 11 lines · Juergen Urban
Quoted 10 lines · On Thursday 19 November 2009 19:00:16 Eric Sandeen wrote:
Quoted 9 lines · Juergen Urban
Hello,
my machine is running very unstable since I use XFS on an external USB harddisc (855 GByte XFS partition on 1TByte). One problem was the stack overflows caused by the large stack use of XFS, USB, SCSI and VFS in Linux 2.6.23.13. NFS on XFS caused much more stack overflows. I think I got around the stack overflows by disabling preemption, SMP and NFS in Linux, but I am not sure about it. I think that I didn't got a message from the stack overflow detection after this.
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
Are you on 4k stacks? To be honest I'd still expect things to be mostly ok stack-wise even if so.
No, I am using 8k stacks.
Hmm.
BTW if you are still seeing stack overflows when testing w/ the newer kernel, we can use some tracing to see what the stack backtrace is:
sysctl -w kernel.stack_tracer_enabled=1
mount -t debugfs none /sys/kernel/debug/
and then look in:
/sys/kernel/debug/tracing/stack_trace/sys/kernel/debug/tracing/stack_max_size
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 24 lines · On Friday 20 November 2009 18:08:12 Eric Sandeen wrote:
Quoted 13 lines · Eric Sandeen
Quoted 12 lines · Juergen Urban
Quoted 11 lines · On Thursday 19 November 2009 19:00:16 Eric Sandeen wrote:
Quoted 9 lines · Juergen Urban
Hello,
my machine is running very unstable since I use XFS on an external USB harddisc (855 GByte XFS partition on 1TByte). One problem was the stack overflows caused by the large stack use of XFS, USB, SCSI and VFS in Linux 2.6.23.13. NFS on XFS caused much more stack overflows. I think I got around the stack overflows by disabling preemption, SMP and NFS in Linux, but I am not sure about it. I think that I didn't got a message from the stack overflow detection after this.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Are you on 4k stacks? To be honest I'd still expect things to be mostly ok stack-wise even if so.
No, I am using 8k stacks.
Hmm.
BTW if you are still seeing stack overflows when testing w/ the newer kernel, we can use some tracing to see what the stack backtrace is:
Quoted 8 lines
sysctl -w kernel.stack_tracer_enabled=1
mount -t debugfs none/sys/kernel/debug/and then look in:
/sys/kernel/debug/tracing/stack_trace/sys/kernel/debug/tracing/stack_max_size
I don't think that I still get stack overflows in Linux 2.6.23.13.
Now I downloaded the new KNOPPIX with Linux 2.6.31.6. I've got also crashes with it, but this also could be a stack overflow, because the kernel configuration includes SMP and preemption. The problem here is that I don't see any kernel message. I just see the blinking keyboard or sometimes even nothing. To get the error I just need to play video with mplayer and don't need to write to the file system at all. I need to wait until I've got a good backtrace, but this will be with Linux 2.6.23.13.
The console tools of my installed Debian can't configure or use /dev/tty[0-9]* of Linux 2.6.31.6. The ioctls are not supported. I installed a newer version of the package, but this doesn't help. I copied the ".config" from Linux 2.6.23.13 to 2.6.31.6. Maybe this is not compatible.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · On Samstag, 21. November 2009 Juergen Urban wrote:
To get the error I just need to play video with mplayer and don't need to write to the file system at all.
This really sounds like a broken hardware. Have you tried running MEMTEST?
mfg zmi
Signature · 5 lines
// Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0660 / 415 65 31 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · On Saturday 21 November 2009 11:51:23 Michael Monnerie wrote:
Quoted 2 lines · On Samstag, 21. November 2009 Juergen Urban wrote:
To get the error I just need to play video with mplayer and don't need to write to the file system at all.
This really sounds like a broken hardware. Have you tried running MEMTEST?
mfg zmi
At the first moment I also thought that it is a hardware bug. I already ran MEMTEST for some hours, but it doesn't find anything. I replaced the main board, the CPU and the power supply. I also tried to replace/remove RAM. But I've done this while chasing for the stack overflow. But maybe this a problem which I introduced by replacing the hardware. So I will try to change all the hardware back.
Can someone explain the purpose of the BUG() in end_page_writeback()? Can I remove the line?
Will XFS also work if I disable all address operations? What is the purpose of the address operations?
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · Juergen Urban
Can someone explain the purpose of the
BUG()inend_page_writeback()? Can I remove the line?
A page that is under writeback is supposed to have the PG_writeback flag set. Hence when writeback is completed (i.e the page is now clean) we need to clear the PG_writeback bit. The BUG is triggered if we are ending writeback on a page that does not have PG_writeback set. IOWs, something is seriously wrong, and could be a memory error or memory corruption.
Quoted 1 line
Will XFS also work if I disable all address operations?
No.
Quoted 1 line
What is the purpose of the address operations?
Reading Documentation/filesystems/vfs.txt might answer your questions....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs