Skip to contentSkip to conversation

XFS internal error xfs_da_do_buf(1) at line 2015 of file fs/xfs/xfs_da_btree.c

DISCUSSION Adam Nielsen 5 participants 8 messages Jan 4, 2009 – Jan 5, 2009

lore
Share & tools
lore
#0 lore raw

Hi all,

I'm having a recurring problem with XFS which started about a day ago. All of a sudden when reading a certain part of the disk (not sure where, but my nightly backups trigger it) I get an infinite loop of these messages appearing in my logs:

xfs_da_do_buf: bno 8388608
dir: inode 3087268096
Filesystem "md0": XFS internal error xfs_da_do_buf(1) at line 2015 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff802eba63
Pid: 4445, comm: metalog Tainted: P 2.6.28-rc2 #3

Call Trace:
 [<ffffffff802eba63>] xfs_da_read_buf+0x24/0x29
 [<ffffffff802eb6aa>] xfs_da_do_buf+0x2d2/0x621
 [<ffffffff80267fb0>] balance_dirty_pages_ratelimited_nr+0x300/0x329
 [<ffffffff802a52af>] block_write_end+0x4a/0x54
 [<ffffffff802eba63>] xfs_da_read_buf+0x24/0x29
 [<ffffffff802ecd0a>] xfs_da_node_lookup_int+0x5b/0x225
 [<ffffffff802ecd0a>] xfs_da_node_lookup_int+0x5b/0x225
 [<ffffffff802f2908>] xfs_dir2_node_lookup+0x43/0xe7
 [<ffffffff802edbff>] xfs_dir2_isleaf+0x19/0x4a
 [<ffffffff802ee357>] xfs_dir_lookup+0x10f/0x14f
 [<ffffffff80297a74>] __d_lookup+0x11a/0x143
 [<ffffffff80314b4b>] xfs_lookup+0x48/0xa5
 [<ffffffff8031d06e>] xfs_vn_lookup+0x3c/0x78
 [<ffffffff8028fafa>] __lookup_hash+0xfa/0x11e
 [<ffffffff8029288d>] do_filp_open+0x159/0x7d7
 [<ffffffff80507856>] _spin_unlock+0x10/0x2a
 [<ffffffff8029a608>] alloc_fd+0x112/0x123
 [<ffffffff8028701e>] do_sys_open+0x48/0xcc
 [<ffffffff8020b3bb>] system_call_fastpath+0x16/0x1b

It doesn't seem to interfere with filesystem use, but metalog is logging thousands of these messages into the system log files (the log files grow at about 1MB/minute.)

Does anyone know what this error means? Do I need to reformat the filesystem?

I've restarted a few times and the error goes away until the next nightly backup triggers it again. Killing metalog does seem to stop the messages, so perhaps one of the log files is the culprit? I'm not sure how to map that inode or bno back to a filename. It's always the same bno/inode and always reports metalog as the offending program.

Any suggestions how to go about diagnosing this problem?

Many thanks,
Adam.

#1 lore raw
Quoted 7 lines · Adam Nielsen

Hi all,

I'm having a recurring problem with XFS which started about a day ago. All of a sudden when reading a certain part of the disk (not sure where, but my nightly backups trigger it) I get an infinite loop of these messages appearing in my logs:

Quoted 5 lines

xfs_da_do_buf: bno 8388608
dir: inode 3087268096
Filesystem "md0": XFS internal error xfs_da_do_buf(1) at line 2015 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff802eba63
Pid: 4445, comm: metalog Tainted: P 2.6.28-rc2 #3

This is a typical result of a power loss scenario with write caches enabled and without barriers. Given that md can't pass through barriers did you disable the write caches on your disk?

Quoted 1 line

Does anyone know what this error means? Do I need to reformat the filesystem?

Run xfs_repair over it to fix up the directory, and make sure to configure your disks properly so that it doesn't happen again..

#2 lore raw
Quoted 3 lines

This is a typical result of a power loss scenario with write caches enabled and without barriers. Given that md can't pass through barriers did you disable the write caches on your disk?

No, I didn't realise I had to do that...in fact I didn't even realise SATA disks *had* write caches, I thought the cache was for reading only...

Quoted 2 lines

Run xfs_repair over it to fix up the directory, and make sure to configure your disks properly so that it doesn't happen again..

Will do, thanks for the advice! Is there any standard way to disable write caching on a SATA disk? hdparm -W seems to do the trick, but then I can't run that until the system is up and running, leaving a small window of opportunity for something to go wrong.

Thanks again,
Adam.

#3 lore raw
Quoted 3 lines · Adam Nielsen

No, I didn't realise I had to do that...in fact I didn't even realise SATA disks *had* write caches, I thought the cache was for reading only...

Which would be the better default (it's what high-end disks generally do by default). I've been wondering for a while how we can make default setups in the presence of lvm/dm more secure, but there hasn't been any progress yet.

Quoted 7 lines
Quoted 2 lines

Run xfs_repair over it to fix up the directory, and make sure to configure your disks properly so that it doesn't happen again..

Will do, thanks for the advice! Is there any standard way to disable write caching on a SATA disk? hdparm -W seems to do the trick, but then I can't run that until the system is up and running, leaving a small window of opportunity for something to go wrong.

On Debian based systems you can add -W0 to /etc/default/hdparm and it gets executed before the root filesystem is remounted read-write, I'm not sure how other distributions handle it.

#4 lore raw
Quoted 3 lines

On Debian based systems you can add -W0 to /etc/default/hdparm and it gets executed before the root filesystem is remounted read-write, I'm not sure how other distributions handle it.

Generally they avoid setting -W0 because it ruins performance and can be very bad for disk lifetime. The barriers code is there for a reason.

Of course certain distributions default to using LVM for all their file systems which is completely and mindbogglingly bogus. That both messes up barriers in some cases and takes a good 10-20% off performance when I've benched it.

LVM is cool - if you need it, most people don't.

Alan

#5 lore raw
Quoted 2 lines · Alan Cox

Generally they avoid setting -W0 because it ruins performance and can be very bad for disk lifetime. The barriers code is there for a reason.

We've done measurements and for modern NCQ/TCQ disks the performance for cache off vs cache on + barriers is close. For ext3 barriers is generally slightly faster, and for XFS it's even or sometimes even cache off is faster depending on the workload.

Quoted 4 lines

Of course certain distributions default to using LVM for all their file systems which is completely and mindbogglingly bogus. That both messes up barriers in some cases and takes a good 10-20% off performance when I've benched it.

The thing is that there's no reason for that at all with just a single underlying disk. There is absolutely no reason for not passing through barriers, and there's also no reason why it should be any slower than our most trivially volume manager, the partition remapping code. In fact there's no reason trivial device mapper tables couldn't be handled by the partition remapping code..

#6 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 5 lines

Of course certain distributions default to using LVM for all their file systems which is completely and mindbogglingly bogus. That both messes up barriers in some cases and takes a good 10-20% off performance when I've benched it.

The thing is that there's no reason for that at all with just a single underlying disk.

I've submitted patches to do exactly that in DM some time ago. Unfortunately they still didn't make it in (as of 2.6.29 git) for unknown reasons.

-Andi

Signature · 1 line
[email protected]
#7 lore raw
In reply to Alan Cox
Quoted 7 lines · Alan Cox
Quoted 3 lines

On Debian based systems you can add -W0 to /etc/default/hdparm and it gets executed before the root filesystem is remounted read-write, I'm not sure how other distributions handle it.

Generally they avoid setting -W0 because it ruins performance and can be very bad for disk lifetime. The barriers code is there for a reason.

First time I read about a possible negative impact of disabled write-cache to disk lifetime. Do you know of an article for further reading?

Signature · 1 line
left blank, right bald