Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1
Hi,
from time to time i'm getting a lot of strange errors like these:
[1217894.093015] [1217894.150529] Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1 [1217894.169930] Call Trace: [1217894.189392] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [1217894.209515] [<ffffffff8117bcb7>] ? xfs_da_do_buf+0x557/0x620 [1217894.229591] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [1217894.249969] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [1217894.269840] [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197 [1217894.289755] [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197 [1217894.309385] [<ffffffff813755ed>] ? tcp_write_xmit+0x883/0x96c [1217894.328966] [<ffffffff8117f9c2>] ? xfs_dir2_block_lookup+0x18/0xb0 [1217894.348653] [<ffffffff8117e71c>] ? xfs_dir_lookup+0xd5/0x147 [1217894.368696] [<ffffffff811a11bd>] ? xfs_lookup+0x47/0xa3 [1217894.388523] [<ffffffff811a90d1>] ? xfs_vn_lookup+0x3c/0x7b [1217894.408019] [<ffffffff810c1de7>] ? __lookup_hash+0xfa/0x11e [1217894.427418] [<ffffffff810c51a8>] ? do_filp_open+0x208/0x934 [1217894.446855] [<ffffffff810c7242>] ? poll_select_copy_remaining+0xd0/0xf3 [1217894.466545] [<ffffffff810cd62a>] ? alloc_fd+0x67/0x10b [1217894.486469] [<ffffffff810b8a60>] ? do_sys_open+0x55/0x103 [1217894.506529] [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b [1217894.526534] ffff8803104df000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [1217894.546783] Filesystem "sdc3": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff8117bdea [1217894.546785] [1217894.606014] Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1 [1217894.626163] Call Trace: [1217894.645625] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [1217894.665465] [<ffffffff8117bcb7>] ? xfs_da_do_buf+0x557/0x620 [1217894.685302] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [1217894.705135] [<ffffffff810c7442>] ? poll_freewait+0x3d/0x8a [1217894.725179] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [1217894.745439] [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197 [1217894.765485] [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197 [1217894.785179] [<ffffffff8117f9c2>] ? xfs_dir2_block_lookup+0x18/0xb0 [1217894.805010] [<ffffffff8117e71c>] ? xfs_dir_lookup+0xd5/0x147 [1217894.824879] [<ffffffff811a11bd>] ? xfs_lookup+0x47/0xa3 [1217894.844689] [<ffffffff811a90d1>] ? xfs_vn_lookup+0x3c/0x7b [1217894.864661] [<ffffffff810c1c0f>] ? do_lookup+0xd5/0x1b3 [1217894.884555] [<ffffffff810c3d04>] ? __link_path_walk+0x8f5/0xda2 [1217894.904367] [<ffffffff813368e3>] ? sock_aio_write+0xd6/0xe1 [1217894.924229] [<ffffffff810c43df>] ? path_walk+0x66/0xcb [1217894.944091] [<ffffffff810c4512>] ? do_path_lookup+0x20/0x41 [1217894.963992] [<ffffffff810c4e39>] ? user_path_at+0x48/0x7f [1217894.984002] [<ffffffff81053986>] ? autoremove_wake_function+0x0/0x2e [1217895.004471] [<ffffffff81059edd>] ? ktime_get_ts+0x68/0xb2 [1217895.024155] [<ffffffff810bd518>] ? vfs_fstatat+0x2c/0x58 [1217895.042995] [<ffffffff810bd6a8>] ? sys_newlstat+0x11/0x30 [1217895.061008] [<ffffffff810baa7e>] ? vfs_write+0x134/0x169 [1217895.079137] [<ffffffff810bab6f>] ? sys_write+0x45/0x6e [1217895.096929] [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b [1217895.115018] ffff8803104df000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [1217895.133667] Filesystem "sdc3": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff8117bdea
I've already repaired the filesystem twice but i'm still getting these errors again.
Kernel: Vanilla 2.6.32.22
XFS Progs Debian Lenny: xfs_repair -Vxfs_repair version 2.9.8
Stfean
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Tue, 26 Oct 2010 08:25:20 +0200 vous écriviez:
Quoted 2 lines
I've already repaired the filesystem twice but i'm still getting these errors again.
I had this problem in the past, but only with much lower kernel versions, what does "xfs_info /dev/sdc3" says? What are the mount options?
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsAm 26.10.2010 09:22, schrieb Emmanuel Florac:
Quoted 1 line
xfs_info/dev/sdc3
Hi here are the details:
root@server361-han:~# xfs_info /dev/sdc3
meta-data=/dev/sdc3 isize=256 agcount=32, agsize=40038461 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=1281230752, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0root@server361-han:~# mount|grep sdc3
/dev/sdc3 on /sdc3 type xfs (rw,noatime,nodiratime,nobarrier,prjquota) /sdc3/serverbackup on /serverbackup type none (rw,bind) /sdc3/serverbackup_rootserver on /serverbackup_rootserver type none (rw,bind)
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Tue, 26 Oct 2010 09:24:57 +0200 vous écriviez:
Quoted 4 lines
/dev/sdc3on /sdc3 type xfs (rw,noatime,nodiratime,nobarrier,prjquota) /sdc3/serverbackup on /serverbackup type none (rw,bind) /sdc3/serverbackup_rootserver on /serverbackup_rootserver type none (rw,bind)
OK, looks fine to me. Aren't there any messages related to /dev/sdc in dmesg? Apparently it's a single drive, any IO errors? What's the smart status like?
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsQuoted 8 lines
Quoted 4 lines
/dev/sdc3on /sdc3 type xfs (rw,noatime,nodiratime,nobarrier,prjquota) /sdc3/serverbackup on /serverbackup type none (rw,bind) /sdc3/serverbackup_rootserver on /serverbackup_rootserver type none (rw,bind)OK, looks fine to me. Aren't there any messages related to
/dev/sdcin dmesg? Apparently it's a single drive, any IO errors? What's the smart status like?
No it is a Raid 5 (3Ware Controller). First Message in dmesg is (attention reverse order):
2010-10-26 02:07:42 [1217871.154330] Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1 2010-10-26 02:07:42 [1217871.178925] Call Trace: 2010-10-26 02:07:42 [1217871.202924] [] ? xfs_da_read_buf+0x24/0x29 2010-10-26 02:07:42 [1217871.226724] [] ? xfs_da_do_buf+0x557/0x620 2010-10-26 02:07:42 [1217871.249821] [] ? xfs_da_read_buf+0x24/0x29 2010-10-26 02:07:42 [1217871.272475] [] ? xfs_da_read_buf+0x24/0x29 2010-10-26 02:07:42 [1217871.294558] [] ? xfs_dir2_block_lookup_int+0x45/0x197 2010-10-26 02:07:42 [1217871.316206] [] ? xfs_dir2_block_lookup_int+0x45/0x197 2010-10-26 02:07:42 [1217871.337603] [] ? xfs_dir2_block_lookup+0x18/0xb0 2010-10-26 02:07:42 [1217871.358986] [] ? xfs_dir_lookup+0xd5/0x147 2010-10-26 02:07:42 [1217871.380585] [] ? xfs_lookup+0x47/0xa3 2010-10-26 02:07:42 [1217871.402313] [] ? xfs_vn_lookup+0x3c/0x7b 2010-10-26 02:07:42 [1217871.424465] [] ? do_lookup+0xd5/0x1b3 2010-10-26 02:07:42 [1217871.445750] [] ? __link_path_walk+0x8f5/0xda2 2010-10-26 02:07:42 [1217871.466323] [] ? sock_aio_write+0xd6/0xe1 2010-10-26 02:07:42 [1217871.487141] [] ? path_walk+0x66/0xcb 2010-10-26 02:07:42 [1217871.507783] [] ? do_path_lookup+0x20/0x41 2010-10-26 02:07:42 [1217871.528673] [] ? user_path_at+0x48/0x7f 2010-10-26 02:07:42 [1217871.549518] [] ? autoremove_wake_function+0x0/0x2e 2010-10-26 02:07:42 [1217871.569930] [] ? vfs_fstatat+0x2c/0x58 2010-10-26 02:07:42 [1217871.589714] [] ? _atomic_dec_and_lock+0x33/0x50 2010-10-26 02:07:42 [1217871.609072] [] ? sys_newlstat+0x11/0x30 2010-10-26 02:07:42 [1217871.627679] [] ? mntput_no_expire+0x23/0xdf 2010-10-26 02:07:42 [1217871.645558] [] ? filp_close+0x5b/0x62 2010-10-26 02:07:41 [1217870.507397] ffff8803104df000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 2010-10-26 02:07:41 [1217870.530516] Filesystem "sdc3": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff8117bdea 2010-10-26 02:07:41 [1217870.530518] 2010-10-26 02:07:41 [1217870.603393] Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1 2010-10-26 02:07:41 [1217870.628925] Call Trace:
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Tue, 26 Oct 2010 09:53:34 +0200
Stefan Priebe - Profihost AG <[email protected]> écrivait:
Quoted 1 line
No it is a Raid 5 (3Ware Controller).
What model? Which firmware? If it's a 96x0 model, notice that 4.x.x firmwares added lots of performance enhancements and solved many problems.
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsHi,
it is a 9650SE-8LPML with Firmware: FE9X 3.06.00.003.
So you mean i should upgrade to 4.x Firmware? Do i then have to do a filesystem repair? Or just wait if the error accours again?
Stefan
Am 26.10.2010 13:01, schrieb Emmanuel Florac:
Quoted 9 lines
Le Tue, 26 Oct 2010 09:53:34 +0200
Stefan Priebe - Profihost AG<[email protected]> écrivait:Quoted 1 line
No it is a Raid 5 (3Ware Controller).
What model? Which firmware? If it's a 96x0 model, notice that 4.x.x firmwares added lots of performance enhancements and solved many problems.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Tue, 26 Oct 2010 13:03:07 +0200
Stefan Priebe - Profihost AG <[email protected]> écrivait:
Quoted 1 line
it is a 9650SE-8LPML with Firmware: FE9X 3.06.00.003.
Augh, this firmware is antique :) The latest is 4.10.xx.xx. Anyway, don't use firmware before 3.08.xx.xx, there were some nasty bugs.
Quoted 1 line
So you mean i should upgrade to 4.x Firmware?
Definitely. It will much improve performances, too. Simply download the firmware file, extract it, and flash the controller with tw_cli :
tw_cli /cXX update fw=prom0006.img
(the firmware is always in the prom00xx.img file).
Quoted 2 lines
Do i then have to do a
filesystem repair? Or just wait if the error accours again?
No, the filesystem will be fine. However you should start a RAID scrub with
tw_cli /cXX/uXX start verify
This will rebuild the parity with the new 4.X format (faster writes) and help detect any hardware fault.
My other advices : generally don't use cfq scheduler with a RAID controller, it will defeat the whole purpose of RAID cache and command reordering abilities. Use noop, generally, and deepen the queue :
echo "noop" > /sys/block/sdc/queue/scheduler
echo 512 > /sys/block/sdc/queue/nr_requests
Don't hesitate to enlarge tremendously the read-ahead cache, too. I generally use about 512 to 1024 sectors per drive as a rule of the thumb, so a 4 drives array will use 2048 to 4096 :
blockdev --setra 4096 /dev/sdc
You should see a 100% write/read speed improvement with these parameters.
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsHi,
thanks for your suggestions.
Quoted 6 lines
My other advices : generally don't use cfq scheduler with a RAID controller, it will defeat the whole purpose of RAID cache and command reordering abilities. Use noop, generally, and deepen the queue :
echo "noop">
/sys/block/sdc/queue/scheduler
echo 512>/sys/block/sdc/queue/nr_requests
Is there a way to make this static to this disk?
Quoted 5 lines
Don't hesitate to enlarge tremendously the read-ahead cache, too. I generally use about 512 to 1024 sectors per drive as a rule of the thumb, so a 4 drives array will use 2048 to 4096 :
blockdev --setra 4096
/dev/sdc
Is there also a way to make this static?
Quoted 2 lines
You should see a 100% write/read speed improvement with these parameters.
That would be great.
Thanks Stefan
Am 26.10.2010 13:25, schrieb Emmanuel Florac:
Quoted 35 lines
Le Tue, 26 Oct 2010 13:03:07 +0200
Stefan Priebe - Profihost AG<[email protected]> écrivait:Quoted 1 line
it is a 9650SE-8LPML with Firmware: FE9X 3.06.00.003.
Augh, this firmware is antique :) The latest is 4.10.xx.xx. Anyway, don't use firmware before 3.08.xx.xx, there were some nasty bugs.
Quoted 1 line
So you mean i should upgrade to 4.x Firmware?
Definitely. It will much improve performances, too. Simply download the firmware file, extract it, and flash the controller with
tw_cli:
tw_cli/cXX update fw=prom0006.img(the firmware is always in the prom00xx.img file).
Quoted 2 lines
Do i then have to do a
filesystem repair? Or just wait if the error accours again?No, the filesystem will be fine. However you should start a RAID scrub with
tw_cli/cXX/uXX start verifyThis will rebuild the parity with the new 4.X format (faster writes) and help detect any hardware fault.
My other advices : generally don't use cfq scheduler with a RAID controller, it will defeat the whole purpose of RAID cache and command reordering abilities. Use noop, generally, and deepen the queue :
echo "noop">
/sys/block/sdc/queue/scheduler
echo 512>/sys/block/sdc/queue/nr_requests
Quoted 9 lines
Don't hesitate to enlarge tremendously the read-ahead cache, too. I generally use about 512 to 1024 sectors per drive as a rule of the thumb, so a 4 drives array will use 2048 to 4096 :
blockdev --setra 4096
/dev/sdcYou should see a 100% write/read speed improvement with these parameters.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Tue, 26 Oct 2010 14:09:12 +0200
Stefan Priebe - Profihost AG <[email protected]> écrivait:
Quoted 7 lines
Quoted 2 lines
echo "noop">
/sys/block/sdc/queue/scheduler
echo 512>/sys/block/sdc/queue/nr_requestsIs there a way to make this static to this disk?
[snip]Quoted 1 line
blockdev --setra 4096
/dev/sdcIs there also a way to make this static?
The easiest way is to add these commands to /etc/rc.local.
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsQuoted 3 lines · On Dienstag, 26. Oktober 2010 Emmanuel Florac wrote:
echo "noop" >
/sys/block/sdc/queue/scheduler
echo 512 >/sys/block/sdc/queue/nr_requests
blockdev --setra 4096/dev/sdc
How about this small script in /etc/init.d/boot.local?
for i in /dev/xvd? /dev/sd? ; do
if test "$i" == "${i%\?}" ; then
echo i=$i
blockdev --setra 1024 $i
j=${i#/dev/}
echo j=$j
echo noop >/sys/block/$j/queue/scheduler
echo 512 >/sys/block/$j/queue/nr_requests fi
done
This works for VMs within XenServer (/dev/xvda) and real servers (/dev/sda), and sets some values for all drives.
BTW: any recommendations for these tuning knobs on VM-ized servers? What should be set in the VMs and what on the host? Has someone got tested numbers? I'm currently testing things, but it's so damn hard to get any useful, reliable results.
Signature · 13 lines
mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services http://proteger.at [gesprochen: Prot-e-schee] Tel: 0660 / 415 65 31 ****** Radiointerview zum Thema Spam ****** http://www.it-podcast.at/archiv.html#podcast-100716 // Wir haben im Moment zwei Häuser zu verkaufen: // http://zmi.at/langegg/ // http://zmi.at/haus2009/
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 15 lines · Michael Monnerie
Quoted 3 lines · On Dienstag, 26. Oktober 2010 Emmanuel Florac wrote:
echo "noop" >
/sys/block/sdc/queue/scheduler
echo 512 >/sys/block/sdc/queue/nr_requests
blockdev --setra 4096/dev/sdcHow about this small script in
/etc/init.d/boot.local?for i in
/dev/xvd?/dev/sd? ; do
if test "$i" == "${i%\?}" ; thenecho i=$i blockdev --setra 1024 $i j=${i#/dev/} echo j=$j echo noop >/sys/block/$j/queue/scheduler echo 512 >/sys/block/$j/queue/nr_requests
Quoted 5 lines
fi
doneThis works for VMs within XenServer (
/dev/xvda) and real servers (/dev/sda), and sets some values for all drives.
I'd suggest that people learn how to tweak udev hotplug rules so that when the device is first created (i.e. during hotplug) the scheduler, queue depth and readahead are set automatically. That way you don't have to rely on devices being discovered before your script runs...
Another benefit of doing it this way is that it is easy to set default rules for different types of devices based on regex matching e.g. different configs for "sd*" vs "dm*" vs "vd*" are trivial to set up.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · On Mittwoch, 27. Oktober 2010 Dave Chinner wrote:
I'd suggest that people learn how to tweak udev hotplug rules so that when the device is first created (i.e. during hotplug) the scheduler, queue depth and readahead are set automatically. That way you don't have to rely on devices being discovered before your script runs...
Another benefit of doing it this way is that it is easy to set default rules for different types of devices based on regex matching e.g. different configs for "sd*" vs "dm*" vs "vd*" are trivial to set up.
Sounds very nice. But the script I use will still work when upgrading the server from openSUSE 11.2 to 11.3, and is therefore the preferred choice for me.
Also, I'd need to find information and learn how to tweak udev hotplug rules. I want to implement this on about 30 VMs on 2 different hosts, with 3 different release states of servers (openSUSE 11.1, 11.2 and 11.3). The chance is high that I'd need two or three different udev tweaks for the different releases, so I don't see the benefit of udev for me. I wrote the script in about the same time I wrote this mail.
That's always the problem between developers ("ah, cool new stuff") and admins ("i need this on 500 servers with least possible work for me, and it must still work after any updates/upgrades").
Signature · 13 lines
mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services http://proteger.at [gesprochen: Prot-e-schee] Tel: 0660 / 415 65 31 ****** Radiointerview zum Thema Spam ****** http://www.it-podcast.at/archiv.html#podcast-100716 // Wir haben im Moment zwei Häuser zu verkaufen: // http://zmi.at/langegg/ // http://zmi.at/haus2009/
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 12 lines · Michael Monnerie
Quoted 10 lines · On Mittwoch, 27. Oktober 2010 Dave Chinner wrote:
I'd suggest that people learn how to tweak udev hotplug rules so that when the device is first created (i.e. during hotplug) the scheduler, queue depth and readahead are set automatically. That way you don't have to rely on devices being discovered before your script runs...
Another benefit of doing it this way is that it is easy to set default rules for different types of devices based on regex matching e.g. different configs for "sd*" vs "dm*" vs "vd*" are trivial to set up.
Quoted 6 lines
Sounds very nice. But the script I use will still work when upgrading the server from openSUSE 11.2 to 11.3, and is therefore the preferred choice for me.
Also, I'd need to find information and learn how to tweak udev hotplug rules.
GIYF.
Quoted 3 lines
I want to implement this on about 30 VMs on 2 different hosts, with 3 different release states of servers (openSUSE 11.1, 11.2 and 11.3).
The udev rule format hasn't changed in a long while. The same rule set should work on all of these.
Quoted 3 lines
The chance is high that I'd need two or three different udev tweaks for the different releases, so I don't see the benefit of udev for me. I wrote the script in about the same time I wrote this mail.
You'd need one regex per device type you want to tweak with different values.
Quoted 3 lines
That's always the problem between developers ("ah, cool new stuff") and admins ("i need this on 500 servers with least possible work for me, and it must still work after any updates/upgrades").
This is not "cool new stuff" - I've seen it used for exactly this purpose for _several years_ by distros. e.g. pulling the identifier string from the device to set hardware device specific tunables, changing default dm/md readahead, etc. It's not new, and is easy to configure generically so it works on a wide array of different machines.
And if you hotplug devices, it just works automatically - you don't need to rerun a script after every hotplug...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · On Mittwoch, 27. Oktober 2010 Dave Chinner wrote:
Quoted 2 lines
Also, I'd need to find information and learn how to tweak udev hotplug rules.
GIYF.
Thanks, I know how to search ;-)
*Finding* the information doesn't mean *understanding* how to do it. It takes time.
Quoted 8 lines
You'd need one regex per device type you want to tweak with different values.
This is not "cool new stuff" - I've seen it used for exactly this purpose for several years by distros. e.g. pulling the identifier string from the device to set hardware device specific tunables, changing default dm/md readahead, etc. It's not new, and is easy to configure generically so it works on a wide array of different machines.
I meant: When upgrading the system from 11.1 to 11.2, I'm sure the system replaces the correspondig script, or at least messes around with them. I know /etc/init.d/boot.local stays the same.
Quoted 2 lines
And if you hotplug devices, it just works automatically - you don't need to rerun a script after every hotplug...
Now that indeed is a pretty good advantage.
Signature · 13 lines
mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services http://proteger.at [gesprochen: Prot-e-schee] Tel: 0660 / 415 65 31 ****** Radiointerview zum Thema Spam ****** http://www.it-podcast.at/archiv.html#podcast-100716 // Wir haben im Moment zwei Häuser zu verkaufen: // http://zmi.at/langegg/ // http://zmi.at/haus2009/
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Wed, 27 Oct 2010 01:23:35 +0200 vous écriviez:
Quoted 4 lines
BTW: any recommendations for these tuning knobs on VM-ized servers? What should be set in the VMs and what on the host?
From my experience, using iSCSI as the transport to export lvs to the
VMs, the best performance is reach by setting scheduler to noop and queue depth on the target and the VM; but setting the readahead only on the VM side.
For different configurations, as usual, YMMV.
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsSo what values do you recommand in general for iSCSI on Serverside and which values on iSCSI client side?
Stefan
Am 27.10.2010 08:04, schrieb Emmanuel Florac:
Quoted 5 lines
Le Wed, 27 Oct 2010 01:23:35 +0200 vous écriviez:
Quoted 2 lines
BTW: any recommendations for these tuning knobs on VM-ized servers? What should be set in the VMs and what on the host?
Quoted 7 lines
From my experience, using iSCSI as the transport to export lvs to the
VMs, the best performance is reach by setting scheduler to noop and queue depth on the target and the VM; but setting the readahead only on the VM side.For different configurations, as usual, YMMV.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Wed, 27 Oct 2010 09:00:06 +0200
Stefan Priebe - Profihost AG <[email protected]> écrivait:
Quoted 3 lines
So what values do you recommand in general for iSCSI on Serverside and which values on iSCSI client side?
Depends upon the hardware. nr_requests should be 512 for 3Ware cards, 256 to 1024 for other controllers (always more than the default 128 anyway).
Same rule goes for read-ahead; however many RAID controllers (Areca, Adaptec) "cheat" and do read-ahead optimisation by themselves (which may harm random access). The default read-ahead value (256) was OK for obsolete ATA drives with a few KB of cache; nowadays all drives have 32 or 64 MB, and controllers 512 MB to several GB, so more read-ahead can do no harm but fill the cache up. I often go up to 65536 or 131072 sectors (for 24 or 48 drives arrays). Keep in mind that this is sequentail-raid optimisation; random access (such as VM work) may give better result with lower read-ahead values. Benchmarking your application is king here.
Just for information : I currently run some servers with kernel 2.6.22.18 (march 2008) and 3Ware firmware 4.10.0.7, so upgrading your 2.6.23 system should be perfectly safe from a driver compatibility point of view.
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsI'm having 2.6.32 Kernel not 2.6.23 :-) Upgrade works fine yesterday.
OK your values are for the iscsi_target right? Should i use the SAME values on the iscsi connector side?
Thanks Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Wed, 27 Oct 2010 12:08:57 +0200
Stefan Priebe - Profihost AG <[email protected]> écrivait:
Quoted 4 lines
I'm having 2.6.32 Kernel not 2.6.23 :-) Upgrade works fine yesterday.
OK your values are for the
iscsi_targetright? Should i use the SAME values on the iscsi connector side?
Yes, the initiator values are more important here. Basically, the iscsi target service should transmit the initiator's requests as they come.
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsQuoted 3 lines
Yes, the initiator values are more important here. Basically, the iscsitarget service should transmit the initiator's requests as they come.
What do you mean by this sentence? Sorry no native english speaker.
Am 27.10.2010 12:12, schrieb Emmanuel Florac:
Quoted 11 lines
Le Wed, 27 Oct 2010 12:08:57 +0200
Stefan Priebe - Profihost AG<[email protected]> écrivait:Quoted 4 lines
I'm having 2.6.32 Kernel not 2.6.23 :-) Upgrade works fine yesterday.
OK your values are for the
iscsi_targetright? Should i use the SAME values on the iscsi connector side?Yes, the initiator values are more important here. Basically, the iscsi target service should transmit the initiator's requests as they come.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Wed, 27 Oct 2010 12:22:47 +0200
Stefan Priebe - Profihost AG <[email protected]> écrivait:
Quoted 4 lines
Quoted 3 lines
Yes, the initiator values are more important here. Basically, the iscsitarget service should transmit the initiator's requests as they come.
What do you mean by this sentence? Sorry no native english speaker.
The settings that you apply on the initiator side are sent to the target ; the target applies them to the underlying device.
This is valid only when exporting a block device (lv, partition), it wouldn't work the same for fileio.
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsYou should keep the driver and firmware ins sync wither kernel. Choose e.g. latest 9.4.3 iso for 2.6.9, and 9.5.3 for RHEL5/SLES11/Fedora13. Do not use 9.5.3 firmware with RHEL4/SLES10 native driver. If I remeber correctly tw_update doesn't even let you mix a downgrade driver with new firmware, and vs.versus.
Iow, If you are on a legacy level, stay with latest legacy 9.4.3 for it.
http://kb.lsi.com/KnowledgebaseArticle14546.aspx
I've use a lot of 9550SXU and 9650SE and various MegaRaid. I find 3ware much easier to work with and better performing with xfs.
//seisqc1> /c2 show all
/c2 Driver Version = 2.26.08.003-2.6.18RH
/c2 Model = 9550SXU-8LP
/c2 Available Memory = 112MB
/c2 Firmware Version = FE9X 3.08.00.029
Rgds,
--ToreL
________________________________________
From: [email protected] [[email protected]] On Behalf Of Stefan Priebe - Profihost AG [[email protected]]
Sent: Tuesday, October 26, 2010 1:03 PM
To: Emmanuel Florac
Cc: [email protected]
Subject: Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1
Hi,
it is a 9650SE-8LPML with Firmware: FE9X 3.06.00.003.
So you mean i should upgrade to 4.x Firmware? Do i then have to do a filesystem repair? Or just wait if the error accours again?
Stefan
Am 26.10.2010 13:01, schrieb Emmanuel Florac:
Quoted 9 lines
Le Tue, 26 Oct 2010 09:53:34 +0200
Stefan Priebe - Profihost AG<[email protected]> écrivait:Quoted 1 line
No it is a Raid 5 (3Ware Controller).
What model? Which firmware? If it's a 96x0 model, notice that 4.x.x firmwares added lots of performance enhancements and solved many problems.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Tue, 26 Oct 2010 12:30:22 +0100
Larsen, Tore Høivaag <[email protected]> écrivait:
Quoted 5 lines
You should keep the driver and firmware ins sync wither kernel. Choose e.g. latest 9.4.3 iso for 2.6.9, and 9.5.3 for RHEL5/SLES11/Fedora13. Do not use 9.5.3 firmware with RHEL4/SLES10 native driver. If I remeber correctly
tw_updatedoesn't even let you mix a downgrade driver with new firmware, and vs.versus.
Yes, it will complain in case your driver/api isn't compatible with the newer firmware. Of course, only actually upgrade if it tells you :
"proceed to update".
However I'm pretty sure 2.6.23 is safe for 4.x firmware (2.6.24 definitely is).
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsHi,
i've done all you advises - updating 3ware, setting other scheduler, block size, ...
but today i got again hundrets of these messages:
[318343.901809] Filesystem "sdc3": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff8117bdea [318343.901811] [318343.901813] Pid: 20299, comm: rsync Not tainted 2.6.32.22intel #1 [318343.901815] Call Trace: [318343.901819] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [318343.901823] [<ffffffff8117bcb7>] ? xfs_da_do_buf+0x557/0x620 [318343.901827] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [318343.901831] [<ffffffff810538a7>] ? bit_waitqueue+0x10/0xa0 [318343.901834] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [318343.901839] [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197 [318343.901843] [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197 [318343.901846] [<ffffffff813751d0>] ? tcp_write_xmit+0x466/0x96c [318343.901850] [<ffffffff8117f9c2>] ? xfs_dir2_block_lookup+0x18/0xb0 [318343.901854] [<ffffffff8117e71c>] ? xfs_dir_lookup+0xd5/0x147 [318343.901857] [<ffffffff811a11bd>] ? xfs_lookup+0x47/0xa3 [318343.901861] [<ffffffff811a90d1>] ? xfs_vn_lookup+0x3c/0x7b [318343.901865] [<ffffffff810c1de7>] ? __lookup_hash+0xfa/0x11e [318343.901869] [<ffffffff810c51a8>] ? do_filp_open+0x208/0x934 [318343.901873] [<ffffffff810c7242>] ? poll_select_copy_remaining+0xd0/0xf3 [318343.901877] [<ffffffff810cd62a>] ? alloc_fd+0x67/0x10b [318343.901880] [<ffffffff810b8a60>] ? do_sys_open+0x55/0x103 [318343.901884] [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b
Stefan
Am 26.10.2010 09:24, schrieb Stefan Priebe - Profihost AG:
Quoted 16 lines
Am 26.10.2010 09:22, schrieb Emmanuel Florac:
Quoted 1 line
xfs_info/dev/sdc3Hi here are the details:
root@server361-han:~# xfs_info /dev/sdc3meta-data=/dev/sdc3 isize=256 agcount=32, agsize=40038461 blks = sectsz=512 attr=0
data = bsize=4096 blocks=1281230752, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Quoted 8 lines
root@server361-han:~# mount|grep sdc3
/dev/sdc3on /sdc3 type xfs (rw,noatime,nodiratime,nobarrier,prjquota) /sdc3/serverbackup on /serverbackup type none (rw,bind) /sdc3/serverbackup_rootserver on /serverbackup_rootserver type none (rw,bind)Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Mon, 01 Nov 2010 09:01:12 +0100 vous écriviez:
Quoted 3 lines
but today i got again hundrets of these messages:
[318343.901809] Filesystem "sdc3": XFS internal error
xfs_da_do_buf(2)
That's weird. What do "uname -a" and "free" tells? I'm wondering if there isn't some low memory condition at work here.
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsQuoted 6 lines
Quoted 3 lines
but today i got again hundrets of these messages:
[318343.901809] Filesystem "sdc3": XFS internal error
xfs_da_do_buf(2)That's weird. What do "uname -a" and "free" tells? I'm wondering if there isn't some low memory condition at work here.
# uname -a Linux server 2.6.32.22intel #1 SMP Tue Sep 21 11:05:46 CEST 2010 x86_64 GNU/Linux
# free
total used free shared buffers cached
Mem: 16470416 15098888 1371528 0 48 6019036
-/+ buffers/cache: 9079804 7390612
Swap: 1469908 0 1469908Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 22 lines · Stefan Priebe - Profihost AG
Hi,
i've done all you advises - updating 3ware, setting other scheduler, block size, ...
but today i got again hundrets of these messages:
[318343.901809] Filesystem "sdc3": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff8117bdea [318343.901811] [318343.901813] Pid: 20299, comm: rsync Not tainted 2.6.32.22intel #1 [318343.901815] Call Trace: [318343.901819] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [318343.901823] [<ffffffff8117bcb7>] ? xfs_da_do_buf+0x557/0x620 [318343.901827] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [318343.901831] [<ffffffff810538a7>] ? bit_waitqueue+0x10/0xa0 [318343.901834] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [318343.901839] [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197 [318343.901843] [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197 [318343.901846] [<ffffffff813751d0>] ? tcp_write_xmit+0x466/0x96c [318343.901850] [<ffffffff8117f9c2>] ? xfs_dir2_block_lookup+0x18/0xb0 [318343.901854] [<ffffffff8117e71c>] ? xfs_dir_lookup+0xd5/0x147
Quoted 8 lines
[318343.901857] [<ffffffff811a11bd>] ? xfs_lookup+0x47/0xa3 [318343.901861] [<ffffffff811a90d1>] ? xfs_vn_lookup+0x3c/0x7b [318343.901865] [<ffffffff810c1de7>] ? __lookup_hash+0xfa/0x11e [318343.901869] [<ffffffff810c51a8>] ? do_filp_open+0x208/0x934 [318343.901873] [<ffffffff810c7242>] ? poll_select_copy_remaining+0xd0/0xf3 [318343.901877] [<ffffffff810cd62a>] ? alloc_fd+0x67/0x10b [318343.901880] [<ffffffff810b8a60>] ? do_sys_open+0x55/0x103 [318343.901884] [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b
Have you run repair to fix all the existing problems after doing all the firmware upgrades, etc?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
no, as Emmanuel told me i don't have todo that.
Stefan
Am 01.11.2010 13:27, schrieb Dave Chinner:
Quoted 32 lines · Stefan Priebe - Profihost AG
Quoted 22 lines
Hi,
i've done all you advises - updating 3ware, setting other scheduler, block size, ...
but today i got again hundrets of these messages:
[318343.901809] Filesystem "sdc3": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff8117bdea [318343.901811] [318343.901813] Pid: 20299, comm: rsync Not tainted 2.6.32.22intel #1 [318343.901815] Call Trace: [318343.901819] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [318343.901823] [<ffffffff8117bcb7>] ? xfs_da_do_buf+0x557/0x620 [318343.901827] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [318343.901831] [<ffffffff810538a7>] ? bit_waitqueue+0x10/0xa0 [318343.901834] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [318343.901839] [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197 [318343.901843] [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197 [318343.901846] [<ffffffff813751d0>] ? tcp_write_xmit+0x466/0x96c [318343.901850] [<ffffffff8117f9c2>] ? xfs_dir2_block_lookup+0x18/0xb0 [318343.901854] [<ffffffff8117e71c>] ? xfs_dir_lookup+0xd5/0x147Quoted 8 lines
[318343.901857] [<ffffffff811a11bd>] ? xfs_lookup+0x47/0xa3 [318343.901861] [<ffffffff811a90d1>] ? xfs_vn_lookup+0x3c/0x7b [318343.901865] [<ffffffff810c1de7>] ? __lookup_hash+0xfa/0x11e [318343.901869] [<ffffffff810c51a8>] ? do_filp_open+0x208/0x934 [318343.901873] [<ffffffff810c7242>] ? poll_select_copy_remaining+0xd0/0xf3 [318343.901877] [<ffffffff810cd62a>] ? alloc_fd+0x67/0x10b [318343.901880] [<ffffffff810b8a60>] ? do_sys_open+0x55/0x103 [318343.901884] [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b
Quoted 6 lines
Have you run repair to fix all the existing problems after doing all the firmware upgrades, etc?
Cheers,
Dave.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
[ please don't top-post - fixed. ]
Quoted 33 lines · Stefan Priebe - Profihost AG
Am 01.11.2010 13:27, schrieb Dave Chinner:
Quoted 32 lines · Stefan Priebe - Profihost AG
Quoted 30 lines
Hi,
i've done all you advises - updating 3ware, setting other scheduler, block size, ...
but today i got again hundrets of these messages:
[318343.901809] Filesystem "sdc3": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff8117bdea [318343.901811] [318343.901813] Pid: 20299, comm: rsync Not tainted 2.6.32.22intel #1 [318343.901815] Call Trace: [318343.901819] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [318343.901823] [<ffffffff8117bcb7>] ? xfs_da_do_buf+0x557/0x620 [318343.901827] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [318343.901831] [<ffffffff810538a7>] ? bit_waitqueue+0x10/0xa0 [318343.901834] [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29 [318343.901839] [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197 [318343.901843] [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197 [318343.901846] [<ffffffff813751d0>] ? tcp_write_xmit+0x466/0x96c [318343.901850] [<ffffffff8117f9c2>] ? xfs_dir2_block_lookup+0x18/0xb0 [318343.901854] [<ffffffff8117e71c>] ? xfs_dir_lookup+0xd5/0x147 [318343.901857] [<ffffffff811a11bd>] ? xfs_lookup+0x47/0xa3 [318343.901861] [<ffffffff811a90d1>] ? xfs_vn_lookup+0x3c/0x7b [318343.901865] [<ffffffff810c1de7>] ? __lookup_hash+0xfa/0x11e [318343.901869] [<ffffffff810c51a8>] ? do_filp_open+0x208/0x934 [318343.901873] [<ffffffff810c7242>] ? poll_select_copy_remaining+0xd0/0xf3 [318343.901877] [<ffffffff810cd62a>] ? alloc_fd+0x67/0x10b [318343.901880] [<ffffffff810b8a60>] ? do_sys_open+0x55/0x103 [318343.901884] [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b
Quoted 4 lines
Quoted 2 lines
Have you run repair to fix all the existing problems after doing all the firmware upgrades, etc?
no, as Emmanuel told me i don't have todo that.
Perhaps you've been told the wrong thing.
You are getting reports of on-disk corruption being encountered. You need to run repair to find and fix all these existing problems. before these problems will go away. Otherwise you'll keep getting these reports every time you read the corrupted directory....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Le Mon, 01 Nov 2010 13:28:46 +0100 vous écriviez:
Quoted 1 line
no, as Emmanuel told me i don't have todo that.
Did you run the "verify" round on the array after the upgrade? That would rebuild all checksum and insure that the RAID is fully coherent. If the verify comes out with any message, there is a sloght chance that the data on disk wasn't entirely coherent or clean.
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsQuoted 5 lines
Did you run the "verify" round on the array after the upgrade? That would rebuild all checksum and insure that the RAID is fully coherent. If the verify comes out with any message, there is a sloght chance that the data on disk wasn't entirely coherent or clean.
shure i have - was done without any problems.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs