suddenly slow writes on XFS Filesystem
Hi,
since a few days i've experienced a really slow fs on one of our backup systems.
I'm not sure whether this is XFS related or related to the Controller / Disks.
It is a raid 10 of 20 SATA Disks and i can only write to them with about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 and 3.3.4 - no difference. Writing to another partition on another xfs array works fine.
Details:
#~ df -h/dev/sdb1 4,6T 4,4T 207G 96% /mnt
#~ df -i/dev/sdb1 4875737052 4659318044 216419008 96% /mnt
#~ xfs_db -c frag -r /dev/sdb1
actual 83160469, ideal 82145389, fragmentation factor 1,22%
#~ xfs_info /dev/sdb1
meta-data=/dev/sdb1 isize=256 agcount=5, agsize=268435392 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=1218967031, imaxpct=5
= sunit=64 swidth=1280 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=64 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0#~ cat /proc/mounts/dev/sdb1 /mnt xfs rw,noatime,nodiratime,attr2,nobarrier,logbufs=8,logbsize=256k,sunit=512,swidth=10240,prjquota 0 0
Any ideas?
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
Quoted 1 line
Hi,
Hi Stefan,
Quoted 10 lines
since a few days i've experienced a really slow fs on one of our backup systems.
I'm not sure whether this is XFS related or related to the Controller / Disks.
It is a raid 10 of 20 SATA Disks and i can only write to them with about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 and 3.3.4 - no difference. Writing to another partition on another xfs array works fine.
How do you measure the 700 KB/s?
Anything in dmesg?
I and likely others here would be interested in vmstat and iostat outputs during slowness.
See:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
Ciao,
Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
Quoted 4 lines
It is a raid 10 of 20 SATA Disks and i can only write to them with about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 and 3.3.4 - no difference. Writing to another partition on another xfs array works fine.
Additionally what RAID is this? SoftRAID or some - which one? - hardware RAID controller? And what disks are used, whats the rpm of these?
Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Martin Steigerwald
Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
Quoted 4 lines
It is a raid 10 of 20 SATA Disks and i can only write to them with about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 and 3.3.4 - no difference. Writing to another partition on another xfs array works fine.
Additionally what RAID is this? SoftRAID or some - which one? - hardware RAID controller? And what disks are used, whats the rpm of these?
I doubt much of this stuff matters. Stefan's filesystem is 96% full, w/~200GB free. This free space is likely heavily fragmented. If he's doing allocation in this fragmented free space I'd think that would fully explain his write performance dropping off a cliff due to massive head seeking.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Am 06.05.2012 17:45, schrieb Stan Hoeppner:
Quoted 7 lines · Martin Steigerwald
Quoted 6 lines
Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
Quoted 4 lines
It is a raid 10 of 20 SATA Disks and i can only write to them with about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 and 3.3.4 - no difference. Writing to another partition on another xfs array works fine.
Quoted 9 lines
Quoted 2 lines
Additionally what RAID is this? SoftRAID or some - which one? - hardware RAID controller? And what disks are used, whats the rpm of these?
I doubt much of this stuff matters. Stefan's filesystem is 96% full, w/~200GB free. This free space is likely heavily fragmented. If he's doing allocation in this fragmented free space I'd think that would fully explain his write performance dropping off a cliff due to massive head seeking.
Thanks Stan that's it. After deleting 200GB-300GB it's running fine again.
What is the general recommandation of free space?
Greets Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Am Sonntag, 6. Mai 2012 schrieb Stan Hoeppner:
Quoted 11 lines · Martin Steigerwald
Quoted 9 lines
Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
Quoted 4 lines
It is a raid 10 of 20 SATA Disks and i can only write to them with about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 and 3.3.4 - no difference. Writing to another partition on another xfs array works fine.
Additionally what RAID is this? SoftRAID or some - which one? - hardware RAID controller? And what disks are used, whats the rpm of these?
Quoted 5 lines
I doubt much of this stuff matters. Stefan's filesystem is 96% full, w/~200GB free. This free space is likely heavily fragmented. If he's doing allocation in this fragmented free space I'd think that would fully explain his write performance dropping off a cliff due to massive head seeking.
Hmmm, I thought about that low free space percentage, too, but then 200 GB free still sounded like quite much to me. Nonetheless, since according to Stefan freeing some space solved the slowness issue, that seems to have been it.
Can free space be defragmented in XFS? xfs_fsr only tries to defragment files as far as I know.
Thanks,
Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Stefan Priebe
Hi,
since a few days i've experienced a really slow fs on one of our backup systems.
I'm not sure whether this is XFS related or related to the Controller / Disks.
It is a raid 10 of 20 SATA Disks and i can only write to them with about 700kb/s while doing random i/o.
What sort of random IO? size, read, write, direct or buffered, data or metadata, etc? iostat -x -d -m 5 and vmstat 5 traces would be useful to see if it is your array that is slow.....
Quoted 7 lines
I tried vanilla Kernel 3.0.30
and 3.3.4 - no difference. Writing to another partition on another xfs array works fine.Details:
#~ df -h/dev/sdb14,6T 4,4T 207G 96% /mnt
Your filesystem is near full - the allocation algorithms definitely slow down as you approach ENOSPC, and IO efficiency goes to hell because of a lack of contiguous free space to allocate from.
Quoted 2 lines
#~ df -i
/dev/sdb14875737052 4659318044 216419008 96% /mnt
You have 4.6 *billion* inodes in your filesystem?
Quoted 1 line
Any ideas?
None until I understand your workload....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 17 lines · Stefan Priebe
Am 06.05.2012 17:45, schrieb Stan Hoeppner:
Quoted 7 lines · Martin Steigerwald
Quoted 6 lines
Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
Quoted 4 lines
It is a raid 10 of 20 SATA Disks and i can only write to them with about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 and 3.3.4 - no difference. Writing to another partition on another xfs array works fine.
Quoted 9 lines
Quoted 2 lines
Additionally what RAID is this? SoftRAID or some - which one? - hardware RAID controller? And what disks are used, whats the rpm of these?
I doubt much of this stuff matters. Stefan's filesystem is 96% full, w/~200GB free. This free space is likely heavily fragmented. If he's doing allocation in this fragmented free space I'd think that would fully explain his write performance dropping off a cliff due to massive head seeking.
Quoted 3 lines
Thanks Stan that's it. After deleting 200GB-300GB it's running fine again.
What is the general recommandation of free space?
Depends on the size of the filesystem. If you've got a 500TB filesystem, then running at 98% full (10TB of free space) is not going to be a big deal. But running at 98% full on a 5TB volume is a big deal because there is relatively little freespace per AG and it will get rapidly fragmented.
So for a 5TB volume, I' say don't run sustained operations at over 90% full. Going above this temporarily won't be a problem, but staying at >95% full will definitely cause accelerated aging of the filesystem.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Hi,
after deleting 400GB it was faster. Now there are still 300GB free but it is slow as hell again ;-(
Am 07.05.2012 03:34, schrieb Dave Chinner:
Quoted 6 lines · Stefan Priebe
Quoted 5 lines
Hi,
since a few days i've experienced a really slow fs on one of our backup systems.
Quoted 8 lines
Quoted 5 lines
I'm not sure whether this is XFS related or related to the Controller / Disks.
It is a raid 10 of 20 SATA Disks and i can only write to them with about 700kb/s while doing random i/o.
What sort of random IO? size, read, write, direct or buffered, data or metadata, etc?
There are 4 rsync processes running and doing backups of other severs.
Quoted 2 lines
iostat -x -d -m 5 and vmstat 5 traces would be useful to see if it is your array that is slow.....
~ # iostat -x -d -m 5
Linux 2.6.40.28intel (server844-han) 05/07/2012 _x86_64_
(8 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 254,80 25,40 1,72 0,16 13,71 0,86 3,08 2,39 67,06 sda 0,00 0,20 0,00 1,20 0,00 0,00
6,50 0,00 0,00 0,00 0,00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 187,40 24,20 1,26 0,19 14,05 0,75 3,56 3,33 70,50 sda 0,00 0,00 0,00 0,40 0,00 0,00
4,50 0,00 0,00 0,00 0,00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 11,20 242,40 92,00 1,56 0,89
15,00 4,70 14,06 1,58 52,68
sda 0,00 0,20 0,00 2,60 0,00 0,02
12,00 0,00 0,00 0,00 0,00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 166,20 24,00 0,99 0,17 12,51 0,57 3,02 2,40 45,56 sda 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00
qDevice: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 188,00 25,40 1,22 0,16 13,23 0,44 2,04 1,78 38,02 sda 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00
# vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 7 0 0 788632 48 12189652 0 0 173 395 13 45 1 16 82 1 [root@server844-han /serverbackup (master)]# vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 4 0 0 778148 48 12189776 0 0 173 395 13 45 1 16 82 1 [root@server844-han /serverbackup (master)]# vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 2 0 0 774372 48 12189876 0 0 173 395 13 45 1 16 82 1 [root@server844-han /serverbackup (master)]# vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 5 0 0 771240 48 12189936 0 0 173 395 13 45 1 16 82 1 [root@server844-han /serverbackup (master)]# vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 6 0 0 768636 48 12190000 0 0 173 395 13 45 1 16 82 1
Quoted 5 lines
Quoted 5 lines
I tried vanilla Kernel 3.0.30
and 3.3.4 - no difference. Writing to another partition on another xfs array works fine.
Quoted 7 lines
Quoted 3 lines
Details:
#~ df -h/dev/sdb14,6T 4,4T 207G 96% /mntYour filesystem is near full - the allocation algorithms definitely slow down as you approach ENOSPC, and IO efficiency goes to hell because of a lack of contiguous free space to allocate from.
I've now 94% used but it is still slow. It seems it was just getting fast with more than 450GB free space.
/dev/sdb1 4,6T 4,3T 310G 94% /mnt
Quoted 3 lines
Quoted 2 lines
#~ df -i
/dev/sdb14875737052 4659318044 216419008 96% /mntYou have 4.6 *billion* inodes in your filesystem?
Yes - it backups around 100 servers with a lot of files.
Greet Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Am 06.05.2012 23:43, schrieb Martin Steigerwald:
Quoted 11 lines
Am Sonntag, 6. Mai 2012 schrieb Stan Hoeppner:
Quoted 10 lines · Martin Steigerwald
Quoted 9 lines
Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
Quoted 4 lines
It is a raid 10 of 20 SATA Disks and i can only write to them with about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 and 3.3.4 - no difference. Writing to another partition on another xfs array works fine.
Additionally what RAID is this? SoftRAID or some - which one? - hardware RAID controller? And what disks are used, whats the rpm of these?
It is a HW Adaptec 5805 Controller with 20x 7200rpm SATA Disks.
Quoted 2 lines
Can free space be defragmented in XFS?
xfs_fsronly tries to defragment files as far as I know.
That is a good question.
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 45 lines · Stefan Priebe - Profihost AG
Hi,
after deleting 400GB it was faster. Now there are still 300GB free but it is slow as hell again ;-(
Am 07.05.2012 03:34, schrieb Dave Chinner:
Quoted 6 lines · Stefan Priebe
Quoted 5 lines
Hi,
since a few days i've experienced a really slow fs on one of our backup systems.
Quoted 8 lines
Quoted 5 lines
I'm not sure whether this is XFS related or related to the Controller / Disks.
It is a raid 10 of 20 SATA Disks and i can only write to them with about 700kb/s while doing random i/o.
What sort of random IO? size, read, write, direct or buffered, data or metadata, etc?
There are 4 rsync processes running and doing backups of other severs.
Quoted 2 lines
iostat -x -d -m 5 and vmstat 5 traces would be useful to see if it is your array that is slow.....
~ # iostat -x -d -m 5
Linux 2.6.40.28intel (server844-han) 05/07/2012 _x86_64_
(8 CPU)Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdb 0,00 0,00 254,80 25,40 1,72 0,16 13,71 0,86 3,08 2,39 67,06 sda 0,00 0,20 0,00 1,20 0,00 0,00 6,50 0,00 0,00 0,00 0,00Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdb 0,00 0,00 187,40 24,20 1,26 0,19 14,05 0,75 3,56 3,33 70,50 sda 0,00 0,00 0,00 0,40 0,00 0,00 4,50 0,00 0,00 0,00 0,00Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdb 0,00 11,20 242,40 92,00 1,56 0,89 15,00 4,70 14,06 1,58 52,68 sda 0,00 0,20 0,00 2,60 0,00 0,02 12,00 0,00 0,00 0,00 0,00Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdb 0,00 0,00 166,20 24,00 0,99 0,17 12,51 0,57 3,02 2,40 45,56 sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
Quoted 6 lines
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 188,00 25,40 1,22 0,16 13,23 0,44 2,04 1,78 38,02 sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00# vmstat
"vmstat 5", not vmstat 5 times.... :/
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
I tried vanilla Kernel 3.0.30
and 3.3.4 - no difference. Writing to another partition on another xfs array works fine.Details:
#~ df -h/dev/sdb14,6T 4,4T 207G 96% /mnt
Quoted 7 lines
Quoted 3 lines
Your filesystem is near full - the allocation algorithms definitely slow down as you approach ENOSPC, and IO efficiency goes to hell because of a lack of contiguous free space to allocate from.
I've now 94% used but it is still slow. It seems it was just getting fast with more than 450GB free space.
/dev/sdb14,6T 4,3T 310G 94% /mnt
Well, you've probably badly fragmented the free space you have. what does the 'xfs_db -r -c freesp <dev>' command tell you?
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
#~ df -i
/dev/sdb14875737052 4659318044 216419008 96% /mntYou have 4.6 *billion* inodes in your filesystem?
Yes - it backups around 100 servers with a lot of files.
So you have what - lots of symlinks? I mean, 4.6 billion inodes alone requires 1.2TB of space, but if I read the fragmentation you only have 82 million files with data extents. The only thing that would other wise use inodes are directories and symlinks....
Still, I can't see how you'd only have 82 million data inodes and 4.5 billion directory inodes - where are all the inodes being consumed? A massive symlink farm?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines
Quoted 2 lines
# vmstat"vmstat 5", not vmstat 5 times.... :/
oh sorry. Sadly the rsync processes do not run right know i've to kill them. Is the output still usable?
# vmstat 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 0 5582136 48 5849956 0 0 176 394 34 54 1 16 82 1 0 1 0 5552180 48 5854280 0 0 2493 2496 3079 2172 1 4 86 9 3 2 0 5601308 48 5857672 0 0 1098 28043 5150 1913 0 10 73 17 0 2 0 5595360 48 5863180 0 0 1098 14336 3945 1897 0 8 69 22 3 2 0 5594088 48 5865280 0 0 432 15897 4209 2366 0 8 71 21 0 2 0 5591068 48 5868940 0 0 854 10989 3519 2107 0 7 70 23 1 1 0 5592004 48 5869872 0 0 180 7886 3605 2436 0 3 76 22
Quoted 3 lines
Quoted 1 line
/dev/sdb14,6T 4,3T 310G 94% /mntWell, you've probably badly fragmented the free space you have. what does the '
xfs_db-r -c freesp <dev>' command tell you?
from to extents blocks pct
1 1 942737 942737 0,87
2 3 671860 1590480 1,47
4 7 461268 2416025 2,23
8 15 1350517 18043063 16,67
16 31 111254 2547581 2,35
32 63 192032 9039799 8,35
64 127 33026 3317194 3,07
128 255 14254 2665812 2,46
256 511 12516 4631200 4,28
512 1023 6942 5031081 4,65
1024 2047 4622 6893270 6,37
2048 4095 3268 9412271 8,70
4096 8191 2135 12716435 11,75
8192 16383 338 3974884 3,67
16384 32767 311 7018658 6,49
32768 65535 105 4511372 4,17
65536 131071 29 2577756 2,38
131072 262143 8 1339796 1,24
262144 524287 10 3950416 3,65
524288 1048575 4 2580085 2,38
1048576 2097151 2 3028028 2,80Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
#~ df -i
/dev/sdb14875737052 4659318044 216419008 96% /mntYou have 4.6 *billion* inodes in your filesystem?
Yes - it backups around 100 servers with a lot of files.
i rechecked this and it seems i sadly copied the wrong output ;-( sorry for that.
Here is the correct one:
#~ df -i/dev/sdb1 975173568 95212355 879961213 10% /mnt
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
Quoted 4 lines
Hi,
after deleting 400GB it was faster. Now there are still 300GB free but it is slow as hell again ;-(
Well, before you had 200 GB free, now you have 300 GB free. Thats not that much more. You have 5 AGs, so if distributed evenly it would be 40 to 60 GB free space per AG.
The 10% Dave recommended of 4,6 TB would be 460 GB.
Are other volumes of similar size are similarily full but faster?
Would be interesting to see whether you are able to sustain faster speeds by keeping the volume above 500 GB free or so.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
It is a raid 10 of 20 SATA Disks and i can only write to them with about 700kb/s while doing random i/o.
What sort of random IO? size, read, write, direct or buffered, data or metadata, etc?
There are 4 rsync processes running and doing backups of other severs.
Which rsync version is involved? I strongly suggest to use rsync version 3 or above.
Quoted 8 lines
Quoted 3 lines
Your filesystem is near full - the allocation algorithms definitely slow down as you approach ENOSPC, and IO efficiency goes to hell because of a lack of contiguous free space to allocate from.
I've now 94% used but it is still slow. It seems it was just getting fast with more than 450GB free space.
/dev/sdb14,6T 4,3T 310G 94% /mnt
Then: Will it stay fast if you keep it above 450 GB free space?
And is it as fast then as it used to be before you faced performance issues?
Ciao,
Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
Quoted 33 lines
Quoted 2 lines
iostat -x -d -m 5 and vmstat 5 traces would be useful to see if it is your array that is slow.....
~ # iostat -x -d -m 5
Linux 2.6.40.28intel (server844-han) 05/07/2012 _x86_64_
(8 CPU)Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %utilsdb 0,00 0,00 254,80 25,40 1,72 0,16 13,71 0,86 3,08 2,39 67,06 sda 0,00 0,20 0,00 1,20 0,00 0,006,50 0,00 0,00 0,00 0,00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %utilsdb 0,00 0,00 187,40 24,20 1,26 0,19 14,05 0,75 3,56 3,33 70,50 sda 0,00 0,00 0,00 0,40 0,00 0,004,50 0,00 0,00 0,00 0,00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 11,20 242,40 92,00 1,56 0,89
15,00 4,70 14,06 1,58 52,68
sda 0,00 0,20 0,00 2,60 0,00 0,02
12,00 0,00 0,00 0,00 0,00Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %utilsdb 0,00 0,00 166,20 24,00 0,99 0,17 12,51 0,57 3,02 2,40 45,56 sda 0,00 0,00 0,00 0,00 0,00 0,00
Quoted 8 lines
0,00 0,00 0,00 0,00 0,00
qDevice: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %utilsdb 0,00 0,00 188,00 25,40 1,22 0,16 13,23 0,44 2,04 1,78 38,02 sda 0,00 0,00 0,00 0,00 0,00 0,000,00 0,00 0,00 0,00 0,00
Disk utilization seems to be quite high but it seems there not near to 90 to 100%. So there might be other overheads involved - like network or (unlikely) CPU.
Did you verify that at the time you perceive slowness the servers you backup can deliver data fast enough?
I would like to now, whether there are really processes waiting for I/O during rsync workload.
Can you try vmstat 5 and
while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
while the backup workload is running and slow?
Like this:
merkaba:~> while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
root 1508 0.0 0.0 0 0 ? D Mai06 0:00 [flush-253:2] root 1508 0.0 0.0 0 0 ? D Mai06 0:00 [flush-253:2] martin 28374 100 0.0 9800 652 pts/7 D+ 10:27 0:02 dd if=/dev/zero of=fhgs root 1508 0.0 0.0 0 0 ? D Mai06 0:00 [flush-253:2]
(this is with an Ext4, so its using the flush daemon, with XFS you probably see xfssyncd or xfsbufd instead if I am not mistaken and if rsync processes are waiting for I/O they should appear there too)
And yes, its important to have vmstat 5 output during workload is happening to see the amount of CPU time that the kernel cannot use for processing cause all processes that are runnable wait for I/O.
Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Am 07.05.2012 10:31, schrieb Martin Steigerwald:
Quoted 3 lines
Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
Did you verify that at the time you perceive slowness the servers you backup can deliver data fast enough?
Yes. It works fine to another partition.
Quoted 5 lines
I would like to now, whether there are really processes waiting for I/O during rsync workload.
Can you try vmstat 5 and
while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
Here it is:
# vmstat 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 0 396688 48 7728136 0 0 229 374 22 32 1 12 85 2 2 0 0 421744 48 7717348 0 0 3722 1169 9015 8389 1 2 95 2 0 1 0 405780 48 7751904 0 0 8937 207 10780 9290 2 2 88 7 0 0 0 486928 48 7733300 0 0 2526 1277 8275 7791 1 2 96 1 0 0 0 416692 48 7778164 0 0 5046 43750 8548 8141 1 2 95 2 0 0 0 444968 48 7777792 0 0 8021 1709 9315 8573 2 2 94 2 2 0 0 357924 48 7946532 0 0 48181 1031 17646 12684 10 4 87 0 1 0 0 348696 48 8137200 0 0 74366 1056 24362 16775 15 5 81 0 1 0 0 391552 48 8279000 0 0 54693 1242 19224 13957 11 4 85 0
# while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done root 12493 2.0 0.2 101780 48392 ? D 15:35 0:24 rsync --daemon root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync --daemon root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync --daemon root 12494 3.9 0.2 128220 44328 ? D 15:35 0:47 rsync --daemon root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync --daemon root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync --daemon root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync --daemon root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync --daemon root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync --daemon root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 21720 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync --daemon root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 21720 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 21980 ? D 15:50 0:11 rsync --daemon root 13582 1.7 0.0 74308 13280 ? D 15:50 0:05 rsync --daemon root 12493 2.0 0.2 87956 48212 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 21980 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86420 48156 ? D 15:35 0:25 rsync --daemon root 12493 2.0 0.2 86420 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync --daemon root 13581 3.5 0.1 76832 22508 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22508 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22508 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22772 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22772 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22772 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22772 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22772 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 3378 0.1 0.0 0 0 ? D 13:59 0:09 [flush-8:16] root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
Quoted 6 lines
Am 07.05.2012 10:31, schrieb Martin Steigerwald:
Quoted 3 lines
Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
Did you verify that at the time you perceive slowness the servers you backup can deliver data fast enough?Yes. It works fine to another partition.
With more free space I suppose?
Quoted 23 lines
Quoted 6 lines
I would like to now, whether there are really processes waiting for I/O during rsync workload.
Can you try vmstat 5 and
while true; do ps aux | grep " D" | grep -v grep ; sleep 1; doneHere it is:
# vmstat 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 0 396688 48 7728136 0 0 229 374 22 32 1 12 85 2 2 0 0 421744 48 7717348 0 0 3722 1169 9015 8389 1 2 95 2 0 1 0 405780 48 7751904 0 0 8937 207 10780 9290 2 2 88 7 0 0 0 486928 48 7733300 0 0 2526 1277 8275 7791 1 2 96 1 0 0 0 416692 48 7778164 0 0 5046 43750 8548 8141 1 2 95 2
Quoted 8 lines
0 0 0 444968 48 7777792 0 0 8021 1709 9315 8573 2 2 94 2 2 0 0 357924 48 7946532 0 0 48181 1031 17646 12684 10 4 87 0 1 0 0 348696 48 8137200 0 0 74366 1056 24362 16775 15 5 81 0 1 0 0 391552 48 8279000 0 0 54693 1242 19224 13957 11 4 85 0
Hmm, there are some I/O wait times, but they are lower than I expected.
In the last lines the throughput is higher than in the first lines. Does that correlate to higher speed in rsync? It may also be excessive writes due to free space fragmentation, but Dave or someone else might be able to say whether such a huge difference can be blamed to free space fragmentation alone.
Quoted 11 lines
# while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done root 12493 2.0 0.2 101780 48392 ? D 15:35 0:24 rsync --daemon root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync --daemon root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync --daemon root 12494 3.9 0.2 128220 44328 ? D 15:35 0:47 rsync --daemon root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync --daemon
Still these rsyncs appear to be in uninteruptible sleep quite consistently. They shouldn´t be in that state when waiting on network socket readyness.
So this workloads seems to be I/O bound to me.
I suggest trying to keep the volume above 500 GB free and see whether that works consistently.
Thanks,
Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 14 lines · Stefan Priebe - Profihost AG
from to extents blocks pct
1 1 942737 942737 0,87
2 3 671860 1590480 1,47
4 7 461268 2416025 2,23
8 15 1350517 18043063 16,67
16 31 111254 2547581 2,35
32 63 192032 9039799 8,35
64 127 33026 3317194 3,07
128 255 14254 2665812 2,46
256 511 12516 4631200 4,28
512 1023 6942 5031081 4,65
1024 2047 4622 6893270 6,37
2048 4095 3268 9412271 8,70
4096 8191 2135 12716435 11,75Quoted 8 lines
8192 16383 338 3974884 3,67 16384 32767 311 7018658 6,49 32768 65535 105 4511372 4,17 65536 131071 29 2577756 2,38 131072 262143 8 1339796 1,24 262144 524287 10 3950416 3,65 524288 1048575 4 2580085 2,38 1048576 2097151 2 3028028 2,80
This shows what I originally suspected. Notice how top heavy this histogram is. Over half of your free space sits on little islands of 8MB or less. 17% is in islands of 60KB or less. This is heavily fragmented free space. Contrast this with an XFS from the opposite end of the aging spectrum that is only 1/3rd full and has seen very few deletes as it has aged:
from to extents blocks pct
1 1 1028 1028 0.01
2 3 1185 2864 0.02
4 7 1656 9007 0.06
8 15 5232 67674 0.41
16 31 36 775 0.00
32 63 7 292 0.00
64 127 4 350 0.00
128 255 6 1286 0.01
256 511 4 1460 0.01
512 1023 5 4278 0.03
1024 2047 5 6965 0.04
2048 4095 5 12935 0.08
4096 8191 1 8179 0.05
8192 16383 2 29570 0.18
16384 32767 2 60352 0.37
32768 65535 3 148594 0.91
65536 131071 2 195575 1.20
131072 262143 2 420917 2.57
524288 1048575 2 1499689 9.17
4194304 6103694 3 13876795 84.88Notice how it is very bottom heavy, and that 85% of the free space is in large islands of 16GB to 24GB.
Stefan, at this point in your filesystem's aging process, it may not matter how much space you keep freeing up, as your deletion of small files simply adds more heavily fragmented free space to the pool. It's the nature of your workload causing this. If you were rsyncing and deleting 500MB files the story would be much different.
What I would suggest is doing an xfsdump to a filesystem on another LUN or machine, expand the size of this LUN by 50% or more (I gather this is an external RAID), format it appropriately, then xfsrestore. This will eliminate your current free space fragmentation, and the 50% size increase will delay the next occurrence of this problem. If you can't expand the LUN, simply do the xfsdump/format/xfsrestore, which will give you contiguous free space.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Martin Steigerwald
Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
Quoted 2 lines
after deleting 400GB it was faster. Now there are still 300GB free but it is slow as hell again ;-(
Well, before you had 200 GB free, now you have 300 GB free. Thats not that much more. You have 5 AGs, so if distributed evenly it would be 40 to 60 GB free space per AG.
The 10% Dave recommended of 4,6 TB would be 460 GB.
As I mentioned, increasing the size of the free space pool will only help for a short duration. Most of the files Stefan is deleting are small files. Thus he's simply punching out small holes of free space, which is the problem he already has. Now if he could delete a handful of large files (10GB+) that are mostly contiguous now, that would help quite a bit, as the resulting free space wouldn't be horribly fragmented.
So the takeaway is that simply deleting files to get more space isn't the right way to go. Deleting larger files that are currently contiguous is the way to go. At least for some temporary breathing room. The "permanent" solution I described in another reply.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Am Montag, 7. Mai 2012 schrieb Stan Hoeppner:
Quoted 7 lines
What I would suggest is doing an xfsdump to a filesystem on another LUN or machine, expand the size of this LUN by 50% or more (I gather this is an external RAID), format it appropriately, then xfsrestore. This will eliminate your current free space fragmentation, and the 50% size increase will delay the next occurrence of this problem. If you can't expand the LUN, simply do the xfsdump/format/xfsrestore, which will give you contiguous free space.
Maybe the new dm thin provisioning could help here? But it might just move the fragmentation from the filesystem to the device mapper layer. I did not yet check the implementation details of the new thin provisioning target.
Signature · 7 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Am 07.05.2012 18:36, schrieb Stan Hoeppner:
Quoted 6 lines
This shows what I originally suspected. Notice how top heavy this histogram is. Over half of your free space sits on little islands of 8MB or less. 17% is in islands of 60KB or less. This is heavily fragmented free space. Contrast this with an XFS from the opposite end of the aging spectrum that is only 1/3rd full and has seen very few deletes as it has aged:
...
Quoted 3 lines
Notice how it is very bottom heavy, and that 85% of the free space is in large islands of 16GB to 24GB.
This totally makes sense too me. Thanks for this explanation.
Quoted 4 lines
Stefan, at this point in your filesystem's aging process, it may not matter how much space you keep freeing up, as your deletion of small files simply adds more heavily fragmented free space to the pool. It's the nature of your workload causing this.
This makes sense - do you have any idea or solution for this? Are Filesystems, Block layers or something else which suits this problem / situation?
Quoted 7 lines
What I would suggest is doing an xfsdump to a filesystem on another LUN or machine, expand the size of this LUN by 50% or more (I gather this is an external RAID), format it appropriately, then xfsrestore. This will eliminate your current free space fragmentation, and the 50% size increase will delay the next occurrence of this problem. If you can't expand the LUN, simply do the xfsdump/format/xfsrestore, which will give you contiguous free space.
But this will only help for a few month or perhaps a year.
Greets,
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 16 lines · Stefan Priebe - Profihost AG
Quoted 3 lines
Quoted 2 lines
# vmstat"vmstat 5", not vmstat 5 times.... :/
oh sorry. Sadly the rsync processes do not run right know i've to kill them. Is the output still usable?
# vmstat 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 0 5582136 48 5849956 0 0 176 394 34 54 1 16 82 1 0 1 0 5552180 48 5854280 0 0 2493 2496 3079 2172 1 4 86 9 3 2 0 5601308 48 5857672 0 0 1098 28043 5150 1913 0 10 73 17
Quoted 8 lines
0 2 0 5595360 48 5863180 0 0 1098 14336 3945 1897 0 8 69 22 3 2 0 5594088 48 5865280 0 0 432 15897 4209 2366 0 8 71 21 0 2 0 5591068 48 5868940 0 0 854 10989 3519 2107 0 7 70 23 1 1 0 5592004 48 5869872 0 0 180 7886 3605 2436 0 3 76 22
It tells me that there is still quite an IO load on the system even when the rsyncs are not running...
Quoted 11 lines
Quoted 3 lines
Quoted 1 line
/dev/sdb14,6T 4,3T 310G 94% /mntWell, you've probably badly fragmented the free space you have. what does the '
xfs_db-r -c freesp <dev>' command tell you?from to extents blocks pct 1 1 942737 942737 0,87 2 3 671860 1590480 1,47 4 7 461268 2416025 2,23 8 15 1350517 18043063 16,67 16 31 111254 2547581 2,35 32 63 192032 9039799 8,35
So that's roughly 3.7 million free space extents of 256kB or less totalling about 32% of the freespace (~100GB). That's pretty badly fragmented, and given the workload, probably unrecoverable. Dump, mkfs and restore is probably the only way to unfragment the free space now, but that would only be a temporary solution if you continue to run at >90% full. Even if you do keep it at below 90% full, such a workload will age the filesystem and slowly fragment free space, but it should take a lot longer to get to this state...
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
/dev/sdb14875737052 4659318044 216419008 96% /mntYou have 4.6 *billion* inodes in your filesystem?
Yes - it backups around 100 servers with a lot of files.
i rechecked this and it seems i sadly copied the wrong output ;-( sorry for that.
Here is the correct one:
#~ df -i/dev/sdb1975173568 95212355 879961213 10% /mnt
That makes more sense. :)
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Stefan Priebe
Am 07.05.2012 18:36, schrieb Stan Hoeppner:
Quoted 4 lines
Stefan, at this point in your filesystem's aging process, it may not matter how much space you keep freeing up, as your deletion of small files simply adds more heavily fragmented free space to the pool. It's the nature of your workload causing this.
This makes sense - do you have any idea or solution for this? Are Filesystems, Block layers or something else which suits this problem / situation?
The problem isn't the block layer nor the filesystem. The problem is a combination of the workload and filling the FS to near capacity.
Any workload that regularly allocates and then deletes large quantities of small files and fills up the filesystem is going to suffer poor performance from free space fragmentation as the water in the FS gets close to the rim of the glass. Two other example workloads are large mail spools on very busy internet mail servers, and maildir storage on IMAP/POP servers.
In your case there are two solutions to this problem, the second of which is also the solution for these mail workloads:
1. Use a backup methodology that writes larger files
2. Give your workload a much larger sandbox to play in
Regarding #1, if you're using rsnapshot your disk space shouldn't be continuously growing, which is does seem to be. If you're not using rsnapshot look into it.
Regarding #2 ...
Quoted 8 lines
Quoted 7 lines
What I would suggest is doing an xfsdump to a filesystem on another LUN or machine, expand the size of this LUN by 50% or more (I gather this is an external RAID), format it appropriately, then xfsrestore. This will eliminate your current free space fragmentation, and the 50% size increase will delay the next occurrence of this problem. If you can't expand the LUN, simply do the xfsdump/format/xfsrestore, which will give you contiguous free space.
But this will only help for a few month or perhaps a year.
So you are saying your backup solution will fill up an additional 2.3TB in less than a year? In that case I'd say you have dramatically undersized your backup storage and/or are not using file compression to your advantage. And you're obviously not using archiving to your advantage or you'd not have the free space fragmentation issue because you'd be dealing with much larger files.
So the best solution to your current problem, and one that will save you disk space and thus $$, is to use a backup solution that makes use of both tar and gzip/bzip2.
You can't fix this fundamental small file free space fragmentation problem by tuning/tweaking XFS, or switching to another filesystem, as again, the problem is the workload, not the block layer or FS.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · Stan Hoeppner
Quoted 1 line · Stefan Priebe
Am 07.05.2012 18:36, schrieb Stan Hoeppner:
....
Quoted 16 lines
Quoted 8 lines
Quoted 7 lines
What I would suggest is doing an xfsdump to a filesystem on another LUN or machine, expand the size of this LUN by 50% or more (I gather this is an external RAID), format it appropriately, then xfsrestore. This will eliminate your current free space fragmentation, and the 50% size increase will delay the next occurrence of this problem. If you can't expand the LUN, simply do the xfsdump/format/xfsrestore, which will give you contiguous free space.
But this will only help for a few month or perhaps a year.
So you are saying your backup solution will fill up an additional 2.3TB in less than a year? In that case I'd say you have dramatically undersized your backup storage and/or are not using file compression to your advantage. And you're obviously not using archiving to your advantage or you'd not have the free space fragmentation issue because you'd be dealing with much larger files.
Quoted 3 lines
So the best solution to your current problem, and one that will save you disk space and thus $$, is to use a backup solution that makes use of both tar and gzip/bzip2.
Why use the slow method? xfsdump will be much faster than tar. :)
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Am 09.05.2012 09:04, schrieb Dave Chinner:
Quoted 5 lines
Quoted 3 lines
So the best solution to your current problem, and one that will save you disk space and thus $$, is to use a backup solution that makes use of both tar and gzip/bzip2.
Why use the slow method? xfsdump will be much faster than tar. :)
I tried to use it in this special case but it is hanging on dumping directories for 3 hours now or is this normal?
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 20 lines · Dave Chinner
Quoted 2 lines · Stan Hoeppner
Quoted 1 line · Stefan Priebe
Am 07.05.2012 18:36, schrieb Stan Hoeppner:
....
Quoted 16 lines
Quoted 8 lines
Quoted 7 lines
What I would suggest is doing an xfsdump to a filesystem on another LUN or machine, expand the size of this LUN by 50% or more (I gather this is an external RAID), format it appropriately, then xfsrestore. This will eliminate your current free space fragmentation, and the 50% size increase will delay the next occurrence of this problem. If you can't expand the LUN, simply do the xfsdump/format/xfsrestore, which will give you contiguous free space.
But this will only help for a few month or perhaps a year.
So you are saying your backup solution will fill up an additional 2.3TB in less than a year? In that case I'd say you have dramatically undersized your backup storage and/or are not using file compression to your advantage. And you're obviously not using archiving to your advantage or you'd not have the free space fragmentation issue because you'd be dealing with much larger files.
Quoted 5 lines
Quoted 3 lines
So the best solution to your current problem, and one that will save you disk space and thus $$, is to use a backup solution that makes use of both tar and gzip/bzip2.
Why use the slow method? xfsdump will be much faster than tar. :)
Certainly faster. I've assumed Stefan is backing up heterogeneous systems so xfsdump is probably not an option across the board. My thinking directly above related to the possibility that his current software may have integrated functions for tar'ing then compressing at the directory level. This would save space, cause less free space fragmentation when old files are deleted, and allow decompressing files that aren't insanely large in the even a single file needs to be restored from a compressed tar file. Of course the latter can be accomplished with xfsdump/restore, but not on heterogeneous systems. And AFAIK there isn't any nifty FOSS backup software that makes use of xfsdump/restore, requiring some serious custom scripting. If there exists such backup software please do share. :)
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
32bit apps and inode64
In reply to Dave ChinnerHello list,
i've discovered some problems on a host with a disk > 1TB. We've some binary 32bit applications which are not able to read some directory anymore after we've formated and installed the system using vanilla 3.7.7 kernel.
Right now we're using 3.0.61 kernel on this host - so 64bit apps work well and newly created files get 32bit inode numbers as inode64 is not the default.
Is there a way to find / get all 64bit inode files / dies and convert them back to 32bit without a reinstall?
Greets,
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: 32bit apps and inode64
Hi Stefan,
Quoted 11 lines · Stefan Priebe - Profihost AG
i've discovered some problems on a host with a disk > 1TB. We've some binary 32bit applications which are not able to read some directory anymore after we've formated and installed the system using vanilla 3.7.7 kernel.
Right now we're using 3.0.61 kernel on this host - so 64bit apps work well and newly created files get 32bit inode numbers as inode64 is not the default.
Is there a way to find / get all 64bit inode files / dies and convert them back to 32bit without a reinstall?
On IRIX you could use xfs_reno to renumber those inodes.
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z
xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu:
http://oss.sgi.com/archives/xfs/2012-11/msg00425.html
It isn't in xfsprogs today.
Regards,
Ben
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: 32bit apps and inode64
I'm on linux. So no recommended way for production systems?
Stefan
Am 15.02.2013 um 22:46 schrieb Ben Myers <[email protected]>:
Quoted 18 lines
Hi Stefan,
Quoted 11 lines · Stefan Priebe - Profihost AG
i've discovered some problems on a host with a disk > 1TB. We've some binary 32bit applications which are not able to read some directory anymore after we've formated and installed the system using vanilla 3.7.7 kernel.
Right now we're using 3.0.61 kernel on this host - so 64bit apps work well and newly created files get 32bit inode numbers as inode64 is not the default.
Is there a way to find / get all 64bit inode files / dies and convert them back to 32bit without a reinstall?
On IRIX you could use
xfs_renoto renumber those inodes.
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z
Quoted 7 lines
xfs_renowas ported to linux in '07 and was most recently reposted by Jeff Liu:
http://oss.sgi.com/archives/xfs/2012-11/msg00425.htmlIt isn't in xfsprogs today.
Regards,
Ben
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: 32bit apps and inode64
In reply to Ben MyersHi Ben,
Quoted 12 lines · Ben Myers
Hi Stefan,
Quoted 9 lines · Stefan Priebe - Profihost AG
i've discovered some problems on a host with a disk > 1TB. We've some binary 32bit applications which are not able to read some directory anymore after we've formated and installed the system using vanilla 3.7.7 kernel.
Right now we're using 3.0.61 kernel on this host - so 64bit apps work well and newly created files get 32bit inode numbers as inode64 is not the default.
Quoted 8 lines
Quoted 2 lines
Is there a way to find / get all 64bit inode files / dies and convert them back to 32bit without a reinstall?
On IRIX you could use
xfs_renoto renumber those inodes.
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z
xfs_renowas ported to linux in '07 and was most recently reposted by Jeff Liu:
http://oss.sgi.com/archives/xfs/2012-11/msg00425.html
The old patch set was belong to the infrastructures of online shrinking support. Recently, I realized that I have made a few mistakes in swap inodes ioctl(2) implementation when I revisit the old patch set at:
http://oss.sgi.com/archives/xfs/2012-11/msg00414.html
Since we have user request and this function is independent to the shrinking feature, I'd like to work on it at first if you like it.
Thanks,
-Jeff
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: 32bit apps and inode64
In reply to Stefan Priebe - Profihost AGQuoted 28 lines · Stefan Priebe - Profihost AG
I'm on linux. So no recommended way for production systems?
Stefan
Am 15.02.2013 um 22:46 schrieb Ben Myers <[email protected]>:
Quoted 22 lines
Hi Stefan,
Quoted 11 lines · Stefan Priebe - Profihost AG
i've discovered some problems on a host with a disk > 1TB. We've some binary 32bit applications which are not able to read some directory anymore after we've formated and installed the system using vanilla 3.7.7 kernel.
Right now we're using 3.0.61 kernel on this host - so 64bit apps work well and newly created files get 32bit inode numbers as inode64 is not the default.
Is there a way to find / get all 64bit inode files / dies and convert them back to 32bit without a reinstall?
On IRIX you could use
xfs_renoto renumber those inodes.
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z
xfs_renowas ported to linux in '07 and was most recently reposted by Jeff Liu:
http://oss.sgi.com/archives/xfs/2012-11/msg00425.htmlIt isn't in xfsprogs today.
Simple answer: mount with inode32, run find to print out all the filenames in the filesystem and their inode number, copy the files with inodes numbers greater than 32 bit to a temporary file and then rename them over the top of the original.
That's effectively all xfs_reno does, anyway, just with faster algorithms (like bulkstat) and a bunch of crash resiliency semantics wrapped around it....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: 32bit apps and inode64
Hi,
Am 17.02.2013 22:33, schrieb Dave Chinner:
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
xfs_renowas ported to linux in '07 and was most recently reposted by Jeff Liu:
http://oss.sgi.com/archives/xfs/2012-11/msg00425.htmlIt isn't in xfsprogs today.
Simple answer: mount with inode32, run find to print out all the filenames in the filesystem and their inode number, copy the files with inodes numbers greater than 32 bit to a temporary file and then rename them over the top of the original.
Thanks, what about directories? Is renaming / moving using mv enough?
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: 32bit apps and inode64
Quoted 7 lines · Stefan Priebe - Profihost AG
Hi,
Am 17.02.2013 22:33, schrieb Dave Chinner:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
xfs_renowas ported to linux in '07 and was most recently reposted by Jeff Liu:
http://oss.sgi.com/archives/xfs/2012-11/msg00425.html
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
It isn't in xfsprogs today.
Simple answer: mount with inode32, run find to print out all the filenames in the filesystem and their inode number, copy the files with inodes numbers greater than 32 bit to a temporary file and then rename them over the top of the original.
Thanks, what about directories? Is renaming / moving using mv enough?
Same thing - create a new temp directory, mv all the entries (which will be a rename operation, anyway) from the old directory into the temp directory, rename the temp dir over the top of the old one.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: 32bit apps and inode64
In reply to Jeff LiuHey Jeff,
Quoted 23 lines · Jeff Liu
Hi Ben,
Quoted 12 lines · Ben Myers
Hi Stefan,
Quoted 9 lines · Stefan Priebe - Profihost AG
i've discovered some problems on a host with a disk > 1TB. We've some binary 32bit applications which are not able to read some directory anymore after we've formated and installed the system using vanilla 3.7.7 kernel.
Right now we're using 3.0.61 kernel on this host - so 64bit apps work well and newly created files get 32bit inode numbers as inode64 is not the default.
Quoted 8 lines
Quoted 2 lines
Is there a way to find / get all 64bit inode files / dies and convert them back to 32bit without a reinstall?
On IRIX you could use
xfs_renoto renumber those inodes.
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z
xfs_renowas ported to linux in '07 and was most recently reposted by Jeff Liu:
http://oss.sgi.com/archives/xfs/2012-11/msg00425.html
Quoted 7 lines
The old patch set was belong to the infrastructures of online shrinking support. Recently, I realized that I have made a few mistakes in swap inodes ioctl(2) implementation when I revisit the old patch set at:
http://oss.sgi.com/archives/xfs/2012-11/msg00414.htmlSince we have user request and this function is independent to the shrinking feature, I'd like to work on it at first if you like it.
That sounds good to me. xfs_reno is a worthwhile feature.
Regards,
Ben
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs