fragmentation question
Hello All,
I have an issue with fragmentation on a particular device thanks for any advice.
-Brady
I have a Dell r510 with 12 disks
2xraid 5 (6 disks each)
raid group1:
48 GB carved out for os mounted as /
remaining space 2.7 TB for xfs mounted as /data1
raid group2:
48 GB for swap
remaining space 2.7 TB for xfs mounted as /data2
The strange thing is that /data1 never gets fragmented where as /data2 is badly fragmented.
I believe increase allocsize would help, but not sure how to explain why /data2(/dev/sdd) always gets fragmented and not /data1(/dev/sdb)
It's a data warehouse application. the I/O is balanced between /data1 and /data2:
output of xfs_db
[root@sdw4 data1]# xfs_db -c frag -r /dev/sdb
actual 14353, ideal 13702, fragmentation factor 4.54%
[root@sdw4 data1]# xfs_db -c frag -r /dev/sdd
actual 408674, ideal 13719, fragmentation factor 96.64%
df output
/dev/sdb 2.7T 967G 1.8T 36% /data1 /dev/sdd 2.7T 1.1T 1.7T 39% /data2
LABEL=/data1 /data1 xfs allocsize=1048576,logbufs=8,noatime,nodiratime 0 0 LABEL=/data2 /data2 xfs allocsize=1048576,logbufs=8,noatime,nodiratime 0 0
raid config output:
[root@sdw4 data1]# omreport storage vdisk
List of Virtual Disks in the System
Controller PERC H700 Integrated (Slot 4)
ID : 0
Status : Ok
Name : boot
State : Ready
Hot Spare Policy violated : Not Assigned
Virtual Disk Bad Blocks : No
Secured : Not Applicable
Progress : Not Applicable
Layout : RAID-5
Size : 48.99 GB (52602470400 bytes)
Device Name : /dev/sda
Bus Protocol : SAS
Media : HDD
Read Policy : No Read Ahead
Write Policy : Force Write Back
Cache Policy : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy : Disabled
ID : 1
Status : Ok
Name : data1
State : Ready
Hot Spare Policy violated : Not Assigned
Virtual Disk Bad Blocks : No
Secured : Not Applicable
Progress : Not Applicable
Layout : RAID-5
Size : 2,742.89 GB (2945150484480 bytes)
Device Name : /dev/sdb
Bus Protocol : SAS
Media : HDD
Read Policy : No Read Ahead
Write Policy : Force Write Back
Cache Policy : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy : Disabled
ID : 2
Status : Ok
Name : swap
State : Ready
Hot Spare Policy violated : Not Assigned
Virtual Disk Bad Blocks : No
Secured : Not Applicable
Progress : Not Applicable
Layout : RAID-5
Size : 48.99 GB (52602470400 bytes)
Device Name : /dev/sdc
Bus Protocol : SAS
Media : HDD
Read Policy : No Read Ahead
Write Policy : Force Write Back
Cache Policy : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy : Disabled
ID : 3
Status : Ok
Name : data2
State : Ready
Hot Spare Policy violated : Not Assigned
Virtual Disk Bad Blocks : No
Secured : Not Applicable
Progress : Not Applicable
Layout : RAID-5
Size : 2,742.89 GB (2945150484480 bytes)
Device Name : /dev/sdd
Bus Protocol : SAS
Media : HDD
Read Policy : No Read Ahead
Write Policy : Force Write Back
Cache Policy : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy : Disabled
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
By the way, the os is RHEL 5.5 kernel 2.6.18-194.11.1.el5
thanks in advance.
-Brady
On 9/7/10 5:45 PM, "Brady Chang" <[email protected]> wrote:
Hello All,
I have an issue with fragmentation on a particular device thanks for any advice.
-Brady
I have a Dell r510 with 12 disks
2xraid 5 (6 disks each)
raid group1:
48 GB carved out for os mounted as /
remaining space 2.7 TB for xfs mounted as /data1
raid group2:
48 GB for swap
remaining space 2.7 TB for xfs mounted as /data2
The strange thing is that /data1 never gets fragmented where as /data2 is badly fragmented.
I believe increase allocsize would help, but not sure how to explain why /data2(/dev/sdd) always gets fragmented and not /data1(/dev/sdb)
It's a data warehouse application. the I/O is balanced between /data1 and /data2:
output of xfs_db
[root@sdw4 data1]# xfs_db -c frag -r /dev/sdb
actual 14353, ideal 13702, fragmentation factor 4.54%
[root@sdw4 data1]# xfs_db -c frag -r /dev/sdd
actual 408674, ideal 13719, fragmentation factor 96.64%
df output
/dev/sdb 2.7T 967G 1.8T 36% /data1 /dev/sdd 2.7T 1.1T 1.7T 39% /data2
LABEL=/data1 /data1 xfs allocsize=1048576,logbufs=8,noatime,nodiratime 0 0 LABEL=/data2 /data2 xfs allocsize=1048576,logbufs=8,noatime,nodiratime 0 0
raid config output:
[root@sdw4 data1]# omreport storage vdisk
List of Virtual Disks in the System
Controller PERC H700 Integrated (Slot 4)
ID : 0
Status : Ok
Name : boot
State : Ready
Hot Spare Policy violated : Not Assigned
Virtual Disk Bad Blocks : No
Secured : Not Applicable
Progress : Not Applicable
Layout : RAID-5
Size : 48.99 GB (52602470400 bytes)
Device Name : /dev/sda
Bus Protocol : SAS
Media : HDD
Read Policy : No Read Ahead
Write Policy : Force Write Back
Cache Policy : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy : Disabled
ID : 1
Status : Ok
Name : data1
State : Ready
Hot Spare Policy violated : Not Assigned
Virtual Disk Bad Blocks : No
Secured : Not Applicable
Progress : Not Applicable
Layout : RAID-5
Size : 2,742.89 GB (2945150484480 bytes)
Device Name : /dev/sdb
Bus Protocol : SAS
Media : HDD
Read Policy : No Read Ahead
Write Policy : Force Write Back
Cache Policy : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy : Disabled
ID : 2
Status : Ok
Name : swap
State : Ready
Hot Spare Policy violated : Not Assigned
Virtual Disk Bad Blocks : No
Secured : Not Applicable
Progress : Not Applicable
Layout : RAID-5
Size : 48.99 GB (52602470400 bytes)
Device Name : /dev/sdc
Bus Protocol : SAS
Media : HDD
Read Policy : No Read Ahead
Write Policy : Force Write Back
Cache Policy : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy : Disabled
ID : 3
Status : Ok
Name : data2
State : Ready
Hot Spare Policy violated : Not Assigned
Virtual Disk Bad Blocks : No
Secured : Not Applicable
Progress : Not Applicable
Layout : RAID-5
Size : 2,742.89 GB (2945150484480 bytes)
Device Name : /dev/sdd
Bus Protocol : SAS
Media : HDD
Read Policy : No Read Ahead
Write Policy : Force Write Back
Cache Policy : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy : Disabled
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Tue, 7 Sep 2010 17:45:41 -0700 vous écriviez:
Quoted 2 lines
I have an issue with fragmentation on a particular device thanks for any advice.
I'd try monitoring io with "iostat -mx 4" for a while. I suppose most write activity goes to data1.
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsQuoted 20 lines · Brady Chang
Hello All,
I have an issue with fragmentation on a particular device thanks for any advice.-Brady
I have a Dell r510 with 12 disks
2xraid 5 (6 disks each)
raid group1:
48 GB carved out for os mounted as /
remaining space 2.7 TB for xfs mounted as /data1 raid group2:
48 GB for swap
remaining space 2.7 TB for xfs mounted as /data2The strange thing is that /data1 never gets fragmented where as /data2 is badly fragmented.
I believe increase allocsize would help, but not sure how to explain why /data2(/dev/sdd) always gets fragmented and not /data1(/dev/sdb)
Quoted 7 lines
It's a data warehouse application. the I/O is balanced between /data1 and /data2:
output ofxfs_db
[root@sdw4 data1]#xfs_db-c frag -r/dev/sdb
actual 14353, ideal 13702, fragmentation factor 4.54% [root@sdw4 data1]#xfs_db-c frag -r/dev/sdd
actual 408674, ideal 13719, fragmentation factor 96.64%
so each file has 30 extents on average (actual/ideal)
Quoted 3 lines
df output
/dev/sdb 2.7T 967G 1.8T 36% /data1 /dev/sdd 2.7T 1.1T 1.7T 39% /data2
1.1T/408674 extents is ~3M per extent, not so good.
How many files are on each fs?
Quoted 4 lines
LABEL=/data1 /data1 xfs
allocsize=1048576,logbufs=8,noatime,nodiratime 0 0 LABEL=/data2 /data2 xfs allocsize=1048576,logbufs=8,noatime,nodiratime 0 0
Everything but the first option is default, BTW.
Is xfs_info output on the 2 filesystems the same?
Otherwise Emmanuel's idea is a good one, maybe it's not as balanced as you think it is, or maybe they have aged differently and have different amounts of freespace (see the freesp command in xfs_db)
Quoted 1 line
By the way, the os is RHEL 5.5 kernel 2.6.18-194.11.1.el5
Was Red Hat support not helpful?
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
thanks guys for the feedback.
iostat shows balanced io between two filesystems.
testing with RHEL 54, no issues there. seems to be happening on RHEL 55 only.
I do not have the latest 55 kernel. I'm going to upgrade to the latest 55 kernel and rerun the test.
On 9/9/10 7:12 AM, "Eric Sandeen" <[email protected]> wrote:
Quoted 20 lines · Brady Chang
Hello All,
I have an issue with fragmentation on a particular device thanks for any advice.-Brady
I have a Dell r510 with 12 disks
2xraid 5 (6 disks each)
raid group1:
48 GB carved out for os mounted as /
remaining space 2.7 TB for xfs mounted as /data1 raid group2:
48 GB for swap
remaining space 2.7 TB for xfs mounted as /data2The strange thing is that /data1 never gets fragmented where as /data2 is badly fragmented.
I believe increase allocsize would help, but not sure how to explain why /data2(/dev/sdd) always gets fragmented and not /data1(/dev/sdb)
Quoted 7 lines
It's a data warehouse application. the I/O is balanced between /data1 and /data2:
output ofxfs_db
[root@sdw4 data1]#xfs_db-c frag -r/dev/sdb
actual 14353, ideal 13702, fragmentation factor 4.54% [root@sdw4 data1]#xfs_db-c frag -r/dev/sdd
actual 408674, ideal 13719, fragmentation factor 96.64%
so each file has 30 extents on average (actual/ideal)
Quoted 3 lines
df output
/dev/sdb 2.7T 967G 1.8T 36% /data1 /dev/sdd 2.7T 1.1T 1.7T 39% /data2
1.1T/408674 extents is ~3M per extent, not so good.
How many files are on each fs?
Quoted 4 lines
LABEL=/data1 /data1 xfs
allocsize=1048576,logbufs=8,noatime,nodiratime 0 0 LABEL=/data2 /data2 xfs allocsize=1048576,logbufs=8,noatime,nodiratime 0 0
Everything but the first option is default, BTW.
Is xfs_info output on the 2 filesystems the same?
Otherwise Emmanuel's idea is a good one, maybe it's not as balanced as you think it is, or maybe they have aged differently and have different amounts of freespace (see the freesp command in xfs_db)
Quoted 1 line
By the way, the os is RHEL 5.5 kernel 2.6.18-194.11.1.el5
Was Red Hat support not helpful?
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Brady Chang
thanks guys for the feedback.
iostat shows balanced io between two filesystems.
testing with RHEL 54, no issues there. seems to be happening on RHEL 55 only.
There were no xfs changes between 5.4 and 5.5 that should be relevant, only a single bugfixes for 5.5, related to fallocate error returns.
Just FWIW.
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Thanks,
after couple runs of tpc-H , RHEL 5.4 /dev/sdd is heavily fragmented. so back to the original problem. it always happen on /dev/sdd
[root@sdw9 data1]# xfs_db -c frag -r /dev/sdb actual 1773, ideal 1731, fragmentation factor 2.37%
[root@sdw9 data1]# xfs_db -c frag -r /dev/sdd actual 43384, ideal 1726, fragmentation factor 96.02%
[root@sdw9 data1]# xfs_info /dev/sdb
meta-data=/dev/sdb isize=256 agcount=32, agsize=22469715 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=719030880, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0[root@sdw9 data1]# xfs_info /dev/sdd
meta-data=/dev/sdd isize=256 agcount=32, agsize=22469715 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=719030880, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=067 files on both /dev/sdb and /dev/sdd.
On 9/9/10 3:06 PM, "Eric Sandeen" <[email protected]> wrote:
Quoted 4 lines · Brady Chang
thanks guys for the feedback.
iostat shows balanced io between two filesystems.
testing with RHEL 54, no issues there. seems to be happening on RHEL 55 only.
There were no xfs changes between 5.4 and 5.5 that should be relevant, only a single bugfixes for 5.5, related to fallocate error returns.
Just FWIW.
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Brady Chang put forth on 9/9/2010 4:59 PM:
Quoted 2 lines
thanks guys for the feedback.
iostat shows balanced io between two filesystems.
Can we please see the "iostat -x" output for the duration of the TPC-H run _only_?
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Stan Hoeppner put forth on 9/9/2010 10:23 PM:
Quoted 6 lines
Brady Chang put forth on 9/9/2010 4:59 PM:
Quoted 2 lines
thanks guys for the feedback.
iostat shows balanced io between two filesystems.Can we please see the "iostat -x" output for the duration of the TPC-H run _only_?
What db engine are you using? Oracle, DB2, MySQL, or PostgreSQL?
Exactly how are you instructing it to split files between /data1 and /data2?
Are you instructing your db engine to split your transaction logs and db files equally across both filesystems?
On which filesystem are you locating your database engine scratch space if any?
Can you run xfs_bmap on the files on each filesystem to determine which are the most fragmented? Doing this may/should produce the smoking gun.
At this point, I'm guessing you have transaction logs and db scratch space allocated to /data2 which is causing the heavy fragmentation.
It would be instructive if you dropped /data2 out of the picture entirely and ran the TPC-H benchy using only /data1. I'm sure you'll see the fragmentation on /data1 in this case.
It seems clear your fragmentation issue is a database management issue, not an XFS issue. Your answers to my questions should tell us which.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
xfs_info output after the TPC-H runs are the same.
[root@sdw9 data1]# xfs_info /dev/sdd
meta-data=/dev/sdd isize=256 agcount=32, agsize=22469715 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=719030880, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
[root@sdw9 data1]# xfs_info /dev/sdb
meta-data=/dev/sdb isize=256 agcount=32, agsize=22469715 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=719030880, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0[root@sdw9 data1]# xfs_db -c frag -r /dev/sdb actual 1799, ideal 1748, fragmentation factor 2.83% [root@sdw9 data1]# xfs_db -c frag -r /dev/sdd actual 54324, ideal 1749, fragmentation factor 96.78%
On 9/9/10 7:12 AM, "Eric Sandeen" <[email protected]> wrote:
Quoted 20 lines · Brady Chang
Hello All,
I have an issue with fragmentation on a particular device thanks for any advice.-Brady
I have a Dell r510 with 12 disks
2xraid 5 (6 disks each)
raid group1:
48 GB carved out for os mounted as /
remaining space 2.7 TB for xfs mounted as /data1 raid group2:
48 GB for swap
remaining space 2.7 TB for xfs mounted as /data2The strange thing is that /data1 never gets fragmented where as /data2 is badly fragmented.
I believe increase allocsize would help, but not sure how to explain why /data2(/dev/sdd) always gets fragmented and not /data1(/dev/sdb)
Quoted 7 lines
It's a data warehouse application. the I/O is balanced between /data1 and /data2:
output ofxfs_db
[root@sdw4 data1]#xfs_db-c frag -r/dev/sdb
actual 14353, ideal 13702, fragmentation factor 4.54% [root@sdw4 data1]#xfs_db-c frag -r/dev/sdd
actual 408674, ideal 13719, fragmentation factor 96.64%
so each file has 30 extents on average (actual/ideal)
Quoted 3 lines
df output
/dev/sdb 2.7T 967G 1.8T 36% /data1 /dev/sdd 2.7T 1.1T 1.7T 39% /data2
1.1T/408674 extents is ~3M per extent, not so good.
How many files are on each fs?
Quoted 4 lines
LABEL=/data1 /data1 xfs
allocsize=1048576,logbufs=8,noatime,nodiratime 0 0 LABEL=/data2 /data2 xfs allocsize=1048576,logbufs=8,noatime,nodiratime 0 0
Everything but the first option is default, BTW.
Is xfs_info output on the 2 filesystems the same?
Otherwise Emmanuel's idea is a good one, maybe it's not as balanced as you think it is, or maybe they have aged differently and have different amounts of freespace (see the freesp command in xfs_db)
Quoted 1 line
By the way, the os is RHEL 5.5 kernel 2.6.18-194.11.1.el5
Was Red Hat support not helpful?
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs