posix_fallocate

Hello,

I use this to preallocate large space but found an issue. Posix_fallocate works right with sizes like 100G, 1T and even 10T on some boxes (on some other can fail after e.g. 7T threshold) but if i tried e.g. 16T the user space process would be "R"unning forever and it is not interruptible. Furthermore some other not related processes like sshd, bash enter D state. There is nothing in kernel log.

I made so far a few logs with ftrace facility for 1G, 100G, 1T and 10T sizes. I noticed that for 1st three sizes the log is as long as abt 1.5M (2M peak) while 10T generates 94M long log. I couldn't retrieve a log for 17T case because "cat /sys ... /trace" enters D.

I would appreciate any help because i gave up with ftrace logs analysis. The xfs_vn_fallocate is covered in abt 11k lines for a 1.5M log case while there are abt 163k lines in 94M log. And all i could see is poss some relationship between time spent in xfs_vn_fallocate subfunctions vs requested space.

Box details:
16 Hitachi 2TB drives (backplane connected), dm, 1 lvm lun of 25T size, kernel 2.6.31.5, more recent kernels neither xfs were not tested.

Krzysztof Błaszkowski put forth on 5/7/2010 3:22 AM:

Quoted 20 lines

Hello,

I use this to preallocate large space but found an issue. Posix_fallocate works right with sizes like 100G, 1T and even 10T on some boxes (on some other can fail after e.g. 7T threshold) but if i tried e.g. 16T the user space process would be "R"unning forever and it is not interruptible. Furthermore some other not related processes like sshd, bash enter D state. There is nothing in kernel log.

I made so far a few logs with ftrace facility for 1G, 100G, 1T and 10T sizes. I noticed that for 1st three sizes the log is as long as abt 1.5M (2M peak) while 10T generates 94M long log. I couldn't retrieve a log for 17T case because "cat /sys ... /trace" enters D.

I would appreciate any help because i gave up with ftrace logs analysis. The xfs_vn_fallocate is covered in abt 11k lines for a 1.5M log case while there are abt 163k lines in 94M log. And all i could see is poss some relationship between time spent in xfs_vn_fallocate subfunctions vs requested space.

Quoted 3 lines

Box details:
16 Hitachi 2TB drives (backplane connected), dm, 1 lvm lun of 25T size, kernel 2.6.31.5, more recent kernels neither xfs were not tested.

32 or 64 bit kernel?

What is the size of the XFS filesystem on the 25TB LVM LUN against which you're running posix_fallocate?

The reason I ask is
that XFS has a 16TB per filesystem limitation on 32 bit kernels. I can only assume that your XFS filesystem is larger than 16TB since you're attempting to posix_fallocate 16TB. But, it's best to ask for confirmation rather than assume, especially given that your problem is appearing near that magical 16TB boundary.

Krzysztof Błaszkowski put forth on 5/7/2010 3:22 AM:

Quoted 20 lines

Hello,

I use this to preallocate large space but found an issue. Posix_fallocate works right with sizes like 100G, 1T and even 10T on some boxes (on some other can fail after e.g. 7T threshold) but if i tried e.g. 16T the user space process would be "R"unning forever and it is not interruptible. Furthermore some other not related processes like sshd, bash enter D state. There is nothing in kernel log.

I made so far a few logs with ftrace facility for 1G, 100G, 1T and 10T sizes. I noticed that for 1st three sizes the log is as long as abt 1.5M (2M peak) while 10T generates 94M long log. I couldn't retrieve a log for 17T case because "cat /sys ... /trace" enters D.

I would appreciate any help because i gave up with ftrace logs analysis. The xfs_vn_fallocate is covered in abt 11k lines for a 1.5M log case while there are abt 163k lines in 94M log. And all i could see is poss some relationship between time spent in xfs_vn_fallocate subfunctions vs requested space.

Quoted 3 lines

Box details:
16 Hitachi 2TB drives (backplane connected), dm, 1 lvm lun of 25T size, kernel 2.6.31.5, more recent kernels neither xfs were not tested.

32 or 64 bit kernel? What is the size of the XFS filesystem on the 25TB LVM LUN against which you're running posix_fallocate? The reason I ask is that XFS has a 16TB per filesystem limitation on 32 bit kernels. I can only assume that your XFS filesystem is larger than 16TB since you're attempting to posix_fallocate 16TB. But, it's best to ask for confirmation rather than assume, especially given that your problem is appearing near that magical 16TB boundary.

Hello Stan,

It seems that your mail server blocks traffic from ".pl" domains so don't be offended if you will not see my reply sent to your address.

(Remote host said: 550 5.7.1 <v007470.home.net.pl[212.85.125.104]>: Client host rejected: We do not accept mail from .pl domains)

Krzysztof Błaszkowski put forth on 5/7/2010 5:07 AM:

Quoted 7 lines

Hello Stan,

It seems that your mail server blocks traffic from ".pl" domains so don't be offended if you will not see my reply sent to your address.

(Remote host said: 550 5.7.1 <v007470.home.net.pl[212.85.125.104]>:
Client host rejected: We do not accept mail from .pl domains)

Sorry about that. Due to the constant battle against spam I've implemented some pretty draconian countermeasures over time, including some ccTLD blocking of SMTP. All my "overseas" contacts are via public mailing lists, as in this case. Very rarely does a conversation need to go "off list", so I've not had much of a problem with this setup. If you'd like I can whitelist your address.

Hello,

I use this to preallocate large space but found an issue. Posix_fallocate works right with sizes like 100G, 1T and even 10T on some boxes (on some other can fail after e.g. 7T threshold) but if i tried e.g. 16T the user space process would be "R"unning forever and it is not interruptible. Furthermore some other not related processes like sshd, bash enter D state. There is nothing in kernel log.

I made so far a few logs with ftrace facility for 1G, 100G, 1T and 10T sizes. I noticed that for 1st three sizes the log is as long as abt 1.5M (2M peak) while 10T generates 94M long log. I couldn't retrieve a log for 17T case because "cat /sys ... /trace" enters D.

I would appreciate any help because i gave up with ftrace logs analysis. The xfs_vn_fallocate is covered in abt 11k lines for a 1.5M log case while there are abt 163k lines in 94M log. And all i could see is poss some relationship between time spent in xfs_vn_fallocate subfunctions vs requested space.

Box details:
16 Hitachi 2TB drives (backplane connected), dm, 1 lvm lun of 25T size, kernel 2.6.31.5, more recent kernels neither xfs were not tested.

Quoted 8 lines · Krzysztof Błaszkowski

Hello,

I use this to preallocate large space but found an issue. Posix_fallocate works right with sizes like 100G, 1T and even 10T on some boxes (on some other can fail after e.g. 7T threshold) but if i tried e.g. 16T the user space process would be "R"unning forever and it is not interruptible. Furthermore some other not related processes like sshd, bash enter D state. There is nothing in kernel log.

Quoted 9 lines · Eric Sandeen

Quoted 8 lines · Krzysztof Błaszkowski

Hello,

I use this to preallocate large space but found an issue. Posix_fallocate works right with sizes like 100G, 1T and even 10T on some boxes (on some other can fail after e.g. 7T threshold) but if i tried e.g. 16T the user space process would be "R"unning forever and it is not interruptible. Furthermore some other not related processes like sshd, bash enter D state. There is nothing in kernel log.

Oh, one thing you should know is that depending on your version of glibc, posix_fallocate may be writing 0s and not using preallocation calls.

Do you know which yours is using? strace should tell you on a small file test.

Anyway, I am seeing things get stuck around 8T it seems...

# touch /mnt/test/bigfile
# xfs_io -c "resvsp 0 16t" /mnt/test/bigfile 

... wait ... in other window ...

# du -hc /mnt/test/bigfile 
8.0G	/mnt/test/bigfile
8.0G	total

# echo t > /proc/sysrq-trigger 
# dmesg | grep -A20 xfs_io
xfs_io        R  running task     3576 29444  29362 0x00000006
 ffff8809cfbb4920 ffffffff81478d9f ffffffffa032d3c5 0000000000000246
 ffff8809cfbb4920 ffffffff814788bc 0000000000000000 ffffffff81ba3510
 ffff8809d3429a68 ffffffffa032b60f ffff8809d3429aa8 000000000000001e
Call Trace:
 [<ffffffff81478d9f>] ? __mutex_lock_common+0x36d/0x392
 [<ffffffffa032d3c5>] ? xfs_icsb_modify_counters+0x17f/0x1ac [xfs]
 [<ffffffffa032b60f>] ? xfs_icsb_unlock_all_counters+0x4d/0x60 [xfs]
 [<ffffffffa032b8bf>] ? xfs_icsb_disable_counter+0x8c/0x95 [xfs]
 [<ffffffff81478e88>] ? mutex_lock_nested+0x3e/0x43
 [<ffffffffa032d3d3>] ? xfs_icsb_modify_counters+0x18d/0x1ac [xfs]
 [<ffffffffa032d536>] ? xfs_mod_incore_sb+0x29/0x6e [xfs]
 [<ffffffffa033052c>] ? _xfs_trans_alloc+0x27/0x61 [xfs]
 [<ffffffffa03303d3>] ? xfs_trans_reserve+0x6c/0x19e [xfs]
 [<ffffffff8106fb45>] ? up_write+0x2b/0x32
 [<ffffffffa0335e55>] ? xfs_alloc_file_space+0x163/0x306 [xfs]
 [<ffffffff8107120a>] ? sched_clock_cpu+0xc3/0xce
 [<ffffffffa0336122>] ? xfs_change_file_space+0x12a/0x2b8 [xfs]
 [<ffffffff8106f9bf>] ? down_write_nested+0x80/0x8b
 [<ffffffffa031b8ce>] ? xfs_ilock+0x30/0xb4 [xfs]
 [<ffffffffa033e0e4>] ? xfs_vn_fallocate+0x80/0xf4 [xfs]
--
R         xfs_io 29444  86014624.786617       162   120  86014624.786617    137655.161327       408.979977 /

# uname -r
2.6.34-0.4.rc0.git2.fc14.x86_64

I'll look into it.

Quoted 9 lines · Eric Sandeen

Quoted 8 lines · Krzysztof Błaszkowski

Hello,

I use this to preallocate large space but found an issue. Posix_fallocate works right with sizes like 100G, 1T and even 10T on some boxes (on some other can fail after e.g. 7T threshold) but if i tried e.g. 16T the user space process would be "R"unning forever and it is not interruptible. Furthermore some other not related processes like sshd, bash enter D state. There is nothing in kernel log.

Oh, one thing you should know is that depending on your version of glibc, posix_fallocate may be writing 0s and not using preallocation calls.

Do you know which yours is using?

strace should tell you on a small
file test.

Anyway, I am seeing things get stuck around 8T it seems...

# touch /mnt/test/bigfile
# xfs_io -c "resvsp 0 16t" /mnt/test/bigfile

... wait ... in other window ...

# du -hc /mnt/test/bigfile
8.0G	/mnt/test/bigfile
8.0G	total

# echo t > /proc/sysrq-trigger

# dmesg | grep -A20 xfs_io
xfs_io        R  running task     3576 29444  29362 0x00000006
 ffff8809cfbb4920 ffffffff81478d9f ffffffffa032d3c5 0000000000000246
 ffff8809cfbb4920 ffffffff814788bc 0000000000000000 ffffffff81ba3510
 ffff8809d3429a68 ffffffffa032b60f ffff8809d3429aa8 000000000000001e
Call Trace:
 [<ffffffff81478d9f>] ? __mutex_lock_common+0x36d/0x392
 [<ffffffffa032d3c5>] ? xfs_icsb_modify_counters+0x17f/0x1ac [xfs]
 [<ffffffffa032b60f>] ? xfs_icsb_unlock_all_counters+0x4d/0x60 [xfs]
 [<ffffffffa032b8bf>] ? xfs_icsb_disable_counter+0x8c/0x95 [xfs]
 [<ffffffff81478e88>] ? mutex_lock_nested+0x3e/0x43
 [<ffffffffa032d3d3>] ? xfs_icsb_modify_counters+0x18d/0x1ac [xfs]
 [<ffffffffa032d536>] ? xfs_mod_incore_sb+0x29/0x6e [xfs]
 [<ffffffffa033052c>] ? _xfs_trans_alloc+0x27/0x61 [xfs]
 [<ffffffffa03303d3>] ? xfs_trans_reserve+0x6c/0x19e [xfs]
 [<ffffffff8106fb45>] ? up_write+0x2b/0x32
 [<ffffffffa0335e55>] ? xfs_alloc_file_space+0x163/0x306 [xfs]
 [<ffffffff8107120a>] ? sched_clock_cpu+0xc3/0xce
 [<ffffffffa0336122>] ? xfs_change_file_space+0x12a/0x2b8 [xfs]
 [<ffffffff8106f9bf>] ? down_write_nested+0x80/0x8b
 [<ffffffffa031b8ce>] ? xfs_ilock+0x30/0xb4 [xfs]
 [<ffffffffa033e0e4>] ? xfs_vn_fallocate+0x80/0xf4 [xfs]
--
R         xfs_io 29444  86014624.786617       162   120  86014624.786617   
137655.161327       408.979977 /

# uname -r
2.6.34-0.4.rc0.git2.fc14.x86_64

I'll look into it.

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 10 lines · Eric Sandeen

Quoted 9 lines · Eric Sandeen

Quoted 8 lines · Krzysztof Błaszkowski

Hello,

I use this to preallocate large space but found an issue. Posix_fallocate works right with sizes like 100G, 1T and even 10T on some boxes (on some other can fail after e.g. 7T threshold) but if i tried e.g. 16T the user space process would be "R"unning forever and it is not interruptible. Furthermore some other not related processes like sshd, bash enter D state. There is nothing in kernel log.

Quoted 2 lines

Oh, one thing you should know is that depending on your version of glibc, posix_fallocate may be writing 0s and not using preallocation calls.

I am absolutely sure that recent libc doesn't emulate this syscall

We stick with 2.6.31.5 which seems to be good for us. We do not change kernels easily, as soon as higher revision arrives because it doesn't make sense from stability point of view. We have seen too many times regression bugs so if we are confident with some revision then there is no point to change this.

Quoted 12 lines · Krzysztof Błaszkowski

Quoted 11 lines · Eric Sandeen

Quoted 9 lines · Eric Sandeen

Quoted 8 lines · Krzysztof Błaszkowski

Hello,

I use this to preallocate large space but found an issue. Posix_fallocate works right with sizes like 100G, 1T and even 10T on some boxes (on some other can fail after e.g. 7T threshold) but if i tried e.g. 16T the user space process would be "R"unning forever and it is not interruptible. Furthermore some other not related processes like sshd, bash enter D state. There is nothing in kernel log.

Quoted 5 lines

Quoted 3 lines

Oh, one thing you should know is that depending on your version of glibc, posix_fallocate may be writing 0s and not using preallocation calls.

I am absolutely sure that recent libc doesn't emulate this syscall

right, recent glibc does not (unless the underlying fs doesn't support it)

...

Quoted 5 lines

We stick with 2.6.31.5 which seems to be good for us. We do not change kernels easily, as soon as higher revision arrives because it doesn't make sense from stability point of view. We have seen too many times regression bugs so if we are confident with some revision then there is no point to change this.

It was just a testing suggestion, but I already tested upstream and the problem persists, now just need to find the time to dig into it.

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 1 line · Eric Sandeen

Krzysztof Błaszkowski wrote:

Quoted 7 lines

Quoted 5 lines

We stick with 2.6.31.5 which seems to be good for us. We do not change kernels easily, as soon as higher revision arrives because it doesn't make sense from stability point of view. We have seen too many times regression bugs so if we are confident with some revision then there is no point to change this.

It was just a testing suggestion, but I already tested upstream and the problem persists, now just need to find the time to dig into it.

I see and I am glad you confirmed this. Do you think that fallocate called many times with fixed size and increasing offset will work better than one time call with huge size @ 0 offset ?

Krzysztof

Quoted 1 line

-Eric

Quoted 2 lines · Krzysztof Błaszkowski

Quoted 1 line · Eric Sandeen

Krzysztof Błaszkowski wrote:

...

Quoted 6 lines

Quoted 6 lines

Quoted 5 lines

We stick with 2.6.31.5 which seems to be good for us. We do not change kernels easily, as soon as higher revision arrives because it doesn't make sense from stability point of view. We have seen too many times regression bugs so if we are confident with some revision then there is no point to change this.

Quoted 6 lines

Quoted 2 lines

It was just a testing suggestion, but I already tested upstream and the problem persists, now just need to find the time to dig into it.

I see and I am glad you confirmed this. Do you think that fallocate called many times with fixed size and increasing offset will work better than one time call with huge size @ 0 offset ?

I'd expect that to work; it's certainly worth a test

, and please send your
results back to the list ;)

thanks,
-Eric

Quoted 3 lines

Krzysztof

Quoted 1 line

-Eric

as long as we don't know the exact nature of this fault we can't consider such test as reliable fix.