xfs: fix the wrong new_size/rnew_size at xfs_iext_realloc_direct()
PATCH 1/1 xfs: fix the wrong new_size/rnew_size at xfs_iext_realloc_direct()
From: Jie Liu <[email protected]>
At xfs_iext_realloc_direct(), the new_size is changed by adding if_bytes if originally the extent records are stored at the inline extent buffer, and we have to switch from it to a direct extent list for those new allocated extents, this is wrong. e.g,
Create a file with three extents which was showing as following,
xfs_io -f -c "truncate 100m" /xfs/testme
for i in $(seq 0 5 10); do
offset=$(($i * $((1 << 20)))) xfs_io -c "pwrite $offset 1m" /xfs/testme
done
Inline
------
irec: if_bytes bytes_diff new_size
1st 0 16 16
2nd 16 16 32
Switching
--------- rnew_size
3rd 32 16 48 + 32 = 80 roundup=128
In this case, the desired value of new_size should be 48, and then it will be roundup to 64 and be assigned to rnew_size.
However, this issue has been covered by resetting the if_bytes to the new_size which is calculated at the begnning of xfs_iext_add() before leaving out this function, and in turn make the rnew_size correctly again. Hence, this can not be detected via xfstestes.
This patch fix above problem and revise the new_size comments at xfs_iext_realloc_direct() to make it more readable. Also, fix the comments while switching from the inline extent buffer to a direct extent list to reflect this change.
--- fs/xfs/xfs_inode_fork.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
fs/xfs/xfs_inode_fork.c +2 -7
@@ -1359,7 +1359,7 @@ xfs_iext_remove_indirect(void xfs_iext_realloc_direct (xfs_ifork_t *ifp,/* inode fork pointer */ -int new_size)/* new size of extents */ +int new_size)/* new size of extents after adding */ {int rnew_size;/* real new size of extents */
@@ -1397,13 +1397,8 @@ xfs_iext_realloc_direct( rnew_size - ifp->if_real_bytes); } } -/* -* Switch from the inline extent buffer to a direct -* extent list. Be sure to include the inline extent -* bytes in new_size. -*/ +/* Switch from the inline extent buffer to a direct extent list */ else { - new_size += ifp->if_bytes;if (!is_power_of_2 (new_size)) { rnew_size =roundup_pow_of_two (new_size); }
Signature · 6 lines
1.7.9.5 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 22 lines · Jeff Liu
From: Jie Liu <[email protected]>
At
xfs_iext_realloc_direct(), thenew_sizeis changed by addingif_bytesif originally the extent records are stored at the inline extent buffer, and we have to switch from it to a direct extent list for those new allocated extents, this is wrong. e.g,Create a file with three extents which was showing as following,
xfs_io-f -c "truncate 100m" /xfs/testmefor i in $(seq 0 5 10); do
offset=$(($i * $((1 << 20)))) xfs_io -c "pwrite $offset 1m" /xfs/testmedone
Inline
------
irec:if_bytesbytes_diffnew_size
1st 0 16 16
2nd 16 16 32
Quoted 6 lines
Switching
---------rnew_size
3rd 32 16 48 + 32 = 80 roundup=128In this case, the desired value of
new_sizeshould be 48, and then it will be roundup to 64 and be assigned tornew_size.
Ok, so it allocates 128 bytes instead of 64 bytes. It tracks that allocation size correctly ifp->if_real_bytes, and all it means is that there are 4 empty extra slots in the extent array. The code already handles having empty slots in the direct extent array, so what impact is there as a result of the oversized initial allocation that is currently happening?
i.e. if fixing the oversized results in more memory allocations due to resizing more regularly, then is there a benefit to changing this code given that the rewrite of the ifp->if_bytes value in the case where we do inline->direct conversion prevents this over-allocation from being a problem...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Hi Dave,
Quoted 37 lines · Dave Chinner
Quoted 22 lines · Jeff Liu
From: Jie Liu <[email protected]>
At
xfs_iext_realloc_direct(), thenew_sizeis changed by addingif_bytesif originally the extent records are stored at the inline extent buffer, and we have to switch from it to a direct extent list for those new allocated extents, this is wrong. e.g,Create a file with three extents which was showing as following,
xfs_io-f -c "truncate 100m" /xfs/testmefor i in $(seq 0 5 10); do
offset=$(($i * $((1 << 20)))) xfs_io -c "pwrite $offset 1m" /xfs/testmedone
Inline
------
irec:if_bytesbytes_diffnew_size
1st 0 16 16
2nd 16 16 32Quoted 6 lines
Switching
---------rnew_size
3rd 32 16 48 + 32 = 80 roundup=128In this case, the desired value of
new_sizeshould be 48, and then it will be roundup to 64 and be assigned tornew_size.Ok, so it allocates 128 bytes instead of 64 bytes. It tracks that allocation size correctly
ifp->if_real_bytes, and all it means is that there are 4 empty extra slots in the extent array. The code already handles having empty slots in the direct extent array, so what impact is there as a result of the oversized initial allocation that is currently happening?
Quoted 5 lines
i.e. if fixing the oversized results in more memory allocations due to resizing more regularly, then is there a benefit to changing this code given that the rewrite of the
ifp->if_bytesvalue in the case where we doinline->directconversion prevents this over-allocation from being a problem...
I guess my current patch subject/description mislead you. The result of the oversized can be ignored since this can be handled in the direct extent array as empty slots.
Actually, what I want to say is that we don't need to perform "new_size += ifp->if_bytes;" again at xfs_iext_realloc_direct() because the new_size at xfs_iext_add() already be the size of extents after adding, just as the variable comments is mentioned.
Thanks,
-Jeff
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 47 lines · Jeff Liu
Hi Dave,
Quoted 37 lines · Dave Chinner
Quoted 22 lines · Jeff Liu
From: Jie Liu <[email protected]>
At
xfs_iext_realloc_direct(), thenew_sizeis changed by addingif_bytesif originally the extent records are stored at the inline extent buffer, and we have to switch from it to a direct extent list for those new allocated extents, this is wrong. e.g,Create a file with three extents which was showing as following,
xfs_io-f -c "truncate 100m" /xfs/testmefor i in $(seq 0 5 10); do
offset=$(($i * $((1 << 20)))) xfs_io -c "pwrite $offset 1m" /xfs/testmedone
Inline
------
irec:if_bytesbytes_diffnew_size
1st 0 16 16
2nd 16 16 32Quoted 6 lines
Switching
---------rnew_size
3rd 32 16 48 + 32 = 80 roundup=128In this case, the desired value of
new_sizeshould be 48, and then it will be roundup to 64 and be assigned tornew_size.Ok, so it allocates 128 bytes instead of 64 bytes. It tracks that allocation size correctly
ifp->if_real_bytes, and all it means is that there are 4 empty extra slots in the extent array. The code already handles having empty slots in the direct extent array, so what impact is there as a result of the oversized initial allocation that is currently happening?Quoted 5 lines
i.e. if fixing the oversized results in more memory allocations due to resizing more regularly, then is there a benefit to changing this code given that the rewrite of the
ifp->if_bytesvalue in the case where we doinline->directconversion prevents this over-allocation from being a problem...
Quoted 3 lines
I guess my current patch subject/description mislead you. The result of the oversized can be ignored since this can be handled in the direct extent array as empty slots.
That's what I thought ;)
Quoted 4 lines
Actually, what I want to say is that we don't need to perform "
new_size+=ifp->if_bytes;" again atxfs_iext_realloc_direct()because thenew_sizeatxfs_iext_add()already be the size of extents after adding, just as the variable comments is mentioned.
Yes, I understand.
What I'm really asking is that whether there is any specific impact you can measure as a result of changing the initial allocation size? i.e. are there workloads where there is a measurable difference in memory footprint or noticable performance impact of having to reallocate the direct array more frequently as files grow and/or shrink?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 59 lines · Dave Chinner
Quoted 47 lines · Jeff Liu
Hi Dave,
Quoted 37 lines · Dave Chinner
Quoted 22 lines · Jeff Liu
From: Jie Liu <[email protected]>
At
xfs_iext_realloc_direct(), thenew_sizeis changed by addingif_bytesif originally the extent records are stored at the inline extent buffer, and we have to switch from it to a direct extent list for those new allocated extents, this is wrong. e.g,Create a file with three extents which was showing as following,
xfs_io-f -c "truncate 100m" /xfs/testmefor i in $(seq 0 5 10); do
offset=$(($i * $((1 << 20)))) xfs_io -c "pwrite $offset 1m" /xfs/testmedone
Inline
------
irec:if_bytesbytes_diffnew_size
1st 0 16 16
2nd 16 16 32Quoted 6 lines
Switching
---------rnew_size
3rd 32 16 48 + 32 = 80 roundup=128In this case, the desired value of
new_sizeshould be 48, and then it will be roundup to 64 and be assigned tornew_size.Ok, so it allocates 128 bytes instead of 64 bytes. It tracks that allocation size correctly
ifp->if_real_bytes, and all it means is that there are 4 empty extra slots in the extent array. The code already handles having empty slots in the direct extent array, so what impact is there as a result of the oversized initial allocation that is currently happening?Quoted 5 lines
i.e. if fixing the oversized results in more memory allocations due to resizing more regularly, then is there a benefit to changing this code given that the rewrite of the
ifp->if_bytesvalue in the case where we doinline->directconversion prevents this over-allocation from being a problem...Quoted 3 lines
I guess my current patch subject/description mislead you. The result of the oversized can be ignored since this can be handled in the direct extent array as empty slots.
That's what I thought ;)
Quoted 4 lines
Actually, what I want to say is that we don't need to perform "
new_size+=ifp->if_bytes;" again atxfs_iext_realloc_direct()because thenew_sizeatxfs_iext_add()already be the size of extents after adding, just as the variable comments is mentioned.
Quoted 8 lines
Yes, I understand.
What I'm really asking is that whether there is any specific impact you can measure as a result of changing the initial allocation size? i.e. are there workloads where there is a measurable difference in memory footprint or noticable performance impact of having to reallocate the direct array more frequently as files grow and/or shrink?
Not yet observed any performance matter, but IMO this problem can cause difference in dynamic memory footprint for creating a large number of files with 3 extents and with additional kmalloc/kfree overhead for 4 extents file.
For the first case, the current code will allocate buffers from kmalloc-128 slab cache rather than kmalloc-64, hence it would occupy more memory until being dropped from the cache, e.g,
# Create 10240 files with 3 extents
for ((i=0; i<10240; i++))
do
xfs_io -f -c 'truncate 10m' /xfs/test_$i xfs_io -c 'pwrite 0 1' /xfs/test_$i 2>&1 >>/dev/null xfs_io -c 'pwrite 1m 1' /xfs/test_$i 2>&1 >>/dev/null xfs_io -c 'pwrite 5m 1' /xfs/test_$i 2>&1 >>/dev/null
done
# cat /proc/slab_info # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab>...
# Non-patched -- before creating files
kmalloc-128 5391 6176 128 32 1
kmalloc-64 21852 25152 64 64 1
# After that -- the number of objects in 128 slab increased significantly, while there basically no change in 64 slab
kmalloc-128 15381 15488 128 32 1
kmalloc-64 21958 25088 64 64 1
# patched -- before creating files kmalloc-128 5751 7072 128 32 1 kmalloc-64 21420 24896 64 64 1
After after
kmalloc-128 6155 6688 128 32 1
kmalloc-64 30464 30464 64 64 1
With this patch, we can reduce the memory footprint for this particular scenario.
For the 2nd case, i.e, 4 extents file. It need to resize the direct extent list to add the fourth extent because rnew_bytes is be re-initialized to 64 at the beginning of xfs_iext_realloc_direct(), however the ifp->if_real_bytes is 128...
I can not think out a convenient approach(perf kmem not works on working laptop for now) to demonstrate the consequence, but using ftrace to figure out the different number of kmalloc. e.g,
# Creating 4096 files with 4 extents and fetch the # of kmalloc. echo 0 > /sys/kernel/debug/tracing/events/kmem/kmalloc/enable echo > /sys/kernel/debug/tracing/trace
for ((i=0; i<4096; i++))
do
xfs_io -f -c 'truncate 10m' /xfs/test_$i xfs_io -c 'pwrite 0 1' /xfs/test_$i 2>&1 >>/dev/null xfs_io -c 'pwrite 1m 1' /xfs/test_$i 2>&1 >>/dev/null xfs_io -c 'pwrite 5m 1' /xfs/test_$i 2>&1 >>/dev/null
done
echo 1 > /sys/kernel/debug/tracing/events/kmem/kmalloc/enable for ((i=0; i<4096; i++))
do
xfs_io -c 'pwrite 8m 1' /xfs/test_$i 2>&1 >>/dev/null
done
cat /sys/kernel/debug/tracing/trace|grep kmalloc|wc -l
# The number of kmalloc calls
Default Patched
110364 103471
Thanks,
-Jeff
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Dave Chinner
Reviewed-byQuoted 14 lines · Jeff Liu
Quoted 12 lines · Dave Chinner
Quoted 11 lines · Jeff Liu
Hi Dave,
Quoted 7 lines · Dave Chinner
Quoted 6 lines · Jeff Liu
From: Jie Liu <[email protected]>
At
xfs_iext_realloc_direct(), thenew_sizeis changed by addingif_bytesif originally the extent records are stored at the inline extent buffer, and we have to switch from it to a direct extent list for those new allocated extents, this is wrong. e.g,
....
Quoted 49 lines
Quoted 5 lines
Quoted 4 lines
Actually, what I want to say is that we don't need to perform "
new_size+=ifp->if_bytes;" again atxfs_iext_realloc_direct()because thenew_sizeatxfs_iext_add()already be the size of extents after adding, just as the variable comments is mentioned.Quoted 8 lines
Yes, I understand.
What I'm really asking is that whether there is any specific impact you can measure as a result of changing the initial allocation size? i.e. are there workloads where there is a measurable difference in memory footprint or noticable performance impact of having to reallocate the direct array more frequently as files grow and/or shrink?
Not yet observed any performance matter, but IMO this problem can cause difference in dynamic memory footprint for creating a large number of files with 3 extents and with additional kmalloc/kfree overhead for 4 extents file.
For the first case, the current code will allocate buffers from kmalloc-128 slab cache rather than kmalloc-64, hence it would occupy more memory until being dropped from the cache, e.g,
# Create 10240 files with 3 extents
for ((i=0; i<10240; i++))
doxfs_io -f -c 'truncate 10m' /xfs/test_$i xfs_io -c 'pwrite 0 1' /xfs/test_$i 2>&1 >>/dev/null xfs_io -c 'pwrite 1m 1' /xfs/test_$i 2>&1 >>/dev/null xfs_io -c 'pwrite 5m 1' /xfs/test_$i 2>&1 >>/dev/nulldone
# cat /proc/slab_info # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab>...# Non-patched -- before creating files
kmalloc-128 5391 6176 128 32 1
kmalloc-64 21852 25152 64 64 1# After that -- the number of objects in 128 slab increased significantly, while there basically no change in 64 slab
kmalloc-128 15381 15488 128 32 1
kmalloc-64 21958 25088 64 64 1# patched -- before creating files kmalloc-128 5751 7072 128 32 1 kmalloc-64 21420 24896 64 64 1
Quoted 5 lines
After after
kmalloc-128 6155 6688 128 32 1
kmalloc-64 30464 30464 64 64 1With this patch, we can reduce the memory footprint for this particular scenario.
Ok, so it's used the kmalloc-64 slab much more effectively and not touched the kmalloc-128 slab. Ok, so thats a measurable difference ;)
Quoted 4 lines
For the 2nd case, i.e, 4 extents file. It need to resize the direct extent list to add the fourth extent because
rnew_bytesis be re-initialized to 64 at the beginning ofxfs_iext_realloc_direct(), however theifp->if_real_bytesis 128...
...
Quoted 3 lines
# The number of kmalloc calls
Default Patched
110364 103471
And that demonstrates the impact in that the array is downsized as the array grows. Ok, I'm convinced there is a net win here :)
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Ben Myers
AppliedQuoted 34 lines · Jeff Liu
From: Jie Liu <[email protected]>
At
xfs_iext_realloc_direct(), thenew_sizeis changed by addingif_bytesif originally the extent records are stored at the inline extent buffer, and we have to switch from it to a direct extent list for those new allocated extents, this is wrong. e.g,Create a file with three extents which was showing as following,
xfs_io-f -c "truncate 100m" /xfs/testmefor i in $(seq 0 5 10); do
offset=$(($i * $((1 << 20)))) xfs_io -c "pwrite $offset 1m" /xfs/testmedone
Inline
------
irec:if_bytesbytes_diffnew_size
1st 0 16 16
2nd 16 16 32Switching
---------rnew_size
3rd 32 16 48 + 32 = 80 roundup=128In this case, the desired value of
new_sizeshould be 48, and then it will be roundup to 64 and be assigned tornew_size.However, this issue has been covered by resetting the
if_bytesto thenew_sizewhich is calculated at the begnning ofxfs_iext_add()before leaving out this function, and in turn make thernew_sizecorrectly again. Hence, this can not be detected via xfstestes.
Quoted 6 lines
This patch fix above problem and revise the
new_sizecomments atxfs_iext_realloc_direct()to make it more readable. Also, fix the comments while switching from the inline extent buffer to a direct extent list to reflect this change.Signed-off-by Jie Liu <[email protected]>
Applied.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs