possible fsync02() xfs slowness regression on power7
This LTP test starting to fail using the latest upstream kernel on one of the power7 systems here, http://tinyurl.com/bngwouj
# ./fsync02 fsync02 1 TFAIL : fsync took too long: 252.000000 seconds; max_block: 214
When it is working, the test is almost returned immediately. The bisecting so far indicated that one or a few of the following could be culprits.
# git log --pretty=oneline 498f7f505dc79934c878c7667840c50c64f232fc..b199c8a4ba11879df87daad496ceee41fdc6aa82 b199c8a4ba11879df87daad496ceee41fdc6aa82 xfs: Pull EFI/EFD handling out from under the AIL lock 9c5f8414efd5eeed9f498d4170337a3eb126341f xfs: fix EFI transaction cancellation. 821eb21d97a8b686649c08b7284d0b9f34d0e138 xfs: connect up buffer reclaim priority hooks 430cbeb86fdcbbdabea7d4aa65307de8de425350 xfs: add a lru to the XFS buffer cache ff57ab21995a8636cfc72efeebb09cc6034d756f xfs: convert xfsbud shrinker to a per-buftarg shrinker. 1a427ab0c1b205d1bda8da0b77ea9d295ac23c57 xfs: convert pag_ici_lock to a spin lock 1a3e8f3da09c7082d25b512a0ffe569391e4c09a xfs: convert inode cache lookups to use RCU locking d95b7aaf9ab6738bef1ebcc52ab66563085e44ac xfs: rcu free inodes 6e857567dbbfe14dd6cc3f7414671b047b1ff5c7 xfs: don't truncate prealloc from frequently accessed inodes 055388a3188f56676c21e92962fc366ac8b5cb72 xfs: dynamic speculative EOF preallocation 622d81494fa32343a4b97b607619656c7a4a6d1a xfs: use KM_NOFS for allocations during attribute list operations dcfcf20512cb517ac18b9433b676183fa1257911 xfs: provide a inode iolock lockdep class 489a150f6454e2cd93d9e0ee6d7c5a361844f62a xfs: factor duplicate code in xfs_alloc_ag_vextent_near into a helper 9f9baab38dacd11fe6095a1e59f3783a305f7020 xfs: clean up xfs_alloc_ag_vextent_exact ecff71e677c6d469f525dcf31ada709d5858307c xfs: simplify xfs_map_at_offset aeea1b1f81800e362a3aca86d769d02e137a8fa7 xfs: refactor xfs_vm_writepage 2fa24f92530edaf86c3b5f662464e0d2e3b3e517 xfs: remove the all_bh flag from xfs_convert_page ed1e7b7e484dfb64168755613d499f32a97409bd xfs: remove xfs_probe_cluster 8ff2957d581582890693affc09920108a67cb05d xfs: simplify xfs_map_blocks a206c817c864583c44e2f418db8e6c7a000fbc38 xfs: kill xfs_iomap
CAI Qian
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · CAI Qian
This LTP test starting to fail using the latest upstream kernel on one of the power7 systems here, http://tinyurl.com/bngwouj
# ./fsync02 fsync02 1 TFAIL : fsync took too long: 252.000000 seconds; max_block: 214When it is working, the test is almost returned immediately. The bisecting so far indicated that one or a few of the following could be culprits.
# git log --pretty=oneline 498f7f505dc79934c878c7667840c50c64f232fc..b199c8a4ba11879df87daad496ceee41fdc6aa82
They are all patches committed more than 2 years ago, and none of them are platform specific. This sounds more like a machine specific issue than a platform specific problem (i.e. lots of RAM, slow, slow disk).
In future when reporting a bug, please tell use hardware you are using as per:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
Given that this is a sparse file test that repeatedly extends the file, this is the likely culprit:
Quoted 1 line
055388a3188f56676c21e92962fc366ac8b5cb72xfs: dynamic speculative EOF preallocation
And this commit in 3.9-rc1:
a1e16c2 xfs: limit speculative prealloc size on sparse files
should fix the problem. Please confirm these commits are the cause and the fix respectively....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
----- 原始邮件 -----
Quoted 27 lines
发件人: "Dave Chinner" <[email protected]>
收件人: "CAI Qian" <[email protected]>
抄送: [email protected], "Steve Best" <[email protected]> 发送时间: 星期四, 2013年 2 月 28日 下午 7:54:10
主题: Re: possiblefsync02()xfs slowness regression on power7Quoted 8 lines · CAI Qian
This LTP test starting to fail using the latest upstream kernel on one of the power7
systems here, http://tinyurl.com/bngwouj# ./fsync02 fsync02 1 TFAIL : fsync took too long: 252.000000 seconds; max_block: 214Quoted 6 lines
When it is working, the test is almost returned immediately. The bisecting so far
indicated that one or a few of the following could be culprits.# git log --pretty=oneline 498f7f505dc79934c878c7667840c50c64f232fc..b199c8a4ba11879df87daad496ceee41fdc6aa82They are all patches committed more than 2 years ago, and none of them are platform specific. This sounds more like a machine specific issue than a platform specific problem (i.e. lots of RAM, slow, slow disk).
Quoted 10 lines
In future when reporting a bug, please tell use hardware you are using as per:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
Given that this is a sparse file test that repeatedly extends the file, this is the likely culprit:
Quoted 2 lines
055388a3188f56676c21e92962fc366ac8b5cb72xfs: dynamic speculative EOF preallocation
Yes, you are right, just confirmed it. Nice catch!
Quoted 4 lines
And this commit in 3.9-rc1:
a1e16c2 xfs: limit speculative prealloc size on sparse files
Just back-ported it and running at the moment and will let you know. Thanks Dave.
Quoted 11 lines
should fix the problem. Please confirm these commits are the cause and the fix respectively....
Cheers,
Dave.
--
Dave Chinner
[email protected]
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines
Quoted 6 lines
And this commit in 3.9-rc1:
a1e16c2 xfs: limit speculative prealloc size on sparse files should fix the problem. Please confirm these commits are the cause and the fix respectively....
Confirmed this fixed the problem. I'd like to request this to be back-ported to stable-3.0, stable-3.4 and stable-3.8. What do you think?
Thanks,
CAI Qian
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · CAI Qian
Quoted 6 lines
Quoted 6 lines
And this commit in 3.9-rc1:
a1e16c2 xfs: limit speculative prealloc size on sparse files should fix the problem. Please confirm these commits are the cause and the fix respectively....
Confirmed this fixed the problem. I'd like to request this to be back-ported to stable-3.0, stable-3.4 and stable-3.8. What do you think?
IMO, no, it is not a backport candidate. The patch has quite a few dependencies, and at least for 3.0 xfs_bmapi_read() doesn't exist and hence is not a trivial backport.
Further, it's take 2 years for this to be noticed, and you haven't explained why the problem exists on your power machine and not any others that it has been tested on. And there's been very few complaints about performance of such workloads over the past 2 years, so either the workload is not important or only your power7 machine is having problems.
Hence I don't see any need to back port it - it's not a critical fix and very few people see the problem so there's no real need to do the backport. Maybe someone else has the time and resources to waste on backporting non-critical fixes to stable kernels, but I don't....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
----- 原始邮件 -----
Quoted 14 lines
发件人: "Dave Chinner" <[email protected]>
收件人: "CAI Qian" <[email protected]>
抄送: [email protected]
发送时间: 星期一, 2013年 3 月 04日 下午 1:55:02
主题: Re: possiblefsync02()xfs slowness regression on power7Quoted 7 lines · CAI Qian
Quoted 7 lines
Quoted 7 lines
And this commit in 3.9-rc1:
a1e16c2 xfs: limit speculative prealloc size on sparse files should fix the problem. Please confirm these commits are the cause
and the fix respectively....
Quoted 7 lines
Quoted 3 lines
Confirmed this fixed the problem. I'd like to request this to be back-ported
to stable-3.0, stable-3.4 and stable-3.8. What do you think?IMO, no, it is not a backport candidate. The patch has quite a few dependencies, and at least for 3.0
xfs_bmapi_read()doesn't exist and hence is not a trivial backport.
It is fine to skip the 3.0 then, but the other stable branches can be applied as it iis and build fine.
Quoted 7 lines
Further, it's take 2 years for this to be noticed, and you haven't explained why the problem exists on your power machine and not any others that it has been tested on. And there's been very few complaints about performance of such workloads over the past 2 years, so either the workload is not important or only your power7 machine is having problems.
Hmm, it could also be possible that it was easy to reproduce now with the new kernel plus new user-spaces as well the new compiler. Also, those systems started to switch to use XFS as root partitions from ext4 only recently, so we have never noticed this in the past when running those LTP tests. XFS could also become more popular than 2-year ago. :)
Quoted 6 lines
Hence I don't see any need to back port it - it's not a critical fix and very few people see the problem so there's no real need to do the backport. Maybe someone else has the time and resources to waste on backporting non-critical fixes to stable kernels, but I don't....
I have time so I can do the back-port for you guys to review if you don't mind. Thanks for your time.
Quoted 8 lines
Cheers,
Dave.
--
Dave Chinner
[email protected]
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs