OOM kills when running fsstress on CIFS
Jeff Layton
Reviewed-by Dave KleikampSince 2.6.34, I've been able to consistently reproduce OOM kills when running fsstress (from the LTP suite) on CIFS. I spent some time yesterday and bisected it down to this patch:
---------------------[snip]---------------------
commit 315e995c63a15cb4d4efdbfd70fe2db191917f7a
Author: Nick Piggin <[email protected]> Date: Wed Apr 21 03:18:28 2010 +0000
[CIFS] use add_to_page_cache_lru
add_to_page_cache_lru is exported, so it should be used. Benefits over using a private pagevec: neater code, 128 bytes fewer stack used, percpu lru ordering is preserved, and finally don't need to flush pagevec before returning so batching may be shared with other LRU insertions.
---------------------[snip]---------------------
Here's how I've been reproducing it:
Mount up a samba share with -o sec=krb5i,nounix,noserverino
Run: fsstress -d /path/to/dir/on/cifs/ -n 1000 -l0 -p8
...within an hour or two, I start getting OOM kills. After backing out the patch above, I was able to run the test overnight. I'm not sure yet what the actual problem is, but there seems to be something wrong with that patch.
Thoughts?
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 23 lines · Jeff Layton
Since 2.6.34, I've been able to consistently reproduce OOM kills when running fsstress (from the LTP suite) on CIFS. I spent some time yesterday and bisected it down to this patch:
---------------------[snip]---------------------
commit315e995c63a15cb4d4efdbfd70fe2db191917f7aAuthor: Nick Piggin <[email protected]> Date: Wed Apr 21 03:18:28 2010 +0000[CIFS] use
add_to_page_cache_lru
add_to_page_cache_lruis exported, so it should be used. Benefits over using a private pagevec: neater code, 128 bytes fewer stack used, percpu lru ordering is preserved, and finally don't need to flush pagevec before returning so batching may be shared with other LRU insertions.Signed-off-by Nick Piggin <[email protected]>Reviewed-by Dave Kleikamp <[email protected]>Signed-off-by Steve French <[email protected]>---------------------[snip]---------------------
Here's how I've been reproducing it:
Mount up a samba share with -o sec=krb5i,nounix,noserverino
Quoted 8 lines
Run: fsstress -d /path/to/dir/on/cifs/ -n 1000 -l0 -p8
...within an hour or two, I start getting OOM kills. After backing out the patch above, I was able to run the test overnight. I'm not sure yet what the actual problem is, but there seems to be something wrong with that patch.
Thoughts?
Yep, it's my fault. The problem is the refcounting. Previously the code hands off the references to the LRU, wheras now the lru takes a new reference. (the other filesystems converted to use this function seemed to more conventionally open-code lru_cache_add).
Can we get rid of a refcount increment anywhere? Otherwise we'll need to just drop the references after adding the pages.
Quoted 33 lines · On Tue, 25 May 2010 21:16:39 +1000 Nick Piggin <[email protected]> wrote:
Quoted 23 lines · Jeff Layton
Since 2.6.34, I've been able to consistently reproduce OOM kills when running fsstress (from the LTP suite) on CIFS. I spent some time yesterday and bisected it down to this patch:
---------------------[snip]---------------------
commit315e995c63a15cb4d4efdbfd70fe2db191917f7aAuthor: Nick Piggin <[email protected]> Date: Wed Apr 21 03:18:28 2010 +0000[CIFS] use
add_to_page_cache_lru
add_to_page_cache_lruis exported, so it should be used. Benefits over using a private pagevec: neater code, 128 bytes fewer stack used, percpu lru ordering is preserved, and finally don't need to flush pagevec before returning so batching may be shared with other LRU insertions.Signed-off-by Nick Piggin <[email protected]>Reviewed-by Dave Kleikamp <[email protected]>Signed-off-by Steve French <[email protected]>---------------------[snip]---------------------
Here's how I've been reproducing it:
Mount up a samba share with -o sec=krb5i,nounix,noserverino
Quoted 8 lines
Run: fsstress -d /path/to/dir/on/cifs/ -n 1000 -l0 -p8
...within an hour or two, I start getting OOM kills. After backing out the patch above, I was able to run the test overnight. I'm not sure yet what the actual problem is, but there seems to be something wrong with that patch.
Thoughts?
Quoted 8 lines
Yep, it's my fault. The problem is the refcounting. Previously the code hands off the references to the LRU, wheras now the lru takes a new reference. (the other filesystems converted to use this function seemed to more conventionally open-code
lru_cache_add).Can we get rid of a refcount increment anywhere? Otherwise we'll need to just drop the references after adding the pages.
The only caller of this function is cifs_readpages, and I don't see where it takes any references. I'm guessing that the pages come from the VFS with the refcount already incremented?
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 41 lines · Jeff Layton
Quoted 38 lines · On Tue, 25 May 2010 21:16:39 +1000 Nick Piggin <[email protected]> wrote:
Quoted 23 lines · Jeff Layton
Since 2.6.34, I've been able to consistently reproduce OOM kills when running fsstress (from the LTP suite) on CIFS. I spent some time yesterday and bisected it down to this patch:
---------------------[snip]---------------------
commit315e995c63a15cb4d4efdbfd70fe2db191917f7aAuthor: Nick Piggin <[email protected]> Date: Wed Apr 21 03:18:28 2010 +0000[CIFS] use
add_to_page_cache_lru
add_to_page_cache_lruis exported, so it should be used. Benefits over using a private pagevec: neater code, 128 bytes fewer stack used, percpu lru ordering is preserved, and finally don't need to flush pagevec before returning so batching may be shared with other LRU insertions.Signed-off-by Nick Piggin <[email protected]>Reviewed-by Dave Kleikamp <[email protected]>Signed-off-by Steve French <[email protected]>---------------------[snip]---------------------
Here's how I've been reproducing it:
Mount up a samba share with -o sec=krb5i,nounix,noserverino
Quoted 8 lines
Run: fsstress -d /path/to/dir/on/cifs/ -n 1000 -l0 -p8
...within an hour or two, I start getting OOM kills. After backing out the patch above, I was able to run the test overnight. I'm not sure yet what the actual problem is, but there seems to be something wrong with that patch.
Thoughts?
Yep, it's my fault. The problem is the refcounting. Previously the code hands off the references to the LRU, wheras now the lru takes a new reference. (the other filesystems converted to use this function seemed to more conventionally open-code
lru_cache_add).
Quoted 7 lines
Quoted 3 lines
Can we get rid of a refcount increment anywhere? Otherwise we'll need to just drop the references after adding the pages.
The only caller of this function is
cifs_readpages, and I don't see where it takes any references. I'm guessing that the pages come from the VFS with the refcount already incremented?
Yep. I think we should just page_cache_release after doing the add_to_page_cache_lru, like the generic code does.
It's a little suboptimal tinkering with refcounts like that, but a cleanup pass to fix it up all over the tree and allow add_to_page_cache_lru to take over a reference would be better place to fix that.
Quoted 50 lines · On Tue, 25 May 2010 21:54:31 +1000 Nick Piggin <[email protected]> wrote:
Quoted 41 lines · Jeff Layton
Quoted 38 lines · On Tue, 25 May 2010 21:16:39 +1000 Nick Piggin <[email protected]> wrote:
Quoted 23 lines · Jeff Layton
Since 2.6.34, I've been able to consistently reproduce OOM kills when running fsstress (from the LTP suite) on CIFS. I spent some time yesterday and bisected it down to this patch:
---------------------[snip]---------------------
commit315e995c63a15cb4d4efdbfd70fe2db191917f7aAuthor: Nick Piggin <[email protected]> Date: Wed Apr 21 03:18:28 2010 +0000[CIFS] use
add_to_page_cache_lru
add_to_page_cache_lruis exported, so it should be used. Benefits over using a private pagevec: neater code, 128 bytes fewer stack used, percpu lru ordering is preserved, and finally don't need to flush pagevec before returning so batching may be shared with other LRU insertions.Signed-off-by Nick Piggin <[email protected]>Reviewed-by Dave Kleikamp <[email protected]>Signed-off-by Steve French <[email protected]>---------------------[snip]---------------------
Here's how I've been reproducing it:
Mount up a samba share with -o sec=krb5i,nounix,noserverino
Quoted 8 lines
Run: fsstress -d /path/to/dir/on/cifs/ -n 1000 -l0 -p8
...within an hour or two, I start getting OOM kills. After backing out the patch above, I was able to run the test overnight. I'm not sure yet what the actual problem is, but there seems to be something wrong with that patch.
Thoughts?
Yep, it's my fault. The problem is the refcounting. Previously the code hands off the references to the LRU, wheras now the lru takes a new reference. (the other filesystems converted to use this function seemed to more conventionally open-code
lru_cache_add).Quoted 7 lines
Quoted 3 lines
Can we get rid of a refcount increment anywhere? Otherwise we'll need to just drop the references after adding the pages.
The only caller of this function is
cifs_readpages, and I don't see where it takes any references. I'm guessing that the pages come from the VFS with the refcount already incremented?
Quoted 8 lines
Yep. I think we should just
page_cache_releaseafter doing theadd_to_page_cache_lru, like the generic code does.It's a little suboptimal tinkering with refcounts like that, but a cleanup pass to fix it up all over the tree and allow
add_to_page_cache_lruto take over a reference would be better place to fix that.
Ok, I tested out the patch that I just sent and it seems to have fixed the problem. Look ok?
Signature · 1 line
Jeff Layton <[email protected]>