Ceph fscache: Fix kernel panic due to a race
Subsystems (2)
cover Ceph fscache: Fix kernel panic due to a race
From: Yunchuan Wen <[email protected]>
The following scripts could easily panic the kernel,
#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 > /proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024
This is due to when writing a page into fscache, the code will assert that the write position does not exceed the object->store_limit_l, which is supposed to be equal to inode->i_size. However, for current implementation, after file writing, the object->store_limit_l is not synchronized with new inode->i_size immediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded the object->store_limit_l, and cause kernel panic. This patch fixes it.
Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initialization
fs/ceph/cache.c | 1 + fs/ceph/cache.h | 10 ++++++++++ fs/ceph/file.c | 3 +++ 3 files changed, 14 insertions(+)
Signature · 1 line
1.7.9.5
PATCH 1/3 Ceph fscache: Add an interface to synchronize object store limit
From: Yunchuan Wen <[email protected]>
Add an interface to explicitly synchronize object->store_limit[_l] with inode->i_size
--- fs/ceph/cache.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
fs/ceph/cache.h +10 -0
@@ -48,6 +48,12 @@ void ceph_readpage_to_fscache(struct inode *inode, struct page *page);void ceph_invalidate_fscache_page (struct inode * inode,struct page *page);void ceph_queue_revalidate (struct inode *inode); +static inline void ceph_fscache_update_objectsize (struct inode *inode) +{ +struct ceph_inode_info *ci =ceph_inode (inode); +fscache_attr_changed (ci->fscache); +} +static inline void ceph_fscache_invalidate (struct inode *inode) {fscache_invalidate (ceph_inode (inode)->fscache);
@@ -127,6 +133,10 @@ static inline void ceph_readpage_to_fscache(struct inode *inode, { } +static inline void ceph_fscache_update_objectsize (struct inode *inode) +{ +} +static inline void ceph_fscache_invalidate (struct inode *inode) { }
Signature · 1 line
1.7.9.5
PATCH 2/3 Ceph fscache: Update object store limit after file writing
In reply to Li WangFrom: Yunchuan Wen <[email protected]>
Synchronize object->store_limit[_l] with new inode->i_size after file writing.
--- fs/ceph/file.c | 3 +++ 1 file changed, 3 insertions(+)
fs/ceph/file.c +3 -0
@@ -786,6 +786,7 @@ retry_snap: goto retry_snap; } }else { +loff_t old_size = inode->i_size;/* * No need to acquire the i_truncate_mutex. Because * the MDS revokes Fwb caps before sending truncate
@@ -796,6 +797,8 @@ retry_snap: written =generic_file_buffered_write (iocb, iov, nr_segs, pos, &iocb->ki_pos, count,0 ); +if (inode->i_size > old_size) +ceph_fscache_update_objectsize (inode);mutex_unlock (&inode->i_mutex); }
Signature · 1 line
1.7.9.5
PATCH 3/3 Ceph fscache: Wait for completion of object initialization
In reply to Li WangFrom: Yunchuan Wen <[email protected]>
The object store limit needs to be updated after writing, and this can be done provided the corresponding object has already been initialized. Current object initialization is done asynchrously, which introduce a race if a file is opened, then immediately followed by a writing, the initialization may have not completed, the code will reach the ASSERT in fscache_submit_exclusive_op() to cause kernel bug.
--- fs/ceph/cache.c | 1 + 1 file changed, 1 insertion(+)
fs/ceph/cache.c +1 -0
@@ -205,6 +205,7 @@ void ceph_fscache_register_inode_cookie(struct ceph_fs_client* fsc, ci->fscache =fscache_acquire_cookie (fsc->fscache, &ceph_fscache_inode_object_def, ci, true); +fscache_check_consistency (ci->fscache); done:mutex_unlock (&inode->i_mutex);
Signature · 1 line
1.7.9.5
Li,
I looked at the patchset am I correct that this only happens when we enable caching in the write path?
- Milosz
Quoted 34 lines · Li Wang
From: Yunchuan Wen <[email protected]>
The following scripts could easily panic the kernel,
#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 >/proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024This is due to when writing a page into fscache, the code will assert that the write position does not exceed the
object->store_limit_l, which is supposed to be equal toinode->i_size. However, for current implementation, after file writing, theobject->store_limit_lis not synchronized with newinode->i_sizeimmediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded theobject->store_limit_l, and cause kernel panic. This patch fixes it.Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initializationfs/ceph/cache.c | 1 + fs/ceph/cache.h | 10 ++++++++++ fs/ceph/file.c | 3 +++ 3 files changed, 14 insertions(+)--
1.7.9.5
Signature · 7 lines
Milosz Tanski CTO 10 East 53rd Street, 37th floor New York, NY 10022 p: 646-253-9055 e: [email protected]
Hi, Milosz
I am not very sure about how to enable caching in write path.
I just compile the latest kernel with "Enable Ceph client caching support", start cachefilesd, and mount cephfs with -o fsc. Then, the kernel will panic easily when the script runs to "dd if=cephfs/foo of=/dev/null bs=8 count=1024"
2013/12/27 Milosz Tanski <[email protected]>:
Quoted 57 lines
Li,
I looked at the patchset am I correct that this only happens when we enable caching in the write path?
- Milosz
Quoted 26 lines · Li Wang
From: Yunchuan Wen <[email protected]>
The following scripts could easily panic the kernel,
#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 >/proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024This is due to when writing a page into fscache, the code will assert that the write position does not exceed the
object->store_limit_l, which is supposed to be equal toinode->i_size. However, for current implementation, after file writing, theobject->store_limit_lis not synchronized with newinode->i_sizeimmediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded theobject->store_limit_l, and cause kernel panic. This patch fixes it.Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initializationQuoted 8 lines
fs/ceph/cache.c | 1 + fs/ceph/cache.h | 10 ++++++++++ fs/ceph/file.c | 3 +++ 3 files changed, 14 insertions(+)--
1.7.9.5--
Milosz Tanski
CTO
10 East 53rd Street, 37th floor
New York, NY 10022p: 646-253-9055
e: [email protected]
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Milosz,
As far as I know, logically, currently fscache does not play
as write cache for Ceph, except that there is a
call to ceph_readpage_to_fscache() in ceph_writepage(), but that is nothing related to our test case. According to our observation, our test case never goes through ceph_writepage(), instead, it goes through ceph_writepages(). So in other words, I donot think this is related to caching in write path.
May I try to explain the panic in more detail,
(1) dd if=/dev/zero of=cephfs/foo bs=8 count=512
(2) echo 3 > /proc/sys/vm/drop_caches
(3) dd if=cephfs/foo of=/dev/null bs=8 count=1024
For statement (1), it is frequently appending a file, soceph_aio_write() frequently updates the inode->i_size,
however, these updates did not immediately reflected toobject->store_limit_l. For statement (3), when we
start reading the second page at [4096, 8192), ceph find that the page
does not be cached in fscache, then it decides to write this page into
fscache, during this process in cachefiles_write_page(), it found that object->store_limit_l < 4096 (page->index << 12), it causes panic. Does it make sense?
Cheers,
Li Wang
Quoted 45 lines · Milosz Tanski
Li,
I looked at the patchset am I correct that this only happens when we enable caching in the write path?
- Milosz
Quoted 34 lines · Li Wang
From: Yunchuan Wen <[email protected]>
The following scripts could easily panic the kernel,
#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 >/proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024This is due to when writing a page into fscache, the code will assert that the write position does not exceed the
object->store_limit_l, which is supposed to be equal toinode->i_size. However, for current implementation, after file writing, theobject->store_limit_lis not synchronized with newinode->i_sizeimmediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded theobject->store_limit_l, and cause kernel panic. This patch fixes it.Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initializationfs/ceph/cache.c | 1 + fs/ceph/cache.h | 10 ++++++++++ fs/ceph/file.c | 3 +++ 3 files changed, 14 insertions(+)--
1.7.9.5
I'm going to look the patches and the issue in full detail. In the meantime do you guys have the oops back trace. I have some other fscache patches that haven't made it upstream yet that might have been masking this issue for me.
Quoted 77 lines · Li Wang
Hi Milosz,
As far as I know, logically, currently fscache does not play
as write cache for Ceph, except that there is a
call toceph_readpage_to_fscache()inceph_writepage(), but that is nothing related to our test case. According to our observation, our test case never goes throughceph_writepage(), instead, it goes throughceph_writepages(). So in other words, I donot think this is related to caching in write path.
May I try to explain the panic in more detail,(1) dd if=/dev/zero of=cephfs/foo bs=8 count=512
(2) echo 3 >/proc/sys/vm/drop_caches
(3) dd if=cephfs/foo of=/dev/null bs=8 count=1024For statement (1), it is frequently appending a file, so
ceph_aio_write()frequently updates theinode->i_size, however, these updates did not immediately reflected toobject->store_limit_l. For statement (3), when we
start reading the second page at [4096, 8192), ceph find that the page does not be cached in fscache, then it decides to write this page into fscache, during this process incachefiles_write_page(), it found thatobject->store_limit_l< 4096 (page->index<< 12), it causes panic. Does it make sense?Cheers,
Li WangQuoted 47 lines · Milosz Tanski
Li,
I looked at the patchset am I correct that this only happens when we enable caching in the write path?
- Milosz
Quoted 35 lines · Li Wang
From: Yunchuan Wen <[email protected]>
The following scripts could easily panic the kernel,
#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 >/proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024This is due to when writing a page into fscache, the code will assert that the write position does not exceed the
object->store_limit_l, which is supposed to be equal toinode->i_size. However, for current implementation, after file writing, theobject->store_limit_lis not synchronized with newinode->i_sizeimmediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded theobject->store_limit_l, and cause kernel panic. This patch fixes it.Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initializationfs/ceph/cache.c | 1 + fs/ceph/cache.h | 10 ++++++++++ fs/ceph/file.c | 3 +++ 3 files changed, 14 insertions(+)--
1.7.9.5
Signature · 7 lines
Milosz Tanski CTO 10 East 53rd Street, 37th floor New York, NY 10022 p: 646-253-9055 e: [email protected]
Hey guys, I'm terribly sorry but apparently this got stuck in my draft mailbox for 3 months. Since we've been running this on both our test / prod clusters I would say this is sufficiently tested.
I've looked at the code and tested this for a week now on test cluster and it looks good. It does fix a real problem and I think we should push it to mainline. Thanks for fixing this Li! The reason I was not seeing this is that I had other fscache patches that were masking this problem :/
Thanks,
- Milosz
P.S: Sorry for the double mail, the first one was not sent as text. I apparently do not know how to use gmail.
Quoted 94 lines · Milosz Tanski
I'm going to look the patches and the issue in full detail. In the meantime do you guys have the oops back trace. I have some other fscache patches that haven't made it upstream yet that might have been masking this issue for me.
Quoted 24 lines · Li Wang
Hi Milosz,
As far as I know, logically, currently fscache does not play
as write cache for Ceph, except that there is a
call toceph_readpage_to_fscache()inceph_writepage(), but that is nothing related to our test case. According to our observation, our test case never goes throughceph_writepage(), instead, it goes throughceph_writepages(). So in other words, I donot think this is related to caching in write path.
May I try to explain the panic in more detail,(1) dd if=/dev/zero of=cephfs/foo bs=8 count=512
(2) echo 3 >/proc/sys/vm/drop_caches
(3) dd if=cephfs/foo of=/dev/null bs=8 count=1024For statement (1), it is frequently appending a file, so
ceph_aio_write()frequently updates theinode->i_size, however, these updates did not immediately reflected toobject->store_limit_l. For statement (3), when we
start reading the second page at [4096, 8192), ceph find that the page does not be cached in fscache, then it decides to write this page into fscache, during this process incachefiles_write_page(), it found thatobject->store_limit_l< 4096 (page->index<< 12), it causes panic. Does it make sense?Quoted 53 lines
Cheers,
Li WangQuoted 47 lines · Milosz Tanski
Li,
I looked at the patchset am I correct that this only happens when we enable caching in the write path?
- Milosz
Quoted 35 lines · Li Wang
From: Yunchuan Wen <[email protected]>
The following scripts could easily panic the kernel,
#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 >/proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024This is due to when writing a page into fscache, the code will assert that the write position does not exceed the
object->store_limit_l, which is supposed to be equal toinode->i_size. However, for current implementation, after file writing, theobject->store_limit_lis not synchronized with newinode->i_sizeimmediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded theobject->store_limit_l, and cause kernel panic. This patch fixes it.Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initializationfs/ceph/cache.c | 1 + fs/ceph/cache.h | 10 ++++++++++ fs/ceph/file.c | 3 +++ 3 files changed, 14 insertions(+)--
1.7.9.5--
Milosz Tanski
CTO
10 East 53rd Street, 37th floor
New York, NY 10022p: 646-253-9055
e: [email protected]
Signature · 7 lines
Milosz Tanski CTO 10 East 53rd Street, 37th floor New York, NY 10022 p: 646-253-9055 e: [email protected]
Hi Milosz,
Thanks, I've added this to the testing branch!
sage
Quoted 128 lines · Milosz Tanski
Hey guys, I'm terribly sorry but apparently this got stuck in my draft mailbox for 3 months. Since we've been running this on both our test / prod clusters I would say this is sufficiently tested.
I've looked at the code and tested this for a week now on test cluster and it looks good. It does fix a real problem and I think we should push it to mainline. Thanks for fixing this Li! The reason I was not seeing this is that I had other fscache patches that were masking this problem :/
Thanks,
- Milosz
P.S: Sorry for the double mail, the first one was not sent as text. I apparently do not know how to use gmail.Quoted 94 lines · Milosz Tanski
I'm going to look the patches and the issue in full detail. In the meantime do you guys have the oops back trace. I have some other fscache patches that haven't made it upstream yet that might have been masking this issue for me.
Quoted 24 lines · Li Wang
Hi Milosz,
As far as I know, logically, currently fscache does not play
as write cache for Ceph, except that there is a
call toceph_readpage_to_fscache()inceph_writepage(), but that is nothing related to our test case. According to our observation, our test case never goes throughceph_writepage(), instead, it goes throughceph_writepages(). So in other words, I donot think this is related to caching in write path.
May I try to explain the panic in more detail,(1) dd if=/dev/zero of=cephfs/foo bs=8 count=512
(2) echo 3 >/proc/sys/vm/drop_caches
(3) dd if=cephfs/foo of=/dev/null bs=8 count=1024For statement (1), it is frequently appending a file, so
ceph_aio_write()frequently updates theinode->i_size, however, these updates did not immediately reflected toobject->store_limit_l. For statement (3), when we
start reading the second page at [4096, 8192), ceph find that the page does not be cached in fscache, then it decides to write this page into fscache, during this process incachefiles_write_page(), it found thatobject->store_limit_l< 4096 (page->index<< 12), it causes panic. Does it make sense?Quoted 53 lines
Cheers,
Li WangQuoted 47 lines · Milosz Tanski
Li,
I looked at the patchset am I correct that this only happens when we enable caching in the write path?
- Milosz
Quoted 35 lines · Li Wang
From: Yunchuan Wen <[email protected]>
The following scripts could easily panic the kernel,
#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 >/proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024This is due to when writing a page into fscache, the code will assert that the write position does not exceed the
object->store_limit_l, which is supposed to be equal toinode->i_size. However, for current implementation, after file writing, theobject->store_limit_lis not synchronized with newinode->i_sizeimmediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded theobject->store_limit_l, and cause kernel panic. This patch fixes it.Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initializationfs/ceph/cache.c | 1 + fs/ceph/cache.h | 10 ++++++++++ fs/ceph/file.c | 3 +++ 3 files changed, 14 insertions(+)--
1.7.9.5--
Milosz Tanski
CTO
10 East 53rd Street, 37th floor
New York, NY 10022p: 646-253-9055
e: [email protected]Signature · 11 lines
Milosz Tanski CTO 10 East 53rd Street, 37th floor New York, NY 10022 p: 646-253-9055 e: [email protected] -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html