Skip to contentSkip to conversation

Ceph fscache: Fix kernel panic due to a race

PATCH In review Li Wang 4 participants 10 messages Dec 26, 2013 – Mar 6, 2014

Subsystems (2)
lore
Share & tools
lore

Li Wang

author
#0 lore raw

cover Ceph fscache: Fix kernel panic due to a race

From: Yunchuan Wen <[email protected]>

The following scripts could easily panic the kernel,

#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 > /proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024

This is due to when writing a page into fscache, the code will assert that the write position does not exceed the object->store_limit_l, which is supposed to be equal to inode->i_size. However, for current implementation, after file writing, the object->store_limit_l is not synchronized with new inode->i_size immediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded the object->store_limit_l, and cause kernel panic. This patch fixes it.

Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initialization

 fs/ceph/cache.c |    1 +
 fs/ceph/cache.h |   10 ++++++++++
 fs/ceph/file.c  |    3 +++
 3 files changed, 14 insertions(+)
Signature · 1 line
1.7.9.5
author
#1 lore raw

PATCH 1/3 Ceph fscache: Add an interface to synchronize object store limit

From: Yunchuan Wen <[email protected]>

Add an interface to explicitly synchronize object->store_limit[_l] with inode->i_size

---
 fs/ceph/cache.h |   10 ++++++++++
 1 file changed, 10 insertions(+)
fs/ceph/cache.h +10 -0
index ba94940..262106b 100644
@@ -48,6 +48,12 @@ void ceph_readpage_to_fscache(struct inode *inode, struct page *page);
 void ceph_invalidate_fscache_page(struct inode* inode, struct page *page);
 void ceph_queue_revalidate(struct inode *inode);
 
+static inline void ceph_fscache_update_objectsize(struct inode *inode)
+{
+	struct ceph_inode_info *ci = ceph_inode(inode);
+	fscache_attr_changed(ci->fscache);
+}
+
 static inline void ceph_fscache_invalidate(struct inode *inode)
 {
 	fscache_invalidate(ceph_inode(inode)->fscache);
@@ -127,6 +133,10 @@ static inline void ceph_readpage_to_fscache(struct inode *inode,
 {
 }
 
+static inline void ceph_fscache_update_objectsize(struct inode *inode)
+{
+}
+
 static inline void ceph_fscache_invalidate(struct inode *inode)
 {
 }
Signature · 1 line
1.7.9.5
author
#2 lore raw

PATCH 2/3 Ceph fscache: Update object store limit after file writing

In reply to Li Wang

From: Yunchuan Wen <[email protected]>

Synchronize object->store_limit[_l] with new inode->i_size after file writing.

---
 fs/ceph/file.c |    3 +++
 1 file changed, 3 insertions(+)
fs/ceph/file.c +3 -0
index 3de8982..b6df7ab 100644
@@ -786,6 +786,7 @@ retry_snap:
 			goto retry_snap;
 		}
 	} else {
+		loff_t old_size = inode->i_size;
 		/*
 		 * No need to acquire the i_truncate_mutex. Because
 		 * the MDS revokes Fwb caps before sending truncate
@@ -796,6 +797,8 @@ retry_snap:
 		written = generic_file_buffered_write(iocb, iov, nr_segs,
 						      pos, &iocb->ki_pos,
 						      count, 0);
+		if (inode->i_size > old_size)
+			ceph_fscache_update_objectsize(inode);
 		mutex_unlock(&inode->i_mutex);
 	}
 
Signature · 1 line
1.7.9.5
author
#3 lore raw

PATCH 3/3 Ceph fscache: Wait for completion of object initialization

In reply to Li Wang

From: Yunchuan Wen <[email protected]>

The object store limit needs to be updated after writing, and this can be done provided the corresponding object has already been initialized. Current object initialization is done asynchrously, which introduce a race if a file is opened, then immediately followed by a writing, the initialization may have not completed, the code will reach the ASSERT in fscache_submit_exclusive_op() to cause kernel bug.

---
 fs/ceph/cache.c |    1 +
 1 file changed, 1 insertion(+)
fs/ceph/cache.c +1 -0
index 8c44fdd..834f9f3 100644
@@ -205,6 +205,7 @@ void ceph_fscache_register_inode_cookie(struct ceph_fs_client* fsc,
 	ci->fscache = fscache_acquire_cookie(fsc->fscache,
 					     &ceph_fscache_inode_object_def,
 					     ci, true);
+	fscache_check_consistency(ci->fscache);
 done:
 	mutex_unlock(&inode->i_mutex);
 
Signature · 1 line
1.7.9.5
#4 lore raw
In reply to Li Wang

Li,

I looked at the patchset am I correct that this only happens when we enable caching in the write path?

- Milosz

Quoted 34 lines · Li Wang

From: Yunchuan Wen <[email protected]>

The following scripts could easily panic the kernel,

#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 > /proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024

This is due to when writing a page into fscache, the code will assert that the write position does not exceed the object->store_limit_l, which is supposed to be equal to inode->i_size. However, for current implementation, after file writing, the object->store_limit_l is not synchronized with new inode->i_size immediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded the object->store_limit_l, and cause kernel panic. This patch fixes it.

Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initialization

 fs/ceph/cache.c |    1 +
 fs/ceph/cache.h |   10 ++++++++++
 fs/ceph/file.c  |    3 +++
 3 files changed, 14 insertions(+)

--
1.7.9.5

Signature · 7 lines
Milosz Tanski
CTO
10 East 53rd Street, 37th floor
New York, NY 10022

p: 646-253-9055
e: [email protected]
#5 lore raw

Hi, Milosz
I am not very sure about how to enable caching in write path.

I just compile the latest kernel with "Enable Ceph client caching support", start cachefilesd, and mount cephfs with -o fsc. Then, the kernel will panic easily when the script runs to "dd if=cephfs/foo of=/dev/null bs=8 count=1024"

2013/12/27 Milosz Tanski <[email protected]>:

Quoted 57 lines

Li,

I looked at the patchset am I correct that this only happens when we enable caching in the write path?

- Milosz

Quoted 26 lines · Li Wang

From: Yunchuan Wen <[email protected]>

The following scripts could easily panic the kernel,

#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 > /proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024

This is due to when writing a page into fscache, the code will assert that the write position does not exceed the object->store_limit_l, which is supposed to be equal to inode->i_size. However, for current implementation, after file writing, the object->store_limit_l is not synchronized with new inode->i_size immediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded the object->store_limit_l, and cause kernel panic. This patch fixes it.

Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initialization

Quoted 8 lines
 fs/ceph/cache.c |    1 +
 fs/ceph/cache.h |   10 ++++++++++
 fs/ceph/file.c  |    3 +++
 3 files changed, 14 insertions(+)

--
1.7.9.5

--
Milosz Tanski
CTO
10 East 53rd Street, 37th floor
New York, NY 10022

p: 646-253-9055
e: [email protected]
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#6 lore raw
In reply to Milosz Tanski

Hi Milosz,
As far as I know, logically, currently fscache does not play
as write cache for Ceph, except that there is a
call to ceph_readpage_to_fscache() in ceph_writepage(), but that is nothing related to our test case. According to our observation, our test case never goes through ceph_writepage(), instead, it goes through ceph_writepages(). So in other words, I donot think this is related to caching in write path.
May I try to explain the panic in more detail,

(1) dd if=/dev/zero of=cephfs/foo bs=8 count=512
(2) echo 3 > /proc/sys/vm/drop_caches
(3) dd if=cephfs/foo of=/dev/null bs=8 count=1024

For statement (1), it is frequently appending a file, so
ceph_aio_write() frequently updates the inode->i_size,
however, these updates did not immediately reflected to
object->store_limit_l. For statement (3), when we
start reading the second page at [4096, 8192), ceph find that the page
does not be cached in fscache, then it decides to write this page into
fscache, during this process in cachefiles_write_page(), it found that object->store_limit_l < 4096 (page->index << 12), it causes panic. Does it make sense?

Cheers,
Li Wang

Quoted 45 lines · Milosz Tanski

Li,

I looked at the patchset am I correct that this only happens when we enable caching in the write path?

- Milosz

Quoted 34 lines · Li Wang

From: Yunchuan Wen <[email protected]>

The following scripts could easily panic the kernel,

#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 > /proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024

This is due to when writing a page into fscache, the code will assert that the write position does not exceed the object->store_limit_l, which is supposed to be equal to inode->i_size. However, for current implementation, after file writing, the object->store_limit_l is not synchronized with new inode->i_size immediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded the object->store_limit_l, and cause kernel panic. This patch fixes it.

Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initialization

  fs/ceph/cache.c |    1 +
  fs/ceph/cache.h |   10 ++++++++++
  fs/ceph/file.c  |    3 +++
  3 files changed, 14 insertions(+)

--
1.7.9.5

#7 lore raw

I'm going to look the patches and the issue in full detail. In the meantime do you guys have the oops back trace. I have some other fscache patches that haven't made it upstream yet that might have been masking this issue for me.

Quoted 77 lines · Li Wang

Hi Milosz,
As far as I know, logically, currently fscache does not play
as write cache for Ceph, except that there is a
call to ceph_readpage_to_fscache() in ceph_writepage(), but that is nothing related to our test case. According to our observation, our test case never goes through ceph_writepage(), instead, it goes through ceph_writepages(). So in other words, I donot think this is related to caching in write path.
May I try to explain the panic in more detail,

(1) dd if=/dev/zero of=cephfs/foo bs=8 count=512
(2) echo 3 > /proc/sys/vm/drop_caches
(3) dd if=cephfs/foo of=/dev/null bs=8 count=1024

For statement (1), it is frequently appending a file, so ceph_aio_write() frequently updates the inode->i_size, however, these updates did not immediately reflected to object->store_limit_l. For statement (3), when we
start reading the second page at [4096, 8192), ceph find that the page does not be cached in fscache, then it decides to write this page into fscache, during this process in cachefiles_write_page(), it found that object->store_limit_l < 4096 (page->index << 12), it causes panic. Does it make sense?

Cheers,
Li Wang

Quoted 47 lines · Milosz Tanski

Li,

I looked at the patchset am I correct that this only happens when we enable caching in the write path?

- Milosz

Quoted 35 lines · Li Wang

From: Yunchuan Wen <[email protected]>

The following scripts could easily panic the kernel,

#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 > /proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024

This is due to when writing a page into fscache, the code will assert that the write position does not exceed the object->store_limit_l, which is supposed to be equal to inode->i_size. However, for current implementation, after file writing, the object->store_limit_l is not synchronized with new inode->i_size immediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded the object->store_limit_l, and cause kernel panic. This patch fixes it.

Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initialization

  fs/ceph/cache.c |    1 +
  fs/ceph/cache.h |   10 ++++++++++
  fs/ceph/file.c  |    3 +++
  3 files changed, 14 insertions(+)

--
1.7.9.5

Signature · 7 lines
Milosz Tanski
CTO
10 East 53rd Street, 37th floor
New York, NY 10022

p: 646-253-9055
e: [email protected]
#8 lore raw

Hey guys, I'm terribly sorry but apparently this got stuck in my draft mailbox for 3 months. Since we've been running this on both our test / prod clusters I would say this is sufficiently tested.

I've looked at the code and tested this for a week now on test cluster and it looks good. It does fix a real problem and I think we should push it to mainline. Thanks for fixing this Li! The reason I was not seeing this is that I had other fscache patches that were masking this problem :/

Thanks,
- Milosz

P.S: Sorry for the double mail, the first one was not sent as text. I apparently do not know how to use gmail.

Quoted 94 lines · Milosz Tanski

I'm going to look the patches and the issue in full detail. In the meantime do you guys have the oops back trace. I have some other fscache patches that haven't made it upstream yet that might have been masking this issue for me.

Quoted 24 lines · Li Wang

Hi Milosz,
As far as I know, logically, currently fscache does not play
as write cache for Ceph, except that there is a
call to ceph_readpage_to_fscache() in ceph_writepage(), but that is nothing related to our test case. According to our observation, our test case never goes through ceph_writepage(), instead, it goes through ceph_writepages(). So in other words, I donot think this is related to caching in write path.
May I try to explain the panic in more detail,

(1) dd if=/dev/zero of=cephfs/foo bs=8 count=512
(2) echo 3 > /proc/sys/vm/drop_caches
(3) dd if=cephfs/foo of=/dev/null bs=8 count=1024

For statement (1), it is frequently appending a file, so ceph_aio_write() frequently updates the inode->i_size, however, these updates did not immediately reflected to object->store_limit_l. For statement (3), when we
start reading the second page at [4096, 8192), ceph find that the page does not be cached in fscache, then it decides to write this page into fscache, during this process in cachefiles_write_page(), it found that object->store_limit_l < 4096 (page->index << 12), it causes panic. Does it make sense?

Quoted 53 lines

Cheers,
Li Wang

Quoted 47 lines · Milosz Tanski

Li,

I looked at the patchset am I correct that this only happens when we enable caching in the write path?

- Milosz

Quoted 35 lines · Li Wang

From: Yunchuan Wen <[email protected]>

The following scripts could easily panic the kernel,

#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 > /proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024

This is due to when writing a page into fscache, the code will assert that the write position does not exceed the object->store_limit_l, which is supposed to be equal to inode->i_size. However, for current implementation, after file writing, the object->store_limit_l is not synchronized with new inode->i_size immediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded the object->store_limit_l, and cause kernel panic. This patch fixes it.

Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initialization

  fs/ceph/cache.c |    1 +
  fs/ceph/cache.h |   10 ++++++++++
  fs/ceph/file.c  |    3 +++
  3 files changed, 14 insertions(+)

--
1.7.9.5

--
Milosz Tanski
CTO
10 East 53rd Street, 37th floor
New York, NY 10022

p: 646-253-9055
e: [email protected]

Signature · 7 lines
Milosz Tanski
CTO
10 East 53rd Street, 37th floor
New York, NY 10022

p: 646-253-9055
e: [email protected]
#9 lore raw

Hi Milosz,

Thanks, I've added this to the testing branch!

sage

Quoted 128 lines · Milosz Tanski

Hey guys, I'm terribly sorry but apparently this got stuck in my draft mailbox for 3 months. Since we've been running this on both our test / prod clusters I would say this is sufficiently tested.

I've looked at the code and tested this for a week now on test cluster and it looks good. It does fix a real problem and I think we should push it to mainline. Thanks for fixing this Li! The reason I was not seeing this is that I had other fscache patches that were masking this problem :/

Thanks,
- Milosz

P.S: Sorry for the double mail, the first one was not sent as text. I apparently do not know how to use gmail.

Quoted 94 lines · Milosz Tanski

I'm going to look the patches and the issue in full detail. In the meantime do you guys have the oops back trace. I have some other fscache patches that haven't made it upstream yet that might have been masking this issue for me.

Quoted 24 lines · Li Wang

Hi Milosz,
As far as I know, logically, currently fscache does not play
as write cache for Ceph, except that there is a
call to ceph_readpage_to_fscache() in ceph_writepage(), but that is nothing related to our test case. According to our observation, our test case never goes through ceph_writepage(), instead, it goes through ceph_writepages(). So in other words, I donot think this is related to caching in write path.
May I try to explain the panic in more detail,

(1) dd if=/dev/zero of=cephfs/foo bs=8 count=512
(2) echo 3 > /proc/sys/vm/drop_caches
(3) dd if=cephfs/foo of=/dev/null bs=8 count=1024

For statement (1), it is frequently appending a file, so ceph_aio_write() frequently updates the inode->i_size, however, these updates did not immediately reflected to object->store_limit_l. For statement (3), when we
start reading the second page at [4096, 8192), ceph find that the page does not be cached in fscache, then it decides to write this page into fscache, during this process in cachefiles_write_page(), it found that object->store_limit_l < 4096 (page->index << 12), it causes panic. Does it make sense?

Quoted 53 lines

Cheers,
Li Wang

Quoted 47 lines · Milosz Tanski

Li,

I looked at the patchset am I correct that this only happens when we enable caching in the write path?

- Milosz

Quoted 35 lines · Li Wang

From: Yunchuan Wen <[email protected]>

The following scripts could easily panic the kernel,

#!/bin/bash
mount -t ceph -o fsc MONADDR:/ cephfs
rm -rf cephfs/foo
dd if=/dev/zero of=cephfs/foo bs=8 count=512
echo 3 > /proc/sys/vm/drop_caches
dd if=cephfs/foo of=/dev/null bs=8 count=1024

This is due to when writing a page into fscache, the code will assert that the write position does not exceed the object->store_limit_l, which is supposed to be equal to inode->i_size. However, for current implementation, after file writing, the object->store_limit_l is not synchronized with new inode->i_size immediately, which introduces a race that if writing a new page into fscache, will reach the ASSERT that write position has exceeded the object->store_limit_l, and cause kernel panic. This patch fixes it.

Yunchuan Wen (3):
Ceph fscache: Add an interface to synchronize object store limit Ceph fscache: Update object store limit after writing Ceph fscache: Wait for completion of object initialization

  fs/ceph/cache.c |    1 +
  fs/ceph/cache.h |   10 ++++++++++
  fs/ceph/file.c  |    3 +++
  3 files changed, 14 insertions(+)

--
1.7.9.5

--
Milosz Tanski
CTO
10 East 53rd Street, 37th floor
New York, NY 10022

p: 646-253-9055
e: [email protected]

Signature · 11 lines
Milosz Tanski
CTO
10 East 53rd Street, 37th floor
New York, NY 10022

p: 646-253-9055
e: [email protected]
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html