Skip to contentSkip to conversation

ceph: request expedited service when flushing caps

PATCH In review Jeff Layton 4 participants 7 messages Mar 31, 2020 – Apr 1, 2020 Ceph client

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ceph: request expedited service when flushing caps

Jan noticed some long stalls when flushing caps using sync() after doing small file creates. For instance, running this:

    $ time for i in $(seq -w 11 30); do echo "Hello World" > hello-$i.txt; sync -f ./hello-$i.txt; done

Could take more than 90s in some cases. The sync() will flush out caps, but doesn't tell the MDS that it's waiting synchronously on the replies.

When ceph_check_caps finds that CHECK_CAPS_FLUSH is set, then set the CEPH_CLIENT_CAPS_SYNC bit in the cap update request. This clues the MDS into that fact and it can then expedite the reply.

URL: https://tracker.ceph.com/issues/44744

---
 fs/ceph/caps.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/ceph/caps.c +5 -2
index 61808793e0c0..6403178f2376 100644
@@ -2111,8 +2111,11 @@ void ceph_check_caps(struct ceph_inode_info *ci, int flags,
 
 		mds = cap->mds;  /* remember mds, so we don't repeat */
 
-		__prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE, 0, cap_used, want,
-			   retain, flushing, flush_tid, oldest_flush_tid);
+		__prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE,
+			   (flags & CHECK_CAPS_FLUSH) ?
+			    CEPH_CLIENT_CAPS_SYNC : 0,
+			   cap_used, want, retain, flushing, flush_tid,
+			   oldest_flush_tid);
 		spin_unlock(&ci->i_ceph_lock);
 
 		__send_cap(mdsc, &arg, ci);
Signature · 1 line
2.25.1
#1 lore raw
Quoted 9 lines · Jeff Layton

Jan noticed some long stalls when flushing caps using sync() after doing small file creates. For instance, running this:

    $ time for i in $(seq -w 11 30); do echo "Hello World" > hello-$i.txt; sync -f ./hello-$i.txt; done

Could take more than 90s in some cases. The sync() will flush out caps, but doesn't tell the MDS that it's waiting synchronously on the replies.

Quoted 7 lines

When ceph_check_caps finds that CHECK_CAPS_FLUSH is set, then set the CEPH_CLIENT_CAPS_SYNC bit in the cap update request. This clues the MDS into that fact and it can then expedite the reply.

URL: https://tracker.ceph.com/issues/44744

Feel free to add my Reviewed-by (and also tested). Also, it may be worth adding a stable tag for this patch, although it would require some massaging to use __send_cap instead of __prep_cap.

Cheers,
--
Luis

Quoted 25 lines
---
 fs/ceph/caps.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/ceph/caps.c +5 -2
index 61808793e0c0..6403178f2376 100644
@@ -2111,8 +2111,11 @@ void ceph_check_caps(struct ceph_inode_info *ci, int flags,
 
 		mds = cap->mds;  /* remember mds, so we don't repeat */
 
-		__prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE, 0, cap_used, want,
-			   retain, flushing, flush_tid, oldest_flush_tid);
+		__prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE,
+			   (flags & CHECK_CAPS_FLUSH) ?
+			    CEPH_CLIENT_CAPS_SYNC : 0,
+			   cap_used, want, retain, flushing, flush_tid,
+			   oldest_flush_tid);
 		spin_unlock(&ci->i_ceph_lock);
 
 		__send_cap(mdsc, &arg, ci);
Signature · 1 line
2.25.1
#2 lore raw
In reply to Jeff Layton
Quoted 28 lines · Jeff Layton

Jan noticed some long stalls when flushing caps using sync() after doing small file creates. For instance, running this:

    $ time for i in $(seq -w 11 30); do echo "Hello World" > hello-$i.txt; sync -f ./hello-$i.txt; done

Could take more than 90s in some cases. The sync() will flush out caps, but doesn't tell the MDS that it's waiting synchronously on the replies.

When ceph_check_caps finds that CHECK_CAPS_FLUSH is set, then set the CEPH_CLIENT_CAPS_SYNC bit in the cap update request. This clues the MDS into that fact and it can then expedite the reply.

URL: https://tracker.ceph.com/issues/44744

---
 fs/ceph/caps.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/ceph/caps.c +0 -0
index 61808793e0c0..6403178f2376 100644
@@ -2111,8 +2111,11 @@ void ceph_check_caps(struct ceph_inode_info *ci, int flags,
 
                mds = cap->mds;  /* remember mds, so we don't repeat */

fs/ceph/caps.c void ceph_check_caps(struct ceph_inode_info *ci, int flags, · line 2120

Quoted 10 lines
fs/ceph/caps.c +5 -2
 
-               __prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE, 0, cap_used, want,
-                          retain, flushing, flush_tid, oldest_flush_tid);
+               __prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE,
+                          (flags & CHECK_CAPS_FLUSH) ?
+                           CEPH_CLIENT_CAPS_SYNC : 0,
+                          cap_used, want, retain, flushing, flush_tid,
+                          oldest_flush_tid);
                spin_unlock(&ci->i_ceph_lock);
 

this is too expensive for syncfs case. mds needs to flush journal for each dirty inode. we'd better to track dirty inodes by session, and only set the flag when flushing the last inode in session dirty list.

Quoted 4 lines
fs/ceph/caps.c +0 -1
                __send_cap(mdsc, &arg, ci);
--

2.25.1

#3 lore raw
Quoted 31 lines · Zheng
Quoted 29 lines · Jeff Layton

Jan noticed some long stalls when flushing caps using sync() after doing small file creates. For instance, running this:

    $ time for i in $(seq -w 11 30); do echo "Hello World" > hello-$i.txt; sync -f ./hello-$i.txt; done

Could take more than 90s in some cases. The sync() will flush out caps, but doesn't tell the MDS that it's waiting synchronously on the replies.

When ceph_check_caps finds that CHECK_CAPS_FLUSH is set, then set the CEPH_CLIENT_CAPS_SYNC bit in the cap update request. This clues the MDS into that fact and it can then expedite the reply.

URL: https://tracker.ceph.com/issues/44744

---
 fs/ceph/caps.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/ceph/caps.c +0 -0
index 61808793e0c0..6403178f2376 100644
@@ -2111,8 +2111,11 @@ void ceph_check_caps(struct ceph_inode_info *ci, int flags,
 
                mds = cap->mds;  /* remember mds, so we don't repeat */
 
Quoted 13 lines

fs/ceph/caps.c void ceph_check_caps(struct ceph_inode_info *ci, int flags, · line 2120

Quoted 9 lines
fs/ceph/caps.c +5 -2
-               __prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE, 0, cap_used, want,
-                          retain, flushing, flush_tid, oldest_flush_tid);
+               __prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE,
+                          (flags & CHECK_CAPS_FLUSH) ?
+                           CEPH_CLIENT_CAPS_SYNC : 0,
+                          cap_used, want, retain, flushing, flush_tid,
+                          oldest_flush_tid);
                spin_unlock(&ci->i_ceph_lock);
 

this is too expensive for syncfs case. mds needs to flush journal for each dirty inode. we'd better to track dirty inodes by session, and only set the flag when flushing the last inode in session dirty list.

Yeah, see the userspace Client::_sync_fs() where we have an internal flags argument which is set on the last cap in the dirty set and tells the actual cap message flushing code to set FLAG_SYNC on the MClientCaps message. I presume the kernel is operating on a similar principle here?
-Greg

Quoted 7 lines
Quoted 6 lines
fs/ceph/caps.c +0 -1
                __send_cap(mdsc, &arg, ci);
--

2.25.1

#4 lore raw
Quoted 30 lines · Gregory Farnum
Quoted 29 lines · Zheng
Quoted 28 lines · Jeff Layton

Jan noticed some long stalls when flushing caps using sync() after doing small file creates. For instance, running this:

    $ time for i in $(seq -w 11 30); do echo "Hello World" > hello-$i.txt; sync -f ./hello-$i.txt; done

Could take more than 90s in some cases. The sync() will flush out caps, but doesn't tell the MDS that it's waiting synchronously on the replies.

When ceph_check_caps finds that CHECK_CAPS_FLUSH is set, then set the CEPH_CLIENT_CAPS_SYNC bit in the cap update request. This clues the MDS into that fact and it can then expedite the reply.

URL: https://tracker.ceph.com/issues/44744

---
 fs/ceph/caps.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/ceph/caps.c +0 -0
index 61808793e0c0..6403178f2376 100644
@@ -2111,8 +2111,11 @@ void ceph_check_caps(struct ceph_inode_info *ci, int flags,
 
                mds = cap->mds;  /* remember mds, so we don't repeat */
 
Quoted 13 lines
Quoted 10 lines

fs/ceph/caps.c void ceph_check_caps(struct ceph_inode_info *ci, int flags, · line 2120

Quoted 9 lines
fs/ceph/caps.c +5 -2
-               __prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE, 0, cap_used, want,
-                          retain, flushing, flush_tid, oldest_flush_tid);
+               __prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE,
+                          (flags & CHECK_CAPS_FLUSH) ?
+                           CEPH_CLIENT_CAPS_SYNC : 0,
+                          cap_used, want, retain, flushing, flush_tid,
+                          oldest_flush_tid);
                spin_unlock(&ci->i_ceph_lock);
 
Quoted 3 lines

this is too expensive for syncfs case. mds needs to flush journal for each dirty inode. we'd better to track dirty inodes by session, and only set the flag when flushing the last inode in session dirty list.

I think this will be more difficult than that...

Quoted 6 lines

Yeah, see the userspace Client::_sync_fs() where we have an internal flags argument which is set on the last cap in the dirty set and tells the actual cap message flushing code to set FLAG_SYNC on the MClientCaps message. I presume the kernel is operating on a similar principle here?

Not today, but we need it to.

The caps are not tracked on a per-session basis (as Zheng points out), and the locking and ordering of these requests is not as straightforward as it is in the (trivial) libcephfs case. Fixing this will be a lot more invasive than I had originally hoped.

It's also not 100% clear to me how we'll gauge which cap will be "last". As we move the last cap on the session's list from dirty to flushing state, we can mark it so that it sets the SYNC flag when it goes out, but what happens if we have a process that is actively dirtying other inodes during this? You might never see the per-session list go empty.

It may also go empty for reasons that have nothing to do with issuing a sync(), (or fsync() or...) so we don't want to universally set this flag in that case.

I'm not sure what the right solution is yet.

Signature · 1 line
Jeff Layton <[email protected]>
#5 lore raw
Quoted 62 lines · Jeff Layton
Quoted 30 lines · Gregory Farnum
Quoted 29 lines · Zheng
Quoted 28 lines · Jeff Layton

Jan noticed some long stalls when flushing caps using sync() after doing small file creates. For instance, running this:

    $ time for i in $(seq -w 11 30); do echo "Hello World" > hello-$i.txt; sync -f ./hello-$i.txt; done

Could take more than 90s in some cases. The sync() will flush out caps, but doesn't tell the MDS that it's waiting synchronously on the replies.

When ceph_check_caps finds that CHECK_CAPS_FLUSH is set, then set the CEPH_CLIENT_CAPS_SYNC bit in the cap update request. This clues the MDS into that fact and it can then expedite the reply.

URL: https://tracker.ceph.com/issues/44744

---
 fs/ceph/caps.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/ceph/caps.c +0 -0
index 61808793e0c0..6403178f2376 100644
@@ -2111,8 +2111,11 @@ void ceph_check_caps(struct ceph_inode_info *ci, int flags,
 
                mds = cap->mds;  /* remember mds, so we don't repeat */
 
Quoted 13 lines
Quoted 10 lines

fs/ceph/caps.c void ceph_check_caps(struct ceph_inode_info *ci, int flags, · line 2120

Quoted 9 lines
fs/ceph/caps.c +5 -2
-               __prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE, 0, cap_used, want,
-                          retain, flushing, flush_tid, oldest_flush_tid);
+               __prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE,
+                          (flags & CHECK_CAPS_FLUSH) ?
+                           CEPH_CLIENT_CAPS_SYNC : 0,
+                          cap_used, want, retain, flushing, flush_tid,
+                          oldest_flush_tid);
                spin_unlock(&ci->i_ceph_lock);
 
Quoted 3 lines

this is too expensive for syncfs case. mds needs to flush journal for each dirty inode. we'd better to track dirty inodes by session, and only set the flag when flushing the last inode in session dirty list.

I think this will be more difficult than that...

Quoted 6 lines

Yeah, see the userspace Client::_sync_fs() where we have an internal flags argument which is set on the last cap in the dirty set and tells the actual cap message flushing code to set FLAG_SYNC on the MClientCaps message. I presume the kernel is operating on a similar principle here?

Not today, but we need it to.

The caps are not tracked on a per-session basis (as Zheng points out), and the locking and ordering of these requests is not as straightforward as it is in the (trivial) libcephfs case. Fixing this will be a lot more invasive than I had originally hoped.

Quoted 7 lines

It's also not 100% clear to me how we'll gauge which cap will be "last". As we move the last cap on the session's list from dirty to flushing state, we can mark it so that it sets the SYNC flag when it goes out, but what happens if we have a process that is actively dirtying other inodes during this? You might never see the per-session list go empty.

It's not necessary to be strict 'last'. just the one before exiting the loop

Quoted 8 lines

It may also go empty for reasons that have nothing to do with issuing a sync(), (or fsync() or...) so we don't want to universally set this flag in that case.

I'm not sure what the right solution is yet.
--
Jeff Layton <[email protected]>

#6 lore raw
Quoted 69 lines · Zheng
Quoted 60 lines · Jeff Layton
Quoted 30 lines · Gregory Farnum
Quoted 29 lines · Zheng
Quoted 28 lines · Jeff Layton

Jan noticed some long stalls when flushing caps using sync() after doing small file creates. For instance, running this:

    $ time for i in $(seq -w 11 30); do echo "Hello World" > hello-$i.txt; sync -f ./hello-$i.txt; done

Could take more than 90s in some cases. The sync() will flush out caps, but doesn't tell the MDS that it's waiting synchronously on the replies.

When ceph_check_caps finds that CHECK_CAPS_FLUSH is set, then set the CEPH_CLIENT_CAPS_SYNC bit in the cap update request. This clues the MDS into that fact and it can then expedite the reply.

URL: https://tracker.ceph.com/issues/44744

---
 fs/ceph/caps.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/ceph/caps.c +0 -0
index 61808793e0c0..6403178f2376 100644
@@ -2111,8 +2111,11 @@ void ceph_check_caps(struct ceph_inode_info *ci, int flags,
 
                mds = cap->mds;  /* remember mds, so we don't repeat */
 
Quoted 13 lines
Quoted 10 lines

fs/ceph/caps.c void ceph_check_caps(struct ceph_inode_info *ci, int flags, · line 2120

Quoted 9 lines
fs/ceph/caps.c +5 -2
-               __prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE, 0, cap_used, want,
-                          retain, flushing, flush_tid, oldest_flush_tid);
+               __prep_cap(&arg, cap, CEPH_CAP_OP_UPDATE,
+                          (flags & CHECK_CAPS_FLUSH) ?
+                           CEPH_CLIENT_CAPS_SYNC : 0,
+                          cap_used, want, retain, flushing, flush_tid,
+                          oldest_flush_tid);
                spin_unlock(&ci->i_ceph_lock);
 
Quoted 3 lines

this is too expensive for syncfs case. mds needs to flush journal for each dirty inode. we'd better to track dirty inodes by session, and only set the flag when flushing the last inode in session dirty list.

I think this will be more difficult than that...

Quoted 5 lines

Yeah, see the userspace Client::_sync_fs() where we have an internal flags argument which is set on the last cap in the dirty set and tells the actual cap message flushing code to set FLAG_SYNC on the MClientCaps message. I presume the kernel is operating on a similar principle here?

Not today, but we need it to.

The caps are not tracked on a per-session basis (as Zheng points out), and the locking and ordering of these requests is not as straightforward as it is in the (trivial) libcephfs case. Fixing this will be a lot more invasive than I had originally hoped.

Quoted 7 lines

It's also not 100% clear to me how we'll gauge which cap will be "last". As we move the last cap on the session's list from dirty to flushing state, we can mark it so that it sets the SYNC flag when it goes out, but what happens if we have a process that is actively dirtying other inodes during this? You might never see the per-session list go empty.

Quoted 2 lines

It's not necessary to be strict 'last'. just the one before exiting the loop

What loop?

I added a little debugging code and ran Jan's reproducer, and it turns out that we generally move the inode to flushing state in ceph_put_wrbuffer_cap_refs while doing writeback under the sync syscall. By the time we get to any sort of looping in the ceph code, the cap flushes have already been issued.

My tentative idea was to just check whether this was the last dirty cap associated with the session and set the flag on it if so, but we don't really have visibility into that info, because we don't determine the session until we move the inode from dirty->flushing.

So at this point, I'm still looking at options for fixing this. I really don't want to just hack this in, as the technical debt in this code is already substantial and that'll just make it worse.

Signature · 1 line
Jeff Layton <[email protected]>