Skip to contentSkip to conversation

ceph: wait on async create before checking caps for syncfs

PATCH In review Jeff Layton 3 participants 17 messages Jun 6, 2022 – Jun 30, 2022 Ceph client

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ceph: wait on async create before checking caps for syncfs

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
 fs/ceph/caps.c | 1 +
 1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +1 -0
index 0a48bf829671..5ecfff4b37c9 100644
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
 		ihold(inode);
 		dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
 		spin_unlock(&mdsc->cap_dirty_lock);
+		ceph_wait_on_async_create(inode);
 		ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
 		iput(inode);
 		spin_lock(&mdsc->cap_dirty_lock);
Signature · 1 line
2.36.1
#1 lore raw
Quoted 21 lines · Jeff Layton

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
  fs/ceph/caps.c | 1 +
  1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +0 -0
index 0a48bf829671..5ecfff4b37c9 100644

fs/ceph/caps.c static void flush_dirty_session_caps(struct ceph_mds_session *s) · line 4395

Quoted 7 lines
fs/ceph/caps.c +1 -0
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
  		ihold(inode);
  		dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
  		spin_unlock(&mdsc->cap_dirty_lock);
+		ceph_wait_on_async_create(inode);
  		ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
  		iput(inode);
  		spin_lock(&mdsc->cap_dirty_lock);

This looks good.

Possibly we can add one dedicated list to store the async creating inodes instead of getting stuck all the others ?

-- Xiubo

#2 lore raw
Quoted 32 lines · Xiubo Li
Quoted 21 lines · Jeff Layton

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
  fs/ceph/caps.c | 1 +
  1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +0 -0
index 0a48bf829671..5ecfff4b37c9 100644

fs/ceph/caps.c static void flush_dirty_session_caps(struct ceph_mds_session *s) · line 4395

Quoted 7 lines
fs/ceph/caps.c +1 -0
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
  		ihold(inode);
  		dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
  		spin_unlock(&mdsc->cap_dirty_lock);
+		ceph_wait_on_async_create(inode);
  		ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
  		iput(inode);
  		spin_lock(&mdsc->cap_dirty_lock);

This looks good.

Quoted 3 lines

Possibly we can add one dedicated list to store the async creating inodes instead of getting stuck all the others ?

I'd be open to that. I think we ought to take this patch first to fix the immediate bug though, before we add extra complexity.

Signature · 1 line
Jeff Layton <[email protected]>
#3 lore raw
Quoted 32 lines · Jeff Layton
Quoted 31 lines · Xiubo Li
Quoted 21 lines · Jeff Layton

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
   fs/ceph/caps.c | 1 +
   1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +0 -0
index 0a48bf829671..5ecfff4b37c9 100644

fs/ceph/caps.c static void flush_dirty_session_caps(struct ceph_mds_session *s) · line 4395

Quoted 7 lines
fs/ceph/caps.c +1 -0
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
   		ihold(inode);
   		dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
   		spin_unlock(&mdsc->cap_dirty_lock);
+		ceph_wait_on_async_create(inode);
   		ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
   		iput(inode);
   		spin_lock(&mdsc->cap_dirty_lock);

This looks good.

Quoted 5 lines
Quoted 3 lines

Possibly we can add one dedicated list to store the async creating inodes instead of getting stuck all the others ?

I'd be open to that. I think we ought to take this patch first to fix the immediate bug though, before we add extra complexity.

Sounds good to me.

I will merge it to the testing branch for now and let's improve it later.

Thanks

-- Xiubo

#4 lore raw
Quoted 36 lines · Xiubo Li
Quoted 35 lines · Jeff Layton
Quoted 34 lines · Xiubo Li
Quoted 21 lines · Jeff Layton

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
   fs/ceph/caps.c | 1 +
   1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +0 -0
index 0a48bf829671..5ecfff4b37c9 100644

fs/ceph/caps.c static void flush_dirty_session_caps(struct ceph_mds_session *s) · line 4395

Quoted 7 lines
fs/ceph/caps.c +1 -0
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
   		ihold(inode);
   		dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
   		spin_unlock(&mdsc->cap_dirty_lock);
+		ceph_wait_on_async_create(inode);
   		ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
   		iput(inode);
   		spin_lock(&mdsc->cap_dirty_lock);

This looks good.

Possibly we can add one dedicated list to store the async creating inodes instead of getting stuck all the others ?

Quoted 7 lines
Quoted 2 lines

I'd be open to that. I think we ought to take this patch first to fix the immediate bug though, before we add extra complexity.

Sounds good to me.

I will merge it to the testing branch for now and let's improve it later.

Can we also mark this for stable? It's a pretty nasty bug, potentially. We should get this into mainline fairly soon.

Thanks,

Signature · 1 line
Jeff Layton <[email protected]>
#5 lore raw
Quoted 37 lines · Jeff Layton
Quoted 36 lines · Xiubo Li
Quoted 35 lines · Jeff Layton
Quoted 34 lines · Xiubo Li
Quoted 21 lines · Jeff Layton

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
    fs/ceph/caps.c | 1 +
    1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +0 -0
index 0a48bf829671..5ecfff4b37c9 100644

fs/ceph/caps.c static void flush_dirty_session_caps(struct ceph_mds_session *s) · line 4395

Quoted 7 lines
fs/ceph/caps.c +1 -0
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
    		ihold(inode);
    		dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
    		spin_unlock(&mdsc->cap_dirty_lock);
+		ceph_wait_on_async_create(inode);
    		ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
    		iput(inode);
    		spin_lock(&mdsc->cap_dirty_lock);

This looks good.

Possibly we can add one dedicated list to store the async creating inodes instead of getting stuck all the others ?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I'd be open to that. I think we ought to take this patch first to fix the immediate bug though, before we add extra complexity.

Sounds good to me.

I will merge it to the testing branch for now and let's improve it later.

Can we also mark this for stable? It's a pretty nasty bug, potentially. We should get this into mainline fairly soon.

Yeah, sure.

Quoted 2 lines

Thanks,

#6 lore raw
In reply to Jeff Layton

The recent series of patches that add "wait on async xxxx" at various places do not seem correct. The correct fix should make mds avoid any wait when handling async requests.

Quoted 32 lines · Jeff Layton

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
 fs/ceph/caps.c | 1 +
 1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +1 -1
index 0a48bf829671..5ecfff4b37c9 100644
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
                ihold(inode);
                dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
                spin_unlock(&mdsc->cap_dirty_lock);
+               ceph_wait_on_async_create(inode);
                ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
                iput(inode);
                spin_lock(&mdsc->cap_dirty_lock);
--

2.36.1

#7 lore raw
Quoted 4 lines · Zheng

The recent series of patches that add "wait on async xxxx" at various places do not seem correct. The correct fix should make mds avoid any wait when handling async requests.

In this case I am thinking what will happen if the async create request is deferred, then the cap flush related request should fail to find the ino.

Should we wait ? Then how to distinguish from migrating a subtree and a deferred async create cases ?

Quoted 32 lines · Jeff Layton
Quoted 31 lines

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
  fs/ceph/caps.c | 1 +
  1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +1 -1
index 0a48bf829671..5ecfff4b37c9 100644
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
                 ihold(inode);
                 dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
                 spin_unlock(&mdsc->cap_dirty_lock);
+               ceph_wait_on_async_create(inode);
                 ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
                 iput(inode);
                 spin_lock(&mdsc->cap_dirty_lock);
--

2.36.1

#8 lore raw
Quoted 7 lines · Xiubo Li
Quoted 4 lines · Zheng

The recent series of patches that add "wait on async xxxx" at various places do not seem correct. The correct fix should make mds avoid any wait when handling async requests.

Quoted 7 lines

In this case I am thinking what will happen if the async create request is deferred, then the cap flush related request should fail to find the ino.

Should we wait ? Then how to distinguish from migrating a subtree and a deferred async create cases ?

async op caps are revoked at freezingtree stage of subtree migration. see Locker::invalidate_lock_caches

Quoted 34 lines
Quoted 33 lines · Jeff Layton
Quoted 31 lines

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
  fs/ceph/caps.c | 1 +
  1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +1 -1
index 0a48bf829671..5ecfff4b37c9 100644
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
                 ihold(inode);
                 dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
                 spin_unlock(&mdsc->cap_dirty_lock);
+               ceph_wait_on_async_create(inode);
                 ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
                 iput(inode);
                 spin_lock(&mdsc->cap_dirty_lock);
--

2.36.1

#9 lore raw
Quoted 11 lines · Zheng
Quoted 10 lines · Xiubo Li
Quoted 4 lines · Zheng

The recent series of patches that add "wait on async xxxx" at various places do not seem correct. The correct fix should make mds avoid any wait when handling async requests.

In this case I am thinking what will happen if the async create request is deferred, then the cap flush related request should fail to find the ino.

Quoted 6 lines
Quoted 3 lines

Should we wait ? Then how to distinguish from migrating a subtree and a deferred async create cases ?

async op caps are revoked at freezingtree stage of subtree migration. see Locker::invalidate_lock_caches

Sorry I may not totally understand this issue.

I think you mean in case of migration and then the MDS will revoke caps for the async create files and then the kclient will send a MclientCap request to mds, right ?

If my understanding is correct, there is another case that:

1, async create a fileA

2, then write a lot of data to it and then release the Fw cap ref, and if we should report the size to MDS, it will send a MclientCap request to MDS too.

3, what if the async create of fileA was deferred due to some reason, then the MclientCap request will fail to find the ino ?

Quoted 32 lines
Quoted 32 lines
Quoted 32 lines · Jeff Layton
Quoted 31 lines

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
   fs/ceph/caps.c | 1 +
   1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +1 -1
index 0a48bf829671..5ecfff4b37c9 100644
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
                  ihold(inode);
                  dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
                  spin_unlock(&mdsc->cap_dirty_lock);
+               ceph_wait_on_async_create(inode);
                  ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
                  iput(inode);
                  spin_lock(&mdsc->cap_dirty_lock);
--

2.36.1

#10 lore raw
Quoted 28 lines · Xiubo Li
Quoted 11 lines · Zheng
Quoted 10 lines · Xiubo Li
Quoted 4 lines · Zheng

The recent series of patches that add "wait on async xxxx" at various places do not seem correct. The correct fix should make mds avoid any wait when handling async requests.

In this case I am thinking what will happen if the async create request is deferred, then the cap flush related request should fail to find the ino.

Quoted 6 lines
Quoted 3 lines

Should we wait ? Then how to distinguish from migrating a subtree and a deferred async create cases ?

async op caps are revoked at freezingtree stage of subtree migration. see Locker::invalidate_lock_caches

Sorry I may not totally understand this issue.

I think you mean in case of migration and then the MDS will revoke caps for the async create files and then the kclient will send a MclientCap request to mds, right ?

If my understanding is correct, there is another case that:

Quoted 9 lines

1, async create a fileA

2, then write a lot of data to it and then release the Fw cap ref, and if we should report the size to MDS, it will send a MclientCap request to MDS too.

3, what if the async create of fileA was deferred due to some reason, then the MclientCap request will fail to find the ino ?

Async op should not be deferred in any case.

Quoted 34 lines
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines · Jeff Layton
Quoted 31 lines

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
   fs/ceph/caps.c | 1 +
   1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +1 -1
index 0a48bf829671..5ecfff4b37c9 100644
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
                  ihold(inode);
                  dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
                  spin_unlock(&mdsc->cap_dirty_lock);
+               ceph_wait_on_async_create(inode);
                  ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
                  iput(inode);
                  spin_lock(&mdsc->cap_dirty_lock);
--

2.36.1

#11 lore raw
Quoted 29 lines · Zheng
Quoted 28 lines · Xiubo Li
Quoted 10 lines · Zheng
Quoted 9 lines · Xiubo Li
Quoted 4 lines · Zheng

The recent series of patches that add "wait on async xxxx" at various places do not seem correct. The correct fix should make mds avoid any wait when handling async requests.

In this case I am thinking what will happen if the async create request is deferred, then the cap flush related request should fail to find the ino.

Quoted 6 lines
Quoted 3 lines

Should we wait ? Then how to distinguish from migrating a subtree and a deferred async create cases ?

async op caps are revoked at freezingtree stage of subtree migration. see Locker::invalidate_lock_caches

Sorry I may not totally understand this issue.

I think you mean in case of migration and then the MDS will revoke caps for the async create files and then the kclient will send a MclientCap request to mds, right ?

If my understanding is correct, there is another case that:

1, async create a fileA

Quoted 8 lines
Quoted 7 lines

2, then write a lot of data to it and then release the Fw cap ref, and if we should report the size to MDS, it will send a MclientCap request to MDS too.

3, what if the async create of fileA was deferred due to some reason, then the MclientCap request will fail to find the ino ?

Async op should not be deferred in any case.

I am still checking the 'mdcache->path_traverse()', in which it seems could forward the request or requeue the request when failing to acquire locks. And also in case [1].

[1] https://github.com/ceph/ceph/blob/main/src/mds/Server.cc#L4501.

Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines · Jeff Layton
Quoted 31 lines

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
    fs/ceph/caps.c | 1 +
    1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +1 -1
index 0a48bf829671..5ecfff4b37c9 100644
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
                   ihold(inode);
                   dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
                   spin_unlock(&mdsc->cap_dirty_lock);
+               ceph_wait_on_async_create(inode);
                   ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
                   iput(inode);
                   spin_lock(&mdsc->cap_dirty_lock);
--

2.36.1

#12 lore raw
In reply to Yan, Zheng
Quoted 29 lines · Zheng
Quoted 28 lines · Xiubo Li
Quoted 10 lines · Zheng
Quoted 9 lines · Xiubo Li
Quoted 4 lines · Zheng

The recent series of patches that add "wait on async xxxx" at various places do not seem correct. The correct fix should make mds avoid any wait when handling async requests.

In this case I am thinking what will happen if the async create request is deferred, then the cap flush related request should fail to find the ino.

Quoted 6 lines
Quoted 3 lines

Should we wait ? Then how to distinguish from migrating a subtree and a deferred async create cases ?

async op caps are revoked at freezingtree stage of subtree migration. see Locker::invalidate_lock_caches

Sorry I may not totally understand this issue.

I think you mean in case of migration and then the MDS will revoke caps for the async create files and then the kclient will send a MclientCap request to mds, right ?

If my understanding is correct, there is another case that:

1, async create a fileA

Quoted 8 lines
Quoted 7 lines

2, then write a lot of data to it and then release the Fw cap ref, and if we should report the size to MDS, it will send a MclientCap request to MDS too.

3, what if the async create of fileA was deferred due to some reason, then the MclientCap request will fail to find the ino ?

Async op should not be deferred in any case.

Recently we have hit a similar bug, caused by deferring a request and requeuing it and the following request was executed before it.

Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines · Jeff Layton
Quoted 31 lines

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
    fs/ceph/caps.c | 1 +
    1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +1 -1
index 0a48bf829671..5ecfff4b37c9 100644
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
                   ihold(inode);
                   dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
                   spin_unlock(&mdsc->cap_dirty_lock);
+               ceph_wait_on_async_create(inode);
                   ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
                   iput(inode);
                   spin_lock(&mdsc->cap_dirty_lock);
--

2.36.1

#13 lore raw
Quoted 36 lines · Xiubo Li
Quoted 33 lines · Zheng
Quoted 32 lines · Xiubo Li
Quoted 10 lines · Zheng
Quoted 9 lines · Xiubo Li
Quoted 4 lines · Zheng

The recent series of patches that add "wait on async xxxx" at various places do not seem correct. The correct fix should make mds avoid any wait when handling async requests.

In this case I am thinking what will happen if the async create request is deferred, then the cap flush related request should fail to find the ino.

Quoted 6 lines
Quoted 3 lines

Should we wait ? Then how to distinguish from migrating a subtree and a deferred async create cases ?

async op caps are revoked at freezingtree stage of subtree migration. see Locker::invalidate_lock_caches

Sorry I may not totally understand this issue.

I think you mean in case of migration and then the MDS will revoke caps for the async create files and then the kclient will send a MclientCap request to mds, right ?

If my understanding is correct, there is another case that:

1, async create a fileA

2, then write a lot of data to it and then release the Fw cap ref, and if we should report the size to MDS, it will send a MclientCap request to MDS too.

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

3, what if the async create of fileA was deferred due to some reason, then the MclientCap request will fail to find the ino ?

Async op should not be deferred in any case.

Recently we have hit a similar bug, caused by deferring a request and requeuing it and the following request was executed before it.

that bug is mds bug.

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines · Jeff Layton
Quoted 31 lines

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
    fs/ceph/caps.c | 1 +
    1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +1 -1
index 0a48bf829671..5ecfff4b37c9 100644
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
                   ihold(inode);
                   dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
                   spin_unlock(&mdsc->cap_dirty_lock);
+               ceph_wait_on_async_create(inode);
                   ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
                   iput(inode);
                   spin_lock(&mdsc->cap_dirty_lock);
--

2.36.1

#14 lore raw
In reply to Yan, Zheng

Are you suggesting that the MDS ought to hold a cap message for an inode before its create request is processed? Note that the MDS won't even be aware that the inode even _exists_ at that point. As far as the MDS knows, it's just be a delegated inode number to the client. At what point does the MDS give up on holding such a cap request if the create request never comes in for some reason?

I don't see the harm in making the client wait until it gets a create reply before sending a cap message. If we want to revert fbed7045f552 instead, we can do that, but it'll cause a regression until the MDS is fixed [1]. Regardless, we need to either take this patch or revert that one.

I move that we take this patch for now to address the softlockups. Once the MDS is fixed we could revert this and fbed7045f552 without causing a regression.

[1]: https://tracker.ceph.com/issues/54107

Quoted 38 lines · Zheng

The recent series of patches that add "wait on async xxxx" at various places do not seem correct. The correct fix should make mds avoid any wait when handling async requests.

Quoted 32 lines · Jeff Layton

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
 fs/ceph/caps.c | 1 +
 1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

fs/ceph/caps.c +1 -1
index 0a48bf829671..5ecfff4b37c9 100644
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
                ihold(inode);
                dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
                spin_unlock(&mdsc->cap_dirty_lock);
+               ceph_wait_on_async_create(inode);
                ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
                iput(inode);
                spin_lock(&mdsc->cap_dirty_lock);
--

2.36.1

Signature · 1 line
Jeff Layton <[email protected]>
#15 lore raw
Quoted 8 lines · Jeff Layton

Are you suggesting that the MDS ought to hold a cap message for an inode before its create request is processed? Note that the MDS won't even be aware that the inode even _exists_ at that point. As far as the MDS knows, it's just be a delegated inode number to the client. At what point does the MDS give up on holding such a cap request if the create request never comes in for some reason?

For an async request, MDS should not process it immediately. If there is any wait when handling async request, it's mds bug. I suggest tracking down any wait, and fix it.

Quoted 55 lines

I don't see the harm in making the client wait until it gets a create reply before sending a cap message. If we want to revert fbed7045f552 instead, we can do that, but it'll cause a regression until the MDS is fixed [1]. Regardless, we need to either take this patch or revert that one.

I move that we take this patch for now to address the softlockups. Once the MDS is fixed we could revert this and fbed7045f552 without causing a regression.

[1]: https://tracker.ceph.com/issues/54107

Quoted 23 lines · Zheng

The recent series of patches that add "wait on async xxxx" at various places do not seem correct. The correct fix should make mds avoid any wait when handling async requests.

Quoted 17 lines · Jeff Layton

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
 fs/ceph/caps.c | 1 +
 1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

Quoted 15 lines
Quoted 13 lines
fs/ceph/caps.c +1 -1
index 0a48bf829671..5ecfff4b37c9 100644
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
                ihold(inode);
                dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
                spin_unlock(&mdsc->cap_dirty_lock);
+               ceph_wait_on_async_create(inode);
                ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
                iput(inode);
                spin_lock(&mdsc->cap_dirty_lock);
--
Quoted 2 lines

2.36.1

--
Jeff Layton <[email protected]>

#16 lore raw
Quoted 10 lines · Zheng
Quoted 8 lines · Jeff Layton

Are you suggesting that the MDS ought to hold a cap message for an inode before its create request is processed? Note that the MDS won't even be aware that the inode even _exists_ at that point. As far as the MDS knows, it's just be a delegated inode number to the client. At what point does the MDS give up on holding such a cap request if the create request never comes in for some reason?

Quoted 4 lines

For an async request, MDS should not process it immediately. If there is any wait when handling async request, it's mds bug. I suggest tracking down any wait, and fix it.

I mean: For an async request, MDS should process it immediately. This is important for preserving request ordering.

Quoted 56 lines
Quoted 56 lines

I don't see the harm in making the client wait until it gets a create reply before sending a cap message. If we want to revert fbed7045f552 instead, we can do that, but it'll cause a regression until the MDS is fixed [1]. Regardless, we need to either take this patch or revert that one.

I move that we take this patch for now to address the softlockups. Once the MDS is fixed we could revert this and fbed7045f552 without causing a regression.

[1]: https://tracker.ceph.com/issues/54107

Quoted 23 lines · Zheng

The recent series of patches that add "wait on async xxxx" at various places do not seem correct. The correct fix should make mds avoid any wait when handling async requests.

Quoted 17 lines · Jeff Layton

Currently, we'll call ceph_check_caps, but if we're still waiting on the reply, we'll end up spinning around on the same inode in flush_dirty_session_caps. Wait for the async create reply before flushing caps.

Fixes fbed7045f552 (ceph: wait for async create reply before sending any cap messages)

URL: https://tracker.ceph.com/issues/55823

---
 fs/ceph/caps.c | 1 +
 1 file changed, 1 insertion(+)

I don't know if this will fix the tx queue stalls completely, but I haven't seen one with this patch in place. I think it makes sense on its own, either way.

Quoted 15 lines
Quoted 13 lines
fs/ceph/caps.c +1 -1
index 0a48bf829671..5ecfff4b37c9 100644
@@ -4389,6 +4389,7 @@ static void flush_dirty_session_caps(struct ceph_mds_session *s)
                ihold(inode);
                dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
                spin_unlock(&mdsc->cap_dirty_lock);
+               ceph_wait_on_async_create(inode);
                ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
                iput(inode);
                spin_lock(&mdsc->cap_dirty_lock);
--
Quoted 2 lines

2.36.1

--
Jeff Layton <[email protected]>