Skip to contentSkip to conversation

ceph: get snap_rwsem read lock in handle_cap_export for ceph_add_cap

PATCH Superseded Niels Dossche 3 participants 6 messages Mar 14, 2022 – Mar 15, 2022 Ceph client

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ceph: get snap_rwsem read lock in handle_cap_export for ceph_add_cap

ceph_add_cap says in its function documentation that the caller should hold the read lock on the session snap_rwsem. Furthermore, not only ceph_add_cap needs that lock, when it calls to ceph_lookup_snap_realm it eventually calls ceph_get_snap_realm which states via lockdep that snap_rwsem needs to be held. handle_cap_export calls ceph_add_cap without that mdsc->snap_rwsem held. Thus, since ceph_get_snap_realm and ceph_add_cap both need the lock, the common place to acquire that lock is inside handle_cap_export.

---
 fs/ceph/caps.c | 2 ++
 1 file changed, 2 insertions(+)
fs/ceph/caps.c +2 -0
index b472cd066d1c..0dd60db285b1 100644
@@ -3903,8 +3903,10 @@ static void handle_cap_export(struct inode *inode, struct ceph_mds_caps *ex,
 		/* add placeholder for the export tagert */
 		int flag = (cap == ci->i_auth_cap) ? CEPH_CAP_FLAG_AUTH : 0;
 		tcap = new_cap;
+		down_read(&mdsc->snap_rwsem);
 		ceph_add_cap(inode, tsession, t_cap_id, issued, 0,
 			     t_seq - 1, t_mseq, (u64)-1, flag, &new_cap);
+		up_read(&mdsc->snap_rwsem);
 
 		if (!list_empty(&ci->i_cap_flush_list) &&
 		    ci->i_auth_cap == tcap) {
Signature · 1 line
2.35.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 19 lines · Niels Dossche

ceph_add_cap says in its function documentation that the caller should hold the read lock on the session snap_rwsem. Furthermore, not only ceph_add_cap needs that lock, when it calls to ceph_lookup_snap_realm it eventually calls ceph_get_snap_realm which states via lockdep that snap_rwsem needs to be held. handle_cap_export calls ceph_add_cap without that mdsc->snap_rwsem held. Thus, since ceph_get_snap_realm and ceph_add_cap both need the lock, the common place to acquire that lock is inside handle_cap_export.

---
  fs/ceph/caps.c | 2 ++
  1 file changed, 2 insertions(+)
fs/ceph/caps.c +0 -0
index b472cd066d1c..0dd60db285b1 100644

fs/ceph/caps.c static void handle_cap_export(struct inode *inode, struct ceph_mds_caps *ex, · line 3912

Quoted 10 lines
fs/ceph/caps.c +2 -0
@@ -3903,8 +3903,10 @@ static void handle_cap_export(struct inode *inode, struct ceph_mds_caps *ex,
  		/* add placeholder for the export tagert */
  		int flag = (cap == ci->i_auth_cap) ? CEPH_CAP_FLAG_AUTH : 0;
  		tcap = new_cap;
+		down_read(&mdsc->snap_rwsem);
  		ceph_add_cap(inode, tsession, t_cap_id, issued, 0,
  			     t_seq - 1, t_mseq, (u64)-1, flag, &new_cap);
+		up_read(&mdsc->snap_rwsem);
 
  		if (!list_empty(&ci->i_cap_flush_list) &&
  		    ci->i_auth_cap == tcap) {

LGTM.

#2 lore raw
In reply to Niels Dossche
Quoted 19 lines · Niels Dossche

ceph_add_cap says in its function documentation that the caller should hold the read lock on the session snap_rwsem. Furthermore, not only ceph_add_cap needs that lock, when it calls to ceph_lookup_snap_realm it eventually calls ceph_get_snap_realm which states via lockdep that snap_rwsem needs to be held. handle_cap_export calls ceph_add_cap without that mdsc->snap_rwsem held. Thus, since ceph_get_snap_realm and ceph_add_cap both need the lock, the common place to acquire that lock is inside handle_cap_export.

---
 fs/ceph/caps.c | 2 ++
 1 file changed, 2 insertions(+)
fs/ceph/caps.c +0 -0
index b472cd066d1c..0dd60db285b1 100644

fs/ceph/caps.c static void handle_cap_export(struct inode *inode, struct ceph_mds_caps *ex, · line 3912

Quoted 10 lines
fs/ceph/caps.c +2 -0
@@ -3903,8 +3903,10 @@ static void handle_cap_export(struct inode *inode, struct ceph_mds_caps *ex,
 		/* add placeholder for the export tagert */
 		int flag = (cap == ci->i_auth_cap) ? CEPH_CAP_FLAG_AUTH : 0;
 		tcap = new_cap;
+		down_read(&mdsc->snap_rwsem);
 		ceph_add_cap(inode, tsession, t_cap_id, issued, 0,
 			     t_seq - 1, t_mseq, (u64)-1, flag, &new_cap);
+		up_read(&mdsc->snap_rwsem);
 
 		if (!list_empty(&ci->i_cap_flush_list) &&
 		    ci->i_auth_cap == tcap) {

Looks good. The other ceph_add_cap callsites already hold this.

Merged into ceph testing branch.

Thanks!

Signature · 1 line
Jeff Layton <[email protected]>
#3 lore raw
Quoted 28 lines · Jeff Layton
Quoted 27 lines · Niels Dossche

ceph_add_cap says in its function documentation that the caller should hold the read lock on the session snap_rwsem. Furthermore, not only ceph_add_cap needs that lock, when it calls to ceph_lookup_snap_realm it eventually calls ceph_get_snap_realm which states via lockdep that snap_rwsem needs to be held. handle_cap_export calls ceph_add_cap without that mdsc->snap_rwsem held. Thus, since ceph_get_snap_realm and ceph_add_cap both need the lock, the common place to acquire that lock is inside handle_cap_export.

---
 fs/ceph/caps.c | 2 ++
 1 file changed, 2 insertions(+)
fs/ceph/caps.c +2 -0
index b472cd066d1c..0dd60db285b1 100644
@@ -3903,8 +3903,10 @@ static void handle_cap_export(struct inode *inode, struct ceph_mds_caps *ex,
 		/* add placeholder for the export tagert */
 		int flag = (cap == ci->i_auth_cap) ? CEPH_CAP_FLAG_AUTH : 0;
 		tcap = new_cap;
+		down_read(&mdsc->snap_rwsem);
 		ceph_add_cap(inode, tsession, t_cap_id, issued, 0,
 			     t_seq - 1, t_mseq, (u64)-1, flag, &new_cap);
+		up_read(&mdsc->snap_rwsem);
 
Quoted 7 lines

fs/ceph/caps.c static void handle_cap_export(struct inode *inode, struct ceph_mds_caps *ex, · line 3912

Quoted 2 lines
fs/ceph/caps.c +0 -0
 		if (!list_empty(&ci->i_cap_flush_list) &&
 		    ci->i_auth_cap == tcap) {

Looks good. The other ceph_add_cap callsites already hold this.

Merged into ceph testing branch.

Oops, spoke too soon. This patch calls down_read (a potentially sleeping function) while holding the i_ceph_lock spinlock. I think you'll need to take the rwsem earlier in the function, before taking the spinlock.

Dropped from testing branch for now...

Signature · 1 line
Jeff Layton <[email protected]>
#4 lore raw
Quoted 38 lines · Jeff Layton
Quoted 28 lines · Jeff Layton
Quoted 27 lines · Niels Dossche

ceph_add_cap says in its function documentation that the caller should hold the read lock on the session snap_rwsem. Furthermore, not only ceph_add_cap needs that lock, when it calls to ceph_lookup_snap_realm it eventually calls ceph_get_snap_realm which states via lockdep that snap_rwsem needs to be held. handle_cap_export calls ceph_add_cap without that mdsc->snap_rwsem held. Thus, since ceph_get_snap_realm and ceph_add_cap both need the lock, the common place to acquire that lock is inside handle_cap_export.

---
 fs/ceph/caps.c | 2 ++
 1 file changed, 2 insertions(+)
fs/ceph/caps.c +2 -0
index b472cd066d1c..0dd60db285b1 100644
@@ -3903,8 +3903,10 @@ static void handle_cap_export(struct inode *inode, struct ceph_mds_caps *ex,
 		/* add placeholder for the export tagert */
 		int flag = (cap == ci->i_auth_cap) ? CEPH_CAP_FLAG_AUTH : 0;
 		tcap = new_cap;
+		down_read(&mdsc->snap_rwsem);
 		ceph_add_cap(inode, tsession, t_cap_id, issued, 0,
 			     t_seq - 1, t_mseq, (u64)-1, flag, &new_cap);
+		up_read(&mdsc->snap_rwsem);
 
Quoted 7 lines

fs/ceph/caps.c static void handle_cap_export(struct inode *inode, struct ceph_mds_caps *ex, · line 3912

Quoted 2 lines
fs/ceph/caps.c +0 -0
 		if (!list_empty(&ci->i_cap_flush_list) &&
 		    ci->i_auth_cap == tcap) {

Looks good. The other ceph_add_cap callsites already hold this.

Merged into ceph testing branch.

Quoted 5 lines

Oops, spoke too soon. This patch calls down_read (a potentially sleeping function) while holding the i_ceph_lock spinlock. I think you'll need to take the rwsem earlier in the function, before taking the spinlock.

Dropped from testing branch for now...

Ah my bad. I notice that handle_cap_export is actually called with the i_ceph_lock spinlock.
I can send a v2 which acquires the down_read lock just before the i_ceph_lock spinlock is taken (i.e. just under the retry label). Does that work for you? If so, I'll send a v2.
Thanks!

#5 lore raw
Quoted 43 lines · Niels Dossche
Quoted 42 lines · Jeff Layton
Quoted 28 lines · Jeff Layton
Quoted 27 lines · Niels Dossche

ceph_add_cap says in its function documentation that the caller should hold the read lock on the session snap_rwsem. Furthermore, not only ceph_add_cap needs that lock, when it calls to ceph_lookup_snap_realm it eventually calls ceph_get_snap_realm which states via lockdep that snap_rwsem needs to be held. handle_cap_export calls ceph_add_cap without that mdsc->snap_rwsem held. Thus, since ceph_get_snap_realm and ceph_add_cap both need the lock, the common place to acquire that lock is inside handle_cap_export.

---
 fs/ceph/caps.c | 2 ++
 1 file changed, 2 insertions(+)
fs/ceph/caps.c +2 -0
index b472cd066d1c..0dd60db285b1 100644
@@ -3903,8 +3903,10 @@ static void handle_cap_export(struct inode *inode, struct ceph_mds_caps *ex,
 		/* add placeholder for the export tagert */
 		int flag = (cap == ci->i_auth_cap) ? CEPH_CAP_FLAG_AUTH : 0;
 		tcap = new_cap;
+		down_read(&mdsc->snap_rwsem);
 		ceph_add_cap(inode, tsession, t_cap_id, issued, 0,
 			     t_seq - 1, t_mseq, (u64)-1, flag, &new_cap);
+		up_read(&mdsc->snap_rwsem);
 
Quoted 7 lines

fs/ceph/caps.c static void handle_cap_export(struct inode *inode, struct ceph_mds_caps *ex, · line 3912

Quoted 2 lines
fs/ceph/caps.c +0 -0
 		if (!list_empty(&ci->i_cap_flush_list) &&
 		    ci->i_auth_cap == tcap) {

Looks good. The other ceph_add_cap callsites already hold this.

Merged into ceph testing branch.

Oops, spoke too soon. This patch calls down_read (a potentially sleeping function) while holding the i_ceph_lock spinlock. I think you'll need to take the rwsem earlier in the function, before taking the spinlock.

Quoted 6 lines
Quoted 1 line

Dropped from testing branch for now...

Ah my bad. I notice that handle_cap_export is actually called with the i_ceph_lock spinlock.
I can send a v2 which acquires the down_read lock just before the i_ceph_lock spinlock is taken (i.e. just under the retry label). Does that work for you? If so, I'll send a v2.
Thanks!

That should be fine.

Signature · 1 line
Jeff Layton <[email protected]>