Skip to contentSkip to conversation

ceph: cancel delayed work instead of flushing on mdsc teardown

PATCH Superseded Jeff Layton 2 participants 3 messages Jul 27, 2021 – Jul 29, 2021 Ceph client

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ceph: cancel delayed work instead of flushing on mdsc teardown

The first thing metric_delayed_work does is check mdsc->stopping, and then return immediately if it's set...which is good since we would have already torn down the metric structures at this point, otherwise.

Worse yet, it's possible that the ceph_metric_destroy call could race with the delayed_work, in which case we could end up a end up accessing destroyed percpu variables.

At this point in the mdsc teardown, the "stopping" flag has already been set, so there's no benefit to flushing the work. Just cancel it instead, and do so before we tear down the metrics structures.

---
 fs/ceph/mds_client.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/ceph/mds_client.c +1 -1
index c43091a30ba8..d3f2baf3c352 100644
@@ -4977,9 +4977,9 @@ void ceph_mdsc_destroy(struct ceph_fs_client *fsc)
 
 	ceph_mdsc_stop(mdsc);
 
+	cancel_delayed_work_sync(&mdsc->metric.delayed_work);
 	ceph_metric_destroy(&mdsc->metric);
 
-	flush_delayed_work(&mdsc->metric.delayed_work);
 	fsc->mdsc = NULL;
 	kfree(mdsc);
 	dout("mdsc_destroy %p done\n", mdsc);
Signature · 1 line
2.31.1
#1 lore raw
Quoted 23 lines · Jeff Layton

The first thing metric_delayed_work does is check mdsc->stopping, and then return immediately if it's set...which is good since we would have already torn down the metric structures at this point, otherwise.

Worse yet, it's possible that the ceph_metric_destroy call could race with the delayed_work, in which case we could end up a end up accessing destroyed percpu variables.

At this point in the mdsc teardown, the "stopping" flag has already been set, so there's no benefit to flushing the work. Just cancel it instead, and do so before we tear down the metrics structures.

---
  fs/ceph/mds_client.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
fs/ceph/mds_client.c +0 -0
index c43091a30ba8..d3f2baf3c352 100644

fs/ceph/mds_client.c void ceph_mdsc_destroy(struct ceph_fs_client *fsc) · line 4982

Quoted 6 lines
fs/ceph/mds_client.c +1 -0
@@ -4977,9 +4977,9 @@ void ceph_mdsc_destroy(struct ceph_fs_client *fsc)
 
  	ceph_mdsc_stop(mdsc);
 
+	cancel_delayed_work_sync(&mdsc->metric.delayed_work);
  	ceph_metric_destroy(&mdsc->metric);
 

In the "ceph_metric_destroy()" it will also do "cancel_delayed_work_sync(&mdsc->metric.delayed_work)".

We can just move the it to the front of the _destory().

Quoted 4 lines
fs/ceph/mds_client.c +0 -1
-	flush_delayed_work(&mdsc->metric.delayed_work);
  	fsc->mdsc = NULL;
  	kfree(mdsc);
  	dout("mdsc_destroy %p done\n", mdsc);
#2 lore raw
Quoted 27 lines · Xiubo Li
Quoted 26 lines · Jeff Layton

The first thing metric_delayed_work does is check mdsc->stopping, and then return immediately if it's set...which is good since we would have already torn down the metric structures at this point, otherwise.

Worse yet, it's possible that the ceph_metric_destroy call could race with the delayed_work, in which case we could end up a end up accessing destroyed percpu variables.

At this point in the mdsc teardown, the "stopping" flag has already been set, so there's no benefit to flushing the work. Just cancel it instead, and do so before we tear down the metrics structures.

---
  fs/ceph/mds_client.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
fs/ceph/mds_client.c +0 -0
index c43091a30ba8..d3f2baf3c352 100644
@@ -4977,9 +4977,9 @@ void ceph_mdsc_destroy(struct ceph_fs_client *fsc)
 
  	ceph_mdsc_stop(mdsc);
 
Quoted 10 lines

fs/ceph/mds_client.c void ceph_mdsc_destroy(struct ceph_fs_client *fsc) · line 4982

Quoted 3 lines
fs/ceph/mds_client.c +1 -0
+	cancel_delayed_work_sync(&mdsc->metric.delayed_work);
  	ceph_metric_destroy(&mdsc->metric);
 

In the "ceph_metric_destroy()" it will also do "cancel_delayed_work_sync(&mdsc->metric.delayed_work)".

We can just move the it to the front of the _destory().

Good point! I'll send a v2 after I test it out.

Quoted 6 lines

fs/ceph/mds_client.c void ceph_mdsc_destroy(struct ceph_fs_client *fsc) · line 4985

Quoted 5 lines
fs/ceph/mds_client.c +0 -1
-	flush_delayed_work(&mdsc->metric.delayed_work);
  	fsc->mdsc = NULL;
  	kfree(mdsc);
  	dout("mdsc_destroy %p done\n", mdsc);

Thanks,

Signature · 1 line
Jeff Layton <[email protected]>