Skip to contentSkip to conversation

fs/ceph/dir: do not repeat ceph_trim_dentries() if no progress possible

PATCH In review Max Kellermann 2 participants 2 messages Jul 7 – Aug 11

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fs/ceph/dir: do not repeat ceph_trim_dentries() if no progress possible

ceph_cap_reclaim_work() re-queues itself for as long as ceph_trim_dentries() returns -EAGAIN, which happens whenever a lease walk exhausts its nr_to_scan budget. This creates a busy loop that consumes CPU without making any progress when there is nothing to
reclaim: with no cap pressure (count==0) and every scanned lease still valid, each pass runs the full scan budget down to zero and returns -EAGAIN, only to be queued again immediately.

The dir-lease walk made this worse. When expire_dir_lease is false (i.e. we have no intention of reclaiming dir leases), __dir_lease_check() returned TOUCH for every valid lease. TOUCH moves the dentry to the tail of the list and resets di->time via __dentry_dir_lease_touch(), so a walk over N valid leases pointlessly rewrote the list, refreshed the timestamps (preventing them from ever aging out) and always drained nr_to_scan, guaranteeing the -EAGAIN requeue.

Fix this in three steps:

- Return KEEP instead of TOUCH when expire_dir_lease is false. If we are not going to reclaim the lease, leave it in place instead of churning the list and resetting its timestamp; the walk then terminates naturally (or via STOP at the first fresh lease).

- Only return -EAGAIN from the first (dentry-lease) walk when something was actually freed. A full batch that frees nothing means retrying the same list immediately is futile; fall through to the dir-lease walk instead.

- After both walks, bail out with success (0) when nothing was freed and there is no cap pressure (count==0). There is no reason to keep retrying when we are not over the cap limit and made no progress.

Under real cap pressure (count>0) the reclaim path is unchanged and still retries via -EAGAIN.

Without this patch, I saw 500 ceph_trim_dentries() calls per second on our web servers. This is very visible in /proc/lock_stat (5 minute capture):

class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg

&mdsc->dentry_list_lock: 126180 128218 0.04 8063.44 15986965.20 124.69 1573354 5296812 0.04 8291.28 74164526.48 14.00
-----------------------

 &mdsc->dentry_list_lock         111736          [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
 &mdsc->dentry_list_lock           2631          [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8
 &mdsc->dentry_list_lock           3878          [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8
 &mdsc->dentry_list_lock           9973          [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0
 -----------------------
 &mdsc->dentry_list_lock         123621          [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8
 &mdsc->dentry_list_lock           1822          [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
 &mdsc->dentry_list_lock           2720          [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0
 &mdsc->dentry_list_lock             55          [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8

With this patch:

class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg

&mdsc->dentry_list_lock: 1203 1215 0.16 408.88 33082.88 27.23 4320501 7357389 0.04 500.64 1961578.00 0.27
-----------------------

 &mdsc->dentry_list_lock           1029          [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
 &mdsc->dentry_list_lock            169          [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0
 &mdsc->dentry_list_lock             16          [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8
 &mdsc->dentry_list_lock              1          [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8
 -----------------------
 &mdsc->dentry_list_lock            158          [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0
 &mdsc->dentry_list_lock            858          [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
 &mdsc->dentry_list_lock            182          [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8
 &mdsc->dentry_list_lock             17          [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8

__dentry_leases_walk() is almost gone. The total wait time is reduced by a factor of 483. That will give some latency gains to ceph_readdir().

Fixes 37c4efc1ddf9 ("ceph: periodically trim stale dentries")
---
 fs/ceph/dir.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
fs/ceph/dir.c +7 -3
index 27ce9e55e947..45cdb611fe2a 100644
@@ -1759,11 +1759,11 @@ static int __dir_lease_check(const struct dentry *dentry,
 	if (ret > 0) {
 		if (time_before(jiffies, di->time + lwc->dir_lease_ttl))
 			return STOP;
+		if (!lwc->expire_dir_lease)
+			return KEEP;
 		/* Move dentry to tail of dir lease list if we don't want
 		 * to delete it. So dentries in the list are checked in a
 		 * round robin manner */
-		if (!lwc->expire_dir_lease)
-			return TOUCH;
 		if (dentry->d_lockref.count > 0 ||
 		    (di->flags & CEPH_DENTRY_REFERENCED))
 			return TOUCH;
@@ -1790,7 +1790,7 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc)
 	lwc.dir_lease = false;
 	lwc.nr_to_scan  = CEPH_CAPS_PER_RELEASE * 2;
 	freed = __dentry_leases_walk(mdsc, &lwc);
-	if (!lwc.nr_to_scan) /* more invalid leases */
+	if (freed > 0 && !lwc.nr_to_scan) /* more invalid leases */
 		return -EAGAIN;
 
 	if (lwc.nr_to_scan < CEPH_CAPS_PER_RELEASE)
@@ -1800,6 +1800,10 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc)
 	lwc.expire_dir_lease = freed < count;
 	lwc.dir_lease_ttl = mdsc->fsc->mount_options->caps_wanted_delay_max * HZ;
 	freed +=__dentry_leases_walk(mdsc, &lwc);
+	if (freed == 0 && count == 0)
+		/* no progress possible currently, retry futile */
+		return 0;
+
 	if (!lwc.nr_to_scan) /* more to check */
 		return -EAGAIN;
 
Signature · 1 line
2.47.3