FAILED: patch "[PATCH] ceph: do not repeat ceph_trim_dentries() if no progress" failed to apply to 5.10-stable tree
Subsystems (2)
[email protected]
maintainerThe patch below does not apply to the 5.10-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
To reproduce the conflict and resubmit, you may use the following commands:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-5.10.y
git checkout FETCH_HEAD
git cherry-pick -x e7d7aa7b730178278109c41fa1b17b06873065d5
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2026090312-grinning-resurrect-3acf@gregkh' --subject-prefix 'PATCH 5.10.y' 'HEAD^..'
Possible dependencies:
thanks,
greg k-h
------------------ original commit in Linus's tree ------------------
From e7d7aa7b730178278109c41fa1b17b06873065d5 Mon Sep 17 00:00:00 2001
From: Max Kellermann <[email protected]>
Date: Tue, 7 Jul 2026 23:42:28 +0200
Subject: [PATCH] ceph: do not repeat ceph_trim_dentries() if no progress
possible
ceph_cap_reclaim_work() re-queues itself for as long as ceph_trim_dentries() returns -EAGAIN, which happens whenever a lease walk exhausts its nr_to_scan budget. This creates a busy loop that consumes CPU without making any progress when there is nothing to
reclaim: with no cap pressure (count==0) and every scanned lease still valid, each pass runs the full scan budget down to zero and returns -EAGAIN, only to be queued again immediately.
The dir-lease walk made this worse. When expire_dir_lease is false (i.e. we have no intention of reclaiming dir leases), __dir_lease_check() returned TOUCH for every valid lease. TOUCH moves the dentry to the tail of the list and resets di->time via __dentry_dir_lease_touch(), so a walk over N valid leases pointlessly rewrote the list, refreshed the timestamps (preventing them from ever aging out) and always drained nr_to_scan, guaranteeing the -EAGAIN requeue.
Fix this in three steps:
- Return KEEP instead of TOUCH when expire_dir_lease is false. If we are not going to reclaim the lease, leave it in place instead of churning the list and resetting its timestamp; the walk then terminates naturally (or via STOP at the first fresh lease).
- Only return -EAGAIN from the first (dentry-lease) walk when something was actually freed. A full batch that frees nothing means retrying the same list immediately is futile; fall through to the dir-lease walk instead.
- After both walks, bail out with success (0) when nothing was freed and there is no cap pressure (count==0). There is no reason to keep retrying when we are not over the cap limit and made no progress.
Under real cap pressure (count>0) the reclaim path is unchanged and still retries via -EAGAIN.
Without this patch, I saw 500 ceph_trim_dentries() calls per second on our web servers. This is very visible in /proc/lock_stat (5 minute capture):
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
&mdsc->dentry_list_lock: 126180 128218 0.04 8063.44 15986965.20 124.69 1573354 5296812 0.04 8291.28 74164526.48 14.00
-----------------------
&mdsc->dentry_list_lock 111736 [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8 &mdsc->dentry_list_lock 2631 [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8 &mdsc->dentry_list_lock 3878 [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8 &mdsc->dentry_list_lock 9973 [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0 ----------------------- &mdsc->dentry_list_lock 123621 [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8 &mdsc->dentry_list_lock 1822 [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8 &mdsc->dentry_list_lock 2720 [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0 &mdsc->dentry_list_lock 55 [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8
With this patch:
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
&mdsc->dentry_list_lock: 1203 1215 0.16 408.88 33082.88 27.23 4320501 7357389 0.04 500.64 1961578.00 0.27
-----------------------
&mdsc->dentry_list_lock 1029 [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8 &mdsc->dentry_list_lock 169 [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0 &mdsc->dentry_list_lock 16 [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8 &mdsc->dentry_list_lock 1 [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8 ----------------------- &mdsc->dentry_list_lock 158 [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0 &mdsc->dentry_list_lock 858 [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8 &mdsc->dentry_list_lock 182 [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8 &mdsc->dentry_list_lock 17 [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8
__dentry_leases_walk() is almost gone. The total wait time is reduced by a factor of 483. That will give some latency gains to ceph_readdir().
fs/ceph/dir.c +7 -3
@@ -1770,11 +1770,11 @@ static int __dir_lease_check(const struct dentry *dentry,if (ret >0 ) {if (time_before (jiffies, di->time + lwc->dir_lease_ttl))return STOP ; +if (!lwc->expire_dir_lease) +return KEEP ;/* Move dentry to tail of dir lease list if we don't want * to delete it. So dentries in the list are checked in a * round robin manner */ -if (!lwc->expire_dir_lease) -return TOUCH ;if (dentry->d_lockref.count >0 || (di->flags &CEPH_DENTRY_REFERENCED ))return TOUCH ;
@@ -1801,7 +1801,7 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc) lwc.dir_lease = false; lwc.nr_to_scan =CEPH_CAPS_PER_RELEASE *2 ; freed =__dentry_leases_walk (mdsc, &lwc); -if (!lwc.nr_to_scan)/* more invalid leases */ +if (freed >0 && !lwc.nr_to_scan)/* more invalid leases */ return -EAGAIN ;if (lwc.nr_to_scan <CEPH_CAPS_PER_RELEASE )
@@ -1811,6 +1811,10 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc) lwc.expire_dir_lease = freed < count; lwc.dir_lease_ttl = mdsc->fsc->mount_options->caps_wanted_delay_max *HZ ; freed +=__dentry_leases_walk (mdsc, &lwc); +if (freed ==0 && count ==0 ) +/* no progress possible currently, retry futile */ +return 0 ; +if (!lwc.nr_to_scan)/* more to check */ return -EAGAIN ;
Sasha Levin
maintainerPATCH 1/1 ceph: do not repeat ceph_trim_dentries() if no progress possible
From: Max Kellermann <[email protected]>
[ Upstream commit e7d7aa7b730178278109c41fa1b17b06873065d5 ]
ceph_cap_reclaim_work() re-queues itself for as long as ceph_trim_dentries() returns -EAGAIN, which happens whenever a lease walk exhausts its nr_to_scan budget. This creates a busy loop that consumes CPU without making any progress when there is nothing to
reclaim: with no cap pressure (count==0) and every scanned lease still valid, each pass runs the full scan budget down to zero and returns -EAGAIN, only to be queued again immediately.
The dir-lease walk made this worse. When expire_dir_lease is false (i.e. we have no intention of reclaiming dir leases), __dir_lease_check() returned TOUCH for every valid lease. TOUCH moves the dentry to the tail of the list and resets di->time via __dentry_dir_lease_touch(), so a walk over N valid leases pointlessly rewrote the list, refreshed the timestamps (preventing them from ever aging out) and always drained nr_to_scan, guaranteeing the -EAGAIN requeue.
Fix this in three steps:
- Return KEEP instead of TOUCH when expire_dir_lease is false. If we are not going to reclaim the lease, leave it in place instead of churning the list and resetting its timestamp; the walk then terminates naturally (or via STOP at the first fresh lease).
- Only return -EAGAIN from the first (dentry-lease) walk when something was actually freed. A full batch that frees nothing means retrying the same list immediately is futile; fall through to the dir-lease walk instead.
- After both walks, bail out with success (0) when nothing was freed and there is no cap pressure (count==0). There is no reason to keep retrying when we are not over the cap limit and made no progress.
Under real cap pressure (count>0) the reclaim path is unchanged and still retries via -EAGAIN.
Without this patch, I saw 500 ceph_trim_dentries() calls per second on our web servers. This is very visible in /proc/lock_stat (5 minute capture):
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
&mdsc->dentry_list_lock: 126180 128218 0.04 8063.44 15986965.20 124.69 1573354 5296812 0.04 8291.28 74164526.48 14.00
-----------------------
&mdsc->dentry_list_lock 111736 [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8 &mdsc->dentry_list_lock 2631 [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8 &mdsc->dentry_list_lock 3878 [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8 &mdsc->dentry_list_lock 9973 [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0 ----------------------- &mdsc->dentry_list_lock 123621 [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8 &mdsc->dentry_list_lock 1822 [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8 &mdsc->dentry_list_lock 2720 [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0 &mdsc->dentry_list_lock 55 [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8
With this patch:
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
&mdsc->dentry_list_lock: 1203 1215 0.16 408.88 33082.88 27.23 4320501 7357389 0.04 500.64 1961578.00 0.27
-----------------------
&mdsc->dentry_list_lock 1029 [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8 &mdsc->dentry_list_lock 169 [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0 &mdsc->dentry_list_lock 16 [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8 &mdsc->dentry_list_lock 1 [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8 ----------------------- &mdsc->dentry_list_lock 158 [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0 &mdsc->dentry_list_lock 858 [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8 &mdsc->dentry_list_lock 182 [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8 &mdsc->dentry_list_lock 17 [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8
__dentry_leases_walk() is almost gone. The total wait time is reduced by a factor of 483. That will give some latency gains to ceph_readdir().
--- fs/ceph/dir.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
fs/ceph/dir.c +7 -3
@@ -1483,11 +1483,11 @@ static int __dir_lease_check(struct dentry *dentry, void *arg)if (ret >0 ) {if (time_before (jiffies, di->time + lwc->dir_lease_ttl))return STOP ; +if (!lwc->expire_dir_lease) +return KEEP ;/* Move dentry to tail of dir lease list if we don't want * to delete it. So dentries in the list are checked in a * round robin manner */ -if (!lwc->expire_dir_lease) -return TOUCH ;if (dentry->d_lockref.count >0 || (di->flags &CEPH_DENTRY_REFERENCED ))return TOUCH ;
@@ -1514,7 +1514,7 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc) lwc.dir_lease = false; lwc.nr_to_scan =CEPH_CAPS_PER_RELEASE *2 ; freed =__dentry_leases_walk (mdsc, &lwc, __dentry_lease_check); -if (!lwc.nr_to_scan)/* more invalid leases */ +if (freed >0 && !lwc.nr_to_scan)/* more invalid leases */ return -EAGAIN ;if (lwc.nr_to_scan <CEPH_CAPS_PER_RELEASE )
@@ -1524,6 +1524,10 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc) lwc.expire_dir_lease = freed < count; lwc.dir_lease_ttl = mdsc->fsc->mount_options->caps_wanted_delay_max *HZ ; freed +=__dentry_leases_walk (mdsc, &lwc, __dir_lease_check); +if (freed ==0 && count ==0 ) +/* no progress possible currently, retry futile */ +return 0 ; +if (!lwc.nr_to_scan)/* more to check */ return -EAGAIN ;
Signature · 1 line
2.53.0