use refcount+RCU method to implement lockless slab shrink
Subsystems (23)
Qi Zheng
reviewercover use refcount+RCU method to implement lockless slab shrink
From: Qi Zheng <[email protected]>
Hi all,
1. Background
=============
We used to implement the lockless slab shrink with SRCU [1], but then kernel test robot reported -88.8% regression in stress-ng.ramfs.ops_per_sec test case [2], so we reverted it [3].
This patch series aims to re-implement the lockless slab shrink using the refcount+RCU method proposed by Dave Chinner [4].
[1]. https://lore.kernel.org/lkml/[email protected]/
[2]. https://lore.kernel.org/lkml/[email protected]/
[3]. https://lore.kernel.org/all/[email protected]/
[4]. https://lore.kernel.org/lkml/[email protected]/
2. Implementation
=================
Currently, the shrinker instances can be divided into the following three types:
a) global shrinker instance statically defined in the kernel, such as workingset_shadow_shrinker.
b) global shrinker instance statically defined in the kernel modules, such as mmu_shrinker in x86.
c) shrinker instance embedded in other structures.
For *case a*, the memory of shrinker instance is never freed. For *case b*, the memory of shrinker instance will be freed after the module is unloaded. But we will call synchronize_rcu() in free_module() to wait for RCU read-side critical section to exit. For *case c*, we need to dynamically allocate these shrinker instances, then the memory of shrinker instance can be dynamically freed alone by calling kfree_rcu(). Then we can use rcu_read_{lock,unlock}() to ensure that the shrinker instance is valid.
The shrinker::refcount mechanism ensures that the shrinker instance will not be run again after unregistration. So the structure that records the pointer of shrinker instance can be safely freed without waiting for the RCU read-side critical section.
In this way, while we implement the lockless slab shrink, we don't need to be blocked in unregister_shrinker() to wait RCU read-side critical section.
PATCH 1 ~ 2: infrastructure for dynamically allocating shrinker instances PATCH 3 ~ 21: dynamically allocate the shrinker instances in case c PATCH 22: introduce pool_shrink_rwsem to implement private synchronize_shrinkers() PATCH 23 ~ 28: implement the lockless slab shrink
PATCH 29: move shrinker-related code into a separate file
3. Testing
==========
3.1 slab shrink stress test
---------------------------
We can reproduce the down_read_trylock() hotspot through the following script:
```
DIR="/root/shrinker/memcg/mnt"
do_create()
{
mkdir -p /sys/fs/cgroup/memory/test
mkdir -p /sys/fs/cgroup/perf_event/test
echo 4G > /sys/fs/cgroup/memory/test/memory.limit_in_bytes
for i in `seq 0 $1`;
do
mkdir -p /sys/fs/cgroup/memory/test/$i;
echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs;
echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs;
mkdir -p $DIR/$i;
done
}
do_mount()
{
for i in `seq $1 $2`;
do
mount -t tmpfs $i $DIR/$i;
done
}
do_touch()
{
for i in `seq $1 $2`;
do
echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs;
echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs;
dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 &
done
}
case "$1" in
touch)
do_touch $2 $3
;;
test)
do_create 4000
do_mount 0 4000
do_touch 0 3000
;;
*)
exit 1
;;
esac
```Save the above script, then run test and touch commands. Then we can use the following perf command to view hotspots:
perf top -U -F 999 [-g]
1) Before applying this patchset:
35.34% [kernel] [k] down_read_trylock 18.44% [kernel] [k] shrink_slab 15.98% [kernel] [k] pv_native_safe_halt 15.08% [kernel] [k] up_read 5.33% [kernel] [k] idr_find 2.71% [kernel] [k] _find_next_bit 2.21% [kernel] [k] shrink_node 1.29% [kernel] [k] shrink_lruvec 0.66% [kernel] [k] do_shrink_slab 0.33% [kernel] [k] list_lru_count_one 0.33% [kernel] [k] __radix_tree_lookup 0.25% [kernel] [k] mem_cgroup_iter
- 82.19% 19.49% [kernel] [k] shrink_slab
- 62.00% shrink_slab
36.37% down_read_trylock
15.52% up_read
5.48% idr_find
3.38% _find_next_bit
+ 0.98% do_shrink_slab2) After applying this patchset:
46.83% [kernel] [k] shrink_slab 20.52% [kernel] [k] pv_native_safe_halt 8.85% [kernel] [k] do_shrink_slab 7.71% [kernel] [k] _find_next_bit 1.72% [kernel] [k] xas_descend 1.70% [kernel] [k] shrink_node 1.44% [kernel] [k] shrink_lruvec 1.43% [kernel] [k] mem_cgroup_iter 1.28% [kernel] [k] xas_load 0.89% [kernel] [k] super_cache_count 0.84% [kernel] [k] xas_start 0.66% [kernel] [k] list_lru_count_one
- 65.50% 40.44% [kernel] [k] shrink_slab
- 22.96% shrink_slab
13.11% _find_next_bit
- 9.91% do_shrink_slab
- 1.59% super_cache_count
0.92% list_lru_count_one
We can see that the first perf hotspot becomes shrink_slab, which is what we expect.
3.2 registeration and unregisteration stress test
-------------------------------------------------
Run the command below to test:
stress-ng --timeout 60 --times --verify --metrics-brief --ramfs 9 &
1) Before applying this patchset:
setting to a 60 second run per stressor
dispatching hogs: 9 ramfs
stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s
(secs) (secs) (secs) (real time) (usr+sys time)
ramfs 880623 60.02 7.71 226.93 14671.45 3753.09
ramfs:
1 System Management Interrupt
for a 60.03s run time:
5762.40s available CPU time
7.71s user time ( 0.13%)
226.93s system time ( 3.94%)
234.64s total time ( 4.07%) load average: 8.54 3.06 2.11
passed: 9: ramfs (9)
failed: 0
skipped: 0
successful run completed in 60.03s (1 min, 0.03 secs)
2) After applying this patchset:
setting to a 60 second run per stressor
dispatching hogs: 9 ramfs
stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s
(secs) (secs) (secs) (real time) (usr+sys time)
ramfs 847562 60.02 7.44 230.22 14120.66 3566.23
ramfs:
4 System Management Interrupts
for a 60.12s run time:
5771.95s available CPU time
7.44s user time ( 0.13%)
230.22s system time ( 3.99%)
237.66s total time ( 4.12%) load average: 8.18 2.43 0.84
passed: 9: ramfs (9)
failed: 0
skipped: 0
successful run completed in 60.12s (1 min, 0.12 secs)
We can see that the ops/s has hardly changed.
This series is based on next-20230613.
Comments and suggestions are welcome.
Thanks,
Qi.
Qi Zheng (29):
mm: shrinker: add shrinker::private_data field
mm: vmscan: introduce some helpers for dynamically allocating shrinker drm/i915: dynamically allocate the i915_gem_mm shrinker drm/msm: dynamically allocate the drm-msm_gem shrinker drm/panfrost: dynamically allocate the drm-panfrost shrinker
dm: dynamically allocate the dm-bufio shrinker
dm zoned: dynamically allocate the dm-zoned-meta shrinker md/raid5: dynamically allocate the md-raid5 shrinker
bcache: dynamically allocate the md-bcache shrinker
vmw_balloon: dynamically allocate the vmw-balloon shrinker
virtio_balloon: dynamically allocate the virtio-balloon shrinker
mbcache: dynamically allocate the mbcache shrinker
ext4: dynamically allocate the ext4-es shrinker
jbd2,ext4: dynamically allocate the jbd2-journal shrinker
NFSD: dynamically allocate the nfsd-client shrinker
NFSD: dynamically allocate the nfsd-reply shrinker
xfs: dynamically allocate the xfs-buf shrinker
xfs: dynamically allocate the xfs-inodegc shrinker
xfs: dynamically allocate the xfs-qm shrinker
zsmalloc: dynamically allocate the mm-zspool shrinker
fs: super: dynamically allocate the s_shrink
drm/ttm: introduce pool_shrink_rwsem
mm: shrinker: add refcount and completion_wait fields
mm: vmscan: make global slab shrink lockless
mm: vmscan: make memcg slab shrink lockless
mm: shrinker: make count and scan in shrinker debugfs lockless
mm: vmscan: hold write lock to reparent shrinker nr_deferred
mm: shrinkers: convert shrinker_rwsem to mutex
mm: shrinker: move shrinker-related code into a separate file
drivers/gpu/drm/i915/gem/i915_gem_shrinker.c | 27 +- drivers/gpu/drm/i915/i915_drv.h | 3 +- drivers/gpu/drm/msm/msm_drv.h | 2 +- drivers/gpu/drm/msm/msm_gem_shrinker.c | 25 +- drivers/gpu/drm/panfrost/panfrost_device.h | 2 +- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 24 +- drivers/gpu/drm/ttm/ttm_pool.c | 15 + drivers/md/bcache/bcache.h | 2 +- drivers/md/bcache/btree.c | 23 +- drivers/md/bcache/sysfs.c | 2 +- drivers/md/dm-bufio.c | 23 +- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- drivers/md/dm-zoned-metadata.c | 25 +- drivers/md/raid5.c | 28 +- drivers/md/raid5.h | 2 +- drivers/misc/vmw_balloon.c | 16 +- drivers/virtio/virtio_balloon.c | 26 +- fs/btrfs/super.c | 2 +- fs/ext4/ext4.h | 2 +- fs/ext4/extents_status.c | 21 +- fs/jbd2/journal.c | 32 +- fs/kernfs/mount.c | 2 +- fs/mbcache.c | 39 +- fs/nfsd/netns.h | 4 +- fs/nfsd/nfs4state.c | 20 +- fs/nfsd/nfscache.c | 33 +- fs/proc/root.c | 2 +- fs/super.c | 40 +- fs/xfs/xfs_buf.c | 25 +- fs/xfs/xfs_buf.h | 2 +- fs/xfs/xfs_icache.c | 27 +- fs/xfs/xfs_mount.c | 4 +- fs/xfs/xfs_mount.h | 2 +- fs/xfs/xfs_qm.c | 24 +- fs/xfs/xfs_qm.h | 2 +- include/linux/fs.h | 2 +- include/linux/jbd2.h | 2 +- include/linux/shrinker.h | 35 +- mm/Makefile | 4 +- mm/shrinker.c | 750 ++++++++++++++++++ mm/shrinker_debug.c | 26 +- mm/vmscan.c | 702 ---------------- mm/zsmalloc.c | 28 +- 44 files changed, 1128 insertions(+), 953 deletions(-) create mode 100644 mm/shrinker.c
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 1/29 mm: shrinker: add shrinker::private_data field
From: Qi Zheng <[email protected]>
To prepare for the dynamic allocation of shrinker instances embedded in other structures, add a private_data field to struct shrinker, so that we can use shrinker::private_data to record and get the original embedded structure.
--- include/linux/shrinker.h | 2 ++ 1 file changed, 2 insertions(+)
include/linux/shrinker.h +2 -0
@@ -70,6 +70,8 @@ struct shrinker {int seeks;/* seeks to recreate an obj */ unsigned flags; +void *private_data; +/* These are for internal use */ struct list_head list;#ifdef CONFIG_MEMCG
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 2/29 mm: vmscan: introduce some helpers for dynamically allocating shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
Introduce some helpers for dynamically allocating shrinker instance, and their uses are as follows:
1. shrinker_alloc_and_init()
Used to allocate and initialize a shrinker instance, the priv_data parameter is used to pass the pointer of the previously embedded structure of the shrinker instance.
2. shrinker_free()
Used to free the shrinker instance when the registration of shrinker fails.
3. unregister_and_free_shrinker()
Used to unregister and free the shrinker instance, and the kfree() will be changed to kfree_rcu() later.
--- include/linux/shrinker.h | 12 ++++++++++++ mm/vmscan.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+)
include/linux/shrinker.h +12 -0
@@ -107,6 +107,18 @@ extern void unregister_shrinker(struct shrinker *shrinker);extern void free_prealloced_shrinker (struct shrinker *shrinker);extern void synchronize_shrinkers (void ); +typedef unsigned long (*count_objects_cb )(struct shrinker *s, +struct shrink_control *sc); +typedef unsigned long (*scan_objects_cb )(struct shrinker *s, +struct shrink_control *sc); + +struct shrinker *shrinker_alloc_and_init (count_objects_cb count, +scan_objects_cb scan,long batch, +int seeks,unsigned flags, +void *priv_data); +void shrinker_free (struct shrinker *shrinker); +void unregister_and_free_shrinker (struct shrinker *shrinker); +#ifdef CONFIG_SHRINKER_DEBUG extern int shrinker_debugfs_add (struct shrinker *shrinker); extern struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker,
mm/vmscan.c +35 -0
@@ -809,6 +809,41 @@ void unregister_shrinker(struct shrinker *shrinker) }EXPORT_SYMBOL (unregister_shrinker); +struct shrinker *shrinker_alloc_and_init (count_objects_cb count, +scan_objects_cb scan,long batch, +int seeks,unsigned flags, +void *priv_data) +{ +struct shrinker *shrinker; + + shrinker =kzalloc (sizeof (struct shrinker ),GFP_KERNEL ); +if (!shrinker) +return NULL ; + + shrinker->count_objects = count; + shrinker->scan_objects = scan; + shrinker->batch = batch; + shrinker->seeks = seeks; + shrinker->flags = flags; + shrinker->private_data = priv_data; + +return shrinker; +} +EXPORT_SYMBOL (shrinker_alloc_and_init); + +void shrinker_free (struct shrinker *shrinker) +{ +kfree (shrinker); +} +EXPORT_SYMBOL (shrinker_free); + +void unregister_and_free_shrinker (struct shrinker *shrinker) +{ +unregister_shrinker (shrinker); +kfree (shrinker); +} +EXPORT_SYMBOL (unregister_and_free_shrinker); +/** * synchronize_shrinkers - Wait for all running shrinkers to complete. *
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 3/29 drm/i915: dynamically allocate the i915_gem_mm shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the i915_gem_mm shrinker, so that it can be freed asynchronously by using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct drm_i915_private.
--- drivers/gpu/drm/i915/gem/i915_gem_shrinker.c | 27 ++++++++++---------- drivers/gpu/drm/i915/i915_drv.h | 3 ++- 2 files changed, 16 insertions(+), 14 deletions(-)
drivers/gpu/drm/i915/gem/i915_gem_shrinker.c +14 -13
@@ -284,8 +284,7 @@ unsigned long i915_gem_shrink_all(struct drm_i915_private *i915)static unsigned long i915_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) { -struct drm_i915_private *i915 = - container_of(shrinker, struct drm_i915_private, mm.shrinker); +struct drm_i915_private *i915 = shrinker->private_data;unsigned long num_objects;unsigned long count;
@@ -302,8 +301,8 @@ i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc)if (num_objects) {unsigned long avg =2 * count / num_objects; - i915->mm.shrinker.batch = -max ((i915->mm.shrinker.batch + avg) >>1 , + i915->mm.shrinker->batch = +max ((i915->mm.shrinker->batch + avg) >>1 ,128ul /* default SHRINK_BATCH */ ); }
@@ -313,8 +312,7 @@ i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc)static unsigned long i915_gem_shrinker_scan (struct shrinker *shrinker,struct shrink_control *sc) { -struct drm_i915_private *i915 = - container_of(shrinker, struct drm_i915_private, mm.shrinker); +struct drm_i915_private *i915 = shrinker->private_data;unsigned long freed; sc->nr_scanned =0 ;
@@ -422,12 +420,15 @@ i915_gem_shrinker_vmap(struct notifier_block *nb, unsigned long event, void *ptrvoid i915_gem_driver_register__shrinker (struct drm_i915_private *i915) { - i915->mm.shrinker.scan_objects = i915_gem_shrinker_scan; - i915->mm.shrinker.count_objects = i915_gem_shrinker_count; - i915->mm.shrinker.seeks =DEFAULT_SEEKS ; - i915->mm.shrinker.batch =4096 ; -drm_WARN_ON (&i915->drm,register_shrinker (&i915->mm.shrinker, -"drm-i915_gem" )); + i915->mm.shrinker =shrinker_alloc_and_init (i915_gem_shrinker_count, + i915_gem_shrinker_scan, +4096 ,DEFAULT_SEEKS ,0 , + i915); +if (i915->mm.shrinker && +register_shrinker (i915->mm.shrinker,"drm-i915_gem" )) { +shrinker_free (i915->mm.shrinker); +drm_WARN_ON (&i915->drm,1 ); + } i915->mm.oom_notifier.notifier_call = i915_gem_shrinker_oom;drm_WARN_ON (&i915->drm,register_oom_notifier (&i915->mm.oom_notifier));
@@ -443,7 +444,7 @@ void i915_gem_driver_unregister__shrinker(struct drm_i915_private *i915) unregister_vmap_purge_notifier(&i915->mm.vmap_notifier));drm_WARN_ON (&i915->drm,unregister_oom_notifier (&i915->mm.oom_notifier)); -unregister_shrinker (&i915->mm.shrinker); +unregister_and_free_shrinker (i915->mm.shrinker); } void i915_gem_shrinker_taints_mutex(struct drm_i915_private *i915,
drivers/gpu/drm/i915/i915_drv.h +2 -1
@@ -163,7 +163,8 @@ struct i915_gem_mm {struct notifier_block oom_notifier;struct notifier_block vmap_notifier; -struct shrinker shrinker; + +struct shrinker *shrinker;#ifdef CONFIG_MMU_NOTIFIER /**
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 4/29 drm/msm: dynamically allocate the drm-msm_gem shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the drm-msm_gem shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct msm_drm_private.
--- drivers/gpu/drm/msm/msm_drv.h | 2 +- drivers/gpu/drm/msm/msm_gem_shrinker.c | 25 ++++++++++++++----------- 2 files changed, 15 insertions(+), 12 deletions(-)
drivers/gpu/drm/msm/msm_drv.h +1 -1
@@ -217,7 +217,7 @@ struct msm_drm_private { } vram;struct notifier_block vmap_notifier; -struct shrinker shrinker; +struct shrinker *shrinker;struct drm_atomic_state *pm_state;
drivers/gpu/drm/msm/msm_gem_shrinker.c +14 -11
@@ -34,8 +34,7 @@ static bool can_block(struct shrink_control *sc)static unsigned long msm_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) { -struct msm_drm_private *priv = - container_of(shrinker, struct msm_drm_private, shrinker); +struct msm_drm_private *priv = shrinker->private_data;unsigned count = priv->lru.dontneed.count;if (can_swap ())
@@ -100,8 +99,7 @@ active_evict(struct drm_gem_object *obj)static unsigned long msm_gem_shrinker_scan (struct shrinker *shrinker,struct shrink_control *sc) { -struct msm_drm_private *priv = - container_of(shrinker, struct msm_drm_private, shrinker); +struct msm_drm_private *priv = shrinker->private_data;struct {struct drm_gem_lru *lru;bool (*shrink)(struct drm_gem_object *obj);
@@ -151,7 +149,7 @@ msm_gem_shrinker_shrink(struct drm_device *dev, unsigned long nr_to_scan)int ret;fs_reclaim_acquire (GFP_KERNEL ); - ret =msm_gem_shrinker_scan (&priv->shrinker, &sc); + ret =msm_gem_shrinker_scan (priv->shrinker, &sc);fs_reclaim_release (GFP_KERNEL );return ret;
@@ -213,10 +211,15 @@ msm_gem_shrinker_vmap(struct notifier_block *nb, unsigned long event, void *ptr)void msm_gem_shrinker_init (struct drm_device *dev) {struct msm_drm_private *priv = dev->dev_private; - priv->shrinker.count_objects = msm_gem_shrinker_count; - priv->shrinker.scan_objects = msm_gem_shrinker_scan; - priv->shrinker.seeks =DEFAULT_SEEKS ; -WARN_ON (register_shrinker (&priv->shrinker,"drm-msm_gem" )); + + priv->shrinker =shrinker_alloc_and_init (msm_gem_shrinker_count, + msm_gem_shrinker_scan,0 , +DEFAULT_SEEKS ,0 , priv); +if (priv->shrinker && +register_shrinker (priv->shrinker,"drm-msm_gem" )) { +shrinker_free (priv->shrinker); +WARN_ON (1 ); + } priv->vmap_notifier.notifier_call = msm_gem_shrinker_vmap;WARN_ON (register_vmap_purge_notifier (&priv->vmap_notifier));
@@ -232,8 +235,8 @@ void msm_gem_shrinker_cleanup(struct drm_device *dev) {struct msm_drm_private *priv = dev->dev_private; -if (priv->shrinker.nr_deferred) { +if (priv->shrinker->nr_deferred) {WARN_ON (unregister_vmap_purge_notifier (&priv->vmap_notifier)); -unregister_shrinker (&priv->shrinker); +unregister_and_free_shrinker (priv->shrinker); } }
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 5/29 drm/panfrost: dynamically allocate the drm-panfrost shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the drm-panfrost shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct panfrost_device.
--- drivers/gpu/drm/panfrost/panfrost_device.h | 2 +- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 24 ++++++++++--------- 2 files changed, 14 insertions(+), 12 deletions(-)
drivers/gpu/drm/panfrost/panfrost_device.h +1 -1
@@ -118,7 +118,7 @@ struct panfrost_device {struct mutex shrinker_lock;struct list_head shrinker_list; -struct shrinker shrinker; +struct shrinker *shrinker;struct panfrost_devfreq pfdevfreq; };
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +13 -11
@@ -18,8 +18,7 @@static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) { -struct panfrost_device *pfdev = - container_of(shrinker, struct panfrost_device, shrinker); +struct panfrost_device *pfdev = shrinker->private_data;struct drm_gem_shmem_object *shmem;unsigned long count =0 ;
@@ -65,8 +64,7 @@ static bool panfrost_gem_purge(struct drm_gem_object *obj)static unsigned long panfrost_gem_shrinker_scan (struct shrinker *shrinker,struct shrink_control *sc) { -struct panfrost_device *pfdev = - container_of(shrinker, struct panfrost_device, shrinker); +struct panfrost_device *pfdev = shrinker->private_data;struct drm_gem_shmem_object *shmem, *tmp;unsigned long freed =0 ;
@@ -100,10 +98,15 @@ panfrost_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc)void panfrost_gem_shrinker_init (struct drm_device *dev) {struct panfrost_device *pfdev = dev->dev_private; - pfdev->shrinker.count_objects = panfrost_gem_shrinker_count; - pfdev->shrinker.scan_objects = panfrost_gem_shrinker_scan; - pfdev->shrinker.seeks =DEFAULT_SEEKS ; -WARN_ON (register_shrinker (&pfdev->shrinker,"drm-panfrost" )); + + pfdev->shrinker =shrinker_alloc_and_init (panfrost_gem_shrinker_count, + panfrost_gem_shrinker_scan,0 , +DEFAULT_SEEKS ,0 , pfdev); +if (pfdev->shrinker && +register_shrinker (pfdev->shrinker,"drm-panfrost" )) { +shrinker_free (pfdev->shrinker); +WARN_ON (1 ); + } }/**
@@ -116,7 +119,6 @@ void panfrost_gem_shrinker_cleanup(struct drm_device *dev) {struct panfrost_device *pfdev = dev->dev_private; -if (pfdev->shrinker.nr_deferred) { -unregister_shrinker (&pfdev->shrinker); - } +if (pfdev->shrinker->nr_deferred) +unregister_and_free_shrinker (pfdev->shrinker); }
Signature · 1 line
2.30.2
Steven Price
maintainerRe: [PATCH 05/29] drm/panfrost: dynamically allocate the drm-panfrost shrinker
Quoted 58 lines · Qi Zheng
From: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the drm-panfrost shrinker, so that it can be freed asynchronously using
kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the structpanfrost_device.Signed-off-by Qi Zheng <[email protected]>--- drivers/gpu/drm/panfrost/panfrost_device.h | 2 +- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 24 ++++++++++--------- 2 files changed, 14 insertions(+), 12 deletions(-)drivers/gpu/drm/panfrost/panfrost_device.h +1
-1@@ -118,7 +118,7 @@ struct panfrost_device {struct mutex shrinker_lock;struct list_head shrinker_list; -struct shrinker shrinker; +struct shrinker *shrinker;struct panfrost_devfreq pfdevfreq; };drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +2
-7@@ -18,8 +18,7 @@static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) { -struct panfrost_device *pfdev = - container_of(shrinker, struct panfrost_device, shrinker); +struct panfrost_device *pfdev = shrinker->private_data;struct drm_gem_shmem_object *shmem;unsigned long count =0 ;@@ -65,8 +64,7 @@ static bool panfrost_gem_purge(struct drm_gem_object *obj)static unsigned long panfrost_gem_shrinker_scan (struct shrinker *shrinker,struct shrink_control *sc) { -struct panfrost_device *pfdev = - container_of(shrinker, struct panfrost_device, shrinker); +struct panfrost_device *pfdev = shrinker->private_data;struct drm_gem_shmem_object *shmem, *tmp;unsigned long freed =0 ;@@ -100,10 +98,15 @@ panfrost_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc)void panfrost_gem_shrinker_init (struct drm_device *dev) {struct panfrost_device *pfdev = dev->dev_private; - pfdev->shrinker.count_objects = panfrost_gem_shrinker_count; - pfdev->shrinker.scan_objects = panfrost_gem_shrinker_scan; - pfdev->shrinker.seeks =DEFAULT_SEEKS ;
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c panfrost_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) · line 109
Quoted 10 lines
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +9
-1-WARN_ON (register_shrinker (&pfdev->shrinker,"drm-panfrost" )); + + pfdev->shrinker =shrinker_alloc_and_init (panfrost_gem_shrinker_count, + panfrost_gem_shrinker_scan,0 , +DEFAULT_SEEKS ,0 , pfdev); +if (pfdev->shrinker && +register_shrinker (pfdev->shrinker,"drm-panfrost" )) { +shrinker_free (pfdev->shrinker); +WARN_ON (1 ); + }
So we didn't have good error handling here before, but this is significantly worse. Previously if register_shrinker() failed then the driver could safely continue without a shrinker - it would waste memory but still function.
However we now have two failure conditions:
* shrinker_alloc_init() returns NULL. No warning and NULL deferences will happen later.
* register_shrinker() fails, shrinker_free() will free pdev->shrinker we get a warning, but followed by a use-after-free later.
I think we need to modify panfrost_gem_shrinker_init() to be able to return an error, so a change something like the below (untested) before your change.
Steve
----8<---
(quoted diff) +0 -0
b/drivers/gpu/drm/panfrost/panfrost_drv.c
index bbada731bbbd..f705bbdea360 100644
drivers/gpu/drm/panfrost/panfrost_drv.c +0 -0
@@ -598,10 +598,14 @@ static int panfrost_probe(struct platform_device
*pdev)
if (err < 0)
goto err_out1;
- panfrost_gem_shrinker_init(ddev);
+ err = panfrost_gem_shrinker_init(ddev);
+ if (err)
+ goto err_out2;
return 0;
+err_out2:
+ drm_dev_unregister(ddev);
err_out1:
pm_runtime_disable(pfdev->dev);
panfrost_device_fini(pfdev);
(quoted diff) +0 -0
b/drivers/gpu/drm/panfrost/panfrost_gem.h
index ad2877eeeccd..863d2ec8d4f0 100644
drivers/gpu/drm/panfrost/panfrost_gem.h +1 -1
@@ -81,7 +81,7 @@ panfrost_gem_mapping_get(struct panfrost_gem_object *bo,void panfrost_gem_mapping_put (struct panfrost_gem_mapping *mapping);void panfrost_gem_teardown_mappings_locked (struct panfrost_gem_object *bo); -void panfrost_gem_shrinker_init (struct drm_device *dev); +int panfrost_gem_shrinker_init (struct drm_device *dev);void panfrost_gem_shrinker_cleanup (struct drm_device *dev);#endif /* __PANFROST_GEM_H__ */
(quoted diff) +0 -0
b/drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c index bf0170782f25..90265b37636f 100644
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +0 -0
@@ -97,13 +97,17 @@ panfrost_gem_shrinker_scan(struct shrinker
*shrinker, struct shrink_control *sc)
*
* This function registers and sets up the panfrost shrinker. */
-void panfrost_gem_shrinker_init(struct drm_device *dev) +int panfrost_gem_shrinker_init(struct drm_device *dev)
{
struct panfrost_device *pfdev = dev->dev_private;
+ int ret;
+
pfdev->shrinker.count_objects = panfrost_gem_shrinker_count; pfdev->shrinker.scan_objects = panfrost_gem_shrinker_scan; pfdev->shrinker.seeks = DEFAULT_SEEKS;
- WARN_ON(register_shrinker(&pfdev->shrinker, "drm-panfrost"));
+ ret = register_shrinker(&pfdev->shrinker, "drm-panfrost");
+
+ return ret;
}
/**
Qi Zheng
reviewerPATCH 6/29 dm: dynamically allocate the dm-bufio shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the dm-bufio shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct dm_bufio_client.
--- drivers/md/dm-bufio.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-)
drivers/md/dm-bufio.c +13 -10
@@ -963,7 +963,7 @@ struct dm_bufio_client {sector_t start; -struct shrinker shrinker; +struct shrinker *shrinker;struct work_struct shrink_work;atomic_long_t need_shrink;
@@ -2385,7 +2385,7 @@ static unsigned long dm_bufio_shrink_scan(struct shrinker *shrink, struct shrink {struct dm_bufio_client *c; - c = container_of(shrink, struct dm_bufio_client, shrinker); + c = shrink->private_data;atomic_long_add (sc->nr_to_scan, &c->need_shrink);queue_work (dm_bufio_wq, &c->shrink_work);
@@ -2394,7 +2394,7 @@ static unsigned long dm_bufio_shrink_scan(struct shrinker *shrink, struct shrinkstatic unsigned long dm_bufio_shrink_count (struct shrinker *shrink,struct shrink_control *sc) { - struct dm_bufio_client *c = container_of(shrink, struct dm_bufio_client, shrinker); +struct dm_bufio_client *c = shrink->private_data;unsigned long count =cache_total (&c->cache);unsigned long retain_target =get_retain_buffers (c);unsigned long queued_for_cleanup =atomic_long_read (&c->need_shrink);
@@ -2507,14 +2507,15 @@ struct dm_bufio_client *dm_bufio_client_create(struct block_device *bdev, unsignINIT_WORK (&c->shrink_work, shrink_work);atomic_long_set (&c->need_shrink,0 ); - c->shrinker.count_objects = dm_bufio_shrink_count; - c->shrinker.scan_objects = dm_bufio_shrink_scan; - c->shrinker.seeks =1 ; - c->shrinker.batch =0 ; - r =register_shrinker (&c->shrinker,"dm-bufio:(%u:%u)" , + c->shrinker =shrinker_alloc_and_init (dm_bufio_shrink_count, + dm_bufio_shrink_scan,0 ,1 ,0 , c); +if (!c->shrinker) + goto bad; + + r =register_shrinker (c->shrinker,"dm-bufio:(%u:%u)" ,MAJOR (bdev->bd_dev),MINOR (bdev->bd_dev));if (r) - goto bad; + goto bad_shrinker;mutex_lock (&dm_bufio_clients_lock); dm_bufio_client_count++;
@@ -2524,6 +2525,8 @@ struct dm_bufio_client *dm_bufio_client_create(struct block_device *bdev, unsignreturn c; +bad_shrinker: +shrinker_free (c->shrinker); bad:while (!list_empty (&c->reserved_buffers)) {struct dm_buffer *b =list_to_buffer (c->reserved_buffers.next);
@@ -2554,7 +2557,7 @@ void dm_bufio_client_destroy(struct dm_bufio_client *c)drop_buffers (c); -unregister_shrinker (&c->shrinker); +unregister_and_free_shrinker (c->shrinker);flush_work (&c->shrink_work);mutex_lock (&dm_bufio_clients_lock);
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 7/29 dm zoned: dynamically allocate the dm-zoned-meta shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the dm-zoned-meta shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct dmz_metadata.
--- drivers/md/dm-zoned-metadata.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
drivers/md/dm-zoned-metadata.c +16 -9
@@ -187,7 +187,7 @@ struct dmz_metadata {struct rb_root mblk_rbtree;struct list_head mblk_lru_list;struct list_head mblk_dirty_list; -struct shrinker mblk_shrinker; +struct shrinker *mblk_shrinker;/* Zone allocation management */ struct mutex map_lock;
@@ -615,7 +615,7 @@ static unsigned long dmz_shrink_mblock_cache(struct dmz_metadata *zmd,static unsigned long dmz_mblock_shrinker_count (struct shrinker *shrink,struct shrink_control *sc) { - struct dmz_metadata *zmd = container_of(shrink, struct dmz_metadata, mblk_shrinker); +struct dmz_metadata *zmd = shrink->private_data;return atomic_read (&zmd->nr_mblks); }
@@ -626,7 +626,7 @@ static unsigned long dmz_mblock_shrinker_count(struct shrinker *shrink,static unsigned long dmz_mblock_shrinker_scan (struct shrinker *shrink,struct shrink_control *sc) { - struct dmz_metadata *zmd = container_of(shrink, struct dmz_metadata, mblk_shrinker); +struct dmz_metadata *zmd = shrink->private_data;unsigned long count;spin_lock (&zmd->mblk_lock);
@@ -2936,17 +2936,22 @@ int dmz_ctr_metadata(struct dmz_dev *dev, int num_dev,*/ zmd->min_nr_mblks =2 + zmd->nr_map_blocks + zmd->zone_nr_bitmap_blocks *16 ; zmd->max_nr_mblks = zmd->min_nr_mblks +512 ; - zmd->mblk_shrinker.count_objects = dmz_mblock_shrinker_count; - zmd->mblk_shrinker.scan_objects = dmz_mblock_shrinker_scan; - zmd->mblk_shrinker.seeks =DEFAULT_SEEKS ; + + zmd->mblk_shrinker =shrinker_alloc_and_init (dmz_mblock_shrinker_count, + dmz_mblock_shrinker_scan, +0 ,DEFAULT_SEEKS ,0 , zmd); +if (!zmd->mblk_shrinker) { +dmz_zmd_err (zmd,"allocate metadata cache shrinker failed" ); + goto err; + }/* Metadata cache shrinker */ - ret =register_shrinker (&zmd->mblk_shrinker,"dm-zoned-meta:(%u:%u)" , + ret =register_shrinker (zmd->mblk_shrinker,"dm-zoned-meta:(%u:%u)" ,MAJOR (dev->bdev->bd_dev),MINOR (dev->bdev->bd_dev));if (ret) {dmz_zmd_err (zmd,"Register metadata cache shrinker failed" ); - goto err; + goto err_shrinker; }dmz_zmd_info (zmd,"DM-Zoned metadata version %d" , zmd->sb_version);
@@ -2982,6 +2987,8 @@ int dmz_ctr_metadata(struct dmz_dev *dev, int num_dev, *metadata = zmd;return 0 ; +err_shrinker: +shrinker_free (zmd->mblk_shrinker); err:dmz_cleanup_metadata (zmd);kfree (zmd);
@@ -2995,7 +3002,7 @@ int dmz_ctr_metadata(struct dmz_dev *dev, int num_dev,*/ void dmz_dtr_metadata (struct dmz_metadata *zmd) { -unregister_shrinker (&zmd->mblk_shrinker); +unregister_and_free_shrinker (zmd->mblk_shrinker);dmz_cleanup_metadata (zmd);kfree (zmd); }
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 8/29 md/raid5: dynamically allocate the md-raid5 shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the md-raid5 shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct r5conf.
--- drivers/md/raid5.c | 28 +++++++++++++++++----------- drivers/md/raid5.h | 2 +- 2 files changed, 18 insertions(+), 12 deletions(-)
drivers/md/raid5.c +17 -11
@@ -7391,7 +7391,7 @@ static void free_conf(struct r5conf *conf)log_exit (conf); -unregister_shrinker (&conf->shrinker); +unregister_and_free_shrinker (conf->shrinker);free_thread_groups (conf);shrink_stripes (conf);raid5_free_percpu (conf);
@@ -7439,7 +7439,7 @@ static int raid5_alloc_percpu(struct r5conf *conf)static unsigned long raid5_cache_scan (struct shrinker *shrink,struct shrink_control *sc) { - struct r5conf *conf = container_of(shrink, struct r5conf, shrinker); +struct r5conf *conf = shrink->private_data;unsigned long ret =SHRINK_STOP ;if (mutex_trylock (&conf->cache_size_mutex)) {
@@ -7460,7 +7460,7 @@ static unsigned long raid5_cache_scan(struct shrinker *shrink,static unsigned long raid5_cache_count (struct shrinker *shrink,struct shrink_control *sc) { - struct r5conf *conf = container_of(shrink, struct r5conf, shrinker); +struct r5conf *conf = shrink->private_data;if (conf->max_nr_stripes < conf->min_nr_stripes)/* unlikely, but not impossible */
@@ -7695,16 +7695,21 @@ static struct r5conf *setup_conf(struct mddev *mddev)* it reduces the queue depth and so can hurt throughput. * So set it rather large, scaled by number of devices. */ - conf->shrinker.seeks =DEFAULT_SEEKS * conf->raid_disks *4 ; - conf->shrinker.scan_objects = raid5_cache_scan; - conf->shrinker.count_objects = raid5_cache_count; - conf->shrinker.batch =128 ; - conf->shrinker.flags =0 ; - ret =register_shrinker (&conf->shrinker,"md-raid5:%s" ,mdname (mddev)); + conf->shrinker =shrinker_alloc_and_init (raid5_cache_count, + raid5_cache_scan,128 , +DEFAULT_SEEKS * conf->raid_disks *4 , +0 , conf); +if (!conf->shrinker) { +pr_warn ("md/raid:%s: couldn't allocate shrinker.\n" , +mdname (mddev)); + goto abort; + } + + ret =register_shrinker (conf->shrinker,"md-raid5:%s" ,mdname (mddev));if (ret) {pr_warn ("md/raid:%s: couldn't register shrinker.\n" ,mdname (mddev)); - goto abort; + goto abort_shrinker; }sprintf (pers_name,"raid%d" , mddev->new_level);
@@ -7717,7 +7722,8 @@ static struct r5conf *setup_conf(struct mddev *mddev) }return conf; - +abort_shrinker: +shrinker_free (conf->shrinker); abort:if (conf)free_conf (conf);
drivers/md/raid5.h +1 -1
@@ -670,7 +670,7 @@ struct r5conf {wait_queue_head_t wait_for_stripe;wait_queue_head_t wait_for_overlap;unsigned long cache_state; -struct shrinker shrinker; +struct shrinker *shrinker;int pool_size;/* number of disks in stripeheads in pool */ spinlock_t device_lock;struct disk_info *disks;
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 9/29 bcache: dynamically allocate the md-bcache shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the md-bcache shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct cache_set.
--- drivers/md/bcache/bcache.h | 2 +- drivers/md/bcache/btree.c | 23 ++++++++++++++--------- drivers/md/bcache/sysfs.c | 2 +- 3 files changed, 16 insertions(+), 11 deletions(-)
drivers/md/bcache/bcache.h +1 -1
@@ -541,7 +541,7 @@ struct cache_set {struct bio_set bio_split;/* For the btree cache */ -struct shrinker shrink; +struct shrinker *shrink;/* For the btree cache and anything allocation related */ struct mutex bucket_lock;
drivers/md/bcache/btree.c +14 -9
@@ -667,7 +667,7 @@ static int mca_reap(struct btree *b, unsigned int min_order, bool flush)static unsigned long bch_mca_scan (struct shrinker *shrink,struct shrink_control *sc) { - struct cache_set *c = container_of(shrink, struct cache_set, shrink); +struct cache_set *c = shrink->private_data;struct btree *b, *t;unsigned long i, nr = sc->nr_to_scan;unsigned long freed =0 ;
@@ -734,7 +734,7 @@ static unsigned long bch_mca_scan(struct shrinker *shrink,static unsigned long bch_mca_count (struct shrinker *shrink,struct shrink_control *sc) { - struct cache_set *c = container_of(shrink, struct cache_set, shrink); +struct cache_set *c = shrink->private_data;if (c->shrinker_disabled)return 0 ;
@@ -752,8 +752,8 @@ void bch_btree_cache_free(struct cache_set *c)closure_init_stack (&cl); -if (c->shrink.list.next) -unregister_shrinker (&c->shrink); +if (c->shrink->list.next) +unregister_and_free_shrinker (c->shrink);mutex_lock (&c->bucket_lock);
@@ -828,14 +828,19 @@ int bch_btree_cache_alloc(struct cache_set *c) c->verify_data =NULL ;#endif - c->shrink.count_objects = bch_mca_count; - c->shrink.scan_objects = bch_mca_scan; - c->shrink.seeks =4 ; - c->shrink.batch = c->btree_pages *2 ; + c->shrink =shrinker_alloc_and_init (bch_mca_count, bch_mca_scan, + c->btree_pages *2 ,4 ,0 , c); +if (!c->shrink) { +pr_warn ("bcache: %s: could not allocate shrinker\n" , + __func__); +return -ENOMEM ; + } -if (register_shrinker (&c->shrink,"md-bcache:%pU" , c->set_uuid)) +if (register_shrinker (c->shrink,"md-bcache:%pU" , c->set_uuid)) {pr_warn ("bcache: %s: could not register shrinker\n" , __func__); +shrinker_free (c->shrink); + }return 0 ; }
drivers/md/bcache/sysfs.c +1 -1
@@ -866,7 +866,7 @@ STORE(__bch_cache_set) sc.gfp_mask =GFP_KERNEL ; sc.nr_to_scan =strtoul_or_return (buf); - c->shrink.scan_objects (&c->shrink, &sc); + c->shrink->scan_objects (c->shrink, &sc); } sysfs_strtoul_clamp(congested_read_threshold_us,
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 10/29 vmw_balloon: dynamically allocate the vmw-balloon shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the vmw-balloon shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct vmballoon.
--- drivers/misc/vmw_balloon.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
drivers/misc/vmw_balloon.c +10 -6
@@ -380,7 +380,7 @@ struct vmballoon {/** * @shrinker: shrinker interface that is used to avoid over-inflation. */ -struct shrinker shrinker; +struct shrinker *shrinker;/** * @shrinker_registered: whether the shrinker was registered.
@@ -1569,7 +1569,7 @@ static unsigned long vmballoon_shrinker_count(struct shrinker *shrinker,static void vmballoon_unregister_shrinker (struct vmballoon *b) {if (b->shrinker_registered) -unregister_shrinker (&b->shrinker); +unregister_and_free_shrinker (b->shrinker); b->shrinker_registered = false; }
@@ -1581,14 +1581,18 @@ static int vmballoon_register_shrinker(struct vmballoon *b)if (!vmwballoon_shrinker_enable)return 0 ; - b->shrinker.scan_objects = vmballoon_shrinker_scan; - b->shrinker.count_objects = vmballoon_shrinker_count; - b->shrinker.seeks =DEFAULT_SEEKS ; + b->shrinker =shrinker_alloc_and_init (vmballoon_shrinker_count, + vmballoon_shrinker_scan, +0 ,DEFAULT_SEEKS ,0 , b); +if (!b->shrinker) +return -ENOMEM ; - r =register_shrinker (&b->shrinker,"vmw-balloon" ); + r =register_shrinker (b->shrinker,"vmw-balloon" );if (r ==0 ) b->shrinker_registered = true; +else +shrinker_free (b->shrinker);return r; }
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 11/29 virtio_balloon: dynamically allocate the virtio-balloon shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the virtio-balloon shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct virtio_balloon.
--- drivers/virtio/virtio_balloon.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)
drivers/virtio/virtio_balloon.c +16 -10
@@ -111,7 +111,7 @@ struct virtio_balloon {struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR ];/* Shrinker to return free pages - VIRTIO_BALLOON_F_FREE_PAGE_HINT */ -struct shrinker shrinker; +struct shrinker *shrinker;/* OOM notifier to deflate on OOM - VIRTIO_BALLOON_F_DEFLATE_ON_OOM */ struct notifier_block oom_nb;
@@ -816,8 +816,7 @@ static unsigned long shrink_free_pages(struct virtio_balloon *vb,static unsigned long virtio_balloon_shrinker_scan (struct shrinker *shrinker,struct shrink_control *sc) { -struct virtio_balloon *vb =container_of (shrinker, - struct virtio_balloon, shrinker); +struct virtio_balloon *vb = shrinker->private_data;return shrink_free_pages (vb, sc->nr_to_scan); }
@@ -825,8 +824,7 @@ static unsigned long virtio_balloon_shrinker_scan(struct shrinker *shrinker,static unsigned long virtio_balloon_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) { -struct virtio_balloon *vb =container_of (shrinker, - struct virtio_balloon, shrinker); +struct virtio_balloon *vb = shrinker->private_data;return vb->num_free_page_blocks *VIRTIO_BALLOON_HINT_BLOCK_PAGES ; }
@@ -847,16 +845,24 @@ static int virtio_balloon_oom_notify(struct notifier_block *nb,static void virtio_balloon_unregister_shrinker (struct virtio_balloon *vb) { -unregister_shrinker (&vb->shrinker); +unregister_and_free_shrinker (vb->shrinker); }static int virtio_balloon_register_shrinker (struct virtio_balloon *vb) { - vb->shrinker.scan_objects = virtio_balloon_shrinker_scan; - vb->shrinker.count_objects = virtio_balloon_shrinker_count; - vb->shrinker.seeks =DEFAULT_SEEKS ; +int ret; + + vb->shrinker =shrinker_alloc_and_init (virtio_balloon_shrinker_count, + virtio_balloon_shrinker_scan, +0 ,DEFAULT_SEEKS ,0 , vb); +if (!vb->shrinker) +return -ENOMEM ; + + ret =register_shrinker (vb->shrinker,"virtio-balloon" ); +if (ret) +shrinker_free (vb->shrinker); -return register_shrinker (&vb->shrinker,"virtio-balloon" ); +return ret; }static int virtballoon_probe (struct virtio_device *vdev)
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 12/29 mbcache: dynamically allocate the mbcache shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the mbcache shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct mb_cache.
--- fs/mbcache.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-)
fs/mbcache.c +21 -18
@@ -37,7 +37,7 @@ struct mb_cache {struct list_head c_list;/* Number of entries in cache */ unsigned long c_entry_count; -struct shrinker c_shrink; +struct shrinker *c_shrink;/* Work for shrinking when the cache has too many entries */ struct work_struct c_shrink_work; };
@@ -293,8 +293,7 @@ EXPORT_SYMBOL(mb_cache_entry_touch);static unsigned long mb_cache_count (struct shrinker *shrink,struct shrink_control *sc) { - struct mb_cache *cache = container_of(shrink, struct mb_cache, - c_shrink); +struct mb_cache *cache = shrink->private_data;return cache->c_entry_count; }
@@ -333,8 +332,8 @@ static unsigned long mb_cache_shrink(struct mb_cache *cache,static unsigned long mb_cache_scan (struct shrinker *shrink,struct shrink_control *sc) { - struct mb_cache *cache = container_of(shrink, struct mb_cache, - c_shrink); +struct mb_cache *cache = shrink->private_data; +return mb_cache_shrink (cache, sc->nr_to_scan); }
@@ -370,26 +369,30 @@ struct mb_cache *mb_cache_create(int bucket_bits) cache->c_hash =kmalloc_array (bucket_count,sizeof (struct hlist_bl_head ),GFP_KERNEL ); -if (!cache->c_hash) { -kfree (cache); - goto err_out; - } +if (!cache->c_hash) + goto err_c_hash; +for (i =0 ; i < bucket_count; i++)INIT_HLIST_BL_HEAD (&cache->c_hash[i]); - cache->c_shrink.count_objects = mb_cache_count; - cache->c_shrink.scan_objects = mb_cache_scan; - cache->c_shrink.seeks =DEFAULT_SEEKS ; -if (register_shrinker (&cache->c_shrink,"mbcache-shrinker" )) { -kfree (cache->c_hash); -kfree (cache); - goto err_out; - } + cache->c_shrink =shrinker_alloc_and_init (mb_cache_count, mb_cache_scan, +0 ,DEFAULT_SEEKS ,0 , cache); +if (!cache->c_shrink) + goto err_shrinker; + +if (register_shrinker (cache->c_shrink,"mbcache-shrinker" )) + goto err_register;INIT_WORK (&cache->c_shrink_work, mb_cache_shrink_worker);return cache; +err_register: +shrinker_free (cache->c_shrink); +err_shrinker: +kfree (cache->c_hash); +err_c_hash: +kfree (cache); err_out:return NULL ; }
@@ -406,7 +409,7 @@ void mb_cache_destroy(struct mb_cache *cache) {struct mb_cache_entry *entry, *next; -unregister_shrinker (&cache->c_shrink); +unregister_and_free_shrinker (cache->c_shrink);/* * We don't bother with any locking. Cache must not be used at this
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 13/29 ext4: dynamically allocate the ext4-es shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the ext4-es shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct ext4_sb_info.
--- fs/ext4/ext4.h | 2 +- fs/ext4/extents_status.c | 21 ++++++++++++--------- 2 files changed, 13 insertions(+), 10 deletions(-)
fs/ext4/ext4.h +1 -1
@@ -1651,7 +1651,7 @@ struct ext4_sb_info {__u32 s_csum_seed;/* Reclaim extents from extent status tree */ -struct shrinker s_es_shrinker; +struct shrinker *s_es_shrinker;struct list_head s_es_list;/* List of inodes with reclaimable extents */ long s_es_nr_inode;struct ext4_es_stats s_es_stats;
fs/ext4/extents_status.c +12 -9
@@ -1596,7 +1596,7 @@ static unsigned long ext4_es_count(struct shrinker *shrink,unsigned long nr;struct ext4_sb_info *sbi; - sbi = container_of(shrink, struct ext4_sb_info, s_es_shrinker); + sbi = shrink->private_data; nr =percpu_counter_read_positive (&sbi->s_es_stats.es_stats_shk_cnt);trace_ext4_es_shrink_count (sbi->s_sb, sc->nr_to_scan, nr);return nr;
@@ -1605,8 +1605,7 @@ static unsigned long ext4_es_count(struct shrinker *shrink,static unsigned long ext4_es_scan (struct shrinker *shrink,struct shrink_control *sc) { -struct ext4_sb_info *sbi =container_of (shrink, - struct ext4_sb_info, s_es_shrinker); +struct ext4_sb_info *sbi = shrink->private_data;int nr_to_scan = sc->nr_to_scan;int ret, nr_shrunk;
@@ -1690,15 +1689,19 @@ int ext4_es_register_shrinker(struct ext4_sb_info *sbi)if (err) goto err3; - sbi->s_es_shrinker.scan_objects = ext4_es_scan; - sbi->s_es_shrinker.count_objects = ext4_es_count; - sbi->s_es_shrinker.seeks =DEFAULT_SEEKS ; - err =register_shrinker (&sbi->s_es_shrinker,"ext4-es:%s" , + sbi->s_es_shrinker =shrinker_alloc_and_init (ext4_es_count, ext4_es_scan, +0 ,DEFAULT_SEEKS ,0 , sbi); +if (!sbi->s_es_shrinker) + goto err4; + + err =register_shrinker (sbi->s_es_shrinker,"ext4-es:%s" , sbi->s_sb->s_id);if (err) - goto err4; + goto err5;return 0 ; +err5: +shrinker_free (sbi->s_es_shrinker); err4:percpu_counter_destroy (&sbi->s_es_stats.es_stats_shk_cnt); err3:
@@ -1716,7 +1719,7 @@ void ext4_es_unregister_shrinker(struct ext4_sb_info *sbi)percpu_counter_destroy (&sbi->s_es_stats.es_stats_cache_misses);percpu_counter_destroy (&sbi->s_es_stats.es_stats_all_cnt);percpu_counter_destroy (&sbi->s_es_stats.es_stats_shk_cnt); -unregister_shrinker (&sbi->s_es_shrinker); +unregister_and_free_shrinker (sbi->s_es_shrinker); }/*
Signature · 1 line
2.30.2
Qi Zheng
reviewerPATCH 14/29 jbd2,ext4: dynamically allocate the jbd2-journal shrinker
In reply to Qi ZhengFrom: Qi Zheng <[email protected]>
In preparation for implementing lockless slab shrink, we need to dynamically allocate the jbd2-journal shrinker, so that it can be freed asynchronously using kfree_rcu(). Then it doesn't need to wait for RCU read-side critical section when releasing the struct journal_s.
--- fs/jbd2/journal.c | 32 +++++++++++++++++++------------- include/linux/jbd2.h | 2 +- 2 files changed, 20 insertions(+), 14 deletions(-)
fs/jbd2/journal.c +19 -13
@@ -1301,7 +1301,7 @@ static int jbd2_min_tag_size(void)static unsigned long jbd2_journal_shrink_scan (struct shrinker *shrink,struct shrink_control *sc) { -journal_t *journal =container_of (shrink, journal_t, j_shrinker); +journal_t *journal = shrink->private_data;unsigned long nr_to_scan = sc->nr_to_scan;unsigned long nr_shrunk;unsigned long count;
@@ -1327,7 +1327,7 @@ static unsigned long jbd2_journal_shrink_scan(struct shrinker *shrink,static unsigned long jbd2_journal_shrink_count (struct shrinker *shrink,struct shrink_control *sc) { -journal_t *journal =container_of (shrink, journal_t, j_shrinker); +journal_t *journal = shrink->private_data;unsigned long count; count =percpu_counter_read_positive (&journal->j_checkpoint_jh_count);
@@ -1415,21 +1415,27 @@ static journal_t *journal_init_common(struct block_device *bdev, journal->j_superblock = (journal_superblock_t *)bh->b_data; journal->j_shrink_transaction =NULL ; - journal->j_shrinker.scan_objects = jbd2_journal_shrink_scan; - journal->j_shrinker.count_objects = jbd2_journal_shrink_count; - journal->j_shrinker.seeks =DEFAULT_SEEKS ; - journal->j_shrinker.batch = journal->j_max_transaction_buffers;if (percpu_counter_init (&journal->j_checkpoint_jh_count,0 ,GFP_KERNEL )) goto err_cleanup; -if (register_shrinker (&journal->j_shrinker,"jbd2-journal:(%u:%u)" , -MAJOR (bdev->bd_dev),MINOR (bdev->bd_dev))) { -percpu_counter_destroy (&journal->j_checkpoint_jh_count); - goto err_cleanup; - } + journal->j_shrinker =shrinker_alloc_and_init (jbd2_journal_shrink_count, + jbd2_journal_shrink_scan, + journal->j_max_transaction_buffers, +DEFAULT_SEEKS ,0 , journal); +if (!journal->j_shrinker) + goto err_shrinker; + +if (register_shrinker (journal->j_shrinker,"jbd2-journal:(%u:%u)" , +MAJOR (bdev->bd_dev),MINOR (bdev->bd_dev))) + goto err_register; +return journal; +err_register: +shrinker_free (journal->j_shrinker); +err_shrinker: +percpu_counter_destroy (&journal->j_checkpoint_jh_count); err_cleanup:brelse (journal->j_sb_buffer);kfree (journal->j_wbuf);
@@ -2190,9 +2196,9 @@ int jbd2_journal_destroy(journal_t *journal)brelse (journal->j_sb_buffer); } -if (journal->j_shrinker.flags &SHRINKER_REGISTERED ) { +if (journal->j_shrinker->flags &SHRINKER_REGISTERED ) {percpu_counter_destroy (&journal->j_checkpoint_jh_count); -unregister_shrinker (&journal->j_shrinker); +unregister_and_free_shrinker (journal->j_shrinker); }if (journal->j_proc_entry)jbd2_stats_proc_exit (journal);
include/linux/jbd2.h +1 -1
@@ -891,7 +891,7 @@ struct journal_s* Journal head shrinker, reclaim buffer's journal head which * has been written back. */ -struct shrinker j_shrinker; +struct shrinker *j_shrinker;/** * @j_checkpoint_jh_count:
Signature · 1 line
2.30.2