mm: fs: invalidate bh_lrus for only cold path
Subsystems (2)
PATCH v2 1/1 mm: fs: invalidate bh_lrus for only cold path
kernel test robot reported the regression of fio.write_iops[1] with [2].
Since lru_add_drain is called frequently, invalidate bh_lrus there could increase bh_lrus cache miss ratio, which needs more IO in the end.
This patch moves the bh_lrus invalidation from the hot path( e.g., zap_page_range, pagevec_release) to cold path(i.e., lru_add_drain_all, lru_cache_disable).
[1] https://lore.kernel.org/lkml/20210520083144.GD14190(a)xsang-OptiPlex-9020/
[2] 8cc621d2f45d, mm: fs: invalidate BH LRU during page migration
---
* from v1 - https://lore.kernel.org/lkml/YK0oQ76zX0uVZCwQ(a)google.com/
* add Reviewed-by - cgoldswo
mm/swap.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
mm/swap.c +13 -3
@@ -642,7 +642,6 @@ void lru_add_drain_cpu(int cpu)pagevec_lru_move_fn (pvec, lru_lazyfree_fn);activate_page_drain (cpu); -invalidate_bh_lrus_cpu (cpu); }/**
@@ -725,6 +724,17 @@ void lru_add_drain(void)local_unlock (&lru_pvecs.lock); } +static void lru_add_and_bh_lrus_drain (void ) +{ +int cpu; + +local_lock (&lru_pvecs.lock); + cpu =smp_processor_id (); +lru_add_drain_cpu (cpu); +local_unlock (&lru_pvecs.lock); +invalidate_bh_lrus_cpu (cpu); +} +void lru_add_drain_cpu_zone (struct zone *zone) {local_lock (&lru_pvecs.lock);
@@ -739,7 +749,7 @@ static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work);static void lru_add_drain_per_cpu (struct work_struct *dummy) { -lru_add_drain (); +lru_add_and_bh_lrus_drain (); }/*
@@ -880,7 +890,7 @@ void lru_cache_disable(void)*/ __lru_add_drain_all (true);#else -lru_add_drain (); +lru_add_and_bh_lrus_drain ();#endif }
Signature · 1 line
2.31.1.818.g46aad6cb9e-goog
Andrew Morton
maintainerQuoted 10 lines · On Tue, 1 Jun 2021 07:54:25 -0700 Minchan Kim <[email protected]> wrote:
kernel test robot reported the regression of fio.write_iops[1] with [2].
Since
lru_add_drainis called frequently, invalidatebh_lrusthere could increasebh_lruscache miss ratio, which needs more IO in the end.This patch moves the
bh_lrusinvalidation from the hotpath( e.g., zap_page_range, pagevec_release)to coldpath(i.e., lru_add_drain_all, lru_cache_disable).
This code is starting to hurt my brain.
What are the locking/context rules for invalidate_bh_lrus_cpu()? AFAICT it offers no protection against two CPUs concurrently running __invalidate_bh_lrus() against the same bh_lru.
So when CONFIG_SMP=y, invalidate_bh_lrus_cpu() must always and only be run on the cpu which owns the bh_lru. In which case why does it have the `cpu' arg?
Your new lru_add_and_bh_lrus_drain() follows these rules by calling invalidate_bh_lrus_cpu() from a per-cpu worker or when CONFIG_SMP=n.
I think. It's all as clear as mud and undocumented. Could you please take a look at this? Comment the locking/context rules thoroughly and check that they are being followed? Not forgetting cpu hotplug... See if there's a way of simplifying/clarifying the code?
The fact that swap.c has those #ifdef CONFIG_SMPs in there is a hint that we're doing something wrong (or poorly) in there. Perhaps that's unavoidable because of all the fancy footwork in __lru_add_drain_all().
Quoted 13 lines · Andrew Morton
Quoted 10 lines · On Tue, 1 Jun 2021 07:54:25 -0700 Minchan Kim <[email protected]> wrote:
kernel test robot reported the regression of fio.write_iops[1] with [2].
Since
lru_add_drainis called frequently, invalidatebh_lrusthere could increasebh_lruscache miss ratio, which needs more IO in the end.This patch moves the
bh_lrusinvalidation from the hotpath( e.g., zap_page_range, pagevec_release)to coldpath(i.e., lru_add_drain_all, lru_cache_disable).
Quoted 7 lines
This code is starting to hurt my brain.
What are the locking/context rules for
invalidate_bh_lrus_cpu()?AFAICT it offers no protection against two CPUs concurrently running
__invalidate_bh_lrus()against the samebh_lru.
The lru_add_drain_per_cpu will run on per-cpu since it's per-cpu work and invalidate_bh_lrus_cpu will run under bh_lru_lock so I couldn't imagine that race can happen.
Quoted 4 lines
So when
CONFIG_SMP=y,invalidate_bh_lrus_cpu()must always and only be run on the cpu which owns thebh_lru. In which case why does it have the `cpu' arg?
I just wanted to express both lru_add_drain_cpu and invalidate_bh_lrus_cpu in lru_add_and_bh_lrus_drain run in the same cpu but look like a bad idea since it makes people confused. Let me remove the cpu argument from invalidate_bh_lrus_cpu.
Quoted 4 lines
Your new
lru_add_and_bh_lrus_drain()follows these rules by callinginvalidate_bh_lrus_cpu()from a per-cpu worker or whenCONFIG_SMP=n.
Quoted 9 lines
I think. It's all as clear as mud and undocumented. Could you please take a look at this? Comment the locking/context rules thoroughly and check that they are being followed? Not forgetting cpu hotplug... See if there's a way of simplifying/clarifying the code?
The fact that
swap.chas those #ifdefCONFIG_SMPsin there is a hint that we're doing something wrong (or poorly) in there. Perhaps that's unavoidable because of all the fancy footwork in__lru_add_drain_all().
Hopefully, this is better.
Quoted 1 line
From
8d58e7ade3ed6c080995dec1395b1e130b3d16b3Mon Sep 17 00:00:00 2001
From: Minchan Kim <[email protected]>
Date: Tue, 25 May 2021 08:19:17 -0700
Subject: [PATCH] mm: fs: invalidate bh_lrus for only cold path
kernel test robot reported the regression of fio.write_iops[1] with [2].
Since lru_add_drain is called frequently, invalidate bh_lrus there could increase bh_lrus cache miss ratio, which needs more IO in the end.
This patch moves the bh_lrus invalidation from the hot path( e.g., zap_page_range, pagevec_release) to cold path(i.e., lru_add_drain_all, lru_cache_disable).
[1] https://lore.kernel.org/lkml/20210520083144.GD14190(a)xsang-OptiPlex-9020/
[2] 8cc621d2f45d, mm: fs: invalidate BH LRU during page migration
--- fs/buffer.c | 8 ++++++-- include/linux/buffer_head.h | 4 ++-- mm/swap.c | 19 ++++++++++++++++--- 3 files changed, 24 insertions(+), 7 deletions(-)
fs/buffer.c +6 -2
@@ -1487,12 +1487,16 @@ void invalidate_bh_lrus(void) }EXPORT_SYMBOL_GPL (invalidate_bh_lrus); -void invalidate_bh_lrus_cpu (int cpu) +/* +* It's called from workqueue context so we need a bh_lru_lock to close +* the race with preemption/irq. +*/ +void invalidate_bh_lrus_cpu (void ) {struct bh_lru *b;bh_lru_lock (); - b =per_cpu_ptr (&bh_lrus, cpu); + b =this_cpu_ptr (&bh_lrus);__invalidate_bh_lrus (b);bh_lru_unlock (); }
include/linux/buffer_head.h +2 -2
@@ -194,7 +194,7 @@ void __breadahead_gfp(struct block_device *, sector_t block, unsigned int size,struct buffer_head *__bread_gfp (struct block_device *,sector_t block,unsigned size,gfp_t gfp);void invalidate_bh_lrus (void ); -void invalidate_bh_lrus_cpu (int cpu); +void invalidate_bh_lrus_cpu (void );bool has_bh_in_lru (int cpu,void *dummy);struct buffer_head *alloc_buffer_head (gfp_t gfp_flags);void free_buffer_head (struct buffer_head * bh);
@@ -408,7 +408,7 @@ static inline int inode_has_buffers(struct inode *inode) { return 0; }static inline void invalidate_inode_buffers (struct inode *inode) {}static inline int remove_inode_buffers (struct inode *inode) {return 1 ; }static inline int sync_mapping_buffers (struct address_space *mapping) {return 0 ; } -static inline void invalidate_bh_lrus_cpu (int cpu) {} +static inline void invalidate_bh_lrus_cpu (void ) {}static inline bool has_bh_in_lru (int cpu,void *dummy) {return 0 ; }#define buffer_heads_over_limit 0
mm/swap.c +16 -3
@@ -642,7 +642,6 @@ void lru_add_drain_cpu(int cpu)pagevec_lru_move_fn (pvec, lru_lazyfree_fn);activate_page_drain (cpu); -invalidate_bh_lrus_cpu (cpu); }/**
@@ -725,6 +724,20 @@ void lru_add_drain(void)local_unlock (&lru_pvecs.lock); } +/* +* It's called from per-cpu workqueue context in SMP case so +* lru_add_drain_cpu and invalidate_bh_lrus_cpu should run on +* the same cpu. It shouldn't be a problem in !SMP case since +* the core is only one and the locks will disable preemption. +*/ +static void lru_add_and_bh_lrus_drain (void ) +{ +local_lock (&lru_pvecs.lock); +lru_add_drain_cpu (smp_processor_id ()); +local_unlock (&lru_pvecs.lock); +invalidate_bh_lrus_cpu (); +} +void lru_add_drain_cpu_zone (struct zone *zone) {local_lock (&lru_pvecs.lock);
@@ -739,7 +752,7 @@ static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work);static void lru_add_drain_per_cpu (struct work_struct *dummy) { -lru_add_drain (); +lru_add_and_bh_lrus_drain (); }/*
@@ -880,7 +893,7 @@ void lru_cache_disable(void)*/ __lru_add_drain_all (true);#else -lru_add_drain (); +lru_add_and_bh_lrus_drain ();#endif }
Signature · 1 line
2.32.0.rc0.204.g9fa02ecfa5-goog
Chris Goldsworthy
Reviewed-byQuoted 178 lines · Minchan Kim
Quoted 14 lines · Andrew Morton
Quoted 10 lines · On Tue, 1 Jun 2021 07:54:25 -0700 Minchan Kim <[email protected]> wrote:
kernel test robot reported the regression of fio.write_iops[1] with [2].
Since
lru_add_drainis called frequently, invalidatebh_lrusthere could increasebh_lruscache miss ratio, which needs more IO in the end.This patch moves the
bh_lrusinvalidation from the hotpath( e.g., zap_page_range, pagevec_release)to coldpath(i.e., lru_add_drain_all, lru_cache_disable).Quoted 7 lines
This code is starting to hurt my brain.
What are the locking/context rules for
invalidate_bh_lrus_cpu()?AFAICT it offers no protection against two CPUs concurrently running
__invalidate_bh_lrus()against the samebh_lru.The
lru_add_drain_per_cpuwill run on per-cpu since it's per-cpu work andinvalidate_bh_lrus_cpuwill run underbh_lru_lockso I couldn't imagine that race can happen.Quoted 4 lines
So when
CONFIG_SMP=y,invalidate_bh_lrus_cpu()must always and only be run on the cpu which owns thebh_lru. In which case why does it have the `cpu' arg?I just wanted to express both
lru_add_drain_cpuandinvalidate_bh_lrus_cpu
inlru_add_and_bh_lrus_drainrun in the same cpu but look like a bad idea
since it makes people confused. Let me remove the cpu argument frominvalidate_bh_lrus_cpu.Quoted 10 lines
Your new
lru_add_and_bh_lrus_drain()follows these rules by callinginvalidate_bh_lrus_cpu()from a per-cpu worker or whenCONFIG_SMP=n.I think. It's all as clear as mud and undocumented. Could you please take a look at this? Comment the locking/context rules thoroughly and check that they are being followed? Not forgetting cpu hotplug... See if
there's a way of simplifying/clarifying the code?Quoted 5 lines
The fact that
swap.chas those #ifdefCONFIG_SMPsin there is a hint that we're doing something wrong (or poorly) in there. Perhaps that's unavoidable because of all the fancy footwork in__lru_add_drain_all().Hopefully, this is better.
From
8d58e7ade3ed6c080995dec1395b1e130b3d16b3Mon Sep 17 00:00:00 2001
From: Minchan Kim <[email protected]>
Date: Tue, 25 May 2021 08:19:17 -0700
Subject: [PATCH] mm: fs: invalidatebh_lrusfor only cold pathkernel test robot reported the regression of fio.write_iops[1] with [2].
Since
lru_add_drainis called frequently, invalidatebh_lrusthere could increasebh_lruscache miss ratio, which needs more IO in the end.This patch moves the
bh_lrusinvalidation from the hotpath( e.g., zap_page_range, pagevec_release)to coldpath(i.e., lru_add_drain_all, lru_cache_disable).[1]
https://lore.kernel.org/lkml/20210520083144.GD14190(a)xsang-OptiPlex-9020/
[2]8cc621d2f45d, mm: fs: invalidate BH LRU during page migrationReported-by kernel test robot <[email protected]>Signed-off-by Minchan Kim <[email protected]>--- fs/buffer.c | 8 ++++++-- include/linux/buffer_head.h | 4 ++-- mm/swap.c | 19 ++++++++++++++++--- 3 files changed, 24 insertions(+), 7 deletions(-)fs/buffer.c +2
-1@@ -1487,12 +1487,16 @@ void invalidate_bh_lrus(void) }EXPORT_SYMBOL_GPL (invalidate_bh_lrus); -void invalidate_bh_lrus_cpu (int cpu) +/* +* It's called from workqueue context so we need a bh_lru_lock to close
+ * the race with preemption/irq.
+ */fs/buffer.c +2
-1+void invalidate_bh_lrus_cpu (void ) {struct bh_lru *b;bh_lru_lock (); - b =per_cpu_ptr (&bh_lrus, cpu); + b =this_cpu_ptr (&bh_lrus);__invalidate_bh_lrus (b);bh_lru_unlock (); }include/linux/buffer_head.h +1
-1@@ -194,7 +194,7 @@ void __breadahead_gfp(struct block_device *, sector_t block, unsigned int size, struct buffer_head *__bread_gfp(struct block_device *,sector_t block,unsigned size,gfp_t gfp);void invalidate_bh_lrus (void ); -void invalidate_bh_lrus_cpu (int cpu); +void invalidate_bh_lrus_cpu (void );bool has_bh_in_lru (int cpu,void *dummy);struct buffer_head *alloc_buffer_head (gfp_t gfp_flags);void
free_buffer_head(struct buffer_head * bh);include/linux/buffer_head.h +1
-1@@ -408,7 +408,7 @@ static inline int inode_has_buffers(struct inode *inode) { return 0; }static inline void invalidate_inode_buffers (struct inode *inode) {}static inline int remove_inode_buffers (struct inode *inode) {return 1 ; }static inline int sync_mapping_buffers (struct address_space *mapping) {return 0 ; } -static inline void invalidate_bh_lrus_cpu (int cpu) {} +static inline void invalidate_bh_lrus_cpu (void ) {}static inline bool
has_bh_in_lru(int cpu, void *dummy){ return 0; } #definebuffer_heads_over_limit0mm/swap.c +15
-2@@ -642,7 +642,6 @@ void lru_add_drain_cpu(int cpu)pagevec_lru_move_fn (pvec, lru_lazyfree_fn);activate_page_drain (cpu); -invalidate_bh_lrus_cpu (cpu); }/** @@ -725,6 +724,20 @@ void lru_add_drain(void)local_unlock (&lru_pvecs.lock); } +/* +* It's called from per-cpu workqueue context in SMP case so +* lru_add_drain_cpu and invalidate_bh_lrus_cpu should run on +* the same cpu. It shouldn't be a problem in !SMP case since +* the core is only one and the locks will disable preemption. +*/ +static void lru_add_and_bh_lrus_drain (void ) +{ +local_lock (&lru_pvecs.lock); +lru_add_drain_cpu (smp_processor_id ()); +local_unlock (&lru_pvecs.lock); +invalidate_bh_lrus_cpu (); +} +void lru_add_drain_cpu_zone (struct zone *zone) {local_lock (&lru_pvecs.lock);@@ -739,7 +752,7 @@ static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work);static void lru_add_drain_per_cpu (struct work_struct *dummy) { -lru_add_drain (); +lru_add_and_bh_lrus_drain (); }/*
mm/swap.c void lru_cache_disable(void) · line 898
Quoted 7 lines
mm/swap.c +1
-1@@ -880,7 +893,7 @@ void lru_cache_disable(void)*/ __lru_add_drain_all (true);#else -lru_add_drain (); +lru_add_and_bh_lrus_drain ();#endif }
Hi Minchan,
This looks good to me. Feel free to add:
Thanks,
Chris.
Signature · 4 lines
The Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Quoted 188 lines · Chris Goldsworthy
Quoted 177 lines · Minchan Kim
Quoted 14 lines · Andrew Morton
Quoted 10 lines · On Tue, 1 Jun 2021 07:54:25 -0700 Minchan Kim <[email protected]> wrote:
kernel test robot reported the regression of fio.write_iops[1] with [2].
Since
lru_add_drainis called frequently, invalidatebh_lrusthere could increasebh_lruscache miss ratio, which needs more IO in the end.This patch moves the
bh_lrusinvalidation from the hotpath( e.g., zap_page_range, pagevec_release)to coldpath(i.e., lru_add_drain_all, lru_cache_disable).Quoted 7 lines
This code is starting to hurt my brain.
What are the locking/context rules for
invalidate_bh_lrus_cpu()?AFAICT it offers no protection against two CPUs concurrently running
__invalidate_bh_lrus()against the samebh_lru.The
lru_add_drain_per_cpuwill run on per-cpu since it's per-cpu work andinvalidate_bh_lrus_cpuwill run underbh_lru_lockso I couldn't imagine that race can happen.Quoted 4 lines
So when
CONFIG_SMP=y,invalidate_bh_lrus_cpu()must always and only be run on the cpu which owns thebh_lru. In which case why does it have the `cpu' arg?I just wanted to express both
lru_add_drain_cpuandinvalidate_bh_lrus_cpu
inlru_add_and_bh_lrus_drainrun in the same cpu but look like a bad idea
since it makes people confused. Let me remove the cpu argument frominvalidate_bh_lrus_cpu.Quoted 10 lines
Your new
lru_add_and_bh_lrus_drain()follows these rules by callinginvalidate_bh_lrus_cpu()from a per-cpu worker or whenCONFIG_SMP=n.I think. It's all as clear as mud and undocumented. Could you please take a look at this? Comment the locking/context rules thoroughly and check that they are being followed? Not forgetting cpu hotplug... See if
there's a way of simplifying/clarifying the code?Quoted 5 lines
The fact that
swap.chas those #ifdefCONFIG_SMPsin there is a hint that we're doing something wrong (or poorly) in there. Perhaps that's unavoidable because of all the fancy footwork in__lru_add_drain_all().Hopefully, this is better.
From
8d58e7ade3ed6c080995dec1395b1e130b3d16b3Mon Sep 17 00:00:00 2001
From: Minchan Kim <[email protected]>
Date: Tue, 25 May 2021 08:19:17 -0700
Subject: [PATCH] mm: fs: invalidatebh_lrusfor only cold pathkernel test robot reported the regression of fio.write_iops[1] with [2].
Since
lru_add_drainis called frequently, invalidatebh_lrusthere could increasebh_lruscache miss ratio, which needs more IO in the end.This patch moves the
bh_lrusinvalidation from the hotpath( e.g., zap_page_range, pagevec_release)to coldpath(i.e., lru_add_drain_all, lru_cache_disable).[1]
https://lore.kernel.org/lkml/20210520083144.GD14190(a)xsang-OptiPlex-9020/
[2]8cc621d2f45d, mm: fs: invalidate BH LRU during page migrationReported-by kernel test robot <[email protected]>Signed-off-by Minchan Kim <[email protected]>--- fs/buffer.c | 8 ++++++-- include/linux/buffer_head.h | 4 ++-- mm/swap.c | 19 ++++++++++++++++--- 3 files changed, 24 insertions(+), 7 deletions(-)fs/buffer.c +6
-2@@ -1487,12 +1487,16 @@ void invalidate_bh_lrus(void) }EXPORT_SYMBOL_GPL (invalidate_bh_lrus); -void invalidate_bh_lrus_cpu (int cpu) +/* +* It's called from workqueue context so we need a bh_lru_lock to close +* the race with preemption/irq. +*/ +void invalidate_bh_lrus_cpu (void ) {struct bh_lru *b;bh_lru_lock (); - b =per_cpu_ptr (&bh_lrus, cpu); + b =this_cpu_ptr (&bh_lrus);__invalidate_bh_lrus (b);bh_lru_unlock (); }include/linux/buffer_head.h +1
-1@@ -194,7 +194,7 @@ void __breadahead_gfp(struct block_device *, sector_t block, unsigned int size, struct buffer_head *__bread_gfp(struct block_device *,sector_t block,unsigned size,gfp_t gfp);void invalidate_bh_lrus (void ); -void invalidate_bh_lrus_cpu (int cpu); +void invalidate_bh_lrus_cpu (void );bool has_bh_in_lru (int cpu,void *dummy);struct buffer_head *alloc_buffer_head (gfp_t gfp_flags);void
free_buffer_head(struct buffer_head * bh);include/linux/buffer_head.h +1
-1@@ -408,7 +408,7 @@ static inline int inode_has_buffers(struct inode *inode) { return 0; }static inline void invalidate_inode_buffers (struct inode *inode) {}static inline int remove_inode_buffers (struct inode *inode) {return 1 ; }static inline int sync_mapping_buffers (struct address_space *mapping) {return 0 ; } -static inline void invalidate_bh_lrus_cpu (int cpu) {} +static inline void invalidate_bh_lrus_cpu (void ) {}static inline bool
has_bh_in_lru(int cpu, void *dummy){ return 0; } #definebuffer_heads_over_limit0mm/swap.c +15
-2@@ -642,7 +642,6 @@ void lru_add_drain_cpu(int cpu)pagevec_lru_move_fn (pvec, lru_lazyfree_fn);activate_page_drain (cpu); -invalidate_bh_lrus_cpu (cpu); }/** @@ -725,6 +724,20 @@ void lru_add_drain(void)local_unlock (&lru_pvecs.lock); } +/* +* It's called from per-cpu workqueue context in SMP case so +* lru_add_drain_cpu and invalidate_bh_lrus_cpu should run on +* the same cpu. It shouldn't be a problem in !SMP case since +* the core is only one and the locks will disable preemption. +*/ +static void lru_add_and_bh_lrus_drain (void ) +{ +local_lock (&lru_pvecs.lock); +lru_add_drain_cpu (smp_processor_id ()); +local_unlock (&lru_pvecs.lock); +invalidate_bh_lrus_cpu (); +} +void lru_add_drain_cpu_zone (struct zone *zone) {local_lock (&lru_pvecs.lock);@@ -739,7 +752,7 @@ static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work);static void lru_add_drain_per_cpu (struct work_struct *dummy) { -lru_add_drain (); +lru_add_and_bh_lrus_drain (); }/*
mm/swap.c void lru_cache_disable(void) · line 898
Quoted 7 lines
mm/swap.c +1
-1@@ -880,7 +893,7 @@ void lru_cache_disable(void)*/ __lru_add_drain_all (true);#else -lru_add_drain (); +lru_add_and_bh_lrus_drain ();#endif }Hi Minchan,
Quoted 3 lines
This looks good to me. Feel free to add:
Reviewed-by Chris Goldsworthy <[email protected]>
Thanks for the review, Chris.
Andrew, could you take a look?
Quoted 189 lines · Minchan Kim
Quoted 188 lines · Chris Goldsworthy
Quoted 177 lines · Minchan Kim
Quoted 14 lines · Andrew Morton
Quoted 10 lines · On Tue, 1 Jun 2021 07:54:25 -0700 Minchan Kim <[email protected]> wrote:
kernel test robot reported the regression of fio.write_iops[1] with [2].
Since
lru_add_drainis called frequently, invalidatebh_lrusthere could increasebh_lruscache miss ratio, which needs more IO in the end.This patch moves the
bh_lrusinvalidation from the hotpath( e.g., zap_page_range, pagevec_release)to coldpath(i.e., lru_add_drain_all, lru_cache_disable).Quoted 7 lines
This code is starting to hurt my brain.
What are the locking/context rules for
invalidate_bh_lrus_cpu()?AFAICT it offers no protection against two CPUs concurrently running
__invalidate_bh_lrus()against the samebh_lru.The
lru_add_drain_per_cpuwill run on per-cpu since it's per-cpu work andinvalidate_bh_lrus_cpuwill run underbh_lru_lockso I couldn't imagine that race can happen.Quoted 4 lines
So when
CONFIG_SMP=y,invalidate_bh_lrus_cpu()must always and only be run on the cpu which owns thebh_lru. In which case why does it have the `cpu' arg?I just wanted to express both
lru_add_drain_cpuandinvalidate_bh_lrus_cpu
inlru_add_and_bh_lrus_drainrun in the same cpu but look like a bad idea
since it makes people confused. Let me remove the cpu argument frominvalidate_bh_lrus_cpu.Quoted 10 lines
Your new
lru_add_and_bh_lrus_drain()follows these rules by callinginvalidate_bh_lrus_cpu()from a per-cpu worker or whenCONFIG_SMP=n.I think. It's all as clear as mud and undocumented. Could you please take a look at this? Comment the locking/context rules thoroughly and check that they are being followed? Not forgetting cpu hotplug... See if
there's a way of simplifying/clarifying the code?Quoted 5 lines
The fact that
swap.chas those #ifdefCONFIG_SMPsin there is a hint that we're doing something wrong (or poorly) in there. Perhaps that's unavoidable because of all the fancy footwork in__lru_add_drain_all().Hopefully, this is better.
From
8d58e7ade3ed6c080995dec1395b1e130b3d16b3Mon Sep 17 00:00:00 2001
From: Minchan Kim <[email protected]>
Date: Tue, 25 May 2021 08:19:17 -0700
Subject: [PATCH] mm: fs: invalidatebh_lrusfor only cold pathkernel test robot reported the regression of fio.write_iops[1] with [2].
Since
lru_add_drainis called frequently, invalidatebh_lrusthere could increasebh_lruscache miss ratio, which needs more IO in the end.This patch moves the
bh_lrusinvalidation from the hotpath( e.g., zap_page_range, pagevec_release)to coldpath(i.e., lru_add_drain_all, lru_cache_disable).[1]
https://lore.kernel.org/lkml/20210520083144.GD14190(a)xsang-OptiPlex-9020/
[2]8cc621d2f45d, mm: fs: invalidate BH LRU during page migrationReported-by kernel test robot <[email protected]>Signed-off-by Minchan Kim <[email protected]>--- fs/buffer.c | 8 ++++++-- include/linux/buffer_head.h | 4 ++-- mm/swap.c | 19 ++++++++++++++++--- 3 files changed, 24 insertions(+), 7 deletions(-)fs/buffer.c +6
-2@@ -1487,12 +1487,16 @@ void invalidate_bh_lrus(void) }EXPORT_SYMBOL_GPL (invalidate_bh_lrus); -void invalidate_bh_lrus_cpu (int cpu) +/* +* It's called from workqueue context so we need a bh_lru_lock to close +* the race with preemption/irq. +*/ +void invalidate_bh_lrus_cpu (void ) {struct bh_lru *b;bh_lru_lock (); - b =per_cpu_ptr (&bh_lrus, cpu); + b =this_cpu_ptr (&bh_lrus);__invalidate_bh_lrus (b);bh_lru_unlock (); }include/linux/buffer_head.h +1
-1@@ -194,7 +194,7 @@ void __breadahead_gfp(struct block_device *, sector_t block, unsigned int size, struct buffer_head *__bread_gfp(struct block_device *,sector_t block,unsigned size,gfp_t gfp);void invalidate_bh_lrus (void ); -void invalidate_bh_lrus_cpu (int cpu); +void invalidate_bh_lrus_cpu (void );bool has_bh_in_lru (int cpu,void *dummy);struct buffer_head *alloc_buffer_head (gfp_t gfp_flags);void
free_buffer_head(struct buffer_head * bh);include/linux/buffer_head.h +1
-1@@ -408,7 +408,7 @@ static inline int inode_has_buffers(struct inode *inode) { return 0; }static inline void invalidate_inode_buffers (struct inode *inode) {}static inline int remove_inode_buffers (struct inode *inode) {return 1 ; }static inline int sync_mapping_buffers (struct address_space *mapping) {return 0 ; } -static inline void invalidate_bh_lrus_cpu (int cpu) {} +static inline void invalidate_bh_lrus_cpu (void ) {}static inline bool
has_bh_in_lru(int cpu, void *dummy){ return 0; } #definebuffer_heads_over_limit0mm/swap.c +15
-2@@ -642,7 +642,6 @@ void lru_add_drain_cpu(int cpu)pagevec_lru_move_fn (pvec, lru_lazyfree_fn);activate_page_drain (cpu); -invalidate_bh_lrus_cpu (cpu); }/** @@ -725,6 +724,20 @@ void lru_add_drain(void)local_unlock (&lru_pvecs.lock); } +/* +* It's called from per-cpu workqueue context in SMP case so +* lru_add_drain_cpu and invalidate_bh_lrus_cpu should run on +* the same cpu. It shouldn't be a problem in !SMP case since +* the core is only one and the locks will disable preemption. +*/ +static void lru_add_and_bh_lrus_drain (void ) +{ +local_lock (&lru_pvecs.lock); +lru_add_drain_cpu (smp_processor_id ()); +local_unlock (&lru_pvecs.lock); +invalidate_bh_lrus_cpu (); +} +void lru_add_drain_cpu_zone (struct zone *zone) {local_lock (&lru_pvecs.lock);@@ -739,7 +752,7 @@ static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work);static void lru_add_drain_per_cpu (struct work_struct *dummy) { -lru_add_drain (); +lru_add_and_bh_lrus_drain (); }/*
mm/swap.c void lru_cache_disable(void) · line 898
Quoted 7 lines
mm/swap.c +1
-1@@ -880,7 +893,7 @@ void lru_cache_disable(void)*/ __lru_add_drain_all (true);#else -lru_add_drain (); +lru_add_and_bh_lrus_drain ();#endif }Hi Minchan,
Quoted 7 lines
Quoted 3 lines
This looks good to me. Feel free to add:
Reviewed-by Chris Goldsworthy <[email protected]>Thanks for the review, Chris.
Andrew, could you take a look?
Hi Andrew,
Have you been able to look over the second version of Minchan's patch?
Thanks,
Chris.
Signature · 4 lines
The Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project