mm: move FOLL_PIN debug accounting under CONFIG_DEBUG_VM
Subsystems (5)
Jens Axboe
maintainerPATCH 1/1 mm: move FOLL_PIN debug accounting under CONFIG_DEBUG_VM
Using FOLL_PIN for mapping user pages caused a performance regression of about 2.7%. Looking at profiles, we see:
+2.71% [kernel.vmlinux] [k] mod_node_page_state
which wasn't there before. The node page state counters are percpu, but with a very low threshold. On my setup, every 108th update ends up needing to punt to two atomic_lond_add()'s, which is causing this above regression.
As these counters are purely for debug purposes, move them under CONFIG_DEBUG_VM rather than do them unconditionally.
---
I added fixes tags, even though it's not a strict fix for this commits. But it does fix a performance regression introduced by those commits. It's a useful hint for backporting.
I'd prefer sticking this at the end of the iov-extract series that is already pulled in, so it can go with those patches.
include/linux/mmzone.h +2 -0
@@ -195,8 +195,10 @@ enum node_stat_item {NR_WRITTEN ,/* page writings since bootup */ NR_THROTTLED_WRITTEN ,/* NR_WRITTEN while reclaim throttled */ NR_KERNEL_MISC_RECLAIMABLE ,/* reclaimable non-slab kernel pages */ +#ifdef CONFIG_DEBUG_VMNR_FOLL_PIN_ACQUIRED ,/* via: pin_user_page(), gup flag: FOLL_PIN */ NR_FOLL_PIN_RELEASED ,/* pages returned via unpin_user_page() */ +#endifNR_KERNEL_STACK_KB ,/* measured in KiB */ #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) NR_KERNEL_SCS_KB, /* measured in KiB */
mm/gup.c +6 -1
@@ -168,7 +168,9 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)*/ smp_mb__after_atomic (); +#ifdef CONFIG_DEBUG_VM node_stat_mod_folio (folio,NR_FOLL_PIN_ACQUIRED , refs); +#endif return folio; }
@@ -180,7 +182,9 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)static void gup_put_folio (struct folio *folio,int refs,unsigned int flags) {if (flags &FOLL_PIN ) { +#ifdef CONFIG_DEBUG_VM node_stat_mod_folio (folio,NR_FOLL_PIN_RELEASED , refs); +#endif if (folio_test_large (folio))atomic_sub (refs,folio_pincount_ptr (folio));else
@@ -236,8 +240,9 @@ int __must_check try_grab_page(struct page *page, unsigned int flags) }else {folio_ref_add (folio,GUP_PIN_COUNTING_BIAS ); } - +#ifdef CONFIG_DEBUG_VM node_stat_mod_folio (folio,NR_FOLL_PIN_ACQUIRED ,1 ); +#endif }return 0 ;
mm/vmstat.c +2 -0
@@ -1227,8 +1227,10 @@ const char * const vmstat_text[] = {"nr_written" ,"nr_throttled_written" ,"nr_kernel_misc_reclaimable" , +#ifdef CONFIG_DEBUG_VM"nr_foll_pin_acquired" ,"nr_foll_pin_released" , +#endif"nr_kernel_stack" , #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) "nr_shadow_call_stack",
Signature · 1 line
Jens Axboe
Quoted 18 lines · Jens Axboe
Using
FOLL_PINfor mapping user pages caused a performance regression of about 2.7%. Looking at profiles, we see:+2.71% [kernel.vmlinux] [k]
mod_node_page_statewhich wasn't there before. The node page state counters are percpu, but with a very low threshold. On my setup, every 108th update ends up needing to punt to two atomic_lond_add()'s, which is causing this above regression.
As these counters are purely for debug purposes, move them under
CONFIG_DEBUG_VMrather than do them unconditionally.Fixes fd20d0c1852e ("block: convert bio_map_user_iov to use iov_iter_extract_pages")Fixes 920756a3306a ("block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages")Signed-off-by Jens Axboe <[email protected]>
Quoted 5 lines
---
I added fixes tags, even though it's not a strict fix for this commits. But it does fix a performance regression introduced by those commits. It's a useful hint for backporting.
I'd just mention them in the commit log instead, but I don't particularly care here as long as the commit ID's are stable.
If still possible, I'd include this as a preparational change for these commits instead. Anyhow
Signature · 3 lines
Thanks, David / dhildenb
Jens Axboe
maintainerQuoted 19 lines · David Hildenbrand
Quoted 18 lines · Jens Axboe
Using
FOLL_PINfor mapping user pages caused a performance regression of about 2.7%. Looking at profiles, we see:+2.71% [kernel.vmlinux] [k]
mod_node_page_statewhich wasn't there before. The node page state counters are percpu, but with a very low threshold. On my setup, every 108th update ends up needing to punt to two atomic_lond_add()'s, which is causing this above regression.
As these counters are purely for debug purposes, move them under
CONFIG_DEBUG_VMrather than do them unconditionally.Fixes fd20d0c1852e ("block: convert bio_map_user_iov to use iov_iter_extract_pages")Fixes 920756a3306a ("block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages")Signed-off-by Jens Axboe <[email protected]>
Quoted 8 lines
Quoted 5 lines
---
I added fixes tags, even though it's not a strict fix for this commits. But it does fix a performance regression introduced by those commits. It's a useful hint for backporting.
I'd just mention them in the commit log instead, but I don't particularly care here as long as the commit ID's are stable.
Sure, I can move that bit into the commit message.
Quoted 2 lines
If still possible, I'd include this as a preparational change for these commits instead. Anyhow
That would be preferable in general, but I don't think it's worth rebasing the series just for that.
Quoted 1 line
Thanks! Will add in conjunction with updating the commit message.
Signature · 1 line
Jens Axboe
Quoted 10 lines · Jens Axboe
Using
FOLL_PINfor mapping user pages caused a performance regression of about 2.7%. Looking at profiles, we see:+2.71% [kernel.vmlinux] [k]
mod_node_page_statewhich wasn't there before. The node page state counters are percpu, but with a very low threshold. On my setup, every 108th update ends up needing to punt to two atomic_lond_add()'s, which is causing this above regression.
Quoted 8 lines
As these counters are purely for debug purposes, move them under
CONFIG_DEBUG_VMrather than do them unconditionally.Fixes fd20d0c1852e ("block: convert bio_map_user_iov to use iov_iter_extract_pages")Fixes 920756a3306a ("block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages")Signed-off-by Jens Axboe <[email protected]>
Yes, that's the exact right fix (in the absence of some magical high-volume, high performance per-cpu counters anyway). In fact, originally these counter operations were behind CONFIG_DEBUG_VM in an early patchset, but the FOLL_PIN feature was new and potentially flaky, and also not yet in the Direct IO fast path. So during code review we decided to enable the debugging information unconditionally.
But now we can no longer afford the debug counters in release builds--but we also no longer need them, because the FOLL_PIN feature has settled in and had enough soak time to be more confident about it.
Over time, I'd kind of started assuming that these counters were necessary for release builds, and it required someone else to wake me up and point out the obvious, so thanks! :)
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 75 lines
---
I added fixes tags, even though it's not a strict fix for this commits. But it does fix a performance regression introduced by those commits. It's a useful hint for backporting.
I'd prefer sticking this at the end of the iov-extract series that is already pulled in, so it can go with those patches.
include/linux/mmzone.h +2
-0@@ -195,8 +195,10 @@ enum node_stat_item {NR_WRITTEN ,/* page writings since bootup */ NR_THROTTLED_WRITTEN ,/* NR_WRITTEN while reclaim throttled */ NR_KERNEL_MISC_RECLAIMABLE ,/* reclaimable non-slab kernel pages */ +#ifdef CONFIG_DEBUG_VMNR_FOLL_PIN_ACQUIRED ,/* via: pin_user_page(), gup flag: FOLL_PIN */ NR_FOLL_PIN_RELEASED ,/* pages returned via unpin_user_page() */ +#endifNR_KERNEL_STACK_KB ,/* measured in KiB */ #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) NR_KERNEL_SCS_KB, /* measured in KiB */mm/gup.c +6
-1@@ -168,7 +168,9 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)*/ smp_mb__after_atomic (); +#ifdef CONFIG_DEBUG_VM node_stat_mod_folio (folio,NR_FOLL_PIN_ACQUIRED , refs); +#endif return folio; }@@ -180,7 +182,9 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)static void gup_put_folio (struct folio *folio,int refs,unsigned int flags) {if (flags &FOLL_PIN ) { +#ifdef CONFIG_DEBUG_VM node_stat_mod_folio (folio,NR_FOLL_PIN_RELEASED , refs); +#endif if (folio_test_large (folio))atomic_sub (refs,folio_pincount_ptr (folio));else @@ -236,8 +240,9 @@ int __must_check try_grab_page(struct page *page, unsigned int flags) }else {folio_ref_add (folio,GUP_PIN_COUNTING_BIAS ); } - +#ifdef CONFIG_DEBUG_VM node_stat_mod_folio (folio,NR_FOLL_PIN_ACQUIRED ,1 ); +#endif }return 0 ;mm/vmstat.c +2
-0@@ -1227,8 +1227,10 @@ const char * const vmstat_text[] = {"nr_written" ,"nr_throttled_written" ,"nr_kernel_misc_reclaimable" , +#ifdef CONFIG_DEBUG_VM"nr_foll_pin_acquired" ,"nr_foll_pin_released" , +#endif"nr_kernel_stack" , #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) "nr_shadow_call_stack",