Hard and soft lockups with FIO and LTP runs on a large system
Subsystems (6)
Many soft and hard lockups are seen with upstream kernel when running a bunch of tests that include FIO and LTP filesystem test on 10 NVME disks. The lockups can appear anywhere between 2 to 48 hours. Originally this was reported on a large customer VM instance with passthrough NVME disks on older kernels(v5.4 based). However, similar problems were reproduced when running the tests on bare metal with latest upstream kernel (v6.10-rc3). Other lockups with different signatures are seen but in this report, only those related to MM area are being discussed. Also note that the subsequent description is related to the lockups in bare metal upstream (and not VM).
The general observation is that the problem usually surfaces when the system free memory goes very low and page cache/buffer consumption hits the ceiling. Most of the times the two contended locks are lruvec and inode->i_lock spinlocks.
- Could this be a scalability issue in LRU list handling and/or page cache invalidation typical to a large system configuration?
- Are there any MM/FS tunables that could help here?
Hardware configuration
======================
Dual socket AMD EPYC 128 Core processor (256 cores, 512 threads)
Memory: 1.5 TB
10 NVME - 3.5TB each
available: 2 nodes (0-1)
node 0 cpus: 0-127,256-383
node 0 size: 773727 MB
node 1 cpus: 128-255,384-511
node 1 size: 773966 MB
Workload details
================
Workload includes concurrent runs of FIO and a few FS tests from LTP.
FIO is run with a size of 1TB on each NVME partition with different combinations of ioengine/blocksize/mode parameters and buffered-IO. Selected FS tests from LTP are run on 256GB partitions of all NVME disks. This is the typical NVME partition layout.
nvme2n1 259:4 0 3.5T 0 disk ├─nvme2n1p1 259:6 0 256G 0 part /data_nvme2n1p1 └─nvme2n1p2 259:7 0 3.2T 0 part
Though many different runs exist in the workload, the combination that results in the problem is buffered-IO run with sync engine.
fio -filename=/dev/nvme1n1p2 -direct=0 -thread -size=1024G \
-rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=sync -bs=4k \
-numjobs=400 -runtime=25000 --time_based -group_reporting -name=mytest
Watchdog threshold was reduced to 5s to reproduce the problem early and all CPU backtrace enabled.
Problem details and analysis
============================
One of the hard lockups which was observed and analyzed in detail is this:
kernel: watchdog: Watchdog detected hard LOCKUP on cpu 284 kernel: CPU: 284 PID: 924096 Comm: cat Not tainted 6.10.0-rc3-lruvec #9 kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: Call Trace: kernel: <NMI> kernel: ? show_regs+0x69/0x80 kernel: ? watchdog_hardlockup_check+0x19e/0x360 <SNIP> kernel: ? native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: </NMI> kernel: <TASK> kernel: ? __pfx_lru_add_fn+0x10/0x10 kernel: _raw_spin_lock_irqsave+0x42/0x50 kernel: folio_lruvec_lock_irqsave+0x62/0xb0 kernel: folio_batch_move_lru+0x79/0x2a0 kernel: folio_add_lru+0x6d/0xf0 kernel: filemap_add_folio+0xba/0xe0 kernel: __filemap_get_folio+0x137/0x2e0 kernel: ext4_da_write_begin+0x12c/0x270 kernel: generic_perform_write+0xbf/0x200 kernel: ext4_buffered_write_iter+0x67/0xf0 kernel: ext4_file_write_iter+0x70/0x780 kernel: vfs_write+0x301/0x420 kernel: ksys_write+0x67/0xf0 kernel: __x64_sys_write+0x19/0x20 kernel: x64_sys_call+0x1689/0x20d0 kernel: do_syscall_64+0x6b/0x110 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e kernel: RIP: 0033:0x7fe21c314887
With all CPU backtraces enabled, many CPUs are waiting for lruvec_lock acquisition. We measured the lruvec spinlock start, end and hold time(htime) using sched_clock(), along with a BUG() if the hold time was more than 10s. The below case shows that lruvec spin lock was held for ~25s.
kernel: vmscan: unlock_page_lruvec_irq: stime 27963327514341, etime 27963324369895, htime 25889317166
kernel: ------------[ cut here ]------------
kernel: kernel BUG at include/linux/memcontrol.h:1677!
kernel: Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
kernel: CPU: 21 PID: 3211 Comm: kswapd0 Tainted: G W 6.10.0-rc3-qspindbg #10
kernel: RIP: 0010:shrink_active_list+0x40a/0x520
And the corresponding trace point for the above:
kswapd0-3211 [021] dN.1. 27963.324332: mm_vmscan_lru_isolate: classzone=0 order=0 nr_requested=1 nr_scanned=156946361 nr_skipped=156946360 nr_taken=1 lru=active_file
This shows that isolate_lru_folios() is scanning through a huge number (~150million) of folios (order=0) with lruvec spinlock held. This is happening because a large number of folios are being skipped to isolate a few ZONE_DMA folios. Though the number of folios to be scanned is bounded (32), there exists a genuine case where this can become unbounded, i.e. in case where folios are skipped.
Meminfo output shows that the free memory is around ~2% and page/buffer cache grows very high when the lockup happens.
MemTotal: 1584835956 kB MemFree: 27805664 kB MemAvailable: 1568099004 kB Buffers: 1386120792 kB Cached: 151894528 kB SwapCached: 30620 kB Active: 1043678892 kB Inactive: 494456452 kB
Often times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inode i_lock (in clear_shadow_entry path)
98.98% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath
|
--98.96%--native_queued_spin_lock_slowpath
|
--98.96%--_raw_spin_lock_irqsave
folio_lruvec_lock_irqsave
|
--98.78%--folio_batch_move_lru
|
--98.63%--deactivate_file_folio
mapping_try_invalidate
invalidate_mapping_pages
invalidate_bdev
blkdev_common_ioctl
blkdev_ioctl
__x64_sys_ioctl
x64_sys_call
do_syscall_64
entry_SYSCALL_64_after_hwframeSome experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.
kernel: watchdog: BUG: soft lockup - CPU#29 stuck for 11s! [fio:2701649] kernel: CPU: 29 PID: 2701649 Comm: fio Tainted: G L 6.10.0-rc3-mglru-irqstrc #24 kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: Call Trace: kernel: <IRQ> kernel: ? show_regs+0x69/0x80 kernel: ? watchdog_timer_fn+0x223/0x2b0 kernel: ? __pfx_watchdog_timer_fn+0x10/0x10 <SNIP> kernel: </IRQ> kernel: <TASK> kernel: ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 kernel: ? native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: _raw_spin_lock+0x38/0x50 kernel: clear_shadow_entry+0x3d/0x100 kernel: ? __pfx_workingset_update_node+0x10/0x10 kernel: mapping_try_invalidate+0x117/0x1d0 kernel: invalidate_mapping_pages+0x10/0x20 kernel: invalidate_bdev+0x3c/0x50 kernel: blkdev_common_ioctl+0x5f7/0xa90 kernel: blkdev_ioctl+0x109/0x270 kernel: x64_sys_call+0x1215/0x20d0 kernel: do_syscall_64+0x7e/0x130
This happens to be contending on inode i_lock spinlock.
Below preemptirqsoff trace points to preemption being disabled for more than 10s and the lock in picture is lruvec spinlock.
# tracer: preemptirqsoff
#
# preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-mglru-irqstrc
# --------------------------------------------------------------------
# latency: 10382682 us, #4/4, CPU#128 | (M:desktop VP:0, KP:0, SP:0
HP:0 #P:512)
# -----------------
# | task: fio-2701523 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: deactivate_file_folio
# => ended at: deactivate_file_folio
#
#
# _------=> CPU#
# / _-----=> irqs-off/BH-disabled
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / _-=> migrate-disable
# ||||| / delay
# cmd pid |||||| time | caller
# \ / |||||| \ | /
fio-2701523 128...1. 0us$: deactivate_file_folio
<-deactivate_file_folio
fio-2701523 128.N.1. 10382681us : deactivate_file_folio
<-deactivate_file_folio
fio-2701523 128.N.1. 10382683us : tracer_preempt_on
<-deactivate_file_folio
fio-2701523 128.N.1. 10382691us : <stack trace>
=> deactivate_file_folio
=> mapping_try_invalidate
=> invalidate_mapping_pages
=> invalidate_bdev
=> blkdev_common_ioctl
=> blkdev_ioctl
=> __x64_sys_ioctl
=> x64_sys_call
=> do_syscall_64
=> entry_SYSCALL_64_after_hwframe2) Increased low_watermark_threshold to 10% to prevent system from entering into extremely low memory situation. Although hard lockups weren't seen, but soft lockups (clear_shadow_entry()) were still seen.
3) AMD has a BIOS setting called NPS (Nodes per socket), using which a socket can be further partitioned into smaller NUMA nodes. With NPS=4, there will be four NUMA nodes in one socket, and hence 8 NUMA nodes in the system. This was done to check if having more number of kswapd threads working on lesser number of folios per node would make a difference. However here too, multiple soft lockups were seen (in clear_shadow_entry() as seen in MGLRU case). No hard lockups were observed.
Any insights/suggestion into these lockups and suggestions are welcome!
Regards,
Bharata.
Hi Bharata,
Quoted 151 lines · Bharata B Rao
Many soft and hard lockups are seen with upstream kernel when running a bunch of tests that include FIO and LTP filesystem test on 10 NVME disks. The lockups can appear anywhere between 2 to 48 hours. Originally this was reported on a large customer VM instance with passthrough NVME disks on older kernels(v5.4 based). However, similar problems were reproduced when running the tests on bare metal with latest upstream kernel (v6.10-rc3). Other lockups with different signatures are seen but in this report, only those related to MM area are being discussed. Also note that the subsequent description is related to the lockups in bare metal upstream (and not VM).
The general observation is that the problem usually surfaces when the system free memory goes very low and page cache/buffer consumption hits the ceiling. Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.- Could this be a scalability issue in LRU list handling and/or page cache invalidation typical to a large system configuration?
- Are there any MM/FS tunables that could help here?Hardware configuration
======================
Dual socket AMD EPYC 128 Core processor (256 cores, 512 threads)
Memory: 1.5 TB
10 NVME - 3.5TB each
available: 2 nodes (0-1)
node 0 cpus: 0-127,256-383
node 0 size: 773727 MB
node 1 cpus: 128-255,384-511
node 1 size: 773966 MBWorkload details
================
Workload includes concurrent runs of FIO and a few FS tests from LTP.FIO is run with a size of 1TB on each NVME partition with different combinations of ioengine/blocksize/mode parameters and buffered-IO. Selected FS tests from LTP are run on 256GB partitions of all NVME disks. This is the typical NVME partition layout.
nvme2n1 259:4 0 3.5T 0 disk ├─nvme2n1p1 259:6 0 256G 0 part /data_nvme2n1p1 └─nvme2n1p2 259:7 0 3.2T 0 partThough many different runs exist in the workload, the combination that results in the problem is buffered-IO run with sync engine.
fio -filename=/dev/nvme1n1p2 -direct=0 -thread -size=1024G \
(quoted diff) +0
-2-rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=sync -bs=4k \ -numjobs=400 -runtime=25000 --time_based -group_reporting -name=mytestWatchdog threshold was reduced to 5s to reproduce the problem early and all CPU backtrace enabled.
Problem details and analysis
============================
One of the hard lockups which was observed and analyzed in detail is this:kernel: watchdog: Watchdog detected hard LOCKUP on cpu 284 kernel: CPU: 284 PID: 924096 Comm: cat Not tainted 6.10.0-rc3-lruvec #9 kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: Call Trace: kernel: <NMI> kernel: ? show_regs+0x69/0x80 kernel: ? watchdog_hardlockup_check+0x19e/0x360 <SNIP> kernel: ? native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: </NMI> kernel: <TASK> kernel: ? __pfx_lru_add_fn+0x10/0x10 kernel: _raw_spin_lock_irqsave+0x42/0x50 kernel: folio_lruvec_lock_irqsave+0x62/0xb0 kernel: folio_batch_move_lru+0x79/0x2a0 kernel: folio_add_lru+0x6d/0xf0 kernel: filemap_add_folio+0xba/0xe0 kernel: __filemap_get_folio+0x137/0x2e0 kernel: ext4_da_write_begin+0x12c/0x270 kernel: generic_perform_write+0xbf/0x200 kernel: ext4_buffered_write_iter+0x67/0xf0 kernel: ext4_file_write_iter+0x70/0x780 kernel: vfs_write+0x301/0x420 kernel: ksys_write+0x67/0xf0 kernel: __x64_sys_write+0x19/0x20 kernel: x64_sys_call+0x1689/0x20d0 kernel: do_syscall_64+0x6b/0x110 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e kernel: RIP:0033:0x7fe21c314887
With all CPU backtraces enabled, many CPUs are waiting for
lruvec_lockacquisition. We measured the lruvec spinlock start, end and hold time(htime) usingsched_clock(), along with aBUG()if the hold time was more than 10s. The below case shows that lruvec spin lock was held for ~25s.kernel: vmscan:
unlock_page_lruvec_irq: stime 27963327514341, etime 27963324369895, htime 25889317166
kernel: ------------[ cut here ]------------
kernel: kernel BUG atinclude/linux/memcontrol.h:1677!
kernel: Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
kernel: CPU: 21 PID: 3211 Comm: kswapd0 Tainted: G W
6.10.0-rc3-qspindbg #10
kernel: RIP: 0010:shrink_active_list+0x40a/0x520And the corresponding trace point for the above:
kswapd0-3211 [021] dN.1. 27963.324332:mm_vmscan_lru_isolate:
classzone=0 order=0 nr_requested=1 nr_scanned=156946361 nr_skipped=156946360 nr_taken=1 lru=active_fileThis shows that
isolate_lru_folios()is scanning through a huge number (~150million) of folios (order=0) with lruvec spinlock held. This is happening because a large number of folios are being skipped to isolate a fewZONE_DMAfolios. Though the number of folios to be scanned is bounded (32), there exists a genuine case where this can become unbounded, i.e. in case where folios are skipped.Meminfo output shows that the free memory is around ~2% and page/buffer cache grows very high when the lockup happens.
MemTotal: 1584835956 kB MemFree: 27805664 kB MemAvailable: 1568099004 kB Buffers: 1386120792 kB Cached: 151894528 kB SwapCached: 30620 kB Active: 1043678892 kB Inactive: 494456452 kBOften times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inode
i_lock(inclear_shadow_entrypath)98.98% fio [kernel.kallsyms] [k]
native_queued_spin_lock_slowpath| --98.96%--native_queued_spin_lock_slowpath | --98.96%--_raw_spin_lock_irqsave folio_lruvec_lock_irqsave | --98.78%--folio_batch_move_lru | --98.63%--deactivate_file_folio mapping_try_invalidate invalidate_mapping_pages invalidate_bdev blkdev_common_ioctl blkdev_ioctl __x64_sys_ioctl x64_sys_call do_syscall_64 entry_SYSCALL_64_after_hwframe
Quoted 4 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.
This is not really an MGLRU issue -- can you please try one of the attached patches? It (truncate.patch) should help with or without MGLRU.
Quoted 25 lines
kernel: watchdog: BUG: soft lockup - CPU#29 stuck for 11s! [fio:2701649]
kernel: CPU: 29 PID: 2701649 Comm: fio Tainted: G L
6.10.0-rc3-mglru-irqstrc #24kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: Call Trace: kernel: <IRQ> kernel: ? show_regs+0x69/0x80 kernel: ? watchdog_timer_fn+0x223/0x2b0 kernel: ? __pfx_watchdog_timer_fn+0x10/0x10 <SNIP> kernel: </IRQ> kernel: <TASK> kernel: ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 kernel: ? native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: _raw_spin_lock+0x38/0x50 kernel: clear_shadow_entry+0x3d/0x100 kernel: ? __pfx_workingset_update_node+0x10/0x10 kernel: mapping_try_invalidate+0x117/0x1d0 kernel: invalidate_mapping_pages+0x10/0x20 kernel: invalidate_bdev+0x3c/0x50 kernel: blkdev_common_ioctl+0x5f7/0xa90 kernel: blkdev_ioctl+0x109/0x270 kernel: x64_sys_call+0x1215/0x20d0 kernel: do_syscall_64+0x7e/0x130
Quoted 4 lines
This happens to be contending on inode
i_lockspinlock.Below preemptirqsoff trace points to preemption being disabled for more than 10s and the lock in picture is lruvec spinlock.
Also if you could try the other patch (mglru.patch) please. It should help reduce unnecessary rotations from deactivate_file_folio(), which in turn should reduce the contention on the LRU lock for MGLRU.
Quoted 52 lines
# tracer: preemptirqsoff # # preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-mglru-irqstrc # -------------------------------------------------------------------- # latency: 10382682 us, #4/4, CPU#128 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:512) # ----------------- # | task: fio-2701523 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # => started at: deactivate_file_folio # => ended at: deactivate_file_folio # # # _------=> CPU# # / _-----=> irqs-off/BH-disabled # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / _-=> migrate-disable # ||||| / delay # cmd pid |||||| time | caller # \ / |||||| \ | / fio-2701523 128...1. 0us$: deactivate_file_folio <-deactivate_file_folio fio-2701523 128.N.1. 10382681us : deactivate_file_folio <-deactivate_file_folio fio-2701523 128.N.1. 10382683us : tracer_preempt_on <-deactivate_file_folio fio-2701523 128.N.1. 10382691us : <stack trace> => deactivate_file_folio => mapping_try_invalidate => invalidate_mapping_pages => invalidate_bdev => blkdev_common_ioctl => blkdev_ioctl => __x64_sys_ioctl => x64_sys_call => do_syscall_64 => entry_SYSCALL_64_after_hwframe2) Increased
low_watermark_thresholdto 10% to prevent system from entering into extremely low memory situation. Although hard lockups weren't seen, but soft lockups (clear_shadow_entry()) were still seen.3) AMD has a BIOS setting called NPS (Nodes per socket), using which a socket can be further partitioned into smaller NUMA nodes. With NPS=4, there will be four NUMA nodes in one socket, and hence 8 NUMA nodes in the system. This was done to check if having more number of kswapd threads working on lesser number of folios per node would make a difference. However here too, multiple soft lockups were seen (in
clear_shadow_entry()as seen in MGLRU case). No hard lockups were observed.
Quoted 1 line
Any insights/suggestion into these lockups and suggestions are welcome!
Thanks!
include/linux/mm_inline.h +10 -0
@@ -182,6 +182,16 @@ static inline bool lru_gen_is_active(struct lruvec *lruvec, int gen)return gen ==lru_gen_from_seq (max_seq) || gen ==lru_gen_from_seq (max_seq -1 ); } +static inline bool lru_gen_should_rotate (struct folio *folio) +{ +int gen =folio_lru_gen (folio); +int type =folio_is_file_lru (folio); +struct lruvec *lruvec =folio_lruvec (folio); +struct lru_gen_folio *lrugen = &lruvec->lrugen; + +return gen !=lru_gen_from_seq (READ_ONCE (lrugen->min_seq[type])); +} +static inline void lru_gen_update_size (struct lruvec *lruvec,struct folio *folio,int old_gen,int new_gen) {
mm/swap.c +1 -1
@@ -692,7 +692,7 @@ void deactivate_file_folio(struct folio *folio)struct folio_batch *fbatch;/* Deactivating an unevictable folio will not accelerate reclaim */ -if (folio_test_unevictable (folio)) +if (folio_test_unevictable (folio) || !lru_gen_should_rotate (folio))return ;folio_get (folio);
mm/truncate.c +30 -37
118 lines not on this page · raw message
Hi Yu Zhao,
Thanks for your patches. See below...
Quoted 4 lines · Yu Zhao
Hi Bharata,
Quoted 1 line · Bharata B Rao
<snip>
Quoted 9 lines
Quoted 5 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.This is not really an MGLRU issue -- can you please try one of the attached patches? It (truncate.patch) should help with or without MGLRU.
With truncate.patch and default LRU scheme, a few hard lockups are seen.
First one is this:
watchdog: Watchdog detected hard LOCKUP on cpu 487
CPU: 487 PID: 11525 Comm: fio Not tainted 6.10.0-rc3 #27 RIP: 0010:native_queued_spin_lock_slowpath+0x81/0x300 Call Trace: <NMI> ? show_regs+0x69/0x80 ? watchdog_hardlockup_check+0x1b4/0x3a0 <SNIP> ? native_queued_spin_lock_slowpath+0x81/0x300 </NMI> <TASK> ? __pfx_folio_activate_fn+0x10/0x10 _raw_spin_lock_irqsave+0x5b/0x70 folio_lruvec_lock_irqsave+0x62/0x90 folio_batch_move_lru+0x9d/0x160 folio_activate+0x95/0xe0 folio_mark_accessed+0x11f/0x160 filemap_read+0x343/0x3d0 <SNIP> blkdev_read_iter+0x6f/0x140 vfs_read+0x25b/0x340 ksys_read+0x67/0xf0 __x64_sys_read+0x19/0x20 x64_sys_call+0x1771/0x20d0
This is the next one:
watchdog: Watchdog detected hard LOCKUP on cpu 219
CPU: 219 PID: 2584763 Comm: fs_racer_dir_cr Not tainted 6.10.0-rc3 #27 RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 Call Trace: <NMI> ? show_regs+0x69/0x80 ? watchdog_hardlockup_check+0x1b4/0x3a0 <SNIP> ? native_queued_spin_lock_slowpath+0x2b4/0x300 </NMI> <TASK> _raw_spin_lock_irqsave+0x5b/0x70 folio_lruvec_lock_irqsave+0x62/0x90 __page_cache_release+0x89/0x2f0 folios_put_refs+0x92/0x230 __folio_batch_release+0x74/0x90 truncate_inode_pages_range+0x16f/0x520 truncate_pagecache+0x49/0x70 ext4_setattr+0x326/0xaa0 notify_change+0x353/0x500 do_truncate+0x83/0xe0 path_openat+0xd9e/0x1090 do_filp_open+0xaa/0x150 do_sys_openat2+0x9b/0xd0 __x64_sys_openat+0x55/0x90 x64_sys_call+0xe55/0x20d0 do_syscall_64+0x7e/0x130 entry_SYSCALL_64_after_hwframe+0x76/0x7e
When this happens, all-CPU backtrace shows a CPU being in isolate_lru_folios().
Quoted 26 lines
Quoted 26 lines
kernel: watchdog: BUG: soft lockup - CPU#29 stuck for 11s! [fio:2701649]
kernel: CPU: 29 PID: 2701649 Comm: fio Tainted: G L
6.10.0-rc3-mglru-irqstrc #24kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: Call Trace: kernel: <IRQ> kernel: ? show_regs+0x69/0x80 kernel: ? watchdog_timer_fn+0x223/0x2b0 kernel: ? __pfx_watchdog_timer_fn+0x10/0x10 <SNIP> kernel: </IRQ> kernel: <TASK> kernel: ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 kernel: ? native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: _raw_spin_lock+0x38/0x50 kernel: clear_shadow_entry+0x3d/0x100 kernel: ? __pfx_workingset_update_node+0x10/0x10 kernel: mapping_try_invalidate+0x117/0x1d0 kernel: invalidate_mapping_pages+0x10/0x20 kernel: invalidate_bdev+0x3c/0x50 kernel: blkdev_common_ioctl+0x5f7/0xa90 kernel: blkdev_ioctl+0x109/0x270 kernel: x64_sys_call+0x1215/0x20d0 kernel: do_syscall_64+0x7e/0x130
Quoted 8 lines
Quoted 4 lines
This happens to be contending on inode
i_lockspinlock.Below preemptirqsoff trace points to preemption being disabled for more than 10s and the lock in picture is lruvec spinlock.
Also if you could try the other patch (mglru.patch) please. It should help reduce unnecessary rotations from
deactivate_file_folio(), which in turn should reduce the contention on the LRU lock for MGLRU.
Currently testing is in progress with mglru.patch and MGLRU enabled. Will get back on the results.
Regards,
Bharata.
Quoted 17 lines · Bharata B Rao
Hi Yu Zhao,
Thanks for your patches. See below...
Quoted 4 lines · Yu Zhao
Hi Bharata,
Quoted 1 line · Bharata B Rao
<snip>
Quoted 6 lines
Quoted 5 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.
Quoted 5 lines
Quoted 3 lines
This is not really an MGLRU issue -- can you please try one of the attached patches? It (truncate.patch) should help with or without MGLRU.
With truncate.patch and default LRU scheme, a few hard lockups are seen.
Thanks.
In your original report, you said:
Most of the times the two contended locks are lruvec and inode->i_lock spinlocks.
...
Often times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inode i_lock (in clear_shadow_entry path)
Based on this new report, does it mean the i_lock is not as contended, for the same path (truncation) you tested? If so, I'll post truncate.patch and add reported-by and tested-by you, unless you have objections.
The two paths below were contended on the LRU lock, but they already batch their operations. So I don't know what else we can do surgically to improve them.
Quoted 92 lines
First one is this:
watchdog: Watchdog detected hard LOCKUP on cpu 487
CPU: 487 PID: 11525 Comm: fio Not tainted 6.10.0-rc3 #27 RIP: 0010:native_queued_spin_lock_slowpath+0x81/0x300 Call Trace: <NMI> ? show_regs+0x69/0x80 ? watchdog_hardlockup_check+0x1b4/0x3a0 <SNIP> ? native_queued_spin_lock_slowpath+0x81/0x300 </NMI> <TASK> ? __pfx_folio_activate_fn+0x10/0x10 _raw_spin_lock_irqsave+0x5b/0x70 folio_lruvec_lock_irqsave+0x62/0x90 folio_batch_move_lru+0x9d/0x160 folio_activate+0x95/0xe0 folio_mark_accessed+0x11f/0x160 filemap_read+0x343/0x3d0 <SNIP> blkdev_read_iter+0x6f/0x140 vfs_read+0x25b/0x340 ksys_read+0x67/0xf0 __x64_sys_read+0x19/0x20 x64_sys_call+0x1771/0x20d0This is the next one:
watchdog: Watchdog detected hard LOCKUP on cpu 219
CPU: 219 PID: 2584763 Comm: fs_racer_dir_cr Not tainted 6.10.0-rc3 #27 RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 Call Trace: <NMI> ? show_regs+0x69/0x80 ? watchdog_hardlockup_check+0x1b4/0x3a0 <SNIP> ? native_queued_spin_lock_slowpath+0x2b4/0x300 </NMI> <TASK> _raw_spin_lock_irqsave+0x5b/0x70 folio_lruvec_lock_irqsave+0x62/0x90 __page_cache_release+0x89/0x2f0 folios_put_refs+0x92/0x230 __folio_batch_release+0x74/0x90 truncate_inode_pages_range+0x16f/0x520 truncate_pagecache+0x49/0x70 ext4_setattr+0x326/0xaa0 notify_change+0x353/0x500 do_truncate+0x83/0xe0 path_openat+0xd9e/0x1090 do_filp_open+0xaa/0x150 do_sys_openat2+0x9b/0xd0 __x64_sys_openat+0x55/0x90 x64_sys_call+0xe55/0x20d0 do_syscall_64+0x7e/0x130 entry_SYSCALL_64_after_hwframe+0x76/0x7eWhen this happens, all-CPU backtrace shows a CPU being in
isolate_lru_folios().Quoted 31 lines
Quoted 30 lines
kernel: watchdog: BUG: soft lockup - CPU#29 stuck for 11s! [fio:2701649]
kernel: CPU: 29 PID: 2701649 Comm: fio Tainted: G L
6.10.0-rc3-mglru-irqstrc #24kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: Call Trace: kernel: <IRQ> kernel: ? show_regs+0x69/0x80 kernel: ? watchdog_timer_fn+0x223/0x2b0 kernel: ? __pfx_watchdog_timer_fn+0x10/0x10 <SNIP> kernel: </IRQ> kernel: <TASK> kernel: ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 kernel: ? native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: _raw_spin_lock+0x38/0x50 kernel: clear_shadow_entry+0x3d/0x100 kernel: ? __pfx_workingset_update_node+0x10/0x10 kernel: mapping_try_invalidate+0x117/0x1d0 kernel: invalidate_mapping_pages+0x10/0x20 kernel: invalidate_bdev+0x3c/0x50 kernel: blkdev_common_ioctl+0x5f7/0xa90 kernel: blkdev_ioctl+0x109/0x270 kernel: x64_sys_call+0x1215/0x20d0 kernel: do_syscall_64+0x7e/0x130This happens to be contending on inode
i_lockspinlock.Below preemptirqsoff trace points to preemption being disabled for more than 10s and the lock in picture is lruvec spinlock.
Quoted 6 lines
Quoted 3 lines
Also if you could try the other patch (mglru.patch) please. It should help reduce unnecessary rotations from
deactivate_file_folio(), which in turn should reduce the contention on the LRU lock for MGLRU.Currently testing is in progress with mglru.patch and MGLRU enabled. Will get back on the results.
Thank you.
Quoted 35 lines · Yu Zhao
Quoted 17 lines · Bharata B Rao
Hi Yu Zhao,
Thanks for your patches. See below...
Quoted 4 lines · Yu Zhao
Hi Bharata,
Quoted 1 line · Bharata B Rao
<snip>
Quoted 6 lines
Quoted 5 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.Quoted 5 lines
Quoted 3 lines
This is not really an MGLRU issue -- can you please try one of the attached patches? It (truncate.patch) should help with or without MGLRU.
With truncate.patch and default LRU scheme, a few hard lockups are seen.
Thanks.
In your original report, you said:
Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.
...
Often times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inodei_lock(inclear_shadow_entrypath)
Quoted 4 lines
Based on this new report, does it mean the
i_lockis not as contended, for the same path (truncation) you tested? If so, I'll post truncate.patch and add reported-by and tested-by you, unless you have objections.
truncate.patch has been tested on two systems with default LRU scheme and the lockup due to inode->i_lock hasn't been seen yet after 24 hours run.
Quoted 4 lines
The two paths below were contended on the LRU lock, but they already batch their operations. So I don't know what else we can do surgically to improve them.
What has been seen with this workload is that the lruvec spinlock is held for a long time from shrink_[active/inactive]_list path. In this path, there is a case in isolate_lru_folios() where scanning of LRU lists can become unbounded. To isolate a page from ZONE_DMA, sometimes scanning/skipping of more than 150 million folios were seen. There is already a comment in there which explains why nr_skipped shouldn't be counted, but is there any possibility of re-looking at this condition?
Regards,
Bharata.
Quoted 37 lines · Bharata B Rao
Quoted 35 lines · Yu Zhao
Quoted 17 lines · Bharata B Rao
Hi Yu Zhao,
Thanks for your patches. See below...
Quoted 4 lines · Yu Zhao
Hi Bharata,
Quoted 1 line · Bharata B Rao
<snip>
Quoted 6 lines
Quoted 5 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.Quoted 5 lines
Quoted 3 lines
This is not really an MGLRU issue -- can you please try one of the attached patches? It (truncate.patch) should help with or without MGLRU.
With truncate.patch and default LRU scheme, a few hard lockups are seen.
Thanks.
In your original report, you said:
Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.
...
Often times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inodei_lock(inclear_shadow_entrypath)
Quoted 7 lines
Quoted 4 lines
Based on this new report, does it mean the
i_lockis not as contended, for the same path (truncation) you tested? If so, I'll post truncate.patch and add reported-by and tested-by you, unless you have objections.truncate.patch has been tested on two systems with default LRU scheme and the lockup due to
inode->i_lockhasn't been seen yet after 24 hours run.
Thanks.
Quoted 5 lines
Quoted 4 lines
The two paths below were contended on the LRU lock, but they already batch their operations. So I don't know what else we can do surgically to improve them.
Quoted 7 lines
What has been seen with this workload is that the lruvec spinlock is held for a long time from shrink_[active/inactive]_list path. In this path, there is a case in
isolate_lru_folios()where scanning of LRU lists can become unbounded. To isolate a page fromZONE_DMA, sometimes scanning/skipping of more than 150 million folios were seen. There is already a comment in there which explains whynr_skippedshouldn't be counted, but is there any possibility of re-looking at this condition?
For this specific case, probably this can help:
@@ -1659,8 +1659,15 @@ static unsigned long
isolate_lru_folios(unsigned long nr_to_scan,
if (folio_zonenum(folio) > sc->reclaim_idx ||
skip_cma(folio, sc)) {
nr_skipped[folio_zonenum(folio)] += nr_pages;
- move_to = &folios_skipped;
- goto move;
+ list_move(&folio->lru, &folios_skipped);
+ if (spin_is_contended(&lruvec->lru_lock)) {
+ if (!list_empty(dst))
+ break;
+ spin_unlock_irq(&lruvec->lru_lock);
+ cond_resched();
+ spin_lock_irq(&lruvec->lru_lock);
+ }
+ continue;
}Quoted 70 lines · Yu Zhao
Quoted 37 lines · Bharata B Rao
Quoted 35 lines · Yu Zhao
Quoted 17 lines · Bharata B Rao
Hi Yu Zhao,
Thanks for your patches. See below...
Quoted 4 lines · Yu Zhao
Hi Bharata,
Quoted 1 line · Bharata B Rao
<snip>
Quoted 6 lines
Quoted 5 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.Quoted 5 lines
Quoted 3 lines
This is not really an MGLRU issue -- can you please try one of the attached patches? It (truncate.patch) should help with or without MGLRU.
With truncate.patch and default LRU scheme, a few hard lockups are seen.
Thanks.
In your original report, you said:
Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.
...
Often times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inodei_lock(inclear_shadow_entrypath)Quoted 7 lines
Quoted 4 lines
Based on this new report, does it mean the
i_lockis not as contended, for the same path (truncation) you tested? If so, I'll post truncate.patch and add reported-by and tested-by you, unless you have objections.truncate.patch has been tested on two systems with default LRU scheme and the lockup due to
inode->i_lockhasn't been seen yet after 24 hours run.Thanks.
Quoted 5 lines
Quoted 4 lines
The two paths below were contended on the LRU lock, but they already batch their operations. So I don't know what else we can do surgically to improve them.
Quoted 7 lines
What has been seen with this workload is that the lruvec spinlock is held for a long time from shrink_[active/inactive]_list path. In this path, there is a case in
isolate_lru_folios()where scanning of LRU lists can become unbounded. To isolate a page fromZONE_DMA, sometimes scanning/skipping of more than 150 million folios were seen. There is already a comment in there which explains whynr_skippedshouldn't be counted, but is there any possibility of re-looking at this condition?For this specific case, probably this can help:
(quoted diff) +0
-2@@ -1659,8 +1659,15 @@ static unsigned long isolate_lru_folios(unsigned long nr_to_scan, if (folio_zonenum(folio) > sc->reclaim_idx || skip_cma(folio, sc)) { nr_skipped[folio_zonenum(folio)] += nr_pages; - move_to = &folios_skipped; - goto move;
Review static unsigned long · line 1672
Quoted 10 lines
(quoted diff) +9
-0+ list_move(&folio->lru, &folios_skipped); + if (spin_is_contended(&lruvec->lru_lock)) { + if (!list_empty(dst)) + break; + spin_unlock_irq(&lruvec->lru_lock); + cond_resched(); + spin_lock_irq(&lruvec->lru_lock); + } + continue; }
Thanks, this helped. With this fix, the test ran for 24hrs without any lockups attributable to lruvec spinlock. As noted in this thread, earlier isolate_lru_folios() used to scan millions of folios and spend a lot of time with spinlock held but after this fix, such a scenario is no longer seen.
However the contention seems to have shifted to other areas and these are the two MM related soft and hard lockups that were observed during this run:
Soft lockup
===========
watchdog: BUG: soft lockup - CPU#425 stuck for 12s! CPU: 425 PID: 145707 Comm: fio Kdump: loaded Tainted: G W 6.10.0-rc3-trkwtrs_trnct_nvme_lruvecresched #21 RIP: 0010:handle_softirqs+0x70/0x2f0 __rmqueue_pcplist+0x4ce/0x9a0 get_page_from_freelist+0x2e1/0x1650 __alloc_pages_noprof+0x1b4/0x12c0 alloc_pages_mpol_noprof+0xdd/0x200 folio_alloc_noprof+0x67/0xe0
Hard lockup
===========
watchdog: Watchdog detected hard LOCKUP on cpu 296
CPU: 296 PID: 150155 Comm: fio Kdump: loaded Tainted: G W L 6.10.0-rc3-trkwtrs_trnct_nvme_lruvecresched #21 RIP: 0010:native_queued_spin_lock_slowpath+0x347/0x430 Call Trace: <NMI> ? watchdog_hardlockup_check+0x1a2/0x370 ? watchdog_overflow_callback+0x6d/0x80 <SNIP> native_queued_spin_lock_slowpath+0x347/0x430 </NMI> <IRQ> _raw_spin_lock_irqsave+0x46/0x60 free_unref_page+0x19f/0x540 ? __slab_free+0x2ab/0x2b0 __free_pages+0x9d/0xb0 __free_slab+0xa7/0xf0 free_slab+0x31/0x100 discard_slab+0x32/0x40 __put_partials+0xb8/0xe0 put_cpu_partial+0x5a/0x90 __slab_free+0x1d9/0x2b0 kfree+0x244/0x280 mempool_kfree+0x12/0x20 mempool_free+0x30/0x90 nvme_unmap_data+0xd0/0x150 [nvme] nvme_pci_complete_batch+0xaf/0xd0 [nvme] nvme_irq+0x96/0xe0 [nvme] __handle_irq_event_percpu+0x50/0x1b0
Regards,
Bharata.
Quoted 75 lines · Bharata B Rao
Quoted 74 lines · Yu Zhao
Quoted 37 lines · Bharata B Rao
Quoted 35 lines · Yu Zhao
Quoted 17 lines · Bharata B Rao
Hi Yu Zhao,
Thanks for your patches. See below...
Quoted 4 lines · Yu Zhao
Hi Bharata,
Quoted 1 line · Bharata B Rao
<snip>
Quoted 6 lines
Quoted 5 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.Quoted 5 lines
Quoted 3 lines
This is not really an MGLRU issue -- can you please try one of the attached patches? It (truncate.patch) should help with or without MGLRU.
With truncate.patch and default LRU scheme, a few hard lockups are seen.
Thanks.
In your original report, you said:
Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.
...
Often times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inodei_lock(inclear_shadow_entrypath)Quoted 7 lines
Quoted 4 lines
Based on this new report, does it mean the
i_lockis not as contended, for the same path (truncation) you tested? If so, I'll post truncate.patch and add reported-by and tested-by you, unless you have objections.truncate.patch has been tested on two systems with default LRU scheme and the lockup due to
inode->i_lockhasn't been seen yet after 24 hours run.Thanks.
Quoted 5 lines
Quoted 4 lines
The two paths below were contended on the LRU lock, but they already batch their operations. So I don't know what else we can do surgically to improve them.
Quoted 7 lines
What has been seen with this workload is that the lruvec spinlock is held for a long time from shrink_[active/inactive]_list path. In this path, there is a case in
isolate_lru_folios()where scanning of LRU lists can become unbounded. To isolate a page fromZONE_DMA, sometimes scanning/skipping of more than 150 million folios were seen. There is already a comment in there which explains whynr_skippedshouldn't be counted, but is there any possibility of re-looking at this condition?For this specific case, probably this can help:
(quoted diff) +0
-0@@ -1659,8 +1659,15 @@ static unsigned long isolate_lru_folios(unsigned long nr_to_scan,if (
folio_zonenum(folio)>sc->reclaim_idx||skip_cma(folio, sc)) { nr_skipped[folio_zonenum(folio)] +=nr_pages; -move_to=&folios_skipped;
- goto move;
+list_move(&folio->lru, &folios_skipped); + if (spin_is_contended(&lruvec->lru_lock)) { + if (!list_empty(dst))
+ break;
Quoted 8 lines
Quoted 6 lines
+
spin_unlock_irq(&lruvec->lru_lock); +cond_resched();
+spin_lock_irq(&lruvec->lru_lock); + }
+ continue;
}Thanks, this helped. With this fix, the test ran for 24hrs without any lockups attributable to lruvec spinlock. As noted in this thread, earlier
isolate_lru_folios()used to scan millions of folios and spend a lot of time with spinlock held but after this fix, such a scenario is no longer seen.
However during the weekend mglru-enabled run (with above fix to isolate_lru_folios() and also the previous two patches: truncate.patch and mglru.patch and the inode fix provided by Mateusz), another hard lockup related to lruvec spinlock was observed.
Here is the hardlock up:
watchdog: Watchdog detected hard LOCKUP on cpu 466
CPU: 466 PID: 3103929 Comm: fio Not tainted 6.10.0-rc3-trnct_nvme_lruvecresched_sirq_inode_mglru #32 RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 Call Trace: <NMI> ? show_regs+0x69/0x80 ? watchdog_hardlockup_check+0x1b4/0x3a0 <SNIP> ? native_queued_spin_lock_slowpath+0x2b4/0x300 </NMI> <IRQ> _raw_spin_lock_irqsave+0x5b/0x70 folio_lruvec_lock_irqsave+0x62/0x90 folio_batch_move_lru+0x9d/0x160 folio_rotate_reclaimable+0xab/0xf0 folio_end_writeback+0x60/0x90 end_buffer_async_write+0xaa/0xe0 end_bio_bh_io_sync+0x2c/0x50 bio_endio+0x108/0x180 blk_mq_end_request_batch+0x11f/0x5e0 nvme_pci_complete_batch+0xb5/0xd0 [nvme] nvme_irq+0x92/0xe0 [nvme] __handle_irq_event_percpu+0x6e/0x1e0 handle_irq_event+0x39/0x80 handle_edge_irq+0x8c/0x240 __common_interrupt+0x4e/0xf0 common_interrupt+0x49/0xc0 asm_common_interrupt+0x27/0x40
Here is the lock holder details captured by all-cpu-backtrace:
NMI backtrace for cpu 75
CPU: 75 PID: 3095650 Comm: fio Not tainted 6.10.0-rc3-trnct_nvme_lruvecresched_sirq_inode_mglru #32 RIP: 0010:folio_inc_gen+0x142/0x430 Call Trace: <NMI> ? show_regs+0x69/0x80 ? nmi_cpu_backtrace+0xc5/0x130 ? nmi_cpu_backtrace_handler+0x11/0x20 ? nmi_handle+0x64/0x180 ? default_do_nmi+0x45/0x130 ? exc_nmi+0x128/0x1a0 ? end_repeat_nmi+0xf/0x53 ? folio_inc_gen+0x142/0x430 ? folio_inc_gen+0x142/0x430 ? folio_inc_gen+0x142/0x430 </NMI> <TASK> isolate_folios+0x954/0x1630 evict_folios+0xa5/0x8c0 try_to_shrink_lruvec+0x1be/0x320 shrink_one+0x10f/0x1d0 shrink_node+0xa4c/0xc90 do_try_to_free_pages+0xc0/0x590 try_to_free_pages+0xde/0x210 __alloc_pages_noprof+0x6ae/0x12c0 alloc_pages_mpol_noprof+0xd9/0x220 folio_alloc_noprof+0x63/0xe0 filemap_alloc_folio_noprof+0xf4/0x100 page_cache_ra_unbounded+0xb9/0x1a0 page_cache_ra_order+0x26e/0x310 ondemand_readahead+0x1a3/0x360 page_cache_sync_ra+0x83/0x90 filemap_get_pages+0xf0/0x6a0 filemap_read+0xe7/0x3d0 blkdev_read_iter+0x6f/0x140 vfs_read+0x25b/0x340 ksys_read+0x67/0xf0 __x64_sys_read+0x19/0x20 x64_sys_call+0x1771/0x20d0 do_syscall_64+0x7e/0x130
Regards,
Bharata.
Quoted 92 lines · Bharata B Rao
Quoted 81 lines · Bharata B Rao
Quoted 80 lines · Yu Zhao
Quoted 38 lines · Bharata B Rao
Quoted 36 lines · Yu Zhao
Quoted 17 lines · Bharata B Rao
Hi Yu Zhao,
Thanks for your patches. See below...
Quoted 4 lines · Yu Zhao
Hi Bharata,
Quoted 1 line · Bharata B Rao
<snip>
Quoted 6 lines
Quoted 5 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.Quoted 6 lines
Quoted 3 lines
This is not really an MGLRU issue -- can you please try one of the attached patches? It (truncate.patch) should help with or without MGLRU.
With truncate.patch and default LRU scheme, a few hard lockups are seen.
Thanks.
In your original report, you said:
Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.
...
Often times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inodei_lock(inclear_shadow_entrypath)Quoted 8 lines
Quoted 4 lines
Based on this new report, does it mean the
i_lockis not as contended, for the same path (truncation) you tested? If so, I'll post truncate.patch and add reported-by and tested-by you, unless you have objections.truncate.patch has been tested on two systems with default LRU scheme and the lockup due to
inode->i_lockhasn't been seen yet after 24 hours run.Thanks.
Quoted 5 lines
Quoted 4 lines
The two paths below were contended on the LRU lock, but they already batch their operations. So I don't know what else we can do surgically to improve them.
Quoted 7 lines
What has been seen with this workload is that the lruvec spinlock is held for a long time from shrink_[active/inactive]_list path. In this path, there is a case in
isolate_lru_folios()where scanning of LRU lists can become unbounded. To isolate a page fromZONE_DMA, sometimes scanning/skipping of more than 150 million folios were seen. There is already a comment in there which explains whynr_skippedshouldn't be counted, but is there any possibility of re-looking at this condition?For this specific case, probably this can help:
(quoted diff) +8
-2@@ -1659,8 +1659,15 @@ static unsigned long isolate_lru_folios(unsigned long nr_to_scan, if (folio_zonenum(folio) > sc->reclaim_idx || skip_cma(folio, sc)) { nr_skipped[folio_zonenum(folio)] += nr_pages; - move_to = &folios_skipped; - goto move; + list_move(&folio->lru, &folios_skipped); + if (spin_is_contended(&lruvec->lru_lock)) { + if (!list_empty(dst)) + break; + spin_unlock_irq(&lruvec->lru_lock); + cond_resched(); + spin_lock_irq(&lruvec->lru_lock); + }Quoted 8 lines
Review static unsigned long · line 1672
Quoted 2 lines
(quoted diff) +1
-0+ continue; }Thanks, this helped. With this fix, the test ran for 24hrs without any lockups attributable to lruvec spinlock. As noted in this thread, earlier
isolate_lru_folios()used to scan millions of folios and spend a lot of time with spinlock held but after this fix, such a scenario is no longer seen.
Quoted 4 lines
However during the weekend mglru-enabled run (with above fix to
isolate_lru_folios()and also the previous two patches: truncate.patch and mglru.patch and the inode fix provided by Mateusz), another hard lockup related to lruvec spinlock was observed.
Thanks again for the stress tests.
I can't come up with any reasonable band-aid at this moment, i.e., something not too ugly to work around a more fundamental scalability problem.
Before I give up: what type of dirty data was written back to the nvme device? Was it page cache or swap?
Quoted 78 lines
Here is the hardlock up:
watchdog: Watchdog detected hard LOCKUP on cpu 466
CPU: 466 PID: 3103929 Comm: fio Not tainted 6.10.0-rc3-trnct_nvme_lruvecresched_sirq_inode_mglru #32 RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 Call Trace: <NMI> ? show_regs+0x69/0x80 ? watchdog_hardlockup_check+0x1b4/0x3a0 <SNIP> ? native_queued_spin_lock_slowpath+0x2b4/0x300 </NMI> <IRQ> _raw_spin_lock_irqsave+0x5b/0x70 folio_lruvec_lock_irqsave+0x62/0x90 folio_batch_move_lru+0x9d/0x160 folio_rotate_reclaimable+0xab/0xf0 folio_end_writeback+0x60/0x90 end_buffer_async_write+0xaa/0xe0 end_bio_bh_io_sync+0x2c/0x50 bio_endio+0x108/0x180 blk_mq_end_request_batch+0x11f/0x5e0 nvme_pci_complete_batch+0xb5/0xd0 [nvme] nvme_irq+0x92/0xe0 [nvme] __handle_irq_event_percpu+0x6e/0x1e0 handle_irq_event+0x39/0x80 handle_edge_irq+0x8c/0x240 __common_interrupt+0x4e/0xf0 common_interrupt+0x49/0xc0 asm_common_interrupt+0x27/0x40Here is the lock holder details captured by all-cpu-backtrace:
NMI backtrace for cpu 75
CPU: 75 PID: 3095650 Comm: fio Not tainted 6.10.0-rc3-trnct_nvme_lruvecresched_sirq_inode_mglru #32 RIP: 0010:folio_inc_gen+0x142/0x430 Call Trace: <NMI> ? show_regs+0x69/0x80 ? nmi_cpu_backtrace+0xc5/0x130 ? nmi_cpu_backtrace_handler+0x11/0x20 ? nmi_handle+0x64/0x180 ? default_do_nmi+0x45/0x130 ? exc_nmi+0x128/0x1a0 ? end_repeat_nmi+0xf/0x53 ? folio_inc_gen+0x142/0x430 ? folio_inc_gen+0x142/0x430 ? folio_inc_gen+0x142/0x430 </NMI> <TASK> isolate_folios+0x954/0x1630 evict_folios+0xa5/0x8c0 try_to_shrink_lruvec+0x1be/0x320 shrink_one+0x10f/0x1d0 shrink_node+0xa4c/0xc90 do_try_to_free_pages+0xc0/0x590 try_to_free_pages+0xde/0x210 __alloc_pages_noprof+0x6ae/0x12c0 alloc_pages_mpol_noprof+0xd9/0x220 folio_alloc_noprof+0x63/0xe0 filemap_alloc_folio_noprof+0xf4/0x100 page_cache_ra_unbounded+0xb9/0x1a0 page_cache_ra_order+0x26e/0x310 ondemand_readahead+0x1a3/0x360 page_cache_sync_ra+0x83/0x90 filemap_get_pages+0xf0/0x6a0 filemap_read+0xe7/0x3d0 blkdev_read_iter+0x6f/0x140 vfs_read+0x25b/0x340 ksys_read+0x67/0xf0 __x64_sys_read+0x19/0x20 x64_sys_call+0x1771/0x20d0 do_syscall_64+0x7e/0x130Regards,
Bharata.
Quoted 7 lines · Yu Zhao
I can't come up with any reasonable band-aid at this moment, i.e., something not too ugly to work around a more fundamental scalability problem.
Before I give up: what type of dirty data was written back to the nvme device? Was it page cache or swap?
With my corporate employee hat on, I would like to note a couple of three things.
1. there are definitely bugs here and someone(tm) should sort them out(R)
however....
2. the real goal is presumably to beat the kernel into shape where production kernels no longer suffer lockups running this workload on this hardware
3. the flamegraph (to be found in [1]) shows expensive debug enabled, notably for preemption count (search for preempt_count_sub to see)
4. I'm told the lruvec problem is being worked on (but no ETA) and I don't think the above justifies considering any hacks or otherwise putting more pressure on it
It is plausible eliminating the aforementioned debug will be good enough.
Apart from that I note percpu_counter_add_batch (+ irq debug) accounts for 5.8% cpu time. This will of course go down if irq tracing is disabled, but so happens I optimized this routine to be faster single-threaded (in particular by dodging the interrupt trip). The patch is hanging out in the mm tree [2] and is trivially applicable for testing.
Even if none of the debug opts can get modified, this should drop percpu_counter_add_batch to 1.5% or so, which may or may not have a side effect of avoiding the lockup problem.
[1]: https://lore.kernel.org/lkml/[email protected]/
[2]: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git/commit/?h=mm-everything&id=51d821654be4286b005ad2b7dc8b973d5008a2ec
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
Quoted 34 lines · Mateusz Guzik
Quoted 7 lines · Yu Zhao
I can't come up with any reasonable band-aid at this moment, i.e., something not too ugly to work around a more fundamental scalability problem.
Before I give up: what type of dirty data was written back to the nvme device? Was it page cache or swap?
With my corporate employee hat on, I would like to note a couple of three things.
1. there are definitely bugs here and someone(tm) should sort them out(R)
however....
2. the real goal is presumably to beat the kernel into shape where production kernels no longer suffer lockups running this workload on this hardware
3. the flamegraph (to be found in [1]) shows expensive debug enabled, notably for preemption count (search forpreempt_count_subto see)
4. I'm told the lruvec problem is being worked on (but no ETA) and I don't think the above justifies considering any hacks or otherwise putting more pressure on itIt is plausible eliminating the aforementioned debug will be good enough.
Apart from that I note
percpu_counter_add_batch(+ irq debug) accounts for 5.8% cpu time. This will of course go down if irq tracing is disabled, but so happens I optimized this routine to be faster single-threaded (in particular by dodging the interrupt trip). The patch is hanging out in the mm tree [2] and is trivially applicable for testing.
Quoted 3 lines
Even if none of the debug opts can get modified, this should drop
percpu_counter_add_batchto 1.5% or so, which may or may not have a side effect of avoiding the lockup problem.
Thanks, A few debug options were turned ON to gather debug data. Will do a full run once with them turned OFF and with the above percpu_counter_add_batch patch.
Regards,
Bharata.
Quoted 5 lines · Yu Zhao
Quoted 4 lines
However during the weekend mglru-enabled run (with above fix to
isolate_lru_folios()and also the previous two patches: truncate.patch and mglru.patch and the inode fix provided by Mateusz), another hard lockup related to lruvec spinlock was observed.
Quoted 8 lines
Thanks again for the stress tests.
I can't come up with any reasonable band-aid at this moment, i.e., something not too ugly to work around a more fundamental scalability problem.
Before I give up: what type of dirty data was written back to the nvme device? Was it page cache or swap?
This is how a typical dstat report looks like when we start to see the problem with lruvec spinlock.
------memory-usage----- ----swap---
used free buff cach| used free|
14.3G 20.7G 1467G 185M| 938M 15G| 14.3G 20.0G 1468G 174M| 938M 15G| 14.3G 20.3G 1468G 184M| 938M 15G| 14.3G 19.8G 1468G 183M| 938M 15G| 14.3G 19.9G 1468G 183M| 938M 15G| 14.3G 19.5G 1468G 183M| 938M 15G|
As you can see, most of the usage is in buffer cache and swap is hardly used. Just to recap from the original post...
====
FIO is run with a size of 1TB on each NVME partition with different combinations of ioengine/blocksize/mode parameters and buffered-IO. Selected FS tests from LTP are run on 256GB partitions of all NVME disks. This is the typical NVME partition layout.
nvme2n1 259:4 0 3.5T 0 disk ├─nvme2n1p1 259:6 0 256G 0 part /data_nvme2n1p1 └─nvme2n1p2 259:7 0 3.2T 0 part
Though many different runs exist in the workload, the combination that results in the problem is buffered-IO run with sync engine.
fio -filename=/dev/nvme1n1p2 -direct=0 -thread -size=1024G \
-rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=sync -bs=4k \
-numjobs=400 -runtime=25000 --time_based -group_reporting -name=mytest
====
Regards,
Bharata.
Quoted 72 lines
However during the weekend mglru-enabled run (with above fix to
isolate_lru_folios()and also the previous two patches: truncate.patch and mglru.patch and the inode fix provided by Mateusz), another hard lockup related to lruvec spinlock was observed.Here is the hardlock up:
watchdog: Watchdog detected hard LOCKUP on cpu 466
CPU: 466 PID: 3103929 Comm: fio Not tainted 6.10.0-rc3-trnct_nvme_lruvecresched_sirq_inode_mglru #32 RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 Call Trace: <NMI> ? show_regs+0x69/0x80 ? watchdog_hardlockup_check+0x1b4/0x3a0 <SNIP> ? native_queued_spin_lock_slowpath+0x2b4/0x300 </NMI> <IRQ> _raw_spin_lock_irqsave+0x5b/0x70 folio_lruvec_lock_irqsave+0x62/0x90 folio_batch_move_lru+0x9d/0x160 folio_rotate_reclaimable+0xab/0xf0 folio_end_writeback+0x60/0x90 end_buffer_async_write+0xaa/0xe0 end_bio_bh_io_sync+0x2c/0x50 bio_endio+0x108/0x180 blk_mq_end_request_batch+0x11f/0x5e0 nvme_pci_complete_batch+0xb5/0xd0 [nvme] nvme_irq+0x92/0xe0 [nvme] __handle_irq_event_percpu+0x6e/0x1e0 handle_irq_event+0x39/0x80 handle_edge_irq+0x8c/0x240 __common_interrupt+0x4e/0xf0 common_interrupt+0x49/0xc0 asm_common_interrupt+0x27/0x40Here is the lock holder details captured by all-cpu-backtrace:
NMI backtrace for cpu 75
CPU: 75 PID: 3095650 Comm: fio Not tainted 6.10.0-rc3-trnct_nvme_lruvecresched_sirq_inode_mglru #32 RIP: 0010:folio_inc_gen+0x142/0x430 Call Trace: <NMI> ? show_regs+0x69/0x80 ? nmi_cpu_backtrace+0xc5/0x130 ? nmi_cpu_backtrace_handler+0x11/0x20 ? nmi_handle+0x64/0x180 ? default_do_nmi+0x45/0x130 ? exc_nmi+0x128/0x1a0 ? end_repeat_nmi+0xf/0x53 ? folio_inc_gen+0x142/0x430 ? folio_inc_gen+0x142/0x430 ? folio_inc_gen+0x142/0x430 </NMI> <TASK> isolate_folios+0x954/0x1630 evict_folios+0xa5/0x8c0 try_to_shrink_lruvec+0x1be/0x320 shrink_one+0x10f/0x1d0 shrink_node+0xa4c/0xc90 do_try_to_free_pages+0xc0/0x590 try_to_free_pages+0xde/0x210 __alloc_pages_noprof+0x6ae/0x12c0 alloc_pages_mpol_noprof+0xd9/0x220 folio_alloc_noprof+0x63/0xe0 filemap_alloc_folio_noprof+0xf4/0x100 page_cache_ra_unbounded+0xb9/0x1a0 page_cache_ra_order+0x26e/0x310 ondemand_readahead+0x1a3/0x360 page_cache_sync_ra+0x83/0x90
Quoted 8 lines
filemap_get_pages+0xf0/0x6a0 filemap_read+0xe7/0x3d0 blkdev_read_iter+0x6f/0x140 vfs_read+0x25b/0x340 ksys_read+0x67/0xf0 __x64_sys_read+0x19/0x20 x64_sys_call+0x1771/0x20d0 do_syscall_64+0x7e/0x130
From the callstack of lock holder, it is looks like a scability issue rather than a deadlock. Unlike legacy LRU management, there is no throttling mechanism for global reclaim under mglru so far.Could we apply the similar method to throttle the reclaim when it is too aggresive. I am wondering if this patch which is a rough version could help on this?
mm/vmscan.c +55 -0
@@ -4520,6 +4520,50 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int swreturn scanned; } +static void lru_gen_throttle (pg_data_t *pgdat,struct scan_control *sc) +{ +struct lruvec *target_lruvec =mem_cgroup_lruvec (sc->target_mem_cgroup, pgdat); + +if (current_is_kswapd ()) { +if (sc->nr.writeback && sc->nr.writeback == sc->nr.taken) +set_bit (PGDAT_WRITEBACK , &pgdat->flags); + +/* Allow kswapd to start writing pages during reclaim.*/ +if (sc->nr.unqueued_dirty == sc->nr.file_taken) +set_bit (PGDAT_DIRTY , &pgdat->flags); + +if (sc->nr.immediate) +reclaim_throttle (pgdat,VMSCAN_THROTTLE_WRITEBACK ); + } + +/* +* Tag a node/memcg as congested if all the dirty pages were marked +* for writeback and immediate reclaim (counted in nr.congested). +* +* Legacy memcg will stall in page writeback so avoid forcibly +* stalling in reclaim_throttle(). +*/ +if (sc->nr.dirty && (sc->nr.dirty /2 < sc->nr.congested)) { +if (cgroup_reclaim (sc) &&writeback_throttling_sane (sc)) +set_bit (LRUVEC_CGROUP_CONGESTED , &target_lruvec->flags); + +if (current_is_kswapd ()) +set_bit (LRUVEC_NODE_CONGESTED , &target_lruvec->flags); + } + +/* +* Stall direct reclaim for IO completions if the lruvec is +* node is congested. Allow kswapd to continue until it +* starts encountering unqueued dirty pages or cycling through +* the LRU too quickly. +*/ +if (!current_is_kswapd () &¤t_may_throttle () && + !sc->hibernation_mode && + (test_bit (LRUVEC_CGROUP_CONGESTED , &target_lruvec->flags) || +test_bit (LRUVEC_NODE_CONGESTED , &target_lruvec->flags))) +reclaim_throttle (pgdat,VMSCAN_THROTTLE_CONGESTED ); +} +static int evict_folios (struct lruvec *lruvec,struct scan_control *sc,int swappiness) {int type;
@@ -4552,6 +4596,16 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap retry: reclaimed =shrink_folio_list (&list, pgdat, sc, &stat, false); sc->nr_reclaimed += reclaimed; + sc->nr.dirty += stat.nr_dirty; + sc->nr.congested += stat.nr_congested; + sc->nr.unqueued_dirty += stat.nr_unqueued_dirty; + sc->nr.writeback += stat.nr_writeback; + sc->nr.immediate += stat.nr_immediate; + sc->nr.taken += scanned; + +if (type) + sc->nr.file_taken += scanned; +trace_mm_vmscan_lru_shrink_inactive (pgdat->node_id, scanned, reclaimed, &stat, sc->priority, type ?LRU_INACTIVE_FILE :LRU_INACTIVE_ANON );
@@ -5908,6 +5962,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)if (lru_gen_enabled () &&root_reclaim (sc)) {lru_gen_shrink_node (pgdat, sc); +lru_gen_throttle (pgdat, sc);return ; }
Signature · 1 line
2.25.1
Quoted 157 lines · zhaoyang.huang
Quoted 73 lines
However during the weekend mglru-enabled run (with above fix to
isolate_lru_folios()and also the previous two patches: truncate.patch and mglru.patch and the inode fix provided by Mateusz), another hard lockup related to lruvec spinlock was observed.Here is the hardlock up:
watchdog: Watchdog detected hard LOCKUP on cpu 466
CPU: 466 PID: 3103929 Comm: fio Not tainted 6.10.0-rc3-trnct_nvme_lruvecresched_sirq_inode_mglru #32 RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 Call Trace: <NMI> ? show_regs+0x69/0x80 ? watchdog_hardlockup_check+0x1b4/0x3a0 <SNIP> ? native_queued_spin_lock_slowpath+0x2b4/0x300 </NMI> <IRQ> _raw_spin_lock_irqsave+0x5b/0x70 folio_lruvec_lock_irqsave+0x62/0x90 folio_batch_move_lru+0x9d/0x160 folio_rotate_reclaimable+0xab/0xf0 folio_end_writeback+0x60/0x90 end_buffer_async_write+0xaa/0xe0 end_bio_bh_io_sync+0x2c/0x50 bio_endio+0x108/0x180 blk_mq_end_request_batch+0x11f/0x5e0 nvme_pci_complete_batch+0xb5/0xd0 [nvme] nvme_irq+0x92/0xe0 [nvme] __handle_irq_event_percpu+0x6e/0x1e0 handle_irq_event+0x39/0x80 handle_edge_irq+0x8c/0x240 __common_interrupt+0x4e/0xf0 common_interrupt+0x49/0xc0 asm_common_interrupt+0x27/0x40Here is the lock holder details captured by all-cpu-backtrace:
NMI backtrace for cpu 75
CPU: 75 PID: 3095650 Comm: fio Not tainted 6.10.0-rc3-trnct_nvme_lruvecresched_sirq_inode_mglru #32 RIP: 0010:folio_inc_gen+0x142/0x430 Call Trace: <NMI> ? show_regs+0x69/0x80 ? nmi_cpu_backtrace+0xc5/0x130 ? nmi_cpu_backtrace_handler+0x11/0x20 ? nmi_handle+0x64/0x180 ? default_do_nmi+0x45/0x130 ? exc_nmi+0x128/0x1a0 ? end_repeat_nmi+0xf/0x53 ? folio_inc_gen+0x142/0x430 ? folio_inc_gen+0x142/0x430 ? folio_inc_gen+0x142/0x430 </NMI> <TASK> isolate_folios+0x954/0x1630 evict_folios+0xa5/0x8c0 try_to_shrink_lruvec+0x1be/0x320 shrink_one+0x10f/0x1d0 shrink_node+0xa4c/0xc90 do_try_to_free_pages+0xc0/0x590 try_to_free_pages+0xde/0x210 __alloc_pages_noprof+0x6ae/0x12c0 alloc_pages_mpol_noprof+0xd9/0x220 folio_alloc_noprof+0x63/0xe0 filemap_alloc_folio_noprof+0xf4/0x100 page_cache_ra_unbounded+0xb9/0x1a0 page_cache_ra_order+0x26e/0x310 ondemand_readahead+0x1a3/0x360 page_cache_sync_ra+0x83/0x90Quoted 8 lines
filemap_get_pages+0xf0/0x6a0 filemap_read+0xe7/0x3d0 blkdev_read_iter+0x6f/0x140 vfs_read+0x25b/0x340 ksys_read+0x67/0xf0 __x64_sys_read+0x19/0x20 x64_sys_call+0x1771/0x20d0 do_syscall_64+0x7e/0x130From the callstack of lock holder, it is looks like a scability issue rather than a deadlock. Unlike legacy LRU management, there is no throttling mechanism for global reclaim under mglru so far.Could we apply the similar method to throttle the reclaim when it is too aggresive. I am wondering if this patch which is a rough version could help on this?
mm/vmscan.c +54
-0@@ -4520,6 +4520,50 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int swreturn scanned; } +static void lru_gen_throttle (pg_data_t *pgdat,struct scan_control *sc) +{ +struct lruvec *target_lruvec =mem_cgroup_lruvec (sc->target_mem_cgroup, pgdat); + +if (current_is_kswapd ()) { +if (sc->nr.writeback && sc->nr.writeback == sc->nr.taken) +set_bit (PGDAT_WRITEBACK , &pgdat->flags); + +/* Allow kswapd to start writing pages during reclaim.*/ +if (sc->nr.unqueued_dirty == sc->nr.file_taken) +set_bit (PGDAT_DIRTY , &pgdat->flags); + +if (sc->nr.immediate) +reclaim_throttle (pgdat,VMSCAN_THROTTLE_WRITEBACK ); + } + +/* +* Tag a node/memcg as congested if all the dirty pages were marked +* for writeback and immediate reclaim (counted in nr.congested). +* +* Legacy memcg will stall in page writeback so avoid forcibly +* stalling in reclaim_throttle(). +*/ +if (sc->nr.dirty && (sc->nr.dirty /2 < sc->nr.congested)) { +if (cgroup_reclaim (sc) &&writeback_throttling_sane (sc)) +set_bit (LRUVEC_CGROUP_CONGESTED , &target_lruvec->flags); + +if (current_is_kswapd ()) +set_bit (LRUVEC_NODE_CONGESTED , &target_lruvec->flags); + } + +/* +* Stall direct reclaim for IO completions if the lruvec is +* node is congested. Allow kswapd to continue until it +* starts encountering unqueued dirty pages or cycling through +* the LRU too quickly. +*/ +if (!current_is_kswapd () &¤t_may_throttle () && + !sc->hibernation_mode && + (test_bit (LRUVEC_CGROUP_CONGESTED , &target_lruvec->flags) || +test_bit (LRUVEC_NODE_CONGESTED , &target_lruvec->flags))) +reclaim_throttle (pgdat,VMSCAN_THROTTLE_CONGESTED ); +} +static int evict_folios (struct lruvec *lruvec,struct scan_control *sc,int swappiness) {int type;@@ -4552,6 +4596,16 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap retry: reclaimed =shrink_folio_list (&list, pgdat, sc, &stat, false); sc->nr_reclaimed += reclaimed; + sc->nr.dirty += stat.nr_dirty; + sc->nr.congested += stat.nr_congested; + sc->nr.unqueued_dirty += stat.nr_unqueued_dirty; + sc->nr.writeback += stat.nr_writeback; + sc->nr.immediate += stat.nr_immediate; + sc->nr.taken += scanned; + +if (type) + sc->nr.file_taken += scanned; +trace_mm_vmscan_lru_shrink_inactive (pgdat->node_id, scanned, reclaimed, &stat, sc->priority, type ?LRU_INACTIVE_FILE :LRU_INACTIVE_ANON );
mm/vmscan.c static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) · line 5967
Quoted 6 lines
mm/vmscan.c +1
-0@@ -5908,6 +5962,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)if (lru_gen_enabled () &&root_reclaim (sc)) {lru_gen_shrink_node (pgdat, sc); +lru_gen_throttle (pgdat, sc);return ; }
Hi Bharata,
This patch arised from a regression Android test case failure which allocated 1GB virtual memory by each over 8 threads on an 5.5GB RAM system. This test could pass on legacy LRU management while failing under MGLRU as a watchdog monitor detected abnormal system-wide schedule status(watchdog can't be scheduled within 60 seconds). This patch with a slight change as below got passed in the test whereas has not been investigated deeply for how it was done. Theoretically, this patch enrolled the similar reclaim throttle mechanism as legacy do which could reduce the contention of lruvec->lru_lock. I think this patch is quite naive for now, but I am hoping it could help you as your case seems like a scability issue of memory pressure rather than a deadlock issue. Thank you!
the change of the applied version(try to throttle the reclaim before instead of after)
if (lru_gen_enabled() && root_reclaim(sc)) {
+ lru_gen_throttle(pgdat, sc);
lru_gen_shrink_node(pgdat, sc);
- lru_gen_throttle(pgdat, sc);
return;
}Quoted 4 lines
mm/vmscan.c +0
-1--2.25.1
Quoted 2 lines · Zhaoyang Huang
On Thu, Jul 25, 2024 at 6:00 PM zhaoyang.huang
<[email protected]> wrote:
<snip>
Quoted 95 lines
Quoted 75 lines
From the callstack of lock holder, it is looks like a scability issue rather than a deadlock. Unlike legacy LRU management, there is no throttling mechanism for global reclaim under mglru so far.Could we apply the similar method to throttle the reclaim when it is too aggresive. I am wondering if this patch which is a rough version could help on this?
mm/vmscan.c +54
-0@@ -4520,6 +4520,50 @@ static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, int swreturn scanned; } +static void lru_gen_throttle (pg_data_t *pgdat,struct scan_control *sc) +{ +struct lruvec *target_lruvec =mem_cgroup_lruvec (sc->target_mem_cgroup, pgdat); + +if (current_is_kswapd ()) { +if (sc->nr.writeback && sc->nr.writeback == sc->nr.taken) +set_bit (PGDAT_WRITEBACK , &pgdat->flags); + +/* Allow kswapd to start writing pages during reclaim.*/ +if (sc->nr.unqueued_dirty == sc->nr.file_taken) +set_bit (PGDAT_DIRTY , &pgdat->flags); + +if (sc->nr.immediate) +reclaim_throttle (pgdat,VMSCAN_THROTTLE_WRITEBACK ); + } + +/* +* Tag a node/memcg as congested if all the dirty pages were marked +* for writeback and immediate reclaim (counted in nr.congested). +* +* Legacy memcg will stall in page writeback so avoid forcibly +* stalling in reclaim_throttle(). +*/ +if (sc->nr.dirty && (sc->nr.dirty /2 < sc->nr.congested)) { +if (cgroup_reclaim (sc) &&writeback_throttling_sane (sc)) +set_bit (LRUVEC_CGROUP_CONGESTED , &target_lruvec->flags); + +if (current_is_kswapd ()) +set_bit (LRUVEC_NODE_CONGESTED , &target_lruvec->flags); + } + +/* +* Stall direct reclaim for IO completions if the lruvec is +* node is congested. Allow kswapd to continue until it +* starts encountering unqueued dirty pages or cycling through +* the LRU too quickly. +*/ +if (!current_is_kswapd () &¤t_may_throttle () && + !sc->hibernation_mode && + (test_bit (LRUVEC_CGROUP_CONGESTED , &target_lruvec->flags) || +test_bit (LRUVEC_NODE_CONGESTED , &target_lruvec->flags))) +reclaim_throttle (pgdat,VMSCAN_THROTTLE_CONGESTED ); +} +static int evict_folios (struct lruvec *lruvec,struct scan_control *sc,int swappiness) {int type;@@ -4552,6 +4596,16 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap retry: reclaimed =shrink_folio_list (&list, pgdat, sc, &stat, false); sc->nr_reclaimed += reclaimed; + sc->nr.dirty += stat.nr_dirty; + sc->nr.congested += stat.nr_congested; + sc->nr.unqueued_dirty += stat.nr_unqueued_dirty; + sc->nr.writeback += stat.nr_writeback; + sc->nr.immediate += stat.nr_immediate; + sc->nr.taken += scanned; + +if (type) + sc->nr.file_taken += scanned; +trace_mm_vmscan_lru_shrink_inactive (pgdat->node_id, scanned, reclaimed, &stat, sc->priority, type ?LRU_INACTIVE_FILE :LRU_INACTIVE_ANON );mm/vmscan.c static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) · line 5967
Quoted 6 lines
mm/vmscan.c +1
-0@@ -5908,6 +5962,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)if (lru_gen_enabled () &&root_reclaim (sc)) {lru_gen_shrink_node (pgdat, sc); +lru_gen_throttle (pgdat, sc);return ; }Hi Bharata,
This patch arised from a regression Android test case failure which allocated 1GB virtual memory by each over 8 threads on an 5.5GB RAM system. This test could pass on legacy LRU management while failing under MGLRU as a watchdog monitor detected abnormal system-wide schedule status(watchdog can't be scheduled within 60 seconds). This patch with a slight change as below got passed in the test whereas has not been investigated deeply for how it was done. Theoretically, this patch enrolled the similar reclaim throttle mechanism as legacy do which could reduce the contention oflruvec->lru_lock. I think this patch is quite naive for now, but I am hoping it could help you as your case seems like a scability issue of memory pressure rather than a deadlock issue. Thank you!
Quoted 8 lines
the change of the applied version(try to throttle the reclaim before instead of after)
if (lru_gen_enabled() && root_reclaim(sc)) { + lru_gen_throttle(pgdat, sc); lru_gen_shrink_node(pgdat, sc); - lru_gen_throttle(pgdat, sc); return; }
Thanks Zhaoyang Huang for the patch, will give this a test and report back.
Regards,
Bharata.
Quoted 71 lines · Yu Zhao
Quoted 37 lines · Bharata B Rao
Quoted 35 lines · Yu Zhao
Quoted 17 lines · Bharata B Rao
Hi Yu Zhao,
Thanks for your patches. See below...
Quoted 4 lines · Yu Zhao
Hi Bharata,
Quoted 1 line · Bharata B Rao
<snip>
Quoted 6 lines
Quoted 5 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.Quoted 5 lines
Quoted 3 lines
This is not really an MGLRU issue -- can you please try one of the attached patches? It (truncate.patch) should help with or without MGLRU.
With truncate.patch and default LRU scheme, a few hard lockups are seen.
Thanks.
In your original report, you said:
Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.
...
Often times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inodei_lock(inclear_shadow_entrypath)Quoted 7 lines
Quoted 4 lines
Based on this new report, does it mean the
i_lockis not as contended, for the same path (truncation) you tested? If so, I'll post truncate.patch and add reported-by and tested-by you, unless you have objections.truncate.patch has been tested on two systems with default LRU scheme and the lockup due to
inode->i_lockhasn't been seen yet after 24 hours run.Thanks.
Quoted 5 lines
Quoted 4 lines
The two paths below were contended on the LRU lock, but they already batch their operations. So I don't know what else we can do surgically to improve them.
Quoted 7 lines
What has been seen with this workload is that the lruvec spinlock is held for a long time from shrink_[active/inactive]_list path. In this path, there is a case in
isolate_lru_folios()where scanning of LRU lists can become unbounded. To isolate a page fromZONE_DMA, sometimes scanning/skipping of more than 150 million folios were seen. There is already a comment in there which explains whynr_skippedshouldn't be counted, but is there any possibility of re-looking at this condition?For this specific case, probably this can help:
(quoted diff) +1
-2@@ -1659,8 +1659,15 @@ static unsigned long isolate_lru_folios(unsigned long nr_to_scan, if (folio_zonenum(folio) > sc->reclaim_idx || skip_cma(folio, sc)) { nr_skipped[folio_zonenum(folio)] += nr_pages; - move_to = &folios_skipped; - goto move; + list_move(&folio->lru, &folios_skipped);
Review static unsigned long · line 1673
Quoted 10 lines
(quoted diff) +8
-0+ if (spin_is_contended(&lruvec->lru_lock)) { + if (!list_empty(dst)) + break; + spin_unlock_irq(&lruvec->lru_lock); + cond_resched(); + spin_lock_irq(&lruvec->lru_lock); + } + continue; }
Hi Yu,
We are seeing lockups and high memory pressure in Meta production due to this lock contention as well. My colleague highlighted it in https://lore.kernel.org/all/[email protected]/ and was pointed to this fix.
We removed skip_cma check as a temporary measure, but this is a proper fix. I might have missed it but didn't see this as a patch on the mailing list. Just wanted to check if you were planning to send it as a patch? Happy to send it on your behalf as well.
Thanks
Quoted 67 lines · Usama Arif
Quoted 63 lines · Yu Zhao
Quoted 37 lines · Bharata B Rao
Quoted 35 lines · Yu Zhao
Quoted 17 lines · Bharata B Rao
Hi Yu Zhao,
Thanks for your patches. See below...
Quoted 4 lines · Yu Zhao
Hi Bharata,
Quoted 1 line · Bharata B Rao
<snip>
Quoted 6 lines
Quoted 5 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.Quoted 5 lines
Quoted 3 lines
This is not really an MGLRU issue -- can you please try one of the attached patches? It (truncate.patch) should help with or without MGLRU.
With truncate.patch and default LRU scheme, a few hard lockups are seen.
Thanks.
In your original report, you said:
Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.
...
Often times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inodei_lock(inclear_shadow_entrypath)Quoted 7 lines
Quoted 4 lines
Based on this new report, does it mean the
i_lockis not as contended, for the same path (truncation) you tested? If so, I'll post truncate.patch and add reported-by and tested-by you, unless you have objections.truncate.patch has been tested on two systems with default LRU scheme and the lockup due to
inode->i_lockhasn't been seen yet after 24 hours run.Thanks.
Quoted 5 lines
Quoted 4 lines
The two paths below were contended on the LRU lock, but they already batch their operations. So I don't know what else we can do surgically to improve them.
Quoted 7 lines
What has been seen with this workload is that the lruvec spinlock is held for a long time from shrink_[active/inactive]_list path. In this path, there is a case in
isolate_lru_folios()where scanning of LRU lists can become unbounded. To isolate a page fromZONE_DMA, sometimes scanning/skipping of more than 150 million folios were seen. There is already a comment in there which explains whynr_skippedshouldn't be counted, but is there any possibility of re-looking at this condition?For this specific case, probably this can help:
Quoted 16 lines
Quoted 6 lines
(quoted diff) +0
-1@@ -1659,8 +1659,15 @@ static unsigned long isolate_lru_folios(unsigned long nr_to_scan, if (folio_zonenum(folio) > sc->reclaim_idx || skip_cma(folio, sc)) { nr_skipped[folio_zonenum(folio)] += nr_pages; - move_to = &folios_skipped;Review static unsigned long · line 1671
Quoted 10 lines
(quoted diff) +9
-1- goto move; + list_move(&folio->lru, &folios_skipped); + if (spin_is_contended(&lruvec->lru_lock)) { + if (!list_empty(dst)) + break; + spin_unlock_irq(&lruvec->lru_lock); + cond_resched(); + spin_lock_irq(&lruvec->lru_lock); + } + continue;
Nitpick:
if () {
...
if (!spin_is_contended(&lruvec->lru_lock))
continue;
if (!list_empty(dst))
break;
spin_unlock_irq(&lruvec->lru_lock);
cond_resched();
spin_lock_irq(&lruvec->lru_lock);
}
Quoted 5 lines
Hi Yu,
We are seeing lockups and high memory pressure in Meta production due to this lock contention as well. My colleague highlighted it in https://lore.kernel.org/all/[email protected]/ and was pointed to this fix.
We removed
skip_cmacheck as a temporary measure, but this is a proper fix. I might have missed it but didn't see this as a patch on the mailing list. Just wanted to check if you were planning to send it as a patch? Happy to send it on your behalf as well.
Please. Thank you.
Quoted 49 lines · Bharata B Rao
Quoted 35 lines · Yu Zhao
Quoted 17 lines · Bharata B Rao
Hi Yu Zhao,
Thanks for your patches. See below...
Quoted 4 lines · Yu Zhao
Hi Bharata,
Quoted 1 line · Bharata B Rao
<snip>
Quoted 6 lines
Quoted 5 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.Quoted 5 lines
Quoted 3 lines
This is not really an MGLRU issue -- can you please try one of the attached patches? It (truncate.patch) should help with or without MGLRU.
With truncate.patch and default LRU scheme, a few hard lockups are seen.
Thanks.
In your original report, you said:
Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.
...
Often times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inodei_lock(inclear_shadow_entrypath)Quoted 4 lines
Based on this new report, does it mean the
i_lockis not as contended, for the same path (truncation) you tested? If so, I'll post truncate.patch and add reported-by and tested-by you, unless you have objections.truncate.patch has been tested on two systems with default LRU scheme and the lockup due to
inode->i_lockhasn't been seen yet after 24 hours run.Quoted 4 lines
The two paths below were contended on the LRU lock, but they already batch their operations. So I don't know what else we can do surgically to improve them.
Quoted 5 lines
What has been seen with this workload is that the lruvec spinlock is held for a long time from shrink_[active/inactive]_list path. In this path, there is a case in
isolate_lru_folios()where scanning of LRU lists can become unbounded. To isolate a page fromZONE_DMA, sometimes scanning/skipping of more than 150 million folios were seen. There is
It seems weird to me to see anything that would require ZONE_DMA allocation on a modern system. Do you know where it comes from?
Quoted 6 lines
already a comment in there which explains why
nr_skippedshouldn't be counted, but is there any possibility of re-looking at this condition?Regards,
Bharata.
Quoted 50 lines · Vlastimil Babka
Quoted 49 lines · Bharata B Rao
Quoted 35 lines · Yu Zhao
Quoted 17 lines · Bharata B Rao
Hi Yu Zhao,
Thanks for your patches. See below...
Quoted 4 lines · Yu Zhao
Hi Bharata,
Quoted 1 line · Bharata B Rao
<snip>
Quoted 6 lines
Quoted 5 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.Quoted 5 lines
Quoted 3 lines
This is not really an MGLRU issue -- can you please try one of the attached patches? It (truncate.patch) should help with or without MGLRU.
With truncate.patch and default LRU scheme, a few hard lockups are seen.
Thanks.
In your original report, you said:
Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.
...
Often times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inodei_lock(inclear_shadow_entrypath)Quoted 4 lines
Based on this new report, does it mean the
i_lockis not as contended, for the same path (truncation) you tested? If so, I'll post truncate.patch and add reported-by and tested-by you, unless you have objections.truncate.patch has been tested on two systems with default LRU scheme and the lockup due to
inode->i_lockhasn't been seen yet after 24 hours run.Quoted 4 lines
The two paths below were contended on the LRU lock, but they already batch their operations. So I don't know what else we can do surgically to improve them.
Quoted 8 lines
Quoted 5 lines
What has been seen with this workload is that the lruvec spinlock is held for a long time from shrink_[active/inactive]_list path. In this path, there is a case in
isolate_lru_folios()where scanning of LRU lists can become unbounded. To isolate a page fromZONE_DMA, sometimes scanning/skipping of more than 150 million folios were seen. There isIt seems weird to me to see anything that would require
ZONE_DMAallocation on a modern system. Do you know where it comes from?
We measured the lruvec spinlock start, end and hold
time(htime) using sched_clock(), along with a BUG() if the hold time was more than 10s. The below case shows that lruvec spin lock was held for ~25s.
vmscan: unlock_page_lruvec_irq: stime 27963327514341, etime 27963324369895, htime 25889317166 (time in ns)
kernel BUG at include/linux/memcontrol.h:1677!
Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI CPU: 21 PID: 3211 Comm: kswapd0 Tainted: G W 6.10.0-rc3-qspindbg #10 RIP: 0010:shrink_active_list+0x40a/0x520 Call Trace: <TASK> shrink_lruvec+0x981/0x13b0 shrink_node+0x358/0xd30 balance_pgdat+0x3a3/0xa60 kswapd+0x207/0x3a0 kthread+0xe1/0x120 ret_from_fork+0x39/0x60 ret_from_fork_asm+0x1a/0x30 </TASK>
As you can see the call stack is from kswapd but not sure what is the exact trigger.
Regards,
Bharata.
On Wed, 17 Jul 2024 16:20:04 +0530 Bharata B Rao <[email protected]>
Quoted 9 lines · Vlastimil Babka
Quoted 3 lines
It seems weird to me to see anything that would require
ZONE_DMAallocation on a modern system. Do you know where it comes from?We measured the lruvec spinlock start, end and hold
time(htime) usingsched_clock(), along with aBUG()if the hold time was more than 10s. The below case shows that lruvec spin lock was held for ~25s.
What is more unusual could be observed perhaps with your hardware config but with 386MiB RAM assigned to each node, the so called tight memory but not extremely tight.
Quoted 6 lines · Hillf Danton
On Wed, 17 Jul 2024 16:20:04 +0530 Bharata B Rao <[email protected]>
Quoted 5 lines · Vlastimil Babka
Quoted 3 lines
It seems weird to me to see anything that would require
ZONE_DMAallocation on a modern system. Do you know where it comes from?
Quoted 7 lines
Quoted 4 lines
We measured the lruvec spinlock start, end and hold
time(htime) usingsched_clock(), along with aBUG()if the hold time was more than 10s. The below case shows that lruvec spin lock was held for ~25s.What is more unusual could be observed perhaps with your hardware config but with 386MiB RAM assigned to each node, the so called tight memory but not extremely tight.
Hardware config is this:
Dual socket AMD EPYC 128 Core processor (256 cores, 512 threads)
Memory: 1.5 TB
10 NVME - 3.5TB each
available: 2 nodes (0-1)
node 0 cpus: 0-127,256-383
node 0 size: 773727 MB
node 1 cpus: 128-255,384-511
node 1 size: 773966 MB
But I don't quite follow what you are hinting at, can you please rephrase or be more verbose?
Regards,
Bharata.
Quoted 4 lines · Yu Zhao
Quoted 4 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.
<snip>
Quoted 6 lines
Quoted 2 lines
Below preemptirqsoff trace points to preemption being disabled for more than 10s and the lock in picture is lruvec spinlock.
Also if you could try the other patch (mglru.patch) please. It should help reduce unnecessary rotations from
deactivate_file_folio(), which in turn should reduce the contention on the LRU lock for MGLRU.
Thanks. With mglru.patch on a MGLRU-enabled system, the below latency trace record is no longer seen for a 30hr workload run.
Quoted 32 lines
Quoted 32 lines
# tracer: preemptirqsoff # # preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-mglru-irqstrc # -------------------------------------------------------------------- # latency: 10382682 us, #4/4, CPU#128 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:512) # ----------------- # | task: fio-2701523 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # => started at: deactivate_file_folio # => ended at: deactivate_file_folio # # # _------=> CPU# # / _-----=> irqs-off/BH-disabled # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / _-=> migrate-disable # ||||| / delay # cmd pid |||||| time | caller # \ / |||||| \ | / fio-2701523 128...1. 0us$: deactivate_file_folio <-deactivate_file_folio fio-2701523 128.N.1. 10382681us : deactivate_file_folio <-deactivate_file_folio fio-2701523 128.N.1. 10382683us : tracer_preempt_on <-deactivate_file_folio fio-2701523 128.N.1. 10382691us : <stack trace> => deactivate_file_folio => mapping_try_invalidate
Quoted 8 lines
Quoted 8 lines
=> invalidate_mapping_pages => invalidate_bdev => blkdev_common_ioctl => blkdev_ioctl => __x64_sys_ioctl => x64_sys_call => do_syscall_64 => entry_SYSCALL_64_after_hwframe
However the contention now has shifted to inode_hash_lock. Around 55 softlockups in ilookup() were observed:
# tracer: preemptirqsoff
#
# preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-trnmglru
# --------------------------------------------------------------------
# latency: 10620430 us, #4/4, CPU#260 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:512)
# -----------------
# | task: fio-3244715 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: ilookup
# => ended at: ilookup
#
#
# _------=> CPU#
# / _-----=> irqs-off/BH-disabled
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / _-=> migrate-disable
# ||||| / delay
# cmd pid |||||| time | caller
# \ / |||||| \ | /
fio-3244715 260...1. 0us$: _raw_spin_lock <-ilookup
fio-3244715 260.N.1. 10620429us : _raw_spin_unlock <-ilookup
fio-3244715 260.N.1. 10620430us : tracer_preempt_on <-ilookup
fio-3244715 260.N.1. 10620440us : <stack trace>
=> _raw_spin_unlock
=> ilookup
=> blkdev_get_no_open
=> blkdev_open
=> do_dentry_open
=> vfs_open
=> path_openat
=> do_filp_open
=> do_sys_openat2
=> __x64_sys_openat
=> x64_sys_call
=> do_syscall_64
=> entry_SYSCALL_64_after_hwframeIt appears that scalability issues with inode_hash_lock has been brought up multiple times in the past and there were patches to address the same.
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
CC'ing FS folks/list for awareness/comments.
Regards,
Bharata.
Quoted 101 lines · Bharata B Rao
Quoted 4 lines · Yu Zhao
Quoted 4 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.<snip>
Quoted 6 lines
Quoted 2 lines
Below preemptirqsoff trace points to preemption being disabled for more than 10s and the lock in picture is lruvec spinlock.
Also if you could try the other patch (mglru.patch) please. It should help reduce unnecessary rotations from
deactivate_file_folio(), which in turn should reduce the contention on the LRU lock for MGLRU.Thanks. With mglru.patch on a MGLRU-enabled system, the below latency trace record is no longer seen for a 30hr workload run.
Quoted 32 lines
Quoted 32 lines
# tracer: preemptirqsoff # # preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-mglru-irqstrc # -------------------------------------------------------------------- # latency: 10382682 us, #4/4, CPU#128 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:512) # ----------------- # | task: fio-2701523 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # => started at: deactivate_file_folio # => ended at: deactivate_file_folio # # # _------=> CPU# # / _-----=> irqs-off/BH-disabled # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / _-=> migrate-disable # ||||| / delay # cmd pid |||||| time | caller # \ / |||||| \ | / fio-2701523 128...1. 0us$: deactivate_file_folio <-deactivate_file_folio fio-2701523 128.N.1. 10382681us : deactivate_file_folio <-deactivate_file_folio fio-2701523 128.N.1. 10382683us : tracer_preempt_on <-deactivate_file_folio fio-2701523 128.N.1. 10382691us : <stack trace> => deactivate_file_folio => mapping_try_invalidateQuoted 8 lines
Quoted 8 lines
=> invalidate_mapping_pages => invalidate_bdev => blkdev_common_ioctl => blkdev_ioctl => __x64_sys_ioctl => x64_sys_call => do_syscall_64 => entry_SYSCALL_64_after_hwframeHowever the contention now has shifted to
inode_hash_lock. Around 55 softlockups inilookup()were observed:# tracer: preemptirqsoff
## preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-trnmglru # -------------------------------------------------------------------- # latency: 10620430 us, #4/4, CPU#260 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:512) # ----------------- # | task: fio-3244715 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # => started at: ilookup # => ended at: ilookup # # # _------=> CPU# # / _-----=> irqs-off/BH-disabled # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / _-=> migrate-disable # ||||| / delay # cmd pid |||||| time | caller # \ / |||||| \ | / fio-3244715 260...1. 0us$: _raw_spin_lock <-ilookup fio-3244715 260.N.1. 10620429us : _raw_spin_unlock <-ilookup fio-3244715 260.N.1. 10620430us : tracer_preempt_on <-ilookup fio-3244715 260.N.1. 10620440us : <stack trace> => _raw_spin_unlock => ilookup => blkdev_get_no_open => blkdev_open => do_dentry_open => vfs_open => path_openat => do_filp_open => do_sys_openat2 => __x64_sys_openat => x64_sys_call => do_syscall_64 => entry_SYSCALL_64_after_hwframe
Quoted 7 lines
It appears that scalability issues with
inode_hash_lockhas been brought up multiple times in the past and there were patches to address the same.https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/lkml/[email protected]/CC'ing FS folks/list for awareness/comments.
Note my patch does not enable RCU usage in ilookup, but this can be trivially added.
I can't even compile-test at the moment, but the diff below should do it. Also note the patches are present here
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/log/?h=vfs.inode.rcu
, not yet integrated anywhere.
That said, if fio you are operating on the same target inode every time then this is merely going to shift contention to the inode spinlock usage in find_inode_fast.
fs/inode.c +0 -0
@@ -1524,10 +1524,14 @@ struct inode *ilookup(struct super_block *sb,
unsigned long ino)
{
struct hlist_head *head = inode_hashtable + hash(sb, ino);
struct inode *inode;
+
again:
- spin_lock(&inode_hash_lock);
- inode = find_inode_fast(sb, head, ino, true);
- spin_unlock(&inode_hash_lock);
+ inode = find_inode_fast(sb, head, ino, false);
+ if (IS_ERR_OR_NULL_PTR(inode)) {
+ spin_lock(&inode_hash_lock);
+ inode = find_inode_fast(sb, head, ino, true);
+ spin_unlock(&inode_hash_lock);+ }
if (inode) {
if (IS_ERR(inode))Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
Quoted 145 lines · Mateusz Guzik
Quoted 101 lines · Bharata B Rao
Quoted 4 lines · Yu Zhao
Quoted 4 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.<snip>
Quoted 6 lines
Quoted 2 lines
Below preemptirqsoff trace points to preemption being disabled for more than 10s and the lock in picture is lruvec spinlock.
Also if you could try the other patch (mglru.patch) please. It should help reduce unnecessary rotations from
deactivate_file_folio(), which in turn should reduce the contention on the LRU lock for MGLRU.Thanks. With mglru.patch on a MGLRU-enabled system, the below latency trace record is no longer seen for a 30hr workload run.
Quoted 32 lines
Quoted 32 lines
# tracer: preemptirqsoff # # preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-mglru-irqstrc # -------------------------------------------------------------------- # latency: 10382682 us, #4/4, CPU#128 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:512) # ----------------- # | task: fio-2701523 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # => started at: deactivate_file_folio # => ended at: deactivate_file_folio # # # _------=> CPU# # / _-----=> irqs-off/BH-disabled # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / _-=> migrate-disable # ||||| / delay # cmd pid |||||| time | caller # \ / |||||| \ | / fio-2701523 128...1. 0us$: deactivate_file_folio <-deactivate_file_folio fio-2701523 128.N.1. 10382681us : deactivate_file_folio <-deactivate_file_folio fio-2701523 128.N.1. 10382683us : tracer_preempt_on <-deactivate_file_folio fio-2701523 128.N.1. 10382691us : <stack trace> => deactivate_file_folio => mapping_try_invalidateQuoted 8 lines
Quoted 8 lines
=> invalidate_mapping_pages => invalidate_bdev => blkdev_common_ioctl => blkdev_ioctl => __x64_sys_ioctl => x64_sys_call => do_syscall_64 => entry_SYSCALL_64_after_hwframeHowever the contention now has shifted to
inode_hash_lock. Around 55 softlockups inilookup()were observed:# tracer: preemptirqsoff
## preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-trnmglru # -------------------------------------------------------------------- # latency: 10620430 us, #4/4, CPU#260 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:512) # ----------------- # | task: fio-3244715 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # => started at: ilookup # => ended at: ilookup # # # _------=> CPU# # / _-----=> irqs-off/BH-disabled # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / _-=> migrate-disable # ||||| / delay # cmd pid |||||| time | caller # \ / |||||| \ | / fio-3244715 260...1. 0us$: _raw_spin_lock <-ilookup fio-3244715 260.N.1. 10620429us : _raw_spin_unlock <-ilookup fio-3244715 260.N.1. 10620430us : tracer_preempt_on <-ilookup fio-3244715 260.N.1. 10620440us : <stack trace> => _raw_spin_unlock => ilookup => blkdev_get_no_open => blkdev_open => do_dentry_open => vfs_open => path_openat => do_filp_open => do_sys_openat2 => __x64_sys_openat => x64_sys_call => do_syscall_64 => entry_SYSCALL_64_after_hwframeQuoted 7 lines
It appears that scalability issues with
inode_hash_lockhas been brought up multiple times in the past and there were patches to address the same.https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/lkml/[email protected]/CC'ing FS folks/list for awareness/comments.
Note my patch does not enable RCU usage in ilookup, but this can be trivially added.
I can't even compile-test at the moment, but the diff below should do it. Also note the patches are present here
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/log/?h=vfs.inode.rcu
, not yet integrated anywhere.That said, if fio you are operating on the same target inode every time then this is merely going to shift contention to the inode spinlock usage in
find_inode_fast.fs/inode.c +7
-3@@ -1524,10 +1524,14 @@ struct inode *ilookup(struct super_block *sb, unsigned long ino) {struct hlist_head *head = inode_hashtable +hash (sb, ino);struct inode *inode; + again: -spin_lock (&inode_hash_lock); - inode =find_inode_fast (sb, head, ino, true); -spin_unlock (&inode_hash_lock); + inode =find_inode_fast (sb, head, ino, false); +if (IS_ERR_OR_NULL_PTR (inode)) { +spin_lock (&inode_hash_lock); + inode =find_inode_fast (sb, head, ino, true); +spin_unlock (&inode_hash_lock); + }if (inode) {
Quoted 2 lines
if (IS_ERR(inode))
I think I expressed myself poorly, so here is take two:
1. inode hash soft lookup should get resolved if you apply
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs.inode.rcu&id=7180f8d91fcbf252de572d9ffacc945effed0060
and the above pasted fix (not compile tested tho, but it should be obvious what the intended fix looks like)
2. find_inode_hash spinlocks the target inode. if your bench only operates on one, then contention is going to shift there and you may still be getting soft lockups. not taking the spinlock in this codepath is hackable, but I don't want to do it without a good justification.
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
Quoted 84 lines · Mateusz Guzik
Quoted 84 lines
Quoted 43 lines
However the contention now has shifted to
inode_hash_lock. Around 55 softlockups inilookup()were observed:# tracer: preemptirqsoff
## preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-trnmglru # -------------------------------------------------------------------- # latency: 10620430 us, #4/4, CPU#260 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:512) # ----------------- # | task: fio-3244715 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # => started at: ilookup # => ended at: ilookup # # # _------=> CPU# # / _-----=> irqs-off/BH-disabled # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / _-=> migrate-disable # ||||| / delay # cmd pid |||||| time | caller # \ / |||||| \ | / fio-3244715 260...1. 0us$: _raw_spin_lock <-ilookup fio-3244715 260.N.1. 10620429us : _raw_spin_unlock <-ilookup fio-3244715 260.N.1. 10620430us : tracer_preempt_on <-ilookup fio-3244715 260.N.1. 10620440us : <stack trace> => _raw_spin_unlock => ilookup => blkdev_get_no_open => blkdev_open => do_dentry_open => vfs_open => path_openat => do_filp_open => do_sys_openat2 => __x64_sys_openat => x64_sys_call => do_syscall_64 => entry_SYSCALL_64_after_hwframeQuoted 7 lines
It appears that scalability issues with
inode_hash_lockhas been brought up multiple times in the past and there were patches to address the same.https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/lkml/[email protected]/CC'ing FS folks/list for awareness/comments.
Note my patch does not enable RCU usage in ilookup, but this can be trivially added.
I can't even compile-test at the moment, but the diff below should do it. Also note the patches are present here
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/log/?h=vfs.inode.rcu
, not yet integrated anywhere.That said, if fio you are operating on the same target inode every time then this is merely going to shift contention to the inode spinlock usage in
find_inode_fast.fs/inode.c +7
-3@@ -1524,10 +1524,14 @@ struct inode *ilookup(struct super_block *sb, unsigned long ino) {struct hlist_head *head = inode_hashtable +hash (sb, ino);struct inode *inode; + again: -spin_lock (&inode_hash_lock); - inode =find_inode_fast (sb, head, ino, true); -spin_unlock (&inode_hash_lock); + inode =find_inode_fast (sb, head, ino, false); +if (IS_ERR_OR_NULL_PTR (inode)) { +spin_lock (&inode_hash_lock); + inode =find_inode_fast (sb, head, ino, true); +spin_unlock (&inode_hash_lock); + }
Quoted 14 lines
Quoted 3 lines
fs/inode.c +0
-0if (inode) {if (IS_ERR(inode))I think I expressed myself poorly, so here is take two:
1. inode hash soft lookup should get resolved if you apply
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs.inode.rcu&id=7180f8d91fcbf252de572d9ffacc945effed0060
and the above pasted fix (not compile tested tho, but it should be obvious what the intended fix looks like)
2.find_inode_hashspinlocks the target inode. if your bench only operates on one, then contention is going to shift there and you may still be getting soft lockups. not taking the spinlock in this codepath is hackable, but I don't want to do it without a good justification.
Thanks Mateusz for the fix. With this patch applied, the above mentioned contention in ilookup() has not been observed for a test run during the weekend.
Regards,
Bharata.
Quoted 101 lines · Bharata B Rao
Quoted 84 lines · Mateusz Guzik
Quoted 84 lines
Quoted 43 lines
However the contention now has shifted to
inode_hash_lock. Around 55 softlockups inilookup()were observed:# tracer: preemptirqsoff
## preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-trnmglru # -------------------------------------------------------------------- # latency: 10620430 us, #4/4, CPU#260 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:512) # ----------------- # | task: fio-3244715 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # => started at: ilookup # => ended at: ilookup # # # _------=> CPU# # / _-----=> irqs-off/BH-disabled # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / _-=> migrate-disable # ||||| / delay # cmd pid |||||| time | caller # \ / |||||| \ | / fio-3244715 260...1. 0us$: _raw_spin_lock <-ilookup fio-3244715 260.N.1. 10620429us : _raw_spin_unlock <-ilookup fio-3244715 260.N.1. 10620430us : tracer_preempt_on <-ilookup fio-3244715 260.N.1. 10620440us : <stack trace> => _raw_spin_unlock => ilookup => blkdev_get_no_open => blkdev_open => do_dentry_open => vfs_open => path_openat => do_filp_open => do_sys_openat2 => __x64_sys_openat => x64_sys_call => do_syscall_64 => entry_SYSCALL_64_after_hwframeQuoted 7 lines
It appears that scalability issues with
inode_hash_lockhas been brought up multiple times in the past and there were patches to address the same.https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/lkml/[email protected]/CC'ing FS folks/list for awareness/comments.
Note my patch does not enable RCU usage in ilookup, but this can be trivially added.
I can't even compile-test at the moment, but the diff below should do it. Also note the patches are present here
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/log/?h=vfs.inode.rcu
, not yet integrated anywhere.That said, if fio you are operating on the same target inode every time then this is merely going to shift contention to the inode spinlock usage in
find_inode_fast.fs/inode.c +7
-3@@ -1524,10 +1524,14 @@ struct inode *ilookup(struct super_block *sb, unsigned long ino) {struct hlist_head *head = inode_hashtable +hash (sb, ino);struct inode *inode; + again: -spin_lock (&inode_hash_lock); - inode =find_inode_fast (sb, head, ino, true); -spin_unlock (&inode_hash_lock); + inode =find_inode_fast (sb, head, ino, false); +if (IS_ERR_OR_NULL_PTR (inode)) { +spin_lock (&inode_hash_lock); + inode =find_inode_fast (sb, head, ino, true); +spin_unlock (&inode_hash_lock); + }Quoted 14 lines
Quoted 3 lines
fs/inode.c +0
-0if (inode) {if (IS_ERR(inode))I think I expressed myself poorly, so here is take two:
1. inode hash soft lookup should get resolved if you apply
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs.inode.rcu&id=7180f8d91fcbf252de572d9ffacc945effed0060
and the above pasted fix (not compile tested tho, but it should be obvious what the intended fix looks like)
2.find_inode_hashspinlocks the target inode. if your bench only operates on one, then contention is going to shift there and you may still be getting soft lockups. not taking the spinlock in this codepath is hackable, but I don't want to do it without a good justification.
Quoted 4 lines
Thanks Mateusz for the fix. With this patch applied, the above mentioned contention in
ilookup()has not been observed for a test run during the weekend.
Ok, I'll do some clean ups and send a proper patch to the vfs folks later today.
Thanks for testing.
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
Quoted 10 lines · Bharata B Rao
Quoted 4 lines · Yu Zhao
Quoted 4 lines
Some experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.<snip>
Quoted 3 lines
Quoted 2 lines
Below preemptirqsoff trace points to preemption being disabled for more than 10s and the lock in picture is lruvec spinlock.
Quoted 6 lines
Quoted 3 lines
Also if you could try the other patch (mglru.patch) please. It should help reduce unnecessary rotations from
deactivate_file_folio(), which in turn should reduce the contention on the LRU lock for MGLRU.Thanks. With mglru.patch on a MGLRU-enabled system, the below latency trace record is no longer seen for a 30hr workload run.
Glad to hear. Will post a patch and add you as reported/tested-by.
Quoted 41 lines
Quoted 32 lines
Quoted 32 lines
# tracer: preemptirqsoff # # preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-mglru-irqstrc # -------------------------------------------------------------------- # latency: 10382682 us, #4/4, CPU#128 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:512) # ----------------- # | task: fio-2701523 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # => started at: deactivate_file_folio # => ended at: deactivate_file_folio # # # _------=> CPU# # / _-----=> irqs-off/BH-disabled # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / _-=> migrate-disable # ||||| / delay # cmd pid |||||| time | caller # \ / |||||| \ | / fio-2701523 128...1. 0us$: deactivate_file_folio <-deactivate_file_folio fio-2701523 128.N.1. 10382681us : deactivate_file_folio <-deactivate_file_folio fio-2701523 128.N.1. 10382683us : tracer_preempt_on <-deactivate_file_folio fio-2701523 128.N.1. 10382691us : <stack trace> => deactivate_file_folio => mapping_try_invalidateQuoted 8 lines
Quoted 8 lines
=> invalidate_mapping_pages => invalidate_bdev => blkdev_common_ioctl => blkdev_ioctl => __x64_sys_ioctl => x64_sys_call => do_syscall_64 => entry_SYSCALL_64_after_hwframe
Quoted 2 lines
However the contention now has shifted to
inode_hash_lock. Around 55 softlockups inilookup()were observed:
This one is from fs/blk, so I'll leave it to those experts.
Quoted 50 lines
# tracer: preemptirqsoff
## preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-trnmglru # -------------------------------------------------------------------- # latency: 10620430 us, #4/4, CPU#260 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:512) # ----------------- # | task: fio-3244715 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # => started at: ilookup # => ended at: ilookup # # # _------=> CPU# # / _-----=> irqs-off/BH-disabled # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / _-=> migrate-disable # ||||| / delay # cmd pid |||||| time | caller # \ / |||||| \ | / fio-3244715 260...1. 0us$: _raw_spin_lock <-ilookup fio-3244715 260.N.1. 10620429us : _raw_spin_unlock <-ilookup fio-3244715 260.N.1. 10620430us : tracer_preempt_on <-ilookup fio-3244715 260.N.1. 10620440us : <stack trace> => _raw_spin_unlock => ilookup => blkdev_get_no_open => blkdev_open => do_dentry_open => vfs_open => path_openat => do_filp_open => do_sys_openat2 => __x64_sys_openat => x64_sys_call => do_syscall_64 => entry_SYSCALL_64_after_hwframeIt appears that scalability issues with
inode_hash_lockhas been brought up multiple times in the past and there were patches to address the same.https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/lkml/[email protected]/CC'ing FS folks/list for awareness/comments.
Regards,
Bharata.
Quoted 11 lines · Bharata B Rao
Many soft and hard lockups are seen with upstream kernel when running a bunch of tests that include FIO and LTP filesystem test on 10 NVME disks. The lockups can appear anywhere between 2 to 48 hours. Originally this was reported on a large customer VM instance with passthrough NVME disks on older kernels(v5.4 based). However, similar problems were reproduced when running the tests on bare metal with latest upstream kernel (v6.10-rc3). Other lockups with different signatures are seen but in this report, only those related to MM area are being discussed. Also note that the subsequent description is related to the lockups in bare metal upstream (and not VM).
Quoted 7 lines
The general observation is that the problem usually surfaces when the system free memory goes very low and page cache/buffer consumption hits the ceiling. Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.- Could this be a scalability issue in LRU list handling and/or page cache invalidation typical to a large system configuration?
Seems to me it could be (except that ZONE_DMA corner case) a general scalability issue in that you tweak some part of the kernel and the contention moves elsewhere. At least in MM we have per-node locks so this means 256 CPUs per lock? It used to be that there were not that many (cores/threads) per a physical CPU and its NUMA node, so many cpus would mean also more NUMA nodes where the locks contention would distribute among them. I think you could try fakenuma to create these nodes artificially and see if it helps for the MM part. But if the contention moves to e.g. an inode lock, I'm not sure what to do about that then.
Quoted 224 lines
- Are there any MM/FS tunables that could help here?
Hardware configuration
======================
Dual socket AMD EPYC 128 Core processor (256 cores, 512 threads)
Memory: 1.5 TB
10 NVME - 3.5TB each
available: 2 nodes (0-1)
node 0 cpus: 0-127,256-383
node 0 size: 773727 MB
node 1 cpus: 128-255,384-511
node 1 size: 773966 MBWorkload details
================
Workload includes concurrent runs of FIO and a few FS tests from LTP.FIO is run with a size of 1TB on each NVME partition with different combinations of ioengine/blocksize/mode parameters and buffered-IO. Selected FS tests from LTP are run on 256GB partitions of all NVME disks. This is the typical NVME partition layout.
nvme2n1 259:4 0 3.5T 0 disk ├─nvme2n1p1 259:6 0 256G 0 part /data_nvme2n1p1 └─nvme2n1p2 259:7 0 3.2T 0 partThough many different runs exist in the workload, the combination that results in the problem is buffered-IO run with sync engine.
fio -filename=/dev/nvme1n1p2 -direct=0 -thread -size=1024G \
(quoted diff) +0
-2-rwmixwrite=30 --norandommap --randrepeat=0 -ioengine=sync -bs=4k \ -numjobs=400 -runtime=25000 --time_based -group_reporting -name=mytestWatchdog threshold was reduced to 5s to reproduce the problem early and all CPU backtrace enabled.
Problem details and analysis
============================
One of the hard lockups which was observed and analyzed in detail is this:kernel: watchdog: Watchdog detected hard LOCKUP on cpu 284 kernel: CPU: 284 PID: 924096 Comm: cat Not tainted 6.10.0-rc3-lruvec #9 kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: Call Trace: kernel: <NMI> kernel: ? show_regs+0x69/0x80 kernel: ? watchdog_hardlockup_check+0x19e/0x360 <SNIP> kernel: ? native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: </NMI> kernel: <TASK> kernel: ? __pfx_lru_add_fn+0x10/0x10 kernel: _raw_spin_lock_irqsave+0x42/0x50 kernel: folio_lruvec_lock_irqsave+0x62/0xb0 kernel: folio_batch_move_lru+0x79/0x2a0 kernel: folio_add_lru+0x6d/0xf0 kernel: filemap_add_folio+0xba/0xe0 kernel: __filemap_get_folio+0x137/0x2e0 kernel: ext4_da_write_begin+0x12c/0x270 kernel: generic_perform_write+0xbf/0x200 kernel: ext4_buffered_write_iter+0x67/0xf0 kernel: ext4_file_write_iter+0x70/0x780 kernel: vfs_write+0x301/0x420 kernel: ksys_write+0x67/0xf0 kernel: __x64_sys_write+0x19/0x20 kernel: x64_sys_call+0x1689/0x20d0 kernel: do_syscall_64+0x6b/0x110 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e kernel: RIP:0033:0x7fe21c314887
With all CPU backtraces enabled, many CPUs are waiting for
lruvec_lockacquisition. We measured the lruvec spinlock start, end and hold time(htime) usingsched_clock(), along with aBUG()if the hold time was more than 10s. The below case shows that lruvec spin lock was held for ~25s.kernel: vmscan:
unlock_page_lruvec_irq: stime 27963327514341, etime 27963324369895, htime 25889317166
kernel: ------------[ cut here ]------------
kernel: kernel BUG atinclude/linux/memcontrol.h:1677!
kernel: Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
kernel: CPU: 21 PID: 3211 Comm: kswapd0 Tainted: G W
6.10.0-rc3-qspindbg #10
kernel: RIP: 0010:shrink_active_list+0x40a/0x520And the corresponding trace point for the above:
kswapd0-3211 [021] dN.1. 27963.324332:mm_vmscan_lru_isolate:
classzone=0 order=0 nr_requested=1 nr_scanned=156946361 nr_skipped=156946360 nr_taken=1 lru=active_fileThis shows that
isolate_lru_folios()is scanning through a huge number (~150million) of folios (order=0) with lruvec spinlock held. This is happening because a large number of folios are being skipped to isolate a fewZONE_DMAfolios. Though the number of folios to be scanned is bounded (32), there exists a genuine case where this can become unbounded, i.e. in case where folios are skipped.Meminfo output shows that the free memory is around ~2% and page/buffer cache grows very high when the lockup happens.
MemTotal: 1584835956 kB MemFree: 27805664 kB MemAvailable: 1568099004 kB Buffers: 1386120792 kB Cached: 151894528 kB SwapCached: 30620 kB Active: 1043678892 kB Inactive: 494456452 kBOften times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inode
i_lock(inclear_shadow_entrypath)98.98% fio [kernel.kallsyms] [k]
native_queued_spin_lock_slowpath| --98.96%--native_queued_spin_lock_slowpath | --98.96%--_raw_spin_lock_irqsave folio_lruvec_lock_irqsave | --98.78%--folio_batch_move_lru | --98.63%--deactivate_file_folio mapping_try_invalidate invalidate_mapping_pages invalidate_bdev blkdev_common_ioctl blkdev_ioctl __x64_sys_ioctl x64_sys_call do_syscall_64 entry_SYSCALL_64_after_hwframeSome experiments tried
======================
1) When MGLRU was enabled many soft lockups were observed, no hard lockups were seen for 48 hours run. Below is once such soft lockup.kernel: watchdog: BUG: soft lockup - CPU#29 stuck for 11s! [fio:2701649]
kernel: CPU: 29 PID: 2701649 Comm: fio Tainted: G L
6.10.0-rc3-mglru-irqstrc #24kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: Call Trace: kernel: <IRQ> kernel: ? show_regs+0x69/0x80 kernel: ? watchdog_timer_fn+0x223/0x2b0 kernel: ? __pfx_watchdog_timer_fn+0x10/0x10 <SNIP> kernel: </IRQ> kernel: <TASK> kernel: ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 kernel: ? native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: _raw_spin_lock+0x38/0x50 kernel: clear_shadow_entry+0x3d/0x100 kernel: ? __pfx_workingset_update_node+0x10/0x10 kernel: mapping_try_invalidate+0x117/0x1d0 kernel: invalidate_mapping_pages+0x10/0x20 kernel: invalidate_bdev+0x3c/0x50 kernel: blkdev_common_ioctl+0x5f7/0xa90 kernel: blkdev_ioctl+0x109/0x270 kernel: x64_sys_call+0x1215/0x20d0 kernel: do_syscall_64+0x7e/0x130This happens to be contending on inode
i_lockspinlock.Below preemptirqsoff trace points to preemption being disabled for more than 10s and the lock in picture is lruvec spinlock.
# tracer: preemptirqsoff # # preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-mglru-irqstrc # -------------------------------------------------------------------- # latency: 10382682 us, #4/4, CPU#128 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:512) # ----------------- # | task: fio-2701523 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # => started at: deactivate_file_folio # => ended at: deactivate_file_folio # # # _------=> CPU# # / _-----=> irqs-off/BH-disabled # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / _-=> migrate-disable # ||||| / delay # cmd pid |||||| time | caller # \ / |||||| \ | / fio-2701523 128...1. 0us$: deactivate_file_folio <-deactivate_file_folio fio-2701523 128.N.1. 10382681us : deactivate_file_folio <-deactivate_file_folio fio-2701523 128.N.1. 10382683us : tracer_preempt_on <-deactivate_file_folio fio-2701523 128.N.1. 10382691us : <stack trace> => deactivate_file_folio => mapping_try_invalidate => invalidate_mapping_pages => invalidate_bdev => blkdev_common_ioctl => blkdev_ioctl => __x64_sys_ioctl => x64_sys_call => do_syscall_64 => entry_SYSCALL_64_after_hwframe2) Increased
low_watermark_thresholdto 10% to prevent system from entering into extremely low memory situation. Although hard lockups weren't seen, but soft lockups (clear_shadow_entry()) were still seen.3) AMD has a BIOS setting called NPS (Nodes per socket), using which a socket can be further partitioned into smaller NUMA nodes. With NPS=4, there will be four NUMA nodes in one socket, and hence 8 NUMA nodes in the system. This was done to check if having more number of kswapd threads working on lesser number of folios per node would make a difference. However here too, multiple soft lockups were seen (in
clear_shadow_entry()as seen in MGLRU case). No hard lockups were observed.Any insights/suggestion into these lockups and suggestions are welcome!
Regards,
Bharata.
Quoted 20 lines · Vlastimil Babka
Quoted 11 lines · Bharata B Rao
Many soft and hard lockups are seen with upstream kernel when running a bunch of tests that include FIO and LTP filesystem test on 10 NVME disks. The lockups can appear anywhere between 2 to 48 hours. Originally this was reported on a large customer VM instance with passthrough NVME disks on older kernels(v5.4 based). However, similar problems were reproduced when running the tests on bare metal with latest upstream kernel (v6.10-rc3). Other lockups with different signatures are seen but in this report, only those related to MM area are being discussed. Also note that the subsequent description is related to the lockups in bare metal upstream (and not VM).
Quoted 7 lines
The general observation is that the problem usually surfaces when the system free memory goes very low and page cache/buffer consumption hits the ceiling. Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.- Could this be a scalability issue in LRU list handling and/or page cache invalidation typical to a large system configuration?
Quoted 9 lines
Seems to me it could be (except that
ZONE_DMAcorner case) a general scalability issue in that you tweak some part of the kernel and the contention moves elsewhere. At least in MM we have per-node locks so this means 256 CPUs per lock? It used to be that there were not that many (cores/threads) per a physical CPU and its NUMA node, so many cpus would mean also more NUMA nodes where the locks contention would distribute among them. I think you could try fakenuma to create these nodes artificially and see if it helps for the MM part. But if the contention moves to e.g. an inode lock, I'm not sure what to do about that then.
See below...
Quoted 1 line
<SNIP>
Quoted 8 lines
Quoted 8 lines
3) AMD has a BIOS setting called NPS (Nodes per socket), using which a socket can be further partitioned into smaller NUMA nodes. With NPS=4, there will be four NUMA nodes in one socket, and hence 8 NUMA nodes in the system. This was done to check if having more number of kswapd threads working on lesser number of folios per node would make a difference. However here too, multiple soft lockups were seen (in
clear_shadow_entry()as seen in MGLRU case). No hard lockups were observed.
These are some softlockups seen with NPS4 mode.
watchdog: BUG: soft lockup - CPU#315 stuck for 11s! [kworker/315:1H:5153] CPU: 315 PID: 5153 Comm: kworker/315:1H Kdump: loaded Not tainted 6.10.0-rc3-enbprftw #12 Workqueue: kblockd blk_mq_run_work_fn RIP: 0010:handle_softirqs+0x70/0x2f0 Call Trace: <IRQ> __irq_exit_rcu+0x68/0x90 irq_exit_rcu+0x12/0x20 sysvec_apic_timer_interrupt+0x85/0xb0 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x1f/0x30 RIP: 0010:iommu_dma_map_page+0xca/0x2c0 dma_map_page_attrs+0x20d/0x2a0 nvme_prep_rq.part.0+0x63d/0x940 [nvme] nvme_queue_rq+0x82/0x210 [nvme] blk_mq_dispatch_rq_list+0x289/0x6d0 __blk_mq_sched_dispatch_requests+0x142/0x5f0 blk_mq_sched_dispatch_requests+0x36/0x70 blk_mq_run_work_fn+0x73/0x90 process_one_work+0x185/0x3d0 worker_thread+0x2ce/0x3e0 kthread+0xe5/0x120 ret_from_fork+0x3d/0x60 ret_from_fork_asm+0x1a/0x30 watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [fio:19820] CPU: 0 PID: 19820 Comm: fio Kdump: loaded Tainted: G L 6.10.0-rc3-enbprftw #12 RIP: 0010:native_queued_spin_lock_slowpath+0x2b8/0x300 Call Trace: <IRQ> </IRQ> <TASK> _raw_spin_lock+0x2d/0x40 clear_shadow_entry+0x3d/0x100 mapping_try_invalidate+0x11b/0x1e0 invalidate_mapping_pages+0x14/0x20 invalidate_bdev+0x40/0x50 blkdev_common_ioctl+0x5f7/0xa90 blkdev_ioctl+0x10d/0x270 __x64_sys_ioctl+0x99/0xd0 x64_sys_call+0x1219/0x20d0 do_syscall_64+0x51/0x120 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x7fc92fc3ec6b </TASK>
The above one (clear_shadow_entry) has since been fixed by Yu Zhao and fix is in mm tree.
We had seen a couple of scenarios with zone lock contention from page free and slab free code paths, as reported here: https://lore.kernel.org/linux-mm/[email protected]/
Would you have any insights on these?
Regards,
Bharata.
Quoted 96 lines · Bharata B Rao
Quoted 20 lines · Vlastimil Babka
Quoted 11 lines · Bharata B Rao
Many soft and hard lockups are seen with upstream kernel when running a bunch of tests that include FIO and LTP filesystem test on 10 NVME disks. The lockups can appear anywhere between 2 to 48 hours. Originally this was reported on a large customer VM instance with passthrough NVME disks on older kernels(v5.4 based). However, similar problems were reproduced when running the tests on bare metal with latest upstream kernel (v6.10-rc3). Other lockups with different signatures are seen but in this report, only those related to MM area are being discussed. Also note that the subsequent description is related to the lockups in bare metal upstream (and not VM).
Quoted 7 lines
The general observation is that the problem usually surfaces when the system free memory goes very low and page cache/buffer consumption hits the ceiling. Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.- Could this be a scalability issue in LRU list handling and/or page cache invalidation typical to a large system configuration?
Quoted 9 lines
Seems to me it could be (except that
ZONE_DMAcorner case) a general scalability issue in that you tweak some part of the kernel and the contention moves elsewhere. At least in MM we have per-node locks so this means 256 CPUs per lock? It used to be that there were not that many (cores/threads) per a physical CPU and its NUMA node, so many cpus would mean also more NUMA nodes where the locks contention would distribute among them. I think you could try fakenuma to create these nodes artificially and see if it helps for the MM part. But if the contention moves to e.g. an inode lock, I'm not sure what to do about that then.See below...
Quoted 1 line
<SNIP>
Quoted 8 lines
Quoted 8 lines
3) AMD has a BIOS setting called NPS (Nodes per socket), using which a socket can be further partitioned into smaller NUMA nodes. With NPS=4, there will be four NUMA nodes in one socket, and hence 8 NUMA nodes in the system. This was done to check if having more number of kswapd threads working on lesser number of folios per node would make a difference. However here too, multiple soft lockups were seen (in
clear_shadow_entry()as seen in MGLRU case). No hard lockups were observed.These are some softlockups seen with NPS4 mode.
watchdog: BUG: soft lockup - CPU#315 stuck for 11s! [kworker/315:1H:5153] CPU: 315 PID: 5153 Comm: kworker/315:1H Kdump: loaded Not tainted 6.10.0-rc3-enbprftw #12 Workqueue: kblockd blk_mq_run_work_fn RIP: 0010:handle_softirqs+0x70/0x2f0 Call Trace: <IRQ> __irq_exit_rcu+0x68/0x90 irq_exit_rcu+0x12/0x20 sysvec_apic_timer_interrupt+0x85/0xb0 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x1f/0x30 RIP: 0010:iommu_dma_map_page+0xca/0x2c0 dma_map_page_attrs+0x20d/0x2a0 nvme_prep_rq.part.0+0x63d/0x940 [nvme] nvme_queue_rq+0x82/0x210 [nvme] blk_mq_dispatch_rq_list+0x289/0x6d0 __blk_mq_sched_dispatch_requests+0x142/0x5f0 blk_mq_sched_dispatch_requests+0x36/0x70 blk_mq_run_work_fn+0x73/0x90 process_one_work+0x185/0x3d0 worker_thread+0x2ce/0x3e0 kthread+0xe5/0x120 ret_from_fork+0x3d/0x60 ret_from_fork_asm+0x1a/0x30 watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [fio:19820] CPU: 0 PID: 19820 Comm: fio Kdump: loaded Tainted: G L 6.10.0-rc3-enbprftw #12 RIP: 0010:native_queued_spin_lock_slowpath+0x2b8/0x300 Call Trace: <IRQ> </IRQ> <TASK> _raw_spin_lock+0x2d/0x40 clear_shadow_entry+0x3d/0x100 mapping_try_invalidate+0x11b/0x1e0 invalidate_mapping_pages+0x14/0x20 invalidate_bdev+0x40/0x50 blkdev_common_ioctl+0x5f7/0xa90 blkdev_ioctl+0x10d/0x270 __x64_sys_ioctl+0x99/0xd0 x64_sys_call+0x1219/0x20d0 do_syscall_64+0x51/0x120 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x7fc92fc3ec6b </TASK>
Quoted 7 lines
The above one (
clear_shadow_entry) has since been fixed by Yu Zhao and fix is in mm tree.We had seen a couple of scenarios with zone lock contention from page free
and slab free code paths, as reported here: https://lore.kernel.org/linux-mm/[email protected]/Would you have any insights on these?
Have you tried enabling memory interleaving policy for your workload?
Karim
PhD Student
Edinburgh University
Quoted 7 lines · Vlastimil Babka
Quoted 5 lines · Bharata B Rao
The general observation is that the problem usually surfaces when the system free memory goes very low and page cache/buffer consumption hits the ceiling. Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.
[snip mm stuff]
There are numerous avoidable i_lock acquires (including some only showing up under load), but I don't know if they play any role in this particular test.
Collecting all traces would definitely help, locked up or not, for example:
bpftrace -e 'kprobe:queued_spin_lock_slowpath { @[kstack()] = count(); }' -o traces
As for clear_shadow_entry mentioned in the opening mail, the content is:
spin_lock(&mapping->host->i_lock);
xa_lock_irq(&mapping->i_pages);
__clear_shadow_entry(mapping, index, entry);
xa_unlock_irq(&mapping->i_pages);
if (mapping_shrinkable(mapping))
inode_add_lru(mapping->host);
spin_unlock(&mapping->host->i_lock);so for all I know it's all about the xarray thing, not the i_lock per se. --
Mateusz Guzik <mjguzik gmail.com>
Quoted 10 lines · Mateusz Guzik
Quoted 7 lines · Vlastimil Babka
Quoted 5 lines · Bharata B Rao
The general observation is that the problem usually surfaces when the system free memory goes very low and page cache/buffer consumption hits the ceiling. Most of the times the two contended locks are lruvec and
inode->i_lockspinlocks.[snip mm stuff]
Quoted 7 lines
There are numerous avoidable
i_lockacquires (including some only showing up under load), but I don't know if they play any role in this particular test.Collecting all traces would definitely help, locked up or not, for example:
bpftrace -e 'kprobe:queued_spin_lock_slowpath { @[kstack()] =count(); }' -o traces
Here are the top 3 traces collected while the full list from a 30s collection duration when the workload was running, is attached.
@[
native_queued_spin_lock_slowpath+1
__remove_mapping+98
remove_mapping+22
mapping_evict_folio+118
mapping_try_invalidate+214
invalidate_mapping_pages+16
invalidate_bdev+60
blkdev_common_ioctl+1527
blkdev_ioctl+265
__x64_sys_ioctl+149
x64_sys_call+4629
do_syscall_64+126
entry_SYSCALL_64_after_hwframe+118
]: 1787212
@[
native_queued_spin_lock_slowpath+1
folio_wait_bit_common+205
filemap_get_pages+1543
filemap_read+231
blkdev_read_iter+111
aio_read+242
io_submit_one+546
__x64_sys_io_submit+132
x64_sys_call+6617
do_syscall_64+126
entry_SYSCALL_64_after_hwframe+118
]: 7922497
@[
native_queued_spin_lock_slowpath+1
clear_shadow_entry+92
mapping_try_invalidate+337
invalidate_mapping_pages+16
invalidate_bdev+60
blkdev_common_ioctl+1527
blkdev_ioctl+265
__x64_sys_ioctl+149
x64_sys_call+4629
do_syscall_64+126
entry_SYSCALL_64_after_hwframe+118]: 10357614
Quoted 11 lines
As for
clear_shadow_entrymentioned in the opening mail, the content is:spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); __clear_shadow_entry(mapping, index, entry); xa_unlock_irq(&mapping->i_pages); if (mapping_shrinkable(mapping)) inode_add_lru(mapping->host); spin_unlock(&mapping->host->i_lock);so for all I know it's all about the xarray thing, not the
i_lockper se.
The soft lockup signature has _raw_spin_lock and not _raw_spin_lock_irq and hence concluded it to be i_lock. Re-pasting the clear_shadow_entry softlockup here again:
kernel: watchdog: BUG: soft lockup - CPU#29 stuck for 11s! [fio:2701649]
kernel: CPU: 29 PID: 2701649 Comm: fio Tainted: G L
6.10.0-rc3-mglru-irqstrc #24
kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: Call Trace: kernel: <IRQ> kernel: ? show_regs+0x69/0x80 kernel: ? watchdog_timer_fn+0x223/0x2b0 kernel: ? __pfx_watchdog_timer_fn+0x10/0x10 <SNIP> kernel: </IRQ> kernel: <TASK> kernel: ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 kernel: ? native_queued_spin_lock_slowpath+0x2b4/0x300 kernel: _raw_spin_lock+0x38/0x50 kernel: clear_shadow_entry+0x3d/0x100 kernel: ? __pfx_workingset_update_node+0x10/0x10 kernel: mapping_try_invalidate+0x117/0x1d0 kernel: invalidate_mapping_pages+0x10/0x20 kernel: invalidate_bdev+0x3c/0x50 kernel: blkdev_common_ioctl+0x5f7/0xa90 kernel: blkdev_ioctl+0x109/0x270 kernel: x64_sys_call+0x1215/0x20d0 kernel: do_syscall_64+0x7e/0x130
Regards,
Bharata.
Quoted 11 lines · Bharata B Rao
Quoted 9 lines · Mateusz Guzik
As for
clear_shadow_entrymentioned in the opening mail, the content is:spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); __clear_shadow_entry(mapping, index, entry); xa_unlock_irq(&mapping->i_pages); if (mapping_shrinkable(mapping)) inode_add_lru(mapping->host); spin_unlock(&mapping->host->i_lock);
Quoted 4 lines
Quoted 1 line
so for all I know it's all about the xarray thing, not the
i_lockper se.The soft lockup signature has
_raw_spin_lockand not_raw_spin_lock_irqand hence concluded it to bei_lock.
I'm not disputing it was i_lock. I am claiming that the i_pages is taken immediately after and it may be that in your workload this is the thing with the actual contention problem, making i_lock a red herring.
I tried to match up offsets to my own kernel binary, but things went haywire.
Can you please resolve a bunch of symbols, like this:./scripts/faddr2line vmlinux clear_shadow_entry+92
and then paste the source code from reported lines? (I presume you are running with some local patches, so opening relevant files in my repo may still give bogus resutls)
Addresses are: clear_shadow_entry+92 __remove_mapping+98 __filemap_add_folio+332
Most notably in __remove_mapping i_lock is conditional:
if (!folio_test_swapcache(folio))
spin_lock(&mapping->host->i_lock);
xa_lock_irq(&mapping->i_pages);and the disasm of the offset in my case does not match either acquire. For all I know i_lock in this routine is *not* taken and all the queued up __remove_mapping callers increase i_lock -> i_pages wait times in clear_shadow_entry.
To my cursory reading i_lock in clear_shadow_entry can be hacked away with some effort, but should this happen the contention is going to shift to i_pages presumably with more soft lockups (except on that lock). I am not convinced messing with it is justified. From looking at other places the i_lock is not a problem in other spots fwiw.
All that said even if it is i_lock in both cases *and* someone whacks it, the mm folk should look into what happens when (maybe i_lock ->) i_pages lock is held. To that end perhaps you could provide a flamegraph or output of perf record -a -g, I don't know what's preferred.
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
Quoted 24 lines · Mateusz Guzik
Quoted 11 lines · Bharata B Rao
Quoted 9 lines · Mateusz Guzik
As for
clear_shadow_entrymentioned in the opening mail, the content is:spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); __clear_shadow_entry(mapping, index, entry); xa_unlock_irq(&mapping->i_pages); if (mapping_shrinkable(mapping)) inode_add_lru(mapping->host); spin_unlock(&mapping->host->i_lock);Quoted 4 lines
Quoted 1 line
so for all I know it's all about the xarray thing, not the
i_lockper se.The soft lockup signature has
_raw_spin_lockand not_raw_spin_lock_irqand hence concluded it to bei_lock.I'm not disputing it was
i_lock. I am claiming that thei_pagesis taken immediately after and it may be that in your workload this is the thing with the actual contention problem, makingi_locka red herring.I tried to match up offsets to my own kernel binary, but things went haywire.
Quoted 8 lines
Can you please resolve a bunch of symbols, like this:
./scripts/faddr2linevmlinux clear_shadow_entry+92and then paste the source code from reported lines? (I presume you are running with some local patches, so opening relevant files in my repo may still give bogus resutls)
Addresses are: clear_shadow_entry+92 __remove_mapping+98 __filemap_add_folio+332
clear_shadow_entry+92
$ ./scripts/faddr2line vmlinux clear_shadow_entry+92 clear_shadow_entry+92/0x180: spin_lock_irq at include/linux/spinlock.h:376 (inlined by) clear_shadow_entry at mm/truncate.c:51
42 static void clear_shadow_entry(struct address_space *mapping,
43 struct folio_batch *fbatch, pgoff_t *indices)
44 {
45 int i;
46
47 if (shmem_mapping(mapping) || dax_mapping(mapping))
48 return;
49
50 spin_lock(&mapping->host->i_lock);
51 xa_lock_irq(&mapping->i_pages);
__remove_mapping+98
$ ./scripts/faddr2line vmlinux __remove_mapping+98 __remove_mapping+98/0x230: spin_lock_irq at include/linux/spinlock.h:376 (inlined by) __remove_mapping at mm/vmscan.c:695
684 static int __remove_mapping(struct address_space *mapping, struct folio *folio,
685 bool reclaimed, struct mem_cgroup *target_memcg)
686 {
687 int refcount;
688 void *shadow = NULL;
689
690 BUG_ON(!folio_test_locked(folio));
691 BUG_ON(mapping != folio_mapping(folio));
692
693 if (!folio_test_swapcache(folio))
694 spin_lock(&mapping->host->i_lock);
695 xa_lock_irq(&mapping->i_pages);
__filemap_add_folio+332
$ ./scripts/faddr2line vmlinux __filemap_add_folio+332 __filemap_add_folio+332/0x480: spin_lock_irq at include/linux/spinlock.h:377 (inlined by) __filemap_add_folio at mm/filemap.c:878
851 noinline int __filemap_add_folio(struct address_space *mapping,
852 struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp)
853 {
854 XA_STATE(xas, &mapping->i_pages, index);
...874 for (;;) {
875 int order = -1, split_order = 0;
876 void *entry, *old = NULL;
877
878 xas_lock_irq(&xas);
879 xas_for_each_conflict(&xas, entry) {
Quoted 10 lines
Most notably in
__remove_mappingi_lockis conditional:if (!folio_test_swapcache(folio)) spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages);and the disasm of the offset in my case does not match either acquire. For all I know
i_lockin this routine is *not* taken and all the queued up__remove_mappingcallers increasei_lock->i_pageswait times inclear_shadow_entry.
So the first two are on i_pages lock and the last one is xa_lock.
Quoted 7 lines
To my cursory reading
i_lockinclear_shadow_entrycan be hacked away with some effort, but should this happen the contention is going to shift toi_pagespresumably with more soft lockups (except on that lock). I am not convinced messing with it is justified. From looking at other places thei_lockis not a problem in other spots fwiw.
Quoted 5 lines
All that said even if it is
i_lockin both cases *and* someone whacks it, the mm folk should look into what happens when (maybei_lock->)i_pageslock is held. To that end perhaps you could provide a flamegraph or output of perf record -a -g, I don't know what's preferred.
I have attached the flamegraph but this is for the kernel that has been running with all the accumulated fixes so far. The original one (w/o fixes) did show considerable time spent on native_queued_spin_lock_slowpath but unfortunately unable to locate it now.
Regards,
Bharata.
Quoted 104 lines · Bharata B Rao
Quoted 24 lines · Mateusz Guzik
Quoted 11 lines · Bharata B Rao
Quoted 9 lines · Mateusz Guzik
As for
clear_shadow_entrymentioned in the opening mail, the content is:spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); __clear_shadow_entry(mapping, index, entry); xa_unlock_irq(&mapping->i_pages); if (mapping_shrinkable(mapping)) inode_add_lru(mapping->host); spin_unlock(&mapping->host->i_lock);Quoted 4 lines
Quoted 1 line
so for all I know it's all about the xarray thing, not the
i_lockper se.The soft lockup signature has
_raw_spin_lockand not_raw_spin_lock_irqand hence concluded it to bei_lock.I'm not disputing it was
i_lock. I am claiming that thei_pagesis taken immediately after and it may be that in your workload this is the thing with the actual contention problem, makingi_locka red herring.I tried to match up offsets to my own kernel binary, but things went haywire.
Quoted 8 lines
Can you please resolve a bunch of symbols, like this:
./scripts/faddr2linevmlinux clear_shadow_entry+92and then paste the source code from reported lines? (I presume you are running with some local patches, so opening relevant files in my repo may still give bogus resutls)
Addresses are: clear_shadow_entry+92 __remove_mapping+98 __filemap_add_folio+332
clear_shadow_entry+92
$ ./scripts/faddr2line vmlinux clear_shadow_entry+92 clear_shadow_entry+92/0x180: spin_lock_irq at include/linux/spinlock.h:376 (inlined by) clear_shadow_entry at mm/truncate.c:5142 static void clear_shadow_entry(struct
address_space*mapping,
43 structfolio_batch*fbatch,pgoff_t
*indices)
44 {
45 int i;
46
47 if (shmem_mapping(mapping)||dax_mapping(mapping))
48 return;
49
50spin_lock(&mapping->host->i_lock);
51xa_lock_irq(&mapping->i_pages);__remove_mapping+98
$ ./scripts/faddr2line vmlinux __remove_mapping+98 __remove_mapping+98/0x230: spin_lock_irq at include/linux/spinlock.h:376 (inlined by) __remove_mapping at mm/vmscan.c:695684 static int __remove_mapping(struct
address_space*mapping, struct folio *folio,
685 bool reclaimed, structmem_cgroup*target_memcg)
686 {
687 int refcount;
688 void *shadow = NULL;
689
690 BUG_ON(!folio_test_locked(folio));
691BUG_ON(mapping != folio_mapping(folio));
692
693 if (!folio_test_swapcache(folio))
694spin_lock(&mapping->host->i_lock);
695xa_lock_irq(&mapping->i_pages);__filemap_add_folio+332
$ ./scripts/faddr2line vmlinux __filemap_add_folio+332 __filemap_add_folio+332/0x480: spin_lock_irq at include/linux/spinlock.h:377 (inlined by) __filemap_add_folio at mm/filemap.c:878851 noinline int __filemap_add_folio(struct
address_space*mapping,
852 struct folio *folio,pgoff_tindex,gfp_tgfp, void
**shadowp)
853 {854 XA_STATE(xas, &mapping->i_pages, index); ...874 for (;;) {
875 int order = -1,split_order= 0;
876 void *entry, *old = NULL;
877
878xas_lock_irq(&xas);
879xas_for_each_conflict(&xas, entry){Quoted 6 lines
Most notably in
__remove_mappingi_lockis conditional:if (!folio_test_swapcache(folio)) spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages);
Quoted 6 lines
Quoted 4 lines
and the disasm of the offset in my case does not match either acquire. For all I know
i_lockin this routine is *not* taken and all the queued up__remove_mappingcallers increasei_lock->i_pageswait times inclear_shadow_entry.So the first two are on
i_pageslock and the last one isxa_lock.
Isn't xa_lock also i_pages->xa_lock, i.e., the same lock?
Quoted 19 lines
Quoted 11 lines
To my cursory reading
i_lockinclear_shadow_entrycan be hacked away with some effort, but should this happen the contention is going to shift toi_pagespresumably with more soft lockups (except on that lock). I am not convinced messing with it is justified. From looking at other places thei_lockis not a problem in other spots fwiw.All that said even if it is
i_lockin both cases *and* someone whacks it, the mm folk should look into what happens when (maybei_lock->)i_pageslock is held. To that end perhaps you could provide a flamegraph or output of perf record -a -g, I don't know what's preferred.I have attached the flamegraph but this is for the kernel that has been running with all the accumulated fixes so far. The original one (w/o fixes) did show considerable time spent on
native_queued_spin_lock_slowpathbut unfortunately unable to locate it now.Regards,
Bharata.
Quoted 104 lines · Bharata B Rao
Quoted 24 lines · Mateusz Guzik
Quoted 11 lines · Bharata B Rao
Quoted 9 lines · Mateusz Guzik
As for
clear_shadow_entrymentioned in the opening mail, the content is:spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); __clear_shadow_entry(mapping, index, entry); xa_unlock_irq(&mapping->i_pages); if (mapping_shrinkable(mapping)) inode_add_lru(mapping->host); spin_unlock(&mapping->host->i_lock);Quoted 4 lines
Quoted 1 line
so for all I know it's all about the xarray thing, not the
i_lockper se.The soft lockup signature has
_raw_spin_lockand not_raw_spin_lock_irqand hence concluded it to bei_lock.I'm not disputing it was
i_lock. I am claiming that thei_pagesis taken immediately after and it may be that in your workload this is the thing with the actual contention problem, makingi_locka red herring.I tried to match up offsets to my own kernel binary, but things went haywire.
Quoted 8 lines
Can you please resolve a bunch of symbols, like this:
./scripts/faddr2linevmlinux clear_shadow_entry+92and then paste the source code from reported lines? (I presume you are running with some local patches, so opening relevant files in my repo may still give bogus resutls)
Addresses are: clear_shadow_entry+92 __remove_mapping+98 __filemap_add_folio+332
clear_shadow_entry+92
$ ./scripts/faddr2line vmlinux clear_shadow_entry+92 clear_shadow_entry+92/0x180: spin_lock_irq at include/linux/spinlock.h:376 (inlined by) clear_shadow_entry at mm/truncate.c:5142 static void clear_shadow_entry(struct
address_space*mapping,
43 structfolio_batch*fbatch,pgoff_t
*indices)
44 {
45 int i;
46
47 if (shmem_mapping(mapping)||dax_mapping(mapping))
48 return;
49
50spin_lock(&mapping->host->i_lock);
51xa_lock_irq(&mapping->i_pages);__remove_mapping+98
$ ./scripts/faddr2line vmlinux __remove_mapping+98 __remove_mapping+98/0x230: spin_lock_irq at include/linux/spinlock.h:376 (inlined by) __remove_mapping at mm/vmscan.c:695684 static int __remove_mapping(struct
address_space*mapping, struct folio *folio,
685 bool reclaimed, structmem_cgroup*target_memcg)
686 {
687 int refcount;
688 void *shadow = NULL;
689
690 BUG_ON(!folio_test_locked(folio));
691BUG_ON(mapping != folio_mapping(folio));
692
693 if (!folio_test_swapcache(folio))
694spin_lock(&mapping->host->i_lock);
695xa_lock_irq(&mapping->i_pages);__filemap_add_folio+332
$ ./scripts/faddr2line vmlinux __filemap_add_folio+332 __filemap_add_folio+332/0x480: spin_lock_irq at include/linux/spinlock.h:377 (inlined by) __filemap_add_folio at mm/filemap.c:878851 noinline int __filemap_add_folio(struct
address_space*mapping,
852 struct folio *folio,pgoff_tindex,gfp_tgfp, void
**shadowp)
853 {854 XA_STATE(xas, &mapping->i_pages, index); ...874 for (;;) {
875 int order = -1,split_order= 0;
876 void *entry, *old = NULL;
877
878xas_lock_irq(&xas);
879xas_for_each_conflict(&xas, entry){Quoted 6 lines
Most notably in
__remove_mappingi_lockis conditional:if (!folio_test_swapcache(folio)) spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages);
Quoted 7 lines
Quoted 4 lines
and the disasm of the offset in my case does not match either acquire. For all I know
i_lockin this routine is *not* taken and all the queued up__remove_mappingcallers increasei_lock->i_pageswait times inclear_shadow_entry.So the first two are on
i_pageslock and the last one isxa_lock.
bottom line though messing with i_lock removal is not justified afaics
Quoted 13 lines
Quoted 7 lines
To my cursory reading
i_lockinclear_shadow_entrycan be hacked away with some effort, but should this happen the contention is going to shift toi_pagespresumably with more soft lockups (except on that lock). I am not convinced messing with it is justified. From looking at other places thei_lockis not a problem in other spots fwiw.Quoted 5 lines
All that said even if it is
i_lockin both cases *and* someone whacks it, the mm folk should look into what happens when (maybei_lock->)i_pageslock is held. To that end perhaps you could provide a flamegraph or output of perf record -a -g, I don't know what's preferred.
Quoted 5 lines
I have attached the flamegraph but this is for the kernel that has been running with all the accumulated fixes so far. The original one (w/o fixes) did show considerable time spent on
native_queued_spin_lock_slowpathbut unfortunately unable to locate it now.
So I think the problems at this point are all mm, so I'm kicking the ball to that side.
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
Quoted 9 lines · Vlastimil Babka
Seems to me it could be (except that
ZONE_DMAcorner case) a general scalability issue in that you tweak some part of the kernel and the contention moves elsewhere. At least in MM we have per-node locks so this means 256 CPUs per lock? It used to be that there were not that many (cores/threads) per a physical CPU and its NUMA node, so many cpus would mean also more NUMA nodes where the locks contention would distribute among them. I think you could try fakenuma to create these nodes artificially and see if it helps for the MM part. But if the contention moves to e.g. an inode lock, I'm not sure what to do about that then.
AMD EPYC BIOSes have an option called NPS (Nodes Per Socket) that can be set to 1, 2, 4 or 8 and that divides the system up into the chosen number of NUMA nodes.
Karim
PhD Student
Edinburgh University