New: Stutters and high kernel CPU usage from list_lru_count_one when cache fills memory

https://bugzilla.kernel.org/show_bug.cgi?id=200627

Bug ID: 200627
Summary: Stutters and high kernel CPU usage from
list_lru_count_one when cache fills memory

           Product: Memory Management
           Version: 2.5
    Kernel Version: 4.18-rc4, 4.16
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Other
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

I've recently noticed stuttering and general sluggishness, in Xorg, Firefox, and other graphical applications, when the memory becomes completely filled with cache. In htop, the stuttering manifests as all CPU cores at 100% usage, mostly in kernel mode.

Doing a perf top shows that list_lru_count_one causes a lot of overhead:

```
Overhead  Shared Object                              Symbol                     
  18.38%  [kernel]                                   [k] list_lru_count_one     
   4.90%  [kernel]                                   [k] nmi                    
   3.27%  [kernel]                                   [k] read_hpet              
   2.66%  [kernel]                                   [k] super_cache_count      
   1.84%  [kernel]                                   [k] shrink_slab.part.52    
   1.63%  [kernel]                                   [k]
shmem_unused_huge_count                                                         
   1.19%  restic                                     [.] 0x00000000002e696c     
   0.98%  restic                                     [.] 0x00000000002e6a2f     
   0.81%  restic                                     [.] 0x00000000002e699b     
   0.80%  restic                                     [.] 0x00000000002e69b6     
   0.79%  restic                                     [.] 0x00000000002e697d     
   0.74%  .perf-wrapped                              [.] rb_next                
   0.62%  [kernel]                                   [k] _aesni_dec4            
   0.57%  restic                                     [.] 0x00000000002e6a18     
   0.56%  [kernel]                                   [k] aesni_xts_crypt8       
   0.51%  restic                                     [.] 0x000000000005676a     
   0.50%  restic                                     [.] 0x00000000002e69de     
   0.50%  restic                                     [.] 0x00000000002e69f1     
   0.43%  restic                                     [.] 0x00000000002e6a10     
   0.43%  restic                                     [.] 0x00000000002e69c9     
   0.41%  .perf-wrapped                              [.] hpp__sort_overhead     
   0.41%  restic                                     [.] 0x00000000002e6996     
   0.40%  [kernel]                                   [k]
update_blocked_averages                                                         
   0.38%  restic                                     [.] 0x00000000002e6a05     
   0.38%  [kernel]                                   [k] __indirect_thunk_start 
   0.37%  [kernel]                                   [k]
copy_user_enhanced_fast_string                                                  
   0.35%  rclone                                     [.] crypto/md5.block
```

I've seen it hit up to 25% overhead, while normally (when the cache hasn't filled up) it only has ~4% overhead. I believe that this is the cause of the stutter.

I've kludged together a workaround, as running echo 3 > /proc/sys/vm/drop_caches every minute keeps the cache from filling up and the system responsive, but I was wondering if this was a potential issue in the kernel.

More details on my workload:

- Running Docker containers connected via NFS to disk; this computer serves ~20 NFSv4.2 shares, though most of them have fairly light IO.
- Running a restic backup with rclone, which requires significant CPU usage and does a lot of disk-waiting on hard drives. (It doesn't impact responsiveness when the cache isn't full, though.)

System:

- Linux 4.18-rc4, NixOS unstable
- Intel i7-4820k
- 20 GB RAM
- AMD RX 580

Let me know if there are any more details I can provide or any tests I can run.

How recently? Were earlier kernels better behaved?

Quoted 1 line

How recently? Were earlier kernels better behaved?

I've seen this issue both on Linux 4.16.15 (admittedly using the -ck patchset) and on vanilla Linux 4.18-rc4 (which is what I'm currently using).

I'm fairly certain that it did not occur on Linux 4.14.50, which I used previously, but I will boot back into it to double-check and let you know.

Sorry, not sure if the previous message registered on bugzilla due to the pgp signature? Including it below.

Quoted 7 lines · Kevin Liu

Quoted 1 line

How recently? Were earlier kernels better behaved?

I've seen this issue both on Linux 4.16.15 (admittedly using the -ck patchset) and on vanilla Linux 4.18-rc4 (which is what I'm currently using).

I'm fairly certain that it did not occur on Linux 4.14.50, which I used previously, but I will boot back into it to double-check and let you know.

And yes, booted back into Linux 4.14.54, there appears to be no issue -- list_lru_count_one reaches 6% overhead at most:

Overhead Shared Object Symbol

5.91% [kernel] [k] list_lru_count_one

5.13% [kernel] [k] nmi

4.08% [kernel] [k] read_hpet

1.26% zma [.] Zone::CheckAlarms

1.16% [kernel] [k] _raw_spin_lock

1.07% restic [.] 0x00000000002e696c

1.06% .perf-wrapped [.] hpp__sort_overhead