Skip to contentSkip to conversation

New: OOMs writing to files from processes with cgroup memory limits

Bug 199297 Andrew Morton 2 participants 2 messages Apr 6, 2018 1 message not in the archive Memory management

lore
Share & tools
lore

Andrew Morton

author maintainer
#1 lore raw
In reply to a message not in the archive

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 74 lines · On Thu, 05 Apr 2018 21:55:26 +0000 [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=199297

            Bug ID: 199297
           Summary: OOMs writing to files from processes with cgroup
                    memory limits
           Product: Memory Management
           Version: 2.5
    Kernel Version: 4.11+
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: Other
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 275113
--> https://bugzilla.kernel.org/attachment.cgi?id=275113&action=edit
script to reproduce issue + kernel config + oom log from dmesg

OVERVIEW:

Processes that have a cgroup memory limit can easily OOM just writing to files. It appears there is no throttling and the process can very quickly exceed the cgroup memory limit. vm.dirty_ratio appears to be applied to global available memory and not available memory for the cgroup, at least in my case.

This issue came to light by using kubernetes and putting memory limits on pods, but is completely reproducible stand-alone.

STEPS TO REPRODUCE:

* create a memory cgroup
* put a memory limit on the cgroup.. say 256M.
* add a shell process to the cgroup
* use dd from that shell to write a bunch of data to a file

See attached simple script that reproduces the issue every time.

dd will end up getting OOMkilled very shortly after starting. OOM logging will show dirty pages above the cgroup limit.

Kernels before 4.11 do not see this behavior. I've tracked the issue to the following commit:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=726d061fbd3658e4bfeffa1b8e82da97de2ca4dd

When I reverse this commit, dd will complete successfully.

I believe there to be a larger issue here, though. I did a bit of debugging. As mentioned above, it doesn't appear there's any throttling. It also doesn't appear that writebacks are fired when dirty pages build up for the cgroup. From what I can tell, vm.dirty* configs would only get applied against the cgroup limit IFF inode_cgwb_enabled(inode) returns true in balance_dirty_pages_ratelimited(). That appears to be returning false for me. I'm using ext4 and CONFIG_CGROUP_WRITEBACK is enabled. The code removed from the above commit seems to be saving things by reclaiming during try_charge(). But from what I can tell, we actually want to throttle in balance_dirty_pages(), instead.. but that's not happening. This code is all foreign to me, but just wanted to dump a bit about what I saw from my debugging.

NOTE: If I set vm.dirty_bytes to a value lower than my cgroup memory limit, I no longer see OOMs... as it appears the process gets throttled correctly.

I'm attaching a script to reproduce the issue, kernel config, and OOM log messages from dmesg for kernel 4.15.0.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.