oom: Don't count on mm-less current process.

On Tue 10-02-15 10:19:34, Johannes Weiner wrote:
[...]

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The code before 9879de7373fc (mm: page_alloc: embed OOM killing naturally into allocation slowpath) was looping on this kind of allocation even though GFP_NOFS didn't trigger OOM killer. This change was not intentional I guess but it makes sense on its own. We shouldn't simply loop in a hope that something happens and we finally make a progress.

Failing __GFP_WAIT allocation is perfectly fine IMO. Why do you think this is a problem?

Btw. this has nothing to do with TIF_MEMDIE and it would be much better to discuss it in a separate thread...

This is really nasty and we have discussed that with Oleg some time ago. We have SIGNAL_GROUP_COREDUMP which prevents the OOM killer from selecting the task. The issue seems to be that OOM killer might inherently race with setting the flag. I have no idea what to do about this, unfortunately.
Oleg?

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Ups, sorry I have missed the follow up emails in this thread. My filters got crazy and the rest got sorted into a different mailbox. Reading the rest now...

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hello. Can we resume TIF_MEMDIE stall discussion?

I'd like to propose

(1) Make several locks killable.

(2) Implement TIF_MEMDIE timeout.

(3) Replace kmalloc() with kmalloc_nofail() and kmalloc_noretry().

for handling TIF_MEMDIE stall problems.

(1) Make several locks killable.

On Linux 3.19, running below command line as an unprivileged user on a system with 4 CPUs / 2GB RAM / no swap can make the system unusable.

  $ for i in `seq 1 100`; do dd if=/dev/zero of=/tmp/file bs=104857600 count=100 & done

---------- An example with ext4 partition ---------- (...snipped...)

[  369.902616] dd              D ffff88007fc12d00     0  9113   6418 0x00000080
[  369.904867]  ffff88007b460890 0000000000012d00 ffff88007b28ffd8 0000000000012d00
[  369.907254]  ffff88007b460890 ffff88007fc12d80 ffff88007a6eb360 0000000000000001
[  369.909855]  ffffffff810946cb 00000000000025f6 ffffffff8108ef1d 0000000000000000
[  369.912054] Call Trace:
[  369.913175]  [<ffffffff810946cb>] ? put_prev_entity+0x5b/0x2c0
[  369.914960]  [<ffffffff8108ef1d>] ? pick_next_entity+0x9d/0x170
[  369.916778]  [<ffffffff8109157e>] ? set_next_entity+0x4e/0x60
[  369.918634]  [<ffffffff81097953>] ? pick_next_task_fair+0x453/0x520
[  369.920530]  [<ffffffff8100c6e0>] ? __switch_to+0x240/0x570
[  369.922263]  [<ffffffff815799f9>] ? schedule_preempt_disabled+0x9/0x10
[  369.924161]  [<ffffffff8157af25>] ? __mutex_lock_slowpath+0xb5/0x120
[  369.926106]  [<ffffffff8157afa6>] ? mutex_lock+0x16/0x25
[  369.927800]  [<ffffffffa01f3acc>] ? ext4_file_write_iter+0x7c/0x3a0 [ext4]
[  369.929778]  [<ffffffff81280fbc>] ? __clear_user+0x1c/0x40
[  369.931491]  [<ffffffff8112c876>] ? iov_iter_zero+0x66/0x2d0
[  369.933235]  [<ffffffff811732a3>] ? new_sync_write+0x83/0xd0
[  369.934977]  [<ffffffff8117397d>] ? vfs_write+0xad/0x1f0
[  369.936703]  [<ffffffff8101b57b>] ? syscall_trace_enter_phase1+0x19b/0x1b0
[  369.938674]  [<ffffffff8117459d>] ? SyS_write+0x4d/0xc0
[  369.940336]  [<ffffffff8157d329>] ? system_call_fastpath+0x12/0x17
(...snipped...)
[  498.421741] SysRq : Manual OOM execution
[  498.423627] kworker/3:3 invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
(...snipped...)
[  498.952807] Out of memory: Kill process 9113 (dd) score 57 or sacrifice child
[  498.954450] Killed process 9113 (dd) total-vm:210340kB, anon-rss:102500kB, file-rss:0kB
(...snipped...)
[  502.068921] SysRq : Manual OOM execution
[  502.070825] kworker/3:3 invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
(...snipped...)
[  502.618222] Out of memory: Kill process 9113 (dd) score 57 or sacrifice child
[  502.620016] Killed process 9113 (dd) total-vm:210340kB, anon-rss:102500kB, file-rss:0kB
(...snipped...)
[  503.900554] SysRq : Manual OOM execution
[  503.902387] kworker/3:3 invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
(...snipped...)
[  504.410444] Out of memory: Kill process 9113 (dd) score 57 or sacrifice child
[  504.412221] Killed process 9113 (dd) total-vm:210340kB, anon-rss:102500kB, file-rss:0kB

(...snipped...)
---------- An example with ext4 partition ----------

---------- An example with xfs partition ---------- (...snipped...)

[  127.135041] Out of memory: Kill process 2505 (dd) score 59 or sacrifice child
[  127.136460] Killed process 2505 (dd) total-vm:210340kB, anon-rss:102464kB, file-rss:1728kB
(...snipped...)
[  243.672302] dd              D ffff88005bd27cb8 12776  2505   2386 0x00100084
[  243.674066]  ffff88005bd27cb8 ffff88005bd27c98 ffff88007850c740 0000000000014080
[  243.676005]  0000000000000000 ffff88005bd27fd8 0000000000014080 ffff88005835d740
[  243.677916]  ffff88007850c740 0000000000000014 ffff8800669bee50 ffff8800669bee54
[  243.679823] Call Trace:
[  243.680478]  [<ffffffff816b2799>] schedule_preempt_disabled+0x29/0x70
[  243.682047]  [<ffffffff816b43d5>] __mutex_lock_slowpath+0x95/0x100
[  243.683548]  [<ffffffff816b83e8>] ? page_fault+0x28/0x30
[  243.684875]  [<ffffffff816b4463>] mutex_lock+0x23/0x37
[  243.686146]  [<ffffffff8129df6c>] xfs_file_buffered_aio_write+0x6c/0x240
[  243.687791]  [<ffffffff813497b5>] ? __clear_user+0x25/0x50
[  243.689121]  [<ffffffff8117294d>] ? iov_iter_zero+0x6d/0x2e0
[  243.690511]  [<ffffffff8129e1b8>] xfs_file_write_iter+0x78/0x110
[  243.691990]  [<ffffffff811beb31>] new_sync_write+0x81/0xb0
[  243.693329]  [<ffffffff811bf2a7>] vfs_write+0xb7/0x1f0
[  243.694581]  [<ffffffff811bfeb6>] SyS_write+0x46/0xb0
[  243.695834]  [<ffffffff81109196>] ? __audit_syscall_exit+0x236/0x2e0
[  243.697376]  [<ffffffff816b64a9>] system_call_fastpath+0x12/0x17
(...snipped...)
[  291.433296] dd              D ffff88005bd27cb8 12776  2505   2386 0x00100084
[  291.433297]  ffff88005bd27cb8 ffff88005bd27c98 ffff88007850c740 0000000000014080
[  291.433298]  0000000000000000 ffff88005bd27fd8 0000000000014080 ffff88005835d740
[  291.433298]  ffff88007850c740 0000000000000014 ffff8800669bee50 ffff8800669bee54
[  291.433299] Call Trace:
[  291.433300]  [<ffffffff816b2799>] schedule_preempt_disabled+0x29/0x70
[  291.433301]  [<ffffffff816b43d5>] __mutex_lock_slowpath+0x95/0x100
[  291.433302]  [<ffffffff816b83e8>] ? page_fault+0x28/0x30
[  291.433303]  [<ffffffff816b4463>] mutex_lock+0x23/0x37
[  291.433304]  [<ffffffff8129df6c>] xfs_file_buffered_aio_write+0x6c/0x240
[  291.433306]  [<ffffffff813497b5>] ? __clear_user+0x25/0x50
[  291.433307]  [<ffffffff8117294d>] ? iov_iter_zero+0x6d/0x2e0
[  291.433308]  [<ffffffff8129e1b8>] xfs_file_write_iter+0x78/0x110
[  291.433309]  [<ffffffff811beb31>] new_sync_write+0x81/0xb0
[  291.433311]  [<ffffffff811bf2a7>] vfs_write+0xb7/0x1f0
[  291.433312]  [<ffffffff811bfeb6>] SyS_write+0x46/0xb0
[  291.433313]  [<ffffffff81109196>] ? __audit_syscall_exit+0x236/0x2e0
[  291.433314]  [<ffffffff816b64a9>] system_call_fastpath+0x12/0x17

(...snipped...)
---------- An example with xfs partition ----------

This is because the OOM killer happily tries to kill a process which is blocked at unkillable mutex_lock(). If locks shown above were killable, we can reduce the possibility of getting stuck.

I didn't check whether it has livelocked or not. But too slow to wait is not acceptable. Oh, why every thread trying to allocate memory has to repeat the loop that might defer somebody who can make progress if CPU time was given? I wish only somebody like kswapd repeats the loop on behalf of all threads waiting at memory allocation slowpath...

(2) Implement TIF_MEMDIE timeout.

While the command line shown above is an artificial stresstest, I'm seeing troubles on real KVM systems where the guests hang entirely with many processes being blocked at jbd2_journal_commit_transaction() or jbd2_journal_get_write_access(). The root cause of guest's stall is not yet identified but is at least independent with TIF_MEMDIE. However, cron jobs which are blocked at those functions after I/O stall begins exhaust all of the system's memory and make the situation worse (e.g. load average exceeded 7000 on a guest with 2 CPUs as of occurrence of the OOM killer livelock).

Unkillable locks in non-critical paths can be replaced with killable locks. But there are critical paths where fail-on-SIGKILL can lead to unwanted results (e.g. filesystem's error action such as remount as r/o or call panic() being taken), there are locks (e.g. rw_semaphore used by mmap_sem) where killable version does not exist, and there are wait_for_completion() calls where killable version does not worth complicating the code.

If TIF_MEMDIE timeout were implemented, we can cope with the OOM killer livelock problem by choosing more OOM victims (for survive strategy) or calling panic() (for debug and reboot strategy).

(3) Replace kmalloc() with kmalloc_nofail() and kmalloc_noretry().

Currently small allocations are implicitly treated like __GFP_NOFAIL unless TIF_MEMDIE is set. But silently changing small allocations like __GFP_NORETRY will cause obscure bugs. If TIF_MEMDIE timeout is implemented, we will no longer worry about unkillable tasks which is retrying forever at memory allocation; instead we kill more OOM victims and satisfy the request. Therefore, we could introduce kmalloc_nofail(size, gfp) which does kmalloc(size, gfp | __GFP_NOFAIL) (i.e. invoke the OOM killer) and kmalloc_noretry(size, gfp) which does kmalloc(size, gfp | __GFP_NORETRY) (i.e. do not invoke the OOM killer), and switch from kmalloc() to either kmalloc_noretry() or kmalloc_nofail(). Those who are doing smaller than PAGE_SIZE bytes allocations would wish to switch from kmalloc() to kmalloc_nofail() and eliminate untested memory allocation failure paths. Those who are well prepared for memory allocation failures would wish to switch from kmalloc() to kmalloc_noretry(). Eventually, kmalloc() which is implicitly treating small allocations like __GFP_NOFAIL and invoking the OOM killer will be abolished.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

How about something like this? This should solve the deadlock problem in the page allocator, but it would also simplify the memcg OOM killer and allow its use by in-kernel faults again.

--

Yes, basic idea would be same with
http://marc.info/?l=linux-mm&m=142002495532320&w=2 .

But Michal and David do not like the timeout approach.
http://marc.info/?l=linux-mm&m=141684783713564&w=2
http://marc.info/?l=linux-mm&m=141686814824684&w=2

Unless they change their opinion in response to the discovery explained at
http://lwn.net/Articles/627419/ , timeout patches will not be accepted.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I'm open to suggestions, but we can't just stick our heads in the sand and pretend that these are just unrelated bugs. They're not. As long as it's legal to enter the allocator with *anything* that can prevent another random task in the system from making progress, we have this deadlock potential. One side has to give up, and it can't be the page allocator because it has to support __GFP_NOFAIL allocations, which are usually exactly the allocations that are buried in hard-to-unwind state that is likely to trip up exiting OOM victims.

The alternative would be lock dependency tracking, but I'm not sure it can be realistically done for production environments.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes I really hate time based solutions for reasons already explained in the referenced links.

Requesting GFP_NOFAIL allocation with locks held is IMHO a bug and should be fixed.
Hopelessly looping in the page allocator without GFP_NOFAIL is too risky as well and we should get rid of this. Why should we still try to loop when previous 1000 attempts failed with OOM killer invocation? Can we simply fail after a configurable number of attempts? This is prone to reveal unchecked allocation failures but those are bugs as well and we shouldn't pretend otherwise.

I am not convinced that GFP_NOFAIL is the biggest problem. Most if OOM livelocks I have seen were either due to GFP_KERNEL treated as GFP_NOFAIL or an incorrect gfp mask (e.g. GFP_FS added where not appropriate). I think we should focus on this part before we start adding heuristics into OOM killer.

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

That's rather naive.

Filesystems do demand paging of metadata within transactions, which means we are guaranteed to be holding locks when doing memory allocation. Indeed, this is what the GFP_NOFS allocation context is supposed to convey - we currently *hold locks* and so reclaim needs to be careful about recursion. I'll also argue that it means the OOM killer cannot kill the process attempting memory allocation for the same reason.

We are also guaranteed to be in a state where memory allocation failure *cannot be tolerated* because failure to complete the modification leaves the filesystem in a "corrupt in memory" state. We don't use GFP_NOFAIL because it's deprecated, but the reality is that we need to ensure memory allocation eventually succeeds because we *cannot go backwards*.

The choice is simple: memory allocation fails, we shut down the filesystem and guarantee that we DOS the entire machine because the filesystems have gone AWOL; or we keep trying memory allocation until it succeeds.

So, memory allocation generally succeeds eventually, so we have these loops around kmalloc(), kmem_cache_alloc() and alloc_page() that ensure allocation succeeds. Those loops also guarantee we get warnings when allocation is repeatedly failing and we might have actually hit a OOM deadlock situation.

Yet the exact situation we need GFP_NOFAIL is the situation that you are calling a bug.

OTOH, why should the memory allocator care what failure policy the callers have?

Having the OOM killer being able to kill the process that triggered it would be a good start. More often than not, that is the process that needs killing, and the oom killer implementation currently cannot do anything about that process. Make the OOM killer only be invoked by kswapd or some other independent kernel thread so that it is independent of the allocation context that needs to invoke it, and have the invoker wait to be told what to do.

That way it can kill the invoking process if that's the one that needs to be killed, and then all "can't kill processes because the invoker holds locks they depend on" go away.

Cheers,

Dave.

Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I am not sure I understand. Do you mean that OOM killer should attempt to select a victim which is doing doing GFP_NOFS allocation or an allocation in general?

Would it be possible to drop the locks and retry the allocations? Is the context which is doing this transaction a killable context?

As pointed in another email this should be done in the page allocator IMO.

It is not about failure policy of the caller. It is about how long the allocator tries before it gives up. A good allocator tries hard but not too much if the caller is able to handle the failure because it is the caller who defines the fallback policy.

Not sure I understand. Do you mean sysctl_oom_kill_allocating_task?

Can you elaborate? AFAICS the process which has triggered the OOM is the easiest to kill victim. It is not blocked on any locks so it just needs to get outside of the kernel.

Again, I am not sure I understand. The OOM killer doesn't block the context which has triggered OOM condition. Allocation is retried after OOM killer invocation and if the current context is the victim the allocation failure is expedited.

Except that killing the messenger is not the best strategy...

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[ cc fsdevel list - watch out for side effect of 9879de7373fc (mm: page_alloc:
embed OOM killing naturally into allocation slowpath) which was merged between 3.19-rc6 and 3.19-rc7 , started from
http://marc.info/?l=linux-mm&m=142348457310066&w=2 ]

Replying in this post picked up from several posts in this thread.

Those who do not want to retry with invoking the OOM killer are using __GFP_WAIT + __GFP_NORETRY allocations.

Those who want to retry with invoking the OOM killer are using __GFP_WAIT allocations.

Those who must retry forever with invoking the OOM killer, no matter how many processes the OOM killer kills, are using __GFP_WAIT + __GFP_NOFAIL allocations.

However, since use of __GFP_NOFAIL is prohibited, I think many of __GFP_WAIT users are expecting that the allocation fails only when "the OOM killer set TIF_MEMDIE flag to the caller but the caller failed to allocate from memory reserves". Also, the implementation before 9879de7373fc (mm: page_alloc: embed OOM killing naturally into allocation slowpath) effectively supported __GFP_WAIT users with such expectation.

But you should not have applied such change without making necessary changes to GFP_NOFS / GFP_NOIO users with such expectation and testing at linux-next.git . Applying such change after 3.19-rc6 is a sucker punch.

It looks wasteful for me that so many threads (greater than number of available CPUs) are sleeping at cond_resched() in shrink_slab() when checking SysRq-t. Imagine 1000 threads sleeping at cond_resched() in shrink_slab() on a machine with only 1 CPU. Each thread gets a chance to try calling reclaim function only when all other threads gave that thread a chance at cond_resched(). Such situation is almost mutually preventing from making progress. I wish the following mechanism.

Prepare a kernel thread (for avoiding being OOM-killed) and let __GFP_WAIT and __GFP_WAIT + __GFP_NOFAIL users to wake up the kernel thread when they failed to allocate from free list. The kernel thread calls shrink_slab() etc. (and also out_of_memory() as needed) and wakes them sleeping at wait_for_event() up.

Failing to allocate from free list is a rare case. Therefore, context switches for asking somebody else for reclaiming memory would be an acceptable overhead. If such mechanism are implemented, 1000 threads except the somebody can save CPU time by sleeping. Avoiding "almost mutually preventing from making progress" situation will drastically shorten the time guarantee even if I beat my machine to death. Such mechanism might be similar to Dave Chinner's

Make the OOM killer only be invoked by kswapd or some other independent kernel thread so that it is independent of the allocation context that needs to invoke it, and have the invoker wait to be told what to do.

suggestion.

I agree with Dave Chinner about this.

I tested on ext4 filesystem, one is stock Linux 3.19 and the other is Linux 3.19 with

- /* The OOM killer does not compensate for light reclaim */
- if (!(gfp_mask & __GFP_FS))
- goto out;

applied. Running a Java-like stressing program (which is multi threaded and likely be chosen by the OOM killer due to huge memory usage) shown below with ext4 filesystem set to remount read-only upon filesystem error.

   # mount -o remount,errors=remount-ro /

---------- Testing program start ----------
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sched.h>

static int file_writer(void *unused)

{
        char buffer[128] = { };
        int fd;
        snprintf(buffer, sizeof(buffer) - 1, "/tmp/file.%u", getpid());
        fd = open(buffer, O_WRONLY | O_CREAT, 0600);
        unlink(buffer);
        while (write(fd, buffer, 1) == 1 && fsync(fd) == 0);
        return 0;
}

static void memory_consumer(void)

{
        const int fd = open("/dev/zero", O_RDONLY);
        unsigned long size;
        char *buf = NULL;
        for (size = 1048576; size < 512UL * (1 << 30); size <<= 1) {
                char *cp = realloc(buf, size);
                if (!cp) {
                        size >>= 1;
                        break;
                }
                buf = cp;
        }
        read(fd, buf, size); /* Will cause OOM due to overcommit */
}

int main(int argc, char *argv[])

{
        int i;
        for (i = 0; i < 100; i++) {
                char *cp = malloc(4 * 1024);
                if (!cp || clone(file_writer, cp + 4 * 1024,
                                 CLONE_SIGHAND | CLONE_VM, NULL) == -1)
                        break;
        }
        memory_consumer();
        while (1)
                pause();
        return 0;
}

---------- Testing program end ----------

The former showed that the ext4 filesystem is remounted read-only due to filesystem errors with 50%+ reproducibility.

----------

[   72.440013] do_get_write_access: OOM for frozen_buffer
[   72.440014] EXT4-fs: ext4_reserve_inode_write:4729: aborting transaction: Out of memory in __ext4_journal_get_write_access
[   72.440015] EXT4-fs error (device sda1) in ext4_reserve_inode_write:4735: Out of memory
(...snipped....)
[   72.495559] do_get_write_access: OOM for frozen_buffer
[   72.495560] EXT4-fs: ext4_reserve_inode_write:4729: aborting transaction: Out of memory in __ext4_journal_get_write_access
[   72.496839] do_get_write_access: OOM for frozen_buffer
[   72.496841] EXT4-fs: ext4_reserve_inode_write:4729: aborting transaction: Out of memory in __ext4_journal_get_write_access
[   72.505766] Aborting journal on device sda1-8.
[   72.505851] EXT4-fs (sda1): Remounting filesystem read-only
[   72.505853] EXT4-fs warning (device sda1): ext4_evict_inode:260: couldn't mark inode dirty (err -12)
[   72.507995] EXT4-fs warning (device sda1): ext4_evict_inode:260: couldn't mark inode dirty (err -12)
[   72.508773] EXT4-fs (sda1): Remounting filesystem read-only
[   72.508775] EXT4-fs warning (device sda1): ext4_evict_inode:260: couldn't mark inode dirty (err -12)
[   72.547799] do_get_write_access: OOM for frozen_buffer
[   72.706692] EXT4-fs warning (device sda1): ext4_evict_inode:260: couldn't mark inode dirty (err -12)
[   73.035416] EXT4-fs warning (device sda1): ext4_evict_inode:260: couldn't mark inode dirty (err -12)
[   73.291732] EXT4-fs warning (device sda1): ext4_evict_inode:260: couldn't mark inode dirty (err -12)
[   73.422171] EXT4-fs warning (device sda1): ext4_evict_inode:260: couldn't mark inode dirty (err -12)
[   73.511862] EXT4-fs warning (device sda1): ext4_evict_inode:260: couldn't mark inode dirty (err -12)
[   73.589174] EXT4-fs warning (device sda1): ext4_evict_inode:260: couldn't mark inode dirty (err -12)
[   73.665302] EXT4-fs warning (device sda1): ext4_evict_inode:260: couldn't mark inode dirty (err -12)

----------

On the other hand, the latter showed that the ext4 filesystem was never remounted read-only because filesystem errors did not occur, though several TIF_MEMDIE stalls which the timeout patch would handle were observed as with the former.

As this is ext4 filesystem, this would use GFP_NOFS. But does using GFP_NOFS + __GFP_NOFAIL at ext4 filesystem solve the problem? I don't think so. The underlying block layer which ext4 filesystem calls would use GFP_NOIO. And memory allocation failures at block layer will result in I/O error which is observed by users as filesystem error. Does passing __GFP_NOFAIL down to block layer solve the problem? I don't think so. There is no means to teach block layer that filesystem layer is doing critical operations where failure results in serious problems. Then, does using GFP_NOIO + __GFP_NOFAIL at block layer solves the problem? I don't think so. It is nothing but bypassing

/* The OOM killer does not compensate for light reclaim */ if (!(gfp_mask & __GFP_FS))

           goto out;

check by passing __GFP_NOFAIL flag.

Killing a user space process or taking filesystem error actions (e.g. remount-ro or kernel panic), which choice is less painful for users? I believe that !(gfp_mask & __GFP_FS) check is a bug and should be removed.

Rather, shouldn't allocations without __GFP_FS get more chance to succeed than allocations with __GFP_FS? If I were the author, I might have added below check instead.

/* This is not a critical allocation. Don't invoke the OOM killer. */ if (gfp_mask & __GFP_FS)

           goto out;

Falling into retry loop with same watermark might prevent rescuer threads from doing memory allocation which is needed for making free memory. Maybe we should use lower watermark for GFP_NOIO and below, middle watermark for GFP_NOFS, high watermark for GFP_KERNEL and above.

IT IS NOT PROHIBITED. It is highly discouraged because GFP_NOFAIL is a strong requirement and the caller should be really aware of the consequences. Especially when the allocation is done under locked context.

This is not what __GFP_WAIT is defined for. It says that the allocator might sleep.

same as GFP_KERNEL == GFP_NOFAIL for small allocations currently which causes a lot of troubles which were not anticipated at the time this was introduced. And we _should_ move away from that model. Because GFP_NOFAIL should be really explicit rather than implicit.

This is a nonsense. OOM was disbaled for !__GFP_FS for ages (since before git era).

Feel free to send patches which are not breaking other loads... [...]

pre-mature OOM killer just because the current allocator context doesn't allow for real reclaim is even worse.

This doesn't make any sense what so ever. So regular GFP_KERNEL|USER allocations wouldn't invoke oom killer. This includes page faults and basically most of allocations.

Michal Hocko
SUSE Labs

Then, at least I expect that filesystem error actions will not be taken so trivially. Can we apply http://marc.info/?l=linux-mm&m=142418465615672&w=2 for Linux 3.19-stable?

I do not understand. What kind of bug would be fixed by that change?

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

That change fixes significant loss of file I/O reliability under extreme memory pressure.

Today I tested how frequent filesystem errors occurs using scripted environment. ( Source code of a.out is http://marc.info/?l=linux-fsdevel&m=142425860904849&w=2 )

----------
#!/bin/sh
: > ~/trial.log
for i in seq 1 100
do

    mkfs.ext4 -q /dev/sdb1 || exit 1
    mount -o errors=remount-ro /dev/sdb1 /tmp || exit 2
    chmod 1777 /tmp
    su - demo -c ~demo/a.out
    if [ -w /tmp/ ]
    then
        echo -n "S" >> ~/trial.log
    else
        echo -n "F" >> ~/trial.log
    fi
    umount /tmp

done
----------

We can see that filesystem errors are occurring frequently if GFP_NOFS / GFP_NOIO allocations give up without retrying. On the other hand, as far as these trials, TIF_MEMDIE stall was not observed if GFP_NOFS / GFP_NOIO allocations give up without retrying. Maybe giving up without retrying is keeping away from hitting stalls for this test case?

Linux 3.19-rc6 (Console log is http://I-love.SAKURA.ne.jp/tmp/serial-20150219-3.19-rc6.txt.xz )

0 filesystem errors out of 100 trials. 2 stalls. SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

Linux 3.19 (Console log is http://I-love.SAKURA.ne.jp/tmp/serial-20150219-3.19.txt.xz )

44 filesystem errors out of 100 trials. 0 stalls. SSFFSSSFSSSFSFFFFSSFSSFSSSSSSFFFSFSFFSSSSSSFFFFSFSSFFFSSSSFSSFFFFFSSSSSFSSFSFSSFSFFFSFFFFFFFSSSSSSSS

Linux 3.19 with http://marc.info/?l=linux-mm&m=142418465615672&w=2 applied. (Console log is http://I-love.SAKURA.ne.jp/tmp/serial-20150219-3.19-patched.txt.xz )

0 filesystem errors out of 100 trials. 2 stalls. SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

If result of Linux 3.19 is what you wanted, we should chime fs developers for immediate action. (But __GFP_NOFAIL discussion between you and Dave is in progress. I don't know whether ext4 and underlying subsystems should start using __GFP_NOFAIL.)

P.S. Just for experimental purpose, Linux 3.19 with below change applied gave better result than retrying GFP_NOFS / GFP_NOIO allocations without invoking the OOM killer. Short-lived small GFP_NOFS / GFP_NOIO allocations can use GFP_ATOMIC instead? How many bytes does blk_rq_map_kern() want?

--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2867,6 +2867,8 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,

           int classzone_idx;

           gfp_mask &= gfp_allowed_mask;

+ if (gfp_mask == GFP_NOFS || gfp_mask == GFP_NOIO)
+ gfp_mask = GFP_ATOMIC;

lockdep_trace_alloc(gfp_mask);

0 filesystem errors out of 100 trials. 0 stalls. SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

[...]

I would suggest reporting this to ext people (in a separate thread please) and see what is the proper fix.

This is expected because those allocations are with locks held and so the chances to release the lock are higher.

[...]

Michal Hocko
SUSE Labs

Unfortunately, timeout based solutions aren't guaranteed to provide anything more helpful. The problem you're referring to is when the oom kill victim is waiting on a mutex and cannot make forward progress even though it has access to memory reserves. Threads that are holding the mutex and allocate in a blockable context will cause the oom killer to defer forever because it sees the presence of a victim waiting to exit.

	TaskA			TaskB
	=====			=====
	mutex_lock(i_mutex)
	allocate memory
	oom kill TaskB
				mutex_lock(i_mutex)

In this scenario, nothing on the system will be able to allocate memory without some type of memory reserve since at least one thread is holding the mutex that the victim needs and is looping forever, unless memory is freed by something else on the system which allows TaskA to allocate and drop the mutex.

In a timeout based solution, this would be detected and another thread would be chosen for oom kill. There's currently no way for the oom killer to select a process that isn't waiting for that same mutex, however. If it does, then the process has been killed needlessly since it cannot make forward progress itself without grabbing the mutex.

Certainly, it would be better to eventually kill something else in the hope that it does not need the mutex and will free some memory which would allow the thread that had originally been deferring forever, TaskA, in the oom killer waiting for the original victim, TaskB, to exit. If that's the solution, then TaskA had been killed unnecessarily itself.

Perhaps we should consider an alternative: allow threads, such as TaskA, that are deferring for a long amount of time to simply allocate with ALLOC_NO_WATERMARKS itself in that scenario in the hope that the allocation succeeding will eventually allow it to drop the mutex. Two
problems: (1) there's no guarantee that the simple allocation is all TaskA needs before it will drop the lock and (2) another thread could immediately grab the same mutex and allocate, in which the same series of events repeats.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

We can see that effectively GFP_NOFAIL allocations with a lock held (e.g. filesystem transaction) exist, can't we?

----------------------------------------
TaskA TaskB TaskC TaskD TaskE
call mutex_lock()
call mutex_lock()
call mutex_lock()
call mutex_lock()
call mutex_lock()
do GFP_NOFAIL allocation
oom kill TaskA
waiting for TaskA to die
will do something with allocated memory
will call mutex_unlock()
will do GFP_NOFAIL allocation
will wait for TaskA to die
will do something with allocated memory
will call mutex_unlock()
will do GFP_NOFAIL allocation
will wait for TaskA to die
will do something with allocated memory will call mutex_unlock()
will do GFP_NOFAIL allocation will wait for TaskA to die will do something with allocated memory will call mutex_unlock()
will do GFP_NOFAIL allocation
----------------------------------------

Allowing ALLOC_NO_WATERMARKS to TaskB helps nothing. We don't want to allow ALLOC_NO_WATERMARKS to TaskC, TaskD, TaskE and TaskA when they do the same sequence TaskB did, or we will deplete memory reserves.

Right. The OOM killer cannot understand that there is such lock dependency.

And do you think there will be a way available for the OOM killer to select a process that isn't waiting for that same mutex in the neare future? (Remembering mutex's address currently waiting for using "struct task_struct" would do, but will not be accepted due to performance penalty. Simplified form would be to check "struct task_struct"->state , but will not be perfect.)

Complaining about unnecessarily killed processes is preventing us from making forward progress.

The memory reserves are something like a balloon. To guarantee forward progress, the balloon must not become empty. All memory managing techniques except the OOM killer are trying to control "deflator of the balloon" via various throttling heuristics. On the other hand, the OOM killer is the only memory managing technique which is trying to control "inflator of the balloon" via several throttling heuristics. The OOM killer is invoked when all memory managing techniques except the OOM killer failed to make forward progress.

Therefore, the OOM killer is responsible for making forward progress for "deflator of the balloon" and is granted the prerogative to send SIGKILL to any process.

Given the fact that the OOM killer cannot understand lock dependency and there are effectively GFP_NOFAIL allocations, it is inevitable that the OOM killer fails to choose one correct process that will make forward progress.

Currently the OOM killer is invoked as one shot mode. This mode helps us to reduce the possibility of depleting the memory reserves and killing processes unnecessarily. But this mode is bothering people with "silently stalling forever" problem when the bullet from the OOM killer missed the target. This mode is also bothering people with "complete system crash" problem when the bullet from SysRq-f missed the target, for they have to use SysRq-i or SysRq-c or SysRq-b which is far more unnecessary kill of processes in order to solve the OOM condition.

My proposal is to allow the OOM killer be invoked as consecutive shots mode. Although consecutive shots mode may increase possibility of killing processes unnecessarily, trying to kill an unkillable process in one shot mode is after all unnecessary kill of processes. The root cause is the same (i.e. the OOM killer cannot understand the dependency). My patch can stop bothering people with "silently stalling forever" / "complete system crash" problems by retrying the oom kill attempt than wait forever.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The mm developers have suggested in the past whether we could solve problems by preallocating memory in advance. Sometimes this is very hard to do because we don't know exactly how much or if we need memory, or in order to do this, we would need to completely restructure the code because the memory allocation is happening deep in the call stack, potentially in some other subsystem.

So I wonder if we can solve the problem by having a subsystem reserving memory in advance of taking the mutexes. We do something like this in ext3/ext4 --- when we allocate a (sub-)transaction handle, we give a worst case estimate of how many blocks we might need to dirty under that handle, and if there isn't enough space in the journal, we block in the start_handle() call while the current transaction is closed, and the transaction handle will be attached to the next transaction.

In the memory allocation scenario, it's a bit more complicated, since the memory might be allocated in a slab that requires a higher-order page allocation, but would it be sufficient if we do something rough where the foreground kernel thread "reserves" a few pages before it starts doing something that requires mutexes. The reservation would be reserved on an accounting basis, and kernel codepath which has reserved pages would get priority over kernel threads running under a task_struct which hsa not reserved pages. If there the system doesn't have enough pages available, then the reservation request would block the process until more memory is available.

This wouldn't necessary help in cases where the memory is required for cleaning dirty pages (although in those cases you really *do* want to let the memory allocation succeed --- so maybe there should be a way to hint to the mm subsystem that a memory allocation should be given higher priority since it might help get the system out of the ham that it is in).

However, for "normal" operations, where blocking a process who was about to execute, say, a read(2) or a open(2) system call early, *before* it takes some mutex, it owuld be good if we could provide a certain amount of admission control when memory pressure is specially high.

Would this be a viable strategy?

Even if this was a hint that wasn't perfect (i.e., it some cases a kernel thread might end up requiring more pages than it had hinted, which would not be considered fatal, although the excess requested pages would be treated the same way as if no reservation was made at all, meaning the memory allocation would be more likely to fail and a GFP_NOFAIL allocation would loop for longer), I would think this could only help us do a better job of "keeping the baloon from getting completely deflated".

Cheers,

- Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This exact discussion is already underway.

My initial proposal:

http://oss.sgi.com/archives/xfs/2015-02/msg00314.html

Why mempools don't work but transaction based reservations will:

http://oss.sgi.com/archives/xfs/2015-02/msg00339.html

Reservation needs to be an accounting mechanisms, not preallocation:

http://oss.sgi.com/archives/xfs/2015-02/msg00456.html
http://oss.sgi.com/archives/xfs/2015-02/msg00457.html
http://oss.sgi.com/archives/xfs/2015-02/msg00458.html

And that's where the discussion currently sits.

Cheers,

Dave.

Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I got two problems (one is stall at io_schedule(), the other is kernel panic due to xfs's assertion failure) using Linux 3.19. I guess those problems are caused by not retrying !GFP_FS allocations under OOM. Will those problems go away by using transaction based reservations? And if yes, are they simple enough to backport to vendor's kernels?

(From http://I-love.SAKURA.ne.jp/tmp/serial-20150225-1.txt.xz )
----------

[ 1225.773411] kworker/3:0H    D ffff88007cadb4f8 11632    27      2 0x00000000
[ 1225.776911]  ffff88007cadb4f8 ffff88007cadb508 ffff88007cac6740 0000000000014080
[ 1225.780670]  ffffffff8101cd19 ffff88007cadbfd8 0000000000014080 ffff88007c28b740
[ 1225.784431]  ffff88007cac6740 ffff88007cadb540 ffff88007f8d4998 ffff88007cadb540
[ 1225.788766] Call Trace:
[ 1225.789988]  [<ffffffff8101cd19>] ? read_tsc+0x9/0x10
[ 1225.792444]  [<ffffffff812acbd9>] ? xfs_iunpin_wait+0x19/0x20
[ 1225.795228]  [<ffffffff816b2590>] io_schedule+0xa0/0x130
[ 1225.797802]  [<ffffffff812a9569>] __xfs_iunpin_wait+0xe9/0x140
[ 1225.800621]  [<ffffffff810af3b0>] ? autoremove_wake_function+0x40/0x40
[ 1225.803770]  [<ffffffff812acbd9>] xfs_iunpin_wait+0x19/0x20
[ 1225.806471]  [<ffffffff812a209c>] xfs_reclaim_inode+0x7c/0x360
[ 1225.809283]  [<ffffffff812a25d7>] xfs_reclaim_inodes_ag+0x257/0x370
[ 1225.812308]  [<ffffffff81340839>] ? radix_tree_gang_lookup_tag+0x89/0xd0
[ 1225.815532]  [<ffffffff8116fe58>] ? list_lru_walk_node+0x148/0x190
[ 1225.817951]  [<ffffffff812a2783>] xfs_reclaim_inodes_nr+0x33/0x40
[ 1225.819373]  [<ffffffff812b3545>] xfs_fs_free_cached_objects+0x15/0x20
[ 1225.820898]  [<ffffffff811c29e9>] super_cache_scan+0x169/0x170
[ 1225.822245]  [<ffffffff8115aed6>] shrink_node_slabs+0x1d6/0x370
[ 1225.823588]  [<ffffffff8115dd2a>] shrink_zone+0x20a/0x240
[ 1225.824830]  [<ffffffff8115e0dc>] do_try_to_free_pages+0x16c/0x460
[ 1225.826230]  [<ffffffff8115e48a>] try_to_free_pages+0xba/0x150
[ 1225.827570]  [<ffffffff81151542>] __alloc_pages_nodemask+0x5b2/0x9d0
[ 1225.829030]  [<ffffffff8119ecbc>] kmem_getpages+0x8c/0x200
[ 1225.830277]  [<ffffffff811a122b>] fallback_alloc+0x17b/0x230
[ 1225.831561]  [<ffffffff811a107b>] ____cache_alloc_node+0x18b/0x1c0
[ 1225.833061]  [<ffffffff811a3b00>] kmem_cache_alloc+0x330/0x5c0
[ 1225.834435]  [<ffffffff8133c9d9>] ? ida_pre_get+0x69/0x100
[ 1225.835719]  [<ffffffff8133c9d9>] ida_pre_get+0x69/0x100
[ 1225.836963]  [<ffffffff8133d312>] ida_simple_get+0x42/0xf0
[ 1225.838248]  [<ffffffff81086211>] create_worker+0x31/0x1c0
[ 1225.839519]  [<ffffffff81087831>] worker_thread+0x3d1/0x4d0
[ 1225.840800]  [<ffffffff81087460>] ? rescuer_thread+0x3a0/0x3a0
[ 1225.842123]  [<ffffffff8108c5e2>] kthread+0xd2/0xf0
[ 1225.843234]  [<ffffffff81010000>] ? perf_trace_xen_mmu_ptep_modify_prot+0x90/0xf0
[ 1225.844978]  [<ffffffff8108c510>] ? kthread_create_on_node+0x180/0x180
[ 1225.846481]  [<ffffffff816b63fc>] ret_from_fork+0x7c/0xb0
[ 1225.847718]  [<ffffffff8108c510>] ? kthread_create_on_node+0x180/0x180
[ 1225.849279] kswapd0         D ffff88007708f998 11552    45      2 0x00000000
[ 1225.850977]  ffff88007708f998 0000000000000000 ffff88007c28b740 0000000000014080
[ 1225.852798]  0000000000000003 ffff88007708ffd8 0000000000014080 ffff880077ff2740
[ 1225.854575]  ffff88007c28b740 0000000000000000 ffff88007948e3a8 ffff88007948e3ac
[ 1225.856358] Call Trace:
[ 1225.856928]  [<ffffffff816b2799>] schedule_preempt_disabled+0x29/0x70
[ 1225.858384]  [<ffffffff816b43d5>] __mutex_lock_slowpath+0x95/0x100
[ 1225.859799]  [<ffffffff816b4463>] mutex_lock+0x23/0x37
[ 1225.860983]  [<ffffffff812a264c>] xfs_reclaim_inodes_ag+0x2cc/0x370
[ 1225.862403]  [<ffffffff8109eb48>] ? __enqueue_entity+0x78/0x80
[ 1225.863742]  [<ffffffff810a5f37>] ? enqueue_entity+0x237/0x8f0
[ 1225.865100]  [<ffffffff81340839>] ? radix_tree_gang_lookup_tag+0x89/0xd0
[ 1225.866659]  [<ffffffff8116fe58>] ? list_lru_walk_node+0x148/0x190
[ 1225.868106]  [<ffffffff812a2783>] xfs_reclaim_inodes_nr+0x33/0x40
[ 1225.869522]  [<ffffffff812b3545>] xfs_fs_free_cached_objects+0x15/0x20
[ 1225.871015]  [<ffffffff811c29e9>] super_cache_scan+0x169/0x170
[ 1225.872338]  [<ffffffff8115aed6>] shrink_node_slabs+0x1d6/0x370
[ 1225.873679]  [<ffffffff8115dd2a>] shrink_zone+0x20a/0x240
[ 1225.874920]  [<ffffffff8115ed2d>] kswapd+0x4fd/0x9c0
[ 1225.876049]  [<ffffffff8115e830>] ? mem_cgroup_shrink_node_zone+0x140/0x140
[ 1225.877654]  [<ffffffff8108c5e2>] kthread+0xd2/0xf0
[ 1225.878762]  [<ffffffff81010000>] ? perf_trace_xen_mmu_ptep_modify_prot+0x90/0xf0
[ 1225.880495]  [<ffffffff8108c510>] ? kthread_create_on_node+0x180/0x180
[ 1225.881996]  [<ffffffff816b63fc>] ret_from_fork+0x7c/0xb0
[ 1225.883336]  [<ffffffff8108c510>] ? kthread_create_on_node+0x180/0x180

----------

(From http://I-love.SAKURA.ne.jp/tmp/serial-20150225-2.txt.xz +
http://I-love.SAKURA.ne.jp/tmp/crash-20150225-2.log.xz )
----------

[  189.586204] Out of memory: Kill process 3701 (a.out) score 834 or sacrifice child
[  189.586205] Killed process 3701 (a.out) total-vm:2167392kB, anon-rss:1465820kB, file-rss:4kB
[  189.586210] Kill process 3702 (a.out) sharing same memory
[  189.586211] Kill process 3714 (a.out) sharing same memory
[  189.586212] Kill process 3748 (a.out) sharing same memory
[  189.586213] Kill process 3755 (a.out) sharing same memory
[  189.593470] XFS: Assertion failed: XFS_FORCED_SHUTDOWN(mp), file: fs/xfs/xfs_inode.c, line: 1701
[  189.593491] ------------[ cut here ]------------
[  189.593492] kernel BUG at fs/xfs/xfs_message.c:106!
[  189.593493] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
[  189.593511] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_mangle ip6table_raw ip6table_filter ip6_tables iptable_mangle iptable_raw iptable_filter ip_tables coretemp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel glue_helper lrw gf128mul ablk_helper cryptd dm_mirror dm_region_hash dm_log microcode dm_mod ppdev parport_pc pcspkr vmw_balloon serio_raw vmw_vmci parport shpchp i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc uinput ata_generic pata_acpi sd_mod ata_piix mptspi libata scsi_transport_spi e1000 mptscsih mptbase floppy
[  189.593512] CPU: 1 PID: 3755 Comm: a.out Not tainted 3.19.0 #42
[  189.593512] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
[  189.593513] task: ffff88007a848740 ti: ffff88005c064000 task.ti: ffff88005c064000
[  189.593517] RIP: 0010:[<ffffffff812af992>]  [<ffffffff812af992>] assfail+0x22/0x30
[  189.593517] RSP: 0000:ffff88005c067af8  EFLAGS: 00010292
[  189.593518] RAX: 0000000000000054 RBX: ffff880079349c00 RCX: 0000000000000050
[  189.593518] RDX: 0000000000005050 RSI: 0000000000000282 RDI: 0000000000000282
[  189.593519] RBP: ffff88005c067af8 R08: 0000000000000282 R09: 0000000000000000
[  189.593519] R10: ffffffff81ec95c8 R11: 656c696166206e6f R12: ffff88005ee92800
[  189.593519] R13: 00000000fffffff4 R14: ffffffff81838140 R15: ffff880064505390
[  189.593520] FS:  00007f62d93e0740(0000) GS:ffff88007f840000(0000) knlGS:0000000000000000
[  189.593521] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  189.593521] CR2: 00007fb901282763 CR3: 0000000077b00000 CR4: 00000000000407e0
[  189.593562] Stack:
[  189.593564]  ffff88005c067b38 ffffffff812ab2d7 ffff880079349e48 ffff88007a6feef0
[  189.593564]  ffff88005c067b38 ffff880079349c00 0000000000000001 ffff880079349db8
[  189.593565]  ffff88005c067b58 ffffffff812acb98 ffff880079349db8 ffff880079349c00
[  189.593565] Call Trace:
[  189.593568]  [<ffffffff812ab2d7>] xfs_inactive_truncate+0x67/0x150
[  189.593569]  [<ffffffff812acb98>] xfs_inactive+0x1c8/0x1f0
[  189.593570]  [<ffffffff812b3216>] xfs_fs_evict_inode+0x86/0xd0
[  189.593572]  [<ffffffff811da0f8>] evict+0xb8/0x190
[  189.593574]  [<ffffffff811daa15>] iput+0xf5/0x180
[  189.593575]  [<ffffffff811d5b58>] __dentry_kill+0x188/0x1f0
[  189.593576]  [<ffffffff811d5c65>] dput+0xa5/0x170
[  189.593577]  [<ffffffff811c0dbd>] __fput+0x16d/0x1e0
[  189.593578]  [<ffffffff811c0e7e>] ____fput+0xe/0x10
[  189.593580]  [<ffffffff8108ac9f>] task_work_run+0xaf/0xf0
[  189.593582]  [<ffffffff81071638>] do_exit+0x2d8/0xbe0
[  189.593583]  [<ffffffff8107a5df>] ? recalc_sigpending+0x1f/0x60
[  189.593584]  [<ffffffff81071fcf>] do_group_exit+0x3f/0xa0
[  189.593585]  [<ffffffff8107d322>] get_signal+0x1d2/0x6f0
[  189.593588]  [<ffffffff810134e8>] do_signal+0x28/0x720
[  189.593589]  [<ffffffff811c1825>] ? __sb_end_write+0x35/0x70
[  189.593591]  [<ffffffff811bf362>] ? vfs_write+0x172/0x1f0
[  189.593592]  [<ffffffff81013c2c>] do_notify_resume+0x4c/0x90
[  189.593594]  [<ffffffff816b6747>] int_signal+0x12/0x17
[  189.593602] Code: 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 f1 41 89 d0 48 c7 c6 48 8b 97 81 48 89 fa 31 c0 48 89 e5 31 ff e8 de fb ff ff <0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 
[  189.593603] RIP  [<ffffffff812af992>] assfail+0x22/0x30
[  189.593604]  RSP <ffff88005c067af8>

----------

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This is a typical "blame the messenger" bug report. XFS is stuck in inode reclaim waiting for log IO completion to occur, along with all the other processes iin xfs_log_force also stuck waiting for the same Io completion.

You need to find where that IO completion that everything is waiting on has got stuck or show that it's not a lost IO and actually an XFS problem. e.g has the IO stack got stuck on a mempool somewhere?

Which is a failure of xfs_trans_reserve(), and through the calling context and parameters can only be from xfs_log_reserve(). That's got a pretty clear cause:

        tic = xlog_ticket_alloc(log, unit_bytes, cnt, client, permanent,
                                KM_SLEEP | KM_MAYFAIL);
        if (!tic)
                return -ENOMEM;

And the reason for the ASSERT is pretty clear: we put it there because we need to know - as developers - what failures (if any) ever come through that path. This is called from evict():

And as such there is no mechanism for actually reporting the error to userspace and in failing here we are about to leak an inode.

When an XFS developer is testing new code, having a failure like that get trapped is immensely useful. However, on production systems, we can just keep going because it's not a fatal error and, even more importantly, the leaked inode will get cleaned up by log recovery next time the filesystem is mounted.

IOWs, when you run CONFIG_XFS_DEBUG=y, you'll often get failures that are valuable to XFS developers but have no runtime effect on production systems.

Cheers,

Dave.

Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I wanted to know whether transaction based reservations can solve these problems. Inside filesystem layer, I guess you can calculate how much memory is needed for your filesystem transaction. But I'm wondering whether we can calculate how much memory is needed inside block layer. If block layer failed to reserve memory, won't file I/O fail under extreme memory pressure? And if __GFP_NOFAIL were used inside block layer, won't the OOM killer deadlock problem arise?

I didn't get a vmcore for this stall. But it seemed to me that

kworker/3:0H is doing

xfs_fs_free_cached_objects()
=> xfs_reclaim_inodes_nr()
=> xfs_reclaim_inodes_ag(mp, SYNC_TRYLOCK | SYNC_WAIT, &nr_to_scan)
=> xfs_reclaim_inode() because mutex_trylock(&pag->pag_ici_reclaim_lock)
was succeessful
=> xfs_iunpin_wait(ip) because xfs_ipincount(ip) was non 0
=> __xfs_iunpin_wait()
=> waiting inside io_schedule() for somebody to unpin

kswapd0 is doing

xfs_fs_free_cached_objects()
=> xfs_reclaim_inodes_nr()

    => xfs_reclaim_inodes_ag(mp, SYNC_TRYLOCK | SYNC_WAIT, &nr_to_scan)
      => not calling xfs_reclaim_inode() because
         mutex_trylock(&pag->pag_ici_reclaim_lock) failed due to kworker/3:0H
      => SYNC_TRYLOCK is dropped for retry loop due to

            if (skipped && (flags & SYNC_WAIT) && *nr_to_scan > 0) {
                    trylock = 0;
                    goto restart;
            }

      => calling mutex_lock(&pag->pag_ici_reclaim_lock) and gets blocked
         due to kworker/3:0H

kworker/3:0H is trying to free memory but somebody needs memory to make forward progress. kswapd0 is also trying to free memory but is blocked by kworker/3:0H already holding the lock. Since kswapd0 cannot make forward progress, somebody can't allocate memory. Finally the system started stalling. Is this decoding correct?

----------

[ 1225.773411] kworker/3:0H    D ffff88007cadb4f8 11632    27      2 0x00000000
[ 1225.776911]  ffff88007cadb4f8 ffff88007cadb508 ffff88007cac6740 0000000000014080
[ 1225.780670]  ffffffff8101cd19 ffff88007cadbfd8 0000000000014080 ffff88007c28b740
[ 1225.784431]  ffff88007cac6740 ffff88007cadb540 ffff88007f8d4998 ffff88007cadb540
[ 1225.788766] Call Trace:
[ 1225.789988]  [<ffffffff8101cd19>] ? read_tsc+0x9/0x10
[ 1225.792444]  [<ffffffff812acbd9>] ? xfs_iunpin_wait+0x19/0x20
[ 1225.795228]  [<ffffffff816b2590>] io_schedule+0xa0/0x130
[ 1225.797802]  [<ffffffff812a9569>] __xfs_iunpin_wait+0xe9/0x140
arch/x86/include/asm/atomic.h:27
fs/xfs/xfs_inode.c:2433
[ 1225.800621]  [<ffffffff810af3b0>] ? autoremove_wake_function+0x40/0x40
[ 1225.803770]  [<ffffffff812acbd9>] xfs_iunpin_wait+0x19/0x20
fs/xfs/xfs_inode.c:2443
[ 1225.806471]  [<ffffffff812a209c>] xfs_reclaim_inode+0x7c/0x360
include/linux/spinlock.h:309
fs/xfs/xfs_inode.h:144
fs/xfs/xfs_icache.c:920
[ 1225.809283]  [<ffffffff812a25d7>] xfs_reclaim_inodes_ag+0x257/0x370
fs/xfs/xfs_icache.c:1105
[ 1225.812308]  [<ffffffff81340839>] ? radix_tree_gang_lookup_tag+0x89/0xd0
[ 1225.815532]  [<ffffffff8116fe58>] ? list_lru_walk_node+0x148/0x190
[ 1225.817951]  [<ffffffff812a2783>] xfs_reclaim_inodes_nr+0x33/0x40
fs/xfs/xfs_icache.c:1166
[ 1225.819373]  [<ffffffff812b3545>] xfs_fs_free_cached_objects+0x15/0x20
[ 1225.820898]  [<ffffffff811c29e9>] super_cache_scan+0x169/0x170
[ 1225.822245]  [<ffffffff8115aed6>] shrink_node_slabs+0x1d6/0x370
[ 1225.823588]  [<ffffffff8115dd2a>] shrink_zone+0x20a/0x240
[ 1225.824830]  [<ffffffff8115e0dc>] do_try_to_free_pages+0x16c/0x460
[ 1225.826230]  [<ffffffff8115e48a>] try_to_free_pages+0xba/0x150
[ 1225.827570]  [<ffffffff81151542>] __alloc_pages_nodemask+0x5b2/0x9d0
[ 1225.829030]  [<ffffffff8119ecbc>] kmem_getpages+0x8c/0x200
[ 1225.830277]  [<ffffffff811a122b>] fallback_alloc+0x17b/0x230
[ 1225.831561]  [<ffffffff811a107b>] ____cache_alloc_node+0x18b/0x1c0
[ 1225.833061]  [<ffffffff811a3b00>] kmem_cache_alloc+0x330/0x5c0
[ 1225.834435]  [<ffffffff8133c9d9>] ? ida_pre_get+0x69/0x100
[ 1225.835719]  [<ffffffff8133c9d9>] ida_pre_get+0x69/0x100
[ 1225.836963]  [<ffffffff8133d312>] ida_simple_get+0x42/0xf0
[ 1225.838248]  [<ffffffff81086211>] create_worker+0x31/0x1c0
[ 1225.839519]  [<ffffffff81087831>] worker_thread+0x3d1/0x4d0
[ 1225.840800]  [<ffffffff81087460>] ? rescuer_thread+0x3a0/0x3a0
[ 1225.842123]  [<ffffffff8108c5e2>] kthread+0xd2/0xf0
[ 1225.843234]  [<ffffffff81010000>] ? perf_trace_xen_mmu_ptep_modify_prot+0x90/0xf0
[ 1225.844978]  [<ffffffff8108c510>] ? kthread_create_on_node+0x180/0x180
[ 1225.846481]  [<ffffffff816b63fc>] ret_from_fork+0x7c/0xb0
[ 1225.847718]  [<ffffffff8108c510>] ? kthread_create_on_node+0x180/0x180
[ 1225.849279] kswapd0         D ffff88007708f998 11552    45      2 0x00000000
[ 1225.850977]  ffff88007708f998 0000000000000000 ffff88007c28b740 0000000000014080
[ 1225.852798]  0000000000000003 ffff88007708ffd8 0000000000014080 ffff880077ff2740
[ 1225.854575]  ffff88007c28b740 0000000000000000 ffff88007948e3a8 ffff88007948e3ac
[ 1225.856358] Call Trace:
[ 1225.856928]  [<ffffffff816b2799>] schedule_preempt_disabled+0x29/0x70
[ 1225.858384]  [<ffffffff816b43d5>] __mutex_lock_slowpath+0x95/0x100
[ 1225.859799]  [<ffffffff816b4463>] mutex_lock+0x23/0x37
arch/x86/include/asm/current.h:14
kernel/locking/mutex.h:22
kernel/locking/mutex.c:103
[ 1225.860983]  [<ffffffff812a264c>] xfs_reclaim_inodes_ag+0x2cc/0x370
fs/xfs/xfs_icache.c:1034
[ 1225.862403]  [<ffffffff8109eb48>] ? __enqueue_entity+0x78/0x80
[ 1225.863742]  [<ffffffff810a5f37>] ? enqueue_entity+0x237/0x8f0
[ 1225.865100]  [<ffffffff81340839>] ? radix_tree_gang_lookup_tag+0x89/0xd0
[ 1225.866659]  [<ffffffff8116fe58>] ? list_lru_walk_node+0x148/0x190
[ 1225.868106]  [<ffffffff812a2783>] xfs_reclaim_inodes_nr+0x33/0x40
fs/xfs/xfs_icache.c:1166
[ 1225.869522]  [<ffffffff812b3545>] xfs_fs_free_cached_objects+0x15/0x20
[ 1225.871015]  [<ffffffff811c29e9>] super_cache_scan+0x169/0x170
[ 1225.872338]  [<ffffffff8115aed6>] shrink_node_slabs+0x1d6/0x370
[ 1225.873679]  [<ffffffff8115dd2a>] shrink_zone+0x20a/0x240
[ 1225.874920]  [<ffffffff8115ed2d>] kswapd+0x4fd/0x9c0
[ 1225.876049]  [<ffffffff8115e830>] ? mem_cgroup_shrink_node_zone+0x140/0x140
[ 1225.877654]  [<ffffffff8108c5e2>] kthread+0xd2/0xf0
[ 1225.878762]  [<ffffffff81010000>] ? perf_trace_xen_mmu_ptep_modify_prot+0x90/0xf0
[ 1225.880495]  [<ffffffff8108c510>] ? kthread_create_on_node+0x180/0x180
[ 1225.881996]  [<ffffffff816b63fc>] ret_from_fork+0x7c/0xb0
[ 1225.883336]  [<ffffffff8108c510>] ? kthread_create_on_node+0x180/0x180

----------

I killed mutex_lock() and memory allocation from shrinker functions in drivers/gpu/drm/ttm/ttm_page_alloc[_dma].c because I observed that kswapd0 was blocked for so long at mutex_lock().

If kswapd0 is blocked forever at e.g. mutex_lock() inside shrinker functions, who else can make forward progress?

Shouldn't we avoid calling functions which could potentially block for unpredictable duration (e.g. unkillable locks and/or completion) from shrinker functions?

Oh, I didn't know this failure is specific to CONFIG_XFS_DEBUG=y ...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes. The per-ag lock is a key throttling point for reclaim when there are many more direct reclaimers than there are allocation groups. System performance drops badly in low memory conditions if we have more than one reclaimer operating on an allocation group at a time as they interfere and contend with each other. Effectively multiple rclaimers within the one AG turn ascending offset order inode writeback into random IO, which is orders of magnitude slower than a single thread can clean and reclaim those same inodes.

Quite simply: if one thread can't make progress due to be stuck waiting for IO, then another hundred threads trying to do the same operations are unlikely to make progress, either.

Thing is, the io layer below XFS that appears to be stuck does GFP_NOIO allocations, and therefore direct reclaim for mempool allocation in the block layer cannot get stuck on GFP_FS level reclaim operations....

Which, to me, is fixing a symptom rather than understanding the root cause of why lower layers are not making progress as they are supposed to.

You can't get into these filesystem shrinkers when you do GFP_NOIO allocations, as the IO path does.

No, because otherwise we can't throttle allocation and reclaim to the rate at which IO can clean dirty objects. i.e. we do this for the same reason we throttle page cache dirtying to the rate at which we can clean dirty pages....

Cheers,

Dave.

Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I'm misunderstanding something. The description for kswapd() function in mm/vmscan.c says "This basically trickles out pages so that we have _some_ free memory available even if there is no other activity that frees anything up".

Forever blocking kswapd0 somewhere inside filesystem shrinker functions is equivalent with removing kswapd() function because it also prevents non filesystem shrinker functions from being called by kswapd0, doesn't it? Then, the description will become "We won't have _some_ free memory available if there is no other activity that frees anything up", won't it?

Does kswapd0 exist only for reducing the delay caused by reclaiming synchronously? Disabling kswapd0 affects nothing about functionality? The system can make forward progress even if nobody can call non filesystem shrinkers, can't it?

If yes, then why do we need to make special handling for excluding kswapd0 at

	while (unlikely(too_many_isolated(zone, file, sc))) {
		congestion_wait(BLK_RW_ASYNC, HZ/10);

		/* We are about to die and free our memory. Return now. */
		if (fatal_signal_pending(current))
			return SWAP_CLUSTER_MAX;
	}

loop inside shrink_inactive_list() ?

I can't understand the difference between "kswapd0 sleeping forever at too_many_isolated() loop inside shrink_inactive_list()" and "kswapd0 sleeping forever at mutex_lock() inside xfs_reclaim_inodes_ag()".

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sure.

Yes, but that's not intentional. Remember, we keep talking about the filesystem not being able to guarantee forwards progress if allocations block forever? Well...

... we've ended up blocking kswapd because it's waiting on a journal commit to complete, and that journal commit is blocked waiting for forwards progress in memory allocation...

Yes, it's another one of those nasty dependencies I keep pointing out that filesystems have, and that can only be solved by guaranteeing we can always make forwards allocation progress from transaction reserve to transaction commit.

The throttling is required to control the unbound parallelism of direct reclaim. If we don't do this, inode cache reclaim causes random inode writeback and thrashes the disks with random IO, causing severe degradation in performance under heavy memory pressure. So we throttle inode reclaim to a single thread per AG so we get nice sequential IO patterns from inode cache reclaim - the difference is that we can reclaim several hundred thousand dirty inodes per second versus a few hundred...

And because memory allocation is bound by reclaim speed, we throttle the direct reclaimers to prevent IO breakdown conditions from occurring and hence keep performance under memory pressure relatively high and mostly predictable.

It's rare that kswapd actually gets stuck like this - I've only ever seen it once, and I've never had anyone running a production system report deadlocks like this...

I don't really care.

The direct reclaim behaviour is a much bigger problem, and the risk of occasionally having problems with kswapd is miniscule in comparison. Sure, you can provoke it, but unless you are intentially doing nasty things to production systems, it will never be a problem that you trip over.

We can't solve every problem with the current memory allcoatin/reclaim design - we've chosen the lesser evil here, and we're going to have to live with it until we get a more robust memory allocation subsystem implementation.

Cheers,

Dave.

Dave Chinner
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

If this is an unexpected deadlock, don't we want below change for xfs_reclaim_inodes_ag() ?

- if (skipped && (flags & SYNC_WAIT) && *nr_to_scan > 0) {
+ if (skipped && (flags & SYNC_WAIT) && *nr_to_scan > 0 && !current_is_kswapd()) {
trylock = 0;
goto restart;
}

I guess we will unlikely see this again, for so far this is observed with only Linux 3.19 which lacks commit cc87317726f8 ("mm: page_alloc: revert inadvertent !__GFP_FS retry behavior change").

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

On Mon 16-02-15 20:23:16, Tetsuo Handa wrote:
[...]

[...]

Well, you are beating your machine to death so you can hardly get any time guarantee. It would be nice to have a better feedback mechanism to know when to back off and fail the allocation attempt which might be blocking OOM victim to pass away. This is extremely tricky because we shouldn't be too eager to fail just because of a sudden memory pressure.

I guess you are talking about direct reclaim and the whole priority loop? Well, this is what I was talking above. Sometimes we really have to go down to low priorities and basically scan the world in order to find something reclaimable. If we bail out too early we might see pre mature allocation failures and which could lead to reduced QoS.

This is the case when the kswapd is _able_ to cope with the memory pressure.

[...]

I think this is a bad approach. GFP_KERNEL != __GFP_NORETRY and we should treat it like that. Killing more victims is a bad solution because it doesn't guarantee any progress (just look at your example of hundreds processes with large RSS hammering the same file - you would have to kill all of them at once).
Besides that any timeout solution is prone to unexpected delays due to reasons which are not related to the allocation latency.

This sounds like a major overkill. We already have gfp flags for that. What would this buy us?

nofail allocations should be discouraged and used only if any other measure would fail.

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

On Sun 21-12-14 09:35:04, Dave Chinner wrote:
[...]

I guess the following would be a first simple step to remove the bug you are mentioning above. It would be simple enough to put into stable as well. What do you think?
---

So I think this patch is definitely going in the right direction, but at least the __GFP_WRITE handling is insane:

(Patch edited to show the resulting code, without the old deleted lines)

Good point!
---

Andrew,
I've noticed you have taken the patch to mm tree already. I have realized I haven't marked it for stable which is worth it IMO because debugging nasty reclaim recursion bugs is definitely a pain and might fix one and even if it doesn't it is rather straightforward and shouldn't break anything. So if nobody has anything against I would mark this for stable 3.16+ AFAICS.

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I already applied it (as commit 45f87de57f8f), so if you think it's stable material - and I agree that it looks that way - you should just email [email protected] about it.

I think it might be a good idea to wait a week or two to make sure it doesn't have any unexpected side effects.

Linus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>