btrfs_extent_map memory consumption results in "Out of memory"
Quoted 67 lines
Hi Filipe,
Quoted 44 lines
Quoted 1 line
I was just wondering about "direct IO writes", so I ran a quick test by fully
removing fio's config option "direct=1" (default value is false).
Quoted 4 lines
Unfortunately, I'm still experiencing the same oom-kill:
[ 4843.936881]oom-
kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allo
Quoted 4 lines
wed=0,global_oom,task_memcg=/,task=fio,pid=649,uid=0
[ 4843.939001] Out of memory: Killed process 649 (fio)total-vm:216868kB, anon-rss:896kB, file-rss:128kB, shmem-rss:2176kB, UID:0 pgtables:100kB
oom_score_a0[ 5306.210082] tmux: server invokedoom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0 ...
Quoted 18 lines
[ 5306.240968] Unreclaimable slab info: [ 5306.241271] Name Used Total [ 5306.242700] btrfs_extent_map 26093KB 26093KBHere's my updated fio config:
[global]
name=fio-rand-write
filename=fio-rand-write
rw=randwrite
bs=4K
numjobs=1time_based
runtime=90000[file1]
size=3G
iodepth=1Quoted 5 lines
"slabtop -s -a" output:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 206080 206080 100% 0.14K 7360 28 29440Kbtrfs_extent_mapI accelerated my testing by running fio test inside a QEMU VM with a limited
amount of RAM (140MB):
Quoted 7 lines
qemu-kvm
-kernel bzImage.v6.6 \
-m 140M \
-drive file=rootfs.btrfs,format=raw,if=none,id=drive0
...Quoted 6 lines
Quoted 1 line
It appears that this issue may not be limited to direct IO writes alone?
In the buffered IO case it's typically much less likely to happen.
The reason why it happens in your test it's because the VM has very little RAM, 140M, which is very unlikely to find in the real world nowadays.I increased the memory to 8GB and ran the test overnight without any OOM errors. Glad memory management mechanism works as expected!
Quoted 4 lines
Pages can only
be released when they are not dirty and not under writeback, and in this case there's no fsync, so the amount of dirty pages (or under writeback) accumulates very quickly.Quoted 8 lines
If pages can not be released, extent maps can not be released either.
If you add "fsync=1" to your fio test, things should change dramatically.
Thanks.
(And btw, try to avoid top posting if possible, as that makes the thread harder to follow.)
Quoted 6 lines
My apologies for the top posting.
Thanks for your help!
--
Abylay Ospan
I did not see if you were using compression mentioned anywhere in the email chain but the extent map issue appears to be compounded significantly by compression. I've run into it under normal loads deleting snapshots on real machines with only 8gb of memory. https://www.spinics.net/lists/linux-btrfs/msg139657.html
Sincerely,
David