Skip to contentSkip to conversation

btrfs_extent_map memory consumption results in "Out of memory"

DISCUSSION Ospan, Abylay 2 participants 5 messages Oct 10, 2023 – Oct 12, 2023 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Greetings Btrfs development team!

I would like to express my gratitude for your outstanding work on Btrfs. However, I recently experienced an 'out of memory' issue as described below.

Steps to reproduce:

1. Run FIO test on a btrfs partition with random write on a 300GB file:

cat <<EOF >> rand.fio
[global]
name=fio-rand-write
filename=fio-rand-write
rw=randwrite
bs=4K
direct=1
numjobs=16
time_based
runtime=90000

[file1]
size=300G
ioengine=libaio
iodepth=16
EOF

fio rand.fio

2. Monitor slab consumption with "slabtop -s -a"

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
25820620 23138538 89% 0.14K 922165 28 3688660K btrfs_extent_map

3. Observe oom-killer:

[49689.294138] ip invoked oom-killer: gfp_mask=0xc2cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP|__GFP_NOMEMALLOC), order=3, oom_score_adj=0
...
[49689.294425] Unreclaimable slab info:
[49689.294426] Name                      Used          Total	
[49689.329363] btrfs_extent_map     3207098KB    3375622KB

...

Memory usage by btrfs_extent_map gradually increases until it reaches a critical point, causing the system to run out of memory.

Test environment: Intel CPU, 8GB RAM (To expedite the reproduction of this issue, I also conducted tests within QEMU with a restricted amount of memory). Linux kernel tested: LTS 5.15.133, and mainline 6.6-rc5

Quick review of the 'fs/btrfs/extent_map.c' code reveals no built-in limitations on memory allocation for extents mapping. Are there any known workarounds or alternative solutions to mitigate this issue?

Thank you!

--
Abylay Ospan

#1 lore raw
Quoted 40 lines · Abylay

Greetings Btrfs development team!

I would like to express my gratitude for your outstanding work on Btrfs. However, I recently experienced an 'out of memory' issue as described below.

Steps to reproduce:

1. Run FIO test on a btrfs partition with random write on a 300GB file:

cat <<EOF >> rand.fio
[global]
name=fio-rand-write
filename=fio-rand-write
rw=randwrite
bs=4K
direct=1
numjobs=16
time_based
runtime=90000

[file1]
size=300G
ioengine=libaio
iodepth=16
EOF

fio rand.fio

2. Monitor slab consumption with "slabtop -s -a"

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
25820620 23138538 89% 0.14K 922165 28 3688660K btrfs_extent_map

3. Observe oom-killer:

[49689.294138] ip invoked oom-killer: gfp_mask=0xc2cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP|__GFP_NOMEMALLOC), order=3, oom_score_adj=0
...
[49689.294425] Unreclaimable slab info:
[49689.294426] Name                      Used          Total	
[49689.329363] btrfs_extent_map     3207098KB    3375622KB

...

Quoted 7 lines

Memory usage by btrfs_extent_map gradually increases until it reaches a critical point, causing the system to run out of memory.

Test environment: Intel CPU, 8GB RAM (To expedite the reproduction of this issue, I also conducted tests within QEMU with a restricted amount of memory). Linux kernel tested: LTS 5.15.133, and mainline 6.6-rc5

Quick review of the 'fs/btrfs/extent_map.c' code reveals no built-in limitations on memory allocation for extents mapping. Are there any known workarounds or alternative solutions to mitigate this issue?

No workarounds really.

So once we add an extent map to the inode's rbtree, it will stay there until:

1) The corresponding pages in the file range get released due to memory pressure or whatever reason decided by the MM side. The release happens in the callback struct address_space_operations::release_folio, which is btrfs_release_folio for btrfs. In your case it's direct IO writes... so there are no pages to release, and therefore extent maps don't get released by that mechanism.

2) The inode is evicted - when evicted of course we drop all extent maps and release all memory.
If some application is keeping a file descriptor open for the inode, and writes keep happening (or reads, since they create an extent map for the range read too), then no extent maps are released...
Databases and other software often do that, keep file descriptors open for long periods, while reads and writes are happening by the some process or others.

The other side effect, even if no memory exhaustion happens, is that it slows down writes, reads, and other operations, due to large red black trees of extent maps.

I don't have a ready solution for that, but I had some thinking about this about a year ago or so. The simplest solution would be to not keep extent maps in memory for direct IO writes/reads... but it may slow down in some cases at least.

Soon I may start some work to improve that.

Thanks.

Quoted 7 lines

Thank you!

--
Abylay Ospan

#2 lore raw

Hi Filipe,

Thanks for the info!

I was just wondering about "direct IO writes", so I ran a quick test by fully removing fio's config option "direct=1" (default value is false). Unfortunately, I'm still experiencing the same oom-kill:

[ 4843.936881] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=fio,pid=649,uid=0
[ 4843.939001] Out of memory: Killed process 649 (fio) total-vm:216868kB, anon-rss:896kB, file-rss:128kB, shmem-rss:2176kB, UID:0 pgtables:100kB oom_score_a0
[ 5306.210082] tmux: server invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
...
[ 5306.240968] Unreclaimable slab info:
[ 5306.241271] Name                      Used          Total
[ 5306.242700] btrfs_extent_map       26093KB      26093KB

Here's my updated fio config:
[global]
name=fio-rand-write
filename=fio-rand-write
rw=randwrite
bs=4K
numjobs=1
time_based
runtime=90000

[file1]
size=3G
iodepth=1

"slabtop -s -a" output:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
206080 206080 100% 0.14K 7360 28 29440K btrfs_extent_map

I accelerated my testing by running fio test inside a QEMU VM with a limited amount of RAM (140MB):

qemu-kvm
-kernel bzImage.v6.6 \
-m 140M \
-drive file=rootfs.btrfs,format=raw,if=none,id=drive0
...

It appears that this issue may not be limited to direct IO writes alone?

Thank you!

-----Original Message-----
From: Filipe Manana <[email protected]>
Sent: Tuesday, October 10, 2023 11:48 AM
To: Ospan, Abylay <[email protected]>
Cc: [email protected]
Subject: RE: [EXTERNAL] btrfs_extent_map memory consumption results in "Out of memory"

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 40 lines · Abylay

Greetings Btrfs development team!

I would like to express my gratitude for your outstanding work on Btrfs. However, I recently experienced an 'out of memory' issue as described below.

Steps to reproduce:

1. Run FIO test on a btrfs partition with random write on a 300GB file:

cat <<EOF >> rand.fio
[global]
name=fio-rand-write
filename=fio-rand-write
rw=randwrite
bs=4K
direct=1
numjobs=16
time_based
runtime=90000

[file1]
size=300G
ioengine=libaio
iodepth=16
EOF

fio rand.fio

2. Monitor slab consumption with "slabtop -s -a"

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
25820620 23138538 89% 0.14K 922165 28 3688660K btrfs_extent_map

3. Observe oom-killer:

[49689.294138] ip invoked oom-killer: 
gfp_mask=0xc2cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP|__GFP_NOMEMALLOC), order=3, oom_score_adj=0 ...
[49689.294425] Unreclaimable slab info:
[49689.294426] Name                      Used          Total
[49689.329363] btrfs_extent_map     3207098KB    3375622KB

...

Quoted 7 lines

Memory usage by btrfs_extent_map gradually increases until it reaches a critical point, causing the system to run out of memory.

Test environment: Intel CPU, 8GB RAM (To expedite the reproduction of this issue, I also conducted tests within QEMU with a restricted amount of memory). Linux kernel tested: LTS 5.15.133, and mainline 6.6-rc5

Quick review of the 'fs/btrfs/extent_map.c' code reveals no built-in limitations on memory allocation for extents mapping. Are there any known workarounds or alternative solutions to mitigate this issue?

No workarounds really.

So once we add an extent map to the inode's rbtree, it will stay there until:

1) The corresponding pages in the file range get released due to memory pressure or whatever reason decided by the MM side. The release happens in the callback struct address_space_operations::release_folio, which is btrfs_release_folio for btrfs. In your case it's direct IO writes... so there are no pages to release, and therefore extent maps don't get released by that mechanism.

2) The inode is evicted - when evicted of course we drop all extent maps and release all memory.
If some application is keeping a file descriptor open for the inode, and writes keep happening (or reads, since they create an extent map for the range read too), then no extent maps are released...
Databases and other software often do that, keep file descriptors open for long periods, while reads and writes are happening by the some process or others.

The other side effect, even if no memory exhaustion happens, is that it slows down writes, reads, and other operations, due to large red black trees of extent maps.

I don't have a ready solution for that, but I had some thinking about this about a year ago or so. The simplest solution would be to not keep extent maps in memory for direct IO writes/reads... but it may slow down in some cases at least.

Soon I may start some work to improve that.

Thanks.

Quoted 7 lines

Thank you!

--
Abylay Ospan

#3 lore raw
Quoted 36 lines · Abylay

Hi Filipe,

Thanks for the info!

I was just wondering about "direct IO writes", so I ran a quick test by fully removing fio's config option "direct=1" (default value is false). Unfortunately, I'm still experiencing the same oom-kill:

[ 4843.936881] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=fio,pid=649,uid=0
[ 4843.939001] Out of memory: Killed process 649 (fio) total-vm:216868kB, anon-rss:896kB, file-rss:128kB, shmem-rss:2176kB, UID:0 pgtables:100kB oom_score_a0
[ 5306.210082] tmux: server invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
...
[ 5306.240968] Unreclaimable slab info:
[ 5306.241271] Name                      Used          Total
[ 5306.242700] btrfs_extent_map       26093KB      26093KB

Here's my updated fio config:
[global]
name=fio-rand-write
filename=fio-rand-write
rw=randwrite
bs=4K
numjobs=1
time_based
runtime=90000

[file1]
size=3G
iodepth=1

"slabtop -s -a" output:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
206080 206080 100% 0.14K 7360 28 29440K btrfs_extent_map

I accelerated my testing by running fio test inside a QEMU VM with a limited amount of RAM (140MB):

Quoted 7 lines

qemu-kvm
-kernel bzImage.v6.6 \
-m 140M \
-drive file=rootfs.btrfs,format=raw,if=none,id=drive0
...

It appears that this issue may not be limited to direct IO writes alone?

In the buffered IO case it's typically much less likely to happen.

The reason why it happens in your test it's because the VM has very little RAM, 140M, which is very unlikely
to find in the real world nowadays. Pages can only be released when they are not dirty and not under writeback,
and in this case there's no fsync, so the amount of dirty pages (or under writeback) accumulates very quickly.
If pages can not be released, extent maps can not be released either.

If you add "fsync=1" to your fio test, things should change dramatically.

Thanks.

(And btw, try to avoid top posting if possible, as that makes the thread harder to follow.)

Quoted 95 lines

Thank you!

-----Original Message-----
From: Filipe Manana <[email protected]>
Sent: Tuesday, October 10, 2023 11:48 AM
To: Ospan, Abylay <[email protected]>
Cc: [email protected]
Subject: RE: [EXTERNAL] btrfs_extent_map memory consumption results in "Out of memory"

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 40 lines · Abylay

Greetings Btrfs development team!

I would like to express my gratitude for your outstanding work on Btrfs. However, I recently experienced an 'out of memory' issue as described below.

Steps to reproduce:

1. Run FIO test on a btrfs partition with random write on a 300GB file:

cat <<EOF >> rand.fio
[global]
name=fio-rand-write
filename=fio-rand-write
rw=randwrite
bs=4K
direct=1
numjobs=16
time_based
runtime=90000

[file1]
size=300G
ioengine=libaio
iodepth=16
EOF

fio rand.fio

2. Monitor slab consumption with "slabtop -s -a"

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
25820620 23138538 89% 0.14K 922165 28 3688660K btrfs_extent_map

3. Observe oom-killer:

[49689.294138] ip invoked oom-killer:
gfp_mask=0xc2cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP|__GFP_NOMEMALLOC), order=3, oom_score_adj=0 ...
[49689.294425] Unreclaimable slab info:
[49689.294426] Name                      Used          Total
[49689.329363] btrfs_extent_map     3207098KB    3375622KB

...

Quoted 7 lines

Memory usage by btrfs_extent_map gradually increases until it reaches a critical point, causing the system to run out of memory.

Test environment: Intel CPU, 8GB RAM (To expedite the reproduction of this issue, I also conducted tests within QEMU with a restricted amount of memory). Linux kernel tested: LTS 5.15.133, and mainline 6.6-rc5

Quick review of the 'fs/btrfs/extent_map.c' code reveals no built-in limitations on memory allocation for extents mapping. Are there any known workarounds or alternative solutions to mitigate this issue?

No workarounds really.

So once we add an extent map to the inode's rbtree, it will stay there until:

1) The corresponding pages in the file range get released due to memory pressure or whatever reason decided by the MM side. The release happens in the callback struct address_space_operations::release_folio, which is btrfs_release_folio for btrfs. In your case it's direct IO writes... so there are no pages to release, and therefore extent maps don't get released by that mechanism.

2) The inode is evicted - when evicted of course we drop all extent maps and release all memory.
If some application is keeping a file descriptor open for the inode, and writes keep happening (or reads, since they create an extent map for the range read too), then no extent maps are released...
Databases and other software often do that, keep file descriptors open for long periods, while reads and writes are happening by the some process or others.

The other side effect, even if no memory exhaustion happens, is that it slows down writes, reads, and other operations, due to large red black trees of extent maps.

I don't have a ready solution for that, but I had some thinking about this about a year ago or so. The simplest solution would be to not keep extent maps in memory for direct IO writes/reads... but it may slow down in some cases at least.

Soon I may start some work to improve that.

Thanks.

Quoted 7 lines

Thank you!

--
Abylay Ospan

#4 lore raw

Hi Filipe,

Quoted 44 lines
Quoted 1 line

I was just wondering about "direct IO writes", so I ran a quick test by fully

removing fio's config option "direct=1" (default value is false).

Quoted 4 lines

Unfortunately, I'm still experiencing the same oom-kill:

[ 4843.936881]

oom-

kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allo

Quoted 4 lines

wed=0,global_oom,task_memcg=/,task=fio,pid=649,uid=0

[ 4843.939001] Out of memory: Killed process 649 (fio)

total-vm:216868kB, anon-rss:896kB, file-rss:128kB, shmem-rss:2176kB, UID:0 pgtables:100kB oom_score_a0 [ 5306.210082] tmux: server invoked

oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0 ...

Quoted 18 lines
[ 5306.240968] Unreclaimable slab info:
[ 5306.241271] Name                      Used          Total
[ 5306.242700] btrfs_extent_map       26093KB      26093KB

Here's my updated fio config:
[global]
name=fio-rand-write
filename=fio-rand-write
rw=randwrite
bs=4K
numjobs=1
time_based
runtime=90000

[file1]
size=3G
iodepth=1

Quoted 5 lines

"slabtop -s -a" output:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
206080 206080 100% 0.14K 7360 28 29440K btrfs_extent_map

I accelerated my testing by running fio test inside a QEMU VM with a limited

amount of RAM (140MB):

Quoted 7 lines

qemu-kvm
-kernel bzImage.v6.6 \
-m 140M \
-drive file=rootfs.btrfs,format=raw,if=none,id=drive0
...

Quoted 6 lines
Quoted 1 line

It appears that this issue may not be limited to direct IO writes alone?

In the buffered IO case it's typically much less likely to happen.

The reason why it happens in your test it's because the VM has very little RAM, 140M, which is very unlikely to find in the real world nowadays.

I increased the memory to 8GB and ran the test overnight without any OOM errors. Glad memory management mechanism works as expected!

Quoted 6 lines

Pages can only
be released when they are not dirty and not under writeback, and in this case there's no fsync, so the amount of dirty pages (or under writeback) accumulates very quickly.
If pages can not be released, extent maps can not be released either.

Quoted 6 lines

If you add "fsync=1" to your fio test, things should change dramatically.

Thanks.

(And btw, try to avoid top posting if possible, as that makes the thread harder to follow.)

My apologies for the top posting.

Thanks for your help!

--
Abylay Ospan