Skip to contentSkip to conversation

Using FIO on systems with <=512MB RAM

DISCUSSION Philippe Schenker 3 participants 6 messages Feb 21, 2022 – Mar 7, 2022

lore
Share & tools
lore
#0 lore raw

Hello

We use FIO to check the eMMC storage on our embedded boards. This works great for our modules that have >= 1GB RAM but on all lower-memory ones (<= 512MB) the OOM killer always kills FIO.

I can hook up a swap which helps but is just painfully slow.

I played with the settings mem, bs, iodepth and size but cannot achieve a configuration that works on low-memory embedded systems.

Can anyone guide me how I can make this work?

Best Regards,
Philippe

#1 lore raw
Quoted 7 lines · Philippe Schenker

Hello

We use FIO to check the eMMC storage on our embedded boards. This works great for our modules that have >= 1GB RAM but on all lower-memory ones (<= 512MB) the OOM killer always kills FIO.

I can hook up a swap which helps but is just painfully slow.

Well yes, your system will be trashing...

Quoted 5 lines

I played with the settings mem, bs, iodepth and size but cannot achieve a configuration that works on low-memory embedded systems.

Can anyone guide me how I can make this work?

See if playing with option "--alloc-size=" helps. You can also reduce the maximum number of jobs with "--max-jobs=". Also try using threads instead of (default) processes for jobs (--thread option). And of course make sure that the total amount of IO buffers you need (num jobs * io depth per job * bs) is not too large for your RAM size.

Quoted 4 lines

Best Regards,
Philippe

Signature · 2 lines
Damien Le Moal
Western Digital Research
#2 lore raw
Quoted 18 lines · Damien Le Moal
Quoted 9 lines · Philippe Schenker

Hello

We use FIO to check the eMMC storage on our embedded boards. This works
great for our modules that have >= 1GB RAM but on all lower-memory ones
(<= 512MB) the OOM killer always kills FIO.

I can hook up a swap which helps but is just painfully slow.

Well yes, your system will be trashing...

Quoted 5 lines

I played with the settings mem, bs, iodepth and size but cannot achieve
a configuration that works on low-memory embedded systems.

Quoted 8 lines
Quoted 1 line

Can anyone guide me how I can make this work?

See if playing with option "--alloc-size=" helps. You can also reduce the maximum number of jobs with "--max-jobs=". Also try using threads instead of (default) processes for jobs (--thread option). And of course
make sure that the total amount of IO buffers you need (num jobs * io depth per job * bs) is not too large for your RAM size.

Thank you very much for your help on this topic. I tried those parameters back and forth, however it seems to always reserve ~400MB on the biggest chunk (checked in /proc/<pid-of-fio>/smaps).

decreasing the max allowed shared memory in /proc/sys/kernel/shmmax to ~64M does solve our issue and fio works also on 256MB ram.

However this leads to the question if this is not a bug? I mean it seems to reserve this memory despite it's not needing it?
What leads me to this assumption is that I can also set mem=malloc instead of mem=shm and I get still the same error.

We use fio 3.17 is this maybe solved in newer versions?

Best Regards,
Philippe

Quoted 7 lines
Quoted 5 lines

Best Regards,
Philippe

#3 lore raw
Quoted 39 lines · Philippe Schenker
Quoted 20 lines · Damien Le Moal
Quoted 9 lines · Philippe Schenker

Hello

We use FIO to check the eMMC storage on our embedded boards. This works
great for our modules that have >= 1GB RAM but on all lower-memory ones
(<= 512MB) the OOM killer always kills FIO.

I can hook up a swap which helps but is just painfully slow.

Well yes, your system will be trashing...

Quoted 6 lines

I played with the settings mem, bs, iodepth and size but cannot achieve
a configuration that works on low-memory embedded systems.

Can anyone guide me how I can make this work?

Quoted 9 lines

See if playing with option "--alloc-size=" helps. You can also reduce
the maximum number of jobs with "--max-jobs=". Also try using threads
instead of (default) processes for jobs (--thread option). And of course
make sure that the total amount of IO buffers you need (num jobs * io
depth per job * bs) is not too large for your RAM size.

Thank you very much for your help on this topic. I tried those parameters back and forth, however it seems to always reserve ~400MB on
the biggest chunk (checked in /proc/<pid-of-fio>/smaps).

decreasing the max allowed shared memory in /proc/sys/kernel/shmmax to ~64M does solve our issue and fio works also on 256MB ram.

Quoted 7 lines

However this leads to the question if this is not a bug? I mean it seems
to reserve this memory despite it's not needing it? What leads me to this assumption is that I can also set mem=malloc instead of mem=shm and I get still the same error.

We use fio 3.17 is this maybe solved in newer versions?

I built fio 3.29 in the meantime and tried it with that and oh wonder I have absolutely no issues, so it seems I hit an old bug.

Sorry for the noise and thanks again Damien for your kind help!

Philippe

Quoted 12 lines

Best Regards,
Philippe

Quoted 7 lines
Quoted 5 lines

Best Regards,
Philippe

#4 lore raw
In reply to Philippe Schenker

I'm developing and maintaining web based gui for fio (w3top). And I can't confirm your concern.

On x64 linux, 380Mb of RAM is enough for both w3top systemd service and fio executable.

On ARM32 the w3top+fio pair needs about 300M of ram.

What engines do you use? I'm about libaio, posixaio, io_uring, sync*** and mmap v2.21 ... v3.28 on headless linux

2/21/2022 4:40 PM, Philippe Schenker пишет:

Quoted 16 lines

Hello

We use FIO to check the eMMC storage on our embedded boards. This works great for our modules that have >= 1GB RAM but on all lower-memory ones (<= 512MB) the OOM killer always kills FIO.

I can hook up a swap which helps but is just painfully slow.

I played with the settings mem, bs, iodepth and size but cannot achieve a configuration that works on low-memory embedded systems.

Can anyone guide me how I can make this work?

Best Regards,
Philippe

#5 lore raw

Hello Volodymyr

It turned out that the fio version we where using was either buggy or the patches our buildsystem (Yocto/Openembedded) was applying did cause that issue.
I updated to the latest fio version in our recipes and I have it running on 256MB RAM systems now.

Best Regards,
Philippe

Quoted 32 lines · Volodymyr Hoida

I'm developing and maintaining web based gui for fio (w3top). And I can't confirm your concern.

On x64 linux, 380Mb of RAM is enough for both w3top systemd service and
fio executable.

On ARM32 the w3top+fio pair needs about 300M of ram.

What engines do you use? I'm about libaio, posixaio, io_uring, sync*** and mmap v2.21 ... v3.28 on headless linux

2/21/2022 4:40 PM, Philippe Schenker пишет:

Quoted 19 lines

Hello

We use FIO to check the eMMC storage on our embedded boards. This works
great for our modules that have >= 1GB RAM but on all lower-memory ones
(<= 512MB) the OOM killer always kills FIO.

I can hook up a swap which helps but is just painfully slow.

I played with the settings mem, bs, iodepth and size but cannot achieve
a configuration that works on low-memory embedded systems.

Can anyone guide me how I can make this work?

Best Regards,
Philippe