Using FIO on systems with <=512MB RAM
Hello
We use FIO to check the eMMC storage on our embedded boards. This works great for our modules that have >= 1GB RAM but on all lower-memory ones (<= 512MB) the OOM killer always kills FIO.
I can hook up a swap which helps but is just painfully slow.
I played with the settings mem, bs, iodepth and size but cannot achieve a configuration that works on low-memory embedded systems.
Can anyone guide me how I can make this work?
Best Regards,
Philippe
Quoted 7 lines · Philippe Schenker
Hello
We use FIO to check the eMMC storage on our embedded boards. This works great for our modules that have >= 1GB RAM but on all lower-memory ones (<= 512MB) the OOM killer always kills FIO.
I can hook up a swap which helps but is just painfully slow.
Well yes, your system will be trashing...
Quoted 5 lines
I played with the settings mem, bs, iodepth and size but cannot achieve a configuration that works on low-memory embedded systems.
Can anyone guide me how I can make this work?
See if playing with option "--alloc-size=" helps. You can also reduce the maximum number of jobs with "--max-jobs=". Also try using threads instead of (default) processes for jobs (--thread option). And of course make sure that the total amount of IO buffers you need (num jobs * io depth per job * bs) is not too large for your RAM size.
Quoted 4 lines
Best Regards,
Philippe
Signature · 2 lines
Damien Le Moal Western Digital Research
Quoted 18 lines · Damien Le Moal
Quoted 9 lines · Philippe Schenker
Hello
We use FIO to check the eMMC storage on our embedded boards. This works
great for our modules that have >= 1GB RAM but on all lower-memory ones
(<= 512MB) the OOM killer always kills FIO.I can hook up a swap which helps but is just painfully slow.
Well yes, your system will be trashing...
Quoted 5 lines
I played with the settings mem, bs, iodepth and size but cannot achieve
a configuration that works on low-memory embedded systems.
Quoted 8 lines
Quoted 1 line
Can anyone guide me how I can make this work?
See if playing with option "--alloc-size=" helps. You can also reduce the maximum number of jobs with "--max-jobs=". Also try using threads instead of (default) processes for jobs (--thread option). And of course
make sure that the total amount of IO buffers you need (num jobs * io depth per job * bs) is not too large for your RAM size.
Thank you very much for your help on this topic. I tried those parameters back and forth, however it seems to always reserve ~400MB on the biggest chunk (checked in /proc/<pid-of-fio>/smaps).
decreasing the max allowed shared memory in /proc/sys/kernel/shmmax to ~64M does solve our issue and fio works also on 256MB ram.
However this leads to the question if this is not a bug? I mean it seems to reserve this memory despite it's not needing it?
What leads me to this assumption is that I can also set mem=malloc instead of mem=shm and I get still the same error.
We use fio 3.17 is this maybe solved in newer versions?
Best Regards,
Philippe
Quoted 7 lines
Quoted 5 lines
Best Regards,
Philippe
Quoted 39 lines · Philippe Schenker
Quoted 20 lines · Damien Le Moal
Quoted 9 lines · Philippe Schenker
Hello
We use FIO to check the eMMC storage on our embedded boards. This works
great for our modules that have >= 1GB RAM but on all lower-memory ones
(<= 512MB) the OOM killer always kills FIO.I can hook up a swap which helps but is just painfully slow.
Well yes, your system will be trashing...
Quoted 6 lines
I played with the settings mem, bs, iodepth and size but cannot achieve
a configuration that works on low-memory embedded systems.Can anyone guide me how I can make this work?
Quoted 9 lines
See if playing with option "--alloc-size=" helps. You can also reduce
the maximum number of jobs with "--max-jobs=". Also try using threads
instead of (default) processes for jobs (--thread option). And of course
make sure that the total amount of IO buffers you need (num jobs * io
depth per job * bs) is not too large for your RAM size.Thank you very much for your help on this topic. I tried those parameters back and forth, however it seems to always reserve ~400MB on
the biggest chunk (checked in /proc/<pid-of-fio>/smaps).decreasing the max allowed shared memory in
/proc/sys/kernel/shmmaxto ~64M does solve our issue and fio works also on 256MB ram.
Quoted 7 lines
However this leads to the question if this is not a bug? I mean it seems
to reserve this memory despite it's not needing it? What leads me to this assumption is that I can also set mem=malloc instead of mem=shm and I get still the same error.We use fio 3.17 is this maybe solved in newer versions?
I built fio 3.29 in the meantime and tried it with that and oh wonder I have absolutely no issues, so it seems I hit an old bug.
Sorry for the noise and thanks again Damien for your kind help!
Philippe
Quoted 12 lines
Best Regards,
PhilippeQuoted 7 lines
Quoted 5 lines
Best Regards,
Philippe
I'm developing and maintaining web based gui for fio (w3top). And I can't confirm your concern.
On x64 linux, 380Mb of RAM is enough for both w3top systemd service and fio executable.
On ARM32 the w3top+fio pair needs about 300M of ram.
What engines do you use? I'm about libaio, posixaio, io_uring, sync*** and mmap v2.21 ... v3.28 on headless linux
2/21/2022 4:40 PM, Philippe Schenker пишет:
Quoted 16 lines
Hello
We use FIO to check the eMMC storage on our embedded boards. This works great for our modules that have >= 1GB RAM but on all lower-memory ones (<= 512MB) the OOM killer always kills FIO.
I can hook up a swap which helps but is just painfully slow.
I played with the settings mem, bs, iodepth and size but cannot achieve a configuration that works on low-memory embedded systems.
Can anyone guide me how I can make this work?
Best Regards,
Philippe
Hello Volodymyr
It turned out that the fio version we where using was either buggy or the patches our buildsystem (Yocto/Openembedded) was applying did cause that issue.
I updated to the latest fio version in our recipes and I have it running on 256MB RAM systems now.
Best Regards,
Philippe
Quoted 32 lines · Volodymyr Hoida
I'm developing and maintaining web based gui for fio (w3top). And I can't confirm your concern.
On x64 linux, 380Mb of RAM is enough for both w3top systemd service and
fio executable.On ARM32 the w3top+fio pair needs about 300M of ram.
What engines do you use? I'm about libaio, posixaio,
io_uring, sync*** and mmap v2.21 ... v3.28 on headless linux2/21/2022 4:40 PM, Philippe Schenker пишет:
Quoted 19 lines
Hello
We use FIO to check the eMMC storage on our embedded boards. This works
great for our modules that have >= 1GB RAM but on all lower-memory ones
(<= 512MB) the OOM killer always kills FIO.I can hook up a swap which helps but is just painfully slow.
I played with the settings mem, bs, iodepth and size but cannot achieve
a configuration that works on low-memory embedded systems.Can anyone guide me how I can make this work?
Best Regards,
Philippe