test-runner: increase RAM size
James Prestwood
PATCH 1/1 test-runner: increase RAM size
This is explained in the comment, but until the issue can be tracked down increase RAM.
--- tools/test-runner | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
tools/test-runner +8 -5
@@ -1816,12 +1816,15 @@ class Main: smp = int(nproc / 2) # - # Increase RAM if valgrind is being used + # TODO: Its unclear exactly what causes it (leak, fragmentation, + # or ?), but after many test runs the kernel starts running out + # of memory. This is sometimes non-fatal, e.g. a python call + # fails, but other times the kernel fully panics. # - if self.args.valgrind: - ram = 512 - else: - ram = 384 + # For now increase the RAM which allows full test runs to + # complete. + # + ram = 1024 print("Using %d cores, %d RAM for VM" % (smp, ram))
Signature · 1 line
2.34.1
Paul Menzel
Dear James,
Am 18.03.22 um 23:36 schrieb James Prestwood:
Quoted 2 lines
This is explained in the comment, but until the issue can be tracked down increase RAM.
Before it was distinguished if run under Valgrind or not. It’s running out of memory regardless of that now?
The comment says, the reason is unclear, but could you document the test setup (QEMU, Linux version, compiler) of the last successful run, and the parameters of the first failure?
Kind regard,
Paul
PS: To be more specific in the commit message, you could use:
test-runner: Increase RAM size to 1 GB
Quoted 29 lines
--- tools/test-runner | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)tools/test-runner +8
-5@@ -1816,12 +1816,15 @@ class Main: smp = int(nproc / 2) # - # Increase RAM if valgrind is being used + # TODO: Its unclear exactly what causes it (leak, fragmentation, + # or ?), but after many test runs the kernel starts running out + # of memory. This is sometimes non-fatal, e.g. a python call + # fails, but other times the kernel fully panics. # - if self.args.valgrind: - ram = 512 - else: - ram = 384 + # For now increase the RAM which allows full test runs to + # complete. + # + ram = 1024 print("Using %d cores, %d RAM for VM" % (smp, ram))
James Prestwood
Hi Paul,
Quoted 10 lines · Paul Menzel
Dear James,
Am 18.03.22 um 23:36 schrieb James Prestwood:
Quoted 2 lines
This is explained in the comment, but until the issue can be tracked down increase RAM.
Before it was distinguished if run under Valgrind or not. It’s running
out of memory regardless of that now?
I think you're referring to the --valgrind option? This runs valgrind on IWD. The issue here is the entire kernel/system is running out of memory which is not a leak from IWD itself but some other test process.
Quoted 5 lines
The comment says, the reason is unclear, but could you document the test
setup (QEMU, Linux version, compiler) of the last successful run, and the parameters of the first failure?
Yeah, I can explain this further in the commit. But basically any full test run exibits this behavior. Usually about 20-25 tests and it OOM's.
I've tried digging into this a few times and come up with nothing. As the tests execute the kernels available memory continues to go down. This memory does get reclaimed for some time and will bounce around at high usage. This does seem somewhat normal as far as I understand how the kernel handles process memory.
In *most* cases its python getting an OOM exception when importing an individual test, i.e. Line 1174:
module = importlib.import_module(os.path.splitext(s)[0])
Othertimes its non-fatal and some random python call (like os.list_dir()) will OOM. And other times the kernel just throws a panic and OOM's.
My guess with the import_module() case is that the test, and all its import dependencies, is quite large to import in this fashion. This is somewhat of a non-standard way of importing modules (rather than 'import <module>' at the top) but its needed for how test-runner operates in a single python interpreter.
Thanks,
James
Quoted 44 lines
Kind regard,
Paul
PS: To be more specific in the commit message, you could use:
test-runner: Increase RAM size to 1 GB
Quoted 33 lines
---
tools/test-runner| 13 ++++++++-----1 file changed, 8 insertions(+), 5 deletions(-)tools/test-runner +0
-0@@ -1816,12 +1816,15 @@ class Main: smp = int(nproc / 2)
#
- # Increase RAM if valgrind is being used + # TODO: Its unclear exactly what causes it (leak, fragmentation,
+ # or ?), but after many test runs the kernel starts running out
+ # of memory. This is sometimes non-fatal, e.g. a python call
+ # fails, but other times the kernel fully panics. #
- if self.args.valgrind:
- ram = 512
- else:
- ram = 384
+ # For now increase the RAM which allows full test runs to
+ # complete.
+ #
+ ram = 1024
print("Using %d cores, %d RAM for VM" % (smp, ram))