Skip to contentSkip to conversation

test-runner: increase RAM size

PATCH In review James Prestwood 2 participants 3 messages Mar 18, 2022 – Mar 20, 2022

lore
Share & tools
lore

James Prestwood

author
#0 lore raw

PATCH 1/1 test-runner: increase RAM size

This is explained in the comment, but until the issue can be tracked down increase RAM.

---
 tools/test-runner | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
tools/test-runner +8 -5
index b0ec87a2..534f08f2 100755
@@ -1816,12 +1816,15 @@ class Main:
 			smp = int(nproc / 2)
 
 		#
-		# Increase RAM if valgrind is being used
+		# TODO: Its unclear exactly what causes it (leak, fragmentation,
+		# or ?), but after many test runs the kernel starts running out
+		# of memory. This is sometimes non-fatal, e.g. a python call
+		# fails, but other times the kernel fully panics.
 		#
-		if self.args.valgrind:
-			ram = 512
-		else:
-			ram = 384
+		# For now increase the RAM which allows full test runs to
+		# complete.
+		#
+		ram = 1024
 
 		print("Using %d cores, %d RAM for VM" % (smp, ram))
 
Signature · 1 line
2.34.1
#1 lore raw

Dear James,

Am 18.03.22 um 23:36 schrieb James Prestwood:

Quoted 2 lines

This is explained in the comment, but until the issue can be tracked down increase RAM.

Before it was distinguished if run under Valgrind or not. It’s running out of memory regardless of that now?

The comment says, the reason is unclear, but could you document the test setup (QEMU, Linux version, compiler) of the last successful run, and the parameters of the first failure?

Kind regard,

Paul

PS: To be more specific in the commit message, you could use:

test-runner: Increase RAM size to 1 GB

Quoted 29 lines
---
  tools/test-runner | 13 ++++++++-----
  1 file changed, 8 insertions(+), 5 deletions(-)
tools/test-runner +8 -5
index b0ec87a2..534f08f2 100755
@@ -1816,12 +1816,15 @@ class Main:
  			smp = int(nproc / 2)
 
  		#
-		# Increase RAM if valgrind is being used
+		# TODO: Its unclear exactly what causes it (leak, fragmentation,
+		# or ?), but after many test runs the kernel starts running out
+		# of memory. This is sometimes non-fatal, e.g. a python call
+		# fails, but other times the kernel fully panics.
  		#
-		if self.args.valgrind:
-			ram = 512
-		else:
-			ram = 384
+		# For now increase the RAM which allows full test runs to
+		# complete.
+		#
+		ram = 1024
 
  		print("Using %d cores, %d RAM for VM" % (smp, ram))
 
author
#2 lore raw

Hi Paul,

Quoted 10 lines · Paul Menzel

Dear James,

Am 18.03.22 um 23:36 schrieb James Prestwood:

Quoted 2 lines

This is explained in the comment, but until the issue can be tracked down increase RAM.

Before it was distinguished if run under Valgrind or not. It’s running
out of memory regardless of that now?

I think you're referring to the --valgrind option? This runs valgrind on IWD. The issue here is the entire kernel/system is running out of memory which is not a leak from IWD itself but some other test process.

Quoted 5 lines

The comment says, the reason is unclear, but could you document the test
setup (QEMU, Linux version, compiler) of the last successful run, and the parameters of the first failure?

Yeah, I can explain this further in the commit. But basically any full test run exibits this behavior. Usually about 20-25 tests and it OOM's.

I've tried digging into this a few times and come up with nothing. As the tests execute the kernels available memory continues to go down. This memory does get reclaimed for some time and will bounce around at high usage. This does seem somewhat normal as far as I understand how the kernel handles process memory.

In *most* cases its python getting an OOM exception when importing an individual test, i.e. Line 1174:

module = importlib.import_module(os.path.splitext(s)[0])

Othertimes its non-fatal and some random python call (like os.list_dir()) will OOM. And other times the kernel just throws a panic and OOM's.

My guess with the import_module() case is that the test, and all its import dependencies, is quite large to import in this fashion. This is somewhat of a non-standard way of importing modules (rather than 'import <module>' at the top) but its needed for how test-runner operates in a single python interpreter.

Thanks,
James

Quoted 44 lines

Kind regard,

Paul

PS: To be more specific in the commit message, you could use:

test-runner: Increase RAM size to 1 GB

Quoted 33 lines

---
  tools/test-runner | 13 ++++++++-----

  1 file changed, 8 insertions(+), 5 deletions(-)
tools/test-runner +0 -0
index b0ec87a2..534f08f2 100755
@@ -1816,12 +1816,15 @@ class Main:
                         smp = int(nproc / 2)


                #
-               # Increase RAM if valgrind is being used +               # TODO: Its unclear exactly what causes it (leak, fragmentation,
+               # or ?), but after many test runs the kernel starts running out
+               # of memory. This is sometimes non-fatal, e.g. a python call
+               # fails, but other times the kernel fully panics. #
-               if self.args.valgrind:
-                       ram = 512
-               else:
-                       ram = 384
+               # For now increase the RAM which allows full test runs to
+               # complete.
+               #
+               ram = 1024

                print("Using %d cores, %d RAM for VM" % (smp, ram))