Skip to contentSkip to conversation

core-image-sato-sdk: test image with 512M memory

PATCH In review [email protected] 4 participants 10 messages Jul 26, 2019 – Aug 19, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 core-image-sato-sdk: test image with 512M memory

From: Kai Kang <[email protected]>

When run do_testimage for core-image-sato-sdk, it fails to pass test case:

| RESULTS - systemd.SystemdBasicTests.test_systemd_failed: FAILED (0.43s)

It is OOM issue and daemon rpc.statd is killed:

| [ 531.306146] Out of memory: Kill process 193 (rpc.statd) score 200 or sacrifice child

Increase the memory of qemu to 512M to avoid such OOM issue.

---
 meta/recipes-sato/images/core-image-sato-sdk.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-sato/images/core-image-sato-sdk.bb +1 -0
index d7cc52b52b..f7963d018e 100644
@@ -9,3 +9,4 @@ IMAGE_FEATURES += "dev-pkgs tools-sdk \
 
 IMAGE_INSTALL += "kernel-devsrc"
 
+TEST_QEMUPARAMS = "-m 512"
Signature · 1 line
2.20.0
#1 lore raw

I think you need to use QB_MEM here.

Also, maybe we should just bump the 256M default?

Alex

Quoted 38 lines · On Fri, 26 Jul 2019 at 12:24, <[email protected]> wrote:

From: Kai Kang <[email protected]>

When run do_testimage for core-image-sato-sdk, it fails to pass test case:

| RESULTS - systemd.SystemdBasicTests.test_systemd_failed: FAILED (0.43s)

It is OOM issue and daemon rpc.statd is killed:

| [ 531.306146] Out of memory: Kill process 193 (rpc.statd) score 200 or sacrifice child

Increase the memory of qemu to 512M to avoid such OOM issue.

---
 meta/recipes-sato/images/core-image-sato-sdk.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-sato/images/core-image-sato-sdk.bb +1 -1
index d7cc52b52b..f7963d018e 100644
@@ -9,3 +9,4 @@ IMAGE_FEATURES += "dev-pkgs tools-sdk \
 
 IMAGE_INSTALL += "kernel-devsrc"
 
+TEST_QEMUPARAMS = "-m 512"
--

2.20.0

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

author
#2 lore raw
Quoted 1 line · Alexander Kanavin

I think you need to use QB_MEM here.

Used TEST_QEMUPARAMS because I found the following line in testimage.bbclass:

# TEST_QEMUPARAMS can be used to pass extra parameters to qemu, e.g. "-m 1024" for setting the amount o f ram to 1 GB.

And it seems QB_MEM is not used when boot the qemu during testimage.

Quoted 2 lines

Also, maybe we should just bump the 256M default?

I hope so.

Regards,
Kai

Quoted 52 lines

Alex

On Fri, 26 Jul 2019 at 12:24, <[email protected] <mailto:[email protected]>> wrote:

    From: Kai Kang <[email protected]
    <mailto:[email protected]>>

    When run do_testimage for core-image-sato-sdk, it fails to pass test
    case:

    | RESULTS - systemd.SystemdBasicTests.test_systemd_failed: FAILED
    (0.43s)

    It is OOM issue and daemon rpc.statd is killed:

    |  [  531.306146] Out of memory: Kill process 193 (rpc.statd)
    score 200 or sacrifice child

    Increase the memory of qemu to 512M to avoid such OOM issue.

    Signed-off-by: Kai Kang <[email protected]
    <mailto:[email protected]>>
    ---
     meta/recipes-sato/images/core-image-sato-sdk.bb
    <http://core-image-sato-sdk.bb> | 1 +
     1 file changed, 1 insertion(+)

    diff --git a/meta/recipes-sato/images/core-image-sato-sdk.bb
    <http://core-image-sato-sdk.bb>
    b/meta/recipes-sato/images/core-image-sato-sdk.bb
    <http://core-image-sato-sdk.bb>
    index d7cc52b52b..f7963d018e 100644
    --- a/meta/recipes-sato/images/core-image-sato-sdk.bb
    <http://core-image-sato-sdk.bb>
    +++ b/meta/recipes-sato/images/core-image-sato-sdk.bb
    <http://core-image-sato-sdk.bb>
    @@ -9,3 +9,4 @@ IMAGE_FEATURES += "dev-pkgs tools-sdk \

     IMAGE_INSTALL += "kernel-devsrc"

    +TEST_QEMUPARAMS = "-m 512"
    -- 
    2.20.0

    -- 
    _______________________________________________
    Openembedded-core mailing list
    [email protected]
    <mailto:[email protected]>
    http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 1 line
Kai Kang
#3 lore raw

I think you need to re-generate the image first (it goes to qemuboot config file). It is already used in ptest images, so it’s good to be consistent.

Alex

Quoted 67 lines · Kang Kai
Quoted 3 lines · Alexander Kanavin

I think you need to use QB_MEM here.

Used TEST_QEMUPARAMS because I found the following line in testimage.bbclass:

# TEST_QEMUPARAMS can be used to pass extra parameters to qemu, e.g. "-m 1024" for setting the amount o f ram to 1 GB.

And it seems QB_MEM is not used when boot the qemu during testimage.

Quoted 2 lines

Also, maybe we should just bump the 256M default?

I hope so.

Regards,
Kai

Quoted 39 lines

Alex

Quoted 36 lines · On Fri, 26 Jul 2019 at 12:24, <[email protected]> wrote:

From: Kai Kang <[email protected]>

When run do_testimage for core-image-sato-sdk, it fails to pass test case:

| RESULTS - systemd.SystemdBasicTests.test_systemd_failed: FAILED (0.43s)

It is OOM issue and daemon rpc.statd is killed:

| [ 531.306146] Out of memory: Kill process 193 (rpc.statd) score 200 or sacrifice child

Increase the memory of qemu to 512M to avoid such OOM issue.

---
 meta/recipes-sato/images/core-image-sato-sdk.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-sato/images/core-image-sato-sdk.bb +1 -0
index d7cc52b52b..f7963d018e 100644
@@ -9,3 +9,4 @@ IMAGE_FEATURES += "dev-pkgs tools-sdk \
 
 IMAGE_INSTALL += "kernel-devsrc"
 
+TEST_QEMUPARAMS = "-m 512"
Signature · 7 lines
2.20.0

-- 
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 1 line
Kai Kang
author
#4 lore raw
Quoted 1 line · Alexander Kanavin

I think you need to re-generate the image first (it goes to qemuboot config file). It is already used in ptest images, so it’s good to be consistent.

Thank. It works for testimage with setting QB_MEM. For the other 2 images which has been set QB_MEM are ptest images just as you said (core-image-sato-ptest-fast and core-image-sato-sdk-ptest). But for core-image-sato-sdk, it affects runqemu beyond testiamge. I just wonder is that acceptable?

Regards,
Kai

Quoted 81 lines

Alex

Quoted 76 lines · On 27 Jul 2019, at 4.35, Kang Kai <[email protected]…
Quoted 1 line · Alexander Kanavin

I think you need to use QB_MEM here.

Used TEST_QEMUPARAMS because I found the following line in testimage.bbclass:

# TEST_QEMUPARAMS can be used to pass extra parameters to qemu, e.g. "-m 1024" for setting the amount o f ram to 1 GB.

And it seems QB_MEM is not used when boot the qemu during testimage.

Quoted 2 lines

Also, maybe we should just bump the 256M default?

I hope so.

Regards,
Kai

Quoted 52 lines

Alex

On Fri, 26 Jul 2019 at 12:24, <[email protected] <mailto:[email protected]>> wrote:

    From: Kai Kang <[email protected]
    <mailto:[email protected]>>

    When run do_testimage for core-image-sato-sdk, it fails to pass test
    case:

    | RESULTS - systemd.SystemdBasicTests.test_systemd_failed:
    FAILED (0.43s)

    It is OOM issue and daemon rpc.statd is killed:

    |  [  531.306146] Out of memory: Kill process 193 (rpc.statd)
    score 200 or sacrifice child

    Increase the memory of qemu to 512M to avoid such OOM issue.

    Signed-off-by: Kai Kang <[email protected]
    <mailto:[email protected]>>
    ---
     meta/recipes-sato/images/core-image-sato-sdk.bb
    <http://core-image-sato-sdk.bb> | 1 +
     1 file changed, 1 insertion(+)

    diff --git a/meta/recipes-sato/images/core-image-sato-sdk.bb
    <http://core-image-sato-sdk.bb>
    b/meta/recipes-sato/images/core-image-sato-sdk.bb
    <http://core-image-sato-sdk.bb>
    index d7cc52b52b..f7963d018e 100644
    --- a/meta/recipes-sato/images/core-image-sato-sdk.bb
    <http://core-image-sato-sdk.bb>
    +++ b/meta/recipes-sato/images/core-image-sato-sdk.bb
    <http://core-image-sato-sdk.bb>
    @@ -9,3 +9,4 @@ IMAGE_FEATURES += "dev-pkgs tools-sdk \

     IMAGE_INSTALL += "kernel-devsrc"

    +TEST_QEMUPARAMS = "-m 512"
    -- 
    2.20.0

    -- 
    _______________________________________________
    Openembedded-core mailing list
    [email protected]
    <mailto:[email protected]>
    http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 1 line
Kai Kang
Signature · 1 line
Kai Kang
#5 lore raw
In reply to [email protected]
Quoted 26 lines · On Fri, 2019-07-26 at 05:23 -0400, [email protected] wrote:

From: Kai Kang <[email protected]>

When run do_testimage for core-image-sato-sdk, it fails to pass test case:

Quoted 2 lines

RESULTS - systemd.SystemdBasicTests.test_systemd_failed: FAILED (0.43s)

It is OOM issue and daemon rpc.statd is killed:

Quoted 2 lines
 [  531.306146] Out of memory: Kill process 193 (rpc.statd) score

200 or sacrifice child

Increase the memory of qemu to 512M to avoid such OOM issue.

---
 meta/recipes-sato/images/core-image-sato-sdk.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-sato/images/core-image-sato-sdk.bb +0 -0
index d7cc52b52b..f7963d018e 100644

meta/recipes-sato/images/core-image-sato-sdk.bb IMAGE_FEATURES += "dev-pkgs tools-sdk \ · line 12

Quoted 4 lines
meta/recipes-sato/images/core-image-sato-sdk.bb +1 -0
@@ -9,3 +9,4 @@ IMAGE_FEATURES += "dev-pkgs tools-sdk \
 
 IMAGE_INSTALL += "kernel-devsrc"
 
+TEST_QEMUPARAMS = "-m 512"

Any idea what is using so much memory in the image when this happens?

Its rather sad that we can't have NFS+systemd with 256MB memory...

Cheers,

Richard

author
#6 lore raw
Quoted 26 lines · On 2019/7/27 上午5:40, [email protected] wrote:
Quoted 25 lines · On Fri, 2019-07-26 at 05:23 -0400, [email protected] wrote:

From: Kai Kang <[email protected]>

When run do_testimage for core-image-sato-sdk, it fails to pass test case:

Quoted 2 lines

RESULTS - systemd.SystemdBasicTests.test_systemd_failed: FAILED (0.43s)

It is OOM issue and daemon rpc.statd is killed:

Quoted 2 lines
  [  531.306146] Out of memory: Kill process 193 (rpc.statd) score

200 or sacrifice child

Increase the memory of qemu to 512M to avoid such OOM issue.

---
  meta/recipes-sato/images/core-image-sato-sdk.bb | 1 +
  1 file changed, 1 insertion(+)
meta/recipes-sato/images/core-image-sato-sdk.bb +0 -0
index d7cc52b52b..f7963d018e 100644
@@ -9,3 +9,4 @@ IMAGE_FEATURES += "dev-pkgs tools-sdk \
 
Quoted 6 lines

meta/recipes-sato/images/core-image-sato-sdk.bb IMAGE_FEATURES += "dev-pkgs tools-sdk \ · line 12

Quoted 3 lines
meta/recipes-sato/images/core-image-sato-sdk.bb +1 -0
  IMAGE_INSTALL += "kernel-devsrc"
 
+TEST_QEMUPARAMS = "-m 512"

Any idea what is using so much memory in the image when this happens?

Its rather sad that we can't have NFS+systemd with 256MB memory...

It caused by stap test case. I minimized the test cases to

TEST_SUITES = "ping date ssh systemd stap kernelmodule gcc "

which could reproduce the error.

And it PASSes testimage that remove stap test from default TEST_SUITES:

TEST_SUITES_remove = 'stap'

But I can't reproduce the OOM failure to run stap tests manually.

Regards,
Kai

Quoted 6 lines

Cheers,

Richard

Signature · 1 line
Kai Kang
author
#7 lore raw
Quoted 36 lines · Kang Kai
Quoted 29 lines · On 2019/7/27 上午5:40, [email protected] wrote:
Quoted 24 lines · On Fri, 2019-07-26 at 05:23 -0400, [email protected] wrote:

From: Kai Kang <[email protected]>

When run do_testimage for core-image-sato-sdk, it fails to pass test case:

Quoted 2 lines

RESULTS - systemd.SystemdBasicTests.test_systemd_failed: FAILED (0.43s)

It is OOM issue and daemon rpc.statd is killed:

Quoted 2 lines

  [  531.306146] Out of memory: Kill process 193 (rpc.statd) score 200 or sacrifice child

Increase the memory of qemu to 512M to avoid such OOM issue.

---
  meta/recipes-sato/images/core-image-sato-sdk.bb | 1 +

  1 file changed, 1 insertion(+)
meta/recipes-sato/images/core-image-sato-sdk.bb +0 -0
index d7cc52b52b..f7963d018e 100644
@@ -9,3 +9,4 @@ IMAGE_FEATURES += "dev-pkgs tools-sdk \
Quoted 2 lines

    IMAGE_INSTALL += "kernel-devsrc"
  +TEST_QEMUPARAMS = "-m 512"

Any idea what is using so much memory in the image when this happens?

Quoted 1 line

Its rather sad that we can't have NFS+systemd with 256MB memory...

It caused by stap test case. I minimized the test cases to

TEST_SUITES = "ping date ssh systemd stap kernelmodule gcc "

Quoted 7 lines

which could reproduce the error.

And it PASSes testimage that remove stap test from default TEST_SUITES:

TEST_SUITES_remove = 'stap'

But I can't reproduce the OOM failure to run stap tests manually.

Hi Richard,

The root cause of test case stap fails is available memory is too low. Compare systemd with sysvinit, memory status are listed:

systemd:

root@qemux86:~# free -h
              total        used        free      shared buff/cache   available
Mem:          239Mi       120Mi        23Mi       8.0Mi 94Mi       100Mi
Swap:            0B          0B          0B

sysvinit:

root@qemux86:~# free -h
              total        used        free      shared buff/cache   available
Mem:          239Mi        45Mi       111Mi       0.0Ki 82Mi       184Mi
Swap:            0B          0B          0B

With systemd, processes of rpc.statd and rpc.mountd take about less than 80M memories. And I compared with Debian 10, they take similar size of memories.

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM TIME+ COMMAND 464 rpcuser   20   0   56316  51336   2192 S   0.0  20.9   0:00.09 rpc.statd 186 root      20   0   30136  27024   2280 S   0.0  11.0 0:00.02 rpc.mountd

Compare to sysvinit, they take only about 2M and 448K:

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM TIME+ COMMAND 567 rpcuser   20   0    3444   2372   1792 S   0.0   1.0 0:00.00 rpc.statd 677 root      20   0    3700    448      0 S   0.0   0.2 0:00.00 rpc.mountd

I didn't figure out the differences between these 2 ways to start the commands:

ExecStart=/usr/sbin/rpc.statd -F $STATD_OPTS

Vs.

test -x "$NFS_STATD" || NFS_STATD=/usr/sbin/rpc.statd start-stop-daemon --start --exec "$NFS_STATD" --pidfile "$STATD_PID"

Regards,
Kai

Quoted 12 lines

Regards,
Kai

Quoted 6 lines

Cheers,

Richard

Signature · 1 line
Kai Kang
author
#8 lore raw
Quoted 77 lines · Kang Kai
Quoted 36 lines · Kang Kai
Quoted 29 lines · On 2019/7/27 上午5:40, [email protected] wrote:
Quoted 24 lines · On Fri, 2019-07-26 at 05:23 -0400, [email protected] wrote:

From: Kai Kang <[email protected]>

When run do_testimage for core-image-sato-sdk, it fails to pass test case:

Quoted 2 lines

RESULTS - systemd.SystemdBasicTests.test_systemd_failed: FAILED (0.43s)

It is OOM issue and daemon rpc.statd is killed:

Quoted 2 lines

  [  531.306146] Out of memory: Kill process 193 (rpc.statd) score 200 or sacrifice child

Increase the memory of qemu to 512M to avoid such OOM issue.

---
  meta/recipes-sato/images/core-image-sato-sdk.bb | 1 +

  1 file changed, 1 insertion(+)
meta/recipes-sato/images/core-image-sato-sdk.bb +0 -0
index d7cc52b52b..f7963d018e 100644
@@ -9,3 +9,4 @@ IMAGE_FEATURES += "dev-pkgs tools-sdk \
Quoted 2 lines

    IMAGE_INSTALL += "kernel-devsrc"
  +TEST_QEMUPARAMS = "-m 512"

Any idea what is using so much memory in the image when this happens?

Quoted 1 line

Its rather sad that we can't have NFS+systemd with 256MB memory...

It caused by stap test case. I minimized the test cases to

TEST_SUITES = "ping date ssh systemd stap kernelmodule gcc "

Quoted 7 lines

which could reproduce the error.

And it PASSes testimage that remove stap test from default TEST_SUITES:

TEST_SUITES_remove = 'stap'

But I can't reproduce the OOM failure to run stap tests manually.

Hi Richard,

The root cause of test case stap fails is available memory is too low. Compare systemd with sysvinit, memory status are listed:

systemd:

root@qemux86:~# free -h
              total        used        free      shared buff/cache   available
Mem:          239Mi       120Mi        23Mi       8.0Mi 94Mi 100Mi
Swap:            0B          0B          0B

sysvinit:

root@qemux86:~# free -h
              total        used        free      shared buff/cache   available
Mem:          239Mi        45Mi       111Mi       0.0Ki 82Mi 184Mi
Swap:            0B          0B          0B

With systemd, processes of rpc.statd and rpc.mountd take about less than 80M memories. And I compared with Debian 10, they take similar size of memories.

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM TIME+ COMMAND 464 rpcuser   20   0   56316  51336   2192 S   0.0  20.9   0:00.09 rpc.statd 186 root      20   0   30136  27024   2280 S   0.0  11.0 0:00.02 rpc.mountd

Compare to sysvinit, they take only about 2M and 448K:

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM TIME+ COMMAND 567 rpcuser   20   0    3444   2372   1792 S   0.0   1.0 0:00.00 rpc.statd 677 root      20   0    3700    448      0 S   0.0   0.2 0:00.00 rpc.mountd

Quoted 8 lines

I didn't figure out the differences between these 2 ways to start the commands:

ExecStart=/usr/sbin/rpc.statd -F $STATD_OPTS

Vs.

test -x "$NFS_STATD" || NFS_STATD=/usr/sbin/rpc.statd start-stop-daemon --start --exec "$NFS_STATD" --pidfile "$STATD_PID"

Hi Richard,

Any more comment here please?

Regards,
Kai

Quoted 21 lines

Regards,
Kai

Quoted 12 lines

Regards,
Kai

Quoted 6 lines

Cheers,

Richard

Signature · 1 line
Kai Kang
#9 lore raw
Quoted 92 lines · Kang Kai
Quoted 82 lines · Kang Kai
Quoted 36 lines · Kang Kai
Quoted 29 lines · On 2019/7/27 上午5:40, [email protected] wrote:
Quoted 24 lines · On Fri, 2019-07-26 at 05:23 -0400, [email protected] wrote:

From: Kai Kang <[email protected]>

When run do_testimage for core-image-sato-sdk, it fails to pass test case:

Quoted 2 lines

RESULTS - systemd.SystemdBasicTests.test_systemd_failed: FAILED (0.43s)

It is OOM issue and daemon rpc.statd is killed:

Quoted 2 lines
  [  531.306146] Out of memory: Kill process 193 (rpc.statd) score

200 or sacrifice child

Increase the memory of qemu to 512M to avoid such OOM issue.

---
  meta/recipes-sato/images/core-image-sato-sdk.bb | 1 +
  1 file changed, 1 insertion(+)
meta/recipes-sato/images/core-image-sato-sdk.bb +0 -0
index d7cc52b52b..f7963d018e 100644

meta/recipes-sato/images/core-image-sato-sdk.bb IMAGE_FEATURES += "dev-pkgs tools-sdk \ · line 10

Quoted 2 lines
meta/recipes-sato/images/core-image-sato-sdk.bb +0 -0
@@ -9,3 +9,4 @@ IMAGE_FEATURES += "dev-pkgs tools-sdk \
    IMAGE_INSTALL += "kernel-devsrc"
  +TEST_QEMUPARAMS = "-m 512"

Any idea what is using so much memory in the image when this happens?

Quoted 1 line

Its rather sad that we can't have NFS+systemd with 256MB memory...

It caused by stap test case. I minimized the test cases to

TEST_SUITES = "ping date ssh systemd stap kernelmodule gcc "

Quoted 7 lines

which could reproduce the error.

And it PASSes testimage that remove stap test from default TEST_SUITES:

TEST_SUITES_remove = 'stap'

But I can't reproduce the OOM failure to run stap tests manually.

Hi Richard,

The root cause of test case stap fails is available memory is too low. Compare systemd with sysvinit, memory status are listed:

systemd:

root@qemux86:~# free -h
              total        used        free      shared buff/cache   
available
Mem:          239Mi       120Mi        23Mi       8.0Mi 94Mi 100Mi
Swap:            0B          0B          0B

sysvinit:

root@qemux86:~# free -h
              total        used        free      shared buff/cache   
available
Mem:          239Mi        45Mi       111Mi       0.0Ki 82Mi 184Mi
Swap:            0B          0B          0B

With systemd, processes of rpc.statd and rpc.mountd take about less than 80M memories. And I compared with Debian 10, they take similar size of memories.

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM TIME+ COMMAND
464 rpcuser   20   0   56316  51336   2192 S   0.0  20.9 0:00.09 rpc.statd

186 root 20 0 30136 27024 2280 S 0.0 11.0 0:00.02
rpc.mountd

Compare to sysvinit, they take only about 2M and 448K:

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM TIME+ COMMAND
  567 rpcuser   20   0    3444   2372   1792 S   0.0   1.0 0:00.00 

rpc.statd
677 root 20 0 3700 448 0 S 0.0 0.2 0:00.00
rpc.mountd

Quoted 8 lines

I didn't figure out the differences between these 2 ways to start the commands:

ExecStart=/usr/sbin/rpc.statd -F $STATD_OPTS

Vs.

test -x "$NFS_STATD" || NFS_STATD=/usr/sbin/rpc.statd start-stop-daemon --start --exec "$NFS_STATD" --pidfile "$STATD_PID"

Quoted 8 lines

Hi Richard,

Any more comment here please?

Regards,
Kai

Hi Richard,

Do you think it's acceptable to use swap as part of this test case? For example, enable 100M swap before running `stap' and disable it afterwards? This ensure no oom caused by other software using too much memory.

Best Regards,
Chen Qi

Quoted 23 lines
Quoted 22 lines

Regards,
Kai

Quoted 12 lines

Regards,
Kai

Quoted 6 lines

Cheers,

Richard