"qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

DISCUSSION Steffen Sledz 5 participants 8 messages Dec 15, 2014 – Jan 1, 2015

Share & tools

Steffen Sledz

author

2014-12-15 10:54 UTC #0 lore raw

After updating our build host from openSUSE 13.1 to 13.2 we're no longer able to build our images with OE (with 13.1 on the same host everything worked well). :(

The first thing we observed was that the do_rootfs stage made a very big load (more than 500) on the machine but the cpu's aren't working for 100%. I could track down the problem to the call

qemu-arm -r 2.6.32 -L /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs -E LD_LIBRARY_PATH=/pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/usr/lib:/pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/lib /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/bin/udevadm hwdb --update --root /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs

which eats swap space without end till the system crashes (more than 100GB swap space ist available).

I've no idea how to inspect this problem further. Any ideas?

Steffen

Signature · 7 lines

DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:[email protected]
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Gaurang Shastri

2014-12-15 11:34 UTC #1 lore raw

Dear Steffen,

I have already raised some similar issue where because of un-appropriate use of "QEMU_OPTIONS" led the post process of udev-hwdb failed.

Anyhow, Richard has created a patch and you can go through the same here at:
http://lists.openembedded.org/pipermail/openembedded-core/2014-December/099544.html

Let's see if that solves your issue.

Regards,
Gaurang Shastri

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Steffen Sledz
Sent: Monday, December 15, 2014 4:24 PM
To: openembedded-core
Subject: [OE-core] "qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

After updating our build host from openSUSE 13.1 to 13.2 we're no longer able to build our images with OE (with 13.1 on the same host everything worked well). :(

The first thing we observed was that the do_rootfs stage made a very big load (more than 500) on the machine but the cpu's aren't working for 100%. I could track down the problem to the call

which eats swap space without end till the system crashes (more than 100GB swap space ist available).

I've no idea how to inspect this problem further. Any ideas?

Steffen

--
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:[email protected]
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle; Amtsgericht Berlin Charlottenburg; HRB 130120 B; Ust.-IDNr. DE273952058 --
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Enrico Scholz

2014-12-15 16:19 UTC #3 lore raw

In reply to Steffen Sledz

Quoted 7 lines · Steffen Sledz

The first thing we observed was that the do_rootfs stage made a very
big load (more than 500) on the machine but the cpu's aren't working
for 100%. I could track down the problem to the call
...
which eats swap space without end till the system crashes (more than 100GB swap space ist available).
I've no idea how to inspect this problem further. Any ideas?

I had a similar issue when /var/tmp in the rootfs was an absolute symlink pointing to toplevel /var/tmp which contained some millions of files.

Can you look (lsof) which files are open by qemu?

Enrico

Steffen Sledz

author

2014-12-16 06:30 UTC #2 lore raw

In reply to Gaurang Shastri

Quoted 9 lines · Gaurang Shastri

From: [email protected] [mailto:[email protected]] On Behalf Of Steffen Sledz
Quoted 8 lines
After updating our build host from openSUSE 13.1 to 13.2 we're no longer able to build our images with OE (with 13.1 on the same host everything worked well). :(
The first thing we observed was that the do_rootfs stage made a very big load (more than 500) on the machine but the cpu's aren't working for 100%. I could track down the problem to the call
qemu-arm -r 2.6.32 -L /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs -E LD_LIBRARY_PATH=/pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/usr/lib:/pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/lib /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/bin/udevadm hwdb --update --root /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs
which eats swap space without end till the system crashes (more than 100GB swap space ist available).

Quoted 8 lines

Quoted 1 line
I've no idea how to inspect this problem further. Any ideas?
I have already raised some similar issue where because of un-appropriate use of "QEMU_OPTIONS" led the post process of udev-hwdb failed.
Anyhow, Richard has created a patch and you can go through the same here at:
http://lists.openembedded.org/pipermail/openembedded-core/2014-December/099544.html
Let's see if that solves your issue.

Unfortunately not.

I've used the latest dizzy-Branch and applied this patch, but the behaviour did not change.

commit 219e793907406eabf632e784e3a11ab9acb77cfb

Author: Richard Purdie <[email protected]>
Date:   Thu Dec 4 19:49:19 2014 +0000

qemu/libc-package: Fix qemu option handling

The 'overrides' here are PACKAGE_ARCH based and hence not overrides as such and the _append wasn't working in many cases. This adjusts the code to use PACKAGE_ARCH as the accessor and ensures the variables work as expected. This fixes various segfaults and ensures postinsts run at build time rather than on the target system.

The bug was introduced in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=7efad8a1b56df6ee07c12ad360c0493d7b1d6d23.

Signed-off-by Richard Purdie <[email protected]>

Signed-off-by Ross Burton <[email protected]>

Regards,
Steffen

Signature · 7 lines

DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:[email protected]
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Steffen Sledz

author

2014-12-16 08:36 UTC #4 lore raw

In reply to Enrico Scholz

Quoted 8 lines · Enrico Scholz

Quoted 6 lines · Steffen Sledz
The first thing we observed was that the do_rootfs stage made a very
big load (more than 500) on the machine but the cpu's aren't working
for 100%. I could track down the problem to the call
...
which eats swap space without end till the system crashes (more than 100GB swap space ist available).

Quoted 6 lines

Quoted 1 line
I've no idea how to inspect this problem further. Any ideas?
I had a similar issue when /var/tmp in the rootfs was an absolute symlink pointing to toplevel /var/tmp which contained some millions of files.
Can you look (lsof) which files are open by qemu?

Crazy! This really seems to be the problem.

I had a big subtree from earlier local openSUSE Build Service runs below toplevel /var/run. After deleting this the build succeeds.

But manually deleting such files before running OE builds cannot really be a reasonable solution. :(

Signature · 7 lines

DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:[email protected]
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Richard Purdie

2014-12-18 10:09 UTC #5 lore raw

Quoted 14 lines · Steffen Sledz

Quoted 13 lines · Enrico Scholz
Quoted 7 lines · Steffen Sledz
The first thing we observed was that the do_rootfs stage made a very
big load (more than 500) on the machine but the cpu's aren't working
for 100%. I could track down the problem to the call
...
which eats swap space without end till the system crashes (more than 100GB swap space ist available).
I've no idea how to inspect this problem further. Any ideas?
I had a similar issue when /var/tmp in the rootfs was an absolute symlink pointing to toplevel /var/tmp which contained some millions of files.

Quoted 7 lines

Quoted 1 line
Can you look (lsof) which files are open by qemu?
Crazy! This really seems to be the problem.
I had a big subtree from earlier local openSUSE Build Service runs below toplevel /var/run. After deleting this the build succeeds.
But manually deleting such files before running OE builds cannot really be a reasonable solution. :(

Agreed, we need to figure out which tool is suffering issues with circular symlinks, then fix that tool. Its not so much a problem with the core build system as the tools we're using though :(

Cheers,

Richard

Enrico Scholz

2014-12-18 11:19 UTC #6 lore raw

Quoted 8 lines · Richard Purdie

Quoted 5 lines
I had a big subtree from earlier local openSUSE Build Service runs below toplevel /var/run. After deleting this the build succeeds.
But manually deleting such files before running OE builds cannot really be a reasonable solution. :(
Agreed, we need to figure out which tool is suffering issues with circular symlinks,

Circular symlinks are not the issue but absolute symlinks (like /tmp -> /var/tmp due to local changes). I can not find absolute symlinks to directories in my images anymore, so Steffen's problem might be caused by local changes too.

Chroot unaware tools (qemu) are the real problem...

Enrico

Khem Raj

2015-01-01 02:45 UTC #7 lore raw

Quoted 14 lines · Enrico Scholz

Quoted 8 lines · Richard Purdie
Quoted 5 lines
I had a big subtree from earlier local openSUSE Build Service runs below toplevel /var/run. After deleting this the build succeeds.
But manually deleting such files before running OE builds cannot really be a reasonable solution. :(
Agreed, we need to figure out which tool is suffering issues with circular symlinks,

Quoted 6 lines

Circular symlinks are not the issue but absolute symlinks (like /tmp -> /var/tmp due to local changes). I can not find absolute symlinks to directories in my images anymore, so Steffen's problem might be caused by local changes too.
Chroot unaware tools (qemu) are the real problem…

I think in this case one could use qemu’s -E option ( -E TMPDIR=/x/y/z) to pass target specific TMPDIR to udevadm