can not build yocto on NFS mounted NAS
Hi,
I tried to build to build bitbake core-image-minimal-dev on some nfs mounted NAS.
Pseudo is not present but is required, building this first before the
main build
Parsing recipes: 100% |###################################################################################################################################|
Time: 00:00:20
Parsing of 819 .bb files complete (0 cached, 819 parsed). 1126 targets,
47 skipped, 0 masked, 0 errors.
Build Configuration:
BB_VERSION = "1.18.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "Ubuntu-12.04" TARGET_SYS = "arm-poky-linux-gnueabi" MACHINE = "beagle-xm-ml" DISTRO = "poky" DISTRO_VERSION = "1.4.1" TUNE_FEATURES = "armv7a vfp neon" TARGET_FPU = "vfp-neon"
meta
meta-yocto
meta-yocto-bsp =
"beagleboard_dylan-9.0.1_LOCAL:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f"
It fails pretty soon with:
ERROR: Error executing a python function in
/yoctotmp/manual-build/poky/meta/recipes-devtools/quilt/quilt-native_0.60.bb:
CalledProcessError: Command 'cp -al /yoctotmp/manual-build/poky/beagle-xm/tmp/work/x86_64-linux/quilt-native/0.60-r0/sysroot-destdir/yoctotmp/manual-build/poky/beagle-xm/tmp/sysroots/x86_64-linux/* /yoctotmp/manual-build/poky/beagle-xm/tmp/work/x86_64-linux/quilt-native/0.60-r0/sstate-build-populate-sysroot/x86_64-linux'
returned non-zero
exit status 1 with output cp: cannot create hard link/yoctotmp/manual-build/poky/beagle-xm/tmp/work/x86_64-linux/quilt-native/0.60-r0/sstate-build-populate-sysroot/x86 _64-linux/usr/share/quilt/compat/awk' to /yoctotmp/manual-build/poky/beagle-xm/tmp/work/x86_64-linux/quilt-native/0.60-r0/sysroot-destdir/yoctotmp/manual-build/poky/beag le-xm/tmp/sysroots/x86_64-linux/usr/share/quilt/compat/awk': Unknown
error 524
A bit of further digging reveals that underneath /yoctotmp/manual-build/poky/beagle-xm/tmp/work/x86_64-linux/quilt-native/0.60-r0/sysroot-destdir/yoctotmp/manual-build/poky/beagle-xm/tmp/sysroots/x86_64-linux/* we have a symbolic link to /yoctotmp/manual-build/poky/beagle-xm/tmp/work/x86_64-linux/quilt-native/0.60-r0/sysroot-destdir/yoctotmp/manual-build/poky/beagle-xm/tmp/sysroots/x86_64-linux/usr/share/quilt/compat/awk -> /usr/bin/gawk
Since hard links can not be made between different file systems (local ext4 and nfs) the cp -al fails.
Is it like this by design, or should I file a bug in bugzilla?
Previous experiments (only SSTATE and DL from NFS/NAS) seem to work and I guess that's also your autobuilder cluster setup.
Regards,
Robert
...Under a government which imprisons any unjustly, the true place for a just man is also a prison. -- Henry David Thoreau
My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1
Quoted 5 lines · Robert Berger
Is it like this by design, or should I file a bug in bugzilla?
Previous experiments (only SSTATE and DL from NFS/NAS) seem to work and I guess that's also your autobuilder cluster setup.
TMPDIR on NFS has never been supported with oe/yocto, and I doubt it will be at any time in the near future, but if that's changed, I'm sure someone will jump into the thread and mention it.
As you say, SSTATE_DIR and DL_DIR work fine there.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 4 lines · Chris Larson
On Thu, Aug 8, 2013 at 6:57 AM, Robert Berger <[email protected] <mailto:[email protected]>> wrote:
Is it like this by design, or should I file a bug in bugzilla?
Quoted 8 lines
Previous experiments (only SSTATE and DL from NFS/NAS) seem to work and I guess that's also your autobuilder cluster setup.
TMPDIR on NFS has never been supported with oe/yocto, and I doubt it will be at any time in the near future, but if that's changed, I'm sure someone will jump into the thread and mention it.
As you say,
SSTATE_DIRandDL_DIRwork fine there.
Interesting. I'm building now - OEROOT on a local ext3 file system, build/tmp on NFS - and it's working just fine. My host is Fedora (15).
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Hi,
Quoted 4 lines · Gary Thomas
Interesting. I'm building now - OEROOT on a local ext3 file system, build/tmp on NFS - and it's working just fine. My host is Fedora (15).
/poky
/beagle-xm
/cache
/conf
/tmp <--- this /tmp?How you do this?
Do you use some magic build setting for this or a symlink to the nfs?
Regards,
Robert
..."It is practically impossible to teach good programming style to students that have had prior exposure to BASIC; as potential programmers they are mentally mutilated beyond hope of regeneration." - Edsger Dijkstra
My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1
Quoted 10 lines · Robert Berger
Hi,
Quoted 4 lines · Gary Thomas
Interesting. I'm building now - OEROOT on a local ext3 file system, build/tmp on NFS - and it's working just fine. My host is Fedora (15).
/poky
/beagle-xm
Quoted 7 lines
/cache /conf /tmp <--- this /tmp?How you do this?
Do you use some magic build setting for this or a symlink to the nfs?
I used the standard setup script, like this:
% . /local/poky/oe-init-build-env /MY_NFS_VOLUME/build_dir
This will create a build tree on MY_NFS_VOLUME named 'build_dir'. Then I adjusted /MY_NFS_VOLUME/build_dir/conf/local.conf as needed and ran bitbake...
Note: I NEVER build inside of my Poky/Yocto tree - always in a separate build directory and quite often on a separate file system.
(sorry for any duplicate messages, I forgot to follow my own advice and CC the list!)
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Hi,
Quoted 9 lines · Gary Thomas
Quoted 4 lines
How you do this?
Do you use some magic build setting for this or a symlink to the nfs?
I used the standard setup script, like this:
% . /local/poky/oe-init-build-env /MY_NFS_VOLUME/build_dir
Quoted 7 lines
This will create a build tree on
MY_NFS_VOLUMEnamed 'build_dir'. Then I adjusted /MY_NFS_VOLUME/build_dir/conf/local.conf as needed and ran bitbake...Note: I NEVER build inside of my Poky/Yocto tree - always in a separate build directory and quite often on a separate file system.
I just tried this and with Ubuntu 12.04.2 LTS it does __NOT__ work (as expected). I'm really surprised that you can have /MY_NFS_VOLUME/build_dir due to the fact that there are hard links required between the /MY_NFS_VOLUME/build_dir and where your host Linux rootfs resides.
So either we have a different NFS setup or cp -al behaves differently.
Do you happen to have an Ubuntu build machine to test? I'm currently on the road so it's a bit tricky for me to try with Fedora.
Regards,
Robert
..."As far as I know we never had an undetected error." - Anonymous
My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1
Quoted 26 lines · Robert Berger
Hi,
Quoted 9 lines · Gary Thomas
Quoted 4 lines
How you do this?
Do you use some magic build setting for this or a symlink to the nfs?
I used the standard setup script, like this:
% . /local/poky/oe-init-build-env /MY_NFS_VOLUME/build_dirQuoted 7 lines
This will create a build tree on
MY_NFS_VOLUMEnamed 'build_dir'. Then I adjusted /MY_NFS_VOLUME/build_dir/conf/local.conf as needed and ran bitbake...Note: I NEVER build inside of my Poky/Yocto tree - always in a separate build directory and quite often on a separate file system.
I just tried this and with Ubuntu 12.04.2 LTS it does
__NOT__work (as expected). I'm really surprised that you can have /MY_NFS_VOLUME/build_dir due to the fact that there are hard links required between the /MY_NFS_VOLUME/build_dir and where your host Linux rootfs resides.
Quoted 4 lines
So either we have a different NFS setup or cp -al behaves differently.
Do you happen to have an Ubuntu build machine to test? I'm currently on the road so it's a bit tricky for me to try with Fedora.
I'm trying this now with my Ubuntu 12.04 laptop and the same NFS server. Looks like it's working just fine, it's already made it through the first phase of a fresh build (building all the necessary 'native' tools), so I think it will work as expected (it's way past where you had problems). I'll let it complete and report back if there were any problems.
n.b. building this way is glacially slow :-(
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Hi,
Quoted 9 lines · Gary Thomas
I'm trying this now with my Ubuntu 12.04 laptop and the same NFS server. Looks like it's working just fine, it's already made it through the first phase of a fresh build (building all the necessary 'native' tools), so I think it will work as expected (it's way past where you had problems). I'll let it complete and report back if there were any problems.
n.b. building this way is glacially slow :-(
So let's assume it's the nfs server. Can you please tell me what's the file system on your nfs server and how you export it?
A possible problem is, that with nfs symbolic links are not followed by default.
My problem might be that I use a FreeBSD based nfs server.
Maybe with something like ext3, ext4 it might work.
Also do you use dylan-9.0.1 or something more recent?
Regards,
Robert
..."Premature optimisation is the root of all evil" -- Knuth
My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1
Quoted 16 lines · Robert Berger
Hi,
Quoted 9 lines · Gary Thomas
I'm trying this now with my Ubuntu 12.04 laptop and the same NFS server. Looks like it's working just fine, it's already made it through the first phase of a fresh build (building all the necessary 'native' tools), so I think it will work as expected (it's way past where you had problems). I'll let it complete and report back if there were any problems.
n.b. building this way is glacially slow :-(
So let's assume it's the nfs server. Can you please tell me what's the file system on your nfs server and how you export it?
Quoted 6 lines
A possible problem is, that with nfs symbolic links are not followed by default.
My problem might be that I use a FreeBSD based nfs server.
Maybe with something like ext3, ext4 it might work.
My NFS server is on a Fedora 16 system. The exported disk is EXT3 file system, using these exports:
/shared *(sync,insecure,rw,no_root_squash)
Quoted 2 lines
Also do you use dylan-9.0.1 or something more recent?
I always work from master, but I've been doing this (and running similar scenarios) since at least "pinky"
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Hi,
Just for the record I was able to remove the hard link with a small patch and now things look good (so far) - although it will consume more disk space.
meta/lib/oe/path.py +1 -1
@@ -90,7 +90,7 @@ def copyhardlinktree(src, dst): if not len(os.listdir(src)): return src = src + "/*" - cmd = 'cp -al %s %s' % (src, dst) + cmd = 'cp -a %s %s' % (src, dst) check_output(cmd, shell=True, stderr=subprocess.STDOUT) def remove(path, recurse=True):
Regards,
Robert
..."But I have a slowly coagulating theory that the size of a project is directly proportional to the possibility that significant bugs will crop up. Exponentiate for each additional programmer involved." - Steven K. Halliburton
My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1
Am 2013-08-18 22:44, schrieb Robert Berger:
Quoted 28 lines
Hi,
Just for the record I was able to remove the hard link with a small patch and now things look good (so far) - although it will consume more disk space.
meta/lib/oe/path.py +1
-1@@ -90,7 +90,7 @@ def copyhardlinktree(src, dst): if not len(os.listdir(src)): return src = src + "/*" - cmd = 'cp -al %s %s' % (src, dst) + cmd = 'cp -a %s %s' % (src, dst) check_output(cmd, shell=True, stderr=subprocess.STDOUT) def remove(path, recurse=True):Regards,
Robert
..."But I have a slowly coagulating theory that the size of a project is directly proportional to the possibility that significant bugs will crop up. Exponentiate for each additional programmer involved." - Steven K. Halliburton
Quoted 8 lines
My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto
Hi,
I have just two points here to ask:
1) Wouldn't simply using SSTATE_MIRRORS be a better solution here, instead?
2) Patch: wouldn't it be nicer to try the cp -al, and catch the CalledProcessError Exception, if it is thrown, and then run the brute force cp -a? Perhaps even as a general approach?
[I didn't test it either practically, just, sometimes one may get the impression in yocto / bitbake, that a bit more exception handling and directly related error messages already are one of the worlds most expensive things to do.. pls, improve this situation!! ;) ]
BR,
Lothar
Hi,
Quoted 5 lines · On 08/20/2013 07:45 PM, [email protected] wrote:
Hi,
I have just two points here to ask:1) Wouldn't simply using
SSTATE_MIRRORSbe a better solution here, instead?
It's not the same.
SSTATE and DL_DIR are on an nfs export anyhow and this works.
The problem here is with TMP dir (and an nfs server with a funny underlying filesystem like UFS) and hard links.
Quoted 4 lines
2) Patch: wouldn't it be nicer to try the cp -al, and catch the CalledProcessError Exception, if it is thrown, and then run the brute force cp -a? Perhaps even as a general approach?
In theory yes, in practice you will find out that if you build something like core-image-sato-sdk "cp -al" is not the only place which creates hard links.
Regards,
Robert
..."There's an infinite number of ways to make a simple problem seem difficult; only a handful to make a difficult problem seem simple" - Jack Crenshaw's Law #1
My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1
Hi,
Gary is, as usual, right ;)
Now I have the following test setup:
nfs server:
Ubuntu 12.04.2 LTS server
<export_dir> *(sync,insecure,rw,no_root_squash)
client:
<nfs_server>:<export_dir> <mount_point> nfs
auto,noatime,nolock,intr,tcp,actimeo=1800 0 0
mkdir -p <mount_point>/${HOSTNAME}/manual-build/
source oe-init-build-env <mount_point>/${HOSTNAME}/manual-build/<board>
I build there happily (no patch to remove hard links):
bitbake core-image-minimal-dev
bitbake core-image-sato-sdk
Regards,
Robert
..."It is practically impossible to teach good programming style to students that have had prior exposure to BASIC; as potential programmers they are mentally mutilated beyond hope of regeneration." - Edsger Dijkstra
My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1
Am 2013-08-21 07:58, schrieb Robert Berger:
Quoted 32 lines
Hi,
Quoted 5 lines · On 08/20/2013 07:45 PM, [email protected] wrote:
Hi,
I have just two points here to ask:1) Wouldn't simply using
SSTATE_MIRRORSbe a better solution here, instead?It's not the same.
SSTATE and
DL_DIRare on an nfs export anyhow and this works.The problem here is with TMP dir (and an nfs server with a funny underlying filesystem like UFS) and hard links.
Quoted 4 lines
2) Patch: wouldn't it be nicer to try the cp -al, and catch the CalledProcessError Exception, if it is thrown, and then run the brute force cp -a? Perhaps even as a general approach?
In theory yes, in practice you will find out that if you build something like core-image-sato-sdk "cp -al" is not the only place which creates hard links.
Regards,
Robert
..."There's an infinite number of ways to make a simple problem seem difficult; only a handful to make a difficult problem seem simple" - Jack Crenshaw's Law #1
Quoted 8 lines
My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto
Hi Robert,
Thank you for the answer. About my second point, I think you missunderstood, or I do not get your answer correctly, respectively.
In brief, I mean:
TRY.... CATCH... EXCEPTION
In detail:
The issue you discovered here, IMHO is definitely a real problem, when it comes to do hardlinks across filesystems.
Grepping for 'cp -al', though, to me only reviels this single place where hardlinks are done (do you know more?), which I showed you in the chat we had apart and for which you posted the fix here in the ML. To my understanding, it is only used by the sstate .bbclass, to refer to the build products after recipe execution. Means, say, using for instance a central SSTATE_DIR folder, you may remove the 'tmp' folder in a particular build folder (referring to the SSTATE_DIR), and still be able to build in your other build folders (having set SSTATE_DIR) relatively fast, since the build products were referenced in SSTATE_DIR by hardlinks. This is done by this 'cp -al', which is applied to every single recipe's build, means it is pretty central!
Thus, simply exchanging 'cp -al' by 'cp -a' would affect all recipes' execution. But running 'cp -al' and extending it by a fallback to 'cp -a' using the already thrown EXCEPTION, does not change the original behavior, but extends it. Perhaps this remark seems a bit hairsplitting. Ok, but, not handling any exception here, IMHO seems to be wrong, too, since it will already be thrown. And if so, it should be fixed, or not? This was my point about. Keeping in mind that if both approaches fail, there could be at least e.g. printstacktrace() like output.. AFAIR, the git fetcher uses a similar fallback approach for handling unaccessable repositories.
Just some remarks, maybe I'm wrong.. but, also currently messing a bit with sstate. BR,
Lothar Rubusch