Could we build tar-replacement firstly and not parallel if tar-replacement is needed to build

PATCH Rongqing Li 3 participants 8 messages Aug 6, 2012 – Aug 7, 2012

Share & tools

Rongqing Li

author

2012-08-06 13:01 UTC #0 lore raw

Hi:

Building failed several times when building tar-replacement-native is needed, The error log shows "bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy". this log proves "an attempt was made to execute a pure-procedure program that is currently open for writing."

Before this error, tar-replacement do_populate_sysroot just starts, (tar-replacement-native-1.26-r2: task do_populate_sysroot: Started) After this error, we see tar-replacement do_populate_sysroot just finish.

So I think the root cause is that do_populate_sysroot sqlite is using a tar command which being opened to written. in fact, all packages need tar to do_popolate_sysroot()

I see building tar-replacement-native is started in bitbake file, could we build tar-replacement firstly, not parallel if needed.

scripts/bitbake +7 -1

index 3772d82..1bb8fed 100755
@@ -134,7 +134,10 @@ if [ $buildpseudo -gt 0 ]; then
             fi
         done
     done
-    bitbake pseudo-native $TARTARGET $additionalopts -c populate_sysroot
+
+    if [ $needptar = "1" ]; then
+       sed -i 's/BB_NUMBER_THREADS =/#BB_NUMBER_THREADS =/g' conf/local.conf
+       bitbake $TARTARGET
+       sed -i 's/#BB_NUMBER_THREADS =/BB_NUMBER_THREADS =/g' conf/local.conf
+    fi
+

+ bitbake pseudo-native $additionalopts -c populate_sysroot

     ret=$?
     if [ "$ret" != "0" ]; then
         exit 1

since building tar-replacement or not depends on host tar version, it is hard to add a tar dependence to any package, and I did not have a good way to fix this bug except the upper.

Any advice and suggestion are welcome

Signature · 2 lines

Best Reagrds,
Roy | RongQing Li

Rongqing Li

author

2012-08-07 06:11 UTC #1 lore raw

Paste a RFC PATCH

[RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not parallel

Compiling tar-replacement or not is decided by version of host tar, if the host tar version is lower than 1.23, Compiling tar-replacement is needed.

When doing popoluate tar-replacement sysroot to write the tar to sysroot, but writing is not finished. other packages probably use the being written tar to unzip file, which will lead to failure and report the below error:
"bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy "

Now we compile tar-replacement firstly and not parallel to ensure that a being written tar command will not be used.

Signed-off-by Roy.Li <[email protected]>

---
 scripts/bitbake |   14 +++++++++++++-
 1 files changed, 13 insertions(+), 1 deletions(-)

scripts/bitbake +13 -1

index 3772d82..eb6b144 100755
@@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
             fi
         done
     done
-    bitbake pseudo-native $TARTARGET $additionalopts -c populate_sysroot
+
+    if [ $needtar = "1" ]; then
+       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS" conf/local.conf |awk -F':' '{print $1}'`
+	test -n "$NUM_THREAD_LINE" &&
+	sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g' conf/local.conf
+
+	bitbake $TARTARGET -c populate_sysroot
+
+	test -n "$NUM_THREAD_LINE" &&
+	sed -i ''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g' conf/local.conf
+    fi
+
+    bitbake pseudo-native $additionalopts -c populate_sysroot
     ret=$?
     if [ "$ret" != "0" ]; then
         exit 1

Signature · 1 line

1.7.4.1

Quoted 56 lines · Rongqing Li

Hi:
Building failed several times when building tar-replacement-native is needed,
The error log shows "bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy". this log proves "an attempt was made to execute a pure-procedure
program that is currently open for writing."
Before this error, tar-replacement do_populate_sysroot just starts, (tar-replacement-native-1.26-r2: task do_populate_sysroot: Started) After this error, we see tar-replacement do_populate_sysroot just finish.
So I think the root cause is that do_populate_sysroot sqlite is using a tar command which being opened to written. in fact, all packages need tar to do_popolate_sysroot()
I see building tar-replacement-native is started in bitbake file, could we build tar-replacement firstly, not parallel if needed.
scripts/bitbake +3 -1
index 3772d82..1bb8fed 100755
@@ -134,7 +134,10 @@ if [ $buildpseudo -gt 0 ]; then
              fi
          done
      done
-    bitbake pseudo-native $TARTARGET $additionalopts -c populate_sysroot
+
+    if [ $needptar = "1" ]; then
+       sed -i 's/BB_NUMBER_THREADS =/#BB_NUMBER_THREADS =/g'
conf/local.conf
scripts/bitbake +2 -0
+       bitbake $TARTARGET
+       sed -i 's/#BB_NUMBER_THREADS =/BB_NUMBER_THREADS =/g'
conf/local.conf
scripts/bitbake +3 -0
+    fi
+
+    bitbake pseudo-native $additionalopts -c populate_sysroot
      ret=$?
      if [ "$ret" != "0" ]; then
          exit 1
since building tar-replacement or not depends on host tar version, it is hard to add a tar dependence to any package, and I did not have a good way to fix this bug except the upper.
Any advice and suggestion are welcome

Signature · 2 lines

Best Reagrds,
Roy | RongQing Li

Richard Purdie

2012-08-07 07:53 UTC #2 lore raw

Quoted 42 lines · Rongqing Li

Paste a RFC PATCH
[RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not parallel
Compiling tar-replacement or not is decided by version of host tar, if the host tar version is lower than 1.23, Compiling tar-replacement is needed.
When doing popoluate tar-replacement sysroot to write the tar to sysroot, but writing is not finished. other packages probably use the being written tar to unzip file, which will lead to failure and report the below error:
"bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy "
Now we compile tar-replacement firstly and not parallel to ensure that a being written tar command will not be used.
Signed-off-by Roy.Li <[email protected]>
---
  scripts/bitbake |   14 +++++++++++++-
  1 files changed, 13 insertions(+), 1 deletions(-)
scripts/bitbake +3 -1
index 3772d82..eb6b144 100755
@@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
              fi
          done
      done
-    bitbake pseudo-native $TARTARGET $additionalopts -c populate_sysroot
+
+    if [ $needtar = "1" ]; then
+       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS" conf/local.conf 
|awk -F':' '{print $1}'`
scripts/bitbake +7 -0
+	test -n "$NUM_THREAD_LINE" &&
+	sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g' conf/local.conf
+
+	bitbake $TARTARGET -c populate_sysroot
+
+	test -n "$NUM_THREAD_LINE" &&
+	sed -i 

Quoted 3 lines

''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g' conf/local.conf
+ fi

We are *NOT* running sed over local.conf. What is the user used a different configuration file for example?

If you're just building tar-native, is there a parallel race possible? I'm not sure that there is?

For the record I hate this script and what it has become, its horrible. It started as a workaround for pseudo-native, its becoming a dumping ground for a whole set of nasty workarounds :(.

Cheers,

Richard

Rongqing Li

author

2012-08-07 08:14 UTC #3 lore raw

Quoted 41 lines · Richard Purdie

Quoted 40 lines · Rongqing Li
Paste a RFC PATCH
[RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not parallel
Compiling tar-replacement or not is decided by version of host tar, if the host tar version is lower than 1.23, Compiling tar-replacement is needed.
When doing popoluate tar-replacement sysroot to write the tar to sysroot, but writing is not finished. other packages probably use the being written tar to unzip file, which will lead to failure and report the below error:
"bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy"
Now we compile tar-replacement firstly and not parallel to ensure that a being written tar command will not be used.
Signed-off-by Roy.Li <[email protected]>
---
   scripts/bitbake |   14 +++++++++++++-
   1 files changed, 13 insertions(+), 1 deletions(-)
scripts/bitbake +3 -1
index 3772d82..eb6b144 100755
@@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
               fi
           done
       done
-    bitbake pseudo-native $TARTARGET $additionalopts -c populate_sysroot
+
+    if [ $needtar = "1" ]; then
+       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS" conf/local.conf
|awk -F':' '{print $1}'`
scripts/bitbake +5 -0
+	test -n "$NUM_THREAD_LINE" &&
+	sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g' conf/local.conf
+
+	bitbake $TARTARGET -c populate_sysroot
+

Quoted 8 lines

Quoted 5 lines
scripts/bitbake +2 -0
+	test -n "$NUM_THREAD_LINE" &&
+	sed -i
''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g' conf/local.conf
+ fi
We are *NOT* running sed over local.conf. What is the user used a different configuration file for example?

I use the sed to disable NUM_THREAD_LINE in conf/local.conf if NUM_THREAD_LINE has been enabled.

After compile $TARTARGET, use the sed to enable NUM_THREAD_LINE in conf/local.conf

Quoted 2 lines

If you're just building tar-native, is there a parallel race possible? I'm not sure that there is?

Even if I just build tar-native, I still have several packages which are needed to building.

$bitbake tar-replacement-native -g

$ cat pn-buildlist
autoconf-native
libtool-native
m4-native
gettext-minimal-native
tar-replacement-native
quilt-native
gnu-config-native
automake-native

Thanks for your reply.
-Roy

Quoted 16 lines

For the record I hate this script and what it has become, its horrible. It started as a workaround for pseudo-native, its becoming a dumping ground for a whole set of nasty workarounds :(.
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Signature · 2 lines

Best Reagrds,
Roy | RongQing Li

Pascal Ouyang

2012-08-07 09:09 UTC #4 lore raw

Quoted 73 lines · Rongqing Li

Quoted 44 lines · Richard Purdie
Quoted 43 lines · Rongqing Li
Paste a RFC PATCH
[RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not parallel
Compiling tar-replacement or not is decided by version of host tar, if the host tar version is lower than 1.23, Compiling tar-replacement is needed.
When doing popoluate tar-replacement sysroot to write the tar to sysroot, but writing is not finished. other packages probably use the being written tar to unzip file, which will lead to failure and report the below error:
"bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy"
Now we compile tar-replacement firstly and not parallel to ensure that a being written tar command will not be used.
Signed-off-by Roy.Li <[email protected]>
---
   scripts/bitbake |   14 +++++++++++++-
   1 files changed, 13 insertions(+), 1 deletions(-)
scripts/bitbake +0 -1
index 3772d82..eb6b144 100755
@@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
               fi
           done
       done
-    bitbake pseudo-native $TARTARGET $additionalopts -c
populate_sysroot
scripts/bitbake +3 -0
+
+    if [ $needtar = "1" ]; then
+       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS" conf/local.conf
|awk -F':' '{print $1}'`
scripts/bitbake +2 -0
+    test -n "$NUM_THREAD_LINE" &&
+    sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g'
conf/local.conf
scripts/bitbake +3 -0
+
+    bitbake $TARTARGET -c populate_sysroot
+
Quoted 8 lines
Quoted 5 lines
scripts/bitbake +2 -0
+    test -n "$NUM_THREAD_LINE" &&
+    sed -i
''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g' conf/local.conf
+ fi
We are *NOT* running sed over local.conf. What is the user used a different configuration file for example?
I use the sed to disable NUM_THREAD_LINE in conf/local.conf if NUM_THREAD_LINE has been enabled.
After compile $TARTARGET, use the sed to enable NUM_THREAD_LINE in conf/local.conf
Quoted 2 lines
If you're just building tar-native, is there a parallel race possible? I'm not sure that there is?
Even if I just build tar-native, I still have several packages which are needed to building.
$bitbake tar-replacement-native -g

Quoted 10 lines

$ cat pn-buildlist
autoconf-native
libtool-native
m4-native
gettext-minimal-native
tar-replacement-native
quilt-native
gnu-config-native
automake-native

To avoid to use the incomplete tar binary, I think we could rename the binary name to "tar-native" before do_populate_sysroot, and then rename back to "tar" after do_populate_sysroot().

Thanks.

- Pascal

Quoted 20 lines

Thanks for your reply.
-Roy
Quoted 16 lines
For the record I hate this script and what it has become, its horrible. It started as a workaround for pseudo-native, its becoming a dumping ground for a whole set of nasty workarounds :(.
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Signature · 1 line

- Pascal

Richard Purdie

2012-08-07 09:14 UTC #6 lore raw

In reply to Rongqing Li

Quoted 52 lines · Rongqing Li

Quoted 41 lines · Richard Purdie
Quoted 40 lines · Rongqing Li
Paste a RFC PATCH
[RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not parallel
Compiling tar-replacement or not is decided by version of host tar, if the host tar version is lower than 1.23, Compiling tar-replacement is needed.
When doing popoluate tar-replacement sysroot to write the tar to sysroot, but writing is not finished. other packages probably use the being written tar to unzip file, which will lead to failure and report the below error:
"bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy"
Now we compile tar-replacement firstly and not parallel to ensure that a being written tar command will not be used.
Signed-off-by Roy.Li <[email protected]>
---
   scripts/bitbake |   14 +++++++++++++-
   1 files changed, 13 insertions(+), 1 deletions(-)
scripts/bitbake +3 -1
index 3772d82..eb6b144 100755
@@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
               fi
           done
       done
-    bitbake pseudo-native $TARTARGET $additionalopts -c populate_sysroot
+
+    if [ $needtar = "1" ]; then
+       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS" conf/local.conf
|awk -F':' '{print $1}'`
scripts/bitbake +5 -0
+	test -n "$NUM_THREAD_LINE" &&
+	sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g' conf/local.conf
+
+	bitbake $TARTARGET -c populate_sysroot
+
Quoted 8 lines
Quoted 5 lines
scripts/bitbake +2 -0
+	test -n "$NUM_THREAD_LINE" &&
+	sed -i
''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g' conf/local.conf
+ fi
We are *NOT* running sed over local.conf. What is the user used a different configuration file for example?

Quoted 5 lines

I use the sed to disable NUM_THREAD_LINE in conf/local.conf if NUM_THREAD_LINE has been enabled.
After compile $TARTARGET, use the sed to enable NUM_THREAD_LINE in conf/local.conf

I understand what you did, I just don't want to do this. Its horrible and buggy.

Quoted 9 lines

Quoted 2 lines
If you're just building tar-native, is there a parallel race possible? I'm not sure that there is?
Even if I just build tar-native, I still have several packages which are needed to building.
$bitbake tar-replacement-native -g

Quoted 9 lines

$ cat pn-buildlist
autoconf-native
libtool-native
m4-native
gettext-minimal-native
tar-replacement-native
quilt-native
gnu-config-native
automake-native

but tar-replacement-native will not build until all the above have executed their populate_sysroot tasks, correct?

So I don't see a task that would run at the same time as tar-replacement:do_populate_sysroot which will be the last task to run?

Cheers,

Richard

Rongqing Li

author

2012-08-07 09:37 UTC #7 lore raw

Quoted 84 lines · Richard Purdie

Quoted 52 lines · Rongqing Li
Quoted 41 lines · Richard Purdie
Quoted 40 lines · Rongqing Li
Paste a RFC PATCH
[RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not parallel
Compiling tar-replacement or not is decided by version of host tar, if the host tar version is lower than 1.23, Compiling tar-replacement is needed.
When doing popoluate tar-replacement sysroot to write the tar to sysroot, but writing is not finished. other packages probably use the being written tar to unzip file, which will lead to failure and report the below error:
"bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy"
Now we compile tar-replacement firstly and not parallel to ensure that a being written tar command will not be used.
Signed-off-by Roy.Li <[email protected]>
---
    scripts/bitbake |   14 +++++++++++++-
    1 files changed, 13 insertions(+), 1 deletions(-)
scripts/bitbake +3 -1
index 3772d82..eb6b144 100755
@@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
                fi
            done
        done
-    bitbake pseudo-native $TARTARGET $additionalopts -c populate_sysroot
+
+    if [ $needtar = "1" ]; then
+       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS" conf/local.conf
|awk -F':' '{print $1}'`
scripts/bitbake +5 -0
+	test -n "$NUM_THREAD_LINE" &&
+	sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g' conf/local.conf
+
+	bitbake $TARTARGET -c populate_sysroot
+
Quoted 8 lines
Quoted 5 lines
scripts/bitbake +2 -0
+	test -n "$NUM_THREAD_LINE" &&
+	sed -i
''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g' conf/local.conf
+ fi
We are *NOT* running sed over local.conf. What is the user used a different configuration file for example?
Quoted 5 lines
I use the sed to disable NUM_THREAD_LINE in conf/local.conf if NUM_THREAD_LINE has been enabled.
After compile $TARTARGET, use the sed to enable NUM_THREAD_LINE in conf/local.conf
I understand what you did, I just don't want to do this. Its horrible and buggy.
Quoted 9 lines
Quoted 2 lines
If you're just building tar-native, is there a parallel race possible? I'm not sure that there is?
Even if I just build tar-native, I still have several packages which are needed to building.
$bitbake tar-replacement-native -g
Quoted 9 lines
$ cat pn-buildlist
autoconf-native
libtool-native
m4-native
gettext-minimal-native
tar-replacement-native
quilt-native
gnu-config-native
automake-native
but tar-replacement-native will not build until all the above have executed their populate_sysroot tasks, correct?

Quoted 8 lines

So I don't see a task that would run at the same time as tar-replacement:do_populate_sysroot which will be the last task to run?
Cheers,
Richard

You are right, seems I only need to build tar-replacement firstly, donot need to disable the BB_NUMBER_THREADS.

-Roy

Quoted 1 line

Signature · 2 lines

Best Reagrds,
Roy | RongQing Li

Rongqing Li

author

2012-08-07 09:39 UTC #5 lore raw

In reply to Pascal Ouyang

Quoted 86 lines · Pascal Ouyang

Quoted 74 lines · Rongqing Li
Quoted 45 lines · Richard Purdie
Quoted 44 lines · Rongqing Li
Paste a RFC PATCH
[RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not parallel
Compiling tar-replacement or not is decided by version of host tar, if the host tar version is lower than 1.23, Compiling tar-replacement is needed.
When doing popoluate tar-replacement sysroot to write the tar to sysroot, but writing is not finished. other packages probably use the being written tar to unzip file, which will lead to failure and report the below error:
"bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy"
Now we compile tar-replacement firstly and not parallel to ensure that a being written tar command will not be used.
Signed-off-by Roy.Li <[email protected]>
---
   scripts/bitbake |   14 +++++++++++++-
   1 files changed, 13 insertions(+), 1 deletions(-)
scripts/bitbake +0 -1
index 3772d82..eb6b144 100755
@@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
               fi
           done
       done
-    bitbake pseudo-native $TARTARGET $additionalopts -c
populate_sysroot
scripts/bitbake +3 -0
+
+    if [ $needtar = "1" ]; then
+       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS"
conf/local.conf
|awk -F':' '{print $1}'`
scripts/bitbake +2 -0
+    test -n "$NUM_THREAD_LINE" &&
+    sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g'
conf/local.conf
scripts/bitbake +3 -0
+
+    bitbake $TARTARGET -c populate_sysroot
+
Quoted 8 lines
Quoted 5 lines
scripts/bitbake +2 -0
+    test -n "$NUM_THREAD_LINE" &&
+    sed -i
''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g' conf/local.conf
+ fi
We are *NOT* running sed over local.conf. What is the user used a different configuration file for example?
I use the sed to disable NUM_THREAD_LINE in conf/local.conf if NUM_THREAD_LINE has been enabled.
After compile $TARTARGET, use the sed to enable NUM_THREAD_LINE in conf/local.conf
Quoted 2 lines
If you're just building tar-native, is there a parallel race possible? I'm not sure that there is?
Even if I just build tar-native, I still have several packages which are needed to building.
$bitbake tar-replacement-native -g
Quoted 10 lines
$ cat pn-buildlist
autoconf-native
libtool-native
m4-native
gettext-minimal-native
tar-replacement-native
quilt-native
gnu-config-native
automake-native

Quoted 8 lines

To avoid to use the incomplete tar binary, I think we could rename the binary name to "tar-native" before do_populate_sysroot, and then rename back to "tar" after do_populate_sysroot().
Thanks.
- Pascal

I think your suggestion is valuable to fix the similar problem, we can adopt this method.

-Roy

Quoted 22 lines

Quoted 20 lines
Thanks for your reply.
-Roy
Quoted 16 lines
For the record I hate this script and what it has become, its horrible. It started as a workaround for pseudo-native, its becoming a dumping ground for a whole set of nasty workarounds :(.
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Signature · 2 lines

Best Reagrds,
Roy | RongQing Li