MAKEALL: Fix kill_children

--continue will allow you to <ctrl-c> the MAKEALL and pick up where you left off.

--rebuild-errors will allow you to rebuild only those boards which had trouble on the last run of MAKEALL, allowing you to quickly test a simple fix on just those boards.

Signed-off-by Joe Hershberger <[email protected]>

---
 MAKEALL | 41 +++++++++++++++++++++++++++++++++--------
 1 file changed, 33 insertions(+), 8 deletions(-)

--continue will allow you to <ctrl-c> the MAKEALL and pick up where you left off.

--rebuild-errors will allow you to rebuild only those boards which had trouble on the last run of MAKEALL, allowing you to quickly test a simple fix on just those boards.

Signed-off-by Joe Hershberger <[email protected]>

When building in parallel, make sure that we look up the children based on the the actual process group id instead of just assuming that the MAKEALL pid is the process group id.

Also ensure that logs from incomplete builds are deleted in the process.

Signed-off-by Joe Hershberger <[email protected]>

---
 MAKEALL | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

MAKEALL +24 -3

index 84a5c92..1f88dc5 100755
@@ -610,6 +610,13 @@ list_target() {
 donep="${LOG_DIR}/._done_"
 skipp="${LOG_DIR}/._skip_"
 
+build_target_killed() {
+       echo "Aborted $target build."
+       # Remove the logs for this board since it was aborted
+       rm -f ${LOG_DIR}/$target.MAKELOG ${LOG_DIR}/$target.ERR
+       exit
+}
+
 build_target() {
        target=$1
        build_idx=$2

@@ -622,6 +629,7 @@ build_target() {
        if [ $BUILD_MANY == 1 ] ; then
                output_dir="${OUTPUT_PREFIX}/${target}"
                mkdir -p "${output_dir}"
+               trap build_target_killed TERM
        else
                output_dir="${OUTPUT_PREFIX}"
        fi

@@ -640,6 +648,8 @@ build_target() {
        fi
 
        if [ $BUILD_MANY == 1 ] ; then
+               trap - TERM
+
                ${MAKE} -s tidy
 
                if [ -s ${LOG_DIR}/${target}.ERR ] ; then

@@ -721,7 +731,9 @@ build_targets() {
                        if [ $BUILD_MANY == 1 ] ; then
                                build_target ${t} ${TOTAL_CNT} &
                        else
+                               CUR_TGT="${t}"
                                build_target ${t} ${TOTAL_CNT}
+                               CUR_TGT=''
                        fi
                fi
 

@@ -745,7 +757,11 @@ build_targets() {
 #-----------------------------------------------------------------------
 
 kill_children() {
-       kill -- "-$1"
+       local pgid=`ps -p $$ --no-headers -o "%r" | tr -d ' '`
+       local children=`pgrep -g $pgid | grep -v $$ | grep -v $pgid`
+
+       kill $children 2> /dev/null
+       wait $children 2> /dev/null
 
        exit
 }

@@ -753,6 +769,9 @@ kill_children() {
 print_stats() {
        if [ "$ONLY_LIST" == 'y' ] ; then return ; fi
 
+       # Only count boards that completed
+       : $((TOTAL_CNT = `find ${skipp}* 2> /dev/null | wc -l`))
+
        rm -f ${donep}* ${skipp}*
 
        if [ $BUILD_MANY == 1 ] && [ -e "${OUTPUT_PREFIX}/ERR" ] ; then

@@ -762,6 +781,9 @@ print_stats() {
                WRN_LIST=`grep -riwL error ${OUTPUT_PREFIX}/ERR/`
                WRN_LIST=`for f in $WRN_LIST ; do echo -n " $(basename $f)" ; done`
                WRN_CNT=`echo $WRN_LIST | wc -w | awk '{print $1}'`
+       else
+               # Remove the logs for any board that was interrupted
+               rm -f ${LOG_DIR}/${CUR_TGT}.MAKELOG ${LOG_DIR}/${CUR_TGT}.ERR
        fi
 
        echo ""

@@ -776,7 +798,7 @@ print_stats() {
        echo "----------------------------------------------------------"
 
        if [ $BUILD_MANY == 1 ] ; then
-               kill_children $$ &
+               kill_children
        fi
 
        exit $RC
--

1.7.11.5

When building in parallel, make sure that we look up the children based on the the actual process group id instead of just assuming that the MAKEALL pid is the process group id.

Also ensure that logs from incomplete builds are deleted in the process.

Signed-off-by Joe Hershberger <[email protected]>

Dear Joe Hershberger,

Quoted 7 lines

When building in parallel, make sure that we look up the children based on the the actual process group id instead of just assuming that the MAKEALL pid is the process group id.

Also ensure that logs from incomplete builds are deleted in the process.

Quoted 1 line

Signed-off-by Joe Hershberger <[email protected]>

[...]

Nice $subject.

btw. is it possible to improve the u-boot build process parallelization?

Dear Joe Hershberger,

Quoted 7 lines

When building in parallel, make sure that we look up the children based on the the actual process group id instead of just assuming that the MAKEALL pid is the process group id.

Also ensure that logs from incomplete builds are deleted in the process.

Quoted 1 line

Signed-off-by Joe Hershberger <[email protected]>

[...]

Nice $subject.

btw. is it possible to improve the u-boot build process parallelization?

Quoted 11 lines

Dear Joe Hershberger,

Quoted 8 lines

When building in parallel, make sure that we look up the children based on the the actual process group id instead of just assuming that the MAKEALL pid is the process group id.

Also ensure that logs from incomplete builds are deleted in the process.

Signed-off-by Joe Hershberger <[email protected]>

Quoted 5 lines

[...]

Nice $subject.

btw. is it possible to improve the u-boot build process parallelization?

In what way?

Dear Andy Fleming,

Quoted 2 lines

On Wed, Oct 31, 2012 at 9:32 PM, Marek Vasut
<[email protected]> wrote:

Quoted 4 lines

Quoted 2 lines

btw. is it possible to improve the u-boot build process parallelization?

In what way?

I recall someone complained the build process wasn't parallelizable enough and thus we need this stuff to run multiple builds at once in MAKEALL.

When building in parallel, make sure that we look up the children based on the the actual process group id instead of just assuming that the MAKEALL pid is the process group id.

Also ensure that logs from incomplete builds are deleted in the process.

Signed-off-by Joe Hershberger <[email protected]>

Quoted 7 lines · Joe Hershberger

When building in parallel, make sure that we look up the children based on the the actual process group id instead of just assuming that the MAKEALL pid is the process group id.

Also ensure that logs from incomplete builds are deleted in the process.

Quoted 1 line

Signed-off-by Joe Hershberger <[email protected]>

Now I see:
/home/trini/bin/uboot-build.sh: line 110: 2024 Terminated /usr/bin/time -o $TIMEFILE -f %e ./MAKEALL $SOC $MACHINE > $LOG 2>&1

With my MAKEALL wrapper. Anything we can do about that?

Hi Tom,

Quoted 15 lines · Tom Rini

Quoted 8 lines · Joe Hershberger

When building in parallel, make sure that we look up the children based on the the actual process group id instead of just assuming that the MAKEALL pid is the process group id.

Also ensure that logs from incomplete builds are deleted in the process.

Signed-off-by Joe Hershberger <[email protected]>

Now I see:
/home/trini/bin/uboot-build.sh: line 110: 2024 Terminated /usr/bin/time -o $TIMEFILE -f %e ./MAKEALL $SOC $MACHINE > $LOG 2>&1

Quoted 1 line

With my MAKEALL wrapper. Anything we can do about that?

I see that as well... I did a little research and it seems like you have to jump through some pretty serious hoops to get rid of that message. I don't think it's worth the effort.

Does it have an impact on something in your wrapper?

When building in parallel, make sure that we look up the children based on the the actual process group id instead of just assuming that the MAKEALL pid is the process group id.

Also ensure that logs from incomplete builds are deleted in the process.

Signed-off-by Joe Hershberger <[email protected]>