current Autobuilder failure
Jiajun, Scott
Seems like there is a sanity test failure mode that then cascades to the next build target or 2 or 3! Can you look into this to determine what's happening and resolve it.
It seems to start with a timeout but then cause problems for the reset of the build of that arch.
Thanks
Sau!
Quoted 7 lines
Jiajun, Scott
Seems like there is a sanity test failure mode that then cascades to the next build target or 2 or 3! Can you look into this to determine what's happening and resolve it.
It's because the timeout value of each case is 360 seconds but now some qemu targets(qemuarm and qemupp) took more than 360s for first time booting. We have a bug to track the booting slowness issue, http://bugzilla.pokylinux.org/show_bug.cgi?id=646. When a previous sanity test is still running, a next build is started. That caused the series failure you saw.
Since only the first time booting takes much time, I wrote a patch to fix the timeout issue on autobuilder. I move test case dmesg to the first one and save the image used for dmesg test. The image then will be used in following test cases, which can save a lot of time. And I also set timeout value to 200s, which means 10 cases will cost 200*10=2000s at maximum, while the timeout value of each sanity test task for each target is 2400s. It can ensure above failure not happen again I think. Could you help to review following patch to check if it solves the issue correctly?
meta/classes/imagetest-qemu.bbclass +16 -3
@@ -5,6 +5,7 @@ TEST_LOG ?= "${LOG_DIR}/qemuimagetests" TEST_RESULT ?= "${TEST_DIR}/result" TEST_TMP ?= "${TEST_DIR}/tmp" TEST_SCEN ?= "sanity" +SHARE_IMAGE ?= "1" python do_qemuimagetest() { qemuimagetest_main(d)
@@ -37,7 +38,7 @@ def qemuimagetest_main(d): """funtion to run each case under scenario""" def runtest(scen, case, fulltestpath): resultpath = bb.data.getVar('TEST_RESULT', d, 1) - testpath = bb.data.getVar('TEST_DIR', d, 1) + tmppath = bb.data.getVar('TEST_TMP', d, 1) """initialize log file for testcase""" logpath = bb.data.getVar('TEST_LOG', d, 1)
@@ -45,10 +46,9 @@ def qemuimagetest_main(d): caselog = os.path.join(logpath, "%s/log_%s.%s" % (scen, case, bb.data.getVar('DATETIME', d, 1))) os.system("touch %s" % caselog) - """export TEST_TMP, TEST_RESULT, DEPLOY_DIR and QEMUARCH""" os.environ["PATH"] = bb.data.getVar("PATH", d, True) - os.environ["TEST_TMP"] = testpath + os.environ["TEST_TMP"] = tmppath os.environ["TEST_RESULT"] = resultpath os.environ["DEPLOY_DIR"] = bb.data.getVar("DEPLOY_DIR", d, True) os.environ["QEMUARCH"] = machine
@@ -56,6 +56,7 @@ def qemuimagetest_main(d): os.environ["DISPLAY"] = bb.data.getVar("DISPLAY", d, True) os.environ["POKYBASE"] = bb.data.getVar("POKYBASE", d, True) os.environ["TOPDIR"] = bb.data.getVar("TOPDIR", d, True) + os.environ["SHARE_IMAGE"] = bb.data.getVar("SHARE_IMAGE", d, True) """run Test Case""" bb.note("Run %s test in scenario %s" % (case, scen))
@@ -97,6 +98,15 @@ def qemuimagetest_main(d): list.append((item, casefile, fulltestcase)) return list + """Clean tmp folder for testing""" + def clean_tmp(): + tmppath = bb.data.getVar('TEST_TMP', d, 1) + + if os.path.isdir(tmppath): + for f in os.listdir(tmppath): + tmpfile = os.path.join(tmppath, f) + os.remove(tmpfile) + """check testcase folder and create test log folder""" testpath = bb.data.getVar('TEST_DIR', d, 1) bb.utils.mkdirhier(testpath)
@@ -150,6 +160,9 @@ def qemuimagetest_main(d): bb.note(line) f.close() + """Clean temp files for testing""" + clean_tmp() + if ret != 0: raise bb.build.FuncFailed("Some testcases fail, pls. check test result and test log!!!")
meta/conf/local.conf.sample +9 -0
@@ -156,6 +156,15 @@ ENABLE_BINARY_LOCALE_GENERATION = "1" # list them like following. #TEST_SCEN = "sanity bat sanity:boot" +# Because of the QEMU booting slowness issue(see bug #646 and #618), we need a +# workaround for sanity test running on autobuilder without timeout problem. +# So we introduce variable SHARE_IMAGE here. It is by default set to 1. Poky +# will use the latest built-out image for first test case running. Then the +# image will be still used in following cases and be removed after testing is end. +# If it is set to 0, latest built-out image will be copied and tested for each +# case, which will take much time. +#SHARE_IMAGE = "1" + # Set GLIBC_GENERATE_LOCALES to the locales you wish to generate should you not # wish to perform the time-consuming step of generating all LIBC locales. # NOTE: If removing en_US.UTF-8 you will also need to uncomment, and set
scripts/qemuimage-testlib +7 -0
@@ -383,9 +383,16 @@ Test_Create_Qemu() TEST_ROOTFS_IMAGE="${TEST_TMP}/${QEMUTARGET}-${QEMUARCH}-test.ext3" CP=`which cp` + + # When SHARE_IMAGE is set, we use the existing image under tmp folder if [ -e "$TEST_ROOTFS_IMAGE" ]; then + if [ ${SHARE_IMAGE} -eq 1 ]; then + ROOTFS_IMAGE="$TEST_ROOTFS_IMAGE" + TEST_ROOTFS_IMAGE="${TEST_TMP}/${QEMUTARGET}-${QEMUARCH}-shared-test.ext3" + fi rm -rf $TEST_ROOTFS_IMAGE fi + $CP $ROOTFS_IMAGE $TEST_ROOTFS_IMAGE if [ $? -ne 0 ]; then
scripts/qemuimage-tests/sanity/compiler +1 -1
@@ -11,7 +11,7 @@ . $POKYBASE/scripts/qemuimage-testlib -TIMEOUT=360 +TIMEOUT=200 RET=1 # Start qemu and check its network
scripts/qemuimage-tests/sanity/connman +1 -1
@@ -11,7 +11,7 @@ . $POKYBASE/scripts/qemuimage-testlib -TIMEOUT=360 +TIMEOUT=200 RET=1 # Start qemu and check its network
scripts/qemuimage-tests/sanity/dmesg +1 -1
@@ -11,7 +11,7 @@ . $POKYBASE/scripts/qemuimage-testlib -TIMEOUT=360 +TIMEOUT=200 RET=1 # Start qemu and check its network
scripts/qemuimage-tests/sanity/rpm_query +1 -1
@@ -11,7 +11,7 @@ . $POKYBASE/scripts/qemuimage-testlib -TIMEOUT=360 +TIMEOUT=200 RET=1 # Start qemu and check its network
scripts/qemuimage-tests/sanity/scp +1 -1
@@ -11,7 +11,7 @@ . $POKYBASE/scripts/qemuimage-testlib -TIMEOUT=360 +TIMEOUT=200 RET=1 SPID=0 i=0
scripts/qemuimage-tests/sanity/shutdown +1 -1
@@ -13,7 +13,7 @@ . $POKYBASE/scripts/qemuimage-testlib -TIMEOUT=360 +TIMEOUT=200 RET=1 i=0
scripts/qemuimage-tests/sanity/ssh +1 -1
@@ -11,7 +11,7 @@ . $POKYBASE/scripts/qemuimage-testlib -TIMEOUT=360 +TIMEOUT=200 RET=1 # Start qemu and check its network
scripts/qemuimage-tests/sanity/zypper_help +1 -1
@@ -11,7 +11,7 @@ . $POKYBASE/scripts/qemuimage-testlib -TIMEOUT=360 +TIMEOUT=200 RET=1 # Start qemu and check its network
scripts/qemuimage-tests/sanity/zypper_search +1 -1
@@ -11,7 +11,7 @@ . $POKYBASE/scripts/qemuimage-testlib -TIMEOUT=360 +TIMEOUT=200 RET=1 # Start qemu and check its network
scripts/qemuimage-tests/scenario/qemuarm/poky-image-lsb +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemuarm/poky-image-sato +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemuarm/poky-image-sdk +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemumips/poky-image-lsb +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemumips/poky-image-sato +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemumips/poky-image-sdk +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemuppc/poky-image-lsb +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemuppc/poky-image-sato +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemuppc/poky-image-sdk +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemux86-64/poky-image-lsb +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sato +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sdk +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemux86/poky-image-lsb +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemux86/poky-image-sato +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
scripts/qemuimage-tests/scenario/qemux86/poky-image-sdk +1 -1
@@ -1,8 +1,8 @@ +sanity shutdown sanity boot sanity ssh sanity scp sanity dmesg -sanity shutdown sanity zypper_help sanity zypper_search sanity rpm_query
Quoted 5 lines
It seems to start with a timeout but then cause problems for the reset of the build of that arch.
Thanks
Sau!
Best Regards,
Jiajun