Skip to contentSkip to conversation

current Autobuilder failure

DISCUSSION Saul Wold 2 participants 2 messages Jan 12, 2011

lore
Share & tools
lore

Saul Wold

author
#0 lore raw

Jiajun, Scott

Seems like there is a sanity test failure mode that then cascades to the next build target or 2 or 3! Can you look into this to determine what's happening and resolve it.

It seems to start with a timeout but then cause problems for the reset of the build of that arch.

Thanks
Sau!

#1 lore raw
Quoted 7 lines

Jiajun, Scott

Seems like there is a sanity test failure mode that then cascades to the next build target or 2 or 3! Can you look into this to determine what's happening and resolve it.

It's because the timeout value of each case is 360 seconds but now some qemu targets(qemuarm and qemupp) took more than 360s for first time booting. We have a bug to track the booting slowness issue, http://bugzilla.pokylinux.org/show_bug.cgi?id=646. When a previous sanity test is still running, a next build is started. That caused the series failure you saw.
Since only the first time booting takes much time, I wrote a patch to fix the timeout issue on autobuilder. I move test case dmesg to the first one and save the image used for dmesg test. The image then will be used in following test cases, which can save a lot of time. And I also set timeout value to 200s, which means 10 cases will cost 200*10=2000s at maximum, while the timeout value of each sanity test task for each target is 2400s. It can ensure above failure not happen again I think. Could you help to review following patch to check if it solves the issue correctly?

meta/classes/imagetest-qemu.bbclass +16 -3
index bcae10d..014fd16 100644
@@ -5,6 +5,7 @@ TEST_LOG ?= "${LOG_DIR}/qemuimagetests"
 TEST_RESULT ?= "${TEST_DIR}/result"
 TEST_TMP ?= "${TEST_DIR}/tmp"
 TEST_SCEN ?= "sanity"
+SHARE_IMAGE ?= "1"
 
 python do_qemuimagetest() {
     qemuimagetest_main(d)
@@ -37,7 +38,7 @@ def qemuimagetest_main(d):
     """funtion to run each case under scenario"""
     def runtest(scen, case, fulltestpath):
         resultpath = bb.data.getVar('TEST_RESULT', d, 1)
-        testpath = bb.data.getVar('TEST_DIR', d, 1)
+        tmppath = bb.data.getVar('TEST_TMP', d, 1)
 
         """initialize log file for testcase"""
         logpath = bb.data.getVar('TEST_LOG', d, 1)
@@ -45,10 +46,9 @@ def qemuimagetest_main(d):
         caselog = os.path.join(logpath, "%s/log_%s.%s" % (scen, case, bb.data.getVar('DATETIME', d, 1)))
         os.system("touch %s" % caselog)
 
-
         """export TEST_TMP, TEST_RESULT, DEPLOY_DIR and QEMUARCH"""
         os.environ["PATH"] = bb.data.getVar("PATH", d, True)
-        os.environ["TEST_TMP"] = testpath
+        os.environ["TEST_TMP"] = tmppath
         os.environ["TEST_RESULT"] = resultpath
         os.environ["DEPLOY_DIR"] = bb.data.getVar("DEPLOY_DIR", d, True)
         os.environ["QEMUARCH"] = machine
@@ -56,6 +56,7 @@ def qemuimagetest_main(d):
         os.environ["DISPLAY"] = bb.data.getVar("DISPLAY", d, True)
         os.environ["POKYBASE"] = bb.data.getVar("POKYBASE", d, True)
         os.environ["TOPDIR"] = bb.data.getVar("TOPDIR", d, True)
+        os.environ["SHARE_IMAGE"] = bb.data.getVar("SHARE_IMAGE", d, True)
 
         """run Test Case"""
         bb.note("Run %s test in scenario %s" % (case, scen))
@@ -97,6 +98,15 @@ def qemuimagetest_main(d):
                        list.append((item, casefile, fulltestcase))
         return list
 
+    """Clean tmp folder for testing"""
+    def clean_tmp():
+        tmppath = bb.data.getVar('TEST_TMP', d, 1)
+
+        if os.path.isdir(tmppath):
+            for f in os.listdir(tmppath):
+                tmpfile = os.path.join(tmppath, f)
+                os.remove(tmpfile)
+
     """check testcase folder and create test log folder"""
     testpath = bb.data.getVar('TEST_DIR', d, 1)
     bb.utils.mkdirhier(testpath)
@@ -150,6 +160,9 @@ def qemuimagetest_main(d):
             bb.note(line)
     f.close()
 
+    """Clean temp files for testing"""
+    clean_tmp()
+
     if ret != 0:
         raise bb.build.FuncFailed("Some testcases fail, pls. check test result and test log!!!")
 
meta/conf/local.conf.sample +9 -0
index 9c58e6b..33a50dc 100644
@@ -156,6 +156,15 @@ ENABLE_BINARY_LOCALE_GENERATION = "1"
 # list them like following.
 #TEST_SCEN = "sanity bat sanity:boot"
 
+# Because of the QEMU booting slowness issue(see bug #646 and #618), we need a
+# workaround for sanity test running on autobuilder without timeout problem.
+# So we introduce variable SHARE_IMAGE here. It is by default set to 1. Poky
+# will use the latest built-out image for first test case running. Then the
+# image will be still used in following cases and be removed after testing is end.
+# If it is set to 0, latest built-out image will be copied and tested for each
+# case, which will take much time.
+#SHARE_IMAGE = "1"
+
 # Set GLIBC_GENERATE_LOCALES to the locales you wish to generate should you not
 # wish to perform the time-consuming step of generating all LIBC locales.
 # NOTE: If removing en_US.UTF-8 you will also need to uncomment, and set
scripts/qemuimage-testlib +7 -0
index e2c2d24..b9afcf5 100644
@@ -383,9 +383,16 @@ Test_Create_Qemu()
        TEST_ROOTFS_IMAGE="${TEST_TMP}/${QEMUTARGET}-${QEMUARCH}-test.ext3"
 
        CP=`which cp`
+
+       # When SHARE_IMAGE is set, we use the existing image under tmp folder
        if [ -e "$TEST_ROOTFS_IMAGE" ]; then
+               if [ ${SHARE_IMAGE} -eq 1 ]; then
+                       ROOTFS_IMAGE="$TEST_ROOTFS_IMAGE"
+                       TEST_ROOTFS_IMAGE="${TEST_TMP}/${QEMUTARGET}-${QEMUARCH}-shared-test.ext3"
+               fi
                rm -rf $TEST_ROOTFS_IMAGE
        fi
+
        $CP $ROOTFS_IMAGE $TEST_ROOTFS_IMAGE
 
        if [ $? -ne 0 ]; then
scripts/qemuimage-tests/sanity/compiler +1 -1
index 0290540..d3168a9 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/sanity/connman +1 -1
index 8c5b7c5..6e19b81 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/sanity/dmesg +1 -1
index 36813dc..d98d7af 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/sanity/rpm_query +1 -1
index 9962115..7663901 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/sanity/scp +1 -1
index ce3489d..ca19857 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 SPID=0
 i=0
scripts/qemuimage-tests/sanity/shutdown +1 -1
index e36b4a9..54d7861 100755
@@ -13,7 +13,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 i=0
 
scripts/qemuimage-tests/sanity/ssh +1 -1
index 084530a..e0ade72 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/sanity/zypper_help +1 -1
index e052501..48e121c 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/sanity/zypper_search +1 -1
index b756db8..9ae69eb 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/scenario/qemuarm/poky-image-lsb +1 -1
index 32d2bd1..0eb1926 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemuarm/poky-image-sato +1 -1
index 7b949a8..b60a89a 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemuarm/poky-image-sdk +1 -1
index 3a8b129..53c9ad5 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemumips/poky-image-lsb +1 -1
index 32d2bd1..0eb1926 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemumips/poky-image-sato +1 -1
index 7b949a8..b60a89a 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemumips/poky-image-sdk +1 -1
index 3a8b129..53c9ad5 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemuppc/poky-image-lsb +1 -1
index 32d2bd1..0eb1926 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemuppc/poky-image-sato +1 -1
index 7b949a8..b60a89a 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemuppc/poky-image-sdk +1 -1
index 3a8b129..53c9ad5 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemux86-64/poky-image-lsb +1 -1
index 32d2bd1..0eb1926 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sato +1 -1
index 7b949a8..b60a89a 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sdk +1 -1
index 3a8b129..53c9ad5 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemux86/poky-image-lsb +1 -1
index 32d2bd1..0eb1926 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemux86/poky-image-sato +1 -1
index 7b949a8..b60a89a 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
scripts/qemuimage-tests/scenario/qemux86/poky-image-sdk +1 -1
index 3a8b129..53c9ad5 100644
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
Quoted 5 lines

It seems to start with a timeout but then cause problems for the reset of the build of that arch.

Thanks
Sau!

Best Regards,
Jiajun