Skip to contentSkip to conversation

qemuimagetest: Use same image during sanity testing

PATCH In review Jiajun Xu 2 participants 3 messages Jan 18, 2011 – Jan 21, 2011

lore
Share & tools
lore

Jiajun Xu

author
#0 lore raw

cover qemuimagetest: Use same image during sanity testing

To reduce the time on sanity testing, we remove variable SHARE_IMAGE and use a new variable TEST_SERIALIZE in local.conf. It is by default set to 1. Poky will copy and boot the to-be tested image for only once. It will not remove or kill the image and test cases will be serialized executed against the same image. If it is set to 0, image is always be copied for each cases, which takes much time. I had a experiment that latest qemuppc sato only takes 7 minutes to finish 9 sanity test cases, which takes more than 20 minutes before.

I also removed sanity case "boot" for target sato/sdk/lsb because the other cases for these targets already cover the check point of "boot".

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: jxu49/master
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=jxu49/master

Thanks,
Jiajun Xu <[email protected]>
---

Jiajun Xu (1):
qemuimagetest: Use same image during sanity testing instead of
copying a new image for each case

 meta/classes/imagetest-qemu.bbclass                |   33 +++++-
 meta/conf/local.conf.sample                        |   10 +-
 scripts/qemuimage-testlib                          |  110 ++++++++++++++------
 scripts/qemuimage-tests/sanity/compiler            |    2 +-
 scripts/qemuimage-tests/sanity/connman             |    2 +-
 scripts/qemuimage-tests/sanity/dmesg               |    2 +-
 scripts/qemuimage-tests/sanity/rpm_query           |    2 +-
 scripts/qemuimage-tests/sanity/scp                 |    2 +-
 scripts/qemuimage-tests/sanity/shutdown            |   11 +-
 scripts/qemuimage-tests/sanity/ssh                 |    2 +-
 scripts/qemuimage-tests/sanity/zypper_help         |    2 +-
 scripts/qemuimage-tests/sanity/zypper_search       |    2 +-
 .../scenario/qemuarm/poky-image-lsb                |    3 +-
 .../scenario/qemuarm/poky-image-sato               |    3 +-
 .../scenario/qemuarm/poky-image-sdk                |    3 +-
 .../scenario/qemumips/poky-image-lsb               |    3 +-
 .../scenario/qemumips/poky-image-sato              |    3 +-
 .../scenario/qemumips/poky-image-sdk               |    3 +-
 .../scenario/qemuppc/poky-image-lsb                |    3 +-
 .../scenario/qemuppc/poky-image-sato               |    3 +-
 .../scenario/qemuppc/poky-image-sdk                |    3 +-
 .../scenario/qemux86-64/poky-image-lsb             |    3 +-
 .../scenario/qemux86-64/poky-image-sato            |    3 +-
 .../scenario/qemux86-64/poky-image-sdk             |    3 +-
 .../scenario/qemux86/poky-image-lsb                |    3 +-
 .../scenario/qemux86/poky-image-sato               |    3 +-
 .../scenario/qemux86/poky-image-sdk                |    3 +-
 27 files changed, 138 insertions(+), 87 deletions(-)
author
#1 lore raw

PATCH 1/1 qemuimagetest: Use same image during sanity testing instead of copying a new image for each case

To reduce the time on sanity testing, we remove variable SHARE_IMAGE and use a new variable TEST_SERIALIZE in local.conf. It is by default set to 1. Poky will copy and boot the to-be tested image for only once. It will not remove or kill the image and test cases will be serialized executed against the same image. If it is set to 0, image is always be copied for each cases, which takes much time. I had a experiment that latest qemuppc sato only takes 7 minutes to finish 9 sanity test cases, which takes more than 20 minutes before.

I also removed sanity case "boot" from sato/sdk/lsb because the other cases for these targets already cover the check point of "boot".

Signed-off-by Jiajun Xu <[email protected]>

---
 meta/classes/imagetest-qemu.bbclass                |   33 +++++-
 meta/conf/local.conf.sample                        |   10 +-
 scripts/qemuimage-testlib                          |  110 ++++++++++++++------
 scripts/qemuimage-tests/sanity/compiler            |    2 +-
 scripts/qemuimage-tests/sanity/connman             |    2 +-
 scripts/qemuimage-tests/sanity/dmesg               |    2 +-
 scripts/qemuimage-tests/sanity/rpm_query           |    2 +-
 scripts/qemuimage-tests/sanity/scp                 |    2 +-
 scripts/qemuimage-tests/sanity/shutdown            |   11 +-
 scripts/qemuimage-tests/sanity/ssh                 |    2 +-
 scripts/qemuimage-tests/sanity/zypper_help         |    2 +-
 scripts/qemuimage-tests/sanity/zypper_search       |    2 +-
 .../scenario/qemuarm/poky-image-lsb                |    3 +-
 .../scenario/qemuarm/poky-image-sato               |    3 +-
 .../scenario/qemuarm/poky-image-sdk                |    3 +-
 .../scenario/qemumips/poky-image-lsb               |    3 +-
 .../scenario/qemumips/poky-image-sato              |    3 +-
 .../scenario/qemumips/poky-image-sdk               |    3 +-
 .../scenario/qemuppc/poky-image-lsb                |    3 +-
 .../scenario/qemuppc/poky-image-sato               |    3 +-
 .../scenario/qemuppc/poky-image-sdk                |    3 +-
 .../scenario/qemux86-64/poky-image-lsb             |    3 +-
 .../scenario/qemux86-64/poky-image-sato            |    3 +-
 .../scenario/qemux86-64/poky-image-sdk             |    3 +-
 .../scenario/qemux86/poky-image-lsb                |    3 +-
 .../scenario/qemux86/poky-image-sato               |    3 +-
 .../scenario/qemux86/poky-image-sdk                |    3 +-
 27 files changed, 138 insertions(+), 87 deletions(-)
meta/classes/imagetest-qemu.bbclass +28 -5
index 28bb218..8301df8 100644
@@ -5,7 +5,9 @@ TEST_LOG ?= "${LOG_DIR}/qemuimagetests"
 TEST_RESULT ?= "${TEST_DIR}/result"
 TEST_TMP ?= "${TEST_DIR}/tmp"
 TEST_SCEN ?= "sanity"
-SHARE_IMAGE ?= "1"
+TEST_STATUS ?= "${TEST_TMP}/status"
+TARGET_IPSAVE ?= "${TEST_TMP}/target_ip"
+TEST_SERIALIZE ?= "1"
 
 python do_qemuimagetest() {
     qemuimagetest_main(d)
@@ -35,6 +37,17 @@ def qemuimagetest_main(d):
     machine = bb.data.getVar('MACHINE', d, 1)
     pname = bb.data.getVar('PN', d, 1)
     
+    """function to save test cases running status"""
+    def teststatus(test, status, index, length):
+        test_status = bb.data.getVar('TEST_STATUS', d, 1)
+        if not os.path.exists(test_status):
+            raise bb.build.FuncFailed("No test status file existing under TEST_TMP")
+
+        f = open(test_status, "w")
+        f.write("\t%-15s%-15s%-15s%-15s\n" % ("Case", "Status", "Number", "Total"))
+        f.write("\t%-15s%-15s%-15s%-15s\n" % (case, status, index, length))
+        f.close()
+
     """funtion to run each case under scenario"""
     def runtest(scen, case, fulltestpath):
         resultpath = bb.data.getVar('TEST_RESULT', d, 1)
@@ -56,11 +69,13 @@ def qemuimagetest_main(d):
         os.environ["DISPLAY"] = bb.data.getVar("DISPLAY", d, True)
         os.environ["POKYBASE"] = bb.data.getVar("POKYBASE", d, True)
         os.environ["TOPDIR"] = bb.data.getVar("TOPDIR", d, True)
-        os.environ["SHARE_IMAGE"] = bb.data.getVar("SHARE_IMAGE", d, True)
+        os.environ["TEST_STATUS"] = bb.data.getVar("TEST_STATUS", d, True)
+        os.environ["TARGET_IPSAVE"] = bb.data.getVar("TARGET_IPSAVE", d, True)
+        os.environ["TEST_SERIALIZE"] = bb.data.getVar("TEST_SERIALIZE", d, True)
 
         """run Test Case"""
         bb.note("Run %s test in scenario %s" % (case, scen))
-        os.system("%s | tee -a %s" % (fulltestpath, caselog))
+        os.system("%s" % fulltestpath)
     
     """Generate testcase list in runtime"""
     def generate_list(testlist):
@@ -119,7 +134,13 @@ def qemuimagetest_main(d):
 
     tmppath = bb.data.getVar('TEST_TMP', d, 1)
     bb.utils.mkdirhier(tmppath)
-    
+
+    """initialize test status file"""
+    test_status = bb.data.getVar('TEST_STATUS', d, 1)
+    if os.path.exists(test_status):
+        os.remove(test_status)
+    os.system("touch %s" % test_status)
+
     """initialize result file"""
     resultpath = bb.data.getVar('TEST_RESULT', d, 1)
     bb.utils.mkdirhier(resultpath)
@@ -142,9 +163,11 @@ def qemuimagetest_main(d):
     fulllist = generate_list(testlist)
 
     """Begin testing"""
-    for test in fulllist:
+    for index,test in enumerate(fulllist):
         (scen, case, fullpath) = test
+        teststatus(case, "running", index, (len(fulllist) - 1))
         runtest(scen, case, fullpath)
+        teststatus(case, "finished", index, (len(fulllist) - 1))
     
     """Print Test Result"""
     ret = 0
meta/conf/local.conf.sample +5 -5
index cb0e548..4567a82 100644
@@ -158,11 +158,11 @@ ENABLE_BINARY_LOCALE_GENERATION = "1"
 
 #Because of the QEMU booting slowness issue(see bug #646 and #618), autobuilder
 #may suffer a timeout issue when running sanity test. We introduce variable
-#SHARE_IMAGE here to fix the issue. It is by default set to 1. Poky will copy
-#latest built-out image and keep using it in sanity testing. If it is set to 0,
-#latest built-out image will be copied and tested for each case, which will take
-#much time.
-#SHARE_IMAGE = "1"
+#TEST_SERIALIZE here to reduce the time on sanity test. It is by default set 
+#to 1. Poky will start image and run cases in the same image without reboot 
+#or kill. If it is set to 0, the image will be copied and tested for each 
+#case, which will take much time.
+#TEST_SERIALIZE = "1"
 
 # Set GLIBC_GENERATE_LOCALES to the locales you wish to generate should you not
 # wish to perform the time-consuming step of generating all LIBC locales.
scripts/qemuimage-testlib +76 -34

175 lines not on this page · raw message

scripts/qemuimage-tests/sanity/compiler +1 -1
index d3168a9..29dbfd9 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=200
+TIMEOUT=400
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/sanity/connman +1 -1
index 6e19b81..fca6a27 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=200
+TIMEOUT=400
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/sanity/dmesg +1 -1
index d98d7af..5ed31b7 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=200
+TIMEOUT=400
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/sanity/rpm_query +1 -1
index 7663901..08017ff 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=200
+TIMEOUT=400
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/sanity/scp +1 -1
index ca19857..c72cdc9 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=200
+TIMEOUT=400
 RET=1
 SPID=0
 i=0
scripts/qemuimage-tests/sanity/shutdown +6 -5
index dde03c9..bc08cf0 100755
@@ -13,11 +13,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-if [ $SHARE_IMAGE -eq 0 ]; then
-	TIMEOUT=200
-elif [ $SHARE_IMAGE -eq 1 ]; then
-	TIMEOUT=500
-fi
+TIMEOUT=400
 
 RET=1
 i=0
@@ -66,6 +62,11 @@ fi
 if [ ${RET} -eq 0 ]; then
 	Test_Info "Shutdown Test PASS"
 	Test_Print_Result "shutdown" 0
+	
+	# Remove TARGET_IPSAVE since no existing qemu running now
+	if [ -e ${TARGET_IPSAVE} ]; then
+		rm -rf ${TARGET_IPSAVE}
+	fi
 	exit 0
 else
 	Test_Info "Shutdown Test FAIL"
scripts/qemuimage-tests/sanity/ssh +1 -1
index e0ade72..2a0e934 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=200
+TIMEOUT=400
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/sanity/zypper_help +1 -1
index 48e121c..1ab6d24 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=200
+TIMEOUT=400
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/sanity/zypper_search +1 -1
index 9ae69eb..d6bcd27 100755
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=200
+TIMEOUT=400
 RET=1
 
 # Start qemu and check its network
scripts/qemuimage-tests/scenario/qemuarm/poky-image-lsb +1 -2
index 0eb1926..4fa6068 100644
@@ -1,8 +1,7 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
+sanity shutdown
scripts/qemuimage-tests/scenario/qemuarm/poky-image-sato +1 -2
index b60a89a..7a6353e 100644
@@ -1,5 +1,3 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
@@ -7,3 +5,4 @@ sanity zypper_help
 sanity zypper_search
 sanity rpm_query
 sanity connman
+sanity shutdown
scripts/qemuimage-tests/scenario/qemuarm/poky-image-sdk +1 -2
index 53c9ad5..42b8e19 100644
@@ -1,5 +1,3 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
@@ -8,3 +6,4 @@ sanity zypper_search
 sanity rpm_query
 sanity compiler
 sanity connman
+sanity shutdown
scripts/qemuimage-tests/scenario/qemumips/poky-image-lsb +1 -2
index 0eb1926..4fa6068 100644
@@ -1,8 +1,7 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
+sanity shutdown
scripts/qemuimage-tests/scenario/qemumips/poky-image-sato +1 -2
index b60a89a..7a6353e 100644
@@ -1,5 +1,3 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
@@ -7,3 +5,4 @@ sanity zypper_help
 sanity zypper_search
 sanity rpm_query
 sanity connman
+sanity shutdown
scripts/qemuimage-tests/scenario/qemumips/poky-image-sdk +1 -2
index 53c9ad5..42b8e19 100644
@@ -1,5 +1,3 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
@@ -8,3 +6,4 @@ sanity zypper_search
 sanity rpm_query
 sanity compiler
 sanity connman
+sanity shutdown
scripts/qemuimage-tests/scenario/qemuppc/poky-image-lsb +1 -2
index 0eb1926..4fa6068 100644
@@ -1,8 +1,7 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
+sanity shutdown
scripts/qemuimage-tests/scenario/qemuppc/poky-image-sato +1 -2
index b60a89a..7a6353e 100644
@@ -1,5 +1,3 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
@@ -7,3 +5,4 @@ sanity zypper_help
 sanity zypper_search
 sanity rpm_query
 sanity connman
+sanity shutdown
scripts/qemuimage-tests/scenario/qemuppc/poky-image-sdk +1 -2
index 53c9ad5..42b8e19 100644
@@ -1,5 +1,3 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
@@ -8,3 +6,4 @@ sanity zypper_search
 sanity rpm_query
 sanity compiler
 sanity connman
+sanity shutdown
scripts/qemuimage-tests/scenario/qemux86-64/poky-image-lsb +1 -2
index 0eb1926..4fa6068 100644
@@ -1,8 +1,7 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
+sanity shutdown
scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sato +1 -2
index b60a89a..7a6353e 100644
@@ -1,5 +1,3 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
@@ -7,3 +5,4 @@ sanity zypper_help
 sanity zypper_search
 sanity rpm_query
 sanity connman
+sanity shutdown
scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sdk +1 -2
index 53c9ad5..42b8e19 100644
@@ -1,5 +1,3 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
@@ -8,3 +6,4 @@ sanity zypper_search
 sanity rpm_query
 sanity compiler
 sanity connman
+sanity shutdown
scripts/qemuimage-tests/scenario/qemux86/poky-image-lsb +1 -2
index 0eb1926..4fa6068 100644
@@ -1,8 +1,7 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
+sanity shutdown
scripts/qemuimage-tests/scenario/qemux86/poky-image-sato +1 -2
index b60a89a..7a6353e 100644
@@ -1,5 +1,3 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
@@ -7,3 +5,4 @@ sanity zypper_help
 sanity zypper_search
 sanity rpm_query
 sanity connman
+sanity shutdown
scripts/qemuimage-tests/scenario/qemux86/poky-image-sdk +1 -2
index 53c9ad5..42b8e19 100644
@@ -1,5 +1,3 @@
-sanity shutdown
-sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
@@ -8,3 +6,4 @@ sanity zypper_search
 sanity rpm_query
 sanity compiler
 sanity connman
+sanity shutdown
Signature · 1 line
1.7.1
#2 lore raw
In reply to Jiajun Xu
Quoted 53 lines · Jiajun Xu

To reduce the time on sanity testing, we remove variable SHARE_IMAGE and use a new variable TEST_SERIALIZE in local.conf. It is by default set to 1. Poky will copy and boot the to-be tested image for only once. It will not remove or kill the image and test cases will be serialized executed against the same image. If it is set to 0, image is always be copied for each cases, which takes much time. I had a experiment that latest qemuppc sato only takes 7 minutes to finish 9 sanity test cases, which takes more than 20 minutes before.

I also removed sanity case "boot" for target sato/sdk/lsb because the other cases for these targets already cover the check point of "boot".

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: jxu49/master
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=jxu49/master

Thanks,
Jiajun Xu<[email protected]>
---

Jiajun Xu (1):
qemuimagetest: Use same image during sanity testing instead of
copying a new image for each case

  meta/classes/imagetest-qemu.bbclass                |   33 +++++-
  meta/conf/local.conf.sample                        |   10 +-
  scripts/qemuimage-testlib                          |  110 ++++++++++++++------
  scripts/qemuimage-tests/sanity/compiler            |    2 +-
  scripts/qemuimage-tests/sanity/connman             |    2 +-
  scripts/qemuimage-tests/sanity/dmesg               |    2 +-
  scripts/qemuimage-tests/sanity/rpm_query           |    2 +-
  scripts/qemuimage-tests/sanity/scp                 |    2 +-
  scripts/qemuimage-tests/sanity/shutdown            |   11 +-
  scripts/qemuimage-tests/sanity/ssh                 |    2 +-
  scripts/qemuimage-tests/sanity/zypper_help         |    2 +-
  scripts/qemuimage-tests/sanity/zypper_search       |    2 +-
  .../scenario/qemuarm/poky-image-lsb                |    3 +-
  .../scenario/qemuarm/poky-image-sato               |    3 +-
  .../scenario/qemuarm/poky-image-sdk                |    3 +-
  .../scenario/qemumips/poky-image-lsb               |    3 +-
  .../scenario/qemumips/poky-image-sato              |    3 +-
  .../scenario/qemumips/poky-image-sdk               |    3 +-
  .../scenario/qemuppc/poky-image-lsb                |    3 +-
  .../scenario/qemuppc/poky-image-sato               |    3 +-
  .../scenario/qemuppc/poky-image-sdk                |    3 +-
  .../scenario/qemux86-64/poky-image-lsb             |    3 +-
  .../scenario/qemux86-64/poky-image-sato            |    3 +-
  .../scenario/qemux86-64/poky-image-sdk             |    3 +-
  .../scenario/qemux86/poky-image-lsb                |    3 +-
  .../scenario/qemux86/poky-image-sato               |    3 +-
  .../scenario/qemux86/poky-image-sdk                |    3 +-
  27 files changed, 138 insertions(+), 87 deletions(-)
Quoted 5 lines

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

Pulled into Master

Thanks
Sau!