Skip to contentSkip to conversation

Improve README and enable Wheezy

RFC PATCH In review Wei Liu 2 participants 16 messages Sep 19, 2013 – Sep 20, 2013

lore
Share & tools
lore

Wei Liu

author
#0 lore raw

cover Improve README and enable Wheezy

Hi Ian and Ian

With the changes to Debian.pm and ts-debian-fixup I can now use Wheezy to run build-* jobs and test-amd64-amd64-{pv,xl}.

Wei Liu (5):
README: add curl and netcat to requirement list
README: describe job dependence
Debian: install libc6-dev-i386 for Wheezy amd64
ts-debian-fixup: remove dead code
ts-debian-fixup: workaround Debian bug #691320

 README            |    8 ++++++++
 ts-debian-fixup   |   32 ++++++++++++++------------------
 ts-xen-build-prep |    4 ++++
 3 files changed, 26 insertions(+), 18 deletions(-)
Signature · 1 line
1.7.10.4
author
#1 lore raw

RFC PATCH 1/5 README: add curl and netcat to requirement list

---
 README |    2 ++
 1 file changed, 2 insertions(+)
README +2 -0
index e9aed52..c845d4d 100644
@@ -5,6 +5,8 @@ To run osstest in standalone mode:
      tcl8.4 tclx8.4 libsqlite3-tcl
      libdbi-perl libdbd-sqlite3-perl
      pax rsync
+     curl
+     netcat
 
  - Write a config file
     ~/.xen-osstest/config
Signature · 1 line
1.7.10.4
author
#2 lore raw

RFC PATCH 2/5 README: describe job dependence

In reply to Wei Liu
---
 README |    6 ++++++
 1 file changed, 6 insertions(+)
README +6 -0
index c845d4d..7f90696 100644
@@ -33,6 +33,12 @@ To run osstest in standalone mode:
                              (strictly, the final `_HOST' is the
 			      uppercase of the host's `ident' in the job)
 
+   However, as test-amd64-{i386,amd64}-xl and other tests depends on
+   some runtime variables generated by build-* jobs, you can run
+   build-* jobs before running actual test jobs. If you don't want to do
+   so you need to insert those missing runvars into standalone.db with
+   sqlite3.
+
  - Don't forget to set the machine to pxeboot in the BIOS.
 
  - Currently you need a serial console at 115200 8n1; see TODO.
Signature · 1 line
1.7.10.4
author
#3 lore raw

RFC PATCH 3/5 Debian: install libc6-dev-i386 for Wheezy amd64

In reply to Wei Liu

Otherwise build-amd64 fails with missing header predefs.h.

---
 ts-xen-build-prep |    4 ++++
 1 file changed, 4 insertions(+)
ts-xen-build-prep +4 -0
index b1e6a6c..66ba000 100755
@@ -175,6 +175,10 @@ sub prep () {
     target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
     # workaround for Debian #595728
 
+    if ($r{arch} eq 'amd64' && $ho->{Suite} =~ m/wheezy/) {
+	target_install_packages($ho, 'libc6-dev-i386');
+    }
+
     if ($ho->{Suite} !~ m/lenny/) {
         target_putfilecontents_stash($ho,60,<<'END','/home/osstest/.hgrc');
 [extensions]
Signature · 1 line
1.7.10.4
#4 lore raw

Re: [PATCH OSSTEST RFC 3/5] Debian: install libc6-dev-i386 for Wheezy amd64

Quoted 1 line · Wei Liu

Otherwise build-amd64 fails with missing header predefs.h.

This package exists in Squeeze too, it might simplify things to make it unconditional?

I wonder why it apparently isn't need for a Squeeze build though. What 32-bit stuff is getting built?

Quoted 21 lines
---
 ts-xen-build-prep |    4 ++++
 1 file changed, 4 insertions(+)
ts-xen-build-prep +4 -0
index b1e6a6c..66ba000 100755
@@ -175,6 +175,10 @@ sub prep () {
     target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
     # workaround for Debian #595728
 
+    if ($r{arch} eq 'amd64' && $ho->{Suite} =~ m/wheezy/) {
+	target_install_packages($ho, 'libc6-dev-i386');
+    }
+
     if ($ho->{Suite} !~ m/lenny/) {
         target_putfilecontents_stash($ho,60,<<'END','/home/osstest/.hgrc');
 [extensions]
author
#5 lore raw

Re: [PATCH OSSTEST RFC 3/5] Debian: install libc6-dev-i386 for Wheezy amd64

Quoted 6 lines · Ian Campbell
Quoted 1 line · Wei Liu

Otherwise build-amd64 fails with missing header predefs.h.

This package exists in Squeeze too, it might simplify things to make it unconditional?

No, it doesn't exist in Lenny -- I see there's branches conditional on Lenny, do we still use that?

Quoted 3 lines

I wonder why it apparently isn't need for a Squeeze build though. What 32-bit stuff is getting built?

I guess that's because Wheezy introduces multiarch? The thing being built is QEMU.

Wei.

Quoted 22 lines
Quoted 21 lines
---
 ts-xen-build-prep |    4 ++++
 1 file changed, 4 insertions(+)
ts-xen-build-prep +4 -0
index b1e6a6c..66ba000 100755
@@ -175,6 +175,10 @@ sub prep () {
     target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
     # workaround for Debian #595728
 
+    if ($r{arch} eq 'amd64' && $ho->{Suite} =~ m/wheezy/) {
+	target_install_packages($ho, 'libc6-dev-i386');
+    }
+
     if ($ho->{Suite} !~ m/lenny/) {
         target_putfilecontents_stash($ho,60,<<'END','/home/osstest/.hgrc');
 [extensions]
#6 lore raw

Re: [PATCH OSSTEST RFC 3/5] Debian: install libc6-dev-i386 for Wheezy amd64

Quoted 10 lines · Wei Liu
Quoted 6 lines · Ian Campbell
Quoted 1 line · Wei Liu

Otherwise build-amd64 fails with missing header predefs.h.

This package exists in Squeeze too, it might simplify things to make it unconditional?

No, it doesn't exist in Lenny -- I see there's branches conditional on Lenny, do we still use that?

I think it is all Squeeze these days.

Quoted 6 lines
Quoted 3 lines

I wonder why it apparently isn't need for a Squeeze build though. What 32-bit stuff is getting built?

I guess that's because Wheezy introduces multiarch? The thing being built is QEMU.

We're building 64-bit qemu and failing because of a missing 32-bit header? That sounds like a bug somewhere, perhaps in the compiler/libc packaging or maybe in some part of our build system. How does the actual error look?

Ian.

author
#7 lore raw

Re: [PATCH OSSTEST RFC 3/5] Debian: install libc6-dev-i386 for Wheezy amd64

Quoted 5 lines · Ian Campbell
Quoted 4 lines · Wei Liu
Quoted 3 lines · Ian Campbell
Quoted 1 line · Wei Liu

Otherwise build-amd64 fails with missing header predefs.h.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

This package exists in Squeeze too, it might simplify things to make it unconditional?

No, it doesn't exist in Lenny -- I see there's branches conditional on Lenny, do we still use that?

I think it is all Squeeze these days.

OK.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

I wonder why it apparently isn't need for a Squeeze build though. What 32-bit stuff is getting built?

Quoted 8 lines
Quoted 2 lines

I guess that's because Wheezy introduces multiarch? The thing being built is QEMU.

We're building 64-bit qemu and failing because of a missing 32-bit header? That sounds like a bug somewhere, perhaps in the compiler/libc packaging or maybe in some part of our build system. How does the actual error look?

Oh sorrrry, it's not QEMU, -j32 and interleaving output fooled me. :-(

XEN_TARGET_ARCH=x86_32 make -f blowfish.mk all
make[5]: Entering directory `/local/scratch/xen.git/tools/tests/x86_emulator'
gcc -O1 -fno-omit-frame-pointer -m32 -march=i686 -g -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wdeclaration-after-statement -Wno-unused-but-set-variable -Wno-unused-local-typedefs -D__XEN_TOOLS__ -MMD -MF .blowfish.bin.d -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -fno-optimize-sibling-calls -mno-tls-direct-seg-refs -fno-stack-protector -fno-exceptions -fno-builtin -msoft-float -c blowfish.c
In file included from /usr/include/stdint.h:26:0,

                 from /usr/lib/gcc/x86_64-linux-gnu/4.7/include/stdint.h:3,
                 from blowfish.c:19:

/usr/include/features.h:323:26: fatal error: bits/predefs.h: No such file or directory compilation terminated.

Wei.

Quoted 1 line

Ian.

#8 lore raw

Re: [PATCH OSSTEST RFC 3/5] Debian: install libc6-dev-i386 for Wheezy amd64

Quoted 33 lines · Wei Liu
Quoted 5 lines · Ian Campbell
Quoted 4 lines · Wei Liu
Quoted 3 lines · Ian Campbell
Quoted 1 line · Wei Liu

Otherwise build-amd64 fails with missing header predefs.h.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

This package exists in Squeeze too, it might simplify things to make it unconditional?

No, it doesn't exist in Lenny -- I see there's branches conditional on Lenny, do we still use that?

I think it is all Squeeze these days.

OK.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

I wonder why it apparently isn't need for a Squeeze build though. What 32-bit stuff is getting built?

Quoted 8 lines
Quoted 2 lines

I guess that's because Wheezy introduces multiarch? The thing being built is QEMU.

We're building 64-bit qemu and failing because of a missing 32-bit header? That sounds like a bug somewhere, perhaps in the compiler/libc packaging or maybe in some part of our build system. How does the actual error look?

Oh sorrrry, it's not QEMU, -j32 and interleaving output fooled me. :-(

Quoted 8 lines

XEN_TARGET_ARCH=x86_32 make -f blowfish.mk all
make[5]: Entering directory `/local/scratch/xen.git/tools/tests/x86_emulator'
gcc -O1 -fno-omit-frame-pointer -m32 -march=i686 -g -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wdeclaration-after-statement -Wno-unused-but-set-variable -Wno-unused-local-typedefs -D__XEN_TOOLS__ -MMD -MF .blowfish.bin.d -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -fno-optimize-sibling-calls -mno-tls-direct-seg-refs -fno-stack-protector -fno-exceptions -fno-builtin -msoft-float -c blowfish.c
In file included from /usr/include/stdint.h:26:0,

                 from /usr/lib/gcc/x86_64-linux-gnu/4.7/include/stdint.h:3,
                 from blowfish.c:19:

/usr/include/features.h:323:26: fatal error: bits/predefs.h: No such file or directory compilation terminated.

Ah, OK, this is test code for the 32-bit instruction decoder/emulator. In principal this probably shouldn't be built if the compiler is not available, but in the meantime your fix looks sensible.

Ian

author
#9 lore raw

RFC PATCH 4/5 ts-debian-fixup: remove dead code

In reply to Wei Liu

Per c7910040 ("Osstest.pm, toolstack()->; always use "cfg", not separate xl configs"), we only pick the config file with .cfg suffix.

---
 ts-debian-fixup |   18 ------------------
 1 file changed, 18 deletions(-)
ts-debian-fixup +0 -18
index d8b51b7..f001418 100755
@@ -163,23 +163,6 @@ sub writecfg () {
     target_putfile_root($ho,10, $cfgstash, $cfgfile);
 }
 
-sub xlcfg () {
-    my $cfg_xend= target_var($gho,'cfgpath');
-    my $cfg_xl= "/etc/xen/$gho->{Name}.xl";
-    store_runvar("$gho->{Guest}_xlpath", $cfg_xl);
-    target_editfile_root($ho, $cfg_xend, undef, $cfg_xl, sub {
-        local ($/)= undef;
-        $_= <EI>;
-        s/^ [ \t]* memory [ \t]* \= [ \t]* ['"] (.+) ['"] [ \t]*
-          $/memory=$1/mx;
-        s/([[,])\s*\n/$1/g;
-        s/,\s*\]/\]/g;
-        s/([^#\n].*)\n/$1;\n/mg;
-        s/\n\n+/\n/g;
-        print EO or die $!;
-    });
-}
-
 savecfg();
 ether();
 target_kernkind_check($gho);
@@ -188,5 +171,4 @@ console();
 filesystems();
 otherfixupcfg();
 writecfg();
-xlcfg();
 unmount();
Signature · 1 line
1.7.10.4
author
#10 lore raw

RFC PATCH 5/5 ts-debian-fixup: workaround Debian bug #691320

In reply to Wei Liu
---
 ts-debian-fixup |   14 ++++++++++++++
 1 file changed, 14 insertions(+)
ts-debian-fixup +14 -0
index f001418..f62bb37 100755
@@ -118,6 +118,20 @@ sub otherfixupcfg () {
     $cfg =~ s/^vcpus.*//mg;
     $cfg .= "\nvcpus = $vcpus\n";
 
+    # Debian bug #691320
+    # Wheezy's xen-create-image writes incorrect memory= option in
+    # config file. This is fixed in newer version of xen-tools but it
+    # is not likely to go in in short term.
+    # On the other hand we should consider making xl support '512M',
+    # '2G' etc. But xend will still need this workaround until Wheezy
+    # is fixed.
+    # The default value 512 comes from ts-debian-install.
+    if ($ho->{Suite} =~ m/wheezy/) {
+	my $memory= guest_var($gho,'memory',512);
+	$cfg =~ s/^memory.*//mg;
+	$cfg .= "\nmemory = $memory\n";
+    }
+
     # PCI passthrough
     # Look for runvars   <gn>_pcipassthrough_<devtype>=<hostident>
     # and pass through all matching devices from the specified host.
Signature · 1 line
1.7.10.4
#11 lore raw

Re: [PATCH OSSTEST RFC 5/5] ts-debian-fixup: workaround Debian bug #691320

Quoted 11 lines · Wei Liu
---
 ts-debian-fixup |   14 ++++++++++++++
 1 file changed, 14 insertions(+)
ts-debian-fixup +0 -0
index f001418..f62bb37 100755
@@ -118,6 +118,20 @@ sub otherfixupcfg () {
     $cfg =~ s/^vcpus.*//mg;

ts-debian-fixup sub otherfixupcfg () { · line 128

Quoted 10 lines
ts-debian-fixup +8 -0
     $cfg .= "\nvcpus = $vcpus\n";
 
+    # Debian bug #691320
+    # Wheezy's xen-create-image writes incorrect memory= option in
+    # config file. This is fixed in newer version of xen-tools but it
+    # is not likely to go in in short term.
+    # On the other hand we should consider making xl support '512M',
+    # '2G' etc. But xend will still need this workaround until Wheezy
+    # is fixed.
+    # The default value 512 comes from ts-debian-install.

Wouldn't it be better to fix the invocation of xne-create-image to use a Mb suffix instead of M, AFAICT that would work on both Squeeze and Wheezy.

I think the fact that Squeeze's accepted just M was unintentional, although it's good that they are willing to fix it to be acceptable again...

Quoted 9 lines
ts-debian-fixup +6 -0
+    if ($ho->{Suite} =~ m/wheezy/) {
+	my $memory= guest_var($gho,'memory',512);
+	$cfg =~ s/^memory.*//mg;
+	$cfg .= "\nmemory = $memory\n";
+    }
+
     # PCI passthrough
     # Look for runvars   <gn>_pcipassthrough_<devtype>=<hostident>
     # and pass through all matching devices from the specified host.
author
#12 lore raw

Re: [PATCH OSSTEST RFC 5/5] ts-debian-fixup: workaround Debian bug #691320

Quoted 14 lines · Ian Campbell
Quoted 13 lines · Wei Liu
---
 ts-debian-fixup |   14 ++++++++++++++
 1 file changed, 14 insertions(+)
ts-debian-fixup +0 -0
index f001418..f62bb37 100755
@@ -118,6 +118,20 @@ sub otherfixupcfg () {
     $cfg =~ s/^vcpus.*//mg;
     $cfg .= "\nvcpus = $vcpus\n";
 
Quoted 13 lines

ts-debian-fixup sub otherfixupcfg () { · line 128

Quoted 8 lines
ts-debian-fixup +8 -0
+    # Debian bug #691320
+    # Wheezy's xen-create-image writes incorrect memory= option in
+    # config file. This is fixed in newer version of xen-tools but it
+    # is not likely to go in in short term.
+    # On the other hand we should consider making xl support '512M',
+    # '2G' etc. But xend will still need this workaround until Wheezy
+    # is fixed.
+    # The default value 512 comes from ts-debian-install.

Wouldn't it be better to fix the invocation of xne-create-image to use a Mb suffix instead of M, AFAICT that would work on both Squeeze and Wheezy.

OK. I tested passing 'Mb', it did work in Wheezy.

I will drop this one.

Wei.

Quoted 14 lines

I think the fact that Squeeze's accepted just M was unintentional, although it's good that they are willing to fix it to be acceptable again...

Quoted 9 lines
ts-debian-fixup +6 -0
+    if ($ho->{Suite} =~ m/wheezy/) {
+	my $memory= guest_var($gho,'memory',512);
+	$cfg =~ s/^memory.*//mg;
+	$cfg .= "\nmemory = $memory\n";
+    }
+
     # PCI passthrough
     # Look for runvars   <gn>_pcipassthrough_<devtype>=<hostident>
     # and pass through all matching devices from the specified host.
#13 lore raw
In reply to Wei Liu
Quoted 2 lines · Wei Liu

With the changes to Debian.pm and ts-debian-fixup I can now use Wheezy to run build-* jobs and test-amd64-amd64-{pv,xl}.

So I guess you've just been through the process of setting up osstest in standalone mode? I wonder if you could put that into a short blog or wiki doc as an illustration for people?

Ian.

author
#14 lore raw
Quoted 8 lines · Ian Campbell
Quoted 2 lines · Wei Liu

With the changes to Debian.pm and ts-debian-fixup I can now use Wheezy to run build-* jobs and test-amd64-amd64-{pv,xl}.

So I guess you've just been through the process of setting up osstest in standalone mode? I wonder if you could put that into a short blog or wiki doc as an illustration for people?

I'm actually writing a blog post on this topic, but still far from complete.

Wei.

Quoted 1 line

Ian.

#15 lore raw
Quoted 5 lines · Wei Liu
Quoted 4 lines · Ian Campbell
Quoted 2 lines · Wei Liu

With the changes to Debian.pm and ts-debian-fixup I can now use Wheezy to run build-* jobs and test-amd64-amd64-{pv,xl}.

Quoted 7 lines
Quoted 4 lines

So I guess you've just been through the process of setting up osstest in standalone mode? I wonder if you could put that into a short blog or wiki doc as an illustration for people?

I'm actually writing a blog post on this topic, but still far from complete.

Perfect, thanks! ;-)