Fixes for switching to Jessie as base OS for test hosts

Compared with last time there are a few new patches to fix things I noticed while testing and the prerequisites for actually deploying (f/w fix on arndale and git cache) are now addressed. It remains the case that the first 7 patches are OK to go in now and the final one to do the switch should perhaps be scheduled separately.

Ian.

root is not a "guest lv", so using target_guest_lv_name is misleading.

target_guest_lv_name also fails to properly handle the d-i vg name correctly, which di_vg_name does. Specifically under Jessie an extra "-vg" is appended to the hostname compared with Wheezy and Squeeze.

---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 153b375..9347c49 100644
@@ -131,7 +131,7 @@ sub uboot_common_kernel_bootargs ($)
 {
     my ($ho) = @_;
 
-    my $root= target_guest_lv_name($ho,"root");
+    my $root= lv_dev_mapper(di_vg_name($ho),"root");
     my $rootdelay= get_host_property($ho, "rootdelay");
 
     my @bootargs;

2.5.3

This has the effect of switching $gho->{Lvdev} from /dev/$VG/$LV to /dev/mapper/$VG-$LV (where $VG and $LV have an s/-/--/ transformation applied).

The two paths point to the samedevice and most call sites don't care about the distinction. Some places which use "kpartx $dev" to Lvdev expect to be able to append a partition number to $dev, while kpartx only creates the /dev/mapper form, meaning such places cannot use Lvdev. By making this switch we allow these places (such as ts-freebsd-install) to use kpartx.

The only other place I'm aware of which requires one form or the other is the Debian initramfs code which expects root=/dev/mapper/vg-lv and does not accept /dev/vg/lv. However this is already handled correctly.

---
v2: New(ish, posted in reply to v1)

---
 Osstest/TestSupport.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 09694f4..d8c5762 100644
@@ -1398,7 +1398,7 @@ sub guest_find_lv ($) {
     $gho->{Vg}= $r{"${gn}_vg"};
     $gho->{Lv}= $r{"${gn}_disk_lv"};
     $gho->{Lvdev}= (defined $gho->{Vg} && defined $gho->{Lv})
-        ? '/dev/'.$gho->{Vg}.'/'.$gho->{Lv} : undef;
+        ? lv_dev_mapper($gho->{Vg},$gho->{Lv}) : undef;
 }
 
 sub guest_find_diskimg($)

2.5.3

prepareguest has already assigned this so we should use it instead of replicating (perhaps wrongly since target_guest_lv_name and target_choose_vg can behave differently if multiple vgs are present).

$gho->{Lvdev} has been adjusted to return /dev/mapper/vg-lv paths which is required to be able to add partition numbers since kpartx does not create the /dev/vg/lv form.

---
 ts-freebsd-install | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index 98dad24..d1f4bd3 100755
@@ -49,7 +49,7 @@ sub prep () {
 
     target_put_guest_image($ho, $gho);
 
-    my $rootpartition_dev = target_guest_lv_name($ho, $gho->{Name}) . "--disk3";
+    my $root_part_nr = 3;
 
     target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
 

@@ -64,7 +64,7 @@ sub prep () {
 
             kpartx -a $gho->{Lvdev}
             mkdir -p $mnt
-            mount -t ufs -o ufstype=ufs2,rw $rootpartition_dev $mnt
+            mount -t ufs -o ufstype=ufs2,rw $gho->{Lvdev}${root_part_nr} $mnt
 END
 
     target_cmd_root($ho, <<END, 15);

2.5.3


_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

---
 Osstest/TestSupport.pm | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

index d8c5762..69da459 100644
@@ -63,7 +63,7 @@ BEGIN {
                       target_install_packages target_install_packages_norec
                       target_jobdir target_extract_jobdistpath_subdir
                       target_extract_jobdistpath
-                      lv_dev_mapper target_guest_lv_name
+                      lv_dev_mapper
 
                       poll_loop tcpconnect await_tcp
                       contents_make_cpio file_simple_write_contents

@@ -721,11 +721,6 @@ sub lv_dev_mapper ($$) {
     return "/dev/mapper/$vg-$lv";
 }    
 
-sub target_guest_lv_name($$) {
-    my ($ho, $lv) = @_;
-    return lv_dev_mapper("$ho->{Name}", $lv);
-}
-
 #---------- dhcp watching ----------
 
 sub dhcp_watch_setup ($$) {

2.5.3

This is not enabled in the arm defconfig (it is in x86) and without it a Debian Jessie userspace with systemd fails to boot with:

systemd[1]: Failed to mount tmpfs at /sys/fs/cgroup: No such file or directory

While we arrange to use sysvinit in the host and in d-i installed guests we do not do so for xen-create-image guests.

Arranging to install sysvinit-core during xen-create-image looks to be non-trivial. I hope it won't be necessary.

---
v2: New patch

---
 ts-kernel-build | 3 +++
 1 file changed, 3 insertions(+)

index a16527b..6d59e6e 100755
@@ -549,6 +549,9 @@ setopt CONFIG_BLK_CPQ_CISS_DA m
 
 setopt CONFIG_FHANDLE y
 
+# Needed for Jessie+ userspace when systemd is used.
+setopt CONFIG_CGROUPS y
+
 # Needed for efibootmgr
 setopt CONFIG_EFI_VARS y
 

2.5.3

Ian Campbell writes ("[PATCH OSSTEST v2 5/8] ts-kernel-build: Explicitly enable CONFIG_CGROUPS"):

If we do decide it's necessary, ISTR that telling x-c-i to add specific additional packages is not too hard.

Ian.

Thanks.

Really? I did look and the advice I saw on the Internet seemed to revolve around adding wrappers around debootstrap and telling x-c-i to use them.

Ian Campbell writes ("Re: [PATCH OSSTEST v2 5/8] ts-kernel-build: Explicitly enable CONFIG_CGROUPS"):

The --role= option seems to provide a suiteable hook. I looked at /etc/xen-tools/role.d/builder and it seems a reasonable template.

Anyway, we can cross this bridge when we come to it.

Ian.

The x86 and arm kernels are inconsistently named upstream, and then renamed in mg-debian-installer-update as:

            ====== KERNEL ============  ====== INITRD ==================
            Debian	   Osstest	Debian		   Osstest
            -----------    -----------  ---------------    -------------
x86/native: linux	=> linux	initrd.gz	=> initrd.gz
x86/xen:    xen/vmlinuz	=> vmlinuz-xen	xen/initrd.gz	=> initrd.gz-xen
arm/native: vmlinuz	=> linux	initrd.gz	=> initrd.gz
arm/xen:    vmlinuz	=> linux	initrd.gz	=> initrd.gz

Cope with this here, by treating the x86/xen (PV) case as the special case that it is (due to the requirement of a separate i686-pae installer kernel).

This goes unnoticed in the distros-debian flights because the kernel is downloaded at runtime via a runvar. It matters once regular (non-distro-debian) flights run with Jessie because then the d-i which is used by the di based tests is taken from the result of mg-debian-installer-update (e.g. for the -qcow2 flights etc).

---
v2: New patch

---
 ts-debian-di-install | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

index 9905691..64b5d44 100755
@@ -153,10 +153,13 @@ sub setup_netboot($$$)
 
 	my $di_path = $c{TftpPath}.'/'.$ho->{Tftp}{DiBase}.'/'.${arch}.'/'.$c{TftpDiVersion}.'-'.$ho->{Suite};
 
-        my $suffix = '';
-        $suffix .= "-xen" if ${arch} =~ m/amd64|i386/;
-	$kernel = "$di_path/vmlinuz$suffix";
-	$ramdisk = "$di_path/initrd.gz$suffix";
+        if (${arch} =~ m/amd64|i386/) {
+	    $kernel = "$di_path/vmlinuz-xen";
+	    $ramdisk = "$di_path/initrd.gz-xen";
+        } else {
+	    $kernel = "$di_path/linux";
+	    $ramdisk = "$di_path/initrd.gz";
+        }
 
 	target_putfile_root($ho, 60, $kernel, "$didir/kernel_${suite}_${arch}");
 	target_putfile_root($ho, 60, $ramdisk, "$didir/ramdisk_${suite}_${arch}");

2.5.3

Ian Campbell writes ("[PATCH OSSTEST v2 6/8] ts-debian-di-install: Use correct per-arch name for kernel."):

This is "Sync mode. Don't return until the partitions are created", which seems to be needed in Jessie. The option also exists in Wheezy, according to the manpage.

Without this the following mount fails having apparently raced against the creation of the device nodes.

---
v2: New patch

---
 ts-freebsd-install | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index d1f4bd3..d630f2a 100755
@@ -62,7 +62,7 @@ sub prep () {
             dd if=$rimagebase.raw of=$gho->{Lvdev} bs=1M
             rm $rimagebase.raw
 
-            kpartx -a $gho->{Lvdev}
+            kpartx -s -a $gho->{Lvdev}
             mkdir -p $mnt
             mount -t ufs -o ufstype=ufs2,rw $gho->{Lvdev}${root_part_nr} $mnt
 END

2.5.3

Ian Campbell writes ("[PATCH OSSTEST v2 7/8] ts-freebsd-install: Pass -s option to kpartx"):

How exciting. (Why on earth is `sync mode' not the default? Don't get me started...)

Ian.

It's always good to have plenty of rope. No, wait, that's mountaineering.

Ian.

mg-debian-installer-update-all has been run on the production instance and TftpDiVersion is also updated to match.

The resulting binaries have also been copied to the Cambridge instance, so update Cambridge config too.

---

---
 Osstest.pm                  | 2 +-
 production-config           | 2 +-
 production-config-cambridge | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

index 4a763c6..fc8334d 100644
@@ -73,7 +73,7 @@ our %c = qw(
 
     Images images
 
-    DebianSuite wheezy
+    DebianSuite jessie
     DebianMirrorSubpath debian
 
     TestHostKeypairPath id_rsa_osstest

index bb8d224..b21153b 100644
@@ -87,7 +87,7 @@ TftpPxeTemplatesReal pxelinux.cfg/%ipaddrhex%
 
 TftpPxeGroup osstest
 # Update with ./mg-debian-installer-update(-all)
-TftpDiVersion 2015-09-07
+TftpDiVersion 2015-09-18
 
 # These should normally be the same.
 # Update with ./mg-cpu-microcode-update

index f801303..5d68ede 100644
@@ -69,7 +69,7 @@ TftpPxeTemplates %name%/pxelinux.cfg
 TftpPxeTemplatesReal pxelinux.cfg/%ipaddrhex%
 
 TftpPxeGroup osstest
-TftpDiVersion 2015-09-07
+TftpDiVersion 2015-09-18
 
 # These should normally be the same.
 MicrocodeUpdateAmd64 microcode.x86.2015-06-12.cpio

2.5.3

Here is a 9/8. This should be deferred and go in along with the final patch to switch to Jessie, i.e. after the rest are in production with Wheezy.

8>----------

From: Ian Campbell <[email protected]>
Date: Wed, 7 Oct 2015 14:04:52 +0100
Subject: [PATCH] make-flight: Use Jessie ISO image for Debian HVM tests

---
Deployment note: debian-8.2.0-{amd64,i386}-CD-1.iso already added to images dir in Colo & Cambridge.

---
 make-flight | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 0291b5f..ed4e6ce 100755
@@ -284,7 +284,7 @@ do_hvm_debian_test_one () {
     test-debianhvm $toolstack $xenarch $dom0arch $qemuu_runvar \
     enable_xsm=$xsm                             \
     $stubdom_runvar $testvars                   \
-    debianhvm_image=debian-7.2.0-$arch-CD-1.iso \
+    debianhvm_image=debian-8.2.0-$arch-CD-1.iso \
     debianhvm_iso_kernel=/$iso_dir/vmlinuz \
     debianhvm_iso_ramdisk=/$iso_dir/initrd.gz \
     bios=$bios \

2.5.3