Fixes for switching to Jessie as base OS for test hosts
cover Fixes for switching to Jessie as base OS for test hosts
5 of these patches are fixes for issues which I noticed when trying to do an adhoc xen-unstable flight with Jessie but which are safe to go in whenever (i.e. they work with Wheezy too).
The 6th patch actually switches to Jessie which we will want to schedule a bit more carefully. In particular this requires:
* Switch to apt-cacher-ng on the cache host, since a bug in Jessie's apt apparently interacts badly with apt-cacher's handling of Ranges in http requests[0]. I've deployed apt-cacher-ng on an alternative port on the production cache and tested that it works with Jessie and Wheezy.
* A firmware upgrade on the Arndale boards, since the new version of gcc seems to trigger an SoC errata when building the kernel.
The reason I care about upgrading to Jessie is that this adds the arm64 architecture which we hope to eventually have hardware for.
Roger, I will only CC you on the ts-install-freebsd script change.
Ian.
[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=795284
PATCH 1/6 Debian: Uninstall flash-kernel when creating our own boot.scr
flash-kernel will run from various kernel postinst hooks and overwrite our own boot scripts. While this might be tollerable for the initial installation we don't want to risk it occuring after we have created our own boot.scr to boot xen.
dpkg --purge succeeds if the package wasn't installed.
This happened to show up with Jessie since it now supports the two boards in our test lab while Wheezy didn't (so flash-kernel didn't know about them and did nothing).
--- Osstest/Debian.pm | 2 ++ 1 file changed, 2 insertions(+)
Osstest/Debian.pm +2 -0
@@ -1106,6 +1106,8 @@ echo Booting bootz \\\${kernel_addr_r} \\\${ramdisk_addr_r}:\\\${filesize} \\\${fdt_addr} EOF +in-target dpkg --purge flash-kernel + in-target mkimage -A arm -T script -d /boot/boot.deb /boot/boot.scr.deb in-target cp /boot/boot.scr.deb /boot/boot.scr END
Signature · 1 line
2.5.3
Re: [PATCH OSSTEST 1/6] Debian: Uninstall flash-kernel when creating our own boot.scr
Ian Campbell writes ("[PATCH OSSTEST 1/6] Debian: Uninstall flash-kernel when creating our own boot.scr"):
Quoted 4 lines
flash-kernel will run from various kernel postinst hooks and overwrite our own boot scripts. While this might be tollerable for the initial installation we don't want to risk it occuring after we have created our own boot.scr to boot xen.
PATCH 2/6 Debian: Search for kernel in /boot as well as / when making u-boot script
In reply to Ian CampbellThe vmlinuz and initrd.img symlinks appear to have moved to /boot when installing Jessie on armhf systems compared to Wheezy.
--- Osstest/Debian.pm | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
Osstest/Debian.pm +15 -2
@@ -1092,8 +1092,21 @@ set -ex r=/target #/ -kernel=`readlink \$r/vmlinuz | sed -e 's|boot/||'` -initrd=`readlink \$r/initrd.img | sed -e 's|boot/||'` +if [ -f \$r/vmlinuz ] ; then + echo "Kernel+initrd symlinks are in /" + kernel=`readlink \$r/vmlinuz | sed -e 's|boot/||'` + initrd=`readlink \$r/initrd.img | sed -e 's|boot/||'` +elif [ -f \$r/boot/vmlinuz ] ; then + echo "Kernel+initrd symlinks are in /boot" + kernel=`readlink \$r/boot/vmlinuz` + initrd=`readlink \$r/boot/initrd.img` +else + echo "No kernel found!" + exit 1 +fi + +echo Using kernel \$kernel +echo Using initrd \$initrd cat >\$r/boot/boot.deb <<EOF setenv bootargs $bootargs
Signature · 1 line
2.5.3
Re: [PATCH OSSTEST 2/6] Debian: Search for kernel in /boot as well as / when making u-boot script
Ian Campbell writes ("[PATCH OSSTEST 2/6] Debian: Search for kernel in /boot as well as / when making u-boot script"):
Quoted 2 lines
The vmlinuz and initrd.img symlinks appear to have moved to /boot when installing Jessie on armhf systems compared to Wheezy.
How inconvenient.
PATCH 3/6 Debian: uboot: Use di_vg_name() and lv_dev_mapper() for root=
In reply to Ian Campbellroot is not a "guest lv", so using target_guest_lv_name is misleading.
target_guest_lv_name also fails to properly handle the guest vg name correctly, which di_vg_name does.
--- Osstest/Debian.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
@@ -130,7 +130,7 @@ sub uboot_common_kernel_bootargs ($) { my ($ho) = @_; - my $root= target_guest_lv_name($ho,"root"); + my $root= lv_dev_mapper(di_vg_name($ho),"root"); my $rootdelay= get_host_property($ho, "rootdelay"); my @bootargs;
Signature · 1 line
2.5.3
Re: [PATCH OSSTEST 3/6] Debian: uboot: Use di_vg_name() and lv_dev_mapper() for root=
Ian Campbell writes ("[PATCH OSSTEST 3/6] Debian: uboot: Use di_vg_name() and lv_dev_mapper() for root="):
Quoted 4 lines
root is not a "guest lv", so using
target_guest_lv_nameis misleading.
target_guest_lv_namealso fails to properly handle the guest vg name correctly, whichdi_vg_namedoes.
WDYM `the guest vg name' ? If it's not a guest lv then is it a guest vg ? Sorry if I am simply confused.
Ian.
Re: [PATCH OSSTEST 3/6] Debian: uboot: Use di_vg_name() and lv_dev_mapper() for root=
Quoted 9 lines · Ian Jackson
Ian Campbell writes ("[PATCH OSSTEST 3/6] Debian: uboot: Use
di_vg_name()andlv_dev_mapper()for root="):Quoted 4 lines
root is not a "guest lv", so using
target_guest_lv_nameis misleading.
target_guest_lv_namealso fails to properly handle the guest vg name correctly, whichdi_vg_namedoes.WDYM `the guest vg name' ? If it's not a guest lv then is it a guest vg ?
Sorry, I think I meant "the d-i vg name". Specifically the default VG name created by D-I changed in Jessie to append "-vg" to the hostname.
i.e. under Squeeze and Wheezy a machine foo would end up with /dev/foo/root while under Jessie it becomes /dev/foo-vg/root.
Quoted 1 line
Sorry if I am simply confused.
I think I said something unhelpfully confusing and wrong.
Ian.
Ian Jackson
Acked-byRe: [PATCH OSSTEST 3/6] Debian: uboot: Use di_vg_name() and lv_dev_mapper() for root=
Ian Campbell writes ("Re: [PATCH OSSTEST 3/6] Debian: uboot: Use di_vg_name() and lv_dev_mapper() for root="):
Quoted 5 lines · Ian Jackson
Quoted 4 lines
Ian Campbell writes ("[PATCH OSSTEST 3/6] Debian: uboot: Use
di_vg_name()andlv_dev_mapper()for root="):Quoted 2 lines
root is not a "guest lv", so using
target_guest_lv_nameis misleading.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
target_guest_lv_namealso fails to properly handle the guest vg name correctly, whichdi_vg_namedoes.WDYM `the guest vg name' ? If it's not a guest lv then is it a guest vg ?
Sorry, I think I meant "the d-i vg name". Specifically the default VG name created by D-I changed in Jessie to append "-vg" to the hostname.
Right.
With that one-word change to the commit message:
Quoted 2 lines
i.e. under Squeeze and Wheezy a machine foo would end up with
/dev/foo/rootwhile under Jessie it becomes/dev/foo-vg/root.
Aha. (You might mention that too in the commit message.)
Ian.
PATCH 4/6 ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name
In reply to Ian Campbellprepareguest has already assigned this so we should use it instead of replicating (perhaps wrongly since target_guest_lv_name and target_choose_vg can behave differently if multiple vgs are present).
--- ts-freebsd-install | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
ts-freebsd-install +2 -2
@@ -49,7 +49,7 @@ sub prep () { target_put_guest_image($ho, $gho); - my $rootpartition_dev = target_guest_lv_name($ho, $gho->{Name}) . "--disk3"; + my $root_part_nr = 3; target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
@@ -64,7 +64,7 @@ sub prep () { kpartx -a $gho->{Lvdev} mkdir -p $mnt - mount -t ufs -o ufstype=ufs2,rw $rootpartition_dev $mnt + mount -t ufs -o ufstype=ufs2,rw $gho->{Lvdev}${root_part_nr} $mnt END target_cmd_root($ho, <<END, 15);
Signature · 7 lines
2.5.3 _______________________________________________ Xen-devel mailing list [email protected] http://lists.xen.org/xen-devel
Re: [PATCH OSSTEST 4/6] ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name
Ian Campbell writes ("[PATCH OSSTEST 4/6] ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name"):
Quoted 3 lines
prepareguest has already assigned this so we should use it instead of replicating (perhaps wrongly since
target_guest_lv_nameandtarget_choose_vgcan behave differently if multiple vgs are present).
Re: [PATCH OSSTEST 4/6] ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name
In reply to Ian CampbellEl 29/09/15 a les 11.44, Ian Campbell ha escrit:
Quoted 6 lines
prepareguest has already assigned this so we should use it instead of replicating (perhaps wrongly since
target_guest_lv_nameandtarget_choose_vgcan behave differently if multiple vgs are present).Signed-off-by Ian Campbell <[email protected]>Cc Roger Pau Monné <[email protected]>
I'm not officially the maintainer of this, but for the record:
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH OSSTEST 4/6] ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name
Roger Pau Monné writes ("Re: [PATCH OSSTEST 4/6] ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name"):
Quoted 11 lines
El 29/09/15 a les 11.44, Ian Campbell ha escrit:
Quoted 6 lines
prepareguest has already assigned this so we should use it instead of replicating (perhaps wrongly since
target_guest_lv_nameandtarget_choose_vgcan behave differently if multiple vgs are present).Signed-off-by Ian Campbell <[email protected]>Cc Roger Pau Monné <[email protected]>I'm not officially the maintainer of this, but for the record:
Acked-by Roger Pau Monné <[email protected]>
Your review and ack are very welcome, thanks.
Ian.
Re: [PATCH OSSTEST 4/6] ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name
In reply to Ian CampbellQuoted 3 lines · Ian Campbell
prepareguest has already assigned this so we should use it instead of replicating (perhaps wrongly since
target_guest_lv_nameandtarget_choose_vgcan behave differently if multiple vgs are present).
This is wrong in practice because
kpartx -a /dev/some-vg/some-device
produces /dev/mapper/some--vg-some--device1 etc but not /dev/some-vg/some -device1.
The right fix probably involves lv_dev_mapper($gho-{Vg}, "$gho->{Lv}3") instead.
I'm in two minds about having guest_find_lv use lv_dev_mapper to populate $gho->{Lvdev} in the first place as an alternative fix though.
Quoted 30 lines
Signed-off-by Ian Campbell <[email protected]>Cc Roger Pau Monné <[email protected]>--- ts-freebsd-install | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)ts-freebsd-install +2
-3@@ -49,7 +49,7 @@ sub prep () { target_put_guest_image($ho, $gho); - my $rootpartition_dev = target_guest_lv_name($ho, $gho->{Name}) . "- -disk3"; + my $root_part_nr = 3; target_cmd_root($ho, "umount $gho->{Lvdev} ||:");@@ -64,7 +64,7 @@ sub prep () { kpartx -a $gho->{Lvdev} mkdir -p $mnt - mount -t ufs -o ufstype=ufs2,rw $rootpartition_dev $mnt + mount -t ufs -o ufstype=ufs2,rw $gho->{Lvdev}${root_part_nr}$mnt
END
Quoted 1 line
target_cmd_root($ho, <<END, 15);
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH OSSTEST 4/6] ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name
Quoted 9 lines · Ian Campbell
Quoted 3 lines · Ian Campbell
prepareguest has already assigned this so we should use it instead of replicating (perhaps wrongly since
target_guest_lv_nameandtarget_choose_vgcan behave differently if multiple vgs are present).This is wrong in practice because
kpartx -a/dev/some-vg/some-device
produces/dev/mapper/some--vg-some--device1etc but not/dev/some-vg/some-device1.
And in case it wasn't clear ->{Lvdev} contains /dev/some-vg/some-device.
Quoted 39 lines
The right fix probably involves
lv_dev_mapper($gho-{Vg}, "$gho->{Lv}3")instead.I'm in two minds about having
guest_find_lvuselv_dev_mapperto populate $gho->{Lvdev} in the first place as an alternative fix though.Quoted 32 lines
Signed-off-by Ian Campbell <[email protected]>Cc Roger Pau Monné <[email protected]>--- ts-freebsd-install | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)ts-freebsd-install +0
-1@@ -49,7 +49,7 @@ sub prep () { target_put_guest_image($ho, $gho); - my $rootpartition_dev = target_guest_lv_name($ho, $gho->{Name}) ."-
ts-freebsd-install +2
-3-disk3"; + my $root_part_nr = 3; target_cmd_root($ho, "umount $gho->{Lvdev} ||:");@@ -64,7 +64,7 @@ sub prep () { kpartx -a $gho->{Lvdev} mkdir -p $mnt - mount -t ufs -o ufstype=ufs2,rw $rootpartition_dev $mnt + mount -t ufs -o ufstype=ufs2,rw $gho ->{Lvdev}${root_part_nr}$mnt
END
Quoted 6 lines
Quoted 1 line
target_cmd_root($ho, <<END, 15);_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH OSSTEST 4/6] ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name
In reply to Ian CampbellQuoted 10 lines · Ian Campbell
Quoted 3 lines · Ian Campbell
prepareguest has already assigned this so we should use it instead of replicating (perhaps wrongly since
target_guest_lv_nameandtarget_choose_vgcan behave differently if multiple vgs are present).This is wrong in practice because
kpartx -a/dev/some-vg/some-device
produces/dev/mapper/some--vg-some--device1etc but not/dev/some-vg/some-device1.
Quoted 5 lines
The right fix probably involves
lv_dev_mapper($gho-{Vg}, "$gho->{Lv}3")instead.I'm in two minds about having
guest_find_lvuselv_dev_mapperto populate $gho->{Lvdev} in the first place as an alternative fix though.
I decided I liked this route more. It's as yet untested other than "perl -c", but I have inserted the patch below before this one.
commit 9ba9a4f1d3fb7f01a1c76e9bff9f4b23187d855b
Author: Ian Campbell <[email protected]> Date: Fri Oct 2 10:31:13 2015 +0100
TestSupport: Use lv_dev_mapper in guest_find_lv
This has the effect of switching $gho->{Lvdev} from /dev/$VG/$LV to /dev/mapper/$VG-$LV (where $VG and $LV have an s/-/--/ transformation applied).
The two paths point to the samedevice and most call sites don't care about the distinction. Some places which use "kpartx $dev" to Lvdev expect to be able to append a partition number to $dev, while kpartx only creates the /dev/mapper form, meaning such places cannot use Lvdev. By making this switch we allow these places (such as ts-freebsd-install) to use kpartx.
The only other place I'm aware of which requires one form or the other is the Debian initramfs code which expects root=/dev/mapper/vg-lv and does not accept /dev/vg/lv. However this is already handled correctly.
Osstest/TestSupport.pm +1 -1
@@ -1382,7 +1382,7 @@ sub guest_find_lv ($) { $gho->{Vg}= $r{"${gn}_vg"}; $gho->{Lv}= $r{"${gn}_disk_lv"}; $gho->{Lvdev}= (defined $gho->{Vg} && defined $gho->{Lv}) - ? '/dev/'.$gho->{Vg}.'/'.$gho->{Lv} : undef; + ? lv_dev_mapper($gho->{Vg},$gho->{Lv}) : undef; } sub guest_find_diskimg($)
Ian Jackson
Acked-byRe: [PATCH OSSTEST 4/6] ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name
Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST 4/6] ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name"):
Quoted 6 lines · Ian Campbell
Quoted 2 lines
I'm in two minds about having
guest_find_lvuselv_dev_mapperto populate $gho->{Lvdev} in the first place as an alternative fix though.I decided I liked this route more. It's as yet untested other than "perl -c", but I have inserted the patch below before this one.
Right.
Quoted 5 lines
commit
9ba9a4f1d3fb7f01a1c76e9bff9f4b23187d855bAuthor: Ian Campbell <[email protected]> Date: Fri Oct 2 10:31:13 2015 +0100TestSupport: Use
lv_dev_mapperinguest_find_lv
Thanks,
Ian.
PATCH 5/6 TestSupport: Remove now unused target_guest_lv_name.
In reply to Ian Campbell--- Osstest/TestSupport.pm | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
Osstest/TestSupport.pm +1 -6
@@ -62,7 +62,7 @@ BEGIN { target_install_packages target_install_packages_norec target_jobdir target_extract_jobdistpath_subdir target_extract_jobdistpath - lv_dev_mapper target_guest_lv_name + lv_dev_mapper poll_loop tcpconnect await_tcp contents_make_cpio file_simple_write_contents
@@ -710,11 +710,6 @@ sub lv_dev_mapper ($$) { return "/dev/mapper/$vg-$lv"; } -sub target_guest_lv_name($$) { - my ($ho, $lv) = @_; - return lv_dev_mapper("$ho->{Name}", $lv); -} - #---------- dhcp watching ---------- sub dhcp_watch_setup ($$) {
Signature · 1 line
2.5.3
Re: [PATCH OSSTEST 5/6] TestSupport: Remove now unused target_guest_lv_name.
Ian Campbell writes ("[PATCH OSSTEST 5/6] TestSupport: Remove now unused target_guest_lv_name."):
Quoted 1 line
Signed-off-by Ian Campbell <[email protected]>
PATCH 6/6 Switch to Debian 8.0 (jessie) as OS for test hosts
In reply to Ian Campbellmg-debian-installer-update-all has been run on the production instance and TftpDiVersion is also updated to match.
The resulting binaries have also been copied to the Cambridge instance, so update Cambridge config too.
---
--- Osstest.pm | 2 +- production-config | 2 +- production-config-cambridge | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
Osstest.pm +1 -1
@@ -73,7 +73,7 @@ our %c = qw( Images images - DebianSuite wheezy + DebianSuite jessie DebianMirrorSubpath debian TestHostKeypairPath id_rsa_osstest
production-config +1 -1
@@ -87,7 +87,7 @@ TftpPxeTemplatesReal pxelinux.cfg/%ipaddrhex% TftpPxeGroup osstest # Update with ./mg-debian-installer-update(-all) -TftpDiVersion 2015-09-07 +TftpDiVersion 2015-09-18 # These should normally be the same. # Update with ./mg-cpu-microcode-update
production-config-cambridge +1 -1
@@ -69,7 +69,7 @@ TftpPxeTemplates %name%/pxelinux.cfg TftpPxeTemplatesReal pxelinux.cfg/%ipaddrhex% TftpPxeGroup osstest -TftpDiVersion 2015-09-07 +TftpDiVersion 2015-09-18 # These should normally be the same. MicrocodeUpdateAmd64 microcode.x86.2015-06-12.cpio
Signature · 1 line
2.5.3
Re: [PATCH OSSTEST 6/6] Switch to Debian 8.0 (jessie) as OS for test hosts
Ian Campbell writes ("[PATCH OSSTEST 6/6] Switch to Debian 8.0 (jessie) as OS for test hosts"):
Quoted 5 lines
mg-debian-installer-update-all has been run on the production instance and TftpDiVersion is also updated to match.
The resulting binaries have also been copied to the Cambridge instance, so update Cambridge config too.
Jolly good.
Subject to the prerequisites,
Quoted 4 lines · Ian Campbell
* Switch to apt-cacher-ng on the cache host, since a bug in Jessie's apt apparently interacts badly with apt-cacher's handling of Ranges in http requests[0]. I've deployed apt-cacher-ng on an alternative port on the production cache and tested that it works with Jessie and Wheezy.
I was assuming we'd want to switch out apt-cacher for apt-cacher-ng on port cache:3142. Instead you suggested just changing the config to use the alternative port 3143.
-----<8-------
Quoted 1 line
From
b9558e5202b6ade9dc91f0ebf5eac2ff94a57ba4Mon Sep 17 00:00:00 2001
From: Ian Campbell <[email protected]>
Date: Tue, 29 Sep 2015 11:24:53 +0100
Subject: [PATCH] production-config: Switch DebianMirrorProxy to port 3143
This is running apt-cacher-ng rather than apt-cacher (which remains on port 3142 for the time being). It seems that apt-cacher exposes a bug (#795284) in Jessie's version of apt.
apt-cacher-ng also seems more popular these days.
---
The arrangements for caching of the Debian mirror in the Cambridge instance are different and so no change is required.
--- production-config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
production-config +1 -1
@@ -103,7 +103,7 @@ XenUseUser osstest # We use the IP address because Citrix can't manage reliable nameservice #DebianMirrorHost debian.uk.xensource.com #DebianMirrorHost 10.80.16.196 -DebianMirrorProxy http://cache:3142/ +DebianMirrorProxy http://cache:3143/ HostProp_NtpServer infra.test-lab.xenproject.org
Signature · 1 line
2.5.3
Ian Jackson
Acked-byIan Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST 0/6] Fixes for switching to Jessie as base OS for test hosts"):
Quoted 9 lines · Ian Campbell
Quoted 4 lines
* Switch to apt-cacher-ng on the cache host, since a bug in Jessie's apt apparently interacts badly with apt-cacher's handling of Ranges in http requests[0]. I've deployed apt-cacher-ng on an alternative port on the production cache and tested that it works with Jessie and Wheezy.
I was assuming we'd want to switch out apt-cacher for apt-cacher-ng on port cache:3142. Instead you suggested just changing the config to use the alternative port 3143.
Ah here we are.
Ian Campbell writes ("[PATCH OSSTEST 0/6] Fixes for switching to Jessie as base OS for test hosts"):
Quoted 4 lines
* Switch to apt-cacher-ng on the cache host, since a bug in Jessie's apt apparently interacts badly with apt-cacher's handling of Ranges in http requests[0]. I've deployed apt-cacher-ng on an alternative port on the production cache and tested that it works with Jessie and Wheezy.
Did we not decide to do this with a change to production-config ?
Ian.
Quoted 2 lines · Ian Campbell
* A firmware upgrade on the Arndale boards, since the new version of gcc seems to trigger an SoC errata when building the kernel.
Now done.
Quoted 2 lines
The reason I care about upgrading to Jessie is that this adds the arm64 architecture which we hope to eventually have hardware for.
I forgot that this will(/should) also enable the -vhd/-raw and -qcow2 tests on ARM (since they rely on d-i for guest install and only Jessie onwards can run in an ARM domU).
Ian.