Switch to Debian buster (= Debian stable)
v2 cover Switch to Debian buster (= Debian stable)
This series looks about as ready as it is going to be. Unfortunately there are still two issues, each of which cropped up once in my final formal retest. See below.
What are people's opinions? Should I push this to osstest pretest soon after the Xen codefreeze (eg, after we get the first push after freeze) ?
The downside would be introducing new low-probability heisenbugs. But in my tests it seems that some existing heisenbugs may be abated. It's hard to say for sure. The upside is a lower risk that Debian will break our stuff and also being able to commission a number of machines which do not work with stretch (Debian oldstable).
The two issues:
1. In one test, libvirt segfaulted. This is with osstest's own-built tested version of libvirt; osstest's own-build tested version of Linux; and osstest's own-built xen.git#master.
http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-i386-libvirt/info.html
http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-i386-libvirt/rimava1---var-log-kern.log.gz
| May 28 21:39:11 rimava1 kernel: [ 1146.326593] libvirtd[16868]: segfault at 820 ip b72535b0 sp abce5090 error 4 in libc-2.28.so[b71f0000+14e000]
2. In one test, the Debian installer hung. The kernel printed many soft lockup messages. This is with osstest's own dom0 kernel and hypervisor, but the 4.19.67-2+deb10u1 kernel from Debian.
http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-amd64-xl-qemut-debianhvm-i386-xsm/info.html
http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-amd64-xl-qemut-debianhvm-i386-xsm/godello1---var-log-xen-qemu-dm-debianhvm.guest.osstest.log.gz
Ian Jackson (49):
ts-logs-capture: Cope if xl shutdown leaves domain running for a bit
ts-xen-build-prep: Install rsync
lvcreate argments: pass --yes -Z y -W y
TestSupport: allow more time for apt
Booting: Use --' rather than ---' to introduce host cmdline
di_installcmdline_core: Pass locale on d-i command line
setupboot_grub2: Drop $submenu variable
ts-leak-check: Ignore buster's udevd too
Bodge systemd random seed arrangements
Debian guests made with xen-tools: Write systemd random seed file
ts-debian-di-install: Provide guest with more RAM
Debian: preseed: use priority= alias
Debian: Specify `priority=critical' rather than locale Honour 'LinuxSerialConsole <suite>' host property
buster: make-hosts-flight: Add to possible suites for hosts flight
buster: Extend grub2 uefi no install workaround
buster: ts-host-install: Extend net.ifnames workaround
buster: Deinstall the "systemd" package
buster: preseed partman-auto-lvm/guided_size
buster: ts-host-install: NTP not honoured bug remains
buster: Extend ARM clock workaround
buster: Extend guest bootloader workaround
Honour DebianImageFile_SUITE_ARCH
buster: Specify DebianImageFile_SUITE_ARCH
20_linux_xen: Copy Debian buster version into our initramfs area 20_linux_xen: Adhoc template substitution
20_linux_xen: Ignore xenpolicy and config files too 20_linux_xen: Support Xen Security Modules (XSM/FLASK) mg-debian-installer-update: support overlay-intramfs-SUITE overlay-initrd-buster/sbin/reopen-console: Copy from Debian overlay-initrd-buster/sbin/reopen-console: Fix #932416
buster: chiark-scripts: Install a new version on buster too
buster: Provide TftpDiVersion
buster: grub, arm64: extend chainloading workaround
buster: setupboot_grub2: Note what files exist in /boot
buster: setupboot_grub2: Handle missing policy file bug
buster: Extend workaround for dhcpd EROFS bug
ts-xen-install: Add $ho argument to some_extradebs
ts-xen-install: Move some_extradebs to Debian.pm Debian.pm: Break out standard_extradebs
Debian.pm: Move standard_extradebs to ts-host-install
buster: Install own linux-libc-dev package (!)
setupboot_grub2: Insist on space after directives
setupboot_grub2: Print line number of entry we are using
setupboot_grub2: Recognise --nounzip for initramfs
setupboot_grub2: Copy hv command line from grub to xen.cfg
setupboot_grub2: Do not boot with XSM policy etc. unless xsm=1 buster 20_linux_xen: Only load policy in XSM-enabled builds
buster: Switch to Debian buster as the default suite
Osstest.pm | 2 +- Osstest/Debian.pm | 128 +++++-- Osstest/TestSupport.pm | 15 +- make-hosts-flight | 2 +- mfi-common | 9 + mg-debian-installer-update | 20 ++ overlay-buster/etc/grub.d/20_linux_xen | 327 ++++++++++++++++++ overlay-initrd-buster/sbin/reopen-console | 126 +++++++ .../override.conf | 3 + overlay/usr/local/bin/random-seed-add | 33 ++ production-config | 5 + ts-debian-di-install | 2 +- ts-debian-fixup | 6 + ts-host-install | 6 +- ts-leak-check | 1 + ts-logs-capture | 1 + ts-xen-build-prep | 6 +- ts-xen-install | 41 +-- 18 files changed, 656 insertions(+), 77 deletions(-) create mode 100755 overlay-buster/etc/grub.d/20_linux_xen create mode 100755 overlay-initrd-buster/sbin/reopen-console create mode 100644 overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf create mode 100755 overlay/usr/local/bin/random-seed-add
Signature · 1 line
2.20.1
PATCH v2 1/49 ts-logs-capture: Cope if xl shutdown leaves domain running for a bit
This seems mostly to affect buster but it could in principle affect earlier releases too I think.
In principle it would be nice to fix this bug, and to have a proper test for it, but a reliable test is hard and an unreliable one is not useful. So I guess we are going to have this workaround indefinitely...
--- ts-logs-capture | 1 + 1 file changed, 1 insertion(+)
ts-logs-capture +1 -0
@@ -272,6 +272,7 @@ sub shutdown_guests () { ( xl shutdown -a -F -w ; echo y ) & ) | ( read x + sleep 10 # xl shutdown is a bit racy :-/ xl list | awk '!/^Domain-0 |^Name / {print $2}' \ | xargs -t -r -n1 xl destroy ||: )
Signature · 1 line
2.20.1
PATCH v2 2/49 ts-xen-build-prep: Install rsync
In reply to Ian Jacksonosstest uses this for transferring configuration, build artefacts, and so on.
In Debian stretch and earlier, rsync happened to be pulled in by something else.
--- ts-xen-build-prep | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
ts-xen-build-prep +1 -1
@@ -197,7 +197,7 @@ END } sub prep () { - my @packages = qw(mercurial + my @packages = qw(mercurial rsync build-essential bin86 bcc iasl bc flex bison cmake libpci-dev libncurses5-dev libssl-dev python-dev
Signature · 1 line
2.20.1
PATCH v2 3/49 lvcreate argments: pass --yes -Z y -W y
In reply to Ian JacksonThe documentation seesm to think this is the default but empirically it isn't. In our environment --yes is fine.
I have reported this to Debian as #953183. Also vaguely related (and discovered by me at the same time) is #953185.
This came up while trying to get things work on buster. I don't know what has changed.
--- Osstest/TestSupport.pm | 2 +- ts-xen-build-prep | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
Osstest/TestSupport.pm +1 -1
@@ -935,7 +935,7 @@ sub lv_create ($$$$) { my ($ho, $vg, $lv, $mb) = @_; my $lvdev = "/dev/$vg/$lv"; target_cmd_root($ho, "lvremove -f $lvdev ||:"); - target_cmd_root($ho, "lvcreate -L ${mb}M -n $lv $vg"); + target_cmd_root($ho, "lvcreate --yes -Z y -W y -L ${mb}M -n $lv $vg"); target_cmd_root($ho, "dd if=/dev/zero of=$lvdev count=10"); return $lvdev; }
ts-xen-build-prep +2 -2
@@ -61,7 +61,7 @@ sub determine_vg_lv () { sub lvextend_stage1 () { target_cmd_root($ho, <<END); set -ex; if ! test -f /root/swap_osstest_enabled; then - lvcreate -L 10G -n swap_osstest_build $vg ||: + lvcreate --yes -Z y -W y -L 10G -n swap_osstest_build $vg ||: mkswap /dev/$vg/swap_osstest_build ||: swapon /dev/$vg/swap_osstest_build touch /root/swap_osstest_enabled
@@ -84,7 +84,7 @@ sub vginfo () { sub lvcreate () { target_cmd_output_root($ho, - "lvdisplay $lv || lvcreate -l 1 -n $lvleaf $vg"); + "lvdisplay $lv || lvcreate --yes -Z y -W y -l 1 -n $lvleaf $vg"); } sub lvextend1 ($$$) {
Signature · 1 line
2.20.1
PATCH v2 4/49 TestSupport: allow more time for apt
In reply to Ian JacksonEmpirically some of these operations can take longer than 30s, especially with a cold cache.
Note that because of host sharing and our on-host apt lock, the timeout needs to be the same for every apt operation: a fast operation could be blocked behind a slow one.
--- Osstest/TestSupport.pm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Osstest/TestSupport.pm +2 -2
@@ -637,12 +637,12 @@ sub target_install_packages_nonfree_nonconcurrent ($@) { my ($ho, @packages) = @_; my $slist= '/etc/apt/sources.list'; my $xsuites= 'contrib non-free'; - target_cmd_root($ho, <<END, 30); + target_cmd_root($ho, <<END, 300); perl -i~ -pe 'next unless m/^deb/; s{ main\$}{\$& $xsuites};' $slist apt-get update END target_run_pkgmanager_install($ho,\@packages); - target_cmd_root($ho, <<END, 30); + target_cmd_root($ho, <<END, 300); mv $slist~ $slist apt-get update END
Signature · 1 line
2.20.1
PATCH v2 5/49 Booting: Use `--' rather than `---' to introduce host cmdline
In reply to Ian JacksonBecause systemd did something obnoxious, the kernel retaliated in the game of Core Wars by hiding all arguments before --' from userspace. So use ---' instead so that all the arguments remain visible.
This in some sense now applies to host installs a change we had already made to Debian HVM guests. See osstest#493b7395
ts-debian-hvm-install: Use ---, and no longer duplicate $gconsole
and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007
Kernel command line handling change breaks d-i user-params functionality
This change is fine for all non-ancient versions of Debian, so I have not made it conditional.
--- Osstest/TestSupport.pm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Osstest/TestSupport.pm +2 -2
@@ -2909,7 +2909,7 @@ label overwrite menu label ^Overwrite menu default kernel $kern - append $dicmd initrd=$initrd -- $hocmd + append $dicmd initrd=$initrd --- $hocmd ipappend $xopts{ipappend} $dtbs default overwrite
@@ -2956,7 +2956,7 @@ sub setup_netboot_di_uefi ($$$$$;%) { set default=0 set timeout=5 menuentry 'overwrite' { - linux $kern $dicmd -- $hocmd + linux $kern $dicmd --- $hocmd initrd $initrd } END
Signature · 1 line
2.20.1
PATCH v2 6/49 di_installcmdline_core: Pass locale on d-i command line
In reply to Ian JacksonIn buster, d-i wants when setting up the network, ie before the preseed is loaded.
We leave it in the preseed too because why not.
I think this change should be fine for older versions of Debian.
--- Osstest/Debian.pm | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Osstest/Debian.pm +2 -1
@@ -699,7 +699,8 @@ sub di_installcmdline_core ($$;@) { "hostname=$tho->{Name}", "$xopts{PreseedScheme}=$ps_url", "netcfg/dhcp_timeout=150", - "netcfg/choose_interface=$netcfg_interface" + "netcfg/choose_interface=$netcfg_interface", + "debian-installer/locale=en_GB", ); my $debconf_priority= $xopts{DebconfPriority};
Signature · 1 line
2.20.1
PATCH v2 7/49 setupboot_grub2: Drop $submenu variable
In reply to Ian JacksonWe really only used this to check how many levels deep in { we are. That can be done by checking $#offsets, which is >0 if we are in a submenu and not otherwise. We lose the ability to report the start line of the submenu, but that's OK.
But as a bonus, we no longer bomb out on nested submenus: previously the first } would cause $submenu to be undef. Now we pop from @offsets and all is fine.
Nested submenus are present in Debian buster.
--- Osstest/Debian.pm | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)
Osstest/Debian.pm +3 -6
@@ -452,15 +452,13 @@ sub setupboot_grub2 ($$$$) { my @offsets = (0); my $entry; my $chainentry; - my $submenu; while (<$f>) { next if m/^\s*\#/ || !m/\S/; if (m/^\s*\}\s*$/) { - die unless $entry || $submenu; - if (!$entry && $submenu) { - logm("Met end of a submenu $submenu->{StartLine}..$.. ". + die unless $entry || $#offsets; + if (!$entry && $#offsets) { + logm("Met end of a submenu at $. (@offsets) ". "Our want kern is $want_kernver"); - $submenu= undef; pop @offsets; $offsets[$#offsets]++; next;
@@ -510,7 +508,6 @@ sub setupboot_grub2 ($$$$) { $offsets[$#offsets]++; } if (m/^\s*submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) { - $submenu={ StartLine =>$., MenuEntryPath => join ">", @offsets }; push @offsets,(0); } if (m/^\s*chainloader\s*\/EFI\/osstest\/xen.efi/) {
Signature · 1 line
2.20.1
PATCH v2 8/49 ts-leak-check: Ignore buster's udevd too
In reply to Ian JacksonFor reasons I don't propose to investigate, on buster udevd shows up like this:
2019-11-26 18:13:48 Z LEAKED [process 2633 /lib/systemd/systemd-udevd] process: root 2633 1555 0 18:10 ? 00:00:00 /lib/systemd/systemd-udevd
This does not match our suppression. Add an additional suppression.
--- ts-leak-check | 1 + 1 file changed, 1 insertion(+)
ts-leak-check +1 -0
@@ -202,6 +202,7 @@ xenstore /vm xenstore /libxl process .* udevd +process .* .*/systemd-udevd process .* /.+/systemd-shim file /var/run/xenstored/db
Signature · 1 line
2.20.1
PATCH v2 9/49 Bodge systemd random seed arrangements
In reply to Ian Jacksonsystemd does not regard the contents of the random seed file as useful for the purposes of placating the kernel's entropy tracker. As a result, the system hangs at boot waiting for entropy.
Fix this by providing a small program which can be used to load a seed file into /dev/random and also call RNDADDTOENTCNT to add the appropriate amount to the kernel's counter.
Arrange to run this program instead of
/lib/systemd/systemd-random-seed load
With systemd the random seed file is in /var/lib/systemd/random-seed rather than /var/lib/urandom/random-seed.
And, provide an initial contents of this file, via a d-i late_command.
Unfortunately we must hardcode the actual numerical value of RNDADDTOENTCNT because we don't have a suitable compiler anywhere nearby. It seems to have the same value on i386, amd64, armhf and arm64, our currently supported architectures.
Thanks to Colin Watson for pointers to the systemd random unit and Matthew Vernon for instructions on overriding just ExecStart.
I think this change should be a no-op on non-systemd systems.
In principle this is a bug in Debian or in systemd, that ought to be reported upstream. However, it has been extensively discussed on debian-devel and it does not seem that any improvement is likely.
--- Osstest/Debian.pm | 18 ++++++++++ .../override.conf | 3 ++ overlay/usr/local/bin/random-seed-add | 33 +++++++++++++++++++ 3 files changed, 54 insertions(+) create mode 100644 overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf create mode 100755 overlay/usr/local/bin/random-seed-add
Osstest/Debian.pm +18 -0
@@ -49,6 +49,7 @@ BEGIN { di_installcmdline_core di_vg_name debian_dhcp_rofs_fix + debian_write_random_seed_command ); %EXPORT_TAGS = ( );
@@ -1087,6 +1088,13 @@ ln -s . /target/boot/boot END } + my $cmd = debian_write_random_seed_command('/target'); + preseed_hook_command($ho, 'late_command', $sfx, <<END); +#!/bin/sh +set -ex +$cmd +END + $preseed_file .= preseed_hook_cmds(); return create_webfile($ho, "preseed$sfx", $preseed_file);
@@ -1612,4 +1620,14 @@ mv '$script.new' '$script' END } +sub debian_write_random_seed_command ($) { + my ($mountpoint) = @_; + my $dir = "$mountpoint/var/lib/systemd"; + return <<END; + umask 077 + test -d $dir || mkdir -m 0755 $dir + dd if=/dev/urandom of=$dir/random-seed bs=1k count=1 +END +} + 1;
overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf new file +3 -0
@@ -0,0 +1,3 @@ +[Service] +ExecStart= +ExecStart=/usr/local/bin/random-seed-add /var/lib/systemd/random-seed
overlay/usr/local/bin/random-seed-add new file +33 -0
@@ -0,0 +1,33 @@ +#!/usr/bin/perl -w +use strict; + +open R, '>', '/dev/random' or die "open /dev/random: $!\n"; +R->autoflush(1); + +sub rndaddtoentcnt ($) { + my ($bits) = @_; + my $x = pack 'L', $bits; + my $r = ioctl R, 0x40045201, $x; + defined $r or die "RNDADDTOENTCNT: $!\n"; +} + +sub process_stdin ($) { + my ($f) = @_; + my $got = read STDIN, $_, 512; + defined $got or die "read $f: $!\n"; + last if !$got; + print R $_ or die "write /dev/random: $!\n"; + my $bits = length($_) * 8; + rndaddtoentcnt($bits); +} + +if (!@ARGV) { + process_stdin('stdin'); +} else { + die "no options supported\n" if $ARGV[0] =~ m/^\-/; + foreach my $f (@ARGV) { + open STDIN, '<', $f or die "open for reading $f: $!\n"; + process_stdin($f); + } +} +
Signature · 1 line
2.20.1
Quoted 7 lines
-----Original Message-----
From: Ian Jackson <[email protected]>
Sent: 29 May 2020 12:19
To: [email protected]; Paul Durrant <[email protected]>
Cc: Ian Jackson <[email protected]>; [email protected]
Subject: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)
Quoted 8 lines
This series looks about as ready as it is going to be. Unfortunately there are still two issues, each of which cropped up once in my final formal retest. See below.
What are people's opinions? Should I push this to osstest pretest soon after the Xen codefreeze (eg, after we get the first push after freeze) ?
That sounds reasonable; I don't think we want to perturb osstest until after the first push.
Quoted 6 lines
The downside would be introducing new low-probability heisenbugs. But in my tests it seems that some existing heisenbugs may be abated. It's hard to say for sure. The upside is a lower risk that Debian will break our stuff and also being able to commission a number of machines which do not work with stretch (Debian oldstable).
I assume we can revert if things go badly wrong and being able to commission more machines would seem to be quite beneficial at this stage.
Paul
Quoted 98 lines
The two issues:
1. In one test, libvirt segfaulted. This is with osstest's own-built tested version of libvirt; osstest's own-build tested version of Linux; and osstest's own-built xen.git#master.
http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-i386-libvirt/info.html http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-i386-libvirt/rimava1---var-log-kern.log.gz
| May 28 21:39:11 rimava1 kernel: [ 1146.326593] libvirtd[16868]: segfault at 820 ipb72535b0spabce5090error 4 in libc-2.28.so[b71f0000+14e000]2. In one test, the Debian installer hung. The kernel printed many soft lockup messages. This is with osstest's own dom0 kernel and hypervisor, but the 4.19.67-2+deb10u1 kernel from Debian.
http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-amd64-xl-qemut-debianhvm-i386-xsm/info.html
http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-amd64-xl-qemut-debianhvm-i386-
xsm/godello1---var-log-xen-qemu-dm-debianhvm.guest.osstest.log.gzIan Jackson (49):
ts-logs-capture: Cope if xl shutdown leaves domain running for a bit
ts-xen-build-prep: Install rsync
lvcreate argments: pass --yes -Z y -W y
TestSupport: allow more time for apt
Booting: Use--' rather than---' to introduce host cmdline
di_installcmdline_core: Pass locale on d-i command line
setupboot_grub2: Drop $submenu variable
ts-leak-check: Ignore buster's udevd too
Bodge systemd random seed arrangements
Debian guests made with xen-tools: Write systemd random seed file
ts-debian-di-install: Provide guest with more RAM
Debian: preseed: use priority= alias
Debian: Specify `priority=critical' rather than locale Honour 'LinuxSerialConsole <suite>' host property
buster: make-hosts-flight: Add to possible suites for hosts flight
buster: Extend grub2 uefi no install workaround
buster: ts-host-install: Extend net.ifnames workaround
buster: Deinstall the "systemd" package
buster: preseed partman-auto-lvm/guided_size
buster: ts-host-install: NTP not honoured bug remains
buster: Extend ARM clock workaround
buster: Extend guest bootloader workaround
HonourDebianImageFile_SUITE_ARCH
buster: SpecifyDebianImageFile_SUITE_ARCH
20_linux_xen: Copy Debian buster version into our initramfs area 20_linux_xen: Adhoc template substitution
20_linux_xen: Ignore xenpolicy and config files too 20_linux_xen: Support Xen Security Modules (XSM/FLASK) mg-debian-installer-update: support overlay-intramfs-SUITE overlay-initrd-buster/sbin/reopen-console: Copy from Debian overlay-initrd-buster/sbin/reopen-console: Fix #932416
buster: chiark-scripts: Install a new version on buster too
buster: Provide TftpDiVersion
buster: grub, arm64: extend chainloading workaround
buster:setupboot_grub2: Note what files exist in /boot
buster:setupboot_grub2: Handle missing policy file bug
buster: Extend workaround for dhcpd EROFS bug
ts-xen-install: Add $ho argument tosome_extradebs
ts-xen-install: Movesome_extradebsto Debian.pm Debian.pm: Break outstandard_extradebs
Debian.pm: Movestandard_extradebsto ts-host-install
buster: Install own linux-libc-dev package (!)
setupboot_grub2: Insist on space after directives
setupboot_grub2: Print line number of entry we are using
setupboot_grub2: Recognise --nounzip for initramfs
setupboot_grub2: Copy hv command line from grub to xen.cfg
setupboot_grub2: Do not boot with XSM policy etc. unless xsm=1 buster 20_linux_xen: Only load policy in XSM-enabled builds
buster: Switch to Debian buster as the default suiteOsstest.pm | 2 +- Osstest/Debian.pm | 128 +++++-- Osstest/TestSupport.pm | 15 +- make-hosts-flight | 2 +- mfi-common | 9 + mg-debian-installer-update | 20 ++ overlay-buster/etc/grub.d/20_linux_xen | 327 ++++++++++++++++++ overlay-initrd-buster/sbin/reopen-console | 126 +++++++ .../override.conf | 3 + overlay/usr/local/bin/random-seed-add | 33 ++ production-config | 5 + ts-debian-di-install | 2 +- ts-debian-fixup | 6 + ts-host-install | 6 +- ts-leak-check | 1 + ts-logs-capture | 1 + ts-xen-build-prep | 6 +- ts-xen-install | 41 +-- 18 files changed, 656 insertions(+), 77 deletions(-) create mode 100755 overlay-buster/etc/grub.d/20_linux_xen create mode 100755 overlay-initrd-buster/sbin/reopen-console create mode 100644 overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf create mode 100755 overlay/usr/local/bin/random-seed-add--
2.20.1
Paul Durrant writes ("RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)"):
Quoted 2 lines
I assume we can revert if things go badly wrong and being able to commission more machines would seem to be quite beneficial at this stage.
Thanks for your opinion.
It would be possible to revert the final switch, yes. Most of the other changes are supposed to work with stretch too.
I haven't done a test run with the new code, but the old version of Debian. I will do that. That will give confidence that we could, in fact, revert things.
Ian.
Ian Jackson writes ("RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)"):
Quoted 4 lines
Paul Durrant writes ("RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)"):
Quoted 2 lines
I assume we can revert if things go badly wrong and being able to commission more machines would seem to be quite beneficial at this stage.
Quoted 8 lines
Thanks for your opinion.
It would be possible to revert the final switch, yes. Most of the other changes are supposed to work with stretch too.
I haven't done a test run with the new code, but the old version of Debian. I will do that. That will give confidence that we could, in fact, revert things.
I have now done this and it resulted in one bugfix.
So if you are still content to go ahead I would like to push this series to osstest's own self-push-gate now.
Ian.
Quoted 18 lines
-----Original Message-----
From: Ian Jackson <[email protected]>
Sent: 08 June 2020 16:34
To: [email protected]
Cc: [email protected]; [email protected]
Subject: RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)Ian Jackson writes ("RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)"):
Quoted 2 lines
Paul Durrant writes ("RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)"):
Quoted 1 line
I assume we can revert if things go badly wrong and being able to commission more machines would
seem to be quite beneficial at this
Quoted 7 lines
Quoted 1 line
stage.
Thanks for your opinion.
It would be possible to revert the final switch, yes. Most of the other changes are supposed to work with stretch too.
Quoted 9 lines
Quoted 3 lines
I haven't done a test run with the new code, but the old version of Debian. I will do that. That will give confidence that we could, in fact, revert things.
I have now done this and it resulted in one bugfix.
So if you are still content to go ahead I would like to push this series to osstest's own self-push-gate now.
Yes, I am content to go ahead.
Paul
PATCH v2 10/49 Debian guests made with xen-tools: Write systemd random seed file
In reply to Ian JacksonWhen the Debian guest is not made with d-i, we must still provide this random seed file. This can be done in ts-debian-fixup.
--- ts-debian-fixup | 6 ++++++ 1 file changed, 6 insertions(+)
ts-debian-fixup +6 -0
@@ -102,6 +102,11 @@ sub console () { logm("extra: $extra"); } +sub randomseed () { + my $cmd = debian_write_random_seed_command($mountpoint); + target_cmd_root($ho, "set -ex\n".cmd); +} + sub filesystems () { my $rootdev= $r{"$gho->{Guest}_rootdev"}; return unless defined($rootdev) && length($rootdev);
@@ -215,6 +220,7 @@ END target_cmd_root($ho, debian_dhcp_rofs_fix($ho, $mountpoint)); console(); +randomseed(); filesystems(); otherfixupcfg(); writecfg();
Signature · 1 line
2.20.1
PATCH v2 11/49 ts-debian-di-install: Provide guest with more RAM
In reply to Ian Jacksonbuster cannot boot in so little because its initramfs and kernel are too large. Bump it to 2G.
However, our armhf test nodes have very little RAM. And the Debian armhf does fit in them as a guest still, so use a smaller value there.
Keying this off the architecture rather than the available host memory is better because you do need the bigger value precisely if you are not using armhf, and this makes osstest less dependent on a completely accurate and populated host properties database.
--- ts-debian-di-install | 2 +- ts-debian-fixup | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
ts-debian-di-install +1 -1
@@ -64,7 +64,7 @@ $gn ||= 'debian'; our $ho= selecthost($whhost); -our $ram_mb= 512; +our $ram_mb= $r{arch} =~ m/^armhf/ ? 768 : 2048; our $disk_mb= 10000; our $guesthost= $gn.
ts-debian-fixup +1 -1
@@ -104,7 +104,7 @@ sub console () { sub randomseed () { my $cmd = debian_write_random_seed_command($mountpoint); - target_cmd_root($ho, "set -ex\n".cmd); + target_cmd_root($ho, "set -ex\n".$cmd); } sub filesystems () {
Signature · 1 line
2.20.1
PATCH v2 12/49 Debian: preseed: use priority= alias
In reply to Ian JacksonThis marginally reduces command line clobber. This alias has been supported approximately forever. (And this code is currently only used when DebconfPriority is set, which it generally isn't.)
--- Osstest/Debian.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
@@ -702,7 +702,7 @@ sub di_installcmdline_core ($$;@) { ); my $debconf_priority= $xopts{DebconfPriority}; - push @cl, "debconf/priority=$debconf_priority" + push @cl, "priority=$debconf_priority" if defined $debconf_priority; push @cl, "rescue/enable=true" if $xopts{RescueMode};
Signature · 1 line
2.20.1
PATCH v2 13/49 Debian: Specify `priority=critical' rather than locale
In reply to Ian JacksonIn buster, it appears that specifying locale on the command line is not sufficient. Rather than adding more things to the command line, instead, just say `priority=critical', by defaulting $debconf_priority to 'critical'.
I think this change should be fine for earlier suites too.
--- Osstest/Debian.pm | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
Osstest/Debian.pm +2 -4
@@ -698,12 +698,10 @@ sub di_installcmdline_core ($$;@) { "$xopts{PreseedScheme}=$ps_url", "netcfg/dhcp_timeout=150", "netcfg/choose_interface=$netcfg_interface", - "debian-installer/locale=en_GB", ); - my $debconf_priority= $xopts{DebconfPriority}; - push @cl, "priority=$debconf_priority" - if defined $debconf_priority; + my $debconf_priority= $xopts{DebconfPriority} // 'critical'; + push @cl, "priority=$debconf_priority"; push @cl, "rescue/enable=true" if $xopts{RescueMode}; if ($r{syslog_server}) {
Signature · 1 line
2.20.1
PATCH v2 14/49 Honour 'LinuxSerialConsole <suite>' host property
In reply to Ian JacksonThis works like LinuxSerialConsole.
I originally wrote this to try to work around #940028, where multiple d-i autoinstalls run in parallel leading to hard-to-debug lossage! Explicitly specing the console causes it to run only on that one.
However, it turns out that explicitly specifying the console does not always work and a better fix is needed. Nevertheless, having added this feature it seems foolish to throw it away.
Currently there are no hosts with this property so no functiaonal change.
--- Osstest/TestSupport.pm | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
Osstest/TestSupport.pm +4 -1
@@ -1447,7 +1447,10 @@ sub get_target_property ($$;$) { sub get_host_native_linux_console ($) { my ($ho) = @_; - my $console = get_host_property($ho, "LinuxSerialConsole", "ttyS0"); + my $console; + $console //= get_host_property($ho, "LinuxSerialConsole $ho->{Suite}") + if $ho->{Suite}; + $console //= get_host_property($ho, "LinuxSerialConsole", "ttyS0"); return $console if $console eq 'NONE'; return "$console,$c{Baud}n8";
Signature · 1 line
2.20.1
PATCH v2 15/49 buster: make-hosts-flight: Add to possible suites for hosts flight
In reply to Ian Jackson--- make-hosts-flight | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
make-hosts-flight +1 -1
@@ -26,7 +26,7 @@ blessing=$4 buildflight=$5 : ${ALL_ARCHES:=amd64 i386 arm64 armhf} -: ${ALL_SUITES:=stretch jessie} +: ${ALL_SUITES:=buster stretch jessie} # ^ most preferred suite first : ${PERHOST_MAXWAIT:=20000} # seconds
Signature · 1 line
2.20.1
PATCH v2 16/49 buster: Extend grub2 uefi no install workaround
In reply to Ian Jacksonsrc:grub2 is RFH in Debian, which is a contributory factor to these patches in #789798 and #792547 languishing.
--- Osstest/Debian.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
@@ -1466,7 +1466,7 @@ d-i partman-auto/expert_recipe string \\ END if (get_host_property($ho, "firmware") eq "uefi") { - die unless $ho->{Suite} =~ m/jessie|stretch/; + die unless $ho->{Suite} =~ m/jessie|stretch|buster/; # Prevent grub-install from making a new Debian boot entry, so # we always reboot from the network. Debian bug #789798 proposes a # properly preseedable solution to this.
Signature · 1 line
2.20.1
PATCH v2 17/49 buster: ts-host-install: Extend net.ifnames workaround
In reply to Ian JacksonReally we should fix this by making a .deb in Debian that we could install. But this is a longer-term project.
--- ts-host-install | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
ts-host-install +1 -1
@@ -282,7 +282,7 @@ END # Don't use "Predictable Network Interface Names" # https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/ - push @hocmdline, "net.ifnames=0" if $ho->{Suite} =~ m/stretch/; + push @hocmdline, "net.ifnames=0" if $ho->{Suite} =~ m/stretch|buster/; push @hocmdline, get_host_property($ho, "linux-boot-append $ho->{Suite}", ''),
Signature · 1 line
2.20.1
PATCH v2 18/49 buster: Deinstall the "systemd" package
In reply to Ian JacksonThis installs a pam rule which causes logins to hang. It also seems to cause some kind of udev wedge.
We are using sysvinit so this package is not desirable. Empirically, removing it makes the system work.
--- Osstest/Debian.pm | 1 + 1 file changed, 1 insertion(+)
Osstest/Debian.pm +1 -0
@@ -842,6 +842,7 @@ sub preseed_base ($$$;@) { if ( $suite !~ /squeeze|wheezy/ ) { preseed_hook_command($ho, 'late_command', $sfx, <<END) in-target apt-get install -y sysvinit-core +in-target apt-get remove -y systemd END }
Signature · 1 line
2.20.1
PATCH v2 19/49 buster: preseed partman-auto-lvm/guided_size
In reply to Ian JacksonOtherwise we get this question:
| You may use the whole volume group for guided partitioning, or part | of it. [...] | Amount of volume group to use for guided partitioning:
--- Osstest/Debian.pm | 1 + 1 file changed, 1 insertion(+)
Osstest/Debian.pm +1 -0
@@ -938,6 +938,7 @@ d-i partman/choose_partition select finish d-i partman/confirm boolean true d-i partman-lvm/confirm boolean true d-i partman-lvm/device_remove_lvm_span boolean true +d-i partman-auto-lvm/guided_size string max d-i partman/confirm_nooverwrite true d-i partman-lvm/confirm_nooverwrite true
Signature · 1 line
2.20.1
PATCH v2 20/49 buster: ts-host-install: NTP not honoured bug remains
In reply to Ian JacksonDebian #778564 remains open.
--- ts-host-install | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
ts-host-install +1 -1
@@ -152,7 +152,7 @@ END my $done= 0; while (<EI>) { if (m/^server\b|^pool\b\s/) { - if ($ho->{Suite} =~ m/lenny|squeeze|wheezy|jessie|stretch/) { + if ($ho->{Suite} =~ m/lenny|squeeze|wheezy|jessie|stretch|buster/) { $_= $done ? "" : "server $ntpserver\n"; } else { m/^server \Q$ntpserver\E\s/ or
Signature · 1 line
2.20.1
PATCH v2 21/49 buster: Extend ARM clock workaround
In reply to Ian Jackson--- Osstest/Debian.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
@@ -248,7 +248,7 @@ END push @xenkopt, $xenkopt; # https://bugs.xenproject.org/xen/bug/45 push @xenkopt, "clk_ignore_unused" - if $ho->{Suite} =~ m/wheezy|jessie|stretch/; + if $ho->{Suite} =~ m/wheezy|jessie|stretch|buster/; $xenkopt = join ' ', @xenkopt; logm("Dom0 Linux options: $xenkopt");
Signature · 1 line
2.20.1
PATCH v2 22/49 buster: Extend guest bootloader workaround
In reply to Ian Jackson--- Osstest/Debian.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
@@ -1064,7 +1064,7 @@ END logm("\$arch is $arch, \$suite is $suite"); if ($xopts{PvMenuLst} && $arch =~ /^arm/ && - $suite =~ /wheezy|jessie|stretch|sid/ ) { + $suite =~ /wheezy|jessie|stretch|buster|sid/ ) { # Debian doesn't currently know what bootloader to install in # a Xen guest on ARM. We install pv-grub-menu above which
Signature · 1 line
2.20.1
PATCH v2 23/49 Honour DebianImageFile_SUITE_ARCH
In reply to Ian JacksonThis lets us specify the whole filename, not just a version. This is needed because for buster we are going to use
debian-10.2.0-ARCH-xfce-CD-1.iso
--- mfi-common | 9 +++++++++ 1 file changed, 9 insertions(+)
mfi-common +9 -0
@@ -522,6 +522,15 @@ job_create_test () { usual_debianhvm_image () { local arch=$1; shift + if [ -n "$DEBIAN_IMAGE_FILE" ]; then + echo $DEBIAN_IMAGE_FILE + return + fi + local file=`getconfig DebianImageFile_${guestsuite}_${arch}` + if [ -n "$file " ]; then + echo $file + return + fi local ver=$DEBIAN_IMAGE_VERSION if [ -z "$ver" ] ; then ver=`getconfig DebianImageVersion_$guestsuite`
Signature · 1 line
2.20.1
PATCH v2 24/49 buster: Specify DebianImageFile_SUITE_ARCH
In reply to Ian Jackson--- production-config | 3 +++ 1 file changed, 3 insertions(+)
production-config +3 -0
@@ -98,6 +98,9 @@ DebianSnapshotBackports_jessie http://snapshot.debian.org/archive/debian/2019020 DebianImageVersion_wheezy 7.2.0 DebianImageVersion_jessie 8.2.0 DebianImageVersion_stretch 9.4.0 +DebianImageFile_buster_amd64 debian-10.2.0-amd64-xfce-CD-1.iso +DebianImageFile_buster_i386 debian-10.2.0-i386-xfce-CD-1.iso + # Update with ./mg-netgrub-loader-update TftpGrubVersion XXXX-XX-XX
Signature · 1 line
2.20.1
PATCH v2 25/49 20_linux_xen: Copy Debian buster version into our initramfs area
In reply to Ian JacksonThis is from 41e42571ebc50fa351cd63ce40044946652c5c72 in Debian's grub package.
We are going to want to modify this to support XSM/FLASK and cope with upstream build outputs.
In this commit we dump the exact file contents across. It's not effective right now because of the ".in" extension. In fact, the file is a template.
At the time of writing I am trying to send our substantive changes upstream via Debian's Gitlab:
https://salsa.debian.org/grub-team/grub/-/merge_requests/18
--- overlay-buster/etc/grub.d/20_linux_xen.in | 299 ++++++++++++++++++++++ 1 file changed, 299 insertions(+) create mode 100644 overlay-buster/etc/grub.d/20_linux_xen.in
overlay-buster/etc/grub.d/20_linux_xen.in new file +299 -0
@@ -0,0 +1,299 @@ +#! /bin/sh +set -e + +# grub-mkconfig helper script. +# Copyright (C) 2006,2007,2008,2009,2010 Free Software Foundation, Inc. +# +# GRUB is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# GRUB is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GRUB. If not, see <http://www.gnu.org/licenses/>. + +prefix="@prefix@" +exec_prefix="@exec_prefix@" +datarootdir="@datarootdir@" + +. "$pkgdatadir/grub-mkconfig_lib" + +export TEXTDOMAIN=@PACKAGE@ +export TEXTDOMAINDIR="@localedir@" + +CLASS="--class gnu-linux --class gnu --class os --class xen" +SUPPORTED_INITS="sysvinit:/lib/sysvinit/init systemd:/lib/systemd/systemd upstart:/sbin/upstart" + +if [ "x${GRUB_DISTRIBUTOR}" = "x" ] ; then + OS=GNU/Linux +else + OS="${GRUB_DISTRIBUTOR} GNU/Linux" + CLASS="--class $(echo ${GRUB_DISTRIBUTOR} | tr 'A-Z' 'a-z' | cut -d' ' -f1|LC_ALL=C sed 's,[^[:alnum:]_],_,g') ${CLASS}" +fi + +# loop-AES arranges things so that /dev/loop/X can be our root device, but +# the initrds that Linux uses don't like that. +case ${GRUB_DEVICE} in + /dev/loop/*|/dev/loop[0-9]) + GRUB_DEVICE=`losetup ${GRUB_DEVICE} | sed -e "s/^[^(]*(\([^)]\+\)).*/\1/"` + # We can't cope with devices loop-mounted from files here. + case ${GRUB_DEVICE} in + /dev/*) ;; + *) exit 0 ;; + esac + ;; +esac + +# btrfs may reside on multiple devices. We cannot pass them as value of root= parameter +# and mounting btrfs requires user space scanning, so force UUID in this case. +if [ "x${GRUB_DEVICE_UUID}" = "x" ] || [ "x${GRUB_DISABLE_LINUX_UUID}" = "xtrue" ] \ + || ! test -e "/dev/disk/by-uuid/${GRUB_DEVICE_UUID}" \ + || ( test -e "${GRUB_DEVICE}" && uses_abstraction "${GRUB_DEVICE}" lvm ); then + LINUX_ROOT_DEVICE=${GRUB_DEVICE} +else + LINUX_ROOT_DEVICE=UUID=${GRUB_DEVICE_UUID} +fi + +# Allow overriding GRUB_CMDLINE_LINUX and GRUB_CMDLINE_LINUX_DEFAULT. +if [ "${GRUB_CMDLINE_LINUX_XEN_REPLACE}" ]; then + GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX_XEN_REPLACE}" +fi +if [ "${GRUB_CMDLINE_LINUX_XEN_REPLACE_DEFAULT}" ]; then + GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_XEN_REPLACE_DEFAULT}" +fi + +case x"$GRUB_FS" in + xbtrfs) + rootsubvol="`make_system_path_relative_to_its_root /`" + rootsubvol="${rootsubvol#/}" + if [ "x${rootsubvol}" != x ]; then + GRUB_CMDLINE_LINUX="rootflags=subvol=${rootsubvol} ${GRUB_CMDLINE_LINUX}" + fi;; + xzfs) + rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true` + bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`" + LINUX_ROOT_DEVICE="ZFS=${rpool}${bootfs%/}" + ;; +esac + +title_correction_code= + +linux_entry () +{ + os="$1" + version="$2" + xen_version="$3" + type="$4" + args="$5" + xen_args="$6" + if [ -z "$boot_device_id" ]; then + boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")" + fi + if [ x$type != xsimple ] ; then + if [ x$type = xrecovery ] ; then + title="$(gettext_printf "%s, with Xen %s and Linux %s (%s)" "${os}" "${xen_version}" "${version}" "$(gettext "${GRUB_RECOVERY_TITLE}")")" + elif [ "${type#init-}" != "$type" ] ; then + title="$(gettext_printf "%s, with Xen %s and Linux %s (%s)" "${os}" "${xen_version}" "${version}" "${type#init-}")" + else + title="$(gettext_printf "%s, with Xen %s and Linux %s" "${os}" "${xen_version}" "${version}")" + fi + replacement_title="$(echo "Advanced options for ${OS}" | sed 's,>,>>,g')>$(echo "$title" | sed 's,>,>>,g')" + if [ x"Xen ${xen_version}>$title" = x"$GRUB_ACTUAL_DEFAULT" ]; then + quoted="$(echo "$GRUB_ACTUAL_DEFAULT" | grub_quote)" + title_correction_code="${title_correction_code}if [ \"x\$default\" = '$quoted' ]; then default='$(echo "$replacement_title" | grub_quote)'; fi;" + grub_warn "$(gettext_printf "Please don't use old title \`%s' for GRUB_DEFAULT, use \`%s' (for versions before 2.00) or \`%s' (for 2.00 or later)" "$GRUB_ACTUAL_DEFAULT" "$replacement_title" "gnulinux-advanced-$boot_device_id>gnulinux-$version-$type-$boot_device_id")" + fi + echo "menuentry '$(echo "$title" | grub_quote)' ${CLASS} \$menuentry_id_option 'xen-gnulinux-$version-$type-$boot_device_id' {" | sed "s/^/$submenu_indentation/" + else + title="$(gettext_printf "%s, with Xen hypervisor" "${os}")" + echo "menuentry '$(echo "$title" | grub_quote)' ${CLASS} \$menuentry_id_option 'xen-gnulinux-simple-$boot_device_id' {" | sed "s/^/$submenu_indentation/" + fi + if [ x$type != xrecovery ] ; then + save_default_entry | grub_add_tab | sed "s/^/$submenu_indentation/" + fi + + if [ -z "${prepare_boot_cache}" ]; then + prepare_boot_cache="$(prepare_grub_to_access_device ${GRUB_DEVICE_BOOT} | grub_add_tab)" + fi + printf '%s\n' "${prepare_boot_cache}" | sed "s/^/$submenu_indentation/" + xmessage="$(gettext_printf "Loading Xen %s ..." ${xen_version})" + lmessage="$(gettext_printf "Loading Linux %s ..." ${version})" + sed "s/^/$submenu_indentation/" << EOF + echo '$(echo "$xmessage" | grub_quote)' + if [ "\$grub_platform" = "pc" -o "\$grub_platform" = "" ]; then + xen_rm_opts= + else + xen_rm_opts="no-real-mode edd=off" + fi + ${xen_loader} ${rel_xen_dirname}/${xen_basename} placeholder ${xen_args} \${xen_rm_opts} + echo '$(echo "$lmessage" | grub_quote)' + ${module_loader} ${rel_dirname}/${basename} placeholder root=${linux_root_device_thisversion} ro ${args} +EOF + if test -n "${initrd}" ; then + # TRANSLATORS: ramdisk isn't identifier. Should be translated. + message="$(gettext_printf "Loading initial ramdisk ...")" + sed "s/^/$submenu_indentation/" << EOF + echo '$(echo "$message" | grub_quote)' + ${module_loader} --nounzip ${rel_dirname}/${initrd} +EOF + fi + sed "s/^/$submenu_indentation/" << EOF +} +EOF +} + +linux_list= +for i in /boot/vmlinu[xz]-* /vmlinu[xz]-* /boot/kernel-*; do + if grub_file_is_not_garbage "$i"; then + basename=$(basename $i) + version=$(echo $basename | sed -e "s,^[^0-9]*-,,g") + dirname=$(dirname $i) + config= + for j in "${dirname}/config-${version}" "${dirname}/config-${alt_version}" "/etc/kernels/kernel-config-${version}" ; do + if test -e "${j}" ; then + config="${j}" + break + fi + done + if (grep -qx "CONFIG_XEN_DOM0=y" "${config}" 2> /dev/null || grep -qx "CONFIG_XEN_PRIVILEGED_GUEST=y" "${config}" 2> /dev/null); then linux_list="$linux_list $i" ; fi + fi +done +if [ "x${linux_list}" = "x" ] ; then + exit 0 +fi + +file_is_not_sym () { + case "$1" in + */xen-syms-*) + return 1;; + *) + return 0;; + esac +} + +xen_list= +for i in /boot/xen*; do + if grub_file_is_not_garbage "$i" && file_is_not_sym "$i" ; then xen_list="$xen_list $i" ; fi +done +prepare_boot_cache= +boot_device_id= + +title_correction_code= + +machine=`uname -m` + +case "$machine" in + i?86) GENKERNEL_ARCH="x86" ;; + mips|mips64) GENKERNEL_ARCH="mips" ;; + mipsel|mips64el) GENKERNEL_ARCH="mipsel" ;; + arm*) GENKERNEL_ARCH="arm" ;; + *) GENKERNEL_ARCH="$machine" ;; +esac + +# Extra indentation to add to menu entries in a submenu. We're not in a submenu +# yet, so it's empty. In a submenu it will be equal to '\t' (one tab). +submenu_indentation="" + +is_top_level=true + +while [ "x${xen_list}" != "x" ] ; do + list="${linux_list}" + current_xen=`version_find_latest $xen_list` + xen_basename=`basename ${current_xen}` + xen_dirname=`dirname ${current_xen}` + rel_xen_dirname=`make_system_path_relative_to_its_root $xen_dirname` + xen_version=`echo $xen_basename | sed -e "s,.gz$,,g;s,^xen-,,g"` + if [ -z "$boot_device_id" ]; then + boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")" + fi + if [ "x$is_top_level" != xtrue ]; then + echo " submenu '$(gettext_printf "Xen hypervisor, version %s" "${xen_version}" | grub_quote)' \$menuentry_id_option 'xen-hypervisor-$xen_version-$boot_device_id' {" + fi + if ($grub_file --is-x86-multiboot2 $current_xen); then + xen_loader="multiboot2" + module_loader="module2" + else + xen_loader="multiboot" + module_loader="module" + fi + while [ "x$list" != "x" ] ; do + linux=`version_find_latest $list` + gettext_printf "Found linux image: %s\n" "$linux" >&2 + basename=`basename $linux` + dirname=`dirname $linux` + rel_dirname=`make_system_path_relative_to_its_root $dirname` + version=`echo $basename | sed -e "s,^[^0-9]*-,,g"` + alt_version=`echo $version | sed -e "s,\.old$,,g"` + linux_root_device_thisversion="${LINUX_ROOT_DEVICE}" + + initrd= + for i in "initrd.img-${version}" "initrd-${version}.img" "initrd-${version}.gz" \ + "initrd-${version}" "initramfs-${version}.img" \ + "initrd.img-${alt_version}" "initrd-${alt_version}.img" \ + "initrd-${alt_version}" "initramfs-${alt_version}.img" \ + "initramfs-genkernel-${version}" \ + "initramfs-genkernel-${alt_version}" \ + "initramfs-genkernel-${GENKERNEL_ARCH}-${version}" \ + "initramfs-genkernel-${GENKERNEL_ARCH}-${alt_version}" ; do + if test -e "${dirname}/${i}" ; then + initrd="$i" + break + fi + done + if test -n "${initrd}" ; then + gettext_printf "Found initrd image: %s\n" "${dirname}/${initrd}" >&2 + else + # "UUID=" magic is parsed by initrds. Since there's no initrd, it can't work here. + linux_root_device_thisversion=${GRUB_DEVICE} + fi + + if [ "x$is_top_level" = xtrue ] && [ "x${GRUB_DISABLE_SUBMENU}" != xy ]; then + linux_entry "${OS}" "${version}" "${xen_version}" simple \ + "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}" "${GRUB_CMDLINE_XEN} ${GRUB_CMDLINE_XEN_DEFAULT}" + + submenu_indentation="$grub_tab$grub_tab" + + if [ -z "$boot_device_id" ]; then + boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")" + fi + # TRANSLATORS: %s is replaced with an OS name + echo "submenu '$(gettext_printf "Advanced options for %s (with Xen hypervisor)" "${OS}" | grub_quote)' \$menuentry_id_option 'gnulinux-advanced-$boot_device_id' {" + echo " submenu '$(gettext_printf "Xen hypervisor, version %s" "${xen_version}" | grub_quote)' \$menuentry_id_option 'xen-hypervisor-$xen_version-$boot_device_id' {" + is_top_level=false + fi + + linux_entry "${OS}" "${version}" "${xen_version}" advanced \ + "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}" "${GRUB_CMDLINE_XEN} ${GRUB_CMDLINE_XEN_DEFAULT}" + for supported_init in ${SUPPORTED_INITS}; do + init_path="${supported_init#*:}" + if [ -x "${init_path}" ] && [ "$(readlink -f /sbin/init)" != "$(readlink -f "${init_path}")" ]; then + linux_entry "${OS}" "${version}" "${xen_version}" "init-${supported_init%%:*}" \ + "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT} init=${init_path}" "${GRUB_CMDLINE_XEN} ${GRUB_CMDLINE_XEN_DEFAULT}" + + fi + done + if [ "x${GRUB_DISABLE_RECOVERY}" != "xtrue" ]; then + linux_entry "${OS}" "${version}" "${xen_version}" recovery \ + "single ${GRUB_CMDLINE_LINUX}" "${GRUB_CMDLINE_XEN}" + fi + + list=`echo $list | tr ' ' '\n' | fgrep -vx "$linux" | tr '\n' ' '` + done + if [ x"$is_top_level" != xtrue ]; then + echo ' }' + fi + xen_list=`echo $xen_list | tr ' ' '\n' | fgrep -vx "$current_xen" | tr '\n' ' '` +done + +# If at least one kernel was found, then we need to +# add a closing '}' for the submenu command. +if [ x"$is_top_level" != xtrue ]; then + echo '}' +fi + +echo "$title_correction_code"
Signature · 1 line
2.20.1
PATCH v2 26/49 20_linux_xen: Adhoc template substitution
In reply to Ian JacksonThis file is a template that various build-time variables get substituted into. Make thos substitutions by hand (actually, by copying the values our file for stretch). And rename the file.
So now we are using our file instead of the grub package's. But it is the same...
---
.../etc/grub.d/{20_linux_xen.in => 20_linux_xen} | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
rename overlay-buster/etc/grub.d/{20_linux_xen.in => 20_linux_xen} (98%)
mode change 100644 => 100755overlay-buster/etc/grub.d/20_linux_xen renamed from overlay-buster/etc/grub.d/20_linux_xen.in +5 -5
@@ -17,14 +17,14 @@ set -e # You should have received a copy of the GNU General Public License # along with GRUB. If not, see <http://www.gnu.org/licenses/>. -prefix="@prefix@" -exec_prefix="@exec_prefix@" -datarootdir="@datarootdir@" +prefix="/usr" +exec_prefix="/usr" +datarootdir="/usr/share" . "$pkgdatadir/grub-mkconfig_lib" -export TEXTDOMAIN=@PACKAGE@ -export TEXTDOMAINDIR="@localedir@" +export TEXTDOMAIN=grub +export TEXTDOMAINDIR="${datarootdir}/locale" CLASS="--class gnu-linux --class gnu --class os --class xen" SUPPORTED_INITS="sysvinit:/lib/sysvinit/init systemd:/lib/systemd/systemd upstart:/sbin/upstart"
Signature · 1 line
2.20.1
PATCH v2 27/49 20_linux_xen: Ignore xenpolicy and config files too
In reply to Ian Jackson"file_is_not_sym" currently only checks for xen-syms. Extend it to disregard xenpolicy (XSM policy files) and files ending .config (which are built by the Xen upstream build system in some configurations and can therefore end up in /boot).
Rename the function accordingly, to "file_is_not_xen_garbage".
--- overlay-buster/etc/grub.d/20_linux_xen | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
overlay-buster/etc/grub.d/20_linux_xen +6 -2
@@ -167,10 +167,14 @@ if [ "x${linux_list}" = "x" ] ; then exit 0 fi -file_is_not_sym () { +file_is_not_xen_garbage () { case "$1" in */xen-syms-*) return 1;; + */xenpolicy-*) + return 1;; + */*.config) + return 1;; *) return 0;; esac
@@ -178,7 +182,7 @@ file_is_not_sym () { xen_list= for i in /boot/xen*; do - if grub_file_is_not_garbage "$i" && file_is_not_sym "$i" ; then xen_list="$xen_list $i" ; fi + if grub_file_is_not_garbage "$i" && file_is_not_xen_garbage "$i" ; then xen_list="$xen_list $i" ; fi done prepare_boot_cache= boot_device_id=
Signature · 1 line
2.20.1
PATCH v2 28/49 20_linux_xen: Support Xen Security Modules (XSM/FLASK)
In reply to Ian JacksonXSM is enabled by adding "flask=enforcing" as a Xen command line argument, and providing the policy file as a grub module.
We make entries for both with and without XSM. If XSM is not compiled into Xen, then there are no policy files, so no change to the boot options.
--- overlay-buster/etc/grub.d/20_linux_xen | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
overlay-buster/etc/grub.d/20_linux_xen +24 -0
@@ -84,6 +84,11 @@ esac title_correction_code= linux_entry () +{ + linux_entry_xsm "$@" false + linux_entry_xsm "$@" true +} +linux_entry_xsm () { os="$1" version="$2"
@@ -91,6 +96,18 @@ linux_entry () type="$4" args="$5" xen_args="$6" + xsm="$7" + # If user wants to enable XSM support, make sure there's + # corresponding policy file. + if ${xsm} ; then + xenpolicy="xenpolicy-$xen_version" + if test ! -e "${xen_dirname}/${xenpolicy}" ; then + return + fi + xen_args="$xen_args flask=enforcing" + xen_version="$(gettext_printf "%s (XSM enabled)" "$xen_version")" + # xen_version is used for messages only; actual file is xen_basename + fi if [ -z "$boot_device_id" ]; then boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")" fi
@@ -140,6 +157,13 @@ EOF sed "s/^/$submenu_indentation/" << EOF echo '$(echo "$message" | grub_quote)' ${module_loader} --nounzip ${rel_dirname}/${initrd} +EOF + fi + if test -n "${xenpolicy}" ; then + message="$(gettext_printf "Loading XSM policy ...")" + sed "s/^/$submenu_indentation/" << EOF + echo '$(echo "$message" | grub_quote)' + ${module_loader} ${rel_dirname}/${xenpolicy} EOF fi sed "s/^/$submenu_indentation/" << EOF
Signature · 1 line
2.20.1
PATCH v2 29/49 mg-debian-installer-update: support overlay-intramfs-SUITE
In reply to Ian JacksonThis lets us patch the installer more easily.
No uses yet.
--- mg-debian-installer-update | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
mg-debian-installer-update +20 -0
@@ -33,6 +33,8 @@ sbase=$site/dists/$suite src=$sbase/main/installer-$arch/current/images/netboot/ +osstest_dir="$(pwd)" + case ${suite}_${arch} in lenny_armhf|squeeze_armhf|lenny_arm64|squeeze_arm64|wheezy_arm64) # No such thing.
@@ -188,6 +190,24 @@ if [ "x$specialkernel" != x ]; then rm -rf x fi +overlay_initrd=$osstest_dir/overlay-initrd-$suite +if [ -e "$overlay_initrd" ]; then + for f in $files; do + s=${f/:*} ; d=${f/*:} + case "$d" in + *initrd*) + echo "adding $overlay_initrd to $d" + ( + set -e + cd "$overlay_initrd" + find -print0 | cpio -0 -Hnewc -o \ + | gzip -9nf + ) >>$d.new + ;; + esac + done +fi + for f in $files; do s=${f/:*} ; d=${f/*:} mv -f $d.new $d
Signature · 1 line
2.20.1
PATCH v2 30/49 overlay-initrd-buster/sbin/reopen-console: Copy from Debian
In reply to Ian JacksonWe are going to patch this file to work around a bug, using the new overlay mechanism.
The first step is to include the file in our overlay so we overwrite it. Currently, this is a no-op, so no functional change.
--- overlay-initrd-buster/sbin/reopen-console | 94 +++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100755 overlay-initrd-buster/sbin/reopen-console
overlay-initrd-buster/sbin/reopen-console new file +94 -0
@@ -0,0 +1,94 @@ +#!/bin/sh + +# First find the enabled consoles from the kernel, noting if one is 'preferred' +# Record these. +# Run the startup scripts on the preferred console + +# In order to have D-I appear on all consoles, modify the inittab to +# add one entry for each console, running debian-installer. +# Finally HUP init so that it runs those installers +# (but doesn't rerun the sysinit startup stuff, including this script) + + +NL=" +" + +LOGGER_UP=0 +LOG_FILE=/var/log/reopen-console + +log() { + # In very early startup we don't have syslog. Log to file that + # we can flush out later so we can at least see what happened + # at early startup + if [ $LOGGER_UP -eq 1 ]; then + logger -t reopen-console "$@" + else + echo "$@" >> $LOG_FILE + fi +} + +flush_logger () { + cat $LOG_FILE | logger -t reopen-console + rm $LOG_FILE +} + +consoles= +preferred= +# Retrieve all enabled consoles from kernel; ignore those +# for which no device file exists + +kernelconsoles="$(cat /proc/consoles)" +for cons in $(echo "$kernelconsoles" | sed -n -r -e 's/(^.*) .*\((.*)\).*$/\1/p' ) +do + log "Looking at console $cons from /proc/consoles" + status=$(echo "$kernelconsoles" | grep $cons | sed -n -r -e 's/(^.*) *.*\((.*)\).*$/\2/p' ) + if [ -e "/dev/$cons" ] && [ $(echo "$status" | grep -o 'E') ]; then + consoles="${consoles:+$consoles$NL}$cons" + log " Adding $cons to consoles list" + fi + # 'C' console is 'most prefered'. + if [ $(echo "$status" | grep -o 'C') ]; then + preferred="$cons" + log " $cons is preferred" + fi +done + +if [ -z "$consoles" ]; then + # Nothing found? Default to /dev/console. + log "Found no consoles! Defaulting to /dev/console" + consoles=console +fi +if [ -z "$preferred" ]; then + #None marked preferred? Use the first one + preferred=$(echo "$consoles" | head -n 1) + log "Found no preferred console. Picking $preferred" +fi + +for cons in $consoles +do + echo "/dev/$cons " >> /var/run/console-devices +done +echo "/dev/$preferred " > /var/run/console-preferred + + +# Add debian-installer lines into inittab - one per console +for cons in $consoles +do + log "Adding inittab entry for $cons" + echo "$cons::respawn:/sbin/debian-installer" >> /etc/inittab +done + +# Run the startup scripts once, using the preferred console +cons=$(cat /var/run/console-preferred) +# Some other session may have that console as ctty. Steal it from them +/sbin/steal-ctty $cons "$@" + +# Now we should have syslog running, so flush our log entries +LOGGER_UP=1 +flush_logger + +# Finally restart init to run debian-installer on discovered consoles +log "Restarting init to start d-i on the consoles we found" +kill -HUP 1 + +exit 0
Signature · 1 line
2.20.1
PATCH v2 31/49 overlay-initrd-buster/sbin/reopen-console: Fix #932416
In reply to Ian JacksonThis bug affects us. Cherry pick the changes to the relevant file from the commit in the upstream debian-installer repo:
https://salsa.debian.org/installer-team/rootskel/commit/0ee43d05b83f8ef5a856f3282e002a111809cef9
--- overlay-initrd-buster/sbin/reopen-console | 36 +++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-)
overlay-initrd-buster/sbin/reopen-console +34 -2
@@ -16,6 +16,17 @@ NL=" LOGGER_UP=0 LOG_FILE=/var/log/reopen-console +# If we're running with preseeding, we have a problem with running d-i +# on multiple consoles. We'll end up running each of those d-i +# instances in parallel with all kinds of hilarious undefined +# behaviour as they trip over each other! If we detect that we're +# preseeding (via any of the possible preseed methods), DO NOT run d-i +# multiple times. Instead, fall back to the older, more simple +# behaviour and run it once. If the user wants to see or interact with +# their preseed on a specific console, they get to tell us which one +# they want to use. +PRESEEDING=0 + log() { # In very early startup we don't have syslog. Log to file that # we can flush out later so we can at least see what happened
@@ -32,6 +43,20 @@ flush_logger () { rm $LOG_FILE } +# If we have a preseed.cfg in the initramfs +if [ -e /preseed.cfg ]; then + log "Found /preseed.cfg; falling back to simple mode for preseeding" + PRESEEDING=1 +fi + +# Have we been told to do preseeding stuff on the boot command line? +for WORD in auto url; do + if (grep -qw "$WORD" /proc/cmdline); then + log "Found \"$WORD\" in the command line; falling back to simple mode for preseeding" + PRESEEDING=1 + fi +done + consoles= preferred= # Retrieve all enabled consoles from kernel; ignore those
@@ -44,7 +69,7 @@ do status=$(echo "$kernelconsoles" | grep $cons | sed -n -r -e 's/(^.*) *.*\((.*)\).*$/\2/p' ) if [ -e "/dev/$cons" ] && [ $(echo "$status" | grep -o 'E') ]; then consoles="${consoles:+$consoles$NL}$cons" - log " Adding $cons to consoles list" + log " Adding $cons to possible consoles list" fi # 'C' console is 'most prefered'. if [ $(echo "$status" | grep -o 'C') ]; then
@@ -64,6 +89,13 @@ if [ -z "$preferred" ]; then log "Found no preferred console. Picking $preferred" fi +# If we're preseeding, do simple stuff here (see above). We just +# want one console. Let's pick the preferred one ONLY +if [ $PRESEEDING = 1 ]; then + log "Running with preseeding. Picking preferred $preferred ONLY" + consoles=$preferred +fi + for cons in $consoles do echo "/dev/$cons " >> /var/run/console-devices
@@ -88,7 +120,7 @@ LOGGER_UP=1 flush_logger # Finally restart init to run debian-installer on discovered consoles -log "Restarting init to start d-i on the consoles we found" +log "Restarting init to start d-i on the console(s) we found" kill -HUP 1 exit 0
Signature · 1 line
2.20.1
PATCH v2 32/49 buster: chiark-scripts: Install a new version on buster too
In reply to Ian JacksonWe need various fixes that are not in buster, sadly.
--- production-config | 1 + 1 file changed, 1 insertion(+)
production-config +1 -0
@@ -107,6 +107,7 @@ TftpGrubVersion XXXX-XX-XX DebianExtraPackages_jessie chiark-scripts_6.0.3~citrix1_all.deb DebianExtraPackages_stretch chiark-scripts_6.0.4~citrix1_all.deb +DebianExtraPackages_buster chiark-scripts_6.0.5~citrix1_all.deb DebianExtraPackages_uefi_i386_jessie extradebs-uefi-i386-2018-04-01/ DebianExtraPackages_uefi_amd64_jessie extradebs-uefi-amd64-2018-04-01/
Signature · 1 line
2.20.1
PATCH v2 33/49 buster: Provide TftpDiVersion
In reply to Ian Jackson--- production-config | 1 + 1 file changed, 1 insertion(+)
production-config +1 -0
@@ -91,6 +91,7 @@ TftpNetbootGroup osstest TftpDiVersion_wheezy 2016-06-08 TftpDiVersion_jessie 2018-06-26 TftpDiVersion_stretch 2020-02-10 +TftpDiVersion_buster 2020-05-19 DebianSnapshotBackports_jessie http://snapshot.debian.org/archive/debian/20190206T211314Z/
Signature · 1 line
2.20.1
PATCH v2 34/49 buster: grub, arm64: extend chainloading workaround
In reply to Ian Jacksonmultiboot[2] isn't supported.
Also link to the bug report.
--- Osstest/Debian.pm | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Osstest/Debian.pm +2 -1
@@ -443,9 +443,10 @@ sub setupboot_grub2 ($$$$) { my $kernkey= (defined $xenhopt ? 'KernDom0' : 'KernOnly'); # Grub2 on jessie/stretch ARM* doesn't do multiboot, so we must chainload. + # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=884770 my $need_uefi_chainload = get_host_property($ho, "firmware") eq "uefi" && - $ho->{Suite} =~ m/jessie|stretch/ && $ho->{Arch} =~ m/^arm/; + $ho->{Suite} =~ m/jessie|stretch|buster/ && $ho->{Arch} =~ m/^arm/; my $parsemenu= sub { my $f= bl_getmenu_open($ho, $rmenu, "$stash/$ho->{Name}--grub.cfg.1");
Signature · 1 line
2.20.1
PATCH v2 35/49 buster: setupboot_grub2: Note what files exist in /boot
In reply to Ian JacksonNothing uses this yet.
--- Osstest/Debian.pm | 5 +++++ 1 file changed, 5 insertions(+)
Osstest/Debian.pm +5 -0
@@ -448,6 +448,11 @@ sub setupboot_grub2 ($$$$) { get_host_property($ho, "firmware") eq "uefi" && $ho->{Suite} =~ m/jessie|stretch|buster/ && $ho->{Arch} =~ m/^arm/; + my %bootfiles = + map { $_ => 1 } + split / /, + target_cmd_output_root($ho, "cd /boot && echo *"); + my $parsemenu= sub { my $f= bl_getmenu_open($ho, $rmenu, "$stash/$ho->{Name}--grub.cfg.1");
Signature · 1 line
2.20.1
PATCH v2 36/49 buster: setupboot_grub2: Handle missing policy file bug
In reply to Ian JacksonThis is a complex interaction between update-grub and the Xen build system on ARM64. Not sure exactly who to blame but since we have our own 20_linux_xen bodge, let's wait until we don't.
--- Osstest/Debian.pm | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
Osstest/Debian.pm +11 -1
@@ -496,7 +496,17 @@ sub setupboot_grub2 ($$$$) { " kernel $entry->{KernVer}, not $want_kernver)"); } elsif ($want_xsm && !defined $entry->{Xenpolicy}) { logm("(skipping entry at $entry->{StartLine}..$.;". - " XSM policy file not present)"); + " XSM policy file not mentioned)"); + } elsif ($ho->{Suite} =~ m/buster/ && + defined $entry->{Xenpolicy} && + !$bootfiles{ + $entry->{Xenpolicy} =~ m{^/?} ? $' : die + }) { + # Our 20_linux_xen bodge with buster's update-grub + # generates entries which mention /boot/xenpolicy-xen + # even though that file doesn't exist on ARM64. + logm("(skipping entry at $entry->{StartLine}..$.;". + " XSM policy file not on disk!)"); } else { # yes! last;
Signature · 1 line
2.20.1
PATCH v2 37/49 buster: Extend workaround for dhcpd EROFS bug
In reply to Ian Jackson--- Osstest/Debian.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
@@ -1624,7 +1624,7 @@ sub debian_dhcp_rofs_fix ($$) { # / is still ro. In stretch, the isc dhcp client spins requesting # an address and then sending a DHCPDECLINE (and then, usually, # eventually works). - return '' unless $ho->{Suite} =~ m/stretch/; + return '' unless $ho->{Suite} =~ m/stretch|buster/; my $script = "$rootdir/lib/udev/ifupdown-hotplug"; <<END.<<'ENDQ'.<<END set -ex
Signature · 1 line
2.20.1
PATCH v2 38/49 ts-xen-install: Add $ho argument to some_extradebs
In reply to Ian JacksonThis is going to move to Debian.pm.
--- ts-xen-install | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
ts-xen-install +4 -4
@@ -71,8 +71,8 @@ sub packages () { if toolstack($ho)->{ExtraPackages}; } -sub some_extradebs ($) { - my ($items) = @_; +sub some_extradebs ($$) { + my ($ho, $items) = @_; my $cfgvar = join('_', @$items); my $specs = $c{$cfgvar}; if (!length $specs) {
@@ -111,11 +111,11 @@ sub extradebs () { my $suite = $ho->{Suite}; # $c{ DebianExtraPackages_<suite> } - some_extradebs([ 'DebianExtraPackages', $suite ]); + some_extradebs($ho, [ 'DebianExtraPackages', $suite ]); # $c{ DebianExtraPackages_<firmware>_<arch>_<suite> } my $firmware = get_host_property($ho, "firmware"); - some_extradebs([ 'DebianExtraPackages', $firmware, $ho->{Arch}, $suite ]); + some_extradebs($ho, [ 'DebianExtraPackages', $firmware, $ho->{Arch}, $suite ]); } sub extract () {
Signature · 1 line
2.20.1
PATCH v2 39/49 ts-xen-install: Move some_extradebs to Debian.pm
In reply to Ian Jackson--- Osstest/Debian.pm | 38 ++++++++++++++++++++++++++++++++++++++ ts-xen-install | 36 ------------------------------------ 2 files changed, 38 insertions(+), 36 deletions(-)
Osstest/Debian.pm +38 -0
@@ -24,6 +24,7 @@ use POSIX; use IO::File; use File::Copy; +use File::Basename; use Osstest; use Osstest::TestSupport;
@@ -50,6 +51,7 @@ BEGIN { di_vg_name debian_dhcp_rofs_fix debian_write_random_seed_command + some_extradebs ); %EXPORT_TAGS = ( );
@@ -1646,4 +1648,40 @@ sub debian_write_random_seed_command ($) { END } +sub some_extradebs ($$) { + my ($ho, $items) = @_; + my $cfgvar = join('_', @$items); + my $specs = $c{$cfgvar}; + if (!length $specs) { + logm("$cfgvar: no extra debs"); + return; + } + my $counter = 0; + my $rsync_installed; + foreach my $spec (split /\s+/, $specs) { + my $path = $spec; + $path = "$c{Images}/$path" unless $path =~ m{^/|^\./}; + my ($ontarget, $dpkgopts); + if ($path =~ m{/$}) { + $ontarget = "extrapackages-$cfgvar-$counter"; $counter++; + $dpkgopts = '-iGROEB'; + logm("$cfgvar: updating packages from directory $path"); + target_install_packages($ho, qw(rsync)) unless $rsync_installed++; + target_putfile_root($ho,300, "$path/.", $ontarget, '-r'); + } elsif ($path =~ m{\.deb$}) { + $path =~ s{_\.deb}{ "_$ho->{Arch}.deb" }e; + logm("$cfgvar: installing $path"); + $ontarget = basename($path); + $dpkgopts = '-iB'; + target_putfile_root($ho,300, $path, $ontarget); + } else { + die "no / or . deb in $spec ?"; + } + target_cmd_root($ho, + "dpkg --force-confold $dpkgopts $ontarget </dev/null", + 300); + target_run_pkgmanager_install($ho, [], 0,1); + } +} + 1;
ts-xen-install +0 -36
@@ -71,42 +71,6 @@ sub packages () { if toolstack($ho)->{ExtraPackages}; } -sub some_extradebs ($$) { - my ($ho, $items) = @_; - my $cfgvar = join('_', @$items); - my $specs = $c{$cfgvar}; - if (!length $specs) { - logm("$cfgvar: no extra debs"); - return; - } - my $counter = 0; - my $rsync_installed; - foreach my $spec (split /\s+/, $specs) { - my $path = $spec; - $path = "$c{Images}/$path" unless $path =~ m{^/|^\./}; - my ($ontarget, $dpkgopts); - if ($path =~ m{/$}) { - $ontarget = "extrapackages-$cfgvar-$counter"; $counter++; - $dpkgopts = '-iGROEB'; - logm("$cfgvar: updating packages from directory $path"); - target_install_packages($ho, qw(rsync)) unless $rsync_installed++; - target_putfile_root($ho,300, "$path/.", $ontarget, '-r'); - } elsif ($path =~ m{\.deb$}) { - $path =~ s{_\.deb}{ "_$ho->{Arch}.deb" }e; - logm("$cfgvar: installing $path"); - $ontarget = basename($path); - $dpkgopts = '-iB'; - target_putfile_root($ho,300, $path, $ontarget); - } else { - die "no / or . deb in $spec ?"; - } - target_cmd_root($ho, - "dpkg --force-confold $dpkgopts $ontarget </dev/null", - 300); - target_run_pkgmanager_install($ho, [], 0,1); - } -} - sub extradebs () { my $suite = $ho->{Suite};
Signature · 1 line
2.20.1
PATCH v2 40/49 Debian.pm: Break out standard_extradebs
In reply to Ian JacksonRefactor this out of ts-xen-install. We are going to run it in ts-host-install.
--- Osstest/Debian.pm | 7 ++++++- ts-xen-install | 3 +-- 2 files changed, 7 insertions(+), 3 deletions(-)
Osstest/Debian.pm +6 -1
@@ -51,7 +51,6 @@ BEGIN { di_vg_name debian_dhcp_rofs_fix debian_write_random_seed_command - some_extradebs ); %EXPORT_TAGS = ( );
@@ -1684,4 +1683,10 @@ sub some_extradebs ($$) { } } +sub standard_extradebs ($) { + my ($ho) = @_; + # $c{ DebianExtraPackages_<suite> } + some_extradebs($ho, [ 'DebianExtraPackages', $ho->{Suite} ]); +} + 1;
ts-xen-install +1 -2
@@ -74,8 +74,7 @@ sub packages () { sub extradebs () { my $suite = $ho->{Suite}; - # $c{ DebianExtraPackages_<suite> } - some_extradebs($ho, [ 'DebianExtraPackages', $suite ]); + standard_extradebs($ho); # $c{ DebianExtraPackages_<firmware>_<arch>_<suite> } my $firmware = get_host_property($ho, "firmware");
Signature · 1 line
2.20.1
PATCH v2 41/49 Debian.pm: Move standard_extradebs to ts-host-install
In reply to Ian JacksonThis makes it effect builds on Debian, too.
--- Osstest/Debian.pm | 1 + ts-host-install | 2 ++ ts-xen-install | 2 -- 3 files changed, 3 insertions(+), 2 deletions(-)
Osstest/Debian.pm +1 -0
@@ -51,6 +51,7 @@ BEGIN { di_vg_name debian_dhcp_rofs_fix debian_write_random_seed_command + some_extradebs standard_extradebs ); %EXPORT_TAGS = ( );
ts-host-install +2 -0
@@ -146,6 +146,8 @@ END qw(amd64-microcode intel-microcode)); } + standard_extradebs($ho); + my $ntpserver = get_target_property($ho, 'NtpServer'); if ($ntpserver) { target_editfile_root($ho, '/etc/ntp.conf', sub {
ts-xen-install +0 -2
@@ -74,8 +74,6 @@ sub packages () { sub extradebs () { my $suite = $ho->{Suite}; - standard_extradebs($ho); - # $c{ DebianExtraPackages_<firmware>_<arch>_<suite> } my $firmware = get_host_property($ho, "firmware"); some_extradebs($ho, [ 'DebianExtraPackages', $firmware, $ho->{Arch}, $suite ]);
Signature · 1 line
2.20.1
PATCH v2 42/49 buster: Install own linux-libc-dev package (!)
In reply to Ian JacksonAs reported here:
https://patchew.org/QEMU/[email protected]/
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960271
the kernel has broken the build of upstream qemu. This made it into a Debian stable kernel update. This breaks our CI runs almost completely, when they run with buster.
I spoke to the Debian kernel folks and apparently there is no intent to fast-track a fix to this. So instead I have made a kernel source package with the patch from that bug report, and built the linux-libc-dev package fromk it. The source is here for the time being:
https://www.chiark.greenend.org.uk/~ijackson/quicksand/2020/libc-kernel-bug.960271/
Deployment note: the source and linux-libc-dev_*.deb are in the images directory on osstest@test-lab.
--- production-config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
production-config +1 -1
@@ -108,7 +108,7 @@ TftpGrubVersion XXXX-XX-XX DebianExtraPackages_jessie chiark-scripts_6.0.3~citrix1_all.deb DebianExtraPackages_stretch chiark-scripts_6.0.4~citrix1_all.deb -DebianExtraPackages_buster chiark-scripts_6.0.5~citrix1_all.deb +DebianExtraPackages_buster chiark-scripts_6.0.5~citrix1_all.deb libc-kernel-bug.960271/linux-libc-dev_4.19.118-2.0iwj_.deb DebianExtraPackages_uefi_i386_jessie extradebs-uefi-i386-2018-04-01/ DebianExtraPackages_uefi_amd64_jessie extradebs-uefi-amd64-2018-04-01/
Signature · 1 line
2.20.1
PATCH v2 43/49 setupboot_grub2: Insist on space after directives
In reply to Ian JacksonThese parsing regexps were all wrong!
--- Osstest/Debian.pm | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
Osstest/Debian.pm +4 -4
@@ -529,17 +529,17 @@ sub setupboot_grub2 ($$$$) { if (m/^\s*submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) { push @offsets,(0); } - if (m/^\s*chainloader\s*\/EFI\/osstest\/xen.efi/) { + if (m/^\s*chainloader\s+\/EFI\/osstest\/xen.efi/) { die unless $entry; $entry->{Hv}= $1; $entry->{Chainload} = 1; } - if (m/^\s*multiboot2?\s*(?:\/boot)?\/(xen\-[0-9][-+.0-9a-z]*\S+)/) { + if (m/^\s*multiboot2?\s+(?:\/boot)?\/(xen\-[0-9][-+.0-9a-z]*\S+)/) { die unless $entry; $entry->{Hv}= $1; $entry->{Chainload} = 0; } - if (m/^\s*multiboot2?\s*(?:\/boot)?\/(vmlinu[xz]-(\S+))\s+(.*)/) { + if (m/^\s*multiboot2?\s+(?:\/boot)?\/(vmlinu[xz]-(\S+))\s+(.*)/) { die unless $entry; $entry->{KernOnly}= $1; $entry->{KernVer}= $2;
@@ -551,7 +551,7 @@ sub setupboot_grub2 ($$$$) { $entry->{KernVer}= $2; $entry->{KernOpts}= $3; } - if (m/^\s*module2?\s*(?:\/boot)?\/(initrd\S+)/) { + if (m/^\s*module2?\s+(?:\/boot)?\/(initrd\S+)/) { $entry->{Initrd}= $1; } if (m/^\s*module2?\s*\/(xenpolicy\S+)/) {
Signature · 1 line
2.20.1
PATCH v2 44/49 setupboot_grub2: Print line number of entry we are using
In reply to Ian Jackson--- Osstest/Debian.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
@@ -562,7 +562,7 @@ sub setupboot_grub2 ($$$$) { die unless $entry->{Title}; - logm("boot check: grub2, found $entry->{Title}"); + logm("boot check: grub2, l.$., found $entry->{Title}"); die unless $entry->{$kernkey}; if (defined $xenhopt) {
Signature · 1 line
2.20.1
PATCH v2 45/49 setupboot_grub2: Recognise --nounzip for initramfs
In reply to Ian JacksonModern versions of update-grub like to add this. We need to spot this so that under EFI we generate the right things in xen.cfg.
--- Osstest/Debian.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
@@ -551,7 +551,7 @@ sub setupboot_grub2 ($$$$) { $entry->{KernVer}= $2; $entry->{KernOpts}= $3; } - if (m/^\s*module2?\s+(?:\/boot)?\/(initrd\S+)/) { + if (m/^\s*module2?\s+(?:--nounzip\s+)*(?:\/boot)?\/(initrd\S+)/) { $entry->{Initrd}= $1; } if (m/^\s*module2?\s*\/(xenpolicy\S+)/) {
Signature · 1 line
2.20.1
PATCH v2 46/49 setupboot_grub2: Copy hv command line from grub to xen.cfg
In reply to Ian JacksonThis reuses all of the stuff that update-grub, etc., have put there.
In particular without this we never have flask=enforcing!
We have to do something about the ${xen_rm_opts} that appear in these entries. In principle there might be many variable expansions, but in practice there is only this one It applies only to x86, and this use of chainloading to xen.efi and reading xen.cfg applies only to arm64. And anyway we weren't putting it these things in before. So OK...
--- Osstest/Debian.pm | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
Osstest/Debian.pm +6 -3
@@ -534,10 +534,11 @@ sub setupboot_grub2 ($$$$) { $entry->{Hv}= $1; $entry->{Chainload} = 1; } - if (m/^\s*multiboot2?\s+(?:\/boot)?\/(xen\-[0-9][-+.0-9a-z]*\S+)/) { + if (m/^\s*multiboot2?\s+(?:\/boot)?\/(xen\-[0-9][-+.0-9a-z]*\S+)\s+(.*)/) { die unless $entry; $entry->{Hv}= $1; $entry->{Chainload} = 0; + $entry->{HvOpts} = $2; } if (m/^\s*multiboot2?\s+(?:\/boot)?\/(vmlinu[xz]-(\S+))\s+(.*)/) { die unless $entry;
@@ -575,7 +576,7 @@ sub setupboot_grub2 ($$$$) { # Propagate relevant fields of the main entry over to the # chain entry for use of subsequent code. foreach (qw(KernVer KernDom0 KernOnly KernOpts - Initrd Xenpolicy)) { + Initrd Xenpolicy HvOpts)) { next unless $entry->{$_}; die if $chainentry->{$_}; $chainentry->{$_} = $entry->{$_};
@@ -604,12 +605,14 @@ sub setupboot_grub2 ($$$$) { if ($need_uefi_chainload) { my $entry= $parsemenu->(); + my $hvopts = $entry->{HvOpts}; + $hvopts =~ s/\$\{\w+\}//g; # delete these and hope! my $xencfg = <<END; [global] default=osstest [osstest] -options=$xenhopt +options=$hvopts kernel=vmlinuz $entry->{KernOpts} END $xencfg .= "ramdisk=initrd.gz\n" if $entry->{Initrd};
Signature · 1 line
2.20.1
PATCH v2 47/49 setupboot_grub2: Do not boot with XSM policy etc. unless xsm=1
In reply to Ian JacksonThis prevents us from passing an XSM policy file, and `flask=enforcing', in supposedly-non-XSM tests.
These bootloader entries can appear because the Xen upstream build ships XSM policy files by default even if XSM is disabled in the hypervisor, causing update-grub to generate useless `XSM enabled' entries.
--- Osstest/Debian.pm | 3 +++ 1 file changed, 3 insertions(+)
Osstest/Debian.pm +3 -0
@@ -499,6 +499,9 @@ sub setupboot_grub2 ($$$$) { } elsif ($want_xsm && !defined $entry->{Xenpolicy}) { logm("(skipping entry at $entry->{StartLine}..$.;". " XSM policy file not mentioned)"); + } elsif (!$want_xsm && defined $entry->{Xenpolicy}) { + logm("(skipping entry at $entry->{StartLine}..$.;". + " XSM policy file, but we don't want XSM)"); } elsif ($ho->{Suite} =~ m/buster/ && defined $entry->{Xenpolicy} && !$bootfiles{
Signature · 1 line
2.20.1
PATCH v2 48/49 buster 20_linux_xen: Only load policy in XSM-enabled builds
In reply to Ian Jackson--- overlay-buster/etc/grub.d/20_linux_xen | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
overlay-buster/etc/grub.d/20_linux_xen +1 -1
@@ -159,7 +159,7 @@ EOF ${module_loader} --nounzip ${rel_dirname}/${initrd} EOF fi - if test -n "${xenpolicy}" ; then + if ${xsm} && test -n "${xenpolicy}" ; then message="$(gettext_printf "Loading XSM policy ...")" sed "s/^/$submenu_indentation/" << EOF echo '$(echo "$message" | grub_quote)'
Signature · 1 line
2.20.1
PATCH v2 49/49 buster: Switch to Debian buster as the default suite
In reply to Ian Jackson--- Osstest.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest.pm +1 -1
@@ -87,7 +87,7 @@ our %c = qw( Images images - DebianSuite stretch + DebianSuite buster DebianMirrorSubpath debian TestHostKeypairPath id_rsa_osstest
Signature · 1 line
2.20.1