Skip to contentSkip to conversation

Upgrade most hosts/guests to buster

PATCH Superseded Ian Jackson 4 participants 51 messages May 19, 2020 – May 29, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Upgrade most hosts/guests to buster

buster is Debian stable right now. We don't want to be using oldstable because Debian have a history of breaking it.

Paul: we should have a conversation about whether this should be pushed soon, or deferred until after the Xen 4.14 release.

I have yet to do a final full formal retest of this series so it may not be final.

Ian Jackson (38):
ts-logs-capture: Cope if xl shutdown leaves domain running for a bit
ts-xen-build-prep: Install rsync
lvcreate argments: pass --yes -Z y -W y
TestSupport: allow more time for apt
Booting: Use --' rather than ---' to introduce host cmdline
di_installcmdline_core: Pass locale on d-i command line
setupboot_grub2: Drop $submenu variable
ts-leak-check: Ignore buster's udevd too
Bodge systemd random seed arrangements
Debian guests: Write systemd random seed file
ts-debian-di-install: Provide guest with more RAM
Debian: preseed: use priority= alias
Debian: Specify `priority=critical' rather than locale Honour 'LinuxSerialConsole <suite>' host property
buster: make-hosts-flight: Add to possible suites for hosts flight
buster: Extend grub2 uefi no install workaround
buster: ts-host-install: Extend net.ifnames workaround
buster: Deinstall the "systemd" package
buster: preseed partman-auto-lvm/guided_size
buster: ts-host-install: NTP not honoured bug remains
buster: Extend ARM clock workaround
buster: Extend guest bootloaer workaround
Honour DebianImageFile_SUITE_ARCH
buster: Specify DebianImageFile_SUITE_ARCH
20_linux_xen: Copy Debian buster version into our initramfs area 20_linux_xen: Adhoc template substitution
20_linux_xen: Ignore xenpolicy and config files too 20_linux_xen: Support Xen Security Modules (XSM/FLASK) mg-debian-installer-update: support overlay-intramfs-SUITE overlay-initrd-buster/sbin/reopen-console: Copy from Debian overlay-initrd-buster/sbin/reopen-console: Fix #932416
buster: chiark-scripts: Install a new version on buster too
buster: Provide TftpDiVersion
buster: grub, arm64: extend chainloading workaround
buster: setupboot_grub2: Note what files exist in /boot
buster: setupboot_grub2: Handle missing policy file bug
buster: Extend workaround for dhcpd EROFS bug
buster: Switch to Debian buster as the default suite

 Osstest.pm                                    |   2 +-
 Osstest/Debian.pm                             |  60 +++-
 Osstest/TestSupport.pm                        |  15 +-
 make-hosts-flight                             |   2 +-
 mfi-common                                    |   9 +
 mg-debian-installer-update                    |  20 ++
 overlay-buster/etc/grub.d/20_linux_xen        | 327 ++++++++++++++++++
 overlay-initrd-buster/sbin/reopen-console     | 126 +++++++
 .../override.conf                             |   3 +
 overlay/usr/local/bin/random-seed-add         |  33 ++
 production-config                             |   5 +
 sg-run-job                                    |   1 +
 ts-debian-di-fixup                            |  29 ++
 ts-debian-di-install                          |   2 +-
 ts-debian-fixup                               |   1 +
 ts-host-install                               |   4 +-
 ts-leak-check                                 |   1 +
 ts-logs-capture                               |   1 +
 ts-xen-build-prep                             |   6 +-
 19 files changed, 617 insertions(+), 30 deletions(-)
 create mode 100755 overlay-buster/etc/grub.d/20_linux_xen
 create mode 100755 overlay-initrd-buster/sbin/reopen-console
 create mode 100644 overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf
 create mode 100755 overlay/usr/local/bin/random-seed-add
 create mode 100755 ts-debian-di-fixup
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/38 ts-logs-capture: Cope if xl shutdown leaves domain running for a bit

This seems mostly to affect buster but it could in principle affect earlier releases too I think.

In principle it would be nice to fix this bug, and to have a proper test for it, but a reliable test is hard and an unreliable one is not useful. So I guess we are going to have this workaround indefinitely...

---
 ts-logs-capture | 1 +
 1 file changed, 1 insertion(+)
ts-logs-capture +1 -0
index 0320a5a5..d75a2fda 100755
@@ -272,6 +272,7 @@ sub shutdown_guests () {
 		( xl shutdown -a -F -w ; echo y ) &
 	    ) | (
 		read x
+		sleep 10 # xl shutdown is a bit racy :-/
 		xl list | awk '!/^Domain-0 |^Name / {print $2}' \
 		| xargs -t -r -n1 xl destroy ||:
 	    )
Signature · 1 line
2.20.1
#2 lore raw

Re: [OSSTEST PATCH 01/38] ts-logs-capture: Cope if xl shutdown leaves domain running for a bit

Quoted 8 lines · Ian Jackson

This seems mostly to affect buster but it could in principle affect earlier releases too I think.

In principle it would be nice to fix this bug, and to have a proper test for it, but a reliable test is hard and an unreliable one is not useful. So I guess we are going to have this workaround indefinitely...

xl shutdown -w waits for either domain shutdown or destruction in wait_for_domain_deaths()
https://github.com/xen-project/xen/blob/master/tools/xl/xl_vmcontrol.c#L183

My understanding is shutdown happens first when the guest stops and destruction happens afterward when all the resources are cleaned up. So your race is that the domain shutdown, but it still shows up in xl list since it hasn't been destroyed.

OpenXT has a hack to only wait for destruction
https://github.com/OpenXT/xenclient-oe/blob/master/recipes-extended/xen/files/xl-shutdown-wait-for-domain-death.patch.
I didn't write it, and the explanation isn't specific, but I think the purpose is to all resources are released before the OpenXT toolstack proceeds with (blktap) cleanup.

Maybe a new xl shutdown flag to wait for domain destruction would be worthwhile?

Regards,
Jason

#3 lore raw

Re: [OSSTEST PATCH 01/38] ts-logs-capture: Cope if xl shutdown leaves domain running for a bit

Jason Andryuk writes ("Re: [OSSTEST PATCH 01/38] ts-logs-capture: Cope if xl shutdown leaves domain running for a bit"):

Quoted 5 lines

My understanding is shutdown happens first when the guest stops and destruction happens afterward when all the resources are cleaned up. So your race is that the domain shutdown, but it still shows up in xl list since it hasn't been destroyed.

Quoted 8 lines

OpenXT has a hack to only wait for destruction
https://github.com/OpenXT/xenclient-oe/blob/master/recipes-extended/xen/files/xl-shutdown-wait-for-domain-death.patch.
I didn't write it, and the explanation isn't specific, but I think the purpose is to all resources are released before the OpenXT toolstack proceeds with (blktap) cleanup.

Maybe a new xl shutdown flag to wait for domain destruction would be worthwhile?

Yes! Repeating the -w flag maybe.

Ian.

#4 lore raw

PATCH 2/38 ts-xen-build-prep: Install rsync

In reply to Ian Jackson

osstest uses this for transferring configuration, build artefacts, and so on.

In Debian stretch and earlier, rsync happened to be pulled in by something else.

---
 ts-xen-build-prep | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ts-xen-build-prep +1 -1
index e9298d54..8e73f763 100755
@@ -197,7 +197,7 @@ END
 }
 
 sub prep () {
-    my @packages = qw(mercurial
+    my @packages = qw(mercurial rsync
                       build-essential bin86 bcc iasl bc
                       flex bison cmake
                       libpci-dev libncurses5-dev libssl-dev python-dev
Signature · 1 line
2.20.1
#5 lore raw

PATCH 3/38 lvcreate argments: pass --yes -Z y -W y

In reply to Ian Jackson

The documentation seesm to think this is the default but empirically it isn't. In our environment --yes is fine.

I have reported this to Debian as #953183. Also vaguely related (and discovered by me at the same time) is #953185.

This came up while trying to get things work on buster. I don't know what has changed.

---
 Osstest/TestSupport.pm | 2 +-
 ts-xen-build-prep      | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
Osstest/TestSupport.pm +1 -1
index 1e7da676..43766ee3 100644
@@ -935,7 +935,7 @@ sub lv_create ($$$$) {
     my ($ho, $vg, $lv, $mb) = @_;
     my $lvdev = "/dev/$vg/$lv";
     target_cmd_root($ho, "lvremove -f $lvdev ||:");
-    target_cmd_root($ho, "lvcreate -L ${mb}M -n $lv $vg");
+    target_cmd_root($ho, "lvcreate --yes -Z y -W y -L ${mb}M -n $lv $vg");
     target_cmd_root($ho, "dd if=/dev/zero of=$lvdev count=10");
     return $lvdev;
 }
ts-xen-build-prep +2 -2
index 8e73f763..dabb9921 100755
@@ -61,7 +61,7 @@ sub determine_vg_lv () {
 sub lvextend_stage1 () {
     target_cmd_root($ho, <<END);
         set -ex; if ! test -f /root/swap_osstest_enabled; then
-            lvcreate -L 10G -n swap_osstest_build $vg ||:
+            lvcreate --yes -Z y -W y -L 10G -n swap_osstest_build $vg ||:
             mkswap /dev/$vg/swap_osstest_build ||:
             swapon /dev/$vg/swap_osstest_build
             touch /root/swap_osstest_enabled
@@ -84,7 +84,7 @@ sub vginfo () {
 
 sub lvcreate () {
     target_cmd_output_root($ho,
-			   "lvdisplay $lv || lvcreate -l 1 -n $lvleaf $vg");
+			   "lvdisplay $lv || lvcreate --yes -Z y -W y -l 1 -n $lvleaf $vg");
 }
 
 sub lvextend1 ($$$) {
Signature · 1 line
2.20.1
#6 lore raw

PATCH 4/38 TestSupport: allow more time for apt

In reply to Ian Jackson

Empirically some of these operations can take longer than 30s, especially with a cold cache.

Note that because of host sharing and our on-host apt lock, the timeout needs to be the same for every apt operation: a fast operation could be blocked behind a slow one.

---
 Osstest/TestSupport.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Osstest/TestSupport.pm +2 -2
index 43766ee3..f4e9414c 100644
@@ -637,12 +637,12 @@ sub target_install_packages_nonfree_nonconcurrent ($@) {
     my ($ho, @packages) = @_;
     my $slist= '/etc/apt/sources.list';
     my $xsuites= 'contrib non-free';
-    target_cmd_root($ho, <<END, 30);
+    target_cmd_root($ho, <<END, 300);
     perl -i~ -pe 'next unless m/^deb/; s{ main\$}{\$& $xsuites};' $slist
     apt-get update
 END
     target_run_pkgmanager_install($ho,\@packages);
-    target_cmd_root($ho, <<END, 30);
+    target_cmd_root($ho, <<END, 300);
     mv $slist~ $slist
     apt-get update
 END
Signature · 1 line
2.20.1
#7 lore raw

PATCH 5/38 Booting: Use `--' rather than `---' to introduce host cmdline

In reply to Ian Jackson

Because systemd did something obnoxious, the kernel retaliated in the game of Core Wars by hiding all arguments before --' from userspace. So use ---' instead so that all the arguments remain visible.

This in some sense now applies to host installs a change we had already made to Debian HVM guests. See osstest#493b7395
ts-debian-hvm-install: Use ---, and no longer duplicate $gconsole
and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007
Kernel command line handling change breaks d-i user-params functionality

This change is fine for all non-ancient versions of Debian, so I have not made it conditional.

---
 Osstest/TestSupport.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Osstest/TestSupport.pm +2 -2
index f4e9414c..ff8103f2 100644
@@ -2909,7 +2909,7 @@ label overwrite
 	menu label ^Overwrite
 	menu default
 	kernel $kern
-	append $dicmd initrd=$initrd -- $hocmd
+	append $dicmd initrd=$initrd --- $hocmd
 	ipappend $xopts{ipappend}
 	$dtbs
 default overwrite
@@ -2956,7 +2956,7 @@ sub setup_netboot_di_uefi ($$$$$;%) {
 set default=0
 set timeout=5
 menuentry 'overwrite' {
-  linux $kern $dicmd -- $hocmd
+  linux $kern $dicmd --- $hocmd
   initrd $initrd
 }
 END
Signature · 1 line
2.20.1
#8 lore raw

PATCH 6/38 di_installcmdline_core: Pass locale on d-i command line

In reply to Ian Jackson

In buster, d-i wants when setting up the network, ie before the preseed is loaded.

We leave it in the preseed too because why not.

I think this change should be fine for older versions of Debian.

---
 Osstest/Debian.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Osstest/Debian.pm +2 -1
index 6e9d2072..ba975b87 100644
@@ -699,7 +699,8 @@ sub di_installcmdline_core ($$;@) {
                "hostname=$tho->{Name}",
                "$xopts{PreseedScheme}=$ps_url",
                "netcfg/dhcp_timeout=150",
-               "netcfg/choose_interface=$netcfg_interface"
+               "netcfg/choose_interface=$netcfg_interface",
+               "debian-installer/locale=en_GB",
                );
 
     my $debconf_priority= $xopts{DebconfPriority};
Signature · 1 line
2.20.1
#9 lore raw

PATCH 7/38 setupboot_grub2: Drop $submenu variable

In reply to Ian Jackson

We really only used this to check how many levels deep in { we are. That can be done by checking $#offsets, which is >0 if we are in a submenu and not otherwise. We lose the ability to report the start line of the submenu, but that's OK.

But as a bonus, we no longer bomb out on nested submenus: previously the first } would cause $submenu to be undef. Now we pop from @offsets and all is fine.

Nested submenus are present in Debian buster.

---
 Osstest/Debian.pm | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
Osstest/Debian.pm +3 -6
index ba975b87..b8bf67dc 100644
@@ -452,15 +452,13 @@ sub setupboot_grub2 ($$$$) {
         my @offsets = (0);
         my $entry;
         my $chainentry;
-        my $submenu;
         while (<$f>) {
             next if m/^\s*\#/ || !m/\S/;
             if (m/^\s*\}\s*$/) {
-                die unless $entry || $submenu;
-                if (!$entry && $submenu) {
-                    logm("Met end of a submenu $submenu->{StartLine}..$.. ".
+                die unless $entry || $#offsets;
+                if (!$entry && $#offsets) {
+                    logm("Met end of a submenu at $. (@offsets) ".
                         "Our want kern is $want_kernver");
-                    $submenu= undef;
                     pop @offsets;
                     $offsets[$#offsets]++;
                     next;
@@ -510,7 +508,6 @@ sub setupboot_grub2 ($$$$) {
                 $offsets[$#offsets]++;
             }
             if (m/^\s*submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
-                $submenu={ StartLine =>$., MenuEntryPath => join ">", @offsets };
                 push @offsets,(0);
             }
             if (m/^\s*chainloader\s*\/EFI\/osstest\/xen.efi/) {
Signature · 1 line
2.20.1
#10 lore raw

PATCH 8/38 ts-leak-check: Ignore buster's udevd too

In reply to Ian Jackson

For reasons I don't propose to investigate, on buster udevd shows up like this:

  2019-11-26 18:13:48 Z LEAKED [process 2633 /lib/systemd/systemd-udevd] process: root      2633  1555  0 18:10 ?        00:00:00 /lib/systemd/systemd-udevd

This does not match our suppression. Add an additional suppression.

---
 ts-leak-check | 1 +
 1 file changed, 1 insertion(+)
ts-leak-check +1 -0
index 41e6245d..f3cca8aa 100755
@@ -202,6 +202,7 @@ xenstore /vm
 xenstore /libxl
 
 process .* udevd
+process .* .*/systemd-udevd
 process .* /.+/systemd-shim
 
 file /var/run/xenstored/db
Signature · 1 line
2.20.1
#11 lore raw

PATCH 9/38 Bodge systemd random seed arrangements

In reply to Ian Jackson

systemd does not regard the contents of the random seed file as useful for the purposes of placating the kernel's entropy tracker. As a result, the system hangs at boot waiting for entropy.

Fix this by providing a small program which can be used to load a seed file into /dev/random and also call RNDADDTOENTCNT to add the appropriate amount to the kernel's counter.

Arrange to run this program instead of
/lib/systemd/systemd-random-seed load

With systemd the random seed file is in /var/lib/systemd/random-seed rather than /var/lib/urandom/random-seed.

Unfortunately we must hardcode the actual numerical value of RNDADDTOENTCNT because we don't have a suitable compiler anywhere nearby. It seems to have the same value on i386, amd64, armhf and arm64, our currently supported architectures.

Thanks to Colin Watson for pointers to the systemd random unit and Matthew Vernon for instructions on overriding just ExecStart.

I think this change should be a no-op on non-systemd systems.

---
 .../override.conf                             |  3 ++
 overlay/usr/local/bin/random-seed-add         | 33 +++++++++++++++++++
 2 files changed, 36 insertions(+)
 create mode 100644 overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf
 create mode 100755 overlay/usr/local/bin/random-seed-add
overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf new file +3 -0
new file mode 100644
index 00000000..f6cc0f84
@@ -0,0 +1,3 @@
+[Service]
+ExecStart=
+ExecStart=/usr/local/bin/random-seed-add /var/lib/systemd/random-seed
overlay/usr/local/bin/random-seed-add new file +33 -0
new file mode 100755
index 00000000..89e75c4d
@@ -0,0 +1,33 @@
+#!/usr/bin/perl -w
+use strict;
+
+open R, '>', '/dev/random' or die "open /dev/random: $!\n";
+R->autoflush(1);
+
+sub rndaddtoentcnt ($) {
+    my ($bits) = @_;
+    my $x = pack 'L', $bits;
+    my $r = ioctl R, 0x40045201, $x;
+    defined $r or die "RNDADDTOENTCNT: $!\n";
+}
+
+sub process_stdin ($) {
+    my ($f) = @_;
+    my $got = read STDIN, $_, 512;
+    defined $got or die "read $f: $!\n";
+    last if !$got;
+    print R $_ or die "write /dev/random: $!\n";
+    my $bits = length($_) * 8;
+    rndaddtoentcnt($bits);
+}
+
+if (!@ARGV) {
+    process_stdin('stdin');
+} else {
+    die "no options supported\n" if $ARGV[0] =~ m/^\-/;
+    foreach my $f (@ARGV) {
+        open STDIN, '<', $f or die "open for reading $f: $!\n";
+        process_stdin($f);
+    }
+}
+
Signature · 1 line
2.20.1
#12 lore raw

PATCH 10/38 Debian guests: Write systemd random seed file

In reply to Ian Jackson

This involves a new ts-debian-di-fixup script, which runs after xen-tools.

---
 Osstest/Debian.pm  | 14 ++++++++++++++
 sg-run-job         |  1 +
 ts-debian-di-fixup | 29 +++++++++++++++++++++++++++++
 ts-debian-fixup    |  1 +
 4 files changed, 45 insertions(+)
 create mode 100755 ts-debian-di-fixup
Osstest/Debian.pm +14 -0
index b8bf67dc..78d8c97e 100644
@@ -49,6 +49,7 @@ BEGIN {
                       di_installcmdline_core
                       di_vg_name
                       debian_dhcp_rofs_fix
+		      debian_write_random_seed
                       );
     %EXPORT_TAGS = ( );
 
@@ -1612,4 +1613,17 @@ mv '$script.new' '$script'
 END
 }
 
+sub debian_write_random_seed ($) {
+    my ($gho) = @_;
+    my $mountpoint = '/mnt';
+    my $ho = $gho->{Host};
+    target_cmd_root($ho, <<END);
+        set -ex
+        mount /dev/$gho->{Vg}/$gho->{Lv} $mountpoint
+        umask 077
+        dd if=/dev/urandom of=$mountpoint/var/lib/systemd/random-seed bs=1k count=1
+END
+    guest_umount_lv($ho, $gho);
+}
+
 1;
sg-run-job +1 -0
index aa7953ac..9255096d 100755
@@ -511,6 +511,7 @@ proc run-job/test-debian {} {
 
 proc install-guest-debian-di {} {
     run-ts . = ts-debian-di-install
+    run-ts . = ts-debian-di-fixup + debian
     run-ts . = ts-guest-start + debian
 }
 
ts-debian-di-fixup new file +29 -0
new file mode 100755
index 00000000..68cda2f5
@@ -0,0 +1,29 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+BEGIN { unshift @INC, qw(.); }
+use Osstest;
+use Osstest::TestSupport;
+use Osstest::Debian;
+
+tsreadconfig();
+
+our ($ho,$gho) = ts_get_host_guest(@ARGV);
+
+debian_write_random_seed($gho);
ts-debian-fixup +1 -0
index fef9836e..45bbcd27 100755
@@ -202,6 +202,7 @@ sub writecfg () {
     target_putfile_root($ho,10, $cfgstash, $cfgfile);
 }
 
+debian_write_random_seed($gho);
 savecfg();
 ether();
 target_kernkind_check($gho);
Signature · 1 line
2.20.1
#13 lore raw

PATCH 11/38 ts-debian-di-install: Provide guest with more RAM

In reply to Ian Jackson

buster cannot boot in so little because its initramfs and kernel are too large. Bump it to 2G.

However, our armhf test nodes have very little RAM. And the Debian armhf does fit in them as a guest still, so use a smaller value there.

Keying this off the architecture rather than the available host memory is better because you do need the bigger value precisely if you are not using armhf, and this makes osstest less dependent on a completely accurate and populated host properties database.

---
 ts-debian-di-install | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ts-debian-di-install +1 -1
index 9abb4956..d84407cf 100755
@@ -64,7 +64,7 @@ $gn ||= 'debian';
 
 our $ho= selecthost($whhost);
 
-our $ram_mb=    512;
+our $ram_mb= $r{arch} =~ m/^armhf/ ? 768 : 2048;
 our $disk_mb= 10000;
 
 our $guesthost= $gn.
Signature · 1 line
2.20.1
#14 lore raw

PATCH 12/38 Debian: preseed: use priority= alias

In reply to Ian Jackson

This marginally reduces command line clobber. This alias has been supported approximately forever. (And this code is currently only used when DebconfPriority is set, which it generally isn't.)

---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index 78d8c97e..c3fbc32c 100644
@@ -702,7 +702,7 @@ sub di_installcmdline_core ($$;@) {
                );
 
     my $debconf_priority= $xopts{DebconfPriority};
-    push @cl, "debconf/priority=$debconf_priority"
+    push @cl, "priority=$debconf_priority"
         if defined $debconf_priority;
     push @cl, "rescue/enable=true" if $xopts{RescueMode};
 
Signature · 1 line
2.20.1
#15 lore raw

PATCH 13/38 Debian: Specify `priority=critical' rather than locale

In reply to Ian Jackson

In buster, it appears that specifying locale on the command line is not sufficient. Rather than adding more things to the command line, instead, just say `priority=critical', by defaulting $debconf_priority to 'critical'.

I think this change should be fine for earlier suites too.

---
 Osstest/Debian.pm | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
Osstest/Debian.pm +2 -4
index c3fbc32c..8380c428 100644
@@ -698,12 +698,10 @@ sub di_installcmdline_core ($$;@) {
                "$xopts{PreseedScheme}=$ps_url",
                "netcfg/dhcp_timeout=150",
                "netcfg/choose_interface=$netcfg_interface",
-               "debian-installer/locale=en_GB",
                );
 
-    my $debconf_priority= $xopts{DebconfPriority};
-    push @cl, "priority=$debconf_priority"
-        if defined $debconf_priority;
+    my $debconf_priority= $xopts{DebconfPriority} // 'critical';
+    push @cl, "priority=$debconf_priority";
     push @cl, "rescue/enable=true" if $xopts{RescueMode};
 
     if ($r{syslog_server}) {
Signature · 1 line
2.20.1
#16 lore raw

PATCH 14/38 Honour 'LinuxSerialConsole <suite>' host property

In reply to Ian Jackson

This works like LinuxSerialConsole.

I originally wrote this to try to work around #940028, where multiple d-i autoinstalls run in parallel leading to hard-to-debug lossage! Explicitly specing the console causes it to run only on that one.

However, it turns out that explicitly specifying the console does not always work and a better fix is needed. Nevertheless, having added this feature it seems foolish to throw it away.

Currently there are no hosts with this property so no functiaonal change.

---
 Osstest/TestSupport.pm | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Osstest/TestSupport.pm +4 -1
index ff8103f2..7eeac49f 100644
@@ -1447,7 +1447,10 @@ sub get_target_property ($$;$) {
 sub get_host_native_linux_console ($) {
     my ($ho) = @_;
 
-    my $console = get_host_property($ho, "LinuxSerialConsole", "ttyS0");
+    my $console;
+    $console //= get_host_property($ho, "LinuxSerialConsole $ho->{Suite}")
+	if $ho->{Suite};
+    $console //= get_host_property($ho, "LinuxSerialConsole", "ttyS0");
     return $console if $console eq 'NONE';
 
     return "$console,$c{Baud}n8";
Signature · 1 line
2.20.1
#17 lore raw

PATCH 15/38 buster: make-hosts-flight: Add to possible suites for hosts flight

In reply to Ian Jackson
---
 make-hosts-flight | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
make-hosts-flight +1 -1
index 92da1c7c..e2c3776a 100755
@@ -26,7 +26,7 @@ blessing=$4
 buildflight=$5
 
 : ${ALL_ARCHES:=amd64 i386 arm64 armhf}
-: ${ALL_SUITES:=stretch jessie}
+: ${ALL_SUITES:=buster stretch jessie}
 # ^ most preferred suite first
 
 : ${PERHOST_MAXWAIT:=20000} # seconds
Signature · 1 line
2.20.1
#18 lore raw

PATCH 16/38 buster: Extend grub2 uefi no install workaround

In reply to Ian Jackson

src:grub2 is RFH in Debian, which is a contributory factor to these patches in #789798 and #792547 languishing.

---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index 8380c428..0958d080 100644
@@ -1459,7 +1459,7 @@ d-i partman-auto/expert_recipe string					\\
 END
 
     if (get_host_property($ho, "firmware") eq "uefi") {
-	die unless $ho->{Suite} =~ m/jessie|stretch/;
+	die unless $ho->{Suite} =~ m/jessie|stretch|buster/;
 	# Prevent grub-install from making a new Debian boot entry, so
 	# we always reboot from the network. Debian bug #789798 proposes a
 	# properly preseedable solution to this.
Signature · 1 line
2.20.1
#19 lore raw

PATCH 17/38 buster: ts-host-install: Extend net.ifnames workaround

In reply to Ian Jackson

Really we should fix this by making a .deb in Debian that we could install. But this is a longer-term project.

---
 ts-host-install | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ts-host-install +1 -1
index 7a72a867..fe26f70f 100755
@@ -282,7 +282,7 @@ END
 
     # Don't use "Predictable Network Interface Names"
     # https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
-    push @hocmdline, "net.ifnames=0" if $ho->{Suite} =~ m/stretch/;
+    push @hocmdline, "net.ifnames=0" if $ho->{Suite} =~ m/stretch|buster/;
 
     push @hocmdline,
         get_host_property($ho, "linux-boot-append $ho->{Suite}", ''),
Signature · 1 line
2.20.1
#20 lore raw

PATCH 18/38 buster: Deinstall the "systemd" package

In reply to Ian Jackson

This installs a pam rule which causes logins to hang. It also seems to cause some kind of udev wedge.

We are using sysvinit so this package is not desirable. Empirically, removing it makes the system work.

---
 Osstest/Debian.pm | 1 +
 1 file changed, 1 insertion(+)
Osstest/Debian.pm +1 -0
index 0958d080..cd8b2de0 100644
@@ -842,6 +842,7 @@ sub preseed_base ($$$;@) {
     if ( $suite !~ /squeeze|wheezy/ ) {
        preseed_hook_command($ho, 'late_command', $sfx, <<END)
 in-target apt-get install -y sysvinit-core
+in-target apt-get remove -y systemd
 END
     }
 
Signature · 1 line
2.20.1
#21 lore raw

PATCH 19/38 buster: preseed partman-auto-lvm/guided_size

In reply to Ian Jackson

Otherwise we get this question:

  | You may use the whole volume group for guided partitioning, or part
  | of it.  [...]
  | Amount of volume group to use for guided partitioning:
---
 Osstest/Debian.pm | 1 +
 1 file changed, 1 insertion(+)
Osstest/Debian.pm +1 -0
index cd8b2de0..bec72788 100644
@@ -938,6 +938,7 @@ d-i partman/choose_partition select finish
 d-i partman/confirm boolean true
 d-i partman-lvm/confirm boolean true
 d-i partman-lvm/device_remove_lvm_span boolean true
+d-i partman-auto-lvm/guided_size string max
 
 d-i partman/confirm_nooverwrite true
 d-i partman-lvm/confirm_nooverwrite true
Signature · 1 line
2.20.1
#22 lore raw

PATCH 20/38 buster: ts-host-install: NTP not honoured bug remains

In reply to Ian Jackson

Debian #778564 remains open.

---
 ts-host-install | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
ts-host-install +1 -1
index fe26f70f..253dbb5d 100755
@@ -152,7 +152,7 @@ END
 	    my $done= 0;
 	    while (<EI>) {
 		if (m/^server\b|^pool\b\s/) {
-		    if ($ho->{Suite} =~ m/lenny|squeeze|wheezy|jessie|stretch/) {
+		    if ($ho->{Suite} =~ m/lenny|squeeze|wheezy|jessie|stretch|buster/) {
 			$_= $done ? "" : "server $ntpserver\n";
 		    } else {
 			m/^server \Q$ntpserver\E\s/ or
Signature · 1 line
2.20.1
#23 lore raw

PATCH 21/38 buster: Extend ARM clock workaround

In reply to Ian Jackson
CC Julien Grall <[email protected]>
CC Stefano Stabellini <[email protected]>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index bec72788..6fed0b75 100644
@@ -248,7 +248,7 @@ END
 	push @xenkopt, $xenkopt;
 	# https://bugs.xenproject.org/xen/bug/45
 	push @xenkopt, "clk_ignore_unused"
-	    if $ho->{Suite} =~ m/wheezy|jessie|stretch/;
+	    if $ho->{Suite} =~ m/wheezy|jessie|stretch|buster/;
 
 	$xenkopt = join ' ', @xenkopt;
 	logm("Dom0 Linux options: $xenkopt");
Signature · 1 line
2.20.1
#25 lore raw

PATCH 22/38 buster: Extend guest bootloader workaround

In reply to Ian Jackson
CC Julien Grall <[email protected]>
CC Stefano Stabellini <[email protected]>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index 6fed0b75..77508d19 100644
@@ -1064,7 +1064,7 @@ END
     logm("\$arch is $arch, \$suite is $suite");
     if ($xopts{PvMenuLst} &&
 	$arch =~ /^arm/ &&
-	$suite =~ /wheezy|jessie|stretch|sid/ ) {
+	$suite =~ /wheezy|jessie|stretch|buster|sid/ ) {
 
 	# Debian doesn't currently know what bootloader to install in
 	# a Xen guest on ARM. We install pv-grub-menu above which
Signature · 1 line
2.20.1
Acked-by on v1 22/38
#26 lore raw

Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround

Hi Ian,

Quoted 3 lines · Ian Jackson
CC Julien Grall <[email protected]>
CC Stefano Stabellini <[email protected]>
Quoted 9 lines
---
  Osstest/Debian.pm | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +0 -0
index 6fed0b75..77508d19 100644

Osstest/Debian.pm END · line 1070

Quoted 8 lines
Osstest/Debian.pm +1 -1
@@ -1064,7 +1064,7 @@ END
      logm("\$arch is $arch, \$suite is $suite");
      if ($xopts{PvMenuLst} &&
  	$arch =~ /^arm/ &&
-	$suite =~ /wheezy|jessie|stretch|sid/ ) {
+	$suite =~ /wheezy|jessie|stretch|buster|sid/ ) {
 
  	# Debian doesn't currently know what bootloader to install in
  	# a Xen guest on ARM. We install pv-grub-menu above which

OOI, what does Debian install for x86 HVM guest? Is there any ticket tracking this issue?

Cheers,

Signature · 1 line
Julien Grall
#27 lore raw

Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround

Julien Grall writes ("Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround"):

Quoted 6 lines · Ian Jackson
Quoted 3 lines
CC Julien Grall <[email protected]>
CC Stefano Stabellini <[email protected]>

Thanks.

Quoted 5 lines
Quoted 2 lines

# Debian doesn't currently know what bootloader to install in

  	# a Xen guest on ARM. We install pv-grub-menu above which

OOI, what does Debian install for x86 HVM guest? Is there any ticket tracking this issue?

On x86, it installes grub. (grub2, x86, PC, to be precise.) I'm not aware of any ticket or bug about this.

Ian.

#28 lore raw

Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround

Hi Ian,

Quoted 10 lines · Ian Jackson

Julien Grall writes ("Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround"):

Quoted 6 lines · Ian Jackson
Quoted 3 lines
CC Julien Grall <[email protected]>
CC Stefano Stabellini <[email protected]>

Thanks.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

# Debian doesn't currently know what bootloader to install in

   	# a Xen guest on ARM. We install pv-grub-menu above which

OOI, what does Debian install for x86 HVM guest? Is there any ticket tracking this issue?

On x86, it installes grub. (grub2, x86, PC, to be precise.)

I have just realized that on x86 you will always have a firmware in the guest. On Arm we commonly boot the kernel directly.

So maybe we are closer to PV here. Do you also install GRUB in that case?

Note that we do support EDK2 at least on Arm64. It would be nice to get some tests for it in Osstest in the future.

Quoted 1 line

I'm not aware of any ticket or bug about this.

It might be worth creating one then.

Signature · 1 line
Julien Grall
#29 lore raw

Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround

Julien Grall writes ("Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround"):

Quoted 6 lines · Ian Jackson
Quoted 5 lines

Julien Grall writes ("Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround"):

Quoted 4 lines · Ian Jackson
Quoted 2 lines

# Debian doesn't currently know what bootloader to install in

   	# a Xen guest on ARM. We install pv-grub-menu above which
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

OOI, what does Debian install for x86 HVM guest? Is there any ticket tracking this issue?

On x86, it installes grub. (grub2, x86, PC, to be precise.)

I have just realized that on x86 you will always have a firmware in the guest. On Arm we commonly boot the kernel directly.

Yes. At leave, for HVM.

Quoted 1 line

So maybe we are closer to PV here. Do you also install GRUB in that case?

It's Complicated. There are several options, but the usual ones are:

1. pygrub: Install some version of grub, which generates /boot/grub.cfg. It doesn't matter very much which version of grub because grub.cfg is read by pygrub in dom0 and that fishes out the kernel and initrd. Many of osstest's tests do this.

2. host kernel: Simply pass the dom0 kernel *and initramfs* as the kernel image to the guest. This works if the kernel has the right modules for the guest storage, which it can easily do. On x86 an amd64 kernel can run an i386 userland.

3. pvgrub.

Quoted 2 lines

Note that we do support EDK2 at least on Arm64. It would be nice to get some tests for it in Osstest in the future.

Is this the same as "EADK" ? I'm afraid I don't follow...

Quoted 3 lines
Quoted 1 line

I'm not aware of any ticket or bug about this.

It might be worth creating one then.

Where should I do that ? I guess I mean, in which bugtracker ?

Thanks,
Ian.

#30 lore raw

Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround

Hi,

Quoted 26 lines · Ian Jackson

Julien Grall writes ("Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround"):

Quoted 6 lines · Ian Jackson
Quoted 5 lines

Julien Grall writes ("Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround"):

Quoted 4 lines · Ian Jackson
Quoted 2 lines

# Debian doesn't currently know what bootloader to install in # a Xen guest on ARM. We install pv-grub-menu above which

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

OOI, what does Debian install for x86 HVM guest? Is there any ticket tracking this issue?

On x86, it installes grub. (grub2, x86, PC, to be precise.)

I have just realized that on x86 you will always have a firmware in the guest. On Arm we commonly boot the kernel directly.

Yes. At leave, for HVM.

Quoted 1 line

So maybe we are closer to PV here. Do you also install GRUB in that case?

It's Complicated. There are several options, but the usual ones are:

1. pygrub: Install some version of grub, which generates /boot/grub.cfg. It doesn't matter very much which version of grub because grub.cfg is read by pygrub in dom0 and that fishes out the kernel and initrd. Many of osstest's tests do this.

Quoted 6 lines

2. host kernel: Simply pass the dom0 kernel *and initramfs* as the kernel image to the guest. This works if the kernel has the right modules for the guest storage, which it can easily do. On x86 an amd64 kernel can run an i386 userland.

3. pvgrub.

Thanks for the explanation. How do you select it in the Osstest today?

Is it a option for the debian installer or you do it manually as part of your install script?

Quoted 4 lines
Quoted 2 lines

Note that we do support EDK2 at least on Arm64. It would be nice to get some tests for it in Osstest in the future.

Is this the same as "EADK" ? I'm afraid I don't follow...

Sorry, I should have been more precise. I meant that we are able to boot a Arm guest using UEFI as we added support in EDK2 (I think in Xen we use the term ovmf).

When using EFI, the guest can boot exactly the same way as it would on baremetal. The toolstack is just loading the firmware in the guest memory.

IIRC we have already regular EFI testing on x86 in Osstest. I am thinking to extend them to Arm at some point.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

I'm not aware of any ticket or bug about this.

It might be worth creating one then.

Where should I do that ? I guess I mean, in which bugtracker ?

From the comment in the code, I would assume this is a bug/enhancement against the Debian installer. But I may have misundertood it.

Cheers,

Signature · 1 line
Julien Grall
#31 lore raw

Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround

Julien Grall writes ("Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround"):

Quoted 8 lines · Ian Jackson
Quoted 7 lines

It's Complicated. There are several options, but the usual ones are:

1. pygrub: Install some version of grub, which generates /boot/grub.cfg. It doesn't matter very much which version of grub because grub.cfg is read by pygrub in dom0 and that fishes out the kernel and initrd. Many of osstest's tests do this.

Quoted 8 lines
Quoted 6 lines

2. host kernel: Simply pass the dom0 kernel *and initramfs* as the kernel image to the guest. This works if the kernel has the right modules for the guest storage, which it can easily do. On x86 an amd64 kernel can run an i386 userland.

3. pvgrub.

Thanks for the explanation. How do you select it in the Osstest today?

I think osstest does all three (not very sure about (2). Installs made with the Debian xen-tools package tend to do (2) by default. Installs made with d-i can do (2) or (3).

Quoted 5 lines
Quoted 1 line

Is this the same as "EADK" ? I'm afraid I don't follow...

Sorry, I should have been more precise. I meant that we are able to boot a Arm guest using UEFI as we added support in EDK2 (I think in Xen we use the term ovmf).

Right.

Quoted 5 lines

When using EFI, the guest can boot exactly the same way as it would on baremetal. The toolstack is just loading the firmware in the guest memory.

IIRC we have already regular EFI testing on x86 in Osstest. I am thinking to extend them to Arm at some point.

Our arm64 boxes are all booting via UEFI right now.

We have to do a different bodge to load xen.efi rather than grub; osstest makes a xen.cfg. That bodge is extended to buster by

Subject: [OSSTEST PATCH 34/38] buster: grub, arm64: extend
chainloading workaround

Quoted 4 lines
Quoted 1 line

Where should I do that ? I guess I mean, in which bugtracker ?

From the comment in the code, I would assume this is a bug/enhancement
against the Debian installer. But I may have misundertood it.

Oh I see. I think amybe the problem was the lack of grub support. Is that all sorted in current Debian unstable/testing ? If so it may well all come out in the wash.

Ian.

#32 lore raw

Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround

Hi,

Quoted 36 lines · Ian Jackson

Julien Grall writes ("Re: [OSSTEST PATCH 22/38] buster: Extend guest bootloader workaround"):

Quoted 8 lines · Ian Jackson
Quoted 7 lines

It's Complicated. There are several options, but the usual ones are:

1. pygrub: Install some version of grub, which generates /boot/grub.cfg. It doesn't matter very much which version of grub because grub.cfg is read by pygrub in dom0 and that fishes out the kernel and initrd. Many of osstest's tests do this.

Quoted 8 lines
Quoted 6 lines

2. host kernel: Simply pass the dom0 kernel *and initramfs* as the kernel image to the guest. This works if the kernel has the right modules for the guest storage, which it can easily do. On x86 an amd64 kernel can run an i386 userland.

3. pvgrub.

Thanks for the explanation. How do you select it in the Osstest today?

I think osstest does all three (not very sure about (2). Installs made with the Debian xen-tools package tend to do (2) by default. Installs made with d-i can do (2) or (3).

Quoted 5 lines
Quoted 1 line

Is this the same as "EADK" ? I'm afraid I don't follow...

Sorry, I should have been more precise. I meant that we are able to boot a Arm guest using UEFI as we added support in EDK2 (I think in Xen we use the term ovmf).

Right.

Quoted 5 lines

When using EFI, the guest can boot exactly the same way as it would on baremetal. The toolstack is just loading the firmware in the guest memory.

IIRC we have already regular EFI testing on x86 in Osstest. I am thinking to extend them to Arm at some point.

Quoted 7 lines

Our arm64 boxes are all booting via UEFI right now.

We have to do a different bodge to load xen.efi rather than grub; osstest makes a xen.cfg. That bodge is extended to buster by

Subject: [OSSTEST PATCH 34/38] buster: grub, arm64: extend
chainloading workaround

We should also be able to use EFI in the guest directly as well :).

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

Where should I do that ? I guess I mean, in which bugtracker ?

From the comment in the code, I would assume this is a bug/enhancement
against the Debian installer. But I may have misundertood it.

Oh I see. I think amybe the problem was the lack of grub support. Is that all sorted in current Debian unstable/testing ? If so it may well all come out in the wash.

I haven't tried a recent Debian installer on Xen on Arm. I will have a try and see what it installs now.

Cheers,

Signature · 1 line
Julien Grall
#33 lore raw

PATCH 23/38 Honour DebianImageFile_SUITE_ARCH

In reply to Ian Jackson

This lets us specify the whole filename, not just a version. This is needed because for buster we are going to use
debian-10.2.0-ARCH-xfce-CD-1.iso

---
 mfi-common | 9 +++++++++
 1 file changed, 9 insertions(+)
mfi-common +9 -0
index b40f057e..640cf328 100644
@@ -522,6 +522,15 @@ job_create_test () {
 
 usual_debianhvm_image () {
   local arch=$1; shift
+  if [ -n "$DEBIAN_IMAGE_FILE" ]; then
+      echo $DEBIAN_IMAGE_FILE
+      return
+  fi
+  local file=`getconfig DebianImageFile_${guestsuite}_${arch}`
+  if [ -n "$file " ]; then
+      echo $file
+      return
+  fi
   local ver=$DEBIAN_IMAGE_VERSION
   if [ -z "$ver" ] ; then
       ver=`getconfig DebianImageVersion_$guestsuite`
Signature · 1 line
2.20.1
#34 lore raw

PATCH 24/38 buster: Specify DebianImageFile_SUITE_ARCH

In reply to Ian Jackson
---
 production-config | 3 +++
 1 file changed, 3 insertions(+)
production-config +3 -0
index 103b8915..f0ddc132 100644
@@ -98,6 +98,9 @@ DebianSnapshotBackports_jessie http://snapshot.debian.org/archive/debian/2019020
 DebianImageVersion_wheezy 7.2.0
 DebianImageVersion_jessie 8.2.0
 DebianImageVersion_stretch 9.4.0
+DebianImageFile_buster_amd64 debian-10.2.0-amd64-xfce-CD-1.iso
+DebianImageFile_buster_i386 debian-10.2.0-i386-xfce-CD-1.iso
+
 
 # Update with ./mg-netgrub-loader-update
 TftpGrubVersion XXXX-XX-XX
Signature · 1 line
2.20.1
#35 lore raw

PATCH 25/38 20_linux_xen: Copy Debian buster version into our initramfs area

In reply to Ian Jackson

This is from 41e42571ebc50fa351cd63ce40044946652c5c72 in Debian's grub package.

We are going to want to modify this to support XSM/FLASK and cope with upstream build outputs.

In this commit we dump the exact file contents across. It's not effective right now because of the ".in" extension. In fact, the file is a template.

At the time of writing I am trying to send our substantive changes upstream via Debian's Gitlab:
https://salsa.debian.org/grub-team/grub/-/merge_requests/18

---
 overlay-buster/etc/grub.d/20_linux_xen.in | 299 ++++++++++++++++++++++
 1 file changed, 299 insertions(+)
 create mode 100644 overlay-buster/etc/grub.d/20_linux_xen.in
overlay-buster/etc/grub.d/20_linux_xen.in new file +299 -0
new file mode 100644
index 00000000..98ef163c
@@ -0,0 +1,299 @@
+#! /bin/sh
+set -e
+
+# grub-mkconfig helper script.
+# Copyright (C) 2006,2007,2008,2009,2010  Free Software Foundation, Inc.
+#
+# GRUB is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# GRUB is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
+
+prefix="@prefix@"
+exec_prefix="@exec_prefix@"
+datarootdir="@datarootdir@"
+
+. "$pkgdatadir/grub-mkconfig_lib"
+
+export TEXTDOMAIN=@PACKAGE@
+export TEXTDOMAINDIR="@localedir@"
+
+CLASS="--class gnu-linux --class gnu --class os --class xen"
+SUPPORTED_INITS="sysvinit:/lib/sysvinit/init systemd:/lib/systemd/systemd upstart:/sbin/upstart"
+
+if [ "x${GRUB_DISTRIBUTOR}" = "x" ] ; then
+  OS=GNU/Linux
+else
+  OS="${GRUB_DISTRIBUTOR} GNU/Linux"
+  CLASS="--class $(echo ${GRUB_DISTRIBUTOR} | tr 'A-Z' 'a-z' | cut -d' ' -f1|LC_ALL=C sed 's,[^[:alnum:]_],_,g') ${CLASS}"
+fi
+
+# loop-AES arranges things so that /dev/loop/X can be our root device, but
+# the initrds that Linux uses don't like that.
+case ${GRUB_DEVICE} in
+  /dev/loop/*|/dev/loop[0-9])
+    GRUB_DEVICE=`losetup ${GRUB_DEVICE} | sed -e "s/^[^(]*(\([^)]\+\)).*/\1/"`
+    # We can't cope with devices loop-mounted from files here.
+    case ${GRUB_DEVICE} in
+      /dev/*) ;;
+      *) exit 0 ;;
+    esac
+  ;;
+esac
+
+# btrfs may reside on multiple devices. We cannot pass them as value of root= parameter
+# and mounting btrfs requires user space scanning, so force UUID in this case.
+if [ "x${GRUB_DEVICE_UUID}" = "x" ] || [ "x${GRUB_DISABLE_LINUX_UUID}" = "xtrue" ] \
+    || ! test -e "/dev/disk/by-uuid/${GRUB_DEVICE_UUID}" \
+    || ( test -e "${GRUB_DEVICE}" && uses_abstraction "${GRUB_DEVICE}" lvm ); then
+  LINUX_ROOT_DEVICE=${GRUB_DEVICE}
+else
+  LINUX_ROOT_DEVICE=UUID=${GRUB_DEVICE_UUID}
+fi
+
+# Allow overriding GRUB_CMDLINE_LINUX and GRUB_CMDLINE_LINUX_DEFAULT.
+if [ "${GRUB_CMDLINE_LINUX_XEN_REPLACE}" ]; then
+  GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX_XEN_REPLACE}"
+fi
+if [ "${GRUB_CMDLINE_LINUX_XEN_REPLACE_DEFAULT}" ]; then
+  GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_XEN_REPLACE_DEFAULT}"
+fi
+
+case x"$GRUB_FS" in
+    xbtrfs)
+	rootsubvol="`make_system_path_relative_to_its_root /`"
+	rootsubvol="${rootsubvol#/}"
+	if [ "x${rootsubvol}" != x ]; then
+	    GRUB_CMDLINE_LINUX="rootflags=subvol=${rootsubvol} ${GRUB_CMDLINE_LINUX}"
+	fi;;
+    xzfs)
+	rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
+	bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
+	LINUX_ROOT_DEVICE="ZFS=${rpool}${bootfs%/}"
+	;;
+esac
+
+title_correction_code=
+
+linux_entry ()
+{
+  os="$1"
+  version="$2"
+  xen_version="$3"
+  type="$4"
+  args="$5"
+  xen_args="$6"
+  if [ -z "$boot_device_id" ]; then
+      boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")"
+  fi
+  if [ x$type != xsimple ] ; then
+      if [ x$type = xrecovery ] ; then
+	  title="$(gettext_printf "%s, with Xen %s and Linux %s (%s)" "${os}" "${xen_version}" "${version}" "$(gettext "${GRUB_RECOVERY_TITLE}")")"
+      elif [ "${type#init-}" != "$type" ] ; then
+	  title="$(gettext_printf "%s, with Xen %s and Linux %s (%s)" "${os}" "${xen_version}" "${version}" "${type#init-}")"
+      else
+	  title="$(gettext_printf "%s, with Xen %s and Linux %s" "${os}" "${xen_version}" "${version}")"
+      fi
+      replacement_title="$(echo "Advanced options for ${OS}" | sed 's,>,>>,g')>$(echo "$title" | sed 's,>,>>,g')"
+      if [ x"Xen ${xen_version}>$title" = x"$GRUB_ACTUAL_DEFAULT" ]; then
+         quoted="$(echo "$GRUB_ACTUAL_DEFAULT" | grub_quote)"
+         title_correction_code="${title_correction_code}if [ \"x\$default\" = '$quoted' ]; then default='$(echo "$replacement_title" | grub_quote)'; fi;"
+         grub_warn "$(gettext_printf "Please don't use old title \`%s' for GRUB_DEFAULT, use \`%s' (for versions before 2.00) or \`%s' (for 2.00 or later)" "$GRUB_ACTUAL_DEFAULT" "$replacement_title" "gnulinux-advanced-$boot_device_id>gnulinux-$version-$type-$boot_device_id")"
+      fi
+      echo "menuentry '$(echo "$title" | grub_quote)' ${CLASS} \$menuentry_id_option 'xen-gnulinux-$version-$type-$boot_device_id' {" | sed "s/^/$submenu_indentation/"
+  else
+      title="$(gettext_printf "%s, with Xen hypervisor" "${os}")"
+      echo "menuentry '$(echo "$title" | grub_quote)' ${CLASS} \$menuentry_id_option 'xen-gnulinux-simple-$boot_device_id' {" | sed "s/^/$submenu_indentation/"
+  fi
+  if [ x$type != xrecovery ] ; then
+      save_default_entry | grub_add_tab | sed "s/^/$submenu_indentation/"
+  fi
+
+  if [ -z "${prepare_boot_cache}" ]; then
+    prepare_boot_cache="$(prepare_grub_to_access_device ${GRUB_DEVICE_BOOT} | grub_add_tab)"
+  fi
+  printf '%s\n' "${prepare_boot_cache}" | sed "s/^/$submenu_indentation/"
+  xmessage="$(gettext_printf "Loading Xen %s ..." ${xen_version})"
+  lmessage="$(gettext_printf "Loading Linux %s ..." ${version})"
+  sed "s/^/$submenu_indentation/" << EOF
+	echo	'$(echo "$xmessage" | grub_quote)'
+        if [ "\$grub_platform" = "pc" -o "\$grub_platform" = "" ]; then
+            xen_rm_opts=
+        else
+            xen_rm_opts="no-real-mode edd=off"
+        fi
+	${xen_loader}	${rel_xen_dirname}/${xen_basename} placeholder ${xen_args} \${xen_rm_opts}
+	echo	'$(echo "$lmessage" | grub_quote)'
+	${module_loader}	${rel_dirname}/${basename} placeholder root=${linux_root_device_thisversion} ro ${args}
+EOF
+  if test -n "${initrd}" ; then
+    # TRANSLATORS: ramdisk isn't identifier. Should be translated.
+    message="$(gettext_printf "Loading initial ramdisk ...")"
+    sed "s/^/$submenu_indentation/" << EOF
+	echo	'$(echo "$message" | grub_quote)'
+	${module_loader}	--nounzip   ${rel_dirname}/${initrd}
+EOF
+  fi
+  sed "s/^/$submenu_indentation/" << EOF
+}
+EOF
+}
+
+linux_list=
+for i in /boot/vmlinu[xz]-* /vmlinu[xz]-* /boot/kernel-*; do
+    if grub_file_is_not_garbage "$i"; then
+    	basename=$(basename $i)
+	version=$(echo $basename | sed -e "s,^[^0-9]*-,,g")
+	dirname=$(dirname $i)
+	config=
+	for j in "${dirname}/config-${version}" "${dirname}/config-${alt_version}" "/etc/kernels/kernel-config-${version}" ; do
+	    if test -e "${j}" ; then
+		config="${j}"
+		break
+	    fi
+	done
+        if (grep -qx "CONFIG_XEN_DOM0=y" "${config}" 2> /dev/null || grep -qx "CONFIG_XEN_PRIVILEGED_GUEST=y" "${config}" 2> /dev/null); then linux_list="$linux_list $i" ; fi
+    fi
+done
+if [ "x${linux_list}" = "x" ] ; then
+    exit 0
+fi
+
+file_is_not_sym () {
+    case "$1" in
+	*/xen-syms-*)
+	    return 1;;
+	*)
+	    return 0;;
+    esac
+}
+
+xen_list=
+for i in /boot/xen*; do
+    if grub_file_is_not_garbage "$i" && file_is_not_sym "$i" ; then xen_list="$xen_list $i" ; fi
+done
+prepare_boot_cache=
+boot_device_id=
+
+title_correction_code=
+
+machine=`uname -m`
+
+case "$machine" in
+    i?86) GENKERNEL_ARCH="x86" ;;
+    mips|mips64) GENKERNEL_ARCH="mips" ;;
+    mipsel|mips64el) GENKERNEL_ARCH="mipsel" ;;
+    arm*) GENKERNEL_ARCH="arm" ;;
+    *) GENKERNEL_ARCH="$machine" ;;
+esac
+
+# Extra indentation to add to menu entries in a submenu. We're not in a submenu
+# yet, so it's empty. In a submenu it will be equal to '\t' (one tab).
+submenu_indentation=""
+
+is_top_level=true
+
+while [ "x${xen_list}" != "x" ] ; do
+    list="${linux_list}"
+    current_xen=`version_find_latest $xen_list`
+    xen_basename=`basename ${current_xen}`
+    xen_dirname=`dirname ${current_xen}`
+    rel_xen_dirname=`make_system_path_relative_to_its_root $xen_dirname`
+    xen_version=`echo $xen_basename | sed -e "s,.gz$,,g;s,^xen-,,g"`
+    if [ -z "$boot_device_id" ]; then
+	boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")"
+    fi
+    if [ "x$is_top_level" != xtrue ]; then
+	echo "	submenu '$(gettext_printf "Xen hypervisor, version %s" "${xen_version}" | grub_quote)' \$menuentry_id_option 'xen-hypervisor-$xen_version-$boot_device_id' {"
+    fi
+    if ($grub_file --is-x86-multiboot2 $current_xen); then
+	xen_loader="multiboot2"
+	module_loader="module2"
+    else
+	xen_loader="multiboot"
+	module_loader="module"
+    fi
+    while [ "x$list" != "x" ] ; do
+	linux=`version_find_latest $list`
+	gettext_printf "Found linux image: %s\n" "$linux" >&2
+	basename=`basename $linux`
+	dirname=`dirname $linux`
+	rel_dirname=`make_system_path_relative_to_its_root $dirname`
+	version=`echo $basename | sed -e "s,^[^0-9]*-,,g"`
+	alt_version=`echo $version | sed -e "s,\.old$,,g"`
+	linux_root_device_thisversion="${LINUX_ROOT_DEVICE}"
+
+	initrd=
+	for i in "initrd.img-${version}" "initrd-${version}.img" "initrd-${version}.gz" \
+	   "initrd-${version}" "initramfs-${version}.img" \
+	   "initrd.img-${alt_version}" "initrd-${alt_version}.img" \
+	   "initrd-${alt_version}" "initramfs-${alt_version}.img" \
+	   "initramfs-genkernel-${version}" \
+	   "initramfs-genkernel-${alt_version}" \
+	   "initramfs-genkernel-${GENKERNEL_ARCH}-${version}" \
+	   "initramfs-genkernel-${GENKERNEL_ARCH}-${alt_version}" ; do
+	    if test -e "${dirname}/${i}" ; then
+		initrd="$i"
+		break
+	    fi
+	done
+	if test -n "${initrd}" ; then
+	    gettext_printf "Found initrd image: %s\n" "${dirname}/${initrd}" >&2
+	else
+    # "UUID=" magic is parsed by initrds.  Since there's no initrd, it can't work here.
+	    linux_root_device_thisversion=${GRUB_DEVICE}
+	fi
+
+	if [ "x$is_top_level" = xtrue ] && [ "x${GRUB_DISABLE_SUBMENU}" != xy ]; then
+	    linux_entry "${OS}" "${version}" "${xen_version}" simple \
+		"${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}" "${GRUB_CMDLINE_XEN} ${GRUB_CMDLINE_XEN_DEFAULT}"
+
+	    submenu_indentation="$grub_tab$grub_tab"
+    
+	    if [ -z "$boot_device_id" ]; then
+		boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")"
+	    fi
+            # TRANSLATORS: %s is replaced with an OS name
+	    echo "submenu '$(gettext_printf "Advanced options for %s (with Xen hypervisor)" "${OS}" | grub_quote)' \$menuentry_id_option 'gnulinux-advanced-$boot_device_id' {"
+	echo "	submenu '$(gettext_printf "Xen hypervisor, version %s" "${xen_version}" | grub_quote)' \$menuentry_id_option 'xen-hypervisor-$xen_version-$boot_device_id' {"
+	   is_top_level=false
+	fi
+
+	linux_entry "${OS}" "${version}" "${xen_version}" advanced \
+	    "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}" "${GRUB_CMDLINE_XEN} ${GRUB_CMDLINE_XEN_DEFAULT}"
+	for supported_init in ${SUPPORTED_INITS}; do
+	    init_path="${supported_init#*:}"
+	    if [ -x "${init_path}" ] && [ "$(readlink -f /sbin/init)" != "$(readlink -f "${init_path}")" ]; then
+		linux_entry "${OS}" "${version}" "${xen_version}" "init-${supported_init%%:*}" \
+		    "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT} init=${init_path}" "${GRUB_CMDLINE_XEN} ${GRUB_CMDLINE_XEN_DEFAULT}"
+
+	    fi
+	done
+	if [ "x${GRUB_DISABLE_RECOVERY}" != "xtrue" ]; then
+	    linux_entry "${OS}" "${version}" "${xen_version}" recovery \
+		"single ${GRUB_CMDLINE_LINUX}" "${GRUB_CMDLINE_XEN}"
+	fi
+
+	list=`echo $list | tr ' ' '\n' | fgrep -vx "$linux" | tr '\n' ' '`
+    done
+    if [ x"$is_top_level" != xtrue ]; then
+	echo '	}'
+    fi
+    xen_list=`echo $xen_list | tr ' ' '\n' | fgrep -vx "$current_xen" | tr '\n' ' '`
+done
+
+# If at least one kernel was found, then we need to
+# add a closing '}' for the submenu command.
+if [ x"$is_top_level" != xtrue ]; then
+  echo '}'
+fi
+
+echo "$title_correction_code"
Signature · 1 line
2.20.1
#36 lore raw

PATCH 26/38 20_linux_xen: Adhoc template substitution

In reply to Ian Jackson

This file is a template that various build-time variables get substituted into. Make thos substitutions by hand (actually, by copying the values our file for stretch). And rename the file.

So now we are using our file instead of the grub package's. But it is the same...

---
 .../etc/grub.d/{20_linux_xen.in => 20_linux_xen}       | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
 rename overlay-buster/etc/grub.d/{20_linux_xen.in => 20_linux_xen} (98%)
 mode change 100644 => 100755
overlay-buster/etc/grub.d/20_linux_xen renamed from overlay-buster/etc/grub.d/20_linux_xen.in +5 -5
old mode 100644
new mode 100755
similarity index 98%
rename from overlay-buster/etc/grub.d/20_linux_xen.in
rename to overlay-buster/etc/grub.d/20_linux_xen
index 98ef163c..fb3ed82f
@@ -17,14 +17,14 @@ set -e
 # You should have received a copy of the GNU General Public License
 # along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
 
-prefix="@prefix@"
-exec_prefix="@exec_prefix@"
-datarootdir="@datarootdir@"
+prefix="/usr"
+exec_prefix="/usr"
+datarootdir="/usr/share"
 
 . "$pkgdatadir/grub-mkconfig_lib"
 
-export TEXTDOMAIN=@PACKAGE@
-export TEXTDOMAINDIR="@localedir@"
+export TEXTDOMAIN=grub
+export TEXTDOMAINDIR="${datarootdir}/locale"
 
 CLASS="--class gnu-linux --class gnu --class os --class xen"
 SUPPORTED_INITS="sysvinit:/lib/sysvinit/init systemd:/lib/systemd/systemd upstart:/sbin/upstart"
Signature · 1 line
2.20.1
#37 lore raw

PATCH 27/38 20_linux_xen: Ignore xenpolicy and config files too

In reply to Ian Jackson

"file_is_not_sym" currently only checks for xen-syms. Extend it to disregard xenpolicy (XSM policy files) and files ending .config (which are built by the Xen upstream build system in some configurations and can therefore end up in /boot).

Rename the function accordingly, to "file_is_not_xen_garbage".

---
 overlay-buster/etc/grub.d/20_linux_xen | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
overlay-buster/etc/grub.d/20_linux_xen +6 -2
index fb3ed82f..01dfcb57 100755
@@ -167,10 +167,14 @@ if [ "x${linux_list}" = "x" ] ; then
     exit 0
 fi
 
-file_is_not_sym () {
+file_is_not_xen_garbage () {
     case "$1" in
 	*/xen-syms-*)
 	    return 1;;
+	*/xenpolicy-*)
+	    return 1;;
+	*/*.config)
+	    return 1;;
 	*)
 	    return 0;;
     esac
@@ -178,7 +182,7 @@ file_is_not_sym () {
 
 xen_list=
 for i in /boot/xen*; do
-    if grub_file_is_not_garbage "$i" && file_is_not_sym "$i" ; then xen_list="$xen_list $i" ; fi
+    if grub_file_is_not_garbage "$i" && file_is_not_xen_garbage "$i" ; then xen_list="$xen_list $i" ; fi
 done
 prepare_boot_cache=
 boot_device_id=
Signature · 1 line
2.20.1
#38 lore raw

PATCH 28/38 20_linux_xen: Support Xen Security Modules (XSM/FLASK)

In reply to Ian Jackson

XSM is enabled by adding "flask=enforcing" as a Xen command line argument, and providing the policy file as a grub module.

We make entries for both with and without XSM. If XSM is not compiled into Xen, then there are no policy files, so no change to the boot options.

---
 overlay-buster/etc/grub.d/20_linux_xen | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
overlay-buster/etc/grub.d/20_linux_xen +24 -0
index 01dfcb57..4d3294a2 100755
@@ -84,6 +84,11 @@ esac
 title_correction_code=
 
 linux_entry ()
+{
+  linux_entry_xsm "$@" false
+  linux_entry_xsm "$@" true
+}
+linux_entry_xsm ()
 {
   os="$1"
   version="$2"
@@ -91,6 +96,18 @@ linux_entry ()
   type="$4"
   args="$5"
   xen_args="$6"
+  xsm="$7"
+  # If user wants to enable XSM support, make sure there's
+  # corresponding policy file.
+  if ${xsm} ; then
+      xenpolicy="xenpolicy-$xen_version"
+      if test ! -e "${xen_dirname}/${xenpolicy}" ; then
+	  return
+      fi
+      xen_args="$xen_args flask=enforcing"
+      xen_version="$(gettext_printf "%s (XSM enabled)" "$xen_version")"
+      # xen_version is used for messages only; actual file is xen_basename
+  fi
   if [ -z "$boot_device_id" ]; then
       boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")"
   fi
@@ -140,6 +157,13 @@ EOF
     sed "s/^/$submenu_indentation/" << EOF
 	echo	'$(echo "$message" | grub_quote)'
 	${module_loader}	--nounzip   ${rel_dirname}/${initrd}
+EOF
+  fi
+  if test -n "${xenpolicy}" ; then
+    message="$(gettext_printf "Loading XSM policy ...")"
+    sed "s/^/$submenu_indentation/" << EOF
+	echo	'$(echo "$message" | grub_quote)'
+	${module_loader}     ${rel_dirname}/${xenpolicy}
 EOF
   fi
   sed "s/^/$submenu_indentation/" << EOF
Signature · 1 line
2.20.1
#39 lore raw

PATCH 29/38 mg-debian-installer-update: support overlay-intramfs-SUITE

In reply to Ian Jackson

This lets us patch the installer more easily.

No uses yet.

---
 mg-debian-installer-update | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
mg-debian-installer-update +20 -0
index f1e682f9..fb4fe2ab 100755
@@ -33,6 +33,8 @@ sbase=$site/dists/$suite
 
 src=$sbase/main/installer-$arch/current/images/netboot/
 
+osstest_dir="$(pwd)"
+
 case ${suite}_${arch} in
     lenny_armhf|squeeze_armhf|lenny_arm64|squeeze_arm64|wheezy_arm64)
         # No such thing.
@@ -188,6 +190,24 @@ if [ "x$specialkernel" != x ]; then
     rm -rf x
 fi
 
+overlay_initrd=$osstest_dir/overlay-initrd-$suite
+if [ -e "$overlay_initrd" ]; then
+    for f in $files; do
+        s=${f/:*} ; d=${f/*:}
+        case "$d" in
+            *initrd*)
+                echo "adding $overlay_initrd to $d"
+                (
+                    set -e
+                    cd "$overlay_initrd"
+                    find -print0 | cpio -0 -Hnewc -o \
+                        | gzip -9nf
+                ) >>$d.new
+                ;;
+        esac
+    done
+fi
+
 for f in $files; do
         s=${f/:*} ; d=${f/*:}
         mv -f $d.new $d
Signature · 1 line
2.20.1
#40 lore raw

PATCH 30/38 overlay-initrd-buster/sbin/reopen-console: Copy from Debian

In reply to Ian Jackson

We are going to patch this file to work around a bug, using the new overlay mechanism.

The first step is to include the file in our overlay so we overwrite it. Currently, this is a no-op, so no functional change.

---
 overlay-initrd-buster/sbin/reopen-console | 94 +++++++++++++++++++++++
 1 file changed, 94 insertions(+)
 create mode 100755 overlay-initrd-buster/sbin/reopen-console
overlay-initrd-buster/sbin/reopen-console new file +94 -0
new file mode 100755
index 00000000..dd354deb
@@ -0,0 +1,94 @@
+#!/bin/sh
+
+# First find the enabled consoles from the kernel, noting if one is 'preferred'
+# Record these.
+# Run the startup scripts on the preferred console
+
+# In order to have D-I appear on all consoles, modify the inittab to
+# add one entry for each console, running debian-installer.
+# Finally HUP init so that it runs those installers
+# (but doesn't rerun the sysinit startup stuff, including this script)
+
+
+NL="
+"
+
+LOGGER_UP=0
+LOG_FILE=/var/log/reopen-console
+
+log() {
+	# In very early startup we don't have syslog. Log to file that
+	# we can flush out later so we can at least see what happened
+	# at early startup
+	if [ $LOGGER_UP -eq 1 ]; then
+	        logger -t reopen-console "$@"
+	else
+		echo "$@" >> $LOG_FILE
+	fi
+}
+
+flush_logger () {
+	cat $LOG_FILE | logger -t reopen-console
+	rm $LOG_FILE
+}
+
+consoles=
+preferred=
+# Retrieve all enabled consoles from kernel; ignore those
+# for which no device file exists
+
+kernelconsoles="$(cat /proc/consoles)"
+for cons in $(echo "$kernelconsoles" | sed -n -r -e 's/(^.*)  .*\((.*)\).*$/\1/p' )
+do
+	log "Looking at console $cons from /proc/consoles"
+	status=$(echo "$kernelconsoles" | grep $cons | sed -n -r -e 's/(^.*) *.*\((.*)\).*$/\2/p' )
+	if [ -e "/dev/$cons" ] && [ $(echo "$status" | grep -o 'E') ]; then
+		consoles="${consoles:+$consoles$NL}$cons"
+		log "   Adding $cons to consoles list"
+	fi
+	# 'C' console is 'most prefered'.
+	if [ $(echo "$status" | grep -o 'C') ]; then
+		preferred="$cons"
+		log "   $cons is preferred"
+	fi
+done
+
+if [ -z "$consoles" ]; then
+	# Nothing found? Default to /dev/console.
+	log "Found no consoles! Defaulting to /dev/console"
+	consoles=console
+fi
+if [ -z "$preferred" ]; then
+	#None marked preferred? Use the first one
+	preferred=$(echo "$consoles" | head -n 1)
+	log "Found no preferred console. Picking $preferred"
+fi
+
+for cons in $consoles
+do
+	echo "/dev/$cons " >> /var/run/console-devices
+done
+echo "/dev/$preferred " > /var/run/console-preferred
+
+
+# Add debian-installer lines into inittab - one per console
+for cons in $consoles
+do
+	log "Adding inittab entry for $cons"
+	echo "$cons::respawn:/sbin/debian-installer" >> /etc/inittab
+done
+
+# Run the startup scripts once, using the preferred console
+cons=$(cat /var/run/console-preferred)
+# Some other session may have that console as ctty. Steal it from them
+/sbin/steal-ctty $cons "$@"
+
+# Now we should have syslog running, so flush our log entries
+LOGGER_UP=1
+flush_logger
+
+# Finally restart init to run debian-installer on discovered consoles
+log "Restarting init to start d-i on the consoles we found"
+kill -HUP 1
+
+exit 0
Signature · 1 line
2.20.1
#41 lore raw

PATCH 31/38 overlay-initrd-buster/sbin/reopen-console: Fix #932416

In reply to Ian Jackson

This bug affects us. Cherry pick the changes to the relevant file from the commit in the upstream debian-installer repo:

https://salsa.debian.org/installer-team/rootskel/commit/0ee43d05b83f8ef5a856f3282e002a111809cef9

---
 overlay-initrd-buster/sbin/reopen-console | 36 +++++++++++++++++++++--
 1 file changed, 34 insertions(+), 2 deletions(-)
overlay-initrd-buster/sbin/reopen-console +34 -2
index dd354deb..13b15a33 100755
@@ -16,6 +16,17 @@ NL="
 LOGGER_UP=0
 LOG_FILE=/var/log/reopen-console
 
+# If we're running with preseeding, we have a problem with running d-i
+# on multiple consoles. We'll end up running each of those d-i
+# instances in parallel with all kinds of hilarious undefined
+# behaviour as they trip over each other! If we detect that we're
+# preseeding (via any of the possible preseed methods), DO NOT run d-i
+# multiple times. Instead, fall back to the older, more simple
+# behaviour and run it once. If the user wants to see or interact with
+# their preseed on a specific console, they get to tell us which one
+# they want to use.
+PRESEEDING=0
+
 log() {
 	# In very early startup we don't have syslog. Log to file that
 	# we can flush out later so we can at least see what happened
@@ -32,6 +43,20 @@ flush_logger () {
 	rm $LOG_FILE
 }
 
+# If we have a preseed.cfg in the initramfs
+if [ -e /preseed.cfg ]; then
+    log "Found /preseed.cfg; falling back to simple mode for preseeding"
+    PRESEEDING=1
+fi
+
+# Have we been told to do preseeding stuff on the boot command line?
+for WORD in auto url; do
+    if (grep -qw "$WORD" /proc/cmdline); then
+	log "Found \"$WORD\" in the command line; falling back to simple mode for preseeding"
+	PRESEEDING=1
+    fi
+done
+
 consoles=
 preferred=
 # Retrieve all enabled consoles from kernel; ignore those
@@ -44,7 +69,7 @@ do
 	status=$(echo "$kernelconsoles" | grep $cons | sed -n -r -e 's/(^.*) *.*\((.*)\).*$/\2/p' )
 	if [ -e "/dev/$cons" ] && [ $(echo "$status" | grep -o 'E') ]; then
 		consoles="${consoles:+$consoles$NL}$cons"
-		log "   Adding $cons to consoles list"
+		log "   Adding $cons to possible consoles list"
 	fi
 	# 'C' console is 'most prefered'.
 	if [ $(echo "$status" | grep -o 'C') ]; then
@@ -64,6 +89,13 @@ if [ -z "$preferred" ]; then
 	log "Found no preferred console. Picking $preferred"
 fi
 
+# If we're preseeding, do simple stuff here (see above). We just
+# want one console. Let's pick the preferred one ONLY
+if [ $PRESEEDING = 1 ]; then
+    log "Running with preseeding. Picking preferred $preferred ONLY"
+    consoles=$preferred
+fi
+
 for cons in $consoles
 do
 	echo "/dev/$cons " >> /var/run/console-devices
@@ -88,7 +120,7 @@ LOGGER_UP=1
 flush_logger
 
 # Finally restart init to run debian-installer on discovered consoles
-log "Restarting init to start d-i on the consoles we found"
+log "Restarting init to start d-i on the console(s) we found"
 kill -HUP 1
 
 exit 0
Signature · 1 line
2.20.1
#42 lore raw

PATCH 32/38 buster: chiark-scripts: Install a new version on buster too

In reply to Ian Jackson

We need various fixes that are not in buster, sadly.

---
 production-config | 1 +
 1 file changed, 1 insertion(+)
production-config +1 -0
index f0ddc132..e3870d47 100644
@@ -107,6 +107,7 @@ TftpGrubVersion XXXX-XX-XX
 
 DebianExtraPackages_jessie chiark-scripts_6.0.3~citrix1_all.deb
 DebianExtraPackages_stretch chiark-scripts_6.0.4~citrix1_all.deb
+DebianExtraPackages_buster chiark-scripts_6.0.5~citrix1_all.deb
 
 DebianExtraPackages_uefi_i386_jessie   extradebs-uefi-i386-2018-04-01/
 DebianExtraPackages_uefi_amd64_jessie  extradebs-uefi-amd64-2018-04-01/
Signature · 1 line
2.20.1
#43 lore raw

PATCH 33/38 buster: Provide TftpDiVersion

In reply to Ian Jackson
---
 production-config | 1 +
 1 file changed, 1 insertion(+)
production-config +1 -0
index e3870d47..6372ac9a 100644
@@ -91,6 +91,7 @@ TftpNetbootGroup osstest
 TftpDiVersion_wheezy 2016-06-08
 TftpDiVersion_jessie 2018-06-26
 TftpDiVersion_stretch 2020-02-10
+TftpDiVersion_buster 2020-05-19
 
 DebianSnapshotBackports_jessie http://snapshot.debian.org/archive/debian/20190206T211314Z/
 
Signature · 1 line
2.20.1
#44 lore raw

PATCH 34/38 buster: grub, arm64: extend chainloading workaround

In reply to Ian Jackson

multiboot[2] isn't supported.

Also link to the bug report.

CC Julien Grall <[email protected]>
CC Stefano Stabellini <[email protected]>
---
 Osstest/Debian.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Osstest/Debian.pm +2 -1
index 77508d19..151677ed 100644
@@ -443,9 +443,10 @@ sub setupboot_grub2 ($$$$) {
     my $kernkey= (defined $xenhopt ? 'KernDom0' : 'KernOnly');
 
     # Grub2 on jessie/stretch ARM* doesn't do multiboot, so we must chainload.
+    # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=884770
     my $need_uefi_chainload =
         get_host_property($ho, "firmware") eq "uefi" &&
-        $ho->{Suite} =~ m/jessie|stretch/ && $ho->{Arch} =~ m/^arm/;
+        $ho->{Suite} =~ m/jessie|stretch|buster/ && $ho->{Arch} =~ m/^arm/;
 
     my $parsemenu= sub {
         my $f= bl_getmenu_open($ho, $rmenu, "$stash/$ho->{Name}--grub.cfg.1");
Signature · 1 line
2.20.1
Acked-by on v1 34/38
#45 lore raw

Re: [OSSTEST PATCH 34/38] buster: grub, arm64: extend chainloading workaround

Hi,

Quoted 7 lines · Ian Jackson

multiboot[2] isn't supported.

Also link to the bug report.

CC Julien Grall <[email protected]>
CC Stefano Stabellini <[email protected]>
Quoted 9 lines
---
  Osstest/Debian.pm | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
Osstest/Debian.pm +0 -0
index 77508d19..151677ed 100644

Osstest/Debian.pm sub setupboot_grub2 ($$$$) { · line 449

Quoted 8 lines
Osstest/Debian.pm +2 -1
@@ -443,9 +443,10 @@ sub setupboot_grub2 ($$$$) {
      my $kernkey= (defined $xenhopt ? 'KernDom0' : 'KernOnly');
 
      # Grub2 on jessie/stretch ARM* doesn't do multiboot, so we must chainload.
+    # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=884770
      my $need_uefi_chainload =
          get_host_property($ho, "firmware") eq "uefi" &&
-        $ho->{Suite} =~ m/jessie|stretch/ && $ho->{Arch} =~ m/^arm/;
+        $ho->{Suite} =~ m/jessie|stretch|buster/ && $ho->{Arch} =~ m/^arm/;

FWIW, the next version of Debian seems to have a newer GRUB version with Xen on Arm support.

Cheers,

Signature · 1 line
Julien Grall
#46 lore raw

Re: [OSSTEST PATCH 34/38] buster: grub, arm64: extend chainloading workaround

Julien Grall writes ("Re: [OSSTEST PATCH 34/38] buster: grub, arm64: extend chainloading workaround"):

Quoted 22 lines · Ian Jackson
Quoted 7 lines

multiboot[2] isn't supported.

Also link to the bug report.

CC Julien Grall <[email protected]>
CC Stefano Stabellini <[email protected]>
Quoted 11 lines
---
  Osstest/Debian.pm | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
Osstest/Debian.pm +0 -0
index 77508d19..151677ed 100644
@@ -443,9 +443,10 @@ sub setupboot_grub2 ($$$$) {
      my $kernkey= (defined $xenhopt ? 'KernDom0' : 'KernOnly');
 
Quoted 9 lines

Osstest/Debian.pm sub setupboot_grub2 ($$$$) { · line 449

Quoted 6 lines
Osstest/Debian.pm +2 -1
      # Grub2 on jessie/stretch ARM* doesn't do multiboot, so we must chainload.
+    # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=884770
      my $need_uefi_chainload =
          get_host_property($ho, "firmware") eq "uefi" &&
-        $ho->{Suite} =~ m/jessie|stretch/ && $ho->{Arch} =~ m/^arm/;
+        $ho->{Suite} =~ m/jessie|stretch|buster/ && $ho->{Arch} =~ m/^arm/;

FWIW, the next version of Debian seems to have a newer GRUB version with Xen on Arm support.

Cool, we can drop this eventually then :-).

Thanks,
Ian.

#47 lore raw

PATCH 35/38 buster: setupboot_grub2: Note what files exist in /boot

In reply to Ian Jackson

Nothing uses this yet.

---
 Osstest/Debian.pm | 5 +++++
 1 file changed, 5 insertions(+)
Osstest/Debian.pm +5 -0
index 151677ed..c0b669c9 100644
@@ -448,6 +448,11 @@ sub setupboot_grub2 ($$$$) {
         get_host_property($ho, "firmware") eq "uefi" &&
         $ho->{Suite} =~ m/jessie|stretch|buster/ && $ho->{Arch} =~ m/^arm/;
 
+    my %bootfiles =
+	map { $_ => 1 }
+	split / /,
+	target_cmd_output_root($ho, "cd /boot && echo *");
+
     my $parsemenu= sub {
         my $f= bl_getmenu_open($ho, $rmenu, "$stash/$ho->{Name}--grub.cfg.1");
     
Signature · 1 line
2.20.1
#48 lore raw

PATCH 36/38 buster: setupboot_grub2: Handle missing policy file bug

In reply to Ian Jackson

This is a complex interaction between update-grub and the Xen build system on ARM64. Not sure exactly who to blame but since we have our own 20_linux_xen bodge, let's wait until we don't.

---
 Osstest/Debian.pm | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
Osstest/Debian.pm +11 -1
index c0b669c9..6c289cc7 100644
@@ -496,7 +496,17 @@ sub setupboot_grub2 ($$$$) {
 			 " kernel $entry->{KernVer}, not $want_kernver)");
 		} elsif ($want_xsm && !defined $entry->{Xenpolicy}) {
 		    logm("(skipping entry at $entry->{StartLine}..$.;".
-			 " XSM policy file not present)");
+			 " XSM policy file not mentioned)");
+		} elsif ($ho->{Suite} =~ m/buster/ &&
+			 defined $entry->{Xenpolicy} &&
+			 !$bootfiles{
+                             $entry->{Xenpolicy} =~ m{^/?} ? $' : die
+						 }) {
+		    # Our 20_linux_xen bodge with buster's update-grub
+		    # generates entries which mention /boot/xenpolicy-xen
+		    # even though that file doesn't exist on ARM64.
+		    logm("(skipping entry at $entry->{StartLine}..$.;".
+			 " XSM policy file not on disk!)");
 		} else {
 		    # yes!
 		    last;
Signature · 1 line
2.20.1
#49 lore raw

PATCH 37/38 buster: Extend workaround for dhcpd EROFS bug

In reply to Ian Jackson
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest/Debian.pm +1 -1
index 6c289cc7..e1ce757e 100644
@@ -1617,7 +1617,7 @@ sub debian_dhcp_rofs_fix ($$) {
     # / is still ro.  In stretch, the isc dhcp client spins requesting
     # an address and then sending a DHCPDECLINE (and then, usually,
     # eventually works).
-    return '' unless $ho->{Suite} =~ m/stretch/;
+    return '' unless $ho->{Suite} =~ m/stretch|buster/;
     my $script = "$rootdir/lib/udev/ifupdown-hotplug";
     <<END.<<'ENDQ'.<<END
 set -ex
Signature · 1 line
2.20.1
#50 lore raw

PATCH 38/38 buster: Switch to Debian buster as the default suite

In reply to Ian Jackson
---
 Osstest.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Osstest.pm +1 -1
index 1e381d8f..63dddd95 100644
@@ -87,7 +87,7 @@ our %c = qw(
 
     Images images
 
-    DebianSuite stretch
+    DebianSuite buster
     DebianMirrorSubpath debian
 
     TestHostKeypairPath id_rsa_osstest
Signature · 1 line
2.20.1